Understanding Linux behaviour

Hello!
I’m playing with multiple network configuration, and I came across a strange behaviour from Linux machines.

This is the RouterOS configuration I’m testing with:

/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment=defconf fast-forward=no name=bridge
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add add-arp=yes address-pool=default-dhcp interface=bridge lease-time=1m name=defconf
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=sfp-sfpplus1
/interface bridge settings
set allow-fast-path=no use-ip-firewall=yes
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/interface ovpn-server server
add mac-address=FE:FA:C6:8B:DB:8E name=ovpn-server1
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
add address=192.168.20.254/24 interface=bridge network=192.168.20.0
add address=192.168.10.254/24 interface=bridge network=192.168.10.0
add address=192.168.30.254/24 interface=bridge network=192.168.30.0
add address=192.168.40.254/24 interface=bridge network=192.168.40.0
/ip dhcp-client
add comment=defconf interface=ether1
/ip dhcp-server lease
add address=192.168.30.10 comment=SERVER mac-address=yy:yy:yy:yy:yy:yy server=defconf
add address=192.168.20.10 comment=PRINTER mac-address=zz:zz:zz:zz:zz:zz server=defconf
add address=192.168.40.30 comment=CLIENT mac-address=jj:jj:jj:jj:jj:jj server=defconf
add address=192.168.40.10 comment=GUEST mac-address=qq:qq:qq:qq:qq:qq server=defconf
add address=192.168.40.20 comment=GUEST mac-address=mm:mm:mm:mm:mm:mm server=defconf
/ip dhcp-server network
add address=192.168.10.0/24 dns-server=192.168.88.1 domain=lan.local gateway=192.168.10.254 netmask=24
add address=192.168.20.0/24 dns-server=192.168.88.1 domain=lan.local gateway=192.168.20.254 netmask=24
add address=192.168.30.0/24 dns-server=192.168.88.1 domain=lan.local gateway=192.168.30.254 netmask=24
add address=192.168.40.0/24 dns-server=192.168.88.1 domain=lan.local gateway=192.168.40.254 netmask=24
add address=192.168.88.0/24 dns-server=192.168.88.1 domain=lan.local gateway=192.168.88.1 netmask=24
/ip dns
set allow-remote-requests=yes
/ip firewall address-list
add address=192.168.10.0/24 list=LAN
add address=192.168.20.0/24 list=LAN
add address=192.168.30.0/24 list=LAN
add address=192.168.88.0/24 list=Service
add address=192.168.20.0/24 list=Printer
add address=192.168.10.0/24 list=Client
add address=192.168.30.0/24 list=Server
add address=192.168.40.0/24 list=Guest
add address=192.168.40.0/24 list=LAN
/ip firewall filter
add action=accept chain=forward dst-address-list=LAN src-address-list=Service
add action=drop chain=forward connection-state=new dst-address-list=Service src-address-list=LAN
add action=accept chain=forward connection-state=established,related dst-address-list=Service src-address-list=LAN
add action=accept chain=forward dst-address-list=Printer src-address-list=Guest
add action=drop chain=forward connection-state=new dst-address-list=Guest src-address-list=Printer
add action=accept chain=forward connection-state=established,related dst-address-list=Guest src-address-list=Printer
add action=accept chain=forward dst-address-list=Printer src-address-list=Client
add action=drop chain=forward connection-state=new dst-address-list=Client src-address-list=Printer
add action=accept chain=forward connection-state=established dst-address-list=Client src-address-list=Printer
add action=accept chain=forward connection-state=related dst-address-list=Client src-address-list=Printer
add action=drop chain=forward dst-address-list=LAN src-address-list=LAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/ip ipsec profile
set [ find default=yes ] dpd-interval=2m dpd-maximum-failures=5
/system clock
set time-zone-name=Europe/Rome
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

While Windows CLIENT machine can use PRINTER and connect to the integrated web server too, the two Linux GUEST cannot, but they can ping and get a reply from PRINTER.

Of course if I allow full forward communication for Guest <=> Printer everything works fine. But why Windows can work even if the firewall rules are active?

Thank you!

Sounds like a linux device issue, or a bad MT configuration issue!

Seems like the latter!!

  1. Remove this setting, its for advanced use...........
    /interface bridge settings
    set allow-fast-path=no use-ip-firewall=yes`

  2. The bridge should only be assigned a single address.
    If you have enough ports you could assign some ports to the bridge with a common subnet, and then assign ports, NOT on the bridge, different addresses and their own subnets.

OR
you could have one bridge and create vlans for all needed subnets.
However what you are currently doing, is not good.
why did you remove the default firewall rules? Your device without input chain rules WHICH EQUALS ALLOW EVERYTHING, is a security risk and should be pulled from the internet.

I’m just playing to better understand how the communication between hosts from different subnets works. The router isn’t directly connected to internet, so I can leave it a bit unprotected.

I came to the conclusion that VLANs are definitely the only solutions to manage subnets, the only drawback is that I need managed switches.

In this specific case, it appears that Windows and Linux handle the TCP/IP stack differently, because the same firewall rules have different effects on the two operating systems, so this kind of configuration is bad under several points of view.

Thanks again for all the insights!

A bit of general info:

generally, IP subnets are overlaid over L2 (ethernet) broadcast domains. Devices within an L2 domain are interconnected by switches (which more or less intelligently pass ethernet frames between their ports). Addressing is done exclusively based on MAC addresses.

VLANs are virtual L2 broadcast domains ... in a sense that physical infrastructure (UTP wires, physical switches) are shared between multiple L2 broadcast domains. Boundaries between those L2 domains are then enforced by VLAN-aware switches which consider 802.1 Q (VLAN) headers when determining where to forward ethernet frames. Addressing is based on MAC addresses and VLAN tags (which are optional for end devices but mandatory for switches ... if switches don't support VLANs then such infrastructure is a half brewed solution which can cause some nasty surprises).

Routing is an L3 operation (IP layer). Routers will have IP addresses on all of their interfaces and will inspect IP header when deciding on the egress interface for each IP packet.
In basic ethernet setup, router's interfaces are physical ethernet ports.
In VLAN setup, router's interfaces are VLAN interfaces (which are anchored off some lower interfaces, such as physical ethernet ports).

Then there are combos between routers and switches. Which are all ROS devices because ROS supports bridges (which are L2 functions, much like physical ethernet switches) ... and on many devices bridge functions are HW-offloaded to underlying physical switch chips (but that doesn't change the logic). On these devices bridge provides also interface which connects L2 entity (bridge the switch-like function) with L3 functions. And in case of VLANs those VLAN interfaces have to be anchored to bridge interface.
Router will happily pass packets between all of its interfaces according to routing rules and tables (there might be some routing filters which will prevent some IP packet to be forwarded).

What I'd like to emphasize is that ethernet and IP stacks operate pretty similarly across all OSes (not identically though) ... bugs excluded. There might be wildly different (G)UIs to configure these stacks, but that's to be expected.

And then there's firewall ... which is L3/L4 (and sometimes L7) function. It in principle operates independently from routing function ... but without looking really closely it might seem like routing and firewalling are intertwined.
Firewalling is by default active on all Windows machines and is becoming more and more frequently enabled on Linux machines (even though they are 'end devices'). Sincer firewalling is, in principle, not bound to any network standards, it comes in a vendor-specific implementation and different defaults.
But that's expected ... as much as it's expected to see e.g. wildly different GUIs and functionality of text editors made by different SW vendors.

1 Like

Thank you very much for all the informations!
I never stop learning new things about networks and ROS, and I really appreciate when someone explain me something I don’t understand.

So I have a new question, reading youre replies: why it’s better to not have multiple IP addresses attached to a bridge interface?

Thank you again!

Any given device needs one IP address per subnet it connects to. It can have multiple IP addresses from same IP subnet but then it becomes a bit complicated when it comes to selecting src IP address for outgoing packets.
And assigning IP address to certain entity makes that entity an IP interface.

Now, bridge (the switch facing interface) is an interface just like any other (e.g. eth0 on linux machine), so it can directly work with a single L2 broadcast domain (either VLAN of physical subnet). Regardless the number of attached IP addresses.

When starting to think about VLANs: L3 (IPv4 or IPv6 or ...) interface doesn't deal with 802.1Q (VLAN) headers, they are all alien to it. Hence need for VLAN interface (the entity created under /interface/vlan) which can be thought of as pipes with two ends: tagged, which attaches to L2 interface (e.g. ether1 or bridge interface), and untagged, which serves as interface (e.g. for assigning IP address). When ethernet frames pass the VLAN interface, it either adds VLAN header (to frames passing from untagged towards tagged end) or strips VLAN header (when passing from tagged tend towards untagged one) ... and does VLAN filtering at the same time (only frames with VLAN headers containing correct VLAN ID are accepted on tagged end).
And then router gets different IP addresses valid in different VLANs on corresponding VLAN interfaces.

Those VLAN interfaces can be attached to bridge interface just fine (but IMO this doesn't mean that bridge has multiple addresses).

more generally, why is it better to not have multiple IP addresses attached to an interface?

There are multiple reasons

  1. there is a single broadcast domain that is being shared by all the subnets. This breaks things like dhcp. If you have dhcp servers for each subnet, every one of the servers will hear the dhcp discover packet and respond with a dhcp offer. This can appear to work at first, because a device with an existing lease won't do another discover when it renews its lease, it will send another dhcp request to request a lease extension from the same dhcp server it received the lease from originally; the server with will then send either a dhcp acknowlege (if it is willing to extend the lease) or a dhcp nak if it is unwilling (or the request was for an ip address that was not previously offered, as will happen in the case when a laptop is moved to a new network).
  2. Security. If devices are on the same broadcast domain, they can "see" broadcast traffic from all subnets. So to join another network only requires changing the static ip address of the device. So if you want to prevent a device from having access to one of the other subnets, you are relying on user ignorance to prevent it.

About the only case where "secondary ip addresses" make sense is during an ip address migration from one subnet to a new one. Having multiple subnets on the server side will allow hosts that have not been switched over to continue to work the old address until their dhcp lease expires. But you should only have a single dhcp server active for the same ip address pool (unless the dhcp servers are configured with high availability and are coordinated).