Local VLAN access ports input router via the bridge rather than the VLAN interface

RouterOS General
1

Hi guys,

I have the following setup with an RB3011 device.
Two WAN ports (ether1 and 2), not bridged and switch chip 1 not used.
A bridge with ether6, 7, 8, 9 and 10 as its ports using hardware offloading. Switch chip VLAN table is used as well. Ether6 is the trunk port while ether7 to 10 are access ports for the VLANs 10 (Office), 20 (Guests), 30 (Private) and 99 (MGMT) respectively.
There is a VLAN interface for all VLANs on top of the bridge. DHCP is enabled for all VLANs except MGMT VLAN.

I’d like to use the allow what you need, block all the rest firewall philosophy.
So I created an interface list called “InternetVLANs” containing VLANs 10, 20 and 30. All traffic flowing in the network should be part of either one of the VLANs. MGMT VLAN doesn’t need internet access so it’s not in the list. Then I created some firewall rules to allow certain kind of traffic (e.g. forward from InternetVLANs to WAN, DNS input to router, etc…).

The issue I encountered:
When I connect a device to the local access ports (e.g. ether 7), DNS requests input the router from the Bridge rather than from the VLAN interface. So this means the traffic is dropped by the default block rule in the firewall. However, when comming through the trunk port the traffic enters the router correctly via the VLAN interface.
Somehow I think this makes sense. Indeed, traffic entering the trunk port is already tagged. However, traffic entering the access port isn’t tagged yet, so why tagging when it should be routed in L3?

The simple solution would be to add the Bridge to the “InternetVLANs” interface list. But I’d rather not do so since this will allow MGMT VLAN traffic that sneaks somehow into the Bridge access to the internet, which I don’t want. I’d rather “fix” the issue that traffic is entering the CPU via the Bridge rather than the VLAN interface. If this is fixable at all…?
One important remark! Despite DNS queries sometimes get blocked because of the default block rule, internet access is more or less working, although very very crappy (i.e. multiple retries before a page gets loaded etc).

So my question would be, any advice from your experts on how to handle this? Someone who’d like to review the setup, especially the bridge and switch config?

Thanks in advance!

2

It is likely you have incomplete switch configuration settings - unless vlan-mode=secure there will be leakage between VLANs. Post the output of /export hide-sensitive

When using the switch chip untagged ingress traffic is tagged with the default-vlan-id, there are some exceptions between switch ports configured with the same default-vlan-id but this would not apply to your case.

3

Here is the complete export:

# nov/05/2019 18:36:35 by RouterOS 6.44.6
# software id = ZBUE-6X7V
#
# model = RouterBOARD 3011UiAS
# serial number = *removed*
/interface bridge
add name=bridge1 protocol-mode=none
/interface ethernet
set [ find default-name=ether1 ] speed=100Mbps
set [ find default-name=ether2 ] speed=100Mbps
set [ find default-name=ether3 ] speed=100Mbps
set [ find default-name=ether4 ] speed=100Mbps
set [ find default-name=ether5 ] speed=100Mbps
set [ find default-name=ether6 ] speed=100Mbps
set [ find default-name=ether7 ] speed=100Mbps
set [ find default-name=ether8 ] speed=100Mbps
set [ find default-name=ether9 ] speed=100Mbps
set [ find default-name=ether10 ] speed=100Mbps
set [ find default-name=sfp1 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
/interface vlan
add comment="Management VLAN" interface=bridge1 name=MGMT vlan-id=99
add comment="My Office VLAN" interface=bridge1 name=VLAN10 vlan-id=10
add comment="My Guests VLAN" interface=bridge1 name=VLAN20 vlan-id=20
add comment="My Private VLAN" interface=bridge1 name=VLAN30 vlan-id=30
/interface ethernet switch port
set 5 vlan-mode=secure
set 6 default-vlan-id=10 vlan-mode=secure
set 7 default-vlan-id=20 vlan-mode=secure
set 8 default-vlan-id=30 vlan-mode=secure
set 9 default-vlan-id=99 vlan-mode=secure
set 11 vlan-mode=secure
/interface list
add comment="This list contains all VLANs that have internet access" name=\
    InternetVLANs
add comment="This list contains all WAN ports" name=WAN
add comment="This list is used to temp allow traffic between Management VLAN a\
    nd Office VLAN (e.g. for troubleshooting purposes)" name=MGMT-VLAN10
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add comment="Pool for Office" name=POOL_10 ranges=\
    192.168.10.100-192.168.10.200
add comment="Pool for Guests" name=POOL_20 ranges=\
    192.168.20.50-192.168.20.200
add comment="Pool for Private" name=POOL_30 ranges=\
    192.168.30.50-192.168.30.200
/ip dhcp-server
add address-pool=POOL_10 disabled=no interface=VLAN10 name=DHCP_10
add address-pool=POOL_20 disabled=no interface=VLAN20 name=DHCP-serv_20
add address-pool=POOL_30 disabled=no interface=VLAN30 name=DHCP-serv_30
/interface bridge port
add bridge=bridge1 interface=ether6
add bridge=bridge1 interface=ether7 pvid=10
add bridge=bridge1 interface=ether8 pvid=20
add bridge=bridge1 interface=ether9 pvid=30
add bridge=bridge1 interface=ether10 pvid=99
/interface ethernet switch vlan
add independent-learning=yes ports=ether6,ether7,switch2-cpu switch=switch2 \
    vlan-id=10
add independent-learning=yes ports=ether6,ether8,switch2-cpu switch=switch2 \
    vlan-id=20
add independent-learning=no ports=ether6,ether9,switch2-cpu switch=switch2 \
    vlan-id=30
add independent-learning=no ports=ether6,ether10,switch2-cpu switch=switch2 \
    vlan-id=99
/interface list member
add interface=VLAN10 list=InternetVLANs
add interface=VLAN20 list=InternetVLANs
add interface=VLAN30 list=InternetVLANs
add interface=ether1 list=WAN
add interface=MGMT list=MGMT-VLAN10
add interface=VLAN10 list=MGMT-VLAN10
/ip address
add address=192.168.10.1/24 interface=VLAN10 network=192.168.10.0
add address=192.168.20.1/24 interface=VLAN20 network=192.168.20.0
add address=192.168.30.1/24 interface=VLAN30 network=192.168.30.0
add address=192.168.99.1/24 interface=MGMT network=192.168.99.0
/ip dhcp-client
add comment="Internet gateway" dhcp-options=hostname,clientid disabled=no \
    interface=ether1
/ip dhcp-server network
add address=192.168.10.0/24 gateway=192.168.10.1
add address=192.168.20.0/24 gateway=192.168.20.1
add address=192.168.30.0/24 gateway=192.168.30.1
/ip dns
set allow-remote-requests=yes cache-max-ttl=4h
/ip firewall filter
add action=accept chain=input comment=\
    "Accept established and related connections" connection-state=\
    established,related
add action=accept chain=input comment=\
    "Accept DNS connections from VLANs with Internet connection" dst-port=53 \
    in-interface-list=InternetVLANs protocol=udp
add action=accept chain=input comment="TEST: Accept DNS from all but WAN" \
    dst-port=53 in-interface-list=!WAN protocol=udp
add action=accept chain=input comment=\
    "TEST: Accept NetBIOS input from all but WAN" dst-port=137 \
    in-interface-list=!WAN protocol=udp
add action=accept chain=input comment=\
    "Allow ICMP traffic (e.g. ping) from non-WAN" in-interface-list=!WAN \
    protocol=icmp
add action=accept chain=input comment=\
    "Accept all connections from Management VLAN" in-interface=MGMT
add action=accept chain=input comment=\
    "Accept WinBox connection from Office VLAN" dst-port=8291 in-interface=\
    VLAN10 protocol=tcp
add action=drop chain=input comment=\
    "TMP: Drop incomming WAN traffic separately to clean up drop log" \
    in-interface-list=WAN
add action=drop chain=input comment=\
    "Basic policy: drop everything unless allowed by a specific rule" \
    disabled=yes log=yes log-prefix=InputChainDrop
add action=fasttrack-connection chain=forward comment=\
    "FastTrack connection to improve troughput for established connections" \
    connection-state=established,related disabled=yes
add action=accept chain=forward comment=\
    "Accept established and related connections" connection-state=\
    established,related
add action=accept chain=forward comment=\
    "Allow outgoing traffic from Internet VLANs" in-interface-list=\
    InternetVLANs out-interface-list=WAN
add action=accept chain=forward comment="Allow inter-VLAN traffic between Mana\
    gement and Office VLAN --> Use only temporary for troubleshooting!!" \
    in-interface-list=MGMT-VLAN10 out-interface-list=MGMT-VLAN10
add action=drop chain=forward comment=\
    "Basic policy: drop everything unless allowed by a specific rule" \
    disabled=yes log=yes log-prefix=ForwardChainDrop
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
/ip route rule
add action=drop comment="No routing from Guest to Office" disabled=yes \
    dst-address=192.168.10.0/24 src-address=192.168.20.0/24
add action=drop comment="No routing from Guest to Management VLAN" disabled=\
    yes dst-address=192.168.99.0/24 src-address=192.168.20.0/24
add action=drop comment="No routing from Private to Office" disabled=yes \
    dst-address=192.168.10.0/24 src-address=192.168.30.0/24
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address=192.168.99.0/24
set api disabled=yes
set winbox address=192.168.99.0/24,192.168.10.0/24
set api-ssl disabled=yes
/lcd
set default-screen=stats-all
/system clock
set time-zone-name=Europe/Brussels
/system ntp client
set enabled=yes primary-ntp=51.255.138.215 secondary-ntp=162.159.200.1
/system ntp server
set enabled=yes
/system package update
set channel=long-term
/system scheduler
add comment="Lookup and set NTP server IP addresses" interval=12h name=\
    CheckNTPservers on-event=setntppool policy=read,write,test start-date=\
    nov/01/2019 start-time=00:00:00
/system script
add dont-require-permissions=no name=setntppool owner=admin policy=\
    read,write,test source="# Check and set NTP servers - \"setntppool\"\r\
    \n\r\
    \n# NTP server pools\r\
    \n:local SYSntpa \"0.pool.ntp.org\";\r\
    \n:local SYSntpb \"1.pool.ntp.org\";\r\
    \n\r\
    \n# Resolve the two ntp pool hostnames\r\
    \n:local ntpipa [:resolve \$SYSntpa];\r\
    \n:local ntpipb [:resolve \$SYSntpb];\r\
    \n\r\
    \n# Get the current settings\r\
    \n:local ntpcura [/system ntp client get primary-ntp];\r\
    \n:local ntpcurb [/system ntp client get secondary-ntp];\r\
    \n\r\
    \n# Define a variable so we know if anything's changed.\r\
    \n:local changea 0;\r\
    \n:local changeb 0;\r\
    \n\r\
    \n# Debug output\r\
    \n:put (\"Old: \" . \$ntpcura . \" New: \" . \$ntpipa);\r\
    \n:put (\"Old: \" . \$ntpcurb . \" New: \" . \$ntpipb);\r\
    \n\r\
    \n# Change primary if required\r\
    \n:if (\$ntpipa != \$ntpcura) do={\r\
    \n    :put \"Changing primary NTP\";\r\
    \n    /system ntp client set primary-ntp=\"\$ntpipa\";\r\
    \n    :set changea 1;\r\
    \n    }\r\
    \n\r\
    \n# Change secondary if required\r\
    \n:if (\$ntpipb != \$ntpcurb) do={\r\
    \n    :put \"Changing secondary NTP\";\r\
    \n    /system ntp client set secondary-ntp=\"\$ntpipb\";\r\
    \n    :set changeb 1;\r\
    \n    }"

According to manual ( https://wiki.mikrotik.com/wiki/Manual:Basic_VLAN_switching#Other_devices_with_built-in_switch_chip and https://wiki.mikrotik.com/wiki/Manual:Switch_Chip_Features#Setup_Examples ):
VLAN mode = secure, configured on the switch chip. Access ports get a PVID, trunk port get none.

/interface ethernet switch port
set 5 vlan-mode=secure
set 6 default-vlan-id=10 vlan-mode=secure
set 7 default-vlan-id=20 vlan-mode=secure
set 8 default-vlan-id=30 vlan-mode=secure
set 9 default-vlan-id=99 vlan-mode=secure
set 11 vlan-mode=secure

Special attention to

Note: For devices with > QCA8337 > and Atheros8327 switch chips a default vlan-header=leave-as-is should be used. When vlan-mode=secure is configured, it ignore switch port vlan-header options. VLAN table entries handle all the egress tagging/untagging and works as vlan-header=leave-as-is on all ports. It means what comes in tagged, goes out tagged as well, only default-vlan-id frames are untagged at the egress of port.

so VLAN-header options are not used (i.e. leave-as-is).


On the bridge I also configured VLAN IDs as a test afterwards, but they should not be required since the switch chip VLAN table is used and VLAN filtering on the bridge is not enabled:

/interface bridge port
add bridge=bridge1 interface=ether6
add bridge=bridge1 interface=ether7 pvid=10
add bridge=bridge1 interface=ether8 pvid=20
add bridge=bridge1 interface=ether9 pvid=30
add bridge=bridge1 interface=ether10 pvid=99

Although in the script hardware offloading is not selected (HW=yes), it is enabled:

[admin@MikroTik] > /interface bridge port print
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
 #     INTERFACE      BRIDGE         HW  PVID PR  PATH-COST INTERNA...    H
 0   H ether6         bridge1        yes    1 0x         10         10     
 1   H ether7         bridge1        yes   10 0x         10         10     
 2 I H ether8         bridge1        yes   20 0x         10         10     
 3 I H ether9         bridge1        yes   30 0x         10         10     
 4 I H ether10        bridge1        yes   99 0x         10         10     
[admin@MikroTik] >

Indepedant learning is not enabled for VLAN 30 and 99, I forgot this. But I don’t believe (I strongly hope) this is not the root of the issue.
Actually, I don’t think independant learning is required at all since I don’t expect the same MAC in different VLANs.

Small note: WAN2 is not yet configured, hence it’s not in the script.

Below an image of the ForwardChainDrop Rule log. I can see outbound DNS querries and HTTPs traffic, entering the router from Bridge 1.
RouterLog_InputDropLog.png
Edit: Trying to get the code tags correct for the layout…

4

Just for testing, I configured the bridge VLAN table and enabled Bridge VLAN filtering, hence HW offloading got disabled.

First thing I noticed was that switch port settings interfered with bridge port settings…
Switch ports were still configured with VLAN mode = secure and PVIDs. I believed the switch configuration didn’t matter since there was no HW offloading active. As a matter of fact, it did! Switch was still adding VLAN tags to the incomming frames on the access port, so the strict Frame Type filtering (configured in Bridge ports) tumbled over the VLAN tags.

After removing the complete switch VLAN configuration, so I had a clean Bridge VLAN filtering setup, everything seems to work as expected.
Traffic originating from the access port enters the router via the VLAN interfaces since no more packets get dropped.

So, in my original setup something was not right between switch config and bridge config. However, I cannot find what using the plenty of manuals and examples available online :confused:
Any help is really appreciated!

I don’t want to use the current config with HW offloading disabled, although I don’t expect much traffic between the local access ports and the rest of the network. I simply want to have it “the right way” (and this means including HW offloading).
Unless someone can prove me that the desired setup is not possible with Mikrotik (or this specific routerboard) :frowning:

5

It will, traffic always passes through the switch chip - to quote from a previous post:

“Looking at Winbox and seeing ether1-5 interfaces you are fooled into thinking that the CPU has five ethernet interfaces, each connected to the corresponding physical port. In reality the CPU has one or two ethernet interfaces wired to a switch chip (either within a single system-on-chip package or separate CPU and switch chips), which in turn is wired to the physical ports. Network traffic is always passing through the switch chip irrespective of if you are using the interfaces individually or in a bridge (with or without VLAN filtering). Atheros implement a proprietary extension to the data flowing between the CPU and switch which specifies which physical port the traffic is associated with so the driver can multiplex/demultiplex the traffic to the logical interfaces.” On a 3011 this is replicated for ether6-10 and the second switch chip.

When using hardware acceleration the CPU cannot determine which of ether[6-10] a packet arrives on, it is delivered to the CPU from the switch2-cpu interface which is a hidden bridge port.

Your original configuration looks fine - I suspect it is a ‘feature’ of RouterOS in that it considers the packet to have first been handled by the bridge rather than the VLAN interface attached to the bridge, hence in: bridge1 in the firewall rule logging, which is different behaviour to a VLAN interface directly attached to an ether interface.

You can add bridge ports as criteria in /ip firewall filter, if this is insufficient you may be able to add packet marks in /interface bridge filter for traffic on a specific VLAN, the use the packet mark as a criteria in /ip firewall filter.

6

EDIT: please ignore this post.

There is a caveat which is not obvious. Frames entering the switch which are to be forwarded to the CPU port are handled in a different way than those forwarded to another ethernet port. So even if you set up the VLAN handling using the /interface ethernet switch branch of the configuration tree, you still have to replicate the same setup in the /interface bridge branch.

So in particular for your case:

/interface ethernet switch port
set 5 vlan-mode=secure
set 6 default-vlan-id=10 vlan-mode=secure
set 7 default-vlan-id=20 vlan-mode=secure
set 8 default-vlan-id=30 vlan-mode=secure
set 9 default-vlan-id=99 vlan-mode=secure
set 11 vlan-mode=secure

/interface ethernet switch vlan
add independent-learning=yes ports=ether6,ether7,switch2-cpu switch=switch2 vlan-id=10
add independent-learning=yes ports=ether6,ether8,switch2-cpu switch=switch2 vlan-id=20
add independent-learning=no ports=ether6,ether9,switch2-cpu switch=switch2 vlan-id=30
add independent-learning=no ports=ether6,ether10,switch2-cpu switch=switch2 vlan-id=99

needs to be replicated by
/interface bridge port
add bridge=bridge1 interface=ether6
add bridge=bridge1 interface=ether7 pvid=10
add bridge=bridge1 interface=ether8 pvid=20
add bridge=bridge1 interface=ether9 pvid=30
add bridge=bridge1 interface=ether10 pvid=99
(which you already did), but also by

/interface bridge vlan
add bridge=bridge1 vlan-ids=10 tagged=ether6,bridge1 untagged=ether7
add bridge=bridge1 vlan-ids=20 tagged=ether6,bridge1 untagged=ether8
add bridge=bridge1 vlan-ids=30 tagged=ether6,bridge1 untagged=ether9
add bridge=bridge1 vlan-ids=99 tagged=ether6,bridge1 untagged=ether10

and, very important, /interface bridge set bridge1 vlan-filtering=yes once you set the above. If you manage the router via one of ether6..ether10, the last step may lock you out, so either manage it via ether3..ether5 or do the last step in safe mode (Ctrl-X before activating vlan-filtering on the bridge).

The “hardware offloading” only handles frames forwarded between wired ports of the switch.

7

Hi guys,

Just wanted to let you know that the issue is resolved.


It not only looks fine, it’s proven to be fine as well :sunglasses:

Before starting to fiddle around with the Bridge VLAN filtering I took another backup. After posting my results on the board and reading some more manuals/examples, I decided to try one more thing before ditching the entire configuration and starting over from scratch.
So, I restored the backup in order to start were I left. After restoring the backup, I wanted to make sure that the situation was exactly as before. Uppon checking the firewall rules and the logs, I noticed that there were no more dropping entries (in the logs, the rules were still in place and enabled). Also internet access was very smooth. I double checked that the firewall rules were enabled, and it turned out to be…

So bottom line, the configuration was indeed fine and finally started working as expected.
I have to admit, I was messing seriously with the configuration before ending up with the above posted configuration. I think the system got a bit entangled due to all the changes. Unfortunately, this means that it will be very hard to replicate the issue since the exact order of every step will be key. And so the bug will probably live on for a while :frowning:

Anyway, thanks all for your input!

8

I’m sorry, I can’t agree with this statement.
Please have a look at the manual: https://wiki.mikrotik.com/wiki/Manual:Basic_VLAN_switching#Other_devices_with_built-in_switch_chip

You can connect a single cable between both switch chips to work around this hardware limitation, another option is to > use Bridge VLAN Filtering, but it disables hardware offloading > (and lowers the total throughput).

Or this manual: https://wiki.mikrotik.com/wiki/Manual:Interface/Bridge#Bridge_VLAN_Filtering

Note: Currently only CRS3xx series devices are capable of using bridge VLAN filtering and hardware offloading at the same time, other devices will not be able to use the benefits of a built-in switch chip when bridge VLAN filtering is enabled. Other devices should be configured according to the method described in the Basic VLAN switching guide. If an improper configuration method is used, your device can cause throughput issues in your network.

9

Yes, correct, I had some total brain eclipse. vlan-filtering coexists with hardware acceleration only on CRS3xx.