Baffled with VLAN filtering not working on bridge!!

nbhatti · December 28, 2022, 5:39pm

Hi, I have CRS317-1G-16S+ running RouterOS version 7.6. The test setup is very simple, I made a simple bridge with VLAN filtering enabled with frame types to accept VLAN tagged only. Ingress filtering is also enabled on the bridge. This bridge has only 2 ports which also have frame types set to admit only VLAN tagged and Ingress filtering enabled as well. And offcourse the VLAN is defined and both 2 ports and the bridge is tagged in that specific VLAN.

This setup is simple enough (I Guess). That’s it on the router, no NAT, no firewall nothing else (for the sake of testing). The setup is to only allow a particular VLAN from the host side going through the bridge to the other side of the switch. Now with this setup, I am not able to ping the IP on the far side of the other port. Torch and packet capture shows that port 11 (server) sends traffic tagged correctly to Mikrotik, but the bridge does not sees that traffic and hence won’t pass to the other port 15 (other switch).

Strangely when I boot into SwOS (Switch OS) and in the VLAN tab I DISABLE Mac learning, thins start to work as expected. But for some reason i am not able to get this simple bridge with VLAN filtering to work.

/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge1 vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp-sfpplus11 ] arp-timeout=2s name=11-TNSR2
set [ find default-name=sfp-sfpplus15 ] arp-timeout=2s name=15-MPPL-WAFAQI rx-flow-control=on tx-flow-control=on
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=11-TNSR2
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=15-MPPL-WAFAQI
/interface bridge vlan
add bridge=bridge1 comment="MPPL VLANs for BGP Only" tagged=11-TNSR2,15-MPPL-WAFAQI,bridge1 vlan-ids=2110

One of 2 ping packets escape when I change any setting on the bridge and then again all timeouts. For example

PING 10.192.1.121 (10.192.1.121) 56(84) bytes of data.
64 bytes from 10.192.1.121: icmp_seq=17 ttl=64 time=0.986 ms
64 bytes from 10.192.1.121: icmp_seq=39 ttl=64 time=0.984 ms
64 bytes from 10.192.1.121: icmp_seq=62 ttl=64 time=0.966 ms
From 10.192.1.122 icmp_seq=756 Destination Host Unreachable
From 10.192.1.122 icmp_seq=757 Destination Host Unreachable
From 10.192.1.122 icmp_seq=758 Destination Host Unreachable
--- 10.192.1.121 ping statistics ---
798 packets transmitted, 3 received, +22 errors, 99.6241% packet loss, time 816038ms

If i create a VLAN interface with vlan 2110 on Mikrotik, I am able to ping the far end and the server can also ping Mikrotik so that’s not the issue of vlan not being passed on the other side. Everything else on default config just all the Firewall and NAT rules removed. I am out of any ideas, possible combinations of different options hence asking for help here. This thing works well with SwOS but I can not use SwOS because I have to create a bond eventually between 2 ports with Active/Backup where SwOS only provides Active/Active LAGG which won’t work in my case.

I need help, any advise any points on how or what to look for. Completely out of ideas here

Thanks.

sindy · December 28, 2022, 6:04pm

What do /interface bridge monitor bridge1 once and /interface bridge port monitor [find where bridge=bridge1] once show while the bridge does not pass traffic?

nbhatti · December 28, 2022, 6:30pm

/interface bridge monitor bridge1
                  state: enabled
    current-mac-address: 2C:C8:1B:A6:8C:8D
            root-bridge: yes
         root-bridge-id: 0x8000.2C:C8:1B:A6:8C:8D
         root-path-cost: 0
              root-port: none
             port-count: 2
  designated-port-count: 2
           fast-forward: no

and

/interface bridge port monitor [find where bridge=bridge1]
              interface: 11-TNSR2        15-MPPL-WAFAQI
                 status: in-bridge       in-bridge
            port-number: 1               2
                   role: designated-port designated-port
              edge-port: yes             yes
    edge-port-discovery: yes             yes
    point-to-point-port: yes             yes
           external-fdb: no              no
           sending-rstp: yes             yes
               learning: yes             yes
             forwarding: yes             yes
       hw-offload-group: switch1         switch1

sindy · December 28, 2022, 6:33pm

OK, and if you start pinging each other from devices connected to both ports (11, 15), what does /interface bridge host print where !local show?

nbhatti · December 28, 2022, 6:40pm

Nothing. Pinging from system connected to port 11 to the router connected at port 15 nothing shows in the bridge host table

[admin@MikroTik-Core-JT-PoP] > /interface bridge host print where !local

[admin@MikroTik-Core-JT-PoP] >
[admin@MikroTik-Core-JT-PoP] > /interface bridge host print
Flags: D - DYNAMIC; L - LOCAL
Columns: MAC-ADDRESS, VID, ON-INTERFACE, BRIDGE
#    MAC-ADDRESS         VID  ON-INTERFACE  BRIDGE
0 DL 2C:C8:1B:A6:8C:8D        bridge1       bridge1
1 DL 2C:C8:1B:A6:8C:8D  2110  bridge1       bridge1
[admin@MikroTik-Core-JT-PoP] >

nbhatti · December 28, 2022, 6:48pm

I noticed something that if I set Edge=No to port 11 and enable BPDU Guard, that ports immediately gets disabled because of BPDU packets received.

BPDU guard changed port role to disabled due to received BPDU frame

Could there be any relevance?

sindy · December 28, 2022, 6:52pm

Yes, the neighboring bridge may have its port disabled by STP. That’s why I was asking regarding the monitor. What kind of device is the neighboring one?

nbhatti · December 28, 2022, 6:56pm

That’s the ISP side (port 15) and they have a juniper switch. So my port 11 is sending BPDU frames and they are tagged as well because of the force vlan tagging? And the other side may have BPDU guard enabled? But even in this case, when I enable traffic sniff should I see traffic trying to go out from port 15?

nbhatti · December 28, 2022, 7:23pm

Ok I made some progress. I added a rule to Drop received BPDUs as per the Wiki

[admin@MikroTik-Core-JT-PoP] > /interface ethernet switch rule print
Flags: X - disabled, I - invalid; D - dynamic
 0 X  switch=switch1 ports=15-MPPL-WAFAQI dst-mac-address=01:80:C2:00:00:00/FF:FF:FF:FF:FF:FF copy-to-cpu=no
      redirect-to-cpu=no mirror=no new-dst-ports=""

 1    switch=switch1 ports=11-TNSR2 dst-mac-address=01:80:C2:00:00:00/FF:FF:FF:FF:FF:FF copy-to-cpu=no redirect-to-cpu=no
      mirror=no new-dst-ports=""

And now with RSTP on, port 11 does not gets disabled by BPDU guard even I set to Edgeport=No.

Bridge table shows the following info now when I ping from the host (behind port 11) to the router towards port 15.

[admin@MikroTik-Core-JT-PoP] > /interface bridge host print where !local
Flags: D - DYNAMIC; E - EXTERNAL
Columns: MAC-ADDRESS, VID, ON-INTERFACE, BRIDGE
#    MAC-ADDRESS         VID  ON-INTERFACE  BRIDGE
2 DE 00:08:30:20:5B:02  2110  11-TNSR2      bridge1
5 DE 78:FE:3D:A7:A7:C3  2110  11-TNSR2      bridge1
6 DE 90:E2:BA:85:31:81  2110  11-TNSR2      bridge1
7 DE 94:00:B0:B8:0B:00  2110  11-TNSR2      bridge1

But still not able to ping the other side from the host.

[admin@MikroTik-Core-JT-PoP] > /interface bridge port monitor [find where bridge=bridge1]
              interface: 11-TNSR2        15-MPPL-WAFAQI
                 status: in-bridge       in-bridge
            port-number: 1               2
                   role: designated-port designated-port
              edge-port: yes             yes
    edge-port-discovery: no              no
    point-to-point-port: yes             yes
           external-fdb: no              no
           sending-rstp: yes             yes
               learning: yes             yes
             forwarding: yes             yes
       hw-offload-group: switch1         switch1

sindy · December 28, 2022, 7:33pm

Except Cisco PVST+, BPDU frames are never tagged, and also the ingress filtering doesn’t affect them - incoming BPDU frames do not ingress the bridge, they are “link local” ones. So yes, since you are sending BPDU, you may trigger a BPDU guard at the ISP side (but it is strange that the same doesn’t happen when running SwOS).

Regarding outgoing traffic, sniffing can only see the complete traffic on a port if hw is set to no on the relevant row of /interface bridge port. Although it doesn’t seem logical to me, even some traffic to or from the CPU (I don’t remember which direction is affected) is not sniffed if hw=yes.

Apparently, setting edge to yes is not enough to prevent BPDUs from being sent to the “wire” (or maybe fibre in this case) - a bridge filter rule is necessary as described here.

nbhatti · December 28, 2022, 8:20pm

Appreciate your inputs and I have already set the rule, because of the rule set the port does not gets disabled now, but still can’t seem to figure out why some packets go through and most not

323 packets transmitted, 11 received, +11 errors, 96.5944% packet loss, time 329323ms

In SwOS under VLANs tab, only when I disable Learning which says, Enables or disables MAC address learning on the defined VLAN. If disabled, then all learned MAC addresses will appear as they have had been learned from VLAN 1. I am able to talk to the other side. Otherwise SwOS also does not works. Really can’t seem to understand why this happens and SwOS does not providers much more than gui to troubleshoot. Any other pointers to look for?

sindy · December 28, 2022, 9:08pm

GIven that there’s almost no traffic yet, I’d set hw=no for both ports, run /tool/sniffer/quick ip-address=10.192.1.121 and start pinging 10.192.1.121 to see whether the frames are dropped on the bridge (i.e. they ingress but don’t egress) or whether the issue is external.

nbhatti · December 29, 2022, 4:13pm

Ingress works fine from port 11, there is no egress. Something is blocking egress from the bridge towards port 15.

[admin@MikroTik-Core-JT-PoP] >  /tool/sniffer/quick ip-address=10.192.1.120/30
Columns: INTERFACE, TIME, NUM, DIR, SRC-MAC, DST-MAC, VLAN, SRC-ADDRESS, DST-ADDRESS, PROTOCOL, SIZE, CPU
INTERFACE  TIME    NUM  DIR  SRC-MAC            DST-MAC            VLAN  SRC-ADDRESS   DST-ADDRESS   PROTOCOL  SIZE  CPU
11-TNSR2   54.66    54  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   55.684   55  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   56.708   56  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   57.732   57  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   58.756   58  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   59.78    59  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   60.804   60  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   61.828   61  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   62.852   62  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   63.876   63  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   64.9     64  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   65.924   65  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   66.948   66  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   67.972   67  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   68.996   68  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   70.02    69  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   71.044   70  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   72.068   71  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   73.092   72  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
11-TNSR2   74.116   73  <-   90:E2:BA:85:31:81  78:FE:3D:A7:A7:C3  2110  10.192.1.122  10.192.1.121  ip:icmp    102    0
-- [Q quit|D dump|C-z pause]

sindy · December 29, 2022, 5:20pm

Does /interface bridge host print show 78:FE:3D:A7:A7:C3 as learned on port 15?

nbhatti · December 29, 2022, 5:37pm

No that’s learned from port 11

[admin@MikroTik-Core-JT-PoP] > interface bridge host print show
Flags: D - DYNAMIC; L - LOCAL; E - EXTERNAL
Columns: MAC-ADDRESS, VID, ON-INTERFACE, BRIDGE
*       MAC-ADDRESS         VID  ON-INTERFACE    BRIDGE
*DF DL  2C:C8:1B:A6:8C:8D        bridge1         bridge1
*DD DL  2C:C8:1B:A6:8C:91        15-MPPL-WAFAQI  bridge1
*E1 D E 00:08:30:20:5B:02  2110  11-TNSR2        bridge1
*DE DL  2C:C8:1B:A6:8C:8D  2110  bridge1         bridge1
*DC DL  2C:C8:1B:A6:8C:91  2110  15-MPPL-WAFAQI  bridge1
*E0 D E 78:FE:3D:A7:A7:C3  2110  11-TNSR2        bridge1
*D9 D E 90:E2:BA:85:31:81  2110  11-TNSR2        bridge1

sindy · December 29, 2022, 6:17pm

But that highlights the issue… The sniff shows that a packet from SRC-MAC 90:E2:BA:85:31:81 to DST-MAC 78:FE:3D:A7:A7:C3 has arrived from outside to 11-TNSR2, so the DST-MAC 78:FE:3D:A7:A7:C3 should indeed be learned at 15, but it is not the case - your /interface bridge host print indicates that both 78:FE:3D:A7:A7:C3 and 90:E2:BA:85:31:81 have been learned at 11-TNSR2, which means that frames with both these source addresses must have come in via that port before. So it seems that another path to the 78:FE:3D:A7:A7:C3 address exists outside the CRS, and the CRS of course doesn’t send a frame back through the same port through which it has received it.