CRS3xx VLAN configuration

astifr · October 20, 2020, 9:33am

Hello to everybody.

I’m trying to configure this scenario,

Switch Cisco ----sfp-sfpplus3 CRS3XX LACP link---- Switch Cisco

All traffic arrive to CR3xx tagged, i.e. with VLAN ID 200.

So, my problem is that packets are not forwarding in the CRS3xx, I can see packets with VLAN tag ingressing sfp-sfpplus3 but not egressing LACP,
That’s is a capture from CRS3xx,

[admin@MKTSW1] > /tool sniffer quick interface=sfp-sfpplus3
INTERFACE                                                    TIME    NUM DIR SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS                         DST-ADDRESS                         PROTOCOL   SIZE CPU FP
sfp-sfpplus3                                                0.229      1 ->  48:8F:5A:C1:13:39 01:80:C2:00:00:00                                                                                802.2        53   1 no
sfp-sfpplus3                                                0.657      2 <-  00:1E:F7:67:32:E0 FF:FF:FF:FF:FF:FF 200    192.168.200.1: who has 192.168.2...                                     arp          64   1 no
sfp-sfpplus3                                                2.229      3 ->  48:8F:5A:C1:13:39 01:80:C2:00:00:00                                                                                802.2        53   1 no
sfp-sfpplus3                                                2.656      4 <-  00:1E:F7:67:32:E0 FF:FF:FF:FF:FF:FF 200    192.168.200.1: who has 192.168.2...                                     arp          64   1 no
[admin@MKTSW1] > /tool sniffer quick interface=LACP1
INTERFACE                                                    TIME    NUM DIR SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS                         DST-ADDRESS                         PROTOCOL   SIZE CPU FP
LACP1                                                       1.019      1 ->  48:8F:5A:C1:13:37 01:80:C2:00:00:00                                                                                802.2        53   1 no
LACP1                                                       3.018      2 ->  48:8F:5A:C1:13:37 01:80:C2:00:00:00                                                                                802.2        53   1 no
LACP1                                                       5.019      3 ->  48:8F:5A:C1:13:37 01:80:C2:00:00:00                                                                                802.2        53   1 no

If I ping from one side, you can see ARP in VLAN 200 traffic ingress trought sfp-sfpplus3 but no egress

LACP Bonding interface is up and running.

That’s is my config,

/interface bridge
add auto-mac=no comment=defconf name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp-sfpplus3 ] auto-negotiation=no
/interface bonding
add mode=802.3ad name=LACP1 slaves=sfp-sfpplus1,sfp-sfpplus2
/interface bridge port
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
add bridge=bridge interface=LACP1
/interface bridge vlan
add bridge=bridge untagged=LACP1,sfp-sfpplus3 vlan-ids=200
/ip address
add address=192.168.88.2/24 comment=defconf interface=ether1 network=192.168.88.0
/ip dns
set servers=8.8.8.8
/ip route
add distance=1 gateway=192.168.88.254
/system identity
set name=MKTSW1

any ideas?

Thank you in advance.

sindy · October 20, 2020, 9:44am

Assuming you want the traffic to be forwarded between sfp-sfpplus3 and the LAG tagged, the record for vlan-ids=200 in /interface bridge vlan must have the ports on the tagged list, not on the untagged one.

astifr · October 20, 2020, 10:14am

Thank you sindy, I can’t believe my fault… sorry.

However, I change the config and trying again but it still not works…
I disable the LACP to simply the troubleshooting.

Switch Cisco ----sfp-sfpplus3 CRS3XX sfp-sfpplus2---- Switch Cisco

That’s the config now,

/interface bridge
add auto-mac=no comment=defconf name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp-sfpplus3 ] auto-negotiation=no
/interface bonding
add disabled=yes mode=802.3ad name=LACP1 slaves=sfp-sfpplus1
/interface bridge port
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
add bridge=bridge interface=LACP1
add bridge=bridge interface=sfp-sfpplus2
/interface bridge vlan
add bridge=bridge tagged=sfp-sfpplus3,sfp-sfpplus2 vlan-ids=200
/ip address
add address=192.168.88.2/24 comment=defconf interface=ether1 network=192.168.88.0
/ip dns
set servers=8.8.8.8
/ip route
add distance=1 gateway=192.168.88.254
/system identity
set name=MKTSW1
/system routerboard settings
set boot-os=router-os

[admin@MKTSW1] > /tool sniffer quick interface=sfp-sfpplus2
INTERFACE                                                    TIME    NUM DIR SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS                         DST-ADDRESS                         PROTOCOL   SIZE CPU FP
sfp-sfpplus2                                                1.358      1 ->  48:8F:5A:C1:13:38 01:80:C2:00:00:00                                                                                802.2        53   0 no
sfp-sfpplus2                                                3.364      2 ->  48:8F:5A:C1:13:38 01:80:C2:00:00:00                                                                                802.2        53   0 no
sfp-sfpplus2                                                5.362      3 ->  48:8F:5A:C1:13:38 01:80:C2:00:00:00                                                                                802.2        53   0 no

[admin@MKTSW1] > /tool sniffer quick interface=sfp-sfpplus3
INTERFACE                                                    TIME    NUM DIR SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS                         DST-ADDRESS                         PROTOCOL   SIZE CPU FP
sfp-sfpplus3                                                0.497      1 <-  00:1E:F7:67:32:E0 FF:FF:FF:FF:FF:FF 200    192.168.200.1: who has 192.168.2...                                     arp          64   1 no
sfp-sfpplus3                                                1.378      2 ->  48:8F:5A:C1:13:39 01:80:C2:00:00:00                                                                                802.2        53   0 no
sfp-sfpplus3                                                2.497      3 <-  00:1E:F7:67:32:E0 FF:FF:FF:FF:FF:FF 200    192.168.200.1: who has 192.168.2...                                     arp          64   1 no
sfp-sfpplus3                                                3.372      4 ->  48:8F:5A:C1:13:39 01:80:C2:00:00:00                                                                                802.2        53   0 no

[admin@MKTSW1] > interface print
Flags: D - dynamic, X - disabled, R - running, S - slave
 #     NAME                                TYPE       ACTUAL-MTU L2MTU  MAX-L2MTU MAC-ADDRESS
 0  R  ether1                              ether            1500  1592      10218 48:8F:5A:C1:13:3F
 1  X  sfp-sfpplus1                        ether            1500  1592      10218 48:8F:5A:C1:13:37
 2  RS sfp-sfpplus2                        ether            1500  1592      10218 48:8F:5A:C1:13:38
 3  RS sfp-sfpplus3                        ether            1500  1592      10218 48:8F:5A:C1:13:39
 4   S sfp-sfpplus4                        ether            1500  1592      10218 48:8F:5A:C1:13:3A
 5   S sfp-sfpplus5                        ether            1500  1592      10218 48:8F:5A:C1:13:3B
 6   S sfp-sfpplus6                        ether            1500  1592      10218 48:8F:5A:C1:13:3C
 7   S sfp-sfpplus7                        ether            1500  1592      10218 48:8F:5A:C1:13:3D
 8   S sfp-sfpplus8                        ether            1500  1592      10218 48:8F:5A:C1:13:3E
 9  X  LACP1                               bond             1500 65535            00:00:00:00:00:00
10  R  ;;; defconf
       bridge                              bridge           1500  1592            48:8F:5A:C1:13:3F

[admin@MKTSW1] > /interface bridge port monitor 0,7
              interface: sfp-sfpplus3    sfp-sfpplus2
                 status: in-bridge       in-bridge
            port-number: 1               7
                   role: designated-port designated-port
              edge-port: yes             no
    edge-port-discovery: yes             yes
    point-to-point-port: yes             yes
           external-fdb: no              no
           sending-rstp: yes             yes
               learning: yes             yes
             forwarding: yes             yes
       hw-offload-group: switch1         switch1

sindy · October 20, 2020, 10:51am

Just for the case, I’d disable also the row in /interface bridge port which makes the LAG a member of the bridge (someone here had an issue with a disabled EoIP affecting the MTU of the bridge, so disabling the membership should be a safer way).

Other than that, your export suggests that you have the default setting of protocol-mode on the bridge, so maybe RSTP is blocking one of the ports or both. So what does /interface bridge monitor bridge show, and what does /interface bridge port monitor 0,1,2,3 after /interface bridge port print show?

And last, what RouterOS version do you run? Someone else here has reported 6.47.4 to break hardware VLAN filtering on an 8227 chip which worked for him with an older ROS version, maybe there is some more generic error there.

astifr · October 20, 2020, 1:39pm

Thank you sindy.

I was using 6.47.4 but after your advise I’m using 6.46.7 long term. The issue still persist…

Unused ports are now disabled and LACP deleted.

That’s the config now,

[admin@MKTSW1] > export hide-sensitive
# jan/14/1970 22:19:12 by RouterOS 6.46.7
# software id = Z2N5-V8B0
#
# model = CRS309-1G-8S+
# serial number = ------
/interface bridge
add admin-mac=48:8F:5A:C1:13:3F auto-mac=no comment=defconf name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] disabled=yes
set [ find default-name=sfp-sfpplus3 ] auto-negotiation=no
set [ find default-name=sfp-sfpplus4 ] disabled=yes
set [ find default-name=sfp-sfpplus5 ] disabled=yes
set [ find default-name=sfp-sfpplus6 ] disabled=yes
set [ find default-name=sfp-sfpplus7 ] disabled=yes
set [ find default-name=sfp-sfpplus8 ] disabled=yes
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/user group
set full policy=local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp
/interface bridge port
add bridge=bridge interface=sfp-sfpplus3
add bridge=bridge interface=sfp-sfpplus2
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface bridge vlan
add bridge=bridge tagged=sfp-sfpplus3,sfp-sfpplus2 vlan-ids=200
/ip address
add address=192.168.88.2/24 comment=defconf interface=ether1 network=192.168.88.0
/ip dns
set servers=8.8.8.8
/ip route
add distance=1 gateway=192.168.88.254
/system identity
set name=MKTSW1
/system package update
set channel=long-term
/system routerboard settings
set boot-os=router-os

And that’s the output of some comands,

[admin@MKTSW1] > interface bridge monitor bridge
                     ;;; defconf
                  state: enabled
    current-mac-address: 48:8F:5A:C1:13:3F
            root-bridge: yes
         root-bridge-id: 0x8000.48:8F:5A:C1:13:3F
         root-path-cost: 0
              root-port: none
             port-count: 2
  designated-port-count: 2
           fast-forward: no

[admin@MKTSW1] >  /interface bridge port print
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload
 #     INTERFACE                                                                          BRIDGE                                                                         HW  PVID PRIORITY  PATH-COST INTERNAL-PATH-COST    HORIZON
 0   H sfp-sfpplus3                                                                       bridge                                                                         yes    1     0x80         10                 10       none
 1   H sfp-sfpplus2                                                                       bridge                                                                         yes    1     0x80         10                 10       none
[admin@MKTSW1] >  /interface bridge port monitor 0,1
              interface: sfp-sfpplus3    sfp-sfpplus2
                 status: in-bridge       in-bridge
            port-number: 2               1
                   role: designated-port designated-port
              edge-port: yes             no
    edge-port-discovery: yes             yes
    point-to-point-port: yes             yes
           external-fdb: no              no
           sending-rstp: yes             yes
               learning: yes             yes
             forwarding: yes             yes
       hw-offload-group: switch1         switch1

hope this help, thank you.

Edit:

I see the same behavior with untagged traffic.
RouterOs creates a Dynamic interface for VLAN 1,

[admin@MKTSW1] > interface bridge vlan print
Flags: X - disabled, D - dynamic
 #   BRIDGE                                                           VLAN-IDS  CURRENT-TAGGED                                                           CURRENT-UNTAGGED
 0   bridge                                                           200       sfp-sfpplus3
                                                                                sfp-sfpplus2
 1 D bridge                                                           1                                                                                  bridge
                                                                                                                                                         sfp-sfpplus3
                                                                                                                                                         sfp-sfpplus2

I can see an ARP request ingressin switch via sfp3 but nothing egressing via sfp2.

sindy · October 20, 2020, 2:35pm

Stop. We’re most likely hunting for a ghost here. Hardware “acceleration” is enabled at both ports, so the traffic between them is forwarded by the switch chip, bypassing the CPU, hence the CPU cannot sniff it.

The frames you can see in the sniffer output are there because they are either sent by the CPU itself or they have a broadcast destination MAC address so the switch broadcasts them also to the CPU port.

To see whether the traffic passes from one port to the other as forwarded by the switch chip itself, you have to sniff externally.

astifr · October 20, 2020, 2:58pm

Yes! you are right again. I will check it and return with the results.

Thank you!