Hello everyone!
I’ve recently replaced an old switch with a new Mikrotik CRS310-8G+2S+ and I’ve stumbled upon a Problem in regards to MSTP and Hardware offloading.
Overview
In order to make troubleshooting easier, consider the following diagram which shows the topology as well as the device which was replaced (green):
All switches have set up and working MSTP for all VLANS (1-4094) and the topology is correctly discovered between all three of them. The cng-chni1-sw5 has all of its ports bridged together without rules. The bridge is, as far as I can tell, correctly set up for VLAN Filtering.
Below is the current config of cng-chni1-sw5:
# 2024-10-29 09:37:09 by RouterOS 7.16.1
# software id = RR52-IQTN
#
# model = CRS310-8G+2S+
# serial number = HG109M9KETK
/interface bridge
add admin-mac=D4:01:C3:0C:19:28 auto-mac=no name=bridge port-cost-mode=short \
protocol-mode=mstp region-name=cngbase vlan-filtering=yes
/interface ethernet
set [ find default-name=ether6 ] comment=ether6 name=cdrop1
set [ find default-name=ether3 ] comment=ether3 name=cdrop2
set [ find default-name=ether1 ] comment=ether1 name=cdrop3
set [ find default-name=ether2 ] comment=ether2 name=cdrop4
set [ find default-name=ether5 ] comment=ether5 name=cdrop5
set [ find default-name=ether4 ] comment=ether4 name=cdrop6
set [ find default-name=ether7 ] disabled=yes
set [ find default-name=ether8 ] comment=ether8 name=sr-wan
set [ find default-name=sfp-sfpplus1 ] comment="sfp-sfpplus1 / cng-chni1-sw1" \
name=uplink1
set [ find default-name=sfp-sfpplus2 ] comment="sfp-sfpplus2 / cng-chni1-sw2" \
name=uplink2
/disk
set usb1 media-interface=none media-sharing=no
/interface ethernet switch port-isolation
set 7 forwarding-override=uplink1
/interface list
add name=lldp-discovery
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/snmp community
set [ find default=yes ] disabled=yes
add addresses=172.24.10.42/32 authentication-protocol=SHA1 name=cmk security=\
authorized
/interface bridge msti
add bridge=bridge identifier=1 vlan-mapping=1-4094
/interface bridge port
add bridge=bridge interface=cdrop3 internal-path-cost=10 path-cost=10
add bridge=bridge interface=cdrop4 internal-path-cost=10 path-cost=10 pvid=64
add bridge=bridge interface=cdrop2 internal-path-cost=10 path-cost=10 pvid=64
add bridge=bridge interface=cdrop6 internal-path-cost=10 path-cost=10 pvid=64
add bridge=bridge interface=cdrop5 internal-path-cost=10 path-cost=10
add bridge=bridge interface=cdrop1 internal-path-cost=10 path-cost=10
add bpdu-guard=yes bridge=bridge edge=yes interface=sr-wan learn=yes \
multicast-router=disabled path-cost=10 point-to-point=no pvid=1099
add bridge=bridge interface=uplink1
add bridge=bridge interface=uplink2
/interface bridge port mst-override
add identifier=1 interface=uplink1 priority=0x40
add identifier=1 interface=uplink2
/ip firewall connection tracking
set udp-timeout=10s
/ip neighbor discovery-settings
set discover-interface-list=lldp-discovery lldp-mac-phy-config=yes \
lldp-max-frame-size=yes lldp-med-net-policy-vlan=1 lldp-vlan-info=yes \
protocol=lldp
/ipv6 settings
set accept-redirects=no accept-router-advertisements=no disable-ipv6=yes
/interface bridge vlan
add bridge=bridge comment="base mgmt" untagged=\
bridge,uplink1,uplink2,cdrop3,cdrop5,cdrop1 vlan-ids=1
add bridge=bridge comment=clients tagged=cdrop1,cdrop3,cdrop5,uplink1,uplink2 \
untagged=cdrop2,cdrop4,cdrop6 vlan-ids=64
add bridge=bridge comment=iot tagged=cdrop5,cdrop1,cdrop3,uplink1,uplink2 \
vlan-ids=80
add bridge=bridge comment=core tagged=uplink1,uplink2,cdrop1 vlan-ids=1001
add bridge=bridge comment=sr-wan tagged=uplink1,uplink2 untagged=sr-wan \
vlan-ids=1099
/interface list member
add interface=uplink1 list=lldp-discovery
add interface=uplink2 list=lldp-discovery
/ip address
add address=172.24.10.15/24 comment=mgmt interface=bridge network=172.24.10.0
/ip dns
set servers=172.24.10.30
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip ipsec profile
set [ find default=yes ] dpd-interval=2m dpd-maximum-failures=5
/ip route
add comment="default route" disabled=no distance=1 dst-address=0.0.0.0/0 \
gateway=172.24.10.1 routing-table=main suppress-hw-offload=no
/snmp
set contact="Kevin Horvat" enabled=yes location=CHNI01
/system identity
set name=cng-chni1-sw5.base.cng.internal
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=172.24.10.30
/tool sniffer
set filter-interface=sr-wan
Problem
Under normal circumstances, the link “uplink1” between cng-chni1-sw5 and cng-chni1-sw1 is selected as the root port of cng-chni1-sw1. Now traffic is forwarded without any problems and server3 can reach server1 and server2 via their respective VLANs.
If now, for any reason, link “uplink1” fails, MSTP should kick in and discover a new Path to cng-chni1-sw1 via cng-chni1-sw2. This also works as expected and “uplink2” is elected as new root path. Forwarding now works for all devices except server2 within VLAN 1099. This server does not get any traffic from any other server nor can it send traffic to any other server unless I disable Hardware offloading on the Bridge Port (sr-wan) to which the server is connected. Then it starts working again and traffic being forwarded as before.
Troubleshooting
I’ve tried to trouble shoot this now for multiple hours and I cannot find where the culprit lies. With the previous switch (the Mikrotik was a drop-in replacement) everything works as expected and failover between “uplink1” and “uplink2” did not cause any prolonged timeouts.
The VLAN forwarding works as expected since all hosts are reachable when “uplink1” is active, so I did rule out any misconfiguration there. Also MSTP works as expected between all three nodes, for all VLANs. Also did a Packet sniff on the sr-wan Bridge Port and it didn’t show any forwarded traffic for when I switched to “uplink2”.
The switch was replaced in order to upgrade Bandwith to 2.5Gbps, thus running in non offloaded mode is not possible (since CPU and switch are only connected via 1.3Gbps).
Has anyone got a clue where the problem lies and what I could further try? I’ve attached the config export of cng-chni1-sw5 below for reference.
Thank you in advance!