Looking for guidance on VLAN, SFP & a Weird TP-Link issue

RouterOS General
1

Good day,

I faced an issue today which in my experience I didn’t know how to diagnose and my solution is half-baked to be honest and I am pretty sure I will face it again, if someone can help me understand what happened and how I avoid it would be great, it is always a learning experience and I am loving it.

I connected my main switch ( crs326-24g-2s+rm ) to an RB5009 sitting in a different room using SFP+ and is acting as a switch, it was connected previously using CAT6, but I needed more bandwidth and thought of future-proofing anyway so went with fiber.

The issue I faced was, after having the SFP modules light up and the 10g light coming up, I added the SFP to the VLAN that I was already using.
but the trunk wasn’t working, I torched the SFP but there was no traffic whatsoever. so I started a speed test and noticed that traffic was going through the tp-link mesh (which I think just had a fallback backhaul to communicate in case of ethernet fail) ..

this is the mesh I am using ( I will change them to hap ax2 and ax3 soon )
https://www.tp-link.com/ae/home-networking/deco/deco-x50/

what I then did was disable the port of the mesh that was connected to the RB5009 and suddenly the SFP port ( trunk ) actually started working and all VLANS restored and working flawlessly.

I then enabled the port of the x50 and got my wifi back up.

this has never happened to me when I was using the CAT6 trunk, I don’t understand why it happened when I switched to SFP, is there any kind of priority for ports?

How do I avoid this issue in the future, as I know if someday electricity goes down, I will face the same issue.

another thing I thought of is if there is a way to delay connecting the ports to the x50s at both ends until the SFP connection is established.

Also I am writing this here, because I know someone WILL face this issue and the mikrotik forum will pop up and hopefully this would help someone in the future.

For reference:
Notes_240705_195525.jpg
Regards!

2

At first glance it sounds like an issue of STP configuration, or absence of STP configuration, or absence of STP support on the Decos. But since the communication did not go back to the mesh link after re-enabling the Deco-facing ports and stayed on the fiber one, I rather suspect the Decos to support STP but use their own mind, but there are still things such an explanation does not cover.

So what does interface bridge monitor [find] and interface bridge port monitor [find] show on both the 326 and the 5009, and what are their complete configurations (complete minus public addreses, usernames and various secret codes, that is)?

3

Just “Disable Mesh” on the two x50
If you want to keep them in mesh mode so as you say they can be backup backhaul; then you will have to (like @sindy says) start playing with STP/RSTP to ensure that the fiber link is a higher STP/RSTP priority than the link over the two x50

4

I have to read more on STP, thank you for pointing it out.

Here is the CRS326 Settings:

# 2024-07-07 14:48:42 by RouterOS 7.15.2
# software id = 9X2M-S4W0
#
# model = CRS326-24G-2S+
# serial number = F5F60FBFDE8C
/interface bridge
add admin-mac=DC:2C:6E:D1:84:9F auto-mac=no comment=defconf name=bridge port-cost-mode=short vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] comment=Router
set [ find default-name=ether2 ] comment="Living Room Switch - RB4011"
set [ find default-name=ether3 ] comment="UNRAID - 999 Tagged"
set [ find default-name=ether4 ] comment=Office-AP
set [ find default-name=ether5 ] comment=5950X
set [ find default-name=ether8 ] comment="To LocalServices Protectli"
set [ find default-name=ether10 ] comment=Printer
set [ find default-name=ether15 ] comment="CCTV - Testing Port - Untagged 999"
set [ find default-name=ether19 ] comment="HIKVISION NVR - Untagged 999"
set [ find default-name=ether20 ] comment="Trunk To Majlis - NEW"
set [ find default-name=ether21 ] comment="Home-Ext-Trunk-Out to Majlis - Not Connected"
set [ find default-name=ether22 ] comment="Home-Ext-Trunk-IN from router"
set [ find default-name=ether23 ] comment="CCTV VLAN Access to CRS318"
set [ find default-name=ether24 ] comment="CCTV VLAN Trunk"
set [ find default-name=sfp-sfpplus1 ] rx-flow-control=auto tx-flow-control=auto
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip smb users
set [ find default=yes ] disabled=yes
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge comment=defconf interface=ether1 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether2 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether3 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether4 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether5 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether6 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether7 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether8 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether9 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether10 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether11 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether12 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether13 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether14 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether15 internal-path-cost=10 path-cost=10 pvid=999
add bridge=bridge comment=defconf interface=ether16 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether17 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether18 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether19 internal-path-cost=10 path-cost=10 pvid=999
add bridge=bridge comment=defconf interface=ether20 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether21 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether22 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether23 internal-path-cost=10 path-cost=10 pvid=999
add bridge=bridge comment=defconf interface=sfp-sfpplus1 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=sfp-sfpplus2 internal-path-cost=10 path-cost=10
add bridge=bridge interface=ether24 internal-path-cost=10 path-cost=10
/ip firewall connection tracking
set udp-timeout=10s
/interface bridge vlan
add bridge=bridge tagged=ether2,ether1,ether20 vlan-ids=666
add bridge=bridge tagged=ether24,bridge,ether22,ether2,ether3,sfp-sfpplus2,sfp-sfpplus1 untagged=ether23,ether19,ether15 vlan-ids=999
add bridge=bridge tagged=ether22,ether21 vlan-ids=555
/ip address
add address=192.168.52.3/24 comment=defconf interface=bridge network=192.168.52.0
/ip dns
set servers=8.8.8.8
/ip firewall filter
add action=drop chain=input comment="drop invalid / untracked for input chain" connection-state=invalid,untracked log=yes
add action=drop chain=forward comment="drop invalid / untracked for forward chain" connection-state=invalid,untracked log=yes
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=192.168.52.1 routing-table=main suppress-hw-offload=no
/ip service
set www address=192.168.152.0/24,192.168.52.0/24
set winbox address=192.168.152.0/24,192.168.52.0/24,192.168.70.0/24
/ip smb shares
set [ find default=yes ] directory=/flash/pub
/system clock
set time-zone-name=Asia/Dubai
/system identity
set name=CRS326
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=192.168.52.1
/system routerboard settings
set boot-os=router-os
/system scheduler
add interval=1d name=dailyb-schedule on-event=dailyb policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=2023-01-23 start-time=03:00:00
/system script
add dont-require-permissions=no name=dailyb owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="/export file=crs326"
/tool romon
set enabled=yes

The RB5009 Settings

# 2024-07-07 14:52:26 by RouterOS 7.15
# software id = 1857-GHLW
#
# model = RB5009UG+S+
# serial number = HFF09833HJK
/interface bridge
add name=bridge port-cost-mode=short vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] comment="2.5G Interface"
set [ find default-name=ether2 ] comment=\
    "VLAN Trunk-Temp ::: Change to SFP+ Later"
set [ find default-name=ether8 ] comment="CCTV_VLAN-999 - To Security AP"
set [ find default-name=sfp-sfpplus1 ] comment=\
    "Future VLAN - Add to Settings now, switch later"
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge interface=ether1 internal-path-cost=10 path-cost=10
add bridge=bridge comment=Trunk interface=ether2 internal-path-cost=10 \
    path-cost=10
add bridge=bridge interface=ether3 internal-path-cost=10 path-cost=10
add bridge=bridge interface=ether4 internal-path-cost=10 path-cost=10
add bridge=bridge interface=ether5 internal-path-cost=10 path-cost=10
add bridge=bridge interface=ether6 internal-path-cost=10 path-cost=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether7 internal-path-cost=10 path-cost=10 pvid=999
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether8 internal-path-cost=10 path-cost=10 pvid=999
add bridge=bridge comment=Trunk interface=sfp-sfpplus1 internal-path-cost=10 \
    path-cost=10
/ip firewall connection tracking
set udp-timeout=10s
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface bridge vlan
add bridge=bridge tagged=ether2,bridge,sfp-sfpplus1 untagged=ether8,ether7 \
    vlan-ids=999
/ip address
add address=192.168.52.4/24 interface=bridge network=192.168.52.0
/ip dns
set servers=192.168.52.50,192.168.52.55
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=192.168.52.1 routing-table=main \
    suppress-hw-offload=no
/system clock
set time-zone-name=Asia/Dubai
/system identity
set name=RB5009-LivingRoom
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=192.168.52.1
/tool romon
set enabled=yes

The output for both Interface bridge monitor [find] & Interface bridge port monitor [find]

CRS326

# 2024-07-07 14:41:35 by RouterOS 7.15.2
# software id = 9X2M-S4W0
#
                     ;;; defconf
                  state: enabled
    current-mac-address: DC:2C:6E:D1:84:9F
            root-bridge: no
         root-bridge-id: 0x8000.08:55:31:9B:20:2C
         root-path-cost: 20
              root-port: ether23
             port-count: 26
  designated-port-count: 18
           fast-forward: no
# 2024-07-07 14:42:05 by RouterOS 7.15.2
# software id = 9X2M-S4W0
#
                 interface: ether1          ether2   ether3          ether4          ether5          ether6          ether7   ether8          ether9          ether10         ether11         ether12  ether13  ether14  ether15  ether16  ether17         ether18         ether19         ether20         ether21         ether22         ether23                  sfp-sfpplus1    sfp-sfpplus2    ether24
                    status: in-bridge       inactive in-bridge       in-bridge       in-bridge       in-bridge       inactive in-bridge       in-bridge       in-bridge       in-bridge       inactive inactive inactive inactive inactive in-bridge       in-bridge       in-bridge       in-bridge       in-bridge       in-bridge       in-bridge                in-bridge       in-bridge       in-bridge
               port-number: 1                        3               4               5               6                        8               9               10              11                                                           17              18              19              20              21              22              23                       25              26              24
                      role: designated-port          designated-port designated-port designated-port designated-port          designated-port designated-port designated-port designated-port                                              designated-port designated-port designated-port designated-port designated-port designated-port root-port                designated-port designated-port designated-port
                 edge-port: yes                      yes             no              yes             yes                      yes             yes             yes             yes                                                          no              yes             yes             no              yes             no              no                       no              yes             yes
       edge-port-discovery: yes                      yes             yes             yes             yes                      yes             yes             yes             yes                                                          yes             yes             yes             yes             yes             yes             yes                      yes             yes             yes
       point-to-point-port: yes                      yes             yes             yes             yes                      yes             yes             yes             yes                                                          yes             yes             yes             yes             yes             yes             yes                      yes             yes             yes
              external-fdb: no                       no              no              no              no                       no              no              no              no                                                           no              no              no              no              no              no              no                       no              no              no
              sending-rstp: yes                      yes             yes             yes             yes                      yes             yes             yes             yes                                                          yes             yes             yes             yes             yes             yes             yes                      yes             yes             yes
                  learning: yes                      yes             yes             yes             yes                      yes             yes             yes             yes                                                          yes             yes             yes             yes             yes             yes             yes                      yes             yes             yes
                forwarding: yes                      yes             yes             yes             yes                      yes             yes             yes             yes                                                          yes             yes             yes             yes             yes             yes             yes                      yes             yes             yes
          actual-path-cost: 10                       10              10              10              10                       10              10              10              10                                                           10              10              10              10              10              10              10                       10              10              10
            root-path-cost:                                                                                                                                                                                                                                                                                                                20                                                       
         designated-bridge:                                                                                                                                                                                                                                                                                                                0x8000.08:55:31:C3:73:25                                 
           designated-cost:                                                                                                                                                                                                                                                                                                                10                                                       
    designated-port-number:                                                                                                                                                                                                                                                                                                                1                                                        
          hw-offload-group: switch1                  switch1         switch1         switch1         switch1                  switch1         switch1         switch1         switch1                                                      switch1         switch1         switch1         switch1         switch1         switch1         switch1                  switch1         switch1         switch1

RB5009

# 2024-07-07 14:39:06 by RouterOS 7.15
# software id = 1857-GHLW
#
                  state: enabled
    current-mac-address: 78:9A:18:CC:D3:3E
            root-bridge: no
         root-bridge-id: 0x8000.08:55:31:9B:20:2C
         root-path-cost: 30
              root-port: sfp-sfpplus1
             port-count: 9
  designated-port-count: 4
           fast-forward: no
# 2024-07-07 14:39:08 by RouterOS 7.15
# software id = 1857-GHLW
#
                 interface: ether1   ether2   ether3          ether4          ether5   ether6          ether7   ether8          sfp-sfpplus1
                    status: inactive inactive in-bridge       in-bridge       inactive in-bridge       inactive in-bridge       in-bridge
               port-number:                   3               4                        6                        8               9
                      role:                   designated-port designated-port          designated-port          designated-port root-port
                 edge-port:                   yes             yes                      yes                      no              no
       edge-port-discovery:                   yes             yes                      yes                      yes             yes
       point-to-point-port:                   yes             yes                      yes                      yes             yes
              external-fdb:                   no              no                       no                       no              no
              sending-rstp:                   yes             yes                      yes                      yes             yes
                  learning:                   yes             yes                      yes                      yes             yes
                forwarding:                   yes             yes                      yes                      yes             yes
          actual-path-cost:                   10              10                       10                       10              10
            root-path-cost:                                                                                                     30
         designated-bridge:                                                                                                     0x8000.DC:2C:6E:D1:84:9F
           designated-cost:                                                                                                     20
    designated-port-number:                                                                                                     25
          hw-offload-group:                   switch1         switch1                  switch1                  switch1         switch1

Thank you both for taking the time to look at it, if something looks wrong please do let me know.

5

As your topology is actually more complex than the one on the drawing, once you read more about STP, you should use the priority setting under /interface/bridge on some devices to explicitly define the primary and secondary root bridges in the topology rather than letting just the MAC addresses determine that - currently, the CRS318 serving the CATV is the root bridge which may not be what you want; however, that is not the reason why the path via the Decos was preferred at some moment, it’s just a generic networking hint.

The path-cost (and, in case of MSTP, also internal-path-cost) settings under /interface/bridge/port are used to control the preference of when multiple L2 paths are available. What you still have to determine is whether the Decos themselves use some kind of STP or some other L2 protocol to prevent L2 loops and optimize the active topology. Removing the ethernet ports to which the Decos are connected from the bridges on both the 316 and the 5009 and sniffing on them should answer this question - if the Decos are not connected to any Mikrotik bridge and you can still see STP BPDUs on the Ethernets, it means that Decos use STP themselves, otherwise they use something proprietary and once they decide to use the wireless path, they start behaving like a piece of cable connecting the port of the 316 with the port of the 5009 from the point of view of STP, just transparently forwarding BPDUs between the Mikrotik devices but ignoring their contents.

6

Apologies for the late response had health related issues.

Thank you for clarifying, I will learn more about STP to expand my knowledge on it on Mikroitk side.

from my research the decos indeed have some STP logic going on, nothing that I can disable as it is very limited interface if used in AP mode. I will switch to Mikrotik APs soon though.

best regards :slight_smile: