RB5009 SFP+ Flapping on HP Switch

dandrzejewski · March 5, 2022, 12:20am

I’ve got an RB5009 connected via its SFP+ port to an SFP+ on my HP E3800. Occasionally and seemingly randomly, I see messages on the HP switch as follows - this sometimes happens a few times in an hour, sometimes it’s a few hours between.

I 03/04/22 17:37:06.29 00076 ports: port 51 is now on-line
I 03/04/22 17:37:05.77 00077 ports: port 51 is now off-line

Nothing appears in the log on the Mikrotik side about the port going up or down. Around this time, I’ll see a tiny amount of packet loss and latency in Smokeping.

I’ve tried swapping the transceivers. I swapped the cable. I even swapped the 850nm transceievers and cable out with an HP DAC. No change. I also changed ports in the HP and the problem followed. Obviously I can’t change ports in the Mikrotik because it only has the one SFP+.

HP Switch is the latest available firmware, Mikrotik was tried on both 7.1.3 and 7.2rc4.

Here’s an export of everything that I think is relevant here, let me know if there’s anything else I should attach. (Also, please don’t judge me for any misconfigurations, this is my home lab and I’m not a network engineer!)

/interface bridge
add comment="admit only vlan-tagged, this includes VLAN 1.  Not sure what this does on the bridge itself." frame-types=admit-only-vlan-tagged igmp-snooping=yes igmp-version=3 multicast-router=disabled name=bridge1 protocol-mode=none vlan-filtering=yes
/interface vlan
add interface=bridge1 name=bridge1-vlan1 vlan-id=1
add interface=bridge1 name=bridge1-vlan50 vlan-id=50
add interface=bridge1 name=bridge1-vlan101 vlan-id=101
add interface=bridge1 name=bridge1-vlan102 vlan-id=102
add interface=bridge1 name=bridge1-vlan103 vlan-id=103
add interface=bridge1 name=bridge1-vlan104 vlan-id=104
/interface ethernet switch port
set 0 mirror-ingress-target=ether8
/interface bridge port
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
add bridge=bridge1 interface=ether4
add bridge=bridge1 interface=ether5
add bridge=bridge1 interface=ether6
add bridge=bridge1 interface=ether7
add bridge=bridge1 interface=ether8
add bridge=bridge1 comment="Remember, PVID is IGNORED because \"admit only VLAN tagged\" is selected. VLAN 1 is tagged on this port!!!" frame-types=admit-only-vlan-tagged interface=sfp-sfpplus1
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,sfp-sfpplus1 vlan-ids=101
add bridge=bridge1 tagged=bridge1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,sfp-sfpplus1 vlan-ids=102
add bridge=bridge1 tagged=bridge1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,sfp-sfpplus1 vlan-ids=103
add bridge=bridge1 tagged=bridge1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,sfp-sfpplus1 vlan-ids=104
add bridge=bridge1 tagged=bridge1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,sfp-sfpplus1 vlan-ids=50
add bridge=bridge1 tagged=bridge1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,sfp-sfpplus1 vlan-ids=500
add bridge=bridge1 comment="This is needed because switch-to-switch is TRUNK - meaning no PVID, only tagged vlans are admitted.  Any \"admit all\" with PVID 1 will automatically be placed in \"current untagged\"" tagged=sfp-sfpplus1,bridge1 vlan-ids=1

Zacharias · March 5, 2022, 10:05am

Well, first i will comment on your configuration…
VLAN-id=1 should not be used in your configuration…
Read here : https://help.mikrotik.com/docs/display/ROS/VLAN
" The IEEE 802.1Q standard has reserved VLAN IDs with special use cases, the following VLAN IDs should not be used in generic VLAN setups: 0, 1, 4095"
Source: link above…

Did you test other SFP modules ?

jbl42 · March 5, 2022, 1:56pm

VLAN-id=1 should not be used in your configuration…
Read here : > https://help.mikrotik.com/docs/display/ROS/VLAN
" The IEEE 802.1Q standard has reserved VLAN IDs with special use cases, the following VLAN IDs should not be used in generic VLAN setups: 0, 1, 4095"
Source: link above…

@zacharias
This is not true. I have a copy of the offcial IEEE 802.1Q standard here. There are 2 reserved VLAN IDs: 0 (0x0000) is reserved for cases were only the priority is used, but not VLAN. 4095 (0xffff) is reserved for implemetation specific usage and shall not be forwarded by switches/bridges. VLAN IDs 1-4094 are alle perfect valid VLAN IDs.
But MikroTik (and many other vendors) use VLAN1 internaly for untagged traffic on bridges, this is why you see a dynamic VLAN with ID1 in MikroTik bridges in certain setups. But this is because MT imlemented it that way, and not part of the standard.

@dandrzejewski
MT still struggles with confirming it, but RB5009 SFP+ port has flapping problems with certain other devices. But those devices all work with anything else than MT. So the problem is on the MT side. We did a lot of experiments on this, don’t let them fool you.
RB4011 had similar flapping issues back when it was new. MT also denied it for months, until it finally got fixed with on of the ROS 6.48 update.

So for now, you have 3 possibilities:

Find a DAC or SFP module not flapping
Put a dumb 10GB switch inbetween. Not ideal, but often helps
Wait for a ROS 7 update fixing the RB5009 SFP+ flapping problems

For some unknown reasons, MT devices tend to have flapping problems. We had it with RB3011, RB4011 (both fixed with ROS updates in the meantime) and RB5009 (not fixed yet).

dandrzejewski · March 5, 2022, 6:41pm

Thanks for the replies everyone!

@zacharias - I haven’t seen any indication that VLAN 1 is being used dynamically for anything in my particular setup, and I haven’t seen any indication of any problems with that VLAN in my network. That said, at some point I’ll change it to a different VLAN ID since the documentation recommends not using it.

@jbl42 - I appreciate your insights. I will have to think about what to do next - this is my home network so it’s not mission-critical or anything, and I rarely if ever come close to saturating that 10gb link, so maybe I’ll either move that link to a 1GB port, or maybe I’ll LACP a few gigabit ports since I have plenty available on the switch it’s connected to. At least for now, until I see fixes listed in the ROS release notes.

I also have been looking at purchasing a small (maybe 8 port) 10GB switch - my HP only has 4 10GB ports and I’d like a few more.

And as I said originally, I tried it with a Mikrotik SFP+ module and an HP DAC… but if it’s highly dependent on the SFP module, I do have a spare 10Gtek SFP+ module that I could try just to see.

Zacharias · March 7, 2022, 4:57pm

@jbl42, VLANs 0 and 4095 are reserved and VLAN with id 1 is the default VLAN used by MikroTik and should not be used…
It is explained here https://en.wikipedia.org/wiki/IEEE_802.1Q

jbl42 · March 7, 2022, 9:43pm

@jbl42, VLANs 0 and 4095 are reserved and VLAN with id 1 is the default VLAN used by MikroTik and should not be used…
It is explained here > https://en.wikipedia.org/wiki/IEEE_802.1Q

Yep. VLAN1 is used by MT implementation as default PVID for all ports, same as for many other vendors too. But in official IEEE_802.1Q standard, it is not treated any special.
ROS bridges create a dynamic VLAN for every bridge port PVID not having a matching static VLAN configured.
If PVID of the bridge itself and all bridge (and where applicable switch) ports is set to something else than 1, no dynamic VLAN1 is created. So ID1 can be used for static VLANs same as any other ID.
Still it is good advice not to use 1 for static VLANs to avoid confusion and hard-to-track L2 misconfigs. No disagreement on that.

We also derailed the original topic quite a lot. To be or not to be of VLAN1 is hardly connected to the OP’s SFP+ flaps.