SFP+ Port Flapping (RB5009 + CRS328-24P-4S+RM)

RouterOS General
1

I have the CRS328-24P-4S+RM connected to the RB5009 using the XS+DA0001 cable and it is disconnecting. I tried other DAC cables and the problem was similar. Tried also disabled “Auto negotiation”.

RB5009 I use eth2 port as WAN
RouterOS: 7.13.4
SwOS: 2.13

 09:33:42 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 09:33:55 interface,info sfp-sfpplus1 link down
 09:33:57 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 09:37:58 interface,info sfp-sfpplus1 link down
 09:37:58 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 10:14:05 interface,info sfp-sfpplus1 link down
 10:14:06 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 10:23:42 interface,info sfp-sfpplus1 link down
 10:23:42 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 10:35:45 interface,info sfp-sfpplus1 link down
 10:35:45 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 10:38:45 interface,info sfp-sfpplus1 link down
 10:38:46 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 10:51:02 interface,info sfp-sfpplus1 link down
 10:51:02 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 10:54:11 interface,info sfp-sfpplus1 link down
 10:54:11 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 10:58:46 interface,info sfp-sfpplus1 link down
 10:58:46 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 11:20:11 interface,info sfp-sfpplus1 link down
 11:20:11 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 11:32:58 interface,info sfp-sfpplus1 link down
 11:32:58 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 11:35:48 interface,info sfp-sfpplus1 link down
 11:35:48 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 11:44:49 interface,info sfp-sfpplus1 link down
 11:44:49 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 12:09:42 interface,info sfp-sfpplus1 link down
 12:09:42 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 12:15:33 interface,info sfp-sfpplus1 link down
 12:15:33 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 12:24:30 interface,info sfp-sfpplus1 link down
 12:24:30 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 12:53:57 interface,info sfp-sfpplus1 link down
 12:53:57 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 13:04:03 interface,info sfp-sfpplus1 link down
 13:04:03 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 13:04:33 interface,info sfp-sfpplus1 link down
 13:04:33 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 13:07:48 interface,info sfp-sfpplus1 link down
 13:07:49 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
2

I have had the same issue for months, even when swapping out with optics. My support case was closed “cannot reproduce”. This didn’t happen with my old RB4011.

3

Same after upgrade.

RouterOS: 7.14
SwOS: 2.16

 04:35:30 interface,info sfp-sfpplus1 link down
 04:35:30 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 04:41:42 interface,info sfp-sfpplus1 link down
 04:41:42 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 04:51:57 interface,info sfp-sfpplus1 link down
 04:51:57 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 05:04:12 interface,info sfp-sfpplus1 link down
 05:04:12 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 05:10:23 interface,info sfp-sfpplus1 link down
 05:10:23 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 05:20:21 interface,info sfp-sfpplus1 link down
 05:20:21 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 05:27:18 interface,info sfp-sfpplus1 link down
 05:27:18 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 05:28:34 interface,info sfp-sfpplus1 link down
 05:28:34 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 05:29:42 interface,info sfp-sfpplus1 link down
 05:29:42 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 05:34:30 interface,info sfp-sfpplus1 link down
 05:34:30 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 05:41:44 interface,info sfp-sfpplus1 link down
 05:41:44 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 05:43:30 interface,info sfp-sfpplus1 link down
 05:43:30 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 05:51:35 interface,info sfp-sfpplus1 link down
 05:51:35 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 06:04:03 interface,info sfp-sfpplus1 link down
 06:04:03 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 06:07:25 interface,info sfp-sfpplus1 link down
4

Don’t know if this provides more info: can you activate debug logging?
Have you tried swapping power supply?

5

I’m experiencing same issue between same devices (RB5009<->CRS328-24P-4S+RM). Is there any update or discovered resolution to that problem?

6

I have the same two devices connected with short OM3 optics, and prior to that with a short DAC. No port flapping at all.
MT devices on RouterOS 7.18.2

OM3 optics: (0.5m)
Vendor name: XZSNET
Vendor part number: XZS-SFP10G-SR
Vendor Revision: A

DAC: (0.5m)
Vendor name: 10GTEK
Vendor part number: CAB-10GSFP-P0.5M-30
Vendor Revision: 01

Happy to help if I can..

7

Do you have any special (not default) setup of your sfp? I was able to minimize numer of sfp errors to zero by turning of auto-negotiation, turning off TX/RX flow control and setting rate select to low. Not sure if this setup if actually resolving this problems or only hide it.

8

All default. Auto-Negotiation… tx/rx flow control off.. SFP rate select High.

Wonder if there was some SFP stability improvements since whatever RouterOS version you have… I’m on 7.18.2.

9

I’m on that version as well. 7.18.2

10

I’m experiencing a similar issue, but with a CRS310-8G+2S+IN. I posted here http://forum.mikrotik.com/t/crs310-8g-2s-in-sfp-auto-negotiation-flapping/183953/1 I have yet to restore the stability we previously had in 7.17.2 or lower, as anything newer breaks auto-negotiation on the SPF+ ports. I have a couple workarounds, but would much prefer these ports auto-negotiate and I never look at them…

11

Same issue with RB5009UPr+S+, CRS328-24P-4S+RM & S+DA0001. ROS 7.19.3. Auto-negotiate or fixed to 10G doesn’t matter.

12

Did you find a solution? I have the same configuration (5009+328+DA001) and the same port flapping problems. If a setup the speed to 5Gb, then the flapping seldom happens. When I set it to 2.5Gb, there’s no flapping at all, but I’d rather have it running at 10Gb…
So changing the DA cable to another brand is not a solution?

13

I’ve seen similar problems (not with RB5009) between MT SFP+ ports but I was using FiberStore (FS) DAC cables. Not had an issue myself with MT or the cables that Baltic Networks sells branded as Maxxwave.

Is the RB5009 the same as the RB4011 in the sense the SFP+ port is CPU controlled instead of the switch chip?

14

Nope, RB5009 has its SFP+ port connected to switch chip ...

15

In the end, I needed to add a CRS309-1G-8S+, and I connected the CRS328-24P-4S+ to it. I’m using the latest version, and after adding this switch, the problem disappeared.

The cable between the router and the main switch is still the same XS+DA0001.

16

RB5009 SFP+ ports are switch chip controlled, not CPU like RB4011. Flapping often comes from incompatible DAC cables. Adding a CRS309-1G-8S+ between RB5009 and CRS328 can fix the issue.