RB5009 : all connected ports flap a few times a day

[mainTAP]

Hello,

All the connected ports on my RB5009 running 7.1.1 are flapping a few times a day. They all go down together and come back up after 5 seconds. All the ports are connected to different devices and there is nothing in the logs indicating what could be happening :

 21:47:46 interface,info ether1 link down
 21:47:46 interface,info ether2-trunk link down
 21:47:46 interface,info ether3-trunk link down
 21:47:46 interface,info ether4-trunk link down
 21:47:51 interface,info ether1 link up (speed 1G, full duplex)
 21:47:51 interface,info ether2-trunk link up (speed 1G, full duplex)
 21:47:51 interface,info ether3-trunk link up (speed 1G, full duplex)
 21:47:51 interface,info ether4-trunk link up (speed 1G, full duplex)

Is anyone experiencing the same ?

[jbl42]

There is s similar report here

[fragtion]

Edit: my (now quoted) post below is now basically irrelevant as it hasn’t actually solved the issue for me

I have a support ticket open with MikroTik about this issue for about 7 days and waiting for a response..

In the meanwhile I think I may have managed to figure out what causes the issue (at least for me). Not 100% certain yet (still testing), but basically I think the issue is caused by attaching a VLAN to a bridge, and then bridging that same VLAN to another, different bridge, like so:

/interface/bridge/add name=bridge_lan
/interface/bridge/add name=bridge_guest
/interface/bridge/port/add bridge=bridge_lan interface=ether1
/interface/bridge/port/add bridge=bridge_lan interface=ether2..
/interface/vlan/add name=vlan_guest interface=bridge_lan > (can be any id..)
/interface/bridge/port/add bridge=bridge_guest interface=vlan_guest

As soon as that last rule is added, then the problem is “armed”. Firstly you may see some mysterious dropping of some forwarded packets (eg 30-70% of some pings) in the first bridge. Then randomly all the ports in the same bridge could flap like this.

What’s interesting is that bridge_guest is not configured to do anything yet for me (except have the vlan_guest as a port), it’s just a new/empty bridge. So it makes little sense why those rules would destabilize the router from our perspective. But I think it’s true, because it fixes the dropped pings issue, and it’s been 2 days now and I’ve had no more issues since > disabling any bridged vlan’s which have a bridge as their parent interface

Hoping to hear if you have a similar VLAN configuration somewhere on your RB5009, then we have a pretty good case for what’s causing this issue and MikroTik would surely need to start working on a fix on behalf of all RB5009 owners ASAP..

On the bright side, it would be a huge relief to confirm that this is indeed a software issue and that our hardware, including switch chip and ports are all perfectly fine. Because the RB5009 has been really impressive for me so far, besides for concerns about this matter

[mainTAP]

I also use VLANS , but not in such configuration, there is just one bridge ( with VLAN filtering ) for the ethernet ports themselves with no vlan_interface attached to the bridge.
I don’t think you should add the VLAN interface to the bridge ( https://help.mikrotik.com/docs/display/ROS/Bridge#Bridge-BridgeVLANFiltering / https://help.mikrotik.com/docs/display/ROS/Layer2+misconfiguration#Layer2misconfiguration-VLANinabridgewithaphysicalinterface )

Are you seeing the ports flapping also when you use the same bridge for both VLANs ?

[fragtion]

Just posting back here to say that even with the vlan’s removed, all my ports flapped twice now again. 6 days of hope down the drain.. xD So, clearly there is still an underlying issue causing this problem and the troubleshooting must go on ..

[mainTAP]

I don’t think it is related to the vlans.
Due to its sporadic nature, it’s tricky to correlate it with something, but I have a feeling that it might be even potentially related to Winbox connection . Every time the ports flapped, I was connected to the router with Winbox and amongst other things had a window with interface stats showing data. For last 5 days I haven’t used Winbox and didn’t have any ports flapping during that time, but had another flap a few minutes ago. Also, I had the router rebooted when setting up a sniffer in winbox and winbox crashed and the router rebooted. Moreover, I noticed a weird behaviour where winbox is showing stats for interfaces but doesn’t refresh the values every time .. in 10-second intervals, it refreshes the values 6 times and last 4 seconds the values stay without updating. I saw this happening quite often and also saw the same when connecting to the device using the android app.
Are you using Winbox and could you possibly relate the port flaps to the times when you were connected to the router?

[fragtion]

You could be on to something

See my post here: https://forum.mikrotik.com/viewtopic.php?t=181472&start=300#p904172
Is this what your traffic graphs look like while the issue is happening?

What I find interesting is, the traffic graphs don’t do that all the time. It seems to happen randomly. Also, the port flapping doesn’t seem to happen every time the graphs are doing that. I’m not entirely convinced it is Winbox, but I can say that most of the time I have Winbox session open to my RB5009’s.

For instance, I’ve had winbox open and showing ether traffic for both my RB5009’s for the past hour and everything is looking smooth right now

At least, on the bright side, if it is winbox-triggered after all, then it’s probably software provoked and again we can be optimistic that it’s not a hardware problem. What seems strange, is if it is Winbox-related, then why it isn’t this issue affecting other Mikrotik (or non-mikrotik) hardware also?

Will keep testing with & without Winbox and keep you posted

[mainTAP]

I haven’t checked the graphs, just the values in the interface window. Currently, it’s showing normally but this issue is also sporadic. But I am quite sure my graphs will look similar as it matches what I was seeing in the interface table.
I am also most of the time connected to the device via Winbox, i don’t think it’s flapping the ports just by connecting to it , but every time it flapped I was connected. As I said I haven’t used Winbox for almost a week and there were no flaps, today I had already 4 flaps …

[brg3466]

It seems it is not a single case of port flipping on RB5009, see this thread. http://forum.mikrotik.com/t/rb5009-sfp-10g-optical-copper-dac-w-intel-x520-da1/153233/13

In my case, my sfp+ interface linkdown at midnight, which clearly indicated it has nothing to do with winbox. Other ethernet interface also has port flipping a couple of times every day.

[mainTAP]

Well, now I’m getting quite convinced it has something to do with Winbox as I haven’t used it since the last flaps 3 days ago and just had all the ports flapped again after around 40minutes having Winbox connected.

I’ve also checked the graphs and yes they also have those gaps :

@brg3466

It seems it is not a single case of port flipping on RB5009, …

Yes, but I think that the individual ports flapping could be a separate issue, here we see all of them going down and back up at the same time.

[brg3466]

It is very annoying to see those port flipping info on the log. There is no reply from Mikrotik support yet.

[brg3466@RB5009] > log print where topics~"interface"
 jan/18 19:18:33 interface,info ether1 link down
 jan/18 19:18:33 interface,info ether2 link down
 jan/18 19:18:33 interface,info ether5 link down
 jan/18 19:18:33 interface,info ether6 link down
 jan/18 19:18:33 interface,info ether8 link down
 jan/18 19:18:33 interface,info sfp-sfpplus1 link down
 jan/18 19:18:34 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/18 19:18:38 interface,info ether2 link up (speed 100M, full duplex)
 jan/18 19:18:38 interface,info ether5 link up (speed 1G, full duplex)
 jan/18 19:18:38 interface,info ether6 link up (speed 1G, full duplex)
 jan/18 19:18:38 interface,info ether8 link up (speed 1G, full duplex)
 jan/18 19:18:38 interface,info ether1 link up (speed 2.5G, full duplex)
 jan/18 22:16:33 interface,info ether6 link down
 jan/19 03:10:52 interface,info sfp-sfpplus1 link down
 jan/19 03:10:52 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/19 05:02:32 interface,info ether6 link up (speed 1G, full duplex)
 jan/19 06:10:08 interface,info sfp-sfpplus1 link down
 jan/19 06:10:08 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/19 18:48:51 interface,info ether6 link down
 jan/19 18:49:21 interface,info ether6 link up (speed 1G, full duplex)
 jan/19 18:54:21 interface,info sfp-sfpplus1 link down
 jan/19 18:54:21 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/20 05:46:28 interface,info sfp-sfpplus1 link down
 jan/20 05:46:28 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/21 11:41:12 interface,info ether6 link down
 jan/21 12:01:21 interface,info ether6 link up (speed 1G, full duplex)
 jan/21 12:33:12 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/21 12:33:14 interface,info ether2 link up (speed 100M, full duplex)
 jan/21 12:33:15 interface,info ether6 link up (speed 1G, full duplex)
 jan/21 12:33:15 interface,info ether8 link up (speed 1G, full duplex)
 jan/21 12:33:16 interface,info ether1 link up (speed 2.5G, full duplex)
 jan/21 12:33:16 interface,info ether5 link up (speed 1G, full duplex)
 jan/21 12:33:27 interface,info ether5 link down
 jan/21 12:33:30 interface,info ether5 link up (speed 1G, full duplex)
 jan/21 22:54:27 interface,info ether6 link down
 jan/22 00:01:23 interface,info ether6 link up (speed 1G, full duplex)
 jan/22 15:32:17 interface,info sfp-sfpplus1 link down
 jan/22 15:32:17 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/22 20:49:48 interface,info sfp-sfpplus1 link down
 jan/22 20:49:48 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 02:39:04 interface,info sfp-sfpplus1 link down
 jan/23 02:39:05 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 12:32:07 interface,info ether6 link down
 jan/23 12:53:21 interface,info ether6 link up (speed 1G, full duplex)
 jan/23 16:52:11 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 16:52:11 interface,info WG11 link up
 jan/23 16:52:11 interface,info WG12 link up
 jan/23 16:52:12 interface,info WG14 link up
 jan/23 16:52:12 interface,info WG15 link up
 jan/23 16:52:12 interface,info WG17 link up
 jan/23 16:52:21 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 16:52:22 interface,info ether2 link up (speed 100M, full duplex)
 jan/23 16:52:24 interface,info ether6 link up (speed 1G, full duplex)
 jan/23 16:52:24 interface,info ether8 link up (speed 1G, full duplex)
 jan/23 16:52:25 interface,info ether1 link up (speed 2.5G, full duplex)
 jan/23 16:52:25 interface,info ether5 link up (speed 1G, full duplex)
 jan/23 16:52:33 interface,info ether5 link down
 jan/23 16:52:36 interface,info ether5 link up (speed 1G, full duplex)
 jan/23 16:55:48 interface,info sfp-sfpplus1 link down
 jan/23 16:55:48 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 16:57:13 interface,info sfp-sfpplus1 link down
 jan/23 16:57:13 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 17:59:46 interface,info ether6 link down
 jan/23 18:08:29 interface,info ether6 link up (speed 1G, full duplex)
 jan/23 18:34:29 interface,info ether6 link down
 jan/23 19:29:22 interface,info ether6 link up (speed 1G, full duplex)
 jan/23 19:29:42 interface,info ether6 link down
 jan/23 19:33:21 interface,info ether6 link up (speed 1G, full duplex)
 jan/23 23:12:13 interface,info ether6 link down
 jan/24 06:27:15 interface,info ether8 link down
 jan/24 06:27:22 interface,info ether8 link up (speed 1G, full duplex)
 jan/24 06:27:38 interface,info ether8 link down
 jan/24 06:27:42 interface,info ether8 link up (speed 1G, full duplex)
 jan/24 13:04:27 interface,info ether6 link up (speed 1G, full duplex)
 jan/24 14:01:51 interface,info ether6 link down
 jan/24 21:43:00 interface,info ether6 link up (speed 1G, full duplex)
 jan/24 21:48:28 interface,info ether6 link down
 jan/24 21:48:38 interface,info ether6 link up (speed 1G, full duplex)
 jan/24 22:03:13 interface,info ether6 link down
 jan/24 22:03:15 interface,info ether6 link up (speed 1G, full duplex)
 jan/24 23:35:04 interface,info ether6 link down
 jan/24 23:39:28 interface,info ether6 link up (speed 1G, full duplex)
 jan/24 23:39:42 interface,info ether6 link down
 jan/24 23:43:28 interface,info ether6 link up (speed 1G, full duplex)
 jan/25 03:39:02 interface,info sfp-sfpplus1 link down
 jan/25 03:39:02 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/25 04:46:39 interface,info sfp-sfpplus1 link down
 jan/25 04:46:39 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/25 17:09:44 interface,info sfp-sfpplus1 link down
 jan/25 17:09:44 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/25 22:39:54 interface,info ether2 link down
 jan/25 22:41:01 interface,info ether2 link up (speed 100M, full duplex)
 jan/25 22:41:23 interface,info ether2 link down
 jan/25 22:41:32 interface,info ether2 link up (speed 100M, full duplex)
 jan/25 22:41:35 interface,info ether2 link down
 jan/25 22:41:41 interface,info ether2 link up (speed 100M, full duplex)
 jan/25 22:42:02 interface,info ether2 link down
 jan/25 22:42:09 interface,info ether2 link up (speed 100M, full duplex)
 jan/25 22:46:53 interface,info ether2 link down
 jan/25 22:46:57 interface,info ether2 link up (speed 100M, full duplex)
 jan/25 22:47:19 interface,info ether2 link down
 jan/25 22:47:27 interface,info ether2 link up (speed 100M, full duplex)
 jan/26 11:25:35 interface,info ether6 link down
 jan/26 12:03:15 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 14:18:37 interface,info ether7 link up (speed 1G, full duplex)
 jan/26 14:18:38 interface,info ether7 link down
 jan/26 14:18:41 interface,info ether7 link up (speed 1G, full duplex)
 jan/26 14:18:42 interface,info ether7 link down
 jan/26 14:18:46 interface,info ether7 link up (speed 1G, full duplex)
 jan/26 14:34:48 interface,info ether6 link down
 jan/26 17:57:48 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 18:11:28 interface,info ether7 link down
 jan/26 18:12:22 interface,info ether7 link up (speed 1G, full duplex)
 jan/26 18:12:23 interface,info ether7 link down
 jan/26 18:12:31 interface,info ether7 link up (speed 1G, full duplex)
 jan/26 18:19:34 interface,info ether7 link down
 jan/26 18:26:01 interface,info ether6 link down
 jan/26 20:36:28 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 20:52:20 interface,info ether6 link down
 jan/26 20:55:11 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 21:02:35 interface,info ether6 link down
 jan/26 21:03:27 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 21:20:07 interface,info ether6 link down
 jan/26 22:06:42 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 22:53:56 interface,info ether6 link down
 jan/26 23:31:27 interface,info sfp-sfpplus1 link down
 jan/26 23:31:27 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/27 04:31:19 interface,info sfp-sfpplus1 link down
 jan/27 04:31:19 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/27 13:06:16 interface,info ether6 link up (speed 1G, full duplex)
 jan/27 20:15:21 interface,info ether6 link down
 jan/27 21:42:50 interface,info ether6 link up (speed 1G, full duplex)
 jan/27 22:40:23 interface,info ether6 link down
 jan/27 22:40:27 interface,info ether6 link up (speed 1G, full duplex)
 jan/27 22:42:07 interface,info ether6 link down
 jan/27 22:45:50 interface,info ether6 link up (speed 1G, full duplex)
 jan/27 22:45:59 interface,info ether6 link down
 jan/27 22:46:23 interface,info ether6 link up (speed 1G, full duplex)
 jan/28 04:25:01 interface,info sfp-sfpplus1 link down
 jan/28 04:25:01 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/28 12:47:55 interface,info ether6 link down
 jan/28 15:21:50 interface,info ether6 link up (speed 1G, full duplex)
 jan/28 19:16:24 interface,info sfp-sfpplus1 link down
 jan/28 19:16:24 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/29 13:32:46 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 13:32:48 interface,info ether7 link down
 jan/29 13:32:51 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 13:32:53 interface,info ether7 link down
 jan/29 13:32:56 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 16:23:00 interface,info sfp-sfpplus1 link down
 jan/29 16:23:00 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/29 16:26:21 interface,info sfp-sfpplus1 link down
 jan/29 16:26:21 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/29 18:09:15 interface,info ether7 link down
 jan/29 18:09:41 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 18:09:42 interface,info ether7 link down
 jan/29 18:09:50 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 20:39:18 interface,info ether7 link down
 jan/29 20:39:43 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 20:39:44 interface,info ether7 link down
 jan/29 20:39:49 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 22:39:49 interface,info ether6 link down
 03:01:51 interface,info ether6 link up (speed 1G, full duplex)

[mainTAP]

I think I might be getting closer …
Other several days with no Winbox connected = no flaps.
A few days with Winbox connected, but showing only the interface table = no flaps and graphs look ok.
As soon as I add DHCP server / lease table = gaps started to appear on graphs immediately and all ports flapped after around 2.5 hours.

[fragtion]

You’re right. Having the DHCP server → Lease table showing, is causing it. This makes sense as I have DHCP server running on both my RB5009’s and often have the leases window open. Lately I haven’t been working on any network config and surprisingly everything has been stable for nearly 2 weeks (with or without winbox running). I’m reluctant to get my hopes up over here BUT you may have just found the problem !!!

Here’s a screen capture of me reproducing the issue: https://youtu.be/A7CE2Q67pg8

I will leave the Leases window open and see if I get another flap of connected ports (it’s been several days now without)

[jbl42]

A few days with Winbox connected, but showing only the interface table = no flaps and graphs look ok.
As soon as I add DHCP server / lease table = gaps started to appear on graphs immediately and all ports flapped after around 2.5 hours.

This. I can reproduce this on a RB5009 running 7.1.1: Having Winbox connected and leave the the DHCP lease Window open, all ports flapped after about 50 minutes. I mostly use the CLI and never had all-port-flaps in the logs before.
Looking at the port LEDs while it happens and regarding the fact all ports flap together, it seems like the switch chip is getting reseted somehow.

The occasional single-port flaps reported by some userers are most likely problems of specific link-pair combinations and not directly related. This is more something like the single port flap issues of the past on RB3011 and SFP of RB4011.
The interesting question is why MT devices in general tends to have flap issues. Except in one case were a port on a Zyxel AP was defective, alle the flaps we experienced had a MT port in the link and were the fault of the MT device fixed with later updates. And we use a lot of gear of different brands.

[gabacho4]

I can reproduce it on my end as well. Curiously though if I run a ping to an external website from my laptop and then monitor the ping times while the ports are flapping, I don’t see a change in response time. You’d think I would be losing packets or, at the very least, experiencing an increase in latency no? Perhaps this is a cosmetic issue or have you all actually seen performance hits as a result?

[gabacho4]

I also notice that when the flap occurs, my CPU use goes from 1% up to 9% or so. I ran tools/profile and “management” is the process that seems to be gobbling the extra cycles. “Unclassified” also seems to jump a little. No idea what what that actually means but it correlates precisely with every flapping event. Once I move off of the leases tab, the flapping stops and I stop experiencing the sudden spikes in CPU use.

EDIT: Using ROS 7.2rc3

[mainTAP]

Perhaps this is a cosmetic issue or have you all actually seen performance hits as a result?

I am definitely getting a few seconds outage.

[fragtion]

OK, My ports just flapped again for the first time in several days because of this discovery. I enabled DHCP lease window at around 21:20 my time. And now around 22:45 I just had ports flap issue.
Seems pretty definitive now !! Nice work @mainTAP

[dimacbz]

v7.1.1 RB5009 Link SFP+ <—> any device with SFP (SFP+) packet loss 35-50%, regardless of the load. Tested on two RB5009.
v6.49.2 CRS328 Link SFP+ <—> any device with SFP (SFP+) packet loss 0%, regardless of the load.

Everyone has this problem with packet loss on the port SFP+?

[fragtion]

Hi all.

Just advising that this problem is also triggered whenever viewing the “IP → ARP List” Window ..

Also, this is not a Winbox problem, as the issue occurs even if you access these menus from Webfig.

Regards