Community discussions

MikroTik App
 
mainTAP
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Tue Oct 02, 2012 4:01 am

RB5009 : all connected ports flap a few times a day

Mon Jan 17, 2022 12:03 am

Hello,

All the connected ports on my RB5009 running 7.1.1 are flapping a few times a day. They all go down together and come back up after 5 seconds. All the ports are connected to different devices and there is nothing in the logs indicating what could be happening :
 21:47:46 interface,info ether1 link down
 21:47:46 interface,info ether2-trunk link down
 21:47:46 interface,info ether3-trunk link down
 21:47:46 interface,info ether4-trunk link down
 21:47:51 interface,info ether1 link up (speed 1G, full duplex)
 21:47:51 interface,info ether2-trunk link up (speed 1G, full duplex)
 21:47:51 interface,info ether3-trunk link up (speed 1G, full duplex)
 21:47:51 interface,info ether4-trunk link up (speed 1G, full duplex)
Is anyone experiencing the same ?
Last edited by mainTAP on Thu Jan 27, 2022 8:44 pm, edited 1 time in total.
 
User avatar
jbl42
Member Candidate
Member Candidate
Posts: 214
Joined: Sun Jun 21, 2020 12:58 pm

Re: RB5009 : all connected ports flap a few times a day

Mon Jan 17, 2022 1:06 am

There is s similar report here
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Mon Jan 17, 2022 4:50 am

Edit: my (now quoted) post below is now basically irrelevant as it hasn't actually solved the issue for me
I have a support ticket open with MikroTik about this issue for about 7 days and waiting for a response..

In the meanwhile I think I may have managed to figure out what causes the issue (at least for me). Not 100% certain yet (still testing), but basically I think the issue is caused by attaching a VLAN to a bridge, and then bridging that same VLAN to another, different bridge, like so:
/interface/bridge/add name=bridge_lan
/interface/bridge/add name=bridge_guest
/interface/bridge/port/add bridge=bridge_lan interface=ether1
/interface/bridge/port/add bridge=bridge_lan interface=ether2..
/interface/vlan/add name=vlan_guest interface=bridge_lan (can be any id..)
/interface/bridge/port/add bridge=bridge_guest interface=vlan_guest
As soon as that last rule is added, then the problem is "armed". Firstly you may see some mysterious dropping of some forwarded packets (eg 30-70% of some pings) in the first bridge. Then randomly all the ports in the same bridge could flap like this.

What's interesting is that bridge_guest is not configured to do anything yet for me (except have the vlan_guest as a port), it's just a new/empty bridge. So it makes little sense why those rules would destabilize the router from our perspective. But I think it's true, because it fixes the dropped pings issue, and it's been 2 days now and I've had no more issues since disabling any bridged vlan's which have a bridge as their parent interface

Hoping to hear if you have a similar VLAN configuration somewhere on your RB5009, then we have a pretty good case for what's causing this issue and MikroTik would surely need to start working on a fix on behalf of all RB5009 owners ASAP..

On the bright side, it would be a huge relief to confirm that this is indeed a software issue and that our hardware, including switch chip and ports are all perfectly fine. Because the RB5009 has been really impressive for me so far, besides for concerns about this matter
Last edited by fragtion on Sat Jan 22, 2022 8:10 pm, edited 1 time in total.
 
mainTAP
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Tue Oct 02, 2012 4:01 am

Re: RB5009 : all connected ports flap a few times a day

Mon Jan 17, 2022 2:35 pm

I also use VLANS , but not in such configuration, there is just one bridge ( with VLAN filtering ) for the ethernet ports themselves with no vlan_interface attached to the bridge.
I don't think you should add the VLAN interface to the bridge ( https://help.mikrotik.com/docs/display/ ... NFiltering / https://help.mikrotik.com/docs/display/ ... linterface )

Are you seeing the ports flapping also when you use the same bridge for both VLANs ?
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Sat Jan 22, 2022 8:14 pm

Just posting back here to say that even with the vlan's removed, all my ports flapped twice now again. 6 days of hope down the drain.. xD So, clearly there is still an underlying issue causing this problem and the troubleshooting must go on ..
 
mainTAP
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Tue Oct 02, 2012 4:01 am

Re: RB5009 : all connected ports flap a few times a day

Tue Jan 25, 2022 12:18 pm

I don't think it is related to the vlans.
Due to its sporadic nature, it's tricky to correlate it with something, but I have a feeling that it might be even potentially related to Winbox connection . Every time the ports flapped, I was connected to the router with Winbox and amongst other things had a window with interface stats showing data. For last 5 days I haven't used Winbox and didn't have any ports flapping during that time, but had another flap a few minutes ago. Also, I had the router rebooted when setting up a sniffer in winbox and winbox crashed and the router rebooted. Moreover, I noticed a weird behaviour where winbox is showing stats for interfaces but doesn't refresh the values every time .. in 10-second intervals, it refreshes the values 6 times and last 4 seconds the values stay without updating. I saw this happening quite often and also saw the same when connecting to the device using the android app.
Are you using Winbox and could you possibly relate the port flaps to the times when you were connected to the router?
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Tue Jan 25, 2022 2:52 pm

You could be on to something

See my post here: viewtopic.php?t=181472&start=300#p904172
Is this what your traffic graphs look like while the issue is happening?

What I find interesting is, the traffic graphs don't do that all the time. It seems to happen randomly. Also, the port flapping doesn't seem to happen every time the graphs are doing that. I'm not entirely convinced it is Winbox, but I can say that most of the time I have Winbox session open to my RB5009's.

For instance, I've had winbox open and showing ether traffic for both my RB5009's for the past hour and everything is looking smooth right now

At least, on the bright side, if it is winbox-triggered after all, then it's probably software provoked and again we can be optimistic that it's not a hardware problem. What seems strange, is if it is Winbox-related, then why it isn't this issue affecting other Mikrotik (or non-mikrotik) hardware also?

Will keep testing with & without Winbox and keep you posted
 
mainTAP
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Tue Oct 02, 2012 4:01 am

Re: RB5009 : all connected ports flap a few times a day

Tue Jan 25, 2022 3:06 pm

I haven't checked the graphs, just the values in the interface window. Currently, it's showing normally but this issue is also sporadic. But I am quite sure my graphs will look similar as it matches what I was seeing in the interface table.
I am also most of the time connected to the device via Winbox, i don't think it's flapping the ports just by connecting to it , but every time it flapped I was connected. As I said I haven't used Winbox for almost a week and there were no flaps, today I had already 4 flaps ...
 
brg3466
Member Candidate
Member Candidate
Posts: 177
Joined: Sat Aug 01, 2015 7:29 am

Re: RB5009 : all connected ports flap a few times a day

Wed Jan 26, 2022 9:05 am

It seems it is not a single case of port flipping on RB5009, see this thread. viewtopic.php?p=907695&hilit=RB5009+aut ... te#p907695

In my case, my sfp+ interface linkdown at midnight, which clearly indicated it has nothing to do with winbox. Other ethernet interface also has port flipping a couple of times every day.
 
mainTAP
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Tue Oct 02, 2012 4:01 am

Re: RB5009 : all connected ports flap a few times a day

Fri Jan 28, 2022 11:59 am

Well, now I'm getting quite convinced it has something to do with Winbox as I haven't used it since the last flaps 3 days ago and just had all the ports flapped again after around 40minutes having Winbox connected.

I've also checked the graphs and yes they also have those gaps :
Image

It seems it is not a single case of port flipping on RB5009, ...
Yes, but I think that the individual ports flapping could be a separate issue, here we see all of them going down and back up at the same time.
 
brg3466
Member Candidate
Member Candidate
Posts: 177
Joined: Sat Aug 01, 2015 7:29 am

Re: RB5009 : all connected ports flap a few times a day

Sun Jan 30, 2022 7:51 pm

It is very annoying to see those port flipping info on the log. There is no reply from Mikrotik support yet.
[brg3466@RB5009] > log print where topics~"interface"
 jan/18 19:18:33 interface,info ether1 link down
 jan/18 19:18:33 interface,info ether2 link down
 jan/18 19:18:33 interface,info ether5 link down
 jan/18 19:18:33 interface,info ether6 link down
 jan/18 19:18:33 interface,info ether8 link down
 jan/18 19:18:33 interface,info sfp-sfpplus1 link down
 jan/18 19:18:34 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/18 19:18:38 interface,info ether2 link up (speed 100M, full duplex)
 jan/18 19:18:38 interface,info ether5 link up (speed 1G, full duplex)
 jan/18 19:18:38 interface,info ether6 link up (speed 1G, full duplex)
 jan/18 19:18:38 interface,info ether8 link up (speed 1G, full duplex)
 jan/18 19:18:38 interface,info ether1 link up (speed 2.5G, full duplex)
 jan/18 22:16:33 interface,info ether6 link down
 jan/19 03:10:52 interface,info sfp-sfpplus1 link down
 jan/19 03:10:52 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/19 05:02:32 interface,info ether6 link up (speed 1G, full duplex)
 jan/19 06:10:08 interface,info sfp-sfpplus1 link down
 jan/19 06:10:08 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/19 18:48:51 interface,info ether6 link down
 jan/19 18:49:21 interface,info ether6 link up (speed 1G, full duplex)
 jan/19 18:54:21 interface,info sfp-sfpplus1 link down
 jan/19 18:54:21 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/20 05:46:28 interface,info sfp-sfpplus1 link down
 jan/20 05:46:28 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/21 11:41:12 interface,info ether6 link down
 jan/21 12:01:21 interface,info ether6 link up (speed 1G, full duplex)
 jan/21 12:33:12 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/21 12:33:14 interface,info ether2 link up (speed 100M, full duplex)
 jan/21 12:33:15 interface,info ether6 link up (speed 1G, full duplex)
 jan/21 12:33:15 interface,info ether8 link up (speed 1G, full duplex)
 jan/21 12:33:16 interface,info ether1 link up (speed 2.5G, full duplex)
 jan/21 12:33:16 interface,info ether5 link up (speed 1G, full duplex)
 jan/21 12:33:27 interface,info ether5 link down
 jan/21 12:33:30 interface,info ether5 link up (speed 1G, full duplex)
 jan/21 22:54:27 interface,info ether6 link down
 jan/22 00:01:23 interface,info ether6 link up (speed 1G, full duplex)
 jan/22 15:32:17 interface,info sfp-sfpplus1 link down
 jan/22 15:32:17 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/22 20:49:48 interface,info sfp-sfpplus1 link down
 jan/22 20:49:48 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 02:39:04 interface,info sfp-sfpplus1 link down
 jan/23 02:39:05 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 12:32:07 interface,info ether6 link down
 jan/23 12:53:21 interface,info ether6 link up (speed 1G, full duplex)
 jan/23 16:52:11 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 16:52:11 interface,info WG11 link up
 jan/23 16:52:11 interface,info WG12 link up
 jan/23 16:52:12 interface,info WG14 link up
 jan/23 16:52:12 interface,info WG15 link up
 jan/23 16:52:12 interface,info WG17 link up
 jan/23 16:52:21 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 16:52:22 interface,info ether2 link up (speed 100M, full duplex)
 jan/23 16:52:24 interface,info ether6 link up (speed 1G, full duplex)
 jan/23 16:52:24 interface,info ether8 link up (speed 1G, full duplex)
 jan/23 16:52:25 interface,info ether1 link up (speed 2.5G, full duplex)
 jan/23 16:52:25 interface,info ether5 link up (speed 1G, full duplex)
 jan/23 16:52:33 interface,info ether5 link down
 jan/23 16:52:36 interface,info ether5 link up (speed 1G, full duplex)
 jan/23 16:55:48 interface,info sfp-sfpplus1 link down
 jan/23 16:55:48 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 16:57:13 interface,info sfp-sfpplus1 link down
 jan/23 16:57:13 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/23 17:59:46 interface,info ether6 link down
 jan/23 18:08:29 interface,info ether6 link up (speed 1G, full duplex)
 jan/23 18:34:29 interface,info ether6 link down
 jan/23 19:29:22 interface,info ether6 link up (speed 1G, full duplex)
 jan/23 19:29:42 interface,info ether6 link down
 jan/23 19:33:21 interface,info ether6 link up (speed 1G, full duplex)
 jan/23 23:12:13 interface,info ether6 link down
 jan/24 06:27:15 interface,info ether8 link down
 jan/24 06:27:22 interface,info ether8 link up (speed 1G, full duplex)
 jan/24 06:27:38 interface,info ether8 link down
 jan/24 06:27:42 interface,info ether8 link up (speed 1G, full duplex)
 jan/24 13:04:27 interface,info ether6 link up (speed 1G, full duplex)
 jan/24 14:01:51 interface,info ether6 link down
 jan/24 21:43:00 interface,info ether6 link up (speed 1G, full duplex)
 jan/24 21:48:28 interface,info ether6 link down
 jan/24 21:48:38 interface,info ether6 link up (speed 1G, full duplex)
 jan/24 22:03:13 interface,info ether6 link down
 jan/24 22:03:15 interface,info ether6 link up (speed 1G, full duplex)
 jan/24 23:35:04 interface,info ether6 link down
 jan/24 23:39:28 interface,info ether6 link up (speed 1G, full duplex)
 jan/24 23:39:42 interface,info ether6 link down
 jan/24 23:43:28 interface,info ether6 link up (speed 1G, full duplex)
 jan/25 03:39:02 interface,info sfp-sfpplus1 link down
 jan/25 03:39:02 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/25 04:46:39 interface,info sfp-sfpplus1 link down
 jan/25 04:46:39 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/25 17:09:44 interface,info sfp-sfpplus1 link down
 jan/25 17:09:44 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/25 22:39:54 interface,info ether2 link down
 jan/25 22:41:01 interface,info ether2 link up (speed 100M, full duplex)
 jan/25 22:41:23 interface,info ether2 link down
 jan/25 22:41:32 interface,info ether2 link up (speed 100M, full duplex)
 jan/25 22:41:35 interface,info ether2 link down
 jan/25 22:41:41 interface,info ether2 link up (speed 100M, full duplex)
 jan/25 22:42:02 interface,info ether2 link down
 jan/25 22:42:09 interface,info ether2 link up (speed 100M, full duplex)
 jan/25 22:46:53 interface,info ether2 link down
 jan/25 22:46:57 interface,info ether2 link up (speed 100M, full duplex)
 jan/25 22:47:19 interface,info ether2 link down
 jan/25 22:47:27 interface,info ether2 link up (speed 100M, full duplex)
 jan/26 11:25:35 interface,info ether6 link down
 jan/26 12:03:15 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 14:18:37 interface,info ether7 link up (speed 1G, full duplex)
 jan/26 14:18:38 interface,info ether7 link down
 jan/26 14:18:41 interface,info ether7 link up (speed 1G, full duplex)
 jan/26 14:18:42 interface,info ether7 link down
 jan/26 14:18:46 interface,info ether7 link up (speed 1G, full duplex)
 jan/26 14:34:48 interface,info ether6 link down
 jan/26 17:57:48 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 18:11:28 interface,info ether7 link down
 jan/26 18:12:22 interface,info ether7 link up (speed 1G, full duplex)
 jan/26 18:12:23 interface,info ether7 link down
 jan/26 18:12:31 interface,info ether7 link up (speed 1G, full duplex)
 jan/26 18:19:34 interface,info ether7 link down
 jan/26 18:26:01 interface,info ether6 link down
 jan/26 20:36:28 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 20:52:20 interface,info ether6 link down
 jan/26 20:55:11 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 21:02:35 interface,info ether6 link down
 jan/26 21:03:27 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 21:20:07 interface,info ether6 link down
 jan/26 22:06:42 interface,info ether6 link up (speed 1G, full duplex)
 jan/26 22:53:56 interface,info ether6 link down
 jan/26 23:31:27 interface,info sfp-sfpplus1 link down
 jan/26 23:31:27 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/27 04:31:19 interface,info sfp-sfpplus1 link down
 jan/27 04:31:19 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/27 13:06:16 interface,info ether6 link up (speed 1G, full duplex)
 jan/27 20:15:21 interface,info ether6 link down
 jan/27 21:42:50 interface,info ether6 link up (speed 1G, full duplex)
 jan/27 22:40:23 interface,info ether6 link down
 jan/27 22:40:27 interface,info ether6 link up (speed 1G, full duplex)
 jan/27 22:42:07 interface,info ether6 link down
 jan/27 22:45:50 interface,info ether6 link up (speed 1G, full duplex)
 jan/27 22:45:59 interface,info ether6 link down
 jan/27 22:46:23 interface,info ether6 link up (speed 1G, full duplex)
 jan/28 04:25:01 interface,info sfp-sfpplus1 link down
 jan/28 04:25:01 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/28 12:47:55 interface,info ether6 link down
 jan/28 15:21:50 interface,info ether6 link up (speed 1G, full duplex)
 jan/28 19:16:24 interface,info sfp-sfpplus1 link down
 jan/28 19:16:24 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/29 13:32:46 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 13:32:48 interface,info ether7 link down
 jan/29 13:32:51 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 13:32:53 interface,info ether7 link down
 jan/29 13:32:56 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 16:23:00 interface,info sfp-sfpplus1 link down
 jan/29 16:23:00 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/29 16:26:21 interface,info sfp-sfpplus1 link down
 jan/29 16:26:21 interface,info sfp-sfpplus1 link up (speed 10G, full duplex)
 jan/29 18:09:15 interface,info ether7 link down
 jan/29 18:09:41 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 18:09:42 interface,info ether7 link down
 jan/29 18:09:50 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 20:39:18 interface,info ether7 link down
 jan/29 20:39:43 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 20:39:44 interface,info ether7 link down
 jan/29 20:39:49 interface,info ether7 link up (speed 1G, full duplex)
 jan/29 22:39:49 interface,info ether6 link down
 03:01:51 interface,info ether6 link up (speed 1G, full duplex)

 
mainTAP
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Tue Oct 02, 2012 4:01 am

Re: RB5009 : all connected ports flap a few times a day

Fri Feb 04, 2022 6:22 pm

I think I might be getting closer ...
Other several days with no Winbox connected = no flaps.
A few days with Winbox connected, but showing only the interface table = no flaps and graphs look ok.
As soon as I add DHCP server / lease table = gaps started to appear on graphs immediately and all ports flapped after around 2.5 hours.
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Fri Feb 04, 2022 8:41 pm

You're right. Having the DHCP server -> Lease table showing, is causing it. This makes sense as I have DHCP server running on both my RB5009's and often have the leases window open. Lately I haven't been working on any network config and surprisingly everything has been stable for nearly 2 weeks (with or without winbox running). I'm reluctant to get my hopes up over here BUT you may have just found the problem !!!

Here's a screen capture of me reproducing the issue: https://youtu.be/A7CE2Q67pg8

I will leave the Leases window open and see if I get another flap of connected ports (it's been several days now without)
 
User avatar
jbl42
Member Candidate
Member Candidate
Posts: 214
Joined: Sun Jun 21, 2020 12:58 pm

Re: RB5009 : all connected ports flap a few times a day

Fri Feb 04, 2022 9:35 pm

A few days with Winbox connected, but showing only the interface table = no flaps and graphs look ok.
As soon as I add DHCP server / lease table = gaps started to appear on graphs immediately and all ports flapped after around 2.5 hours.
This. I can reproduce this on a RB5009 running 7.1.1: Having Winbox connected and leave the the DHCP lease Window open, all ports flapped after about 50 minutes. I mostly use the CLI and never had all-port-flaps in the logs before.
Looking at the port LEDs while it happens and regarding the fact all ports flap together, it seems like the switch chip is getting reseted somehow.

The occasional single-port flaps reported by some userers are most likely problems of specific link-pair combinations and not directly related. This is more something like the single port flap issues of the past on RB3011 and SFP of RB4011.
The interesting question is why MT devices in general tends to have flap issues. Except in one case were a port on a Zyxel AP was defective, alle the flaps we experienced had a MT port in the link and were the fault of the MT device fixed with later updates. And we use a lot of gear of different brands.
 
gabacho4
Member
Member
Posts: 329
Joined: Mon Dec 28, 2020 12:30 pm
Location: Earth

Re: RB5009 : all connected ports flap a few times a day

Fri Feb 04, 2022 9:51 pm

I can reproduce it on my end as well. Curiously though if I run a ping to an external website from my laptop and then monitor the ping times while the ports are flapping, I don't see a change in response time. You'd think I would be losing packets or, at the very least, experiencing an increase in latency no? Perhaps this is a cosmetic issue or have you all actually seen performance hits as a result?
 
gabacho4
Member
Member
Posts: 329
Joined: Mon Dec 28, 2020 12:30 pm
Location: Earth

Re: RB5009 : all connected ports flap a few times a day

Fri Feb 04, 2022 9:59 pm

I also notice that when the flap occurs, my CPU use goes from 1% up to 9% or so. I ran tools/profile and "management" is the process that seems to be gobbling the extra cycles. "Unclassified" also seems to jump a little. No idea what what that actually means but it correlates precisely with every flapping event. Once I move off of the leases tab, the flapping stops and I stop experiencing the sudden spikes in CPU use.

EDIT: Using ROS 7.2rc3
Last edited by gabacho4 on Fri Feb 04, 2022 10:33 pm, edited 1 time in total.
 
mainTAP
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Tue Oct 02, 2012 4:01 am

Re: RB5009 : all connected ports flap a few times a day

Fri Feb 04, 2022 10:12 pm

Perhaps this is a cosmetic issue or have you all actually seen performance hits as a result?
I am definitely getting a few seconds outage.
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Fri Feb 04, 2022 10:54 pm

OK, My ports just flapped again for the first time in several days because of this discovery. I enabled DHCP lease window at around 21:20 my time. And now around 22:45 I just had ports flap issue.
Seems pretty definitive now !! Nice work @mainTAP 👍🏻
You do not have the required permissions to view the files attached to this post.
Last edited by fragtion on Wed Feb 09, 2022 3:42 pm, edited 2 times in total.
 
dimacbz
just joined
Posts: 2
Joined: Sat Jul 02, 2016 11:04 pm

Re: RB5009 : all connected ports flap a few times a day

Sat Feb 05, 2022 10:20 pm

v7.1.1 RB5009 Link SFP+ <---> any device with SFP (SFP+) packet loss 35-50%, regardless of the load. Tested on two RB5009.
v6.49.2 CRS328 Link SFP+ <---> any device with SFP (SFP+) packet loss 0%, regardless of the load.

Everyone has this problem with packet loss on the port SFP+?
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Tue Feb 08, 2022 7:37 pm

Hi all.

Just advising that this problem is also triggered whenever viewing the “IP → ARP List” Window ..

Also, this is not a Winbox problem, as the issue occurs even if you access these menus from Webfig.

Regards
 
User avatar
jbl42
Member Candidate
Member Candidate
Posts: 214
Joined: Sun Jun 21, 2020 12:58 pm

Re: RB5009 : all connected ports flap a few times a day

Wed Feb 09, 2022 2:12 pm

Just advising that this problem is also triggered whenever viewing the “IP → ARP List” Window ..
Also, this is not a Winbox problem, as the issue occurs even if you access these menus from Webfig.
The IP/Arp window has a column "bridge port", same as the DHCP lease windows. This info is coming from the bridge code / switch chip.
I can also reproduce the issue using the monitor command on CLI for ARP or DHCP leases.

It looks like constant polling of this information messes up something in lower layers causing the switch chip to reset, flapping all ports.
Doesn't matter if it is Winbox, Webfig or the CLI monitor.

Another issue probably related:
If I export the config to the CLI, it makes a pause of some seconds with high CPU spike before the /interface/bridge/ports section appears.
Again, getting bridge port info behaves weird.
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Wed Feb 09, 2022 2:56 pm

Just advising that this problem is also triggered whenever viewing the “IP → ARP List” Window ..
Also, this is not a Winbox problem, as the issue occurs even if you access these menus from Webfig.
The IP/Arp window has a column "bridge port", same as the DHCP lease windows. This info is coming from the bridge code / switch chip.
I can also reproduce the issue using the monitor command on CLI for ARP or DHCP leases.

It looks like constant polling of this information messes up something in lower layers causing the switch chip to reset, flapping all ports.
Doesn't matter if it is Winbox, Webfig or the CLI monitor.

Another issue probably related:
If I export the config to the CLI, it makes a pause of some seconds with high CPU spike before the /interface/bridge/ports section appears.
Again, getting bridge port info behaves weird.
Nice find. So for now the best workaround is to disable the "Bridge Port" column in winbox, then we can safely enter DHCP leases & ARP list windows again without disrupting router stability.
 
User avatar
jbl42
Member Candidate
Member Candidate
Posts: 214
Joined: Sun Jun 21, 2020 12:58 pm

Re: RB5009 : all connected ports flap a few times a day

Wed Feb 09, 2022 3:30 pm

So for now the best workaround is to disable the "Bridge Port" column in winbox, then we can safely enter DHCP leases & ARP list windows again without disrupting router stability.
It is not sure the information is not polled when the column is hidden in Winbox.
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Wed Feb 09, 2022 3:39 pm

So for now the best workaround is to disable the "Bridge Port" column in winbox, then we can safely enter DHCP leases & ARP list windows again without disrupting router stability.
It is not sure the information is not polled when the column is hidden in Winbox.
It's not, I confirmed :D
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Thu Feb 10, 2022 12:54 pm

This issue is not fixed yet in v7.1.5 (in case anyone's curious).
Last edited by fragtion on Wed Mar 23, 2022 3:44 pm, edited 2 times in total.
 
MayestroPW
Frequent Visitor
Frequent Visitor
Posts: 52
Joined: Wed Oct 26, 2016 3:28 pm

Re: RB5009 : all connected ports flap a few times a day

Mon Feb 14, 2022 3:32 pm

7.2.rc3 the same
 
brg3466
Member Candidate
Member Candidate
Posts: 177
Joined: Sat Aug 01, 2015 7:29 am

Re: RB5009 : all connected ports flap a few times a day

Mon Feb 14, 2022 7:12 pm

Got reply from mikrotik support :

quote:
Hello,
If you plan to use RouterOS version 7, then we currently suggest v7.2rc2/v7.2rc3.
Those versions contain lots of SFP improvements for CRS3xx devices and could solve this issue as well.
unqutoe
 
mainTAP
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Tue Oct 02, 2012 4:01 am

Re: RB5009 : all connected ports flap a few times a day

Fri Feb 18, 2022 5:46 pm

Got reply from mikrotik support :
thank you, but that doesn't seem to be related to the issue in this thread.

I got a suggestion from mikrotik to change interface queue types and sizes and observe how it affects the issue:
But unfortunately I don't have time to experiment with this right now, so if anyone wants to give it a go :
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Sat Feb 19, 2022 5:36 pm

Got reply from mikrotik support :
thank you, but that doesn't seem to be related to the issue in this thread.

I got a suggestion from mikrotik to change interface queue types and sizes and observe how it affects the issue:
But unfortunately I don't have time to experiment with this right now, so if anyone wants to give it a go :
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

It doesn't work.

One has to wonder to what extent MikroTik tests their own solutions before suggesting them as feedback? For a while now, this issue has been easy to reproduce and affects any/all RB5009's (with or without out-of-the-box config)
Last edited by fragtion on Wed Apr 20, 2022 11:45 pm, edited 1 time in total.
 
gabacho4
Member
Member
Posts: 329
Joined: Mon Dec 28, 2020 12:30 pm
Location: Earth

Re: RB5009 : all connected ports flap a few times a day

Sat Feb 19, 2022 6:11 pm

I too was confused and distressed by the support. They should have plenty of RB5009s available for testing and we’ve very clearly shown that the issue can be duplicated easily. I even sent them a supout that I’m sure they could use to exactly replicate my router setup. Unless they knew that changing the queue characteristics would fix the problem, why are they asking us to do something they themselves can/should test?

This issue doesn’t break my router. I just can’t have certain config windows open. But it’s nuts not to have seen a confirmation that this is indeed a bug that is being seriously addressed.
 
kusmierz
just joined
Posts: 8
Joined: Sun May 09, 2021 9:06 am

Re: RB5009 : all connected ports flap a few times a day

Sat Apr 30, 2022 7:52 pm

anyone have tested 7.3beta37 (2022-Apr-25)?

First line from the changelog:
*) bonding - fixed LACP flapping for RB5009 and CCR2004-16G-2S+ devices;
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Sat Apr 30, 2022 8:02 pm

anyone have tested 7.3beta37 (2022-Apr-25)?

First line from the changelog:
*) bonding - fixed LACP flapping for RB5009 and CCR2004-16G-2S+ devices;
Yes I'm running that version. The "bridge port" issue with rb5009 in arp table or dhcp leases, causing gaps in traffic graph and eventual port flapping, remains unfixed in 7.3beta37
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: RB5009 : all connected ports flap a few times a day

Fri Sep 02, 2022 5:05 pm

Hi guys

Some great news! It looks like this port flapping issue on the rb5009 caused by "bridge port", appears to be finally fixed as of 7.6beta4 (I'm not sure which of the changelog entries addressed this issue, presumably one of the l3hw ones). Please test and see if you can confirm the same on your side :)
 
hoseinkashefi
just joined
Posts: 1
Joined: Wed Jan 11, 2023 7:19 pm

Re: RB5009 : all connected ports flap a few times a day

Wed Jan 11, 2023 7:22 pm

I have the same problem. it flaps every 7 days at the exact time.
ethernet link is a layer 3 connection.
Routerboard 760i / v6.48.6

Who is online

Users browsing this forum: GoogleOther [Bot], inteq, NxtGen [Bot] and 21 guests