RB3011 port flopping - bad design

soooc · Sun Dec 17, 2017 2:20 pm

Hello,

do you have trouble with port flopping? Here is "answer" from support.This "trouble" is writen nowhere in specification. RB3011 is CRAP and Mikrotik makes bad device.

The reason for port flapping is that you have different kind of devices connected to the same switch group. When you have 1Gbps devices connected to the same switch group with 100Mbps devices, the switch buffer needs to be cleared before forwarding traffic to the next port, but when you have mixed speeds it can't keep up clearing out the buffers so it flops the port. This is a RB3011 switch chip specification, there isn't much you can do, you must use the same speed Ethernet devices to avoid this.

whitbread · Sun Dec 17, 2017 9:52 pm

wtf - the longer you read about experience with new devices the longer you tend to keep good old RB2011's, but with every second power failure I loose one ot those either.

sup5 · Sun Dec 17, 2017 10:59 pm

That's the worst explanation of egress buffer overflow and port flapping I ever read.

So at least this should be mitigatable by employing flow-control.

A switch must *never* drop the Layer-1 connectivity, when having issues handling large loads of traffic. So this simply is a sign of resignation over the chosen design.

Cha0s · Mon Dec 18, 2017 4:38 pm

I confirm the issue. I've never experienced the port flappings everyone mentions since RB3011's release.
I now know why. I didn't mix Fast ethernet and Gigabit devices on the same switch group.

I just tried it and indeed after I maxed out a fast ethernet all ports flapped.

Steveocee · Tue Dec 19, 2017 5:01 pm

Oh dear, this is a major slip up on MT behalf. The RB3011 really did have a bright future but then had a huge amount of teething problems when launched (probably due to new arm architecture) and then there was the "loop detected" issue as well.

I wasn't aware of this however am now a little dubious of this as shortly I had planned on connecting 3 10/100 devices to an RB3011 in a remote location.

Nothing quite says buy a CCR like your existing model failing.....

haik01 · Tue Dec 26, 2017 9:50 pm

Is it a possibility to put a dumb 10/100 switch after the RB3011? I know, it should not be like that.

Is going back to RB2011 an option? I never had this problem, mixed Gbit and fast ethernet.

And still, how do YOU control the users equipment? Today all users have "old laptops", with 100 Mbit NIC's. Tomorrow one guy buys a new laptop and that one has Gigabit connection. So what, after the new laptop he get's worse performance or reliability? Strange.

Tue Dec 26, 2017 10:25 pm

I wonder if other RouterBOARD models with the same switch-chip model (QCA8337) suffer from the same problem. According to this wiki page the models in question are: hAP ac, OmniTIK 5 ac (including OmniTIK 5 ac PoE), the old hEX model (RB750Gr2), hEX PoE and PowerBox Pro.

soooc · Fri Dec 29, 2017 6:06 pm

From support

This is not an issue, this is how this specific switch chip works.

I don't know any other switch with L1 disable and enable during buffer overload. I think, there is something bad in ROS a RB3011.

@normis ?

Cha0s · Fri Dec 29, 2017 6:20 pm

I don't know any other switch with L1 disable and enable during buffer overload. I think, there is something bad in ROS a RB3011.

Me neither.

This is either a bad software design on Mikrotik's part, or bad device design by Mikrotik for using such a switch chip (if indeed the issue is there - any qualcomm/atheros datasheet or documentation pdf confirming this would answer this question).

Either way, this is Mikrotik's fault IMHO. And even worse, they haven't posted any announcement on this, so people are still buying a faulty device (yes I consider this a fault since it doesn't work as advertised).

I wonder if this is the reason that RB3011 was pretty much left on it's own and we never saw any other models after the RM one. Did they realized they messed up and simply dropped the whole series?

Does anyone know any other devices (routerboards or not) using this specific switch chip? I wonder if we can independently reproduce Mikrotik's claims.

Fri Dec 29, 2017 10:14 pm

Does anyone know any other devices (routerboards or not) using this specific switch chip? I wonder if we can independently reproduce Mikrotik's claims.

I have listed other RouterBOARDs where the same switch chip is used several posts above.
They seem to be: hAP ac, OmniTIK 5 ac (including OmniTIK 5 ac PoE), the old hEX model (RB750Gr2), hEX PoE and PowerBox Pro.

Cha0s · Fri Dec 29, 2017 10:16 pm

Does anyone know any other devices (routerboards or not) using this specific switch chip? I wonder if we can independently reproduce Mikrotik's claims.
I have listed other RouterBOARDs where the same switch chip is used several posts above.
They seem to be: hAP ac, OmniTIK 5 ac (including OmniTIK 5 ac PoE), the old hEX model (RB750Gr2), hEX PoE and PowerBox Pro.

Oops I totally missed your post!

I don't have any of these devices to check it out.

Steveocee · Thu Mar 01, 2018 1:49 pm

It looks like I may now be suffering from this. Have taken relevant screenshot and supout files and sent to support but reading back over RB3011 port issues it looks like I am now falling foul of this.

Sheriff1972 · Fri Mar 02, 2018 10:26 pm

i am looking forward to an update on this one.

Steveocee · Sat Mar 03, 2018 4:01 pm

i am looking forward to an update on this one.

There won’t be one. Simply don’t mix 10/100 and 10/100/1000 on the same switch group. I’m going to uplink to a CRS112 and use that for for non gigabit devices.

gotsprings · Thu Mar 08, 2018 7:31 pm

10/100 modem plugged into Port1 going to cause a problem for ports 2-5?

Steveocee · Thu Mar 08, 2018 7:46 pm

10/100 modem plugged into Port1 going to cause a problem for ports 2-5?

Probably yes. May be better grabbing an SFP to RJ and using that if you have it spare.

gotsprings · Fri Mar 09, 2018 11:19 am

The one I see regularly...
Plugging an Xbox directly into the 3011... Something about the network card on those things.

Fri Mar 09, 2018 2:28 pm

Hello, the discussed RB3011 issue is not hardware related, despite the quoted email in the first post.
We have discovered that it tends to occur due to software after several day uptime, therefore it is harder to reproduce and debug it. Despite all that, we are working on it and plan to apply fixes in upcoming RouterOS versions.

Sheriff1972 · Fri Mar 09, 2018 3:06 pm

Thank you for the update. The fix will be applied to current and RC software release channels?

Steveocee · Fri Mar 09, 2018 10:34 pm

Hello, the discussed RB3011 issue is not hardware related, despite the quoted email in the first post.
We have discovered that it tends to occur due to software after several day uptime, therefore it is harder to reproduce and debug it. Despite all that, we are working on it and plan to apply fixes in upcoming RouterOS versions.

Will make me very happy if that is the case.

honzam · Sat Mar 10, 2018 12:45 pm

Hello, the discussed RB3011 issue is not hardware related, despite the quoted email in the first post.
We have discovered that it tends to occur due to software after several day uptime, therefore it is harder to reproduce and debug it. Despite all that, we are working on it and plan to apply fixes in upcoming RouterOS versions.

Very good news!!
It's great that you wrote it publicly on the forum.
Thanks

nescafe2002 · Sat Mar 10, 2018 1:58 pm

We have discovered that it tends to occur due to software after several day uptime

Does this mean that a scheduled daily reboot will fix this problem for now?

gotsprings · Sat Mar 10, 2018 5:33 pm

We have discovered that it tends to occur due to software after several day uptime
Does this mean that a scheduled daily reboot will fix this problem for now?

Nothing scream hack GARBAGE like "requiring daily reboots".
PLEASE DON'T MAKE ME ADD DAILY REBOOTS TO MY SYSTEMS.

soooc · Sun Mar 11, 2018 1:24 pm

Now It starts flopping on 2 Hex Poe - same switch chipset. Traffic over Hex is about 200 Mbit and there is one 100 Mbit port. I will migrate it to 1Gbit, but it is user router and it spends more money. Will new release fix Hex Poe?

Steveocee · Tue Mar 20, 2018 4:55 pm

Good news!!!

"Hello,

We have repeated the RB3011 problem, determined that the cause is purely software related and now we are working on fixes which should be done soon.
Thank you for reporting.

Best regards,
Janis B."

Love that MikroTik actual take fault reports seriously!

Sheriff1972 · Thu Apr 12, 2018 9:53 am

Do we know how soon is soon?

Sheriff1972 · Sat Apr 28, 2018 7:43 am

I guess not...!

soooc · Thu May 24, 2018 9:30 pm

Any progress?

axc27 · Sun Jun 03, 2018 4:25 pm

Any news on the progress?

My problem with port flopping on the hAP ac might be related to the same problem. Therefore, I am looking foreward to news on the topic...

/axc27

Sheriff1972 · Tue Jun 19, 2018 8:27 pm

Hello All,

Just checking to see if this is being looked into anymore?

It's been a while

jhardy · Wed Jun 20, 2018 1:51 am

Still hoping for word of a solution. I've found that instead of rebooting the whole switch, I can just bounce the offending interface interface and that clears it up, at least for awhile.

Sheriff1972 · Wed Jun 20, 2018 9:13 am

Well that is something.

It has been very quiet on this for a long time, I had just wondered if the team had given up.

sjoukes · Wed Jul 11, 2018 5:13 pm

Any updates on this topic?

gotsprings · Wed Jul 11, 2018 7:01 pm

Good news!!!

"Hello,

We have repeated the RB3011 problem, determined that the cause is purely software related and now we are working on fixes which should be done soon.
Thank you for reporting.

Best regards,
Janis B."

Love that MikroTik actual take fault reports seriously!

SONY
Soon Only Not Yet

huntermic · Thu Jul 12, 2018 12:10 pm

Probably something that isn't easy to fix so i guess they will eventually say it will be fixed in version 7, that's another way of saying it won't be fixed thsi century

gutoloureiro · Fri Jul 20, 2018 5:33 pm

Is it a possibility to put a dumb 10/100 switch after the RB3011? I know, it should not be like that.

Is going back to RB2011 an option? I never had this problem, mixed Gbit and fast ethernet.

And still, how do YOU control the users equipment? Today all users have "old laptops", with 100 Mbit NIC's. Tomorrow one guy buys a new laptop and that one has Gigabit connection. So what, after the new laptop he get's worse performance or reliability? Strange.

Hello, I have purchased a RB3011 and migrated an RB2011, copy all settings via the terminal. However, when I put the RB3011 in the air, the port that reaches my link (ETH1), is in constant UP / DOWN. Due to this situation, I put the rb2011 back and this error does not occur.

gotsprings · Sat Jul 21, 2018 7:06 pm

Xbox does that exact thing. Have to put a switch in-between or run wireless.

Sheriff1972 · Sat Jul 21, 2018 7:10 pm

How about they just fix the issue....

gotsprings · Sat Jul 21, 2018 9:27 pm

How about they just fix the issue....

Not sure if the Xbox thing IS the same thing as the port flopping... but have seen this since the 2011s came out.

soooc · Sun Aug 12, 2018 11:06 pm

Any progress?

We stopped buying this HW, last good product in Mikrotiks offer is RB1100AHx4, but its twice expensive

Thu Oct 04, 2018 1:38 pm

Hello, RB3011 Port Flapping problem is addressed by means of a new CPU Flow Control setting in RouterOS v6.43.
If you have experienced this problem, it is recommended to upgrade to the v6.43 and apply following RouterOS command to prevent lockups between RB3011 switch chips and CPU.

/interface ethernet switch set switch1,switch2 cpu-flow-control=no

NathanA · Thu Oct 04, 2018 4:21 pm

Hello, RB3011 Port Flapping problem is addressed by means of a new CPU Flow Control setting in RouterOS v6.43.
If you have experienced this problem, it is recommended to upgrade to the v6.43 and apply following RouterOS command to prevent lockups between RB3011 switch chips and CPU.
Code: Select all
/interface ethernet switch set switch1,switch2 cpu-flow-control=no

`
Is there any downside to enabling this option, or any scenario in which it would be best to leave cpu-flow-control enabled? If not, why did you make it an option instead of just making this a universal change in the way 6.43+ behaves on this hardware?

-- Nathan

honzam · Thu Oct 04, 2018 5:28 pm

Hello, RB3011 Port Flapping problem is addressed by means of a new CPU Flow Control setting in RouterOS v6.43.

Hello. It is fixed since 6.43? Why no info in changelog? No info on forum until now

Thanks

soooc · Mon Oct 08, 2018 3:22 pm

Still not working

3011.png

padrecito007 · Sun Oct 21, 2018 7:36 pm

You're not alone.
We are millions.
We are also millions ignored by mikrotik.
I assure you that the mikrotik company will not do anything.

Miracle · Mon Oct 22, 2018 5:13 am

My RB3011 port 5,6 flapping every day.
I add 2nd power supply over port POE, port flapping gone.

sebastian · Thu Nov 15, 2018 7:57 pm

hello.
We applied to @becs suggestions with RouterOS command and firmware upgrade.

Hello, RB3011 Port Flapping problem is addressed by means of a new CPU Flow Control setting in RouterOS v6.43.
If you have experienced this problem, it is recommended to upgrade to the v6.43 and apply following RouterOS command to prevent lockups between RB3011 switch chips and CPU.
Code: Select all
Code: Select all
/interface ethernet switch set switch1,switch2 cpu-flow-control=no

Problem still persists as you can see in image below.

3011.png

Halesk2k · Fri Nov 16, 2018 9:55 am

I'm not sure to understand everything.

Can someone confirm this happen only with 100Mb/s directly connected devices ?

If my 100Mb/s devices are indirectly connected through a random 1 GB/s switch, there's no issue ?

Many thanks

soooc · Sat Dec 08, 2018 2:59 pm

Any sollution?

Still not working.

/interface ethernet switch set switch1,switch2 cpu-flow-control=no

No effect of this setting!

caspat · Sat Jan 26, 2019 2:35 pm

Is RB4011 have the same behavior?

bon2 · Sat Jun 22, 2019 5:33 am

I have the same or similar problem,
I open a case with MT but they are useless as ashtray on a motorbike,
I have a ROS v 6.44.3 with Intel PRO Ethernet interfaces
The problem appear in few hours and only on LAN interface. (the MT have only two interfaces LAN and WAN)
The traffic is no more of 150Mb/s
If I switch the LAN interface to 100Mb/s the problem didn't appear

The same config on CHR (Cloud Hosted Router) on ESXi host, work well with no problems, with the same cables and same Ethernet cards

Please MT fix this problem,

I am planing to migrate to a different platform if no "FIX" of this port flapping problem

Tue Jun 25, 2019 12:17 pm

Could you please give us your ticket number?

honzam · Tue Jul 02, 2019 6:43 pm

Still not fixed in 6.45.1 with cpu-flow-control=no

[Ticket#2019062722005304]

frogale · Sun Jul 21, 2019 8:39 pm

Same problem here.. RB3011 port 5-10 flaping randomly many times per day.. No 100M and gigabit mix.. Any progress on this?

angriukas · Fri Jul 26, 2019 1:49 pm

Have same problem on CRS326-24G-2S+ and CRS125-24G-1S

Steveocee · Fri Aug 02, 2019 3:56 pm

Have same problem on CRS326-24G-2S+ and CRS125-24G-1S

You have a different problem.

lenciso · Sun Oct 06, 2019 8:13 pm

I also have the same problem and I already realized the suggested corrections. Mikrotik team, I hope they solve this problem, because in the last month I have changed several routers RB2011 to RB3011 and most of them present this problem.

lenciso · Tue Oct 15, 2019 4:23 pm

I also have the same problem and I already realized the suggested corrections. Mikrotik team, I hope they solve this problem, because in the last month I have changed several routers RB2011 to RB3011 and most of them present this problem.

The solution was: Do not mix giga ethernet interfaces with Fast ethernet interfaces in the same switch group.

honzam · Tue Nov 12, 2019 7:41 pm

I also have the same problem and I already realized the suggested corrections. Mikrotik team, I hope they solve this problem, because in the last month I have changed several routers RB2011 to RB3011 and most of them present this problem.
The solution was: Do not mix giga ethernet interfaces with Fast ethernet interfaces in the same switch group.

This is not solution. We need software fix.
Please @Mikrotik

InoX · Tue Nov 12, 2019 10:41 pm

I also have the same problem and I already realized the suggested corrections. Mikrotik team, I hope they solve this problem, because in the last month I have changed several routers RB2011 to RB3011 and most of them present this problem.
The solution was: Do not mix giga ethernet interfaces with Fast ethernet interfaces in the same switch group.

More like an workaround.

gargola · Fri Mar 20, 2020 6:37 pm

RB3011UiAS v6.45.8 all ports in switch 2 flapping constantly, all connections are at 1Gb Mikrotik's response on this looks like a joke.

gargola · Thu Mar 26, 2020 11:47 pm

Good knews short story. At least for me it is fixed with the 6.45.6 in the RB3011UiAS.

Long story I was having 3 RB3011UiAS with the port flapping issues, then I upgraded them to the 6.45.6 and it was fixed. That was several months ago, then I had to replace on of the 3 RB3011UiAS in one site and I installed a new one but with the firmware 6.45.8 (this version is the one I found quickly in the Download section) and that was when my port flapping issue came back in that site. after some days having the problem, I looked at the download archive section and found the same version of my other 2 RB3011UiAS that doesn't has any issue, so I did the downgrade from 6.45.8 to 6.45.6 and problem fixed.

2 days now with my adjacencies up.

Something different should be between the same version but the 6.45.6 is the good one.

tm3558 · Sat May 16, 2020 7:46 pm

In RouterOS version 6.45.9 "long-term" (7 may 2020) there is this fix:

*) system - improved system stability when forwarding traffic from switch chip to CPU (introduced in v6.43);

Anyone tested 6.45.9 yet? Anything changed with the flapping issue?

krafg · Sat May 16, 2020 9:07 pm

In older posts I see that PowerBox Pro have the same processor that RB2011 and probably can be affected.

I use PowerBox Pro since ROS 6.40, then upgraded to 6.45, then 6.46 and now 6.46.6 and I never had problems.

It's a little fact.

Regards.

TheSirStumfy · Sat Jun 06, 2020 9:22 am

The problem has started for me in 6.47.

Never had problems before.

Capture.JPG

jvanhambelgium · Sat Jun 06, 2020 9:40 am

Same here! Since upgrade to 6.47 on my RB3011.
I've generated supout.rif and forwarded it to Mikrotik.

In my case, it seems to be ports ether3 (1Gits/s, Unify AP groundfloor) and ether5 (1Gits/s, some D-LINK 8-port small switch connected on the other end on a floor)seeing transitions, but ether5 much more then port 3

TheSirStumfy · Sat Jun 06, 2020 9:58 am

Same here! Since upgrade to 6.47 on my RB3011.
I've generated supout.rif and forwarded it to Mikrotik.

In my case, it seems to be ports ether3 (1Gits/s, Unify AP groundfloor) and ether5 (1Gits/s, some D-LINK 8-port small switch connected on the other end on a floor)seeing transitions, but ether5 much more then port 3

BTW jvanhambelgium, do you perhaps have any SFP modules in? Im fishing for a fix here

In the mean time i will try the "FIX" by moving the 100M links to my switch, to see if that has any effect.

jvanhambelgium · Sat Jun 06, 2020 10:10 am

Same here! Since upgrade to 6.47 on my RB3011.
I've generated supout.rif and forwarded it to Mikrotik.

In my case, it seems to be ports ether3 (1Gits/s, Unify AP groundfloor) and ether5 (1Gits/s, some D-LINK 8-port small switch connected on the other end on a floor)seeing transitions, but ether5 much more then port 3
BTW jvanhambelgium, do you perhaps have any SFP modules in? Im fishing for a fix here

In the mean time i will try the "FIX" by moving the 100M links to my switch, to see if that has any effect.

No SFP in use here.
My other "switch" module (ports 6-10) contains also a mix of 100 / 1G ports and I do not experience any transitions with that!
The first switch-fabric with ports 1-5 has also 100 + 1G mix.
Not sure if I can take some 100M ports out of the affected switch-modules as a "workaround" but I'll check.

jvanhambelgium · Sat Jun 06, 2020 10:46 am

I've grouped some ports together now, the first switch-module (1-5) now only has 1Gbits/s clients.
At present no more transitions/flappings. Will evaluate over some time.

The other port-group (6-10) now contains some 100Mbit/s but also still 1 client with 1Gbits/s link (= ISP modem). However I don't see any flappings on that switch-block.

So let's keep an eye on it for now...

TheSirStumfy · Sat Jun 06, 2020 10:53 am

Im moving off 100M clients to an external switch 1 by 1 to try to see if this is somehow specific port related.

My flapps have been very spread out (over hours) so testing will take some time.

But if your config stays stable jvanhambelgium (since you had problems on fabric 1-5) it could be safe to assume the bug was reintroduced in 6.47?

jvanhambelgium · Sat Jun 06, 2020 10:57 am

But if your config stays stable jvanhambelgium (since you had problems on fabric 1-5) it could be safe to assume the bug was reintroduced in 6.47?

I think so,I've never seen this happening actually. I did run a couple of versions behind, so I went from 6.44 or something straight to 6.47
I'll be evaluating also further today if anything pops up in the logs.

jvanhambelgium · Sat Jun 06, 2020 1:41 pm

Nope, did not really fix it

I've seen some transitions again ... in the last hour.

TheSirStumfy · Sat Jun 06, 2020 7:06 pm

After I moved off the AP on port 9 it stopped flapping for the whole day now..

Will continue to monitor, but still this would need to get checked out by MT.

TheSirStumfy · Sun Jun 07, 2020 6:30 pm

Now happening also on a all 1G switch, so the "do not mix speeds" workaround does not seem to work.

Capture.JPG

krafg · Sun Jun 07, 2020 7:40 pm

So downgrade to an older version until it will be fixed.

Regards.

rextended · Mon Jun 08, 2020 6:00 pm

I have one on my office "RouterBOARD 3011UiAS" from 9 Jul 2016, and is still working flawlessly (factory is 6.35.3, now have 6.44.6)

[admin@XXX XXX] /interface ethernet switch> export  verbose 
# jun/08/2020 17:12:34 by RouterOS 6.44.6
# software id = XXXX-XXXX
#
# model = RouterBOARD 3011UiAS
# serial number = 78XD0XDX49CX
/interface ethernet switch
set 0 cpu-flow-control=yes mirror-source=none mirror-target=none name=switch1
set 1 cpu-flow-control=yes mirror-source=none mirror-target=none name=switch2
/interface ethernet switch port
set 0 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 1 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 2 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 3 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 4 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 5 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 6 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 7 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 8 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 9 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 10 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 11 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled

jvanhambelgium · Mon Jun 08, 2020 6:19 pm

I have one on my office "RouterBOARD 3011UiAS" from 9 Jul 2016, and is still working flawlessly (factory is 6.35.3, now have 6.44.6)

Mine was working fine with 6.44.x too, but some days ago I moved to the latest 6.47 stable.
Today I was asked by Mikrotik support to make some config-adaption on my RB3011 which I performed this morning at around 11AM
I have to say, the last 24 have been rather quite on "link down / link up" messages. I did have 1 flap on a downstair Wireless AP. around noon and a few around 5PM.

6/8/20 5:04:11.000 PM interface,info MikroTik: ether3 link down
6/8/20 5:04:03.000 PM interface,info MikroTik: ether3 link down
6/8/20 5:03:58.000 PM interface,info MikroTik: ether3 link down
6/8/20 5:03:56.000 PM interface,info MikroTik: ether3 link down
6/8/20 12:29:20.000 PM interface,info MikroTik: ether3 link down

So let's wait a bit more. I don't seem to have really impact from these at the user level.

TheSirStumfy · Mon Jun 08, 2020 11:41 pm

I have one on my office "RouterBOARD 3011UiAS" from 9 Jul 2016, and is still working flawlessly (factory is 6.35.3, now have 6.44.6)
Mine was working fine with 6.44.x too, but some days ago I moved to the latest 6.47 stable.
Today I was asked by Mikrotik support to make some config-adaption on my RB3011 which I performed this morning at around 11AM
I have to say, the last 24 have been rather quite on "link down / link up" messages. I did have 1 flap on a downstair Wireless AP. around noon and a few around 5PM.

6/8/20 5:04:11.000 PM interface,info MikroTik: ether3 link down
6/8/20 5:04:03.000 PM interface,info MikroTik: ether3 link down
6/8/20 5:03:58.000 PM interface,info MikroTik: ether3 link down
6/8/20 5:03:56.000 PM interface,info MikroTik: ether3 link down
6/8/20 12:29:20.000 PM interface,info MikroTik: ether3 link down

So let's wait a bit more. I don't seem to have really impact from these at the user level.

what was the config adoption? Id love to give it a try as well.

jvanhambelgium · Tue Jun 09, 2020 12:21 am

what was the config adoption? Id love to give it a try as well.
I was asked to disable flow-control on the CPU and performed following on my 3011

/interface ethernet switch
set 0 cpu-flow-control=no name="Switch 1"
set 1 cpu-flow-control=no name="Switch 2"

Although today it seems in general the amount of flapping is pretty low, they did pop up on the interface servicing the wireless AP when running a 600-second "iperf3" test with my phone.
In the iperf3-test you clearly see (I do per-second stats) 2 seconds of 0-transfer and then it picks up again.

TheSirStumfy · Tue Jun 09, 2020 8:31 am

Ok thank you, will give it a try.

I did notice a correlation between flapping and CPU usage.
During down time (at night) i get almost 0 flaps, and at high CPU usage times (mostly VPN clients) during day time, the flaps return.

jvanhambelgium · Tue Jun 09, 2020 7:34 pm

True, I've performed a downgrade back to where I came from at release 6.46.4
I had a today some annoying drops between a linked switch causing glitching in video-conf calls etc.
So let's evaluate how 6.46.4 does ..... and then perhaps upgrade step-by-step to 6.46.5 , then 6.46.6 etc to see where this started...

I've also reverted the change on the "cpu-flow-control" that I was asked to perform by Mikrotik support.

jvanhambelgium · Tue Jun 09, 2020 8:27 pm

Now I'm completely clueless ... even now the issues remain present ... so on 6.46.4 the flapping also occurs, although very limited so far.
Apparently I don't have enough data in my Splunk to go very far back in time to see when these messages first started to appear...

TheSirStumfy · Tue Jun 09, 2020 10:17 pm

Double post ignore this one

TheSirStumfy · Tue Jun 09, 2020 10:21 pm

For me at least for now the flapping has stopped (for today) after the flow control disable. Need to monitor this for more days.

I save logs for 4 months only, but i remembered i have a server on the router since 2019 - 06.
So i went to check on those logs and now see that this has been happening all this time. Rarer than now, used to be around once every 2-3 days but still.
I did not notice this on the server since its in a bonded interface, and since flaps usually occur on one switch at a time, link never went down completely.

This was supposed to be fixed in 6.36 4 years ago, but is clearly still an issue.

TheSirStumfy · Wed Jun 10, 2020 7:43 am

Now I'm completely clueless ... even now the issues remain present ... so on 6.46.4 the flapping also occurs, although very limited so far.
Apparently I don't have enough data in my Splunk to go very far back in time to see when these messages first started to appear...

Ok 24h of no flapps after disabling CPU flow control even after heavy client VPN usage (CPU was constantly around 30-50% usage) .

Probably the best thing to do now is to stay on 6.47 since the version states "switch - correctly enable and disable CPU Flow Control on RB3011UiAS;" and see if this will ever be fixed properly.

jvanhambelgium · Wed Jun 10, 2020 10:26 am

This morning I've been hammering the "ether1" port which had a lot of flapping yesterday with traffic ... strangely enough while yesterday I have 15 flaps / hour today all seems rather silent...
Currently I'm running 6.46.4 and I've disabled the flow-control with the command suggested by Mikrotik support.

Flags: I - invalid
0 name="Switch 1" type=QCA-8337 mirror-source=none mirror-target=none cpu-flow-control=no
1 name="Switch 2" type=QCA-8337 mirror-source=none mirror-target=none cpu-flow-control=no

So really, I don't understand this and it might explain why it deemed so difficult for Mikrotik to have a confirmed/solid fix in place for this one.

Later today I am going to upgrade back to 4.47 again

and re-evaluate.

Pawels · Thu Jul 16, 2020 4:32 pm

Later today I am going to upgrade back to 4.47 again and re-evaluate.

Hello - any results?
How does it work now after the upgrade to 4.47?

TheSirStumfy · Mon Jul 27, 2020 9:49 pm

For me the combination of 6.47 and disabled CPU flow control on both switch chips worked.

jvanhambelgium · Mon Jul 27, 2020 10:21 pm

Later today I am going to upgrade back to 4.47 again and re-evaluate.
Hello - any results?
How does it work now after the upgrade to 4.47?

Nope, I'm running 4.46.6 "testing release" for quite some time now (25 days), "hardly" any portflaps anymore.
Not completely free of flaps, but difficult to pinpoint who is to blame : mikrotik or the LAN-switch on the other end..

aletrevizoli · Sun Aug 09, 2020 3:38 am

I have around 12 RB3011 - 6.45.9 version
no one have port flap except one
this one have three interface with high traffic first one eth5, second eth8 and eth9 is my trunk (so eth5 and eth8 goes through there).
All eth6 to eth10 goes down simultaneously and after 2 seconds goes up again. It's happen around 3 or 4 times per day.
Eth1 to Eth5 never goes down.
Most strange is I have others RB3011 that have more traffic and never showed this behavior.
I check firmware of all, and all are 6.45.9 too.
I tried right now o cpu-flow-control=no command....I will monitor it for some hours to check if this resolve this issue.
I tried earlier to exchange this rb3011 thinking that could be an switch chip problem, but not resolved.

aletrevizoli · Tue Aug 11, 2020 4:09 pm

Complementing the previous information.
cpu-flow-control=no make worst port flapping on 6.45.9.
What I did that resolve my problem:
I revert to cpu-flow-control=yes.
I upgrade firmware and software to 6.47.1
I checked that /tools graphing had some interfaces configured, deleted all
I disabled packages, mpls, wireless, hotspot, that I'm not using.
I balanced interfaces with high traffic between the two switch chips, trunk in first(eth1) and second with more high traffic on (eth9).
My SFP was not in use, so it was disabled before, I enable it, just a kick.

Two days now without any problem.

voxter · Wed Sep 09, 2020 1:24 am

Im curious for the people who were asking about SFP's - I have a brand new RB3011UiAS-RM that replaced a 2011.

I'm having port flapping constantly but i _am_ using an SFP which I believe is bound to switch2? Im not able to confirm this for sure yet.

Anyways, the SFP behavior is: It will display the SFP module (GPON SFP) information/details, then it will establish link. It does DHCP to the ISP then after traffic starts to route over it, the link goes down, as do all my ports on ethernet 1-5 .. then they come back after a few seconds

This is on firmware 6.47.3
I've tried disabling CPU flow control on both switch chips, that didn't seem to help at all.

Any advice for SFP users with this problem? I am currently about to downgrade to 6.45.9 as it seemed to be working for other users. Very frustrating.

TheSirStumfy · Wed Sep 09, 2020 8:53 am

I rechecked my logs regarding the issue, and even with CPU flow control off on latest FW/OS I still get flops on SC1.
Much more rare now (weeks instead of multi per day) but still.

I do have a SFP connection too.

Just seems to me this will never be fixed.
CRS devices seem to get constant fixes in every patch, but this issue on the RB series seems to be forgotten.

---Last two flops follow---

2020-09-02 09:39:38 Information daemon ether4 link up (speed 1G, full duplex)
2020-09-02 09:39:38 Information daemon ether2 link up (speed 1G, full duplex)
2020-09-02 09:39:37 Information daemon ether5 link up (speed 1G, full duplex)
2020-09-02 09:39:37 Information daemon ether3 link up (speed 1G, full duplex)
2020-09-02 09:39:37 Information daemon ether1 link up (speed 1G, full duplex)
2020-09-02 09:39:34 Information daemon ether5 link down
2020-09-02 09:39:34 Information daemon ether4 link down
2020-09-02 09:39:34 Information daemon ether3 link down
2020-09-02 09:39:34 Information daemon ether2 link down
2020-09-02 09:39:34 Information daemon ether1 link down
2020-08-19 08:17:47 Information daemon ether2 link up (speed 1G, full duplex)
2020-08-19 08:17:46 Information daemon ether5 link up (speed 1G, full duplex)
2020-08-19 08:17:46 Information daemon ether4 link up (speed 1G, full duplex)
2020-08-19 08:17:46 Information daemon ether3 link up (speed 1G, full duplex)
2020-08-19 08:17:46 Information daemon ether1 link up (speed 1G, full duplex)
2020-08-19 08:17:42 Information daemon ether5 link down
2020-08-19 08:17:42 Information daemon ether4 link down
2020-08-19 08:17:42 Information daemon ether3 link down
2020-08-19 08:17:42 Information daemon ether2 link down
2020-08-19 08:17:42 Information daemon ether1 link down

jvanhambelgium · Wed Sep 09, 2020 9:32 am

Any advice for SFP users with this problem? I am currently about to downgrade to 6.45.9 as it seemed to be working for other users. Very frustrating.

Not using any SFP on my RB3011 but I'm running 6.46.6 (testing) without port-flaps for months now.
I'm pretty sure that if I upgrade to the latest "stable" I will be having troubles again, so I stay on 6.46.6 unless really required to upgrade (eg. serious security issue)

mt99 · Tue Sep 15, 2020 6:03 am

So I wonder if anyone happened to see this in the 6.46.7 (long term) changelog:

*) switch - correctly enable and disable CPU Flow Control on RB3011UiAS;

Apologies if this isn't news, as I typically only pay attention to long term. I also don't want to get anyone's hopes up :)

TheSirStumfy · Tue Sep 15, 2020 8:49 am

So I wonder if anyone happened to see this in the 6.46.7 (long term) changelog:

*) switch - correctly enable and disable CPU Flow Control on RB3011UiAS;

Apologies if this isn't news, as I typically only pay attention to long term. I also don't want to get anyone's hopes up :)

Thanks for that.

Yeah I noted that in my post from Jun 10. Still flops regardless of the setting, though much less with DISABLED.

alyandon · Mon Sep 28, 2020 4:17 pm

Similar issue here on 6.46.7 - I started noticing the port flapping during backup runs where the client and server are both connected to switch 2. After further investigation, I'm able to trigger the port flap on-demand by running iperf between any pair of hosts plugged into switch 2. Disabling cpu flow control seems to help.

b0fh · Thu Dec 24, 2020 1:08 am

After upgrade from 6.47.8 to 6.48, first switch is hardly flopping. Cpu flow control change doesn't resolving problem. On 6.47.8 there were no any even sign of flopping.
Rolled back to 6.47.8, will watch on it.
UPD: rolling back to 6.47.8 seems to be fixed everything.
BTW, in 6.48 notes we have "improved arm stability" item...

Screenshot at Dec 24 00-35-54.png

mikolajm · Tue Jan 05, 2021 4:57 pm

I have just upgraded RB3011 from 6.47.8 to 6.48 and while the previous seemed stable, I've started having the port flopping issue with the latter. Ports 1, 4, 5, 8, and 10 are 1Gbps, 3 and 7 are 100Mbps. Only ports 7, 8 and 10 (switch2) have been flopping so far.
Some weeks ago I also had the flopping problem, which at the time I thought were caused by bad cables, but now I'm starting to think they could have been caused by the issue from this thread.

ragno · Fri Jan 08, 2021 3:30 am

Same problem here... downgrading to 6.46.8 solved the problem.

cazimodo · Wed Jan 13, 2021 10:52 am

Created a user to say this: On version 6.48 I tried running 1Gbps and 100Mbps separately on switch 1 and 2 respectively. No luck. Turned off flow control. No luck. The changes affect things, the flapping patterns change. But flapping continues. I downgraded to 6.47.8 and the flapping is gone.

[Edit] Oh yes forgot. I changed the power supply to a different one. I powered it with really big POE power supply rather than regular power brick. No luck on any of those either.

uCZBpmK6pwoZg7LR · Thu Jan 14, 2021 8:49 pm

In my config started bit better after complete removal All interface rules in Tools -> Graphing. From my side i can tell that it is no relation to 100mb or 1gb
. I have 3 devices which are 100mb only. And links flapped untill i removed Graphing. Most probabaly exist multiply issues

1) Combination 100 and 1000 devices
which possible to solve with cpu-flow-control setting
2) Another issue which with same sympthoms . Which was solved with Tools -> Graphing removal.
At least now i have 4 minutes without flaps. Before had flaps every 30 seconds on 3 ports in same moment.

Firmware latest 6.48

leopardus2 · Fri Jan 15, 2021 1:34 pm

My RB3011 started flapping on ports 6-10 after a reconfiguration, and I'm running 7.1beta3 .
Apparently, OS version doesn't really count anything re. this flapping issue.

I really would like to see some extra debugging about the interfaces. I tried to set debug logging but nothing is printed in logs....

SpartanX · Mon Jan 25, 2021 9:20 pm

Flipping (ha ha) typical!

I was just about to pull the trigger on one of these (literally right now), thinking I had found the perfect router with this one since the switch chips will do hardware VLAN unlike the 4011.

So much for that. Oh well, at least for once I found out about an issue before purchase.

Any news from staff that the issue has been resolved would be greatly appreciated. At least before I buy something else.

jvanhambelgium · Mon Jan 25, 2021 11:20 pm

Flipping (ha ha) typical!

I was just about to pull the trigger on one of these (literally right now), thinking I had found the perfect router with this one since the switch chips will do hardware VLAN unlike the 4011.

So much for that. Oh well, at least for once I found out about an issue before purchase.

Any news from staff that the issue has been resolved would be greatly appreciated. At least before I buy something else.

My RB3011 is running for many many months without any "flap" whatsoever on 6.47.7 (and 6.47.8 would also be fine)
Stay away from 6.48 obviously, probably the worst "stable" RouterOS release I've even seen since years.
I've seen "stable" releases earlier that also had a port-flapping issue, but if you avoid them, the RB3011 is pretty neat box !

pohutukawa · Wed Jan 27, 2021 10:14 pm

Flipping (ha ha) typical!

I was just about to pull the trigger on one of these (literally right now), thinking I had found the perfect router with this one since the switch chips will do hardware VLAN unlike the 4011.

So much for that. Oh well, at least for once I found out about an issue before purchase.

Any news from staff that the issue has been resolved would be greatly appreciated. At least before I buy something else.
My RB3011 is running for many many months without any "flap" whatsoever on 6.47.7 (and 6.47.8 would also be fine)
Stay away from 6.48 obviously, probably the worst "stable" RouterOS release I've even seen since years.
I've seen "stable" releases earlier that also had a port-flapping issue, but if you avoid them, the RB3011 is pretty neat box !

Been seeing something very similar → viewtopic.php?f=2&t=104622&p=841161#p841161

Pohutukawa

richard_s · Wed Jan 27, 2021 11:38 pm

I had a port flapping problem, I turned off the auto negotiate and set the data rate this fixed the problem.

Hope this helps

Richard

leopardus2 · Thu Jan 28, 2021 11:10 am

I wish it was that simple.

TheSirStumfy · Thu Jan 28, 2021 11:40 am

Setting speeds does help a bit I noticed as well, but does not fully fix the problem sadly.

In my observation I could not find a single definitive evidence what would cause the issue. The flaps just seem completely random. They even happen during the night with almost 0 load on the system.

For example last flaps ware:

SWC1: 16.1.2021 8:18
SWC2: 18.1.2021 14:27
SWC2: 25.1.2021 2:01

There seems to be no correlation between other events around the flaps in the logs, Nothing special happening, just BOOB gone for 2 seconds.

Event though MT has stated that this is not a HW issue, I am beginning to doubt it, since it has not gotten a fix in years now.

plani · Fri Jan 29, 2021 12:52 pm

Adding my experience to the port flapping issue on an RB3011 UiAS:

After receiving my RB3011 I updated it by setting it to the stable-branch and going for 6.48. I noticed that the ports on switch2 where randomly going up/down. Connected at that time where ether6 to a Mikrotik CRS as Trunk-Port and ether9, connected to the same CRS as an accesspoint. Assuming that I did a configuration mistake and not really sure if I knew what I was doing I disabled ether9, changed cables etc. and finally, being desperate did a reboot, thinking that a "loop" might break. Well, that worked.

All of a sudden, ether1 and ether2 started flapping two days ago. The only changes done during the time in question was to change routing on the vpn-gateway attached to ether2 and working on access-lists on capsman. The flapping was again very irregular. 2-3 flapps every 10 minutes then again flapping 2-3 times per minute for several minutes. Changing speed and auto-negotiation on ether1 and the connected fritzbox made no difference. Maybe the vpn-gateway has an issue, e.g. faulty usb-ether on the NanoPI R2S? Disconnecting the vpn-gateway connected to ether2 on one side and the fritzbox on the other side made no difference, except that only ether1 flapped. During that time and up to now, no flapping on ether-ports on the second switch-chip.
Sample log looks like this:

	2021-01-29 08:12:53	(no unique labels)	ether2 link down
	2021-01-29 08:12:53	(no unique labels)	ether1 link down
	2021-01-29 08:12:41	(no unique labels)	ether2 link up (speed 1G, full duplex)
	2021-01-29 08:12:40	(no unique labels)	ether1 link up (speed 1G, full duplex)
	2021-01-29 08:12:38	(no unique labels)	ether2 link down
	2021-01-29 08:12:38	(no unique labels)	ether1 link down
	2021-01-29 08:11:58	(no unique labels)	ether2 link up (speed 1G, full duplex)
	2021-01-29 08:11:57	(no unique labels)	ether1 link up (speed 1G, full duplex)
	2021-01-29 08:11:55	(no unique labels)	ether2 link down
	2021-01-29 08:11:55	(no unique labels)	ether1 link down
	2021-01-29 08:11:12	(no unique labels)	ether2 link up (speed 1G, full duplex)
	2021-01-29 08:11:10	(no unique labels)	ether1 link up (speed 1G, full duplex)
	2021-01-29 08:11:08	(no unique labels)	ether2 link down
	2021-01-29 08:11:08	(no unique labels)	ether1 link down

Then I downgraded by setting the update-channel to long-term and "downgraded" to ROS 6.46.8 longterm. Not one up/down over the last 2 hours. Not sure if this just a lucky moment, or rather lucky hours, or if this is now stable. In case flapping comes back, I'll post an update here.

TheSirStumfy · Tue Feb 09, 2021 8:39 am

After 3 days of testing it seems that MT finally fixed this in 6.48.1 as stated in the patch notes.

No flaps now for the mentioned 3 days even under heavy load conditions.

Here's hoping it stays that way.

b0fh · Thu Feb 11, 2021 6:52 pm

6.48.1 works good for me too. 1 day without any flopping

cazimodo · Mon Feb 22, 2021 9:16 am

6.48.1 works for me too. 48 hours without any port flopping

leopardus2 · Mon Feb 22, 2021 11:05 am

need the same fix in 7.x

Who is online