Community discussions

MikroTik App
 
JRayfield
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 66
Joined: Tue Aug 19, 2014 8:15 pm

Ethernet Port Flapping on MikroTik Routers

Tue Oct 19, 2021 4:54 am

I've been using MikroTik routers with Mimosa equipment for about 4 years now. An ongoing problem that I've had is port flapping of the ethernet port on the MikroTik that is connected to the Mimosa radio. When possible (not needing over 100 Mbps of speed) I've set the Mimosa radios to 100 Mbps (instead of Auto) and set the MikroTik routers to Auto at 100 Mpbs or 1000 Mpbs. This drastically reduces or eliminates the port flapping problem.

Other people have had the same trouble with MikroTik and Mimosa equipment. Some of these have switched to other brands of routers (such as Ubiquiti) and not had any problems between the Mimosa and these other brands.

It appears to be some kind of incompatibility between the Mimosa and MikroTik equipment. Has anyone come up with a solution to this problem, to get Mimosa and MikroTik equipment to work together at 1 Gbps ethernet speeds?

John Rayfield, Jr.
 
mada3k
Long time Member
Long time Member
Posts: 687
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: Ethernet Port Flapping on MikroTik Routers

Tue Oct 19, 2021 12:05 pm

Are you using long or short cables? Shielded or un-shielded? common ground?

Unfortunally port-flap is common in some Mikrotik equipment. Its probably some hardware issue, maybe variying quality of the Ethernet transformers. Running 100MBps, or sometimes enabling flow-control helps.
 
eddieb
Member
Member
Posts: 322
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Ethernet Port Flapping on MikroTik Routers

Tue Oct 19, 2021 12:36 pm

I must disagree with the statement "common"

I am using dozens of MT devices for more than 5 years and have seen this only once.
AND that was caused by a bad cable between some cheap switch and a MT device ...
 
JRayfield
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 66
Joined: Tue Aug 19, 2014 8:15 pm

Re: Ethernet Port Flapping on MikroTik Routers

Wed Oct 20, 2021 11:07 pm

Thanks for the thoughts on this. In our case, and the case of another WISP (located in Arizona), this problem has been 'common'. In fact, I just had a call from another customer who was complaining of 'slow speed'. I checked the log in the Mimosa C5x and it shows that the ethernet port has been flapping. And that C5x is still set on Auto. Time to set it at 100.

John Rayfield, Jr.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19100
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Ethernet Port Flapping on MikroTik Routers

Wed Oct 20, 2021 11:34 pm

What is common is the non-MT equipment being used and one particular brand. I suspect they are not following ethernet protocols properly.................
 
ConnyMercier
Forum Veteran
Forum Veteran
Posts: 724
Joined: Tue Dec 17, 2019 1:08 pm

Re: Ethernet Port Flapping on MikroTik Routers

Thu Oct 21, 2021 2:50 am

Just look around the Mimosa-Forum....
21-10-_2021_01-46-36.png
You do not have the required permissions to view the files attached to this post.
 
JRayfield
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 66
Joined: Tue Aug 19, 2014 8:15 pm

Re: Ethernet Port Flapping on MikroTik Routers

Sat Oct 23, 2021 3:33 am

First of all, I don't accept just anyone's comments regarding problems with any product.
It's possible that Mimosa has a problem, but the Mimosa units appear to work fine with Ubiquiti Air Cube routers. So no one can claim with 100% certainty that this problem is all Mimosa's.

So Mimosa doesn't work well with MikroTik but works fine with Ubiquiti. And, it appears that Ubiquiti and MikroTik work ok.

Mimosa -- Ubiquiti = OK
Mikrotik -- Ubiquiti = OK
Mimosa -- MikroTik = Bad

If Ubiquiti had a decent wifi product for WISPs, I'd use those. But there's no way that I'll do that.

This sure makes it harder to pin down where the problem really is. An engineer who I know told me that he suspects that the problem is due to a timing difference between Mimosa and MikroTik devices. That would explain why the problem occurs only at the higher speed (1 Gbps). I suspect that he's correct. But so far, no one has been able to get either company to consider that their product might have a problem like that. I guess I need to set up some tests in my lab and see what I can find. I've done testing (mostly software) for large manufacturers before, so I guess it's time to do some more testing.

John
 
jishi
just joined
Posts: 12
Joined: Mon Oct 25, 2021 4:29 pm

Re: Ethernet Port Flapping on MikroTik Routers

Mon Oct 25, 2021 4:53 pm

I have an RB4011 that is connected to a fiber-switch (https://www.ctsystem.com/se/product/pro ... 12&pid=117) that is experiencing port-flapping on the WAN port. So far I've tested the following:

Doesn't help:
Changing LAN cable
Changing WAN port (from ether1 to ether7 to be on a different switch altogether)
Changing switch port (in the HLS3106)
Changing firmware (tried 7.1rc3 and 7.1rc4, and downgraded to 6.49)

What does help:
Lowering negotiation to only allow 100Mbit (but that's not an option since I have 250/250).
Putting a dumb switch between the fiber switch and the RB4011

Rebooting doesn't really help, but if I do a shutdown, and power on (a power cycle basically) it seems to be good for 3-4 days, and then frequency of the flaps increases, to about 5-10 per day.

Given how many reports there is on port flapping, I'd say there is definitely a problem on the mikrotik side, with some sort of tolerance problem.

I have ordered an RJ-45 SFP and I'm gonna try using that for WAN just to see if the flapping stops. But it feels stupid that I have to buy additional hardware to work around this problem, given that I already have way more ports than have use for... also keeping that dump switch running in between just to avoid flapping also seems like 🤯

I've had multiple WAN cables between the fiber-switch and the mikrotik, but only traffic flowing through one of them. It's only the port with traffic that flaps, the other one doesn't seem to flap at all, so it is also related to throughput it seems.
 
JRayfield
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 66
Joined: Tue Aug 19, 2014 8:15 pm

Re: Ethernet Port Flapping on MikroTik Routers

Tue Nov 02, 2021 1:08 am

Thanks for posting this information, Jishi. This might be very useful to me. I've managed to get MikroTik interested in looking into this problem further, since I have been able to duplicate this problem in my lab on the bench. I'm going to try to get some more data on this problem and forward it to MikroTik.

John
 
jishi
just joined
Posts: 12
Joined: Mon Oct 25, 2021 4:29 pm

Re: Ethernet Port Flapping on MikroTik Routers

Tue Nov 16, 2021 6:11 pm

I have been running 6.49 for a while now, and the port flapping is significantly less but still exist. It flaps 1-3 times a day, but sometimes 1-2 days go by without any flapping. Might be kernel related, since 7.x is running a newer kernel, I don't know.

I haven't yet switched to using the SFP, but I will try to do that just to see if it mitigates the flapping altogether.
 
JRayfield
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 66
Joined: Tue Aug 19, 2014 8:15 pm

Re: Ethernet Port Flapping on MikroTik Routers

Tue Nov 16, 2021 6:15 pm

Good info, Jishi. I'll do some testing here in our lab with 6.49 and I might update one router to the newest 7.x firmware to see what happens, too.

Thanks!
John
 
jishi
just joined
Posts: 12
Joined: Mon Oct 25, 2021 4:29 pm

Re: Ethernet Port Flapping on MikroTik Routers

Wed Nov 24, 2021 11:53 pm

Been running it on the SFP (S-RJ01) as WAN now for over a week, and not a single flap has happened. Will report back in a week or two if it is still the same.
 
jishi
just joined
Posts: 12
Joined: Mon Oct 25, 2021 4:29 pm

Re: Ethernet Port Flapping on MikroTik Routers

Wed Dec 15, 2021 1:06 am

A couple of weeks has passed and still no flapping with the SFP module. Even upgraded to 7.1rc7 and still no flapping. Given that I don't get flaps with the mikrotik SFP, I will safely assume that the flapping isn't caused by my fiber switch, but rather the switch in the RB4011.
 
JRayfield
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 66
Joined: Tue Aug 19, 2014 8:15 pm

Re: Ethernet Port Flapping on MikroTik Routers

Wed Dec 15, 2021 4:58 pm

I think you're right, Jishi. I want to set up a test on the bench here in my lab with the latest firmware running on an RB951G with a Mimosa C5x radio and see if the new firmware makes any difference with the ethernet ports.

John
 
jishi
just joined
Posts: 12
Joined: Mon Oct 25, 2021 4:29 pm

Re: Ethernet Port Flapping on MikroTik Routers

Fri Jan 28, 2022 3:27 pm

I'm now running dual "WANs" between the fiberbox and my RB4011, just for fun.

One gets a public IP and the other one is behind CGNAT (so SFP is the public one and ether1 is the CGNATed one). I route different vlans over different gateways, and I have less traffic on ether1, but I still se occasional port flaps. However, anything that is sensitive to the flapping goes over the SFP interface now so I'm less affected by it.

But it is still frustrating that this problem doesn't seem to get any traction, since it seems to affect multiple hardware devices (given all the flapping reporting) and it would be good to hear if this is a software or hardware problem, and that Mikrotik is considering to at least adjust future hardware if it is actually a hardware fault. And if software, at least an attempt at some sort of fix would be appreciated.
 
john231
Frequent Visitor
Frequent Visitor
Posts: 91
Joined: Wed Aug 08, 2018 12:11 pm

Re: Ethernet Port Flapping on MikroTik Routers

Fri Feb 25, 2022 8:40 am

One of the fixes might be to downgrade to 6.48.6 in the long term channel.

https://www.reddit.com/r/mikrotik/comme ... eros_6493/
 
JRayfield
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 66
Joined: Tue Aug 19, 2014 8:15 pm

Re: Ethernet Port Flapping on MikroTik Routers

Fri Feb 25, 2022 5:21 pm

This is interesting with reference to 6.48.6. I'll try that and see what happens. It's pretty easy to duplicate this problem. I really don't understand why MikroTik engineers haven't been able to see this for themselves.

John
 
Gaspode
just joined
Posts: 1
Joined: Thu Apr 14, 2022 11:36 am

Re: Ethernet Port Flapping on MikroTik Routers

Thu Apr 14, 2022 11:48 am

Tangentally related, and posted more for future google search results, but I have a CRS305-1G-4S+IN with FS.com 10GB fibre SFP+ modules, and over the last few days I decided to switch from my previously rock solid RouterOS to SwOs to see what that was like.

With SwOS 2.13p and 2.13, and my typical "endpoints not being powered up exept for a few hours at the end of the day when I get home from work but switch up all day" use case, my fibre ports were flapping (down/up) every few seconds when my TrueNas server with an Intel x520 and FS.com 10 GB fibre SFP+ module, or windows desktop with the same NIC config were powered up.

The flapping gradually slowed down over the course of 10 minutes, but was still pretty regular.

Switching back to RouterOS 6.49.5 and they are all rock solid.

Who is online

Users browsing this forum: No registered users and 94 guests