Hi folks,

After recently having stability problems with 2 switches, both with RouterOS 7.14.2, I noted the following warning in the switch logs:

interface, warning <interface> excessive or late collission, link duplex mismatch?

This warning coincided with the switches starting to drop connectivity. Over time, more and more connections were dropped, until the switches became completely dead.

The culprit here was an optical link using a SFP+ interface in each of the switches. The endpoint was an older switch that only had gigabit capability.From the beginning, the SFP+ interfaces were set to automatic detection of link speed, and getting an active link was problematic. The old 1 Gbit switch had to be rebooted once or twice to connect. After setting a fixed link speed to GbaseX on the SFP+ ports, the problem was solved, and the connections were stable.

When the switches started to ill behave, the rest of the network was also affected, similar to existence of a network loop.

IMHO, this error should result in immediate shut down of the offending port, and not cause switch instability. The error should also be displayed as an error, and not as a warning in the logs.

As the error does not cause immediate problems, but after some time, makes it very hard to track down. One way to immediately provoke the error seems to be sending wake up packets (WOL).

So, please MikroTik, correct this bug in RouterOS. Close the offending port immediately, and log an error, and make sure the switch remains stable.

There should also be a very clear warning in the documentation, that link speed should be set manually if the switch interface has got higher capability, than on the other end of the connection. Or fix the automatic detection, so it works reliably. Automatic detection of link speed does not work even between MikroTik switches, where the two end points have got different capabilities. For example connecting a QSFP+ port with 100 Gbit capability on one switch, to a switch having only 40 Gbit capability with a DAC-cable XQ+DA0001 or XQ+DA0003, wont work. You need to set maximum link speed to 40 GbitCR on the faster switch, otherwise you're out of luck.

Best regards,

Peter

I have a few devices that give me similar errors in ROS logs. The devices use 100Mbps Ethernet link and are pretty old. Aside from these warnings, I see no other issues whatsoever. So they do not affect my network - at least, not to my knowledge.

So I'd not want ROS to block ports if / when a collision occurs. Dropping a 'warning' to the system log (the way it's done now) is the best option, in my opinion.

Hi Experimentator,

I cannot vouch for the problem existing, or not, in earlier versions of RouterOS. All switches I manage are regularly updated to the latest RouterOS version. In this case 7.14.2. At least one switch had 7.14.1 installed, and it also displayed this bad behavior. In any case, the behavior is a serious bug. The switch should always behave well, offending ports shut down, and a log record appended. That would save huge amounts of time when locating problems.

Best regards,

Peter Milesson

Deadpete,

You can restrict the advertised speeds on the link. For example to restrict ether1 to 100M/full or 1G/full, you may use the following.
This could be an alternative to forcing speed/duplex, as this will also disabled MDI-X (applies mostly to wired).

Regarding the effects, that's a bit concerning. If you are certain you have a loop-free topology, then I would look into spanning-tree to see whether there are waves of spanning-tree recalculations when the link misbehaves.

Remember that when you do not use autonegotiation, you have to manually set both speed and duplex mode AT EACH END.
When not setting full duplex, the default is half duplex (for reasons of backward compatibility to 25 years ago), and you will get this warning when the other end is configured for full duplex.
Only when you are sure that the end that experiences this problem is correctly configured for full duplex (or both ends are set to autonegotiate), this is a bug. Otherwise it is just a configuration error.

I cannot vouch for the problem existing, or not, in earlier versions of RouterOS.

Hi Peter,

I just meant that in my case the very same error message may appear on Ethernet ports connected to some legacy (non-Mikrotik) hardware. And in my specific situation these errors do not lead to any noticeable issues on my network. Therefore, I don't think a Mikrotik switch should disable ports if / as it detects a network collision.

I understand your experience is different. I just wanted to say that shutting the ports down just like that may (and likely will) in some cases lead to other issues - i.e. a loss of remote access to the site.

Andrey

Hi vingjfg,

Thanks for the information. I prefer to set a fixed speed. A 100 Mbit speed is unacceptable, as the link would be more or less constantly saturated. With the set 1 Gbit, the old switch in the other end has got no problem to connect. I have scanned the networks for potential loops, without any positive indication. There were never any loops before switch replacement, and should not be afterwards either.

Best regards,

Peter

Hi pe1chl, Experimentator.

Both ends are set to full duplex. Nothing else is acceptable. The problem is auto negotiation when the Mikrotik switch port has got much higher capabilities, than the other end. This also applies when both ends are Mikrotik devices, for example a CRS51 C0-8XS-2XQ-IN connected to a CRS354-48P-4S+2Q+RM over the QSFP+ ports on both devices (DAC-cable). The CRS510-8XS-2XQ-IN is 100 Gbit, and the CRS354-48P-4S+2Q+RM is 40 Gbit. You just don't get a link, unless you set the port on the CRS51 C0-8XS-2XQ-IN to 40 Gbit.

The problem here is that the faster Mikrotik switch fails if the link is auto negotiated. The behavior when the error message appears is unacceptable. IMHO, the error handling is buggy on the Mikrotik side (at least with RouterOS 7.14.1 and 7.14.2). The Mikrotik switch should not fail under any circumstances when this occurs. Either it should close down the offending port, or correct and ignore the error.

@Experimentator: IMHO, the port should be closed, which will force a quick reaction from the IT support people. The port is not closed (should not be) on the remote end, but on the offending Mikrotik device. The appropriate steps (setting a fixed speed) could then be taken to make the link work again. As indicated in numerous posts on internet, if the cause of the behavior is failing hardware, bad connections, or cables, the connection is probably dubious at best. When the error is corrected, communications will be restored. I prefer a hard fail to a soft fail. A hard fail requires quick action. A soft fail is often just overlooked indefinitely. Man is a lazy creature...

Final words: The Mikrotik end should not fail in this way. No log entries, except the one in the subject, nothing else indicating what's going on, just slowly stopping to work.

Best regards,

Peter

When you want to define what the device should do in your opinion, please create a ticket at the MikroTik support site!

"You just don't get a link, unless you set the port on the CRS51 C0-8XS-2XQ-IN to 40 Gbit."

But what do you mean with that? You turned off auto negotiate? Or did you remove all higher speeds from the Advertise section on that port?

Hi pe1chl,

I turned off auto negotiation and set a fixed port speed on the CRS51-C0-8XS-2XQ to 40GbitCR for the 100 Gbit QSFPPlus port. Otherwise no link.

Thanks for reminding me about filing a bug report. I definitely consider it a bug, and I just didn't have time yet to file a bug report. If auto negotiation for SFP/SFP+/QSFP/QSFP+ ports doesn't work, there is no sense having the option to set it. With fixed speed, it works perfectly. But it took me some time to get the idea to try setting fixed speeds. I also set fixed speed on the CRS354-48P-4S+2Q+RM QSFP+ port. It worked with the default auto negotiation setting. It's probably redundant, but it doesn't hurt anyway.

I guess that the auto negotiation does not work for optical connections and DAC-cables (probably use the same code as for optical transceivers). With connections over UTP cables, auto negotiation seems to work if the ports in both ends differ in capability, never had any problems with that.

Best regards,

Peter