Troubleshooting ethernet port failure; rate setting

RouterOS General
1

Lately (after upgrading to v6.28 and v.6.29.1) we occur several ethernet interfaces that stop passing traffic on routers that otherwise funcion normally.

Under v6.27 we started to set port rate setting to manual (instead of the OS default ‘auto’) because we found in multiple hop links this improves the throughput over the link (4=>10+ routers. board to board ethernet connections and wireless NV2 radio links)

But although we think to see some improvement on these link by setting all port in that link to ‘manual’ AND we started upgrading to 6.29.1 to get better throughput over the bridged interfaces we are now in a situation (days to week later) that some ports suddenly start to stop passing traffic. Only reboots or in some case power cycles bring them up working again.

Now we can go back in OS but maybe its more ‘rate setting’ related.
Some parts of the total links are also gigabit ethernet connections where at more distant towers with less traffic we still have fast ethernet.

Question1:
Would have the rate setting any impact on the throughput over two ethernet interconnected interfaces?

Q2:
Would it make such connections really more stable if set to ‘manual’ instead of ‘auto’

Q3:
Why, and is it, really important to have both end at the same setting? (We we know actually, one ‘auto’ rate port connected to a ‘manual rate’ port don’t go well. They are much more unstable and sometimes stop to work. But whould ‘auto’ no be smart enough to connect to ‘any’ kind of port?)


If someone could give us some real opinions about the port rate setting we maybe can at least leave that issue behind us… (and go onto the os v.6.29.1 related…)

2

When you don’t understand how automatic vs manual ethernet setting really works, and have to question what
will happen when you set one end to manual and the other to automatic:

ALWAYS leave the setting at automatic!!!

The manual setting was the only way 15-20 years ago, but today there really can be a lot of trouble when you
use it without knowing the implications w.r.t. half/full duplex setting. Just because auto mode cannot negotiate
with a port that is set to manual. This is not something particular to MikroTik, it is a general aspect of ethernet.

With 1Gbps ethernet, the automatic mode is really the only thing that can be used.

3

well, we started to use manual setting because some port that occasional port failure improved in manual setting.
Secondly some one on this forum (no time to look it up) actually stated manual setting was ALWAYS to be preferred to stabilize links especial when these routes consist out of mixed fast ethernet, gigabit AND NV2 radio links…

So this creates even more confusion now…

Obviously we are aware that the manual settings like rate and full duplex or simplex had to be set the same on both ends and if one port is giga where the other is fastE both have to be set to 100Mbps.

We also noted that several routers set to manual take a second or two to connect and let traffic pass where on two routers on both ends the setting towards ‘manual’ is performed almost at the same time the traffic flow hardly even stops…

We also noted that some long many routers paths improved with all ethernet interfaces along the line set to ‘manual’ compared to the same in ‘auto’.
But I don’t understand why that can be. Because as a network dummy I don’t have the knowledge to give it some fundamental basis. So I was hoping someone else could.
To hear now the opposite surprises me quit a bit… :astonished:

I’m not going to argue you are wrong. You might be right, bit if that is the case I would like to learn why…

(Did I not mention we speak about a ALL Mikrotik router network. The only place were 3rd party routes come involved is at the client where we install a TP-link router. Hence for ease we leave the Mt-CPE ethernet port connecting to is in ‘auto’ mode.)

(Just another reason that conflicts with your statement; We recently hooked up a client that happens to be a network tech himself for years. First thing he asked when we were troubleshooting his not so perfect speeds was if we could set the port rate to ‘manual’! Because he also stated it works better… )

4

Spend several hours now reading some info online including a years spanning discussion between network operators what is best.

First my impression was getting firm ‘auto’ negotiate was preferred. But in reading along the line, even on relative ‘new’ equipments some defend hardly that 'manual (but set same on both ends!) are preferred in many networks…

But since I’m working in an 99% MT routerboard environment my bet would be ‘auto’ would be best. And after all, it worked for years with us.
Its only now we are dealing with partly gigabit sections in our network combined with the ‘fastpath’ features and ‘hardware queues only’ interface setting we did improve link throughputs but at the same time have much more link issues…

Some questions:

If some node is to be reached over 2 or more tdma wireless links and in-between routers are set to manual, or even ‘auto’ is half duplex to be preferred over full? After all, the wireless links are half duplex and some state for sync purposes 2 or more tdma links should have same tdma period size. So why not set the ethernet to half duplex too? Its still faster than the fastest link and it would eliminate any crosstalk over the cables if at any point a ‘made to the purpose’ cable is not made cat5e approvable? This way traffic between gateway and remote host will traverse in sync from start to end and back?

Or is it better to keep full duplex as the setting?
(some docs state that although interface auto negotiation might set to full duplex it doesn’t mean the quality of the link is better than it would be in half duplex? Its just that both ends declare its possible it is set that way. But especial under high load package loss might occur that would not happen if link was set to half duplex…)

5

If one both ends of a wireless (tdma) link two routers are ethernet connected, would it make any difference if of one end the routers set their rate in the ‘auto’ process where on the other end of the link the both routers would be set manually by the operator? (presume both ends are set to 100Mbps)

And would it make a difference if on one end we use half duplex where the other end is set to full duplex?


What about at the router?
We have one rb1100AH were several routes to several parts of our network all come together in this router to be transported over one other link towards a remote gateway router.
Should ALL ports be the same? or can it be that one remote link ‘comes in’ with its interface set to manual where another link ‘comes in’ on an interface has ‘auto’ set?
Again its considered that any link from this router further to the end user follows its route to this end user in all the same settings for all other routers it passes… (All auto/manual, full/half)

If one port setting of this router indeed would make an influence to another interface’s (part of another IP network) performance it basically means the WHOLE network should be using the same setting?

Than auto would be preferred since we are talking now so many routers the change some manual mismatching setting might occur becomes higher than the change an auto setting might prove to be failing…

6

check ethernet interface counters for errors, its a good signal of some negotiation problem

7

When you use manual configuration and the parameters are different, it will not work OK.
E.g. when one end is half duplex and the other end is full duplex, you will have many collisions and lost packets.

When one end is auto and the other end is manual, the auto end will choose half duplex.
So when you have set full duplex on the manual end, you have created yourself a problem.

My experience with auto mode on all different kind of equipment and with good cabling has been very good for
over 10 years. Before that, there were some issues when the automatic mode was first implemented by different
manufacturers and sometimes there were disagreements. But I think that has all been solved long ago.

You are right in that people copy this “manual” recommendation from others in “standard practice” documents, but
I think it causes more trouble than it fixes. When you have negotiation problems on your ethernet links (the WiFi
side has nothing to do with this!! so NV2 is not relevant), you must make sure you have good cabling first. Because
even when you may mask the problem by setting manual, it will still not work 100%.

8

Ok, some new experiences and comments for you guys:

First the comments.

Yes, in reading I found many have the ‘manual’ practise still in place as sort of 'thing from the past when things were different so we just keep on using this practise?. Even when several other, very experienced network operators state that nowadays ‘auto’ is the preferred way since it works in 99,999% of the cases.

But these statements are up to today contested by some others, with the same base of experiences, stating that ‘manual’ is preferred on carrier and server networks since it eliminates possible problems from ‘auto’ setting. They defend very hard their working method to still, up to today, ‘manual’ settings. Off course with the mentioning that it has to be guaranteed both ends use the same setting and Gigabit is the exclusion. Giga should always be set to auto…

The ‘auto’ camp now actually contested the ‘manual’ guys in saying that if they have issues on negotiation and therefore using ‘manual’ its not the ethernet handshaking giving the problem, but more probably a cable issue or router issue.

And so it goes up and down…

Some came up with very recent manuals or white papers for a Cisco router where possible negotiation issues are to be solved by going to ‘manual’. The manual (Cisco white paper for that series of devices) states even that ‘manual’ is to be preferred!
Their was also a website from a very big network operator related to ethernet issues and their conclusion was that in most fast ethernet connections manual is to be preferred… (with the mentioning that every change of router or port or cable has to be certified and administered so no duplex or speed mismatches can occur. They claim a good operator’s network administration should have ruled out any human error in this… :open_mouth: )

But ok, since I am working in an all MT routerboard environment and even the oldest units are from 2006 or so I suppose the MT drivers and embedded protocols are fully compliant to the latest standards (My network is never more than 4 OS versions behind and firmware is updated each time the OS is.)
This would make me think ‘auto’ is the preferred setting in my network.

I also distilled that if you have a multiple hop link running over several cable connected AND wireless connected routers, the wireless setting by themselves make no difference? So as long as one or the other medium is not the bottle neck they have no influence on eachothers performance?
This thinking is actually fed since some on the forum said that port flapping some have is probably a result from poor sync between NV2 and Gigabit interfaces on MT stuff… He mentioned that even MT had said to him it was better to have all interfaces the same and not to mix networks in partially gigabit and partially fastethernet when the wireless parts are in NV2. Basically he stated that the whole network should be made gigabit before you really would start using it in parts??
(It sounds a bit weird to me because some devices like rb750UP still have no Gigabit replacement so making a full gigabit network is impossible if you use the different MT devices…)


You guys didn’t answer yet my question about what should be the setting on a multiple port router that connect to different type of networks…

Last night I had a rb1100AH that I set one port wrongly in regard to his opponent and almost all other ports stopped working inmediately! The only port still worked was the one connected to my office network (Gigabit connection to a cloud switch) and the port to my internet gateway (CCR router).
I could only bring the other ports backup by disabling the one I must have made the misalignment and reboot the rb1100AH. After that I could via a backdoor route login to the one router (netmetal) creating the issue and set is to ‘auto’ with all other setting enable. Only after that it worked again… (for abt 7 hours. The rb1100AH just crashed again… I’ll tell more about this in the next post)
But this proofs imho that a mismatch in one interface will make other interface to go down as well!
Or do you guys disagree on that?

9

Now the new crash just happend on my rb1100AH again. Well crash, the same ports ceased to pass traffic again…

First emergency move I made (running network, 600 clients, 70-80meg traffic etc) is just to reboot the whole lot of attached routers at once. (So not the rb1100AH).
After the reboot all came up and worked again… but;

The interface connected to the netmetal (yes, the one gave me grieve last night) is on ‘auto’ and all other options enabled, like its counterpart interface on the Netmetal, but every so many seconds it just stopped. The old bloody ‘port flapping’ is back on my network!

So what to do? After all its all Gigabit. Settings are gigabit, negotiated rates are gigabit and I could run 300megas or more to that router…
Well, I had to do something so set both ends to ‘manual’ and max rate to 100Mb. The connection is now stable… but completely opposed to the recommendations and practises for gigabit connections…

Maybe the cable? But it ran for months… and I need a bloody elevation platform to change it…

And comments?

10

I would replug the easy accessed side first ..surely doesn’t hurt ..maybe you are lucky :smiley:

11

Well, after so many years of troubleshooting this never struck my mind… :frowning: