New Ethernet port flap issue enquiery, PLS JOIN!

RouterOS General
21

Just for completation:

What makes this problem even worse is fact, that such ethernet down/up events makes switch (that is conected to the MK) to empty its dynamic MAC entries in its table for that port. So when it happened, unicast flood in the subnetwork where this switch is “root” is inevitable, because traffic sent to the MK port before the flap, is sent then immediately to the all ports (without the uplink of course), because switch does not know the learned MACs on the MK port as before. If you have more MKs in such network (or more on the same switch), some weaker models a la RB133 etc. may not handle such traffic (CPU 100 percent) and may reboot, or inaccessible for several seconds or even minute (if watchog sent it to reboot).

This is the worst thing I am afraid of. If MK handles e.g. 10-20Mbit of traffic and decides to “close” the door, then switch sents this traffic to all ports (and eventually other MK if present) and reboot/inaccessibility avalanche is here!. Sometimes, the outage is 2-3 seconds, but from logs I posted above - some of them I found were 20-30 seconds long! So until router sends ARP request to retrieve if such host behind MK is alive, unicast flood in the subnetwork will not dismiss. Of course, some smarter swicthes can reduce unicast floods by settings thresholds, but this is just a crutch.

Such traffic spikes and unicast floods were more times a day detected in our network, just due to this problem.

22

Up to now, not. But as it looks, I will have to :slight_smile:. BTW thanks for explanation - I thought, that this forum is extensively under the look of developers.

23

:slight_smile: :slight_smile: I´ll bet they follow every topic if it is interesting enough for them… they only won’t acknowledge this otherwise users will start making claims.
So if a user want to make a claim or serious report they expect you to do a lit bit more effort and make an official support request with some additional information so they than are in a better situation to administer the follow-up.
(Not meaning though this always works 100%. They are also humans… :frowning: )

24

Oh, thanks for the karma! You earned one too because I think this is a very helpfull report you made for these MT guys. :slight_smile:

25

It is likely not related, but that box went down on me twice today. All leds shining and all, but no ping response from any port nor traffic being processed. First measure was to change a PSU. I’ll see if it will fail again or not.
Despite all that the port flapping is still the same:

18:59:44 interface,info ether1 link down 
18:59:48 interface,info ether1 link up (speed 10M, full duplex) 
19:14:00 dhcp,info server1 deassigned 192.168.21.51 from F0:DE:F1:0D:39:0F 
19:38:03 interface,info ether1 link down 
19:38:10 interface,info ether1 link up (speed 1000M, full duplex) 
19:38:10 dhcp,info server1 assigned 192.168.21.51 to F0:DE:F1:0D:39:0F 
19:59:42 interface,info ether1 link down 
19:59:46 interface,info ether1 link up (speed 10M, full duplex) 
20:04:27 interface,info ether1 link down 
20:04:34 interface,info ether1 link up (speed 1000M, full duplex) 
20:06:39 interface,info ether1 link down 
20:06:40 interface,info ether1 link up (speed 10M, full duplex) 
20:06:41 interface,info ether1 link down 
20:06:43 interface,info ether1 link up (speed 10M, full duplex) 
20:24:39 dhcp,info server1 deassigned 192.168.21.51 from F0:DE:F1:0D:39:0F 
20:50:44 interface,info ether1 link down 
20:50:51 interface,info ether1 link up (speed 1000M, full duplex) 
20:50:53 dhcp,info server1 assigned 192.168.21.51 to F0:DE:F1:0D:39:0F 
20:57:47 interface,info ether1 link down 
20:57:50 interface,info ether1 link up (speed 10M, full duplex) 
21:10:53 dhcp,info server1 deassigned 192.168.21.51 from F0:DE:F1:0D:39:0F 
aug/31 23:11:53 interface,info ether1 link down 
aug/31 23:12:00 interface,info ether1 link up (speed 1000M, full duplex) 
aug/31 23:12:01 dhcp,info server1 assigned 192.168.21.51 to F0:DE:F1:0D:39:0F 
aug/31 23:33:30 interface,info ether2 link down 
aug/31 23:33:32 interface,info ether2 link up (speed 10M, full duplex) 
aug/31 23:34:38 interface,info ether1 link down 
aug/31 23:34:40 interface,info ether1 link up (speed 10M, full duplex) 
aug/31 23:35:02 interface,info ether1 link down 
aug/31 23:52:00 dhcp,info server1 deassigned 192.168.21.51 from F0:DE:F1:0D:39:0F 
07:24:46 interface,info ether2 link down 
07:24:49 interface,info ether2 link up (speed 1000M, full duplex) 
12:31:59 interface,info ether2 link down 
12:32:00 interface,info ether2 link up (speed 10M, full duplex) 
12:56:46 interface,info ether5 link down 
12:56:48 interface,info ether5 link up (speed 100M, full duplex) 
13:03:18 interface,info ether5 link down 
13:03:20 interface,info ether5 link up (speed 100M, full duplex) 
13:18:37 interface,info ether5 link down 
13:18:38 interface,info ether5 link up (speed 100M, full duplex) 
13:28:09 interface,info ether5 link down 
13:28:11 interface,info ether5 link up (speed 100M, full duplex) 
14:34:00 interface,info ether2 link down 
14:34:03 interface,info ether2 link up (speed 1000M, full duplex) 
00:20:59 interface,info ether2 link down 
00:21:01 interface,info ether2 link up (speed 10M, full duplex) 
11:02:15 interface,info ether2 link down 
11:02:17 interface,info ether2 link up (speed 1000M, full duplex) 
13:46:40 interface,info ether2 link down 
13:46:42 interface,info ether2 link up (speed 10M, full duplex) 
13:47:32 interface,info ether2 link down 
14:06:46 interface,info ether2 link up (speed 100M, full duplex) 
14:07:24 interface,info ether2 link down 
14:07:27 interface,info ether2 link up (speed 1000M, full duplex) 
14:08:03 interface,info ether2 link down 
14:08:07 interface,info ether2 link up (speed 1000M, full duplex)
26

Dear
Mikrotik Forum

I apologize for my write but its 1:31am 03/89/2011.

My scenery.:

Main Router (OSPF) — RB1000(switch)/routing --RB433AH(PtP)/bridge – RB433AH(PtP)/bridge —RB1100AH(PPPoE)/OSPF— RB493AH/Bridge 5 port (where is connected 4 AP) — RB433AH/Two Radios/Bridge—End

Ten days ago after to upgrade all our Mikrotik equipment two 5.6V we note that one AP have to many drop and disconnection and over the RB493AH I see the ether9 down/up.- On this moment we reported to Mikrotik this problem and they recommend to me that Enable/RSTP on Bridge port because they think that the problem was a Loop.

A.- I enable the RSTP but the problem continue
B.- I read it on this forum that maybe the problem was the ethernet cable (I changed) but the
problem continue
C.- I change the RB433AH on the AP and the Radios but the problem continue
D.- I change the power supply and Poe and the problem continue

You can see my graphic where you understand that every 5 or 10 minutes the AP disconnect from the RB493AH by 3 sec and after come back but this problem affect to all client that is connect on this AP.

I spoke with the MT specialist on Mexico and he recommend that we need to add the mac address of our client on the access list, at the same time I ask to Janis/Mikrotik about it and he sent me back and email recommend that we need to do this.

Well yesterday 02/09/2011 aprox 23:00PM I added the clients that connect to this AP on the access list and right now 2:00AM on 03/09/2011 I not have the flap problem on the Port9.

I dont know if the problem go but I will be post on the next 2 days if the problem is 100% is fix it..

I hope that this information help you..

Best regards

Wiyat
flap3.jpg
flap1.jpg
flap2.jpg

27

wiyat; I’ve been reading your post with interest.
It is always best to have wireless clients that are allowed to associate to AP in the access list. But that’s another topic.
I find it curious that in your case putting AP’s clients in its access list would solve the port flap on a router that is only attached to this AP via a Ethernet interface and cable.
I don’t think one router would be influenced by another’s associated clients. Lets see what happens. I’ve had occasions where a port flap issue disappeared only to come back after some days…

But keep us posted what your findings are. In your case it is clear your 349AH has a port flap issue and its real, since your AP looses contact when your Etherenet port 9 is down.

28

Dear
Rudy

The problem is on the Firmware 2.29 that this firmware coming with any error and I dont know how I can put back to the older version of firmware on my RB433AH.

Do you know how I can do this, please specify from where I can download the before version.

We have install 50 AP RB433AH and we not have any problem on 49 equipment and only this one, the only change that I do was that I upgrade the firmware to 2.29 and after this situation I have a error on my ethernet port down/up (FLAP).

If you know how I can downgrade the firmware to the before version of 2.29 and tell me from where I can download to test right now I will be testing now.

Thank you..

29

How to downgrade the firmware:
@adi55 wrote this:

Downgrade procedure is quite simple:

  1. Download the .fwf firmware file for corresponding product firmware from > http://routerboard.com/index.php?showProduct=52> .
    Drag&drop this file into File Window from Winbox.
  2. Reboot.
  3. Last operations:
    from terminal window:
    /system routerboard print
    and check the “update-firmware” to see if the downgrade version was “seen”. If it is then you will put:
    /system routerboard upgrade
    which, in this case will do a “downgrade” firmware.

http://forum.mikrotik.com/t/downgrade-the-firmware/46351/14

and the warning from @willy:

Do not downgrade the firmware to lower than the device released! Be careful! Downgrade only if you exactly know what will happen (for example device runs with this version before).

same topic.

Please let us know how you’re going. It might be a solution…

30

If that is the firmware issue then where to get 2.27 for a RB750G to test it out?

31

Dear
WirelessRudy and Ivoshiee

The main point is that i dont found on internet a site from where I can download the older version to download.- On Mikrotik Forum exist a topic that show how to do this but when do you go to routerboard.com the firmware file is actual 2.36.

Mikrotik.Wiki not recommend to do this “downgrade to the before version” apparently we can damage the board is better to upgrade to the last firmware version.- For this reason you dont found a older firmware version on Internet.

I have two idea:

  1. Using a netinstall I try to download to the original firmware version, if this no working its ok.
  2. I will be update to the 2.36 version and I hope that my problem will be correct.

At the same time I report to Mikrotik about this situation where we testing the 5.6 Roteros version and 2.29 firmware in other AP that is on production and we have the Flap problem too, automatically after to upgrade the firmware on this RB433AH the flap problem start.

Remember this is a Theory about the firmware because maybe the problem is my RB433AH that have 14 months that is working and maybe the firmware its not compatible with the chip I dont know.

I write you after that we test our solution and I hope that this work.

Thank u

32

I haven’t found a proper firmware and thus I haven’t reverted the RB750G firmware, yet.

Another variant of that port flip is reported here:
http://forum.mikrotik.com/t/script-to-check-the-ethernet-status/49916/1

33

I took 4 new units in use this week, I updated to 5.6 but did not update the firmware. I’m putting them in production and will monitor them. If they don’t have the port flap coming up in a week or so I will still update the firmware and see what happens then…

34

Good plan for the starters.

35

New Issue developed today!"

rb493AH running for over one year.
On Ethernet port 8 a backhaul link is connected that consist out of two in bridge mode working NanoStationM’s
This link works for 8 months without issues.
Since this morning Ethernet port 8 of rb493AH started to flip on and off. This is a real one since units and customers behind that link lost their connection.

I set the Ethernet to 10Mb manual and the problem is gone although the speed is maximal 6Mb over that back-haul which is not enough!
The moment I switch the port manual to 100Mb or set negotiation to ´auto´ the port starts to flicker on and off.

Just send a support request for this one, but no ticket no. received yet.

I am going to switch units off, and back on (power cycle) to see if that might make any difference.

36

Hi Rudy not sure if this of any help but i had a issue with a 493ah where on one ethernet port going to a ptp link was going down and by just disabling and few moments later re-enabling would solve the problem for hours this happened not every day but about every three days, the only entry in the log about the down time was reporting ospf route loss after trying a lot to pin point the source i traced the fault to a DHCP conflict coming back from a internal router in a household into a rb750 further down the ptp link chain, this was after ethernet from 493 to 433 wireless link to 433 (PtP) ethernet to 750 and also using nat masquerade from 10.152.x.x to 192.168.10.x ???

37

OK, I took the Ethernet cable out of this flapping port 8 (All other ports are in use and all connected to as many more backhauls + one to main gateway to internet)
Waited 2 mins and put it back in. Port still flapped in 100Mb mode.

Powered rb493AH down, waited 2 mins and powered up again (all cables attached, whole network down!).
Router came up and I switched port 8 to 100Mb manual. Connected up fine.
Now, 30mins later port did not flap a single time!

Obviously their is something wrong with the software or the Ethernet driver.
Now I have to sit and wait to see what is happening.

In the meantime waiting for v5.7 that was promised last week… :frowning:
Hopefully the reason for the delay is that they are still working on this issue before they bring 5.7 out. Otherwise they can start on 5.8!

MT: Ticket#2011091366000173 refers to this specific issue. If you answer to that mail I can send you 2 more supout.rif’s. One at 100Mb but flapping and one after the power cycle.

(It might be an idea to have support request to have the ability to add more files to the mail. At least 2, 1 before an issue and 1 after a change/reboot/power cycle.)

38

I am begining to wonder if it is hardware related, you remember that I said the other day, that an AP appeared to freeze the other day, and that although I was unable to contact it over the wire, that the customers appeared unaffected.

And that was because, the AP had a second wlan interface (backhaul), so no data was physically leaving the router via its ethernet port. But within 5 mins I had customers complaining that they had lost internet, they were all connected via various AP’s on the same tower, but their cables all went to a RB750, and all the traffic left the RB750 via port 2 to the backhaul within the AP mentioned above.

Rebooting all the routers by power cycling made no difference, however as soon as I physically touched the ethernet cable plugged up to port 2 on the RB750, everything sprang back into life.

I now believe that there may be a faulty batch of RJ45 connectors out there. The RJ45 connector has magnetics that are delicate and also sensitive to over-soldering, I know because I trashed one trying to remove it from the pcb, although it appeared intact, the heat had damaged the internal magnetics.

Maybe these are put in by hand and maybe on the assembly line there is a guy/gal who keeps the soldering iron on for too long. Maybe that is causing dry/oxidised joints within the RJ45 socket that appear to fail when there is temperature changes.

39

Well, you might be true.
But it seems that I’ve an unreasonably big amount of this ´bad badge´ coming my way! According MT I was actually the only one having the issue to start with! :laughing:

Anyway, your story sounds plausible with my limited knowledge compared to you. :wink:
But how does this relate to the fact that most port flapping units also see the issue disappear if the software is set to ´manual´ and the rate to 10Mb?

And what would be the reason most flaps happen during daytime hours while issues seem to greatly disappear overnight? Several users more or less have reported this behaviour in the forum.

Would be interesting if MT finally could shine a light on this and if they are working on the issue. So far its damn silent from their end. :frowning:

40

Hi everyone, I got the same problem occuring several places.

RB1100 v5.6 firm 2.30 was flipping ether1-5.
We tried various auto-negotiate on/off and speed settings, but we didnt notice any difference(no, we did not try 10Mbps, as its just wont carry the traffic).
Next, we replaced the RB1100 with a brand new one, but the issue is the same.
Tonight we are going to replace it with a new RB1200…and see what happens.
We are an ISP and this issue is serious… So I really hope MT will come up with the solution fast.

There isnt any high CPU loud (16-28%), total bw around 20-70Mbps max, and not alot of small packages…all fine really.

Here is the log from the newly replaced RB1100…

sep/11 03:02:10 interface,info ether4 link down 
sep/11 03:02:10 interface,info ether5 link down 
sep/11 03:02:13 interface,info ether5 link up (speed 100M, full duplex) 
sep/11 03:02:14 interface,info ether1 link up (speed 1000M, full duplex) 
sep/11 03:02:14 interface,info ether2 link up (speed 1000M, full duplex) 
sep/11 03:02:14 interface,info ether3 link up (speed 1000M, full duplex) 
sep/11 03:02:14 interface,info ether4 link up (speed 1000M, full duplex) 
sep/11 09:28:16 interface,info ether3 link down 
sep/11 09:28:19 interface,info ether3 link up (speed 1000M, full duplex) 
sep/12 06:46:19 interface,info ether4 link down 
sep/12 06:46:22 interface,info ether4 link up (speed 1000M, full duplex) 
sep/12 06:46:50 route,ospf,info OSPFv2 neighbor 5.5.5.5: state change from Full to Down 
sep/12 07:33:38 interface,info ether1 link down 
sep/12 07:33:38 interface,info ether2 link down 
sep/12 07:33:38 interface,info ether3 link down 
sep/12 07:33:38 interface,info ether4 link down 
sep/12 07:33:38 interface,info ether5 link down 
sep/12 07:33:41 interface,info ether5 link up (speed 100M, full duplex) 
sep/12 07:33:42 interface,info ether1 link up (speed 1000M, full duplex) 
sep/12 07:33:42 interface,info ether2 link up (speed 1000M, full duplex) 
sep/12 07:33:42 interface,info ether3 link up (speed 1000M, full duplex) 
sep/12 07:33:42 interface,info ether4 link up (speed 1000M, full duplex) 
sep/12 10:37:11 interface,info ether3 link down 
sep/12 10:37:14 interface,info ether3 link up (speed 1000M, full duplex) 
sep/12 18:56:49 interface,info ether1 link down 
sep/12 18:56:49 interface,info ether2 link down 
sep/12 18:56:49 interface,info ether3 link down 
sep/12 18:56:49 interface,info ether4 link down 
sep/12 18:56:49 interface,info ether5 link down 
sep/12 18:56:51 interface,info ether5 link up (speed 100M, full duplex) 
sep/12 18:56:52 interface,info ether1 link up (speed 1000M, full duplex) 
sep/12 18:56:52 interface,info ether2 link up (speed 1000M, full duplex) 
sep/12 18:56:52 interface,info ether3 link up (speed 1000M, full duplex) 
sep/12 18:56:53 interface,info ether1 link down 
sep/12 18:56:53 interface,info ether2 link down 
sep/12 18:56:53 interface,info ether3 link down 
sep/12 18:56:53 interface,info ether5 link down 
sep/12 18:56:55 interface,info ether5 link up (speed 100M, full duplex) 
sep/12 18:56:56 interface,info ether1 link up (speed 1000M, full duplex) 
sep/12 18:56:56 interface,info ether2 link up (speed 1000M, full duplex) 
sep/12 18:56:56 interface,info ether3 link up (speed 1000M, full duplex) 
sep/12 18:56:56 interface,info ether4 link up (speed 1000M, full duplex) 
07:21:35 interface,info ether1 link down 
07:21:35 interface,info ether2 link down 
07:21:35 interface,info ether3 link down 
07:21:35 interface,info ether4 link down 
07:21:35 interface,info ether5 link down 
07:21:37 interface,info ether5 link up (speed 100M, full duplex) 
07:21:38 interface,info ether2 link up (speed 1000M, full duplex) 
07:21:38 interface,info ether3 link up (speed 1000M, full duplex) 
07:21:38 interface,info ether4 link up (speed 1000M, full duplex) 
07:21:39 interface,info ether2 link down 
07:21:39 interface,info ether3 link down 
07:21:39 interface,info ether4 link down 
07:21:39 interface,info ether5 link down 
07:21:41 interface,info ether5 link up (speed 100M, full duplex) 
07:21:42 interface,info ether1 link up (speed 1000M, full duplex) 
07:21:42 interface,info ether2 link up (speed 1000M, full duplex) 
07:21:42 interface,info ether3 link up (speed 1000M, full duplex) 
07:21:42 interface,info ether4 link up (speed 1000M, full duplex)

We got a PowerRouter lying around, but there isnt enough ether ports on it…but first lets see how the RB1200 handles it.