CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

ddt · Fri Apr 24, 2015 2:26 am

We deployed our first CCR1009 at a remote site, and it looks to be a wonderful midway unit for routing between trying to software bridge on a CRS125 and spending a lot on other CCR models that are a bit of overkill.

However, ever since we deployed this CCR1009, I've been noticing this every few hours:

apr/22 17:23:35 interface,info ether1 link down
apr/22 17:23:35 interface,info ether2 link down
apr/22 17:23:35 interface,info ether3 link down
apr/22 17:23:35 interface,info ether4 link down
apr/22 17:23:35 route,ospf,info OSPFv2 neighbor X.X.128.33: state change from Full to Down
apr/22 17:23:35 route,ospf,info OSPFv2 neighbor X.X.128.47: state change from Full to Down
apr/22 17:23:37 interface,info ether2 link up (speed 100M, full duplex)
apr/22 17:23:37 interface,info ether3 link up (speed 100M, full duplex)
apr/22 17:23:38 interface,info ether1 link up (speed 1G, full duplex)
apr/22 17:23:38 interface,info ether4 link up (speed 1G, full duplex)
apr/22 17:24:28 route,ospf,info Database Description packet has different master status flag
apr/22 17:24:28 route,ospf,info     new master flag=false
apr/22 17:24:28 route,ospf,info OSPFv2 neighbor X.X.128.33: state change from Full to 2-Way

It's not doing it at set times throughout the day, it seems pretty sporadic. However, ether1 (X.X.128.33) and ether4 (X.X.128.47) are the two routes out of the unit, so I believe this is dropping packets.

I'm noticing it happen on ether1-ether4, which correlates to the switch chip, even though we're not switching from this unit at all. I've combed over the unit's full export and nothing is really different from our previous incarnation of this unit being a CRS125 (and having CPU issues from that).

When doing a ping from one hop away, (out the ether4) I see when this happens (X.X.136.106 is down ether2 on the above, X.X.65.17 is upstream bridging between ether1 and ether4 on the above device via OSPF:

  SEQ HOST                                     SIZE TTL TIME  STATUS             
  727 X.X.136.106                            56  63 4ms  
  728 X.X.136.106                            56  63 1ms  
  729 X.X.136.106                                         timeout            
  730 X.X.136.106                                         timeout            
  731 X.X.136.106                                         timeout            
  732 X.X.136.106                                         timeout            
  733 X.X.136.106                            56  63 1ms  
  734 X.X.128.162                            84  64 0ms   redirect host      
  735 X.X.136.106                                         timeout            
  736 X.X.65.17                               84  64 134ms TTL exceeded       
  737 X.X.65.17                               84  64 134ms TTL exceeded       
  738 X.X.65.17                               84  64 118ms TTL exceeded       
  739 X.X.65.17                               84  64 130ms TTL exceeded

Thoughts? Why would ether1-ether4 just be dropping?

ddt · Fri Apr 24, 2015 8:35 pm

Last night I upgraded from 6.27 to 6.28, and also made sure that the firmware was up to date (v3.22).

Everything is up to date on this thing, yet ether1-ether4 keep randomly dropping for 2 seconds every few hours.

We're most likely going to swap out this unit for our shelf spare CCR1036, but I really want to be able to trust this model. The price point (plus the switch chip and SFP+ port) is perfect for our needs in places where a CRS125 doing software bridge routing instead of just plain switching doesn't cut it. However the CCR1036 is super overkill as it never goes over 0% CPU usage at the 5-6 sites we have it-- the CCR1036 costs a bit too much for how much we need to deploy them in places where a CRS125 really won't do and the lack for SFP+ really hurts it.

uri · Tue May 26, 2015 10:49 pm

Hi, I'm having the exact same problem. I have a CCR1009 ROS 6.27, and ether1 to ether4 are going down and up (within few seconds) on a irregular basis.

Have you found any solution?

uri · Tue May 26, 2015 10:57 pm

Hi, I'm having the same issue! Have you had any solution?

Fri May 29, 2015 11:32 am

Have you all sent SUPOUT. RIF to support@mikrotik.com ?

maznu · Fri May 29, 2015 1:08 pm

We had a similar problem with a CRS226 running v6.27. All 24 ports would flap for a few seconds down/up/down/up/down/up, randomly between 8-36 hours. We tried v6.28, but other bugs caused even bigger problems, so we downgraded to v6.27. MikroTik suggested we upgrade to 6.29. So now we're running v6.29 - it's only been 10 hours so far... but seems ok?

We have two CCR1009 boxes - and they have been just fine on v6.27, though, for several weeks. No flapping at all.

maznu · Tue Jun 02, 2015 10:31 am

[quote="maznu"]MikroTik suggested we upgrade to 6.29. So now we're running v6.29 - it's only been 10 hours so far... but seems ok?[/quote]

Alas, yesterday our CRS226 - running v6.29 - flapped all its switch ports 5 times.

justlovingIT · Tue Jun 02, 2015 1:04 pm

We noticed the same on the CRS125 as well as CRS226 with 6.28

After upgrading to 6.29 yesterday this seems to have gone now. (At least I hope so)

maznu · Thu Jun 04, 2015 9:52 pm

No joy with v6.29 for us on the CRS226. Trying v6.29.1 now...

jamesrossell · Tue Jun 23, 2015 11:27 am

Hey,

we are seeing the same / similar issues on CCR1009-8G-1S-1S+ running 6.28.

routerboard: yes
model: CCR1009-8G-1S-1S+
serial-number: 4AB2047433A3
current-firmware: 3.13
upgrade-firmware: 3.22

08:44:54 interface,info ether4 link down 
08:44:54 interface,info ether1 link down 
08:44:54 interface,info ether2 link down 
08:44:54 interface,info ether3 link down 
08:44:54 pppoe,ppp,info AAISP: terminating... - disconnected 
08:44:54 pppoe,ppp,info AAISP: disconnected 
08:44:54 pppoe,ppp,info AAISP: initializing... 
08:44:54 pppoe,ppp,info AAISP: waiting for packets... 
08:44:54 pppoe,ppp,info AAISP: terminating... 
08:44:54 pppoe,ppp,info AAISP: disconnected 
08:44:54 pppoe,ppp,info AAISP: initializing... 
08:44:54 pppoe,ppp,info AAISP: waiting for packets... 
08:44:54 route,ospf,info OSPFv2 neighbor 134.0.17.140: state change from Full to Down 
08:44:54 route,ospf,info OSPFv3 neighbor 134.0.17.140: state change from Full to Down 
08:45:02 interface,info ether4 link up (speed 1G, full duplex) 
08:45:02 interface,info ether1 link up (speed 100M, full duplex) 
08:45:02 interface,info ether2 link up (speed 100M, full duplex) 
08:45:02 interface,info ether3 link up (speed 1G, full duplex)

Anyone any joy with this problem?

James

pukkita · Tue Jun 30, 2015 12:12 pm

Make sure firmware is updated.

maznu · Wed Jul 01, 2015 12:09 pm

[quote="pukkita"]Make sure firmware is updated.[/quote]

That's not helping, pukkita. 6.27, 6.28, 6.29, 6.29.1, and 6.30rc22 all seem to have the same problem.

pukkita · Wed Jul 01, 2015 12:10 pm

Make sure firmware is updated.
That's not helping, pukkita. 6.27, 6.28, 6.29, 6.29.1, and 6.30rc22 all seem to have the same problem.

Those are RouterOS versions, not firmware versions. I mean firmware version in /System > Routerboard

maznu · Wed Jul 01, 2015 12:14 pm

Sorry, we update the firmware at the same time as updating the software.

All on 3.22, which is the most up-to-date for CRS226.

pukkita · Wed Jul 01, 2015 12:22 pm

Don't know about CRS226, but on CCR1009 (-1S-1S+ and -1S-1S+PC) with firmware 3.22 I'm not having any port flapping on switch chip ports as the original poster, using versions 6.27 and 6.19.

A general preventive measure with routers behaving weird, specially due to previous bugs, is a reset to no defaults (exporting the configuration to a rsc file before), after firmware/RouterOS has been updated, then import the rsc file back to make sure all gets initialized correctly.

Only thing I notice with versions >= 6.27 on CCRs is SFP "micro cuts", with nothing showing in the logs (juts stop passing traffic, does not seem SFP module brand/model related). Almost not noticeable unless you're monitoring it. Doesn't happen with 6.19.

maznu · Wed Jul 01, 2015 12:24 pm

[quote="pukkita"]A general preventive measure with routers behaving weird, specially due to previous bugs, is a reset to no defaults (exporting the configuration to a rsc file before), then reload the configuration to make sure all gets initialized correctly.
[/quote]

Have done that. Have also replaced with a completely different brand new CRS226, with a different PSU. Same problem: after about 5 days, all ports flap (sometimes once, sometimes several times in a day). After another approximately 5 days, either all ports flap some more, or kernel crash.

pukkita · Wed Jul 01, 2015 12:29 pm

Have not yet used CRS226 but per your comments it looks not enough stable to put it in production.

No problems with CRS125 and CRS109 so far (6.27 and 3.22).

we are seeing the same / similar issues on CCR1009-8G-1S-1S+ running 6.28.

routerboard: yes
model: CCR1009-8G-1S-1S+
serial-number: 4AB2047433A3
current-firmware: 3.13
upgrade-firmware: 3.22

@Jamesrossell have you upgraded that CCR1009 firmware?.

maznu · Wed Jul 01, 2015 12:33 pm

[quote="pukkita"]Have not yet used CRS226 but per your comments it looks not enough stable to put it in production. [/quote]

Very few people seem to be reporting this problem. Also, it is strange that the person who started this thread has the same problem with the CCR1009. We also have two CCR1009, but are not using the switch features (all acting as separate ports), and they have been fine - well, one crashed last night at 23:59:60 because of a leap second bug.

[quote="pukkita"]No problems with CRS125 and CRS109 so far.
[/quote]

Two CRS125 are ok for us. Two CCR1009-1S+ are ok. Just this CRS226 is being a problem. However, it seems like the problem is not the CRS but the software, maybe, because of the original poster of this thread. :(

pukkita · Wed Jul 01, 2015 12:38 pm

maznu, all CCR and CRS (and some RBs) have had port flapping problems at the beggining (just launched).

Seems history is repeating with CRS226, sending supout.rif and opening a ticket is the course of action to take.

As it happens with all versions, I'd do it with the 6.30rc as they're already focused on that version to release.

maznu · Wed Jul 01, 2015 12:41 pm

[quote="pukkita"]Seems history is repeating with CRS226, sending supout.rif and opening a ticket is the course of action to take.
[/quote]

We opened the ticket many weeks ago now. Each time it happens, we tell MikroTik and send a supout, they say nothing. It happens again, they tell us to install the next version... it happens again... and here we are, now testing 6.30rc22... and other threads discussing the same problem suggest that it is not fixed in 6.30 yet. But maybe the supout files will help this time... maybe... I hope...!

pukkita · Wed Jul 01, 2015 12:59 pm

I feel your pain... patience and hope mikrotik get it sorted out quickly.

In the meanwhile (response from mikrotik support for CRS212 in http://forum.mikrotik.com/viewtopic.php?f=2&t=97778 ):

Configuration should be changed to utilize switch chip features
instead of cpu on this device. Most of CRS series devices are
designed to be used as switches and cpu's are not that powerful and
the moment it seems that there are some bottlenecks reached
somewhere in hardware and they cause switch to reset it's port
state. From your end it looks like all ports have flapped.
We will try to correct these bottleneck cause lock-ups in
software, but in the mean while you should reconfigure your CRS212 to work as switch.

Wed Jul 01, 2015 1:26 pm

Possible port flapping problems on CCR1009 ether1-ether4 ports are fixed since RouterOS v6.29.

But the CRS226 interface resets seem to appear due to insufficient CPU resources in routing or bridging applications.
We are working to address this issue in upcoming RouterOS version.

maznu · Wed Jul 01, 2015 1:32 pm

[quote="becs"]But the CRS226 interface resets seem to appear due to insufficient CPU resources in routing or bridging applications.
We are working to address this issue in upcoming RouterOS version.[/quote]

Thanks, becs. Glad to hear you're looking at this.

The CRS226 we have had problems with is not doing much routing: it participates in our OSPF and OSPFv3, but only carries management traffic. It has three adjacencies, and the OSPF/v3 routing tables are only a few tens of subnets. CPU load is ~23% and not increasing (the graphs are almost perfectly flat)... and then after about five days, interface reset.

Edit: it does also have an EOIP and bridge interfaces which are for the management traffic. But again, that is only used for us to be able to query with SNMP.

StubArea51 · Wed Jul 01, 2015 5:48 pm

Just curious....are the flapping ports connected to other MikroTik equipment or another vendor? If it's another vendor, which one is it?

Haven't seen this with the CCR1009 in our lab, but it only connects to other MT equip...may hook it up to a Cisco 6509 and see what happens.

maznu · Thu Jul 02, 2015 2:21 pm

[quote="IPANetEngineer"]Just curious....are the flapping ports connected to other MikroTik equipment or another vendor? If it's another vendor, which one is it?
[/quote]

For us: CCR1009 (x2), MikroTik mAP 2n (x1), and two connections to other suppliers who use a mix of Cisco and another vendor's equipment (unknown). All ports flap at the same time, so I'm not sure that this is a useful datapoint!

digitexwireless · Wed Jan 06, 2016 5:25 am

Did anyone figure out a fix for this? Had to run out and replace a CCR1009 tonight. Put a RB2011 in its place, but let face it, they were never meant to push 400+ meg. The box at the tower is just too small for a rack mount size so we are using the CCR1009-8g-1s-1s+PC. It began dropping all ports and then right back up. Then it just failed altogether. We cannot pass traffic on port 1-4 at all when it stops working. We are running ros 6.33.3. Tried downgrading to 6.32.3, no change.

I may have a different problem like failing hardware, I don't know. Brought it back to the NOC and have been hammering it with a bandwidth test and cannot get it to lockup now. (surprise)

maznu · Wed Jan 06, 2016 10:45 am

There's been a really useful thread about this on some other hardware (but related to the switch chip, probably) here: http://forum.mikrotik.com/viewtopic.php?f=1&t=97895

CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Re: CCR1009: ether1 through ether4 sporadically drop then instantly come back up - switch chip problem?

Who is online