Ive got a CCR1016-12G feeding a CRS226 and CRS125.

The 226 is always every few hours (may go up to 20 hours) sooner or later drops. The CCR shows a link down against ether 4 (running to ether 1 on CRS226). What's interesting the CRS shows a cycle on the logs of all the used ether ports.
The CRS 125 never its on ether 5 on CCR.

So I set up a bonded balance-tlb layer 2/3 on the CRS226. I used ether1 and 9 since 9 belongs to the next internal switch.

I then have ether4 and 5 on the CCR feeding this, the CRS125 is now on ether 6

So its still happening, but Im seeing ether4 and 5 drop almost the same time.

Ive had the CCR and CRS226 replaced still doing this cycling of the ethers used on the CRS226

I'm now trying 6.30rc22 as support say this has a kernel crash fix that has been around the last few versions..

We have had the same problems with the CRS226: http://forum.mikrotik.com/viewtopic.php ... 73#p485373

We even tried a different CRS226, but after 5 days and 16 hours, the same problems: all ports go down for a second, then come back up. Another 20 minutes later, they all go down for a few seconds, then back up, then back down for a second, then back up.

We have been advised by MikroTik to try 6.30rc22 as well. I will post here in about 6 days when we know more!

We have had the same problems with the CRS226: http://forum.mikrotik.com/viewtopic.php ... 73#p485373

We even tried a different CRS226, but after 5 days and 16 hours, the same problems: all ports go down for a second, then come back up. Another 20 minutes later, they all go down for a few seconds, then back up, then back down for a second, then back up.

We have been advised by MikroTik to try 6.30rc22 as well. I will post here in about 6 days when we know more!

Glad I'm not alone it was driving me crazy, especially I dumped a old 16 port POS 4k buffer non managed.

Anyway it's been up hours both the CRS and CCR and I've not seen any random kernel failures on the CCR and the CRS it's so far not cycled the ethers!

I'll keep you posted it will NEVER go 24h stay tuned!

[quote="105547111"]I'll keep you posted it will NEVER go 24h stay tuned!
[/quote]

Thanks - and good luck! :)

Uptime over 8h and no drops on the CCR and no ether cycle on the CRS226

This IS looking good

Not fixed

Right at 9hrs 50 up time all the ethers reset on the CRS226 ;(

But Ive tried few restarts the CCR random kernel reboot issue looks good.

Did you send support request on Mikrotik?

Yes

Ticket [Ticket#2015062366000063] RE: CRS226-24G-2S-RM

It included the supout that shows a clean reboot then 9 hours later all the ethers in use cycled

[quote="105547111"]It included the supout that shows a clean reboot then 9 hours later all the ethers in use cycled :-)
[/quote]

Oh dear. I guess I will be saying the same thing in a few days' time, then. I hope our supout files help, at least!

Our ticket is #2015052766000629 (opened almost a month ago now).

Whats interesting the logs in the CRS show the ethers all cycle that are in use, down first then immediately up.

But in interfaces the link downs remain at 0????

But on reverse end (eth4 and 5 on the CCR) I see ether 4 and 5 immediately cycled in sequence very fast down and up.

This is clearly a interesting CRS226 bug the CRS126 is perfect....

Any response that in 6.30 would be fixed?

[quote="pchott"]Any response that in 6.30 would be fixed?[/quote]

All I know is what Janis from MT Support sent in my exchange on the ticket:

Please upgrade to RouterOS release candidate version 6.30rc22.
If the changes in the latest version do not solve this problem completely, supout
files from that version will allow us to continue debugging.

I've provided Janis with supout and remote access to our CRS226.

I can confirm on RC22 I saw the ethers all cycle still for us about the 8 to 12 hour range normally but definitely within 24 hours like clockwork. It's had two sets of resets since running.

I had RC24 running but I've updated to RC26 now and it only had 6 hours uptime so I'm far away from saying it's fixed...,

Hopefully RC26 stay around - I'll update this thread should I see the CRS226 have a cycle again. What's interesting it's not just Ethernet. I set up a SFP link between the CRS226 and CRS125 yes it's doing that as well.

It's looking like a low level reset on all the switch chips, but not a full kernel failure. What's interesting always the CRS226 claims no ether drops on some ports, but the CRS125 and the CCR1016 on the other ends show link downs.....

I also know it's happening as all IPv6 traffic through the switch dies for about 3 minutes as it has to wait for RADVD advertisement from the CCR, plus I'm monitoring winbox over IPv6.

Tick tick RC26 running about a hour so far....

Just lost it all again 19:25

Sending to support

Random reboot after about 18-24 hours running v6.30rc22:

14:04:16 system,error,critical router was rebooted without proper shutdown

No flaps yet, but random reboots isn't better.

[quote="maznu"]No flaps yet, but random reboots isn't better.
[/quote]

Well, we just had the flaps:

14:18:34 interface,info ether02-mmr link down
14:18:34 interface,info ether23-wap1 link down
14:18:34 interface,info ether01-mmr link down
14:18:34 interface,info ether06-mmr-metronet link down
14:18:34 interface,info ether21-mmr-exa link down

14:18:35 interface,info ether02-mmr link up (speed 1G, full duplex)
14:18:35 interface,info ether23-wap1 link up (speed 100M, full duplex)
14:18:35 interface,info ether01-mmr link up (speed 1G, full duplex)
14:18:35 interface,info ether21-mmr-exa link up (speed 100M, full duplex)

...repeating...

14:20:24 interface,info ether02-mmr link up (speed 1G, full duplex)
14:20:24 interface,info ether23-wap1 link up (speed 100M, full duplex)
14:20:24 interface,info ether01-mmr link up (speed 1G, full duplex)
14:20:24 interface,info ether21-mmr-exa link up (speed 100M, full duplex)

Tried 6.30rc28 for ten minutes. Ports were flapping so much (every couple of minutes) that the switch was unusable.

Trying 6.30rc29 now... hasn't flapped in a whole 1 hour so far!

[quote="maznu"]Trying 6.30rc29 now... hasn't flapped in a whole 1 hour so far!
[/quote]

6.30rc29: kernel crash and switch reboot after under 10 hours. I can't keep on testing firmware that is this unreliable, so am downgrading to v6.29.1 which at least will run for five days before all the ports flap.

Trying 6.30rc29 now... hasn't flapped in a whole 1 hour so far!

6.30rc29: kernel crash and switch reboot after under 10 hours. I can't keep on testing firmware that is this unreliable, so am downgrading to v6.29.1 which at least will run for five days before all the ports flap.

You'll love this.... As I've still got a several days left to return the 226 for a full refund, I ordered a 125-RM. I've thrown it in place of the 226 and RC28 17 hours strong (never ever).

So I played devils advocate wanting to still help eradicate this bug, changed IPs on the 226, plugged eth1 and 3 into a spare MT951G on eth4 and 5 let it run. NO LOAD NOTHING. Simply ether 1 and 3 as inputs and I had a winbox open to monitor.

After the magic about 9H bang winbox gone. Shows eth4 and 5 dropped on the 951G

Even no use it goes in 9H.

Don't forget this is my second 226 the SAME issue exactly.

Cheers!

[quote="105547111"]As I've still got a several days left to return the 226 for a full refund, I ordered a 125-RM[/quote]

Funnily enough, our 125 arrives tomorrow morning. We'll swap out the 226 for the 125 (there are two other 125s on our network, been running fine for 60 days). And then make the 226 available for MikroTik support.

Interesting to hear your problems continue with no traffic. That suggests it is not CPU load, but maybe some other resource exhaustion.

Good luck, MikroTik!

Thank you, 105547111 :-)

As I've still got a several days left to return the 226 for a full refund, I ordered a 125-RM

Funnily enough, our 125 arrives tomorrow morning. We'll swap out the 226 for the 125 (there are two other 125s on our network, been running fine for 60 days). And then make the 226 available for MikroTik support.

Interesting to hear your problems continue with no traffic. That suggests it is not CPU load, but maybe some other resource exhaustion.

Good luck, MikroTik!

Thank you, 105547111

Just an update. Its been now 1 day, 14 hours and its STILL not seen a single link down on the new 125. The 226 just again a hour ago flipped ether 1 and 3 again, on the floor!

You'll notice CPU usage is WAY down on the 125. The 226 has now a bridge (unused) and a single IP on ether 1, no load nothing and ether 3 is a second link incoming and it still drops them, and CPU sits 13 to 32%!! Mostly around the mid 20's... Seems high... Especially just sitting at less than 1 packet a sec total??

Now Ive got two 125's both sit 13 to 17 % but fairly constantly, despite one's heavily loaded traffic wise, averages about 1000 to 3000 packets / sec constantly ether 1.

I'm wondering if the CPU is not keeping up with the switch chips and causing intermittent 'reset' of the chip since its loosing contact? The display is also very 'laggy' on the 226, but responsive on the 125. Ive always noticed this and may be a cause of the issue. The CCR its also responsive. Try scrolling interfaces on the 226 its SLOW...

Looking the 125 has a 600 MHz MIPS 74KC (Version 4.12) and the 226 has same CPU (Later version V8.5) but only 400MHz

File transfers are way slower to it also. I could do 3 to 4 firmwares in a 125 to a 226 uploads slow as CPU maxes out..

I'd compare resources both and play around. Ive got to pack up mine now, but Ive got two more sipouts for support showing link downs with nothing connected!


Here's my ticket (201561966000722)... Mention it to support, lets see if they combine the tickets... As much as I'd like to see this fixed as I still plan to deploy these if possible, the two SFP ports are great feature....




PS - Small update Almost 2 hrs later... v6.30 is officially out today. Ive just updated and lost my unbroken no link downs @ almost 2 days.... but I'm sure it will have no more with 125's...

[quote="105547111"]I'd compare resources both and play around. Ive got to pack up mine now, but Ive got two more sipouts for support showing link downs with nothing connected!
[/quote]

Glad to hear your 125 is working ok. Similarly our third 125 has replaced the 226 (for now). The only change we had to make was to MTU: with our 226 we used a L2MTU of 9216 (MTU 9000). Now we have to set the L2MTU to 4064 (MTU 4000) instead, because the 125 does not support the largest jumbo packets.

I'd compare resources both and play around. Ive got to pack up mine now, but Ive got two more sipouts for support showing link downs with nothing connected!

Glad to hear your 125 is working ok. Similarly our third 125 has replaced the 226 (for now). The only change we had to make was to MTU: with our 226 we used a L2MTU of 9216 (MTU 9000). Now we have to set the L2MTU to 4064 (MTU 4000) instead, because the 125 does not support the largest jumbo packets.

I hadn't even tweaked L2 MTU yet Its nice it being stable.....

Still all good. No drops or kernel fails

rb433AH running now for just over 3 hours without interface crash after 6.29.1 => 6.30... fingers crossed!

The board actually was upgrades 2 or 3 hours before that but on the bridged networks some other units still had v.6.29.1 running.
Some of them hardwired through rb705UP's or other boards and some remote behind some wireless links.
Only after ALL units in the IP network domain were upgraded from 6.26.1 to 6.30 the package losses to units dropped considerably and only after the very last (wireless remote) board was upgraded the rb433AH is still running.

But still some 8-9 hours to go before bed and hoping to get some sleep. I'm only at ease when its still on in 24 hours.....

Weird actually that routers even at the other end of a wireless link are at times brought down (ethernet port stopped passing traffic) were other wired routers to the same board stayed up without issues.

Like its a virus spreading and affecting every unit not resistant enough (but configs all sort of the same..)

Weird actually that routers even at the other end of a wireless link are at times brought down (ethernet port stopped passing traffic) were other wired routers to the same board stayed up without issues.

Like its a virus spreading and affecting every unit not resistant enough (but configs all sort of the same..)

That's exactly what I experienced also. I found that basically everything downstream from the 226 connections all go away. It's like physical Ethernet disconnection and reconnection within a second. I had not only downstream routers that got kicked, every MT I'm monitoring gets winbox kicked off, most wireless devices, servers, etc. anything that's directly connected that's capable shows Ethernet drops.

But now I've replaced it with a 125, I'm now almost 72 hours not a single drop

Yes, upstream and downstream, over wireless links routers were effected. Not everytime though.
The first time we installed 6.29.1 we did so on basically three different backbone links. Each existing out of several wireless and wired connections with multiple MT routers (mainly rb433AH's and some new rb911/912's and one rb800) and they only come together in three physical ethernet ports in one rb1100AH router. But all with their own IP domain (IP-network, thus routed and separated)
We had one of the links that I now fixed (by downgrading) existing out of 4 wireless links and 12 nodes away that gave lots of problems halfway under 6.29.1. (halfway = 2 wireless links and 5 nodes away) The other 2 rb1100AH connected links also went down! Other links connected to the same rb1100AH still running 3.27 all still kept on working.(rb1100AH also ran 6.29.1 at that moment)
Only after upgrading to 6.30 the 'infection' symptoms disappeared. But that one link giving the most of the problems (rb433AH ether2 wired to rb750UP) still stopped passing traffic every so many hours...even with 6.30)

These units still giving issues are now downgraded to 6.27 and eare up and running for 26 hours now....

(To picture from from where I monitor; I am basically directly on the last router before the internet gateway. So except this gateway router ALL my units are downstream. I am telling this because it is normal that if one router in a link goes down all routers behind it downstreams are effected. Not because THEY are effected, but because the traffic towards and from it is broken...)

One other proof of 'infection':

On the problematic backbone line existing out of 4 wireless link and 12 nodes (from my PC) it was node 9 (rb433AH) and 10 (rb750) that gave problems.
Sometimes the problems were just lots of package losses (initiating a crash to come....) but it still ran. At the same time even package losses (on a separate ping stream) towards node 7 (rb800) builded up more and more package losses.
The package loss occurred almost in sync, but less often, as the ping losses towards node 9 (Ok, because node 7 stops, the ping towards node 9 also got broken.)
But more interesting, after the build up of ping losses (and indeed ver slow access) to node 10, or even winbox disconnect to 10 I could still reach node 7, but after some 10 or 15 seconds that one also stopped passing traffic. I lost node 7, but still connected to node 6 (rb433AH, end of a wireless backhaul) In this node 6 I could see no more rx traffic on the port connected to node 7. Luckily netwatch performed on a rb750UP parallel to this node 7 and a power cycle brought everything back up.
I could than login to node 9 again, still no rx from node 10, reboot and rx from node 10 came backup and all worked again. Other routers downstream towards end according logs were not affected.

I had two instances were this same link went down this way and the two other backbones running 6.29.1 some nodes away from that rb1100AH (=node 4) also stopped passing traffic! Complete separate links! But can't be a coincidence... anyway, after 6.30 this issue is gone it seems...

Having switched to the 125, with identical software and router configuration, it's been rock solid stable for almost 10 days.

uptime: 1w2d19h37m11s

Definitely a problem with the CRS226.

I reverted my rb433AH from 6.30 to 6.27 now 3 days ago and since then not had a ethernet port failure anymore. Reverted other units back in my network and my network (thus my life!) all back to normal.
Had a rb911 as backhaul that had a kernel crash every 10-20 hours,under v6.30 . Now with 3.27 running for 30 hours without one......

Still have other boards (including rb433AH's) at other places in my network that still run v.6.30 and never gave any issue.

So don't know what is now exactly the issue. Maybe the combination of 6.30 and some poorer (sub-standard) cables? or faulty ethernet port that otherwise work but not under v.6.30? I don't know....

It's something in the newer versions. I've had Ethernet drops from two CRS226 (as I had it replaced once, as well as the CCR1016). It's only 1 meter (3 feet) and I've tried multiple cables and ports, it's not that!

In the last two days before I returned the replacement 226 it had TWO Ethernet drops sitting on the floor with nothing connected other than two incoming and I even looped a few Ethernet ports to spare RB951G and they all cycled!

I've got a pair of full online UPS and I've tried both power feeds..

Ever since I've returned the CRS226 and changed to CRS125 there are no Ethernet drops not one and keeps that way until I upgrade and reboot, 10 days and more! That's how it's supposed to be...

But as support told me as this issue effects only certain hardware and it's usually after lengthy up times it's going to take a long time to track down.

They confirmed the supout does show all the Ethernet ports going down and up but there's no internal reason why... It simply looks like you physically removed and reinserted all those at once...

I've been through support with all my downstream devices that all cycle. I gave support full remote access to the CRS226, CRS125 and CCR1016! and it's the software / hardware combination, but NOT any physical hardware fault....

If your got ether drops I'd advise running as you've done older firmware unless you can tolerate random but consistent Ethernet drops...

From Janis today:

"We have reproduced this problem, we have a setup in lab where ports are flapping on CRS226. The developer may fix it any day, it should be soon."

Fingers crossed!!

Has this issue been fixed? I'd like to purchase a CRS226-24G-2S+RM for the SFP+ ports.

This problem has been fixed in the latest RouterOS v6.31 release candidate versions.

Do we have a release date for 6.31 final?

Thanks!

Any news regarding described problem?
We have simillar issues at 6.31, port flapping at higher trafic system rebooting.

Thinking about i will return devices to supplier.

[quote="madmucho"]Any news regarding described problem? [/quote]

We replaced our CRS226 with a CRS125, and it has been stable for a few weeks.

We have not yet tried 6.31 on CRS226, so please do not interpret my silence as "problem fixed" — I do not yet know.

Any news regarding described problem?

We replaced our CRS226 with a CRS125, and it has been stable for a few weeks.

We have not yet tried 6.31 on CRS226, so please do not interpret my silence as "problem fixed" — I do not yet know.

Thank you for information,
we thinking about same, CRS125 works ok on our customers, but we design 226 for its 10GBit support , if we get it back to 1GB cons for mikrotik is gone they will continue with cisco.

Hope that someone from mikrotik will listen.

Update,
confirmed as bug and seems to be fixed at latest RC version 6.33 rc22 currently works on our lab.
i will keep updates here if works ok after release of final version.

I must inform you readers that problems with CRS226 still presist, workaround for buggy state is bridge all ports to cpu and remove them from switchchip.
6.32.3 still problem.

[quote="madmucho"]problems with CRS226 still presist[/quote]

Are you joking?! The CRS226 will still flap all its ports and reset randomly, and MikroTik haven't found a solution yet?

[quote="madmucho"]6.32.3 still problem.[/quote]

I see that 6.33 has landed. I have two CRS226 switches which we pulled from production. I shall get these running in our lab, send several hundred Mbit/sec across them, and see what happens over the next few days.

problems with CRS226 still presist

Are you joking?! The CRS226 will still flap all its ports and reset randomly, and MikroTik haven't found a solution yet?

No joke, customer kick us out with this solution, we replaced them with CRS125 models, works after replacement ok.
No special config just common switch on L2 with size >600 mac adress. I spend very long time to debug that with Mikrotik support. Is something wierd in on low level on this model.

That is our experience.

[quote="madmucho"]I spend very long time to debug that with Mikrotik support. Is something wierd in on low level on this model.[/quote]

Thanks for your information, madmucho.

We've been testing a CRS226 running v6.33 in our lab for five days now (only a few MAC addresses, only ~70Mbit/sec of traffic). No resets yet... but we only started to see problems after maybe two weeks of uptime before.

I will report back soon with our findings.

I spend very long time to debug that with Mikrotik support. Is something wierd in on low level on this model.

Thanks for your information, madmucho.

We've been testing a CRS226 running v6.33 in our lab for five days now (only a few MAC addresses, only ~70Mbit/sec of traffic). No resets yet... but we only started to see problems after maybe two weeks of uptime before.

I will report back soon with our findings.

We have same behavior on previous routeros 6.32 in our lab, that problem appears in real trafic one client on switch and uplink to customer segment, we monitor client, he goes down and uplinks flap. on our table looks ok. But not much mac there, if you can put device on your production edge, from there i make support files.

v6.33: 10 days in lab, no flaps or crashes (just 70Mbit/s test traffic continuous). That's much better than we ever had with the CRS226 before. But we will test for another 5-7 days more.

v6.33: 16 days, still going strong.

But I notice this:

[quote]What's new in 6.33.1 (2015-Nov-17 09:55):
*) CRS2xx - fixed occasional switchip resets (broken in 6.33);
[/quote]

I guess that means the good developers at MikroTik are looking at this thread? ;-)

Still stable?

v6.33 firmware 3.22 CRS226-24G-2S+

Uptime 30d 15:28:47
Link Downs 0
Last Link Up Time Nov/07/2015 19:26:14

I guess it's time to go to production...

Please let us know,
maibe i get another chance to that model in some project.

[quote="madmucho"]Please let us know[/quote]

I will report back soon, madmucho. The plan is to move the switches on Friday night this week...

Please let us know

I will report back soon, madmucho. The plan is to move the switches on Friday night this week...

maznu: How does it look in production?

I have it in production and so far no big problems.