Community discussions

MikroTik App
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Aug 06, 2020 1:16 am

Hi,

We started deploying 2004s into our network and have issues with one we are trying to add into our bgp core.

It rebooted every 10-14 days so we took it out and replaced in with another one. We can ran memory test and got memory errors on the first one.

This was 12 days ago....

Today the new rebooted same way making lots of issues in OSPF 0.0.0.0 to the extent we had to reboot 2 other routers running 6.45.9 to make traffic resume again. Another CCR1016 got so corrupted that /export compact, snmp etc dident work.

The second CCR2004 (6.47.1) is connected with console cable now and doing remote memory test we get errors on this one too. Broken batch or something else? Feels like too much coincidence perhaps?

Error in address=0x00000000C0004768, W=0xC0004768 R=0x00000000 X=0xC0004768
Error in address=0x00000000C000476C, W=0xC000476C R=0x00000000 X=0xC000476C
Error in address=0x00000000C0004770, W=0xC0004770 R=0x00000000 X=0xC0004770
Error in address=0x00000000C0004774, W=0xC0004774 R=0x00000000 X=0xC0004774

etc etc etc.

/M
Last edited by mhugo on Wed Dec 30, 2020 1:36 pm, edited 1 time in total.
 
joegoldman
Long time Member
Long time Member
Posts: 562
Joined: Mon May 27, 2013 2:05 am

Re: 2004 hardware issues?

Thu Aug 06, 2020 1:57 am

Something like this is better sent to support@mikrotik.com to start a real case - this is a discussion forum not a proper support channel.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Aug 06, 2020 11:09 pm

Hi,

I asked the question here because it's a forum. I'm fully aware that it's not a support channel.

Mikrotik has no answer for the reboots. Seems they were able to reproduce the memory tester issue so it's confirmed that the memory tester is broken.

They don't think it's related to the reboot.

The system reboots randomly but seldom. I'm trying to attach a console to get output but this is at production site.

If anyone experiences 2004 reboots and tests memory like I did it's not faulty memory being the cause.

The one we have issues with runs 3 full bgp feeds but not much traffic.

The 2004s with no bgp has not rebooted.

/Mikael
 
User avatar
cwachs
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Sun Aug 16, 2020 5:04 am

I'm having lockups on one of our two 2004s as well. It's an edge router with 4 bgp sessions (no full tables). About every 36 hours, it locks up where we can't access it via winbox, ssh and snmp stops. Sometimes it passes traffic through it, other times no traffic will pass.

We got a console server on it now. Last lockup reported nothing at all in the console but I still had console access. When I tried to get it to generate a supout.rif file via console, that failed but immediately after, it came back to life and I could generate one via winbox.

Our crash 2 days ago it was passing traffic but we lost all access to it including console. Had to pull power to reboot it.

Support ticket is open but so far, no info.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sun Aug 16, 2020 5:59 pm

I have been told by Mikrotik the bootloader memory test has been fixed and fix will be included at next release.

As for the logging we have connected to another mikrotik at the same site and logging to locai file - hoping for a crash to happen.


/M
 
User avatar
cwachs
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Sun Aug 16, 2020 6:03 pm

What are you all logging to "echo" in hopes of getting useful info in case of a crash? We have "critical, warning, health, system and event" echoing and nothing was on the console at all for our last crash.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Aug 18, 2020 1:07 pm

Not from the log but supposedly there should come some crash information on the console disregarding logging settings.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Fri Aug 21, 2020 4:30 pm

Bootloader fix was included in 6.47.2
 
ofca
Member Candidate
Member Candidate
Posts: 202
Joined: Fri Aug 20, 2004 7:18 pm

Re: 2004 hardware issues?

Fri Aug 21, 2020 7:21 pm

We are seeing same issues with one of 8 CCR2004s: uninvited reboots 1-2 weeks apart. Each one is running 6.47.1, each one is running BGP, so it would seem that it may be a hardware problem. SFP28 is in use on all, so it's also not the reason.
 
User avatar
cwachs
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Fri Aug 21, 2020 7:23 pm

I am shipping one 2004 back under RMA and our other one reboots every 1 -2 weeks. Was on 6.47 and I just put it on 6.47.2 two days ago. Both are running BGP and OSPF.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sat Sep 05, 2020 1:13 am

Mikrotik is running some special debug packages on one of our routers. Its either a software bug or something deep in the hardware since they mentioned involving the CPU vendor.

Without the debug packages nothing came on console at crash time, so hoping for a new crash soon so this can be resolved. It has currently been up 7 days.

We have 24 2004s in boxes so would be nice to solve this issue.....
 
User avatar
cwachs
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Sat Sep 05, 2020 1:55 am

we are running the same debug firmware on ours. Every one of our deployed 2004s is having some sort of problem (either random reboots or crashes or both).
 
ofca
Member Candidate
Member Candidate
Posts: 202
Joined: Fri Aug 20, 2004 7:18 pm

Re: 2004 hardware issues?

Sat Sep 05, 2020 3:46 am

2004 that used to reboot every now and then seems stable so far:
/system resource> print 
             uptime: 3w15h17m35s
            version: 6.47.1 (stable)

/system routerboard> print 
             model: CCR2004-1G-12S+2XS
  factory-firmware: 6.46.3
  current-firmware: 6.47.1

I gave it something to do, so it didn't die out of boredom:
> /tool bandwidth-test (...) direction=both protocol=udp local-tx-speed=20000000000 remote-tx-speed=20000000000
                status: running
              duration: 1w6d20h41m35s
            tx-current: 20.0Gbps
  tx-10-second-average: 20.0Gbps
      tx-total-average: 20.0Gbps
            rx-current: 20.0Gbps
  rx-10-second-average: 19.9Gbps
      rx-total-average: 20.0Gbps
          lost-packets: 0
           random-data: no
             direction: both
               tx-size: 9000
               rx-size: 9000
      connection-count: 20
        local-cpu-load: 70%
       remote-cpu-load: 74%
...but it seems there's either no correlation between load and reboots, or there is and load prevents reboots ;)

Anyway, my $0.03 to the case. No instabilities seen on other 2004s either.
 
bugtoodd
just joined
Posts: 6
Joined: Thu Jun 29, 2017 5:54 pm

Re: 2004 hardware issues?

Wed Sep 09, 2020 10:04 am

We are seeing the same problems on two CCR 2004 our of 10 deployed. No Connection tracking enabled. Support says that unless we have the debug package installed with the console, there's no way to catch the problem. Today it happend again on the same units with version 6.47.2: it looks like an hardware issue to me, we will deploy the debug and console and see if we can help finding the problem.
 
User avatar
cwachs
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Sun Sep 13, 2020 7:38 pm

We just had our 2004 crash with the debug firmware installed. The last line of the console output is:
[admin@AUW-LOOKOUT-EDGE-02] > LOOPER: read_raw read failed: EOF
died with signal
Nothing before that for hours. After the crash, we got 2 physical link up/down messages in console (about 2 minutes after the crash). Nothing else. Router will not respond to console input and we can't log into it. Nor is it passing traffic. I have sent the debug log to support with our open ticket with them. Hopefully this console message is useful to them and they can get this fixed.
 
Abner
just joined
Posts: 8
Joined: Sat Dec 21, 2019 11:31 pm

Re: 2004 hardware issues?

Sun Oct 04, 2020 12:30 am

Any new info about this?


I wonder if this has something to do with it ->

What's new in 6.48beta40 (2020-Sep-14 13:34):
*) arm64 - improved reboot reason reporting in log;
 
User avatar
cwachs
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Sun Oct 04, 2020 12:32 am

That is related. It was added to help support troubleshoot this. We are running that firmware at the request of Mikrotik to gather more information when it crashes. Has not done anything yet to stop the crashing.
 
angriukas
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Fri Nov 22, 2013 9:20 am
Contact:

Re: 2004 hardware issues?

Fri Oct 09, 2020 6:42 pm

We also have issues with CCR2004, there is no BGP/OSPF in our case. Reboot fully random, sometimes few times per day, sometimes once in two weeks.
CPU load do not exceed 6%, average load 20-25Mbps with rare spikes to 50Mbps. ROS: 6.47.4.
We love MT devices, ...but those reboots are horrible.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Oct 27, 2020 9:55 am

Hi,

We are running 6.48beta48 on some of the 2004s that was rebooting, It seems to have solved the reboots, but it seems that we now face an issue where the routing protocols stops working instead.

Are any of you experiencing the same?

/Mikael
 
FezzFest
newbie
Posts: 43
Joined: Wed Jun 03, 2015 12:03 am

Re: 2004 hardware issues?

Wed Oct 28, 2020 4:40 pm

Just had the first unexpected reboot on our CCR2004. Running 6.47.4, no autosupout.rif on flash. Very light load, 20-30Mbit/s of traffic.
 
User avatar
cwachs
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Wed Oct 28, 2020 4:48 pm

After almost a month of troubleshooting with Mikrotik, our 2004's have been RMAd. I would strongly encourage all of you with 2004s that are rebooting, freezing, etc to open support tickets with Mikrotik. They need data to solve this problem and the more data, hopefully the faster it can be solved.
 
jkaufman
just joined
Posts: 11
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Wed Oct 28, 2020 4:51 pm

I opened a ticket with Mikrotik and sent a support file and have not heard back. Did they require that you upgrade to the beta before RMA?
 
User avatar
cwachs
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Wed Oct 28, 2020 5:02 pm

We did not go to ROS 7 beta but we did run a couple different version 6 betas as part of the test prior to RMA.
 
FezzFest
newbie
Posts: 43
Joined: Wed Jun 03, 2015 12:03 am

Re: 2004 hardware issues?

Sat Oct 31, 2020 9:23 pm

Experienced our second crash in one week today. No autosupout.rif was generated, sent manual supout.rif to support@mikrotik.com.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Mon Nov 02, 2020 7:58 am

Hi,

Right now we have one 2004 not reachable by winbox, but routing works. It runt 6.48beta48.

The same box dropped ospf last time it had issues.

Previously is rebooted when it had issues and I think that was actually better.

We swapped this with another 2004 so its either a general hardware issue or a software isssue.

Did any of you get improvements after swapping for RMAd boxes as this might have been an issue with early production?

/Mikael
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Nov 24, 2020 12:38 pm

Hi!

Did anyone here get any more updates from MT on this? We are seeing processor hangs in the beta and reboots in stable still. We have a lot of 2004s waiting to go to production, so its frustrating :)

/M
 
User avatar
cwachs
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Tue Nov 24, 2020 2:44 pm

Nothing with us. RMA'd two of ours and bought CCR 1036s to go in their place. My gut tells me it's going to be a while.
 
User avatar
vectieba
just joined
Posts: 24
Joined: Mon Oct 26, 2009 3:04 pm
Location: South Africa

Re: 2004 hardware issues?

Wed Nov 25, 2020 9:07 am

We have been having similar issues, our previous 2004 just rebooted randomly, we also had problems with PPPoE accounts connecting showing Dynamic but not Running, we then replaced it and it went well for about two weeks. We just had the same issue with the PPPoE accounts again and had to do a reboot to get it working.
 
n3xus
just joined
Posts: 4
Joined: Wed Aug 06, 2014 8:39 pm

Re: 2004 hardware issues?

Wed Nov 25, 2020 2:01 pm

Reading this topic, i am on hold, before buying... I need 6 of them
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Nov 26, 2020 12:19 am

We were told the new beta solved the issue, but we did some fast testing and the whole router behaves strange and becomes unreachable even with mac telnet and romon when OSPF is running.

So test very carefully....
 
bugtoodd
just joined
Posts: 6
Joined: Thu Jun 29, 2017 5:54 pm

Re: 2004 hardware issues?

Fri Nov 27, 2020 1:58 am

6.47.8 (stable) also includes the fix for the freeze/reboot issue and I can confirm that it installs fine and no issues with OSPF/BGP.
We are now testing it on 3 units starting today.
 
glueck05
just joined
Posts: 23
Joined: Fri Jan 26, 2018 12:49 pm

Re: 2004 hardware issues?

Fri Nov 27, 2020 10:10 am

6.47.8 (stable) also includes the fix for the freeze/reboot issue and I can confirm that it installs fine and no issues with OSPF/BGP.
We are now testing it on 3 units starting today.
Same here, we started with two units (doing OSPF and BGP).

regards,
Glueck
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sun Nov 29, 2020 3:12 pm

We have rolled out on a couple too.

I did see this in the release that someone seems to have experienced a reboot even with the new software.

I got the feeling also that BGP convergece with full tables was much slower, but dident time it. Ill try to get some time in the lab to reproduce if possible.

/M
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1313
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: 2004 hardware issues?

Sun Nov 29, 2020 6:14 pm

I need to upgrade the CCR2004 in my lab and see if the stability improves. We've used them for several clients but have also had some stability issues.

I think this router will be amazing after a few more months of bug fixes from MikroTik. This is pretty typical of a new router release...it takes a little while to get stable.
Global - MikroTik Support & Consulting - English | Español | Serbian | Danish +1 855-645-7684
https://iparchitechs.com/ecosystem/mikr ... consulting mikrotiksupport@iparchitechs.com
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 02, 2020 1:29 pm

4 months since the issue started - so im hoping the "amazing" starts now :)

So far its been stable for us since upgrade.

/M
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Thu Dec 03, 2020 3:07 am

I have two 2004s (and a few spares on the shelf), which are neighbors to each other in a small (mid 20s) mpls network, not a lot of throughput, not a lot of firewall rules(typical usage ~1%). Started with 6.47, tried 6.47.7 (which was better) but every so often they would just reboot. And typically within 12 hours of each other. I cant have them doing that, so Ive also ordered a few 1036s as well, and some should be here tomorrow. I also hadnt previously seen when I chose them, the limitations with using rj01 sfps (close proximity and power issues within sfp chassis).

Before the one rebooted this morning, I had logged into it ( I had made a few changes just prior) and saw that overall cpu utilization had jumped to 25% (one of the cores was maxed). It did finally crash and reboot, which seemed to take about 5 minutes for it to come back. Quite honestly I thought I was going to have to netinstall. Before it crashed, I tried to get a support file, but it crashed before it could complete. I had sent in autosup files into them, and they suggested newer versions for debug reasons. Chalk it up to experience I guess, and hang on to them until they become a better work horse.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Dec 03, 2020 4:34 pm

I have two 2004s (and a few spares on the shelf), which are neighbors to each other in a small (mid 20s) mpls network, not a lot of throughput, not a lot of firewall rules(typical usage ~1%). Started with 6.47, tried 6.47.7 (which was better) but every so often they would just reboot. And typically within 12 hours of each other. I cant have them doing that, so Ive also ordered a few 1036s as well, and some should be here tomorrow. I also hadnt previously seen when I chose them, the limitations with using rj01 sfps (close proximity and power issues within sfp chassis).

Before the one rebooted this morning, I had logged into it ( I had made a few changes just prior) and saw that overall cpu utilization had jumped to 25% (one of the cores was maxed). It did finally crash and reboot, which seemed to take about 5 minutes for it to come back. Quite honestly I thought I was going to have to netinstall. Before it crashed, I tried to get a support file, but it crashed before it could complete. I had sent in autosup files into them, and they suggested newer versions for debug reasons. Chalk it up to experience I guess, and hang on to them until they become a better work horse.
Hi this seems to have been fixed in latest stable - 6.47.8
 
abyss
Frequent Visitor
Frequent Visitor
Posts: 78
Joined: Wed Sep 21, 2005 10:51 am

Re: 2004 hardware issues?

Fri Dec 04, 2020 7:52 am

Hi,
Our CCR2004 has rebooted without proper shutdown during night with 6.47.8. Not fixed yet. It's not a power problem because this device is in a rack with dual power circuit.
Reboot happens after 3.8 days of running time the 04 December
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Fri Dec 04, 2020 8:01 am


Hi this seems to have been fixed in latest stable - 6.47.8


Thanks for the update, but with the old version it would last up to ~20 some days before it would quit. Sometimes as quick as 8 or so.I do hope they figure it out soon, I have about 10 of these now that will hold paper down, until they are proven again.


Hi,
Our CCR2004 has rebooted without proper shutdown during night with 6.47.8. Not fixed yet. It's not a power problem because this device is in a rack with dual power circuit.
Reboot happens after 3.8 days of running time the 04 December

Thats what I was afraid of. Were you able to log into it shortly before it rebooted? Or do you keep cpu statistics? If so were they higher than normal before reboot?
Hi this seems to have been fixed in latest stable - 6.47.8
 
abyss
Frequent Visitor
Frequent Visitor
Posts: 78
Joined: Wed Sep 21, 2005 10:51 am

Re: 2004 hardware issues?

Fri Dec 04, 2020 9:06 am

Thats what I was afraid of. Were you able to log into it shortly before it rebooted? Or do you keep cpu statistics? If so were they higher than normal before reboot?
Yes, of course, nothing really special. It's not related to trafic, because it happens during night when it's much lower.
CCR2004_Uptime.jpg
CCR2004_trafic.jpg
CCR2004_CPU.jpg
CCR2004_Memory.jpg

CCR2004_winbox.jpg
You do not have the required permissions to view the files attached to this post.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Fri Dec 04, 2020 9:15 am

We just had a 2004 have all interfaces go down and up again. Also seems OSPF dident fully recover, so in the end we rebooted it.

/Mikael
 
markonen
just joined
Posts: 8
Joined: Tue Aug 11, 2020 4:28 pm

Re: 2004 hardware issues?

Fri Dec 04, 2020 10:04 am

Went and looked at the uptimes of our six longest-deployed 2004s:

11w3d18h44m51s, S/N D4F00...
10w6d18h10m42s, S/N D4F10...
8w1d16h32s, S/N D4F10...
5w1d16h1m47s, S/N D4F10...
1w6d17h29m41s, S/N D4F00...
1d22h28m59s, S/N C8A60...

The last two have clearly rebooted without human intervention, not 100% sure about the rest.
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Fri Dec 04, 2020 3:26 pm

Thats what I was afraid of. Were you able to log into it shortly before it rebooted? Or do you keep cpu statistics? If so were they higher than normal before reboot?
Yes, of course, nothing really special. It's not related to trafic, because it happens during night when it's much lower.

CCR2004_Uptime.jpg

CCR2004_trafic.jpg

CCR2004_CPU.jpg

CCR2004_Memory.jpg

CCR2004_winbox.jpg

I was more curious if yours had a spike in "unclassified" on one of the cores like I caught mine doing this last time.

Have you guys seen this about the S+RJ10? Not sure if it affects you, you may be using fiber, I know I didnt catch it until recently.

https://wiki.mikrotik.com/wiki/S%2BRJ10 ... l_guidance
 
murrayis
just joined
Posts: 6
Joined: Tue Sep 29, 2020 11:57 pm

Re: 2004 hardware issues?

Mon Dec 07, 2020 1:20 am

We've experienced all that everyone has listed including;
  • Random Reboots
  • Interface Resets
  • System Deadlocks
  • System Slow Downs
We've tried special firmware's, Debugging firmware's, Logging console outputs for months and unfortunately this issues caused out CCR1072's to be unstable and experience similar issues.

We've pulled all CCR2004's from our network and we have stability back again however due to the lack of support and willingness to fix these issues as a priority and getting support responses back like "Are you sure it rebooted" has caused us to make the rash decision to drop Mikrotik from our Core network and spend Hundreds of Thousands of dollars and switch to Juniper.
 
jkaufman
just joined
Posts: 11
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 7:03 am

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:22 pm

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.
What version were you on previously?
 
jkaufman
just joined
Posts: 11
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:40 pm

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.
What version were you on previously?
6.47.4
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:42 pm

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.
What version were you on previously?
6.47.4

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.
 
jkaufman
just joined
Posts: 11
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:45 pm

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.
The longest uptime I saw was 14 or so days. I did NOT upgrade the routerboard firmware yet, just the OS. I have gone ahead and ordered a different CCR model to replace the 2004.
Last edited by jkaufman on Mon Dec 07, 2020 4:47 pm, edited 1 time in total.
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:46 pm

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.
The longest uptime I saw was 14 or so days. I did NOT upgrade the routerboard firmware yet. I have gone ahead and ordered a different CCR model to replace the 2004.
I think my longest was ~mid 20 days. Dont think I ever reached anything above 30. Was there a reason you upgraded to the new version?
 
jkaufman
just joined
Posts: 11
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:47 pm

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.
The longest uptime I saw was 14 or so days. I did NOT upgrade the routerboard firmware yet. I have gone ahead and ordered a different CCR model to replace the 2004.
I think my longest was ~mid 20 days. Dont think I ever reached anything above 30. Was there a reason you upgraded to the new version?
The new version was said to have improvements in stability for the arm based devices.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 9:32 pm

Hi,

We havent seen any reboots in 6.47.8 but one 2004 dropped and reconnected all intefaces same second. Was asked by Mikrotik if it was a cable issue - which I dont think since nobody was in the rack or the other sites it connects to. Also disconnecting and reconnecting 8 cables same second would be the fastest person alive.

Besides this we do have some oddity in the ospf in network since a couple versions. Having more than 500 mikrotiks in my network this is honestly not what we are used to. Its been quite stable for years.

Im also surprised by the speed Mikrotik is attacking these issues - Its been months. And when you have an issue like that you need to talk with your customers, not just tell them it will be fixed when found even if it was done in a very polite way.

/M
 
jkaufman
just joined
Posts: 11
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 9:42 pm

Hi,

We havent seen any reboots in 6.47.8 but one 2004 dropped and reconnected all intefaces same second. Was asked by Mikrotik if it was a cable issue - which I dont think since nobody was in the rack or the other sites it connects to. Also disconnecting and reconnecting 8 cables same second would be the fastest person alive.

Besides this we do have some oddity in the ospf in network since a couple versions. Having more than 500 mikrotiks in my network this is honestly not what we are used to. Its been quite stable for years.

Im also surprised by the speed Mikrotik is attacking these issues - Its been months. And when you have an issue like that you need to talk with your customers, not just tell them it will be fixed when found even if it was done in a very polite way.

/M
Did you just upgrade the RouterOS or did you upgrade the firmware as well?
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 10:24 pm

Did you just upgrade the RouterOS or did you upgrade the firmware as well?
Both

/Mikael
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 11:14 pm

Hi,

We havent seen any reboots in 6.47.8 but one 2004 dropped and reconnected all intefaces same second. Was asked by Mikrotik if it was a cable issue - which I dont think since nobody was in the rack or the other sites it connects to. Also disconnecting and reconnecting 8 cables same second would be the fastest person alive.

Besides this we do have some oddity in the ospf in network since a couple versions. Having more than 500 mikrotiks in my network this is honestly not what we are used to. Its been quite stable for years.

Im also surprised by the speed Mikrotik is attacking these issues - Its been months. And when you have an issue like that you need to talk with your customers, not just tell them it will be fixed when found even if it was done in a very polite way.

/M

I may have seen something similar.

When an even happens on one ospf port, it sometimes seems to happen simultaneously on one or several other ports. And the site I've had this happen at has 24/7 camera monitoring, so I know nobody touched ours either.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 12:46 am


I may have seen something similar.

When an even happens on one ospf port, it sometimes seems to happen simultaneously on one or several other ports. And the site I've had this happen at has 24/7 camera monitoring, so I know nobody touched ours either.
If you feel like it then please mail Mikrotik support and reference SUP-35544 which is the ticket we have with this behavior.

/M
 
murrayis
just joined
Posts: 6
Joined: Tue Sep 29, 2020 11:57 pm

Re: 2004 hardware issues?

Tue Dec 08, 2020 6:09 am

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.
 
markonen
just joined
Posts: 8
Joined: Tue Aug 11, 2020 4:28 pm

Re: 2004 hardware issues?

Tue Dec 08, 2020 9:44 am

Have y'all had these issues with the v7 betas?

I know it's a hard question since v7's own bugs can mask this. But as the 2004 was designed for ROS7, I wonder if some of these problems are specific to ROS6.

Unfortunately I don't have any spares to try v7 on...
 
User avatar
Larsa
Member Candidate
Member Candidate
Posts: 237
Joined: Sat Aug 29, 2015 7:40 pm

Re: 2004 hardware issues?

Tue Dec 08, 2020 10:41 am

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.
How much? ;-)
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 12:38 pm

Have y'all had these issues with the v7 betas?

I know it's a hard question since v7's own bugs can mask this. But as the 2004 was designed for ROS7, I wonder if some of these problems are specific to ROS6.

Unfortunately I don't have any spares to try v7 on...
Dont try beta3 - It gives massive pingloss for 2004s both on IP and ARP. Same with latest 6.48beta.

/M
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 12:39 pm

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.
What was the reason for rejecting?

/Mikael
 
abyss
Frequent Visitor
Frequent Visitor
Posts: 78
Joined: Wed Sep 21, 2005 10:51 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 1:15 pm

I've replaced the CCR2004 with a CCR1072 and it's running like a charm !

I hope that Mikrotik will soon consider making a premium hardware product line in parallel of the cheap hardware race !
RouterOS has so much potential, it's a shame that this software has no more premium rock solid hardware...
 
User avatar
Kickoleg
Member Candidate
Member Candidate
Posts: 129
Joined: Tue Mar 11, 2014 3:13 pm
Location: Yverdon-les-Bains, Suisse

Re: 2004 hardware issues?

Tue Dec 08, 2020 1:19 pm

I've replaced the CCR2004 with a CCR1072 and it's running like a charm !

I hope that Mikrotik will soon consider making a premium hardware product line in parallel of the cheap hardware race !
RouterOS has so much potential, it's a shame that this software has no more premium rock solid hardware...
+1
MTCNA, MTCUME, MTCRE, MTCWE, MTCTCE certified
 
Paternot
Forum Veteran
Forum Veteran
Posts: 786
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: 2004 hardware issues?

Tue Dec 08, 2020 1:35 pm

I've replaced the CCR2004 with a CCR1072 and it's running like a charm !

I hope that Mikrotik will soon consider making a premium hardware product line in parallel of the cheap hardware race !
RouterOS has so much potential, it's a shame that this software has no more premium rock solid hardware...
You are atacking the wrong problem: this looks like software related, not hardware. I agree that it should be more stable, but (barring some defective units) this kind of complain is usually solved down the line, with a new RoS version.

No, it shouldn't happen - You are quite right about it. But is the software testing that needs improving...
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 3:54 pm

You are atacking the wrong problem: this looks like software related, not hardware. I agree that it should be more stable, but (barring some defective units) this kind of complain is usually solved down the line, with a new RoS version.
I fully agree this is software, but actually MT was a little unsure of this in the beginning so even hardware could be solved with software at some cost I suspect.

I have two wishes this year - First is hoping for stable 2004s on 6.47.x soon and of course stable ROS 7 for new year, but most importantly stable 2004s.

/M
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 11:33 pm


If you feel like it then please mail Mikrotik support and reference SUP-35544 which is the ticket we have with this behavior.

/M
Ill have to put them back in service to do that lol. Ive since replaced them. I will have to try them and either with the older version you started with, or wait for a newer release.

You are atacking the wrong problem: this looks like software related, not hardware. I agree that it should be more stable, but (barring some defective units) this kind of complain is usually solved down the line, with a new RoS version.

No, it shouldn't happen - You are quite right about it. But is the software testing that needs improving...

Wrong to you or not, when you've try a few versions of software, and NEED your equipment to continue running, you have to look for alternative hardware until they have something stable for that hardware. When the problem goes away, you know it was a combination of that hardware/software. When you don't have a unit that's stable, you have to try something else.
Have y'all had these issues with the v7 betas?

I know it's a hard question since v7's own bugs can mask this. But as the 2004 was designed for ROS7, I wonder if some of these problems are specific to ROS6.

Unfortunately I don't have any spares to try v7 on...
Can't run v7 here, running mpls/vpls, and from what I have seen thats not supported yet. Hopefully soon!
 
murrayis
just joined
Posts: 6
Joined: Tue Sep 29, 2020 11:57 pm

Re: 2004 hardware issues?

Wed Dec 09, 2020 1:08 am

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.
What was the reason for rejecting?

/Mikael
Nothing wrong with the hardware....
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 09, 2020 5:53 pm

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.
What was the reason for rejecting?
Nothing wrong with the hardware....
Thats probably true and they will claim software is without warranty I guess which is the real issue. While correct according to license I think they dont understand that we are returning customers. Hopefully.,,

We have 14 of them if it makes you feel better - One we use to test stability.

/M
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Dec 10, 2020 9:09 pm

Hi again,

Today we again experienced a drop of interfaces on a 2004. The drop this time was not all the interfaces, but still all ospf was lost. The other 2004s we are running in test has less traffic and no problems, but one thing I realized is that we are using an SFP28 (DAC Cable) port and that the sfpplus10 is a S-RJ01 which draws much more power than the rest of the interfaces that are normal optical SFPs.

As you can see this happened very fast but because of BGP convergence with CCRs plus no graceful restart it becomes a much longer outage.

Has anyone else seen something like this? We have on the positive side seen no reboots with 6.47.8

13:49:03 snmp,warning timeout while waiting for program 20
13:52:18 interface,info sfp-sfpplus10 link down
13:52:18 interface,info sfp28-2.transit.AS65600 link down
13:52:18 route,ospf,info OSPFv2 neighbor 185.230.33.240: state change from Full to
Down
13:52:18 route,ospf,info OSPFv2 neighbor 192.168.33.235: state change from Full to
Down
13:52:18 route,ospf,info OSPFv2 neighbor 192.168.33.8: state change from Full to Do
wn
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.1
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.8
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.3
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.6
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.234
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=185.230.33.40
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=185.230.33.240
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.231
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.235
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=5.157.4.1
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=193.110.13.254
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.2
13:52:18 route,bgp,error Unexpected UPDATE message
13:52:18 route,bgp,error RemoteAddress=192.168.33.3
13:52:18 route,bgp,error Unexpected UPDATE message
13:52:18 route,bgp,error RemoteAddress=192.168.33.235
13:52:18 route,bgp,info Connection terminated
13:52:18 route,bgp,info RemoteAddress=5.157.4.1
13:52:18 route,bgp,info Connection opened by remote host
13:52:18 route,bgp,info RemoteAddress=193.110.13.254
13:52:18 route,bgp,info Connection opened by remote host
13:52:18 route,bgp,info RemoteAddress=5.157.4.1
13:52:18 route,ospf,info Discarding Database Description packet: wrong neighbor st
ate
13:52:18 route,ospf,info state=Down
13:52:18 route,ospf,info Discarding Database Description packet: wrong neighbor st
ate
13:52:18 route,ospf,info state=Down
13:52:20 interface,info sfp28-2.transit.AS65600 link up (speed 10G, full duplex)
13:52:21 interface,info sfp-sfpplus10 link up (speed 1G, full duplex)
13:52:40 route,bgp,info Connection opened by remote host
13:52:40 route,bgp,info RemoteAddress=192.168.33.234
13:52:41 route,bgp,info Connection opened by remote host
13:52:41 route,bgp,info RemoteAddress=185.230.33.240
13:52:41 route,bgp,info Connection opened by remote host
13:52:41 route,bgp,info RemoteAddress=185.230.33.40
13:52:42 route,bgp,info Connection opened by remote host
13:52:42 route,bgp,info RemoteAddress=192.168.33.3
13:52:42 route,bgp,info Connection opened by remote host
13:52:42 route,bgp,info RemoteAddress=192.168.33.6
13:52:44 route,bgp,info Connection opened by remote host
13:52:44 route,bgp,info RemoteAddress=192.168.33.235
13:52:45 route,bgp,info Connection opened by remote host
13:52:45 route,bgp,info RemoteAddress=192.168.33.1
13:52:47 route,bgp,info Connection opened by remote host
13:52:47 route,bgp,info RemoteAddress=192.168.33.231
13:53:20 route,bgp,info TCP connection established
13:53:20 route,bgp,info RemoteAddress=192.168.33.8
13:53:20 route,bgp,info TCP connection established
13:53:20 route,bgp,info RemoteAddress=192.168.33.2
13:53:20 route,bgp,info Connection opened by remote host
13:53:20 route,bgp,info RemoteAddress=192.168.33.8
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Thu Dec 10, 2020 9:32 pm

Hi again,

Today we again experienced a drop of interfaces on a 2004. The drop this time was not all the interfaces, but still all ospf was lost. The other 2004s we are running in test has less traffic and no problems, but one thing I realized is that we are using an SFP28 (DAC Cable) port and that the sfpplus10 is a S-RJ01 which draws much more power than the rest of the interfaces that are normal optical SFPs.

As you can see this happened very fast but because of BGP convergence with CCRs plus no graceful restart it becomes a much longer outage.

Has anyone else seen something like this? We have on the positive side seen no reboots with 6.47.8

13:49:03 snmp,warning timeout while waiting for program 20
It was my understanding the newer version was supposed to have numerous snmp fixes. I saw that issue lock myself out of a few 1100ahx4's of mine, requiring reboot to regain access.

When mine dropped bfd/ospf connectons, it was using fiber sfp's. Even the two routers connected together with a short fiber jumper. (The other was a longer run to a separate building, though underground and a dark fiber link) I have previously found out that the RJ-01's dont seem to like to hold stable, when used to connect between two routers (RJ-01s on each end), hence why I switched that to fiber. But they seem fine going to a standard ethernet port on one end of the cable. I just didnt have enough fiber sfps on original install, but had RJ-01s.

I have also been experimenting with passive DACs, and they seem better, but I also get them to mess up (ospf/bfds) with the 2004s. (Running 6.47.8) Though I have been pushing them harder for testing purposes.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Dec 10, 2020 10:33 pm

When mine dropped bfd/ospf connectons, it was using fiber sfp's. Even the two routers connected together with a short fiber jumper. (The other was a longer run to a separate building, though underground and a dark fiber link) I have previously found out that the RJ-01's dont seem to like to hold stable, when used to connect between two routers (RJ-01s on each end), hence why I switched that to fiber. But they seem fine going to a standard ethernet port on one end of the cable. I just didnt have enough fiber sfps on original install, but had RJ-01s.
We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

I note you write dropped ospf/bfd but did your interfaces go down too?

/M
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Fri Dec 11, 2020 3:06 am

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

I note you write dropped ospf/bfd but did your interfaces go down too?

/M
I have over 100 wireless microwave paths in my network that spans across the state, I cant live without bfds! Im going from memory, but I believe I have seen the interface go down, but IIRC it wasnt as common as the ospf/bfd drops. IIRC when the interface went down there was usually a RJ-01 involved. Id really like to buy another ~20 or so of these 2004s, they really seem like a nice router, especially at their price point, but I need the reliability for constant voice traffic.
 
User avatar
jspool
Member
Member
Posts: 425
Joined: Sun Oct 04, 2009 4:06 am
Location: Oregon

Re: 2004 hardware issues?

Sat Dec 12, 2020 2:38 am

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

BFD typically works fine on CHR, and most ARM based Mikrotik's. I use it with no issue on CHR & RB4011. I haven't picked up a 2004 yet as I typically let them stabilize a year before putting them in production.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sat Dec 12, 2020 9:01 am

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.
BFD typically works fine on CHR, and most ARM based Mikrotik's. I use it with no issue on CHR & RB4011. I haven't picked up a 2004 yet as I typically let them stabilize a year before putting them in production.
It might be tilera specific -

> -----Original Message-----
> From: Maris B. [MikroTik Support] <support@mikrotik.com>
> Sent: 04 January 2019 11:05
> Subject: Re: [Ticket#2019010322004734] Router acting wierdly
>
> Hello,
>
> Currently there are known problems with BFD on CCR series routers. I would suggest
> to turn off BFD on those devices until problem is resolved.
>
> Best regards,
> Maris B.
>
> 01/03/2019 23:18 - Mikael Hugo wrote:
>
> > Hi,
> >
> > See this autosupout. The router started loosing OSPF neighbours ang
> > giving BFD errors on multiple links with some hours inbetween.
> >
> > We upgraded to .10 and rebooted.

and lastly a follow up on the progress of fixing it from 24th july 2020.

There were no changes regarding BFD in ROS v6.
It may work and it may not work in specific setups, so general recommendation is not to use it in ROSv6.
Māris B.

/Mikael
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Sun Dec 13, 2020 4:00 am

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.
BFD typically works fine on CHR, and most ARM based Mikrotik's. I use it with no issue on CHR & RB4011. I haven't picked up a 2004 yet as I typically let them stabilize a year before putting them in production.
It might be tilera specific -

> -----Original Message-----
> From: Maris B. [MikroTik Support] <support@mikrotik.com>
> Sent: 04 January 2019 11:05
> Subject: Re: [Ticket#2019010322004734] Router acting wierdly
>
> Hello,
>
> Currently there are known problems with BFD on CCR series routers. I would suggest
> to turn off BFD on those devices until problem is resolved.
>
> Best regards,
> Maris B.
>
> 01/03/2019 23:18 - Mikael Hugo wrote:
>
> > Hi,
> >
> > See this autosupout. The router started loosing OSPF neighbours ang
> > giving BFD errors on multiple links with some hours inbetween.
> >
> > We upgraded to .10 and rebooted.

and lastly a follow up on the progress of fixing it from 24th july 2020.

There were no changes regarding BFD in ROS v6.
It may work and it may not work in specific setups, so general recommendation is not to use it in ROSv6.
Māris B.

/Mikael
There have been some ospf changes from 6.47 to 6.47.7 and 6.47.8. I only recently noticed when setting up some test environment equipment, that it no longer complains about l2mtu mis-match between neighbors anymore. As you likely know, it will cause you issues if they dont match, that isnt apparent at first. I went to update that in the thread about 6.47.8 but it appears to be locked. I am not sure which version killed it, I went straight from 6.47 to 6.47.7 and then to .8 when they had arm stability upgrades in the change log. I am not sure what all other ospf changes were made, but I had realized that one.

I have ospf\bfds running on few CCR (tile) processor devices, and havent noticed issues, until the 2004 (which is arm64 actually)
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 16, 2020 8:04 pm

Hi,

Are any of you experiencing reboots still? We actually havent had any on 6.47.8 yet and the interfaces going down could have been from an RJ45 sfp plug which does makes me wonder about power in the sfp cages on the 2004, but it wasent even connected to anything so was easy to remove.

/Mikael
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Wed Dec 16, 2020 9:16 pm

Hi,

Are any of you experiencing reboots still? We actually havent had any on 6.47.8 yet and the interfaces going down could have been from an RJ45 sfp plug which does makes me wonder about power in the sfp cages on the 2004, but it wasent even connected to anything so was easy to remove.

/Mikael
I have one left in the field, and it has 6.47.8, I just looked and it has an uptime of 20 days. This has been about the upper limit from before. This device not really being used hard at the moment, though it is part of a smaller ospf/mpls network. We shall see if it exceeds ~35 days or so. Its going to become more important soon, so I will start to monitor it a little more closely. The ports have gone up and down some, but thats because the backhaul has been changing, and equipment being moved around there, so those details are not worth watching at the moment, unfortunately.
 
murrayis
just joined
Posts: 6
Joined: Tue Sep 29, 2020 11:57 pm

Re: 2004 hardware issues?

Thu Dec 17, 2020 11:02 pm

We've had one reboot since being on 6.47.8

Device was installed on the 12th this month and rebooted on the 14th :(
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Fri Dec 18, 2020 12:50 am

We've had one reboot since being on 6.47.8

Device was installed on the 12th this month and rebooted on the 14th :(
Do you have any logging running? CPU performance, etc? Traffic?

What sfp's are you using?
 
murrayis
just joined
Posts: 6
Joined: Tue Sep 29, 2020 11:57 pm

Re: 2004 hardware issues?

Fri Dec 18, 2020 7:26 am

We've had one reboot since being on 6.47.8

Device was installed on the 12th this month and rebooted on the 14th :(
Do you have any logging running? CPU performance, etc? Traffic?

What sfp's are you using?
2 x 10gbps 10Gtek SFP+ MM Modules in a LAG

Logs show nothing per normal on Disk or Syslog just "Router was rebooted without a proper shutdown"

Syslog: Device rebooted after 2 days 14 hours 33 minutes -> 202s
 
markonen
just joined
Posts: 8
Joined: Tue Aug 11, 2020 4:28 pm

Re: 2004 hardware issues?

Fri Dec 18, 2020 9:37 am

2 x 10gbps 10Gtek SFP+ MM Modules in a LAG
Sorry to hijack the thread, but what's the performance like on that LAG, and the CPU load?

I've been reluctant to set up bonding on my 2004s as its done in software vs my current setup of hardware bonding on a CRS317.
 
jkaufman
just joined
Posts: 11
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 21, 2020 5:17 pm

Still seeing reboots. Max uptime so far on 6.47.8 is 8 days
 
server8
Member
Member
Posts: 486
Joined: Fri Apr 22, 2011 1:27 pm

Re: 2004 hardware issues?

Tue Dec 22, 2020 9:07 am

Still seeing reboots. Max uptime so far on 6.47.8 is 8 days
We have three of these in production no bgp just OSPF routing and vlan, we still seeing random reboots every X days....
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Dec 22, 2020 5:02 pm

Hi,

Ours have not yet rebooted - all have full BGP - 25 days since last.

We run "simple" ospf/bgp routing. Nothing else. Not too many interfaces (all optical) in either and only 500 mbit of traffic.

The ones with reboots - what differs from ours?

/M
 
jkaufman
just joined
Posts: 11
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Tue Dec 22, 2020 11:52 pm

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Wed Dec 23, 2020 12:29 am

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.
What types of sfp's is yours using ?
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 23, 2020 12:48 am

All fs.com optics in ours.
 
jkaufman
just joined
Posts: 11
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Wed Dec 23, 2020 1:13 am

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.
What types of sfp's is yours using ?
I am using the mikrotik gigabit RJ45 SFPs (S-RJ01)
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Wed Dec 23, 2020 3:02 am

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.
What types of sfp's is yours using ?
I am using the mikrotik gigabit RJ45 SFPs (S-RJ01)
Do you notice any of your interfaces dropping (going up and down) randomly when not expecting it?

On a positive note, the one I still have in service is up to 26 days of uptime.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 23, 2020 10:09 pm

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.
What types of sfp's is yours using ?
I am using the mikrotik gigabit RJ45 SFPs (S-RJ01)
Do you notice any of your interfaces dropping (going up and down) randomly when not expecting it?

On a positive note, the one I still have in service is up to 26 days of uptime.
We did with S-RJ01 on the only 2004 that had one and disabled it because it wasent even connected. We have had no interface drops since then.

Im reading in the changelog of 6.48 and im not sure why it states arm and arm64 stability since these were already in 6.47.8. Does anyone know if they fixed even more stuff?

/Mikael
 
server8
Member
Member
Posts: 486
Joined: Fri Apr 22, 2011 1:27 pm

Re: 2004 hardware issues?

Thu Dec 24, 2020 12:59 pm

Im reading in the changelog of 6.48 and im not sure why it states arm and arm64 stability since these were already in 6.47.8. Does anyone know if they fixed even more stuff?
+1
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sun Dec 27, 2020 6:29 pm

Hi,

We had 2 reboots on 6.47.8 in the last 24 hours. Seems they last longer but still reboots.

/Mikael
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Sun Dec 27, 2020 8:34 pm

Hi,

We had 2 reboots on 6.47.8 in the last 24 hours. Seems they last longer but still reboots.

/Mikael
Mine was going strong until you said that, it died 2.5 hours ago lol, time to pull it I suppose. Has anyone had better success 6.46 long term versions?
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sun Dec 27, 2020 8:36 pm

Mine was going strong until you said that, it died 2.5 hours ago lol, time to pull it I suppose. Has anyone had better success 6.46 long term versions?
Please send ticket to Mikrotik too so they dont think its just me :)

6.46.x is worse im afraid.

/M
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Sun Dec 27, 2020 8:44 pm

Sent another one in.

SUP-37419 (new)
SUP-35544 (yours)
SUP-30924 (my previous)
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 30, 2020 12:09 pm

Sent another one in.

SUP-37419 (new)
SUP-35544 (yours)
SUP-30924 (my previous)
Yeah and sent in some more too - one asking if 6.48 does in fact solve anything more but got a wierd response so I've asked for clarification.

No responses on the main issue in the reboot.

/M
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Wed Dec 30, 2020 11:45 pm

Sent another one in.

SUP-37419 (new)
SUP-35544 (yours)
SUP-30924 (my previous)
Yeah and sent in some more too - one asking if 6.48 does in fact solve anything more but got a wierd response so I've asked for clarification.

No responses on the main issue in the reboot.

/M
I havent tried 6.48, but the thread from 6.48 doesnt look stable at all.

I pulled mine last 2004 from service for now until they are stable. I think I have about 6 of the dang things. They suggested custom firmware for them, but I cant let it keep running like that. I will have to setup a separate test setup to do so.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 30, 2020 11:52 pm


I havent tried 6.48, but the thread from 6.48 doesnt look stable at all.

I pulled mine last 2004 from service for now until they are stable. I think I have about 6 of the dang things. They suggested custom firmware for them, but I cant let it keep running like that. I will have to setup a separate test setup to do so.
Ive been watchinh the 6.48 thread too and it doesent look too good.

Did they suggest custom firmware recently? The only feedback ive been really recieving is that it should be fixed and then no more response on the older tickets.

If it makes you feel any better we have 20 of them :)
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Thu Dec 31, 2020 4:02 am


I havent tried 6.48, but the thread from 6.48 doesnt look stable at all.

I pulled mine last 2004 from service for now until they are stable. I think I have about 6 of the dang things. They suggested custom firmware for them, but I cant let it keep running like that. I will have to setup a separate test setup to do so.
Ive been watchinh the 6.48 thread too and it doesent look too good.

Did they suggest custom firmware recently? The only feedback ive been really recieving is that it should be fixed and then no more response on the older tickets.

If it makes you feel any better we have 20 of them :)
Yes, they suggested custom firmware and wanted serial out information. Though the response was unfortunately after I pulled the unit from service. This unit was located about two hours from my office. A little too far to gamble. I had two in a redundant setup that were ten minutes from my office, which would have been more convenient. But when they started to reboot within hours of each other, I just couldn't risk it.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Dec 31, 2020 1:28 pm

Yes, they suggested custom firmware and wanted serial out information. Though the response was unfortunately after I pulled the unit from service. This unit was located about two hours from my office. A little too far to gamble. I had two in a redundant setup that were ten minutes from my office, which would have been more convenient. But when they started to reboot within hours of each other, I just couldn't risk it.
Please give them my ticket id. We have a couple that can run the custom firmware.

Seems the MT support suggests very different things depending on who picks up the ticket.
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: 2004 hardware issues?

Thu Dec 31, 2020 3:35 pm

Yes, they suggested custom firmware and wanted serial out information. Though the response was unfortunately after I pulled the unit from service. This unit was located about two hours from my office. A little too far to gamble. I had two in a redundant setup that were ten minutes from my office, which would have been more convenient. But when they started to reboot within hours of each other, I just couldn't risk it.
Please give them my ticket id. We have a couple that can run the custom firmware.

Seems the MT support suggests very different things depending on who picks up the ticket.
I had given yours and my prior. I wasnt thinking at the time or I would have update my prior. So I am at fault as well.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Jan 04, 2021 9:32 am

Hi,

Got confirmation that the fixes are in 6.47.8 so no additional fixes in 6.48.

Mikrotik is under the impression this is fixed so please keep reporting if it crashes for you.

We had 2 reboots with 6.47.8, but it's much better and some routers has been up for 38 days now.

/Mikael
 
hoeser
just joined
Posts: 2
Joined: Wed Jan 13, 2021 5:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Jan 13, 2021 5:46 pm

Two crashes so far on 6.48 - doesn't seem fixed to me. Hugely disappointing.
 
welan
newbie
Posts: 39
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 12:58 pm

Same here, 6..48 and a lot of reboots.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 1:04 pm

Same here, 6..48 and a lot of reboots.
We had 2 reboots on 6.47.8 14 days ago in same day for 2 routers. After that nothing. We don't run 6.48.

Are you making tickets? We need to keep the pressure on mikrotik so they realize it's an ongoing issue.

/M
 
welan
newbie
Posts: 39
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 1:08 pm

Just opened and supout sent, i believe it is something to do with ospf and or mpls. I have other 2004 routers but they don't reboot that often, some have uptime as much as 20 days. these instead restart every 7-8 hours (OSPF + MPLS, no firewall, mix of 10G 1G interfaces)
Same here, 6..48 and a lot of reboots.
We had 2 reboots on 6.47.8 14 days ago in same day for 2 routers. After that nothing. We don't run 6.48.

Are you making tickets? We need to keep the pressure on mikrotik so they realize it's an ongoing issue.

/M
 
mikeeg02
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri Mar 30, 2018 2:28 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 6:22 pm

Just opened and supout sent, i believe it is something to do with ospf and or mpls. I have other 2004 routers but they don't reboot that often, some have uptime as much as 20 days. these instead restart every 7-8 hours (OSPF + MPLS, no firewall, mix of 10G 1G interfaces)
Same here, 6..48 and a lot of reboots.
We had 2 reboots on 6.47.8 14 days ago in same day for 2 routers. After that nothing. We don't run 6.48.

Are you making tickets? We need to keep the pressure on mikrotik so they realize it's an ongoing issue.

/M
The ones I had in service were running ospf, mpls, vpls, and not even high volumes of traffic. They had a max uptime of ~30 days before rebooting. Two were side by side in an effective redundant setup, and within ~8 hours of one, the second would reboot. Again after an max of ~30 days. Unfortunately, mine all have been pulled from service at this time. I had better results with 6.47.8, and Im not sure 6.48rc1 was any different in uptime with no other config changes.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 6:25 pm

Just opened and supout sent, i believe it is something to do with ospf and or mpls. I have other 2004 routers but they don't reboot that often, some have uptime as much as 20 days. these instead restart every 7-8 hours (OSPF + MPLS, no firewall, mix of 10G 1G interfaces)
Hi - we actually removed MPLS from our network recently because it was misbehaving and we used it so little. This could be a reson the reboots ceased to occur as often.

The routers rebooting for me on 6.47.8 runs ospf and full BGP. The load is only a couple of hundred megabits. The odd thing was it was only 2 of them. The rest are now above 40 days some reaching 50 days uptime.

2004s with no routing just standing idle never reboots.

/Mikael
 
hoeser
just joined
Posts: 2
Joined: Wed Jan 13, 2021 5:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 7:50 pm

No OSPF or MPLS on mine. Averages between 7 and 30 days between reboots. Low traffic load (averages probably 30 to 50 mbit).

Created SUP-38895.
 
sundalez
just joined
Posts: 9
Joined: Tue Dec 27, 2016 6:42 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Jan 19, 2021 12:39 pm

We have 7 of these routers in production.
My experience with them is:
1 we use as a route reflector. 115 days uptime. (never rebooted). roughly 2 mill routes in RT
4 we use for passing traffic for clients. 3-13 days uptime. (only local routes and default)
2 we have sitting waiting to be put into production. No reboots since we turned them on 37 days ago.

It definitely seems to be related to when these routers are passing traffic. Not really how much traffic, but just passing traffic.

Of the ones passing traffic, we pass between a few 100 mbit to 2 gigabits/s. and they alle seem to reboot randomly equally between them regardless of load.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Jan 19, 2021 12:43 pm

We have 7 of these routers in production.
My experience with them is:
1 we use as a route reflector. 115 days uptime. (never rebooted). roughly 2 mill routes in RT
4 we use for passing traffic for clients. 3-13 days uptime. (only local routes and default)
2 we have sitting waiting to be put into production. No reboots since we turned them on 37 days ago.

It definitely seems to be related to when these routers are passing traffic. Not really how much traffic, but just passing traffic.

Of the ones passing traffic, we pass between a few 100 mbit to 2 gigabits/s. and they alle seem to reboot randomly equally between them regardless of load.
Hi,

We have a similar experience. We have some with full bgp not passing traffic that never reboots, but the ones that do will reboot but some.of these too have never rebooted since 6.47.8. It feels very random like a lockup in fib when traffic has some specific pattern.

/Mikael.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 102
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Jan 20, 2021 9:11 pm

Hi,

We had one crash today - first in some time. Uptime 34 days. The 2004 next to it has been up 45 days and pushes even more traffic so no clear difference.

Made new ticket as we are not getting response on the old ones - SUP-39418

/Mikael

Who is online

Users browsing this forum: No registered users and 48 guests