Community discussions

MikroTik App
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Aug 06, 2020 1:16 am

Hi,

We started deploying 2004s into our network and have issues with one we are trying to add into our bgp core.

It rebooted every 10-14 days so we took it out and replaced in with another one. We can ran memory test and got memory errors on the first one.

This was 12 days ago....

Today the new rebooted same way making lots of issues in OSPF 0.0.0.0 to the extent we had to reboot 2 other routers running 6.45.9 to make traffic resume again. Another CCR1016 got so corrupted that /export compact, snmp etc dident work.

The second CCR2004 (6.47.1) is connected with console cable now and doing remote memory test we get errors on this one too. Broken batch or something else? Feels like too much coincidence perhaps?

Error in address=0x00000000C0004768, W=0xC0004768 R=0x00000000 X=0xC0004768
Error in address=0x00000000C000476C, W=0xC000476C R=0x00000000 X=0xC000476C
Error in address=0x00000000C0004770, W=0xC0004770 R=0x00000000 X=0xC0004770
Error in address=0x00000000C0004774, W=0xC0004774 R=0x00000000 X=0xC0004774

etc etc etc.

/M
Last edited by mhugo on Wed Dec 30, 2020 1:36 pm, edited 1 time in total.
 
joegoldman
Forum Veteran
Forum Veteran
Posts: 766
Joined: Mon May 27, 2013 2:05 am

Re: 2004 hardware issues?

Thu Aug 06, 2020 1:57 am

Something like this is better sent to support@mikrotik.com to start a real case - this is a discussion forum not a proper support channel.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Aug 06, 2020 11:09 pm

Hi,

I asked the question here because it's a forum. I'm fully aware that it's not a support channel.

Mikrotik has no answer for the reboots. Seems they were able to reproduce the memory tester issue so it's confirmed that the memory tester is broken.

They don't think it's related to the reboot.

The system reboots randomly but seldom. I'm trying to attach a console to get output but this is at production site.

If anyone experiences 2004 reboots and tests memory like I did it's not faulty memory being the cause.

The one we have issues with runs 3 full bgp feeds but not much traffic.

The 2004s with no bgp has not rebooted.

/Mikael
 
cwachs
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Sun Aug 16, 2020 5:04 am

I'm having lockups on one of our two 2004s as well. It's an edge router with 4 bgp sessions (no full tables). About every 36 hours, it locks up where we can't access it via winbox, ssh and snmp stops. Sometimes it passes traffic through it, other times no traffic will pass.

We got a console server on it now. Last lockup reported nothing at all in the console but I still had console access. When I tried to get it to generate a supout.rif file via console, that failed but immediately after, it came back to life and I could generate one via winbox.

Our crash 2 days ago it was passing traffic but we lost all access to it including console. Had to pull power to reboot it.

Support ticket is open but so far, no info.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sun Aug 16, 2020 5:59 pm

I have been told by Mikrotik the bootloader memory test has been fixed and fix will be included at next release.

As for the logging we have connected to another mikrotik at the same site and logging to locai file - hoping for a crash to happen.


/M
 
cwachs
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Sun Aug 16, 2020 6:03 pm

What are you all logging to "echo" in hopes of getting useful info in case of a crash? We have "critical, warning, health, system and event" echoing and nothing was on the console at all for our last crash.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Aug 18, 2020 1:07 pm

Not from the log but supposedly there should come some crash information on the console disregarding logging settings.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Fri Aug 21, 2020 4:30 pm

Bootloader fix was included in 6.47.2
 
ofca
Member Candidate
Member Candidate
Posts: 228
Joined: Fri Aug 20, 2004 7:18 pm

Re: 2004 hardware issues?

Fri Aug 21, 2020 7:21 pm

We are seeing same issues with one of 8 CCR2004s: uninvited reboots 1-2 weeks apart. Each one is running 6.47.1, each one is running BGP, so it would seem that it may be a hardware problem. SFP28 is in use on all, so it's also not the reason.
 
cwachs
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Fri Aug 21, 2020 7:23 pm

I am shipping one 2004 back under RMA and our other one reboots every 1 -2 weeks. Was on 6.47 and I just put it on 6.47.2 two days ago. Both are running BGP and OSPF.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sat Sep 05, 2020 1:13 am

Mikrotik is running some special debug packages on one of our routers. Its either a software bug or something deep in the hardware since they mentioned involving the CPU vendor.

Without the debug packages nothing came on console at crash time, so hoping for a new crash soon so this can be resolved. It has currently been up 7 days.

We have 24 2004s in boxes so would be nice to solve this issue.....
 
cwachs
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Sat Sep 05, 2020 1:55 am

we are running the same debug firmware on ours. Every one of our deployed 2004s is having some sort of problem (either random reboots or crashes or both).
 
ofca
Member Candidate
Member Candidate
Posts: 228
Joined: Fri Aug 20, 2004 7:18 pm

Re: 2004 hardware issues?

Sat Sep 05, 2020 3:46 am

2004 that used to reboot every now and then seems stable so far:
/system resource> print 
             uptime: 3w15h17m35s
            version: 6.47.1 (stable)

/system routerboard> print 
             model: CCR2004-1G-12S+2XS
  factory-firmware: 6.46.3
  current-firmware: 6.47.1

I gave it something to do, so it didn't die out of boredom:
> /tool bandwidth-test (...) direction=both protocol=udp local-tx-speed=20000000000 remote-tx-speed=20000000000
                status: running
              duration: 1w6d20h41m35s
            tx-current: 20.0Gbps
  tx-10-second-average: 20.0Gbps
      tx-total-average: 20.0Gbps
            rx-current: 20.0Gbps
  rx-10-second-average: 19.9Gbps
      rx-total-average: 20.0Gbps
          lost-packets: 0
           random-data: no
             direction: both
               tx-size: 9000
               rx-size: 9000
      connection-count: 20
        local-cpu-load: 70%
       remote-cpu-load: 74%
...but it seems there's either no correlation between load and reboots, or there is and load prevents reboots ;)

Anyway, my $0.03 to the case. No instabilities seen on other 2004s either.
 
bugtoodd
just joined
Posts: 10
Joined: Thu Jun 29, 2017 5:54 pm

Re: 2004 hardware issues?

Wed Sep 09, 2020 10:04 am

We are seeing the same problems on two CCR 2004 our of 10 deployed. No Connection tracking enabled. Support says that unless we have the debug package installed with the console, there's no way to catch the problem. Today it happend again on the same units with version 6.47.2: it looks like an hardware issue to me, we will deploy the debug and console and see if we can help finding the problem.
 
cwachs
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Sun Sep 13, 2020 7:38 pm

We just had our 2004 crash with the debug firmware installed. The last line of the console output is:
[admin@AUW-LOOKOUT-EDGE-02] > LOOPER: read_raw read failed: EOF
died with signal
Nothing before that for hours. After the crash, we got 2 physical link up/down messages in console (about 2 minutes after the crash). Nothing else. Router will not respond to console input and we can't log into it. Nor is it passing traffic. I have sent the debug log to support with our open ticket with them. Hopefully this console message is useful to them and they can get this fixed.
 
Abner
just joined
Posts: 14
Joined: Sat Dec 21, 2019 11:31 pm

Re: 2004 hardware issues?

Sun Oct 04, 2020 12:30 am

Any new info about this?


I wonder if this has something to do with it ->

What's new in 6.48beta40 (2020-Sep-14 13:34):
*) arm64 - improved reboot reason reporting in log;
 
cwachs
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Sun Oct 04, 2020 12:32 am

That is related. It was added to help support troubleshoot this. We are running that firmware at the request of Mikrotik to gather more information when it crashes. Has not done anything yet to stop the crashing.
 
angriukas
Member Candidate
Member Candidate
Posts: 103
Joined: Fri Nov 22, 2013 9:20 am
Contact:

Re: 2004 hardware issues?

Fri Oct 09, 2020 6:42 pm

We also have issues with CCR2004, there is no BGP/OSPF in our case. Reboot fully random, sometimes few times per day, sometimes once in two weeks.
CPU load do not exceed 6%, average load 20-25Mbps with rare spikes to 50Mbps. ROS: 6.47.4.
We love MT devices, ...but those reboots are horrible.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Oct 27, 2020 9:55 am

Hi,

We are running 6.48beta48 on some of the 2004s that was rebooting, It seems to have solved the reboots, but it seems that we now face an issue where the routing protocols stops working instead.

Are any of you experiencing the same?

/Mikael
 
FezzFest
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Wed Jun 03, 2015 12:03 am

Re: 2004 hardware issues?

Wed Oct 28, 2020 4:40 pm

Just had the first unexpected reboot on our CCR2004. Running 6.47.4, no autosupout.rif on flash. Very light load, 20-30Mbit/s of traffic.
 
cwachs
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Wed Oct 28, 2020 4:48 pm

After almost a month of troubleshooting with Mikrotik, our 2004's have been RMAd. I would strongly encourage all of you with 2004s that are rebooting, freezing, etc to open support tickets with Mikrotik. They need data to solve this problem and the more data, hopefully the faster it can be solved.
 
jkaufman
just joined
Posts: 12
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Wed Oct 28, 2020 4:51 pm

I opened a ticket with Mikrotik and sent a support file and have not heard back. Did they require that you upgrade to the beta before RMA?
 
cwachs
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Wed Oct 28, 2020 5:02 pm

We did not go to ROS 7 beta but we did run a couple different version 6 betas as part of the test prior to RMA.
 
FezzFest
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Wed Jun 03, 2015 12:03 am

Re: 2004 hardware issues?

Sat Oct 31, 2020 9:23 pm

Experienced our second crash in one week today. No autosupout.rif was generated, sent manual supout.rif to support@mikrotik.com.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Mon Nov 02, 2020 7:58 am

Hi,

Right now we have one 2004 not reachable by winbox, but routing works. It runt 6.48beta48.

The same box dropped ospf last time it had issues.

Previously is rebooted when it had issues and I think that was actually better.

We swapped this with another 2004 so its either a general hardware issue or a software isssue.

Did any of you get improvements after swapping for RMAd boxes as this might have been an issue with early production?

/Mikael
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Nov 24, 2020 12:38 pm

Hi!

Did anyone here get any more updates from MT on this? We are seeing processor hangs in the beta and reboots in stable still. We have a lot of 2004s waiting to go to production, so its frustrating :)

/M
 
cwachs
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Tue Apr 29, 2014 5:55 am

Re: 2004 hardware issues?

Tue Nov 24, 2020 2:44 pm

Nothing with us. RMA'd two of ours and bought CCR 1036s to go in their place. My gut tells me it's going to be a while.
 
User avatar
vectieba
just joined
Posts: 24
Joined: Mon Oct 26, 2009 3:04 pm
Location: South Africa

Re: 2004 hardware issues?

Wed Nov 25, 2020 9:07 am

We have been having similar issues, our previous 2004 just rebooted randomly, we also had problems with PPPoE accounts connecting showing Dynamic but not Running, we then replaced it and it went well for about two weeks. We just had the same issue with the PPPoE accounts again and had to do a reboot to get it working.
 
n3xus
just joined
Posts: 4
Joined: Wed Aug 06, 2014 8:39 pm

Re: 2004 hardware issues?

Wed Nov 25, 2020 2:01 pm

Reading this topic, i am on hold, before buying... I need 6 of them
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Nov 26, 2020 12:19 am

We were told the new beta solved the issue, but we did some fast testing and the whole router behaves strange and becomes unreachable even with mac telnet and romon when OSPF is running.

So test very carefully....
 
bugtoodd
just joined
Posts: 10
Joined: Thu Jun 29, 2017 5:54 pm

Re: 2004 hardware issues?

Fri Nov 27, 2020 1:58 am

6.47.8 (stable) also includes the fix for the freeze/reboot issue and I can confirm that it installs fine and no issues with OSPF/BGP.
We are now testing it on 3 units starting today.
 
glueck05
newbie
Posts: 37
Joined: Fri Jan 26, 2018 12:49 pm

Re: 2004 hardware issues?

Fri Nov 27, 2020 10:10 am

6.47.8 (stable) also includes the fix for the freeze/reboot issue and I can confirm that it installs fine and no issues with OSPF/BGP.
We are now testing it on 3 units starting today.
Same here, we started with two units (doing OSPF and BGP).

regards,
Glueck
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sun Nov 29, 2020 3:12 pm

We have rolled out on a couple too.

I did see this in the release that someone seems to have experienced a reboot even with the new software.

I got the feeling also that BGP convergece with full tables was much slower, but dident time it. Ill try to get some time in the lab to reproduce if possible.

/M
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: 2004 hardware issues?

Sun Nov 29, 2020 6:14 pm

I need to upgrade the CCR2004 in my lab and see if the stability improves. We've used them for several clients but have also had some stability issues.

I think this router will be amazing after a few more months of bug fixes from MikroTik. This is pretty typical of a new router release...it takes a little while to get stable.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 02, 2020 1:29 pm

4 months since the issue started - so im hoping the "amazing" starts now :)

So far its been stable for us since upgrade.

/M
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Thu Dec 03, 2020 3:07 am

I have two 2004s (and a few spares on the shelf), which are neighbors to each other in a small (mid 20s) mpls network, not a lot of throughput, not a lot of firewall rules(typical usage ~1%). Started with 6.47, tried 6.47.7 (which was better) but every so often they would just reboot. And typically within 12 hours of each other. I cant have them doing that, so Ive also ordered a few 1036s as well, and some should be here tomorrow. I also hadnt previously seen when I chose them, the limitations with using rj01 sfps (close proximity and power issues within sfp chassis).

Before the one rebooted this morning, I had logged into it ( I had made a few changes just prior) and saw that overall cpu utilization had jumped to 25% (one of the cores was maxed). It did finally crash and reboot, which seemed to take about 5 minutes for it to come back. Quite honestly I thought I was going to have to netinstall. Before it crashed, I tried to get a support file, but it crashed before it could complete. I had sent in autosup files into them, and they suggested newer versions for debug reasons. Chalk it up to experience I guess, and hang on to them until they become a better work horse.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Dec 03, 2020 4:34 pm

I have two 2004s (and a few spares on the shelf), which are neighbors to each other in a small (mid 20s) mpls network, not a lot of throughput, not a lot of firewall rules(typical usage ~1%). Started with 6.47, tried 6.47.7 (which was better) but every so often they would just reboot. And typically within 12 hours of each other. I cant have them doing that, so Ive also ordered a few 1036s as well, and some should be here tomorrow. I also hadnt previously seen when I chose them, the limitations with using rj01 sfps (close proximity and power issues within sfp chassis).

Before the one rebooted this morning, I had logged into it ( I had made a few changes just prior) and saw that overall cpu utilization had jumped to 25% (one of the cores was maxed). It did finally crash and reboot, which seemed to take about 5 minutes for it to come back. Quite honestly I thought I was going to have to netinstall. Before it crashed, I tried to get a support file, but it crashed before it could complete. I had sent in autosup files into them, and they suggested newer versions for debug reasons. Chalk it up to experience I guess, and hang on to them until they become a better work horse.
Hi this seems to have been fixed in latest stable - 6.47.8
 
abyss
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Sep 21, 2005 10:51 am

Re: 2004 hardware issues?

Fri Dec 04, 2020 7:52 am

Hi,
Our CCR2004 has rebooted without proper shutdown during night with 6.47.8. Not fixed yet. It's not a power problem because this device is in a rack with dual power circuit.
Reboot happens after 3.8 days of running time the 04 December
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Fri Dec 04, 2020 8:01 am


Hi this seems to have been fixed in latest stable - 6.47.8


Thanks for the update, but with the old version it would last up to ~20 some days before it would quit. Sometimes as quick as 8 or so.I do hope they figure it out soon, I have about 10 of these now that will hold paper down, until they are proven again.


Hi,
Our CCR2004 has rebooted without proper shutdown during night with 6.47.8. Not fixed yet. It's not a power problem because this device is in a rack with dual power circuit.
Reboot happens after 3.8 days of running time the 04 December

Thats what I was afraid of. Were you able to log into it shortly before it rebooted? Or do you keep cpu statistics? If so were they higher than normal before reboot?
Hi this seems to have been fixed in latest stable - 6.47.8
 
abyss
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Sep 21, 2005 10:51 am

Re: 2004 hardware issues?

Fri Dec 04, 2020 9:06 am

Thats what I was afraid of. Were you able to log into it shortly before it rebooted? Or do you keep cpu statistics? If so were they higher than normal before reboot?
Yes, of course, nothing really special. It's not related to trafic, because it happens during night when it's much lower.
CCR2004_Uptime.jpg
CCR2004_trafic.jpg
CCR2004_CPU.jpg
CCR2004_Memory.jpg

CCR2004_winbox.jpg
You do not have the required permissions to view the files attached to this post.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Fri Dec 04, 2020 9:15 am

We just had a 2004 have all interfaces go down and up again. Also seems OSPF dident fully recover, so in the end we rebooted it.

/Mikael
 
markonen
just joined
Posts: 23
Joined: Tue Aug 11, 2020 4:28 pm

Re: 2004 hardware issues?

Fri Dec 04, 2020 10:04 am

Went and looked at the uptimes of our six longest-deployed 2004s:

11w3d18h44m51s, S/N D4F00...
10w6d18h10m42s, S/N D4F10...
8w1d16h32s, S/N D4F10...
5w1d16h1m47s, S/N D4F10...
1w6d17h29m41s, S/N D4F00...
1d22h28m59s, S/N C8A60...

The last two have clearly rebooted without human intervention, not 100% sure about the rest.
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Fri Dec 04, 2020 3:26 pm

Thats what I was afraid of. Were you able to log into it shortly before it rebooted? Or do you keep cpu statistics? If so were they higher than normal before reboot?
Yes, of course, nothing really special. It's not related to trafic, because it happens during night when it's much lower.

CCR2004_Uptime.jpg

CCR2004_trafic.jpg

CCR2004_CPU.jpg

CCR2004_Memory.jpg

CCR2004_winbox.jpg

I was more curious if yours had a spike in "unclassified" on one of the cores like I caught mine doing this last time.

Have you guys seen this about the S+RJ10? Not sure if it affects you, you may be using fiber, I know I didnt catch it until recently.

https://wiki.mikrotik.com/wiki/S%2BRJ10 ... l_guidance
 
murrayis
just joined
Posts: 21
Joined: Tue Sep 29, 2020 11:57 pm

Re: 2004 hardware issues?

Mon Dec 07, 2020 1:20 am

We've experienced all that everyone has listed including;
  • Random Reboots
  • Interface Resets
  • System Deadlocks
  • System Slow Downs
We've tried special firmware's, Debugging firmware's, Logging console outputs for months and unfortunately this issues caused out CCR1072's to be unstable and experience similar issues.

We've pulled all CCR2004's from our network and we have stability back again however due to the lack of support and willingness to fix these issues as a priority and getting support responses back like "Are you sure it rebooted" has caused us to make the rash decision to drop Mikrotik from our Core network and spend Hundreds of Thousands of dollars and switch to Juniper.
 
jkaufman
just joined
Posts: 12
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 7:03 am

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:22 pm

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.
What version were you on previously?
 
jkaufman
just joined
Posts: 12
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:40 pm

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.
What version were you on previously?
6.47.4
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:42 pm

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.
What version were you on previously?
6.47.4

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.
 
jkaufman
just joined
Posts: 12
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:45 pm

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.
The longest uptime I saw was 14 or so days. I did NOT upgrade the routerboard firmware yet, just the OS. I have gone ahead and ordered a different CCR model to replace the 2004.
Last edited by jkaufman on Mon Dec 07, 2020 4:47 pm, edited 1 time in total.
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:46 pm

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.
The longest uptime I saw was 14 or so days. I did NOT upgrade the routerboard firmware yet. I have gone ahead and ordered a different CCR model to replace the 2004.
I think my longest was ~mid 20 days. Dont think I ever reached anything above 30. Was there a reason you upgraded to the new version?
 
jkaufman
just joined
Posts: 12
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 4:47 pm

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.
The longest uptime I saw was 14 or so days. I did NOT upgrade the routerboard firmware yet. I have gone ahead and ordered a different CCR model to replace the 2004.
I think my longest was ~mid 20 days. Dont think I ever reached anything above 30. Was there a reason you upgraded to the new version?
The new version was said to have improvements in stability for the arm based devices.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 9:32 pm

Hi,

We havent seen any reboots in 6.47.8 but one 2004 dropped and reconnected all intefaces same second. Was asked by Mikrotik if it was a cable issue - which I dont think since nobody was in the rack or the other sites it connects to. Also disconnecting and reconnecting 8 cables same second would be the fastest person alive.

Besides this we do have some oddity in the ospf in network since a couple versions. Having more than 500 mikrotiks in my network this is honestly not what we are used to. Its been quite stable for years.

Im also surprised by the speed Mikrotik is attacking these issues - Its been months. And when you have an issue like that you need to talk with your customers, not just tell them it will be fixed when found even if it was done in a very polite way.

/M
 
jkaufman
just joined
Posts: 12
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 9:42 pm

Hi,

We havent seen any reboots in 6.47.8 but one 2004 dropped and reconnected all intefaces same second. Was asked by Mikrotik if it was a cable issue - which I dont think since nobody was in the rack or the other sites it connects to. Also disconnecting and reconnecting 8 cables same second would be the fastest person alive.

Besides this we do have some oddity in the ospf in network since a couple versions. Having more than 500 mikrotiks in my network this is honestly not what we are used to. Its been quite stable for years.

Im also surprised by the speed Mikrotik is attacking these issues - Its been months. And when you have an issue like that you need to talk with your customers, not just tell them it will be fixed when found even if it was done in a very polite way.

/M
Did you just upgrade the RouterOS or did you upgrade the firmware as well?
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Mon Dec 07, 2020 10:24 pm

Did you just upgrade the RouterOS or did you upgrade the firmware as well?
Both

/Mikael
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Mon Dec 07, 2020 11:14 pm

Hi,

We havent seen any reboots in 6.47.8 but one 2004 dropped and reconnected all intefaces same second. Was asked by Mikrotik if it was a cable issue - which I dont think since nobody was in the rack or the other sites it connects to. Also disconnecting and reconnecting 8 cables same second would be the fastest person alive.

Besides this we do have some oddity in the ospf in network since a couple versions. Having more than 500 mikrotiks in my network this is honestly not what we are used to. Its been quite stable for years.

Im also surprised by the speed Mikrotik is attacking these issues - Its been months. And when you have an issue like that you need to talk with your customers, not just tell them it will be fixed when found even if it was done in a very polite way.

/M

I may have seen something similar.

When an even happens on one ospf port, it sometimes seems to happen simultaneously on one or several other ports. And the site I've had this happen at has 24/7 camera monitoring, so I know nobody touched ours either.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 12:46 am


I may have seen something similar.

When an even happens on one ospf port, it sometimes seems to happen simultaneously on one or several other ports. And the site I've had this happen at has 24/7 camera monitoring, so I know nobody touched ours either.
If you feel like it then please mail Mikrotik support and reference SUP-35544 which is the ticket we have with this behavior.

/M
 
murrayis
just joined
Posts: 21
Joined: Tue Sep 29, 2020 11:57 pm

Re: 2004 hardware issues?

Tue Dec 08, 2020 6:09 am

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.
 
markonen
just joined
Posts: 23
Joined: Tue Aug 11, 2020 4:28 pm

Re: 2004 hardware issues?

Tue Dec 08, 2020 9:44 am

Have y'all had these issues with the v7 betas?

I know it's a hard question since v7's own bugs can mask this. But as the 2004 was designed for ROS7, I wonder if some of these problems are specific to ROS6.

Unfortunately I don't have any spares to try v7 on...
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: 2004 hardware issues?

Tue Dec 08, 2020 10:41 am

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.
How much? ;-)
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 12:38 pm

Have y'all had these issues with the v7 betas?

I know it's a hard question since v7's own bugs can mask this. But as the 2004 was designed for ROS7, I wonder if some of these problems are specific to ROS6.

Unfortunately I don't have any spares to try v7 on...
Dont try beta3 - It gives massive pingloss for 2004s both on IP and ARP. Same with latest 6.48beta.

/M
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 12:39 pm

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.
What was the reason for rejecting?

/Mikael
 
abyss
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Sep 21, 2005 10:51 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 1:15 pm

I've replaced the CCR2004 with a CCR1072 and it's running like a charm !

I hope that Mikrotik will soon consider making a premium hardware product line in parallel of the cheap hardware race !
RouterOS has so much potential, it's a shame that this software has no more premium rock solid hardware...
 
User avatar
Kickoleg
Member Candidate
Member Candidate
Posts: 129
Joined: Tue Mar 11, 2014 3:13 pm
Location: Yverdon-les-Bains, Suisse

Re: 2004 hardware issues?

Tue Dec 08, 2020 1:19 pm

I've replaced the CCR2004 with a CCR1072 and it's running like a charm !

I hope that Mikrotik will soon consider making a premium hardware product line in parallel of the cheap hardware race !
RouterOS has so much potential, it's a shame that this software has no more premium rock solid hardware...
+1
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 953
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: 2004 hardware issues?

Tue Dec 08, 2020 1:35 pm

I've replaced the CCR2004 with a CCR1072 and it's running like a charm !

I hope that Mikrotik will soon consider making a premium hardware product line in parallel of the cheap hardware race !
RouterOS has so much potential, it's a shame that this software has no more premium rock solid hardware...
You are atacking the wrong problem: this looks like software related, not hardware. I agree that it should be more stable, but (barring some defective units) this kind of complain is usually solved down the line, with a new RoS version.

No, it shouldn't happen - You are quite right about it. But is the software testing that needs improving...
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Dec 08, 2020 3:54 pm

You are atacking the wrong problem: this looks like software related, not hardware. I agree that it should be more stable, but (barring some defective units) this kind of complain is usually solved down the line, with a new RoS version.
I fully agree this is software, but actually MT was a little unsure of this in the beginning so even hardware could be solved with software at some cost I suspect.

I have two wishes this year - First is hoping for stable 2004s on 6.47.x soon and of course stable ROS 7 for new year, but most importantly stable 2004s.

/M
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Tue Dec 08, 2020 11:33 pm


If you feel like it then please mail Mikrotik support and reference SUP-35544 which is the ticket we have with this behavior.

/M
Ill have to put them back in service to do that lol. Ive since replaced them. I will have to try them and either with the older version you started with, or wait for a newer release.

You are atacking the wrong problem: this looks like software related, not hardware. I agree that it should be more stable, but (barring some defective units) this kind of complain is usually solved down the line, with a new RoS version.

No, it shouldn't happen - You are quite right about it. But is the software testing that needs improving...

Wrong to you or not, when you've try a few versions of software, and NEED your equipment to continue running, you have to look for alternative hardware until they have something stable for that hardware. When the problem goes away, you know it was a combination of that hardware/software. When you don't have a unit that's stable, you have to try something else.
Have y'all had these issues with the v7 betas?

I know it's a hard question since v7's own bugs can mask this. But as the 2004 was designed for ROS7, I wonder if some of these problems are specific to ROS6.

Unfortunately I don't have any spares to try v7 on...
Can't run v7 here, running mpls/vpls, and from what I have seen thats not supported yet. Hopefully soon!
 
murrayis
just joined
Posts: 21
Joined: Tue Sep 29, 2020 11:57 pm

Re: 2004 hardware issues?

Wed Dec 09, 2020 1:08 am

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.
What was the reason for rejecting?

/Mikael
Nothing wrong with the hardware....
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 09, 2020 5:53 pm

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.
What was the reason for rejecting?
Nothing wrong with the hardware....
Thats probably true and they will claim software is without warranty I guess which is the real issue. While correct according to license I think they dont understand that we are returning customers. Hopefully.,,

We have 14 of them if it makes you feel better - One we use to test stability.

/M
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Dec 10, 2020 9:09 pm

Hi again,

Today we again experienced a drop of interfaces on a 2004. The drop this time was not all the interfaces, but still all ospf was lost. The other 2004s we are running in test has less traffic and no problems, but one thing I realized is that we are using an SFP28 (DAC Cable) port and that the sfpplus10 is a S-RJ01 which draws much more power than the rest of the interfaces that are normal optical SFPs.

As you can see this happened very fast but because of BGP convergence with CCRs plus no graceful restart it becomes a much longer outage.

Has anyone else seen something like this? We have on the positive side seen no reboots with 6.47.8

13:49:03 snmp,warning timeout while waiting for program 20
13:52:18 interface,info sfp-sfpplus10 link down
13:52:18 interface,info sfp28-2.transit.AS65600 link down
13:52:18 route,ospf,info OSPFv2 neighbor 185.230.33.240: state change from Full to
Down
13:52:18 route,ospf,info OSPFv2 neighbor 192.168.33.235: state change from Full to
Down
13:52:18 route,ospf,info OSPFv2 neighbor 192.168.33.8: state change from Full to Do
wn
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.1
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.8
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.3
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.6
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.234
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=185.230.33.40
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=185.230.33.240
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.231
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.235
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=5.157.4.1
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=193.110.13.254
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.2
13:52:18 route,bgp,error Unexpected UPDATE message
13:52:18 route,bgp,error RemoteAddress=192.168.33.3
13:52:18 route,bgp,error Unexpected UPDATE message
13:52:18 route,bgp,error RemoteAddress=192.168.33.235
13:52:18 route,bgp,info Connection terminated
13:52:18 route,bgp,info RemoteAddress=5.157.4.1
13:52:18 route,bgp,info Connection opened by remote host
13:52:18 route,bgp,info RemoteAddress=193.110.13.254
13:52:18 route,bgp,info Connection opened by remote host
13:52:18 route,bgp,info RemoteAddress=5.157.4.1
13:52:18 route,ospf,info Discarding Database Description packet: wrong neighbor st
ate
13:52:18 route,ospf,info state=Down
13:52:18 route,ospf,info Discarding Database Description packet: wrong neighbor st
ate
13:52:18 route,ospf,info state=Down
13:52:20 interface,info sfp28-2.transit.AS65600 link up (speed 10G, full duplex)
13:52:21 interface,info sfp-sfpplus10 link up (speed 1G, full duplex)
13:52:40 route,bgp,info Connection opened by remote host
13:52:40 route,bgp,info RemoteAddress=192.168.33.234
13:52:41 route,bgp,info Connection opened by remote host
13:52:41 route,bgp,info RemoteAddress=185.230.33.240
13:52:41 route,bgp,info Connection opened by remote host
13:52:41 route,bgp,info RemoteAddress=185.230.33.40
13:52:42 route,bgp,info Connection opened by remote host
13:52:42 route,bgp,info RemoteAddress=192.168.33.3
13:52:42 route,bgp,info Connection opened by remote host
13:52:42 route,bgp,info RemoteAddress=192.168.33.6
13:52:44 route,bgp,info Connection opened by remote host
13:52:44 route,bgp,info RemoteAddress=192.168.33.235
13:52:45 route,bgp,info Connection opened by remote host
13:52:45 route,bgp,info RemoteAddress=192.168.33.1
13:52:47 route,bgp,info Connection opened by remote host
13:52:47 route,bgp,info RemoteAddress=192.168.33.231
13:53:20 route,bgp,info TCP connection established
13:53:20 route,bgp,info RemoteAddress=192.168.33.8
13:53:20 route,bgp,info TCP connection established
13:53:20 route,bgp,info RemoteAddress=192.168.33.2
13:53:20 route,bgp,info Connection opened by remote host
13:53:20 route,bgp,info RemoteAddress=192.168.33.8
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Thu Dec 10, 2020 9:32 pm

Hi again,

Today we again experienced a drop of interfaces on a 2004. The drop this time was not all the interfaces, but still all ospf was lost. The other 2004s we are running in test has less traffic and no problems, but one thing I realized is that we are using an SFP28 (DAC Cable) port and that the sfpplus10 is a S-RJ01 which draws much more power than the rest of the interfaces that are normal optical SFPs.

As you can see this happened very fast but because of BGP convergence with CCRs plus no graceful restart it becomes a much longer outage.

Has anyone else seen something like this? We have on the positive side seen no reboots with 6.47.8

13:49:03 snmp,warning timeout while waiting for program 20
It was my understanding the newer version was supposed to have numerous snmp fixes. I saw that issue lock myself out of a few 1100ahx4's of mine, requiring reboot to regain access.

When mine dropped bfd/ospf connectons, it was using fiber sfp's. Even the two routers connected together with a short fiber jumper. (The other was a longer run to a separate building, though underground and a dark fiber link) I have previously found out that the RJ-01's dont seem to like to hold stable, when used to connect between two routers (RJ-01s on each end), hence why I switched that to fiber. But they seem fine going to a standard ethernet port on one end of the cable. I just didnt have enough fiber sfps on original install, but had RJ-01s.

I have also been experimenting with passive DACs, and they seem better, but I also get them to mess up (ospf/bfds) with the 2004s. (Running 6.47.8) Though I have been pushing them harder for testing purposes.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Dec 10, 2020 10:33 pm

When mine dropped bfd/ospf connectons, it was using fiber sfp's. Even the two routers connected together with a short fiber jumper. (The other was a longer run to a separate building, though underground and a dark fiber link) I have previously found out that the RJ-01's dont seem to like to hold stable, when used to connect between two routers (RJ-01s on each end), hence why I switched that to fiber. But they seem fine going to a standard ethernet port on one end of the cable. I just didnt have enough fiber sfps on original install, but had RJ-01s.
We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

I note you write dropped ospf/bfd but did your interfaces go down too?

/M
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Fri Dec 11, 2020 3:06 am

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

I note you write dropped ospf/bfd but did your interfaces go down too?

/M
I have over 100 wireless microwave paths in my network that spans across the state, I cant live without bfds! Im going from memory, but I believe I have seen the interface go down, but IIRC it wasnt as common as the ospf/bfd drops. IIRC when the interface went down there was usually a RJ-01 involved. Id really like to buy another ~20 or so of these 2004s, they really seem like a nice router, especially at their price point, but I need the reliability for constant voice traffic.
 
User avatar
jspool
Member
Member
Posts: 468
Joined: Sun Oct 04, 2009 4:06 am
Location: Oregon

Re: 2004 hardware issues?

Sat Dec 12, 2020 2:38 am

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

BFD typically works fine on CHR, and most ARM based Mikrotik's. I use it with no issue on CHR & RB4011. I haven't picked up a 2004 yet as I typically let them stabilize a year before putting them in production.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sat Dec 12, 2020 9:01 am

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.
BFD typically works fine on CHR, and most ARM based Mikrotik's. I use it with no issue on CHR & RB4011. I haven't picked up a 2004 yet as I typically let them stabilize a year before putting them in production.
It might be tilera specific -

> -----Original Message-----
> From: Maris B. [MikroTik Support] <support@mikrotik.com>
> Sent: 04 January 2019 11:05
> Subject: Re: [Ticket#2019010322004734] Router acting wierdly
>
> Hello,
>
> Currently there are known problems with BFD on CCR series routers. I would suggest
> to turn off BFD on those devices until problem is resolved.
>
> Best regards,
> Maris B.
>
> 01/03/2019 23:18 - Mikael Hugo wrote:
>
> > Hi,
> >
> > See this autosupout. The router started loosing OSPF neighbours ang
> > giving BFD errors on multiple links with some hours inbetween.
> >
> > We upgraded to .10 and rebooted.

and lastly a follow up on the progress of fixing it from 24th july 2020.

There were no changes regarding BFD in ROS v6.
It may work and it may not work in specific setups, so general recommendation is not to use it in ROSv6.
Māris B.

/Mikael
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Sun Dec 13, 2020 4:00 am

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.
BFD typically works fine on CHR, and most ARM based Mikrotik's. I use it with no issue on CHR & RB4011. I haven't picked up a 2004 yet as I typically let them stabilize a year before putting them in production.
It might be tilera specific -

> -----Original Message-----
> From: Maris B. [MikroTik Support] <support@mikrotik.com>
> Sent: 04 January 2019 11:05
> Subject: Re: [Ticket#2019010322004734] Router acting wierdly
>
> Hello,
>
> Currently there are known problems with BFD on CCR series routers. I would suggest
> to turn off BFD on those devices until problem is resolved.
>
> Best regards,
> Maris B.
>
> 01/03/2019 23:18 - Mikael Hugo wrote:
>
> > Hi,
> >
> > See this autosupout. The router started loosing OSPF neighbours ang
> > giving BFD errors on multiple links with some hours inbetween.
> >
> > We upgraded to .10 and rebooted.

and lastly a follow up on the progress of fixing it from 24th july 2020.

There were no changes regarding BFD in ROS v6.
It may work and it may not work in specific setups, so general recommendation is not to use it in ROSv6.
Māris B.

/Mikael
There have been some ospf changes from 6.47 to 6.47.7 and 6.47.8. I only recently noticed when setting up some test environment equipment, that it no longer complains about l2mtu mis-match between neighbors anymore. As you likely know, it will cause you issues if they dont match, that isnt apparent at first. I went to update that in the thread about 6.47.8 but it appears to be locked. I am not sure which version killed it, I went straight from 6.47 to 6.47.7 and then to .8 when they had arm stability upgrades in the change log. I am not sure what all other ospf changes were made, but I had realized that one.

I have ospf\bfds running on few CCR (tile) processor devices, and havent noticed issues, until the 2004 (which is arm64 actually)
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 16, 2020 8:04 pm

Hi,

Are any of you experiencing reboots still? We actually havent had any on 6.47.8 yet and the interfaces going down could have been from an RJ45 sfp plug which does makes me wonder about power in the sfp cages on the 2004, but it wasent even connected to anything so was easy to remove.

/Mikael
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Wed Dec 16, 2020 9:16 pm

Hi,

Are any of you experiencing reboots still? We actually havent had any on 6.47.8 yet and the interfaces going down could have been from an RJ45 sfp plug which does makes me wonder about power in the sfp cages on the 2004, but it wasent even connected to anything so was easy to remove.

/Mikael
I have one left in the field, and it has 6.47.8, I just looked and it has an uptime of 20 days. This has been about the upper limit from before. This device not really being used hard at the moment, though it is part of a smaller ospf/mpls network. We shall see if it exceeds ~35 days or so. Its going to become more important soon, so I will start to monitor it a little more closely. The ports have gone up and down some, but thats because the backhaul has been changing, and equipment being moved around there, so those details are not worth watching at the moment, unfortunately.
 
murrayis
just joined
Posts: 21
Joined: Tue Sep 29, 2020 11:57 pm

Re: 2004 hardware issues?

Thu Dec 17, 2020 11:02 pm

We've had one reboot since being on 6.47.8

Device was installed on the 12th this month and rebooted on the 14th :(
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Fri Dec 18, 2020 12:50 am

We've had one reboot since being on 6.47.8

Device was installed on the 12th this month and rebooted on the 14th :(
Do you have any logging running? CPU performance, etc? Traffic?

What sfp's are you using?
 
murrayis
just joined
Posts: 21
Joined: Tue Sep 29, 2020 11:57 pm

Re: 2004 hardware issues?

Fri Dec 18, 2020 7:26 am

We've had one reboot since being on 6.47.8

Device was installed on the 12th this month and rebooted on the 14th :(
Do you have any logging running? CPU performance, etc? Traffic?

What sfp's are you using?
2 x 10gbps 10Gtek SFP+ MM Modules in a LAG

Logs show nothing per normal on Disk or Syslog just "Router was rebooted without a proper shutdown"

Syslog: Device rebooted after 2 days 14 hours 33 minutes -> 202s
 
markonen
just joined
Posts: 23
Joined: Tue Aug 11, 2020 4:28 pm

Re: 2004 hardware issues?

Fri Dec 18, 2020 9:37 am

2 x 10gbps 10Gtek SFP+ MM Modules in a LAG
Sorry to hijack the thread, but what's the performance like on that LAG, and the CPU load?

I've been reluctant to set up bonding on my 2004s as its done in software vs my current setup of hardware bonding on a CRS317.
 
jkaufman
just joined
Posts: 12
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Mon Dec 21, 2020 5:17 pm

Still seeing reboots. Max uptime so far on 6.47.8 is 8 days
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: 2004 hardware issues?

Tue Dec 22, 2020 9:07 am

Still seeing reboots. Max uptime so far on 6.47.8 is 8 days
We have three of these in production no bgp just OSPF routing and vlan, we still seeing random reboots every X days....
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Tue Dec 22, 2020 5:02 pm

Hi,

Ours have not yet rebooted - all have full BGP - 25 days since last.

We run "simple" ospf/bgp routing. Nothing else. Not too many interfaces (all optical) in either and only 500 mbit of traffic.

The ones with reboots - what differs from ours?

/M
 
jkaufman
just joined
Posts: 12
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Tue Dec 22, 2020 11:52 pm

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Wed Dec 23, 2020 12:29 am

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.
What types of sfp's is yours using ?
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 23, 2020 12:48 am

All fs.com optics in ours.
 
jkaufman
just joined
Posts: 12
Joined: Wed Dec 21, 2011 6:26 am

Re: 2004 hardware issues?

Wed Dec 23, 2020 1:13 am

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.
What types of sfp's is yours using ?
I am using the mikrotik gigabit RJ45 SFPs (S-RJ01)
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Wed Dec 23, 2020 3:02 am

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.
What types of sfp's is yours using ?
I am using the mikrotik gigabit RJ45 SFPs (S-RJ01)
Do you notice any of your interfaces dropping (going up and down) randomly when not expecting it?

On a positive note, the one I still have in service is up to 26 days of uptime.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 23, 2020 10:09 pm

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.
What types of sfp's is yours using ?
I am using the mikrotik gigabit RJ45 SFPs (S-RJ01)
Do you notice any of your interfaces dropping (going up and down) randomly when not expecting it?

On a positive note, the one I still have in service is up to 26 days of uptime.
We did with S-RJ01 on the only 2004 that had one and disabled it because it wasent even connected. We have had no interface drops since then.

Im reading in the changelog of 6.48 and im not sure why it states arm and arm64 stability since these were already in 6.47.8. Does anyone know if they fixed even more stuff?

/Mikael
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: 2004 hardware issues?

Thu Dec 24, 2020 12:59 pm

Im reading in the changelog of 6.48 and im not sure why it states arm and arm64 stability since these were already in 6.47.8. Does anyone know if they fixed even more stuff?
+1
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sun Dec 27, 2020 6:29 pm

Hi,

We had 2 reboots on 6.47.8 in the last 24 hours. Seems they last longer but still reboots.

/Mikael
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Sun Dec 27, 2020 8:34 pm

Hi,

We had 2 reboots on 6.47.8 in the last 24 hours. Seems they last longer but still reboots.

/Mikael
Mine was going strong until you said that, it died 2.5 hours ago lol, time to pull it I suppose. Has anyone had better success 6.46 long term versions?
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Sun Dec 27, 2020 8:36 pm

Mine was going strong until you said that, it died 2.5 hours ago lol, time to pull it I suppose. Has anyone had better success 6.46 long term versions?
Please send ticket to Mikrotik too so they dont think its just me :)

6.46.x is worse im afraid.

/M
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Sun Dec 27, 2020 8:44 pm

Sent another one in.

SUP-37419 (new)
SUP-35544 (yours)
SUP-30924 (my previous)
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 30, 2020 12:09 pm

Sent another one in.

SUP-37419 (new)
SUP-35544 (yours)
SUP-30924 (my previous)
Yeah and sent in some more too - one asking if 6.48 does in fact solve anything more but got a wierd response so I've asked for clarification.

No responses on the main issue in the reboot.

/M
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Wed Dec 30, 2020 11:45 pm

Sent another one in.

SUP-37419 (new)
SUP-35544 (yours)
SUP-30924 (my previous)
Yeah and sent in some more too - one asking if 6.48 does in fact solve anything more but got a wierd response so I've asked for clarification.

No responses on the main issue in the reboot.

/M
I havent tried 6.48, but the thread from 6.48 doesnt look stable at all.

I pulled mine last 2004 from service for now until they are stable. I think I have about 6 of the dang things. They suggested custom firmware for them, but I cant let it keep running like that. I will have to setup a separate test setup to do so.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Wed Dec 30, 2020 11:52 pm


I havent tried 6.48, but the thread from 6.48 doesnt look stable at all.

I pulled mine last 2004 from service for now until they are stable. I think I have about 6 of the dang things. They suggested custom firmware for them, but I cant let it keep running like that. I will have to setup a separate test setup to do so.
Ive been watchinh the 6.48 thread too and it doesent look too good.

Did they suggest custom firmware recently? The only feedback ive been really recieving is that it should be fixed and then no more response on the older tickets.

If it makes you feel any better we have 20 of them :)
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Thu Dec 31, 2020 4:02 am


I havent tried 6.48, but the thread from 6.48 doesnt look stable at all.

I pulled mine last 2004 from service for now until they are stable. I think I have about 6 of the dang things. They suggested custom firmware for them, but I cant let it keep running like that. I will have to setup a separate test setup to do so.
Ive been watchinh the 6.48 thread too and it doesent look too good.

Did they suggest custom firmware recently? The only feedback ive been really recieving is that it should be fixed and then no more response on the older tickets.

If it makes you feel any better we have 20 of them :)
Yes, they suggested custom firmware and wanted serial out information. Though the response was unfortunately after I pulled the unit from service. This unit was located about two hours from my office. A little too far to gamble. I had two in a redundant setup that were ten minutes from my office, which would have been more convenient. But when they started to reboot within hours of each other, I just couldn't risk it.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: 2004 hardware issues?

Thu Dec 31, 2020 1:28 pm

Yes, they suggested custom firmware and wanted serial out information. Though the response was unfortunately after I pulled the unit from service. This unit was located about two hours from my office. A little too far to gamble. I had two in a redundant setup that were ten minutes from my office, which would have been more convenient. But when they started to reboot within hours of each other, I just couldn't risk it.
Please give them my ticket id. We have a couple that can run the custom firmware.

Seems the MT support suggests very different things depending on who picks up the ticket.
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: 2004 hardware issues?

Thu Dec 31, 2020 3:35 pm

Yes, they suggested custom firmware and wanted serial out information. Though the response was unfortunately after I pulled the unit from service. This unit was located about two hours from my office. A little too far to gamble. I had two in a redundant setup that were ten minutes from my office, which would have been more convenient. But when they started to reboot within hours of each other, I just couldn't risk it.
Please give them my ticket id. We have a couple that can run the custom firmware.

Seems the MT support suggests very different things depending on who picks up the ticket.
I had given yours and my prior. I wasnt thinking at the time or I would have update my prior. So I am at fault as well.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Jan 04, 2021 9:32 am

Hi,

Got confirmation that the fixes are in 6.47.8 so no additional fixes in 6.48.

Mikrotik is under the impression this is fixed so please keep reporting if it crashes for you.

We had 2 reboots with 6.47.8, but it's much better and some routers has been up for 38 days now.

/Mikael
 
hoeser
just joined
Posts: 17
Joined: Wed Jan 13, 2021 5:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Jan 13, 2021 5:46 pm

Two crashes so far on 6.48 - doesn't seem fixed to me. Hugely disappointing.
 
welan
newbie
Posts: 39
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 12:58 pm

Same here, 6..48 and a lot of reboots.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 1:04 pm

Same here, 6..48 and a lot of reboots.
We had 2 reboots on 6.47.8 14 days ago in same day for 2 routers. After that nothing. We don't run 6.48.

Are you making tickets? We need to keep the pressure on mikrotik so they realize it's an ongoing issue.

/M
 
welan
newbie
Posts: 39
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 1:08 pm

Just opened and supout sent, i believe it is something to do with ospf and or mpls. I have other 2004 routers but they don't reboot that often, some have uptime as much as 20 days. these instead restart every 7-8 hours (OSPF + MPLS, no firewall, mix of 10G 1G interfaces)
Same here, 6..48 and a lot of reboots.
We had 2 reboots on 6.47.8 14 days ago in same day for 2 routers. After that nothing. We don't run 6.48.

Are you making tickets? We need to keep the pressure on mikrotik so they realize it's an ongoing issue.

/M
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 6:22 pm

Just opened and supout sent, i believe it is something to do with ospf and or mpls. I have other 2004 routers but they don't reboot that often, some have uptime as much as 20 days. these instead restart every 7-8 hours (OSPF + MPLS, no firewall, mix of 10G 1G interfaces)
Same here, 6..48 and a lot of reboots.
We had 2 reboots on 6.47.8 14 days ago in same day for 2 routers. After that nothing. We don't run 6.48.

Are you making tickets? We need to keep the pressure on mikrotik so they realize it's an ongoing issue.

/M
The ones I had in service were running ospf, mpls, vpls, and not even high volumes of traffic. They had a max uptime of ~30 days before rebooting. Two were side by side in an effective redundant setup, and within ~8 hours of one, the second would reboot. Again after an max of ~30 days. Unfortunately, mine all have been pulled from service at this time. I had better results with 6.47.8, and Im not sure 6.48rc1 was any different in uptime with no other config changes.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 6:25 pm

Just opened and supout sent, i believe it is something to do with ospf and or mpls. I have other 2004 routers but they don't reboot that often, some have uptime as much as 20 days. these instead restart every 7-8 hours (OSPF + MPLS, no firewall, mix of 10G 1G interfaces)
Hi - we actually removed MPLS from our network recently because it was misbehaving and we used it so little. This could be a reson the reboots ceased to occur as often.

The routers rebooting for me on 6.47.8 runs ospf and full BGP. The load is only a couple of hundred megabits. The odd thing was it was only 2 of them. The rest are now above 40 days some reaching 50 days uptime.

2004s with no routing just standing idle never reboots.

/Mikael
 
hoeser
just joined
Posts: 17
Joined: Wed Jan 13, 2021 5:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jan 14, 2021 7:50 pm

No OSPF or MPLS on mine. Averages between 7 and 30 days between reboots. Low traffic load (averages probably 30 to 50 mbit).

Created SUP-38895.
 
sundalez
just joined
Posts: 11
Joined: Tue Dec 27, 2016 6:42 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Jan 19, 2021 12:39 pm

We have 7 of these routers in production.
My experience with them is:
1 we use as a route reflector. 115 days uptime. (never rebooted). roughly 2 mill routes in RT
4 we use for passing traffic for clients. 3-13 days uptime. (only local routes and default)
2 we have sitting waiting to be put into production. No reboots since we turned them on 37 days ago.

It definitely seems to be related to when these routers are passing traffic. Not really how much traffic, but just passing traffic.

Of the ones passing traffic, we pass between a few 100 mbit to 2 gigabits/s. and they alle seem to reboot randomly equally between them regardless of load.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Jan 19, 2021 12:43 pm

We have 7 of these routers in production.
My experience with them is:
1 we use as a route reflector. 115 days uptime. (never rebooted). roughly 2 mill routes in RT
4 we use for passing traffic for clients. 3-13 days uptime. (only local routes and default)
2 we have sitting waiting to be put into production. No reboots since we turned them on 37 days ago.

It definitely seems to be related to when these routers are passing traffic. Not really how much traffic, but just passing traffic.

Of the ones passing traffic, we pass between a few 100 mbit to 2 gigabits/s. and they alle seem to reboot randomly equally between them regardless of load.
Hi,

We have a similar experience. We have some with full bgp not passing traffic that never reboots, but the ones that do will reboot but some.of these too have never rebooted since 6.47.8. It feels very random like a lockup in fib when traffic has some specific pattern.

/Mikael.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Jan 20, 2021 9:11 pm

Hi,

We had one crash today - first in some time. Uptime 34 days. The 2004 next to it has been up 45 days and pushes even more traffic so no clear difference.

Made new ticket as we are not getting response on the old ones - SUP-39418

/Mikael
 
murrayis
just joined
Posts: 21
Joined: Tue Sep 29, 2020 11:57 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Jan 29, 2021 12:13 am

It would be great to get a comment from Mikrotik on this matter - It would also be nice to get a refund on the hardware we cannot use due to it being "Faulty"
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Jan 29, 2021 8:24 am

It would be great to get a comment from Mikrotik on this matter - It would also be nice to get a refund on the hardware we cannot use due to it being "Faulty"
Mikrotik cannot reproduce the issues so they need our help.

"RouterOS 6.48 has improvements for arm64 stability, which for some of our clients has solved an unexpected reboot issue. So if you say that other issues could you please provide all possible information about them and how we can reproduce them at our lab.

About interface drops, we still do not know what may cause them."

So please make tickets and feed them info. We havent had reboots (6.47.8 and 6.48) for more than 30 days so we have had nothing to feed back.

/Mikael
 
murrayis
just joined
Posts: 21
Joined: Tue Sep 29, 2020 11:57 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Feb 01, 2021 12:55 am

We've reported it to Mikrotik on multiple occasions an they've been able to replicate it.

Unfortunately now we have all the CCR2004's out of production so we cannot provide any further production testing - The risk was too high leaving it in the network.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Feb 01, 2021 5:10 pm

We've reported it to Mikrotik on multiple occasions an they've been able to replicate it.

Unfortunately now we have all the CCR2004's out of production so we cannot provide any further production testing - The risk was too high leaving it in the network.
Hi,

Did you try with 6.48 that has fixes for these issues? I have several running 6.47.8 that had some fixes too and most 2004s hasent rebooted since. 3 of them have all and some are not at 65 days uptime. 2 with 35 and one without almost any traffic that rebooted 8 days ago.

/M
 
murrayis
just joined
Posts: 21
Joined: Tue Sep 29, 2020 11:57 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Feb 02, 2021 1:45 am

Yep Tried 6.48 and crashed 5 days later.
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Feb 02, 2021 2:20 am

We've reported it to Mikrotik on multiple occasions an they've been able to replicate it.

Unfortunately now we have all the CCR2004's out of production so we cannot provide any further production testing - The risk was too high leaving it in the network.
This is where I am at on the subject as well. My longest was low 30s for days of uptime. Even with 6.48rc1
 
Robervaldd
newbie
Posts: 40
Joined: Tue Nov 15, 2016 2:11 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Feb 02, 2021 9:03 pm

I have two ccr2004 in our network with traffic of about 1.5gb many problems of packet loss without solution until the moment!
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Feb 02, 2021 9:43 pm

I have two ccr2004 in our network with traffic of about 1.5gb many problems of packet loss without solution until the moment!
Hi,

Are you using 1gbit sfps and especially any rj45s plugs? There is a separate thread for that topic, but when we had them in lab we had issues with rj45 plugs.

/Mikael
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Feb 03, 2021 3:03 pm

Could be interesting

6.49 released.

switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;

viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707

It's a beta, so test it out first.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Feb 11, 2021 11:40 pm

Could be interesting

6.49 released.

switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;

viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707

It's a beta, so test it out first.
We tried it and rolled back. It lost one of the 1 gig ports that started going up and down a lot. Rest seemed ok.

Think its too many changes in one. Would be nice if they could have a 2004 branch based on long-term.
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Feb 13, 2021 3:15 pm

Do NOT buy CCR2004's.

I have x4 CCR2004's (version 6.47.1) in production running OSFP and x2 Full BGP tables and I am experiencing these issues.

Last night I experienced a full OSPF/BGP lockup where the Routers had to be manually rebooted to recover (1 hour drive to the Data Centre at god forsaken hours). Last night was the first time the Routers didn't recover themselves after this weird behaviour. Last Friday night was the first time I noticed the OSPF/BGP lockup, which it auto recovered without intervention from me. I thought this was a DDOS attack or a SNMP issue because we are running Full BGP tables and had SNMP turned on and https://wiki.mikrotik.com/wiki/Manual:BGP_HowTo_%26_FAQ (scroll to the very bottom) says to not have SNMP turned on with Full BGP tables so I turned off SNMP on all our Routers. I experienced another OSPF/BGP lockup on Wednesday that auto recovered. Saying this a few months ago I noticed one of our CCR2004 Router reboot, which I didn't look that far into due to me still setting them up and configuring them. Something worthy of note is that I do notice x1 CPU core maxing out all of the time. It is almost as though every time our eBGP feeds have prefix updates x1 CPU core is maxing out, strange stuff.

Due to this complete OSFP/BGP lockup I decided to go full cowboy and upgrade the version to 6.49beta11. You should NOT run 6.49beta11 in PROD since our BGP peers started flapping like mad on this version. I have now upgraded the version of all our x4 CCR2004's to 6.48.1, which according to this thread still experienced this intermittent OSFP/BGP lockup and reboots. I did want to be a completely insane cowboy and run v7, but I saw someone posted there is a traffic flow bug that we require to work.

It looks like I purchased these CCR2004 Routers on the 6th of July 2020 before this thread was made (this helps clear my conscience on the decision a little). I am incredibly disappointed in these Routers and the only thing that makes sense to me that is causing these issues is the ARM 64bit CPU. I assume other Mikrotik CCR Routers don't have such a critical flaw right? My number one priority is replacing these CCR2004's with something else. I am thinking either the CCR1016-12S-1S+ or CCR1072-1G-8S+ might be a good option? The CCR1072-1G-8S+ is very pricey so I might go with the CCR1016-12S-1S+ once I figure out if all of my 10Gbps cross-connects will be compatible with 1000BASE-LX instead. We require Routers that can handle QinQ, OSPF, x2 Full BGP feeds, 200 ish mangle rules, x8 simple queue's for each customer and about 100Mbps - 1Gbps of traffic.
 
markonen
just joined
Posts: 23
Joined: Tue Aug 11, 2020 4:28 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Feb 14, 2021 10:08 am

I run five 2004s in production with full BGP tables (and a couple more without BGP) and they don’t suffer from the reboots. I’ve tried to pattern match the things people are reporting in this thread and OSPF stands out—I don’t run OSPF but many people reporting reboots here seem to.
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Feb 14, 2021 4:38 pm

What version of RouterOS are you running? I might swap to that.
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Feb 14, 2021 4:47 pm

I run five 2004s in production with full BGP tables (and a couple more without BGP) and they don’t suffer from the reboots. I’ve tried to pattern match the things people are reporting in this thread and OSPF stands out—I don’t run OSPF but many people reporting reboots here seem to.
If you are running a RouterOS version described in this thread that is having the issues I might look at re-designing my network to use RIP/BGP instead of OSFP/BGP lol. Or even just BGP... I am pretty sure Mikrotik don't support EIGRP/IS-IS right?
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Feb 14, 2021 4:51 pm

What version of RouterOS are you running? I might swap to that.
This bug claims to be fixed in 6.47.8 and then some more in 6.48. Ive had reboots on 6.48 but not on 6.47.8 so running 6.47.9 now.

These is also a version 6.49rc11 that fixes some more, but it broke sfp support.

/Mikael
Last edited by mhugo on Sun Feb 14, 2021 9:34 pm, edited 1 time in total.
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Feb 14, 2021 6:20 pm

Ive had reboots on 6.48 but not on 6.47.8 so running 6.49.9 now.
[/quote]

Saying that do you think 6.47.8 is the gold standard since you haven't had any OSPF/BGP lockups or reboots on it?

I am very curious to know what version @markonen is running since he hasn't suffered OSPF/BGP lockups or reboots.
 
markonen
just joined
Posts: 23
Joined: Tue Aug 11, 2020 4:28 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Feb 14, 2021 7:17 pm

I have two that are running 6.47.8 with 75 days and 66 days of uptime, as well as a bunch that I just upgraded from 6.47.4 to 6.48.1 that were between 80 and 120 days of uptime before the upgrade.

Just BGP, static routes and raw firewall rules.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Feb 14, 2021 9:35 pm

I have two that are running 6.47.8 with 75 days and 66 days of uptime, as well as a bunch that I just upgraded from 6.47.4 to 6.48.1 that were between 80 and 120 days of uptime before the upgrade.

Just BGP, static routes and raw firewall rules.
Out of curiosity - how much bgp do you run on them? Full feeds?

/Mikael
 
markonen
just joined
Posts: 23
Joined: Tue Aug 11, 2020 4:28 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Feb 15, 2021 8:00 am

Full feeds from one or two transits (over both IPv4 and IPv6, so 2-4 sessions) + some peering. So one CPU core is more or less pegged doing BGP.
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Feb 15, 2021 8:39 am

Full feeds from one or two transits (over both IPv4 and IPv6, so 2-4 sessions) + some peering. So one CPU core is more or less pegged doing BGP.
Thank you so much for this information @markonen

My plan of attack is to downgrade to 6.47.8 with OSPF and x2 Full BGP Tables.

If the OSPF/BGP lockup or reboot happens again I will re-design my network to be a RIP/BGP or Static Routes/BGP design.
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Feb 15, 2021 8:42 am

@markonen what are your BGP peer uptimes? Maybe you have experienced a BGP lockup without noticing it?

I have been impacted by OSPF/BGP lockups a lot more than reboots.
 
markonen
just joined
Posts: 23
Joined: Tue Aug 11, 2020 4:28 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Feb 15, 2021 12:48 pm

Longest peer uptime is 32 days but IIRC they had scheduled maintenance then. I haven't really paid attention to these enough to really say whether it's the remote or some change we did that reset the other sessions (or if it was indeed a lockup, I guess).
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Feb 15, 2021 5:11 pm

Very interesting, It potentially could of been a lockup. I really am hoping it wasn't because that would mean I have to buy new Routers :(
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Feb 15, 2021 7:03 pm

Very interesting, It potentially could of been a lockup. I really am hoping it wasn't because that would mean I have to buy new Routers :(
This could be related to port extender issues. See 6.49 beta. We are running 3 of these in production but beware there is a bug if any of the interfaces at 1G and not 10G. They all come up in 10G and becomes unstable.

/M
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Feb 16, 2021 2:37 pm

What version of RouterOS are you running? I might swap to that.
This bug claims to be fixed in 6.47.8 and then some more in 6.48. Ive had reboots on 6.48 but not on 6.47.8 so running 6.47.9 now.

These is also a version 6.49rc11 that fixes some more, but it broke sfp support.

/Mikael
@mhugo In regards to the reboots you experienced on 6.48, how certain are you these weren't power related? Is your Router in a Data Centre power connected to A and B rails?
 
PacketNomad
just joined
Posts: 1
Joined: Wed Dec 30, 2020 12:35 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Feb 17, 2021 12:47 am

Full feeds from one or two transits (over both IPv4 and IPv6, so 2-4 sessions) + some peering. So one CPU core is more or less pegged doing BGP.
Out of interest, what is your route count? and do you see any packet loss?
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Feb 17, 2021 10:41 am

Full feeds from one or two transits (over both IPv4 and IPv6, so 2-4 sessions) + some peering. So one CPU core is more or less pegged doing BGP.
Out of interest, what is your route count? and do you see any packet loss?
We see slight packet loss on routers even with 150k routes. I don't think it's more or less on the ones with 2-3 full.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Feb 19, 2021 11:31 pm

Got a reboot on one now on 6.49beta11.

/Mikael
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Feb 23, 2021 5:47 pm

I had another OSPF/BGP lockup running 6.48.1, it auto recovered.
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Feb 23, 2021 6:52 pm

I have just re-designed my network to be Static Routes/BGP instead of OSPF/BGP.

I hope removing OSPF from our design has resolved these issues. If not I think my only option is to replace these CCR2004's.

Does anyone have any thoughts on Full BGP tables being too much for the CCR2004's to handle (CPU, etc)?
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Feb 23, 2021 10:15 pm

I have just re-designed my network to be Static Routes/BGP instead of OSPF/BGP.

I hope removing OSPF from our design has resolved these issues. If not I think my only option is to replace these CCR2004's.

Does anyone have any thoughts on Full BGP tables being too much for the CCR2004's to handle (CPU, etc)?
I dont have lockups in ospf/bgp. We have seen interface down/ups and resets.

/Mikael
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 953
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Feb 24, 2021 4:36 am

Does anyone have any thoughts on Full BGP tables being too much for the CCR2004's to handle (CPU, etc)?
Take a look at the "Performance Status" section.
https://help.mikrotik.com/docs/display/ ... col+Status

They show the results for 6.44 and 7.1beta3.
Yes, You wouldn't use 7.1beta3 in production - but 6.44 is a good example of what to expect from the 6.x series.
 
byko
just joined
Posts: 3
Joined: Wed Dec 18, 2019 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Feb 24, 2021 9:50 am

6.48 ospf and bgp
no snmp, firewall, nat

reboots once per day

SOLUTION: turn off routeros monitoring in dude .
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Feb 24, 2021 2:33 pm

I dont have lockups in ospf/bgp. We have seen interface down/ups and resets.
Hmm, maybe that is what I have been experiencing too. I haven't noticed any syslogs for the interface going up/down though. The syslogs that I have noticed are "OSFPv2 neighbor x.x.x.x; state change from Full to Down.

I'll have to wait and see if removed OSPF from my network design has resolved the issue or not. @markonen is running Static Routes/BGP and hasn't experienced any of these issues we are, fingers crossed he was onto something.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Feb 24, 2021 2:35 pm

I dont have lockups in ospf/bgp. We have seen interface down/ups and resets.
Hmm, maybe that is what I have been experiencing too. I haven't noticed any syslogs for the interface going up/down though. The syslogs that I have noticed are "OSFPv2 neighbor x.x.x.x; state change from Full to Down.

I'll have to wait and see if removed OSPF from my network design has resolved the issue or not. @markonen is running Static Routes/BGP and hasn't experienced any of these issues we are, fingers crossed he was onto something.
BGP on loopback is not tied to the interface like OSPF is.

This interface flapping is supposedly fixed in 6.49beta11.

/M
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Feb 24, 2021 2:50 pm

I tried 6.49beta11 and BGP was flapping like mad on it, certainly don't run that version in production.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Feb 24, 2021 2:59 pm

I tried 6.49beta11 and BGP was flapping like mad on it, certainly don't run that version in production.
Are you using 1Gs - optics or rj45?
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Feb 25, 2021 3:42 pm

Are you using 1Gs - optics or rj45?
Yes, some of my cross-connects are 1000BASE-LX 1Gbps. I am pretty sure the Router <-> Router 10Gbps iBGP was going up and down on the 6.49beta11.
 
mod3m
just joined
Posts: 5
Joined: Thu Jan 16, 2020 10:29 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Feb 27, 2021 6:07 am

Also faced this problem. It worked for more than a year, first at CCR1009 - uptime for more than 365 days. About a month ago, they installed CCR2004, on the same kind of config, worked for 3 weeks, rebooted. Today again, less than a week has passed, rebooted.

The config is as minimal as possible, only BGP and iBGP. 4 active BGP sessions, 2 full-view. Latest long term firmware. Traffic is about 1.5 gigabits at its peak. After rebooting, one of the peering session does not get up, you need to switch it by hand. The link is connected with copper, through the sfp module 10G GL-CC-SFP-030.

SNMP enabled. Firewall, OSPF disabled
Last edited by mod3m on Sat Feb 27, 2021 6:28 am, edited 1 time in total.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Feb 28, 2021 10:02 pm

Are you using 1Gs - optics or rj45?
Yes, some of my cross-connects are 1000BASE-LX 1Gbps. I am pretty sure the Router <-> Router 10Gbps iBGP was going up and down on the 6.49beta11.
1G in 6.49beta11 did not seem to fully work as they negotiated in 10G even when setting to low speed.

Just swapped some 2004s for 1072. Feels sad when router is running 200-300 mbit to have to buy 1072 with less ports and higher cost due to Mikrotiks inability to fix the problem and even worse not really communicate about it.

Starting to feel like having stable 2004s require routeros 7. I would suggest mikrotik to skip mpls for the first release and make a 7.1 with that to focus efforts and perhaps solve some peoples issues who can do without mpls from the beginning.

/M
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Mar 01, 2021 1:02 pm

Just swapped some 2004s for 1072.
That is sad you had to resort to replacing the CCR2004's. Do you have any plan for returning them and getting your money back or anything along those lines?
Starting to feel like having stable 2004s require routeros 7
Yes I am starting to get that feeling as well. If I get one more lockup or reboot after changing my network design from OSPF/BGP to Static Routes/BGP then I am swapping in either CCR1036-8G-2S+'s or CCR1072-1G-8S+'s.

If ROS 7 is required for stable CCR2004's then I hope they get OSPF, BGP, VLANs, Traffic Flow, QinQ, DHCP, Simple Queues, Firewall, Routing Filters, and RADIUS operational.
 
OlofL
Member Candidate
Member Candidate
Posts: 113
Joined: Mon Oct 12, 2015 2:37 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Mar 03, 2021 10:15 pm

I now have one CCR2004 in prod. Uptime only 10 days, but no issues so far.
Running 6.48.1.
Very simple configuration-
Only two static routes, some mangle rules and a few firewall rules.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Mar 03, 2021 10:17 pm

I now have one CCR2004 in prod. Uptime only 10 days, but no issues so far.
Running 6.48.1.
Very simple configuration-
Only two static routes, some mangle rules and a few firewall rules.
I dont think (or at least hope) you will have any issues. I have had no reboots on the 2004s with simple configs. The issues are the ones with ospf (we have 50 in some areas) and/or BGP with full or at least 150K routes.

/Mikael
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Mar 04, 2021 1:51 pm

I had another lockup after changing from a OSPF/BGP to Static Routes/BGP Network Design.

CCR1036-8G-2S+'s have been ordered and I will be swapping out all of my CCR2004's ASAP.

In my opinion Mikrotik should stop selling CCR2004's until ROSv7 is up to scratch and has confirmed resolved these issues. Having CCR2004's in my network has damaged my ISPs reputation, I am not happy.
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Mar 04, 2021 1:56 pm

Very simple configuration-
Only two static routes, some mangle rules and a few firewall rules.
For your sake I hope the issues I have experienced are caused due to running x2 Full BGP tables. If I was you I would be planning for it to experience lockups/reboots.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Mar 04, 2021 2:20 pm

Very simple configuration-
Only two static routes, some mangle rules and a few firewall rules.
For your sake I hope the issues I have experienced are caused due to running x2 Full BGP tables. If I was you I would be planning for it to experience lockups/reboots.
It's not I think. We have had issues with less than full bgp at 100k and no issues with routers with multiple full bgp.

/M
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Mar 04, 2021 2:44 pm

CCR1036-8G-2S+'s have been ordered and I will be swapping out all of my CCR2004's ASAP.
Thats the route I went (replaced with 1036s). No reboots since, 100+ days. Not running bgp here, but an ospf/mpls network. Max uptime was 30ish days with 2004s for me. No matter what version.
 
cwachs
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Tue Apr 29, 2014 5:55 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Mar 05, 2021 1:59 am

We pulled both our 2004's from service and replaced them with 1036's as well.

One of our 2004s was sent to Mikrotik (at their request through a SUP). We opened the ticket back in July. The RMA was approved in August and the 2004 was boxed and shipped to Baltic Networks for return to Mikrotik in early September. Last week (March), it was shipped back to me. The box was never opened and the router was never touched. They shipped it back to Baltic Networks stating it was repaired so Baltic returned it to me.

So unimpressed with this entire process. Sorry for all of you that bought more of these and are trying to run businesses with them. I'm, lucky I only got stuck with two of them.
 
Rudiger420
just joined
Posts: 23
Joined: Sat Feb 13, 2021 2:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Mar 06, 2021 1:26 pm

One of our 2004s was sent to Mikrotik (at their request through a SUP). We opened the ticket back in July. The RMA was approved in August and the 2004 was boxed and shipped to Baltic Networks for return to Mikrotik in early September. Last week (March), it was shipped back to me. The box was never opened and the router was never touched. They shipped it back to Baltic Networks stating it was repaired so Baltic returned it to me.
It sounds like I have no chance at getting my money back then :( I guess I have just sunk $4,000 down the drain until ROS7 is up to scratch.
Thats the route I went (replaced with 1036s). No reboots since, 100+ days. Not running bgp here, but an ospf/mpls network.
It sounds like I have made the right choice getting 1036s then.

I am a new Mikrotik customer so getting CCR2004's hasn't been a good first time experience.
 
murrayis
just joined
Posts: 21
Joined: Tue Sep 29, 2020 11:57 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Mar 22, 2021 2:55 am

Tested the latest RC Friday just past and the 2004 lasted all of 8 hours before rebooting...
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Mar 24, 2021 12:47 am

Tested the latest RC Friday just past and the 2004 lasted all of 8 hours before rebooting...
For me they have been stable - we have 7 running RC and it solved most of packetloss.

/Mikael
 
hugues
just joined
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Apr 16, 2021 4:57 pm

Hello, 3 CCR2004 Here, we are experiencing packet loss issues :

Image

We do have another issue related to OSPF and neighbors getting down for no reason :

Image

i tried woth both 6.49 Beta and 6.48.2 stable, but i didn't see any difference. Is anyone at MikroTik working on this issue ?
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Apr 16, 2021 5:35 pm

Hello, 3 CCR2004 Here, we are experiencing packet loss issues :

What speeds are going through it...what protocols are you running?
 
hugues
just joined
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Apr 18, 2021 6:38 am

Hello, 3 CCR2004 Here, we are experiencing packet loss issues :

What speeds are going through it...what protocols are you running?
Less than 1gbps per router, with OSPF and 200k routes v4+v6 BGP.
 
wintech2003
just joined
Posts: 10
Joined: Fri Jun 09, 2006 6:56 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Apr 29, 2021 10:05 pm

Anybody tried 6.48.2 (2021-Apr-09 10:17)?
*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
*) switch - improved system stability with 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Apr 30, 2021 12:03 am

Running 14 on 6.48.2 they modified the timing so we have had two issues with them getting bad routes after large network changes but not reboots.

Waiting for a stable 7 for some working bgp....
 
wintech2003
just joined
Posts: 10
Joined: Fri Jun 09, 2006 6:56 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Apr 30, 2021 12:14 pm

Waiting for a stable 7 for some working bgp....
"Soon" :D
 
bugtoodd
just joined
Posts: 10
Joined: Thu Jun 29, 2017 5:54 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun May 02, 2021 1:56 pm

I have now three different units running on 6.48.2 still experiencing random reboots: they last from a few hours (6 hours) to several days before rebooting: they are rebooted by watchdog timer (router was rebooted without proper shutdown by watchdog timer) - I have console cables on all of them logging on a server the output, but there's nothing printed there before the reboot.
I also offered to Support several times to install the debug firmware, but they are not responding to tickets anymore.
 
hoeser
just joined
Posts: 17
Joined: Wed Jan 13, 2021 5:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon May 17, 2021 5:29 pm

I have now three different units running on 6.48.2 still experiencing random reboots: they last from a few hours (6 hours) to several days before rebooting: they are rebooted by watchdog timer (router was rebooted without proper shutdown by watchdog timer) - I have console cables on all of them logging on a server the output, but there's nothing printed there before the reboot.
I also offered to Support several times to install the debug firmware, but they are not responding to tickets anymore.
Exact same scenario here, single unit, very low throughput use case. Lasts anywhere from 4 to 20 days on average before a reboot. Nothing dumped to the console. I'm definitely getting tired of this router and I've lost confidence with Mikrotik as a result of using it.
 
mhugo
Member Candidate
Member Candidate
Topic Author
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon May 17, 2021 8:53 pm

Exact same scenario here, single unit, very low throughput use case. Lasts anywhere from 4 to 20 days on average before a reboot. Nothing dumped to the console. I'm definitely getting tired of this router and I've lost confidence with Mikrotik as a result of using it.
We actually dont have that issue and have 20 in production. We run full bgp and ospf but nothing else on them. How are you using the router?

/Mikael
 
hoeser
just joined
Posts: 17
Joined: Wed Jan 13, 2021 5:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon May 17, 2021 11:31 pm

Exact same scenario here, single unit, very low throughput use case. Lasts anywhere from 4 to 20 days on average before a reboot. Nothing dumped to the console. I'm definitely getting tired of this router and I've lost confidence with Mikrotik as a result of using it.
We actually dont have that issue and have 20 in production. We run full bgp and ospf but nothing else on them. How are you using the router?

/Mikael
I use it as a relative simple NAT firewall and edge router running a 1000mbit down/50 up commercial cable connection with a /29 public and a couple dozen internal private addresses and about 50 rules in the firewall. No BGP, no OSPF.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jun 10, 2021 12:35 pm

I am small ISP from India.
I bought 2004 for better specs.
but now, this CCR reboots in week.
Previously, I was on 6.47.7 and now on 6.48.3.
but no improvements.
Actual 6.48.3 reboots more often.
.
1G SFP modules on 3 sfp slots.
traffic between 100Mbps to 200Mbps. 170+ PPP clients.
 
hoeser
just joined
Posts: 17
Joined: Wed Jan 13, 2021 5:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Jun 14, 2021 5:14 pm

I am small ISP from India.
I bought 2004 for better specs.
but now, this CCR reboots in week.
Previously, I was on 6.47.7 and now on 6.48.3.
but no improvements.
Actual 6.48.3 reboots more often.
.
1G SFP modules on 3 sfp slots.
traffic between 100Mbps to 200Mbps. 170+ PPP clients.
After a couple of weeks of running 6.48.3 I can report the same. Reboots are more frequent. I'm just about ready to throw this thing out the window.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 2855
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Jun 14, 2021 5:28 pm

@hoeser:

Could you please use "Post reply" button instead of quotting previous posts all the time.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Jun 14, 2021 7:10 pm

Anybody tried 6.48.2 (2021-Apr-09 10:17)?
*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
*) switch - improved system stability with 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
6.48.2 still packet loss (0.4 %) on SFP28 10 giga and still reboot around every 2 weeks: 2.5 Gb/s peak traffic and OSPF, no bgp
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jul 01, 2021 10:29 pm

For the diehards still trying to use these, any updates on stability? Been quiet in here.
 
stuckoff
just joined
Posts: 2
Joined: Wed Jan 31, 2018 3:28 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Jul 04, 2021 10:30 pm

Not much.
24.06.2021 04:35 - router was rebooted without proper shutdown by watchdog timer (uptime ~7 days)
04.07.2021 15:25 - router was rebooted without proper shutdown by watchdog timer (uptime ~9.5 days)
So reboots continues.
This is with the latest stable release (6.38.3)

Regards,
stuckoff
 
stuckoff
just joined
Posts: 2
Joined: Wed Jan 31, 2018 3:28 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Jul 04, 2021 10:33 pm

Hi All,

Is there anyone that is using the LTS release (6.47.10) on CCR2004 with BGP?
Is it unstable as the "stable" release?

Regards,
stuckoff
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Jul 05, 2021 8:55 am

We tried several versions and now we are on the last, but still reboots every 5/10 days and packet loss <%0,2
 
Abner
just joined
Posts: 14
Joined: Sat Dec 21, 2019 11:31 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Jul 05, 2021 11:53 pm

Up and Running.. 2 Full tables and both 10G and 1G Links. To me, most stable is 6.47.8.

Vendor/Hardware Mikrotik CCR2004-1G-12S+2XS
Operating system Mikrotik RouterOS 6.47.8 (Level 6)
Uptime 47D 6h 6m 51s


Hi All,

Is there anyone that is using the LTS release (6.47.10) on CCR2004 with BGP?
Is it unstable as the "stable" release?

Regards,
stuckoff
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Jul 14, 2021 11:34 pm

I am becoming crazy with reboots anyone has better results with 6.48.3?
 
Cablenut9
Long time Member
Long time Member
Posts: 542
Joined: Fri Jan 08, 2021 5:30 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Jul 14, 2021 11:43 pm

anyone has better results with 6.48.3?
https://tryitands.ee

Anyway, considering all the fixes in 6.48.3, I would expect there to be some improvement with the CCR2004.
 
owireless
just joined
Posts: 4
Joined: Wed Dec 02, 2020 2:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Jul 20, 2021 9:18 am

Wish I knew about this thread before buying a 2004.

Now sitting with an unresponsive 2004 in a datacentre, having to route our traffic through a colleagues network.

Running 6.48.3 / BGP and OSPF.

Had one reboot with just OSPF running, no packet loss or interface drops like others. Just watchdog timer rebooting. Now that BGP is on, it lasts a few days between reboot (and stall as it is).

No copper, just 3x UF-SM-10G
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Jul 23, 2021 12:21 pm

Guys, any update about reboot issue ?
I raised ticket and they are now asking me shitty CLI things, as if this is unknown to them.
If such poor support from mikrotik, why network guys suggest it ?
I am totally disappointed by mikrotik and never buy any device again because poor support.
Also, referring same to others, don't buy mikrotik
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Jul 23, 2021 2:23 pm

... and they are now asking me shitty CLI things...
... why network guys suggest it ?

Mikrotik devices (running ROS in particular) are not really something to recommend to people with attitude towards CLI as you have. Most network guys, who know their stuff (be it Cisco, Juniper, ... or Mikrotik), like CLI very much, hence recommendations for Mikrotik as home router.
 
jackrabbit
just joined
Posts: 10
Joined: Tue Jul 07, 2020 1:28 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Jul 23, 2021 5:34 pm

Experiencing continued CCR2004 watchdog reboots on 6.48.3. Using BGP and OSPF. No copper SFPs, all 10G fiber transceivers. Average reboot every 4-12 days.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Jul 23, 2021 11:50 pm

CCR2004 trash hardware not usable in a professional network.

Guys, any update about reboot issue ?
I raised ticket and they are now asking me shitty CLI things, as if this is unknown to them.
If such poor support from mikrotik, why network guys suggest it ?
I am totally disappointed by mikrotik and never buy any device again because poor support.
Also, referring same to others, don't buy mikrotik
 
Cablenut9
Long time Member
Long time Member
Posts: 542
Joined: Fri Jan 08, 2021 5:30 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Jul 24, 2021 12:02 am

CCR2004 trash hardware not usable in a professional network.
What's the alternative? The equivalent Cisco would cost 100 times as much.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Jul 24, 2021 9:39 am

What's the alternative? The equivalent Cisco would cost 100 times as much.
I agree but we have 4 of these in production and it's a disaster long watchdog reboots every weeks, we move back to old CCR1016 for non 10 gbits link for 10 gibt routed link we are looking for a solutions
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Jul 29, 2021 6:23 pm

It seems be people are using this router for BGP sessions to upstream AND NAT/Connection State tracking.

NAT/Connection Tracking should be disabled on edge routers: https://mum.mikrotik.com/presentations/ ... 231376.pdf
 
toxicfusion
Member Candidate
Member Candidate
Posts: 267
Joined: Mon Jan 14, 2013 6:02 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Jul 30, 2021 6:45 am

Question for those running CCR2004 in production and having issues.

Is consensus to run 6.47.8 LTS or has anyone tried 6.47.10 LTS???

Lastly, What happens if you reduce the CPU clockspeed, underclock CPU? Have you gained stability?
 
ZPrime
just joined
Posts: 22
Joined: Wed May 07, 2014 3:14 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 03, 2021 2:44 am

I have a CCR2004 running 6.47.9 (LTS) and it has been up for over 138 days right now.

It is not running any dynamic routing though - all static routes. It does have a fairly comprehensive firewall in place (33 filter rules with fasttrack enabled when possible, along with 14 NAT rules, and 85 total items in Address Lists). I have 5x 10Gb in use - two 2x LACP bonding interfaces, and one more 10Gb that is an uplink to a different router for WAN access (this is acting as a server firewall, so 2x10Gb LACP goes to the server switch, and another 2x10Gb is "client facing", with 1x10Gb going up to the WAN router).

This particular CCR2004 has not crashed that I know of.

However, we have two or three other CCR2004s in production that are doing iBGP against a juniper (not a ton of prefixes, maybe 20-30?) and don't seem to like to stay up more than 10-15 days at a time. So, something in the dynamic routing seems to be the root cause here, but I'm not 100% on that.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 03, 2021 9:14 am

We are running 4 ccr2004 all with ospf and have random crash weekly
 
toxicfusion
Member Candidate
Member Candidate
Posts: 267
Joined: Mon Jan 14, 2013 6:02 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Aug 04, 2021 8:56 pm

We are running 4 ccr2004 all with ospf and have random crash weekly
What happens if you underclock the CPU? Reduce the CPU Frequency a few steps. Does the device become stable? This is out of curiosity. Just trying to find solution for those that have CRS2004 units that crash while using routing protocols.
 
popcorrin
Member Candidate
Member Candidate
Posts: 196
Joined: Wed Mar 11, 2009 12:55 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Aug 05, 2021 7:20 pm

I'm always impressed with the professionalism of the mikrotik customer base. The reboot issue with the CCR2004 has been going on for an extremely long time. Many customers have invested a substantial sum of money in this particular router. Yet, the customers display extreme patience and restraint. Mikrotik however has been much less than professional. They have yet to fix the issue and worse, they can't even communicate with their loyal customer base on the status of this issue. What they are doing to fix it, how long before we can expect a solution. If no solution is on the horizon, what are they going to compensate us with. This needs to be addressed and addressed immediately.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Aug 06, 2021 9:56 am

What happens if you underclock the CPU? Reduce the CPU Frequency a few steps. Does the device become stable? This is out of curiosity. Just trying to find solution for those that have CRS2004 units that crash while using routing protocols.
We swapped 2 of 4 with ccr106 tonight, no frequency control under system routerboard settings
 
djdrastic
Member
Member
Posts: 367
Joined: Wed Aug 01, 2012 2:14 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Aug 06, 2021 10:14 am

I'm always impressed with the professionalism of the mikrotik customer base. The reboot issue with the CCR2004 has been going on for an extremely long time. Many customers have invested a substantial sum of money in this particular router. Yet, the customers display extreme patience and restraint. Mikrotik however has been much less than professional. They have yet to fix the issue and worse, they can't even communicate with their loyal customer base on the status of this issue. What they are doing to fix it, how long before we can expect a solution. If no solution is on the horizon, what are they going to compensate us with. This needs to be addressed and addressed immediately.
This is very reminiscent of CCR's on the 5.XX code train. They were just a liability until 6.X train came out to get some stability on that platform.
If you think this is bad, just look at the CCR1072 issues . Been out for years now , cost like $3K and little to no input or feedback from Mikrotik on all the ongoing issues.
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6694
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Aug 06, 2021 11:54 am

I'm running CCR2004-1G-12S+2XS as one of upstream gateways, (OSPF, IBGP, >100 firewall rules). The latest reboot was after upgrade to 6.48.3 (both RouterOS and firmware), there were no unexpected reboots on it at the current version.
 
blingblouw
Member
Member
Posts: 345
Joined: Wed Aug 25, 2010 9:43 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Aug 06, 2021 12:00 pm

weird response, this is a pretty active thread, I don't think everyone here is lying?

This thread is the reason I don't have any 2004's, I don't know how people can have routers restarting every few weeks.
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6694
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Aug 06, 2021 12:33 pm

blingblouw, this was not a general response and even not response to a specific topic. It was just like sharing some experience.
We are researching multiple support files, that were sent to us regarding 2004 issues. Currently there are not too many open tickets though.
And many issues were addressed with the latest version, we are focusing on SFP compatibility and interoperation, that might lead to the described problems.

blingblouw, or any other person, do you have ongoing reboots with the latest current/longterm/beta? Update us with the latest support/autosupout files (support@mikrotik.com), so we can proceed with our researches.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Aug 06, 2021 2:51 pm

Currently there are not too many open tickets though.
Maybe not too many tickets opened because people are borried to open ticket and wait months/year without any real solutions? We have had 4 x CCR2004 in production and all have weekly random reboots... please Sergejs have respect of our IQ: buggy hardware.

Old CCR1016 outperform CCR2004 in uptime stability with 0% packet loss!!
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6694
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Aug 06, 2021 3:19 pm

Yes, that's why for us it is important to get as many support.rif files as possible, even if there is no 100% resolution within 1-3 days, it could help us to proceed quicker.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Aug 06, 2021 4:27 pm

Tonight we 'll swap 1 of 2 remaining CCR2004 in produtcion with cisco. I 'll open a new ticket after next watchdog reboot on the last CCR2004 in producition .
 
itsmekiran
just joined
Posts: 3
Joined: Sat Aug 07, 2021 7:02 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Aug 07, 2021 7:16 pm

Yes, that's why for us it is important to get as many support.rif files as possible, even if there is no 100% resolution within 1-3 days, it could help us to proceed quicker.
Hi , I've few routers in my production acting as PPPOE Concentrator and getting rebooted , initially we were running the Mikrotik 2003 with 6.48.3 [stable ] Version and the reboot pattern used to around 2 days

post raising a case with Support , who suggested to 6.49beta54 the reboot pattern changed to 6 days across all the router the business is getting impacted due to this ongoing issue and the customer were left with a pathetic issue

FYI the current ongoing case is SUP-55410 , see if you can help i'm hereby ready to give you access to the production network in case if want to end this

Rgds,
Kiran Kumar
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6694
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Aug 09, 2021 8:55 am

Hello, Kiran.
Thank you for the provided ticket number.
Yes, beta version contain the latest fixes and patches, even for 2004 that might impact it in rare cases, that are not yet implemented (are going to be implemented) in current RouterOS version.

Regarding your provided support output file from beta54, provided file is generated after manual /system reboot (somebody actually launched the command), and there is no usable information for us.
Make the file immediately after unexpected reboot, it is very important.
 
oneiric
just joined
Posts: 1
Joined: Mon Sep 30, 2019 3:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 10, 2021 7:20 am

+ 1 for CCR2004 random watchdog reboot. Ticket SUP-55742 lodged with supportrif directly after - no comment on ticket since lodging. Fingers crossed fix in 6.48.4 ? :(
Last edited by oneiric on Tue Aug 10, 2021 7:33 am, edited 1 time in total.
 
itsmekiran
just joined
Posts: 3
Joined: Sat Aug 07, 2021 7:02 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 10, 2021 8:34 am

Hello, Kiran.
Thank you for the provided ticket number.
Yes, beta version contain the latest fixes and patches, even for 2004 that might impact it in rare cases, that are not yet implemented (are going to be implemented) in current RouterOS version.

Regarding your provided support output file from beta54, provided file is generated after manual /system reboot (somebody actually launched the command), and there is no usable information for us.
Make the file immediately after unexpected reboot, it is very important.
Hi , Sergejs ,

For you clarification we have never done a manual / system reboot , we have enabled watchdog which is helping to reboot the router during the stage when it goes to hung may be that was triggering the reboot , if we disable the watchdog the system goes to hung state and we require a manual presence to reboot during such instances which is being hard for us to reach during odd hours

secondly the system has rebooted and we could capture the support file immediately within 3 mins , see if you can help with this new support file , attached under the same support ticket SUP-55410
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 10, 2021 5:11 pm

We have the samen problem with 3 CCR2004 routers.

1 we are using for OSPF / BGP (reboot around day 40/45) running v6.47.10
2 we are using with OSPF + PPPoE server (reboots around every 21 days) running v6.47.9 and v6.48.3

I created a support ticket: SUP-56952

There is one thing that caught my eye, when there are a lot of OSPF changes (for example when a link gets disconnected) the router wil crash also, this can also happen after just a few hours of uptime. Our OSPF table is very small, only around 150 routes. Maybe it is meaningful information.. i don't know.
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6694
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Aug 11, 2021 8:56 am

Thank you for the provided information and support output files. All information is important, and might help us, thank you for the help.

Additionally for setups with many queues and link flapping, we would recommend to do the following,
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default
 
owireless
just joined
Posts: 4
Joined: Wed Dec 02, 2020 2:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Aug 11, 2021 11:03 am

Can you try lowering your TCP connection time out to less than the standard 24 hours? Not sure if you have any tracking, but it made a huge difference to ours. Now we're stuck with ports randomly dropping and killing the router.
We have the samen problem with 3 CCR2004 routers.

1 we are using for OSPF / BGP (reboot around day 40/45) running v6.47.10
2 we are using with OSPF + PPPoE server (reboots around every 21 days) running v6.47.9 and v6.48.3

I created a support ticket: SUP-56952

There is one thing that caught my eye, when there are a lot of OSPF changes (for example when a link gets disconnected) the router wil crash also, this can also happen after just a few hours of uptime. Our OSPF table is very small, only around 150 routes. Maybe it is meaningful information.. i don't know.
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Aug 11, 2021 11:09 am

Can you try lowering your TCP connection time out to less than the standard 24 hours? Not sure if you have any tracking, but it made a huge difference to ours. Now we're stuck with ports randomly dropping and killing the router.

On the first CCR2004 we have no connection tracking active.
The 2 other CCr2004 we have it active so queue will work, i will try this.
 
owireless
just joined
Posts: 4
Joined: Wed Dec 02, 2020 2:45 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Aug 13, 2021 8:47 pm

Ours is at 30 minutes, and since making that change, we've had no watchdog reboots. YMMV.
Can you try lowering your TCP connection time out to less than the standard 24 hours? Not sure if you have any tracking, but it made a huge difference to ours. Now we're stuck with ports randomly dropping and killing the router.

On the first CCR2004 we have no connection tracking active.
The 2 other CCr2004 we have it active so queue will work, i will try this.
 
itsmekiran
just joined
Posts: 3
Joined: Sat Aug 07, 2021 7:02 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Aug 19, 2021 7:29 am

I've made changes as suggested by Sergejs , i could see the 2004 hasn't rebooted for almost 9 days which we haven't seen so far while its working as PPPOE AC

Along with the suggested changes - we've disabled snmp , logging to memory , increased hold queue on the other end - Cisco to 4096 and could see the tx drops went away

will monitor for another couple of days
 
NCalex
just joined
Posts: 2
Joined: Tue Feb 09, 2021 2:30 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Aug 19, 2021 12:38 pm

Just submited a suppout (SUP-57679) after another watchdog reboot on 6.48.3 (Previously on 6.47.10), Device is used as an OSPF Router for a small Area (170 Routes +/-) with traffic around the 1.5Gbit mark. Traffic loss was getting bad so i opted for an upgrade just for the sake of trying.
Rebooted 2 times in the last 3h. I am beyond curious by now what the problem could be.
Swapping now for a CRS317 as Aggregation and a CCR1036-8G-2S+EM for Routing.
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6694
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Aug 20, 2021 8:57 am

Just submited a suppout (SUP-57679) after another watchdog reboot on 6.48.3 (Previously on 6.47.10), Device is used as an OSPF Router for a small Area (170 Routes +/-) with traffic around the 1.5Gbit mark. Traffic loss was getting bad so i opted for an upgrade just for the sake of trying.
Rebooted 2 times in the last 3h. I am beyond curious by now what the problem could be.
Swapping now for a CRS317 as Aggregation and a CCR1036-8G-2S+EM for Routing.
Thank you very much for the feedback, we have provided you with the necessary instructions (attached files are from earlier RouterOS versions).
 
mod3m
just joined
Posts: 5
Joined: Thu Jan 16, 2020 10:29 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Aug 23, 2021 4:29 pm

The same use 6.49beta30 issued by tech support. But also, 26 days - is a reboot. Then they turned off the watchdog, there was a record of ~60 days. Then the hardware freezes. Rebooted manually down/up 220v. As wrote above, we noticed that, as a rule, when the link on the bgp session falls, after a while, reboots took place (but not always).
There was an offer from those support to provide them witch a console. but there is no technical feasibility yet
 
joarc
just joined
Posts: 16
Joined: Wed May 08, 2019 1:52 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Aug 23, 2021 6:49 pm

We have about 12 CCR2004, one of our oldest just crossed the 365 days without any incident(*) mark. We have it running 6.48.2, it has OSPF, MPLS/LDP/VPLS, and has both 1G and 10G optics. It is working perfectly, although it doesn't have much throughput.

Our other CCR2004 all have almost the same configuration, just most of them only have 10G optics and no 1G optics and also some run NAT. Since 6.48.2 (and 6.49beta30 specifically) all reboots have stopped. We use cisco coded FS.com optics, for both 10G and 1G.

If your device still reboots, it is most likely something specific to your configuration/feature you use, or the optics you use, as we have not experienced one reboot since installing 6.48.2.

We talked with Mikrotik support, and setup our spare routers (which either would have been sold to our customers or setup in our own network) to assist them as fast and as much as possible, as we were able to replicate the issue fairly easily. Although it took some time(well, a few weeks), they managed to recreate and resolve all of the issues we experienced.

Although Mikrotiks handling of this issue has been... disappointing, we haven't lost faith in them as some other in this thread might have. Mikrotik isn't alone with having bugs or huge issues with their networking hardware. We have had other, bigger, providers do too, and those issues also last for weeks until they find what is wrong and fix it. So in the bigger picture, Mikrotik isn't slow, it just isn't always easy to find and fix bugs.

Also, this is one hardware product they have. They have almost a hundred different devices so in that aspect, this is actually a very small issue. But of course, it is still an issue.

(*Edit: to clarify, it has been rebooted once, to be upgraded to 6.48.2 when it was pushed into stable. The routers actual uptime is 126 days.)
Last edited by joarc on Tue Aug 24, 2021 1:46 pm, edited 1 time in total.
 
mod3m
just joined
Posts: 5
Joined: Thu Jan 16, 2020 10:29 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Aug 23, 2021 7:54 pm

We have about 12 CCR2004, one of our oldest just crossed the 365 days uptime mark. We have it running 6.48.2, it has OSPF, MPLS/LDP/VPLS, and has both 1G and 10G optics. It is working perfectly, although it doesn't have much throughput.

Our other CCR2004 all have almost the same configuration, just most of them only have 10G optics and no 1G optics and also some run NAT. Since 6.48.2 (and 6.49beta30 specifically) all reboots have stopped. We use cisco coded FS.com optics, for both 10G and 1G.

If your device still reboots, it is most likely something specific to your configuration/feature you use, or the optics you use, as we have not experienced one reboot since installing 6.48.2.

We talked with Mikrotik support, and setup our spare routers (which either would have been sold to our customers or setup in our own network) to assist them as fast and as much as possible, as we were able to replicate the issue fairly easily. Although it took some time(well, a few weeks), they managed to recreate and resolve all of the issues we experienced.

Although Mikrotiks handling of this issue has been... disappointing, we haven't lost faith in them as some other in this thread might have. Mikrotik isn't alone with having bugs or huge issues with their networking hardware. We have had other, bigger, providers do too, and those issues also last for weeks until they find what is wrong and fix it. So in the bigger picture, Mikrotik isn't slow, it just isn't always easy to find and fix bugs.

Also, this is one hardware product they have. They have almost a hundred different devices so in that aspect, this is actually a very small issue. But of course, it is still an issue.
Do you have a bgp? FV tables? Initially we had 2 FV and then caught up to 4, not counting peering operator. Now we are testing from 1 FV and 1 default.
comrades suggested turning off the SNMP, but so far, unfortunately, we cannot, because we get graphs from interfaces by load
from the very beginning, the config was primitive, only bgp and security settings for filtering admin addresses. did nothing else
10G interface loading about 15-25% of traffic about 2.5 gbits at the peak, 1G - management
the most important thing. so as not to be unfounded. until CCR2004 used CCR1009, with half a year uptime and the same configs. actually, here is the chart on 21 jan, set in CCR2004, and you could see the reboots in the end.
Image
ps at the beginning of the graph, this is not a reboot, but work was in progress
Last edited by mod3m on Mon Aug 23, 2021 8:22 pm, edited 2 times in total.
 
joarc
just joined
Posts: 16
Joined: Wed May 08, 2019 1:52 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Aug 23, 2021 8:17 pm

We use BGP, but not on the CCR2004s, only on CCR1072 at the moment, as they have been handling our peers and transits well. We currently get two full tables from two different transit operators, and have three main BGP-routers directly connected to eachother. I do not understand what you mean by "FV tables". We have SNMP turned on without issue, and our OSPF contains about 360-ish routes.

These CCR2004s are only access routers, and do not usually have customers directly connected to the 2004s, although on some smaller sites we do have them connected directly. We almost always put a CRS317 in-between our routers and customers ( if no special configuration are required, but even then it usually goes on a CRS317 ).

Traffic thruput varries during the time of day, but on of the more central ones have peaks at 1G, average 400 mbits, and around 100 mbit of that is traffic staying inside our network. We almost always use 10G optics when possible, only to end users do we use 1G optics.

We also try to have a basic configuration, as less config is easier config.

I should also mention, that during testing with Mikrotik we did temporarily disable SNMP, but during our tests, they found and fixed it for 6.48.2, so after some time the routers were running after upgrade without SNMP and then we turned it on again.
 
kos
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Mon Oct 31, 2016 11:51 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 24, 2021 9:48 am

We have about 12 CCR2004, one of our oldest just crossed the 365 days uptime mark. We have it running 6.48.2...
Release 6.48.2 - 2021-04-13

May be it is not uptime but period of time without any unwanted reboots? If it is so, than there is another ROS version with 8 reboot free months?
 
joarc
just joined
Posts: 16
Joined: Wed May 08, 2019 1:52 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 24, 2021 1:44 pm

Yes, sorry, that's what i meant. We did upgrade it but our monitoring didn't see it as have "gone down", so it is still "okay" since last incident which was on 22 august 2020. It's actual uptime is 126 days, which is when it was upgraded.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Aug 25, 2021 1:24 am

@sergejs : please look SUP-51953
I submitted lot supouts there.
Latest one was immediately after reboot.
Please look into them.
I am with no OSPF or BGP.
Using just as regular Router.
I am running 200 ppp client with SNAT.
Please help me as this going too frequent.
The latest update is, support asked me to do CLI things which out of my knowledge and not possible for me as my ccr is in remote location where we have very limited power source. i can't keep running any pc there.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Aug 25, 2021 1:33 am

I've made changes as suggested by Sergejs , i could see the 2004 hasn't rebooted for almost 9 days which we haven't seen so far while its working as PPPOE AC

Along with the suggested changes - we've disabled snmp , logging to memory , increased hold queue on the other end - Cisco to 4096 and could see the tx drops went away

will monitor for another couple of days
Can you please be more specific ?
What changes you did ?
I didn't understand much from me.
I am also facing same issue lot.
My ccr2004 reboots on 2 or 3 days maximum.
 
User avatar
macgaiver
Forum Guru
Forum Guru
Posts: 1764
Joined: Wed May 18, 2005 5:57 pm
Location: Sol III, Sol system, Sector 001, Alpha Quadrant

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Aug 25, 2021 6:16 pm

Now that v7.1rc is out, and next CCR2004 is shipping with RouterOS v7.0.4 or 5 from factory, might be a good time to try out your configuration with that v7. At least it will give you definite answer on HW or SW issue
 
dsarlis
just joined
Posts: 1
Joined: Thu Aug 26, 2021 5:25 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Aug 26, 2021 5:51 pm

I'm using the CCR2004 in my home LAN only - yes, I know, overkill, but, I wanted something that would take the GPON SFP from my provider (Bell Canada). In my case, I have their 1Gig service, since I know the Mikrotik's won't support 2.5G negotiation on their optical ports, but I am ok with that. Bell is in the middle of rolling out some initial 10G SFP GPON ports, so, I can "live" with just the 1Gig service for now.

I am seeing watchdog resets almost daily on my CCR2004, and it's setup only with:
1. Regular PPPoE termination from ISP (Bell) - Bridged VLANs on 35 and 36.
2. Firewall IS enabled because:
a. I need an ingress rule on the PPPoE link to kill Winbox/Dude access, TCP port 8291. Someone in Asia is constantly probing that port
b. I am using fasttrack on just my local LAN traffic
c. Connection Tracking IS enabled as well (because of the few rules I have)
3. My IPTV setup is a little "funky" - I actually take the HH3000 from Bell, and just hook into it's WAN port, connect it back into the CCR2004 via a S+RJ10. I then "fake" the HH3000 to PPPoE authenticate on the CCR2004 acting as a PPPoE server, but just for my IPTV traffic on VLAN 36.
4. I use DHCP server to supply all my home LAN's IP addresses

Other than that - my CCR2004 isn't doing much more - no OSPF, no BGP, no MPLS.

One observation - I have been playing with the TCP Established Timeout on the Connection Tracker - when I set it to multiple hours (I currently have it set to 1 hour), like 4 say, it resets far quicker.

Another observation - since I work from home, and am on MS Teams all day long, when I DO set the TCP Establish Timeout to a multiple hour setting, Teams gets weird - I get the "We ran into a problem. Reconnecting..." message, and that goes away when I lower the value. At the 1 hour value I have it now, the CCR2004 has not reset today.....yet.

Other than that - the way I am using the CCR2004 is like I'm constantly using a firehose to put out a lighter - WAY overkill for what I need. And yet - it's been resetting every day the last week or so.

I had been using a 500M/500M Bell service just with the RB750Gr3, for almost 18 months, and never reset once! So, with this "upgrade" to the CCR2004, I'm a little shocked at the constant Watchdog resets every day.....

Dennis
Last edited by dsarlis on Thu Aug 26, 2021 6:14 pm, edited 3 times in total.
 
rob82
just joined
Posts: 2
Joined: Thu May 04, 2017 12:13 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Aug 30, 2021 1:28 pm

We've just replace 2/3rd of our CCR2004's with CCR1036's yesterday...
And then I just got an e-mail from support:
*****
The CCR2004 router stability improvements and the most optimal resource allocation settings are included since RouterOS v6.48.2.

*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
*) switch - improved system stability with 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
The v6.47.10, unfortunately, does not contain all of them. Please upgrade to the current v6.48.3. And if there is a problem when using v6.48.3 on CCR2004, send us supout.rif for analysis.
*****
We'll try this on the CCR2004's that are still operational, but I wanted to give you guys a headsup.
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Aug 30, 2021 7:36 pm

Now that v7.1rc is out, and next CCR2004 is shipping with RouterOS v7.0.4 or 5 from factory, might be a good time to try out your configuration with that v7. At least it will give you definite answer on HW or SW issue
Has anyone the possibility to try if v7.1rc is more stable?
I wan't to but on the routing status page it stills says /31 adress not supported, that we really need unfortunately.

https://help.mikrotik.com/docs/display/ ... col+Status
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 31, 2021 12:14 am

I wan't to but on the routing status page it stills says /31 adress not supported, that we really need unfortunately.
6.x does not support /31 addresses either. If they release it, it will be a brand new feature in 7.x.
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 31, 2021 9:53 am

6.x does not support /31 addresses either. If they release it, it will be a brand new feature in 7.x.
You are correct.

But in v6 you can do it as:
IP-Adress=10.0.0.2
Network=10.0.0.3

And the other site/router the opposite.

Do you know if it works in v7? If so i wan't to look if we can upgrade 2 of the 3 CCR2004 who are acting as a PPPoE server with OSPF.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 31, 2021 11:07 am

We've just replace 2/3rd of our CCR2004's with CCR1036's yesterday...
And then I just got an e-mail from support:
*****
The CCR2004 router stability improvements and the most optimal resource allocation settings are included since RouterOS v6.48.2.

*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
*) switch - improved system stability with 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
The v6.47.10, unfortunately, does not contain all of them. Please upgrade to the current v6.48.3. And if there is a problem when using v6.48.3 on CCR2004, send us supout.rif for analysis.
*****
We'll try this on the CCR2004's that are still operational, but I wanted to give you guys a headsup.
Already on 6.48.3 Stable but having same issue.
Reboot or 2 in week.
No support from mikrotik
 
rest
just joined
Posts: 1
Joined: Tue Aug 31, 2021 9:08 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 31, 2021 9:12 pm

Hi there,

we're also having the exact same issue, our setup (just in case it helps the Support Team):
- 2x CCR2004-1G-12S+2XS (both on v6.48.4)
- interconnected with 2x XS+DA0001
- links to switches: Q+BC0003-S+

used features:
- OSPF, BGP (1 eBGP Peer with only default route)
- MPLS/VPLS

Hangups occur on a 2 to 3 day interval.
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Aug 31, 2021 11:11 pm

Now that v7.1rc is out, and next CCR2004 is shipping with RouterOS v7.0.4 or 5 from factory, might be a good time to try out your configuration with that v7. At least it will give you definite answer on HW or SW issue
Has anyone the possibility to try if v7.1rc is more stable?
I wan't to but on the routing status page it stills says /31 adress not supported, that we really need unfortunately.

https://help.mikrotik.com/docs/display/ ... col+Status
I just updated 1 of the 3 CCR2004 routers we have to 7.1rc2.
After reconfiguring OSPF i have no problems so far, also PPPoE and L2TP server are working fine for know.

I hope this resolves the reboot problems, i will share my experience.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 02, 2021 12:43 am

Do you know if it works in v7? If so i wan't to look if we can upgrade 2 of the 3 CCR2004 who are acting as a PPPoE server with OSPF.
Yes, this workaround should still work fine in v7.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Sep 04, 2021 10:15 am



Has anyone the possibility to try if v7.1rc is more stable?
I wan't to but on the routing status page it stills says /31 adress not supported, that we really need unfortunately.

https://help.mikrotik.com/docs/display/ ... col+Status
I just updated 1 of the 3 CCR2004 routers we have to 7.1rc2.
After reconfiguring OSPF i have no problems so far, also PPPoE and L2TP server are working fine for know.

I hope this resolves the reboot problems, i will share my experience.
not reboot from upgrade till now ?
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Sep 04, 2021 11:44 am



I just updated 1 of the 3 CCR2004 routers we have to 7.1rc2.
After reconfiguring OSPF i have no problems so far, also PPPoE and L2TP server are working fine for know.

I hope this resolves the reboot problems, i will share my experience.
not reboot from upgrade till now ?
No reboot, for now stable.
I hope it stays so!
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Sep 07, 2021 7:35 am



not reboot from upgrade till now ?
No reboot, for now stable.
I hope it stays so!
Still same from August 31 ? No reboots ?
 
edrichdelange
just joined
Posts: 3
Joined: Sat Aug 21, 2021 9:33 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Sep 07, 2021 10:03 am



I just updated 1 of the 3 CCR2004 routers we have to 7.1rc2.
After reconfiguring OSPF i have no problems so far, also PPPoE and L2TP server are working fine for know.

I hope this resolves the reboot problems, i will share my experience.
not reboot from upgrade till now ?
I installed one with just OSPF and BGP, not even moving traffic other than management. Longest uptime since I converted was 5 days
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Sep 07, 2021 10:31 am



No reboot, for now stable.
I hope it stays so!
Still same from August 31 ? No reboots ?
No reboots yet.
 
rob82
just joined
Posts: 2
Joined: Thu May 04, 2017 12:13 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Sep 08, 2021 2:47 pm

We've just replace 2/3rd of our CCR2004's with CCR1036's yesterday...
And then I just got an e-mail from support:
*****
The CCR2004 router stability improvements and the most optimal resource allocation settings are included since RouterOS v6.48.2.

*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
*) switch - improved system stability with 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
The v6.47.10, unfortunately, does not contain all of them. Please upgrade to the current v6.48.3. And if there is a problem when using v6.48.3 on CCR2004, send us supout.rif for analysis.
*****
We'll try this on the CCR2004's that are still operational, but I wanted to give you guys a headsup.
We are running 2 CCR2004's with BGP and more features 'stable' for little over a week on V6.48.4 (software AND firmware updated).
I'm not giving any guarantees yet, but these are our first 8 days without a reboot since the implementation of these CCR2004's.
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 09, 2021 11:10 am



Still same from August 31 ? No reboots ?
No reboots yet.
Unfortunately reboot after 8 days and 8 hours.. v7.1rc2
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 09, 2021 11:36 am



No reboots yet.
Unfortunately reboot after 8 days and 8 hours.. v7.1rc2
That's bad news.
I was hoping on 7.1 and planned to upgrade.
.
New ccr 2004 16G 2S+ comes with stable 7.0.4 version.
Can anyone get that upgrade file ?
It's stable, so I think no reboot will happen there.
 
biomesh
Long time Member
Long time Member
Posts: 561
Joined: Fri Feb 10, 2012 8:25 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 09, 2021 1:17 pm

The 2004-16G-2S+ uses different hardware (the same switch chip in the 5009) which is not in the original 2004. Until more people buy the 16G-2S+ version it will be hard to say what the real issue is with these devices.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 09, 2021 3:24 pm

The 2004-16G-2S+ uses different hardware (the same switch chip in the 5009) which is not in the original 2004. Until more people buy the 16G-2S+ version it will be hard to say what the real issue is with these devices.
Can you please tell us both chip model no. In old 2004 and in new 2004
 
User avatar
sch
Frequent Visitor
Frequent Visitor
Posts: 75
Joined: Tue Feb 26, 2013 1:05 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 09, 2021 3:32 pm

 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Sep 10, 2021 10:48 am



No reboots yet.
Unfortunately reboot after 8 days and 8 hours.. v7.1rc2
Another reboot of this router after 15 hours and yet another reboot after 5 hours..
So the reboot problem still exist..
 
vmnx
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Sun May 23, 2010 12:13 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Sep 10, 2021 3:11 pm

Another reboot of this router after 15 hours and yet another reboot after 5 hours..
So the reboot problem still exist..
I don't understand. Is it so hard for MT to trace what causes reboots and fix it? Even it is happening so often... That's pathetic...
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Sep 10, 2021 3:12 pm

Yes, I observed same.
router works fine for week when you upgrade firmware or change some configs.
for me, I was on 48.3 and upgraded to 48.4, then no reboot for week.
but now 2 reboots daily.
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Sep 10, 2021 4:13 pm

Another reboot of this router after 15 hours and yet another reboot after 5 hours..
So the reboot problem still exist..
I don't understand. Is it so hard for MT to trace what causes reboots and fix it? Even it is happening so often... That's pathetic...
Complete agree with you.. after more then a year (after the topic starts) still no fix for it..
If it's een hardware problem it would be nice to have some sort of return/discount for example for buying another type.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Sep 11, 2021 10:50 am

At this pint I think it's an hardware problem that can't be solved by MT

CCR2004 for now is an expensive paperweight
 
kraljek
just joined
Posts: 18
Joined: Fri Apr 24, 2015 8:13 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Sep 12, 2021 8:46 am

At this pint I think it's an hardware problem that can't be solved by MT

CCR2004 for now is an expensive paperweight
But what now? Lot of them is probably out of warranty. Mikrotik will not accept returns!
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Sep 12, 2021 3:48 pm

At this pint I think it's an hardware problem that can't be solved by MT

CCR2004 for now is an expensive paperweight
But what now? Lot of them is probably out of warranty. Mikrotik will not accept returns!
What for those who already tickets open for 6 months ?
My ticket also open from 6 months.
No resolution till now.
We need to force them for fix or replacement / refund.
 
vmnx
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Sun May 23, 2010 12:13 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Sep 12, 2021 6:42 pm

At this pint I think it's an hardware problem that can't be solved by MT

CCR2004 for now is an expensive paperweight
In this case MT should recall this product imediatly, until upgrade of this model is made.
Or do they keep on selling fault hardware and don't care about this? Mikrotik, what is your position???
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Sep 12, 2021 9:54 pm

In this case MT should recall this product imediatly, until upgrade of this model is made.
Or do they keep on selling fault hardware and don't care about this? Mikrotik, what is your position???
forgot it :-)

CCR2004 works well we are all not able to use it correctly!!!
 
rpra
newbie
Posts: 45
Joined: Tue Jan 24, 2012 8:43 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Sep 14, 2021 4:26 pm

Confirming reboots on 6.48.4 after 5 days and then withing few hours.
Used as PE with one ldp/vpls endpoint, 1,5k ospf routes.

It was waiting for almost a year after first try with no luck :)
 
Kanta
just joined
Posts: 17
Joined: Tue May 15, 2018 7:54 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Sep 15, 2021 3:23 pm

I run scheduled reboots on my own 2004s, rebooting them at 03:45 every 3 days to make sure no random reboot happens during peak times. Helps a bunch in controlling and preventing those random reboots.

Do not buy them, reboots are sporadic, get something else like 1036 with a 328 for the ports, 2004 is such a pain in the ass.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 16, 2021 10:26 am



Unfortunately reboot after 8 days and 8 hours.. v7.1rc2
Another reboot of this router after 15 hours and yet another reboot after 5 hours..
So the reboot problem still exist..
will you please try 7.1rc3 ?
 
edrichdelange
just joined
Posts: 3
Joined: Sat Aug 21, 2021 9:33 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 16, 2021 11:11 am

[/quote]

will you please try 7.1rc3 ?
[/quote]

Currently running 6 days without a reboot, which is my v7 record on the 2004

Regards
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 16, 2021 11:59 am


will you please try 7.1rc3 ?

Currently running 6 days without a reboot, which is my v7 record on the 2004

Regards
Please observe for more than 15 days.
My max uptime is 15 days.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 16, 2021 9:59 pm

Last rebbot 2 days ago after 47 days of uptime, we run only OSPF, no bgp with 4 gb/s of traffic
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Sep 17, 2021 7:00 am

Last rebbot 2 days ago after 47 days of uptime, we run only OSPF, no bgp with 4 gb/s of traffic

On which version ?
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Sep 17, 2021 12:21 pm



Another reboot of this router after 15 hours and yet another reboot after 5 hours..
So the reboot problem still exist..
will you please try 7.1rc3 ?
I have now 6d 15h uptime on v7.1rc3.
 
stealth
newbie
Posts: 30
Joined: Fri May 26, 2006 4:55 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Sep 17, 2021 2:40 pm

You guys scare me. I need to replace our IGW router which is smt like 4-core ivy bridge core i7. It had a freeze a week ago, the first and only in like 7 years. So I bought CCR2004-1g-12s-x2xs as a replacement. Should I not even try to deploy it ? Some of you stated that not all units are freezing and some are stable. Have you tried to switch configs and places ?

Can anybody with spontaneous reboots corelate that to memory corruption ? Have you tried memory test during the boot ? I have tried and it seems like it hung just after several minutes after starting.
Image
Edit: seems like it really takes some time to finish [hours].
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Sep 17, 2021 8:42 pm

On which version ?
6.48.2
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Sep 18, 2021 12:16 am

You guys scare me. I need to replace our IGW router which is smt like 4-core ivy bridge core i7. It had a freeze a week ago, the first and only in like 7 years. So I bought CCR2004-1g-12s-x2xs as a replacement. Should I not even try to deploy it ? Some of you stated that not all units are freezing and some are stable. Have you tried to switch configs and places ?

Can anybody with spontaneous reboots corelate that to memory corruption ? Have you tried memory test during the boot ? I have tried and it seems like it hung just after several minutes after starting.
Image
Edit: seems like it really takes some time to finish [hours].

Upgrade to latest v7.1rc3 and test this again.
See if it changes something.
Mikrotik is hoping fix this issue on v7.
Even, new model of ccr 2004 ( with 16 lan ports) comes with stable version 7.0.4 which not available to download.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Sep 18, 2021 12:18 am

On which version ?
6.48.2
Test on latest v7. 1rc3 build.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Sep 18, 2021 12:42 am

Guys, can upgrade from v6 to v7 without loosing any configs ? Does upgrading from stable to beta loose any configuration ?
Also, v6 backup file will work on v7 ?
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sun Sep 19, 2021 5:56 am

will you please try 7.1rc3 ?
I have now 6d 15h uptime on v7.1rc3.
Did you lost any configuration after upgrade from v6 to v7 ?
And, backup file from v6 will work on v7 or not ?
 
anf
just joined
Posts: 13
Joined: Mon Dec 22, 2008 11:26 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Sep 21, 2021 5:41 am

Recently I replaced CCR1009 with CCR2004 in a hope that it will improve the performance. Our current setup includes two BGP routers with full routing table from each of two upstream ISP, OSPF instance for downstream and traffic around 3Gbps at the peak level. Because of BGP full routing tables CCR1009 were always using 100% on one of the nine cores. Two issues I've faced so far:

1. I wasn't able to configure my setup properly using 7.1rc3 RouterOS version because the lack of tools to analyze BGP peer state. For example I couldn't clearly see how many prefixes were announced from each side of the peer. There was some level of routing features inconsistency which made it work then stop and then make it work again after the reboot with the same configuration. So I had to roll back it to RouterOS version 6.
2. In version 6 mentioned earlier feature set was working as expected but router has crashed unexpectedly after about 20 hours of uptime without supout.riff file just leaving a "router was rebooted without proper shutdown by watchdog timer" message in the logs. Previously used CCR1009 were working over a several years without such problems.

Overall according to my observations performance looks better but not too much and reliability is definitely a biggest concern at the moment. Keep watching CCR2004 but it looks like we will have to make a Linux based backup router which I expect will give us better performance and service reliability.
 
brunoforcajovem
just joined
Posts: 6
Joined: Thu Sep 09, 2021 6:32 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Sep 21, 2021 3:32 pm

will you please try 7.1rc3 ?
I have now 6d 15h uptime on v7.1rc3.
Dear how is your 2004 with 7.1rc3? what is the restart time
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Sep 22, 2021 12:51 pm

will you please try 7.1rc3 ?
I have now 6d 15h uptime on v7.1rc3.
I have lower the tcp-established-timeout from 1 day (default) to 1 hour, i don't know if this has any effect, but i read about it on the forum.

for now i have a uptime of 11d 15h 17m on v7.1rc3 it it anyway a record for me :)
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 23, 2021 6:42 am



I have now 6d 15h uptime on v7.1rc3.
I have lower the tcp-established-timeout from 1 day (default) to 1 hour, i don't know if this has any effect, but i read about it on the forum.

for now i have a uptime of 11d 15h 17m on v7.1rc3 it it anyway a record for me :)
Please update to 7.1rc4.
Your unexpected reboot was related to 7.1rc3, it is fixed in rc4.
Please try it.
 
korsakik
just joined
Posts: 6
Joined: Fri Apr 02, 2021 7:19 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Sep 24, 2021 6:20 am



I have now 6d 15h uptime on v7.1rc3.
I have lower the tcp-established-timeout from 1 day (default) to 1 hour, i don't know if this has any effect, but i read about it on the forum.

for now i have a uptime of 11d 15h 17m on v7.1rc3 it it anyway a record for me :)
Annotation 2021-09-24 131939.jpg
work fine for me
You do not have the required permissions to view the files attached to this post.
 
kraljek
just joined
Posts: 18
Joined: Fri Apr 24, 2015 8:13 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Sep 27, 2021 9:31 pm

I have information that try this, this resolve problem with rebooting. This fix 99% problems with rebooting. There is a some more small problems with routing, what will be resolved in next updates.

Please do the followig,

install 6.48.4 and the latest /system routerboard firmware on the device.
Make sure to set the following settings for queues,
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

Please contact support, if you will have problems with reboots again after that!
 
yaorushdie
just joined
Posts: 2
Joined: Thu Sep 23, 2021 11:01 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Sep 28, 2021 9:57 am

I have information that try this, this resolve problem with rebooting. This fix 99% problems with rebooting. There is a some more small problems with routing, what will be resolved in next updates.

Please do the followig,

install 6.48.4 and the latest /system routerboard firmware on the device.
Make sure to set the following settings for queues,
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

Please contact support, if you will have problems with reboots again after that!
There's an issue that running BGP over VPLS/MPLS on CCR2004 with Juniper MX240:

Log shows on MX240:
rpd[30259]: bgp_io_mgmt_cb:1777: NOTIFICATION sent to 172.x.x.x (Internal AS xxx): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 172.x.x.x (Internal AS xxx), socket buffer sndacc: 38 rcvacc: 0 , socket buffer sndccc: 38 rcvccc: 0 TCP state: 4, snd_una: 2029403502 snd_nxt: 2029403540 snd_wnd: 64981 rcv_nxt: 3011881775 rcv_adv: 3011898159, hold timer 90s, hold timer remain 0s, last sent 17s, TCP port (local 59394, remote 179)

It looks there's no bgp hello packet receive on MX240 so bgp down after hold time(90 second).

We don't know this issue exist after which CCR2004 stable version, bgp can works normal on Juniper MX240 when CCR2004 using version 6.47.10, but we still have CCR2004 reboot problem even we using Mikrotik's SFP on CCR2004.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Sep 28, 2021 6:24 pm



I have now 6d 15h uptime on v7.1rc3.
I have lower the tcp-established-timeout from 1 day (default) to 1 hour, i don't know if this has any effect, but i read about it on the forum.

for now i have a uptime of 11d 15h 17m on v7.1rc3 it it anyway a record for me :)
any update on uptime on 7.1 ?
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Sep 30, 2021 12:32 pm



I have lower the tcp-established-timeout from 1 day (default) to 1 hour, i don't know if this has any effect, but i read about it on the forum.

for now i have a uptime of 11d 15h 17m on v7.1rc3 it it anyway a record for me :)
any update on uptime on 7.1 ?
Still on 7.1rc3 and a uptime of 19 days. So certainly much more stable since changing TCP Timeout
 
brunoforcajovem
just joined
Posts: 6
Joined: Thu Sep 09, 2021 6:32 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Oct 01, 2021 6:09 pm



any update on uptime on 7.1 ?
Still on 7.1rc3 and a uptime of 19 days. So certainly much more stable since changing TCP Timeout
It seems to me that the 2004 path is MK's V7, I've always questioned that here in Brazil, 2004 doesn't work well with V6
 
popcorrin
Member Candidate
Member Candidate
Posts: 196
Joined: Wed Mar 11, 2009 12:55 am

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Oct 05, 2021 5:25 pm

I have information that try this, this resolve problem with rebooting. This fix 99% problems with rebooting. There is a some more small problems with routing, what will be resolved in next updates.

Please do the followig,

install 6.48.4 and the latest /system routerboard firmware on the device.
Make sure to set the following settings for queues,
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

Please contact support, if you will have problems with reboots again after that!
Does this seem to work for most, anyone else having success with this. Personally I'm not too comfortable with switching to v7, so it would be nice to keep it on v6 if this work.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Oct 07, 2021 8:37 am

I have information that try this, this resolve problem with rebooting. This fix 99% problems with rebooting. There is a some more small problems with routing, what will be resolved in next updates.

Please do the followig,

install 6.48.4 and the latest /system routerboard firmware on the device.
Make sure to set the following settings for queues,
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

Please contact support, if you will have problems with reboots again after that!
Does this seem to work for most, anyone else having success with this. Personally I'm not too comfortable with switching to v7, so it would be nice to keep it on v6 if this work.
I was using this. but limit raised to 500 packets.
it works.
later 6.49rc1 came with ccr2004 related changelog.
So, I reverted that limit to default 50 packets and changed connection tracking settings to default ( 1 Day ) and upgraded to 6.49rc1.
I am with 10 Days uptime and no reboot till now.
So, you can try 6.49rc.
 
June03
just joined
Posts: 3
Joined: Fri Oct 01, 2021 5:50 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Sat Oct 09, 2021 10:17 am



Does this seem to work for most, anyone else having success with this. Personally I'm not too comfortable with switching to v7, so it would be nice to keep it on v6 if this work.
I was using this. but limit raised to 500 packets.
it works.
later 6.49rc1 came with ccr2004 related changelog.
So, I reverted that limit to default 50 packets and changed connection tracking settings to default ( 1 Day ) and upgraded to 6.49rc1.
I am with 10 Days uptime and no reboot till now.
So, you can try 6.49rc.
I have a CCr2004-16g-2s+ with reboot issues. Support did ask me to do that as well, it didn't work, then they asked me to download 7.1rc5 and its been going for 2 days now.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Mon Oct 11, 2021 8:39 am

Upgraded to 6.48.4 as suggested in last ticket, reboot after 4 days, borried
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Oct 12, 2021 12:15 pm



any update on uptime on 7.1 ?
Still on 7.1rc3 and a uptime of 19 days. So certainly much more stable since changing TCP Timeout
FYI No reboot yet, running 31d 14h 51m
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Oct 12, 2021 3:00 pm



Still on 7.1rc3 and a uptime of 19 days. So certainly much more stable since changing TCP Timeout
FYI No reboot yet, running 31d 14h 51m
Thank you for update.
That's great, finally we have something now.
Also for me, I am on 6.49rc1 and with 20 days uptime, no reboot till now.
Some fixes are there in changelog.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Oct 12, 2021 6:42 pm

ROS 6.48.2 I have had 48 days uptime, in the next 2 months later this lucky age :-) I have had almost 20 reboots,5 day,2 day less 1 day.
 
brunoforcajovem
just joined
Posts: 6
Joined: Thu Sep 09, 2021 6:32 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Oct 13, 2021 10:36 pm

6.48.5 with 1 day and 5 hours crashed the RB, going back to 6.47.10 I made a scipt that every 3 days it reboots. My 2004 just does OSPF. It's a lot of suffering
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Oct 13, 2021 11:01 pm

6.48.5 with 1 day and 5 hours crashed the RB, going back to 6.47.10 I made a scipt that every 3 days it reboots. My 2004 just does OSPF. It's a lot of suffering
If its just doing ospf you may consider 7.1rc4. Been pretty stable for mine as a firewall. 30 days tops with any 6 release. There's a slight configuration change but it seems good.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Oct 14, 2021 12:23 pm



Still on 7.1rc3 and a uptime of 19 days. So certainly much more stable since changing TCP Timeout
FYI No reboot yet, running 31d 14h 51m
will you please tell us what all the changes you made ?
I updated 7.1rc4 yesterday and got 3 reboots till now. tcp timeout is for 1hr
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Oct 14, 2021 12:27 pm



FYI No reboot yet, running 31d 14h 51m
will you please tell us what all the changes you made ?
I updated 7.1rc4 yesterday and got 3 reboots till now. tcp timeout is for 1hr
I am still runing 7.1rc3 on this router and set tcp established timeout to 1hr
That's all i changed.. currently 33d and 15hr uptime
 
brunoforcajovem
just joined
Posts: 6
Joined: Thu Sep 09, 2021 6:32 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Fri Oct 15, 2021 9:13 pm

6.48.5 with 1 day and 5 hours crashed the RB, going back to 6.47.10 I made a scipt that every 3 days it reboots. My 2004 just does OSPF. It's a lot of suffering
If its just doing ospf you may consider 7.1rc4. Been pretty stable for mine as a firewall. 30 days tops with any 6 release. There's a slight configuration change but it seems good.
I have another 2004, we already put the V7 on it, the only problem presented was the IPv6 did not close adjacent, as behind it there is a BRAS for pppoe, I'm thinking of making a tunnel and taking IPv6 to the edges, the only solution I found to use to 2004 with the V7. Next week I'm going to change this 2004 and put the other one with the V7. Let us pray!!!
 
rpra
newbie
Posts: 45
Joined: Tue Jan 24, 2012 8:43 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Tue Oct 19, 2021 9:19 pm

Tried 6.49 - no luck, reboot after 8 days.

Can't test 7.1rc4 - VPLS does not pass MTU > 1500, looks like it's a bug.
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Oct 20, 2021 10:37 am

Tried 6.49 - no luck, reboot after 8 days.
Have you tried this setting?

/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default
 
sander123
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Wed Oct 28, 2020 11:58 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Oct 20, 2021 10:58 am



will you please tell us what all the changes you made ?
I updated 7.1rc4 yesterday and got 3 reboots till now. tcp timeout is for 1hr
I am still runing 7.1rc3 on this router and set tcp established timeout to 1hr
That's all i changed.. currently 33d and 15hr uptime
FYI

We have still no reboot after 39d 13h still running 7.1rc3 only thing we changed is the tcp established timeout.
We also updates a second CCR2004 it is running 7.1rc4 for 6d 9h now.

We dit not set the queue/pfifo setting.
 
rpra
newbie
Posts: 45
Joined: Tue Jan 24, 2012 8:43 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Oct 20, 2021 4:04 pm

Tried 6.49 - no luck, reboot after 8 days.
Have you tried this setting?

/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default
Yes, it was set on 6.48.4 and 6.49.

I've got a one new CCR2004-1G-12S+2XS, maybe give it a try? Is there a chance that problem was fixed in later hardware?
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Wed Oct 20, 2021 5:25 pm

Im not sure we are ever going to get a fix in v6.

Sanders is about leading the way with uptime on 2004s at this point on 7.1rc3, and I am seeing good results with 7.1rc4. But I havent put them back into production, as mpls/vpls is a requirement and vpls is still broken.
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Oct 21, 2021 4:46 pm

Im not sure we are ever going to get a fix in v6.

Sanders is about leading the way with uptime on 2004s at this point on 7.1rc3, and I am seeing good results with 7.1rc4. But I havent put them back into production, as mpls/vpls is a requirement and vpls is still broken.
for me, 7.1rc4 rebooted twice in day after upgrade, then I quickly shifted to ccr 1009 for now..
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Oct 21, 2021 4:52 pm

Im not sure we are ever going to get a fix in v6.

Sanders is about leading the way with uptime on 2004s at this point on 7.1rc3, and I am seeing good results with 7.1rc4. But I havent put them back into production, as mpls/vpls is a requirement and vpls is still broken.
for me, 7.1rc4 rebooted twice in day after upgrade, then I quickly shifted to ccr 1009 for now..
What are you using your 2004 for ? Ospf? Mpls? Nat? Etc?
 
RohanAJoshi
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Jun 10, 2021 12:29 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Oct 21, 2021 4:55 pm



for me, 7.1rc4 rebooted twice in day after upgrade, then I quickly shifted to ccr 1009 for now..
What are you using your 2004 for ? Ospf? Mpls? Nat? Etc?
I am running small isp with 200+ ppp clients + SNAT, thats it, single link, NO BGP ETC.
1009 is fully stable with same configuration.
 
issme
just joined
Posts: 13
Joined: Wed Sep 16, 2020 11:38 pm

Re: The big CCR2004 reboot thread (was 2004 hardware issues?)

Thu Oct 21, 2021 9:39 pm

Have installed three CCR2004-1G-12S+2XS units over the past several weeks.

Running 6.48.4 RouterOS and firmware on all three units, no reboots yet since each was installed... but definitely watching for a reboot to occur.
Uptime: 37d 12:50:20
Uptime: 27d 12:31:48
Uptime: 12d 16:13:14

Using as internal IPv4 routers only - running OSPF, along with some very light bridging for a couple VLANs.

No NAT/BGP/MPLS/IPv6/PPP. No changes from default queue/conntrack timeout configs. Have disabled hotspot, ipv6, mpls, ppp, and wireless packages.

Using 10G LR and DAC SFP+ transceivers, and also 1G LR SFP transceivers.

Who is online

Users browsing this forum: lurker888 and 28 guests