Community discussions

 
ElviN
just joined
Posts: 3
Joined: Tue May 15, 2018 1:04 pm

Re: CCR1072 watchdog reboot

Mon Jul 30, 2018 11:34 am

Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker
No, device rebooted again with kernel error fail.
Problem not solved
 
Jeanluck
Member Candidate
Member Candidate
Posts: 266
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Thu Aug 09, 2018 9:50 pm

I have a CCR1072 with RouterOS 6.38.3 and it works perfectly. I wanted to update it to 6.42.6 but reading this thread I'm afraid...
Has anyone ever tested if the problems go away with 6.42.6?

Can anyone tell me if these problems exist only after version 6.38? (I need to know if the problem is determined by the RouterOS version or by the configuration, in which case I understand that I should already have problems)
 
tduchch
newbie
Posts: 26
Joined: Tue Aug 24, 2004 3:58 pm

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 5:16 am

These random reboots are becoming very annoying. Looking for a replacement solution, but have not found one with at least 3 SFP+ ports. We do a lot of MPLS and VPLS on this box, have not been able to use CHR on either VMWare or Hyper-V.

Recently upgraded to 6.42.9 Long Term, and still get random watchdog reboots. We have a serial logging device connected to the console port, and there is no output before the reboot. Of course, nothing in the logs.

Has anybody received any support from Mikrotik on this issue?
Any other hardware that someone has used in place of the 1072?

Terry
 
Jeanluck
Member Candidate
Member Candidate
Posts: 266
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 1:33 pm

Works fine for me with 6.38.3, can you use this version?
 
strods
MikroTik Support
MikroTik Support
Posts: 1367
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 6:40 pm

There are multiple possible reasons why router is rebooting:

1) Hardware problem;
2) Software issue;
3) Overloaded device.

All of these three possible issues can be masked under log messages "rebooted due to Kernel Failure" , "rebooted due to Watchdog timeout", etc.

There is no reason to believe that issue, that you have on router, is the same issue as other user has just because log messages are equal. Only reason why you could believe that issue is the same is if problem appears for the first time at the same RouterOS version for multiple devices or you know how to trigger it.

In any other case contact MikroTik support. Only support has ability to see crashes and help you in order to diagnose issue and resolve the problem in its roots.

Multiple times we have seen reports "when you will fix Kernel Failure bug". This is equivalent with "when RouterOS will be a bug free version". There is no bug free software, but you can resolve your problem only by cooperating with MikroTik support. It is also the fastest way how to reach solution, after an upgrade to the latest version (if problem is already fixed by MikroTik).
 
strods
MikroTik Support
MikroTik Support
Posts: 1367
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 6:46 pm

If you have CCR device that is crashing, then I recommend that you make sure that latest RouterBOOT version is installed on your router. v6.42 includes latest firmware for TILE devices which could help due to this fix:

!) tile - improved system performance and stability ("/system routerboard upgrade" required);
 
tduchch
newbie
Posts: 26
Joined: Tue Aug 24, 2004 3:58 pm

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 7:06 pm

Yes we are running latest long term version 6.42.9
We have had this problem in several versions.
We have swapped out different hardware
We have opened support ticket, were told the only way to trouble shoot this was to turn off watch dog and see what appears on the console port.
The router is a two hour drive away, and is a core router for all our network, letting it lock up is not an option.
As stated, we have a logging device on the console port and nothing appeared before the reboot.
The device can reboot during high traffic, or very low. No pattern exists.
CPU load at high traffic is around 25%, profile has 10% Firewall and 10% network.
CPU does not seem to spike before reboot.
High traffic is just approaching 2Gbps, should be no problem for this device.
Sometimes it runs for days, sometimes only for hours.

I will open another ticket with support, hopefully this can be figured out.
 
kos
just joined
Posts: 14
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Mon Oct 15, 2018 1:50 pm

Hi strods,

as I mentioned earlier, CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled.
To achieve reboot there is no no need of any other configuration, except two IP addresses, two routes and connection-tracking enabled.
 
DeCoen
just joined
Posts: 1
Joined: Tue Oct 03, 2017 11:33 am
Location: NL
Contact:

Re: CCR1072 watchdog reboot

Thu Nov 15, 2018 11:22 am

Hi strods,

as I mentioned earlier, CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled.
To achieve reboot there is no no need of any other configuration, except two IP addresses, two routes and connection-tracking enabled.
How often do you see a reboot? We have 10x 1072 and we are on 6.38.7, its reboots about once every few months, sometimes few weeks in between.
See screenshot, connection tracking is on, and with a load of more then 1 gig .
We are planning to upgrade to 6.42.9 , should we wait?

thanks,
You do not have the required permissions to view the files attached to this post.
 
kos
just joined
Posts: 14
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Mon Nov 19, 2018 10:35 am

Your traffic is distributed over two interfaces. That is why you are not experiencing more frequent reboots.

I am speaking about situation where two interface are loaded to 1G in every direction (1Gbps bidirectional forwarding). So 1Gbps entering sfp+1 and going out trough sfp+2, and in the same time 1Gbps is entering sfp+2 and going out trough sfp+1.
 
doush
Long time Member
Long time Member
Posts: 594
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Fri Nov 23, 2018 10:39 am

Frequent reboots EVERY DAY !
"Router was rebooted without proper shutdown by watchdog timer"

This issue is still not resolved and expect an answer or a possible reason from Mikrotik.

Mikrotik Support is pretty much less useless as of now.

As I say, it happens every EVERY DAY and mikrotik is silent about the issue. This behavior is not right.

CCR1072 is a useless device as it has this major fundamental flaw and should be avoided for anyone who are considering about using it in any deployment.

An reply from Mikrotik would be nice.
 
whoknew
Member Candidate
Member Candidate
Posts: 141
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Mon Nov 26, 2018 3:53 pm

We are having the same issue with the watchdog timer reboot.

disabling the watchdog causes the router to lock up until power is taken away. Full reboot with pulling power cables from both Power supplies.

We are looking for a solution to this. Our CCR1036 that this replaced did not have this issue. We are @ less than 10% load always, less than 2Gbps/300Mbps and only have static routes on this router as it is our core.
 
bradnz
just joined
Posts: 22
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Wed Nov 28, 2018 9:35 am

Hi all.

I have a CCR-1072 on 6.43.2 and this error was in the log "critical router was rebooted without proper shutdown by watchdog timer". I have sent the SUPOUT log to mikrotik and they are saying the following:

"For debugging you should turn off watchdog and test again.
"/system watchdog set watchdog-timer=no"

Connect now to your router over serial console, make sure that you have accessed RouterOS command line interface and leave console running.

Now router either:

1) might be stuck (freeze). If that happens, then you have to generate supout file on the router through serial console. Now you can reboot device.

2) might be stuck (freeze) and become unavailable over serial console. If that happens, then reboot router and then generate supout file.

3) will reboot. If that happens, then after reboot generate supout file.

Send supout file and full serial console output (within text file) to us for investigation."

This is a production router that I have already swapped for our cold spare in the hope it was just a power supply / hardware issue, but it would appear, based on this, that its not. I can let it just freeze because its in a data centre 20 mins from the office. 20mins of downtime would be unacceptable to our customers. What are your thoughts? Should I just replace it with a 1036, or a 1009? Are these any better? It runs BGP, only have about 300 BGP routes, runs at about 9% CPU most of the time, and has almost all its memory available. It just cant be that hard for this router to do surely.

Any suggestions would be appreciated.
 
Lifz
just joined
Posts: 18
Joined: Tue Feb 26, 2013 1:05 pm

Re: CCR1072 watchdog reboot

Wed Nov 28, 2018 9:39 am

At first upgrade your CCR to latest RouterOS and firmware version. It may help.
 
kos
just joined
Posts: 14
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Wed Nov 28, 2018 11:28 am

Upgrading will not help! 1009 is better choice then 1036.
 
bradnz
just joined
Posts: 22
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Thu Nov 29, 2018 5:55 am

I have upgraded to 6.43.4. I doubt it will help.

I actually wondered if this had something to do with the number of connections and the router not being able to handle it.

Either way, Im looking for an alternative hardware device now. I tried to change to the 1009 last night, but Im concerned about the performance, and also I use SFP+ that arent compatible with it. The interfaces never came up.

Incidentally, if I wanted to trial my theory of the number of connections does anyone know of tool I can use to do this?
 
kos
just joined
Posts: 14
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Thu Nov 29, 2018 11:16 am

RouterOS traffic-generator.

Don't be an optimist, because Mikrotik support claims "There are no hidden fixes"
 
whoknew
Member Candidate
Member Candidate
Posts: 141
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Fri Nov 30, 2018 8:24 pm

We are on 6.43.4 and the problem still exists. If you disable the watchdog timer, the router will hard lockup, all lights on and you will have to pull both power supplies.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 266
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Fri Nov 30, 2018 10:11 pm

I had to change the CCR1072 because I just couldn't find a solution. New hardware and everything solved, with the same .backup of the unit that failed.

Do you have the option to change the hardware and check if it fixes the problem?
(remember to do a reset-macs if you load the .backup)
 
JULIOLIMA
just joined
Posts: 1
Joined: Wed Dec 05, 2018 2:05 pm

Re: CCR1072 watchdog reboot

Wed Dec 05, 2018 2:20 pm

A few weeks ago we are going through the same problem, several restarts during the day, causing various disruptions, dissatisfaction and cancellations, we have replaced 1072 by another we have the settings in the zero hand, no procedure solved the problem, Mikrotik support so far, only speculation and no definitive solution ...
 
whoknew
Member Candidate
Member Candidate
Posts: 141
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Thu Dec 06, 2018 3:32 pm

There is a clear problem with the CCR 1072. Mikrotik we need refunds for these units or for the problem to be addressed. Please issue a statement here on the forum.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 266
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Thu Dec 06, 2018 5:46 pm

I use the 6.38.3 without problems, except high cpu for pppoe disconnections (no nat, no masquerade, no reason...)
Precisely I don't update for fear of what I read is happening...
 
guipoletto
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Mon Sep 19, 2011 5:31 am

Re: CCR1072 watchdog reboot

Thu Dec 06, 2018 9:13 pm

I use the 6.38.3 without problems, except high cpu for pppoe disconnections (no nat, no masquerade, no reason...)
Precisely I don't update for fear of what I read is happening...
That is probably Conntrack clearing the connections table after the PPPOE disconnections. it can be very disruptive indeed.
Are ALL your PPPOE connections NAT'ed?

> if you don't use NAT at all, you can disable Conntrack
> if you only need NAT for part of the connections, you can try to create "action=no-track" rules in the RAW table of the firewall, to bypass Conntrack.

As it's a bit off-topic, you should open a new topic if you want to dig further in the conntrack hole. :)
 
whoknew
Member Candidate
Member Candidate
Posts: 141
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Mon Dec 10, 2018 6:08 pm

Our CCR1072 has around 2Gbps going through it currently (about to go up to 4Gbps). Constant watchdog reboots, if I disable watchdog, I can only recover the CCR1072 by pulling power from both power supplies. Our setup is as follows:

6.42.6 is the firmware we are on.
We have 38 static routes (nothing dynamic)
No NAT rules.
No Queue rules.
No PPPoE
3 Firewall rules, 1 of which blocks winbox, ssh and telnet. 2 allow our internal subnets.
No Mangle rules.
2 SFP+'s only.
Less than 10% CPU usage.

We have a CCR1036-8G-2S+ and it does not reboot with the same configuration as we had it in place and running with an uptime of 372 days prior. I will be going back to our CCR1036 again in the middle of this month, a weekly lockup and reboot is uncalled for.
 
doush
Long time Member
Long time Member
Posts: 594
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 1:22 am

It is time for Mikrotik to seriously consider refunds of these CCR1072 units.
$3000+ for a router which can not hold even straight 3days of uptime is ridiculous.
 
User avatar
Murmaider
Member Candidate
Member Candidate
Posts: 112
Joined: Fri Oct 30, 2015 10:10 am

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 3:06 pm

Ours which had been running on version 6.38 for months and months without any issues, was upgraded to 6.42.9 a month ago and today it randomly rebooted with no clear reason why.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 266
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 3:39 pm

That confirms what I thought.... I will not move from the 6.38.3 that works perfect! In the 6.42.1 changed the firmware to improve performance for CCR1072, I don't know if it will have anything to do ...
 
doush
Long time Member
Long time Member
Posts: 594
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 3:55 pm

v6.38 is vulnerable.
We cant use that
 
Jeanluck
Member Candidate
Member Candidate
Posts: 266
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 4:35 pm

Closing 1072 well with IP/Services with subntest + ip firewall, is protected. If I have to choose between restarting it every day by watchdog or protecting it well manually....
 
doush
Long time Member
Long time Member
Posts: 594
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Wed Dec 12, 2018 1:57 pm

Would be good to have a special 6.38.x version from Mikrotik with security patches applied.
So at least we can try it out.
 
whoknew
Member Candidate
Member Candidate
Posts: 141
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Wed Dec 12, 2018 4:56 pm

Can we get a Mikrotik response please. The 1072 is ideal with the redundant PSU's but I cannot be rebooting it once or more a week.

At the very least like Doush said, give us a v6.38.xx that is patched. specifically for the CCR1072.
 
User avatar
Murmaider
Member Candidate
Member Candidate
Posts: 112
Joined: Fri Oct 30, 2015 10:10 am

Re: CCR1072 watchdog reboot

Thu Dec 13, 2018 12:54 pm

Can we get a Mikrotik response please. The 1072 is ideal with the redundant PSU's but I cannot be rebooting it once or more a week.

At the very least like Doush said, give us a v6.38.xx that is patched. specifically for the CCR1072.
I 2nd this, otherwise I'm just going to downgrade to 6.38.7 and firewall off the security issues.

Who is online

Users browsing this forum: No registered users and 19 guests