Page 1 of 1

CCR1072 watchdog reboot

Posted: Mon Jun 12, 2017 7:08 pm
by cinatus
We have several CCR1072s in our core and in the last 2 days we have had 2 watchdog reboots. one with the router on 6.38.5 and the other with the router on 6.39.2. What could be causing it and what should I do to prevent it in the future.

Re: CCR1072 watchdog reboot

Posted: Mon Jun 12, 2017 10:08 pm
by berlo
We fighted too for same reboot. We're talking with the support and they suspect an hardware failure, but i can reproduce reboots always with same conditions and are:

- If you use traffic-flow with selected interfaces (i mean anything different then interfaces: ALL), we have continous reboot every 3-4 hours

- If you overclock CPU to 1200Mhz in some condition it reboot.

This is what i did:
- Forced the firmware upgrade (it was already 3.33 version, but i forced the reinstall). Revert back CPU from 1200 to 1000Mhz. Put interfaces to ALL in traffic-flow configuration with that values

> /ip traffic-flow print
enabled: yes
interfaces: all
cache-entries: 2M
active-flow-timeout: 2m
inactive-flow-timeout: 1m

With that changes i not experienced more reboots (i still monitor it has only two days passed, but prior i have more frequent reboot).

If you can, attach serial console and keep it open, this is what i saw after reboots:

MikroTik Login: (0,0) hv_warning: L2$ correctable data ECC error at PA 0xf8a8ff30
(0,0) hv_panic: got processor error: PC 0xffff_fff7_0051_e7c0, ICS/PL 0x6
(0,0) SBOX_ERROR: 0x0000_0000_0000_0000
(0,0) MEM_ERROR_CBOX_ADDR: 0x0000_0000_f8a8_fd78
(0,0) MEM_ERROR_CBOX_STATUS: 0x0000_0000_001c_0405
(0,0) L2 data ram 2-bit error detected.
(0,0) MEM_ERROR_MBOX_ADDR: 0x0000_0000_0000_0000
(0,0) MEM_ERROR_MBOX_STATUS: 0x0000_0000_0000_0000
(0,0) XDN_DEMUX_ERROR: 0x0000_0000_0000_0000

Re: CCR1072 watchdog reboot

Posted: Wed Jun 14, 2017 6:38 pm
by cinatus
Thank you very much for the input. I will see what that does in the next maintenance window.

Re: CCR1072 watchdog reboot

Posted: Thu Jun 29, 2017 11:32 pm
by Murmaider
I too am experiencing this.

We do however have our units overclocked to 1200Mhz... I wonder if that may be the issue.

Re: CCR1072 watchdog reboot

Posted: Thu Jun 29, 2017 11:49 pm
by berlo
Yes, we have downgraded to 1000Mhz and we not had more unexpected reboot

Re: CCR1072 watchdog reboot

Posted: Fri Jun 30, 2017 9:17 am
by Murmaider
Yes, we have downgraded to 1000Mhz and we not had more unexpected reboot
I'm going to give this a try, thanks a lot.

Re: CCR1072 watchdog reboot

Posted: Tue Jul 04, 2017 2:47 pm
by LynxChaus
Yes, we have downgraded to 1000Mhz and we not had more unexpected reboot
Same symptom as I reported here. Check PSU.

Re: CCR1072 watchdog reboot

Posted: Tue Jul 04, 2017 2:58 pm
by berlo
1036 is single PSU, 1072 is redundant. Also we experienced same issue on 3 different CCRs.

Is a different case

Re: CCR1072 watchdog reboot

Posted: Thu Aug 17, 2017 9:42 pm
by rvzweb
I have the same issue on CCR 1072 - cpu is 1000 MHz - fw is 6.40.1
Any ideas?

Re: CCR1072 watchdog reboot

Posted: Thu Aug 17, 2017 11:19 pm
by berlo
me too on lab router. Disabling watchdog we see the CPU goes to 100% due to networking process and router became unusable.

Downgrade to 6.40 fix the issue.

Re: CCR1072 watchdog reboot

Posted: Wed Nov 01, 2017 3:09 pm
by pamafer
I have the same problem, CCR1072 with 6.40.3, I only use BGP router mode, I do not use IP Traffic, the peak load never exceeds 6% of all CPUs, and the bandwidth never exceeds 1Gbps, anyway I suffer spontaneous reboots at any time, days can pass without problem and suddenly the Watchdog reboots the system. Any ideas?!

Re: CCR1072 watchdog reboot

Posted: Thu Nov 02, 2017 9:17 pm
by nocmonkee
We started experiencing these watchdog reboots on our CCR1072s running 6.40.1. Is this a reported bug? Does downgrading to 6.40.0 really stabilize it? Is it fixed in newer versions?

nov/02/2017 05:13:15 system,error,critical router was rebooted without proper shut
down, probably kernel failure
nov/02/2017 05:13:15 system,error,critical kernel failure in previous boot
nov/02/2017 05:13:15 system,error,critical router was rebooted without proper shut
down, probably kernel failure

Re: CCR1072 watchdog reboot

Posted: Thu Nov 02, 2017 9:22 pm
by berlo
hi,
at the moment we have 21 ccr1072 with 6.41rc44 all up with 17 days without issue. We do bgp + filtering + ospf. Nothing else. Try to upgrade to this release, if you still experience reboot you can exclude these service as reboot cause.

We experienced reboots with cpu upgraded to 1200Mhz, at 1000mhz never experienced it.

Re: CCR1072 watchdog reboot

Posted: Thu Nov 02, 2017 9:57 pm
by nocmonkee
We currently are not running any dynamic routing protocols. The purpose of our CCRs is NAT. They also manage dhcpd and upnpd with lacp, multiple rfc1918 vlans 20G in and one vlan 20G out.

Re: CCR1072 watchdog reboot

Posted: Fri Nov 03, 2017 12:34 pm
by pamafer
hi,
at the moment we have 21 ccr1072 with 6.41rc44 all up with 17 days without issue. We do bgp + filtering + ospf. Nothing else. Try to upgrade to this release, if you still experience reboot you can exclude these service as reboot cause.

We experienced reboots with cpu upgraded to 1200Mhz, at 1000mhz never experienced it.
Hi berlo, in the same way when we knew we had it in 1200Mhz of CPU we suffered much more frequent reboots but now we have two 1072 to 1000Mhz of CPU and suffers the reboots only one of them, it is the router that has less workload !!!. So it is a contradiction!

Re: CCR1072 watchdog reboot

Posted: Fri Nov 03, 2017 1:17 pm
by berlo
have you tried disabling whatdog and keep serial console opened?

You should see the error

Re: CCR1072 watchdog reboot

Posted: Fri Nov 03, 2017 1:48 pm
by pamafer
have you tried disabling whatdog and keep serial console opened?

You should see the error

I just tried to disable the watchdog, but not the console, I'll try it and comment! Thanks

Re: CCR1072 watchdog reboot

Posted: Fri Nov 03, 2017 1:54 pm
by berlo
You need to have console opened, because if is kernel panic or memory error or similar you can't see the error message

Re: CCR1072 watchdog reboot

Posted: Sat Nov 04, 2017 7:39 am
by sakirozkan
hi,
at the moment we have 21 ccr1072 with 6.41rc44 all up with 17 days without issue. We do bgp + filtering + ospf. Nothing else. Try to upgrade to this release, if you still experience reboot you can exclude these service as reboot cause.

We experienced reboots with cpu upgraded to 1200Mhz, at 1000mhz never experienced it.
It is still working without any issue. What is the uptime of ccr1072 now.

Re: CCR1072 watchdog reboot

Posted: Sat Nov 04, 2017 1:29 pm
by berlo
Yes and now ccr was raised to 28 in all Europe. All are working fine and we never experienced more random reboots. Also we experienced better performance on routes with > 1kk routes installed disabled route cache. You loose some % CPU, about 10% more, but you will not experiencing packetloss/stop forwarding when router will forward > 2mil pps

Re: CCR1072 watchdog reboot

Posted: Mon Nov 20, 2017 6:10 pm
by sakirozkan
is there any development of the watchdog reboot subject. What version are u using for solution. Our 1072 is reboots every 2 weeks.

Re: CCR1072 watchdog reboot

Posted: Wed Nov 29, 2017 11:55 pm
by mac86
there is some update about this post ?

I've a ccr 1072 and get watchdog reboot and kernel panic reboot

this is about watchDog reboot
nov/29/2017 10:35:40 system,error,critical router was rebooted without proper shutdown by watchdog timer

and this is about kernel Failure reboot
nov/28/2017 10:33:45 system,error,critical router was rebooted without proper shutdown, probably kernel failure
nov/28/2017 10:33:45 system,error,critical kernel failure in previous boot
really fix it downgrading to 6.40 ?
it's about a hardware problem ?

In the meantime, I've checked routerboard clock on 1000mhz, disable watchdog, and I've disable traffic flow too.

thank you.

Re: CCR1072 watchdog reboot

Posted: Thu Nov 30, 2017 1:26 am
by msbr
We have several CCR1072s in our core and in the last 2 days we have had 2 watchdog reboots. one with the router on 6.38.5 and the other with the router on 6.39.2. What could be causing it and what should I do to prevent it in the future.
Hi
I have same problem with CCR1019.
So I disable whatdog



Enviado desde mi iPhone utilizando Tapatalk

Re: CCR1072 watchdog reboot

Posted: Mon Dec 04, 2017 1:25 pm
by Murmaider
Yes and now ccr was raised to 28 in all Europe. All are working fine and we never experienced more random reboots. Also we experienced better performance on routes with > 1kk routes installed disabled route cache. You loose some % CPU, about 10% more, but you will not experiencing packetloss/stop forwarding when router will forward > 2mil pps
If I am reading this right, is fastpath only reducing the CPU by 10%?
Is the benefit of the disabled route cache better than using fastpath?

Re: CCR1072 watchdog reboot

Posted: Mon Dec 04, 2017 1:32 pm
by berlo
the CPU usage in fastpath is always lower, so on high normal traffico fastpath still the only one solution.

But if without fastpath the ccr can handle the traffic, you can keep route cache disable that will help under ddos where you experiencing stop packet forwarding.

Re: CCR1072 watchdog reboot

Posted: Wed Jan 24, 2018 6:11 pm
by Murmaider
@berlo - did you need to reboot after you disabled route cache?

Re: CCR1072 watchdog reboot

Posted: Wed Jan 24, 2018 9:37 pm
by berlo
No, is on the fly. All changes can be done without reboot. The only issues are dummy rules that are not removed automatically, but need to reboot it to reactivate fast path

Re: CCR1072 watchdog reboot

Posted: Thu Jan 25, 2018 7:04 am
by strods
If you have a router which reboots itself, then conversation in forum will only be a guessing game. If you want to find out for sure what is the problem, then send supout file (generated after reboot) to support@mikrotik.com. We are the only ones who can tell what the problem was. Tracing the cause of the issue might take a while, but still it is the best solution how to get rid of the problem.

In case of Watchdog reboots - they are caused by software. Basically router says to itself "/system reboot" at the point when router becomes inaccessible. In order to trace an issue you have to disabled Watchdog and now router either will reboot or get stuck. In both cases after reboot (either router rebooted itself or you have to power cycle it) generate new supout file and again - send it to support@mikrotik.com

Re: CCR1072 watchdog reboot

Posted: Sat Jan 27, 2018 7:53 pm
by buset1974
i also have the same problems,

mikrotik also have the same request to us to disable the watchdog and try to send supout file white the problem happen, well it's make sense but we have to accept risk the device wont reboot or freeze until we cycle the power and with will cause very long downtime.

but i just realize so many people having problem with me.

we already replace the device with the same type twice and both having the symptom, so it software problem i guests
We have many CCR1072 operate but only 1 device with this particular configuration that having problems.
the one that have problem is run internet BGP full route both ipv4 & ipv6 and running on 1000 mhz.cpu.

We have other CCR1072s even running more complicated configuration like MPLS with OSPF + BGP running, but all running fine.

thx

Re: CCR1072 watchdog reboot

Posted: Sun Jan 28, 2018 4:20 am
by buset1974
No, is on the fly. All changes can be done without reboot. The only issues are dummy rules that are not removed automatically, but need to reboot it to reactivate fast path
hi @berio,

sorry i just deep read your thread, i am using version 6.41 (stable) but still have randomly reboot.
so the DDOS packet caused the problem? i also have suspicion about this ddos things , because we having this similar problem around 1 year ago when one of our site having DDOS , not big though just around 50-100 mbps but it's caused the router keep reboot. At that time mikrotik give us solution to change all the interface queue type from "only-hardware-queue" to default or default-small and it's solved for view months after i change the queue to "only-hardware-queue" again.

so if i'am not wrong you suggestion to disable route cache right?
what is router cache anyway? if i disable the route cache is there will be other impact regarding performance? because the router running for about 300.000 routes (bgp) inside?

thx

Re: CCR1072 watchdog reboot

Posted: Sun Jan 28, 2018 4:29 am
by buset1974
have you tried disabling whatdog and keep serial console opened?

You should see the error
Hi berio,

if i tried this (put serial on and disable watchdog), when freeze can i still reboot the router after see the error and capture the supout file?
because the router is not on 24 hrs man on site.
but i can arrange a pc with other link so when it happen i can still remote the pc.


thx

Re: CCR1072 watchdog reboot

Posted: Mon Jan 29, 2018 5:07 pm
by berlo
hi,
route cache should not cause reboot, but stop on packet forwarding. If your device have enough performance to forward traffic in slow path you can try disable the cache. You will see CPU usage increasing.

If you got kernel panic you need to hard reboot the router, so you need a managed pdu or someone that power cycle it.

Re: CCR1072 watchdog reboot

Posted: Mon Jan 29, 2018 6:12 pm
by buset1974
hi,
route cache should not cause reboot, but stop on packet forwarding. If your device have enough performance to forward traffic in slow path you can try disable the cache. You will see CPU usage increasing.

If you got kernel panic you need to hard reboot the router, so you need a managed pdu or someone that power cycle it.
hi Berlo,

yes i've disable the cache and it's still rebooting.
Tonight i will set cache enable again and try to change all interface queue to default-ethernet

thx

Re: CCR1072 watchdog reboot

Posted: Tue Jan 30, 2018 1:47 pm
by BMena
I was also having my CCR 1072 rebooting, but mine was "without proper shutdown" instead of "by watchdog".
Sent a e-mail to support@mikrotik and he said he thinks the hardware is faulty..

But then another CCR 1072 I have, about 300km away, showed the same problem.

My solution for both was disconecting one power supply.

Before unpluging, I noticed under system health that while it was reading PSU1 and 2 voltage of 12.1v, only one was outputing Current.

Still can't test the supposedly bad PSU I removed cause I don't have a spare 1072 atm.

I have a third 1072 working fine where both PSU share the outputed current.

Re: CCR1072 watchdog reboot

Posted: Wed Jan 31, 2018 5:36 pm
by buset1974
i set cache enable again and try to change all interface queue to default-ethernet around 2 days and the device still randomly rebooting.
so many people having similar problem and mikrotik still did not have any clue
CCR1072 is mikrotik most expensive RB and premium router type though, they should take some action regarding this issues

hx

Re: CCR1072 watchdog reboot

Posted: Thu Mar 08, 2018 1:17 pm
by buset1974
I was also having my CCR 1072 rebooting, but mine was "without proper shutdown" instead of "by watchdog".
Sent a e-mail to support@mikrotik and he said he thinks the hardware is faulty..

But then another CCR 1072 I have, about 300km away, showed the same problem.

My solution for both was disconecting one power supply.

Before unpluging, I noticed under system health that while it was reading PSU1 and 2 voltage of 12.1v, only one was outputing Current.

Still can't test the supposedly bad PSU I removed cause I don't have a spare 1072 atm.

I have a third 1072 working fine where both PSU share the outputed current.
well is it solved now? by disconnecting 1 psu?

I also have the same answer, they said it hardware problem, i told them we have tried on 3 routers and all have the same issues.
so if it's an hardware issues mean all of my router have to be RMA, and also hundred of thousand other router in all over the world, so it means it's factory failure and they have to fix it.


thx

Re: CCR1072 watchdog reboot

Posted: Thu Mar 08, 2018 1:31 pm
by mrz
Upgrade to v6.41.2

Then upgrade bootloader to v6.41.2
/system routerboard upgrade

And reboot twice.

This should fix kernel crashes that previously was thought as hardware failure. Also it could fix other abnormal router behavior.

Re: CCR1072 watchdog reboot

Posted: Thu Mar 08, 2018 1:43 pm
by buset1974
Upgrade to v6.41.2

Then upgrade bootloader to v6.41.2
/system routerboard upgrade

And reboot twice.

This should fix kernel crashes that previously was thought as hardware failure. Also it could fix other abnormal router behavior.
right now i'am using 6.41.2, i've tried all the firmware from 6.41.2 and mikrotik gave me also 3.42.5 and 6.99.
but the problem still exists.
i've sent all the supout from all of the firmware version.

btw are u maris from mikrotik?

thx

Re: CCR1072 watchdog reboot

Posted: Sat Mar 24, 2018 2:48 pm
by BMena
Could you find a fix? After 20 days my CCR rebooted out of the blue again...
So it becoming stable after removing only 1 PSU was just coincidence. Put the PSU back in place, better to have it rebooting than risking it stopping.

Re: CCR1072 watchdog reboot

Posted: Sat Mar 24, 2018 8:27 pm
by mac86
Could you find a fix? After 20 days my CCR rebooted out of the blue again...
So it becoming stable after removing only 1 PSU was just coincidence. Put the PSU back in place, better to have it rebooting than risking it stopping.
have you upgraded your hardware routerboard, with last RouterOs version?
(not just router OS)
/system routerboard upgrade

Re: CCR1072 watchdog reboot

Posted: Sat Mar 24, 2018 11:27 pm
by doush
We will deploy a CCR1072 soon on our NOC.
Does all CCR1072 s have the same problem ?

Should we postpone the deployment ?

Re: CCR1072 watchdog reboot

Posted: Tue Apr 10, 2018 3:04 pm
by BMena
Could you find a fix? After 20 days my CCR rebooted out of the blue again...
So it becoming stable after removing only 1 PSU was just coincidence. Put the PSU back in place, better to have it rebooting than risking it stopping.
have you upgraded your hardware routerboard, with last RouterOs version?
(not just router OS)
/system routerboard upgrade
Yup, I did.
We will deploy a CCR1072 soon on our NOC.
Does all CCR1072 s have the same problem ?

Should we postpone the deployment ?
Unless you're in real need. I have three, one each city, the reboots aren't awfully frequent but they happen.
The one I have that doesn't reboot have a bug with the ping tool, sometimes it stops working at all for some hours.

Re: CCR1072 watchdog reboot

Posted: Tue Apr 10, 2018 4:40 pm
by Wibernet
We are having this exact same issue, our CCR1072 randomly reboots.

Did the firmware upgrade fix this issue?

I must say I cannot tell people to avoid this device more. Since installing we have had major issues with latency spikes until we removed all NAT (Worked perfectly in CCR1036) we have had these random reboots disconnecting all of our clients for 10 minutes randomly. Nightmare.

Re: CCR1072 watchdog reboot

Posted: Tue Apr 10, 2018 10:10 pm
by doush
OMG !
We were thinking about replacing our core 1036 with 1072 but it seems that there are major issues with it.

Re: CCR1072 watchdog reboot

Posted: Mon Apr 23, 2018 6:08 pm
by sakirozkan
Is there any development with new versions. I use 6.38.7 for this reason.
We want to upgrade 1072 for "Vulnerability exploiting the Winbox port" Anyone use 6.42 version with 1072 without watchdog reboot error.

Re: CCR1072 watchdog reboot

Posted: Tue May 08, 2018 12:24 am
by heddita
We have this problem happening as well. It got worse recently so I came across this thread. It was previously on version 6.38.5, then up to 40.1 and now it's at 42.1. I just changed the CPU speed to 1000Mhz. I don't know why it comes at 1200Mhz if it causes issues. So far it's been up for 1:40 so not a lot of time has gone by... We've had about 5 unexpected reboots have happened today. Two of them caused the router to just hang completely. 1036's only have 2 10g ports and I need 3 ;_; I'm going to move the NAT to another box if it reboots itself again. This router literally only has OSPF running with a few vlans + NAT. At the time it rebooted, it was early morning so the traffic at the time was low, around 300mbps... It's not like it's using it's full force is what I mean. Well, I lit a candle, we'll see how it goes.

Re: CCR1072 watchdog reboot

Posted: Tue May 08, 2018 12:46 am
by berlo
The one I have that doesn't reboot have a bug with the ping tool, sometimes it stops working at all for some hours.
Disable route cache, it fixed for me.

Re: CCR1072 watchdog reboot

Posted: Tue May 15, 2018 1:34 pm
by ElviN
Hi!
Read the topic....I'm shoked....
I laughed frankly when i read it: "We are the only ones who can tell what the problem was".
We have been in correspondence with support for almost a year now about the gaps in arbitrary reboots of the device... We are sending many supout.rif files. And what a requests of support: "have a problem with power, DDOS, changed CPU".
We disabled whatchdog, changed power suply, we have 2 CCR1072. The last thing we did, we getting new device in our resseller and what can we see,(i think you guess) his arbitrary rebooted again and again....
Guys, are you seriously? A lot of your clients have the same problem, and you know about this and don't recognize the problem.

We are bought your top device and you say to reduce the performance by almost 20% (CPU downgrade). It's abnormal...

I do not recommend buying this device

Re: CCR1072 watchdog reboot

Posted: Tue May 22, 2018 5:21 pm
by VagnerBecker
Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker

Re: CCR1072 watchdog reboot

Posted: Tue Jul 10, 2018 10:45 am
by kos
Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker
Unfortunately they are not! Neither in ROS 6.42.2 nor in any other ROS version!!!

CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled, due to watchdog timer reboots!!!

Five to ten times cheaper devices like CCR1009 and RB1100AHx4 are working fine in same test conditions!

The worst thing is that they are continuing selling this faulty device!

Contacting Mikrotik support is a lost of time!

Re: CCR1072 watchdog reboot

Posted: Mon Jul 30, 2018 11:34 am
by ElviN
Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker
No, device rebooted again with kernel error fail.
Problem not solved

Re: CCR1072 watchdog reboot

Posted: Thu Aug 09, 2018 9:50 pm
by Jeanluck
I have a CCR1072 with RouterOS 6.38.3 and it works perfectly. I wanted to update it to 6.42.6 but reading this thread I'm afraid...
Has anyone ever tested if the problems go away with 6.42.6?

Can anyone tell me if these problems exist only after version 6.38? (I need to know if the problem is determined by the RouterOS version or by the configuration, in which case I understand that I should already have problems)

Re: CCR1072 watchdog reboot

Posted: Tue Oct 09, 2018 5:16 am
by tduchch
These random reboots are becoming very annoying. Looking for a replacement solution, but have not found one with at least 3 SFP+ ports. We do a lot of MPLS and VPLS on this box, have not been able to use CHR on either VMWare or Hyper-V.

Recently upgraded to 6.42.9 Long Term, and still get random watchdog reboots. We have a serial logging device connected to the console port, and there is no output before the reboot. Of course, nothing in the logs.

Has anybody received any support from Mikrotik on this issue?
Any other hardware that someone has used in place of the 1072?

Terry

Re: CCR1072 watchdog reboot

Posted: Tue Oct 09, 2018 1:33 pm
by Jeanluck
Works fine for me with 6.38.3, can you use this version?

Re: CCR1072 watchdog reboot

Posted: Tue Oct 09, 2018 6:40 pm
by strods
There are multiple possible reasons why router is rebooting:

1) Hardware problem;
2) Software issue;
3) Overloaded device.

All of these three possible issues can be masked under log messages "rebooted due to Kernel Failure" , "rebooted due to Watchdog timeout", etc.

There is no reason to believe that issue, that you have on router, is the same issue as other user has just because log messages are equal. Only reason why you could believe that issue is the same is if problem appears for the first time at the same RouterOS version for multiple devices or you know how to trigger it.

In any other case contact MikroTik support. Only support has ability to see crashes and help you in order to diagnose issue and resolve the problem in its roots.

Multiple times we have seen reports "when you will fix Kernel Failure bug". This is equivalent with "when RouterOS will be a bug free version". There is no bug free software, but you can resolve your problem only by cooperating with MikroTik support. It is also the fastest way how to reach solution, after an upgrade to the latest version (if problem is already fixed by MikroTik).

Re: CCR1072 watchdog reboot

Posted: Tue Oct 09, 2018 6:46 pm
by strods
If you have CCR device that is crashing, then I recommend that you make sure that latest RouterBOOT version is installed on your router. v6.42 includes latest firmware for TILE devices which could help due to this fix:

!) tile - improved system performance and stability ("/system routerboard upgrade" required);

Re: CCR1072 watchdog reboot

Posted: Tue Oct 09, 2018 7:06 pm
by tduchch
Yes we are running latest long term version 6.42.9
We have had this problem in several versions.
We have swapped out different hardware
We have opened support ticket, were told the only way to trouble shoot this was to turn off watch dog and see what appears on the console port.
The router is a two hour drive away, and is a core router for all our network, letting it lock up is not an option.
As stated, we have a logging device on the console port and nothing appeared before the reboot.
The device can reboot during high traffic, or very low. No pattern exists.
CPU load at high traffic is around 25%, profile has 10% Firewall and 10% network.
CPU does not seem to spike before reboot.
High traffic is just approaching 2Gbps, should be no problem for this device.
Sometimes it runs for days, sometimes only for hours.

I will open another ticket with support, hopefully this can be figured out.

Re: CCR1072 watchdog reboot

Posted: Mon Oct 15, 2018 1:50 pm
by kos
Hi strods,

as I mentioned earlier, CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled.
To achieve reboot there is no no need of any other configuration, except two IP addresses, two routes and connection-tracking enabled.

Re: CCR1072 watchdog reboot

Posted: Thu Nov 15, 2018 11:22 am
by DeCoen
Hi strods,

as I mentioned earlier, CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled.
To achieve reboot there is no no need of any other configuration, except two IP addresses, two routes and connection-tracking enabled.
How often do you see a reboot? We have 10x 1072 and we are on 6.38.7, its reboots about once every few months, sometimes few weeks in between.
See screenshot, connection tracking is on, and with a load of more then 1 gig .
We are planning to upgrade to 6.42.9 , should we wait?

thanks,

Re: CCR1072 watchdog reboot

Posted: Mon Nov 19, 2018 10:35 am
by kos
Your traffic is distributed over two interfaces. That is why you are not experiencing more frequent reboots.

I am speaking about situation where two interface are loaded to 1G in every direction (1Gbps bidirectional forwarding). So 1Gbps entering sfp+1 and going out trough sfp+2, and in the same time 1Gbps is entering sfp+2 and going out trough sfp+1.

Re: CCR1072 watchdog reboot

Posted: Fri Nov 23, 2018 10:39 am
by doush
Frequent reboots EVERY DAY !
"Router was rebooted without proper shutdown by watchdog timer"

This issue is still not resolved and expect an answer or a possible reason from Mikrotik.

Mikrotik Support is pretty much less useless as of now.

As I say, it happens every EVERY DAY and mikrotik is silent about the issue. This behavior is not right.

CCR1072 is a useless device as it has this major fundamental flaw and should be avoided for anyone who are considering about using it in any deployment.

An reply from Mikrotik would be nice.

Re: CCR1072 watchdog reboot

Posted: Mon Nov 26, 2018 3:53 pm
by whoknew
We are having the same issue with the watchdog timer reboot.

disabling the watchdog causes the router to lock up until power is taken away. Full reboot with pulling power cables from both Power supplies.

We are looking for a solution to this. Our CCR1036 that this replaced did not have this issue. We are @ less than 10% load always, less than 2Gbps/300Mbps and only have static routes on this router as it is our core.

Re: CCR1072 watchdog reboot

Posted: Wed Nov 28, 2018 9:35 am
by bradnz
Hi all.

I have a CCR-1072 on 6.43.2 and this error was in the log "critical router was rebooted without proper shutdown by watchdog timer". I have sent the SUPOUT log to mikrotik and they are saying the following:

"For debugging you should turn off watchdog and test again.
"/system watchdog set watchdog-timer=no"

Connect now to your router over serial console, make sure that you have accessed RouterOS command line interface and leave console running.

Now router either:

1) might be stuck (freeze). If that happens, then you have to generate supout file on the router through serial console. Now you can reboot device.

2) might be stuck (freeze) and become unavailable over serial console. If that happens, then reboot router and then generate supout file.

3) will reboot. If that happens, then after reboot generate supout file.

Send supout file and full serial console output (within text file) to us for investigation."

This is a production router that I have already swapped for our cold spare in the hope it was just a power supply / hardware issue, but it would appear, based on this, that its not. I can let it just freeze because its in a data centre 20 mins from the office. 20mins of downtime would be unacceptable to our customers. What are your thoughts? Should I just replace it with a 1036, or a 1009? Are these any better? It runs BGP, only have about 300 BGP routes, runs at about 9% CPU most of the time, and has almost all its memory available. It just cant be that hard for this router to do surely.

Any suggestions would be appreciated.

Re: CCR1072 watchdog reboot

Posted: Wed Nov 28, 2018 9:39 am
by Lifz
At first upgrade your CCR to latest RouterOS and firmware version. It may help.

Re: CCR1072 watchdog reboot

Posted: Wed Nov 28, 2018 11:28 am
by kos
Upgrading will not help! 1009 is better choice then 1036.

Re: CCR1072 watchdog reboot

Posted: Thu Nov 29, 2018 5:55 am
by bradnz
I have upgraded to 6.43.4. I doubt it will help.

I actually wondered if this had something to do with the number of connections and the router not being able to handle it.

Either way, Im looking for an alternative hardware device now. I tried to change to the 1009 last night, but Im concerned about the performance, and also I use SFP+ that arent compatible with it. The interfaces never came up.

Incidentally, if I wanted to trial my theory of the number of connections does anyone know of tool I can use to do this?

Re: CCR1072 watchdog reboot

Posted: Thu Nov 29, 2018 11:16 am
by kos
RouterOS traffic-generator.

Don't be an optimist, because Mikrotik support claims "There are no hidden fixes"

Re: CCR1072 watchdog reboot

Posted: Fri Nov 30, 2018 8:24 pm
by whoknew
We are on 6.43.4 and the problem still exists. If you disable the watchdog timer, the router will hard lockup, all lights on and you will have to pull both power supplies.

Re: CCR1072 watchdog reboot

Posted: Fri Nov 30, 2018 10:11 pm
by Jeanluck
I had to change the CCR1072 because I just couldn't find a solution. New hardware and everything solved, with the same .backup of the unit that failed.

Do you have the option to change the hardware and check if it fixes the problem?
(remember to do a reset-macs if you load the .backup)

Re: CCR1072 watchdog reboot

Posted: Wed Dec 05, 2018 2:20 pm
by JULIOLIMA
A few weeks ago we are going through the same problem, several restarts during the day, causing various disruptions, dissatisfaction and cancellations, we have replaced 1072 by another we have the settings in the zero hand, no procedure solved the problem, Mikrotik support so far, only speculation and no definitive solution ...

Re: CCR1072 watchdog reboot

Posted: Thu Dec 06, 2018 3:32 pm
by whoknew
There is a clear problem with the CCR 1072. Mikrotik we need refunds for these units or for the problem to be addressed. Please issue a statement here on the forum.

Re: CCR1072 watchdog reboot

Posted: Thu Dec 06, 2018 5:46 pm
by Jeanluck
I use the 6.38.3 without problems, except high cpu for pppoe disconnections (no nat, no masquerade, no reason...)
Precisely I don't update for fear of what I read is happening...

Re: CCR1072 watchdog reboot

Posted: Thu Dec 06, 2018 9:13 pm
by guipoletto
I use the 6.38.3 without problems, except high cpu for pppoe disconnections (no nat, no masquerade, no reason...)
Precisely I don't update for fear of what I read is happening...
That is probably Conntrack clearing the connections table after the PPPOE disconnections. it can be very disruptive indeed.
Are ALL your PPPOE connections NAT'ed?

> if you don't use NAT at all, you can disable Conntrack
> if you only need NAT for part of the connections, you can try to create "action=no-track" rules in the RAW table of the firewall, to bypass Conntrack.

As it's a bit off-topic, you should open a new topic if you want to dig further in the conntrack hole. :)

Re: CCR1072 watchdog reboot

Posted: Mon Dec 10, 2018 6:08 pm
by whoknew
Our CCR1072 has around 2Gbps going through it currently (about to go up to 4Gbps). Constant watchdog reboots, if I disable watchdog, I can only recover the CCR1072 by pulling power from both power supplies. Our setup is as follows:

6.42.6 is the firmware we are on.
We have 38 static routes (nothing dynamic)
No NAT rules.
No Queue rules.
No PPPoE
3 Firewall rules, 1 of which blocks winbox, ssh and telnet. 2 allow our internal subnets.
No Mangle rules.
2 SFP+'s only.
Less than 10% CPU usage.

We have a CCR1036-8G-2S+ and it does not reboot with the same configuration as we had it in place and running with an uptime of 372 days prior. I will be going back to our CCR1036 again in the middle of this month, a weekly lockup and reboot is uncalled for.

Re: CCR1072 watchdog reboot

Posted: Tue Dec 11, 2018 1:22 am
by doush
It is time for Mikrotik to seriously consider refunds of these CCR1072 units.
$3000+ for a router which can not hold even straight 3days of uptime is ridiculous.

Re: CCR1072 watchdog reboot

Posted: Tue Dec 11, 2018 3:06 pm
by Murmaider
Ours which had been running on version 6.38 for months and months without any issues, was upgraded to 6.42.9 a month ago and today it randomly rebooted with no clear reason why.

Re: CCR1072 watchdog reboot

Posted: Tue Dec 11, 2018 3:39 pm
by Jeanluck
That confirms what I thought.... I will not move from the 6.38.3 that works perfect! In the 6.42.1 changed the firmware to improve performance for CCR1072, I don't know if it will have anything to do ...

Re: CCR1072 watchdog reboot

Posted: Tue Dec 11, 2018 3:55 pm
by doush
v6.38 is vulnerable.
We cant use that

Re: CCR1072 watchdog reboot

Posted: Tue Dec 11, 2018 4:35 pm
by Jeanluck
Closing 1072 well with IP/Services with subntest + ip firewall, is protected. If I have to choose between restarting it every day by watchdog or protecting it well manually....

Re: CCR1072 watchdog reboot

Posted: Wed Dec 12, 2018 1:57 pm
by doush
Would be good to have a special 6.38.x version from Mikrotik with security patches applied.
So at least we can try it out.

Re: CCR1072 watchdog reboot

Posted: Wed Dec 12, 2018 4:56 pm
by whoknew
Can we get a Mikrotik response please. The 1072 is ideal with the redundant PSU's but I cannot be rebooting it once or more a week.

At the very least like Doush said, give us a v6.38.xx that is patched. specifically for the CCR1072.

Re: CCR1072 watchdog reboot

Posted: Thu Dec 13, 2018 12:54 pm
by Murmaider
Can we get a Mikrotik response please. The 1072 is ideal with the redundant PSU's but I cannot be rebooting it once or more a week.

At the very least like Doush said, give us a v6.38.xx that is patched. specifically for the CCR1072.
I 2nd this, otherwise I'm just going to downgrade to 6.38.7 and firewall off the security issues.

Re: CCR1072 watchdog reboot

Posted: Mon Dec 17, 2018 11:45 pm
by ElviN
Hello!
People, don't upgrade to 6.43.7!!!!
After upgrading we getting freezing of device third time in the last two days...

Re: CCR1072 watchdog reboot

Posted: Tue Dec 18, 2018 1:49 am
by rime
Hi,
uptime: 13w2d23h16m46s
version: 6.43 (stable)
build-time: Sep/06/2018 12:44:56
free-memory: 14.8GiB
total-memory: 15.8GiB
cpu: tilegx
cpu-count: 72
cpu-frequency: 1000MHz
cpu-load: 2%
free-hdd-space: 76.5MiB
total-hdd-space: 128.0MiB
architecture-name: tile
board-name: CCR1072-1G-8S+
platform: MikroTik
No problem here.

Re: CCR1072 watchdog reboot

Posted: Fri Dec 21, 2018 12:51 am
by Deywid
Unfortunately the CCR1072 has a chronic problem that has not been resolved until now.
The Mikortik team does not position or make any statement about the situation.
3 units in production CCR1072 with the same symptoms when doing BGP using IPV6 restarts sporadically every day by the watchdog
If the watchdog is not active, it freezes and only works after the two sources are turned on.
It does not matter the updated version or not.

We can cite the same with the nickname of (1072 of frozen)

Frozen kill

Infelizmente o CCR1072 tem um problema crônico que não foi resolvido até agora.
A equipe Mikortik não posiciona ou faz qualquer declaração sobre a situação.
3 unidades em produção CCR1072 com os mesmos sintomas ao fazer BGP usando IPV6 reinicia esporadicamente todos os dias pelo watchdog
Se o watchdog não estiver ativo, ela congela e só funciona após deligar as duas fontes.
Não importa a versão atualizada ou não.

Podemos citar o mesmo com apelido de (1072 do congelada)

Congeladas matar

Re: CCR1072 watchdog reboot

Posted: Fri Dec 21, 2018 5:41 pm
by matheusazevedo
Hello guys, I have the same problem here with a CCR1036-8G-2S+. Anybody else?

Re: CCR1072 watchdog reboot

Posted: Wed Jan 02, 2019 10:37 pm
by bradnz
I 2nd this, otherwise I'm just going to downgrade to 6.38.7 and firewall off the security issues.
Did you end up doing this and has it restarted since? Im about to do this. I had upgraded to 6.43.4 and it was fine for about 28 days, then it restarted twice in the space of 2 days. I simply cant turn off Watchdog to get the required log details - the data centre this is in is a 30mins drive to get there, and our customers wouldnt be happy at all with that amount of downtime.

If there is hope downgrading the FW will sort this out I would rather just do that.,

Re: CCR1072 watchdog reboot

Posted: Sun Jan 06, 2019 7:48 pm
by Deywid
We are still waiting for mikrotik's official position on the CCR1072 freezes without any solution or information.

Complete wrapping of your best product, fix, worst product.

Re: CCR1072 watchdog reboot

Posted: Mon Jan 14, 2019 11:08 pm
by bradnz
I tried to downgrade to 6.38.3 and 6.38.7 and in both instances I had to recover the device from a boot loop using netinstall - the kernel wouldn't be found as it couldn't mount the drive. I have now upgraded to 6.43.8. Such a pain in the ass and I have now got really pissed off customers.

If this upgrade doesn't do anything to help, I'm probably going to start the process of getting a refund for the devices from our distributor and look at an alternative vendor. Its just ridiculous.

Re: CCR1072 watchdog reboot

Posted: Tue Jan 15, 2019 7:19 pm
by Deywid
SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.

Re: CCR1072 watchdog reboot

Posted: Tue Jan 15, 2019 11:13 pm
by whoknew
SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
So....you went from a $2,000 USD router to a $35,000 USD router?

Re: CCR1072 watchdog reboot

Posted: Wed Jan 16, 2019 12:51 am
by nz_monkey
SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
So....you went from a $2,000 USD router to a $35,000 USD router?
A MX80 is not a $35000 router! Where the hell do you buy your gear from ?

Re: CCR1072 watchdog reboot

Posted: Wed Jan 16, 2019 11:16 am
by kos
CCR1072 price is 3050$!

Re: CCR1072 watchdog reboot

Posted: Fri Jan 18, 2019 8:07 pm
by Deywid
That's not all.
Mx-80 = $ 6,700
CCR1072 = $ 3,000

When the plane is falling, paying double to survive is acceptable.

Re: CCR1072 watchdog reboot

Posted: Fri Jan 18, 2019 11:38 pm
by Kevo
For those whose 1072 is rebooting are you running at 1200MHz or 1000MHz. We are running the long-term/bugfix version and haven't had any reboots or lock ups. It has always been set to 1000MHz. It came that way as far as I know.

I'm wondering what feature or setting makes the difference. We have 3 interfaces in use and run about 3Gbps through it during peak times.

We are only doing routing OSPF, BGP, and a bit of basic firewalling.

Re: CCR1072 watchdog reboot

Posted: Sat Jan 19, 2019 1:26 pm
by Deywid
1000Mhz
BGP Com Ipv6 e Ipv4
OSPF
Firewall Basic
No full route
IBGP

Re: CCR1072 watchdog reboot

Posted: Mon Jan 21, 2019 1:44 pm
by kos
It not depends on features but traffic pattern and interface distribution.

Reboot requirements:
- connection-tracking activated
- clear routing between two interfaces
- 1G bidirectional traffic, 600B packets (each interface handles 1G in each direction)

If the packets are bigger:
- ~2,5G bidirectional traffic, 1500B packets

If the traffic is mostly unidirectional:
- ~2G unidirectional traffic, 600B packets
- ~5G unidirectional traffic, 1500B packets

Re: CCR1072 watchdog reboot

Posted: Tue Jan 22, 2019 8:57 pm
by Kevo
I've just completed removing any connection tracking from ours. We had some DDoS issues and from what I've read connection tracking can be a big problem in that case. So I've rearranged my firewall rules and adapted things to run in the raw table as much as possible and turned off connection tracking. So far things seem good, but no real test to speak of yet. We really didn't need connection tracking on this router it was just not something I really considered much until the DDoS stuff started.

Hopefully this will keep us out of trouble until MT can resolve the issue completely with a future update or something.

Re: CCR1072 watchdog reboot

Posted: Tue Jan 22, 2019 10:50 pm
by doush
We have 6Gbit/s on it.
CT is on.
We have moved most of the rules to RAW so around 450Mbit of traffic is currently NATed and processed by connection tracking table.
Rebooted again with a very small dDOS attack yesterday.

MT doesnt accept that there is a problem. So nothing will be fixed.
Check the beta thread.
viewtopic.php?f=21&t=139057&start=300#p707452

Re: CCR1072 watchdog reboot

Posted: Tue Jan 29, 2019 1:59 pm
by fr0zonza
We have a similar issue except it affects ALL of my CCR routers (1009-1036) at different locations at the same time.
Interfaces flap at the exact same time and sometime stay down until i disable and enable. In extreme cases some routers lock up completely.
On my LHG radios i get
"12:55:41 interface,warning wlan60-1: bridge port received packet with own address
as source address (xxxxxxxxxx), probably loop"

Has been going on for months and still no help from support.
We run a fully routed network so no chance of any loops.

Re: CCR1072 watchdog reboot

Posted: Tue Feb 19, 2019 10:31 am
by cabijuan
I have 3 CCR 1072 and in all the same thing happens to me, I'm pissed with Mikrotik, it can not be that your top-of-the-line router will pass this to you and do not say anything about it.

Re: CCR1072 watchdog reboot

Posted: Mon Feb 25, 2019 3:03 am
by bradnz
I've just completed removing any connection tracking from ours. We had some DDoS issues and from what I've read connection tracking can be a big problem in that case. So I've rearranged my firewall rules and adapted things to run in the raw table as much as possible and turned off connection tracking. So far things seem good, but no real test to speak of yet. We really didn't need connection tracking on this router it was just not something I really considered much until the DDoS stuff started.
I have just removed all NAT and Mangle rules, which means that CT is actually not operating on the device at all. its behind a Transparent Fortigate Firewall, so shouldnt be an issue anyway. I have moved NAT and PAT services to a CISCO 3925 router now, and havent yet seen a reboot. I wondered if the CT turned off would have actually just been the answer rather than having to install another router for this purpose of NAT / PAT. A bit annoying. Do you think if I just turned off CT it would have been ok? How have you found things have been since you did this?

Re: CCR1072 watchdog reboot

Posted: Mon Feb 25, 2019 10:34 am
by kos
Yes, when connection-tracking is disabled the device performs much better.

When I contacted Mikrotik support about device reboots they requested access to device. I provided them a test setup with one CCR1072 to act as traffic-generator and another one acting as DUT. No configuration on DUT, just routing and 4 FW rules to prevent unauthorized access.

After a few weeks of meaningless tests, the answer was - DUT is overloaded, you can't see it but it is.

With 1G?????? That is not even close to your test results published on your web site?????!!!!

- Our tests are performed with connection-tracking disabled!

That's all! You are ******!

I will say it again, CCR1009, RB1100x4 and RB4011 are working as expected in absolutely same test condition in which CCR1072 reboots it self!!!

Re: CCR1072 watchdog reboot

Posted: Mon Feb 25, 2019 10:38 am
by cabijuan
Another restart this Saturday, is desperate, I have the router at 400km, please Mikrotik say something about it.

Re: CCR1072 watchdog reboot

Posted: Mon Feb 25, 2019 4:28 pm
by cdemers
Email mikrotik support, they don't monitor the user forums for people having problems. Normus and a few others frequent here, but best just to email them with a support ticket and supout file.


Sent from my SM-A520W using Tapatalk


Re: CCR1072 watchdog reboot

Posted: Mon Feb 25, 2019 5:32 pm
by cabijuan
I always send it, and the only thing they tell me is update, I already have 5 different versions. It does not help to send it, they have no idea what the restarting does.

Re: CCR1072 watchdog reboot

Posted: Wed Mar 27, 2019 11:54 am
by djdrastic
Reading this thread as I have to upgrade a 1016 due to lack of SFP+ ports . Has worked 1y without skipping a beat barring the security updates.
Am I better off getting a EdgeRouter Infinity or building a cheap X86 box ?

Re: CCR1072 watchdog reboot

Posted: Sat Mar 30, 2019 4:34 pm
by doush
Dont upgrade to CCR1072. It is not a stable product to work with.
Frequent reboots etc..

Re: CCR1072 watchdog reboot

Posted: Thu Apr 04, 2019 4:53 pm
by whoknew
SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
So....you went from a $2,000 USD router to a $35,000 USD router?
A MX80 is not a $35000 router! Where the hell do you buy your gear from ?
Where are you buying them from. Distributor list price all show $19,000+ USD. I have a friend who works for a larger ISP and they can get them for around $6,800 directly from Juniper.

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 12:56 am
by wildbill442
Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker
Unfortunately they are not! Neither in ROS 6.42.2 nor in any other ROS version!!!

CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled, due to watchdog timer reboots!!!

Five to ten times cheaper devices like CCR1009 and RB1100AHx4 are working fine in same test conditions!

The worst thing is that they are continuing selling this faulty device!

Contacting Mikrotik support is a lost of time!
Looks like I'm having a similar issue...

viewtopic.php?f=2&t=152192

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 6:45 am
by Fluke
Out of curiosity - what is the device temperature? (system health print)

I have a device that runs at ~53C and reboots occasionally, and another one that runs at ~39C and works OK.

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 11:17 am
by Jeanluck
My CCR1072 works at 37-39

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 12:00 pm
by cabijuan
forget about the temperature, it is a problem in the kernel that mikrotik does not know how to solve, nor does it pay attention to us. Hopefully with the V7 this is solved, it is the only hope I have left. the top-of-the-range router does this happen, it's nonsense

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 4:51 pm
by mac86
We've downgraded CCR1072 to 6.44.5 and no more reboots. :D

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 6:08 pm
by Jeanluck
can anyone confirm that version 6.44.5 is stable with the CCR1072?

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 6:21 pm
by wildbill442
can anyone confirm that version 6.44.5 is stable with the CCR1072?
We’ve just downgraded to latest long term, I’ll update in 48hrs.

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 6:36 pm
by Jeanluck
Please let us know how it works in a few days

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 7:31 pm
by cabijuan
I have 6.43.8, and I have reboots.

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 7:57 pm
by Jeanluck
Thanks, 6.43 discarded then... let's see if there's luck with 6.44.
I have the 6.38.3 and only had one reboot in 3 years

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 7:58 pm
by mac86
can anyone confirm that version 6.44.5 is stable with the CCR1072?

Yes, I can confirm it.
4 weeks without reboots, and going on.....

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 8:16 pm
by Jeanluck
What reboot frequency did you have previously?

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 8:34 pm
by miltont
Up 60+ Days

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 10:49 pm
by wildbill442
What reboot frequency did you have previously?
One to multiple times in a 24HR window.

Mikrotik support wanted me to turn off watchdog and log error / generate supout.rif once locked. The generated supout.rif files I provided after the watchdog reboot didn't yield any useful information. I opted to netinstall the latest long term release track (6.44.5). We also replaced all the SFP modules as a precaution during the maintenance window.

Mikrotik Support's Recommendation:
Before that, I suggest that you upgrade to the latest "stable" version (if there is an actual bug, then it might be already fixed). For example, v6.45.6 fixes a Watchdog reboot caused by h323 firewall helper. If your router did process voice call traffic, then the issue might be already resolved.

What's new in 6.45.6 (2019-Sep-10 09:06):
*) conntrack - improved system stability when using h323 helper (introduced in v6.45);


These are our edge routers and no NAT is being performed and we don't use h323 internally, but if some malformed h323 packet traversing these routers was causing the reboot then this may have been the cause behind the sporadic reboots. Again I'll post results of the downgrade to the Latest Long Term Release track after ample time has passed. So far ~15hrs and no reboot, this won't be conclusive until we make it past the 24-48hr mark. From what other users above have posted I'm feeling optimistic.

Re: CCR1072 watchdog reboot

Posted: Tue Sep 17, 2019 11:34 pm
by wildbill442
Here's a sequence of events on how this issue started for us:

It began with a kernel failure on one of our edge routers, I'll call it EDGE A. To fix this we decided to do a netinstall on EDGE A and bring the router up to the latest stable release at the time (6.45.5) due to CVE fixes etc. Prior to the netinstall we were running 6.42.6 on both edge routers and other than the kernel failure on EDGE A we were not experiencing reboots. We also upgraded our other edge router, EDGE B, in the same maintenance window to 6.45.5. The reboots continued after the netinstall on EDGE A only the error changed from "reboot due to kernel failure" to "reboot by watchdog timer". At this time we were under the impression there was a hardware issue with the EDGE A router so we moved the BGP peer and other connections to EDGE B and overnighted new hardware. The reboots then started happening on EDGE B. We opened a ticket with Mikrotik support, but after going through the weekend and into Monday with no response and the routers rebooting sporadically we decided to netinstall the latest longterm release track (6.44.5). Mikrotik support responded after we were done with our downgrade and reaffirmed our suspicion that there was something wrong in software and to upgrade to latest stable release (6.45.6) due to a stability improvement relating to H323 and watchdog reboots. As we did not need any of the "new" features in the Stable Release tree, and receiving this information after downgrading to Long-Term Release tree, I think we're going to stay here unless the issue persists.

Again I won't know definitively if this was the root cause until ample time has passed, so I'll update after we get passed that 48hr mark.