Community discussions

 
cinatus
just joined
Topic Author
Posts: 15
Joined: Fri Nov 18, 2005 4:50 am

CCR1072 watchdog reboot

Mon Jun 12, 2017 7:08 pm

We have several CCR1072s in our core and in the last 2 days we have had 2 watchdog reboots. one with the router on 6.38.5 and the other with the router on 6.39.2. What could be causing it and what should I do to prevent it in the future.
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Mon Jun 12, 2017 10:08 pm

We fighted too for same reboot. We're talking with the support and they suspect an hardware failure, but i can reproduce reboots always with same conditions and are:

- If you use traffic-flow with selected interfaces (i mean anything different then interfaces: ALL), we have continous reboot every 3-4 hours

- If you overclock CPU to 1200Mhz in some condition it reboot.

This is what i did:
- Forced the firmware upgrade (it was already 3.33 version, but i forced the reinstall). Revert back CPU from 1200 to 1000Mhz. Put interfaces to ALL in traffic-flow configuration with that values

> /ip traffic-flow print
enabled: yes
interfaces: all
cache-entries: 2M
active-flow-timeout: 2m
inactive-flow-timeout: 1m

With that changes i not experienced more reboots (i still monitor it has only two days passed, but prior i have more frequent reboot).

If you can, attach serial console and keep it open, this is what i saw after reboots:

MikroTik Login: (0,0) hv_warning: L2$ correctable data ECC error at PA 0xf8a8ff30
(0,0) hv_panic: got processor error: PC 0xffff_fff7_0051_e7c0, ICS/PL 0x6
(0,0) SBOX_ERROR: 0x0000_0000_0000_0000
(0,0) MEM_ERROR_CBOX_ADDR: 0x0000_0000_f8a8_fd78
(0,0) MEM_ERROR_CBOX_STATUS: 0x0000_0000_001c_0405
(0,0) L2 data ram 2-bit error detected.
(0,0) MEM_ERROR_MBOX_ADDR: 0x0000_0000_0000_0000
(0,0) MEM_ERROR_MBOX_STATUS: 0x0000_0000_0000_0000
(0,0) XDN_DEMUX_ERROR: 0x0000_0000_0000_0000
 
cinatus
just joined
Topic Author
Posts: 15
Joined: Fri Nov 18, 2005 4:50 am

Re: CCR1072 watchdog reboot

Wed Jun 14, 2017 6:38 pm

Thank you very much for the input. I will see what that does in the next maintenance window.
 
User avatar
Murmaider
Member Candidate
Member Candidate
Posts: 121
Joined: Fri Oct 30, 2015 10:10 am

Re: CCR1072 watchdog reboot

Thu Jun 29, 2017 11:32 pm

I too am experiencing this.

We do however have our units overclocked to 1200Mhz... I wonder if that may be the issue.
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Thu Jun 29, 2017 11:49 pm

Yes, we have downgraded to 1000Mhz and we not had more unexpected reboot
 
User avatar
Murmaider
Member Candidate
Member Candidate
Posts: 121
Joined: Fri Oct 30, 2015 10:10 am

Re: CCR1072 watchdog reboot

Fri Jun 30, 2017 9:17 am

Yes, we have downgraded to 1000Mhz and we not had more unexpected reboot
I'm going to give this a try, thanks a lot.
 
LynxChaus
just joined
Posts: 24
Joined: Tue Jul 08, 2014 2:24 pm

Re: CCR1072 watchdog reboot

Tue Jul 04, 2017 2:47 pm

Yes, we have downgraded to 1000Mhz and we not had more unexpected reboot
Same symptom as I reported here. Check PSU.
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Tue Jul 04, 2017 2:58 pm

1036 is single PSU, 1072 is redundant. Also we experienced same issue on 3 different CCRs.

Is a different case
 
rvzweb
just joined
Posts: 3
Joined: Thu Sep 11, 2014 9:49 pm

Re: CCR1072 watchdog reboot

Thu Aug 17, 2017 9:42 pm

I have the same issue on CCR 1072 - cpu is 1000 MHz - fw is 6.40.1
Any ideas?
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Thu Aug 17, 2017 11:19 pm

me too on lab router. Disabling watchdog we see the CPU goes to 100% due to networking process and router became unusable.

Downgrade to 6.40 fix the issue.
 
User avatar
pamafer
just joined
Posts: 3
Joined: Wed Nov 01, 2017 2:42 pm
Location: Argentina
Contact:

Re: CCR1072 watchdog reboot

Wed Nov 01, 2017 3:09 pm

I have the same problem, CCR1072 with 6.40.3, I only use BGP router mode, I do not use IP Traffic, the peak load never exceeds 6% of all CPUs, and the bandwidth never exceeds 1Gbps, anyway I suffer spontaneous reboots at any time, days can pass without problem and suddenly the Watchdog reboots the system. Any ideas?!
 
User avatar
nocmonkee
just joined
Posts: 4
Joined: Thu May 04, 2017 8:07 am

Re: CCR1072 watchdog reboot

Thu Nov 02, 2017 9:17 pm

We started experiencing these watchdog reboots on our CCR1072s running 6.40.1. Is this a reported bug? Does downgrading to 6.40.0 really stabilize it? Is it fixed in newer versions?

nov/02/2017 05:13:15 system,error,critical router was rebooted without proper shut
down, probably kernel failure
nov/02/2017 05:13:15 system,error,critical kernel failure in previous boot
nov/02/2017 05:13:15 system,error,critical router was rebooted without proper shut
down, probably kernel failure
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Thu Nov 02, 2017 9:22 pm

hi,
at the moment we have 21 ccr1072 with 6.41rc44 all up with 17 days without issue. We do bgp + filtering + ospf. Nothing else. Try to upgrade to this release, if you still experience reboot you can exclude these service as reboot cause.

We experienced reboots with cpu upgraded to 1200Mhz, at 1000mhz never experienced it.
 
User avatar
nocmonkee
just joined
Posts: 4
Joined: Thu May 04, 2017 8:07 am

Re: CCR1072 watchdog reboot

Thu Nov 02, 2017 9:57 pm

We currently are not running any dynamic routing protocols. The purpose of our CCRs is NAT. They also manage dhcpd and upnpd with lacp, multiple rfc1918 vlans 20G in and one vlan 20G out.
 
User avatar
pamafer
just joined
Posts: 3
Joined: Wed Nov 01, 2017 2:42 pm
Location: Argentina
Contact:

Re: CCR1072 watchdog reboot

Fri Nov 03, 2017 12:34 pm

hi,
at the moment we have 21 ccr1072 with 6.41rc44 all up with 17 days without issue. We do bgp + filtering + ospf. Nothing else. Try to upgrade to this release, if you still experience reboot you can exclude these service as reboot cause.

We experienced reboots with cpu upgraded to 1200Mhz, at 1000mhz never experienced it.
Hi berlo, in the same way when we knew we had it in 1200Mhz of CPU we suffered much more frequent reboots but now we have two 1072 to 1000Mhz of CPU and suffers the reboots only one of them, it is the router that has less workload !!!. So it is a contradiction!
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Fri Nov 03, 2017 1:17 pm

have you tried disabling whatdog and keep serial console opened?

You should see the error
 
User avatar
pamafer
just joined
Posts: 3
Joined: Wed Nov 01, 2017 2:42 pm
Location: Argentina
Contact:

Re: CCR1072 watchdog reboot

Fri Nov 03, 2017 1:48 pm

have you tried disabling whatdog and keep serial console opened?

You should see the error

I just tried to disable the watchdog, but not the console, I'll try it and comment! Thanks
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Fri Nov 03, 2017 1:54 pm

You need to have console opened, because if is kernel panic or memory error or similar you can't see the error message
 
sakirozkan
newbie
Posts: 40
Joined: Sat Jun 14, 2014 12:19 pm

Re: CCR1072 watchdog reboot

Sat Nov 04, 2017 7:39 am

hi,
at the moment we have 21 ccr1072 with 6.41rc44 all up with 17 days without issue. We do bgp + filtering + ospf. Nothing else. Try to upgrade to this release, if you still experience reboot you can exclude these service as reboot cause.

We experienced reboots with cpu upgraded to 1200Mhz, at 1000mhz never experienced it.
It is still working without any issue. What is the uptime of ccr1072 now.
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Sat Nov 04, 2017 1:29 pm

Yes and now ccr was raised to 28 in all Europe. All are working fine and we never experienced more random reboots. Also we experienced better performance on routes with > 1kk routes installed disabled route cache. You loose some % CPU, about 10% more, but you will not experiencing packetloss/stop forwarding when router will forward > 2mil pps
 
sakirozkan
newbie
Posts: 40
Joined: Sat Jun 14, 2014 12:19 pm

Re: CCR1072 watchdog reboot

Mon Nov 20, 2017 6:10 pm

is there any development of the watchdog reboot subject. What version are u using for solution. Our 1072 is reboots every 2 weeks.
 
User avatar
mac86
Member Candidate
Member Candidate
Posts: 118
Joined: Sat Nov 25, 2006 12:52 am
Location: bahia blanca - argentina
Contact:

Re: CCR1072 watchdog reboot

Wed Nov 29, 2017 11:55 pm

there is some update about this post ?

I've a ccr 1072 and get watchdog reboot and kernel panic reboot

this is about watchDog reboot
nov/29/2017 10:35:40 system,error,critical router was rebooted without proper shutdown by watchdog timer

and this is about kernel Failure reboot
nov/28/2017 10:33:45 system,error,critical router was rebooted without proper shutdown, probably kernel failure
nov/28/2017 10:33:45 system,error,critical kernel failure in previous boot
really fix it downgrading to 6.40 ?
it's about a hardware problem ?

In the meantime, I've checked routerboard clock on 1000mhz, disable watchdog, and I've disable traffic flow too.

thank you.
http://www.netpro-ar.com
MTINE - MTCWE - MTCRE - MTCTCE - MTCNA
Patagonia Argentina IT Consultant

Mikrotik user from V2.7.7 - Aug/2003
 
msbr
just joined
Posts: 8
Joined: Thu Sep 17, 2015 10:30 pm
Location: Mendoza
Contact:

Re: CCR1072 watchdog reboot

Thu Nov 30, 2017 1:26 am

We have several CCR1072s in our core and in the last 2 days we have had 2 watchdog reboots. one with the router on 6.38.5 and the other with the router on 6.39.2. What could be causing it and what should I do to prevent it in the future.
Hi
I have same problem with CCR1019.
So I disable whatdog



Enviado desde mi iPhone utilizando Tapatalk
 
User avatar
Murmaider
Member Candidate
Member Candidate
Posts: 121
Joined: Fri Oct 30, 2015 10:10 am

Re: CCR1072 watchdog reboot

Mon Dec 04, 2017 1:25 pm

Yes and now ccr was raised to 28 in all Europe. All are working fine and we never experienced more random reboots. Also we experienced better performance on routes with > 1kk routes installed disabled route cache. You loose some % CPU, about 10% more, but you will not experiencing packetloss/stop forwarding when router will forward > 2mil pps
If I am reading this right, is fastpath only reducing the CPU by 10%?
Is the benefit of the disabled route cache better than using fastpath?
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Mon Dec 04, 2017 1:32 pm

the CPU usage in fastpath is always lower, so on high normal traffico fastpath still the only one solution.

But if without fastpath the ccr can handle the traffic, you can keep route cache disable that will help under ddos where you experiencing stop packet forwarding.
 
User avatar
Murmaider
Member Candidate
Member Candidate
Posts: 121
Joined: Fri Oct 30, 2015 10:10 am

Re: CCR1072 watchdog reboot

Wed Jan 24, 2018 6:11 pm

@berlo - did you need to reboot after you disabled route cache?
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Wed Jan 24, 2018 9:37 pm

No, is on the fly. All changes can be done without reboot. The only issues are dummy rules that are not removed automatically, but need to reboot it to reactivate fast path
 
User avatar
strods
MikroTik Support
MikroTik Support
Posts: 1407
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: CCR1072 watchdog reboot

Thu Jan 25, 2018 7:04 am

If you have a router which reboots itself, then conversation in forum will only be a guessing game. If you want to find out for sure what is the problem, then send supout file (generated after reboot) to support@mikrotik.com. We are the only ones who can tell what the problem was. Tracing the cause of the issue might take a while, but still it is the best solution how to get rid of the problem.

In case of Watchdog reboots - they are caused by software. Basically router says to itself "/system reboot" at the point when router becomes inaccessible. In order to trace an issue you have to disabled Watchdog and now router either will reboot or get stuck. In both cases after reboot (either router rebooted itself or you have to power cycle it) generate new supout file and again - send it to support@mikrotik.com
 
buset1974
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Sep 13, 2006 12:12 pm
Location: Jakarta

Re: CCR1072 watchdog reboot

Sat Jan 27, 2018 7:53 pm

i also have the same problems,

mikrotik also have the same request to us to disable the watchdog and try to send supout file white the problem happen, well it's make sense but we have to accept risk the device wont reboot or freeze until we cycle the power and with will cause very long downtime.

but i just realize so many people having problem with me.

we already replace the device with the same type twice and both having the symptom, so it software problem i guests
We have many CCR1072 operate but only 1 device with this particular configuration that having problems.
the one that have problem is run internet BGP full route both ipv4 & ipv6 and running on 1000 mhz.cpu.

We have other CCR1072s even running more complicated configuration like MPLS with OSPF + BGP running, but all running fine.

thx
 
buset1974
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Sep 13, 2006 12:12 pm
Location: Jakarta

Re: CCR1072 watchdog reboot

Sun Jan 28, 2018 4:20 am

No, is on the fly. All changes can be done without reboot. The only issues are dummy rules that are not removed automatically, but need to reboot it to reactivate fast path
hi @berio,

sorry i just deep read your thread, i am using version 6.41 (stable) but still have randomly reboot.
so the DDOS packet caused the problem? i also have suspicion about this ddos things , because we having this similar problem around 1 year ago when one of our site having DDOS , not big though just around 50-100 mbps but it's caused the router keep reboot. At that time mikrotik give us solution to change all the interface queue type from "only-hardware-queue" to default or default-small and it's solved for view months after i change the queue to "only-hardware-queue" again.

so if i'am not wrong you suggestion to disable route cache right?
what is router cache anyway? if i disable the route cache is there will be other impact regarding performance? because the router running for about 300.000 routes (bgp) inside?

thx
 
buset1974
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Sep 13, 2006 12:12 pm
Location: Jakarta

Re: CCR1072 watchdog reboot

Sun Jan 28, 2018 4:29 am

have you tried disabling whatdog and keep serial console opened?

You should see the error
Hi berio,

if i tried this (put serial on and disable watchdog), when freeze can i still reboot the router after see the error and capture the supout file?
because the router is not on 24 hrs man on site.
but i can arrange a pc with other link so when it happen i can still remote the pc.


thx
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Mon Jan 29, 2018 5:07 pm

hi,
route cache should not cause reboot, but stop on packet forwarding. If your device have enough performance to forward traffic in slow path you can try disable the cache. You will see CPU usage increasing.

If you got kernel panic you need to hard reboot the router, so you need a managed pdu or someone that power cycle it.
 
buset1974
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Sep 13, 2006 12:12 pm
Location: Jakarta

Re: CCR1072 watchdog reboot

Mon Jan 29, 2018 6:12 pm

hi,
route cache should not cause reboot, but stop on packet forwarding. If your device have enough performance to forward traffic in slow path you can try disable the cache. You will see CPU usage increasing.

If you got kernel panic you need to hard reboot the router, so you need a managed pdu or someone that power cycle it.
hi Berlo,

yes i've disable the cache and it's still rebooting.
Tonight i will set cache enable again and try to change all interface queue to default-ethernet

thx
 
BMena
just joined
Posts: 24
Joined: Thu May 17, 2012 8:51 pm

Re: CCR1072 watchdog reboot

Tue Jan 30, 2018 1:47 pm

I was also having my CCR 1072 rebooting, but mine was "without proper shutdown" instead of "by watchdog".
Sent a e-mail to support@mikrotik and he said he thinks the hardware is faulty..

But then another CCR 1072 I have, about 300km away, showed the same problem.

My solution for both was disconecting one power supply.

Before unpluging, I noticed under system health that while it was reading PSU1 and 2 voltage of 12.1v, only one was outputing Current.

Still can't test the supposedly bad PSU I removed cause I don't have a spare 1072 atm.

I have a third 1072 working fine where both PSU share the outputed current.
 
buset1974
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Sep 13, 2006 12:12 pm
Location: Jakarta

Re: CCR1072 watchdog reboot

Wed Jan 31, 2018 5:36 pm

i set cache enable again and try to change all interface queue to default-ethernet around 2 days and the device still randomly rebooting.
so many people having similar problem and mikrotik still did not have any clue
CCR1072 is mikrotik most expensive RB and premium router type though, they should take some action regarding this issues

hx
 
buset1974
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Sep 13, 2006 12:12 pm
Location: Jakarta

Re: CCR1072 watchdog reboot

Thu Mar 08, 2018 1:17 pm

I was also having my CCR 1072 rebooting, but mine was "without proper shutdown" instead of "by watchdog".
Sent a e-mail to support@mikrotik and he said he thinks the hardware is faulty..

But then another CCR 1072 I have, about 300km away, showed the same problem.

My solution for both was disconecting one power supply.

Before unpluging, I noticed under system health that while it was reading PSU1 and 2 voltage of 12.1v, only one was outputing Current.

Still can't test the supposedly bad PSU I removed cause I don't have a spare 1072 atm.

I have a third 1072 working fine where both PSU share the outputed current.
well is it solved now? by disconnecting 1 psu?

I also have the same answer, they said it hardware problem, i told them we have tried on 3 routers and all have the same issues.
so if it's an hardware issues mean all of my router have to be RMA, and also hundred of thousand other router in all over the world, so it means it's factory failure and they have to fix it.


thx
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5934
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: CCR1072 watchdog reboot

Thu Mar 08, 2018 1:31 pm

Upgrade to v6.41.2

Then upgrade bootloader to v6.41.2
/system routerboard upgrade

And reboot twice.

This should fix kernel crashes that previously was thought as hardware failure. Also it could fix other abnormal router behavior.
 
buset1974
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Sep 13, 2006 12:12 pm
Location: Jakarta

Re: CCR1072 watchdog reboot

Thu Mar 08, 2018 1:43 pm

Upgrade to v6.41.2

Then upgrade bootloader to v6.41.2
/system routerboard upgrade

And reboot twice.

This should fix kernel crashes that previously was thought as hardware failure. Also it could fix other abnormal router behavior.
right now i'am using 6.41.2, i've tried all the firmware from 6.41.2 and mikrotik gave me also 3.42.5 and 6.99.
but the problem still exists.
i've sent all the supout from all of the firmware version.

btw are u maris from mikrotik?

thx
 
BMena
just joined
Posts: 24
Joined: Thu May 17, 2012 8:51 pm

Re: CCR1072 watchdog reboot

Sat Mar 24, 2018 2:48 pm

Could you find a fix? After 20 days my CCR rebooted out of the blue again...
So it becoming stable after removing only 1 PSU was just coincidence. Put the PSU back in place, better to have it rebooting than risking it stopping.
 
User avatar
mac86
Member Candidate
Member Candidate
Posts: 118
Joined: Sat Nov 25, 2006 12:52 am
Location: bahia blanca - argentina
Contact:

Re: CCR1072 watchdog reboot

Sat Mar 24, 2018 8:27 pm

Could you find a fix? After 20 days my CCR rebooted out of the blue again...
So it becoming stable after removing only 1 PSU was just coincidence. Put the PSU back in place, better to have it rebooting than risking it stopping.
have you upgraded your hardware routerboard, with last RouterOs version?
(not just router OS)
/system routerboard upgrade
http://www.netpro-ar.com
MTINE - MTCWE - MTCRE - MTCTCE - MTCNA
Patagonia Argentina IT Consultant

Mikrotik user from V2.7.7 - Aug/2003
 
doush
Long time Member
Long time Member
Posts: 621
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Sat Mar 24, 2018 11:27 pm

We will deploy a CCR1072 soon on our NOC.
Does all CCR1072 s have the same problem ?

Should we postpone the deployment ?
 
BMena
just joined
Posts: 24
Joined: Thu May 17, 2012 8:51 pm

Re: CCR1072 watchdog reboot

Tue Apr 10, 2018 3:04 pm

Could you find a fix? After 20 days my CCR rebooted out of the blue again...
So it becoming stable after removing only 1 PSU was just coincidence. Put the PSU back in place, better to have it rebooting than risking it stopping.
have you upgraded your hardware routerboard, with last RouterOs version?
(not just router OS)
/system routerboard upgrade
Yup, I did.
We will deploy a CCR1072 soon on our NOC.
Does all CCR1072 s have the same problem ?

Should we postpone the deployment ?
Unless you're in real need. I have three, one each city, the reboots aren't awfully frequent but they happen.
The one I have that doesn't reboot have a bug with the ping tool, sometimes it stops working at all for some hours.
 
Wibernet
just joined
Posts: 13
Joined: Thu Mar 16, 2017 8:22 am

Re: CCR1072 watchdog reboot

Tue Apr 10, 2018 4:40 pm

We are having this exact same issue, our CCR1072 randomly reboots.

Did the firmware upgrade fix this issue?

I must say I cannot tell people to avoid this device more. Since installing we have had major issues with latency spikes until we removed all NAT (Worked perfectly in CCR1036) we have had these random reboots disconnecting all of our clients for 10 minutes randomly. Nightmare.
 
doush
Long time Member
Long time Member
Posts: 621
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Tue Apr 10, 2018 10:10 pm

OMG !
We were thinking about replacing our core 1036 with 1072 but it seems that there are major issues with it.
 
sakirozkan
newbie
Posts: 40
Joined: Sat Jun 14, 2014 12:19 pm

Re: CCR1072 watchdog reboot

Mon Apr 23, 2018 6:08 pm

Is there any development with new versions. I use 6.38.7 for this reason.
We want to upgrade 1072 for "Vulnerability exploiting the Winbox port" Anyone use 6.42 version with 1072 without watchdog reboot error.
 
heddita
just joined
Posts: 7
Joined: Mon Dec 07, 2009 5:48 am

Re: CCR1072 watchdog reboot

Tue May 08, 2018 12:24 am

We have this problem happening as well. It got worse recently so I came across this thread. It was previously on version 6.38.5, then up to 40.1 and now it's at 42.1. I just changed the CPU speed to 1000Mhz. I don't know why it comes at 1200Mhz if it causes issues. So far it's been up for 1:40 so not a lot of time has gone by... We've had about 5 unexpected reboots have happened today. Two of them caused the router to just hang completely. 1036's only have 2 10g ports and I need 3 ;_; I'm going to move the NAT to another box if it reboots itself again. This router literally only has OSPF running with a few vlans + NAT. At the time it rebooted, it was early morning so the traffic at the time was low, around 300mbps... It's not like it's using it's full force is what I mean. Well, I lit a candle, we'll see how it goes.
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: CCR1072 watchdog reboot

Tue May 08, 2018 12:46 am

The one I have that doesn't reboot have a bug with the ping tool, sometimes it stops working at all for some hours.
Disable route cache, it fixed for me.
 
ElviN
just joined
Posts: 4
Joined: Tue May 15, 2018 1:04 pm

Re: CCR1072 watchdog reboot

Tue May 15, 2018 1:34 pm

Hi!
Read the topic....I'm shoked....
I laughed frankly when i read it: "We are the only ones who can tell what the problem was".
We have been in correspondence with support for almost a year now about the gaps in arbitrary reboots of the device... We are sending many supout.rif files. And what a requests of support: "have a problem with power, DDOS, changed CPU".
We disabled whatchdog, changed power suply, we have 2 CCR1072. The last thing we did, we getting new device in our resseller and what can we see,(i think you guess) his arbitrary rebooted again and again....
Guys, are you seriously? A lot of your clients have the same problem, and you know about this and don't recognize the problem.

We are bought your top device and you say to reduce the performance by almost 20% (CPU downgrade). It's abnormal...

I do not recommend buying this device
 
VagnerBecker
just joined
Posts: 18
Joined: Wed Mar 13, 2013 3:49 pm
Location: Brasil

Re: CCR1072 watchdog reboot

Tue May 22, 2018 5:21 pm

Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Tue Jul 10, 2018 10:45 am

Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker
Unfortunately they are not! Neither in ROS 6.42.2 nor in any other ROS version!!!

CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled, due to watchdog timer reboots!!!

Five to ten times cheaper devices like CCR1009 and RB1100AHx4 are working fine in same test conditions!

The worst thing is that they are continuing selling this faulty device!

Contacting Mikrotik support is a lost of time!
 
ElviN
just joined
Posts: 4
Joined: Tue May 15, 2018 1:04 pm

Re: CCR1072 watchdog reboot

Mon Jul 30, 2018 11:34 am

Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker
No, device rebooted again with kernel error fail.
Problem not solved
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Thu Aug 09, 2018 9:50 pm

I have a CCR1072 with RouterOS 6.38.3 and it works perfectly. I wanted to update it to 6.42.6 but reading this thread I'm afraid...
Has anyone ever tested if the problems go away with 6.42.6?

Can anyone tell me if these problems exist only after version 6.38? (I need to know if the problem is determined by the RouterOS version or by the configuration, in which case I understand that I should already have problems)
 
tduchch
newbie
Posts: 26
Joined: Tue Aug 24, 2004 3:58 pm

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 5:16 am

These random reboots are becoming very annoying. Looking for a replacement solution, but have not found one with at least 3 SFP+ ports. We do a lot of MPLS and VPLS on this box, have not been able to use CHR on either VMWare or Hyper-V.

Recently upgraded to 6.42.9 Long Term, and still get random watchdog reboots. We have a serial logging device connected to the console port, and there is no output before the reboot. Of course, nothing in the logs.

Has anybody received any support from Mikrotik on this issue?
Any other hardware that someone has used in place of the 1072?

Terry
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 1:33 pm

Works fine for me with 6.38.3, can you use this version?
 
User avatar
strods
MikroTik Support
MikroTik Support
Posts: 1407
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 6:40 pm

There are multiple possible reasons why router is rebooting:

1) Hardware problem;
2) Software issue;
3) Overloaded device.

All of these three possible issues can be masked under log messages "rebooted due to Kernel Failure" , "rebooted due to Watchdog timeout", etc.

There is no reason to believe that issue, that you have on router, is the same issue as other user has just because log messages are equal. Only reason why you could believe that issue is the same is if problem appears for the first time at the same RouterOS version for multiple devices or you know how to trigger it.

In any other case contact MikroTik support. Only support has ability to see crashes and help you in order to diagnose issue and resolve the problem in its roots.

Multiple times we have seen reports "when you will fix Kernel Failure bug". This is equivalent with "when RouterOS will be a bug free version". There is no bug free software, but you can resolve your problem only by cooperating with MikroTik support. It is also the fastest way how to reach solution, after an upgrade to the latest version (if problem is already fixed by MikroTik).
 
User avatar
strods
MikroTik Support
MikroTik Support
Posts: 1407
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 6:46 pm

If you have CCR device that is crashing, then I recommend that you make sure that latest RouterBOOT version is installed on your router. v6.42 includes latest firmware for TILE devices which could help due to this fix:

!) tile - improved system performance and stability ("/system routerboard upgrade" required);
 
tduchch
newbie
Posts: 26
Joined: Tue Aug 24, 2004 3:58 pm

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 7:06 pm

Yes we are running latest long term version 6.42.9
We have had this problem in several versions.
We have swapped out different hardware
We have opened support ticket, were told the only way to trouble shoot this was to turn off watch dog and see what appears on the console port.
The router is a two hour drive away, and is a core router for all our network, letting it lock up is not an option.
As stated, we have a logging device on the console port and nothing appeared before the reboot.
The device can reboot during high traffic, or very low. No pattern exists.
CPU load at high traffic is around 25%, profile has 10% Firewall and 10% network.
CPU does not seem to spike before reboot.
High traffic is just approaching 2Gbps, should be no problem for this device.
Sometimes it runs for days, sometimes only for hours.

I will open another ticket with support, hopefully this can be figured out.
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Mon Oct 15, 2018 1:50 pm

Hi strods,

as I mentioned earlier, CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled.
To achieve reboot there is no no need of any other configuration, except two IP addresses, two routes and connection-tracking enabled.
 
DeCoen
just joined
Posts: 1
Joined: Tue Oct 03, 2017 11:33 am
Location: NL
Contact:

Re: CCR1072 watchdog reboot

Thu Nov 15, 2018 11:22 am

Hi strods,

as I mentioned earlier, CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled.
To achieve reboot there is no no need of any other configuration, except two IP addresses, two routes and connection-tracking enabled.
How often do you see a reboot? We have 10x 1072 and we are on 6.38.7, its reboots about once every few months, sometimes few weeks in between.
See screenshot, connection tracking is on, and with a load of more then 1 gig .
We are planning to upgrade to 6.42.9 , should we wait?

thanks,
You do not have the required permissions to view the files attached to this post.
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Mon Nov 19, 2018 10:35 am

Your traffic is distributed over two interfaces. That is why you are not experiencing more frequent reboots.

I am speaking about situation where two interface are loaded to 1G in every direction (1Gbps bidirectional forwarding). So 1Gbps entering sfp+1 and going out trough sfp+2, and in the same time 1Gbps is entering sfp+2 and going out trough sfp+1.
 
doush
Long time Member
Long time Member
Posts: 621
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Fri Nov 23, 2018 10:39 am

Frequent reboots EVERY DAY !
"Router was rebooted without proper shutdown by watchdog timer"

This issue is still not resolved and expect an answer or a possible reason from Mikrotik.

Mikrotik Support is pretty much less useless as of now.

As I say, it happens every EVERY DAY and mikrotik is silent about the issue. This behavior is not right.

CCR1072 is a useless device as it has this major fundamental flaw and should be avoided for anyone who are considering about using it in any deployment.

An reply from Mikrotik would be nice.
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Mon Nov 26, 2018 3:53 pm

We are having the same issue with the watchdog timer reboot.

disabling the watchdog causes the router to lock up until power is taken away. Full reboot with pulling power cables from both Power supplies.

We are looking for a solution to this. Our CCR1036 that this replaced did not have this issue. We are @ less than 10% load always, less than 2Gbps/300Mbps and only have static routes on this router as it is our core.
 
bradnz
newbie
Posts: 27
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Wed Nov 28, 2018 9:35 am

Hi all.

I have a CCR-1072 on 6.43.2 and this error was in the log "critical router was rebooted without proper shutdown by watchdog timer". I have sent the SUPOUT log to mikrotik and they are saying the following:

"For debugging you should turn off watchdog and test again.
"/system watchdog set watchdog-timer=no"

Connect now to your router over serial console, make sure that you have accessed RouterOS command line interface and leave console running.

Now router either:

1) might be stuck (freeze). If that happens, then you have to generate supout file on the router through serial console. Now you can reboot device.

2) might be stuck (freeze) and become unavailable over serial console. If that happens, then reboot router and then generate supout file.

3) will reboot. If that happens, then after reboot generate supout file.

Send supout file and full serial console output (within text file) to us for investigation."

This is a production router that I have already swapped for our cold spare in the hope it was just a power supply / hardware issue, but it would appear, based on this, that its not. I can let it just freeze because its in a data centre 20 mins from the office. 20mins of downtime would be unacceptable to our customers. What are your thoughts? Should I just replace it with a 1036, or a 1009? Are these any better? It runs BGP, only have about 300 BGP routes, runs at about 9% CPU most of the time, and has almost all its memory available. It just cant be that hard for this router to do surely.

Any suggestions would be appreciated.
 
User avatar
Lifz
newbie
Posts: 40
Joined: Tue Feb 26, 2013 1:05 pm

Re: CCR1072 watchdog reboot

Wed Nov 28, 2018 9:39 am

At first upgrade your CCR to latest RouterOS and firmware version. It may help.
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Wed Nov 28, 2018 11:28 am

Upgrading will not help! 1009 is better choice then 1036.
 
bradnz
newbie
Posts: 27
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Thu Nov 29, 2018 5:55 am

I have upgraded to 6.43.4. I doubt it will help.

I actually wondered if this had something to do with the number of connections and the router not being able to handle it.

Either way, Im looking for an alternative hardware device now. I tried to change to the 1009 last night, but Im concerned about the performance, and also I use SFP+ that arent compatible with it. The interfaces never came up.

Incidentally, if I wanted to trial my theory of the number of connections does anyone know of tool I can use to do this?
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Thu Nov 29, 2018 11:16 am

RouterOS traffic-generator.

Don't be an optimist, because Mikrotik support claims "There are no hidden fixes"
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Fri Nov 30, 2018 8:24 pm

We are on 6.43.4 and the problem still exists. If you disable the watchdog timer, the router will hard lockup, all lights on and you will have to pull both power supplies.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Fri Nov 30, 2018 10:11 pm

I had to change the CCR1072 because I just couldn't find a solution. New hardware and everything solved, with the same .backup of the unit that failed.

Do you have the option to change the hardware and check if it fixes the problem?
(remember to do a reset-macs if you load the .backup)
 
JULIOLIMA
just joined
Posts: 1
Joined: Wed Dec 05, 2018 2:05 pm

Re: CCR1072 watchdog reboot

Wed Dec 05, 2018 2:20 pm

A few weeks ago we are going through the same problem, several restarts during the day, causing various disruptions, dissatisfaction and cancellations, we have replaced 1072 by another we have the settings in the zero hand, no procedure solved the problem, Mikrotik support so far, only speculation and no definitive solution ...
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Thu Dec 06, 2018 3:32 pm

There is a clear problem with the CCR 1072. Mikrotik we need refunds for these units or for the problem to be addressed. Please issue a statement here on the forum.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Thu Dec 06, 2018 5:46 pm

I use the 6.38.3 without problems, except high cpu for pppoe disconnections (no nat, no masquerade, no reason...)
Precisely I don't update for fear of what I read is happening...
 
guipoletto
Frequent Visitor
Frequent Visitor
Posts: 92
Joined: Mon Sep 19, 2011 5:31 am

Re: CCR1072 watchdog reboot

Thu Dec 06, 2018 9:13 pm

I use the 6.38.3 without problems, except high cpu for pppoe disconnections (no nat, no masquerade, no reason...)
Precisely I don't update for fear of what I read is happening...
That is probably Conntrack clearing the connections table after the PPPOE disconnections. it can be very disruptive indeed.
Are ALL your PPPOE connections NAT'ed?

> if you don't use NAT at all, you can disable Conntrack
> if you only need NAT for part of the connections, you can try to create "action=no-track" rules in the RAW table of the firewall, to bypass Conntrack.

As it's a bit off-topic, you should open a new topic if you want to dig further in the conntrack hole. :)
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Mon Dec 10, 2018 6:08 pm

Our CCR1072 has around 2Gbps going through it currently (about to go up to 4Gbps). Constant watchdog reboots, if I disable watchdog, I can only recover the CCR1072 by pulling power from both power supplies. Our setup is as follows:

6.42.6 is the firmware we are on.
We have 38 static routes (nothing dynamic)
No NAT rules.
No Queue rules.
No PPPoE
3 Firewall rules, 1 of which blocks winbox, ssh and telnet. 2 allow our internal subnets.
No Mangle rules.
2 SFP+'s only.
Less than 10% CPU usage.

We have a CCR1036-8G-2S+ and it does not reboot with the same configuration as we had it in place and running with an uptime of 372 days prior. I will be going back to our CCR1036 again in the middle of this month, a weekly lockup and reboot is uncalled for.
 
doush
Long time Member
Long time Member
Posts: 621
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 1:22 am

It is time for Mikrotik to seriously consider refunds of these CCR1072 units.
$3000+ for a router which can not hold even straight 3days of uptime is ridiculous.
 
User avatar
Murmaider
Member Candidate
Member Candidate
Posts: 121
Joined: Fri Oct 30, 2015 10:10 am

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 3:06 pm

Ours which had been running on version 6.38 for months and months without any issues, was upgraded to 6.42.9 a month ago and today it randomly rebooted with no clear reason why.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 3:39 pm

That confirms what I thought.... I will not move from the 6.38.3 that works perfect! In the 6.42.1 changed the firmware to improve performance for CCR1072, I don't know if it will have anything to do ...
 
doush
Long time Member
Long time Member
Posts: 621
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 3:55 pm

v6.38 is vulnerable.
We cant use that
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 4:35 pm

Closing 1072 well with IP/Services with subntest + ip firewall, is protected. If I have to choose between restarting it every day by watchdog or protecting it well manually....
 
doush
Long time Member
Long time Member
Posts: 621
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Wed Dec 12, 2018 1:57 pm

Would be good to have a special 6.38.x version from Mikrotik with security patches applied.
So at least we can try it out.
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Wed Dec 12, 2018 4:56 pm

Can we get a Mikrotik response please. The 1072 is ideal with the redundant PSU's but I cannot be rebooting it once or more a week.

At the very least like Doush said, give us a v6.38.xx that is patched. specifically for the CCR1072.
 
User avatar
Murmaider
Member Candidate
Member Candidate
Posts: 121
Joined: Fri Oct 30, 2015 10:10 am

Re: CCR1072 watchdog reboot

Thu Dec 13, 2018 12:54 pm

Can we get a Mikrotik response please. The 1072 is ideal with the redundant PSU's but I cannot be rebooting it once or more a week.

At the very least like Doush said, give us a v6.38.xx that is patched. specifically for the CCR1072.
I 2nd this, otherwise I'm just going to downgrade to 6.38.7 and firewall off the security issues.
 
ElviN
just joined
Posts: 4
Joined: Tue May 15, 2018 1:04 pm

Re: CCR1072 watchdog reboot

Mon Dec 17, 2018 11:45 pm

Hello!
People, don't upgrade to 6.43.7!!!!
After upgrading we getting freezing of device third time in the last two days...
 
rime
just joined
Posts: 7
Joined: Wed Jul 12, 2017 8:57 pm

Re: CCR1072 watchdog reboot

Tue Dec 18, 2018 1:49 am

Hi,
uptime: 13w2d23h16m46s
version: 6.43 (stable)
build-time: Sep/06/2018 12:44:56
free-memory: 14.8GiB
total-memory: 15.8GiB
cpu: tilegx
cpu-count: 72
cpu-frequency: 1000MHz
cpu-load: 2%
free-hdd-space: 76.5MiB
total-hdd-space: 128.0MiB
architecture-name: tile
board-name: CCR1072-1G-8S+
platform: MikroTik
No problem here.
 
Deywid
just joined
Posts: 6
Joined: Fri Dec 21, 2018 12:40 am

Re: CCR1072 watchdog reboot

Fri Dec 21, 2018 12:51 am

Unfortunately the CCR1072 has a chronic problem that has not been resolved until now.
The Mikortik team does not position or make any statement about the situation.
3 units in production CCR1072 with the same symptoms when doing BGP using IPV6 restarts sporadically every day by the watchdog
If the watchdog is not active, it freezes and only works after the two sources are turned on.
It does not matter the updated version or not.

We can cite the same with the nickname of (1072 of frozen)

Frozen kill

Infelizmente o CCR1072 tem um problema crônico que não foi resolvido até agora.
A equipe Mikortik não posiciona ou faz qualquer declaração sobre a situação.
3 unidades em produção CCR1072 com os mesmos sintomas ao fazer BGP usando IPV6 reinicia esporadicamente todos os dias pelo watchdog
Se o watchdog não estiver ativo, ela congela e só funciona após deligar as duas fontes.
Não importa a versão atualizada ou não.

Podemos citar o mesmo com apelido de (1072 do congelada)

Congeladas matar
 
matheusazevedo
just joined
Posts: 2
Joined: Fri Dec 21, 2018 3:48 pm

Re: CCR1072 watchdog reboot

Fri Dec 21, 2018 5:41 pm

Hello guys, I have the same problem here with a CCR1036-8G-2S+. Anybody else?
 
bradnz
newbie
Posts: 27
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Wed Jan 02, 2019 10:37 pm

I 2nd this, otherwise I'm just going to downgrade to 6.38.7 and firewall off the security issues.
Did you end up doing this and has it restarted since? Im about to do this. I had upgraded to 6.43.4 and it was fine for about 28 days, then it restarted twice in the space of 2 days. I simply cant turn off Watchdog to get the required log details - the data centre this is in is a 30mins drive to get there, and our customers wouldnt be happy at all with that amount of downtime.

If there is hope downgrading the FW will sort this out I would rather just do that.,
 
Deywid
just joined
Posts: 6
Joined: Fri Dec 21, 2018 12:40 am

Re: CCR1072 watchdog reboot

Sun Jan 06, 2019 7:48 pm

We are still waiting for mikrotik's official position on the CCR1072 freezes without any solution or information.

Complete wrapping of your best product, fix, worst product.
 
bradnz
newbie
Posts: 27
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Mon Jan 14, 2019 11:08 pm

I tried to downgrade to 6.38.3 and 6.38.7 and in both instances I had to recover the device from a boot loop using netinstall - the kernel wouldn't be found as it couldn't mount the drive. I have now upgraded to 6.43.8. Such a pain in the ass and I have now got really pissed off customers.

If this upgrade doesn't do anything to help, I'm probably going to start the process of getting a refund for the devices from our distributor and look at an alternative vendor. Its just ridiculous.
 
Deywid
just joined
Posts: 6
Joined: Fri Dec 21, 2018 12:40 am

Re: CCR1072 watchdog reboot

Tue Jan 15, 2019 7:19 pm

SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Tue Jan 15, 2019 11:13 pm

SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
So....you went from a $2,000 USD router to a $35,000 USD router?
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1818
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: CCR1072 watchdog reboot

Wed Jan 16, 2019 12:51 am

SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
So....you went from a $2,000 USD router to a $35,000 USD router?
A MX80 is not a $35000 router! Where the hell do you buy your gear from ?
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Wed Jan 16, 2019 11:16 am

CCR1072 price is 3050$!
 
Deywid
just joined
Posts: 6
Joined: Fri Dec 21, 2018 12:40 am

Re: CCR1072 watchdog reboot

Fri Jan 18, 2019 8:07 pm

That's not all.
Mx-80 = $ 6,700
CCR1072 = $ 3,000

When the plane is falling, paying double to survive is acceptable.
You do not have the required permissions to view the files attached to this post.
 
Kevo
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Wed Oct 12, 2011 1:38 am

Re: CCR1072 watchdog reboot

Fri Jan 18, 2019 11:38 pm

For those whose 1072 is rebooting are you running at 1200MHz or 1000MHz. We are running the long-term/bugfix version and haven't had any reboots or lock ups. It has always been set to 1000MHz. It came that way as far as I know.

I'm wondering what feature or setting makes the difference. We have 3 interfaces in use and run about 3Gbps through it during peak times.

We are only doing routing OSPF, BGP, and a bit of basic firewalling.
 
Deywid
just joined
Posts: 6
Joined: Fri Dec 21, 2018 12:40 am

Re: CCR1072 watchdog reboot

Sat Jan 19, 2019 1:26 pm

1000Mhz
BGP Com Ipv6 e Ipv4
OSPF
Firewall Basic
No full route
IBGP
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Mon Jan 21, 2019 1:44 pm

It not depends on features but traffic pattern and interface distribution.

Reboot requirements:
- connection-tracking activated
- clear routing between two interfaces
- 1G bidirectional traffic, 600B packets (each interface handles 1G in each direction)

If the packets are bigger:
- ~2,5G bidirectional traffic, 1500B packets

If the traffic is mostly unidirectional:
- ~2G unidirectional traffic, 600B packets
- ~5G unidirectional traffic, 1500B packets
 
Kevo
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Wed Oct 12, 2011 1:38 am

Re: CCR1072 watchdog reboot

Tue Jan 22, 2019 8:57 pm

I've just completed removing any connection tracking from ours. We had some DDoS issues and from what I've read connection tracking can be a big problem in that case. So I've rearranged my firewall rules and adapted things to run in the raw table as much as possible and turned off connection tracking. So far things seem good, but no real test to speak of yet. We really didn't need connection tracking on this router it was just not something I really considered much until the DDoS stuff started.

Hopefully this will keep us out of trouble until MT can resolve the issue completely with a future update or something.
 
doush
Long time Member
Long time Member
Posts: 621
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Tue Jan 22, 2019 10:50 pm

We have 6Gbit/s on it.
CT is on.
We have moved most of the rules to RAW so around 450Mbit of traffic is currently NATed and processed by connection tracking table.
Rebooted again with a very small dDOS attack yesterday.

MT doesnt accept that there is a problem. So nothing will be fixed.
Check the beta thread.
viewtopic.php?f=21&t=139057&start=300#p707452
 
fr0zonza
just joined
Posts: 1
Joined: Tue Jan 29, 2019 1:30 pm

Re: CCR1072 watchdog reboot

Tue Jan 29, 2019 1:59 pm

We have a similar issue except it affects ALL of my CCR routers (1009-1036) at different locations at the same time.
Interfaces flap at the exact same time and sometime stay down until i disable and enable. In extreme cases some routers lock up completely.
On my LHG radios i get
"12:55:41 interface,warning wlan60-1: bridge port received packet with own address
as source address (xxxxxxxxxx), probably loop"

Has been going on for months and still no help from support.
We run a fully routed network so no chance of any loops.
 
cabijuan
just joined
Posts: 5
Joined: Mon Mar 12, 2018 12:49 pm

Re: CCR1072 watchdog reboot

Tue Feb 19, 2019 10:31 am

I have 3 CCR 1072 and in all the same thing happens to me, I'm pissed with Mikrotik, it can not be that your top-of-the-line router will pass this to you and do not say anything about it.
 
bradnz
newbie
Posts: 27
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Mon Feb 25, 2019 3:03 am

I've just completed removing any connection tracking from ours. We had some DDoS issues and from what I've read connection tracking can be a big problem in that case. So I've rearranged my firewall rules and adapted things to run in the raw table as much as possible and turned off connection tracking. So far things seem good, but no real test to speak of yet. We really didn't need connection tracking on this router it was just not something I really considered much until the DDoS stuff started.
I have just removed all NAT and Mangle rules, which means that CT is actually not operating on the device at all. its behind a Transparent Fortigate Firewall, so shouldnt be an issue anyway. I have moved NAT and PAT services to a CISCO 3925 router now, and havent yet seen a reboot. I wondered if the CT turned off would have actually just been the answer rather than having to install another router for this purpose of NAT / PAT. A bit annoying. Do you think if I just turned off CT it would have been ok? How have you found things have been since you did this?
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Mon Feb 25, 2019 10:34 am

Yes, when connection-tracking is disabled the device performs much better.

When I contacted Mikrotik support about device reboots they requested access to device. I provided them a test setup with one CCR1072 to act as traffic-generator and another one acting as DUT. No configuration on DUT, just routing and 4 FW rules to prevent unauthorized access.

After a few weeks of meaningless tests, the answer was - DUT is overloaded, you can't see it but it is.

With 1G?????? That is not even close to your test results published on your web site?????!!!!

- Our tests are performed with connection-tracking disabled!

That's all! You are ******!

I will say it again, CCR1009, RB1100x4 and RB4011 are working as expected in absolutely same test condition in which CCR1072 reboots it self!!!
Last edited by kos on Mon Feb 25, 2019 3:22 pm, edited 1 time in total.
 
cabijuan
just joined
Posts: 5
Joined: Mon Mar 12, 2018 12:49 pm

Re: CCR1072 watchdog reboot

Mon Feb 25, 2019 10:38 am

Another restart this Saturday, is desperate, I have the router at 400km, please Mikrotik say something about it.
 
cdemers
Member Candidate
Member Candidate
Posts: 184
Joined: Sun Feb 26, 2006 3:32 pm
Location: Canada
Contact:

Re: CCR1072 watchdog reboot

Mon Feb 25, 2019 4:28 pm

Email mikrotik support, they don't monitor the user forums for people having problems. Normus and a few others frequent here, but best just to email them with a support ticket and supout file.


Sent from my SM-A520W using Tapatalk

 
cabijuan
just joined
Posts: 5
Joined: Mon Mar 12, 2018 12:49 pm

Re: CCR1072 watchdog reboot

Mon Feb 25, 2019 5:32 pm

I always send it, and the only thing they tell me is update, I already have 5 different versions. It does not help to send it, they have no idea what the restarting does.
 
djdrastic
Member
Member
Posts: 305
Joined: Wed Aug 01, 2012 2:14 pm

Re: CCR1072 watchdog reboot

Wed Mar 27, 2019 11:54 am

Reading this thread as I have to upgrade a 1016 due to lack of SFP+ ports . Has worked 1y without skipping a beat barring the security updates.
Am I better off getting a EdgeRouter Infinity or building a cheap X86 box ?
 
doush
Long time Member
Long time Member
Posts: 621
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Sat Mar 30, 2019 4:34 pm

Dont upgrade to CCR1072. It is not a stable product to work with.
Frequent reboots etc..
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Thu Apr 04, 2019 4:53 pm

SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
So....you went from a $2,000 USD router to a $35,000 USD router?
A MX80 is not a $35000 router! Where the hell do you buy your gear from ?
Where are you buying them from. Distributor list price all show $19,000+ USD. I have a friend who works for a larger ISP and they can get them for around $6,800 directly from Juniper.
 
wildbill442
Forum Guru
Forum Guru
Posts: 1050
Joined: Wed Dec 08, 2004 7:29 am
Location: Sacramento, CA

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 12:56 am

Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker
Unfortunately they are not! Neither in ROS 6.42.2 nor in any other ROS version!!!

CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled, due to watchdog timer reboots!!!

Five to ten times cheaper devices like CCR1009 and RB1100AHx4 are working fine in same test conditions!

The worst thing is that they are continuing selling this faulty device!

Contacting Mikrotik support is a lost of time!
Looks like I'm having a similar issue...

viewtopic.php?f=2&t=152192
William Burnett
Network Engineer
 
Fluke
just joined
Posts: 2
Joined: Wed Aug 02, 2017 1:46 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 6:45 am

Out of curiosity - what is the device temperature? (system health print)

I have a device that runs at ~53C and reboots occasionally, and another one that runs at ~39C and works OK.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 11:17 am

My CCR1072 works at 37-39
 
cabijuan
just joined
Posts: 5
Joined: Mon Mar 12, 2018 12:49 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 12:00 pm

forget about the temperature, it is a problem in the kernel that mikrotik does not know how to solve, nor does it pay attention to us. Hopefully with the V7 this is solved, it is the only hope I have left. the top-of-the-range router does this happen, it's nonsense
 
User avatar
mac86
Member Candidate
Member Candidate
Posts: 118
Joined: Sat Nov 25, 2006 12:52 am
Location: bahia blanca - argentina
Contact:

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 4:51 pm

We've downgraded CCR1072 to 6.44.5 and no more reboots. :D
http://www.netpro-ar.com
MTINE - MTCWE - MTCRE - MTCTCE - MTCNA
Patagonia Argentina IT Consultant

Mikrotik user from V2.7.7 - Aug/2003
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 6:08 pm

can anyone confirm that version 6.44.5 is stable with the CCR1072?
 
wildbill442
Forum Guru
Forum Guru
Posts: 1050
Joined: Wed Dec 08, 2004 7:29 am
Location: Sacramento, CA

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 6:21 pm

can anyone confirm that version 6.44.5 is stable with the CCR1072?
We’ve just downgraded to latest long term, I’ll update in 48hrs.
William Burnett
Network Engineer
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 6:36 pm

Please let us know how it works in a few days
 
cabijuan
just joined
Posts: 5
Joined: Mon Mar 12, 2018 12:49 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 7:31 pm

I have 6.43.8, and I have reboots.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 7:57 pm

Thanks, 6.43 discarded then... let's see if there's luck with 6.44.
I have the 6.38.3 and only had one reboot in 3 years
 
User avatar
mac86
Member Candidate
Member Candidate
Posts: 118
Joined: Sat Nov 25, 2006 12:52 am
Location: bahia blanca - argentina
Contact:

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 7:58 pm

can anyone confirm that version 6.44.5 is stable with the CCR1072?

Yes, I can confirm it.
4 weeks without reboots, and going on.....
http://www.netpro-ar.com
MTINE - MTCWE - MTCRE - MTCTCE - MTCNA
Patagonia Argentina IT Consultant

Mikrotik user from V2.7.7 - Aug/2003
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 8:16 pm

What reboot frequency did you have previously?
 
miltont
newbie
Posts: 48
Joined: Wed Nov 11, 2015 3:58 am

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 8:34 pm

Up 60+ Days
You do not have the required permissions to view the files attached to this post.
 
wildbill442
Forum Guru
Forum Guru
Posts: 1050
Joined: Wed Dec 08, 2004 7:29 am
Location: Sacramento, CA

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 10:49 pm

What reboot frequency did you have previously?
One to multiple times in a 24HR window.

Mikrotik support wanted me to turn off watchdog and log error / generate supout.rif once locked. The generated supout.rif files I provided after the watchdog reboot didn't yield any useful information. I opted to netinstall the latest long term release track (6.44.5). We also replaced all the SFP modules as a precaution during the maintenance window.

Mikrotik Support's Recommendation:
Before that, I suggest that you upgrade to the latest "stable" version (if there is an actual bug, then it might be already fixed). For example, v6.45.6 fixes a Watchdog reboot caused by h323 firewall helper. If your router did process voice call traffic, then the issue might be already resolved.

What's new in 6.45.6 (2019-Sep-10 09:06):
*) conntrack - improved system stability when using h323 helper (introduced in v6.45);


These are our edge routers and no NAT is being performed and we don't use h323 internally, but if some malformed h323 packet traversing these routers was causing the reboot then this may have been the cause behind the sporadic reboots. Again I'll post results of the downgrade to the Latest Long Term Release track after ample time has passed. So far ~15hrs and no reboot, this won't be conclusive until we make it past the 24-48hr mark. From what other users above have posted I'm feeling optimistic.
William Burnett
Network Engineer
 
wildbill442
Forum Guru
Forum Guru
Posts: 1050
Joined: Wed Dec 08, 2004 7:29 am
Location: Sacramento, CA

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 11:34 pm

Here's a sequence of events on how this issue started for us:

It began with a kernel failure on one of our edge routers, I'll call it EDGE A. To fix this we decided to do a netinstall on EDGE A and bring the router up to the latest stable release at the time (6.45.5) due to CVE fixes etc. Prior to the netinstall we were running 6.42.6 on both edge routers and other than the kernel failure on EDGE A we were not experiencing reboots. We also upgraded our other edge router, EDGE B, in the same maintenance window to 6.45.5. The reboots continued after the netinstall on EDGE A only the error changed from "reboot due to kernel failure" to "reboot by watchdog timer". At this time we were under the impression there was a hardware issue with the EDGE A router so we moved the BGP peer and other connections to EDGE B and overnighted new hardware. The reboots then started happening on EDGE B. We opened a ticket with Mikrotik support, but after going through the weekend and into Monday with no response and the routers rebooting sporadically we decided to netinstall the latest longterm release track (6.44.5). Mikrotik support responded after we were done with our downgrade and reaffirmed our suspicion that there was something wrong in software and to upgrade to latest stable release (6.45.6) due to a stability improvement relating to H323 and watchdog reboots. As we did not need any of the "new" features in the Stable Release tree, and receiving this information after downgrading to Long-Term Release tree, I think we're going to stay here unless the issue persists.

Again I won't know definitively if this was the root cause until ample time has passed, so I'll update after we get passed that 48hr mark.
William Burnett
Network Engineer
 
hytanium
Member Candidate
Member Candidate
Posts: 200
Joined: Thu Jan 18, 2007 9:10 pm

Re: CCR1072 watchdog reboot

Tue Oct 08, 2019 5:05 am

We are moving over 8Gbps on our 1072 at peak and have started seeing this random reboot watchdog error. We replaced with another 1072 thinking it was hardware related...once again, same random issue. Sometimes we go days without a reboot, sometimes a couple reboots in a day. It has been very disruptive.

I have disabled the h323 service as someone pointed out it may be part of the issue. It does seem to be load or random packet related. Any insight would be helpful. We are on 6.45.6.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 272
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Oct 08, 2019 11:41 am

Could be some random attack? Try to register cpu usage every second looking for high cpu usage

Who is online

Users browsing this forum: No registered users and 17 guests