Community discussions

 
ElviN
just joined
Posts: 4
Joined: Tue May 15, 2018 1:04 pm

Re: CCR1072 watchdog reboot

Mon Jul 30, 2018 11:34 am

Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker
No, device rebooted again with kernel error fail.
Problem not solved
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Thu Aug 09, 2018 9:50 pm

I have a CCR1072 with RouterOS 6.38.3 and it works perfectly. I wanted to update it to 6.42.6 but reading this thread I'm afraid...
Has anyone ever tested if the problems go away with 6.42.6?

Can anyone tell me if these problems exist only after version 6.38? (I need to know if the problem is determined by the RouterOS version or by the configuration, in which case I understand that I should already have problems)
 
tduchch
newbie
Posts: 26
Joined: Tue Aug 24, 2004 3:58 pm

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 5:16 am

These random reboots are becoming very annoying. Looking for a replacement solution, but have not found one with at least 3 SFP+ ports. We do a lot of MPLS and VPLS on this box, have not been able to use CHR on either VMWare or Hyper-V.

Recently upgraded to 6.42.9 Long Term, and still get random watchdog reboots. We have a serial logging device connected to the console port, and there is no output before the reboot. Of course, nothing in the logs.

Has anybody received any support from Mikrotik on this issue?
Any other hardware that someone has used in place of the 1072?

Terry
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 1:33 pm

Works fine for me with 6.38.3, can you use this version?
 
User avatar
strods
MikroTik Support
MikroTik Support
Posts: 1406
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 6:40 pm

There are multiple possible reasons why router is rebooting:

1) Hardware problem;
2) Software issue;
3) Overloaded device.

All of these three possible issues can be masked under log messages "rebooted due to Kernel Failure" , "rebooted due to Watchdog timeout", etc.

There is no reason to believe that issue, that you have on router, is the same issue as other user has just because log messages are equal. Only reason why you could believe that issue is the same is if problem appears for the first time at the same RouterOS version for multiple devices or you know how to trigger it.

In any other case contact MikroTik support. Only support has ability to see crashes and help you in order to diagnose issue and resolve the problem in its roots.

Multiple times we have seen reports "when you will fix Kernel Failure bug". This is equivalent with "when RouterOS will be a bug free version". There is no bug free software, but you can resolve your problem only by cooperating with MikroTik support. It is also the fastest way how to reach solution, after an upgrade to the latest version (if problem is already fixed by MikroTik).
 
User avatar
strods
MikroTik Support
MikroTik Support
Posts: 1406
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 6:46 pm

If you have CCR device that is crashing, then I recommend that you make sure that latest RouterBOOT version is installed on your router. v6.42 includes latest firmware for TILE devices which could help due to this fix:

!) tile - improved system performance and stability ("/system routerboard upgrade" required);
 
tduchch
newbie
Posts: 26
Joined: Tue Aug 24, 2004 3:58 pm

Re: CCR1072 watchdog reboot

Tue Oct 09, 2018 7:06 pm

Yes we are running latest long term version 6.42.9
We have had this problem in several versions.
We have swapped out different hardware
We have opened support ticket, were told the only way to trouble shoot this was to turn off watch dog and see what appears on the console port.
The router is a two hour drive away, and is a core router for all our network, letting it lock up is not an option.
As stated, we have a logging device on the console port and nothing appeared before the reboot.
The device can reboot during high traffic, or very low. No pattern exists.
CPU load at high traffic is around 25%, profile has 10% Firewall and 10% network.
CPU does not seem to spike before reboot.
High traffic is just approaching 2Gbps, should be no problem for this device.
Sometimes it runs for days, sometimes only for hours.

I will open another ticket with support, hopefully this can be figured out.
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Mon Oct 15, 2018 1:50 pm

Hi strods,

as I mentioned earlier, CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled.
To achieve reboot there is no no need of any other configuration, except two IP addresses, two routes and connection-tracking enabled.
 
DeCoen
just joined
Posts: 1
Joined: Tue Oct 03, 2017 11:33 am
Location: NL
Contact:

Re: CCR1072 watchdog reboot

Thu Nov 15, 2018 11:22 am

Hi strods,

as I mentioned earlier, CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled.
To achieve reboot there is no no need of any other configuration, except two IP addresses, two routes and connection-tracking enabled.
How often do you see a reboot? We have 10x 1072 and we are on 6.38.7, its reboots about once every few months, sometimes few weeks in between.
See screenshot, connection tracking is on, and with a load of more then 1 gig .
We are planning to upgrade to 6.42.9 , should we wait?

thanks,
You do not have the required permissions to view the files attached to this post.
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Mon Nov 19, 2018 10:35 am

Your traffic is distributed over two interfaces. That is why you are not experiencing more frequent reboots.

I am speaking about situation where two interface are loaded to 1G in every direction (1Gbps bidirectional forwarding). So 1Gbps entering sfp+1 and going out trough sfp+2, and in the same time 1Gbps is entering sfp+2 and going out trough sfp+1.
 
doush
Long time Member
Long time Member
Posts: 616
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Fri Nov 23, 2018 10:39 am

Frequent reboots EVERY DAY !
"Router was rebooted without proper shutdown by watchdog timer"

This issue is still not resolved and expect an answer or a possible reason from Mikrotik.

Mikrotik Support is pretty much less useless as of now.

As I say, it happens every EVERY DAY and mikrotik is silent about the issue. This behavior is not right.

CCR1072 is a useless device as it has this major fundamental flaw and should be avoided for anyone who are considering about using it in any deployment.

An reply from Mikrotik would be nice.
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Mon Nov 26, 2018 3:53 pm

We are having the same issue with the watchdog timer reboot.

disabling the watchdog causes the router to lock up until power is taken away. Full reboot with pulling power cables from both Power supplies.

We are looking for a solution to this. Our CCR1036 that this replaced did not have this issue. We are @ less than 10% load always, less than 2Gbps/300Mbps and only have static routes on this router as it is our core.
 
bradnz
newbie
Posts: 25
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Wed Nov 28, 2018 9:35 am

Hi all.

I have a CCR-1072 on 6.43.2 and this error was in the log "critical router was rebooted without proper shutdown by watchdog timer". I have sent the SUPOUT log to mikrotik and they are saying the following:

"For debugging you should turn off watchdog and test again.
"/system watchdog set watchdog-timer=no"

Connect now to your router over serial console, make sure that you have accessed RouterOS command line interface and leave console running.

Now router either:

1) might be stuck (freeze). If that happens, then you have to generate supout file on the router through serial console. Now you can reboot device.

2) might be stuck (freeze) and become unavailable over serial console. If that happens, then reboot router and then generate supout file.

3) will reboot. If that happens, then after reboot generate supout file.

Send supout file and full serial console output (within text file) to us for investigation."

This is a production router that I have already swapped for our cold spare in the hope it was just a power supply / hardware issue, but it would appear, based on this, that its not. I can let it just freeze because its in a data centre 20 mins from the office. 20mins of downtime would be unacceptable to our customers. What are your thoughts? Should I just replace it with a 1036, or a 1009? Are these any better? It runs BGP, only have about 300 BGP routes, runs at about 9% CPU most of the time, and has almost all its memory available. It just cant be that hard for this router to do surely.

Any suggestions would be appreciated.
 
User avatar
Lifz
newbie
Posts: 39
Joined: Tue Feb 26, 2013 1:05 pm

Re: CCR1072 watchdog reboot

Wed Nov 28, 2018 9:39 am

At first upgrade your CCR to latest RouterOS and firmware version. It may help.
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Wed Nov 28, 2018 11:28 am

Upgrading will not help! 1009 is better choice then 1036.
 
bradnz
newbie
Posts: 25
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Thu Nov 29, 2018 5:55 am

I have upgraded to 6.43.4. I doubt it will help.

I actually wondered if this had something to do with the number of connections and the router not being able to handle it.

Either way, Im looking for an alternative hardware device now. I tried to change to the 1009 last night, but Im concerned about the performance, and also I use SFP+ that arent compatible with it. The interfaces never came up.

Incidentally, if I wanted to trial my theory of the number of connections does anyone know of tool I can use to do this?
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Thu Nov 29, 2018 11:16 am

RouterOS traffic-generator.

Don't be an optimist, because Mikrotik support claims "There are no hidden fixes"
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Fri Nov 30, 2018 8:24 pm

We are on 6.43.4 and the problem still exists. If you disable the watchdog timer, the router will hard lockup, all lights on and you will have to pull both power supplies.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Fri Nov 30, 2018 10:11 pm

I had to change the CCR1072 because I just couldn't find a solution. New hardware and everything solved, with the same .backup of the unit that failed.

Do you have the option to change the hardware and check if it fixes the problem?
(remember to do a reset-macs if you load the .backup)
 
JULIOLIMA
just joined
Posts: 1
Joined: Wed Dec 05, 2018 2:05 pm

Re: CCR1072 watchdog reboot

Wed Dec 05, 2018 2:20 pm

A few weeks ago we are going through the same problem, several restarts during the day, causing various disruptions, dissatisfaction and cancellations, we have replaced 1072 by another we have the settings in the zero hand, no procedure solved the problem, Mikrotik support so far, only speculation and no definitive solution ...
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Thu Dec 06, 2018 3:32 pm

There is a clear problem with the CCR 1072. Mikrotik we need refunds for these units or for the problem to be addressed. Please issue a statement here on the forum.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Thu Dec 06, 2018 5:46 pm

I use the 6.38.3 without problems, except high cpu for pppoe disconnections (no nat, no masquerade, no reason...)
Precisely I don't update for fear of what I read is happening...
 
guipoletto
Frequent Visitor
Frequent Visitor
Posts: 92
Joined: Mon Sep 19, 2011 5:31 am

Re: CCR1072 watchdog reboot

Thu Dec 06, 2018 9:13 pm

I use the 6.38.3 without problems, except high cpu for pppoe disconnections (no nat, no masquerade, no reason...)
Precisely I don't update for fear of what I read is happening...
That is probably Conntrack clearing the connections table after the PPPOE disconnections. it can be very disruptive indeed.
Are ALL your PPPOE connections NAT'ed?

> if you don't use NAT at all, you can disable Conntrack
> if you only need NAT for part of the connections, you can try to create "action=no-track" rules in the RAW table of the firewall, to bypass Conntrack.

As it's a bit off-topic, you should open a new topic if you want to dig further in the conntrack hole. :)
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Mon Dec 10, 2018 6:08 pm

Our CCR1072 has around 2Gbps going through it currently (about to go up to 4Gbps). Constant watchdog reboots, if I disable watchdog, I can only recover the CCR1072 by pulling power from both power supplies. Our setup is as follows:

6.42.6 is the firmware we are on.
We have 38 static routes (nothing dynamic)
No NAT rules.
No Queue rules.
No PPPoE
3 Firewall rules, 1 of which blocks winbox, ssh and telnet. 2 allow our internal subnets.
No Mangle rules.
2 SFP+'s only.
Less than 10% CPU usage.

We have a CCR1036-8G-2S+ and it does not reboot with the same configuration as we had it in place and running with an uptime of 372 days prior. I will be going back to our CCR1036 again in the middle of this month, a weekly lockup and reboot is uncalled for.
 
doush
Long time Member
Long time Member
Posts: 616
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 1:22 am

It is time for Mikrotik to seriously consider refunds of these CCR1072 units.
$3000+ for a router which can not hold even straight 3days of uptime is ridiculous.
 
User avatar
Murmaider
Member Candidate
Member Candidate
Posts: 121
Joined: Fri Oct 30, 2015 10:10 am

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 3:06 pm

Ours which had been running on version 6.38 for months and months without any issues, was upgraded to 6.42.9 a month ago and today it randomly rebooted with no clear reason why.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 3:39 pm

That confirms what I thought.... I will not move from the 6.38.3 that works perfect! In the 6.42.1 changed the firmware to improve performance for CCR1072, I don't know if it will have anything to do ...
 
doush
Long time Member
Long time Member
Posts: 616
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 3:55 pm

v6.38 is vulnerable.
We cant use that
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Dec 11, 2018 4:35 pm

Closing 1072 well with IP/Services with subntest + ip firewall, is protected. If I have to choose between restarting it every day by watchdog or protecting it well manually....
 
doush
Long time Member
Long time Member
Posts: 616
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Wed Dec 12, 2018 1:57 pm

Would be good to have a special 6.38.x version from Mikrotik with security patches applied.
So at least we can try it out.
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Wed Dec 12, 2018 4:56 pm

Can we get a Mikrotik response please. The 1072 is ideal with the redundant PSU's but I cannot be rebooting it once or more a week.

At the very least like Doush said, give us a v6.38.xx that is patched. specifically for the CCR1072.
 
User avatar
Murmaider
Member Candidate
Member Candidate
Posts: 121
Joined: Fri Oct 30, 2015 10:10 am

Re: CCR1072 watchdog reboot

Thu Dec 13, 2018 12:54 pm

Can we get a Mikrotik response please. The 1072 is ideal with the redundant PSU's but I cannot be rebooting it once or more a week.

At the very least like Doush said, give us a v6.38.xx that is patched. specifically for the CCR1072.
I 2nd this, otherwise I'm just going to downgrade to 6.38.7 and firewall off the security issues.
 
ElviN
just joined
Posts: 4
Joined: Tue May 15, 2018 1:04 pm

Re: CCR1072 watchdog reboot

Mon Dec 17, 2018 11:45 pm

Hello!
People, don't upgrade to 6.43.7!!!!
After upgrading we getting freezing of device third time in the last two days...
 
rime
just joined
Posts: 7
Joined: Wed Jul 12, 2017 8:57 pm

Re: CCR1072 watchdog reboot

Tue Dec 18, 2018 1:49 am

Hi,
uptime: 13w2d23h16m46s
version: 6.43 (stable)
build-time: Sep/06/2018 12:44:56
free-memory: 14.8GiB
total-memory: 15.8GiB
cpu: tilegx
cpu-count: 72
cpu-frequency: 1000MHz
cpu-load: 2%
free-hdd-space: 76.5MiB
total-hdd-space: 128.0MiB
architecture-name: tile
board-name: CCR1072-1G-8S+
platform: MikroTik
No problem here.
 
Deywid
just joined
Posts: 6
Joined: Fri Dec 21, 2018 12:40 am

Re: CCR1072 watchdog reboot

Fri Dec 21, 2018 12:51 am

Unfortunately the CCR1072 has a chronic problem that has not been resolved until now.
The Mikortik team does not position or make any statement about the situation.
3 units in production CCR1072 with the same symptoms when doing BGP using IPV6 restarts sporadically every day by the watchdog
If the watchdog is not active, it freezes and only works after the two sources are turned on.
It does not matter the updated version or not.

We can cite the same with the nickname of (1072 of frozen)

Frozen kill

Infelizmente o CCR1072 tem um problema crônico que não foi resolvido até agora.
A equipe Mikortik não posiciona ou faz qualquer declaração sobre a situação.
3 unidades em produção CCR1072 com os mesmos sintomas ao fazer BGP usando IPV6 reinicia esporadicamente todos os dias pelo watchdog
Se o watchdog não estiver ativo, ela congela e só funciona após deligar as duas fontes.
Não importa a versão atualizada ou não.

Podemos citar o mesmo com apelido de (1072 do congelada)

Congeladas matar
 
matheusazevedo
just joined
Posts: 2
Joined: Fri Dec 21, 2018 3:48 pm

Re: CCR1072 watchdog reboot

Fri Dec 21, 2018 5:41 pm

Hello guys, I have the same problem here with a CCR1036-8G-2S+. Anybody else?
 
bradnz
newbie
Posts: 25
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Wed Jan 02, 2019 10:37 pm

I 2nd this, otherwise I'm just going to downgrade to 6.38.7 and firewall off the security issues.
Did you end up doing this and has it restarted since? Im about to do this. I had upgraded to 6.43.4 and it was fine for about 28 days, then it restarted twice in the space of 2 days. I simply cant turn off Watchdog to get the required log details - the data centre this is in is a 30mins drive to get there, and our customers wouldnt be happy at all with that amount of downtime.

If there is hope downgrading the FW will sort this out I would rather just do that.,
 
Deywid
just joined
Posts: 6
Joined: Fri Dec 21, 2018 12:40 am

Re: CCR1072 watchdog reboot

Sun Jan 06, 2019 7:48 pm

We are still waiting for mikrotik's official position on the CCR1072 freezes without any solution or information.

Complete wrapping of your best product, fix, worst product.
 
bradnz
newbie
Posts: 25
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Mon Jan 14, 2019 11:08 pm

I tried to downgrade to 6.38.3 and 6.38.7 and in both instances I had to recover the device from a boot loop using netinstall - the kernel wouldn't be found as it couldn't mount the drive. I have now upgraded to 6.43.8. Such a pain in the ass and I have now got really pissed off customers.

If this upgrade doesn't do anything to help, I'm probably going to start the process of getting a refund for the devices from our distributor and look at an alternative vendor. Its just ridiculous.
 
Deywid
just joined
Posts: 6
Joined: Fri Dec 21, 2018 12:40 am

Re: CCR1072 watchdog reboot

Tue Jan 15, 2019 7:19 pm

SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Tue Jan 15, 2019 11:13 pm

SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
So....you went from a $2,000 USD router to a $35,000 USD router?
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1815
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: CCR1072 watchdog reboot

Wed Jan 16, 2019 12:51 am

SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
So....you went from a $2,000 USD router to a $35,000 USD router?
A MX80 is not a $35000 router! Where the hell do you buy your gear from ?
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Wed Jan 16, 2019 11:16 am

CCR1072 price is 3050$!
 
Deywid
just joined
Posts: 6
Joined: Fri Dec 21, 2018 12:40 am

Re: CCR1072 watchdog reboot

Fri Jan 18, 2019 8:07 pm

That's not all.
Mx-80 = $ 6,700
CCR1072 = $ 3,000

When the plane is falling, paying double to survive is acceptable.
You do not have the required permissions to view the files attached to this post.
 
Kevo
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Wed Oct 12, 2011 1:38 am

Re: CCR1072 watchdog reboot

Fri Jan 18, 2019 11:38 pm

For those whose 1072 is rebooting are you running at 1200MHz or 1000MHz. We are running the long-term/bugfix version and haven't had any reboots or lock ups. It has always been set to 1000MHz. It came that way as far as I know.

I'm wondering what feature or setting makes the difference. We have 3 interfaces in use and run about 3Gbps through it during peak times.

We are only doing routing OSPF, BGP, and a bit of basic firewalling.
 
Deywid
just joined
Posts: 6
Joined: Fri Dec 21, 2018 12:40 am

Re: CCR1072 watchdog reboot

Sat Jan 19, 2019 1:26 pm

1000Mhz
BGP Com Ipv6 e Ipv4
OSPF
Firewall Basic
No full route
IBGP
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Mon Jan 21, 2019 1:44 pm

It not depends on features but traffic pattern and interface distribution.

Reboot requirements:
- connection-tracking activated
- clear routing between two interfaces
- 1G bidirectional traffic, 600B packets (each interface handles 1G in each direction)

If the packets are bigger:
- ~2,5G bidirectional traffic, 1500B packets

If the traffic is mostly unidirectional:
- ~2G unidirectional traffic, 600B packets
- ~5G unidirectional traffic, 1500B packets
 
Kevo
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Wed Oct 12, 2011 1:38 am

Re: CCR1072 watchdog reboot

Tue Jan 22, 2019 8:57 pm

I've just completed removing any connection tracking from ours. We had some DDoS issues and from what I've read connection tracking can be a big problem in that case. So I've rearranged my firewall rules and adapted things to run in the raw table as much as possible and turned off connection tracking. So far things seem good, but no real test to speak of yet. We really didn't need connection tracking on this router it was just not something I really considered much until the DDoS stuff started.

Hopefully this will keep us out of trouble until MT can resolve the issue completely with a future update or something.
 
doush
Long time Member
Long time Member
Posts: 616
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Tue Jan 22, 2019 10:50 pm

We have 6Gbit/s on it.
CT is on.
We have moved most of the rules to RAW so around 450Mbit of traffic is currently NATed and processed by connection tracking table.
Rebooted again with a very small dDOS attack yesterday.

MT doesnt accept that there is a problem. So nothing will be fixed.
Check the beta thread.
viewtopic.php?f=21&t=139057&start=300#p707452
 
fr0zonza
just joined
Posts: 1
Joined: Tue Jan 29, 2019 1:30 pm

Re: CCR1072 watchdog reboot

Tue Jan 29, 2019 1:59 pm

We have a similar issue except it affects ALL of my CCR routers (1009-1036) at different locations at the same time.
Interfaces flap at the exact same time and sometime stay down until i disable and enable. In extreme cases some routers lock up completely.
On my LHG radios i get
"12:55:41 interface,warning wlan60-1: bridge port received packet with own address
as source address (xxxxxxxxxx), probably loop"

Has been going on for months and still no help from support.
We run a fully routed network so no chance of any loops.
 
cabijuan
just joined
Posts: 5
Joined: Mon Mar 12, 2018 12:49 pm

Re: CCR1072 watchdog reboot

Tue Feb 19, 2019 10:31 am

I have 3 CCR 1072 and in all the same thing happens to me, I'm pissed with Mikrotik, it can not be that your top-of-the-line router will pass this to you and do not say anything about it.
 
bradnz
newbie
Posts: 25
Joined: Mon Jan 22, 2018 4:34 am

Re: CCR1072 watchdog reboot

Mon Feb 25, 2019 3:03 am

I've just completed removing any connection tracking from ours. We had some DDoS issues and from what I've read connection tracking can be a big problem in that case. So I've rearranged my firewall rules and adapted things to run in the raw table as much as possible and turned off connection tracking. So far things seem good, but no real test to speak of yet. We really didn't need connection tracking on this router it was just not something I really considered much until the DDoS stuff started.
I have just removed all NAT and Mangle rules, which means that CT is actually not operating on the device at all. its behind a Transparent Fortigate Firewall, so shouldnt be an issue anyway. I have moved NAT and PAT services to a CISCO 3925 router now, and havent yet seen a reboot. I wondered if the CT turned off would have actually just been the answer rather than having to install another router for this purpose of NAT / PAT. A bit annoying. Do you think if I just turned off CT it would have been ok? How have you found things have been since you did this?
 
kos
newbie
Posts: 25
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR1072 watchdog reboot

Mon Feb 25, 2019 10:34 am

Yes, when connection-tracking is disabled the device performs much better.

When I contacted Mikrotik support about device reboots they requested access to device. I provided them a test setup with one CCR1072 to act as traffic-generator and another one acting as DUT. No configuration on DUT, just routing and 4 FW rules to prevent unauthorized access.

After a few weeks of meaningless tests, the answer was - DUT is overloaded, you can't see it but it is.

With 1G?????? That is not even close to your test results published on your web site?????!!!!

- Our tests are performed with connection-tracking disabled!

That's all! You are ******!

I will say it again, CCR1009, RB1100x4 and RB4011 are working as expected in absolutely same test condition in which CCR1072 reboots it self!!!
Last edited by kos on Mon Feb 25, 2019 3:22 pm, edited 1 time in total.
 
cabijuan
just joined
Posts: 5
Joined: Mon Mar 12, 2018 12:49 pm

Re: CCR1072 watchdog reboot

Mon Feb 25, 2019 10:38 am

Another restart this Saturday, is desperate, I have the router at 400km, please Mikrotik say something about it.
 
cdemers
Member Candidate
Member Candidate
Posts: 184
Joined: Sun Feb 26, 2006 3:32 pm
Location: Canada
Contact:

Re: CCR1072 watchdog reboot

Mon Feb 25, 2019 4:28 pm

Email mikrotik support, they don't monitor the user forums for people having problems. Normus and a few others frequent here, but best just to email them with a support ticket and supout file.


Sent from my SM-A520W using Tapatalk

 
cabijuan
just joined
Posts: 5
Joined: Mon Mar 12, 2018 12:49 pm

Re: CCR1072 watchdog reboot

Mon Feb 25, 2019 5:32 pm

I always send it, and the only thing they tell me is update, I already have 5 different versions. It does not help to send it, they have no idea what the restarting does.
 
djdrastic
Member
Member
Posts: 305
Joined: Wed Aug 01, 2012 2:14 pm

Re: CCR1072 watchdog reboot

Wed Mar 27, 2019 11:54 am

Reading this thread as I have to upgrade a 1016 due to lack of SFP+ ports . Has worked 1y without skipping a beat barring the security updates.
Am I better off getting a EdgeRouter Infinity or building a cheap X86 box ?
 
doush
Long time Member
Long time Member
Posts: 616
Joined: Thu Jun 04, 2009 3:11 pm

Re: CCR1072 watchdog reboot

Sat Mar 30, 2019 4:34 pm

Dont upgrade to CCR1072. It is not a stable product to work with.
Frequent reboots etc..
 
whoknew
Member Candidate
Member Candidate
Posts: 145
Joined: Wed Oct 13, 2010 8:51 pm

Re: CCR1072 watchdog reboot

Thu Apr 04, 2019 4:53 pm

SOLVED

We get after a lot of work and trouble solving the problem of CCR1072 no longer restart or freeze.

Solved the problem was relatively easy, after more than 3 months waiting for the mikrotik team to position on the problem, solved.

We traded the CCR1072 for an MX-80 Juniper.
So....you went from a $2,000 USD router to a $35,000 USD router?
A MX80 is not a $35000 router! Where the hell do you buy your gear from ?
Where are you buying them from. Distributor list price all show $19,000+ USD. I have a friend who works for a larger ISP and they can get them for around $6,800 directly from Juniper.
 
wildbill442
Forum Guru
Forum Guru
Posts: 1050
Joined: Wed Dec 08, 2004 7:29 am
Location: Sacramento, CA

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 12:56 am

Hey, guys

After upgrading CCR1072 and CRS317-1G-16+ to version 6.42.2, the kernel crashes have stopped. I hope they have really solved this great problem.

Best Regards,

Vagner Felipe Becker
Unfortunately they are not! Neither in ROS 6.42.2 nor in any other ROS version!!!

CCR1072 is not capable to handle 1Gbps bidirectional IMIX traffic and 25k sessions with conection-tracking enabled, due to watchdog timer reboots!!!

Five to ten times cheaper devices like CCR1009 and RB1100AHx4 are working fine in same test conditions!

The worst thing is that they are continuing selling this faulty device!

Contacting Mikrotik support is a lost of time!
Looks like I'm having a similar issue...

viewtopic.php?f=2&t=152192
William Burnett
Network Engineer
 
Fluke
just joined
Posts: 2
Joined: Wed Aug 02, 2017 1:46 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 6:45 am

Out of curiosity - what is the device temperature? (system health print)

I have a device that runs at ~53C and reboots occasionally, and another one that runs at ~39C and works OK.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 11:17 am

My CCR1072 works at 37-39
 
cabijuan
just joined
Posts: 5
Joined: Mon Mar 12, 2018 12:49 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 12:00 pm

forget about the temperature, it is a problem in the kernel that mikrotik does not know how to solve, nor does it pay attention to us. Hopefully with the V7 this is solved, it is the only hope I have left. the top-of-the-range router does this happen, it's nonsense
 
User avatar
mac86
Member Candidate
Member Candidate
Posts: 118
Joined: Sat Nov 25, 2006 12:52 am
Location: bahia blanca - argentina
Contact:

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 4:51 pm

We've downgraded CCR1072 to 6.44.5 and no more reboots. :D
http://www.netpro-ar.com
MTINE - MTCWE - MTCRE - MTCTCE - MTCNA
Patagonia Argentina IT Consultant

Mikrotik user from V2.7.7 - Aug/2003
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 6:08 pm

can anyone confirm that version 6.44.5 is stable with the CCR1072?
 
wildbill442
Forum Guru
Forum Guru
Posts: 1050
Joined: Wed Dec 08, 2004 7:29 am
Location: Sacramento, CA

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 6:21 pm

can anyone confirm that version 6.44.5 is stable with the CCR1072?
We’ve just downgraded to latest long term, I’ll update in 48hrs.
William Burnett
Network Engineer
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 6:36 pm

Please let us know how it works in a few days
 
cabijuan
just joined
Posts: 5
Joined: Mon Mar 12, 2018 12:49 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 7:31 pm

I have 6.43.8, and I have reboots.
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 7:57 pm

Thanks, 6.43 discarded then... let's see if there's luck with 6.44.
I have the 6.38.3 and only had one reboot in 3 years
 
User avatar
mac86
Member Candidate
Member Candidate
Posts: 118
Joined: Sat Nov 25, 2006 12:52 am
Location: bahia blanca - argentina
Contact:

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 7:58 pm

can anyone confirm that version 6.44.5 is stable with the CCR1072?

Yes, I can confirm it.
4 weeks without reboots, and going on.....
http://www.netpro-ar.com
MTINE - MTCWE - MTCRE - MTCTCE - MTCNA
Patagonia Argentina IT Consultant

Mikrotik user from V2.7.7 - Aug/2003
 
Jeanluck
Member Candidate
Member Candidate
Posts: 271
Joined: Tue Apr 19, 2011 7:07 pm

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 8:16 pm

What reboot frequency did you have previously?
 
miltont
newbie
Posts: 48
Joined: Wed Nov 11, 2015 3:58 am

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 8:34 pm

Up 60+ Days
You do not have the required permissions to view the files attached to this post.
 
wildbill442
Forum Guru
Forum Guru
Posts: 1050
Joined: Wed Dec 08, 2004 7:29 am
Location: Sacramento, CA

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 10:49 pm

What reboot frequency did you have previously?
One to multiple times in a 24HR window.

Mikrotik support wanted me to turn off watchdog and log error / generate supout.rif once locked. The generated supout.rif files I provided after the watchdog reboot didn't yield any useful information. I opted to netinstall the latest long term release track (6.44.5). We also replaced all the SFP modules as a precaution during the maintenance window.

Mikrotik Support's Recommendation:
Before that, I suggest that you upgrade to the latest "stable" version (if there is an actual bug, then it might be already fixed). For example, v6.45.6 fixes a Watchdog reboot caused by h323 firewall helper. If your router did process voice call traffic, then the issue might be already resolved.

What's new in 6.45.6 (2019-Sep-10 09:06):
*) conntrack - improved system stability when using h323 helper (introduced in v6.45);


These are our edge routers and no NAT is being performed and we don't use h323 internally, but if some malformed h323 packet traversing these routers was causing the reboot then this may have been the cause behind the sporadic reboots. Again I'll post results of the downgrade to the Latest Long Term Release track after ample time has passed. So far ~15hrs and no reboot, this won't be conclusive until we make it past the 24-48hr mark. From what other users above have posted I'm feeling optimistic.
William Burnett
Network Engineer
 
wildbill442
Forum Guru
Forum Guru
Posts: 1050
Joined: Wed Dec 08, 2004 7:29 am
Location: Sacramento, CA

Re: CCR1072 watchdog reboot

Tue Sep 17, 2019 11:34 pm

Here's a sequence of events on how this issue started for us:

It began with a kernel failure on one of our edge routers, I'll call it EDGE A. To fix this we decided to do a netinstall on EDGE A and bring the router up to the latest stable release at the time (6.45.5) due to CVE fixes etc. Prior to the netinstall we were running 6.42.6 on both edge routers and other than the kernel failure on EDGE A we were not experiencing reboots. We also upgraded our other edge router, EDGE B, in the same maintenance window to 6.45.5. The reboots continued after the netinstall on EDGE A only the error changed from "reboot due to kernel failure" to "reboot by watchdog timer". At this time we were under the impression there was a hardware issue with the EDGE A router so we moved the BGP peer and other connections to EDGE B and overnighted new hardware. The reboots then started happening on EDGE B. We opened a ticket with Mikrotik support, but after going through the weekend and into Monday with no response and the routers rebooting sporadically we decided to netinstall the latest longterm release track (6.44.5). Mikrotik support responded after we were done with our downgrade and reaffirmed our suspicion that there was something wrong in software and to upgrade to latest stable release (6.45.6) due to a stability improvement relating to H323 and watchdog reboots. As we did not need any of the "new" features in the Stable Release tree, and receiving this information after downgrading to Long-Term Release tree, I think we're going to stay here unless the issue persists.

Again I won't know definitively if this was the root cause until ample time has passed, so I'll update after we get passed that 48hr mark.
William Burnett
Network Engineer

Who is online

Users browsing this forum: No registered users and 5 guests