Community discussions

MUM Europe 2020
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Possible fix for hAP ac2 rebooting randomly

Sun Nov 17, 2019 11:24 pm

Update! The issue has reappeared (after update to RouterOS 6.46/6.46.1) and procedure mentioned in this post didn't help.
A working solution (verified by several users on several devices) is removal of NTP package. More info in this post.

One of my hAP ac2 (RBD52G-5HacD2HnD-TC) routers was rebooting randomly (at random intervals varying from a few hours to around 20 days - on average a reboot every 2-7 days). The issue started soon after updating to RouterOS 6.43/6.43.1.
Router had quite complex configuration (including IPsec and several bridges).
Reinstalling router (/export + netinstall + running the exported configuration script in console) didn't help.
Changing power supply did not help. Changing router (changing hardware to a new hAP ac2 device) helped for couple of months, then reboots started to happen again.

As I suspected a thermal issue (overheating) I tried to underclock the router to 672 MHz by

Code: Select all

/system routerboard settings set cpu-frequency=672MHz
/system reboot

After underclocking the random reboots dissappeared. After around 30 days I restored the default clock speed of 716 MHz by

Code: Select all

/system routerboard settings set cpu-frequency=716MHz
/system reboot

Restoring the default clock speed did not cause reboots to reappear. Currently there have been no random reboots running at the default clock speed for 45 days.

Has anybody else faced random reboots of hAP ac2 (or possibly other ARM devices)?
I would be glad to know whether underclocking (+ rebooting) + restoring the default clock speed (+ rebooting again) would have helped for anybody else...

P.S. In case I would have any updates, I will post them here.
Last edited by Davis on Sun Feb 09, 2020 9:35 pm, edited 2 times in total.
 
Zacharias
Forum Guru
Forum Guru
Posts: 1433
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 12:51 pm

A reboot can happen in case the device for some reason stays unresponsive for a minute. This is due to watchdog.
So, since you say you have a complex setup, in case the CPU stays high and the device freezes then watchdog would reboot the device.
But that's an assumption.
Why dont you update to latest firmware and ROS ?
 
Kampfwurst
Frequent Visitor
Frequent Visitor
Posts: 89
Joined: Mon Mar 24, 2014 2:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 3:06 pm

I had the same.
viewtopic.php?f=3&t=153252&p=759046#p759046

I replaced it and for now it runns stable, (since over one week)

maybe there are some faulty prozessors in the hAP ac2.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 7:14 pm

Why dont you update to latest firmware and ROS ?
The router is always kept up to date.
To be more precise, when the reboots reappeared after the hardware swap they become more often - the maximum time between reboots went down to 11 days.
And 11 days after underclocking (and disappearing of the random reboots) I have installed RouterOS 6.45.6, so there is a chance that it was a coincidence during those 11 days and RouterOS 6.45.6 was the actual fix (however I don't think this is the case, because 6.45.6 changelog does not mention stability fixes for issues introduced before 6.45).

Majority of the time router has almost no load (0-10 Mbps traffic) and there are a few hours long traffic spikes, however there is no observed correlation between load and the reboots.
Reboots always happened with message "system;error;critical: router was rebooted without proper shutdown" and supout file was never created by the reboots.

I had the same.
viewtopic.php?f=3&t=153252&p=759046#p759046

I replaced it and for now it runns stable, (since over one week)
Very interesting! Symptoms sound familiar.
In case (after some weeks) your new router would start to randomly reboot as well, please try underclocking+rebooting+restoring the default clockspeed+rebooting again!
Currently I suspect that underclocking + restoring the default clockspeed (and rebooting after each change) is what fixed my router (currently I don't think running 30 days underclocked is what made the difference).
 
Zacharias
Forum Guru
Forum Guru
Posts: 1433
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 7:23 pm

I ve worked with hundreds hap ac and i had never a single problem like yours. Ofcorce this does not mean that your hap may not have a hardware problem.
However your problem does not happen every day so that makes it difficult for me to be hardware related.
I would netinstall the device and program it from the beginning again...
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 8:30 pm

I ve worked with hundreds hap ac and i had never a single problem like yours. Ofcorce this does not mean that your hap may not have a hardware problem.
However your problem does not happen every day so that makes it difficult for me to be hardware related.
I would netinstall the device and program it from the beginning again...
The router is hAP ac2 (hAP ac2 and hAP ac have different hardware).
As the OP states: Changing router (changing hardware to a new hAP ac2 device) helped for couple of months, then reboots started to happen again.
As the OP states: Reinstalling router (/export + netinstall + running the exported configuration script in console) didn't help.
The issue seems to be resolved now. As the OP states: Currently there have been no random reboots running at the default clock speed for 45 days.
 
Zacharias
Forum Guru
Forum Guru
Posts: 1433
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 10:35 pm

When a router reboots after 20 days, it is not a hardware problem to me.
It cant have a faulty hardware and work perfect for 20 days!
The issue started soon after updating to RouterOS 6.43/6.43.1
Also, since the problem started after update it is obvious it is not hardware related...
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 10:49 pm

When a router reboots after 20 days, it is not a hardware problem to me.
It cant have a faulty hardware and work perfect for 20 days!
The issue started soon after updating to RouterOS 6.43/6.43.1
Also, since the problem started after update it is obvious it is not hardware related...
Not all hardware issues are equally easy to troubleshoot. There are non-trivial things like single bit errors in memory and overheating...
As swapping hardware helped for some time (and netinstall didn't help at all) I like to assume that hardware could be one of the factors. And as underclocking + restoring the default clock speed (+ rebooting after both changes) has fixed the issue (or at least helped for some time) there might be some software factor involved as well.
 
Zacharias
Forum Guru
Forum Guru
Posts: 1433
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 11:06 pm

There are non-trivial things like single bit errors in memory
Memory errors and no supout file ?
Also, why would it overheat ? Unless it is out in the sun...
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 11:21 pm

Memory errors and no supout file ?
Also, why would it overheat ? Unless it is out in the sun...
In the same way I could say "Software issue and no supout file?"...
These are just guesses and thanks for your contribution!
The router is in 18-25 °C room, with no airflow obstructions (not in closet, not behind curtains etc.), and never gets direct sunlight.
 
Zacharias
Forum Guru
Forum Guru
Posts: 1433
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 11:58 pm

You say you changed the hap ac2 to a new one...
What are the possibilities the first and the second one as well are faulty ?
 
sindy
Forum Guru
Forum Guru
Posts: 4220
Joined: Mon Dec 04, 2017 9:19 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Nov 19, 2019 12:03 am

What are the possibilities the first and the second one as well are faulty ?
The chances are quite high if they come from the same production lot for which the same faulty lot of some chips was used (field experience, not with Mikrotik in particular).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Nov 19, 2019 12:10 am

You say you changed the hap ac2 to a new one...
What are the possibilities the first and the second one as well are faulty ?
Software-only scenario doesn't explain why swapping hardware helped for some time...
And I don't think this conversation is providing any new information on the topic (just consuming time of others who will read this topic).

The chances are quite high if they come from the same production lot for which the same faulty lot of some chips was used (field experience, not with Mikrotik in particular).
This shouldn't be the case as the new hAP ac2 seems to be manufactured several months later (had different amount of RAM and newer factory software version).
Last edited by Davis on Tue Nov 19, 2019 12:23 am, edited 1 time in total.
 
Zacharias
Forum Guru
Forum Guru
Posts: 1433
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Tue Nov 19, 2019 12:11 am

What are the possibilities the first and the second one as well are faulty ?
The chances are quite high if they come from the same production lot for which the same faulty lot of some chips was used (field experience, not with Mikrotik in particular).
Ok, now we made a whole production lot faulty.
And we also diagnosed that is particularly a faulty chip!
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sat Dec 14, 2019 7:10 pm

Another 27 days without reboots (in total 72 days running at default clock speed without reboots - more than after changing hardware).

I have also disproved my second hypothesis - that reboots might have been caused by "/interface bridge nat" rules. As I had made changes to "/interface bridge nat" rules together with underclocking there was some probability that "/interface bridge nat" rules were causing the reboots. To test this around 28 days ago I had reverted "/interface bridge nat" configuration to the state where it was before underclocking (during the time period where random reboots were happening). And I have not observed any reboots for 28 days (with the same "/interface bridge nat" rules that were present before underclocking).
Other configuration changes done together with underclocking (and during first 11 days after underclocking) were only to "/ip firewall filter" and "/ip firewall mangle" rules that were created after the reboot issue started (so I don't think changes to these rules might have affected the reboots).

To recap - the reboot issue has been resolved for around 72 days (running at default clockspeed).
Most likely the reboots were fixed by underclocking (+ rebooting) + restoring the default clock speed (+ rebooting again).
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sat Jan 04, 2020 10:41 am

Within a few days after installing RouterOS 6.46 similar random reboots started on another hAP ac2 router (with very similar configuration).
After some more days (and after installing 6.46.1) also the first router randomly rebooted, however the second router experienced the reboots more often.
Underclocking the second router to 672 MHz did not prevent the reboots.
So underclocking is not a solution (at least not with RouterOS 6.46/6.46.1, and the long period without reboots might have been mainly caused by "lucky" software versions).
 
gotsprings
Forum Veteran
Forum Veteran
Posts: 843
Joined: Mon May 14, 2012 9:30 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 05, 2020 5:07 am

Sure it's not a voltage thing or something else rebooting?

I have some pretty intricate settings on a few hAP AC2s that have had several months of uptimes. I never messed with the processor speed.

I always updated routerOS and firmware.

I have had some issues with cap radios on the running cap. But nothing that auto rebooted.
"It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so."
Mark Twain
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Jan 06, 2020 2:08 am

Sure it's not a voltage thing or something else rebooting?
Most likely it isn't a power interruption or something like that. Once the second router rebooted while I was in the room and ceiling light was on - I didn't see a glitch in the light. Also I have tried to change power adapter of the second router - it didn't help either.
The first router is near UPS (router itself is not connected via UPS) and there are no data about power fluctuations in UPS logs.

Are you using IPSec on your hAP ac2 (RBD52G-5HacD2HnD-TC) routers?
 
sindy
Forum Guru
Forum Guru
Posts: 4220
Joined: Mon Dec 04, 2017 9:19 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Jan 06, 2020 7:06 am

I had an issue like this a few weeks ago, but much less random, on one of my hAP ac2s. Whenever traffic was forwarded between a GRE over IPsec tunnel and a policy-based IPsec tunnel, both passing through the same WAN interface, the router was consistently restarting (although some of the traffic did pass through each time, so I assume it was a particular size or contents of a packet being forwarded that crashed it). The remedy was to replace the policy-based IPsec tunnel by a GRE over IPsec one; replacement of the power adaptor was the first thing to come on my mind but it didn't help.

As usually with configurations involving IPsec and 3rd party networks, I hesitate to send supout files to support, so I haven't reported the issue to Rīga as I haven't had time yet to reproduce it in the lab.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Jan 07, 2020 3:50 am

I had an issue like this a few weeks ago, but much less random, on one of my hAP ac2s. Whenever traffic was forwarded between a GRE over IPsec tunnel and a policy-based IPsec tunnel, both passing through the same WAN interface, the router was consistently restarting (although some of the traffic did pass through each time, so I assume it was a particular size or contents of a packet being forwarded that crashed it).
Could that actually be a race condition (instead of a particular traffic)?
Did the reboots occur some seconds after the connection establishment? Was the time between connection establishment and reboot varying notably (e.g. sometimes being 5 seconds and sometimes 30 seconds)?

What algorithms and PFS ground are you using? I am using AES-256(-CBC), SHA256 and modp4096.
 
Kampfwurst
Frequent Visitor
Frequent Visitor
Posts: 89
Joined: Mon Mar 24, 2014 2:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Jan 07, 2020 8:45 am

I think it has something to do with the cpu frequenz. I have two hAP ac2. One had the problem with the reboots. On this one i was playing arround with the cpu frequenz. I changed the power supply, installed it new with netinstall...

At the end I set up a new config with only ip and then i connected it to a RB1100x4 for testing with the Bandwith Test. After some time (mostly 5-10min) under load it startet to reboot.
In the config was only one IP over DHCP and the BW test. Nothing else.

I changed the device. Now i have a new hAP ac2 running for 30days. The second one is running now für 70days
 
aradicev
just joined
Posts: 4
Joined: Tue Apr 29, 2014 5:39 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 19, 2020 8:31 pm

I am experiencing very similar thing with mine hAP ac2 router. It reboots randomly within 24 hrs and it does it random number of times. It started somewhere begging of December when it rebooted each 2-4 days. The under clocking did not help, my router rebooted also when under clocked. I have also tried with changing of the AC adapter, but i didn't help either.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 19, 2020 8:57 pm

I am experiencing very similar thing with mine hAP ac2 router. It reboots randomly within 24 hrs and it does it random number of times. It started somewhere begging of December when it rebooted each 2-4 days. The under clocking did not help, my router rebooted also when under clocked. I have also tried with changing of the AC adapter, but i didn't help either.
Did it start after installing RouterOS 6.46?
Does the router have a simple (default or close to default) or complex configuration?
What features are being used (are you using IPSec, GRE/IPIP/EoIP tunnels, bridge filter/nat rules, MPLS etc.)?
What packages are installed (/system package print)? Is the NTP package installed?
 
rbfast
just joined
Posts: 10
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Jan 25, 2020 11:50 pm

Hi.For 4 months now I have hap ac2 and have been observing observing unexpected router reboots. On average, this happens a couple of times a week. Replacing the power supply did not solve the problem, as well as lower processor frequencies. I installed the simplest configuration a week ago after flashing and resetting the configuration with Netinstall, today I got a reboot again. Now I know for sure that this is a defective router, but replacing it under warranty is not so simple.
 
Zacharias
Forum Guru
Forum Guru
Posts: 1433
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 7:51 pm

Is ROS and firmware updated ?
 
rbfast
just joined
Posts: 10
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 9:02 pm

updated to 6.44.6
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 9:36 pm

updated to 6.44.6
Is there any reason why you are not using latest RouterOS (6.46.2)?

When I had the problem first time the hAP ac2 router I swapped by warranty did not reboot for 2 weeks when run in default configuration (and probably no network connected). But that was on March 2019 (with latest RouterOS then).

Currently I have removed NTP package and that seems to have helped in my case - I have not got any reboots for around 14 days on the router which since upgrading to 6.46 was rebooting every few days (maximum time between reboots was 7 days). I am still running RouterOS 6.46.1 as I don't want the tests to be affected by RouterOS upgrades (I am planning to upgrade later).
 
rbfast
just joined
Posts: 10
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 10:38 pm

Updated to 6.46.2. I do not have an ntp package installed. Earlier, I already tried hundreds of ways to find out what the problem is.
If you turn off watchdog, then the router just hangs. There are suspicions that the problem is in the defective memory, but I do not know how to check it.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 11:53 pm

Updated to 6.46.2. I do not have an ntp package installed. Earlier, I already tried hundreds of ways to find out what the problem is.
If you turn off watchdog, then the router just hangs. There are suspicions that the problem is in the defective memory, but I do not know how to check it.
If your router hangs/reboots with a fairly simple configuration, you should contact the distributor and try to get swapped by warranty (get RMA). It is better to provide a supout file generated during/after the reboot (if crash doesn't generate a supout file, then make a "/system scheduler" entry that will produce one on startup after the crash) or to make sure that distributor will be able to reproduce crash and create a supout file (AFAIK in many cases supout file is mandatory for RMA).
You can contact the distributor to get details about RMA process or you can create a support case with MikroTik, send them the supout file and get an advice from MikroTik whether you should contact distributor with RMA request.
P.S. Supout file might contain some sensitive information and it will contain "/export hide-sensitive" output.
 
mkx
Forum Guru
Forum Guru
Posts: 3745
Joined: Thu Mar 03, 2016 10:23 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 11:55 pm

If you turn off watchdog, then the router just hangs.

When router hangs, does it respond over wireless? Do you have any non-default config regarding switch chip?

I've had frequent hangs (cured by watchdog reboots) of my RBD52G with ROS 6.42 (factory default) and some 6.43 ... but I had some non-basic switch chip config (VLANs configured via /interface ethernet switch subtree) ... when I reconfigured that to use bridge vlan-filtering, hangs/reboots ceased.
I've never tried to connect via wireless while device was hung, I'm using it only as wired router.
BR,
Metod
 
rbfast
just joined
Posts: 10
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Jan 27, 2020 10:50 pm

Updated to 6.46.2. I do not have an ntp package installed. Earlier, I already tried hundreds of ways to find out what the problem is.
If you turn off watchdog, then the router just hangs. There are suspicions that the problem is in the defective memory, but I do not know how to check it.
If your router hangs/reboots with a fairly simple configuration, you should contact the distributor and try to get swapped by warranty (get RMA). It is better to provide a supout file generated during/after the reboot (if crash doesn't generate a supout file, then make a "/system scheduler" entry that will produce one on startup after the crash) or to make sure that distributor will be able to reproduce crash and create a supout file (AFAIK in many cases supout file is mandatory for RMA).
You can contact the distributor to get details about RMA process or you can create a support case with MikroTik, send them the supout file and get an advice from MikroTik whether you should contact distributor with RMA request.
P.S. Supout file might contain some sensitive information and it will contain "/export hide-sensitive" output.
Of course, supout.rif does not form during reboot. I see no reason to form it through Scheduler after a reboot. There will be nothing useful in this file. Too resolve the problem I decided to try forcibly rebooting the router through the sheduler once a day at night.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Jan 28, 2020 1:26 am

When router hangs, does it respond over wireless? Do you have any non-default config regarding switch chip?

I've had frequent hangs (cured by watchdog reboots) of my RBD52G with ROS 6.42 (factory default) and some 6.43 ...
Interestingly, I am using switch rules. However I never had the issue with 6.42, the issue initially started with 6.43 (without any changes in switch rules). Afterwards switch rules were changed, but I didn't see any notable correlation between switch rule changes and reboots aplearing/disappearing (there was only one possible correlation event and it was 7 days apart).

Of course, supout.rif does not form during reboot. I see no reason to form it through Scheduler after a reboot. There will be nothing useful in this file. Too resolve the problem I decided to try forcibly rebooting the router through the sheduler once a day at night.
I was asked to generate supout file for RMA after the reboot.
Android has last_kmsg for obtaining kernel crash information after reboot, maybe MikroTik has something similar (I don't know).
Are the intervals between two random reboots of your router always bigger than 24 hours?
 
Issarin
just joined
Posts: 5
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Jan 29, 2020 6:47 am

Hi, I'm not native english speaker but want to share my experience.

I have my hap ac2 for about 6 months. Before updating to 6.46, sometimes it reboots randomly but not so often, maybe weeks or almost a month between each reboot. Last week I updated to 6.46.2 (from 6.45.x) and it started to reboot very often, 2-3 times a day so I downgraded to 6.45.7 yesterday and it's over 24 hours now without any reboot.

I think my config is quite simple but with one IPSEC for site to site VPN and I also use NTP package, other than that is just normal setting.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Wed Jan 29, 2020 2:30 pm

I have my hap ac2 for about 6 months. Before updating to 6.46, sometimes it reboots randomly but not so often, maybe weeks or almost a month between each reboot. Last week I updated to 6.46.2 (from 6.45.x) and it started to reboot very often, 2-3 times a day so I downgraded to 6.45.7 yesterday and it's over 24 hours now without any reboot.

I think my config is quite simple but with one IPSEC for site to site VPN and I also use NTP package, other than that is just normal setting.
Thanks a lot for sharing this info!

Can you uninstall NTP package and check whether reboots stop happening?
Please note that NTP client (/system ntp client) is available also without the NTP package (NTP package is required only for NTP server).
 
Issarin
just joined
Posts: 5
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Jan 29, 2020 3:21 pm

Thanks a lot for sharing this info!

Can you uninstall NTP package and check whether reboots stop happening?
Please note that NTP client (/system ntp client) is available also without the NTP package (NTP package is required only for NTP server).
I plan to try that as well but first I want to see how long I can get with 6.45.7 plus NTP before it reboots again. Then I will uninstall NTP and may also try to upgrade to 6.46.2 to see if there is any difference. With this kind of problem, it takes so much time to try something and verify the result....

Anyway, I will post back if I get something.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Wed Jan 29, 2020 8:29 pm

I plan to try that as well but first I want to see how long I can get with 6.45.7 plus NTP before it reboots again. Then I will uninstall NTP and may also try to upgrade to 6.46.2 to see if there is any difference. With this kind of problem, it takes so much time to try something and verify the result....

Anyway, I will post back if I get something.
That will be an interesting test! For me 6.45.7 worked for 38 days without a single reboot (then I upgraded to 6.46).
I have no idea whether these 38 days without reboots are somehow related to the potential fix described in the OP.
 
rbfast
just joined
Posts: 10
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Jan 29, 2020 10:22 pm

It seems to me that we are using the wrong approach to solving the problem. We do not need to look for a way to stop accidental reboots, but rather make the router reboot every 5 minutes) So we will quickly find the reasons for the reboots. Let's remember under what conditions the router was rebooted most often.What can influence the behavior of the router from what we know? Firmware version, availability of NTP package, what else?
 
gowa
just joined
Posts: 6
Joined: Sun Dec 04, 2016 12:13 pm

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jan 30, 2020 12:32 am

yet another possible fix:
viewtopic.php?f=2&t=153666&p=771975#p771975

I blamed my USB drive and logging on it :)
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jan 30, 2020 2:29 am

It seems to me that we are using the wrong approach to solving the problem. We do not need to look for a way to stop accidental reboots, but rather make the router reboot every 5 minutes) So we will quickly find the reasons for the reboots. Let's remember under what conditions the router was rebooted most often.What can influence the behavior of the router from what we know? Firmware version, availability of NTP package, what else?
As the reboots did not start immediately after my hardware swap (described in OP) there are quite some chances that reboots are not easily reproducible on any hardware.

yet another possible fix:
viewtopic.php?f=2&t=153666&p=771975#p771975

I blamed my USB drive and logging on it :)
hAP ac and hAP ac2 are very different hardware.

Issarin, In case you would like to report this to MikroTik as a bug, I think before uninstalling NTP package you should upgrade to 6.46.2 (latest stable version), wait for a reboot and create supout file (and save it to PC). Supout file creation can also be scheduled on every boot (using /system scheduler - I can help with this if needed).
The supout file might contain some sensitive data (and it will contain "/export hide-sensitive") - this is one of the reasons why I am not very eager to generate it myself.
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 158
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jan 30, 2020 10:02 pm

Had one location with a hAP ac2 with the same symptoms.
The owner was using a lousy power strip and of course, without an UPS.

Changed the power strip to a more sturdy one and added an UPS.
No more problems since.
 
rbfast
just joined
Posts: 10
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jan 30, 2020 10:22 pm

By the way, before I installed the power filter, the router rebooted much more often. The filter is like this: https://www.apc.com/shop/ru/ru/products ... /P-PM1W-RS
Last edited by rbfast on Thu Jan 30, 2020 11:50 pm, edited 2 times in total.
 
rbfast
just joined
Posts: 10
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jan 30, 2020 11:46 pm

I found an easy way to cause a spontaneous restart of the router due to a kernel error caused by out of memory. As a result of such a reboot, the autosupout.rif file is not generated on the Hap AC2, and such a file is formed on the RB951Ui-2HnD.I can make a video on this subject if the public has interest
 
r00t
Member Candidate
Member Candidate
Posts: 278
Joined: Tue Nov 28, 2017 2:14 am

Re: Possible fix for hAP ac2 rebooting randomly

Fri Jan 31, 2020 2:49 am

What is the general state of this issue? Have it been officially confirmed to exist by Mikrotik? Or they are still playing the "can not reproduce" game?
Maybe best would be to give them one of the rebooting hAPs on MUM (negotiate exchange for a new one... this should be doable) so they can finally see it in action?

As for using power filters, if power input was really the culprit, it would make a difference running it on clean lab powersupply or just battery (no noise at all). But hAP uses switching DC-DC converter inside so noise or ripple on power input shouldn't matter that much.
I found an easy way to cause a spontaneous restart of the router due to a kernel error caused by out of memory.
I hope you reported it to support and it will be fixed, but I don't think this rebooting issue is related to RAM usage, graphing it doesn't reveal any fast filling of RAM before the reboot. Or maybe we have multiple issues here...
 
rbfast
just joined
Posts: 10
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 01, 2020 6:41 pm

You need to run the Bandwidth test at the same time as the Packet sniffer. In the Bandwidth test window in the TEST TO field, specify the address of the router itself, do not forget the username and password below. Run the packet sniffer tool with the limit set to 128,000 kb set in the Memory limit field. In 5-10 seconds, what I love so much now will happen. If this behavior is incorrect for the Mikrotik router, please contact technical support. I don’t want to do this myself. I do not believe that they will do anything.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 02, 2020 12:51 am

You need to run the Bandwidth test at the same time as the Packet sniffer. In the Bandwidth test window in the TEST TO field, specify the address of the router itself, do not forget the username and password below. Run the packet sniffer tool with the limit set to 128,000 kb set in the Memory limit field. In 5-10 seconds, what I love so much now will happen. If this behavior is incorrect for the Mikrotik router, please contact technical support. I don’t want to do this myself. I do not believe that they will do anything.
Why do you think this is related to the random reboots? :-o :?: This is not a rhetoric question!

hAP ac2 has 128 MB or 256 MB RAM (I guess your device probably has 128 MB RAM). If you run something that eats-up all the RAM (on device that has no swap space on disk) either the RAM-eating process will terminate or the device will crash/reboot.
I would be against restricting maximum size of the sniffer buffer, because either:
  • it would still be possible to exhaust the memory by combining several RAM hungry things (e.g. placing maximum amount of files on the "RAM disk" and running sniffer with maximum buffer size);
  • all the RAM limits would be unreasonably small (so they could not exhaust RAM if taken together) - this would be the worst thing - a massive reduction in functionality of the router with extremely small benefit;
  • some complex memory management/dynamic restrictions logic would need to be implemented (that would be notable development efforts with relatively small benefit for end-users).

If the sniffer buffer is in usermode process, then a-kind-of generic fix would be to implement out-of-memory killer (a Linux kernel feature that would kill the RAM-eating processes upon RAM exhaustion). That might be a good idea, however that might lead to situations where router starts to malfunction instead of rebooting and functioning normally (so this might actually be not so good idea afterall)...
I hope this gives you an answer why MikroTik might not be eager to fix issues like this one...
 
rbfast
just joined
Posts: 10
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 02, 2020 10:18 pm

Rebooted again today ((Rolled back to 6.43.13.
 
Issarin
just joined
Posts: 5
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 09, 2020 8:13 pm

Here is what I have gathered so far;
- 6.45.7 + NTP : 96 hours passed without reboot but I decided to stop and upgrade to 6.46.2
- 6.46.2 + NTP : I got random reboots with the interval of 36, 4, 15 and 10 hours
- 6.46.2 (NO NTP) : Now it's over 60 hours and going without reboot
The supout file might contain some sensitive data (and it will contain "/export hide-sensitive") - this is one of the reasons why I am not very eager to generate it myself.
Yes, also my concern.
Maybe, after a few more days, if there is still no reboot then I might go back to 6.46.2 + NTP again with minimal configuration to get Supout for Mikrotik.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 09, 2020 9:00 pm

Here is what I have gathered so far;
- 6.45.7 + NTP : 96 hours passed without reboot but I decided to stop and upgrade to 6.46.2
- 6.46.2 + NTP : I got random reboots with the interval of 36, 4, 15 and 10 hours
- 6.46.2 (NO NTP) : Now it's over 60 hours and going without reboot
Maybe, after a few more days, if there is still no reboot then I might go back to 6.46.2 + NTP again with minimal configuration to get Supout for Mikrotik.
Great testing results! Thanks a lot!
If you won't be able to reproduce the issue with minimal configuration, maybe IPSec is also required for the reboots to happen...
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 09, 2020 9:27 pm

My results after disabling NTP package:
  • First hAP ac2 router (that is described in OP) has had NTP package disabled for 3 weeks and had 0 random reboots.
  • Second hAP ac2 router (that started to reboot randomly on December 2019) has had NTP package disabled for 4 weeks and had only 1 random reboot (during very unusual 35 Mbps IPSec to wifi traffic; also power glitch cannot be excluded).

Also 3 other users on 5 (hAP ac2) devices have confirmed that removing NTP package has resolved the random reboots:
viewtopic.php?f=21&t=156825#p773118
viewtopic.php?f=21&t=156825#p773712
viewtopic.php?f=2&t=153988#p774170

So looks like NTP package is related to the problem. Also at least in some cases (on my routers) this seems to be related to RouterOS version - after installing RouterOS 6.46 the issue re-appeared on the first router (that previously had random reboots) and started to happen on the second router that had no random reboots previously.
To me it is still unclear:
  • How it is related to the hardware (why swapping hardware temporarily helped back in spring 2019). The old hardware was rebooting usually within 7 days, the swapped hardware worked for around 20 days before first RouterOS update (and around a month afterwards).
  • Whether/how/why the underclocking (described in OP) helped. The underclocked router worked 10 days without reboots before first RouterOS update.
  • Whether (in conjunction with NTP package) this is related to IPSec or some other features that are not enabled in default configuration. Traveller, DimaFIX Do you also have IPSec (or something else non-default) enabled on all problematic routers? Update! Traveller and DimaFIX have confirmed that all their problematic routers also have IPSec enabled. Looks like all the users facing this problem have IPSec enabled and NTP package installed.
Last edited by Davis on Tue Feb 11, 2020 12:57 am, edited 1 time in total.
 
Traveller
newbie
Posts: 28
Joined: Thu Apr 05, 2018 10:12 am

Re: Possible fix for hAP ac2 rebooting randomly

Mon Feb 10, 2020 5:02 pm

I have 4 hap ac2 routers in service. My two: the main router and the router as an access point and two strangers. All have ipsec l2tp vpn. I communicate with them through this vpn. Three routers were rebooted with the ntp package. Any router is not rebooted without an ntp package. The access point now stands with the beta version of the ROS 7 and does not rebooted. ntp server is built in to ROS 7. Install ROS 7 to the home main router going to frequent reboots. An access point with an enabled ntp server is already on the network for 62 days. My router and access point have been exchanged one by one. It did not give results. My conclusion: In addition to the ntp server, an even higher load on the router is required for rebooting.
 
User avatar
DimaFIX
just joined
Posts: 10
Joined: Wed Apr 18, 2018 7:46 pm
Location: Ukraine

Re: Possible fix for hAP ac2 rebooting randomly

Mon Feb 10, 2020 10:42 pm

Davis, Yes, I'm using IPSec in an L2TP tunnel on my router.
After removing the NTP packet, the reboots stopped for 2 days.
Then I installed this package again, turned off watchdog, and after 38 hours the router crashed without showing any signs of life.
Now I have enabled watchdog and removed the NTP package. Now the router works 12 hours without problems.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 12:53 am

Thanks! Looks like in all cases not only NTP package was installed, but also IPSec was enabled.
In my case it is not L2TP, but other protocols over IPSec.
 
rooted
just joined
Posts: 22
Joined: Tue Feb 04, 2020 5:58 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 4:59 am

What is the average load on the devices that are rebooting?
 
Reinis
MikroTik Support
MikroTik Support
Posts: 73
Joined: Wed Jan 02, 2019 12:14 pm
Location: Latvia
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 8:01 am

You will not ever be able to determine the cause of the reboot in the forum. If your router is rebooting by itself, then please contact support@mikrotik.com providing supout.rif file and any details on how to reproduce the issue. If there is a specific workaround, provide details about that too. Only MikroTik staff can determine the precise cause of the problem and if it is caused by the software, then we want to fix this problem as soon as possible. If you even manage to workaround a problem based on other user experience, then the issue is still there and others suffer from it.
Please understand that the current topic information lacks consistency and might be different issues with the same result, thus provide only your own test and troubleshooting results. All bug reports are very welcome.
 
Kampfwurst
Frequent Visitor
Frequent Visitor
Posts: 89
Joined: Mon Mar 24, 2014 2:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 8:56 am

if it helps to fine the bug.
On my hAP ac2 is version 6.45.7 with NTP since 69 days running without reboot.
I have also 3 IPSec connections running
 
User avatar
macgaiver
Forum Guru
Forum Guru
Posts: 1730
Joined: Wed May 18, 2005 5:57 pm
Location: Sol III, Sol system, Sector 001, Alpha Quadrant

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 9:52 am

What's new in 6.47beta32 (2020-Feb-10 11:45):

*) arm - improved watchdog and kernel panic reporting in log after reboots on RB3011 and IPQ4018/IPQ4019 devices ("/system routerboard upgrade" required);
I have the same issue, from my perspective it looks like issue is related to specific wireless client combination (right mix), that more likely causes this issue, usually it takes blend of IPhones/Androids/Chromecasts and smart scale to cause that issue :) on average once every week.

Now, after upgrading instead of just "rebooted without proper shutdown" i know in my case it is caused "by watchdog", now plan is to disable watchdog and allow it to crash/hang, and then manually reboot to check if something is in supout.rif file
With great knowledge comes great responsibility, because of ability to recognize id... incompetent people much faster.
 
User avatar
CZFan
Forum Guru
Forum Guru
Posts: 1578
Joined: Sun Oct 09, 2016 8:25 pm
Location: South Africa, Randburg
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 9:44 pm

Davis, Yes, I'm using IPSec in an L2TP tunnel on my router.
After removing the NTP packet, the reboots stopped for 2 days.
Then I installed this package again, turned off watchdog, and after 38 hours the router crashed without showing any signs of life.
Now I have enabled watchdog and removed the NTP package. Now the router works 12 hours without problems.

Did you check if you are not maybe being DDoSed on port 123 (NTP) causing the reboots?
MTCNA, MTCTCE, MTCRE & MTCINE
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Thu Feb 13, 2020 3:24 am

What is the average load on the devices that are rebooting?
In my case load on the rebooting devices was close to zero (well under 1 Mbps in many cases; below 10 Mbps in vast majority of cases).
The reboots did not seem to be correlated with load (reboots were not concentrated in peak-load time periods).

I have the same issue, from my perspective it looks like issue is related to specific wireless client combination (right mix), that more likely causes this issue, usually it takes blend of IPhones/Androids/Chromecasts and smart scale to cause that issue :) on average once every week.
Looks like this is another issue. In most cases one of my routers that was rebooting actually had 0 wifi clients connected.

Did you check if you are not maybe being DDoSed on port 123 (NTP) causing the reboots?
I am pretty sure I was not DoS-ed. In my case NTP was opened only from local network and during testing for RMA a router was connected to an isolated network.

You will not ever be able to determine the cause of the reboot in the forum. If your router is rebooting by itself, then please contact support@mikrotik.com providing supout.rif file and any details on how to reproduce the issue. If there is a specific workaround, provide details about that too. Only MikroTik staff can determine the precise cause of the problem and if it is caused by the software, then we want to fix this problem as soon as possible. If you even manage to workaround a problem based on other user experience, then the issue is still there and others suffer from it.
Please understand that the current topic information lacks consistency and might be different issues with the same result, thus provide only your own test and troubleshooting results. All bug reports are very welcome.
Thanks for looking into this! There is a recent support ticket regarding this issue from Traveller.

Most likely there is one common issue - random reboots on routers with IPSec and NTP package. Removal of NTP package has helped for 7 devices of 4 users (including myself).

According to my experience this issue might not be reliably reproducible on all hardware - for me one router was rebooting every few days, netinstall+running config script didn't help at all, however (after some time though) swapping hardware (and doing exactly the same netinstall + running config script on the new hardware) seemingly "resolved" the situation (I was able to connect the rebooting router to an isolated network in another place and continue to observe reboots there, but the new hardware had no random reboots). After a couple of months the issue appeared on the new hardware as well. I.e. importing the config of a problematic router on a test device might not be enough to reproduce the issue in lab.
However, assuming that the hardware susceptible to this issue is spread evenly among all the hardware (or that "wear and tear" of a router increases probability to have this issue) I would suggest the following:
  • Configure a pool of hAP ac2 routers with the config from Traveller's ticket.
  • Create the "other end" for all the IPSec tunnels. During my testing (on isolated network, during spring 2019, for RMA purpose) I had IPSec up and some traffic flowing from LAN side of the problematic router to the "other end" of IPSec (the other end just dropped all incoming IPSec encapsulated packets).
  • Ensure that test routers can reach the NTP server(s) configured. During my testing I had internet connectivity for the problematic router, but I did not have any NTP clients.
  • Hopefully some of the routers will reboot randomly...

I completely agree with "If you even manage to workaround a problem based on other user experience, then the issue is still there and others suffer from it.", also I would like to add that it is not known whether there can be less frequent random reboots (that would be considered to be isolated cases with unknown causes by the users) also without NTP package. Also it is not known how widespread is this issue (I would guess that many users who have random reboots+NTP package+IPSec on hAP ac2 actually have this issue).
Looks like the issue might (this is just a guess - the issue might have been introduced earlier) have started in 6.43 (my observation is 6.43/6.43.1, Traveller's observation is 6.43.15).

A few very technical points (that might serve as input for developers) I would like to make are:
  • most likely there have been no code changes in NTP package in recent versions of RouterOS, however there have been IPSec related changes
  • IPSec has some kernel code, NTP most likely doesn't, and kernel code has many more ways for crashing/locking-up a system
  • IPSec implementation might be sensitive to changes in system time (however I don't know whether MikroTik implementation counts the remaining time of SAs in kernel or user mode, I don't know how CLOCK_MONOTONIC and CLOCK_REALTIME are used etc.)
  • NTP client of NTP package and the built-in NTP client (I guess part of system package) might use different Linux APIs (adjtimex(), settimeofday() etc.) to set the time
  • logically unrelated changes in code can affect race conditions as memory layout changes can have notable effects on execution times of certain procedures (this could explain why the issue seemingly disappears in some versions and then comes back in later versions - of course this is just a guess)
 
Issarin
just joined
Posts: 5
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Thu Feb 13, 2020 6:43 am

What is the average load on the devices that are rebooting?
For my case, it's home use so most of the time it's just idling and in addition to that sometime the reboot happened in the late night which no one was using anything.

If you won't be able to reproduce the issue with minimal configuration, maybe IPSec is also required for the reboots to happen...
I just tried 6.46.2 + NTP again (I did keep IPSec) and it rebooted after 28 hours. Already sent Mikrotik the Supout file.
 
Traveller
newbie
Posts: 28
Joined: Thu Apr 05, 2018 10:12 am

Re: Possible fix for hAP ac2 rebooting randomly

Thu Feb 13, 2020 9:08 am

There is a recent support ticket regarding this issue from Traveller.
The answer was expected:
Unfortunately, there is no information about the problem in the supout file.
But we have made movement on this issue:
However, yesterday we released a new beta version with improvements regarding a problem troubleshooting exactly for the hAP ac^2 units.
I will not install beta versions on working responsible equipment. But perhaps someone this problem will begin to be fixed.
 
rooted
just joined
Posts: 22
Joined: Tue Feb 04, 2020 5:58 pm

Re: Possible fix for hAP ac2 rebooting randomly

Fri Feb 14, 2020 2:52 am

I have a mixture of 32+ devices connecting to the hAP ac² which include 3 Chromecast, many Android devices, and some iOS and haven't suffered a reboot.

I'm going to enable NTP and see if anything changes, but being I've never had a reboot NTP may be a determining factor.

 
gurnec
just joined
Posts: 6
Joined: Wed Jul 14, 2010 9:42 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 15, 2020 2:28 am

If it's of any use, I'm including a log of reboots and upgrades from a single AC2 that's been running for about 10 months where I have with the NTP package installed, IPSec enabled (but very rarely used), and watchdog enabled. Although there have been many config changes over time, the ones I just mentioned have been static since the beginning.
Installed Apr/12/1019
   Now Running 6.44.2
apr/26/2019 14:19:12 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.44.3
jul/01/2019 16:42:29 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.45.1
jul/22/2019 19:35:44 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.45.2
aug/05/2019 18:21:46 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.45.3
aug/18/2019 06:22:49 system,error,critical router was rebooted without proper shutdown
sep/06/2019 21:47:37 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.45.5
oct/10/2019 22:38:26 system,error,critical router was rebooted without proper shutdown
oct/12/2019 16:48:37 system,error,critical router was rebooted without proper shutdown
oct/26/2019 04:28:40 system,error,critical router was rebooted without proper shutdown
oct/27/2019 04:27:37 system,error,critical router was rebooted without proper shutdown
oct/28/2019 20:06:15 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.45.7
nov/18/2019 01:00:03 system,error,critical router was rebooted without proper shutdown
nov/18/2019 07:37:49 system,error,critical router was rebooted without proper shutdown
nov/29/2019 01:09:46 system,error,critical router was rebooted without proper shutdown
dec/03/2019 18:09:15 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.46
dec/08/2019 10:19:04 system,error,critical router was rebooted without proper shutdown
dec/09/2019 19:14:04 system,error,critical router was rebooted without proper shutdown
dec/12/2019 00:22:00 system,error,critical router was rebooted without proper shutdown
dec/12/2019 06:37:05 system,error,critical router was rebooted without proper shutdown
dec/14/2019 00:12:33 system,error,critical router was rebooted without proper shutdown
dec/14/2019 05:04:54 system,error,critical router was rebooted without proper shutdown
dec/17/2019 16:58:58 system,error,critical router was rebooted without proper shutdown
dec/20/2019 23:28:53 system,error,critical router was rebooted without proper shutdown
jan/05/2020 09:27:50 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.46.1
jan/06/2020 15:26:26 system,error,critical router was rebooted without proper shutdown
jan/07/2020 22:27:58 system,error,critical router was rebooted without proper shutdown
jan/17/2020 20:37:36 system,error,critical router was rebooted without proper shutdown
jan/18/2020 16:15:49 system,error,critical router was rebooted without proper shutdown
jan/20/2020 14:32:13 system,error,critical router was rebooted without proper shutdown
jan/21/2020 13:30:42 system,error,critical router was rebooted without proper shutdown
jan/25/2020 22:26:18 system,error,critical router was rebooted without proper shutdown
jan/30/2020 22:49:14 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.46.2
jan/31/2020 07:46:11 system,error,critical router was rebooted without proper shutdown
feb/03/2020 15:58:29 system,error,critical router was rebooted without proper shutdown
feb/04/2020 16:25:24 system,error,critical router was rebooted without proper shutdown
feb/06/2020 17:59:23 system,error,critical router was rebooted without proper shutdown
feb/07/2020 13:17:24 system,error,critical router was rebooted without proper shutdown
feb/07/2020 16:13:36 system,error,critical router was rebooted without proper shutdown
feb/10/2020 07:58:24 system,error,critical router was rebooted without proper shutdown
feb/11/2020 14:42:14 system,error,critical router was rebooted without proper shutdown
feb/11/2020 20:21:46 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
  Now Running 6.46.3 and NTP Package Disabled
For this device, the reboots started somewhere around the 6.45.3 - 6.45.5 timeframe. I disabled the NTP package (still have the NTP client running) about 3 days ago, and so far no reboots. (I also upgraded to a newer firmware... perhaps not the smartest move to make two changes at once I admit.)

If I do see a reboot, I'll be sure to report back.
 
Issarin
just joined
Posts: 5
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 15, 2020 1:56 pm

Just an update.

Mikrotik didn't find any problem from my supout and they asked me to try 6.47beta32.
I already installed that along with the NTP package and has been running for two days. I will update again after a few more days.
 
rbfast
just joined
Posts: 10
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 15, 2020 6:23 pm

Hello. I requested help from Mikrotik a week ago. They watched the router for a week. During all this time there has not been a single spontaneous reboot. The router has been updated to 6.47beta32. Maybe it helped.
 
User avatar
Davis
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 16, 2020 3:19 am

I have a mixture of 32+ devices connecting to the hAP ac² which include 3 Chromecast, many Android devices, and some iOS and haven't suffered a reboot.

I'm going to enable NTP and see if anything changes, but being I've never had a reboot NTP may be a determining factor.

Looks like IPSec+NTP package (with NTP client configured) might be the combination that causes the random reboots...
And thanks for experimenting/helping!

Issarin, Thanks for the update!
 
rooted
just joined
Posts: 22
Joined: Tue Feb 04, 2020 5:58 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 16, 2020 5:40 am

Looks like IPSec+NTP package (with NTP client configured) might be the combination that causes the random reboots...
And thanks for experimenting/helping!

Issarin, Thanks for the update!
Yeah it's been 2 days with NTP enabled and nothing has gone awry, I don't have IPSec enabled but I will try to find a server and enable it also.

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 143 guests