Community discussions

MikroTik App
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Possible fix for hAP ac2 rebooting randomly

Sun Nov 17, 2019 11:24 pm

Update! The issue has reappeared (after update to RouterOS 6.46/6.46.1) and procedure mentioned in this post didn't help.
A working solution (verified by several users on several devices) is removal of NTP package. More info in this post.

One of my hAP ac2 (RBD52G-5HacD2HnD-TC) routers was rebooting randomly (at random intervals varying from a few hours to around 20 days - on average a reboot every 2-7 days). The issue started soon after updating to RouterOS 6.43/6.43.1.
Router had quite complex configuration (including IPsec and several bridges).
Reinstalling router (/export + netinstall + running the exported configuration script in console) didn't help.
Changing power supply did not help. Changing router (changing hardware to a new hAP ac2 device) helped for couple of months, then reboots started to happen again.

As I suspected a thermal issue (overheating) I tried to underclock the router to 672 MHz by

Code: Select all

/system routerboard settings set cpu-frequency=672MHz
/system reboot

After underclocking the random reboots dissappeared. After around 30 days I restored the default clock speed of 716 MHz by

Code: Select all

/system routerboard settings set cpu-frequency=716MHz
/system reboot

Restoring the default clock speed did not cause reboots to reappear. Currently there have been no random reboots running at the default clock speed for 45 days.

Has anybody else faced random reboots of hAP ac2 (or possibly other ARM devices)?
I would be glad to know whether underclocking (+ rebooting) + restoring the default clock speed (+ rebooting again) would have helped for anybody else...

P.S. In case I would have any updates, I will post them here.
Last edited by Davis on Sun Feb 09, 2020 9:35 pm, edited 2 times in total.
 
Zacharias
Forum Guru
Forum Guru
Posts: 3459
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 12:51 pm

A reboot can happen in case the device for some reason stays unresponsive for a minute. This is due to watchdog.
So, since you say you have a complex setup, in case the CPU stays high and the device freezes then watchdog would reboot the device.
But that's an assumption.
Why dont you update to latest firmware and ROS ?
 
Kampfwurst
Member Candidate
Member Candidate
Posts: 107
Joined: Mon Mar 24, 2014 2:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 3:06 pm

I had the same.
viewtopic.php?f=3&t=153252&p=759046#p759046

I replaced it and for now it runns stable, (since over one week)

maybe there are some faulty prozessors in the hAP ac2.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 7:14 pm

Why dont you update to latest firmware and ROS ?
The router is always kept up to date.
To be more precise, when the reboots reappeared after the hardware swap they become more often - the maximum time between reboots went down to 11 days.
And 11 days after underclocking (and disappearing of the random reboots) I have installed RouterOS 6.45.6, so there is a chance that it was a coincidence during those 11 days and RouterOS 6.45.6 was the actual fix (however I don't think this is the case, because 6.45.6 changelog does not mention stability fixes for issues introduced before 6.45).

Majority of the time router has almost no load (0-10 Mbps traffic) and there are a few hours long traffic spikes, however there is no observed correlation between load and the reboots.
Reboots always happened with message "system;error;critical: router was rebooted without proper shutdown" and supout file was never created by the reboots.

I had the same.
viewtopic.php?f=3&t=153252&p=759046#p759046

I replaced it and for now it runns stable, (since over one week)
Very interesting! Symptoms sound familiar.
In case (after some weeks) your new router would start to randomly reboot as well, please try underclocking+rebooting+restoring the default clockspeed+rebooting again!
Currently I suspect that underclocking + restoring the default clockspeed (and rebooting after each change) is what fixed my router (currently I don't think running 30 days underclocked is what made the difference).
 
Zacharias
Forum Guru
Forum Guru
Posts: 3459
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 7:23 pm

I ve worked with hundreds hap ac and i had never a single problem like yours. Ofcorce this does not mean that your hap may not have a hardware problem.
However your problem does not happen every day so that makes it difficult for me to be hardware related.
I would netinstall the device and program it from the beginning again...
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 8:30 pm

I ve worked with hundreds hap ac and i had never a single problem like yours. Ofcorce this does not mean that your hap may not have a hardware problem.
However your problem does not happen every day so that makes it difficult for me to be hardware related.
I would netinstall the device and program it from the beginning again...
The router is hAP ac2 (hAP ac2 and hAP ac have different hardware).
As the OP states: Changing router (changing hardware to a new hAP ac2 device) helped for couple of months, then reboots started to happen again.
As the OP states: Reinstalling router (/export + netinstall + running the exported configuration script in console) didn't help.
The issue seems to be resolved now. As the OP states: Currently there have been no random reboots running at the default clock speed for 45 days.
 
Zacharias
Forum Guru
Forum Guru
Posts: 3459
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 10:35 pm

When a router reboots after 20 days, it is not a hardware problem to me.
It cant have a faulty hardware and work perfect for 20 days!
The issue started soon after updating to RouterOS 6.43/6.43.1
Also, since the problem started after update it is obvious it is not hardware related...
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 10:49 pm

When a router reboots after 20 days, it is not a hardware problem to me.
It cant have a faulty hardware and work perfect for 20 days!
The issue started soon after updating to RouterOS 6.43/6.43.1
Also, since the problem started after update it is obvious it is not hardware related...
Not all hardware issues are equally easy to troubleshoot. There are non-trivial things like single bit errors in memory and overheating...
As swapping hardware helped for some time (and netinstall didn't help at all) I like to assume that hardware could be one of the factors. And as underclocking + restoring the default clock speed (+ rebooting after both changes) has fixed the issue (or at least helped for some time) there might be some software factor involved as well.
 
Zacharias
Forum Guru
Forum Guru
Posts: 3459
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 11:06 pm

There are non-trivial things like single bit errors in memory
Memory errors and no supout file ?
Also, why would it overheat ? Unless it is out in the sun...
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 11:21 pm

Memory errors and no supout file ?
Also, why would it overheat ? Unless it is out in the sun...
In the same way I could say "Software issue and no supout file?"...
These are just guesses and thanks for your contribution!
The router is in 18-25 °C room, with no airflow obstructions (not in closet, not behind curtains etc.), and never gets direct sunlight.
 
Zacharias
Forum Guru
Forum Guru
Posts: 3459
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Mon Nov 18, 2019 11:58 pm

You say you changed the hap ac2 to a new one...
What are the possibilities the first and the second one as well are faulty ?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Nov 19, 2019 12:03 am

What are the possibilities the first and the second one as well are faulty ?
The chances are quite high if they come from the same production lot for which the same faulty lot of some chips was used (field experience, not with Mikrotik in particular).
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Nov 19, 2019 12:10 am

You say you changed the hap ac2 to a new one...
What are the possibilities the first and the second one as well are faulty ?
Software-only scenario doesn't explain why swapping hardware helped for some time...
And I don't think this conversation is providing any new information on the topic (just consuming time of others who will read this topic).

The chances are quite high if they come from the same production lot for which the same faulty lot of some chips was used (field experience, not with Mikrotik in particular).
This shouldn't be the case as the new hAP ac2 seems to be manufactured several months later (had different amount of RAM and newer factory software version).
Last edited by Davis on Tue Nov 19, 2019 12:23 am, edited 1 time in total.
 
Zacharias
Forum Guru
Forum Guru
Posts: 3459
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Tue Nov 19, 2019 12:11 am

What are the possibilities the first and the second one as well are faulty ?
The chances are quite high if they come from the same production lot for which the same faulty lot of some chips was used (field experience, not with Mikrotik in particular).
Ok, now we made a whole production lot faulty.
And we also diagnosed that is particularly a faulty chip!
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sat Dec 14, 2019 7:10 pm

Another 27 days without reboots (in total 72 days running at default clock speed without reboots - more than after changing hardware).

I have also disproved my second hypothesis - that reboots might have been caused by "/interface bridge nat" rules. As I had made changes to "/interface bridge nat" rules together with underclocking there was some probability that "/interface bridge nat" rules were causing the reboots. To test this around 28 days ago I had reverted "/interface bridge nat" configuration to the state where it was before underclocking (during the time period where random reboots were happening). And I have not observed any reboots for 28 days (with the same "/interface bridge nat" rules that were present before underclocking).
Other configuration changes done together with underclocking (and during first 11 days after underclocking) were only to "/ip firewall filter" and "/ip firewall mangle" rules that were created after the reboot issue started (so I don't think changes to these rules might have affected the reboots).

To recap - the reboot issue has been resolved for around 72 days (running at default clockspeed).
Most likely the reboots were fixed by underclocking (+ rebooting) + restoring the default clock speed (+ rebooting again).
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sat Jan 04, 2020 10:41 am

Within a few days after installing RouterOS 6.46 similar random reboots started on another hAP ac2 router (with very similar configuration).
After some more days (and after installing 6.46.1) also the first router randomly rebooted, however the second router experienced the reboots more often.
Underclocking the second router to 672 MHz did not prevent the reboots.
So underclocking is not a solution (at least not with RouterOS 6.46/6.46.1, and the long period without reboots might have been mainly caused by "lucky" software versions).
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 05, 2020 5:07 am

Sure it's not a voltage thing or something else rebooting?

I have some pretty intricate settings on a few hAP AC2s that have had several months of uptimes. I never messed with the processor speed.

I always updated routerOS and firmware.

I have had some issues with cap radios on the running cap. But nothing that auto rebooted.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Jan 06, 2020 2:08 am

Sure it's not a voltage thing or something else rebooting?
Most likely it isn't a power interruption or something like that. Once the second router rebooted while I was in the room and ceiling light was on - I didn't see a glitch in the light. Also I have tried to change power adapter of the second router - it didn't help either.
The first router is near UPS (router itself is not connected via UPS) and there are no data about power fluctuations in UPS logs.

Are you using IPSec on your hAP ac2 (RBD52G-5HacD2HnD-TC) routers?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Jan 06, 2020 7:06 am

I had an issue like this a few weeks ago, but much less random, on one of my hAP ac2s. Whenever traffic was forwarded between a GRE over IPsec tunnel and a policy-based IPsec tunnel, both passing through the same WAN interface, the router was consistently restarting (although some of the traffic did pass through each time, so I assume it was a particular size or contents of a packet being forwarded that crashed it). The remedy was to replace the policy-based IPsec tunnel by a GRE over IPsec one; replacement of the power adaptor was the first thing to come on my mind but it didn't help.

As usually with configurations involving IPsec and 3rd party networks, I hesitate to send supout files to support, so I haven't reported the issue to Rīga as I haven't had time yet to reproduce it in the lab.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Jan 07, 2020 3:50 am

I had an issue like this a few weeks ago, but much less random, on one of my hAP ac2s. Whenever traffic was forwarded between a GRE over IPsec tunnel and a policy-based IPsec tunnel, both passing through the same WAN interface, the router was consistently restarting (although some of the traffic did pass through each time, so I assume it was a particular size or contents of a packet being forwarded that crashed it).
Could that actually be a race condition (instead of a particular traffic)?
Did the reboots occur some seconds after the connection establishment? Was the time between connection establishment and reboot varying notably (e.g. sometimes being 5 seconds and sometimes 30 seconds)?

What algorithms and PFS ground are you using? I am using AES-256(-CBC), SHA256 and modp4096.
 
Kampfwurst
Member Candidate
Member Candidate
Posts: 107
Joined: Mon Mar 24, 2014 2:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Jan 07, 2020 8:45 am

I think it has something to do with the cpu frequenz. I have two hAP ac2. One had the problem with the reboots. On this one i was playing arround with the cpu frequenz. I changed the power supply, installed it new with netinstall...

At the end I set up a new config with only ip and then i connected it to a RB1100x4 for testing with the Bandwith Test. After some time (mostly 5-10min) under load it startet to reboot.
In the config was only one IP over DHCP and the BW test. Nothing else.

I changed the device. Now i have a new hAP ac2 running for 30days. The second one is running now für 70days
 
aradicev
just joined
Posts: 12
Joined: Tue Apr 29, 2014 5:39 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 19, 2020 8:31 pm

I am experiencing very similar thing with mine hAP ac2 router. It reboots randomly within 24 hrs and it does it random number of times. It started somewhere begging of December when it rebooted each 2-4 days. The under clocking did not help, my router rebooted also when under clocked. I have also tried with changing of the AC adapter, but i didn't help either.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 19, 2020 8:57 pm

I am experiencing very similar thing with mine hAP ac2 router. It reboots randomly within 24 hrs and it does it random number of times. It started somewhere begging of December when it rebooted each 2-4 days. The under clocking did not help, my router rebooted also when under clocked. I have also tried with changing of the AC adapter, but i didn't help either.
Did it start after installing RouterOS 6.46?
Does the router have a simple (default or close to default) or complex configuration?
What features are being used (are you using IPSec, GRE/IPIP/EoIP tunnels, bridge filter/nat rules, MPLS etc.)?
What packages are installed (/system package print)? Is the NTP package installed?
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Jan 25, 2020 11:50 pm

Hi.For 4 months now I have hap ac2 and have been observing observing unexpected router reboots. On average, this happens a couple of times a week. Replacing the power supply did not solve the problem, as well as lower processor frequencies. I installed the simplest configuration a week ago after flashing and resetting the configuration with Netinstall, today I got a reboot again. Now I know for sure that this is a defective router, but replacing it under warranty is not so simple.
 
Zacharias
Forum Guru
Forum Guru
Posts: 3459
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 7:51 pm

Is ROS and firmware updated ?
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 9:02 pm

updated to 6.44.6
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 9:36 pm

updated to 6.44.6
Is there any reason why you are not using latest RouterOS (6.46.2)?

When I had the problem first time the hAP ac2 router I swapped by warranty did not reboot for 2 weeks when run in default configuration (and probably no network connected). But that was on March 2019 (with latest RouterOS then).

Currently I have removed NTP package and that seems to have helped in my case - I have not got any reboots for around 14 days on the router which since upgrading to 6.46 was rebooting every few days (maximum time between reboots was 7 days). I am still running RouterOS 6.46.1 as I don't want the tests to be affected by RouterOS upgrades (I am planning to upgrade later).
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 10:38 pm

Updated to 6.46.2. I do not have an ntp package installed. Earlier, I already tried hundreds of ways to find out what the problem is.
If you turn off watchdog, then the router just hangs. There are suspicions that the problem is in the defective memory, but I do not know how to check it.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 11:53 pm

Updated to 6.46.2. I do not have an ntp package installed. Earlier, I already tried hundreds of ways to find out what the problem is.
If you turn off watchdog, then the router just hangs. There are suspicions that the problem is in the defective memory, but I do not know how to check it.
If your router hangs/reboots with a fairly simple configuration, you should contact the distributor and try to get swapped by warranty (get RMA). It is better to provide a supout file generated during/after the reboot (if crash doesn't generate a supout file, then make a "/system scheduler" entry that will produce one on startup after the crash) or to make sure that distributor will be able to reproduce crash and create a supout file (AFAIK in many cases supout file is mandatory for RMA).
You can contact the distributor to get details about RMA process or you can create a support case with MikroTik, send them the supout file and get an advice from MikroTik whether you should contact distributor with RMA request.
P.S. Supout file might contain some sensitive information and it will contain "/export hide-sensitive" output.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Jan 26, 2020 11:55 pm

If you turn off watchdog, then the router just hangs.

When router hangs, does it respond over wireless? Do you have any non-default config regarding switch chip?

I've had frequent hangs (cured by watchdog reboots) of my RBD52G with ROS 6.42 (factory default) and some 6.43 ... but I had some non-basic switch chip config (VLANs configured via /interface ethernet switch subtree) ... when I reconfigured that to use bridge vlan-filtering, hangs/reboots ceased.
I've never tried to connect via wireless while device was hung, I'm using it only as wired router.
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Jan 27, 2020 10:50 pm

Updated to 6.46.2. I do not have an ntp package installed. Earlier, I already tried hundreds of ways to find out what the problem is.
If you turn off watchdog, then the router just hangs. There are suspicions that the problem is in the defective memory, but I do not know how to check it.
If your router hangs/reboots with a fairly simple configuration, you should contact the distributor and try to get swapped by warranty (get RMA). It is better to provide a supout file generated during/after the reboot (if crash doesn't generate a supout file, then make a "/system scheduler" entry that will produce one on startup after the crash) or to make sure that distributor will be able to reproduce crash and create a supout file (AFAIK in many cases supout file is mandatory for RMA).
You can contact the distributor to get details about RMA process or you can create a support case with MikroTik, send them the supout file and get an advice from MikroTik whether you should contact distributor with RMA request.
P.S. Supout file might contain some sensitive information and it will contain "/export hide-sensitive" output.
Of course, supout.rif does not form during reboot. I see no reason to form it through Scheduler after a reboot. There will be nothing useful in this file. Too resolve the problem I decided to try forcibly rebooting the router through the sheduler once a day at night.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Jan 28, 2020 1:26 am

When router hangs, does it respond over wireless? Do you have any non-default config regarding switch chip?

I've had frequent hangs (cured by watchdog reboots) of my RBD52G with ROS 6.42 (factory default) and some 6.43 ...
Interestingly, I am using switch rules. However I never had the issue with 6.42, the issue initially started with 6.43 (without any changes in switch rules). Afterwards switch rules were changed, but I didn't see any notable correlation between switch rule changes and reboots aplearing/disappearing (there was only one possible correlation event and it was 7 days apart).

Of course, supout.rif does not form during reboot. I see no reason to form it through Scheduler after a reboot. There will be nothing useful in this file. Too resolve the problem I decided to try forcibly rebooting the router through the sheduler once a day at night.
I was asked to generate supout file for RMA after the reboot.
Android has last_kmsg for obtaining kernel crash information after reboot, maybe MikroTik has something similar (I don't know).
Are the intervals between two random reboots of your router always bigger than 24 hours?
 
Issarin
just joined
Posts: 7
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Jan 29, 2020 6:47 am

Hi, I'm not native english speaker but want to share my experience.

I have my hap ac2 for about 6 months. Before updating to 6.46, sometimes it reboots randomly but not so often, maybe weeks or almost a month between each reboot. Last week I updated to 6.46.2 (from 6.45.x) and it started to reboot very often, 2-3 times a day so I downgraded to 6.45.7 yesterday and it's over 24 hours now without any reboot.

I think my config is quite simple but with one IPSEC for site to site VPN and I also use NTP package, other than that is just normal setting.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Wed Jan 29, 2020 2:30 pm

I have my hap ac2 for about 6 months. Before updating to 6.46, sometimes it reboots randomly but not so often, maybe weeks or almost a month between each reboot. Last week I updated to 6.46.2 (from 6.45.x) and it started to reboot very often, 2-3 times a day so I downgraded to 6.45.7 yesterday and it's over 24 hours now without any reboot.

I think my config is quite simple but with one IPSEC for site to site VPN and I also use NTP package, other than that is just normal setting.
Thanks a lot for sharing this info!

Can you uninstall NTP package and check whether reboots stop happening?
Please note that NTP client (/system ntp client) is available also without the NTP package (NTP package is required only for NTP server).
 
Issarin
just joined
Posts: 7
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Jan 29, 2020 3:21 pm

Thanks a lot for sharing this info!

Can you uninstall NTP package and check whether reboots stop happening?
Please note that NTP client (/system ntp client) is available also without the NTP package (NTP package is required only for NTP server).
I plan to try that as well but first I want to see how long I can get with 6.45.7 plus NTP before it reboots again. Then I will uninstall NTP and may also try to upgrade to 6.46.2 to see if there is any difference. With this kind of problem, it takes so much time to try something and verify the result....

Anyway, I will post back if I get something.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Wed Jan 29, 2020 8:29 pm

I plan to try that as well but first I want to see how long I can get with 6.45.7 plus NTP before it reboots again. Then I will uninstall NTP and may also try to upgrade to 6.46.2 to see if there is any difference. With this kind of problem, it takes so much time to try something and verify the result....

Anyway, I will post back if I get something.
That will be an interesting test! For me 6.45.7 worked for 38 days without a single reboot (then I upgraded to 6.46).
I have no idea whether these 38 days without reboots are somehow related to the potential fix described in the OP.
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Jan 29, 2020 10:22 pm

It seems to me that we are using the wrong approach to solving the problem. We do not need to look for a way to stop accidental reboots, but rather make the router reboot every 5 minutes) So we will quickly find the reasons for the reboots. Let's remember under what conditions the router was rebooted most often.What can influence the behavior of the router from what we know? Firmware version, availability of NTP package, what else?
 
gowa
just joined
Posts: 6
Joined: Sun Dec 04, 2016 12:13 pm

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jan 30, 2020 12:32 am

yet another possible fix:
viewtopic.php?f=2&t=153666&p=771975#p771975

I blamed my USB drive and logging on it :)
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jan 30, 2020 2:29 am

It seems to me that we are using the wrong approach to solving the problem. We do not need to look for a way to stop accidental reboots, but rather make the router reboot every 5 minutes) So we will quickly find the reasons for the reboots. Let's remember under what conditions the router was rebooted most often.What can influence the behavior of the router from what we know? Firmware version, availability of NTP package, what else?
As the reboots did not start immediately after my hardware swap (described in OP) there are quite some chances that reboots are not easily reproducible on any hardware.

yet another possible fix:
viewtopic.php?f=2&t=153666&p=771975#p771975

I blamed my USB drive and logging on it :)
hAP ac and hAP ac2 are very different hardware.

Issarin, In case you would like to report this to MikroTik as a bug, I think before uninstalling NTP package you should upgrade to 6.46.2 (latest stable version), wait for a reboot and create supout file (and save it to PC). Supout file creation can also be scheduled on every boot (using /system scheduler - I can help with this if needed).
The supout file might contain some sensitive data (and it will contain "/export hide-sensitive") - this is one of the reasons why I am not very eager to generate it myself.
 
User avatar
inteq
Member
Member
Posts: 402
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jan 30, 2020 10:02 pm

Had one location with a hAP ac2 with the same symptoms.
The owner was using a lousy power strip and of course, without an UPS.

Changed the power strip to a more sturdy one and added an UPS.
No more problems since.
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jan 30, 2020 10:22 pm

By the way, before I installed the power filter, the router rebooted much more often. The filter is like this: https://www.apc.com/shop/ru/ru/products ... /P-PM1W-RS
Last edited by rbfast on Thu Jan 30, 2020 11:50 pm, edited 2 times in total.
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jan 30, 2020 11:46 pm

I found an easy way to cause a spontaneous restart of the router due to a kernel error caused by out of memory. As a result of such a reboot, the autosupout.rif file is not generated on the Hap AC2, and such a file is formed on the RB951Ui-2HnD.I can make a video on this subject if the public has interest
 
r00t
Long time Member
Long time Member
Posts: 672
Joined: Tue Nov 28, 2017 2:14 am

Re: Possible fix for hAP ac2 rebooting randomly

Fri Jan 31, 2020 2:49 am

What is the general state of this issue? Have it been officially confirmed to exist by Mikrotik? Or they are still playing the "can not reproduce" game?
Maybe best would be to give them one of the rebooting hAPs on MUM (negotiate exchange for a new one... this should be doable) so they can finally see it in action?

As for using power filters, if power input was really the culprit, it would make a difference running it on clean lab powersupply or just battery (no noise at all). But hAP uses switching DC-DC converter inside so noise or ripple on power input shouldn't matter that much.
I found an easy way to cause a spontaneous restart of the router due to a kernel error caused by out of memory.
I hope you reported it to support and it will be fixed, but I don't think this rebooting issue is related to RAM usage, graphing it doesn't reveal any fast filling of RAM before the reboot. Or maybe we have multiple issues here...
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 01, 2020 6:41 pm

You need to run the Bandwidth test at the same time as the Packet sniffer. In the Bandwidth test window in the TEST TO field, specify the address of the router itself, do not forget the username and password below. Run the packet sniffer tool with the limit set to 128,000 kb set in the Memory limit field. In 5-10 seconds, what I love so much now will happen. If this behavior is incorrect for the Mikrotik router, please contact technical support. I don’t want to do this myself. I do not believe that they will do anything.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 02, 2020 12:51 am

You need to run the Bandwidth test at the same time as the Packet sniffer. In the Bandwidth test window in the TEST TO field, specify the address of the router itself, do not forget the username and password below. Run the packet sniffer tool with the limit set to 128,000 kb set in the Memory limit field. In 5-10 seconds, what I love so much now will happen. If this behavior is incorrect for the Mikrotik router, please contact technical support. I don’t want to do this myself. I do not believe that they will do anything.
Why do you think this is related to the random reboots? :-o :?: This is not a rhetoric question!

hAP ac2 has 128 MB or 256 MB RAM (I guess your device probably has 128 MB RAM). If you run something that eats-up all the RAM (on device that has no swap space on disk) either the RAM-eating process will terminate or the device will crash/reboot.
I would be against restricting maximum size of the sniffer buffer, because either:
  • it would still be possible to exhaust the memory by combining several RAM hungry things (e.g. placing maximum amount of files on the "RAM disk" and running sniffer with maximum buffer size);
  • all the RAM limits would be unreasonably small (so they could not exhaust RAM if taken together) - this would be the worst thing - a massive reduction in functionality of the router with extremely small benefit;
  • some complex memory management/dynamic restrictions logic would need to be implemented (that would be notable development efforts with relatively small benefit for end-users).

If the sniffer buffer is in usermode process, then a-kind-of generic fix would be to implement out-of-memory killer (a Linux kernel feature that would kill the RAM-eating processes upon RAM exhaustion). That might be a good idea, however that might lead to situations where router starts to malfunction instead of rebooting and functioning normally (so this might actually be not so good idea afterall)...
I hope this gives you an answer why MikroTik might not be eager to fix issues like this one...
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 02, 2020 10:18 pm

Rebooted again today ((Rolled back to 6.43.13.
 
Issarin
just joined
Posts: 7
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 09, 2020 8:13 pm

Here is what I have gathered so far;
- 6.45.7 + NTP : 96 hours passed without reboot but I decided to stop and upgrade to 6.46.2
- 6.46.2 + NTP : I got random reboots with the interval of 36, 4, 15 and 10 hours
- 6.46.2 (NO NTP) : Now it's over 60 hours and going without reboot
The supout file might contain some sensitive data (and it will contain "/export hide-sensitive") - this is one of the reasons why I am not very eager to generate it myself.
Yes, also my concern.
Maybe, after a few more days, if there is still no reboot then I might go back to 6.46.2 + NTP again with minimal configuration to get Supout for Mikrotik.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 09, 2020 9:00 pm

Here is what I have gathered so far;
- 6.45.7 + NTP : 96 hours passed without reboot but I decided to stop and upgrade to 6.46.2
- 6.46.2 + NTP : I got random reboots with the interval of 36, 4, 15 and 10 hours
- 6.46.2 (NO NTP) : Now it's over 60 hours and going without reboot
Maybe, after a few more days, if there is still no reboot then I might go back to 6.46.2 + NTP again with minimal configuration to get Supout for Mikrotik.
Great testing results! Thanks a lot!
If you won't be able to reproduce the issue with minimal configuration, maybe IPSec is also required for the reboots to happen...
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 09, 2020 9:27 pm

My results after disabling NTP package:
  • First hAP ac2 router (that is described in OP) has had NTP package disabled for 3 weeks and had 0 random reboots.
  • Second hAP ac2 router (that started to reboot randomly on December 2019) has had NTP package disabled for 4 weeks and had only 1 random reboot (during very unusual 35 Mbps IPSec to wifi traffic; also power glitch cannot be excluded).

Also 3 other users on 5 (hAP ac2) devices have confirmed that removing NTP package has resolved the random reboots:
viewtopic.php?f=21&t=156825#p773118
viewtopic.php?f=21&t=156825#p773712
viewtopic.php?f=2&t=153988#p774170

So looks like NTP package is related to the problem. Also at least in some cases (on my routers) this seems to be related to RouterOS version - after installing RouterOS 6.46 the issue re-appeared on the first router (that previously had random reboots) and started to happen on the second router that had no random reboots previously.
To me it is still unclear:
  • How it is related to the hardware (why swapping hardware temporarily helped back in spring 2019). The old hardware was rebooting usually within 7 days, the swapped hardware worked for around 20 days before first RouterOS update (and around a month afterwards).
  • Whether/how/why the underclocking (described in OP) helped. The underclocked router worked 10 days without reboots before first RouterOS update.
  • Whether (in conjunction with NTP package) this is related to IPSec or some other features that are not enabled in default configuration. Traveller, DimaFIX Do you also have IPSec (or something else non-default) enabled on all problematic routers? Update! Traveller and DimaFIX have confirmed that all their problematic routers also have IPSec enabled. Looks like all the users facing this problem have IPSec enabled and NTP package installed.
Last edited by Davis on Tue Feb 11, 2020 12:57 am, edited 1 time in total.
 
Traveller
newbie
Posts: 35
Joined: Thu Apr 05, 2018 10:12 am

Re: Possible fix for hAP ac2 rebooting randomly

Mon Feb 10, 2020 5:02 pm

I have 4 hap ac2 routers in service. My two: the main router and the router as an access point and two strangers. All have ipsec l2tp vpn. I communicate with them through this vpn. Three routers were rebooted with the ntp package. Any router is not rebooted without an ntp package. The access point now stands with the beta version of the ROS 7 and does not rebooted. ntp server is built in to ROS 7. Install ROS 7 to the home main router going to frequent reboots. An access point with an enabled ntp server is already on the network for 62 days. My router and access point have been exchanged one by one. It did not give results. My conclusion: In addition to the ntp server, an even higher load on the router is required for rebooting.
 
User avatar
DimaFIX
just joined
Posts: 12
Joined: Wed Apr 18, 2018 7:46 pm
Location: Ukraine

Re: Possible fix for hAP ac2 rebooting randomly

Mon Feb 10, 2020 10:42 pm

Davis, Yes, I'm using IPSec in an L2TP tunnel on my router.
After removing the NTP packet, the reboots stopped for 2 days.
Then I installed this package again, turned off watchdog, and after 38 hours the router crashed without showing any signs of life.
Now I have enabled watchdog and removed the NTP package. Now the router works 12 hours without problems.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 12:53 am

Thanks! Looks like in all cases not only NTP package was installed, but also IPSec was enabled.
In my case it is not L2TP, but other protocols over IPSec.
 
User avatar
rooted
Member Candidate
Member Candidate
Posts: 129
Joined: Tue Feb 04, 2020 5:58 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 4:59 am

What is the average load on the devices that are rebooting?
 
Reinis
MikroTik Support
MikroTik Support
Posts: 87
Joined: Wed Jan 02, 2019 12:14 pm
Location: Latvia
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 8:01 am

You will not ever be able to determine the cause of the reboot in the forum. If your router is rebooting by itself, then please contact support@mikrotik.com providing supout.rif file and any details on how to reproduce the issue. If there is a specific workaround, provide details about that too. Only MikroTik staff can determine the precise cause of the problem and if it is caused by the software, then we want to fix this problem as soon as possible. If you even manage to workaround a problem based on other user experience, then the issue is still there and others suffer from it.
Please understand that the current topic information lacks consistency and might be different issues with the same result, thus provide only your own test and troubleshooting results. All bug reports are very welcome.
 
Kampfwurst
Member Candidate
Member Candidate
Posts: 107
Joined: Mon Mar 24, 2014 2:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 8:56 am

if it helps to fine the bug.
On my hAP ac2 is version 6.45.7 with NTP since 69 days running without reboot.
I have also 3 IPSec connections running
 
User avatar
macgaiver
Forum Guru
Forum Guru
Posts: 1764
Joined: Wed May 18, 2005 5:57 pm
Location: Sol III, Sol system, Sector 001, Alpha Quadrant

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 9:52 am

What's new in 6.47beta32 (2020-Feb-10 11:45):

*) arm - improved watchdog and kernel panic reporting in log after reboots on RB3011 and IPQ4018/IPQ4019 devices ("/system routerboard upgrade" required);
I have the same issue, from my perspective it looks like issue is related to specific wireless client combination (right mix), that more likely causes this issue, usually it takes blend of IPhones/Androids/Chromecasts and smart scale to cause that issue :) on average once every week.

Now, after upgrading instead of just "rebooted without proper shutdown" i know in my case it is caused "by watchdog", now plan is to disable watchdog and allow it to crash/hang, and then manually reboot to check if something is in supout.rif file
 
User avatar
CZFan
Forum Guru
Forum Guru
Posts: 2098
Joined: Sun Oct 09, 2016 8:25 pm
Location: South Africa, Krugersdorp (Home town of Brad Binder)
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 11, 2020 9:44 pm

Davis, Yes, I'm using IPSec in an L2TP tunnel on my router.
After removing the NTP packet, the reboots stopped for 2 days.
Then I installed this package again, turned off watchdog, and after 38 hours the router crashed without showing any signs of life.
Now I have enabled watchdog and removed the NTP package. Now the router works 12 hours without problems.

Did you check if you are not maybe being DDoSed on port 123 (NTP) causing the reboots?
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Thu Feb 13, 2020 3:24 am

What is the average load on the devices that are rebooting?
In my case load on the rebooting devices was close to zero (well under 1 Mbps in many cases; below 10 Mbps in vast majority of cases).
The reboots did not seem to be correlated with load (reboots were not concentrated in peak-load time periods).

I have the same issue, from my perspective it looks like issue is related to specific wireless client combination (right mix), that more likely causes this issue, usually it takes blend of IPhones/Androids/Chromecasts and smart scale to cause that issue :) on average once every week.
Looks like this is another issue. In most cases one of my routers that was rebooting actually had 0 wifi clients connected.

Did you check if you are not maybe being DDoSed on port 123 (NTP) causing the reboots?
I am pretty sure I was not DoS-ed. In my case NTP was opened only from local network and during testing for RMA a router was connected to an isolated network.

You will not ever be able to determine the cause of the reboot in the forum. If your router is rebooting by itself, then please contact support@mikrotik.com providing supout.rif file and any details on how to reproduce the issue. If there is a specific workaround, provide details about that too. Only MikroTik staff can determine the precise cause of the problem and if it is caused by the software, then we want to fix this problem as soon as possible. If you even manage to workaround a problem based on other user experience, then the issue is still there and others suffer from it.
Please understand that the current topic information lacks consistency and might be different issues with the same result, thus provide only your own test and troubleshooting results. All bug reports are very welcome.
Thanks for looking into this! There is a recent support ticket regarding this issue from Traveller.

Most likely there is one common issue - random reboots on routers with IPSec and NTP package. Removal of NTP package has helped for 7 devices of 4 users (including myself).

According to my experience this issue might not be reliably reproducible on all hardware - for me one router was rebooting every few days, netinstall+running config script didn't help at all, however (after some time though) swapping hardware (and doing exactly the same netinstall + running config script on the new hardware) seemingly "resolved" the situation (I was able to connect the rebooting router to an isolated network in another place and continue to observe reboots there, but the new hardware had no random reboots). After a couple of months the issue appeared on the new hardware as well. I.e. importing the config of a problematic router on a test device might not be enough to reproduce the issue in lab.
However, assuming that the hardware susceptible to this issue is spread evenly among all the hardware (or that "wear and tear" of a router increases probability to have this issue) I would suggest the following:
  • Configure a pool of hAP ac2 routers with the config from Traveller's ticket.
  • Create the "other end" for all the IPSec tunnels. During my testing (on isolated network, during spring 2019, for RMA purpose) I had IPSec up and some traffic flowing from LAN side of the problematic router to the "other end" of IPSec (the other end just dropped all incoming IPSec encapsulated packets).
  • Ensure that test routers can reach the NTP server(s) configured. During my testing I had internet connectivity for the problematic router, but I did not have any NTP clients.
  • Hopefully some of the routers will reboot randomly...

I completely agree with "If you even manage to workaround a problem based on other user experience, then the issue is still there and others suffer from it.", also I would like to add that it is not known whether there can be less frequent random reboots (that would be considered to be isolated cases with unknown causes by the users) also without NTP package. Also it is not known how widespread is this issue (I would guess that many users who have random reboots+NTP package+IPSec on hAP ac2 actually have this issue).
Looks like the issue might (this is just a guess - the issue might have been introduced earlier) have started in 6.43 (my observation is 6.43/6.43.1, Traveller's observation is 6.43.15).

A few very technical points (that might serve as input for developers) I would like to make are:
  • most likely there have been no code changes in NTP package in recent versions of RouterOS, however there have been IPSec related changes
  • IPSec has some kernel code, NTP most likely doesn't, and kernel code has many more ways for crashing/locking-up a system
  • IPSec implementation might be sensitive to changes in system time (however I don't know whether MikroTik implementation counts the remaining time of SAs in kernel or user mode, I don't know how CLOCK_MONOTONIC and CLOCK_REALTIME are used etc.)
  • NTP client of NTP package and the built-in NTP client (I guess part of system package) might use different Linux APIs (adjtimex(), settimeofday() etc.) to set the time
  • logically unrelated changes in code can affect race conditions as memory layout changes can have notable effects on execution times of certain procedures (this could explain why the issue seemingly disappears in some versions and then comes back in later versions - of course this is just a guess)
 
Issarin
just joined
Posts: 7
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Thu Feb 13, 2020 6:43 am

What is the average load on the devices that are rebooting?
For my case, it's home use so most of the time it's just idling and in addition to that sometime the reboot happened in the late night which no one was using anything.

If you won't be able to reproduce the issue with minimal configuration, maybe IPSec is also required for the reboots to happen...
I just tried 6.46.2 + NTP again (I did keep IPSec) and it rebooted after 28 hours. Already sent Mikrotik the Supout file.
 
Traveller
newbie
Posts: 35
Joined: Thu Apr 05, 2018 10:12 am

Re: Possible fix for hAP ac2 rebooting randomly

Thu Feb 13, 2020 9:08 am

There is a recent support ticket regarding this issue from Traveller.
The answer was expected:
Unfortunately, there is no information about the problem in the supout file.
But we have made movement on this issue:
However, yesterday we released a new beta version with improvements regarding a problem troubleshooting exactly for the hAP ac^2 units.
I will not install beta versions on working responsible equipment. But perhaps someone this problem will begin to be fixed.
 
User avatar
rooted
Member Candidate
Member Candidate
Posts: 129
Joined: Tue Feb 04, 2020 5:58 pm

Re: Possible fix for hAP ac2 rebooting randomly

Fri Feb 14, 2020 2:52 am

I have a mixture of 32+ devices connecting to the hAP ac² which include 3 Chromecast, many Android devices, and some iOS and haven't suffered a reboot.

I'm going to enable NTP and see if anything changes, but being I've never had a reboot NTP may be a determining factor.

 
gurnec
just joined
Posts: 6
Joined: Wed Jul 14, 2010 9:42 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 15, 2020 2:28 am

If it's of any use, I'm including a log of reboots and upgrades from a single AC2 that's been running for about 10 months where I have with the NTP package installed, IPSec enabled (but very rarely used), and watchdog enabled. Although there have been many config changes over time, the ones I just mentioned have been static since the beginning.
Installed Apr/12/1019
   Now Running 6.44.2
apr/26/2019 14:19:12 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.44.3
jul/01/2019 16:42:29 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.45.1
jul/22/2019 19:35:44 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.45.2
aug/05/2019 18:21:46 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.45.3
aug/18/2019 06:22:49 system,error,critical router was rebooted without proper shutdown
sep/06/2019 21:47:37 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.45.5
oct/10/2019 22:38:26 system,error,critical router was rebooted without proper shutdown
oct/12/2019 16:48:37 system,error,critical router was rebooted without proper shutdown
oct/26/2019 04:28:40 system,error,critical router was rebooted without proper shutdown
oct/27/2019 04:27:37 system,error,critical router was rebooted without proper shutdown
oct/28/2019 20:06:15 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.45.7
nov/18/2019 01:00:03 system,error,critical router was rebooted without proper shutdown
nov/18/2019 07:37:49 system,error,critical router was rebooted without proper shutdown
nov/29/2019 01:09:46 system,error,critical router was rebooted without proper shutdown
dec/03/2019 18:09:15 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.46
dec/08/2019 10:19:04 system,error,critical router was rebooted without proper shutdown
dec/09/2019 19:14:04 system,error,critical router was rebooted without proper shutdown
dec/12/2019 00:22:00 system,error,critical router was rebooted without proper shutdown
dec/12/2019 06:37:05 system,error,critical router was rebooted without proper shutdown
dec/14/2019 00:12:33 system,error,critical router was rebooted without proper shutdown
dec/14/2019 05:04:54 system,error,critical router was rebooted without proper shutdown
dec/17/2019 16:58:58 system,error,critical router was rebooted without proper shutdown
dec/20/2019 23:28:53 system,error,critical router was rebooted without proper shutdown
jan/05/2020 09:27:50 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.46.1
jan/06/2020 15:26:26 system,error,critical router was rebooted without proper shutdown
jan/07/2020 22:27:58 system,error,critical router was rebooted without proper shutdown
jan/17/2020 20:37:36 system,error,critical router was rebooted without proper shutdown
jan/18/2020 16:15:49 system,error,critical router was rebooted without proper shutdown
jan/20/2020 14:32:13 system,error,critical router was rebooted without proper shutdown
jan/21/2020 13:30:42 system,error,critical router was rebooted without proper shutdown
jan/25/2020 22:26:18 system,error,critical router was rebooted without proper shutdown
jan/30/2020 22:49:14 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
   Now Running 6.46.2
jan/31/2020 07:46:11 system,error,critical router was rebooted without proper shutdown
feb/03/2020 15:58:29 system,error,critical router was rebooted without proper shutdown
feb/04/2020 16:25:24 system,error,critical router was rebooted without proper shutdown
feb/06/2020 17:59:23 system,error,critical router was rebooted without proper shutdown
feb/07/2020 13:17:24 system,error,critical router was rebooted without proper shutdown
feb/07/2020 16:13:36 system,error,critical router was rebooted without proper shutdown
feb/10/2020 07:58:24 system,error,critical router was rebooted without proper shutdown
feb/11/2020 14:42:14 system,error,critical router was rebooted without proper shutdown
feb/11/2020 20:21:46 system,info,critical Firmware upgraded successfully, please reboot for changes to take effect!
  Now Running 6.46.3 and NTP Package Disabled
For this device, the reboots started somewhere around the 6.45.3 - 6.45.5 timeframe. I disabled the NTP package (still have the NTP client running) about 3 days ago, and so far no reboots. (I also upgraded to a newer firmware... perhaps not the smartest move to make two changes at once I admit.)

If I do see a reboot, I'll be sure to report back.
 
Issarin
just joined
Posts: 7
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 15, 2020 1:56 pm

Just an update.

Mikrotik didn't find any problem from my supout and they asked me to try 6.47beta32.
I already installed that along with the NTP package and has been running for two days. I will update again after a few more days.
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 15, 2020 6:23 pm

Hello. I requested help from Mikrotik a week ago. They watched the router for a week. During all this time there has not been a single spontaneous reboot. The router has been updated to 6.47beta32. Maybe it helped.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 16, 2020 3:19 am

I have a mixture of 32+ devices connecting to the hAP ac² which include 3 Chromecast, many Android devices, and some iOS and haven't suffered a reboot.

I'm going to enable NTP and see if anything changes, but being I've never had a reboot NTP may be a determining factor.

Looks like IPSec+NTP package (with NTP client configured) might be the combination that causes the random reboots...
And thanks for experimenting/helping!

Issarin, Thanks for the update!
 
User avatar
rooted
Member Candidate
Member Candidate
Posts: 129
Joined: Tue Feb 04, 2020 5:58 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 16, 2020 5:40 am

Looks like IPSec+NTP package (with NTP client configured) might be the combination that causes the random reboots...
And thanks for experimenting/helping!

Issarin, Thanks for the update!
Yeah it's been 2 days with NTP enabled and nothing has gone awry, I don't have IPSec enabled but I will try to find a server and enable it also.
 
netflow
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Sat Oct 01, 2016 3:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Feb 17, 2020 8:35 pm

I removed NTP package but still have the same issue. It may just decrease likelihood but for sure it is not the solution. Common point between NTP, wireless and IPSec could be RTC clock. Running 6.47beta32, hoping to generate an autosupout file for support.
 
rbfast
just joined
Posts: 11
Joined: Sat Jan 25, 2020 11:41 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Feb 18, 2020 10:19 pm

Hello! 16 days without reboots on 6.47beta32
 
Issarin
just joined
Posts: 7
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 22, 2020 2:16 pm

For me, I've already tried v6.47beta32 & v6.47beta35 and it's still rebooting after 2-3 days without auto-generated supout file.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 23, 2020 7:57 pm

For me, I've already tried v6.47beta32 & v6.47beta35 and it's still rebooting after 2-3 days without auto-generated supout file.
I suggest to schedule creation of supout file on every boot, wait for a reboot to happen and send the supout file to MikroTik.
Creation of supout file on boot can be scheduled by command:
/system scheduler add name=create-supout on-event="/system sup-output" start-time=startup
 
Issarin
just joined
Posts: 7
Joined: Wed Jan 22, 2020 2:15 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Feb 26, 2020 2:09 pm

I suggest to schedule creation of supout file on every boot, wait for a reboot to happen and send the supout file to MikroTik.
Creation of supout file on boot can be scheduled by command:
/system scheduler add name=create-supout on-event="/system sup-output" start-time=startup
Yes, I did that and sent the file to Mikrotik but they said it didn't record any error that they can investigate.

My understanding is that they are trying to get auto-supout working which should log the error they want to see so they asked me to try the beta version. I believe they improve this function in beta version but still not working in my case.
 
netflow
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Sat Oct 01, 2016 3:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Fri Mar 27, 2020 11:18 pm

Received a new unit after returning the first under warranty. Unfortunately I can still make it unresponsive almost immediately when using wifi.

Since it is not directly related with reboots (mine hangs forever without being able to reboot by itself), I have put information here.
 
aradicev
just joined
Posts: 12
Joined: Tue Apr 29, 2014 5:39 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Mar 28, 2020 1:57 am

I am experiencing very similar thing with mine hAP ac2 router. It reboots randomly within 24 hrs and it does it random number of times. It started somewhere begging of December when it rebooted each 2-4 days. The under clocking did not help, my router rebooted also when under clocked. I have also tried with changing of the AC adapter, but i didn't help either.
Did it start after installing RouterOS 6.46?
Does the router have a simple (default or close to default) or complex configuration?
What features are being used (are you using IPSec, GRE/IPIP/EoIP tunnels, bridge filter/nat rules, MPLS etc.)?
What packages are installed (/system package print)? Is the NTP package installed?
Sorry for responding quite late to your post. My router was still with warranty so I have sent it to the service and after fighting with them for almost 2 months I got a new device. However the new device also reboots randomly, well in about 2 plus some hours interval. So it's less than the former and with a different message. The former had the message "system,error,critical router was rebooted without proper shutdown" and the latter "system,error,critical router was rebooted without proper shutdown by watchdog timer". So I do not think that those two problems are related.
For the former device I cannot be sure that it started with the upgrade to 6.46, all I know is that the reboots got more frequent during time.
I'm running a simple configuration, basically I'm using it as an AP and a switch (all interfaces are in a bridge) and the NTP package is installed.

I have uninstalled the NTP package from the new device and am currently testing it. For now it runs for 2 days and some 13 hours, so it's due for a reboot but I hope it will not. I'll keep you posted how it goes.
 
scyld
just joined
Posts: 3
Joined: Sat Mar 28, 2020 12:05 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Mar 30, 2020 7:57 pm

Hello,

I have had the same issue with hAP ac². Current stable (6.46.6) and testing (6.47beta49) releases of RouterOS.
The resolution for now is to disable NTP Server. This fixes the unexpected reboot problem.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Wed Apr 01, 2020 2:40 am

I have had the same issue with hAP ac². Current stable (6.46.6) and testing (6.47beta49) releases of RouterOS.
The resolution for now is to disable NTP Server. This fixes the unexpected reboot problem.
What do you meant by "disable NTP Server" - uninstalling NTP package or just disabling NTP server in the configuration (these things internally are not the same)?
 
aradicev
just joined
Posts: 12
Joined: Tue Apr 29, 2014 5:39 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Apr 01, 2020 8:20 pm

I have uninstalled the NTP package from the new device and am currently testing it. For now it runs for 2 days and some 13 hours, so it's due for a reboot but I hope it will not. I'll keep you posted how it goes.
My device is now running without reboot for more than 7 days, usually it would have rebooted for at least 2 times by now. So It seams that the NTP package V6.46 is responsible for the crashes.
 
Kampfwurst
Member Candidate
Member Candidate
Posts: 107
Joined: Mon Mar 24, 2014 2:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Apr 01, 2020 8:53 pm

is it possible to install a older NTP package on the new version?
 
User avatar
macsrwe
Forum Guru
Forum Guru
Posts: 1007
Joined: Mon Apr 02, 2007 5:43 am
Location: Arizona, USA
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Wed Apr 01, 2020 11:09 pm

ROS does not allow this. You would have to downgrade all of ROS to that version.
 
User avatar
bogoevski
just joined
Posts: 6
Joined: Mon Apr 13, 2020 11:22 am

Re: Possible fix for hAP ac2 rebooting randomly

Mon Apr 13, 2020 11:48 am

Hello,
I am new to this forum and I Want to share my experience with this problem.

I have the similar experience on my HAP AC^2 with very simple configuration, running versions bellow.
v6.45.8 (longterm) reboots
v6.45.4 (stable) reboots
v6.46.5 (stable) reboots
My router reboots so frequently by watchdog timer, 2-3 hours was best uptime in my case, that's due to large number of devices synchronizing over NTP. Disabling watchdog, freezing the router.
I been trying with so many versions to get it work, but...
Dealing with CPU frequency didn't help. Disabling/uninstalling NTP package solves the reboot problem but anyway I need NTP server as my network has a lot of devices that needs to be time synchronized and their network has no access to the internet, so I cannot use any external/public NTP.

As @Isaarin and @Kampfwurst said I downgraded rOS and firmware to v6.45.7
4 days so far I have no single reboot.

I believe it won't reboot, as the power supply on my site is very stable with few backups, solar power, wind power, battery banks, AC mains...
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Apr 13, 2020 12:23 pm

I need NTP server as my network has a lot of devices that needs to be time synchronized and their network has no access to the internet, so I cannot use any external/public NTP.
If even 6.45.7 is not a long-term solution to the problem, a workaround could be to permit access to public NTP servers (and nowhere else) to just one of those devices and let the other ones synchronize from it.

I believe it won't reboot, as the power supply on my site is very stable with few backups, solar power, wind power, battery banks, AC mains...
That sounds impressive, what's the whole topology? Do you use both the PoE input on ether1 and the DC jack input on the hAP ac² itself, one per each 24VDC source group?
 
User avatar
macsrwe
Forum Guru
Forum Guru
Posts: 1007
Joined: Mon Apr 02, 2007 5:43 am
Location: Arizona, USA
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Tue Apr 14, 2020 1:18 am

If even 6.45.7 is not a long-term solution to the problem, a workaround could be to permit access to public NTP servers (and nowhere else) to just one of those devices and let the other ones synchronize from it.

I read this as: let some other device on your network run NTP server. That's great as long as NTP server is crashing only hAP ac2s, but I think I have heard that the problem is more widespread than that.
 
User avatar
bogoevski
just joined
Posts: 6
Joined: Mon Apr 13, 2020 11:22 am

Re: Possible fix for hAP ac2 rebooting randomly

Tue Apr 14, 2020 1:20 am

I need NTP server as my network has a lot of devices that needs to be time synchronized and their network has no access to the internet, so I cannot use any external/public NTP.
If even 6.45.7 is not a long-term solution to the problem, a workaround could be to permit access to public NTP servers (and nowhere else) to just one of those devices and let the other ones synchronize from it.

I believe it won't reboot, as the power supply on my site is very stable with few backups, solar power, wind power, battery banks, AC mains...
That sounds impressive, what's the whole topology? Do you use both the PoE input on ether1 and the DC jack input on the hAP ac² itself, one per each 24VDC source group?
That's a strict policy, no internet to this kind of sensitive devices.

All the devices on the site are using 12V DC, The power topology... Photovoltaics and wind turbine are connected to PWM charging controller, the controller cares for a bank of eight 12V/100Ah VRLA batteries, the bank sum is 12V/800Ah, enough power for site to run if no AC, no sun and no wind for 7 days :) The 220VAC charger also charges the batteries if their voltage drops bellow floating level, in my case 13.67V.
Summary: All the devices are powering from DC power distribution unit which is powered directly from the battery bank.
 
User avatar
bogoevski
just joined
Posts: 6
Joined: Mon Apr 13, 2020 11:22 am

Re: Possible fix for hAP ac2 rebooting randomly

Tue Apr 14, 2020 1:23 am

If even 6.45.7 is not a long-term solution to the problem, a workaround could be to permit access to public NTP servers (and nowhere else) to just one of those devices and let the other ones synchronize from it.

I read this as: let some other device on your network run NTP server. That's great as long as NTP server is crashing only hAP ac2s, but I think I have heard that the problem is more widespread than that.
I have to buy that device, I already bought a device which "can" be a NTP server.
No other device on my network can run NTP...
 
User avatar
bogoevski
just joined
Posts: 6
Joined: Mon Apr 13, 2020 11:22 am

Re: Possible fix for hAP ac2 rebooting randomly

Fri Apr 17, 2020 5:54 pm

Update:
Today it started with random reboots, even staying at v6.45.7 is not a solution.
This device is I believe, first ever Mikrotik's garbage.
 
netflow
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Sat Oct 01, 2016 3:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Apr 18, 2020 12:23 am

Could you guys try the following:
  • Connect to wifi with a smartphone and run speedtest using the app
  • While speedtest is running, attempt writing to unit flash (eg: by running a backup)

Using this 2 simple steps I can crash it pretty much on demand with any ROS version up to 6.47Beta54 within less than 10 minutes. You might need to start over 2-3 times to find the right spot.
I suspect NTP package do periodically write to flash.

Without any wifi traffic or low traffic I can run days without issues.

REM: Actually unit is in a weird state where it looses all connectivity but serial connectivity. If you have watchdog enable it will no longer be able to ping anyone else and reboot itself. Using the serial console and inspecting the interfaces while the ethernet connectivity is lost does not reveal anything strange or suspicious. Unit looks absolutely normal except no packet can enter or leave the unit anymore.
Last edited by netflow on Sat Apr 18, 2020 12:28 am, edited 1 time in total.
 
Kampfwurst
Member Candidate
Member Candidate
Posts: 107
Joined: Mon Mar 24, 2014 2:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Apr 18, 2020 12:27 am

I have v6.45.7 running without problems.
Could you guys try the following:
  • Connect to wifi with a smartphone and run speedtest using the app
  • While speedtest is running, attempt writing to unit flash (eg: by running a backup)
Using this 2 simple steps I can crash it pretty much on demand with any ROS version up to 6.47Beta54 within less than 10 minutes. You might need to start over 2-3 times to find the right spot.
I suspect NTP package do periodically write to flash.

Without any wifi traffic or low traffic I can run days without issues.
I have connected a usb drive for the logs. So this can be the reason why I have no problems
 
User avatar
macsrwe
Forum Guru
Forum Guru
Posts: 1007
Joined: Mon Apr 02, 2007 5:43 am
Location: Arizona, USA
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sat Apr 18, 2020 1:58 am

Using this 2 simple steps I can crash it pretty much on demand with any ROS version up to 6.47Beta54 within less than 10 minutes.

Is this specific to hAP ac2, or is it reproducible on arbitrary models?
 
aradicev
just joined
Posts: 12
Joined: Tue Apr 29, 2014 5:39 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Apr 18, 2020 2:03 am

I have connected a usb drive for the logs. So this can be the reason why I have no problems
I had a USB connected to which the router was logging and it crashed anyway until i uninstalled the NTP package.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sat Apr 18, 2020 2:51 am

Could you guys try the following:
  • Connect to wifi with a smartphone and run speedtest using the app
  • While speedtest is running, attempt writing to unit flash (eg: by running a backup)

Using this 2 simple steps I can crash it pretty much on demand with any ROS version up to 6.47Beta54 within less than 10 minutes. You might need to start over 2-3 times to find the right spot.
I suspect NTP package do periodically write to flash.

Without any wifi traffic or low traffic I can run days without issues.

REM: Actually unit is in a weird state where it looses all connectivity but serial connectivity. If you have watchdog enable it will no longer be able to ping anyone else and reboot itself. Using the serial console and inspecting the interfaces while the ethernet connectivity is lost does not reveal anything strange or suspicious. Unit looks absolutely normal except no packet can enter or leave the unit anymore.
I have seen many reboots of hAP ac2 (with IPsec configured + NTP package installed) when no devices are connected to wifi.
And I am not sure whether the issue you have observed is related to the IPsec + NTP package issue.

Regarding your test case - can you try something like this and check whether the issue (reboot/lockup trigerred by simultaneous load on wifi and SPI flash) is reproducible afterwards?
/system resource irq set [find users="78b5000.spi"] cpu=2
/system resource irq set [find users="ipq4019_ahb"] cpu=3

To undo this change:
/system resource irq set [find users="78b5000.spi"] cpu=auto
/system resource irq set [find users="ipq4019_ahb"] cpu=auto
 
netflow
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Sat Oct 01, 2016 3:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Apr 18, 2020 11:56 pm

Thanks for suggestion however it crash exactly in the same way.
It failed once I executed
export compact file=flash/export
while doing a speedtest on wifi 5Ghz channel.

Regarding your test case - can you try something like this and check whether the issue (reboot/lockup trigerred by simultaneous load on wifi and SPI flash) is reproducible afterwards?
/system resource irq set [find users="78b5000.spi"] cpu=2
/system resource irq set [find users="ipq4019_ahb"] cpu=3
I connected to serial console, attempted to change cpu once it was in the weird state but it did not restore any connectivity.
Counter on 78b5000.spi is 1. So it does not seems to be used on my system. The only CPU assignable IRQ that seems to steadily increase counters on my system is IRQ 201.

I wonder what I could reset to revert to normal state (beside rebooting)...
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Apr 19, 2020 2:30 am

Thanks for suggestion however it crash exactly in the same way.
It failed once I executed
export compact file=flash/export
while doing a speedtest on wifi 5Ghz channel.

I connected to serial console, attempted to change cpu once it was in the weird state but it did not restore any connectivity.
Counter on 78b5000.spi is 1. So it does not seems to be used on my system. The only CPU assignable IRQ that seems to steadily increase counters on my system is IRQ 201.

I wonder what I could reset to revert to normal state (beside rebooting)...
There can be several different errors that lead to a "similar" crash. Unless we have some "debugging info" to compare, I think it is hard to tell whether it crashes in the same way or not.

Regarding the
/system resource irq set
commands - they are to be executed before creating the load (not when everything has already locked up).

Have you informed MikroTik about the reproduction scenario (doing export while wifi is active that leads to crash)? If no, I suggest to do that! You can also try to do this reproduction scenario on a router with default configuration (then you will know whether it has something to do with something in your config). Also I suggest to experiment with different wifi clients (only one client connected to wifi at the same time!) - maybe the issue has something to do with a particular wifi client you have!

Anyway - the mechanism for triggering the lockup/reboot in your case is very different from the IPsec + NTP package mechanism.
 
User avatar
bogoevski
just joined
Posts: 6
Joined: Mon Apr 13, 2020 11:22 am

Re: Possible fix for hAP ac2 rebooting randomly

Sun Apr 19, 2020 10:53 pm

I had two more HAP AC^2 at other location.
Got them at my desk this morning and found that all the 3 units I have are prone to the same reboot problem.
If I install NTP, even there are no sync requests from any device, they reboots frequently.

Mikrotik have to FIX THIS!

Angry, very angry!
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Apr 20, 2020 1:55 am

I had two more HAP AC^2 at other location.
Got them at my desk this morning and found that all the 3 units I have are prone to the same reboot problem.
If I install NTP, even there are no sync requests from any device, they reboots frequently.
Do you have IPsec configured on the devices that are rebooting?
Did you have NTP client (of the NTP package) configured on those devices?

I think you can try to make a supout file (if autosupout is not generated) after the reboot on test device and send the supout file to MikroTik (supout file might contain some sensitive information and it will contain "/export hide-sensitive" output).
 
aradicev
just joined
Posts: 12
Joined: Tue Apr 29, 2014 5:39 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Apr 20, 2020 1:58 am

Mikrotik have to FIX THIS!

Angry, very angry!
I agree with you, did someone informed them about the problem?
 
aradicev
just joined
Posts: 12
Joined: Tue Apr 29, 2014 5:39 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon Apr 20, 2020 2:09 am


Do you have IPsec configured on the devices that are rebooting?
Did you have NTP client (of the NTP package) configured on those devices?

I think you can try to make a supout file (if autosupout is not generated) after the reboot on test device and send the supout file to MikroTik (supout file might contain some sensitive information and it will contain "/export hide-sensitive" output).
In my case the IPsec was not configured and only the NTP client was configured. Actually my configuration is very simple, I'm using the device as AP and switch. The supout file was never generated after the crash. After I have uninstalled the NTP package everything is fine, no reboots since than.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon Apr 20, 2020 2:30 am

Mikrotik have to FIX THIS!

Angry, very angry!
I agree with you, did someone informed them about the problem?
I think additional bugreports (with supout files!) might be helpful for MikroTik.
 
User avatar
bogoevski
just joined
Posts: 6
Joined: Mon Apr 13, 2020 11:22 am

Re: Possible fix for hAP ac2 rebooting randomly

Mon Apr 20, 2020 8:46 pm

I think there is no need for additional bugreports.
MikroTik has to simulate the reported problem, if as I believe every HAP AC^2 have.
 
MetUys
newbie
Posts: 31
Joined: Mon Mar 17, 2014 1:19 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Apr 29, 2020 2:08 pm

Hi All,

I am also experiencing this problem: hAP ac2 rebooting randomly.
I originally thought it was a config issue as I was playing with the HA_Mikrotik setup (https://github.com/svlsResearch/ha-mikrotik), however I had read about the NTP suggestion and after disabling the NTP package (not uninstalling the package) the setup stabilized and has been fine for roughly 2 weeks.
running version v6.45.8 including firmware updated.

As for IPSec I have some config related to it, but not actively being used:
- L2TP Client with "use IPSec" set and IPSec secret present (but interface is disabled)
- PPP L2TP Server disabled but "use IPsec" is set to yes, and IPSec Secret configured.
- IPSec itself: only default in there (police/proposals/groups/profiles/Mode configs etc)

Reboot was happening late at night (every night for a few days), everyone sleeping so phones etc not doing any huge traffic. I dont have many cable connected devices, and those were all off at night.

I too want the NTP package so I have now rolled back to v6.45.6 (and firmware too), NTP package enabled and server configured (didnt actually need to configure it, settings were still present once it was enabled the package)
I chose this version to roll back to as I have been using the same config with NTP on it for a long time on hAP ac2 devices (bar the HA configuration side obviously).
Will update here if I pick up any issues.

As for NTP DDoS, its plausible... will circle back to that concept later if needed.
 
User avatar
macsrwe
Forum Guru
Forum Guru
Posts: 1007
Joined: Mon Apr 02, 2007 5:43 am
Location: Arizona, USA
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Wed Apr 29, 2020 11:41 pm

NTP DDoS is extremely low probability, since only one router model is reporting this issue so far (correct me if I am wrong). Also, to prevent DDoS, your firewall should be blocking "new connection" and "unrelated" NTP traffic from the WAN interface(s) (don't block all incoming traffic on the NTP port, or responses to your NTP client queries will never arrive).
 
MetUys
newbie
Posts: 31
Joined: Mon Mar 17, 2014 1:19 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon May 04, 2020 1:44 pm

Hi,
100% I think Public NTP attack is a low probability, but I believe in accurate feedback: I normally have NTP restricted, but I noticed I had disabled the "drop all !LAN" while doing the HA_mikrotik testing, so accurately it was plausible. I have re-enabled the restriction and no reboots whatsoever, but am still running the older version.
If anyone would like me to, I'm happy to roll up to the "6.45.8" version again, while leaving the restriction "drop all !LAN" in place and see if I get the reboots? We will then know with more certainty that its probably not an NTP exploit?
 
aradicev
just joined
Posts: 12
Joined: Tue Apr 29, 2014 5:39 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon May 04, 2020 3:17 pm

Hi,
100% I think Public NTP attack is a low probability, but I believe in accurate feedback: I normally have NTP restricted, but I noticed I had disabled the "drop all !LAN" while doing the HA_mikrotik testing, so accurately it was plausible. I have re-enabled the restriction and no reboots whatsoever, but am still running the older version.
If anyone would like me to, I'm happy to roll up to the "6.45.8" version again, while leaving the restriction "drop all !LAN" in place and see if I get the reboots? We will then know with more certainty that its probably not an NTP exploit?
I had this firewall filter ON and the router rebooted anyway. Also my router was not exposed to the internet, so it's not the result of an NTP exploit for sure.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Mon May 04, 2020 6:03 pm

I had this firewall filter ON and the router rebooted anyway. Also my router was not exposed to the internet, so it's not the result of an NTP exploit for sure.
I don't think the reboots are caused by malformed packets from NTP clients (to router's NTP server).
I had reboot problems on routers that allowed NTP connections only from LAN. Also I had reproduced reboots on router with RouterOS 6.43.x (back in year 2019) that was connected to the Internet via another router (so the problematic router was not reachable directly from the internet, and most likely it had no LAN NTP clients as well).
 
dzavy
just joined
Posts: 1
Joined: Wed May 13, 2020 1:01 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed May 13, 2020 1:08 pm

I can confirm the same applies to RBcAPGi-5acD2nD - I have 5 of them deployed at home, saw them crashing/rebooting couple times a day for no obvious reason. They have been super stable since I removed the ntp package. It was set to sync from internal router/firewall only so I'm pretty sure there couldn't have been any sort of NTP attack. There was no IPsec configured on them.
 
asavah
just joined
Posts: 1
Joined: Tue Dec 04, 2018 9:53 pm

Re: Possible fix for hAP ac2 rebooting randomly

Mon May 18, 2020 6:46 pm

This (ntp server issue) has been known for years.
I opened a ticket with mikrotik support about this at Tue, Oct 23, 2018, 8:06 AM
After a couple mails with support I was asked to try disabling some of the packages.
Oct 25, 2018, 9:15 AM
In some cases, multicast and ntp packages can make problems, but usually, it will be seen in debug logs in a suppout file. At this moment we can suggest to:
1) disable multicast;
2) enable multicast and disable ntp;
3) disable both ntp and multicast;
If reboots stop, then we will know what was an issue.
Since I disabled and after that uninstalled ntp package my hAP ac2 has been rock stable.

So they did know about this for years and still the issue is present.
Realy nice developers!
Even the bugs are stable.
Last edited by asavah on Mon May 18, 2020 9:51 pm, edited 1 time in total.
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Fri Jul 10, 2020 10:51 pm

This (ntp server issue) has been known for years.
I opened a ticket with mikrotik support about this at Tue, Oct 23, 2018, 8:06 AM
After a couple mails with support I was asked to try disabling some of the packages.
Looks like we have had the same issue since around the same time.
And that's why I like to publish such things in forums etc. :wink:
 
jandafields
Forum Guru
Forum Guru
Posts: 1515
Joined: Mon Sep 19, 2005 6:12 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 27, 2021 6:49 am

I just ran across this exact issue today. I am using NTP and IPSEC with HAP AC2

Is the current fix still to disable NTP? Or is there a better fix available now?
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sat Feb 27, 2021 8:45 pm

I just ran across this exact issue today. I am using NTP and IPSEC with HAP AC2

Is the current fix still to disable NTP? Or is there a better fix available now?
Removal of NTP package is still a working fix and I am not aware of other/better solutions.
 
jandafields
Forum Guru
Forum Guru
Posts: 1515
Joined: Mon Sep 19, 2005 6:12 pm

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 28, 2021 12:48 am

I just ran across this exact issue today. I am using NTP and IPSEC with HAP AC2

Is the current fix still to disable NTP? Or is there a better fix available now?
Removal of NTP package is still a working fix and I am not aware of other/better solutions.

Ok, thank you. Also, does this ONLY affect the HAPac2? Or does this affect all devices that RouterOS runs on?
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Feb 28, 2021 2:44 am

Also, does this ONLY affect the HAPac2? Or does this affect all devices that RouterOS runs on?
This issue was observed primary on hAP ac2, however a long time ago (before hAP ac2 were introduced) I had RB951G with very similar configuration and it rebooted around once a month (I don't remember more details, but I think the reboots "ended" when I changed RB951G to hAP ac2 with the same configuration). Whether reboots of RB951G were caused by this issue I have no idea.
So it is possible that this issue affects also other devices.
 
jandafields
Forum Guru
Forum Guru
Posts: 1515
Joined: Mon Sep 19, 2005 6:12 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Mar 02, 2021 11:58 pm

I removed the NTP package and only used SNTP CLIENT ... and it still rebooted after a day.

So, now I have disabled SNTP CLIENT to see what happens. Has anyone else had a reboot while using sntp client, but not having ntp package installed?
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Mar 03, 2021 1:27 pm

I don't think I have seen hAP AC2 rebooting.

I don't have IP Cloud time turned on.
I set the sntp client to use time.nist.gov, time.google.com

Maybe try those settings???
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Possible fix for hAP ac2 rebooting randomly

Wed Mar 03, 2021 6:11 pm

I didn't see my hAP ac2 reboot since 6.43.something. I've always had ntp package installed and active. I'm also using IPsec (implicitly, I'm running ipip tunnels). Current uptime with 6.47.9 is 20 days (since I upgraded ROS).
 
User avatar
Davis
Member Candidate
Member Candidate
Topic Author
Posts: 117
Joined: Mon Aug 01, 2011 12:27 pm
Location: Latvia, Riga
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Sun Mar 07, 2021 9:10 pm

I removed the NTP package and only used SNTP CLIENT ... and it still rebooted after a day.

So, now I have disabled SNTP CLIENT to see what happens. Has anyone else had a reboot while using sntp client, but not having ntp package installed?
In my case after removing NTP package from 2 hAP ac2 routers I have had only 1 random reboot (in total) during around 14 months (details about the reboot are described here, no random reboots have happened after that). Removal of NTP package helped for several other users as well (more details in the link above).

Maybe you have another problem, or maybe in some cases these reboots can happen also with built-in SNTP client.
How often do these reboots happen? Have you tried disabling SNTP client and time synchronization from cloud (and rebooting, and setting time manually)?
Have you contacted MikroTik support? There should be more information about the reboot in supout file (that should be created automatically after the random reboot or, in the worst case, can be created manually after the reboot).
 
jandafields
Forum Guru
Forum Guru
Posts: 1515
Joined: Mon Sep 19, 2005 6:12 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Mar 30, 2021 9:41 pm

With NTP package removed, and SNTP client disabled, it was still rebooting every 5 days. I sent several supout files to support, after some of these reboots, but they never could find the problem.

I think I have fixed it though, at least I am up to 11 days now without any reboots. I turned off the IP -> Cloud -> Update Time. I didn't even realize that was there.
 
rsalmar
newbie
Posts: 30
Joined: Mon Oct 23, 2017 6:22 am

Re: Possible fix for hAP ac2 rebooting randomly

Wed Apr 28, 2021 5:19 pm

jandafields - I'm curious if that fixed it? I am not using NTP and get random crashes, same symptoms. I've now turned this Cloud time off and will see. What exactly does this do?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Possible fix for hAP ac2 rebooting randomly

Wed Apr 28, 2021 6:12 pm

I removed the NTP package and only used SNTP CLIENT ... and it still rebooted after a day.

So, now I have disabled SNTP CLIENT to see what happens. Has anyone else had a reboot while using sntp client, but not having ntp package installed?
In my case after removing NTP package from 2 hAP ac2 routers I have had only 1 random reboot (in total) during around 14 months (details about the reboot are described here, no random reboots have happened after that). Removal of NTP package helped for several other users as well (more details in the link above).

Maybe you have another problem, or maybe in some cases these reboots can happen also with built-in SNTP client.
How often do these reboots happen? Have you tried disabling SNTP client and time synchronization from cloud (and rebooting, and setting time manually)?
Have you contacted MikroTik support? There should be more information about the reboot in supout file (that should be created automatically after the random reboot or, in the worst case, can be created manually after the reboot).

Who install separate packages on hAP?

ntp is not default package and must be installed apart.

I'm using hAP ac2 on more than 200 clients, as PPPoE access over Cambium 450, and no one randomly reboots...
And I do it because Cambium software is really shit for my opinion.
 
SVD
just joined
Posts: 3
Joined: Thu Jul 15, 2021 6:19 pm

Re: Possible fix for hAP ac2 rebooting randomly

Thu Jul 15, 2021 6:27 pm

Had similar problems with my Hap AC2.
In my case - it was radiomodule overheat.
Wifi speed was degrading over time - and finally it becamed totally unresponding untill reboot.
Changed Tx power to 15dbm and problem was gone.
After disassembly I've found that thermal compound was totally dry and seems to be not enough. So I've repaced it with high quality one and added vent holes to the case - will test it further.
 
SVD
just joined
Posts: 3
Joined: Thu Jul 15, 2021 6:19 pm

Re: Possible fix for hAP ac2 rebooting randomly

Tue Jul 27, 2021 12:55 pm

And no problems so far with default Tx power. Also it's much colder now.
You do not have the required permissions to view the files attached to this post.

Who is online

Users browsing this forum: coreshock, johnson73 and 72 guests