Possible fix for hAP ac2 rebooting randomly

[Davis]

What is the average load on the devices that are rebooting?

In my case load on the rebooting devices was close to zero (well under 1 Mbps in many cases; below 10 Mbps in vast majority of cases).
The reboots did not seem to be correlated with load (reboots were not concentrated in peak-load time periods).

I have the same issue, from my perspective it looks like issue is related to specific wireless client combination (right mix), that more likely causes this issue, usually it takes blend of IPhones/Androids/Chromecasts and smart scale to cause that issue on average once every week.

Looks like this is another issue. In most cases one of my routers that was rebooting actually had 0 wifi clients connected.

Did you check if you are not maybe being DDoSed on port 123 (NTP) causing the reboots?

I am pretty sure I was not DoS-ed. In my case NTP was opened only from local network and during testing for RMA a router was connected to an isolated network.

You will not ever be able to determine the cause of the reboot in the forum. If your router is rebooting by itself, then please contact support@mikrotik.com providing supout.rif file and any details on how to reproduce the issue. If there is a specific workaround, provide details about that too. Only MikroTik staff can determine the precise cause of the problem and if it is caused by the software, then we want to fix this problem as soon as possible. If you even manage to workaround a problem based on other user experience, then the issue is still there and others suffer from it.
Please understand that the current topic information lacks consistency and might be different issues with the same result, thus provide only your own test and troubleshooting results. All bug reports are very welcome.

Thanks for looking into this! There is a recent support ticket regarding this issue from Traveller.

Most likely there is one common issue - random reboots on routers with IPSec and NTP package. Removal of NTP package has helped for 7 devices of 4 users (including myself).

According to my experience this issue might not be reliably reproducible on all hardware - for me one router was rebooting every few days, netinstall+running config script didn’t help at all, however (after some time though) swapping hardware (and doing exactly the same netinstall + running config script on the new hardware) seemingly “resolved” the situation (I was able to connect the rebooting router to an isolated network in another place and continue to observe reboots there, but the new hardware had no random reboots). After a couple of months the issue appeared on the new hardware as well. I.e. importing the config of a problematic router on a test device might not be enough to reproduce the issue in lab.
However, assuming that the hardware susceptible to this issue is spread evenly among all the hardware (or that “wear and tear” of a router increases probability to have this issue) I would suggest the following:

• Configure a pool of hAP ac2 routers with the config from Traveller’s ticket.
• Create the “other end” for all the IPSec tunnels. During my testing (on isolated network, during spring 2019, for RMA purpose) I had IPSec up and some traffic flowing from LAN side of the problematic router to the “other end” of IPSec (the other end just dropped all incoming IPSec encapsulated packets).
• Ensure that test routers can reach the NTP server(s) configured. During my testing I had internet connectivity for the problematic router, but I did not have any NTP clients.
• Hopefully some of the routers will reboot randomly…

I completely agree with “If you even manage to workaround a problem based on other user experience, then the issue is still there and others suffer from it.”, also I would like to add that it is not known whether there can be less frequent random reboots (that would be considered to be isolated cases with unknown causes by the users) also without NTP package. Also it is not known how widespread is this issue (I would guess that many users who have random reboots+NTP package+IPSec on hAP ac2 actually have this issue).
Looks like the issue might (this is just a guess - the issue might have been introduced earlier) have started in 6.43 (my observation is 6.43/6.43.1, Traveller’s observation is 6.43.15).

A few very technical points (that might serve as input for developers) I would like to make are:

• most likely there have been no code changes in NTP package in recent versions of RouterOS, however there have been IPSec related changes
• IPSec has some kernel code, NTP most likely doesn’t, and kernel code has many more ways for crashing/locking-up a system
• IPSec implementation might be sensitive to changes in system time (however I don’t know whether MikroTik implementation counts the remaining time of SAs in kernel or user mode, I don’t know how CLOCK_MONOTONIC and CLOCK_REALTIME are used etc.)
• NTP client of NTP package and the built-in NTP client (I guess part of system package) might use different Linux APIs (adjtimex(), settimeofday() etc.) to set the time
• logically unrelated changes in code can affect race conditions as memory layout changes can have notable effects on execution times of certain procedures (this could explain why the issue seemingly disappears in some versions and then comes back in later versions - of course this is just a guess)