Community discussions

MikroTik App
 
woro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Sun May 24, 2015 12:47 am

Periodic crashes in 7.1rc4

Sun Oct 03, 2021 12:20 am

Hi,

I'm running 7.1rc4 on an RB4011 since a while.
At some point I noticed that it reboots periodically (really always somewhen after 3,5 - 4 hours runtime).
Bildschirmfoto_2021-10-02_23-14-19.png
Unfortunately I do not have much more information than that. I obviously could provide a supout but the forum says to report issues here and not to support@.
Overall my configuration is not too complex. The device is just my central home network router and capsman device for two additional hAP-ac^2. Those are running 7.1rc4 as well but do not crash. They have almost no configuration besides running as CAPs.

Currently I do not have logs from just before the crashes.
Any pointers how to debug further?
You do not have the required permissions to view the files attached to this post.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 8827
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 2:15 am

send a supout to Mikrotik. they should be interested........
I'd rather manage rats than software. Follow my advice at your own risk! (Sob & mkx forced me to write that!)
MTUNA Certified, by the Ascerbic Llama!
 
holvoetn
Member Candidate
Member Candidate
Posts: 148
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 10:54 am

What I find intriguing here is that after each crash it goes back to more or less the same "base" time.
Oct 1 2021, 21h58 and some seconds.

I would expect the internal clock goes up as the device operates normally ?
 
woro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Sun May 24, 2015 12:47 am

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 1:00 pm

Bildschirmfoto_2021-10-03_11-59-20.png
So that looks fine at least, right?
You do not have the required permissions to view the files attached to this post.
 
joegoldman
Long time Member
Long time Member
Posts: 672
Joined: Mon May 27, 2013 2:05 am

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 2:07 pm

Bildschirmfoto_2021-10-03_11-59-20.png

So that looks fine at least, right?
The config is fine - its obviously being set by ntp - but the curious point the other poster was making is that the boot time is always the same to ~2 days ago, all your 'reboot without proper shutdown messages' default back to oct 01 23:57:49 - is this when you upgraded/installed 7.1rc4? Sounds like some kind of memory leak and nothing is really saving on the underlying system so resets back to that point like it was refresh.

Have you tried a proper reboot cycle? Let it shutdown gracefully and come back up..
Alternatively I'd say do a net-install fresh install of rc4 and use a recent backup to restore config. Take a supout and sent it to support though in case its an error with the upgrade process or similar.
 
woro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Sun May 24, 2015 12:47 am

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 4:47 pm

No idea why after boot it falls back to that old date.
I performed a clean shutdown (actually more than one) in the meantime yesterday.
Also I installed 7.1rc4 a while ago not only at Oct 1st.
I'm also not totally sure but I think the problem appeared not immediately after the 7.1r4 update. I noticed it the first time after I installed a few extra packages. Not sure if I didn't notice before or it really started after that.
I installed container, zerotier and user-manager because I wanted to start playing with those. I didn't touch anything for those yet. First thing when I noticed my reboot issue I disabled and eventually removed those packages one after the other to see if it helps but it did not.

A supout I have sent today to support and a netinstall I can try (as soon as people around me can live a while w/o internet connection ;-))
 
msatter
Forum Guru
Forum Guru
Posts: 2319
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 5:02 pm

After an reboot I have a warning in Terminal that the time has been adjusted by the NTP-client.

Normal on a reboot the that is 1970-01-01 00:00 and that was on v6.

V7 could just like a RaspberryPI save the last know date and time when doing a reboot and use that as start from time. However if you boot your device weeks later that save data is well out of date.

It could be the same here that the last known time is saved and reused on boot in the time before a NTP sync or the user manually setting date and time.

In your case it seems that the saved time is never (re)set because the router hard crashed.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
holvoetn
Member Candidate
Member Candidate
Posts: 148
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 5:33 pm

In your case it seems that the saved time is never (re)set because the router hard crashed.
That was exactly the point I was making with my earlier remark.
Going on a stretch ... the crash might be related to this internal time not advancing as it should ? Some code not reading the 'real' time, or something related.
Speculation, I know.

For fun did a reset on one of my mAPLite's running 7.1rc4
[holvoetn@mAPLite91] > log print
 16:21:10 system,info router rebooted
 16:21:13 interface,info wireguard link up
 16:21:15 script,warning NETWATCH - WG Down
 16:21:15 system,info led trigger changed
 16:21:16 wireless,info 2C:C8:1B:47:0B:3D@wlan1 established connection on 2452000,
 SSID HomeLab
 16:21:17 dhcp,info dhcp-client on wlan1 got IP address 192.168.92.27
 16:21:23 system,critical,info ntp change time Oct/03/2021 14:21:23 => Oct/03/2021
 14:21:41
 16:21:55 system,info,account user holvoetn logged in from 10.255.255.1 via winbox
 16:22:00 system,info led trigger changed
 16:22:00 script,info NETWATCH - WG Up
 16:22:22 system,info,account user holvoetn logged in from 10.255.255.1 via telnet

[holvoetn@mAPLite91] > 
There is an NTP correction but that's only the time it took to reboot.
So why does the device of OP go back to Oct 1st ?

Something to try: go back to a previous version (downgrade) and then upgrade again to 7.1rc4 ?
Only the analysis of supout might tell what's happening ...
 
jookraw
newbie
Posts: 49
Joined: Mon Aug 19, 2019 3:06 pm

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 6:22 pm

Do you have the container package installed?

I'm starting to think that the container package is to blame. I had it installed on my hAP-ac2, it was rebooting all the time, but, seems that just removing does not fix it, just a full config reset.
I'll test my theory on my RB5009 that is not in "production" yet
 
woro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Sun May 24, 2015 12:47 am

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 6:39 pm

Yes, above I wrote that it feels like my problem started after I installed container, zerotier and user-manager. But removing those did not change anything. So this experience seems to match.
 
woro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Sun May 24, 2015 12:47 am

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 6:57 pm

I'm starting to think that the container package is to blame. I had it installed on my hAP-ac2, it was rebooting all the time, but, seems that just removing does not fix it, just a full config reset.
I'll test my theory on my RB5009 that is not in "production" yet
/system reset-configuration
worked for you?
 
jookraw
newbie
Posts: 49
Joined: Mon Aug 19, 2019 3:06 pm

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 7:13 pm

I'm starting to think that the container package is to blame. I had it installed on my hAP-ac2, it was rebooting all the time, but, seems that just removing does not fix it, just a full config reset.
I'll test my theory on my RB5009 that is not in "production" yet
/system reset-configuration
worked for you?
on the hAPac2 in the end I had to do a netinstall, because it was crashing too much, but there is another issue with the hapac2 that is l2MTU, that also make it crash, so, I had 2 issues that were locking up the device.
I just installed it on my RB5009 to test, lets see
 
msatter
Forum Guru
Forum Guru
Posts: 2319
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Periodic crashes in 7.1rc4

Sun Oct 03, 2021 7:36 pm

When reboot it will always sync just as when it crashes. The difference is that on a reboot the time is not that far apart, this because of when you reboot the time is written.

On a hard crash the current time is not written and is much more in the past. I assume Mirotik left this kind of 'canary' in there to detect normal reboots or crashes.

Big difference, crash an small difference, a manual reboot/triggered reboot.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
jookraw
newbie
Posts: 49
Joined: Mon Aug 19, 2019 3:06 pm

Re: Periodic crashes in 7.1rc4

Mon Oct 04, 2021 1:10 pm

My RB5009 is up for 18h after installing the container package, but it is arm64.
I'll install on my hAP-ac2 (that uses the same arch than RB4011) to see if it crashes...
 
holvoetn
Member Candidate
Member Candidate
Posts: 148
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: Periodic crashes in 7.1rc4

Mon Oct 04, 2021 3:03 pm

When reboot it will always sync just as when it crashes. The difference is that on a reboot the time is not that far apart, this because of when you reboot the time is written.

On a hard crash the current time is not written and is much more in the past. I assume Mirotik left this kind of 'canary' in there to detect normal reboots or crashes.

Big difference, crash an small difference, a manual reboot/triggered reboot.
I understand where you are going with your thinking but it is not logical.
Why on subsequent crashes always going back to Oct 1st even though OP said he did downgrade/upgrade after Oct 1st ? Upgrading also involves rebooting at least twice, no ? Package and firmware.
I did another test this morning when leaving home: took my mAP, simply pulled the plug and took it with me to work. I think that classifies as an unexpected shutdown.
Upon reboot 2 hours later, the log file correctly showed this unexpected event and upon NTP-start a correction of just a bit under 4 hours. More then almost 2 hours which my commute took this morning but not a difference of days.
 
msatter
Forum Guru
Forum Guru
Posts: 2319
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Periodic crashes in 7.1rc4

Mon Oct 04, 2021 10:06 pm

That four looks to a pivotal point here. The crashes are about fours apart and your last time set also went back four hours. Then maybe the updating of the time triggered the crashes of TS.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
woro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Sun May 24, 2015 12:47 am

Re: Periodic crashes in 7.1rc4

Mon Oct 04, 2021 10:27 pm

Just an update because I tried to do a netinstall of the device a few times now.

I really don't get netinstall to do anything. I followed the manual really very much in detail. I brought the device into netboot via /system routerboard settings set boot-device=try-ethernet-once-then-nand and had prepared netinstall. When I try to install, it shortly tells me "updating" but no progress bar or anything. Goes into "ready" state quickly. After a reboot everything is back as before.
I tried that often and now I'm scared to not be able to recover my device when doing the next step using the reset button to reset and enter netboot.
Tried on a spare RB2011 doing the same and it worked flawlessly with the same laptop and setup.
Will continue trying but need to have some Plan B to have connectivity... (to be continued)

BTW, support reacted but apparently no idea. Asked for a serial console connection which I cannot provide easily due to missing hardware.
 
msatter
Forum Guru
Forum Guru
Posts: 2319
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Periodic crashes in 7.1rc4

Mon Oct 04, 2021 10:49 pm

I have the same problem that Netinstall statred but does continue. Have you tried to run it as Administrator on Win10.

I solved the my 4011 being unusable with RC4 was to go back to 6.48.4 and fromntherecupgrade to v7RC4. It running smoothly now.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
woro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Sun May 24, 2015 12:47 am

Re: Periodic crashes in 7.1rc4

Tue Oct 05, 2021 11:10 am

Last night I downgraded to 6.48.4 and back to 7.1rc4 as mentioned by msatter.

Unfortunately still seeing exactly the same behavior.
 
intrepidsilence
just joined
Posts: 3
Joined: Thu Nov 02, 2017 8:56 pm

Re: Periodic crashes in 7.1rc4

Fri Oct 08, 2021 2:13 pm

I don't know if this is relevant to this discussion but I wanted to post it here anyway in case it leads to something useful.

After upgrading to 7.1rc4 it takes my RB4011GS somewhere between 6 and 18 hours to corrupt itself to the point that it must be reset and rolled back to rc3 before it starts working again.

For some reason it stop routing DNS traffic to my DNS server. It also seems to cause port flapping. The base routing out to the Internet still works. Very strange behavior and I was unable to capture anything that helped me pinpoint what was going on. I have gone through this a few times now and I feel that i have proven that it happens after the upgrade to rc4 so I won't be doing that again. I guess I will give it another go in the next rc if the relase notes have anything resembling an acknowledgement and fix.
 
Lilarcor
newbie
Posts: 45
Joined: Sun Oct 08, 2017 3:16 am

Re: Periodic crashes in 7.1rc4

Fri Oct 08, 2021 5:27 pm

I got same issue. Rb5009 v7.1rc4
 
woro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Sun May 24, 2015 12:47 am

Re: Periodic crashes in 7.1rc4

Fri Oct 08, 2021 6:02 pm

I got same issue. Rb5009 v7.1rc4
Same as in crashing/rebooting every 3h45m or same as in "starts dying after 6-18 hours"?

Just for the record. Finally connected a serial console a few minutes ago to possibly see something around the next crash/reboot.
 
woro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Sun May 24, 2015 12:47 am

Re: Periodic crashes in 7.1rc4

Sat Oct 09, 2021 12:27 am

Just for the record. This is all which shows up through the serial console after the "unexpected reboot":
MikroTik 7.1rc4 (testing)
MikroTik Login: :00000050
 
 
RouterBOOT booter 7.1rc4
 
RB4011iGS+5HacQ2HnD
 
CPU frequ..
 
loading kernel... OK
setting up elf image... OK
jumping to kernel code
Seems quite useless unless the 00000050 says anything.
 
ferilagi
just joined
Posts: 5
Joined: Mon Jun 01, 2020 6:35 am

Re: Periodic crashes in 7.1rc4

Tue Oct 12, 2021 1:53 pm

mine rb5009, crashed too, every 3-5 hours

Who is online

Users browsing this forum: oskarsk and 9 guests