Community discussions

 
Zeitkind
just joined
Topic Author
Posts: 5
Joined: Sat Jul 05, 2014 1:36 pm

RB2011 crashes when metarouter shuts down

Mon Jul 07, 2014 11:34 am

Hi,
Model: RB2011UiAS-2HnD, Firmware 3.10, ROS 6.15
When stopping a running metarouter (openwrt with kernel 3.3.8, 24MB RAM, two virtual NICs) via its console (halt), openwrt shuts down but also the RB2011 reboots with "router was rebooted without proper shutdown by watchdog timer".

Anyone seen this before? Known bug?

tia Hermann

PS: Unable to sign up at http://bugs.mikrotik-routeros.com, because I don't see any captcha.. :/
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24197
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: RB2011 crashes when metarouter shuts down

Mon Jul 07, 2014 12:14 pm

is not related to MikroTik, contact the site owner
No answer to your question? How to write posts
 
Zeitkind
just joined
Topic Author
Posts: 5
Joined: Sat Jul 05, 2014 1:36 pm

Re: RB2011 crashes when metarouter shuts down

Mon Jul 07, 2014 1:37 pm

Oh, though that site was from Mikrotik.. good to know!
Is there an official bugtracking site from Mikrotik?
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24197
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: RB2011 crashes when metarouter shuts down

Mon Jul 07, 2014 1:55 pm

No, you must contact support@mikrotik.com if you think you have stumbled on a bug.
No answer to your question? How to write posts
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: RB2011 crashes when metarouter shuts down

Mon Jul 07, 2014 1:59 pm

Hi,
on MIPSBE

When stopping a running metarouter using Shut-down via internal console) router reboots with "router was rebooted without proper shutdown by watchdog timer".
i left specific bug in quotes

as a workaround you can use /metarouter shut-down <mr>

That will shuwdown your guest safely.
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: RB2011 crashes when metarouter shuts down

Mon Jul 07, 2014 2:00 pm

 
jarda
Forum Guru
Forum Guru
Posts: 7602
Joined: Mon Oct 22, 2012 4:46 pm

Re: RB2011 crashes when metarouter shuts down

Mon Jul 07, 2014 2:25 pm

PS: Unable to sign up at http://bugs.mikrotik-routeros.com, because I don't see any captcha.. :/
Seems to be dead Web for one year. No need to sign in...
 
Zeitkind
just joined
Topic Author
Posts: 5
Joined: Sat Jul 05, 2014 1:36 pm

Re: RB2011 crashes when metarouter shuts down

Mon Jul 07, 2014 4:15 pm

Yeah, thx, found it. So shutting down is impossible for now. Wonder what fs is used, / get's mounted as roots with rw enabled.
I compiled attitude-adjustment (with fixed openssl etc.) some weeks ago, but it seems a good idea to not save any userdata inside the mr till this bug is fixed.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: RB2011 crashes when metarouter shuts down

Mon Jul 07, 2014 4:25 pm

see my post above.
 
Zeitkind
just joined
Topic Author
Posts: 5
Joined: Sat Jul 05, 2014 1:36 pm

Re: RB2011 crashes when metarouter shuts down

Mon Jul 07, 2014 5:13 pm

Tried that, but shutting down with metarouter shut-down mr1 doesn't work (for me). The state gets changed to shutting down, but the halt signal is simply ignored by the mr OS (openwrt build with kernel 3.3.8). Even worse, a following reboot inside the metarouter crashes the rb2011 (reboots).
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: RB2011 crashes when metarouter shuts down

Tue Jul 08, 2014 10:16 am

compile guest with ACPI support, as ACPI message of shutdown is sent by '/metarouter shutdown' at least RouterOS supports this.
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 793
Joined: Tue Aug 03, 2004 9:01 am

Re: RB2011 crashes when metarouter shuts down

Thu Nov 27, 2014 1:40 am

Janis,

I ran into this bug while trying to chase down a different problem with MikroTik Support (ticket #2014110766000602).

ACPI appears to be x86-only. So what equivalent option do I pick during kernel build configuration for both MIPS and PowerPC to support host-initiated shutdown?

Thanks,

-- Nathan
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 793
Joined: Tue Aug 03, 2004 9:01 am

Re: RB2011 crashes when metarouter shuts down

Mon Dec 01, 2014 3:59 am

as a workaround you can use /metarouter shut-down <mr>

That will shuwdown your guest safely.
Hey JanisK,

Unfortunately, this is not accurate. I ran into this bug on a RB450G while I was trying to hunt down a different bug (random "kernel failure" reboots that I thought might be related to MetaROUTER but which turned out to be a CAPsMAN problem in 6.20, which is now apparently fixed [EDIT: I now have reason to doubt that this was a CAPsMAN bug as originally claimed, but was instead simply a different manifestation of another MetaROUTER instability bug that keeps morphing and changing its symptoms between ROS versions and generally making a nuisance of itself; it might even be related to the bug being discussed in this thread]), and it does not matter if you ask RouterOS to tell the MetaROUTER to shut down with '/metarouter shut-down mr1', or if you tell the MetaROUTER itself to shut down. Either way can cause a crash. Crashes are random, and the bug doesn't always happen, so you were probably just lucky and all of your crashes during testing happened to you when you told the MetaROUTER to shut down in its own console, and not when you used the '/metarouter shut-down' command. But I have had the host RouterOS crash several times when shutting down with '/metarouter shut-down' in the same way, so that command is definitely not "safe" from the bug.

I got a RB2011UiAS-2HnD out to test further, and I can still reproduce this bug on 6.22. So this bug still has not been fixed. It is not limited to the RB2011, because as I said, it happens every once in a while on an RB450G that I have MetaROUTERs on. It doesn't happen as often or as easily on a 450G, though, so using an RB2011 to test for it and reproduce it is best (it happens quite often on the 2011!).

This is a very annoying bug because like the original posters both to this thread and the other one, I have discovered that if the RouterBoard gets rebooted when MetaROUTERs are still running, it can lead to filesystem corruption. So I need to shut the MetaROUTERs down if I ever need to pull the power on a RouterBoard, but shutting them down now risks a crash and reboot which can STILL cause corruption. Mostly this corruption only seems to impact the MetaROUTER, but occasionally I have even seen it corrupt some of the host's files. Shutting a MetaROUTER down cleanly is not super-important in-and-of-itself because MetaFS just stores the files from the MetaROUTER directly on the host filesystem, but there seems to be some kind of problem with YAFFS if a MetaFS is mounted and a clean shutdown is not performed that can lead to data corruption (probably another bug, and has been happening since at least RouterOS 5.x).

So this MR shutdown crashing really needs to get fixed. Please bring this bug back to the attention of the developers. It is very easy to test for and observe on an RB2011.

Oh, and by the way, it can also crash not just on MetaROUTER shutdown, but also on MetaROUTER reboot (either within the MetaROUTER console, or via '/metarouter reboot'). And I have even had it crash when just disabling the MetaROUTER ('/metarouter disable') without shutting it down! Bad, bad, bad...

(By the way, as far as the data corruption goes, so far I have never had a RouterBoard model that uses UBIFS instead of YAFFS have some data become corrupt if it loses power with MetaROUTERs still running. If RouterBOOT on all models supports UBIFS, you should give users the option of choosing UBIFS during Netinstall. It would be great to get a more stable filesystem on boards that use YAFFS now.)
compile guest with ACPI support, as ACPI message of shutdown is sent by '/metarouter shutdown' at least RouterOS supports this.
Unfortunately, this also turns out not to be accurate. "ACPI" is an x86-specific power management technology, and so no ACPI options will show up during kernel build if you are building for non-x86 architectures like MIPS.

There is a "power management" submenu in the build options, but because "ACPI" does not apply to MIPS architectures, I had no idea what options I was supposed to pick in order to make an OpenWRT kernel that could accept shutdown and reboot commands from the RouterOS host. (I also don't feel I got good support from MikroTik about the MR shutdown crash bug or about this specific question during my interactions with MT support in ticket 2014110766000602. I asked for information about how to make this work, and he didn't consult the developers, but just told me that MT has no control over OpenWRT images that other people make. Well, no duh. That wasn't my question. The person I exchanged e-mails with also seemed to think my 450G board must have a failing NAND that was to blame for my issues and didn't realize that these crashes were being caused by a bug that you guys already know about. Don't MT support people have access to an internal bug database that they can search so that we don't waste gobs of time going round and round about a bug that is already known to you?)

I eventually figured this out for myself, so I will post the solution for other people's benefit. It turns out that MetaROUTER has some kind of message passing capability between host and guest that has nothing to do with ACPI. The MT kernel patches that add MetaROUTER guest support already listen for "restart" or "halt" messages, and then take action by signaling the init process. The reason why most OpenWRT MR images do not actually shut down or reboot when they are passed that message is because the RouterOS init process apparently expects different signals for "restart" (SIGINT) and "halt" (SIGWINCH) than the OpenWRT init (busybox) does, which wants to hear SIGTERM for reboot and SIGUSR1 for halt. So all you have to do to enable this functionality is to make a slight change to the MikroTik kernel patches. Here is a patch made against the official OpenWRT MetaROUTER 1.2 patch for kernel 2.6.31.10, which most people should be able to easily adapt to whatever kernel and patches they are already using:
diff -p -r -d -u -N linux-2.6.31.10/arch/mips/metarouter/setup.c linux-2.6.31.10.new/arch/mips/metarouter/setup.c
--- linux-2.6.31.10/arch/mips/metarouter/setup.c
+++ linux-2.6.31.10.new/arch/mips/metarouter/setup.c
@@ -77,12 +77,12 @@ static irqreturn_t ctrl_interrupt(int ir
 		printk("RESTART\n");
 		init = find_task_by_pid_ns(1, &init_pid_ns);
 		if (init)
-			send_sig(SIGINT, init, 1);
+			send_sig(SIGTERM, init, 1);
 	} else if (strncmp(buf, "halt", len) == 0) {
 	    printk("HALT\n");
 		init = find_task_by_pid_ns(1, &init_pid_ns);
 		if (init)
-			send_sig(SIGWINCH, init, 1);
+			send_sig(SIGUSR1, init, 1);
 	}
 
 	return IRQ_HANDLED;
To use this patch, you don't even have to enable any "power management" features in your kernel config. It all just works with these changes.

Good luck,

-- Nathan
 
User avatar
bajodel
Long time Member
Long time Member
Posts: 545
Joined: Sun Nov 24, 2013 8:30 am
Location: Italy

Re: RB2011 crashes when metarouter shuts down

Wed Feb 18, 2015 10:17 pm

woow.. nice work Nathan!!

Who is online

Users browsing this forum: MSN [Bot] and 99 guests