Bug? Custom MetaROUTER kernels no longer work on PPC w/ ROS6

As the subject says, it would appear that something changed in RouterOS 6 for PowerPC that broke the ability to run custom Linux kernels built with MikroTik’s MetaROUTER patches (v1.2).

RouterOS 6 for MIPS still works fine: I can continue to use my custom images on MIPSBE boards after upgrading from RouterOS 5 to RouterOS 6 without a problem. But on PPC, a custom image will not boot. Nothing ever shows up on the console. And it is not just that the console is broken: this image automatically runs a DHCP client on eth0, but after waiting over 10 minutes, no network traffic has been seen coming from the VIF attached to the guest. So I know that it is not even running.

When I downgrade back to 5.26, the exact same images start working again.

I have no idea what on earth might have changed that would break just custom PowerPC kernels and not the kernel that comes with RouterOS. I examined the PPC MetaROUTER guest code in MikroTik’s patches to Linux kernel 3.3.5 (the version that RouterOS 6 is built on top of), and was surprised to see that there are no significant differences between that code and the v1.2 MetaROUTER patches for 2.6.31, and certainly none that would explain why these kernels won’t boot under RouterOS 6’s MetaROUTER host. (Specifically, I looked at arch/powerpc/platforms/85xx/metarouter.c and arch/powerpc/include/asm/vm.h)

Furthermore, I have successfully compiled Linux 2.6.35 PPC MetaROUTER guest kernels from MikroTik’s own 2.6.35 patches, and these kernels boot just fine under RouterOS 5, but these same kernels also won’t boot up under RouterOS 6. I compared the MetaROUTER guest code between these 2.6.35 patches and the 3.3.5 patches, and there are NO differences at all between them! The code is exactly the same.

Is there anybody else out there who is familiar with Linux on PowerPC and who has sharper eyes than I who might be able to tell me either what I’m doing wrong or how the issue can be worked around? The fact that nothing even shows up in the console makes it incredibly hard to know where to start…

I ran my tests on an RB1100 (800MHz, single-core). I can supply test MetaROUTER guest images upon request.

– Nathan

Hello,
I have the same problem on my RB800 ROS6.5 . I tried four metarouter builds and none of them was working.
On RB951G-2HnD ROS6.4 there is no problem with metarouter…

Rajo

please look in the forum where users report till what build number you can use openwrt where patch from wiki applies cleanly.

Right. RB951G is MIPSBE, and non-RouterOS MetaROUTER guests still work in RouterOS 6.x. RB800 is PowerPC, and non-RouterOS MetaROUTER guests do NOT work in RouterOS 6.x.

janisk, you are missing the point. This has nothing to do with the OpenWRT patch and build process. I am not having problems building OpenWRT with official MetaROUTER patches. I am checking out a version of OpenWRT from their SVN that your official patch applies to cleanly. The problem is that the same OpenWRT TGZ that boots just fine on PPC RouterOS 5.x does not even boot on PPC RouterOS 6.x. Something changed in the PowerPC MetaROUTER host code, and MikroTik has not told us what that change is (the necessary guest changes are not included in the Linux 3.3.5 patches that MikroTik published).

MikroTik either needs to reverse the incompatible change in RouterOS 6.6 so that older OpenWRT images continue to work, or publish the guest changes so that we can rebuild our PowerPC OpenWRT images to work under RouterOS 6.x (and hopefully the same images can work on both 5.x and 6.x, like they already do on MIPSBE).

– Nathan

I tried two PPC metarouter images on RouterOS 6.6, but both images still freezing at boot time (without any message)
(Both images working under 5.x)

2janisk: please could you try to run some metarouter on PPC routerboard ?

Update: Sergejs responded to my ticket (2013100466000193) on December 9: “Thank you very much for pointing that out. It will be difficult for us to look for the solution, however we will see what we can do.” So now we wait. :slight_smile:

(I’m not sure why it will be difficult…MikroTik’s own kernel runs just fine in RouterOS 6 MetaROUTER, so it seems like they obviously know how to make it work under some circumstances, and there must be something different about their kernel vs. ours.)

– Nathan

Crap. I just bought an RB800, and was going to use their MetaRouter tech to be able to run a more powerful version of OpenVPN, as well as AICCU. Hopefully they find the solution soon :frowning:

@NathanA, how do I get added to that ticket as an interested party?

For OpenWRT use RouterOS 6.15 where this problem is resolved.

Today is best day! I’ll give it a shot promptly!

MetaROUTER on PowerPC is broken again on 6.21, 6.21.1, and 6.22, and this time for all guests, including RouterOS. If you use MetaROUTER on PowerPC, do not upgrade to these versions. I opened a ticket (2014111366000271).

– Nathan

Thanks for posting here to let us know, I appreciate it.

FYI, RouterOS 6.23rc7 fixes this bug. Thanks, MikroTik devs, for getting this one turned around so quickly.

– Nathan

not-booting down, console-reconnection-hang to go.

Actually, RB1xxx reboots are probably more serious than the console bug, which is merely inconvenient because it means I have to make sure I stay away from the console and only talk to MetaROUTERs over the network. :slight_smile:

– Nathan

it was hard to test console bug without MetaROUTER guests booting.

Regarding RB1100 crashes - most probably we will wait for PPC SMP virtualization to arrive.

Well, since there is no way of knowing when that will arrive, I will still continue to run a few tests. If I make the bug easy to reproduce, perhaps you will discover that the fix is easy to code.

One thought: maybe it is related to PCI bus issues? On RB1xxx, some ethernet interfaces are PCI-based. On RB850Gx2, SoC ethernet is exposed via a custom bus/interface/API.

– Nathan


EDIT:

I’m not sure I understand your meaning here. Are you saying that you had trouble testing the console bug because your RouterBoard kept crashing and rebooting? If so, then actually, that is really funny…

I suppose you could just use a hacked RB850Gx2 to test for other bugs without constant crashing. :smiley:

RB800 is fine for MetaROUTER tests.

And not-booting at all i would not call a-constant-crashing.

Oh, you are talking about the fix that came in 6.23rc7. Got it. I wasn’t sure what you were referring to.

– Nathan

This means the RB800’s are good to go for MetaROUTER stuff? (well for the RC7 at least)

Thanks,
David