I'm curious if anyone else has tried to create a truly minimal MetaROUTER image.

"Truly minimal", in this context, would mean a non-Linux firmware image whose sole function would be to service the "VIRQ" virtual interrupts associated with the virtual interfaces (and optionally the virtual console).

Such an approach would yield tiny RAM usage and use less CPU time, allowing multiple Meta images even on tiny devices like the RB750. There doesn't seem much point in running many megabytes of virtualized Linux code when all that is needed is packet processing.

The barriers to this seem to be trying to infer the VIRQ interface API and relevant hardware-specific initialization by deciphering the Mikrotik patches to Linux. I'm not aware of any formal documentation, and I presume that it might change over time.

For anyone else interested, it seems like the best results can be achieved by searching the patched Linux kernel source code for "VIRQ_BASE" and "get_virq_nr". (Different code references VIRQ differently.) For the hardware-specific initialization, it is hard to tell what is relevant and what is just specific to features needed for Linux.

A MetaROUTER image appears to just be a .tgz file containing an uncompressed .elf file of the kernel plus any file system files.

Given this, a "truly minimal" image would just be a .tgz file containing an .elf file named "kernel" with the raw VIRQ service routine MIPS code.

I've implemented the minimal image that I spoke of the previous post.

The two most relevant things I've learned are:

1) Even though the memory-size parameter can be set as low as 4 (MBytes), NOTHING will run with less than 8 (Mbytes). Even a simple two instruction assembly program (less than a dozen bytes) won't run with less than 8 MBytes. Is there something that Mikrotik is doing in initialization that prevents operation with less than 8?

2) Even said two instruction assembly program (an endless loop with a "wait" instruction) will cause the Mikrotik device to regularly reboot itself due to watchdog timeouts. This is with a RB750 (no G), and only the G model is listed on the Manual page as having Random Freezing.

Testing was with a RB750 with RouterOS 5.24

In further testing, I've noticed a *huge* difference in MetaROUTER stability by changing guest code slightly.

An RB750 will watchdog reset in no more than a few minutes (and often much less than that) with this simple test program (in assembly):

kernel_entry:
wait
j kernel_entry

but seems to be quite stable with this test program:

kernel_entry:
j kernel_entry

I've done more complex tests enabling or disabling interrupts, using the console, etc., etc. However, it is the "wait" instruction that seems to be the common denominator in causing a watchdog reset.

Note that:

a) "wait" should be the right instruction to use to voluntarily free the CPU when there are no tasks to be performed.

b) The MIPS Linux kernel is replete with uses of "wait" (including Mikrotik's own patches for MetaROUTER).

So, this observation perhaps jibes with past observations that the problem is related to the frequency of interrupts (since "wait" in the MIPS 24K CPU core stalls the pipeline until the next interrupt).

It is difficult to blame the configuration of the guest code with something so straightforward. With such a simple program, Mikrotik should have a simple test case to debug their code.

For completeness, I've attached two MIPS MetaROUTER images for testing (a whopping 281 and 282 bytes respectively).

USE AT YOUR OWN RISK. DO NOT USE THESE ON A DEVICE IN SERVICE. BE FAMILIAR WITH USING NETBOOT BEFORE USING. ALTHOUGH IT IS PROBABLY UNLIKELY, YOU MAY DESTROY YOUR DEVICE.

loop-withwait.tgz should cause watchdog resets.
loop-nowait.tgz should run without issue.

Once the watchdog starts to fire (loop-withwait.tgz), it becomes tricky to recover the device. In my experience, it will only run for a few seconds before it reboots again, so recovery is a race to disable the MetaROUTER instance before the next reboot.

Note that they don't do anything functional. All one will see is that the status goes quickly from "booting" to "running".

The point here is to show a simple, clear example of the limitation in MetaROUTER (so that Mikrotik can fix it).

for how long should run this for watchdog timer to reboot the router.

Attempting to crash RB2011 running 6.0rc13

for how long should run this for watchdog timer to reboot the router.

Attempting to crash RB2011 running 6.0rc13

According to Mikrotik's documentation (see "Random Freezing"):

http://wiki.mikrotik.com/wiki/Manual:Metarouter

An RB2011 is supposed to be a unit without problems. So, perhaps it is not the right unit to test with.

I'm seeing watchdog resets no more than a minute or so apart on a RB750 (RouterOS 5.24)

all listed devices have to be without issues.

Tried your image on RB450G - it went on a reboot loop with pretty decent frequency. Test router delivered to developers.

btw, RB2011 still holds. and tests are done with RouterOS 6.0rc13

Great! It is reassuring that the test was reproducible, at least on some models.

you can try this build:
http://www.mikrotik.com/download/share/ ... .0rc14.npk

those that are no tin 6.0rc:
http://www.mikrotik.com/download/share/ ... .25rc1.npk

I used two RB750 units for testing.

I upgraded one from 5.24 to 5.25rc1; I upgraded the other from 5.xx to 6.0rc14.

The RB750 with 5.25rc1 ran the MetaROUTER image "loop-withwait.tgz" without issue (a definite improvement over 5.24). However, as with 5.24, the "memory-size" parameter had to be at least "8". Anything less than "8" resulted in the image status getting stuck in "booting", even though the image only addresses memory at a lot less than 8MB.

The RB750 with 6.0rc14 initially wouldn't run the MetaROUTER image "loop-withwait.tgz"; it claimed that there wasn't enough memory. It wouldn't allow a "memory-size" parameter less than "8". The system resources showed 10 available. After many minutes, it seemed to free up more memory, and it eventually allowed me to start the image. After that, it seemed to work without failure.

So, great work and thank you. However, it would be desirable (particularly if 6.0 is going to use more RAM) if MetaROUTER images could run with less RAM.

for 6.0 we will address issue of that 8MB is the minimum amount amount assigned

majbthrd,

Great work! It looks like this was especially helpful in tracking down a rather nasty mipsbe MetaROUTER bug.

Are you planning on sharing/publishing your source code and build environment/instructions for your truly minimal MR image? Sounds like a fascinating project.

-- Nathan

Thanks, Nathan. I'm working on an example in my spare time. I was reading through the Mikrotik feature request list for inspiration.

An example I found there was a request for a more configurable DNS server that allowed different servers to be polled for different Top Level Domains. Yes, this could be done with a 12-16MB Linux MetaROUTER image, but 1MB should be plenty for a custom minimal image.

Custom packet mangling and custom intrusion detection would be other examples of how a minimal image could be used.

When boiled down to its essence, MetaROUTER could be viewed as a lightweight way for RouterOS to hand packets to custom code, and for said custom code to hand packets to RouterOS. As such, I see MetaROUTER as a way to add complementary functionality to existing Mikrotik RouterOS functionality.

However, as long as there is a minimum 8MB memory limit, the appeal is somewhat lost. The concept seems much more compelling when one can allocate only, say, 1 or 2 MB.

Yeah, it does sound rather useful, although I can see that there would be pros and cons to both approaches. Most people who are looking to leverage the MetaROUTER image import feature are probably most interested in taking already-existing code and using it in a RouterOS environment in some way. Most existing code doesn't live in a bubble, and doesn't reinvent every conceivable wheel that it doesn't need to (an OS kernel, the ANSI C library, sockets/networking layers, and so forth), so (for example) they are written to use third-party code libraries, some of which might be pretty large themselves. If you are only going to be running one MetaROUTER instance, then making it as lean and as mean as possible makes a lot of sense. If you are going to be running multiple little isolated MetaROUTER instances, though, depending on what you are doing in each instance, that might potentially add up to a lot of duplicate code, both on disk as well as resident in memory. If all of your custom tasks share one MR instance, then those applets can also share code (shared libraries).

With your current "kernel", do you already implement the necessary hooks to access VIF interfaces that are presented to the guest, or is that one of your next steps?

-- Nathan

Thanks NathanA for linking to this thread, I too think this is a interesting topic.
Lots of thanks to majbthrd, as it really seems that your minimal two to three instruction images helped MikroTik to finally get a grip on their MetaROUTER hypervisor code.

Nathan, although I concur with you opinion, that reinventing the wheel isn't necessary, this might not be necessary. There are many librarieres for pretty minimal embedded systems. So these might be a good starting point. Right now I'm wondering if porting FreeRTOS and the LwIP stack to MetaROUTER might be a good idea...

majbthrd if you could share you minimal build environment, that would be a good starting point. My development skills with assembler are a bit rusty, but I might be able to pick up.

minimal amount of RAM assigned to guest will be reduced in furhter RouterOS builds. It will be 2MB for imported images.

It is a work-in-progress, but here is some source code:

http://code.google.com/p/metarouter-apps/

The "tincan" app monitors for the creation and removal of virtual interfaces. Upon a key-press from the console, it prints out a hex representation of a bit-field showing which interfaces are currently active.

Very cool; thanks for sharing. Are you planning to implement a lightweight IP stack yourself, or port one over from someplace else, or do you know yet?

-- Nathan

I don't know yet.

There is an applicable expression: "If you have a hammer, everything starts to look like a nail."

I do embedded design as a profession, and so was naturally curious to see what would be involved in getting a more "embedded" low-level version of code running in a MetaROUTER guest instance (perhaps regardless of whether or not it makes practical sense).

The only way to reach that goal was to pore over lots of the Mikrotik-patched Linux kernel code in order to learn how the guest virtual machine interacted with the host, and then start writing new code from scratch.

It was (and still is) a laborious and slow process. Normally in an embedded environment, one can toggle GPIO pins or something rudimentary to debug whether the code is actually running as expected. However, the virtualization (by design) traps any memory access outside the assigned memory range, so none of the peripherals are available. Also, hopefully (but not necessarily) an embedded environment has a documented interface, but the interface in this case had to be inferred from the source code and through lots of experimentation.

As timberwolf said, lwIP / uIP could be used as a lightweight network stack.

The dilemma here is that Linux rules as a generic programming platform. All the substantial underlying hard work has been done by many, many volunteers. At the cost of CPU and memory inefficiency, a simple program can take mere seconds to write, without the programmer having to worry about the important details of how everything was set up to make it happen.

So, in considering making some sort of generic platform for writing MetaROUTER apps, there is some (early?) threshold point where it just doesn't make sense to re-invent the wheel.

However, my (hammer-wielding?) gut tells me there *should* be some applications where it does make sense for a stripped-down piece of code (that might not even need a proper "network stack" per se). I'm just not sure where those threshold points are yet.