containerlab (https://containerlab.dev/) is on the run and it is an incredible new way of doing network labs. To run CHR it wraps it in a a docker container, which has some drawbacks especiall for large labs startup time but much better then running lets say within gn3. So my question to the folks at Mikrotik:
Are there any plans to release CHR as docker images instead of full blown VMs? A lot of the vendors already do this? This would be another huge step using CHR in containerized envs.
I wonder what would be the benefit? CHR, being primarily used for routing, needs access to network interfaces … as direct as possible for better performance. Even with VMs this might b ean issue due to virtualization layer (but it does pass NICs as piece of hardware). And the second issue might be even bigger issue: for certain functions is MT more or less UI for linux kernel functions (e.g. firewall is UI to iptables). In containers there is no “native” kernel, so kernel functions are usually not available.
It could be that some other vendors offer their “routers” in container environment … some vendors actually have all those functions implemented as userland executables and those are usually perfectly portable to container environment.
I think there’s a case for a containerized version of RouterOS to provide the UI on top of what, as you describe, are largly built in kernel functions. All containers are are syntactic sugar on top of lower level namespacing primitives, and before moving most of my network needs to physical Mikrotik devices, I had a large number of soft-routers running in containers that would just persistently claim hardware network devices. If you want to do exotic things there’s always Intel’s SR-IOV, but in general just doing a move of the device into the correct namespace is sufficient.
… which are very different kernel API than actual kernel API. I think many of us remember how much time took MT to get at least bare minimum of ROS v7 running … because of moving from kernel version 3.3.x to currently used 5.6.y. And IMO difference in API between both kernel versions is less than difference between kernel API and docker virtualization layer.
I still think that ROS is not fit for containerization, some other NOSes are much better candidates.
It is possible to run CHR with QEMU inside Docker container and in such way provided ROS kernel is used, that is probably doing container posted in #4. Still, I bet performance is be poor on such setup.
Yeah QEMU inside Docker is an “common” approach, to uncommon problem, today. I think Mikrotik has trouble supporting the various cloud hosting providers as it is, and Docker has whole ecosystem stuff folks would expect. While you used to be run RouterOS-on-RouterOS using even more basic MetaROUTER in version prior to V7, you cannot even run RouterOS under the new /container feature… so I don’t think Docker likely. But most container platforms do offer “host” mode for networking, which will bridge a network adapter. If some container cluster thing, theoretically, it could act as the VXLAN router for the cluster.
My RouterOS “schema diff” website at https://tikoci.github.io/restraml uses a GitHub Action, which automatically run some JavaScript/REST API calls to a CHR hosted inside GitHub runner, whole series of things to bring it. i.e. inside Microsoft servers for GitHub Actions, the build starts in a container, then runs QEMU, then Docker Compose to both build and run a Dockerfile/project for CHR on Docker EvilFreelancer/docker-routeros, with “build tasks” later involving JS via REST to extract all the commands using /console/inspect, and the CHR dies when the outer build container does. Then it repeats the process, including a reboot of CHR inside GitHub cloud, to install extra-packages. Anyhow, the build script are here, and link to rest of project to see how this is done: https://github.com/tikoci/restraml/tree/main/.github/workflows with a modified version of @EvilFreelancer’s Dockerfile that accepts an RouterOS version to re-build for a new version:
On performance… It takes it 5 minutes, majority ~4-8 minutes of extracting the data needed for https://tikoci.github.io/restraml – while there is a version that will connect to “real” router, same script take 1 hour to run against a RB1100AHx4. The GitHub runners are pretty modest x64, 4 cores / 16 GB. While it’s running, it makes at least ~50K REST APIs per build, and growing as the command surface grows. I run most beta/rc – since it’s automated – through that mess of stuff – with CHR inside two Dockers. Somehow in maybe 100+ various rc/builds, it’s always works.
So there is a “use case”, but I think most involve automated testing of scripts. Using a “real” VMs on desktop/server is likely better plan, than involving all the Docker things.
There is a bunch of use cases but mine is around labs and building digital twins with containerlab (https://containerlab.dev/) which is the greatest thing that has happend in labbing envs for ages in my opinion.
So today the approach is: download CHR.vmdk, convert to a “Docker” image with vrnetlab (https://github.com/vrnetlab/vrnetlab/tree/master/routeros). While that works it’s still quite resource intense (especially if you want a lab representing your real infrastructure (48port switches etc)) compared to platforms that provide docker images since it runs a VM inside of Docker shrug).
So please @mikrotik let us have our beloved CHR as a container image - I bet you internally have them already to do all kinds of testing CI/CD stuff..
Please @mikrotik, concentrate on actual hardware and make ROS full-featured and rock-solid. 99.999% of us don’t care about running ROS in network simulators. And we even less care about how heavy-weight might be running ROS in container environment.
If we go down to personal level … then it’s somebody else who’s trippin … by requesting something really marginal (both in terms of usability and demand). Even if MT devs spend only 5 seconds to get you container image of CHR it would be IMO 5 seconds wasted (over spending them on fixing routing functionality or something equally important). But I’m still convinced that they would have to spend lots of effort to provide “native container images” … and then there would be demand for images specific for 17 different containerization platforms.
And, BTW, 99.999% was beginning of next sentence: “99.999% of us don’t care about running ROS in network simulators.” … it doesn’t quantify the “full-featured and rock-solid” sentence.
Dude why are you even in this thread? It was clear from your first post that your horizon is very limited. Why publicly make that clear a second and third time? We got it thanks.
Sure few “users” of RouterOS may use a network simulator… If more folks tested RouterOS in simulators, generally - including Mikrotik – it might ferret out bugs quicker and reduce deployment issues. And, other vendors use simulators as a sales tool to show folks “see, you don’t have to use cisco”. i.e. why Nokia supports containerlab.
But the question is should they have some Docker image, at least for ANY lab testing or automation is pretty reasonable request. In fact, I’ve long thought that /container should be able to run RouterOS inside RouterOS - just like MetaROUTER. So was a “lost feature” in V7 IMO.
And beyond lab testing, there is VXLAN support. So for small-to-medium container environments, CHR might make a good container-based router. Different market, and Mikrotik does focus on too many virticles. But I long used CHR on every VMWare server at a customer, for the single purpose to enable remote management since CHR on host can see all the networks, and enable a VPNs to all the guest VM on host.
Now to @mkx quality points… CHR images already have issues on some hosting providers since they are not proper EUFI images, so some “real” VM hosts won’t boot CHR (or require complex steps, like dd or gdisk). So I don’t see Mikrotik making a Docker image for a while. First, the docs state: “Warning: Hypervisors that provide paravirtualization are not supported.”, which is what “Docker” uses. Plus, CHR is a licensed product, so managing the license is needed to get full speed from CHR - but that is tricky in the “toss-away” world of containers, since there outbound calls needed to add license and not API around CHR licensing. Without a license, you’re testing at 1MB/s speeds.
But I’d recommend you reach out the Containerlab team. I’ve never used it, but a quick check of the source shows they do have RouterOS support in code. See https://github.com/srl-labs/containerlab/tree/main/nodes/vr_ros. Now I can’t say it wired up, or works with V7… but someone did at least think to include RouterOS already. And there are OSS docker wrappers for CHR that run QEMU inside Docker, see posts above.
Sorry i might have been unclear in my statements. With the power of https://github.com/hellt/vrnetlab its easy to create a docker image and run it in containerlab from Mikrotik CHR vmdks. This works fine when running small test labs. This VM inside Docker works fine for a bunch of instances but when trying to go for real digital twins and a bit larger networks it becomes clear a real docker image would be much more scalable. Other vendors indeed have these options for quite some time and of course I think Mikrotik has the potential to come up with a a different licensing model in containerzed envs in contrast to what the CHR model is (although 1Mbit without license brings you quite far in terms of network automation and config testing).
That said many vendors already have docker images internally for their CI/CD piplines and testing. So maybe the same counts for Mikrotik. So in my opinion it should not hurt to ask if there are any thoughts or plans inside Mikrotik that go in this direction…