Stopped containers respond to ping

RouterOS Containers
1

Hi,

This is a discussion I would like to start following my ticket SUP-153396.

Issue. Take a CHR or a real device with RouterOS v7.15 or older. Add any container (e.g. alpine) and start it. Make sure it responds to pings from the host and/or other machines. Then stop it and try to ping it again - you will get responses as if the container was up and running.

Implication. In my view, it’s weird and counter-intuitive to have stopped containers responding to pings. If I ping something and it responds, I normally treat it to be in a running state, definitely not stopped. No ICMP-based container monitoring can be performed this way.

Position of the support. Unfortunately, I didn’t manage to make the support agree this is a problem. They basically think RouterOS container system behaves as expected. Below I’m quoting Olegs’ responses.

Since the VETH interface always is up, and you have assigned an IP address to it, you can ping it. Unless you disable VETH interface and IP address, ping will receive/responses packets. This is how it works.

Comparison to Docker. My understanding is as follows. Both Docker and RouterOS use Linux network/user namespaces, create a pair of veth interfaces and assign IP addresses on two sides. Unlike Docker, RouterOS uses static and always enabled veth interfaces, unless manually disabled by the user. In other words, RouterOS doesn’t remove or disable veth interfaces, even the container side, when the container stops.

We cannot disclose all information about the container implementation in RouterOS. All we can say about the RouterOS container source code is that there is no similarity with Docker. RouterOS container for the most part based on the same Linux features. The biggest difference with standard Docker is that we use user namespace separation. Docker also has this feature, but it is not by default - > https://docs.docker.com/engine/security/userns-remap/

Community, if you use containers, do you think stopped containers should respond to pings? Could such an implementation of the container system, whatever MikroTik has under the hood of RouterOS, be considered acceptable?

Regards,

2

Your question is based on a misapprehension: that started containers respond to pings. They don’t. It’s the VETH that responds to pings, because it owns the IP you’re pinging. Since the VETH lifetime is independent of any containers attached to it, it responds to pings as long as it is up.

If you don’t see that this distinction matters, consider the following thought experiment:

If you have two containers running on the same VETH and stop one, should its IP go on pinging?

3

veth interfaces can respond by themselves to ICMP ping? did not know that. But apparently they do so.

4

Yes, you are right, that’s what the support is using for their defence. Then, please, answer one question: why should anybody want a VETH independent of any container? As for me, it’s a nonsense.

So, technically speaking, stopped containers respond to ping because the host’s kernel does it on both sides of veth. What I don’t understand, and the support refused to comment on, is why MikroTik decided not to turn a veth off when its container stops.

5

Didn’t know it’s possible to run 2 containers on the same veth. What’s the use of such a setup?

6

Apples and oranges.
Veth interface is something else then a container.

Stopped containers do NOT respond to ping. Veth interface does.
So technically you’re barking against the wrong tree, if you get my analogy …

7

I don’t think it’s a matter of “want” but more a reflection of the bare-bones nature of RouterOS’s container runtime. It has no equivalent of “podman network create” for example, much less the even more elaborate behviors of “docker network create”.

Instead, it leaves up to you, the network admin, the provisioning of the network, including the VETH used to transition from the container’s internal namespace to that used by the router’s running kernel.


why MikroTik decided not to turn a veth off when its container stops.

Because RouterOS’ container feature didn’t create it automatically in the background; you did, manually. The interface isn’t RouterOS’s to delete out from under you.

Contrast rootful Podman, which works more like you expect:


$ sudo podman network create --subnet 192.0.2.0/24 veth-test
$ sudo podman run --net veth-test --rm -it --entrypoint=/bin/sh alpine
/ # ip link 
1: lo: …blah blah blah…
2: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether ba:1b:95:94:89:81 brd ff:ff:ff:ff:ff:ff
/ # ip addr
…blah blah blah…
2: eth0@if4: …blah blah blah… inet 192.0.2.3/24 …

Hm! An at-sign! Strange. What happens on the host side?


$ ip link
…blah blah blah…
4: veth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman0 state UP mode DEFAULT group default qlen 1000
    link/ether f2:0e:c1:d2:e1:56 brd ff:ff:ff:ff:ff:ff link-netns netns-21047033-6b46-7304-88f1-6cff32d1b111

Yep, it’s a VETH, just like on RouterOS, only automatically created by the far more featureful Podman container runtime.

Realize that this is not the default situation, however, either with Podman or Docker Engine. Normally you take its default network, which is set up using NAT like the RouterOS docs show. If you start Alpine under that network, it’s your host IP on the outside, and it continues to ping when the container stops, doesn’t it?


What’s the use of such a setup?

What’s the use of having a router that responds to ping, SSH, HTTP (WebFig) and TCP port 8291 (WinBox) all on the same IP, and goes on pinging when you disable the WebFig service?

Part of your misapprehension is assuming there is a 1:1 correspondence between IPs and containers. There isn’t, and there should not be.

Instead of continuing to struggle against these truths, how about you tell us what service it is you’re monitoring for availability? Ping testing is an extremely crude liveness test. If you go looking around for container health check scripts for Docker and Podman and such, I think you will find that comparatively few do this. Instead, they will do something like issue an HTTP API call, asking it for some trivial service that can be done without having logged in first.

8

I liked your analogy!)

Let’s put it this way. I think, RouterOS container system architecture could have been improved in a way that veth loses its running state once its container stops.

9

Thanks for contributing your time to this issue and writing such a long message. I appreciate your opinion.

10

A couple of last messages made me think about another issue of only one veth allowed per container.

MikroTik seems to think of veth as a parent entity, and containers as child entities. This way, if a child entity stops, no parent entity should be stopped. It also means that a child entity can have only one parent entity.

I look at it in the opposite way. For me, a container is a parent entity, and a veth is a child entity. No child entity should be running, if a parent entity stops. And a parent entity can have multiple children.

11

The part you quoted isn’t an opinion.

Consider a Kubernetes node, running many containers. Typically, they’ll all be behind a single IP owned by the “ingress” container. When individual containers within that node die, the VETH underlying it all doesn’t die.

Regardless, you must keep in mind that container.npk is about an order of magnitude smaller than crun (100k vs 1.5MB), the bare-bones container runner at the heart of Podman. Add the rest of Podman (20-40MB), and it becomes two orders of magnitude size difference. Switch to Docker Engine as your comparison (400MB), and it’s another order of magnitude still. You cannot expect such a tiny container runner to behave in every single respect like Podman or Docker or K8S or Rancher or…

Take container.npk on its own merits. Use it like it was intended, not like “Docker in a Router”.

12

@vovan700i If you really want to achieve this for some reason, you may try with workaround and create short interval scheduler script which checks if container is running and if not disables its VETH interface. Ofc. for next container start VETH first needs to be enabled again while such scheduler is disabled (to avoid VETH to be disabled if triggers while container is not started), when container is started scheduler can be enabled also. This flow for starting container like that can be scripted also.

13

Honestly, OP did not explain why pinging a container is even useful or what problem should be solved. Even in full-blown Openshift/Kubernetes environments my healthchecks target against the exposed service and not a dumb, stupid ping that tells you nothing about the container/service health.

14

Thanks for sharing, thought of such a workaround myself.

15

Container monitoring was just an example. Will repeat, that as far as I’m concerned, it’s a nonsense to have a running veth with a stopped container. Because I treat container as a parent entity and its veth as a child entity.

16

If something is little, it doesn’t automatically mean it should be treated as true and correct. I explained in multiple messages above my view on containers and veths. Unfortunately, it doesn’t match the ones of MikroTik, other guys who responded here and yours.

But you mentioned a very interesting (undocumented?) feature of having multiple containers per one veth. Thanks for that. Will experiment with it and try to understand whether it could be of any use.

17

The more smarts you add, the bigger it gets. There isn’t room left even for something the size of crun on some ROS devices that support containers today, much less Podman scale or larger.

TANSTAAFL.

18

You are overthinking.
If you want pingable container just add extra ip address from same subnet in container manually. Ping that address

19

And 4 months later in 7.16 they seem to have silently implemented the approach I proposed here (or at least a part of):

*) container - clear VETH address on container exit and mark interface as running only when VETH is in use;

Thanks.