and now rc7, still with no sign of the containers.
The good Mikrotik giveth, and the good Mikrotik taketh away…
To be fair, they probably are focusing on 7.1 stable. This means finishing features already on 6.49.1. And I think it’s better this way.
I like the idea of RoS dockers - but it’s far better to get a stable release first.
Thanks for the amazing new feature. I hope it is not too late to celebrate it.
This feature also emphasizes the lack of x64 OS support with a memory of more than 2GB.
Hi!
I just upgraded os up to 7.1 stable and there is no /containers at all
/system/package> /system/routerboard/print
routerboard: yes
model: 2011UiAS
serial-number:
firmware-type: ar9344
factory-firmware: 3.14
current-firmware: 6.46.4
upgrade-firmware: 7.1
Any ideas how can I use it?
Also what about CCRs - are container support exists?
for ARM-based boards it should be much simple to build docker images, using RaspberyPi for example
What’s new in 7.1rc5 (2021-Oct-25 20:15):
!) container - package is getting updated and will be made available in future, if interested in container feature please use 7.1rc4;
If you downgrade to 7.1rc4 you still musi install container package from “extra packages”
It’s been quite some time since 7.1rc4, has Docker support been dumped?
That solved my problem being stuck on “extracting”. Mikrotik instructions are not really good
Will docker be in a future release?
We will re-release container package soon. We wanted to improve security of the package, and make sure it does not have access to things it should not access. We will sandbox it more, re-evaluate all security aspects and will release it soon, when it is ready and completely secure.
Hi!
Just before starting, need to understand all steps.
Before asking questions, let me explain how the network in docker works (simplest case)
In Linux you always should configure 2 points - veth0 and veth1, for example. So it creates a “virtual patch-cord”, and after it is created you can place one end of this “patch-cord” into the network namespace, and do what you need.
Here is simple Linux example
ip link add dev <FIRST_END_OF_THE_VIRTUAL_PATCH_CORD> \
type veth \
peer name <SECOND_END_OF_THE_VIRTUAL_PATCH_CORD>
So after this command is executed you’ll have 2 interfaces:
<FIRST_END_OF_THE_VIRTUAL_PATCH_CORD>
and
<SECOND_END_OF_THE_VIRTUAL_PATCH_CORD>
and you can place one into a network namespace:
ip link set dev <SECOND_END_OF_THE_VIRTUAL_PATCH_CORD> netns <MY_NETWORK_NAMESAPCES>
Docker/k8s on Linux do exactly the same!
So you can see eth0 in docker which is another end of virtual patch-cord
Example
- Run any container
- enter container’s network namespace with command “nsenter -t -t” and check interfaces
PID here is the process ID of the process running inside the container (in my example below it is sshd)
(I build a container that runs sshd in an endless loop in order to be able to change config and restart sshd without container restart, it is not the best practice but good for testing)
ps -auxfw | grep sshd
root 2177 0.0 0.0 1496 384 ? Ss 11:44 0:00 \_ /bin/sh -c mkdir -p /run/sshd && while true; do /usr/sbin/sshd -D; sleep 10; done
root 2282 0.0 0.5 9216 4840 ? S 11:44 0:00 \_ sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
PID here is 2177
go to container’s network namespace:
nsenter -t 2177 -n
Check interfaces:
ip -o link show
22: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:ac:11:00:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0
Find peer’s ifindex:
ethtool -S eth0
NIC statistics:
peer_ifindex: 23
So we can see - peer index is 23 (and ifindex for eth0 inside container is 22)
Outside the container, you can see:
ip -o link
23: vethd6fa430@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default \ link/ether 9a:5e:37:b4:48:83 brd ff:ff:ff:ff:ff:ff link-netnsid 1
And find peer’s ifindex:
ethtool -S vethd6fa430
NIC statistics:
peer_ifindex: 22
So now we exactly know - eth0 (ifindex is 22) in container is connected to vethd6fa430 (ifindex is 23)
Next, docker daemon adds veth interface (in the system, not in the container) into the bridge
brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242006ddc40 no vethd6fa430
Check IP addresses:
ip a s dev docker0
7: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:00:6d:dc:40 brd ff:ff:ff:ff:ff:ff
inet 172.17.42.1/16 brd 172.17.255.255 scope global docker0
And inside container default route points on bridge’s IP : 172.17.42.1
# ip a s
22: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
inet 172.17.0.1/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
Check routing settings:
ip route
default via 172.17.42.1 dev eth0
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.1
The last question is: how ip address comes to the container? There is DHCP requests on docker0 interface, but container image may be built w/o dhcp client, so I guess IP is configured by docker daemon
Now let’s go back to Mikrotik
Docs reccomends to do
/interface/veth/add name=veth1 address=172.17.0.2/16 gateway=172.17.0.1
It creates a virtual eth interface but I do not understand how it works
- what is the gateway here?
- what is address here?
- why do we need ip address on SLAVE interface (member of the bridge docker)?
Create a bridge for containers and add veth to it
/interface/bridge/add name=docker
/ip/address/add address=172.17.0.1/16 interface=docker
/interface/bridge/port add bridge=docker interface=veth1
And, the last question is how to assign static addresses inside containers (one of the 172.17.0.0/16 network but predictable)?
I am not able to “map” Mikrotik commands to Linux commands.
Also do not see how to remove container files
After container is created , storage 2df45158-b892-4e06-af32-9ed00c0a1b9a is created and when container removed, I was not delete it
The workaround is router reboot - container storage was deleted after reboot
Some details:
[sirmax@RB4011iGS+5HacQ2HnD] /file> print detail
0 name="console-dump.txt" type=".txt file" size=471 creation-time=dec/25/2021 18:01:51 contents=
...
1 name="ubuntu-ssh-2.tar" type=".tar file" size=159.5MiB creation-time=dec/27/2021 14:22:21
2 name="2df45158-b892-4e06-af32-9ed00c0a1b9a" type="container store" creation-time=dec/27/2021 14:31:30
[sirmax@RB4011iGS+5HacQ2HnD] /file> remove 2
action failed (6)
The preview does work when I test this a while back, but I think they know there were missing holes and significant limitation in original support.
Totally get trying to plan/test stuff BEFORE containers in anticipation of an eventual release. But it’s hard to predict what and when containers come out again, but I do suspect it will be significantly different. If I was a betting man, I’m not sure you’d initially see complex network topology allowed directly in containers – rather, the interface be routable/bridgeable by the rest of Mikrotik’s functions. Idea of course is the Mikrotik is a router, so NFV is likely what they’re targeting, not being full-service, general container host.
If I was PLANNING a deployment, Mikrotik may simply not be able to run just any Docker/etc installer, for a variety of reasons. But if you do have something running in v7.1rc4, even if there are bug in disks, limits on networks, etc, you’d likely be pretty well set for containers when they come out.
For example, this poster reports that interprocess communications using SysV IPC conventions isn’t supported - didn’t test it myself, but believe him that yeah a few containers might not like some of Mikrotik’s kernel choices:
This would be enabled on a “regular Linux”, but perhaps not on a Mikrotik since they likely don’t need it for routing/etc (e.g. Mikrotik seems more BSD sockets and signals, than AT&T UNIX IPC people). While this poster suggests “turn this on” – it’s just not that simple since it might compromise something else on a RouterOS. I suspect there are other things like that may come up on why a specific container cannot run on a Mikrotik.
Totally all for containers, just seem early to be planning for deployment (starting testing I can see, but likely better when they re-release it)
will this be part of v7.2?
Finally I got container up and running on RB4011
Steps to reproduce:
Do docker pull ubuntu:focal on ARM (not ARM64!!!) host. My system is RaspberryPi
# uname -a
Linux raspberrypi-dcc 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux
# lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 9.8 (stretch)
Release: 9.8
Codename: stretch
Because of a bug in docker, I was not able to use docker build command so I started the container –privileged , do docker exec, and install sshd in it.
Also changed root password and allowed root login via ssh and login using password. (apt update do not work because blocked by security if container is not privileged and docker build has no --privileged option. Hope it is fixed on new systems, my one is pretty old)
After that I saved image using docker commit command
Next, I created a simple Dockerfile in order to add CMD:
FROM ubuntu-ssh-base:1
CMD while true; do /usr/sbin/sshd -D -e -ddd ; sleep 10; done
ubuntu-ssh-base:1 here is the name of the image I saved with docker commit
and run
docker build . -t ubuntu-ssh:1
docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu-ssh 1 4eea6dba0384 28 hours ago 157MB
ubuntu-ssh-base 1 fe2990098441 30 hours ago 157MB
...
As result, you’ll have image ubuntu-ssh :1
Save image to file:
docker save 4eea6dba0384> ubuntu-ssh-1.tar,
and upload it to router:
scp ubuntu-ssh-1.tar sirmax@192.168.1.22
(Please test in on linux before uploading to the router - just to know it works!)
On router:
(do not want /16 network for docker, another part is like example)
/interface/veth/add name=veth1 address=172.17.0.2/29 gateway=172.17.0.1
/interface/bridge/add name=docker
/ip/address/add address=172.17.0.1/29 interface=docker
/container/add file=ubuntu-ssh-1.tar interface=veth1 hostname=ubuntu1 workdir=/ubuntu logging=yes
Re-check:
/container/export
# dec/28/2021 21:03:49 by RouterOS 7.1rc4
/container
add file=ubuntu-ssh-1.tar hostname=ubuntu1 interface=veth1 logging=yes
And finally /container start 0
Now I’m able to log in via ssh to host 172.17.0.2 (it is a routed network)
What I found:
Kernel:
root@ubuntu1:/# uname -a
Linux ubuntu1 5.6.3 #15 SMP Mon Sep 20 07:31:39 UTC 2021 armv7l armv7l armv7l GNU/Linux
Cmd line
root@ubuntu1:/# cat /proc/cmdline
root=/dev/ram0 ubifs_support console=ttyS0,115200 benand_no_swecc=2 parts=1 boot_part_size=6291456 board=4011 Board=4011 board_rev=r2 ver=6.47.9 bver=6.47.9 hw_opt=00790002 boot=1 mlc=11
Modules
root@ubuntu1:/# cat /proc/modules
traffic_gen 24576 0 - Live 0x7f497000
al_crypto 90112 0 - Live 0x7f47b000
des_generic 16384 0 - Live 0x7f476000
libdes 24576 2 al_crypto,des_generic, Live 0x7f46f000
a9300_pci 16384 0 - Live 0x7f425000
sch_sfq 16384 4 - Live 0x7f298000
a9300 315392 1 a9300_pci, Live 0x7fbe7000
pcie_qcom 28672 0 - Live 0x7f467000
veth 24576 0 - Live 0x7f45c000
8021q 24576 1 - Live 0x7f455000
ipt_SAME 16384 0 - Live 0x7f450000
xt_NETMAP 16384 0 - Live 0x7f44b000
xt_REDIRECT 16384 0 - Live 0x7f446000
xt_MASQUERADE 16384 1 - Live 0x7f441000
xt_nat 16384 0 - Live 0x7f43c000
iptable_nat 16384 1 - Live 0x7f437000
ipt_TARPIT 16384 0 - Live 0x7f3db000
ipt_REJECT 16384 0 - Live 0x7f42f000
nf_reject_ipv4 16384 1 ipt_REJECT, Live 0x7f42a000
oprofile 28672 0 - Live 0x7f41d000
iptable_filter 16384 1 - Live 0x7f415000
nf_defrag_ipv4 16384 0 - Live 0x7f40d000
ipt_psd 16384 0 - Live 0x7f405000
ip_tables 24576 2 iptable_nat,iptable_filter, Live 0x7f3fb000
ipt_snif 16384 0 - Live 0x7f3f3000
snif 16384 1 ipt_snif, Live 0x7f3eb000
ipt_ulog 16384 0 - Live 0x7f3e3000
xt_tls 16384 0 - Live 0x7f3d3000
xt_layer7 20480 0 - Live 0x7f3ca000
xt_HL 16384 0 - Live 0x7f3c2000
xt_DSCP 16384 0 - Live 0x7f3ba000
xt_TCPMSS 16384 0 - Live 0x7f3b2000
xt_CT 16384 0 - Live 0x7f3aa000
xt_policy 16384 3 - Live 0x7f3a2000
xt_addrtype 16384 0 - Live 0x7f39a000
xt_hl 16384 0 - Live 0x7f392000
xt_realm 16384 0 - Live 0x7f38a000
xt_physdev 16384 0 - Live 0x7f382000
xt_length 16384 0 - Live 0x7f37a000
xt_connbytes 16384 0 - Live 0x7f372000
xt_helper 16384 0 - Live 0x7f36a000
xt_tcpmss 16384 0 - Live 0x7f362000
xt_dscp 16384 0 - Live 0x7f35a000
xt_hashlimit 20480 0 - Live 0x7f351000
xt_statistic 16384 0 - Live 0x7f349000
xt_string 16384 0 - Live 0x7f341000
xt_connmark 16384 0 - Live 0x7f339000
xt_conntrack 16384 5 - Live 0x7f331000
xt_multiport 16384 1 - Live 0x7f329000
xt_mark 16384 0 - Live 0x7f324000
xt_mac 16384 0 - Live 0x7f31f000
xt_tcpudp 16384 0 - Live 0x7f31a000
ts_kmp 16384 0 - Live 0x7f262000
xt_misc 20480 4 - Live 0x7f304000
x_tables 24576 38 ipt_SAME,xt_NETMAP,xt_REDIRECT,xt_MASQUERADE,xt_nat,ipt_TARPIT,ipt_REJECT,iptable_filter,ipt_psd,ip_tables,ipt_snif,ipt_ulog,xt_tls,xt_layer7,xt_HL,xt_DSCP,xt_TCPMSS,xt_CT,xt_policy,xt_addrtype,xt_hl,xt_realm,xt_physdev,xt_length,xt_connbytes,xt_helper,xt_tcpmss,xt_dscp,xt_hashlimit,xt_statistic,xt_string,xt_connmark,xt_conntrack,xt_multiport,xt_mark,xt_mac,xt_tcpudp,xt_misc, Live 0x7f2f9000
nf_nat_pptp 16384 0 - Live 0x7f2f1000
nf_conntrack_pptp 16384 1 nf_nat_pptp, Live 0x7f2e9000
nf_nat_sip 20480 0 - Live 0x7f2e0000
nf_conntrack_sip 28672 1 nf_nat_sip, Live 0x7f2d5000
nf_nat_h323 16384 0 - Live 0x7f2cd000
nf_conntrack_h323 45056 1 nf_nat_h323, Live 0x7f2be000
nf_nat_irc 16384 0 - Live 0x7f2b6000
nf_conntrack_irc 16384 1 nf_nat_irc, Live 0x7f2ae000
nf_nat_tftp 16384 0 - Live 0x7f2a6000
nf_conntrack_tftp 16384 1 nf_nat_tftp, Live 0x7f270000
nf_nat_ftp 16384 0 - Live 0x7f26b000
nf_conntrack_ftp 16384 1 nf_nat_ftp, Live 0x7f183000
nf_conntrack_ipv4 16384 0 - Live 0x7f29e000
qca9984_pci 77824 0 - Live 0x7f284000
nf_nat 36864 12 ipt_SAME,xt_NETMAP,xt_REDIRECT,xt_MASQUERADE,xt_nat,iptable_nat,nf_nat_pptp,nf_nat_sip,nf_nat_h323,nf_nat_irc,nf_nat_tftp,nf_nat_ftp, Live 0x7f276000
ath 143360 2 a9300,qca9984_pci,[permanent], Live 0x7f23e000
nf_conntrack_netlink 36864 0 - Live 0x7f234000
nfnetlink 16384 1 nf_conntrack_netlink, Live 0x7f17e000
rb 16384 1 - Live 0x7f22f000
vrf 20480 0 [permanent], Live 0x7f229000
ts 20480 1 - Live 0x7f223000
ticker 16384 1 - Live 0x7f18a000
al 53248 0 - Live 0x7f215000
rtl8367 24576 0 - Live 0x7f313000
stm8_fan 24576 0 - Live 0x7f30c000
prestera_dx_mac 466944 0 - Live 0x7f1a2000
poe_simple 16384 2 - Live 0x7f0e1000
ledgroup 16384 0 - Live 0x7f19d000
af_key 32768 0 - Live 0x7f190000
wlan 360448 6 a9300,qca9984_pci,ath, Live 0x7f125000
xfrm_user 28672 0 - Live 0x7f11d000
capsmanglue 12288 1 wlan, Live 0x7f01f000
btest 24576 0 - Live 0x7f116000
xfrm_algo 16384 2 af_key,xfrm_user, Live 0x7f0cd000
ulog 16384 2 ipt_ulog, Live 0x7f09f000
bridge2 131072 1 prestera_dx_mac, Live 0x7f0f5000
switch 57344 4 al,rtl8367,prestera_dx_mac,bridge2, Live 0x7f0e6000
phy_helper 45056 4 al,rtl8367,prestera_dx_mac,switch, Live 0x7f0d5000
packet_hook 131072 18 traffic_gen,xt_layer7,xt_misc,ath,al,prestera_dx_mac,wlan,bridge2,switch, Live 0x7f0ac000
tunnel6 16384 1 packet_hook, Live 0x7f0a7000
nf_conntrack 94208 26 xt_NETMAP,xt_REDIRECT,xt_MASQUERADE,xt_nat,xt_CT,xt_connbytes,xt_helper,xt_connmark,xt_conntrack,xt_misc,nf_nat_pptp,nf_conntrack_pptp,nf_nat_sip,nf_conntrack_sip,nf_nat_h323,nf_conntrack_h323,nf_nat_irc,nf_conntrack_irc,nf_nat_tftp,nf_conntrack_tftp,nf_nat_ftp,nf_conntrack_ftp,nf_conntrack_ipv4,nf_nat,nf_conntrack_netlink,packet_hook, Live 0x7f087000
jiffies 16384 44 - Live 0x7f082000
ipv6 319488 56 vrf,bridge2,packet_hook,[permanent], Live 0x7f033000
nf_defrag_ipv6 16384 2 nf_conntrack,ipv6, Live 0x7f02e000
logring 32768 136 - Live 0x7f023000
unix 32768 227 - Live 0x7f016000
panics 16384 0 - Live 0x7f00e000
flash 40960 10 a9300,qca9984_pci,rb,ts,al,stm8_fan,prestera_dx_mac,poe_simple,phy_helper,panics, Live 0x7f000000
Known issues and workarounds:
1 - I recreated container few times and only first creation really untar file (checked disk usage with /system/resource/print)
Workaround: changed workdir and container name (from ubuntu → ubuntu1)
2 - /file/print sees files during untar:
/file/print
# NAME TYPE SIZE CREATION-TIME
0 console-dump.txt .txt file 471 dec/25/2021 18:01:51
1 autosupout.old.rif .rif file 287.3KiB dec/28/2021 19:55:24
2 ubuntu-ssh-1.tar .tar file 158.5MiB dec/28/2021 20:17:33
3 2df45158-b892-4e06-af32-9ed00c0a1b9a container store dec/28/2021 20:19:58
4 autosupout.rif .rif file 360.4KiB dec/28/2021 20:20:17
5 e4ed5448-a0aa-4cad-b1ee-95babd14d70f directory dec/28/2021 20:20:30
6 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/boot directory apr/15/2020 13:09:51
7 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/dev directory sep/21/2021 19:13:02
8 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc directory dec/28/2021 20:20:30
9 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/adduser.conf .conf file 3028 sep/21/2021 18:53:34
10 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/alternatives directory dec/28/2021 20:20:30
11 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/alternatives/README file 100 mar/23/2020 08:12:54
12 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/apt directory dec/28/2021 20:20:30
13 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/apt/apt.conf.d directory dec/28/2021 20:20:30
14 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/apt/apt.conf.d/01-vendor-ubuntu .d/01-vendor-ubuntu file 92 apr/09/2020 12:21:07
15 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/apt/apt.conf.d/01autoremove .d/01autoremove file 630 apr/09/2020 12:21:07
16 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/apt/apt.conf.d/01autoremove-kernels .d/01autoremove-kernels file 623 sep/21/2021 18:53:36
17 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/apt/apt.conf.d/70debconf .d/70debconf file 182 aug/03/2019 12:51:13
18 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/apt/apt.conf.d/docker-autoremove-suggests .d/docker-autoremove-suggests file 44 sep/21/2021 19:12:58
19 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/apt/apt.conf.d/docker-clean .d/docker-clean file 318 sep/21/2021 19:12:58
20 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/apt/apt.conf.d/docker-gzip-indexes .d/docker-gzip-indexes file 70 sep/21/2021 19:12:58
21 e4ed5448-a0aa-4cad-b1ee-95babd14d70f/etc/apt/apt.conf.d/docker-no-languages .d/docker-no-languages file 27 sep/21/2021 19:12:58
It would be great to run PRIVILEGED container and do insmod 
with own module
Or join RB as k8s worker node
Am I missing something, because I can’t see container.npk in extra packages for 7.1.1 nor for 7.2rc1?
Was it removed for now?
Hello,
I just want’d to try out the container when i saw this.
I did it anyway on the 7.1rc3 and noticed something strange.
Does the veth network range must be only 172.17.0.0/16 to get container to work? i tried 172.172.0.0/16 and got nowhere. issue is that for people having docker on their computer that make the container ip range not accessible on mikrotik devices. And if you want container on two mikrotik devices you get the same issue.
Thanks for any details on that part
Probably in most applications of a docker container running on the router, there will be dstnat/srcnat rules to translate between the address used on the local network and this docker subnet.
(of course it is not really required, but it seems that many home network admins feel uncomfortable with the concept of routing and solve everything using NAT)
Still, it would of course be best when the subnet can be configured arbitrarily. It would be a bad idea to set it to 172.172.0.0/16 but there are lots of other addresses that could be used (in one of the reserved ranges for local addresses).
You can use any address but you must remove and re-create the container after you changed any configuration including network configuration.