Horrible container performance from 7.14 up to 7.15rc2

RouterOS Containers
1

Ever since updating to 7.14, and all version I’ve updated to so far, my containers perform incredibly slowly.
(see updates below, it happens in 7.14 too)

    • A container will say it is running but takes 10+ minutes for the programs running in it to become accessible, sometimes it doesn’t even succeed even though the container is listed as “running”.


    • The response from the program running in the container is horribly slow


    • I have tried this with multiple different containers with the same result


    • I have tried using the local flash storage or a USB-attach SSD with the same result


    • I have completely removed all containers and started again, but it does not improve


    • Sometimes rebooting the router appears to fix it, but of the container is restarted while RouterOS is running the problem comes right back


    • CPU and Memory stats show that the containers take up very little CPU and Memory, so it’s not a problem with those resources.

Has anyone else experiencing this? I’m on a

RB5009UPr+S+

.

2

Check your container/config, maybe you have set an insufficient amount of ram?

ram-high	RAM usage limit.  ( 0 for unlimited)
3

The limit is 0, and memory utilization on the unit has shown less than 25% of ram utilized at all times.

It is definitely not a problem with resources being unavailable, as I mention in the post CPU has also very low.

4

Have you deleted and re-created the VETH interface used by the container?

Sometime stuff goes bad between versions with VETH interface I’ve found. Never slowness you’re describing. But still may be worth it to remove/add VETH interface, even with same IP.

5
  • Stopped Container


  • Changed container from veth1 to veth3


  • Deleted veth1


  • Re-created veth1


  • Changed container from veth3 to veth1


  • Started Container


  • No change in the problem :frowning:
6

I take it back, it’s not just 7.15. I am seeing the same problems on 7.14.2, and on a different device (hAP ax^3)

  • Containers start, and show running status


  • I can shell into them and ps to see their processes are running.


  • The IPs of the containers are pingable from outside


  • I can ping out of the container successfully


  • Netstat in the container shows no processes listening on any port

It will remain this way for potentially forever, never moving forward.

Unless I shell into the container and run the exact same command that is already running (from ps).

THEN…

  • The application starts up.


  • I can access it on the network.


  • netstat in the container shows the process is listening

However soon after the container is killed by routeros.

If I restart the container after this it is the same pattern as above and the service in the container may never become available.

This is reproduce-able across

  • multiple containers


  • multiple different container images (different applications that is)


  • multiple major version of ROS (7.14 & 7.15 tested)


  • multiple filesystems (local flash, or usb storage)

It is also worthy of note that the majority of the time, ROS fails to log any logs from any of the containers. Once in a while there will be a startup or a shutdown of a container that shows up in logs (info,containers,debug) but very inconsistently despite the same start and stop process being repeated while testing.

7

More info from inside the containers

I updated the entrypoint and cmd as needed to invoke strace as the first process that launches the program to capture a trace of the activity, so as to ascertain of the application is mis-behaving.

You can see the strace did start successfully and try to launch the program

/ # ps -ef
PID   USER     TIME  COMMAND
    1 root      0:00 strace -ffyo /etc/vmagent/strace.out /vmagent-prod -remoteWrite.url=http://xxxxxx:9009/api/v1/push -promscra
    9 root      0:00 strace -ffyo /etc/vmagent/strace.out /vmagent-prod -remoteWrite.url=http://xxxxxx:9009/api/v1/push -promscra
   10 root      0:00 ps -ef

You can it has written a file out already

/etc/vmagent # ls -la /etc/vmagent/
total 20
drwxr-xr-x    2 root     root          4096 May 28 00:04 .
drwxr-xr-x   20 root     root          4096 May 27 23:36 ..
-rw-r--r--    1 root     root            10 May 27 23:36 .type
-rw-r--r--    1 root     root           476 May 27 23:43 prometheus.conf
-rw-r--r--    1 root     root           199 May 28 00:05 strace.out.11

You can see that the program has barely gotten anywhere, a full minute after the container starts up

/etc/vmagent # cat /etc/vmagent/strace.out.11 
execve("/vmagent-prod", ["/vmagent-prod", "-remoteWrite.url=http://xxxxxx"..., "-promscrape.config=/etc/vmagent/"...], 0xfffffffffd50 /* 3 vars */) = 0
set_tid_address(0xeef210)               = 11
brk(NULL)                               = 0xf3e000
brk(0xf40000)                           = 0xf40000
mmap(0xf3e000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xf3e000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff7ffd000

10 minutes later… it has managed to make 10 more calls, when there should be hundreds of calls by now

execve("/vmagent-prod", ["/vmagent-prod", "-remoteWrite.url=http://xxxxxx"..., "-promscrape.config=/etc/vmagent/"...], 0xfffffffffd50 /* 3 vars */) = 0
set_tid_address(0xeef210)               = 11
brk(NULL)                               = 0xf3e000
brk(0xf40000)                           = 0xf40000
mmap(0xf3e000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xf3e000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff7ffd000
munmap(0xfffff7ffd000, 4096)            = 0
sched_getaffinity(0, 8192, [0 1 2 3])   = 8
openat(AT_FDCWD</>, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = -1 ENOENT (No such file or directory)
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff7fbe000
mmap(NULL, 131072, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff7f9e000
mmap(NULL, 1048576, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff7e9e000
mmap(NULL, 8388608, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff769e000
mmap(NULL, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff369e000
mmap(NULL, 536870912, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffd369e000
mmap(NULL, 536870912, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffb369e000

It will slowly continue to make calls at an unbelievably slow pace for hours and still never manage to complete the number of system calls for the application to startup.

It is as if ROS is throttling the container, except CPU and Memory on the system show there is no resource problem.


[admin@MikroTik] /system/resource> print            
                   uptime: 38m56s
                  version: 7.14.3 (stable)
               build-time: 2024-04-17 12:47:58
         factory-software: 7.5
              free-memory: 657.4MiB
             total-memory: 928.0MiB
                      cpu: ARM64
                cpu-count: 4
            cpu-frequency: 1320MHz
                 cpu-load: 1%
           free-hdd-space: 90.2MiB
          total-hdd-space: 128.0MiB
  write-sect-since-reboot: 2387
         write-sect-total: 388926
               bad-blocks: 0%
        architecture-name: arm64
               board-name: hAP ax^3
                 platform: MikroTik
                 
[admin@MikroTik] /system/resource> cpu/print 
Columns: CPU, LOAD, IRQ, DISK
#  CPU   LOAD  IRQ  DISK
0  cpu0  1%    0%   1%  
1  cpu1  1%    0%   1%  
2  cpu2  2%    1%   0%  
3  cpu3  1%    0%   0%
8

I regularlly use traefik container on my main test router, I have not seen any slowness. Now I don’t have a lot of traffic going to/from. But they’ve started quickly and worked in the various 7.15beta.

I know you’re saying all containers slow, on all platforms. But perhaps it’s something in some subset of containers. RouterOS does not offer all the “capacities” (–cap-add), so it possible code gets on a slow path in user mode since it does not have access to kernel for something… dunno.

But you’re saying things were working fine in 7.13, and just since 7.14 and now beta/rc’s it dramatically different. Can you repo it on older version and it’s fast? That be the kinda thing that proves if there is a bug.

9

Try to load system by running stress-ng inside container (eg. stress-ng --cpu 4 --aggressive). Does it loads all cpu cores to 100% when monitoring cpu from ROS /system/resource/monitor ?
You can also try with yabs.sh benchmark script inside container and share result.

10

FYI even Pi-hole does this and I know there are dozens of people at least running that.

11

Anything running in a /container/shell works as you would expect it to. I can try running stress-ng --cpu 4 --aggressive as the entrypoint on a container like I do with strace, but based on everything else I’ve tried it will most likely never even load it’s own code far enough to attempt to load the system.

12

I know there are folks with iperf container too. Just no reports on the forum, other than yours. Normally there a flurry if wide-spread issue…

I’m believe you’re seeing something. But you need one example, of one thing, that repo’s the slowness you’re seeing. The “all containers are slow”… but, not at shell… etc. is kinda hard to follow.

13

Maybe it’s the disk you are using?

14

Are you using one of the official container images for this, or is it something you’ve built locally?

15

As I stated higher up in my post, i’ve tried the local flash on the device as well as external storage. The storage strange has ranged from USB sticks, to SSDs, to NVME sticks.

16

yes, correct. I was using victoriametrics/vmagent for the container output I provided above.

I’ve also built custom images (not for victoriametrics but for other applications) and experienced the same problem with the first pid of the container appeared to be throttled, but any additional pid (ie: executing a shell into the container and running any commands after) works without any performance issue.

I’ve even tried making the first process launched in custom containers simply be an entrypoint script that launches the app afterwards, but the process that launched from the script (when it was eventually able finish running a two-line script at a snails pace) appeared to inherit the slowness.

17

As I’ve stated, I’ve tried this with multiple images of different applications and experienced the same result with the same output from the containers. I only posted output from one example container because the results were consistent with the other containers.

18

FWIW I have containers running on RB5009 (iperf, PiHole) and AX3 (netinstall) and none are experiencing performance issues.
Both devices have been following test build releases quite frequently (currently 7.15rc5).

Maybe it’s time to show your config ?

19

That, or a minimal reproducible test case that is expected to show the same result everywhere.

It’s unreasonable to expect third-party testers to set up complex things like VictoriaMetrics, but if you instead give us something that can be tried in under a minute without external dependencies, you will get many more eyes on your problem.

20

I got my ax^3 yesterday, updated to 7.15, and am also seeing this problem.

The only present container is linuxserver/duckdns:latest, and it takes 10 minutes to start. 2 env strings (domain name and token), no mounted dirs. Storage is USB2 flash (to avoid dealing with USB3 interference) that is fairly fast AFAIK.

It’s a very simple DDNS program. It would be awesome if someone tried to repro