Community discussions

MikroTik App
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Horrible container performance from 7.14 up to 7.15rc2

Fri May 10, 2024 11:32 pm

Ever since updating to 7.14, and all version I've updated to so far, my containers perform *incredibly* slowly.
(see updates below, it happens in 7.14 too)
  • - A container will say it is running but takes 10+ minutes for the programs running in it to become accessible, sometimes it doesn't even succeed even though the container is listed as "running".
  • - The response from the program running in the container is horribly slow
  • - I have tried this with multiple different containers with the same result
  • - I have tried using the local flash storage or a USB-attach SSD with the same result
  • - I have completely removed all containers and started again, but it does not improve
  • - Sometimes rebooting the router appears to fix it, but of the container is restarted while RouterOS is running the problem comes right back
  • - CPU and Memory stats show that the containers take up very little CPU and Memory, so it's not a problem with those resources.
Has anyone else experiencing this? I'm on a
RB5009UPr+S+
.
Last edited by autonomous on Tue May 28, 2024 1:17 am, edited 1 time in total.
 
User avatar
sch
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Tue Feb 26, 2013 1:05 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Mon May 13, 2024 8:31 am

Check your container/config, maybe you have set an insufficient amount of ram?
ram-high	RAM usage limit.  ( 0 for unlimited)
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Wed May 15, 2024 5:15 am

The limit is 0, and memory utilization on the unit has shown less than 25% of ram utilized at all times.

It is definitely not a problem with resources being unavailable, as I mention in the post CPU has also very low.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4008
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Horrible container performance from 7.14 up to 7.15rc2

Wed May 15, 2024 5:51 am

Have you deleted and re-created the VETH interface used by the container?

Sometime stuff goes bad between versions with VETH interface I've found. Never slowness you're describing. But still may be worth it to remove/add VETH interface, even with same IP.
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Thu May 23, 2024 12:12 am

  • Stopped Container
  • Changed container from veth1 to veth3
  • Deleted veth1
  • Re-created veth1
  • Changed container from veth3 to veth1
  • Started Container
  • No change in the problem :(
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Tue May 28, 2024 1:17 am

I take it back, it's not just 7.15. I am seeing the same problems on 7.14.2, and on a different device (hAP ax^3)
  • Containers start, and show running status
  • I can shell into them and ps to see their processes are running.
  • The IPs of the containers are pingable from outside
  • I can ping out of the container successfully
  • Netstat in the container shows no processes listening on any port
It will remain this way for potentially *forever*, never moving forward.

Unless I shell into the container and run the exact same command that is already running (from ps).

THEN...
  • The application starts up.
  • I can access it on the network.
  • netstat in the container shows the process is listening
However soon after the container is killed by routeros.

If I restart the container after this it is the same pattern as above and the service in the container may never become available.

This is reproduce-able across
  • multiple containers
  • multiple different container images (different applications that is)
  • multiple major version of ROS (7.14 & 7.15 tested)
  • multiple filesystems (local flash, or usb storage)
It is also worthy of note that the majority of the time, ROS fails to log any logs from any of the containers. Once in a while there will be a startup or a shutdown of a container that shows up in logs (info,containers,debug) but very inconsistently despite the same start and stop process being repeated while testing.
Last edited by autonomous on Tue May 28, 2024 3:21 am, edited 1 time in total.
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Tue May 28, 2024 3:19 am

More info from inside the containers

I updated the entrypoint and cmd as needed to invoke strace as the first process that launches the program to capture a trace of the activity, so as to ascertain of the application is mis-behaving.

You can see the strace did start successfully and try to launch the program
/ # ps -ef
PID   USER     TIME  COMMAND
    1 root      0:00 strace -ffyo /etc/vmagent/strace.out /vmagent-prod -remoteWrite.url=http://xxxxxx:9009/api/v1/push -promscra
    9 root      0:00 strace -ffyo /etc/vmagent/strace.out /vmagent-prod -remoteWrite.url=http://xxxxxx:9009/api/v1/push -promscra
   10 root      0:00 ps -ef
You can it has written a file out already
/etc/vmagent # ls -la /etc/vmagent/
total 20
drwxr-xr-x    2 root     root          4096 May 28 00:04 .
drwxr-xr-x   20 root     root          4096 May 27 23:36 ..
-rw-r--r--    1 root     root            10 May 27 23:36 .type
-rw-r--r--    1 root     root           476 May 27 23:43 prometheus.conf
-rw-r--r--    1 root     root           199 May 28 00:05 strace.out.11
You can see that the program has barely gotten anywhere, a full minute after the container starts up
/etc/vmagent # cat /etc/vmagent/strace.out.11 
execve("/vmagent-prod", ["/vmagent-prod", "-remoteWrite.url=http://xxxxxx"..., "-promscrape.config=/etc/vmagent/"...], 0xfffffffffd50 /* 3 vars */) = 0
set_tid_address(0xeef210)               = 11
brk(NULL)                               = 0xf3e000
brk(0xf40000)                           = 0xf40000
mmap(0xf3e000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xf3e000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff7ffd000
10 minutes later... it has managed to make 10 more calls, when there should be hundreds of calls by now
execve("/vmagent-prod", ["/vmagent-prod", "-remoteWrite.url=http://xxxxxx"..., "-promscrape.config=/etc/vmagent/"...], 0xfffffffffd50 /* 3 vars */) = 0
set_tid_address(0xeef210)               = 11
brk(NULL)                               = 0xf3e000
brk(0xf40000)                           = 0xf40000
mmap(0xf3e000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xf3e000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff7ffd000
munmap(0xfffff7ffd000, 4096)            = 0
sched_getaffinity(0, 8192, [0 1 2 3])   = 8
openat(AT_FDCWD</>, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = -1 ENOENT (No such file or directory)
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff7fbe000
mmap(NULL, 131072, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff7f9e000
mmap(NULL, 1048576, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff7e9e000
mmap(NULL, 8388608, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff769e000
mmap(NULL, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffff369e000
mmap(NULL, 536870912, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffd369e000
mmap(NULL, 536870912, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffb369e000

It will slowly continue to make calls at an unbelievably slow pace for hours and still never manage to complete the number of system calls for the application to startup.

It is as if ROS is throttling the container, except CPU and Memory on the system show there is no resource problem.

[admin@MikroTik] /system/resource> print            
                   uptime: 38m56s
                  version: 7.14.3 (stable)
               build-time: 2024-04-17 12:47:58
         factory-software: 7.5
              free-memory: 657.4MiB
             total-memory: 928.0MiB
                      cpu: ARM64
                cpu-count: 4
            cpu-frequency: 1320MHz
                 cpu-load: 1%
           free-hdd-space: 90.2MiB
          total-hdd-space: 128.0MiB
  write-sect-since-reboot: 2387
         write-sect-total: 388926
               bad-blocks: 0%
        architecture-name: arm64
               board-name: hAP ax^3
                 platform: MikroTik
                 
[admin@MikroTik] /system/resource> cpu/print 
Columns: CPU, LOAD, IRQ, DISK
#  CPU   LOAD  IRQ  DISK
0  cpu0  1%    0%   1%  
1  cpu1  1%    0%   1%  
2  cpu2  2%    1%   0%  
3  cpu3  1%    0%   0%                   
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4008
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Horrible container performance from 7.14 up to 7.15rc2

Tue May 28, 2024 8:32 pm

I regularlly use traefik container on my main test router, I have not seen any slowness. Now I don't have a lot of traffic going to/from. But they've started quickly and worked in the various 7.15beta.

I know you're saying all containers slow, on all platforms. But perhaps it's something in some subset of containers. RouterOS does not offer all the "capacities" (--cap-add), so it possible code gets on a slow path in user mode since it does not have access to kernel for something... dunno.

But you're saying things were working fine in 7.13, and just since 7.14 and now beta/rc's it dramatically different. Can you repo it on older version and it's fast? That be the kinda thing that proves if there is a bug.
 
optio
Forum Veteran
Forum Veteran
Posts: 863
Joined: Mon Dec 26, 2022 2:57 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Tue May 28, 2024 9:42 pm

Try to load system by running stress-ng inside container (eg. stress-ng --cpu 4 --aggressive). Does it loads all cpu cores to 100% when monitoring cpu from ROS /system/resource/monitor ?
You can also try with yabs.sh benchmark script inside container and share result.
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Wed May 29, 2024 7:07 am

I regularlly use traefik container on my main test router, I have not seen any slowness. Now I don't have a lot of traffic going to/from. But they've started quickly and worked in the various 7.15beta.

I know you're saying all containers slow, on all platforms. But perhaps it's something in some subset of containers. RouterOS does not offer all the "capacities" (--cap-add), so it possible code gets on a slow path in user mode since it does not have access to kernel for something... dunno.

But you're saying things were working fine in 7.13, and just since 7.14 and now beta/rc's it dramatically different. Can you repo it on older version and it's fast? That be the kinda thing that proves if there is a bug.
FYI even Pi-hole does this and I know there are dozens of people at least running that.
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Wed May 29, 2024 7:08 am

Try to load system by running stress-ng inside container (eg. stress-ng --cpu 4 --aggressive). Does it loads all cpu cores to 100% when monitoring cpu from ROS /system/resource/monitor ?
You can also try with yabs.sh benchmark script inside container and share result.
Anything running in a /container/shell works as you would expect it to. I can try running stress-ng --cpu 4 --aggressive as the entrypoint on a container like I do with strace, but based on everything else I've tried it will most likely never even load it's own code far enough to attempt to load the system.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4008
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Horrible container performance from 7.14 up to 7.15rc2

Wed May 29, 2024 7:26 am

FYI even Pi-hole does this and I know there are dozens of people at least running that.
I know there are folks with iperf container too. Just no reports on the forum, other than yours. Normally there a flurry if wide-spread issue...

I'm believe you're seeing something. But you need one example, of one thing, that repo's the slowness you're seeing. The "all containers are slow"... but, not at shell... etc. is kinda hard to follow.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4008
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Horrible container performance from 7.14 up to 7.15rc2

Wed May 29, 2024 7:27 am

Maybe it's the disk you are using?
 
User avatar
tangent
Forum Guru
Forum Guru
Posts: 1586
Joined: Thu Jul 01, 2021 3:15 pm
Contact:

Re: Horrible container performance from 7.14 up to 7.15rc2

Wed May 29, 2024 3:59 pm

Are you using one of the official container images for this, or is it something you've built locally?
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Thu May 30, 2024 8:08 am

Maybe it's the disk you are using?
As I stated higher up in my post, i've tried the local flash on the device as well as external storage. The storage strange has ranged from USB sticks, to SSDs, to NVME sticks.
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Thu May 30, 2024 8:09 am

Are you using one of the official container images for this, or is it something you've built locally?
yes, correct. I was using victoriametrics/vmagent for the container output I provided above.

I've also built custom images (not for victoriametrics but for other applications) and experienced the same problem with the first pid of the container appeared to be throttled, but any additional pid (ie: executing a shell into the container and running any commands after) works without any performance issue.

I've even tried making the first process launched in custom containers simply be an entrypoint script that launches the app afterwards, but the process that launched from the script (when it was eventually able finish running a two-line script at a snails pace) appeared to inherit the slowness.
Last edited by autonomous on Thu May 30, 2024 8:16 am, edited 2 times in total.
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Thu May 30, 2024 8:11 am

FYI even Pi-hole does this and I know there are dozens of people at least running that.
I know there are folks with iperf container too. Just no reports on the forum, other than yours. Normally there a flurry if wide-spread issue...

I'm believe you're seeing something. But you need one example, of one thing, that repo's the slowness you're seeing. The "all containers are slow"... but, not at shell... etc. is kinda hard to follow.
As I've stated, I've tried this with multiple images of different applications and experienced the same result with the same output from the containers. I only posted output from one example container because the results were consistent with the other containers.
 
holvoetn
Forum Guru
Forum Guru
Posts: 6115
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: Horrible container performance from 7.14 up to 7.15rc2

Thu May 30, 2024 9:02 am

FWIW I have containers running on RB5009 (iperf, PiHole) and AX3 (netinstall) and none are experiencing performance issues.
Both devices have been following test build releases quite frequently (currently 7.15rc5).

Maybe it's time to show your config ?
 
User avatar
tangent
Forum Guru
Forum Guru
Posts: 1586
Joined: Thu Jul 01, 2021 3:15 pm
Contact:

Re: Horrible container performance from 7.14 up to 7.15rc2

Thu May 30, 2024 6:01 pm

Maybe it's time to show your config ?

That, or a minimal reproducible test case that is expected to show the same result everywhere.

It's unreasonable to expect third-party testers to set up complex things like VictoriaMetrics, but if you instead give us something that can be tried in under a minute without external dependencies, you will get many more eyes on your problem.
 
akabyshev
just joined
Posts: 4
Joined: Mon Jan 09, 2023 2:00 am

Re: Horrible container performance from 7.14 up to 7.15rc2

Sat Jun 01, 2024 5:21 pm

I got my ax^3 yesterday, updated to 7.15, and am also seeing this problem.

The only present container is linuxserver/duckdns:latest, and it takes 10 minutes to start. 2 env strings (domain name and token), no mounted dirs. Storage is USB2 flash (to avoid dealing with USB3 interference) that is fairly fast AFAIK.

It's a very simple DDNS program. It would be awesome if someone tried to repro
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4008
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Horrible container performance from 7.14 up to 7.15rc2

Sat Jun 01, 2024 7:49 pm

Maybe there is an issue, IDK.

But to clarify on the "duckdns" sub-case...

1. Does it show "starting" for 10 minutes, or it goes to "running" and just does respond?
2. Assuming logging is enabled, does anything appear during the 10 minutes?
3. Does "/container/shell [find tag~"duckdns"]" get you the terminal of the container during this window?

I just think some less subject measurement be helpful to narrow this down. Since duckdns is a dns server, using dns test tool be useful. On "Linux" there is `dnsperf`, (or with WSL `apt install dnsperf` on Windows, or `brew install dnsperf` on Mac w/homebrew)... I have an old /bin/sh script that ran it, which I adapted here:
{
FILE=~/dnsperf.queries
DNSSERVER=192.168.88.1
CONNECTIONS=20 
QUERIESPERSEC=100
DURATION=10
TESTHOST=google.com

echo "** DNSPERF TEST **"
echo "Running dnsperf to $DNSSERVER for $DURATION sec, using $CONNECTIONS connections, at max rate of $QUERIESPERSEC queries per sec."
echo "... using the following query file at $FILE:"
echo "$TESTHOST A" > $FILE   
cat $FILE
echo "Starting dnsperf...\n"
dnsperf -s $DNSSERVER  -c $CONNECTIONS -l $DURATION -Q $QUERIESPERSEC -d $FILE   
}
Anyway, I just think it good with the container to run some test tool against them - even outside the specific issue. This is generally not too hard since the whole idea of a container it does one things, so easy to test using off-the-shelf tools.

And, in fact some slight difference in how long "ssh" when running testing script is how the XZ vulnerability was caught.... And here some cut-and-paste of duckdns from 7.14 and 7.15 showing it changed, might get more action from Mikrotik since there is some concrete thing to point to.
 
akabyshev
just joined
Posts: 4
Joined: Mon Jan 09, 2023 2:00 am

Re: Horrible container performance from 7.14 up to 7.15rc2

Sun Jun 02, 2024 10:43 pm

Sorry - duckdns is not a server, just a simple API caller for some DynamicDNS service. Useful script nevertheless, thank you!
Yesterday, that container was shown as "started" immediately, logging was enabled and it took 5+ minutes for very first log lines to start coming.

BUT HEAR ME OUT - today containers are working normally! Container-specific log messages start coming in 3 seconds.

Only noteworthy thing that happened was a reboot (for unrelated reasons). Also, I wiped the memory log by setting lines from 500 to 1 and to 500 again. OP, could you try it please?

Now I can't reproduce the problem. I'll keep you all posted
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Mon Jun 03, 2024 3:09 am

Sorry - duckdns is not a server, just a simple API caller for some DynamicDNS service. Useful script nevertheless, thank you!
Yesterday, that container was shown as "started" immediately, logging was enabled and it took 5+ minutes for very first log lines to start coming.

BUT HEAR ME OUT - today containers are working normally! Container-specific log messages start coming in 3 seconds.

Only noteworthy thing that happened was a reboot (for unrelated reasons). Also, I wiped the memory log by setting lines from 500 to 1 and to 500 again. OP, could you try it please?

Now I can't reproduce the problem. I'll keep you all posted
I also experience some some intermittency with containers having this issue. Sometimes it will run fine, only to starting funny again on a future restart of routeros or the container itself.


On my RB5009UPr+S+ (not to be confused with my hAP x3) I have a custom built container that runs Rsyslog+Promtail together for sending RouterOS logs to Loki that was experiencing this issue. I had to do a re-image of the OS with netinstall, restore of the backup taken just before the restore, and re-installation of container package just to get it to work again. (I was also experiencing it with technitium DNS server as a container at the same time, which also simply began to work after netinstall with the same identical configuration applied)

Only to have it begin exhibiting the same problem again (with both containers) after updating from 7.15-rc4 to 7.15 official.

I've seen it happen from a simple reboot of RouterOS as well, though. Something is definitely going on in the backend, but we'll never be able to see it because access to the information on the host that would allow for debugging this is restricted by the vendor.
 
autonomous
just joined
Topic Author
Posts: 19
Joined: Sun May 07, 2023 8:32 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Mon Jun 03, 2024 5:22 am

Sorry - duckdns is not a server, just a simple API caller for some DynamicDNS service. Useful script nevertheless, thank you!
Yesterday, that container was shown as "started" immediately, logging was enabled and it took 5+ minutes for very first log lines to start coming.

BUT HEAR ME OUT - today containers are working normally! Container-specific log messages start coming in 3 seconds.

Only noteworthy thing that happened was a reboot (for unrelated reasons). Also, I wiped the memory log by setting lines from 500 to 1 and to 500 again. OP, could you try it please?

Now I can't reproduce the problem. I'll keep you all posted
Also I've tried all of those things including setting my memory log to 1 and back, no difference.
 
anettoph
just joined
Posts: 3
Joined: Tue Aug 27, 2024 1:48 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Thu Aug 29, 2024 12:12 pm

hAP ax^3 7.15.3 (stable)
FROM alpine:3.20
CMD ["/bin/sh" , "-c", "while true; do echo 'test'; sleep 1; done"]

DOCKER_BUILDKIT=1  docker buildx build --platform linux/arm64/v8 --output=type=docker --tag mikrotik-test:latest .
docker save mikrotik-test:latest > mikrotik-test.tar
1.png
2.png
cpu load 0-1%
You do not have the required permissions to view the files attached to this post.
 
anettoph
just joined
Posts: 3
Joined: Tue Aug 27, 2024 1:48 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Mon Sep 02, 2024 11:23 am

In my case reason was in RAM High.
ram-high RAM usage limit (in bytes). ( 0 for unlimited)
P.S. Plz add G/M/K suffix
 
felted67
newbie
Posts: 25
Joined: Sat Mar 16, 2013 5:24 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Mon Sep 02, 2024 1:55 pm

Hi everyone,

perhaps it might be a storage problem ?

I discovered that using a "normal" USB3-Memory-Stick in a RB5009 slows down the performance massive.
When running my project in a chr-environment it is very much faster.

USB-Devices have mainly only 1 (or 2) read/write-threads at the same time. This can be a problem when
massive reads/write are done.

Regards,
Detlef
 
holvoetn
Forum Guru
Forum Guru
Posts: 6115
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: Horrible container performance from 7.14 up to 7.15rc2

Mon Sep 02, 2024 3:49 pm

I discovered that using a "normal" USB3-Memory-Stick in a RB5009 slows down the performance massive.
When running my project in a chr-environment it is very much faster.
Be careful (expecially with RB5009) when using USB3 as storage.
For some USB devices it will after reboot come back up as USB2. Huge difference !

USB reset solves the issue (or they need to wait a bit longer after reboot).
Support is aware of the issue (since some months already) but no ETA for a real fix.
So a workaround needs to be used if you are affected by this.
 
felted67
newbie
Posts: 25
Joined: Sat Mar 16, 2013 5:24 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Mon Sep 02, 2024 5:32 pm

Well, I saw this issue already, thanks.

The worst case is that USB2 / USB3 is mapped to a different slot.
That means if you installed on USB3/slot, switching to USB2 gives no access then
to your container.

Using my container-image I just finished downloads on a "normal"-disk-installation in about
30 minutes. When using a USB-device (SanDISK / normal status), it takes about 2 hours !!!

Edit: I will try to mount a virtual USB device to one of my CHR-installations. This will perhaps
give a clue if thats a hardware- or software-problem in routeros.

Regards,
Detlef
 
andrebrandao
just joined
Posts: 1
Joined: Tue Sep 17, 2024 3:24 pm

Re: Horrible container performance from 7.14 up to 7.15rc2

Tue Sep 17, 2024 3:42 pm

Ever since updating to 7.14, and all version I've updated to so far, my containers perform *incredibly* slowly.
(see updates below, it happens in 7.14 too)
  • - A container will say it is running but takes 10+ minutes for the programs running in it to become accessible, sometimes it doesn't even succeed even though the container is listed as "running".
  • - The response from the program running in the container is horribly slow
  • - I have tried this with multiple different containers with the same result
  • - I have tried using the local flash storage or a USB-attach SSD with the same result
  • - I have completely removed all containers and started again, but it does not improve
  • - Sometimes rebooting the router appears to fix it, but of the container is restarted while RouterOS is running the problem comes right back
  • - CPU and Memory stats show that the containers take up very little CPU and Memory, so it's not a problem with those resources.
Has anyone else experiencing this? I'm on a
RB5009UPr+S+
.

Hello friend, I hope you are well.

I had the same problem as you and I did the same tests without success, today I decided to do a general analysis again and I solved my problem. I have 3 containers (pihole, observium and smokeping) running on a mini pc Router OS x86 (intel N100 processor, 16Gb RAM and 500GB NVME) 7.16rc4 (I had the same problem in 7.15), since 2 of my containers are for ICMP and SNMP monitoring I noticed that the collection time was wrong and they were apparently using data from my router OS. I did the following steps and the performance improved a lot:

1 - I stopped all the containers (I didn't need to remove them);

2 - I activated NTP-client and adjusted it so that the Timezone, clock and date were correct;

3 - I adjusted my environment variables (envs) for each container (TZ=Brazil/East);

4 - I turned on all the containers again and checked via "shell" with the "date" command if all the timezone, date and clock data were correct.

After that my machines are running very well, like a normal Linux/Docker container.

I hope this can help you, this topic helped me a lot with containers in Router OS and I hope to be able to contribute to you with this experience. I hope Mikrotik can release the possibility of manipulating container resources, because I see a great future for Router OS doing virtualization (there is a huge market right now for ISPs).

Who is online

Users browsing this forum: No registered users and 6 guests