Mikroscope: sub-second kernel telemetry from inside RouterOS (container agent, open source, MIT)

On my RB5009 the kernel log was printing this line every two seconds:

br0: received packet on eth1 with own address as source address

The frames were 2.00–2.01 s apart, which is the STP hello interval: a layer-2 loop through a mesh access point. RouterOS's own log and the port monitor showed a healthy device the whole time. STP kept the port blocked, so nothing melted down, but for days the bridge sent that port about 0.6 packets a second. I only saw it because I was reading the kernel from inside the router for something else.


Caption: The second time it happened, 2026-09-20 to 09-24 (UTC): 1 800 own-address records an hour in the kernel log, and ether2 receiving a median 13 packets/s (hourly means) while the bridge sent it 0.6. RouterOS's log and port monitor showed nothing.

That something else is mikroscope, and it is now public.

GitHub: GitHub - jmrplens/mikroscope: Sub-second, kernel-level telemetry for container-capable MikroTik RouterOS devices: a static Go agent that runs on the router and a CLI/collector that installs it, records, plots and forwards. · GitHub
Docs (English and Spanish): mikroscope — sub-second kernel telemetry for MikroTik RouterOS

What it is

Two small MIT-licensed programs:

  • An agent that runs on the router in a RouterOS container (a static Go binary in a scratch image, about 6.4 MiB for arm64). Because the container shares the router's kernel, it reads the router's own /proc, /sys, /dev/kmsg and perf counters at 1 to 100 Hz (10 Hz by default): per-core CPU ticks, interrupts, softirqs, softnet drops and time squeezes, memory, load, disk I/O. With privileged=yes it also reads the kernel log as timestamped events. It keeps a short ring in memory (60 s by default) and serves it over HTTP on a /30 veth.
  • A CLI on your machine that installs, checks and removes the agent over SSH. It also records a window while you change something and draws it as a chart, or runs as a collector that merges a 1 Hz RouterOS API tier (interface counters) and forwards everything to Prometheus, InfluxDB 3, PostgreSQL, a SQL script, Loki, OTLP, Graphite, Elasticsearch, Telegraf, a file or stdout. It ships ready-made Grafana dashboards and alert rules.


Caption: The Overview row of the bundled InfluxDB dashboard: my RB5009, last 6 h.

The agent ships raw tick deltas, never percentages. For comparison, cpu-load over the API is roughly a one-second trailing mean. I fitted it against /proc/stat read at 10 Hz on the RB5009 over two separate hours: a 1.0 s mean reaching the API 0.6 s late (r = 0.9825 over 3 499 samples), and a 1.1 s mean 0.1 s late. A core saturated for 100 ms moves a four-core one-second average by only 2.5 %.

What it costs the router

Measured on an RB5009UG+S+ (RouterOS 7.24.2, 2026-09-18) from the agent's own cgroup. Each row is one 300 s window at steady state:

rate CPU of one core RSS gaps / drops
10 Hz (default) 2.69 % 13.2 MiB 0 / 0
50 Hz 9.63 % 23.3 MiB 0 / 0
100 Hz 16.83 % 45.7 MiB 0 / 0

This is one board. The docs explain how to measure it on yours.

Things it found on my own network

  • The layer-2 loop above, twice. The second time the port was effectively out of service for four days. Now there is an alert for a bridge port that receives traffic while the bridge sends it nothing, which RouterOS reports as running and error-free.
  • A Home Assistant integration polling the router over the API pushed the timer interrupts from about 2 500 to about 35 000 per second, in bursts on one core at a time. /tool profile barely showed it. Disabling the integration brought it back within 30 s.
  • Short egress-queue drop bursts on the port to my server: 3 337 packets in six one-second bursts over 6.5 h, peaking at 436 packets/s. It's working as designed, so it's on a dashboard panel and deliberately not an alert.


Caption: The Home Assistant item above, on the bundled dashboard, 2026-09-23 08:00 to 09-24 10:00 UTC: arch_timer goes from about 2 500 to about 35 000 interrupts a second at 12:08, when the Home Assistant integration connected, and drops back the moment I disabled it.

Installing

Requirements: RouterOS 7.24 or later, the container package, and device-mode container=yes (the physical button press). arm64, arm or x86_64; no MIPS or TILE.

export MIKROSCOPE_ROUTER=admin@192.168.88.1
mikroscope doctor   # read-only preflight; names the fix for anything missing
mikroscope plan     # every RouterOS command it would run, nothing written
mikroscope install --remote-image jmrplens/mikroscope-agent:1.3.0

The router pulls the image from Docker Hub by its full reference, registry-1.docker.io/…. You don't need to change /container/config registry-url or add a registry login. I checked that on 7.24.4: a host in remote-image= overrides the global setting, and Docker Hub serves the image anonymously.

Every object it creates carries a mikroscope: comment. uninstall removes exactly those objects and verifies nothing is left. There is also a .rsc route if you'd rather paste a script, and a published tar for offline installs.

Honest status

  • It has run on one device, my RB5009UG+S+ (RouterOS 7.24.2, then 7.24.4). The arm and x86_64 builds are cross-built and tested in CI but have never run on hardware.
  • Three of the eleven sinks have taken real router samples (file, Prometheus, InfluxDB 3). The other eight are tested against the real products in containers.
  • Everything the docs claim carries the device, version and date it was measured on, and there is a "what is not claimed" box on almost every page.

Where I'd love help

  • Other boards. If you run it on a hAP ax³, CCR2004, RB4011, a hEX refresh or a CHR, the output of mikroscope status and one measured window would tell me whether the cost holds. So would which kernel ethN each port is: the kernel-to-RouterOS port map is measured per board, and only the RB5009's exists so far.
  • Faults you know how to provoke that RouterOS's own counters hide.

Feedback, issues and "this is wrong on my device" are all welcome.

uhh....

As usuall AI is very wordy but what is the point?

I would say - paraphrasing B. Marley - "No config, no help".

What " Honest status" means? No clue at all. Cen you explain this? Please.

I believe it is clear what it means with the content of that section. I do not understand why you would dedicate time to a response that is so useless.

Hi there—regarding the suggestion: it is actually running on one of those black units right now. Port 6 on the previous one had stopped working for no apparent reason—out of the blue, with no video streaming load. It showed as enabled, but the LEDs weren't lighting up. I replaced it with an identical new unit, and it’s currently running with significant traffic on Port 6—without issues (so far, anyway). As for the "offending" unit, after a reset, it’s working flawlessly, though without any significant load. There are no visible issues; it executes scripts without skipping phrases or characters. We’re keeping an eye on it. Best regards.

Is there a how to install the necessary container on routeros without relying on the "mikroscope" blackbox cli?

WTF, once again!! Stop whining about AI and address the actual content FFS.

Thank you very much for the feedback! The problem with the first unit is strange, but if it hasn't been reproduced, I will wait to see if it happens again to someone else.

Again, thank you!

Yes, of course, you can create a .rsc and analyze it, nothing runs on the router and you adjust it as you want: Four ways to install the agent on RouterOS | mikroscope

mikroscope plan --rsc \
  --remote-image jmrplens/mikroscope-agent:latest \
  --out install.rsc

jmrplens :

After a site visit to a client who complained that two out of four LAN ports on an RB_AXx2 weren't working—and reported frequent reboots—I performed a reset and reconfigured the firewall properly; I saved a screenshot of what I saw from the WAN side. It’s clear to me what happened (Layer 2 BFA + flooding). Not everyone has a proper configuration culture. MikroTik doesn't hold up against that...
Image --->

I think something similar happened to the problematic 5009 as well, even though I didn't find any traces of such an act.

Regards

Thanks for sharing it, that's a nice catch. With no proper firewall in front of it, a flood from the WAN filling the conntrack table sounds very plausible, and the reset plus a real firewall was surely the right fix.

The only thing I'd keep open is where the connections came from. That same "connection tracking table full" can also come from inside, one LAN host with P2P, an infected machine or something scanning, and from the log line alone I don't think you can tell the two apart. If it ever happens again, it might be worth looking before the reset:

/ip/firewall/connection/tracking/print
/tool/torch interface=WAN1

and IP > Firewall > Connections in WinBox sorted by Src. Address. If most of the table is one or two addresses, that would settle it one way or the other. I'd also guess the WAN1 flap at 01:38 is unrelated, since it's two hours before the first warning, but you know that network better than I do.

About your RB5009, it could be the same thing, of course. The fact that the port stayed dark until you swapped the unit made me think more of the PHY, but without traces it's hard to say either way.

In case it helps next time: this is one of the things mikroscope watches from inside the router. The agent reads the conntrack population from the kernel's slab (5 times a second at the default rate) next to the table's limit, and the collector flags it when the table is above 80 % and still rising, so you would see when it started to fill and how fast:

If you ever run the agent on that client's router and it happens again, I'd be glad to take a look at the recording.

For the sake of the topic, I made an effort to copy this too, before it "died" --->

PS_17:28:07--->

This manifests via the following "Active" interfaces; there is no "LAN" interface. There is only "WAN" and the one shown in the crop—I have indicated which interface is which. To stop the attack, I am applying a renaming method that I am not disclosing here. However, I do want to understand the attack mechanism and how it occurs. I apologize for the lack of names, but those who understand the subject don't need them.

PS_Image --->

/interface ovpn-server server
add auth=sha1,md5 mac-address=xx:xx:xx:44:55:67 name=ovpn-server1

Regards_x2

Thanks for posting it, that's a much clearer picture.

I'm only looking at a crop, but I'd read it as the router being used as a DNS reflector:

  • Everything is UDP, new, on input, to port 53 on the router, and all the same size (len 66), which is what a scripted DNS query looks like.
  • The sources look more like victims than attackers: a whole /24 (185.148.1.x) with random ports, plus a few hosts that keep coming back with fixed service ports (443, 6672, 32002, 1947). In a reflection attack the source address is spoofed to be the target, so if the router answers, the answers go to them, larger than the questions.
  • The src-mac is the same on every line, but that only says it arrived through the ISP's gateway, not who sent it.

If /ip dns had allow-remote-requests=yes and nothing on input dropped WAN traffic, every one of those packets is a new conntrack entry, which would fit the "connection tracking table full" from before, and the load alone could explain the reboots. Even if DNS doesn't answer, the flood fills the table when nothing drops it.

If that's it, the defconf input rules cover it ("drop all not coming from LAN", in-interface-list=!LAN), as long as the LAN list really holds the LAN side. I read your second crop as that router having only the old discover / mac-winbox / mactel lists and no LAN list; if the firewall was written against a list that isn't there, nothing gets dropped. For a heavy flood, dropping it in raw also saves the conntrack work:

/ip firewall raw add chain=prerouting in-interface=WAN1 protocol=udp dst-port=53 action=drop
/ip firewall raw add chain=prerouting in-interface=WAN1 protocol=tcp dst-port=53 action=drop

The ovpn-server is a separate question, and the one I'd worry about more. If neither you nor the client created it, I'd treat the router as compromised rather than misconfigured: a reset keeps the same RouterOS install, so I'd go for a Netinstall to the current version, new passwords, only the services that are really needed (/ip service, /ip socks, /ip proxy), and then a look at /system scheduler, /system script and /user for anything nobody added.

Of course you know that site and I'm guessing from two crops, so take it as a hypothesis.

Hello everyone,

jmrplens :

You guessed it ..

Based on my observations at the client's site, everything you’ve written so far is absolutely correct—though, in the rush of juggling other tasks, I did overlook a few things myself. I failed to mention that the interfaces in question were active, yet there was no "LAN" interface list—exactly! And another puzzling detail: rule (drop all not coming from LAN) was missing. It’s truly strange, because without it, the resulting mess is quite something. And just to bring a smile to your faces—I had to freeze the logs so the screenshot wouldn't come out blurry. Compromised—and how.
Next time something like this happens, I’ll record a video!

I can think of a few scenarios, including client interference—though the latter seems highly unlikely. I hadn't seen a system in such a mess in a long time. What I found interesting was that the service call was marked "best effort" because, while the client would move the cable between ports—resulting in no lights or internet—the connection would eventually come back up at some point. Naturally, the mention of "ports" piqued my curiosity regarding whether this was actually the core issue here. Those are the results.

Best regards,,,

jmrplens :

PS/ smells of roses .. and :anatomical_heart:

Thanks for coming back to confirm it, good to hear it pointed the right way.

One more guess about the ports, since that's what brought you there. As far as I know a port's lights come from the link itself, the PHY and the switch chip, so CPU load alone shouldn't make them go dark. But you mentioned frequent reboots, and while the router reboots every port is dark until it's back up. If the flood was what kept rebooting it, the client moving the cable would land in one of those windows and see a dead port that later "came back by itself".