CHR 7.23.2: large in-memory packet capture causes /tool sniffer crash and generates autosupout.rif

RouterOS General
1

Hello,

I would like to report an issue with /tool sniffer on RouterOS CHR.

Environment:

  • RouterOS CHR 7.23.2 stable
  • x86_64 on VMware Virtual Platform
  • 16 vCPU / 16 GB RAM

We use /tool sniffer for continuous in-memory packet capture with these characteristics:

  • file-name=""
  • memory-limit between 2 GB and 4 GB
  • memory-scroll=yes
  • only-headers=no
  • Continuous in-memory capture

When the issue occurs, the CHR does not reboot and RouterOS remains reachable. However, /tool sniffer itself stops/crashes unexpectedly, and RouterOS automatically generates an autosupout.rif file.

Expected behavior:
When memory-limit is reached, old packets should be evicted and the capture should continue running normally.

Actual behavior:
After running a multi-GB in-memory capture continuously, the sniffer may crash unexpectedly and can no longer retain the pre-incident capture buffer.

As a temporary mitigation, we have disabled this in-memory capture method.

Could you please confirm whether this is a known issue in CHR 7.23.2, and whether there are recommended limits or constraints for multi-GB memory capture with memory-scroll=yes? An autosupout.rif file has been generated and can be provided privately for analysis.

2

Sniffers a 32-bit process, so it's capture buffer is capped at ~4GB address space on linux. A 32-bit process running on 64-bit kernel means it is also going to be storing two heap allocations per packet. So real consumption runs well above your memory limit most likely. Past that ceiling the allocation fails and the process dies.

On top of that, memory-limit is converted from KB to bytes in 32-bit arithmetic, so a value of 4GB or above wraps. So at exactly 4GB the effective limit becomes 0, and with memory-scroll=yes each new packet flushes the buffer. That's why you lose the pre-incident capture rather than getting eviction.

Drop memory-limit to something well under the ceiling, a few hundred MB and you should be stable. If you need multi-GB retention , in memory, a 32-bit process is the wrong mechanism for it. But you can stream your capture to an external collector. Let wireshark or tcpdump hold the volume on a 64-bit box if you really need it

/tool sniffer set streaming-enabled=yes streaming-server= filter-stream=yes

3

What is the purpose of using this dumb way of doing packet captures, repeatedly?

4

You really should open ticket with MT support and provide autosupout.rif file ... that's the only chance that MT will do anything about it.

IMO even if what @rd.searcher wrote is correct, ROS should take care of overheads and not push itself into memiry problems.

5

It would be better to create a ramdisk (disk type tmpfs) and let the sniffer log to a file in the ramdisk.

Not only would that work correctly (I think), you also can download the capture when you suspect something interesting in it.

Unfortunately there is no name-rotation facility like in log to disk.