MQTT frequently flooding LOG with "memory | mqtt warning | couldn't read control header (-1) [Bad file descriptor]"

RouterOS General
1

Hi Folks,

observing frequently (= a couple of times per day), but not periodically or reproducable starting a flood of (blue colored) messages in the log = hundreads and thousand of same log Messages per second - seams to be unstoppable:

999 | Jan/04/2026 11:46:33 | memory | mqtt, warning | couldn't read control header (-1) [Bad file descriptor]

Context:

  • System: RB3011 with 7.18.23 and 7.21rc3 and 7.21rc4 (same with all versions)
  • MT-Script: sending metrics every minute via mqtt to my broker (consumed by influxDB/grafana) with connect/disconnect to the broker in any run = I don't keep the session open during the minute
  • Runs hours w/o any problem and suddenly it starts = not reproducable or periodically
  • Profile "total": If it is running normal: "mqtt" is 0% (=disapears), during sending ~ 1,5% CPU usage
  • Profile "total": If the flood starts "mqtt" is constantly +/- 30% CPU usage. Plus logging the CPU is almost 100%
  • Reboot of the RB3011 always helps - until it starts again. I don't know any other option to stop it or heal it by now.

Questions:

  • Any idea what could be the root cause and how to heal it w/o rebooting? Why it runs hours and suddenly... fuzzy is not easy to investigate...
  • Script: Is it possible to check the CPU usage of "mqtt" frequently by a script to react somehow? Happy for any script snippet...
  • Script: Is it possible to kill the mqtt-process via script and restart it? Would be self-healing at least the symptomes.

Appreciate any idea for my questions. In best case to avoid the root cause.

grafik
grafik671×483 10.2 KB

2

I've always used auto-connect=yes on the broker. I'd imagine manually re-connecting uses more resources but IDK.

Now it should not leak resources when connecting+disconnecting, which the error message suggests is happening... You may want to grab a supout.rif when it's in bad state (as well as one when it's working) and open a ticket with MikroTik.

3

Thanks Amm0,
I changed my broker settings to auto-connect=yes now and removed connect/disconnect. We will see... :slight_smile:

Interesting: The auto-connect option is not offered in WinBox 3.x --> to be set via Terminal only

4

Please create a support ticket for us and generate a rif file when it happens.

https://mikrotik.com/support

5

Within the last 24h no incident anymore :slight_smile:
it seems to be the solution: auto-connect=yes instead of connect/disconnect. Thank you

6

Seems to be solved. Maybe you can reproduce it by yourself (w/o .rif from me)

my old solution (Script scheduled every minute - maybe it forces an incident with a higher frequency) and broker setting with auto-connect=no

/iot mqtt connect broker=$broker

/system script run "0021_RB3011_Status_System_Server_Metrics"
/system script run "0022_RB3011_Status_WAN"
/system script run "0023_RB3011_Status_LAN_WLAN"
/system script run "0024_RB3011_Status_Container"
/system script run "0026_hAP_Status_System_Server_Metrics"
/system script run "0029_RB3011_Status_check" 

/iot mqtt disconnect broker=$broker

and in the sub-scripts something like this

[...]
:local topic "$mqttBucketStatus/$mqttHost/$mqttMeasurmentContainer/$mqttProcessed"

/iot mqtt publish broker=$broker topic="$topic/val/Container" message=[/container print count-only]
/iot mqtt publish broker=$broker topic="$topic/val/ContainerRunning" message=[/container print count-only where status=running]
[...]
7

Hi denissMT,

Unfortunately it was not solved... --> as suggested: opened SUP-207873 + support.rif