MQTT frequently flooding LOG with "memory | mqtt warning | couldn't read control header (-1) [Bad file descriptor]"

RouterOS General
1

Hi Folks,

observing frequently (= a couple of times per day), but not periodically or reproducable starting a flood of (blue colored) messages in the log = hundreads and thousand of same log Messages per second - seams to be unstoppable:

999 | Jan/04/2026 11:46:33 | memory | mqtt, warning | couldn't read control header (-1) [Bad file descriptor]

Context:

  • System: RB3011 with 7.18.23 and 7.21rc3 and 7.21rc4 (same with all versions)
  • MT-Script: sending metrics every minute via mqtt to my broker (consumed by influxDB/grafana) with connect/disconnect to the broker in any run = I don't keep the session open during the minute
  • Runs hours w/o any problem and suddenly it starts = not reproducable or periodically
  • Profile "total": If it is running normal: "mqtt" is 0% (=disapears), during sending ~ 1,5% CPU usage
  • Profile "total": If the flood starts "mqtt" is constantly +/- 30% CPU usage. Plus logging the CPU is almost 100%
  • Reboot of the RB3011 always helps - until it starts again. I don't know any other option to stop it or heal it by now.

Questions:

  • Any idea what could be the root cause and how to heal it w/o rebooting? Why it runs hours and suddenly... fuzzy is not easy to investigate...
  • Script: Is it possible to check the CPU usage of "mqtt" frequently by a script to react somehow? Happy for any script snippet...
  • Script: Is it possible to kill the mqtt-process via script and restart it? Would be self-healing at least the symptomes.

Appreciate any idea for my questions. In best case to avoid the root cause.

grafik
grafik671×483 10.2 KB

2

I've always used auto-connect=yes on the broker. I'd imagine manually re-connecting uses more resources but IDK.

Now it should not leak resources when connecting+disconnecting, which the error message suggests is happening... You may want to grab a supout.rif when it's in bad state (as well as one when it's working) and open a ticket with MikroTik.

3

Thanks Amm0,
I changed my broker settings to auto-connect=yes now and removed connect/disconnect. We will see... :slight_smile:

Interesting: The auto-connect option is not offered in WinBox 3.x --> to be set via Terminal only

4

Please create a support ticket for us and generate a rif file when it happens.

https://mikrotik.com/support

5

Within the last 24h no incident anymore :slight_smile:
it seems to be the solution: auto-connect=yes instead of connect/disconnect. Thank you

6

Seems to be solved. Maybe you can reproduce it by yourself (w/o .rif from me)

my old solution (Script scheduled every minute - maybe it forces an incident with a higher frequency) and broker setting with auto-connect=no

/iot mqtt connect broker=$broker

/system script run "0021_RB3011_Status_System_Server_Metrics"
/system script run "0022_RB3011_Status_WAN"
/system script run "0023_RB3011_Status_LAN_WLAN"
/system script run "0024_RB3011_Status_Container"
/system script run "0026_hAP_Status_System_Server_Metrics"
/system script run "0029_RB3011_Status_check" 

/iot mqtt disconnect broker=$broker

and in the sub-scripts something like this

[...]
:local topic "$mqttBucketStatus/$mqttHost/$mqttMeasurmentContainer/$mqttProcessed"

/iot mqtt publish broker=$broker topic="$topic/val/Container" message=[/container print count-only]
/iot mqtt publish broker=$broker topic="$topic/val/ContainerRunning" message=[/container print count-only where status=running]
[...]
7

Hi denissMT,

Unfortunately it was not solved... --> as suggested: opened SUP-207873 + support.rif

8

After 7 days without any issue anymore I take it as solved now :slight_smile:

3 things seems to me as crucial as solution of the root cause

  • auto-connect=true (and never us connect/disconnect for MQTT again) --> Thank you @Amm0 as I think this was the main root cause
  • keep-alive longer than my scheduled script frequency. running a script every 60s and do not care about the default keep-alive=60 is a kind of stupid - or missing knowledge. or both.
  • Update to >= 7.21rc5

Thank you @denissMT for your support!

9

Just raising my hand with the same issue -- only happened once so far, on 7.23.2 I believe, and the broker config was already using auto-connect=true. Couldn't fix the issue without rebooting. Will keep an eye out.

10

It's worth collecting a supout.rif at the time of the issue. You can also enable a bit more logging on topics=mqtt,!packet,!raw to memory (perhaps also increasing memory-lines /system/logging/action/set memory memory-lines=10000), so if it does happen again, you/MikroTik have a bit more detail on ponder. @denissMT/friends are pretty good at getting to bottom of these things, but have a supout.rif (which will include memory logs).

The other thing to look at is keep-alive= (and corresponding one in broker). Docs imply to keep them aligned, but I suppose if both are 60s (default in RouterOS and mosquito)... but my wild-ass-guess is it possible there is some race where 59.999 vs 60.001 so keepalive<>timeout "get crossed"... although I have not studied the actual interactions nor specs on how this works in MQTT.