Single-core 100% CPU with 550+ dynamic FQ_CODEL simple queues — can't refresh queue-type without reboot

RouterOS General
1

Hi all, hoping for community/engineer insight on a persistent CPU issue that I've spent 4+ days investigating.

Setup

  • Router: RouterOS x86, Intel 4 cores @ 3.3 GHz, 4 GB RAM
  • RouterOS: 7.12.1 (stable), Nov 2023 build
  • Role: PPPoE server for ~560 FTTH customers (WISP)
  • PPP profiles: 9 profiles, all with queue-type=FQ CODEL (rate 3-20 Mbps per package)
  • Simple queues: 557 dynamic (auto-created per PPPoE session), all FQ_CODEL
  • Active PPPoE sessions: 559
  • FastTrack: enabled (chain=forward action=fasttrack-connection src-address-list=!blocked dst-address-list=!blocked)
  • Firewall: 17 filter / 12 NAT / 7 mangle rules
  • Connection tracking: ~36k concurrent

Symptom

Exactly one CPU core is constantly at 100%, while the other 3 idle at 8–20%.

The 100%-core rotates between cpu0/1/2/3 (single kernel thread being rescheduled).

Total CPU averages ~35% = (100 + 3×~12)/4.

This happens BOTH during traffic peaks (~800 Mbps aggregate) AND during idle (0 Mbps at 00:11 local time). Traffic is NOT the driver.

Currently no packet drops (rx-drop/tx-drop = 0 on WAN interfaces). This is a spare-capacity concern before I lose margin, not an active service impact yet.

What I've Ruled Out With Data

Hypothesis Result
Traffic-bound :cross_mark: 425 Mbps → 12% CPU, but 152 Mbps → 76% CPU
Packet-bound :cross_mark: Same pps (~65k) gives 12% or 90%
Broadcast flood :cross_mark: zero ifInNUcastPkts
NIC IRQ affinity :cross_mark: core with zero IRQ also pegs
Profiler stuck :cross_mark: locked constant, core rotates
Winbox mgmt session :cross_mark: constant 550 KB/s, CPU varies wildly
Scheduler task :cross_mark: /system scheduler empty
PPPoE auth loop :cross_mark: constant 13-15 fails/10min, CPU varies
Post-reboot recovery :cross_mark: CPU climbs 8% → 100% within 30 min as sessions reconnect (queue re-creation is the load)
FastTrack enabled :warning: Total CPU dropped 100%→36%, BUT single-core-100% persists (queues NOT bypassed)

Current Prime Suspect

Hypothesis: RouterOS processes simple queues single-threaded. 557 dynamic PPPoE queues with FQ_CODEL scheduler saturate one core because FQ_CODEL is significantly more CPU-heavy than pfifo, especially when running per-session.

FastTrack doesn't help because it bypasses firewall but NOT the queue subsystem — 97.6% of traffic still hits queue processing.

The Blocker: Cannot Test The Hypothesis

Yesterday I tried to prove/disprove this without a reboot:

  1. Changed queue-type on all 9 PPP profiles from FQ CODEL to default (pfifo) via /ppp/profile/set :white_check_mark: verified applied
  2. Kicked 10 test sessions via /ppp/active/remove :white_check_mark: 8 auto-reconnected within 3 min
  3. Checked queue-type of reconnected sessions → still FQ_CODEL :cross_mark:

Dynamic PPPoE simple queues appear to be persistent across session kicks — the queue is not deleted on /ppp active remove, just its statistics reset. When the session reconnects, it re-attaches to the existing queue (still FQ_CODEL).

Attempted workarounds — also blocked:

  • /queue/simple/set on dynamic queue → "can't edit dynamic object"
  • /queue/simple/remove on dynamic queue → "can't remove dynamic object"

Questions

  1. Am I right that dynamic PPPoE simple queues are single-threaded per queue-instance, and FQ_CODEL vs pfifo is what's saturating one core?

  2. Is there any way to force a dynamic simple queue to refresh its queue field from the parent PPP profile without a full reboot or deleting the PPP secret? I can accept short (seconds) glitches per-customer but I can't drop 560 customers for 5-10 min just to test a hypothesis.

  3. My /tool profile has been locked with "profile already running" for months — only a fresh reboot clears it. Any way to unstuck this without reboot? Without it I can't identify the exact process eating the pegged core.

  4. For ~600 PPPoE customers with per-customer rate limiting on x86 RouterOS 7.x, is per-customer dynamic simple queue the right architecture? Would you recommend Queue Tree + PCQ with mangle-based classification for better multi-core scaling? Any real-world examples of WISPs at this scale that solved it?

  5. Known issue with FQ_CODEL scaling on 7.12.1 at 500+ concurrent queue instances?

Environment context (for anyone considering the same setup)

  • Rural WISP, single upstream + backup
  • Dual WAN (primary + backup ISP)
  • Address-list has ~27k static entries (public CDN/speedtest ranges + ASN routing lists) — dormant, not in packet path, confirmed not the CPU cause
  • No hotspot, no DHCP other than one small subnet
  • FastTrack was added mid-investigation and reduced TOTAL CPU load significantly (100%→36%) but didn't eliminate the single-core 100% pattern

Thanks in advance to anyone who can shed light or has been through this. Happy to share any additional data — logs, /export, screenshots.

2

I would start with upgrading as your ros version is very old. Do not know if this solves your problem but in your current version there is a security issue with PPP that I would suggest you should patch. Moving to long-term would be a good start. A lot has happened since 2023.

3

After reading a full featured AI generated and formatted post with so many hypotheses (sigh!!!) I did a simple search for
"mikrotik pppoe is single core"

and the first answer was: PPPOE SERVER ONLY USES 1 CORE

It has been a long time problem since .... always.

4

This post hints that perhaps there might be changes in this area with very recent versions of routeros.