I did some Google and found that a lot of people complain that the PPPoE on RouterOS eats too much CPU, like the PPPoE on Linux is single-threaded. Is this causing the high CPU usage?
I found someone suggesting using VyOS or accel-ppp. But I just paid 250$ for my RouterOS L6 license
PPPoE is apparently single threaded on Linux or Unix variants. I have not seen that VyOS (Debian Linux) is different. I do not know anything about accel-ppp.
That would not account for two of your cores being 100%, nor would I expect that of the PPPoE core on common loads. I have not seen that problem on lesser hardware than you are deploying (not ROS but still PPPoE on 'nix). What are the load factors (i.e. queuing not %utilisation)?
I found out why, I use KVM to run my RouterOS. When I run a speed test, the CPU will wait for NIC’s interrupt.
Inside RouterOS, everything is fine, the CPU usage seems pretty OK.
In the host system however, the vCPU will have nothing to schedule for and wait for IRQ, to make the guest IRQ handle faster, the KVM has a halt polling system, which is a loop that waits for IRQ to come. That is why CPU #3 and #4 will come to 100% usage.