Context:
I come from an Ubiquiti Edgerouter Lite and the moment I opened Webfig for the first time I almost got traumatized by the sheer amount of option and configurations available.
My setup is a normal home networking setup with very few devices.
I have a Gigabit Fiber internet connection with 1Gbit down and 200Mbit up.
My current network structure is really simple:
Fiber ONT (bridge mode) -> Mikrotik -> various devices.
The issue
When connected straight into the ONT I can saturate the download bandwidth and won't have any issues.
The upload, though, is a completely different story.
Although I have 200Mbit, for reasons unknown when I saturate the upload I can reach up to 210Mbit/s and the latency and bufferbloat skyrocket to various seconds (yes, 1000ms and more).
The easy solution is to cap the upload bandwidth, I tried this with Google Chrome developer tools at first by limiting the upload bandwidth to 190Mbit/s and everything was nice and smooth.
When I tried to enable QoS to limit the upload bandwidth on my Edgerouter lite, performance collapsed: download couldn't go faster than 200Mbit and 150Mbit in upload.
This lead me to find another solution: enter the Mikrotik.
The new issue
My Mikrotik has the standard, default configuration: LAN port 1 is Internet, the other 4 (and wireless) are a hardware bridge (with hardware offloading enabled, so they are managed by the switch chip).
In order to limit the upload I had to disable Fasttrack otherwise queues wouldn't work.
Then I created a queue to limit only the upload to 180Mbit/s and to keep the bufferbloat down I used a small PFIFO queue with 5 packet queue.
I have zero packet loss and very low latency,
The download on the other hand... the performance took quite the hit, down to 400Mbit/s.
In order to get back to at least 800Mbit/s without packet loss I had to create a huge PFIFO queue with 10000 packets.
Luckily the latency and bufferbloat is still amazingly low, but I'm still unable to get the 940Mbit/s I had with Fasttrack.
On top of this, the whole box is super hot and I had to put a CPU fan in front of it to cool it down.
When I look at the Resource/CPU page on Webfig I see two of the four cores on 100% and the other two at very low values.
Question
Why is it not using all four cores?
Am I doing this right?
Are there any better solutions to get full performance on download?
I have already done a search here and it looks like I can't enable Fasttrack only for downloads and, unfortunately, it looks like I can't just assign a no-queue on the download.