I have a 500/250Mbps fiber. My ISP has installed an ONT and I have static IPv4 from them.
Router is connected over ethernet cable, ISP link has a VLAN and PPPoE connection. MTU and MRU are 1492.
I am getting consistent 60+ (60-70 ms) upload latency on bufferbloat test.
I have tried using queue tree with max limit set to 250M and fq-codel parented to the PPPoE interface (let’s call it wan1). This kind of setup used to work fine for my previous cable ISP which didn’t involve PPPoE (cable modem was in bridge mode) – result is no change in upload latency.
I then tried parenting the queue tree to VLAN interface, and finally to the ether1 interface but upload latency stayed the same.
I then tried using sfp-plus1 port via copper (since ether1 is behind a switch1 on this router and sfp isn’t) but upload latency stayed the same.
I tried using a simple queue too and the result was the same.
I then tried setting rx/tx rate limit (250M/500M) in the PPP profile I am using for PPPoE connection. This sligthly reduced download latency (from 20 ms to 9 ms) but did nothing for upload latency.
Finally I tried changing only-hardware-queue option on the physical interface to other queue types. PCQ with specified rate didn’t do almost anything. FQ-CoDeL also couldn’t do much since it doesn’t have a rate / bandwidth parameter. Then I set CAKE with bandwidth limit 250M and RTT 30 and the upload latency dropped to 0 ms. Max upload speed was 225Mbps (without shaping it’s 235Mbps out of rated 250Mbps).
Is there perhaps any known issue with only-hardware-queue option on CCR 2004 and extra 60 ms latency?
Should I set rx/tx rate on PPP profile or not? [Answer: apparently not as this just adds a dynamic simple queue with those limits]
Is there anything else I can do to improve download latency?
Is there any recommended setup for shaping on PPPoE links?
Please advise.
EDIT1: Edited to clarify that even if I create a simple queue with CAKE in both directions as long as I don’t set a shaper directly on physical interface I have +60ms latency on upload. To me that definitely seems like some bug.
The first thing is, most of the time, setting the queue bandwidth to the same bandwidth as your booked internet connection is not a good idea for two reasons. Some ISPs give you extra bandwidth for the overhead, some don’t. For example, my DSL reseller sells a 168 Mbit/s contract. The main dsl company that owns the cables, sells the same contract as a 150 Mbit/s connection. One is advertising raw bandwidth and the other is advertising net bandwidth.
Also if your ISP give you extra bandwidth, it’s still a problem to set the limits of the queue to the max bandwidth for your connection. At same point in the routing chain the ISP will limit your internet connection to your booked bandwidth. For that he’s also using some sort of queue but 99% of time not cake nor fq_cadel. Just a simple FIFO (first in first out) queue to limit the bandwidth and thats the reason you get bufferbloat on your end.
Now what will happen if your cake queue and the FIFO queue of your ISP are both set to 500 Mbit/s. You start downloading at max speed. The packets will first arrive at the FIFO queue because your a downloading stuff. The FIFO queue will fill up because you are maxing out your booked bandwidth and packets get queued or droped aka bufferbloat. There is no way your cake queue can help there to mitigate bufferbloat. There is no direct way for you to control this. The only thing you can try to do is to prevent the FIFO queue to filling up. To do so you need the set your cake queue to a lower limit.
In the upload dircation you have more control, because the packets arrive first at your queue but as soon as you hit your upload limit for the FIFO queue you will get also bufferbloat. So i would also set the upload limit in the cake queue lower.
The second thing that came to mind is a CPU bottleneck. I think this is not the case for you, but theoretically possible. You can use the profile tool set to CPU=all to check that. PPPoE in RouterOS is software based and sadly single thread.
There is no need to set same special PPP profile settings nor interface interface queues. Just make a simple queue. If you experience bufferbloat lower the limit for the right dircation.
I have tried setting lower limits than what I get from the ISP, but it didn’t change the fact that as long as I have a hardware-only-queue on the physical port on which the ONT is connected there’s extra 60ms latency on upload. Only if I put CAKE directly on that port the latency is gone.
This queue setup eliminates 60ms latency and gets me an A on bufferbloat test:
However, it feels wrong to have to put CAKE directly on ether1 to elminate upload latency.
There might be something I am doing wrong with the switch, bridge, or VLAN configuration but to me it still looks like some bug either in the switch driver or in the RouterOS itself.
I’d really love if someone from MT would chime in with some advice.