Hey there.

I apologize if this has already been beaten to absolute death here on the Mikrotik forum, but I was having trouble searching for advice and figured my use case is somewhat unique.

I'd like to preface this by saying I only discovered what bufferbloat is a little over a week ago. I've learned so much in the last 7 days, but I clearly have a long way to go before I fully understand what is actually going on under the hood when I make adjustments here.

Some background info first:

• I'm using the hAP lite (RB941-2ND-TC) running on v7.1rc4.

• My modem is a 4G LTE router/modem running in bridge mode supplied by my ISP.

• My bandwidth is highly variable. I assume this is due to my 4G connection (Only one 4G tower in the area, but a very strong signal). At lowest, around 5Mbps down /5 + Mbps up.

• I know this particular Mikrotik Router is quite underpowered, but as far as I understand, a more powerful CPU will be useless as my bandwidth is very low.

• Changing my ISP is not at all an option. I live very rurally in New Zealand and it's a miracle I get any cell signal at all in my area, let alone a very strong 4G signal.

• I very closely monitor all network traffic, unless I've missed something, there's no way my issues are coming from a user saturating bandwidth that I wasn't aware of.

• hAP lite is configured with WiFi disabled. I have one ethernet cable running directly into my AP/Mesh WiFi setup. My Desktop is wired to the AP.

• My network has 6 very active users who all have several devices. Their bandwidth usage is basically just social media/video streaming, however.

• Nobody is particularly sensitive to streaming quality, so I can largely ignore concerns regarding the quality that video will be streamed at.

• One particular user in my household has a constant Netflix stream running during all waking hours of the day, they like to leave this on while they do other things. The bandwidth I allocate to the network is essentially constantly saturated (which is why I'm looking at QoS/Queue Trees in the first place)

My goal with my setup is to reduce my latency/bufferbloat as much as humanly possible while gaming on my desktop PC. Ideally, I'd like to simply not have to think about my latency at all. Over the last decade living with very poor bandwidth (much worse than what I have now), I have grown a somewhat "paranoid" habit of monitoring my ping. For example, I have set up a tool on my desktop that sends me a notification in the Windows 10 action centre as soon as my ping spikes above 100ms.

Set and forget would be a dream come true.


The current configuration I've put together based on some time-consuming research is as follows (I'm more than happy to provide any additional info on request):
This is a latency graph from the last 10 minutes. Latency is on the Y-Axis, time on the X-Axis. Even though there's a little over 2Mbps of network usage, the latency is pretty inconsistent. (As I'm writing this I tested disabling the queue tree entirely and immediately I could see the "All-Download" connection mark reach 8mbps pretty regularly)


I have a couple of questions:

• Clearly this process is very specific to each individual user. Trial and error has been my philosophy throughout this whole process. What tools can or should I use to trial and error my particular use case? At the moment I have been using PingPlotter 5, set to the global gateway of my country. I've been monitoring whenever it shows that my latency has increased beyond what I believe to be unacceptable. To saturate my available bandwidth I found TikTok to be surprisingly useful here. Specifically, I scroll as fast as it will let me and it seems that my bandwidth limits how fast I can scroll. Not very scientific but it seems to work well enough.

• Is there anything sticking out as something I've done incorrectly or perhaps that I could immediately improve? I mentioned earlier that I'm very new to this level of network configuration and I'd be very grateful if anyone was able to share their informed opinion.

• Despite the extremely low bandwidth limits I set within the queue tree, I still experience some pretty unpleasant latency issues. I can very clearly see that my config works to some extent. I can temporarily disable the entire queue tree and watch my latency go absolutely crazy.

I've definitely made some improvement but I still have to monitor the latency at all times and adjust the limits on the fly while I'm playing to achieve more acceptable latency. Also, to be crystal clear, my bandwidth never goes below 5/5. I've read that if you have highly inconsistent bandwidth, I should just set it at the absolute lowest values. It actually typically sits much higher than that, but 5mbps down is enough for my household's use cases and latency is our goal here.

Here's a real-world scenario that took place with the settings in the attached images:
I was playing a game and my ping started rising, so I decreased the available bandwidth for devices other than my own and it helped a bit. I had to repeat this several times until the latency was better. After I settled on the limit value (I think it was 800Kbps down for all devices other than my own), it was almost impossible to stream video because the limit was so low, but the ping reported by my game was almost completely stable.


I have a theory that explains my current situation (forgive my lack of technical language here):

My available bandwidth is restricted by what the local cell tower is capable of outputting. I know that this cell tower in question is the only one in the area for miles. I used to get zero cell service at all (not even 3G), and one day I woke up with a signal. I can drive in both directions on my road and both ways the signal drops out after a relatively short distance.

This is where the assumptions start.

Anyone within range of the tower can take some of what's available. The cell tower, much like a local network, will become saturated sooner or later if enough bandwidth is requested from it. Once it is saturated, I'm guessing that it has to prioritize users in some way or another and basically divide the available bandwidth up between us all (my modem, my neighbors, anyone in my own household that decides to use cellular instead of WiFi).

Given that the tower sees my modem as a single device, just like a cellphone, our available bandwidth is entirely dependant on how much my neighbors decide they want to use.

So, if I'm constantly lowering my "global minus my desktop" limit, I'm essentially increasing the available bandwidth to anyone else connected to the tower. Given that they could very easily be streaming video or something else that would potentially saturate the bandwidth, it's almost entirely pointless for me to lower the limit as I'm currently doing?

There's a multitude of unsubstantiated assumptions in the above theory, but given that I can make my desktop's latency spike by conducting a speedtest on my cellular connection on my phone, I tend to think this may be the case.

My hope is that I've overlooked something (or several things) and I can adjust my setup to improve both my bandwidth, my latency, and the time I am spending twiddling with the settings in WinBox. (I'm starting to wish I had purchased a model compatible with TheDude). I can't think of a possible explanation for why my latency remains so poor despite how little bandwidth gaming seems to require.

TLDR: My bandwidth is unstable. I've made solid improvements, but I still have unbearable latency issues. I can fix this by restricting the network so severely that streaming a video at the lowest possible quality still buffers (Not really much point streaming Netflix at 144p resolution). Is there any more I can do about this on top of what I've shown in the attached images?

Any advice or direction towards some more reading about all this would be greatly appreciated!
Thanks for your attention!

You are the first person I've found on mikrotik fiddling with their fq_codel implementation. (I'm one of the authors of fq_codel, cake, etc)

And you are tackling the worst problem "out there" for bufferbloat - inbound shaping a low rate lte connection. These sorts of connections are not only low rate,
but wildly variable in their ability to deliver packets at any given instant in time.

And you are only 7 days into it, and doing pretty well.

My zeroth suggestion is to try it at 5Mbits, not 20Mbits. You want it to be closer to the minimum you experience, not the max. Period. You need to retain control of the
queue.

My first suggestion is to switch to cake, and kill all the rules you have there. I've been trying to find out if mikrotik supports nat for cake or not, but if you are natting on the router, applying the nat option to cake will give per host/per flow flow queuing, meaning that your low rate gaming traffic will win more often than netflix. In general, however, this is not hugely required.

One feature of cake, probably not exposed by mikrotik, is that it's possible to dynamically change the shaping rate based on some other set of supplied statistics.

If you are going to stick with fq_codel, use a quantum of 300.

For reference, this is the paper on cake: https://arxiv.org/abs/1804.07617

For reference, this is the paper on cake: https://arxiv.org/abs/1804.07617

Hi Dave, this is my result from dslreports using CAKE,


And this is my CAKE configurations (basically untouched, default Mikrotik parameter):

Since im using per-IP bandwidth limitations for my client,
i wonder if there's any better calculation or configuration for this usecases.

Also for CAKE RAM/memory usage, it will preserve approx 2M of RAM for each simple queue with CAKE profile/scheduler,
not sure if this only happen on RouterOS or default CAKE behaviour/requirements.

You are the first person I've found on mikrotik fiddling with their fq_codel implementation. (I'm one of the authors of fq_codel, cake, etc)

And you are tackling the worst problem "out there" for bufferbloat - inbound shaping a low rate lte connection. These sorts of connections are not only low rate,
but wildly variable in their ability to deliver packets at any given instant in time.

And you are only 7 days into it, and doing pretty well.

My zeroth suggestion is to try it at 5Mbits, not 20Mbits. You want it to be closer to the minimum you experience, not the max. Period. You need to retain control of the
queue.

My first suggestion is to switch to cake, and kill all the rules you have there. I've been trying to find out if mikrotik supports nat for cake or not, but if you are natting on the router, applying the nat option to cake will give per host/per flow flow queuing, meaning that your low rate gaming traffic will win more often than netflix. In general, however, this is not hugely required.

One feature of cake, probably not exposed by mikrotik, is that it's possible to dynamically change the shaping rate based on some other set of supplied statistics.

If you are going to stick with fq_codel, use a quantum of 300.

Thanks so much for your response! I tried adjusting my config as you suggested (CAKE + a stricter overall limit to 5/5 on inbound/outbound traffic) and it made things a little better, which I certainly appreciated. Unfortunately, it seems that I can't get CAKE to run stable on my particular setup. Perhaps due to CAKE being a relatively new addition to RouterOS (running beta software), or that my particular Mikrotik router is a very low-end device. Regardless, fq_codel with the parameters you suggested made a noticeable difference, so thank you.

After even more digging into this topic I found some interesting information on another forum (can't find the link to what they said exactly, unfortunately), but it was something to the effect of:
You can try limiting the "download" traffic all you want, but you're really just telling the router to throw away some of what it has already received, and thus not actually improving the saturation of your link to the internet. It may appear that the limit is working as intended (in my case, my family members can no longer stream Netflix when I hard limit them), but in reality, you still have a large amount of traffic saturating your link.

This got me to thinking, what if I could prevent my devices from even requesting that much data in the first place? So I implemented a far stricter limit on my outgoing traffic (which I guess would include requests to outside sources for packets) and hopefully, that will improve things even further! It actually seems to be working. I can remove my ingress limit entirely and my strict egress limit seems to be preventing the inbound traffic from getting very high at all.

Do you think the underlying principles behind what I'm trying here make sense? I can't really find anything else confirming or denying this to be the case, but I personally think it makes sense.

Thanks again for your help!

There has been a lot of work towards making the autorate feature work better on LTE over here:

https://forum.openwrt.org/t/cakes-autor ... ss/108848/