Community discussions

MikroTik App
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

For ISPS: Motivations and methods for implementing fq_codel and cake

Wed Dec 15, 2021 12:31 am

This enormous thread, debugging fq_codel and cake on mikrotik ( viewtopic.php?t=179307 ), spawned this question from @mducharme ... and it seemed best to fork it here.

> dtaht:
> Lastly, I do not know how much wred is deployed anymore. 5 tuple FQ - all by itself - seems to be gaining traction.

From my interactions with engineers working for much bigger ISPs than the one I work for (where queuing in software is possible), wred is still the gold standard for most large providers. My understanding is that everything Cisco and Juniper is wred. It can handle huge bandwidth amounts due to offloading to the ASIC, but is almost certainly much worse than any of the newer AQM solutions. I believe those running Cisco and Juniper have no ability to even consider codel or fq_codel or cake on the service provider side.

I suppose the difference is how much these AQM technologies are really designed to be used on the client side vs the service provider side. If I was daring enough to upgrade our core routers to 7.1 (I am not), what would make the difference - should we do cake queues for each customer at the ISP end, or should we deploy local cake queues to their routers, or should we do it on both ends? And if we should do at both ends, then why?

I ask this because most of the documentation I have seen about cake and fq_codel is about what the client should do rather than what the ISP should do. There is a lot of useful information geared towards the regular end user, but very little geared towards ISPs who want to improve services for their users.
 
tcroghan
just joined
Posts: 7
Joined: Wed Dec 12, 2018 9:42 pm

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Fri Dec 17, 2021 1:43 am

@dtaht,

Thank you for forking and I look forward to your thoughts and information.
 
User avatar
deadkat
Frequent Visitor
Frequent Visitor
Posts: 57
Joined: Sun Nov 15, 2020 11:14 pm
Location: Alabama, USA

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Tue Dec 21, 2021 9:48 am

+1
I'm excited to hear about how this might be addressed on the ISP side
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Wed Dec 22, 2021 7:20 pm

I'm sorry it's taken me so long to address this. Backstory first:

fq_codel and cake were "bottom up" efforts throughout, developed upon jim gettys seeing a real problem with how the internet had degraded, that nobody understood, that he kept digging into and into, with the aid of many of the other founders of the Internet such as Van Jacobson, Vint Cerf, Dave Reed, Dave Clark, Stuart Cheshire, and a whole bunch of younger geeks that rose to the occasion and cared throughout the linux and BSD communities.

To meet with widespread ignorance and disbelief about how much better the internet could be, if we somehow controlled queue delay better, we started a project to, in part, provide an existence proof back in 2011: https://www.bufferbloat.net/projects/ma ... Annotated/

- but we didn't know the final shape of the answers, or even what to work on first. Everything was broken.

In my case I wanted a (variable) line rate wondershaper that worked on wifi. https://www.bufferbloat.net/projects/bl ... _Must_Die/

Progress followed after the development of "BQL", which finally made a whole bunch of AQM and FQ papers written in the 90s and 00s line up with the then deployed reality on linux mainline and in our openwrt (cerowrt) testbed. A key moment for me was realizing that all that prior work... was essentially obsolete until that day BQL arrived, and it was a greenfield problem where rapid innovation was possible: https://conferences.sigcomm.org/sigcomm ... es/137.pdf *IF* we did rigorous science AND engineering.

Once we had codel and fq_codel - that knocked network latency "out of the park", we took it to the ietf to standardize ( https://datatracker.ietf.org/group/aqm/documents/ ) , in the hope that folks implementing new code, ISPs, vendors, and those writing RFPs would still pay attention to RFCs, and meanwhile took the fq_codel theory into ethernet and a htb based shaper we ended up calling sqm ("smart queue management") and provided as a reference for all Linuxes here: https://github.com/tohojo/sqm-scripts which has a few bits of tricky math in it.

It took 3+ more years to fix wifi, with really, really amazing results, and early adopters such as gfiber, eero, openwrt and all the third party firmwares, took up this stuff first. ( https://lwn.net/Articles/705884/ ). ( I don't know what mikrotik products now have this code in 7.1, btw)

I certainly fully understood then and now that it would take at least another 5 years before we could "Cross the chasm" into a fuller deployment in more common business gear. Things like the development of cake continued meeting the needs of the userbase, the wifi work got baked into a lot more chipsets, the BSDs gained fq-codel support, and so on, so we have a pretty good basic, proven, infrastructure, now, that can essentially go "on by default", for anyone with a modern enough kernel, the wit, and the way.

To this day we struggle with explaining the problem we solved, and how important it is to deploy to make your network applications like videoconferencing and web traffic share better, business cases, and so on. https://blog.apnic.net/2020/01/22/buffe ... -over-yet/ but we are trying hard to make the case that better QoE is very important for ISPs to deliver, as opposed to more and more "bad" bandwidth, moving forward. https://blog.apnic.net/2021/12/02/worki ... -frontier/
Last edited by dtaht on Fri Jan 07, 2022 7:38 pm, edited 1 time in total.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Wed Dec 22, 2021 9:28 pm

two last bits of backstory: I founded an ISP in 1993 (sold it in 1999), and was trying to get my WISP off the ground in Nicaragua in 2007, but my fresh shiny new 802.11n network failed when it rained. There were numerous times over the course of my history in these markets where I would have killed for a better "RED" on my backhaul, or a scalable SFQ ( https://datatracker.ietf.org/doc/rfc7806/ ) , but I didn't really realize the extent to which the overbuffering problem had got out of hand for several more years: https://the-edge.taht.net/post/Did_Buff ... Net_Radio/

And people mostly seemed not to notice, like a frog in boiling water, we just abandoned applications that used to just work. The internet morphed, we went from an era where the internet degraded gracefully, where skype "just worked", where bittorrent (+SFQ) didn't mess up your home network, and email flowed freely, and we had client/server databases and rather minimal web traffic, to something at least, at home, you couldn't share....

I'd figured that when free.fr adopted fq-codel in 2012, 3 months after we shipped it (with their own minimally buffered DSL driver so no HTB was needed), it would be a year or two, tops, before my former vendors in the WISP market (cambium, ubnt, mikrotik) would see the benefits of fixing bufferbloat, ship products with the fixes, and I could return to my mountaintop in nicaragua and finish expanding throughout the Rivas department. Hah!

Anyway, having figured out how big the problem was, worldwide, I got older, and no longer have much urge to restart my wisp anymore, nor do I have the capital to do it.

I'm here on these forums this month, working for free, as mikrotik's implementation of cake prior to this was buggy, and mikrotik the last vendor in my former market I cared about improving. Most of my work is volunteer, paid for by small grants and if I'm being useful here, please feel free to toss something into my patreon: https://www.patreon.com/dtaht
Last edited by dtaht on Fri Jan 07, 2022 7:40 pm, edited 1 time in total.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Wed Dec 22, 2021 9:52 pm

Let me get the biggest negative motivations out of the way first.

1) ISPs have perverse incentives to discourage actual use of the bandwidth. The more bandwidth you can sell, and the less your users use it, the more money they make. Monopoly ISPs, especially, have a tendency to not move very fast.

2) The principal way for an ISP to make more money from a customer has always been to sell more bandwidth, and some hosted services.

To counter these two assertions I've tried a variety of arguments over the last decade.

1A) ISPs selling good *service* retain customers. 1B) If you are in a competitive market, selling better service gets you customers. 1C) Better service reduces support calls 1D) Improving videoconferencing right now is very important and yet that's very low bandwidth, yet. more than games, and fq-codel really shines at that. I quote Van Jacobson a lot:

“FQ_Codel provides great isolation... if you've got low-rate videoconferencing and low rate web traffic they never get dropped. A lot of issues with IW10 go away, because all the other traffic sees is the front of the queue. You don't know how big its window is, but you don't care because you are not affected by it. FQ_Codel increases utilization across your entire networking fabric, especially for bidirectional traffic... If we're sticking code into boxes to deploy codel, don't do that. Deploy fq_codel. It's just an across the board win.” - Van Jacobson | IETF 84 Talk

He's the guy that designed RED, and has been trying to replace it for a long time with something that worked better. He and kathie came up with codel to replace it, and eric dumazet and I attached a better form of fq to it, and we knocked out longstanding issues like tcp global synchronization, near/far bandwidth disparities, vastly improved web page PLT, and reduced typical queue depths from sometimes seconds, to a few milliseconds, with it across every technology we have.

2A) I figured that a software upgrade to better bandwidth is about the cheapest rollout of new stuff possible, and all of us in the bufferbloat effort have been demonstrating vast improvements in DSL behaviors in particular for 10 years now, in addition to wifi, cable, fiber, etc. 2B) A sale or rental of a better router is also worthwhile. 2C) It costs serious money to dig holes in the ground to deploy more bandwidth, why not deploy better bandwidth with what you have? 2D) The customers on the edge of your service range, or too poor to get more service, and stuck WFH, really need this stuff. You, a customer of your own ISP, need this stuff. 2E) More services - like home security - are along the edge now and those need to work well all the time and interfere with the other services minimally.

Historically, for example, ISPs don't get along with gamers very well, because... their networks often have wildly variable latencies, and indeed all the gamer routers adopted fq_Codel very quickly, but getting the gamers off your back would be nice to do by default - and low latency is needed for videoconferencing, and more than one person sharing a link, so if only - wearing that bright shiny optimism I'd had back in 2012 - fq-codel were on by default in the ISPs router, we could move onto building out more useful services along the edge. ISPs know more about their network than the customer supplied routers can guess, and there's only 3 configuration variables to cake - upload speed, download speed, and link technology.

Lastly, there is the spectre of competition from starlink and from 5G. The ISP CAN do better than 5G (which has really terrible bufferbloat) by innovating in this area, providing orders of magnitude less queuing delay and shorter physical RTTs than either technology can. Web traffic, in particular, is almost entirely bound by the RTT, and the more you can do to have an RTT advantage over those two technologies the longer you will survive as an ISP.

Now, the deployment model to date was user driven - the home routers we built were fast enough to do download and upload shaping on their own, and we didn't need any ISPs to co-operate, but our end goal was that the ISPs would replace their HTB + FIFO (typical) or HTB + SFQ (much better but has scaling limits above about 20Mbits or so) in the routers they delivered to customers and in their headends with htb + fq_codel or (now) cake.

There's more to talk about in the last mile arguments here, I'll do a post on backhaul and on key metrics like "utilization", and then actually try to answer the questions positied at the start of this thread, but it's just generally been my hope that the tech (which is open source software, ietf standardized) would, essentially, sell itself in terms of delivering a vastly better internet for everybody if you just turned it on at every potential bottleneck link,
Last edited by dtaht on Thu Dec 23, 2021 5:30 pm, edited 3 times in total.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Utilization is often a lousy metric

Thu Dec 23, 2021 6:05 am

Utilization as a metric is useful for backhauls and other links that have a large amount of already statistically mixed traffic from many sources. It's well known also that it's also a lousy metric, in that 50% utilization over an interval might actually mean 100% utilization for half that interval and 0 for the other half, and there is not a lot of useful information in that average, but as a rule of thumb, ISPs upgrading a link once it cracks 50 and especially 80% utilization over a 5 minute interval has been a pretty good one. That said, what is really going on when you have any degree of measurable utilization to the packets themselves? If you have packet drops on a link that appears only 50% utilized, is that a line... or statistical sampling... error? What's the delay when that happens? What's the jitter as you go back and forth?

Similar problems with statistical sampling are your classic pingplotter tests which do show, over time, delay and loss, but usually not on an interval that makes sense. Packet loss is essential for the internet's correct operation, if you are NOT showing packet loss, it's possible you are doing it wrong, which is in part where the overbuffering across the internet came from. Sporadic delay statistics didn't show the actual behaviors of the TCP stacks when under load when, first, bittorrent hit, and later, netflix, and few test rigorously for a continuous up/down/udp/icmp ping load that represents the worst case scenario for a network - someone doing an upload, while doing a download, while making a videoconferencing call. 'Which really, isn't such a worst case after all.

We've tried to fix this test regime with the flent suite, particularly the rrul and rtt_fair tests that we used to tune things up over on the link this convo started. A better measure, than any form of utilization, is to actively measure periodically, what actually happens to your network when it is 100% utilized. If you know what happens then, and it's good, you're done, and you can move onto tracking more useful statistics such as queue backlog,
drops, marks and FQ reschedules rather than just utilization as to making decisions for adding more links or routing traffic differently.

Some other passive measurements of TCP rtt have appeared (Kathie Nichol's Passive Ping utility) which can measure the actual transit time of those kind of flows and make other adjustments. Preseem and LibreQos are doing this.

So anyway, this info basically fed forward into https://datatracker.ietf.org/doc/rfc7567/ - that every fast to slow transition have some form of AQM on it. The IETF considers FQ optional, I don't, but the world has bifurcated into switches with short buffers for the data center and bigger ones for the outside world, where inbetween those two possibilities is the idea of AQM being universally on. Networks degrade gradually under excessive load with a quality AQM tech in place (which RED is not, and pie and codel are). A massive failure (like level3 going down) results in increased packet loss, but no increase in internet latency, the biggest flows slow down, the smallest are less touched, it's far less catastrophic.

Now along the last mile, utilization is a particularly lousy metric. A web page usually takes about 3 seconds to load, and it's actually hard to peak at more than 20Mbit at any given instant. Then the link goes idle for 10s of seconds, and then :wham: another web page, and that sort of self inflicted damage on other flows - like voip or videoconferencing - is really hard to measure, except in human annoyances.

I am big on explicit fair queuing, everywhere, but especially along the last mile, in wifi, and other wireless technologies, because unlike in a data center where you might "step down" a 100Gbit link across 10 LAG 10 gig ports (this is essentially physical fair queuing), you've got just the one link, and multiple devices on the other end and bandwidth that is often physically variable from 1Mbit to a Gbit. Microbursts at 100Gbit are enormous bursts at 10Mbit, and it helps a lot to break those up back into packets every time you step down. Even after you've got a good statistical distribution of packets, signalling needs to be provided e2e to tell the transports to slow down, with a minimal amount of buffering. Prior to fq-codel, fq-pie, or cake, there were two schools of thought - WRED, which used dscp markings and dpi to find some way to reduce queue length, and SFQ, which broke up flows 127 ways, and in-between, and especially on wireless, nothing worked!!

RED is really finicky to configure and only suited to isochronous links. SFQ starts having trouble scaling above 20Mbit with its default queue lengths. DRR can be made to scale too but five tuple FQ is much more effective and simple than a ton of dpi, and rewriting the dscp fields that happens in the DC today. BTW Cisco Nexus 9000's AFD is pretty good...

Anyway that gets us up to 2012 when pie and codel appeared on the scene and solved most of RED's problems. Neither are finicky, and can be used without FQ in a system that is already well statistically mixed, or re-done in a WRED-like configuration, running at data center speeds, and especially on variable rate wireless technologies, but it turned out five tuple fq-codel (later fq-pie and cake) solved a lot more problems on the servers, hosts and home routers, as well, and thus fq-codel is what's seen the most deployment to date. Applied at line rate, everywhere, it smooths out network operations and improves bidirectional throughput, in particular.
Last edited by dtaht on Fri Jan 07, 2022 7:42 pm, edited 1 time in total.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Thu Dec 23, 2021 5:13 pm

"My understanding is that everything Cisco and Juniper is wred. It can handle huge bandwidth amounts due to offloading to the ASIC, but is almost certainly much worse than any of the newer AQM solutions. I believe those running Cisco and Juniper have no ability to even consider codel or fq_codel or cake on the service provider side. "

Cisco has AFD now. Both providers have switches that can do P4 for which there is fq-codel. But it is still early days for the big iron and offloads there.

"I suppose the difference is how much these AQM technologies are really designed to be used on the client side vs the service provider side. If I was daring enough to upgrade our core routers to 7.1 (I am not), what would make the difference - should we do cake queues for each customer at the ISP end, or should we deploy local cake queues to their routers, or should we do it on both ends? And if we should do at both ends, then why?"

It would be pretty cool if fq-codel or cake were backported to the most popular stable shipping mikrotik release. That's how we got deployment on ubnt's gear, initially.

You just used the word "core routers" and mikrotik in the same sentence. My impression was that the big iron was core, mikrotik more of a leaf? In all honesty I don't know much about mikrotik's stuff outside of their wireless gear and a few switches. Another thing I don't know much about is the prevalance of MPLS, or even the possibility of running a fiber interface on whatever standards deloyed for it at a physical line rate?

Ideally an ISP would running fq_codel everywhere at line rate (using sch_fq instead for tcp heavy servers), and replace an existing HTB + FIFO shaper (if you have any, and I really hope that the default for most ISPs in the mikrotik market is presently HTB + SFQ) with HTB + FQ_CODEL, or cake, flowing in the customer's direction, and the CPE, cake on the uplink.

Inbound shaping is more cpu intensive than outbound (and less accurate), and yet at the same time most home routers have so much more CPU that stuff at the headends that it may well be that a primitive shaper at the ISP is what remains, in use, in which case, inbound/outbound shaping via cake with better communication from the ISP of the actual configured rate to the CPE or router behind it (And and ongoing changes thereof), is one of my deepest wishes. There are PPPoe and DHCP messages for that now, and cake was designed to take a message and be able to dynamically change the
bandwidth to the link without interruption of service. Most of the confusing noise on the web today is from users attempting to correctly infer the ISP's set rate, and it really needn't be this complicated. Evenroute, for example, has means of scraping quite a few DSL modems built in, but it would be so much better to have a dsl modem or ONT that told you it's rate so a cake could set it.

"I ask this because most of the documentation I have seen about cake and fq_codel is about what the client should do rather than what the ISP should do. There is a lot of useful information geared towards the regular end user, but very little geared towards ISPs who want to improve services for their users. "

Until now there was very little direct ISP deployment (though evenroute and eero I am told do do deals with ISPs). I'd like to change that. My hope was that a string of smaller, more nimble mikrotik based ISPs would leap all over this stuff, share data, build better backends, or, like preseem/libreQos, etc, build something that leveraged this tech that could actually have a business model. A blog from an ISP that deploys this would be nice. I LOVED that Comcast published this B/A comparison study: https://arxiv.org/pdf/2107.13968.pdf they were rolling out DOCSIS-PIE and had one modem with a broken implementation of that so they used that opportunity to do some really compelling, unique science.

Me, wearing my scientist's hat getting a chance to look at and publish an ISP's data as they deploy, would be nice. There's more problems in this world than just network queuing, too.

It's kind of hard to monetize or support or write extensive documentation on something as simple to just turn on as fq_codel or cake are, essentially one line of configuration, each. In writing this and the page we started from ( viewtopic.php?t=179307 ) my hope has been to provide a deeper understanding of network behaviors, better benchmarking and analysis tools, and clear demonstrations of the end-user benefits of deploying these technologies to all the users. The rest is up to y'all. I'm very interested in feature requests for cake, and in benchmarks of enabling fq_codel on the higher end mikrotik gear. Benchmarks of web PLT, MOS scores and statistics about loss, marks, backlogs, and delay I'd love to see.

If I have any one feature request of mikrotik's future releases, is that I'd really like those statistics to be available, and secondly, we'd put in a couple nice optimizations for ipsec and wireguard in linux 5.7 for fq-codel that are very useful for site-site vpns.
 
tcroghan
just joined
Posts: 7
Joined: Wed Dec 12, 2018 9:42 pm

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Thu Dec 23, 2021 8:43 pm

David,

Thank you for your thoughts, I think info like this really helps people understand where your project is coming from and working towards. I am hoping to pick your brain some more though.

There are plenty of us running Mikrotik all the way from the customer to our core and back out to the internet. I have ROS v7.1 running for 90% of my customers and working on the last couple hundred. It's been a rough couple days dealing with v7.1, but I have been working on understanding CAKE and fq_codel for most of the last year and how to go about implementing it.

Right now I am running fq_codel on each of the customer facing interfaces with Mikrotik's simple queues, below I have the config I am using (for everyone's reference)
(Basically the Mikrotik default config)
/queue/type/add fq-codel-ecn=yes fq-codel-flows=1024 fq-codel-interval=100ms fq-codel-limit=10240 fq-codel-memlimit=32.0MiB fq-codel-quantum=1514 fq-codel-target=5ms kind=fq-codel name=fq_codel

/queue/simple/add bucket-size=0.1/0.1 burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s disabled=no limit-at=0/0 max-limit=6200k/15400k name="CustomerXXX" packet-marks=" priority=8/8 queue=fq_codel/fq_codel target=CustomerVLANXXX !time

So right now my customer's are being bandwidth limited by fq_codel + HTB, which is where I would see the vast majority of people being able to deploy this tech. Building a communication system between CPE and a customers router, an ISP's billing system and and and, just seems impractical without a unified standard (that doesn't exist) or a unified system which few people will be willing to enslave themselves to (and Mikrotik probably won't build anyways). Sure in theory you could force your customers to run your router, then we could tie everything together with some scripts and a decent billing system, but many of us are techies and don't like putting that kind of requirement on the people we serve. I also don't have a decent billing system, so I am going to build for what I can handle right now.

Notes for those who are reading, 7.1 seems to have a lower CPU load compared to 6.48.x. fq_codel also puts less strain on the CPU then my previous ethernet-default setting, which was a simple FIFO + HTB that is default from Mikrotik (Yes I know it was terrible, but it was the most stable one on my 1009s and I had fewer complaints with it then some of the other queues I tried in 6.x.x and I didn't have much knowledge of queues when I originally set this up.) RAM utilization is significantly higher with fq_codel though, routers with ~100 active customers went from ~380 MB of ram usage to over 560 MB of ram. There's plenty of room for that in my 1009s, but I figure it would be a good FYI for people.

Which leaves me with some questions I am hoping you can enlighten me and others on:

You mention running fq_codel with a HTB, do you have any thoughts about the size/settings of the HTB?

What do you mean by "fq_codel everywhere at line rate"? Do you mean setting the bandwidth that should be possible on each interface of a router/switch? I suspect that Mikrotik's Switches will not be able to handle this, not enough CPU and there's no hardware offload support at the moment. Rather unfortunate, but if we can have a router on each side of the switch that might be transitioning from 10G to 1G this could be mitigated by running fq_codel there correct?

Would you recommend CAKE being run with an HTB as well? From my initial testing having the HTB helps a lot with the CPU load on a router, but now that I have flent working I think CAKE without the HTB performs better.

That said, I have not been able to sit down and get a full understanding of what one is looking for with flent testing or what "good" is. At the moment my work has been to get the Ping CDF plot with a RTT Fair Realtime Response Under Load test to be as vertical as possible. This weekend I should be able to get some testing done and maybe setup my own server to contribute to the greater effort.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Thu Dec 23, 2021 10:48 pm

Hi Dave,

Thanks for the excellent information!

Our core and edge is all MikroTik. Some of the higher end routers like the CCR1036's can actually handle a fair bit of traffic. Ours is only about 2.3Gbps total traffic, but I know someone who has around 8Gbps of traffic at peak and they handle it just fine.

We deployed one of the recommended WISP models for MikroTik, which is to have PPPoE concentrators at the hub and to layer 2 tunnel all customers from the towers back to the hub with VPLS. So we have our three concentrators in our main datacenter that actually have the queues (currently RED for each customer) that do traffic shaping / rate limiting for the customer's connection. Because the concentrators are in our main datacenter, there is significant bandwidth there and all customer upload that makes it to the concentrator will make it to the internet.

Things get more complicated on the download side. They get shaped on the PPPoE concentrator but the concentrator has no knowledge of downstream congestion. This downstream traffic then moves to the core router (another MikroTik) which has a series of HTB for each layer 2 connection to the sites, to ensure that enterprise traffic and management traffic gets prioritized over retail traffic. This is again done with just a series of RED queues, so all retail traffic to a series of towers in a given area passes through a single RED queue. This has no knowledge of individual customers and streams and will just drop random packets when it is congested. After this it goes through a series of backhaul PTP links, some of which may be congested, and they do their own QoS (I'm not quite sure what mechanism they use, they do not say), before getting to the actual tower. After that, it goes over PtMP which in some cases may also be congested, before it gets to the customers router.

So to summarize this layout:

Internet <--> Edge <--> PPPoE Concentrator (individual customer RED queues) <--> Core router (red queues across third party L2 links to our tower, HTB 8 priority) <--> Tower1 <--wireless-link-with-qos--> Tower 2 <--wireless-link-with-qos--> PtMP AP <--> Customer SU <--> Customer router (PPPoE)

So I'm wondering given situations like this, how smart are fq_codel and cake for figuring out that there is downstream congestion in this type of scenario? The concentrator itself does not have visibility regarding what happens at the core router and the tower wireless links and the PtMP AP in the community.

Also CPU usage is a concern. Since MikroTik routers are software based, different queue types will consume different amounts of CPU. So I wonder if you could give some information as to the CPU usage of things like codel, fq_codel and cake compared with some of the legacy queue types like red and pfifo. If a PPPoE concentrator is already at fairly high load, I am wondering if something like cake could completely overwhelm it.

I am also wondering how they would work over backhaul links with hundreds of customers. Again most of the things I see about fq_codel and cake are for individual customer shaping, but I am wondering how well this translates to situations where you have 600 individual customers traffic passing through a single fq_codel or cake queue and you have congestion. In my case it is more complicated with MPLS which these queue types will not understand, but for other ISPs there may be no MPLS involved and the queue would still have some visibility into the traffic.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Thu Dec 23, 2021 10:51 pm

It would be pretty cool if fq-codel or cake were backported to the most popular stable shipping mikrotik release. That's how we got deployment on ubnt's gear, initially.
It might be very hard for them to backport as the kernel that RouterOS v6 runs on is very old (3.3.5), although I don't know how old UBNT's kernel was in this situation.

Regarding the prevalence of MPLS, it is everywhere in the larger ISP setups. It is not as common with ISPs of our size, but we have enterprise customers wanting layer 2 transparent LAN services in rural areas that they can't get elsewhere, so MPLS makes a lot of sense for us.

Unfortunately I have found that AQM that is not MPLS aware sometimes behaves really strangely when MPLS frames pass through it. For instance I tried to put SFQ on our core router for one of those third party layer 2 links to where our towers are, and performance dropped from like 800Mbps to 200-300Mbps. I reverted right away. It seems that SFQ was seeing all MPLS traffic as being a single stream and so they were all put through the same pfifo within the MPLS, so 600 customers all crammed into a single one of the 1024 SFQ pfifo substreams.

I suspect something similar may happen with fq_codel and cake when the traffic for 600 customers passes through a single queue and all have MPLS labels.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Fri Jan 07, 2022 3:05 am

I have been away for the holidays and not paying too much attention to the internet. I'll be back for real next week. Thank y'all for sharing your topologies. Quick question: What statistics are you currently able to collect on the behavior of your bottleneck links (via snmp... or?)

Also, a current linux tree I checked has support for doing some degree of flow dissection on mpls: https://github.com/torvalds/linux/blob/ ... issector.c but I don't know of what linux version that capability arrived, nor how deep the flows can be dissected to get at the headers. Re:

"Unfortunately I have found that AQM that is not MPLS aware sometimes behaves really strangely when MPLS frames pass through it. For instance I tried to put SFQ"

There is a lot of confusion in the market of the meaning of the term AQM. Some want to fold "FQ" into that, and here is a case where conflating these terms hurts. An AQM is simpler than FQ - it just pounds on the queue until some "target" is met, and does not care about individual flows, so RED/Codel/Cobalt/Pie over MPLS would (probably) have worked in your case. SFQ needs to see the headers, and tends to have short queue lengths in the first place but it is not "actively" managing the queue length.

But: a "pure" AQM will drop a lot more (and often mistakenly) packets and have wider latency variability than a fq+{aqm}. Still, FQ by itself does not "solve" bufferbloat, it just bypasses it for large categories of flow types, and also makes more possible delay or delay + ecn based congestion controls like ledbat, BBR that work more often. FQ+AQM wins for more of everything (but as one side effect lebat's delay budget is 100ms, codel's is 5ms, so ledbat "degrades" to reno behavior. We showed that 5ms and reno behavior was in general far better than ledbat's behavior (for limited numbers of flows for either FQ or AQM solutions) back in the early days ( https://perso.telecom-paristech.fr/dros ... mnet-b.pdf ) )

Anyway, after *years* of discussion on the bufferbloat and ietf mailing lists, we ended up trying to create a new term that meant more and covered more of the bases, called "SQM".

https://www.bufferbloat.net/projects/ce ... anagement/

Maybe it will take off. (apple referred to it recently) Maybe it will make wikipedia. Maybe we need to tighten the definition or file for a trademark. Maybe it will be futile. I have no idea. Certainly killing off "QoS", also, in this context, has long been on my mind.

And at least modern versions of the linux flow dissector can distinguish between flows on MPLS to some extent and MPLS + "SFQ or Fq-Codel" are now worth re-testing on miktrotik 7.1. Same goes for red, codel or pie standalone on mpls as a point in comparison.
Last edited by dtaht on Sun Jan 09, 2022 8:24 pm, edited 2 times in total.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sun Jan 09, 2022 8:13 pm

I note I am enjoying "jamming with an audience" and hope in the end we end up with a quality reference implementation guide for mikrotik's stuff (very sad to hear ipv6 is currently borked). I think there is a wonderful "pile of money" and genuine benefit to be created in just america in the $70B rural broadband funds (see broadband.money) but have *no* idea how to get it, and finding funding for my basic role as an educator (or for more research) has always been hard. Also, I'd really love to somehow put together some quality animations of (for example) the difference between SFQ, DRR, and FQ-codel's fair queuing implementations, but am graphically challenged. So if anyone has ideas for keeping a roof over my head, or can throw in resources of any kind, I'd really appreciate it.

I just got back from a two week trip, and *every* hotel I stayed at (mostly cheap ones like best western) had a wifi implementation totally unusable for videoconferencing.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sun Jan 09, 2022 8:38 pm

It would be pretty cool if fq-codel or cake were backported to the most popular stable shipping mikrotik release. That's how we got deployment on ubnt's gear, initially.
It might be very hard for them to backport as the kernel that RouterOS v6 runs on is very old (3.3.5), although I don't know how old UBNT's kernel was in this situation.
BQL landed in linux 3.3. It was the core innovation in making all this stuff work correctly and near zero cpu impact at line rate, but it was improved multiple times (and required explicit driver support to enable which landed for hundreds of ethernet devices over many releases). Ubnt was stuck at linux 3.10, which is as far back as we ported fq_codel and cake. So mikrotik really, really does need to move forward, and I'm also generally always pointing at ipv6 also really needing a kernel that is less than 4 years old. I do kind of wish the 7.1 release was based off a 5.10 or other *stable* kernel. I would like to find a mikrotik product that has known "bql" support in the device drivers.
Regarding the prevalence of MPLS, it is everywhere in the larger ISP setups. It is not as common with ISPs of our size, but we have enterprise customers wanting layer 2 transparent LAN services in rural areas that they can't get elsewhere, so MPLS makes a lot of sense for us.

Unfortunately I have found that AQM that is not MPLS aware sometimes behaves really strangely when MPLS frames pass through it. For instance I tried to put SFQ on our core router for one of those third party layer 2 links to where our towers are, and performance dropped from like 800Mbps to 200-300Mbps. I reverted right away. It seems that SFQ was seeing all MPLS traffic as being a single stream and so they were all put through the same pfifo within the MPLS, so 600 customers all crammed into a single one of the 1024 SFQ pfifo substreams.

I suspect something similar may happen with fq_codel and cake when the traffic for 600 customers passes through a single queue and all have MPLS labels.
@mducharme

I exited the ISP market before there were many independent implementations of MPLS, and viewed it at the time as another CISCO attempt at "vendor lock". I was also painfully aware (at the time) that ipv6 didn't work worth a darn on a bunch of "core"-ish routers. I am under the impression that it's possible now to get away from MPLS but to me that would be some other document with some other expertise contributed!

That said, as I wrote a post or two up, I *think* linux's packet dissector may now up to the job of FQing MPLS (and should always have been capable of some form of AQM)

However, two other things - older versions of SFQ had only 127 queues, (configurable) and a very short (128?) queue depth and at least part of your thoughput problem at the time was that the default SFQ queue depth was too short at that speed even and especially with everything landing in one queue. Probably.
pfifo_fast defaults to 1000 packets.
Last edited by dtaht on Sun Jan 09, 2022 10:55 pm, edited 1 time in total.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sun Jan 09, 2022 10:24 pm

David,

Thank you for your thoughts, I think info like this really helps people understand where your project is coming from and working towards. I am hoping to pick your brain some more though.

There are plenty of us running Mikrotik all the way from the customer to our core and back out to the internet. I have ROS v7.1 running for 90% of my customers and working on the last couple hundred. It's been a rough couple days dealing with v7.1, but I have been working on understanding CAKE and fq_codel for most of the last year and how to go about implementing it.

Right now I am running fq_codel on each of the customer facing interfaces with Mikrotik's simple queues, below I have the config I am using (for everyone's reference)
(Basically the Mikrotik default config)
/queue/type/add fq-codel-ecn=yes fq-codel-flows=1024 fq-codel-interval=100ms fq-codel-limit=10240 fq-codel-memlimit=32.0MiB fq-codel-quantum=1514 fq-codel-target=5ms kind=fq-codel name=fq_codel

/queue/simple/add bucket-size=0.1/0.1 burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s disabled=no limit-at=0/0 max-limit=6200k/15400k name="CustomerXXX" packet-marks=" priority=8/8 queue=fq_codel/fq_codel target=CustomerVLANXXX !time

So right now my customer's are being bandwidth limited by fq_codel + HTB, which is where I would see the vast majority of people being able to deploy this tech. Building a communication system between CPE and a customers router, an ISP's billing system and and and, just seems impractical without a unified standard (that doesn't exist) or a unified system which few people will be willing to enslave themselves to (and Mikrotik probably won't build anyways). Sure in theory you could force your customers to run your router, then we could tie everything together with some scripts and a decent billing system, but many of us are techies and don't like putting that kind of requirement on the people we serve. I also don't have a decent billing system, so I am going to build for what I can handle right now.

Notes for those who are reading, 7.1 seems to have a lower CPU load compared to 6.48.x. fq_codel also puts less strain on the CPU then my previous ethernet-default setting, which was a simple FIFO + HTB that is default from Mikrotik (Yes I know it was terrible, but it was the most stable one on my 1009s and I had fewer complaints with it then some of the other queues I tried in 6.x.x and I didn't have much knowledge of queues when I originally set this up.) RAM utilization is significantly higher with fq_codel though, routers with ~100 active customers went from ~380 MB of ram usage to over 560 MB of ram. There's plenty of room for that in my 1009s, but I figure it would be a good FYI for people.
At rates below 1gbit, you can safely reduce the fq_codel memlimit for packet buffering to 8MB (or less if you want to do some math). fq_codel's basic overhead is 64 bytes per queue; where you are most likely using up the most more in your configuration is in coping with misbehaved traffic, which will hit the memlimit before being dropped robustly via the "drop_batch" facility. In general fq_codel by itself barely shows up as a blip on the cpu meter compared to a simple fifo and by having shorter queues in general, you can actually keep memory "hotter" in the L1 or L2 cache.

I keep hoping to hear of a successful application of fq_codel + bql at line 10Gbit rate on mikrotik. :)

Cake has about 2.5 times more cpu overhead than fq_codel does. My hope was that if you ended up using less tc filter rules with it that it would
be comparable in many scenarios to a complicated htb + fq_codel setup, and that (selfishly) people would prefer the relative ease of cake setup and additional benefits, and throw hardware at the problem until it subsides.

HTB is is the 95% of the overhead killer, both in cpu locks and general context switching overhead, no matter the sub-qdisc. We've been trying to kill off the locks for years but the major innovations there are now happening mostly in the ebpf world. I don't currently understand what the bucket-size parameter you are using here translates to. Another optimization for your max-limit=6200k/15400k is you can use a smaller quantum 300 and
get less latency for the smaller packets at a cost of up to 6x more fq-codel-related cpu in some cases. My 2012 "wall" for when to use quantum 300
or not with 1989 style mips hardware was about 40Mbits or less. Cake uses its own curve for the quantum.
Which leaves me with some questions I am hoping you can enlighten me and others on:

You mention running fq_codel with a HTB, do you have any thoughts about the size/settings of the HTB?
The current state of the art is embedded in the sqm-scripts. They have a debug mode where you can put in your parameters for bandwidth and
then see the params for htb. Cake will also output its calculations. Having a separate htb config "calculator" would make some sense but its relative to the speed of the cpu, the context switch time, the other workloads. Right now the ebpf work is easier to reason about. I guess my rule of thumb is
basically start worrying when "top" shows more than 50% of a single core in use.
What do you mean by "fq_codel everywhere at line rate"? Do you mean setting the bandwidth that should be possible on each interface of a router/switch?
You should be able to set the line rate bandwidth (e.g for ethernet 10Mbit/100Mbit/1Gbit/2.5Gbit/10Gbit) and get adaquate backpressure from the
the device to be able to apply an aqm without a shaper.

I've been getting this a lot, where people think this is a shaper-only needed technology. It's certainly a major use case but totally not the intent of algorithms which were designed to intelligently step down from any fast -> slow transition, everywhere, and especially on wireless tech, as we thought would happen in 1992 when RED was standardized. I have high hopes for wider deployment in 2.5gbit to 1gbit interfaces, for example. Nowadays, fq_codel is the default qdisc at ethernet line rate for most of linux, and all of ios and osx now. It just sits there, breaking up microbursts, keeping queue lengths short, pretty invisibly. OpenWrt actually removed pfifo-fast from their codebase entirely. It would be great if mikrotik did that too. The biggest application for fq_codel is it's on by default, in hundreds of wifi products more or less silently "doing it's job". If it wasn't in there, wifi mesh routers like eero's in particular, would have been still-born. I'm rather proud of the wifi work we did as finally solving the p2mp problem wifi has always had as well as the wifi "performance anomaly. Essentially had been working that problem for 16 years. http://the-edge.blogspot.com/2010/10/wh ... based.html

fq_codel and pie work beautifully with ethernet pause frames as another example and would massively improved ethernet over powerline.

Although that "line rate" wifi implementation got folded into linux 4.12 I don't know to what extent that too made mikrotik's products at this point either! Qualcomm did do a pretty decent out of tree firmware implementation eventually I'm told, but the outputs of the codebase we created are shown over here: https://forum.openwrt.org/t/aql-and-the ... vely/59002 but right now it's the latest mediatek mt7x chips that are really rocking the world latency wise with their wifi 6 implementation of our stuff.

anyway to sum up: fq_codel at the native line rate of the intereface is useful all by itself. You can see it in action on nearly any linux box today via tc -s qdisc show. On osx it's netstat -qq -I the_interface_name. On linux wifi on the supported chipsets if you see an "aqm" file in sysfs it's there. I'd really like mikrotik (and the whole world) to kill off their per-packet fifos entirely, as openwrt did, and the internet everywhere, would get subtly better for all applications, especially when under load.

It's setting up the shapers at something other than the native line rate that's so darn finicky.

Is there some other phrase I could use besides line rate? native rate? that is in common use in the world?

Arguably what I would design for an ISP <-> multi-customer interface would be a "veth"-like interface fq-codel or cake on it) with a bql-like multiplexor at the bottom (in hw), (not a token bucket), and I'd punt the uplink shaping to the customer device via some message, and use a routing table for their IP address ranges.

I suspect that Mikrotik's Switches will not be able to handle this, not enough CPU and there's no hardware offload support at the moment. Rather unfortunate, but if we can have a router on each side of the switch that might be transitioning from 10G to 1G this could be mitigated by running fq_codel there correct?

Would you recommend CAKE being run with an HTB as well? From my initial testing having the HTB helps a lot with the CPU load on a router, but now that I have flent working I think CAKE without the HTB performs better.

That said, I have not been able to sit down and get a full understanding of what one is looking for with flent testing or what "good" is. At the moment my work has been to get the Ping CDF plot with a RTT Fair Realtime Response Under Load test to be as vertical as possible. This weekend I should be able to get some testing done and maybe setup my own server to contribute to the greater effort.
I imagine most of mikrotik's switches are short buffered and lack advanced features, maybe not even red?

There is a role for transparent middleboxes like preseem's in networks. It's also pretty easy to roll your own... but (I have no business relationship
with preseem) I adore having the great statistics they provide. LibreQos has a long way to go there, and so far, I still don't know what sort of stats
can be usefully obtained from mikrotik.

in general, no, run cake without an htb and use the native integral shaper. Two exceptions - if there's a hardware offload for the htb, use that, and if you have a complex sharing setup where you are trying to multiplex a ton of users onto a link htb might be a better choice than what would be
my preferred choice of per customer DRR + cake bandwidth X.

PS aiming for a perfectly flat CDF is futile and makes me nervous about people "engineering to the test'. I've ranted already about speedtest but didn't point out that if you optimize for the first 3 seconds or so of speedtet and ignore the rest, it's still a bit useful. I'd really hate to wake up 10 years from now to find that all the new hardware offloads "optimized for rrul"... by offloading exactly 4 up and 4 downloads, and having 3 channels for udp traffic and one for icmp, for example! Feel free to get to the upper range of the cdf with say 10ms extra latency at the 98% percentile at 10Mbit on that test and move on. Also try to get a feel for the behavior of that test at 100Mbit rather than 10 (as the latency will get flatter) but hit THAT with like 128 flows to watch it go to hell in its own way.... and look at both the long term stats for other notable glitches like some we've encountered on the other thread... the best result we had from the rtt_fair test was really pleasing... rrul + PLT is a great benchmark....

aggh... the prospect of a network world designed for the rrul test series would be a much better one than the speedtest world we are in now, but I can think of all sorts of traffic types (like videoconferencing and videocamera feeds) that are worth having more information about than what those tests currently do! I sometimes wake up at night terrified someone will find an exploitable flaw in fq-codel or cakes behaviors and have 4B+ users descend upon me and my project for not being able to fix it quickly on the infitesmal budget we have had.
Last edited by dtaht on Sun Jan 09, 2022 11:28 pm, edited 1 time in total.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sun Jan 09, 2022 10:52 pm

Over here is my demo of how the entire field/world mis-understood the benefits and problems of wifi packet aggregation in the 802.11n standard and later.

https://www.youtube.com/watch?v=Rb-UnHDw02o&t=1540s

If somehow millions more folk spent 8 minutes of time watching that segment demand for the "right stuff" would skyrocket. I'd love to do a string of animations about how wifi really works. And I didn't manage to cover media access delay or EDCA in that talk, and most of the marketplace still has a "brick wall" where you get a little too far from the AP and performance drops off a cliff because they don't get the aqm component right...

https://blog.cerowrt.org/post/real_results/

but that sells more APs per household. :(

It wasn't just wifi aggregation we fixed in that project, but also per station scheduling, and airtime fairness, and it does hurt me to know that hundreds of millions of APs and billions of wifi devices still don't have these fixes in them clogging up wifi for everything else. I really hope that the wifi-6e products entering the market place, on the new 6Ghz spectrum, are getting all these fixes right, but I've only got preliminary (and wonderful) results back on the mediatek mt7x chips so far.

anyway, thx for listening. I look forward to happy deployment stories and more questions as we go along. I'm still looking for a representative
mikrotik box to add to my testbed...
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Mon Jan 10, 2022 6:19 pm

I do kind of wish the 7.1 release was based off a 5.10 or other *stable* kernel.
People were asking for Wireguard support which at the time I believe was not yet available in any LTS kernel, so they went up to the current kernel at that time. I'm sure they will be upgrading to a newer LTS kernel in the coming months.
I am under the impression that it's possible now to get away from MPLS but to me that would be some other document with some other expertise contributed!
There is no need to "get away" from MPLS - it is not something that those who run it wish to abandon. There are no real suitable substitutes, but there is also no downsides to using it. A lot of us want MPLS-SR (segment routing) support from MikroTik. MPLS should not be viewed as something old that people are trying to get away from. WISPs and other ISPs should be moving towards MPLS, not trying to get away from it.

There is a standard for carriers that one of our upstreams uses called carrier ethernet (part of MEF) which they use to provision layer 2 circuits from us. It is painful to hear them configuring things though - we are sitting on the phone when they reconfigure like 20-30 different routers one at a time for our circuit, and they forget something so it doesn't work and then it is another hour of troubleshooting, all in the middle of the night during the change window. Meanwhile on our MPLS network, we just pop up a tunnel between two routers and we have provisioned a similar layer 2 circuit for our own customers and it takes a small fraction of the time (5-10 minutes), and it is so simple that there are rarely any mistakes made.

So yes, if we wanted to "get away" from MPLS, there are alternatives that we could deploy that are much more work to manage and maintain, and provide no real benefits, all to get rid of MPLS which isn't broken in the first place. I want to use the better solution, not deploy something that is worse to make life harder for myself.

At the ISP level I would really like some of these queuing disciplines to have native support for MPLS. The typical way to do QoS on an MPLS network is with EXP bits and not DSCP for the actual IP packet being transported. It would be ideal if something like cake could support prioritization based on MPLS EXP bits for backhaul links.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Mon Jan 10, 2022 8:05 pm

At the ISP level I would really like some of these queuing disciplines to have native support for MPLS. The typical way to do QoS on an MPLS network is with EXP bits and not DSCP for the actual IP packet being transported. It would be ideal if something like cake could support prioritization based on MPLS EXP bits for backhaul links.
yea! the first actionable feature request I've had since the start of these threads. But I pointed out that a pure AQM might already work, and that the flow_dissector might already work also for fq-codel derived algos, with this kernel release. I am not in a position to test...

Secondly, cake supports putting things into 3 or 4 or 8 classes of service not just on the dscp bits, but on any "tc" filter you can write. Can a tc filter see the EXP bits? Can that be expressed in RouterOS?

Thirdly, there's the spectre of ECN, I still have not the foggiest idea if access to those bits is working, nor what the current standard is for sticking them (if possible?) into MPLS.

I asked some questions based on this of the bufferbloat folk on the bloat and cake mailing lists.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Tue Jan 11, 2022 4:08 pm

From the list:

If you just want to use cake with priority tins based on the MPLS "Traffic Class" (TC) field (i.e. the renamed original "EXP" field, see RFC5462), I think you can use a tc flower filter (https://man7.org/linux/man-pages/man8/tc-flower.8.html) matching on mpls_tc values. See here for some examples:
I
https://www.redhat.com/sysadmin/mpls-tc-linux-kernel

/Jonas

I'd asked:

Dave Taht <dave.taht@gmail.com>
Jan 10, 2022, 10:21 AM (19 hours ago)
to Cake, bloat

I noticed that sometime in the past 8 years the flow_dissector gained
support for dissecting mpls packets. I don't know how deep that rabbit
hole
goes.

Over here on this mikrotik thead
viewtopic.php?p=904422#p904422 the question
was asked about cake, the exp bits, and mpls.

In looking over this, would we slam cake onto the vrf? or?

https://blog.swineson.me/en/use-linux-a ... ls-router/

I have precisely zero experience with mpls. Is there an mpls expert in
the house?
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Tue Jan 11, 2022 7:52 pm

If you just want to use cake with priority tins based on the MPLS "Traffic Class" (TC) field (i.e. the renamed original "EXP" field, see RFC5462), I think you can use a tc flower filter (https://man7.org/linux/man-pages/man8/tc-flower.8.html) matching on mpls_tc values. See here for some examples:
I
https://www.redhat.com/sysadmin/mpls-tc-linux-kernel
Thanks, I was looking at this tc flower functionality before, but I did not really see any examples that showed how to use the EXP to do strict priority queueing where 7 is the highest, 6 is the second highest, 5 is the third highest, etc., ideally with one or two below-best-effort slots as priority 1 (and possibly 2 as well). It might be due to my unfamiliarity with tc.

Also, I asked MikroTik to provide a way to filter based on MPLS EXP bits into a queue, and it looks like the request made it to the list. However, it also appears that probably the solution would involve creating 8 different cake queues (one for each priority) in an HTB situation with a single parent queue acting as the token bucket. But from what you have been saying, it doesn't seem like cake is really designed to be used in an HTB with 8 different cake instances drawing from the same token bucket?
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Tue Jan 11, 2022 9:14 pm

I am not big on strict priority queues except under strictly controlled circumstances.

Cake does soft admission control which gets the game theory more right, all classes are guaranteed service, all classes can borrow.

For strict priority queues I might leverage the prio qdisc + fq_codel sub-qdiscs, and in the case of mis-marking, may the network gods have mercy on your soul. Or you can try the 8 tin version of cake and program selecting those via tc flower or diffserv markings

One long term thought was that we designed the cake 'models' to be pluggable, essentially. If there was a model for how you wanted your traffic to behave, it's one lookup table and one piece of custom code to create. So if there was a common ISP "model" we knew of, we could fold that into cake's C directly. The only things that made sense to us (from surveying a lot of existing shapers at the time) was besteffort, diffserv3 (wondershaper), diffserv4 (wifi), and precedence (8). MPLS support didn't exist in linux at all at that time!!

https://github.com/dtaht/sch_cake/blob/ ... ke.c#L2570

tc qdisc add dev thevrfdevice root cake bandwidth 20Mbit diffserv8. I don't know how you'd do ingress
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Tue Jan 11, 2022 9:36 pm

Hi Dave,

Thanks for the info. I can provide a little more info to you as well.

There is somewhat of a standard for ISP QoS when it comes to VLAN priority and MPLS EXP bits. Both VLAN priority and EXP bits (3 bit values) are handled on the following scale:

highest to lowest: 7,6,5,4,3,0,2,1

i.e. two levels below best effort. I was trying to do some research before on where exactly this scheme came from and the best I could determine is that it seems to have originated from some Cisco wifi QoS spec for WMM. But this scheme appears to be pretty standard across the vendors I have encountered and so is probably the closest thing to an ISP standard.

In our case we mark all of the packets with EXP and those EXP bits are copied to VLAN priority for radio transit as some radios cannot read the EXP bits directly. We use the following mapping internally:

7: routing protocols + monitoring system ICMP
6: management traffic (staff logging into devices)
5: enterprise customer priority traffic (traffic that they tell us they want to be handled priority in the case of congestion due to unexpected interference that takes a radio link down far below normal capacity)
4: enterprise customer normal traffic
3: retail customer priority traffic
0: retail customer normal traffic
2: background
1: scavenger

On our network we are not currently using 1, 2, and 3. 3 is intended for future use for retail VoIP. All traffic marked 4-7 also has a limit applied to prevent the traffic from dominating the link in event of some kind of misconfiguration or attack.

The enterprise customers buy "dedicated bandwidth" circuits from us where retail customer traffic should always be dropped if needed to ensure that they always get the capacity they are paying for. In some cases our backhauls are congested and so without this strict priority it is hard for us to ensure that the enterprise customers always get their bandwidth allocation.

Approximately 90% of our traffic at any given time is retail customer normal traffic, which falls into the best effort queue.
 
ojnas
just joined
Posts: 1
Joined: Wed Mar 09, 2016 11:02 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Tue Jan 11, 2022 10:35 pm

Thanks, I was looking at this tc flower functionality before, but I did not really see any examples that showed how to use the EXP to do strict priority queueing where 7 is the highest, 6 is the second highest, 5 is the third highest, etc., ideally with one or two below-best-effort slots as priority 1 (and possibly 2 as well).
I don't know about strict priority queueing but for using cake priority tins I think you could use the functionality described in the section "OVERRIDING CLASSIFICATION WITH TC FILTERS" here, i.e. using "action skbedit priority":

https://man7.org/linux/man-pages/man8/tc-cake.8.html

And the filter would be something like "protocol mpls_uc flower mpls_tc 7" if you for example wanted to match MPLS traffic class (i.e. EXP) 7. However, the documentation is sparse and I have not tried any of this (I don't have an MPLS network to experiment with) so I don't know if it would actually work.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Thu Jan 13, 2022 12:28 am

I would be happy to learn that this has not deployed; https://datatracker.ietf.org/doc/html/rfc5129 or if it did, what variant.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Fri Jan 14, 2022 3:32 am

I am going to take a break from this thread for a while. The bitag report I'd been working on was released yesterday and I have to
go deal with the political fallout. Please reshare; https://www.bitag.org/latency-explained.php

1) Can someone confirm that 7.2 perhaps has a working ipv6?
2) I'd love to know if the hashing algos in fq_codel and cake can actually get into an mpls header and FQ on this encapsulation.
3) dying to hear of higher end mikrotik boxes being tested.

I have talks scheduled with nznog and with ICSI later this month and have to go heads down on writing those. I've appreciated learning a bit more about ISP models for dealing with commercial and retail traffic and will think upon that and MPLS whilst I'm away.

Happy debloating!
 
zainarbani
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Thu Jul 22, 2021 9:42 am
Location: Pati, Indonesia

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Mon Jan 17, 2022 12:24 pm

I suppose the difference is how much these AQM technologies are really designed to be used on the client side vs the service provider side. If I was daring enough to upgrade our core routers to 7.1 (I am not), what would make the difference - should we do cake queues for each customer at the ISP end, or should we deploy local cake queues to their routers, or should we do it on both ends? And if we should do at both ends, then why?
in my opinion, deploy fq_codel/cake to customer routers might better if you're using ROS as your main backbone server.
using AQM on beta ROS 7 (as your core ISP router) might simply cause CPU bottleneck at the moments.
if you're on Linux, OpenWRT, PFSense, ... it might be possible to do fq_codel/cake at ISP end using LibreQOS which include xdp-cpumap-tc
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Tue Jan 18, 2022 3:47 am

1) Can someone confirm that 7.2 perhaps has a working ipv6?
I don't fully understand what you mean here as IPv6 is working in 7.1.1. There are some issues or things not implemented yet, but none of those qualify as IPv6 "not working"
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Tue Jan 18, 2022 4:10 pm

ipv6 and these queue types have issues documented here viewtopic.php?t=181705

I don't have access to the bug tracker: SUP-70209
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Tue Jan 18, 2022 7:40 pm

ipv6 and these queue types have issues documented here viewtopic.php?t=181705
This only impacts using cake with a simple queue shaper. It works fine with IPv6 when using queue trees instead or interface queues.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Wed Jan 19, 2022 3:28 am

I don't have insight into mikrotik terminology.

To me, an interface queue is one line of code that can look like this:

tc qdisc replace dev eth0 root fq_codel # This runs the egress interface at the L2 negotiated line rate (ethernet 10mbit, 100mbit, 1gbit, 2.5gbit, 10gbit)
tc qdisc replace dev eth0 root cake # This also runs at the L2 rate.
tc qdisc replace dev eth0 root cake bandwidth 200Mbit # bandwidth 200Mbit or the L2 rate, whichever is lesser

In none of these cases is there any difference between how ipv4 or ipv6 is handled. Unless there's a bug elsewhere.

An inbound shaper looks like this (taken from sqm example: https://github.com/tohojo/sqm-scripts/b ... nctions.sh ),
and elides the insertion of several kernel modules and other error checking...

tc qdisc replace dev eth0 handle ffff: ingress
ip link add name ifb4eth0 type ifb
ifconfig ifb4eth0 up # or ip set dev ifb0 up
tc qdisc replace dev ifb4eth0 root cake bandwidth 200Mbit # or htb and leaf classes
tc filter replace dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb4eth0
# This filter is ipv4 or ipv6 agnostic also

I see in several examples folk doing x.y.z.0/24 rather than this catchall rule. In that case I would expect folk to also be matching an ipv6 pattern for ipv6 enablement (and also icmp and arp might need to be handled specially)

To me a per customer shaper might use a "veth" interface. Anyway I don't have a mental model of the differences in mikrotiks' terminology, setup, etc.!

And the other thing that I don't understand is the need to specify a bucket of any kind with this setup.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Wed Jan 19, 2022 5:46 am

Hi Dave,

I can show you a screenshot in case it helps to clarify things.
Screenshot 2022-01-18 194412.png
I believe this "interface queue" is equivalent to your "tc qdisc replace dev eth0 root cake", it only affects egress and not ingress. There is no "bandwidth" setting exposed, so if you want to set a certain bandwidth amount you have to use HTB. This works for both IPv4 and IPv6 without issue.
You do not have the required permissions to view the files attached to this post.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Wed Jan 19, 2022 5:53 am

Queue trees can be either interface-attached (in which case they affect only egress traffic) or "global" attached, in which case they can affect ingress or egress traffic depending on how the packets are marked. Packets can have a mark applied using either firewall mangle rules or bridge filter rules.

Here is an example of a queue tree setup that I have at home using fq_codel on RouterOS v7. My understanding is that this is a fairly direct application of HTB, but I don't know what the "packet mark" corresponds with. By default, packets will have a mark called "no-mark".
Screenshot 2022-01-18 195254.png
You do not have the required permissions to view the files attached to this post.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Wed Jan 19, 2022 7:36 pm

Simple queues allow both ingress and egress to be limited at the same time. They are often used at the ISP side for rate limiting of a single customer. It is this that currently has issues with IPv6.
Screenshot 2022-01-19 093239.png
Screenshot 2022-01-19 093345.png
Screenshot 2022-01-19 093436.png
I hope all these screenshots potentially help to identify what tc parameters might be used underneath for each. I am unfortunately unfamiliar with tc outside of MikroTik (although I use Linux often enough) and do not always understand your specific terms for this reason as they are things that you would only encounter when using tc manually at the CLI instead of in MikroTik's implementation.
You do not have the required permissions to view the files attached to this post.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Wed Jan 19, 2022 9:54 pm

thx for attempting to meet my mind in the middle! Perhaps there is a mikrotik person that can fill in more of my blanks?
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Wed Jan 19, 2022 10:25 pm

thx for attempting to meet my mind in the middle! Perhaps there is a mikrotik person that can fill in more of my blanks?
Most likely this would require somebody to explain what is actually happening on the back end (CLI equivalent) for these configuration options I've shown you above for the different classes of queues you can have (simple queue, queue tree, interface queue). MikroTik does not share this data publicly to my knowledge, but maybe their support would be willing to provide details. Without further details, we can probably assume that interface queue is the same as what you think, just with no ability to specify the bandwidth amount (in which case you are stuck with it assuming line rate). For queue trees, since they use HTB, it is probably HTB even in the case of where there is only one queue tree instead of a heirarchy, but who knows what other settings they may be passing as defaults that are not shown in the UI. For simple queues, those also apparently use HTB underneath because you can also have parent-child heirarchies of simple queues.

So it looks to me like there are only two real ways of using cake on MikroTik:

- interface queue, which provides no bandwidth control and handles egress only
- HTB with only one queue inside the heirarchy (implemented using either simple queues or queue trees)

Is it suitable to have only these configuration options or is this too limiting? You don't seem to generally recommend the use of cake with HTB and a token bucket at all, so it seems like the current feature set potentially locks the user into a configuration that may not be ideal (a useless HTB when you really only have the one queue inside). The question is how much of a problem does this cause, and what impact does it end up having on end users?
 
noyp
just joined
Posts: 7
Joined: Tue Jun 21, 2016 2:05 pm

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Thu Jan 20, 2022 1:42 pm

Queue trees can be either interface-attached (in which case they affect only egress traffic) or "global" attached, in which case they can affect ingress or egress traffic depending on how the packets are marked. Packets can have a mark applied using either firewall mangle rules or bridge filter rules.

Here is an example of a queue tree setup that I have at home using fq_codel on RouterOS v7. My understanding is that this is a fairly direct application of HTB, but I don't know what the "packet mark" corresponds with. By default, packets will have a mark called "no-mark".

Screenshot 2022-01-18 195254.png
You might want to check this openwrt cake script with dscp mangling to see if it can be ported to mikrotik

https://forum.openwrt.org/t/ultimate-sq ... ript/53209
https://github.com/hisham2630/Ultimate- ... New-Script
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Fri Jan 21, 2022 3:13 am

You might want to check this openwrt cake script with dscp mangling to see if it can be ported to mikrotik
Are you asking for help with this? or trying to help me? If the latter, this is not helpful for my specific use case. If the former, I can explain how you could probably adapt this for RouterOS v7.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Fri Jan 21, 2022 6:00 pm

thx for attempting to meet my mind in the middle! Perhaps there is a mikrotik person that can fill in more of my blanks?
Most likely this would require somebody to explain what is actually happening on the back end (CLI equivalent) for these configuration options I've shown you above for the different classes of queues you can have (simple queue, queue tree, interface queue). MikroTik does not share this data publicly to my knowledge, but maybe their support would be willing to provide details. Without further details, we can probably assume that interface queue is the same as what you think, just with no ability to specify the bandwidth amount (in which case you are stuck with it assuming line rate). For queue trees, since they use HTB, it is probably HTB even in the case of where there is only one queue tree instead of a heirarchy, but who knows what other settings they may be passing as defaults that are not shown in the UI. For simple queues, those also apparently use HTB underneath because you can also have parent-child heirarchies of simple queues.

So it looks to me like there are only two real ways of using cake on MikroTik:

- interface queue, which provides no bandwidth control and handles egress only
- HTB with only one queue inside the heirarchy (implemented using either simple queues or queue trees)

Is it suitable to have only these configuration options or is this too limiting? You don't seem to generally recommend the use of cake with HTB and a token bucket at all, so it seems like the current feature set potentially locks the user into a configuration that may not be ideal (a useless HTB when you really only have the one queue inside). The question is how much of a problem does this cause, and what impact does it end up having on end users?
Re:

- interface queue, which provides no bandwidth control and handles egress only

It's a more generic problem than that, I suppose. Being unable to provide any parameters at all to any more advanced qdisc on the base interface queue means you can't set flows (for sfq/fq_codel/etc), rtt, or other stuff. I hope they add that feature. I will keep repeating that fq_codel is a general improvement over pfifo_fast for most line rate interfaces, in general and hope more try it (and cake) there.

- HTB with only one queue inside the hierarchy (implemented using either simple queues or queue trees)

I can see why mikrotik is so tied to HTB, as when it was developed and for this market, it was the ideal solution. By and large in my world we've tried to improve it but it is still very "locky" and doesn't scale as well to multiple cpus as we'd like, and bursty, which is behind the moves elsewhere to XDP and eBPF. That said, mikrotik folk get a given box expecting htb's behaviors and costs, and thus I am not opposed to using htb + qdisc in mikrotik's case. Ideally everywhere you have a FIFO or SFQ today, put in fq_codel or cake, and watch the network improve. Being able to use cake's native shaper is a nice to have, but not required.

In order to cope with the ipv6 problem, they really need to be able to allow multiple filters pointing into the same customer qdisc. I'm not in a position to file these feature requests.

For inspiration in general I recommend looking over how preseem does things.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sat Jan 22, 2022 4:40 am

It's a more generic problem than that, I suppose. Being unable to provide any parameters at all to any more advanced qdisc on the base interface queue means you can't set flows (for sfq/fq_codel/etc), rtt, or other stuff. I hope they add that feature. I will keep repeating that fq_codel is a general improvement over pfifo_fast for most line rate interfaces, in general and hope more try it (and cake) there.
This isn't really the case, because the one thing I didn't take a screenshot of was the "queue types" tab that allows you to define settings for each queue type. The only reason I didn't screenshot this is because I was fairly sure that someone else had shared this with you before in one of these threads. For instance the following is the one for CAKE:
Screenshot 2022-01-21 183623.png
So you do get access to the cake settings in this way, and you can create another queue type that also uses cake as a "queue kind" but has different settings. However when I saw this in your examples:

tc qdisc replace dev eth0 root cake bandwidth 200Mbit # bandwidth 200Mbit or the L2 rate, whichever is lesser

Based on the syntax of the command, I interpreted that bandwidth setting as being not something cake specific but instead something that is generally available for interface queues (where you could set bandwidth 200Mbit for a pfifo queue if you wanted). Or is my assumption wrong and is this something specific to cake, and the same as the "Bandwidth Limit" setting under the cake settings in the screenshot above?

Similarly it appears that the ability to be able to handle ingress on an interface queue with the equivalent of "tc qdisc replace dev eth0 handle ffff: ingress" is not provided.
You do not have the required permissions to view the files attached to this post.
Last edited by mducharme on Sat Jan 22, 2022 4:47 am, edited 3 times in total.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sat Jan 22, 2022 4:44 am

For inspiration in general I recommend looking over how preseem does things.
I have looked at Preseem, but it is a non-starter for us because of its complete lack of MPLS support.
 
noyp
just joined
Posts: 7
Joined: Tue Jun 21, 2016 2:05 pm

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sat Jan 22, 2022 8:18 am

You might want to check this openwrt cake script with dscp mangling to see if it can be ported to mikrotik
Are you asking for help with this? or trying to help me? If the latter, this is not helpful for my specific use case. If the former, I can explain how you could probably adapt this for RouterOS v7.
Im using one of your queue tree script with dscp mangling now that cake is possible should i just change the queue type to cake in queue tree. Or is there anything more that should be configured in mangle or queue tree.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sat Jan 22, 2022 7:24 pm

Im using one of your queue tree script with dscp mangling now that cake is possible should i just change the queue type to cake in queue tree. Or is there anything more that should be configured in mangle or queue tree.
With cake you don't need the 8 leaf queues anymore for the different priorities, only the parent queue. The script you showed for openwrt had some additional mangling of certain websites to mark them in certain ways which might be useful to you, these markings can be done in RouterOS using address lists to store the sites and packet mark rules in the mangle table.
 
noyp
just joined
Posts: 7
Joined: Tue Jun 21, 2016 2:05 pm

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sun Jan 23, 2022 4:47 am

Im using one of your queue tree script with dscp mangling now that cake is possible should i just change the queue type to cake in queue tree. Or is there anything more that should be configured in mangle or queue tree.
With cake you don't need the 8 leaf queues anymore for the different priorities, only the parent queue. The script you showed for openwrt had some additional mangling of certain websites to mark them in certain ways which might be useful to you, these markings can be done in RouterOS using address lists to store the sites and packet mark rules in the mangle table.
Thanks, so for 8 leaf queues it needs to be remove and cake queue for the parent. What packet mark do put for the parent queue ? no-mark ? is cake aware of the dscp value from mangle ?
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sun Jan 23, 2022 6:52 am

What packet mark do put for the parent queue ? no-mark ? is cake aware of the dscp value from mangle ?
Yes, use no-mark for the parent, and remove any mangle rules that apply any mark other than no-mark. And, yes, cake is aware of the dscp rule from mangle.
 
noyp
just joined
Posts: 7
Joined: Tue Jun 21, 2016 2:05 pm

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Sun Jan 23, 2022 7:46 am

What packet mark do put for the parent queue ? no-mark ? is cake aware of the dscp value from mangle ?
Yes, use no-mark for the parent, and remove any mangle rules that apply any mark other than no-mark. And, yes, cake is aware of the dscp rule from mangle.
Thanks i'll tweak your queue tree script for cake
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Mon Jan 24, 2022 6:29 pm

@mducharme

No, I had failed to "connect the dots" on your terminology and subguis. (I'm still seeking a decent high end mikrotik box to try) Sounds like cake's shaper (bandwidth param) can be used natively using that last panel you showed, without an htb. Yay!

cake's "bandwidth" is not an "addon" parameter that could be used directly by some other qdisc like sfq, but integral to it, like the other params.

The "container" htb and tbf qdiscs inherit the default qdisc (in the bad ole days, pfifo_fast). There were a lot of shapers out there that used the default qdisc without explicit setup, so we'd hoped as this sysctl became common:

net.core.default_qdisc=fq_codel

that typical htb and tbf setups would just pick that up, just as they (sadly) picked up the change in pfifo_fast from 100 to 1000 packets by default in 2006.

Other shaper implementations (like I tend to think mikrotik is like) probably explicitly lay out all the params for you via the gui (with inheritance). Things to do on a migration would include not using packet limits anymore.

While I hope that we produce better documentation out of these long threads, I really long for getting detailed statistics out of these setups, in part to verify they are actually doing what we think they are!
Last edited by dtaht on Mon Jan 24, 2022 8:03 pm, edited 1 time in total.
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Mon Jan 24, 2022 6:39 pm

For inspiration in general I recommend looking over how preseem does things.
I have looked at Preseem, but it is a non-starter for us because of its complete lack of MPLS support.
I've been working on testing Cake on a CCR2116 and was able to shape at 10G line rate with no issue - i'm working on getting IPv6 up in my test network so I can try Cake with IPv6 as well but having some odd issues with OSPFv3 in 7.2rc1.

That said, being able to apply Cake with label switched traffic is a great idea and I'll probably try that next!
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Mon Jan 24, 2022 6:41 pm

In a more ISP-ideal world the ISP router would shape down to the customer, and the CPE router shape the up. It's more cpu efficient, and packets
get dropped (and acks filtered) before they hit the bottleneck link. Our efforts to do both sides of the shaping on the consumer end router was more an act of desperation than efficiency, although the advantages of handing both sides of this task to the far edge can be pretty compelling.
 
dtaht
Member Candidate
Member Candidate
Topic Author
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: For ISPS: Motivations and methods for implementing fq_codel and cake

Mon Feb 07, 2022 1:31 am

I am not much of a lobbyist, but if there is a mikrotik group representing WISP interests?, i'd like to talk to them about how to spend a billion $ better:

https://docs.google.com/document/d/1FjR ... PlPr8/edit

Who is online

Users browsing this forum: No registered users and 13 guests