Community discussions

MikroTik App
 
discoengineer
just joined
Topic Author
Posts: 12
Joined: Wed Jul 20, 2016 2:40 pm

FQ_Codel and Mikrotik CCR CPU Utilization

Fri Feb 10, 2023 2:12 pm

Hi Guys,

Ive recently upgraded one of my CCR1036-2S+ to version 7 which is being used as a router for my clients. Im performing traffic policy on it using simple queues (Majority CIR Connections) for each customer. Approx. have 350-400 simple queues on it. Ive tested fq_codel for simple queues to counter bufferbloat on 4 simple queues and it has worked as intended but Im a bit confused to implement it for all the simple queues just because Im not sure how will the CCR's CPU behave if fq_codel algo is selected in all simple queues. Can anyone share his/her experience of using this algo for such a large number of simple queues and the impact it has on the CPU of the CCR? There are no PPPoE sessions on this CCR. Just SVIs for each customer and in simple queues the target is set to the each customer's IP. Will using this algo for such a no. of queues increase the CPU utilization/choke any core during peak traffic (max. 2 Gbps in total)?
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Fri Feb 10, 2023 10:17 pm

I would just implement it and go measure, and be ready to roll back. Pound it flat with artificial traffic at 4am?

(I am one of the authors of fq_codel and cake, but I do not have enough data either, on how well this stuff scales on given bits of mikrotik hardware). In most cases it is the per customer shaper that dominates the cpu by a factor of 9, and the underlying queue type be it a fifo, sfq, fq_codel or cake, adds only a tiny bit (cake is about 2.5 times as slow as fq_codel but does more, and again, it is the shaping cost that dominates). If you are low on memory, for folk running at less than a gbit, you can use memlimit 8mbyte rather than the default 32Mbyte.

On x86 gear we have individual implementations of cake scaling to 10Gbit/core. On 500mhz single core mips, cake scales to only about 80Mbit. your mileage will be somewhere between those. :/

See also.

viewtopic.php?t=179307
 
discoengineer
just joined
Topic Author
Posts: 12
Joined: Wed Jul 20, 2016 2:40 pm

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Sat Feb 11, 2023 12:54 pm

Thank you for the detailed explanation. You are right! I will implement and check what impact it has on the CPU. I read your post and my my, what a gem of information it is. Ill test it out and share results as well. I have 15.2 GB memory free on the CCR so I think wont be needing to change memlimit. Ill share my findings in a few days.
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Sat Feb 11, 2023 2:11 pm

I do hope a table of performance metrics appears for more mikrotik gear. Sometimes I just hope they will put the fq_codel or cake algorithms more directly into their PCQ implementation. I am blogging more and more, and hoped that someday that insanely long thread on cake for mikrotik got boiled down into a compact usage guide. I look forward to hearing about your results!

https://blog.cerowrt.org/post/ has a few pieces on flent and such...
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Fri Feb 17, 2023 4:19 pm

Thank you for the detailed explanation. You are right! I will implement and check what impact it has on the CPU. I read your post and my my, what a gem of information it is. Ill test it out and share results as well. I have 15.2 GB memory free on the CCR so I think wont be needing to change memlimit. Ill share my findings in a few days.
I am really intensely curious as to how well this went? Did you try it, your router melt down, and lose your life to a lynch mob of unhappy users? Or... ?
 
Kevo
Frequent Visitor
Frequent Visitor
Posts: 67
Joined: Wed Oct 12, 2011 1:38 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Sat Feb 18, 2023 3:27 am

I recently added cake as an interface queue on one of my CCR routers that tops out at about 700Mbps of traffic. I left the bandwidth limit off as a test, and I haven't noticed any change and I haven't had any user complaints. I set it up using the dual srchost and dsthost options. The CPU usage on this CCR barely ever breaks 0% even with cake running. I have only 3 raw firewall rules since it's just a router, so there's not much extra taxing the CPU. I'm curious to know if cake will actually do anything without the bandwidth limit. This router is fed by wireless so I'm not sure the bandwidth limit would help all that much anyway since there is likely to be less bandwidth when there are issues and that's when I'd hope to see cake help keep things running well. So now I'm just waiting to see how it goes the next time we have a bad storm and the link drops modulations.
 
User avatar
sirbryan
Member
Member
Posts: 303
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Sat Feb 18, 2023 6:27 pm

Generally, queues need to get full before they become effective. If there's no pressure on the traffic, all the packets get thrown out there as fast as possible (FIFO). The bottleneck becomes the final delivery mechanism, i.e. loop to the house or WiFi to the end-user device.

The only place where I've seen QOS/COS useful without the queue being filled was on Cisco 3550's feeding Motorola (now Cambium) Canopy AP's. When we deployed VoIP, we were seeing (hearing) poor audio quality. Employing COS rules on the switch to get the DSCP-tagged VoIP packets out first, in addition to Canopy's QoS rules, allowed the VoIP packets to always get priority, regardless of how lightly loaded the AP/SM were.

Now (showing my ignorance), if Cake/fq-codel do reorder high-priority packets regardless of queue pressure, then I can see leveraging those queues on interfaces without shapers. In my simple WiFi speed testing at home, however, I don't see any results/benefits until I start maxing out the radio's capacity and apply a shaper that matches.
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Feb 22, 2023 8:55 pm

The word priority does not apply to fq_codel or cake (in besteffort mode). A better word would be "sparsity", the relevant paper on this is here: https://ieeexplore.ieee.org/stamp/stamp ... er=8469111 - but in a nutshell it translates to flows having an arrival rate of less than the departure rate of all the other flows, go out first. This is a natural fit for voip (20ms interval), in particular, but applies to many other forms of req/response traffic. Only flows that build a queue are (slightly) deprioritized, interleaved over all the other flows.

As for the effectiveness of the algorithms on less than full bandwidth, voip is very subject to very hard to see in the aggregate tiny bursts of latency - a 60ms delay or loss is audible, and 100ms, very much so. If you are not measuring your link at a nyquist frequency (half the samplerate, e.g. for voip that would 10ms), and instead looking at 5 minute averages, everything might seem fine, but the voice quality still be poor due to microbursts elsewhere in the network. So in general, always apply FQ in some form, AND classification if you can do it, even if your link never appears to saturate over a 5 minute interval - and someday, if you can, find a way to sample via nyquist's methods!

"you will see things, you people won't believe...
sawtooths flying through a gate..."

We are effectively sampling these days using ebpf at 10ms and I hope the technology makes it into more devices such as mikrotiks. Here's a demo (click run test, then select one of the slower sites)

https://payne.taht.net/

A much better metric for voip would be something like Glitches Per Minute - where someday, if we could get calls and videoconferences down to just glitches per hour, it would be a better world.
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Sat Feb 25, 2023 8:28 pm

I am, btw, fairly certain that more than a few > 1gbit ISP uplinks to the internet are behaving this badly:

https://blog.cerowrt.org/post/juniper/
 
User avatar
sirbryan
Member
Member
Posts: 303
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Mon Feb 27, 2023 5:43 am

I loaded just a few queues on my CCR1036. Initially they were all set as Cake with 600M to 2G for each queue, with a total of 6 or 7 queues.

Today, as the total utilization approached 2Gbps, the CPU load jumped to 100%. This is with all packets being tagged and assigned to one of the half-dozen queues. I changed the main queue to fq-codel and the utilization drops to 18-20%.

I'll have to try this on an ARM router (CCR2116) to see if it can handle the Cake work better than the 1036.
 
User avatar
sirbryan
Member
Member
Posts: 303
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Tue Feb 28, 2023 9:28 pm

Update:

I also put some rules on an RB4011 running 7.4.1, initially with Cake shaping all traffic going out to about six AP's, each on their own VLAN on the same ethernet port to the AP switch. Each queue was Cake with Internet RTT, Ethernet overhead, Ack-filtering, and diffserv4 (all else defaults). The queues themselves rarely hit 50-70% (only one ever hit 90%). CPU utilization was around 20%.

Today I got a call from a customer who complained that the last four days he'd been seeing packet loss, but hadn't made any changes on his end. I ran mtr to trace the path to his radio and found that I was losing packets at the router itself. After a few minutes of tinkering with the backhaul to rule it out, I disabled the queues on the router. I saw an immediate improvement, both to the router and to his radio. He likewise saw improvements in whatever tools he was using to test from his PC.

I changed the queue to fq-codel, and while the CPU utilization was lower, I still saw packet loss to/through the router, just not as much. As of now, I've disabled all queueing on the RB4011, turned fasttrack back on, and we're humming along at 6% load with 200-300Mbps continuous traffic.
 
User avatar
sirbryan
Member
Member
Posts: 303
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Tue Feb 28, 2023 9:49 pm

Update update:

For both the 1036 and the 4011, I disabled the shaper queues and created basic queues for the interfaces. The CPU load was causing more problems than perceivable benefits on both routers.

Even with basic settings for Cake on the 4011, I still saw packet loss increase. With fq_codel on the interfaces, it's working as expected.

On the 1036, Cake besteffort on the two SFP+ interfaces pushing 1Gbps has the CPU at 2-3%. With fq_codel on the interfaces, it's at 0%. No perceivable losses either way.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Tue Feb 28, 2023 11:45 pm



I'll have to try this on an ARM router (CCR2116) to see if it can handle the Cake work better than the 1036.

CCR2116 is DRAMATICALLY better than the tile platform. Tile is very slow for general purpose compute. Having a ton of crappy CPUs is ok... The ARMv8 cores in the CCR2116 must be like 10x faster per core for general purpose compute. I can get over 8Gbps one way on a single cake shaper between two CCR2116 with onboard mikrotik speed test.
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Mar 01, 2023 2:25 am

Packet loss is not a particularly good metric to use against a cake or fq_codel instance, as it uses packet loss to control congestion if RFC3168 is not enabled by the endpoints. In exchange for decreased latency, you get more packet loss. So in both fq_codel and cake you should have seen an increase in packet loss, and an improvement in network latency. Did the network "feel" better? Did videoconferencing and voip work better? are better questions to ask.

Now, if cake is actually mis-behaving due to cpu load or otherwise, and randomly inducing packet loss rather than intelligently dropping flows, then you are right to disable it.

What I try to do in a circumstance like this is to get a packet capture of a few test flows, and look at them via wireshark. It could very well be that your queues and offloads and/or applications are behaving better in this case with the fast path enabled, or there is a bug related to the offload + cake, or that the customer finds the range of latencies they currently experience acceptable.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Mar 01, 2023 3:08 am

Packet loss is not a particularly good metric to use against a cake or fq_codel instance, as it uses packet loss to control congestion if RFC3168 is not enabled by the endpoints. In exchange for decreased latency, you get more packet loss. So in both fq_codel and cake you should have seen an increase in packet loss, and an improvement in network latency. Did the network "feel" better? Did videoconferencing and voip work better? are better questions to ask.

Now, if cake is actually mis-behaving due to cpu load or otherwise, and randomly inducing packet loss rather than intelligently dropping flows, then you are right to disable it.

What I try to do in a circumstance like this is to get a packet capture of a few test flows, and look at them via wireshark. It could very well be that your queues and offloads and/or applications are behaving better in this case with the fast path enabled, or there is a bug related to the offload + cake, or that the customer finds the range of latencies they currently experience acceptable.
Dave, I see dramatically less packet loss with cake. Cake may be dropping packets as part of traffic control, but by managing the congestion other flows drop a lot fewer.

But back on topic, that CCR1036 has too slow single core performance IMO to do large shapers.

To the OP, as long as it's individual shapers and NO top level shaper, then CCR1036 will spread the shapers out on separate CPUs. If you do a top level shaper, everything in that shaper tree will be trapped on a single core.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2990
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Mar 01, 2023 5:40 am

i agree about limited single core performance on Tile Architecture, but in the test realized by sirbryan he replicated the situation in a rb4011 which has much better single core performance (it has OoO A15 CPU) at a rate normal for that router doing shapping 200-300mbps
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Mar 01, 2023 6:14 am

i agree about limited single core performance on Tile Architecture, but in the test realized by sirbryan he replicated the situation in a rb4011 which has much better single core performance (it has OoO A15 CPU) at a rate normal for that router doing shapping 200-300mbps
Rb4011 isn’t dramatically faster. It’s a 32bit arm v7.

That said, I have rb4011 running up to about 650Mbps aggregate with cake. I can get that reliably. I use them for backup backhaul links so I have fq-codel or cake handling the constriction.
 
User avatar
sirbryan
Member
Member
Posts: 303
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Mar 01, 2023 11:52 pm

To the customer, the feel was horrible, as he was no longer able to reliably play his games. He spent the previous four days testing things with friends, updating drivers, twiddling with settings on PC, etc. to no avail. Latency was all over the place and just in my ping tests we were losing one or two every 10-15 seconds. MTR reported 6-8% packet loss to the router and 4-5% to the radio.

I've reconfigured the queue setup on the 4011 and will report back. After some more digging, I think I "overconfigured" it initially, burdening the CPU with unnecessary tasks.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Mar 01, 2023 11:55 pm

I've reconfigured the queue setup on the 4011 and will report back. After some more digging, I think I "overconfigured" it initially, burdening the CPU with unnecessary tasks.
Feel fry to post a sanitized config.

also note, you cannot use fasttrack with shaping. I've seen it a few times that someone has the fasttrack firewall rule and shaping and it's not behaving as desired, only shaping things that have hit other firewall rules and so on.
 
User avatar
sirbryan
Member
Member
Posts: 303
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 02, 2023 12:03 am

Here's what I've whittled it down to:
/queue type
add fq-codel-flows=10240 fq-codel-limit=1024 fq-codel-memlimit=320.0MiB fq-codel-quantum=300 kind=fq-codel name=fq-codel
add cake-diffserv=besteffort cake-mpu=84 cake-overhead=38 cake-overhead-scheme=ethernet cake-rtt-scheme=internet kind=cake name=cake-interface
/queue tree
add limit-at=180M max-limit=180M name="LTU 192 " packet-mark=no-mark parent=vlan4001 queue=cake-interface
add limit-at=180M max-limit=180M name="LTU 203" packet-mark=no-mark parent=vlan4002 queue=cake-interface
add limit-at=180M max-limit=180M name="LTU 110" packet-mark=no-mark parent=vlan4004 queue=cake-interface
add limit-at=200M max-limit=200M name="LTU 227" packet-mark=no-mark parent=vlan4003 queue=cake-interface
add limit-at=150M max-limit=150M name=Airmax packet-mark=no-mark parent=vlan4010 queue=cake-interface
There's only one firewall rule presently, which is to drop any traffic destined for the public IP addresses assigned to the router from non-internal IP's.

The queues show the same amount of traffic whether or not fasttrack is enabled. Since these queues aren't tied to "global", I imagine that fasttrack should work. Either way, it's not impacting the CPU much at this point as traffic is just under 100Mbps right now.

(To clarify, this particular config is working with zero packet loss, so far. So the problem I was seeing was likely due to mangling everything coming into the router.)
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 02, 2023 1:59 am

I don't like how queue trees behave when explicit directionality isn't set via a packet mark.

Am I extrapolating that you have a vlan per customer from this tree?

If it were me, I would separate this out to:
mangle add packet mark LTU192DL where dst is vlan4001
mangle add packet mark LTU192UL where src is vlan4001

Then 2 queue trees for the separate DL and UL.

Also, I'd do diffserv4 instead of besteffort.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 02, 2023 2:06 am

Here's an example of how I build these. This is a CCR2004 so I can get away with a bit more than on a 4011 but principals are the same. This is a very effective model. I don't use vlans per so this is IP matched
match ip and mark UL and DL packets separately via mangle
Add a queue tree with NO markers to identify network structure.
add individual queue with packet mark and shaped speed.

currently doing fq_codel for 'groups' and cake for individuals.
Screenshot 2023-03-01 at 5.04.18 PM.png
You do not have the required permissions to view the files attached to this post.
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 02, 2023 8:42 pm

thank you for filling in here. I note that in a properly functioning fq_codel setup, you should see drops or marks. Not seeing them is bothersome.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 02, 2023 8:53 pm

to clarify here for the readers, the fq_codel and cake in routeros <=7.8 does not show drops or queued packets. I have a ticket open with mikrotik support on this that is marked 'pending'. Until that is resolve, you wont know if packets are being dropped.
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Sat Mar 04, 2023 4:54 am

I don't know if you also requested RFC3168 marks be logged also? A surprising amount of packets are being marked by fq_codel/cake etc, today, being signaled end to end by at least some (apple?s?) devices requesting ECN support. RFC3168 ecn is a way to do congestion control without dropping packets, and is baked into most OSes today, accepted by most TCP servers today, but rarely initiated by clients. I kind of suspect some embedded clients are negotiating it given just how much of it I have seen of late - or mismarking it - can't tell without logging it!

RFC3168 ECN is enabled by default in fq_codel and cake.

It's a one liner to enable in windows and linux transports.

https://www.bufferbloat.net/projects/ce ... nable_ECN/

Apple uses a variety of heuristics as to when they enable it, nowadays.

I always thought that the ECN support in fq_codel would make it easier for those with tough SLA requirements regarding both limiting drops AND latency (how?), to suggest further deployment of ECN enabled transports to their clients and servers.
 
Babujnik
newbie
Posts: 32
Joined: Fri May 05, 2017 2:15 pm

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Mar 15, 2023 6:20 pm

Here's an example of how I build these. This is a CCR2004 so I can get away with a bit more than on a 4011 but principals are the same. This is a very effective model. I don't use vlans per so this is IP matched
match ip and mark UL and DL packets separately via mangle
Add a queue tree with NO markers to identify network structure.
add individual queue with packet mark and shaped speed.

currently doing fq_codel for 'groups' and cake for individuals.

Screenshot 2023-03-01 at 5.04.18 PM.png
do You use FastTrack with this setup ?
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Mar 15, 2023 7:26 pm

Here's an example of how I build these. This is a CCR2004 so I can get away with a bit more than on a 4011 but principals are the same. This is a very effective model. I don't use vlans per so this is IP matched
match ip and mark UL and DL packets separately via mangle
Add a queue tree with NO markers to identify network structure.
add individual queue with packet mark and shaped speed.

currently doing fq_codel for 'groups' and cake for individuals.

Screenshot 2023-03-01 at 5.04.18 PM.png
do You use FastTrack with this setup ?
No. you cannot. fasttrack bypasses queues.
 
massinia
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Jun 09, 2022 7:20 pm

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Mar 15, 2023 7:51 pm

No. you cannot. fasttrack bypasses queues.
Actually it works very well for me but obviously only for upload and with Queue Tree
Image
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Wed Mar 15, 2023 8:01 pm

No. you cannot. fasttrack bypasses queues.
Actually it works very well for me but obviously only for upload and with Queue Tree
Image
Only because you are interface routing. To match by IP or packet marks etc you have to hit the firewall, which fasttrack bypasses.
 
Babujnik
newbie
Posts: 32
Joined: Fri May 05, 2017 2:15 pm

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 16, 2023 10:22 am

that's why I'm asking :)

as soon as I add rules before fasttrack:
 3    ;;; cust: guests download
      chain=forward action=accept connection-state=established,related dst-address=192.168.100.0/24 log=no log-prefix="" 

 4    ;;; cust: guests upload
      chain=forward action=accept connection-state=established,related src-address=192.168.100.0/24 log=no log-prefix="" 
to use those for simple rules, the general speed for that network drops from 800-900mbps to max 400 :D

just thinking how to make it work in best way.
I have a NAS device for which I would like to limit upload during daytime, but give full speed over night.
with rules like above, simple queue is working fine, but would have to disable them for night to give full speed. plus during daytime it's cutting download due to skipping fasttrack..
 
User avatar
sirbryan
Member
Member
Posts: 303
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 16, 2023 11:52 am

I don't like how queue trees behave when explicit directionality isn't set via a packet mark.

Am I extrapolating that you have a vlan per customer from this tree?
Each of those VLANs goes to an AP. I have 10-25 customers per AP. The queueing is on the VLAN's egress.

My hope was to take just enough of the edge off before traffic hits the AP as opposed to letting the AP's max out.

So far, so good.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 16, 2023 4:11 pm

that's why I'm asking :)

as soon as I add rules before fasttrack:
 3    ;;; cust: guests download
      chain=forward action=accept connection-state=established,related dst-address=192.168.100.0/24 log=no log-prefix="" 

 4    ;;; cust: guests upload
      chain=forward action=accept connection-state=established,related src-address=192.168.100.0/24 log=no log-prefix="" 
to use those for simple rules, the general speed for that network drops from 800-900mbps to max 400 :D

just thinking how to make it work in best way.
I have a NAS device for which I would like to limit upload during daytime, but give full speed over night.
with rules like above, simple queue is working fine, but would have to disable them for night to give full speed. plus during daytime it's cutting download due to skipping fasttrack..
Then you need faster hardware. rb4011 can handle about a gig of cake queues as long as it's not doing NAT, if you NAT on the same box that's more like 600Mbps.

You could modify your fast track rules to allow est/rel for the rest of the hosts. ie, dst-address!=nas.ip.ad.ress in your rules.

Or look at the nas and see if it has a built in upload scheduler.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 16, 2023 4:13 pm

Each of those VLANs goes to an AP. I have 10-25 customers per AP. The queueing is on the VLAN's egress.

My hope was to take just enough of the edge off before traffic hits the AP as opposed to letting the AP's max out.

So far, so good.
you just loose flexability because you can only really shape 'downloads' to the individual AP. What if your uploads get overwhelmed? You don't have an interface shaper that will handle that if you have more than 1 AP.
 
User avatar
sirbryan
Member
Member
Posts: 303
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 16, 2023 4:56 pm

you just loose flexability because you can only really shape 'downloads' to the individual AP. What if your uploads get overwhelmed? You don't have an interface shaper that will handle that if you have more than 1 AP.
To date that hasn't been a problem. At some future point I'll load queueing + shapers onto customers' IP's, or better yet onto their routers on the outbound interface.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Thu Mar 16, 2023 4:59 pm

you just loose flexability because you can only really shape 'downloads' to the individual AP. What if your uploads get overwhelmed? You don't have an interface shaper that will handle that if you have more than 1 AP.
To date that hasn't been a problem. At some future point I'll load queueing + shapers onto customers' IP's, or better yet onto their routers on the outbound interface.
Ideally you shape downloads on the head end with a big shaper tree, so you don't transport packets you will end up dropping. And you shape uploads twice, once at the customer for plan enforcement, and once at or right above the AP to keep in from getting overwhelmed. maybe also at each backhaul if you have constraints.
 
PortalNET
Member Candidate
Member Candidate
Posts: 126
Joined: Sun Apr 02, 2017 7:24 pm

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Sat Mar 18, 2023 2:50 pm

I would just implement it and go measure, and be ready to roll back. Pound it flat with artificial traffic at 4am?

(I am one of the authors of fq_codel and cake, but I do not have enough data either, on how well this stuff scales on given bits of mikrotik hardware). In most cases it is the per customer shaper that dominates the cpu by a factor of 9, and the underlying queue type be it a fifo, sfq, fq_codel or cake, adds only a tiny bit (cake is about 2.5 times as slow as fq_codel but does more, and again, it is the shaping cost that dominates). If you are low on memory, for folk running at less than a gbit, you can use memlimit 8mbyte rather than the default 32Mbyte.

On x86 gear we have individual implementations of cake scaling to 10Gbit/core. On 500mhz single core mips, cake scales to only about 80Mbit. your mileage will be somewhere between those. :/

See also.

viewtopic.php?t=179307

Hiya.. that certainly looks like an easy comment, but when u are running on a production hardware mikrotik with over 1000 pppoe-sessions.. it does not seem the smartest thing to do.. beta-testing on a production hardware with lots of pppos-sessions runnig..
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Sat Mar 18, 2023 4:43 pm

Hiya.. that certainly looks like an easy comment, but when u are running on a production hardware mikrotik with over 1000 pppoe-sessions.. it does not seem the smartest thing to do.. beta-testing on a production hardware with lots of pppos-sessions runnig..
Certainly is, that's why you need to bench test. Further, I find that ipq4018 devices aren't stable on cake when you overrun the queue. easy to replicate by doing a bandwidth test at 2x the shaper, the device will crash. That's probably because not many people are doing this across all hardware types.

I'm running 1G queue trees 8 levels deep ( the limit with the current kernel builds ) and all cake on ccr2004 and ccr2116 AND rb4011 though it's right at the edge on the rb4011 and I've had to clock it up to 1.8Ghz. Stable.
 
discoengineer
just joined
Topic Author
Posts: 12
Joined: Wed Jul 20, 2016 2:40 pm

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Fri Mar 31, 2023 9:50 pm

Thank you for the detailed explanation. You are right! I will implement and check what impact it has on the CPU. I read your post and my my, what a gem of information it is. Ill test it out and share results as well. I have 15.2 GB memory free on the CCR so I think wont be needing to change memlimit. Ill share my findings in a few days.
I am really intensely curious as to how well this went? Did you try it, your router melt down, and lose your life to a lynch mob of unhappy users? Or... ?
Hi. So at last i got the time to test it out. I did implement FQ_Codel using simple queue for a customer of mine on the CCR with a capping of 800 Mbps and to my disappointment, the CCR's CPU got clocked at 100%. In addition, the throughput for the customer also dropped to half of what Ive set in simple queue. Ive not tested it out on customers with less than 100 Mbps circuits yet. But it seems CCR cant handle it on high capacity circuits.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Fri Mar 31, 2023 9:54 pm


Hi. So at last i got the time to test it out. I did implement FQ_Codel using simple queue for a customer of mine on the CCR with a capping of 800 Mbps and to my disappointment, the CCR's CPU got clocked at 100%. In addition, the throughput for the customer also dropped to half of what Ive set in simple queue. Ive not tested it out on customers with less than 100 Mbps circuits yet. But it seems CCR cant handle it on high capacity circuits.
I assume that this is on your 1036 right? Those TILE CPUs just can't cut it. CCR2116 is THE mikrotik shaping box to get...
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 999
Joined: Fri Jun 26, 2020 4:37 pm

Re: FQ_Codel and Mikrotik CCR CPU Utilization

Sun Apr 02, 2023 12:29 pm

What is the situation with BQL driver support on MikroTik ROSv7.8/7.9?

Who is online

Users browsing this forum: Ahrefs [Bot], alan3664, Bing [Bot], deadmaus911, donmunyak, Greyhard, sybadi and 92 guests