Community discussions

MikroTik App
 
mikegleasonjr
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 55
Joined: Tue Aug 07, 2018 3:14 am

Simple queue going over max-limit

Thu Jun 02, 2022 10:18 pm

Hi,

Something really weird is happening. Fasttrack is disabled. The simple queue works fine for all traffic except when there's a steam download.

Whenever I do speed tests (even by multiple clients at once) or I download large files, the queue barely go over its max limit. But when I start a steam game download, the queue overshoots almost always and sometimes up to 18 mbps (I have a 100mbit connection, so it's almost up to 20%). When I go to the connections tab of the firewall, I see that steam opens up multiple TCP connections. They're not fasttracked.

What's odd and indicates a bug, sometimes when I play the queue settings, it stop overshooting. But when I reboot the router (RB4011), it starts happening again.

My wan is pppoe over a VLAN on a fiber ONT (bridged) with the Mikrotik router.

In the screenshot below, you can see I lowered the max-limit of the queue to 80M but still my pppoe traffic overshoots to 96M.

Here's my full config:

/interface bridge
	add admin-mac=XX:XX:XX:XX:XX:XX auto-mac=no frame-types=admit-only-vlan-tagged name=bridge1 vlan-filtering=yes

/interface vlan
	add comment=lan interface=bridge1 name=vlan10 vlan-id=10
	add comment=guest interface=bridge1 name=vlan20 vlan-id=20
	add comment=voip interface=bridge1 name=vlan30 vlan-id=30
	add comment=cams interface=bridge1 name=vlan40 vlan-id=40
	add comment=iam-wan interface=bridge1 name=vlan881 vlan-id=881

/interface pppoe-client
	add add-default-route=yes disabled=no interface=vlan881 max-mru=1492 max-mtu=1492 name=pppoe-iam user=*****

/interface list
	add name=WAN
	add name=LAN
	add name=GUEST
	add name=VOIP
	add name=CAMS

/ip pool
	add name=vlan10 ranges=*********
	add name=vlan20 ranges=*********
	add name=vlan30 ranges=*********
	add name=vlan40 ranges=*********

/ip dhcp-server
	add address-pool=vlan10 interface=vlan10 lease-time=1d name=vlan10
	add address-pool=vlan20 interface=vlan20 lease-time=1d name=vlan20
	add address-pool=vlan30 interface=vlan30 lease-time=1d name=vlan30
	add address-pool=vlan40 interface=vlan40 lease-time=1d name=vlan40

/queue type
	add cake-flowmode=dual-srchost cake-nat=yes cake-overhead=36 kind=cake name=cake-out
	add cake-diffserv=besteffort cake-flowmode=dual-dsthost cake-nat=yes cake-overhead=36 cake-wash=yes kind=cake name=cake-in

/queue simple
	add bucket-size=0.002/0.002 max-limit=80M/47500k name=wan queue=cake-in/cake-out target=pppoe-iam

/interface bridge port
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether1
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether2
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether3
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether4
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether5
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether6
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether7
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether8
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether9
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether10
	add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=sfp-sfpplus1

/ip neighbor discovery-settings
	set discover-interface-list=LAN

/interface bridge vlan
	add bridge=bridge1 tagged=bridge1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,sfp-sfpplus1 vlan-ids=10
	add bridge=bridge1 tagged=bridge1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,sfp-sfpplus1 vlan-ids=20
	add bridge=bridge1 tagged=bridge1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,sfp-sfpplus1 vlan-ids=30
	add bridge=bridge1 tagged=bridge1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,sfp-sfpplus1 vlan-ids=40
	add bridge=bridge1 tagged=bridge1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,sfp-sfpplus1 vlan-ids=881

/interface list member
	add interface=pppoe-iam list=WAN
	add interface=vlan10 list=LAN
	add interface=vlan20 list=GUEST
	add interface=vlan30 list=VOIP
	add interface=vlan40 list=CAMS

/ip address
	add address=******* interface=vlan10 network=******
	add address=******* interface=vlan20 network=******
	add address=******* interface=vlan30 network=******
	add address=******* interface=vlan40 network=******

/ip cloud
	set ddns-enabled=yes

/ip dhcp-server network
	add address=***** dns-server=******** gateway=******** ntp-server=*******
	add address=***** dns-server=******** gateway=******** ntp-server=*******
	add address=***** dns-server=******** gateway=******** ntp-server=*******
	add address=***** dns-server=******** gateway=******** ntp-server=*******

/ip dns
	set allow-remote-requests=yes cache-max-ttl=0s servers=********

/ip firewall address-list
	add address=******* comment=asterisk list=VOIP-WHITELIST
	add address=******** comment=traefik list=DSTNAT-WHITELIST
	add address=******* comment=scale list=DSTNAT-WHITELIST

/ip firewall filter
	add action=accept chain=input connection-state=established,related,untracked
	add action=drop chain=input connection-state=invalid
	add action=accept chain=input protocol=icmp
	add action=accept chain=input dst-port=13231 protocol=udp
	add action=accept chain=input dst-address=127.0.0.1
	add action=accept chain=input dst-port=123 in-interface-list=!WAN protocol=udp
	add action=jump chain=input dst-port=53 jump-target=dns protocol=tcp
	add action=jump chain=input dst-port=53 jump-target=dns protocol=udp
	add action=drop chain=input in-interface-list=!LAN
	add action=accept chain=forward connection-state=established,related
	add action=accept chain=forward in-interface-list=LAN
	add action=accept chain=forward in-interface-list=GUEST out-interface-list=WAN
	add action=accept chain=forward in-interface-list=VOIP out-interface-list=WAN src-address-list=VOIP-WHITELIST
	add action=accept chain=forward connection-nat-state=dstnat dst-address-list=DSTNAT-WHITELIST in-interface-list=WAN
	add action=drop chain=forward
	add action=drop chain=dns in-interface-list=WAN
	add action=drop chain=dns in-interface-list=CAMS
	add action=drop chain=dns in-interface-list=VOIP src-address-list=!VOIP-WHITELIST
	add action=accept chain=dns

/ip firewall nat
	add action=masquerade chain=srcnat comment=wan ipsec-policy=out,none out-interface-list=WAN
	add action=dst-nat chain=dstnat comment=******* dst-port=**in-interface-list=WAN protocol=tcp to-addresses=***** to-ports=**
	add action=dst-nat chain=dstnat comment=******* dst-port=** in-interface-list=WAN protocol=tcp to-addresses=***** to-ports=**
	add action=dst-nat chain=dstnat comment=******* dst-port=** in-interface-list=WAN protocol=tcp to-addresses=***** to-ports=**
	add action=dst-nat chain=dstnat comment=******* dst-port=** in-interface-list=WAN protocol=udp to-addresses=***** to-ports=**

/system clock
	set time-zone-autodetect=no time-zone-name=********

/system identity
	set name=rb4011

/system ntp client
	set enabled=yes

/system ntp server
	set enabled=yes manycast=yes

/system ntp client servers
	add address=*******
	add address=*******
	add address=*******
	add address=*******

/system package update
	set channel=testing

/tool bandwidth-server
	set enabled=no

/tool mac-server
	set allowed-interface-list=LAN

/tool mac-server mac-winbox
	set allowed-interface-list=LAN
You do not have the required permissions to view the files attached to this post.
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Simple queue going over max-limit

Thu Jun 02, 2022 10:30 pm

viewtopic.php?p=931315#p931315

I would also like some answers on this but even Mikrotik support I contacted few times about this never gave any useful info..
 
mikegleasonjr
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 55
Joined: Tue Aug 07, 2018 3:14 am

Re: Simple queue going over max-limit

Thu Jun 02, 2022 10:47 pm

viewtopic.php?p=931315#p931315

I would also like some answers on this but even Mikrotik support I contacted few times about this never gave any useful info..
wow ok I thought I was crazy. Thing is, I am glad we finally narrowed down something that is happening (IMHO) for a long time. Probably a bug in the queues.

While fiddling with the settings, sometimes it starts to shape the traffic correctly. It happened two times over the course of two days for me. Then I revert all my changes and go back to the initial config. Then everything keeps working fine, until I reboot the router.

The common denominator I see is that we're on pppoe over fiber.
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Simple queue going over max-limit

Thu Jun 02, 2022 10:52 pm

viewtopic.php?p=931315#p931315

I would also like some answers on this but even Mikrotik support I contacted few times about this never gave any useful info..
wow ok I thought I was crazy. Thing is, I am glad we finally narrowed down something that is happening (IMHO) for a long time. Probably a bug in the queues.

While fiddling with the settings, sometimes it starts to shape the traffic correctly. It happened two times over the course of two days for me. Then I revert all my changes and go back to the initial config. Then everything keeps working fine, until I reboot the router.

The common denominator I see is that we're on pppoe over fiber.
I have 100% same issue as you show in screen and I noticed this issue over year now and it's super easy reproducible by almost any major game downloaders like Steam, Epic games, Adobe programs etc..

I did very dirty and very complicated workaround by making queue which matches such downloaders with multi connections and then I set speed almost half from desired one.

So queue is set to 30mb in order to actually download at 60mbs and not to kill my internet..
 
mikegleasonjr
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 55
Joined: Tue Aug 07, 2018 3:14 am

Re: Simple queue going over max-limit

Thu Jun 02, 2022 11:23 pm


I did very dirty and very complicated workaround by making queue which matches such downloaders with multi connections and then I set speed almost half from desired one.
That's true I noticed it does the same thing with the Epic Games client.

What's the general route you've taken ? You're marking packets coming from IPs you're constantly keeping an eye to every time you start a download ?
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Simple queue going over max-limit

Thu Jun 02, 2022 11:30 pm


I did very dirty and very complicated workaround by making queue which matches such downloaders with multi connections and then I set speed almost half from desired one.
That's true I noticed it does the same thing with the Epic Games client.

What's the general route you've taken ? You're marking packets coming from IPs you're constantly keeping an eye to every time you start a download ?
Very big list of adresses and script which dynamically collects them from dns and adds more + some extra packet marks by connection number + rate limits..

It's very hard to tune and get it to work properly but for now it solved my problems.
 
Zoxc
just joined
Posts: 17
Joined: Fri Aug 13, 2021 4:01 pm

Re: Simple queue going over max-limit

Sun Jun 05, 2022 2:42 pm

Do you see the same behavior when using CAKE's shaper? That should be preferred if you're matching a slow ISP link. Make the simple queue unlimited and enter your bandwidth in the CAKE queue types to give it a try.
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Simple queue going over max-limit

Sun Jun 05, 2022 3:19 pm

Do you see the same behavior when using CAKE's shaper? That should be preferred if you're matching a slow ISP link. Make the simple queue unlimited and enter your bandwidth in the CAKE queue types to give it a try.
70mbit isn't exactly slow link, and queue doesnt matter, it completly chokes dsl line and triggers bufferbloat. I did try setting cake with limit, and it still doesn't respecet even that limit and goes way over..
 
Zoxc
just joined
Posts: 17
Joined: Fri Aug 13, 2021 4:01 pm

Re: Simple queue going over max-limit

Sun Jun 05, 2022 7:15 pm

I'd like to see a bandwidth measurement from the device doing the downloading too, to see if the queue works, without relying on measurements on the router.
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Simple queue going over max-limit

Sun Jun 05, 2022 7:24 pm

I'd like to see a bandwidth measurement from the device doing the downloading too, to see if the queue works, without relying on measurements on the router.
If I run speedtest or simple driver download from nvidia it's respecting set limit exactly as set in queue as we mentioned already.
 
mikegleasonjr
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 55
Joined: Tue Aug 07, 2018 3:14 am

Re: Simple queue going over max-limit

Tue Jun 07, 2022 2:43 pm

I'd like to see a bandwidth measurement from the device doing the downloading too, to see if the queue works, without relying on measurements on the router.

See previous answer.

But also the fact that my buffer bloats shoots up tells me the measurements on the router are accurate.
 
mikegleasonjr
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 55
Joined: Tue Aug 07, 2018 3:14 am

Re: Simple queue going over max-limit

Tue Jun 07, 2022 2:45 pm

Do you see the same behavior when using CAKE's shaper? That should be preferred if you're matching a slow ISP link. Make the simple queue unlimited and enter your bandwidth in the CAKE queue types to give it a try.
Cake comes later in the equation, so if the queue gives too much packets already (more than your internet connection speed), cake can't do nothing about it. It improves things but still buffer bloats shoots up because more packets are pushed in a pipe that can't handle it
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: Simple queue going over max-limit

Tue Jun 07, 2022 2:48 pm

I started the other thread (linked in post 2). Unfortunate that there are duplicate threads now but on the bright side it's a good testament that the bug is becoming more noticed and prevalent and should be taken more seriously by support.

Besides for game downloaders as mentioned by @ivicask, another easy way to trigger this is with torrents. For some reason torrents easily exceed mikrotik qos as well.

I created a support ticket about this issue, SUP-81506, but the support assistant (Arturs C) requested a supout which has stalled the ticket (I don't see why an elaborate supout is needed to reproduce an issue that is well described and attested on the forums... I've already wasted enough time on this issue and am sure that the supout won't reveal anything of interest so I haven't bothered to do that yet)

If someone has time to reproduce this bug on a clean config and send evidence to mikrotik support along with the pesky supout file which they refuse to proceed on the ticket without, then that would be great. I don't have the time right now but if I manage to do this in future I will update this post accordingly.

At first I thought it was incoming UDP traffic causing the problem, but it happens with TCP also. It would be helpful to test and rule out if this happens only with pppoe over ethernet, or if it is a general problem with qos which could affect any connection type
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Simple queue going over max-limit

Tue Jun 07, 2022 3:10 pm

I started the other thread (linked in post 2). Unfortunate that there are duplicate threads now but on the bright side it's a good testament that the bug is becoming more noticed and prevalent and should be taken more seriously by support.

Besides for game downloaders as mentioned by @ivicask, another easy way to trigger this is with torrents. For some reason torrents easily exceed mikrotik qos as well.

I created a support ticket about this issue, SUP-81506, but the support assistant (Arturs C) requested a supout which has stalled the ticket (I don't see why an elaborate supout is needed to reproduce an issue that is well described and attested on the forums... I've already wasted enough time on this issue and am sure that the supout won't reveal anything of interest so I haven't bothered to do that yet)

If someone has time to reproduce this bug on a clean config and send evidence to mikrotik support along with the pesky supout file which they refuse to proceed on the ticket without, then that would be great. I don't have the time right now but if I manage to do this in future I will update this post accordingly.

It will be a great day when this issue is fixed... nothing more frustrating than QoS failing dismally to do what it was implemented for.
I also gave them supout and it was useless, i gave them full details and easy way to reproduce the issue, but they didint care much i think.

BTW i have no issue with torrents, its 100% respecting limits for me, torrents use multiple connections with different ports while this problematic downloaders are opening multiple connections to same port like 443, 80 or 8080..
You do not have the required permissions to view the files attached to this post.
Last edited by ivicask on Tue Jun 07, 2022 3:33 pm, edited 2 times in total.
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: Simple queue going over max-limit

Tue Jun 07, 2022 3:30 pm

BTW i have no issue with torrents, its 100% respecting limits for me.
It could be something to do with the torrent client (local or remote), or the number of seeds? In my case with qBittorrent and a torrent with thousands of seeders, I did experience the issue with torrents too. But that was with a simple queue, not queue tree marking torrent packets, so maybe that changes the outcome somehow for torrents?
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Simple queue going over max-limit

Tue Jun 07, 2022 3:39 pm

BTW i have no issue with torrents, its 100% respecting limits for me.
It could be something to do with the torrent client (local or remote), or the number of seeds? In my case with qBittorrent and a torrent with thousands of seeders, I did experience the issue with torrents too. But that was with a simple queue, not queue tree marking torrent packets, so maybe that changes the outcome somehow for torrents?
I also tested with qbit, and it opens also 200+ connections but both simple and queue tree are respecting limits for me in this case.

Try lowering bucket sizes to 0.001 and try codel queue with lowered packet and quantum limits, thats how i have maybe that helps you.
You do not have the required permissions to view the files attached to this post.
 
fragtion
Member Candidate
Member Candidate
Posts: 257
Joined: Fri Nov 13, 2009 10:08 pm
Location: Johannesburg, South Africa

Re: Simple queue going over max-limit

Thu Jul 07, 2022 5:38 am

I think I've been able to "work around" this issue somewhat, but unfortunately the technique only works for queue tree, not simple queues

Basically the idea is to create a new "Queue Type" with "pfifo" (or bfifo) kind, and set a higher than normal "Queue Size". In my case anything more than around 1000 packets or 5120000 bytes seems to be effective, but I suppose YMMV if this even works for you?

So in my case the main queues are fq_codel, then I assign this custom queue type to all the children queues in the tree, something like this:
looksllikethis.png
My pings have never been more stable under full load...torrents blazing, I can browse or play games very responsively but I was hoping to achieve all of this with a single simple fq_codel or cake queue targeting the pppoe interface so I suppose that will remain a pipedream for the time being

I'm guessing the bug has something to do with not enough packets being processed by the queues by default? Or failing to shape some kind of transport overhead? hopefully more users can chime in with some feedback on this
You do not have the required permissions to view the files attached to this post.

Who is online

Users browsing this forum: No registered users and 22 guests