Queue tree not working as expected

RouterOS General
1

My queue tree does not work as expected, I wonder why.

I have a HAP AC2 router with ISP theoretical max. speed= 500Mbps down, 22 Mbps up. The WAN interface is called “ether1-UPC”. The LAN part uses all remaining ports in a bridge called “bridge”.

I wanted to create a queue tree that:

  • limits ssh connection download speed to 10M
  • ensures higher priority for ssh connections for all servers
  • except for a backup server (e.g. 111.11.111.11, real IP is hidden here) - that should have low priority

Here are my mangle rules:

/ip firewall mangle
add action=mark-connection chain=prerouting comment="SSH connection" dst-port=22,2222 new-connection-mark=ssh_con protocol=tcp
add action=mark-connection chain=prerouting comment="SSH connection" new-connection-mark=ssh_con protocol=tcp src-port=22,2222
add action=mark-connection chain=prerouting comment="backup connection" new-connection-mark=backup_con src-address=111.11.111.11
add action=mark-connection chain=prerouting comment="backup connection" dst-address=111.11.111.11 new-connection-mark=backup_con
add action=mark-packet chain=forward comment="backup packet" connection-mark=backup_con new-packet-mark=backup
add action=mark-packet chain=forward comment="SSH packet" connection-mark=ssh_con new-packet-mark=ssh

Here is my queue tree:

/queue tree
add max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=1024M name=ssh_to_bridge packet-mark=ssh parent=local_out \
    priority=4
add comment="Backup server SSH, low priority" max-limit=1024M name=backup_to_bridge packet-mark=backup parent=local_out

In the next step, I have created a random file on two external servers: www and backup, with the following commands:

head -c 4G </dev/urandom > www.dat # www server
head -c 4G </dev/urandom > backup.dat # backup server

The following command works just fine:

scp root@backup:/root/backup.dat .

Speed is 1.1M/sec, as expected. Meanwhile, I can see that counters for “backup” packets are increasing.

Same thing happens with www:

scp root@www:/root/www.dat .

Speed is 1.0M/sec, as expected, counters for “ssh” packets are increasing.

If I start both of them at the same time, then actually the backup connection is sometimes faster than the www connection.

But the ssh_to_bridge queue has higher priority (4) than backup_to_bridge (8). Even if I change priority to ssh_to_bridge to 1, there is almost no difference in the speeds. Probably I do not understand how queue trees work, but I’m not sure what I’m doing wrong.

2

Anybody, please.

3

I haven’t dealt with this for some time, but something is telling me that the priority only comes into play when limit-at is exceeded, and limit-at=0 (which is set by default if you don’t specify it) is handled in a special way. Try setting it also for the backup queue, where it is currently not set at all, to the same value like for the primary server, and see what happens. If it helps, you can reduce it even lower for the backup, but not to 0.

If it helps, read also this post (except my first paragraph there) - the way you’ve configured your mangle rules, you waste the CPU by matching every single packet against multiple mangle rules.

4

Queue Tree configuration seems inconsistent and might confuse the queue mechanism.

Parent queue max limit is set to 10M which is responsible for distributing bandwidth between leaf queues, but leaf queues max limits are set to 1024M (1Gb/s)

5

Changed queue tree config to this:

/queue tree
add limit-at=10M max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=\
    local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup \
    parent=local_out

I think that limit-at and max-limit values are now consistent. I’m still getting about the same speed when downloading from both locations at the same time.

Meanwhile, counters are increasing for both leaf queues.

6

You accidentally posted your profile url.

7

Sorry, this is what I intended to post.

8

Okay so for every mark-connection rule, I need to add connection-state=new. For example:

 chain=prerouting action=mark-connection new-connection-mark=ssh_con connection-state=new protocol=tcp dst-port=22,2222

Thanks for catching that. I still don’t understand why queue priorities are not working. I understand that the speed of incoming traffic cannot be controlled directly. The “local_out” queue has parent=bridge, and it is directly controlling the speed of the traffic going out on bridge. (Indirectly, it is controlling the incoming traffic too, because ssh is based on TCP, and TCP can sense congestion.)

9

In your particular case this is sufficient, as the order of the action=mark-connection rules is such that connections to/from the backup server address first get the ssh_con mark by one of the first rule, and then this value is overwritten to backup_con by the third one, but in general it is even better to use (also*) connection-mark=no-mark (and revert the order of rules in your case, so that backup_con would be assigned first).

*also because there is yet another option, to assign some connection-mark to each connection, so the only packets matching connection-mark=no-mark are those matching also connection-state=new.

Btw, since you only deal with connections initiated from your side, your action=mark-connection rules which match on src-port and src-address are completely useless, as their dst- twins do the job.

But all the above doesn’t explain why priority doesn’t work.

10

Ive tested this somewhat thoroughly in 6.47.8. It has changed a little from 6.47.

I believe in this case, the priority is only working for the “limit at” portion. (Committed information rate) above the limit at things aren’t guaranteed.

The “max limit” on the child is treated like your burst. When “max limit” on the parent is available, traffic can flow to the max limit of the child.

During congestion, (max limit on the parent) traffic will be prioritized for the “limit at” portion of the children. When the combined traffic exceeds the parent “max limit”, only the “limit at” (CIR) is guaranteed, and both children will be queued

To test this you should be able to make a continuous ssh (priority 1) traffic flow at 10k (as you have listed as its cir) and you should not be able to override that with the other traffic.
If you set the ssh traffic to 20k, and the combined is more then your parent, both will get queued.

On the hardware I am using for testing, the parent never drops packets, so its very important that the sum of your “limit at” on the children does not exceed the desired output on the parent interface.

At least thats been my experience.

11

Thank you for the tips! These are my new mangle rules:

/ip firewall mangle
add action=mark-connection chain=prerouting comment="SSH connection" protocol=tcp dst-port=22,2222 connection-mark=no-mark new-connection-mark=ssh_con 
add action=mark-connection chain=prerouting comment="SSH connection" protocol=tcp src-port=22,2222 connection-mark=no-mark new-connection-mark=ssh_con 
add action=mark-connection chain=prerouting comment="backup connection" src-address=164.68.125.74 connection-mark=no-mark new-connection-mark=backup_con
add action=mark-connection chain=prerouting comment="backup connection" dst-address=164.68.125.74 connection-mark=no-mark new-connection-mark=backup_con
add action=mark-packet chain=forward comment="SSH packet" connection-mark=ssh_con packet-mark=no-mark new-packet-mark=ssh
add action=mark-packet chain=forward comment="backup packet" connection-mark=backup_con packet-mark=no-mark new-packet-mark=backup

I understand that one part of connection marks is not used. I wanted to specify priorities with the upload direction too, but I got stuck at the beginning.

I know for a fact that connections can only have a single mark assigned. I have some doubts about packet marks. Can you please confirm, that a packet can have multiple marks assigned?

If what you said is true, then the only way to prioritize all traffic is to put the full available bandwith into the limit-at of the root (parent) queue. Is that right?

My end goal is to prioritize all TCP traffic, and be able to use the full available bandwidth. I just used a global 10k/10M limit in my first example because I wanted to create a synthetic test, where the max. speed possible (limited by the ISP, the remote ssh servers and other local traffic on my LAN) does not interfere with the test. The “full available bandwidth” is not a fixed value. I’m connected to an ISP, and my line is shared with other customers. My theoretical speed limit is 500M/22M, but the actual speed limit is less (240M/20M usually, but it changes over time).

Now, I’m not sure how to put all traffic into a single parent queue, because the max. available bandwith is not fixed. I can try to use the theoretical maximum. For example:

/queue tree
add max-limit=22M name=local_out parent=bridge
add comment="SSH  high priority" limit-at=22M max-limit=22M name=ssh_to_bridge packet-mark=ssh parent=local_out priority=4
add comment="Backup server SSH, low priority" limit-at=22M max-limit=22M name=backup_to_bridge packet-mark=backup parent=local_out priority=8

The problem here is that the sum of limit-all values exceeds the parent’s limit-at value. This can’t be right! I can’t use limit-at=1, max-limit=22M either, because priorities are ignored for the max-limit part (at least that is what you just wrote), so in that case everything would have the same priority.

Instead of using priorites, I can divide the bandwidth by giving fixed limit-at values:

/queue tree
add max-limit=22M name=local_out parent=bridge
add comment="SSH  high priority" limit-at=16M max-limit=16M name=ssh_to_bridge packet-mark=ssh parent=local_out priority=4
add comment="Backup server SSH, low priority" limit-at=6M max-limit=6M name=backup_to_bridge packet-mark=backup parent=local_out priority=8

But this is also bad, because it will not give all available bw. to the second queue. The second queue will never get more than 6M, even if the first one is not getting any traffic.

To reach my end goal, how do I express this with queues:

  • use all available bandwidth for all traffic in a single parent queue
  • prioritize traffic between its child queues with the given priorities (e.g. queue with priority=1 should have 8 times the bandwidth of queue with priority=8)

I just can’t figure this out.

12

I believe adjusting bucket size is where you will find your answer. I still have my test setup together. I can try to verify this evening.

13

No. At most one packet-mark can be assigned to any given packet. However, a queue may match on multiple packet-mark values.

Regarding prioritizing the upload direction, the connection-mark value is common for both directions of a connection. So you can translate the same connection-mark to a packet-mark e.g. depending on in-interface. But if the root parents of queues are interfaces, you can use the same packet-mark for both directions, as only queues parented in the actual out-interface are taken into account for selection up to the packet-mark value.

14

That would be great, thank you. I just went through the token bucket algorithm wiki again ( https://wiki.mikrotik.com/wiki/Manual:HTB-Token_Bucket_Algorithm ). I think that the bucket size is only relevant for bursts. For packets flowing continuously at constant rates, the bucket size is irrelevant.

What you said about queue priorities not being used for packets exceeding max-limit seems to be the exact opposite what they say in the wiki in https://wiki.mikrotik.com/wiki/Manual:HTB

This is what they say about priorities in that wiki:

We already know that limit-at (CIR) to all queues will be given out no matter what.

Priority is responsible for distribution of remaining parent queues traffic to child queues so that they are able to reach max-limit

Queue with higher priority will reach its max-limit before the queue with lower priority. 8 is the lowest priority, 1 is the highest.

Make a note that priority only works:

  • for leaf queues - priority in inner queue have no meaning.
  • if max-limit is specified (not 0)

So this queue should have worked:

/queue tree
add limit-at=10M max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=\
    local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup \
    parent=local_out

But it does not work. I get 50-50% speed from these two directions on average.

I’m clueless.

15

There are some setups where the LAN side has multiple bridges and interfaces. Is it okay use in-interface=WAN and out-interface=WAN? I’m not too comfortable with using bridges as interfaces, it becomes hard for me to tell wether an encapsulated / tunelled etc. package gets mangled and queued twice or not.

Example - ssh connections have the same connection mark, regardless of what side (LAN/WAN) initiated the connection.

  /ip firewall mangle chain=prerouting connection-mark=no-mark connection-state=new \
		protocol=tcp dst-port=22,2222 \
		action=mark-connection mark=ssh_con \
		comment="SSH"
  /ip firewall mangle chain=prerouting connection-mark=no-mark connection-state=new \
		protocol=tcp dst-port=22,2222 \
		action=mark-connection mark=ssh_con \
		comment="SSH"

Packets are differentiated based on their input/output interface:

  /ip firewall mange add chain=forward connection-mark=ssh_con packet-mark=no-mark \
                in-interface=ether1-WAN \
		action=mark-packet  new-packet-mark=ssh_to_wan comment="SSH coming in from WAN"
		
  /ip firewall mange add chain=forward connection-mark=ssh_con packet-mark=no-mark \
                out-interface=ether1-WAN \
		action=mark-packet  new-packet-mark=ssh_to_wan comment="SSH going out to WAN"			
		
  /ip firewall mange add chain=forward connection-mark=ssh_con packet-mark=no-mark \
		action=mark-packet  new-packet-mark=ssh_internal comment="SSH between internal LANs (fallback)"
16

You might have a look at my QoS script, because it is set up in a similar way to yours, but I know it works:

http://forum.mikrotik.com/t/fasttrack-friendly-qos-script/102401/1

I suspect your issues are due to the way you are marking the packets rather than the queue settings. Are you using fasttrack on your device?

Make sure you are monitoring the “average rate” in the “statistics” tab for the child queue trees as you are doing your testing.

17

I found a simple work around.


/queue tree
add limit-at=10M max-limit=10M name=local_out parent=bridge
add max-limit=10M name=queue_local_out parent=local_out
add comment=“SSH 10k guaranteed, high priority” limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=queue_local_out priority=1
add comment=“Backup server SSH, low priority” limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup parent=queue_local_out


The hardware I am using doesnt support traffic limiting on the interface. Perhaps yours doesnt either, and the software wont allow it now. But if you add a child queue to the interface queue, and then make your packet queues parents the interface child queue, it works how you want it to.

Interface queue (local_out)
—> Child (queue_local_out)
---------> Priority1 (ssh_to_bridge)
---------> Priority8 (backup_to_bridge)

18

Your intermediate queue, queue_local_out, is not necessary here. The following configuration should behave in exactly the same way:

/queue tree
add max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup parent=local_out
19

Without the intermediate queue on this release, the prioritized traffic it not queued/dropped/limited at properly.
CIR works as intended, but the max-limit is the problem.
As he noted, it pretty much is split 50/50 until max-limit is reached on the child queues

EDIT:
It appears in this release, the Max-limit on the interface only forces packets on the child queues that are above their respective limit-at rate (CIR) to be queued. So if you have two priorities, and the combined traffic exceeds interface max-limit, the one(s) above their respective CIR is becomes queued, but it still exceeds the max-limit on the interface. If both priorities are exceeding their respective CIR but still under their respective max-limit, and you want the lower priority to drop traffic, adding interface sub-parent we will call it, fixes this. I believe this is the problem he is experiencing.

20

This is what I get with my queue setup described in the previous post without the intermediate queue:
Screenshot 2020-12-13 154838.png
I do believe that is the exact behavior the OP wants.