Queue tree not working as expected

nagylzs · Tue Dec 08, 2020 10:23 pm

My queue tree does not work as expected, I wonder why.

I have a HAP AC2 router with ISP theoretical max. speed= 500Mbps down, 22 Mbps up. The WAN interface is called "ether1-UPC". The LAN part uses all remaining ports in a bridge called "bridge".

I wanted to create a queue tree that:

* limits ssh connection download speed to 10M
* ensures higher priority for ssh connections for all servers
* except for a backup server (e.g. 111.11.111.11, real IP is hidden here) - that should have low priority

Here are my mangle rules:

/ip firewall mangle
add action=mark-connection chain=prerouting comment="SSH connection" dst-port=22,2222 new-connection-mark=ssh_con protocol=tcp
add action=mark-connection chain=prerouting comment="SSH connection" new-connection-mark=ssh_con protocol=tcp src-port=22,2222
add action=mark-connection chain=prerouting comment="backup connection" new-connection-mark=backup_con src-address=111.11.111.11
add action=mark-connection chain=prerouting comment="backup connection" dst-address=111.11.111.11 new-connection-mark=backup_con
add action=mark-packet chain=forward comment="backup packet" connection-mark=backup_con new-packet-mark=backup
add action=mark-packet chain=forward comment="SSH packet" connection-mark=ssh_con new-packet-mark=ssh

Here is my queue tree:

/queue tree
add max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=1024M name=ssh_to_bridge packet-mark=ssh parent=local_out \
    priority=4
add comment="Backup server SSH, low priority" max-limit=1024M name=backup_to_bridge packet-mark=backup parent=local_out

In the next step, I have created a random file on two external servers: www and backup, with the following commands:

head -c 4G </dev/urandom > www.dat # www server
head -c 4G </dev/urandom > backup.dat # backup server

The following command works just fine:

scp root@backup:/root/backup.dat .

Speed is 1.1M/sec, as expected. Meanwhile, I can see that counters for "backup" packets are increasing.

Same thing happens with www:

scp root@www:/root/www.dat .

Speed is 1.0M/sec, as expected, counters for "ssh" packets are increasing.

If I start both of them at the same time, then actually the backup connection is sometimes faster than the www connection.

But the ssh_to_bridge queue has higher priority (4) than backup_to_bridge (8). Even if I change priority to ssh_to_bridge to 1, there is almost no difference in the speeds. Probably I do not understand how queue trees work, but I'm not sure what I'm doing wrong.

nagylzs · Thu Dec 10, 2020 3:15 pm

Anybody, please.

sindy · Thu Dec 10, 2020 6:39 pm

I haven't dealt with this for some time, but something is telling me that the priority only comes into play when limit-at is exceeded, and limit-at=0 (which is set by default if you don't specify it) is handled in a special way. Try setting it also for the backup queue, where it is currently not set at all, to the same value like for the primary server, and see what happens. If it helps, you can reduce it even lower for the backup, but not to 0.

If it helps, read also this post (except my first paragraph there) - the way you've configured your mangle rules, you waste the CPU by matching every single packet against multiple mangle rules.

CZFan · Thu Dec 10, 2020 9:29 pm

Queue Tree configuration seems inconsistent and might confuse the queue mechanism.

Parent queue max limit is set to 10M which is responsible for distributing bandwidth between leaf queues, but leaf queues max limits are set to 1024M (1Gb/s)

nagylzs · Sat Dec 12, 2020 9:07 pm

Changed queue tree config to this:

/queue tree
add limit-at=10M max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=\
    local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup \
    parent=local_out

I think that limit-at and max-limit values are now consistent. I'm still getting about the same speed when downloading from both locations at the same time.

Meanwhile, counters are increasing for both leaf queues.

nagylzs · Sat Dec 12, 2020 9:09 pm

If it helps, read also this post (except my first paragraph there) - the way you've configured your mangle rules, you waste the CPU by matching every single packet against multiple mangle rules.

You accidentally posted your profile url.

sindy · Sat Dec 12, 2020 9:51 pm

Sorry, this is what I intended to post.

nagylzs · Sat Dec 12, 2020 11:00 pm

Sorry, this is what I intended to post.

Okay so for every mark-connection rule, I need to add connection-state=new. For example:

 chain=prerouting action=mark-connection new-connection-mark=ssh_con connection-state=new protocol=tcp dst-port=22,2222

Thanks for catching that. I still don't understand why queue priorities are not working. I understand that the speed of incoming traffic cannot be controlled directly. The "local_out" queue has parent=bridge, and it is directly controlling the speed of the traffic going out on bridge. (Indirectly, it is controlling the incoming traffic too, because ssh is based on TCP, and TCP can sense congestion.)

sindy · Sun Dec 13, 2020 11:04 am

Okay so for every mark-connection rule, I need to add connection-state=new.

In your particular case this is sufficient, as the order of the action=mark-connection rules is such that connections to/from the backup server address first get the ssh_con mark by one of the first rule, and then this value is overwritten to backup_con by the third one, but in general it is even better to use (also*) connection-mark=no-mark (and revert the order of rules in your case, so that backup_con would be assigned first).

*also because there is yet another option, to assign some connection-mark to each connection, so the only packets matching connection-mark=no-mark are those matching also connection-state=new.

Btw, since you only deal with connections initiated from your side, your action=mark-connection rules which match on src-port and src-address are completely useless, as their dst- twins do the job.

But all the above doesn't explain why priority doesn't work.

mikeeg02 · Sun Dec 13, 2020 6:19 pm

Changed queue tree config to this:
Code: Select all
/queue tree
add limit-at=10M max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=\
    local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup \
    parent=local_out
I think that limit-at and max-limit values are now consistent. I'm still getting about the same speed when downloading from both locations at the same time.

Meanwhile, counters are increasing for both leaf queues.

Ive tested this somewhat thoroughly in 6.47.8. It has changed a little from 6.47.

I believe in this case, the priority is only working for the "limit at" portion. (Committed information rate) above the limit at things aren't guaranteed.

The "max limit" on the child is treated like your burst. When "max limit" on the parent is available, traffic can flow to the max limit of the child.

During congestion, (max limit on the parent) traffic will be prioritized for the "limit at" portion of the children. When the combined traffic exceeds the parent "max limit", only the "limit at" (CIR) is guaranteed, and both children will be queued

To test this you should be able to make a continuous ssh (priority 1) traffic flow at 10k (as you have listed as its cir) and you should not be able to override that with the other traffic.
If you set the ssh traffic to 20k, and the combined is more then your parent, both will get queued.

On the hardware I am using for testing, the parent never drops packets, so its very important that the sum of your "limit at" on the children does not exceed the desired output on the parent interface.

At least thats been my experience.

nagylzs · Sun Dec 13, 2020 7:13 pm

Thank you for the tips! These are my new mangle rules:

/ip firewall mangle
add action=mark-connection chain=prerouting comment="SSH connection" protocol=tcp dst-port=22,2222 connection-mark=no-mark new-connection-mark=ssh_con 
add action=mark-connection chain=prerouting comment="SSH connection" protocol=tcp src-port=22,2222 connection-mark=no-mark new-connection-mark=ssh_con 
add action=mark-connection chain=prerouting comment="backup connection" src-address=164.68.125.74 connection-mark=no-mark new-connection-mark=backup_con
add action=mark-connection chain=prerouting comment="backup connection" dst-address=164.68.125.74 connection-mark=no-mark new-connection-mark=backup_con
add action=mark-packet chain=forward comment="SSH packet" connection-mark=ssh_con packet-mark=no-mark new-packet-mark=ssh
add action=mark-packet chain=forward comment="backup packet" connection-mark=backup_con packet-mark=no-mark new-packet-mark=backup

I understand that one part of connection marks is not used. I wanted to specify priorities with the upload direction too, but I got stuck at the beginning.

I know for a fact that connections can only have a single mark assigned. I have some doubts about packet marks. Can you please confirm, that a packet can have multiple marks assigned?

Ive tested this somewhat thoroughly in 6.47.8. It has changed a little from 6.47.

I believe in this case, the priority is only working for the "limit at" portion. (Committed information rate) above the limit at things aren't guaranteed.

The "max limit" on the child is treated like your burst. When "max limit" on the parent is available, traffic can flow to the max limit of the child.

During congestion, (max limit on the parent) traffic will be prioritized for the "limit at" portion of the children. When the combined traffic exceeds the parent "max limit", only the "limit at" (CIR) is guaranteed, and both children will be queued

If what you said is true, then the only way to prioritize all traffic is to put the full available bandwith into the limit-at of the root (parent) queue. Is that right?

My end goal is to prioritize all TCP traffic, and be able to use the full available bandwidth. I just used a global 10k/10M limit in my first example because I wanted to create a synthetic test, where the max. speed possible (limited by the ISP, the remote ssh servers and other local traffic on my LAN) does not interfere with the test. The "full available bandwidth" is not a fixed value. I'm connected to an ISP, and my line is shared with other customers. My theoretical speed limit is 500M/22M, but the actual speed limit is less (240M/20M usually, but it changes over time).

Now, I'm not sure how to put all traffic into a single parent queue, because the max. available bandwith is not fixed. I can try to use the theoretical maximum. For example:

/queue tree
add max-limit=22M name=local_out parent=bridge
add comment="SSH  high priority" limit-at=22M max-limit=22M name=ssh_to_bridge packet-mark=ssh parent=local_out priority=4
add comment="Backup server SSH, low priority" limit-at=22M max-limit=22M name=backup_to_bridge packet-mark=backup parent=local_out priority=8

The problem here is that the sum of limit-all values exceeds the parent's limit-at value. This can't be right! I can't use limit-at=1, max-limit=22M either, because priorities are ignored for the max-limit part (at least that is what you just wrote), so in that case everything would have the same priority.

Instead of using priorites, I can divide the bandwidth by giving fixed limit-at values:

/queue tree
add max-limit=22M name=local_out parent=bridge
add comment="SSH  high priority" limit-at=16M max-limit=16M name=ssh_to_bridge packet-mark=ssh parent=local_out priority=4
add comment="Backup server SSH, low priority" limit-at=6M max-limit=6M name=backup_to_bridge packet-mark=backup parent=local_out priority=8

But this is also bad, because it will not give all available bw. to the second queue. The second queue will never get more than 6M, even if the first one is not getting any traffic.

To reach my end goal, how do I express this with queues:

* use all available bandwidth for all traffic in a single parent queue
* prioritize traffic between its child queues with the given priorities (e.g. queue with priority=1 should have 8 times the bandwidth of queue with priority=8)

I just can't figure this out.

mikeeg02 · Sun Dec 13, 2020 7:30 pm

I believe adjusting bucket size is where you will find your answer. I still have my test setup together. I can try to verify this evening.

sindy · Sun Dec 13, 2020 7:54 pm

I wanted to specify priorities with the upload direction too, but I got stuck at the beginning.

I know for a fact that connections can only have a single mark assigned. I have some doubts about packet marks. Can you please confirm, that a packet can have multiple marks assigned?

No. At most one packet-mark can be assigned to any given packet. However, a queue may match on multiple packet-mark values.

Regarding prioritizing the upload direction, the connection-mark value is common for both directions of a connection. So you can translate the same connection-mark to a packet-mark e.g. depending on in-interface. But if the root parents of queues are interfaces, you can use the same packet-mark for both directions, as only queues parented in the actual out-interface are taken into account for selection up to the packet-mark value.

nagylzs · Sun Dec 13, 2020 10:03 pm

I believe adjusting bucket size is where you will find your answer. I still have my test setup together. I can try to verify this evening.

That would be great, thank you. I just went through the token bucket algorithm wiki again ( https://wiki.mikrotik.com/wiki/Manual:H ... _Algorithm ). I think that the bucket size is only relevant for bursts. For packets flowing continuously at constant rates, the bucket size is irrelevant.

What you said about queue priorities not being used for packets exceeding max-limit seems to be the exact opposite what they say in the wiki in https://wiki.mikrotik.com/wiki/Manual:HTB

This is what they say about priorities in that wiki:

We already know that limit-at (CIR) to all queues will be given out no matter what.

Priority is responsible for distribution of remaining parent queues traffic to child queues so that they are able to reach max-limit

Queue with higher priority will reach its max-limit before the queue with lower priority. 8 is the lowest priority, 1 is the highest.

Make a note that priority only works:

* for leaf queues - priority in inner queue have no meaning.
* if max-limit is specified (not 0)

So this queue should have worked:

/queue tree
add limit-at=10M max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=\
    local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup \
    parent=local_out

But it does not work. I get 50-50% speed from these two directions on average.

I'm clueless.

nagylzs · Mon Dec 14, 2020 12:04 am

Regarding prioritizing the upload direction, the connection-mark value is common for both directions of a connection. So you can translate the same connection-mark to a packet-mark e.g. depending on in-interface. But if the root parents of queues are interfaces, you can use the same packet-mark for both directions, as only queues parented in the actual out-interface are taken into account for selection up to the packet-mark value.

There are some setups where the LAN side has multiple bridges and interfaces. Is it okay use in-interface=WAN and out-interface=WAN? I'm not too comfortable with using bridges as interfaces, it becomes hard for me to tell wether an encapsulated / tunelled etc. package gets mangled and queued twice or not.

Example - ssh connections have the same connection mark, regardless of what side (LAN/WAN) initiated the connection.

  /ip firewall mangle chain=prerouting connection-mark=no-mark connection-state=new \
		protocol=tcp dst-port=22,2222 \
		action=mark-connection mark=ssh_con \
		comment="SSH"
  /ip firewall mangle chain=prerouting connection-mark=no-mark connection-state=new \
		protocol=tcp dst-port=22,2222 \
		action=mark-connection mark=ssh_con \
		comment="SSH"

Packets are differentiated based on their input/output interface:

  /ip firewall mange add chain=forward connection-mark=ssh_con packet-mark=no-mark \
                in-interface=ether1-WAN \
		action=mark-packet  new-packet-mark=ssh_to_wan comment="SSH coming in from WAN"
		
  /ip firewall mange add chain=forward connection-mark=ssh_con packet-mark=no-mark \
                out-interface=ether1-WAN \
		action=mark-packet  new-packet-mark=ssh_to_wan comment="SSH going out to WAN"			
		
  /ip firewall mange add chain=forward connection-mark=ssh_con packet-mark=no-mark \
		action=mark-packet  new-packet-mark=ssh_internal comment="SSH between internal LANs (fallback)"

mducharme · Mon Dec 14, 2020 12:20 am

There are some setups where the LAN side has multiple bridges and interfaces. Is it okay use in-interface=WAN and out-interface=WAN? I'm not too comfortable with using bridges as interfaces, it becomes hard for me to tell wether an encapsulated / tunelled etc. package gets mangled and queued twice or not.

You might have a look at my QoS script, because it is set up in a similar way to yours, but I know it works:

viewtopic.php?f=9&t=113308

I suspect your issues are due to the way you are marking the packets rather than the queue settings. Are you using fasttrack on your device?

Make sure you are monitoring the "average rate" in the "statistics" tab for the child queue trees as you are doing your testing.

mikeeg02 · Mon Dec 14, 2020 1:12 am

Regarding prioritizing the upload direction, the connection-mark value is common for both directions of a connection. So you can translate the same connection-mark to a packet-mark e.g. depending on in-interface. But if the root parents of queues are interfaces, you can use the same packet-mark for both directions, as only queues parented in the actual out-interface are taken into account for selection up to the packet-mark value.
There are some setups where the LAN side has multiple bridges and interfaces. Is it okay use in-interface=WAN and out-interface=WAN? I'm not too comfortable with using bridges as interfaces, it becomes hard for me to tell wether an encapsulated / tunelled etc. package gets mangled and queued twice or not.

Example - ssh connections have the same connection mark, regardless of what side (LAN/WAN) initiated the connection.
Code: Select all
  /ip firewall mangle chain=prerouting connection-mark=no-mark connection-state=new \
		protocol=tcp dst-port=22,2222 \
		action=mark-connection mark=ssh_con \
		comment="SSH"
  /ip firewall mangle chain=prerouting connection-mark=no-mark connection-state=new \
		protocol=tcp dst-port=22,2222 \
		action=mark-connection mark=ssh_con \
		comment="SSH"		
Packets are differentiated based on their input/output interface:
Code: Select all
  /ip firewall mange add chain=forward connection-mark=ssh_con packet-mark=no-mark \
                in-interface=ether1-WAN \
		action=mark-packet  new-packet-mark=ssh_to_wan comment="SSH coming in from WAN"
		
  /ip firewall mange add chain=forward connection-mark=ssh_con packet-mark=no-mark \
                out-interface=ether1-WAN \
		action=mark-packet  new-packet-mark=ssh_to_wan comment="SSH going out to WAN"			
		
  /ip firewall mange add chain=forward connection-mark=ssh_con packet-mark=no-mark \
		action=mark-packet  new-packet-mark=ssh_internal comment="SSH between internal LANs (fallback)"

I found a simple work around.

/queue tree
add limit-at=10M max-limit=10M name=local_out parent=bridge
add max-limit=10M name=queue_local_out parent=local_out
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=queue_local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup parent=queue_local_out

The hardware I am using doesnt support traffic limiting on the interface. Perhaps yours doesnt either, and the software wont allow it now. But if you add a child queue to the interface queue, and then make your packet queues parents the interface child queue, it works how you want it to.

Interface queue (local_out)
---> Child (queue_local_out)
---------> Priority1 (ssh_to_bridge)
---------> Priority8 (backup_to_bridge)

mducharme · Mon Dec 14, 2020 1:20 am

I found a simple work around.

/queue tree
add limit-at=10M max-limit=10M name=local_out parent=bridge
add max-limit=10M name=queue_local_out parent=local_out
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=queue_local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup parent=queue_local_out

Your intermediate queue, queue_local_out, is not necessary here. The following configuration should behave in exactly the same way:

/queue tree
add max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup parent=local_out

mikeeg02 · Mon Dec 14, 2020 1:22 am

I found a simple work around.

/queue tree
add limit-at=10M max-limit=10M name=local_out parent=bridge
add max-limit=10M name=queue_local_out parent=local_out
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=queue_local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup parent=queue_local_out
Your intermediate queue, queue_local_out, is not necessary here. The following configuration should behave in exactly the same way:
Code: Select all
/queue tree
add max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup parent=local_out

Without the intermediate queue on this release, the prioritized traffic it not queued/dropped/limited at properly.
CIR works as intended, but the max-limit is the problem.
As he noted, it pretty much is split 50/50 until max-limit is reached on the child queues

EDIT:
It appears in this release, the Max-limit on the interface only forces packets on the child queues that are above their respective limit-at rate (CIR) to be queued. So if you have two priorities, and the combined traffic exceeds interface max-limit, the one(s) above their respective CIR is becomes queued, but it still exceeds the max-limit on the interface. If both priorities are exceeding their respective CIR but still under their respective max-limit, and you want the lower priority to drop traffic, adding interface sub-parent we will call it, fixes this. I believe this is the problem he is experiencing.

mducharme · Mon Dec 14, 2020 1:50 am

Without the intermediate queue on this release, the prioritized traffic it not queued/dropped/limited at properly.
CIR works as intended, but the max-limit is the problem.
As he noted, it pretty much is split 50/50 until max-limit is reached on the child queues

This is what I get with my queue setup described in the previous post without the intermediate queue:

Screenshot 2020-12-13 154838.png

I do believe that is the exact behavior the OP wants.

mikeeg02 · Mon Dec 14, 2020 1:51 am

Without the intermediate queue on this release, the prioritized traffic it not queued/dropped/limited at properly.
CIR works as intended, but the max-limit is the problem.
As he noted, it pretty much is split 50/50 until max-limit is reached on the child queues
This is what I get with my queue setup without the intermediate queue:

Screenshot 2020-12-13 154838.png

With 6.47.8 and what hardware?

mducharme · Mon Dec 14, 2020 1:53 am

With 6.47.8 and what hardware?

6.48 beta58, RB4011 wifi model, fasttrack disabled (of course)

I don't think 6.47.8 should behave any differently.

mikeeg02 · Mon Dec 14, 2020 2:01 am

With 6.47.8 and what hardware?
6.48 beta58, RB4011 wifi model, fasttrack disabled (of course)

I don't think 6.47.8 should behave any differently.

Interestingly enough. I was able to fully duplicate what the OP complained about this morning before I left the house. Now that Ive tested with the sub-parent, and returned to a queue more like described above, it works as it should. I know 6.47 does operate differently than 6.47.8 does.

mducharme · Mon Dec 14, 2020 2:09 am

Interestingly enough. I was able to fully duplicate what the OP complained about this morning before I left the house. Now that Ive tested with the sub-parent, and returned to a queue more like described above, it works as it should. I know 6.47 does operate differently than 6.47.8 does.

I've been using queue trees in this setup for a very long time and haven't noticed any real deviation between versions, at least nothing that extreme. I expect the OP's issue is one of these two things:

- packets are not being marked properly, so some traffic that he expects to be divided between the two queues is actually sharing a single queue due to a logic error in the packet marking, or
- packets are being marked properly, but an external limitation on the SSH bandwidth prevents it from exceeding 5Mbps (not the queue tree's fault), and the backup queue is correctly using whatever is left over that the SSH is not using. If this is the case then the SSH bandwidth should still only be 5Mbps even if the queue tree is disabled.

My tests involved using two Debian ISO mirrors, downloading the ISO, and marking them independently (one as "ssh", one as "backup", based on the IP of the mirror). If I swap the priorities of the ssh and backup queues, the bandwidth amounts reverse as expected. I tried a third Debian mirror that downloaded quite slowly, only at about 3Mbps, and so of course when I used that third Debian mirror instead as the "high priority" queue, the result was that it was only using 3Mbps even though the queue was higher priority. In that case, I think the mirror itself is limiting the transfer rate per user.

Here is the exact config I used, for reference:

/ip firewall mangle
add action=mark-packet chain=postrouting new-packet-mark=backup passthrough=yes src-address=67.212.86.194
add action=mark-packet chain=postrouting new-packet-mark=ssh passthrough=yes src-address=204.157.3.70
/queue tree
add max-limit=10M name=local_out parent=bridge
add comment="SSH 10k guaranteed, high priority" limit-at=10k max-limit=10M name=ssh_to_bridge packet-mark=ssh parent=local_out priority=1
add comment="Backup server SSH, low priority" limit-at=10k max-limit=10M name=backup_to_bridge packet-mark=backup parent=local_out

mducharme · Mon Dec 14, 2020 2:19 am

I would add that once you start mixing connection marks and packet marks, it starts to get quite easy to have a logic error where packets end up getting marked in unexpected ways, or do not get marked at all. In such cases it makes sense to start with a really simple mark-packet mangle rule like I tested with and avoid the connection marks temporarily. Once you know that it works, then you can try moving to a combination of connection marks and packet marks.

mikeeg02 · Mon Dec 14, 2020 2:36 am

Interestingly enough. I was able to fully duplicate what the OP complained about this morning before I left the house. Now that Ive tested with the sub-parent, and returned to a queue more like described above, it works as it should. I know 6.47 does operate differently than 6.47.8 does.
I've been using queue trees in this setup for a very long time and haven't noticed any real deviation between versions, at least nothing that extreme. I expect the OP's issue is one of these two things:

- packets are not being marked properly, so traffic that he expects to be divided between the two queues is actually sharing a single queue due to a logic error in the packet marking, or
- packets are being marked properly, but an external limitation on the SSH bandwidth prevents it from exceeding 5Mbps (not the queue tree's fault), and the backup queue is correctly using whatever is left over that the SSH is not using. If this is the case then the SSH bandwidth should still only be 5Mbps even if the queue tree is disabled.

My tests involved using two Debian ISO mirrors, downloading the ISO, and marking them independently (one as "ssh", one as "backup", based on the IP of the mirror). If I swap the priorities of the ssh and backup queues, the bandwidth amounts reverse as expected. I tried a third Debian mirror that downloaded quite slowly, only at about 3Mbps, and so of course when I used that third Debian mirror instead as the "high priority" queue, the result was that it was only using 3Mbps even though the queue was higher priority. In that case, I think the mirror itself is limiting the transfer rate per user.
Code: Select all
/ip firewall mangle
add action=mark-packet chain=postrouting new-packet-mark=backup passthrough=yes src-address=67.212.86.194
add action=mark-packet chain=postrouting new-packet-mark=ssh passthrough=yes src-address=204.157.3.70

I definitely appreciate and trust your information, you've certainly helped me in the past. I wouldnt have said anything if I didnt see it with my own eyes this morning! Ill have to try to figure out how I had the same condition.

I was using bridge filters to mark packets, had only changed queue settings. I wasnt worried about his connection/marking. I figured he had that worked out. The queuing was the oddity.

mducharme · Mon Dec 14, 2020 2:57 am

I was using bridge filters to mark packets, had only changed queue settings. I wasnt worried about his connection/marking. I figured he had that worked out. The queuing was the oddity.

MPLS is pretty much the only time I use bridge filters to mark packets, or possibly if I wanted to do QoS on bridged traffic it would make sense. If the traffic is routed, it makes a lot more sense to use firewall mangle rules to mark the packets.

In the OPs case, looking at his first post, it is possible for the connection to be briefly renamed from ssh_con back to backup_con and then back to ssh_con, given the rules he has there. Due to the multithreaded nature of packet processing, this could result in some packets that are actually for backup to be flagged as ssh because there might be a second thread that is processing some other packet and marks it during the instant that the connection backup_con is renamed to ssh_con. I also do not see whether passthrough=yes or passthrough=no in the mangle rules in the original post, which can potentially have a huge impact. If passthrough=no on the connection mark rule, it will never actually make it to the packet mark rule.

Using connection marks in tandem with packet marks is much more efficient, but it is also a lot easier to make a logic error, in my experience. I'm pretty certain that is the case here.

mikeeg02 · Mon Dec 14, 2020 3:22 am

I believe I have found how created my problem. I believe it was related to queue size. I was doing testing previously with larger queue sizes and higher bandwidth affects on processor performance and such. I outsmarted myself. My apologies.

nagylzs · Mon Dec 14, 2020 8:06 am

One difference I noticed is that mducharme used postrouting chain for packet marking instead of forward chain. I looked at the packet flow diagrams and I can see that HTB Global queue tree is processed at the end of the input chain, and also at the end of postrouting ( https://wiki.mikrotik.com/wiki/File:Pfd.png ). What is the advantage of mangling in postrouting instead of forward?

I'm confused by the fact that each routed and forwarded packet goes though both forward and postrouting chains ( prerouting -> forward -> postrouting ), so it seems that they go through the HTB Global queue tree twice??

For the record, I have fasttrack turned off completely, and I can download from both of the servers at least 100Mbps when I delete the queues. So the problem is not with the servers or the ISP.

Today I'm going to dedicate a brand new HAP AC2 for this problem, reset all configuration and do more tests using public HTTP servers instead of SSH. It will take a while. Then I'll come back and tell my experiences.

Once again, thank you for your help!

sindy · Mon Dec 14, 2020 9:22 am

One difference I noticed is that mducharme used postrouting chain for packet marking instead of forward chain. I looked at the packet flow diagrams and I can see that HTB Global queue tree is processed at the end of the input chain, and also at the end of postrouting ( https://wiki.mikrotik.com/wiki/File:Pfd.png ). What is the advantage of mangling in postrouting instead of forward?

I'm confused by the fact that each routed and forwarded packet goes though both forward and postrouting chains ( prerouting -> forward -> postrouting ), so it seems that they go through the HTB Global queue tree twice??

From the end: if you look at the routing diagram at the packet flow page, you'll see that packets which get to the input chain never get to postrouting, and vice versa. A significant exception from this may be tunneled traffic where the received transport packet is handled in input and the payload one extracted from it is handled in either input again or postrouting (and similar treatment takes place when payload packets are sent via a tunnel), but that's another can of worms so let's leave it aside for now.

So except transit between two tunnel interfaces, the packet flow is not input -> forward -> output as you seem to expect. For received packets, routing decides whether the packet is for the router itself and will be handled in input, or whether it is for another destination and will be handled by forward.

The difference between mangling in forward and in postrouting is that forward only handles transit packets, i.e. those the router forwards from one interface to another, whereas postrouting handles both forwarded packets and packets sent by the router itself (i.e. router's own DNS queries, downloads etc.) previously handled in chain output. So for same treatment of forward and output traffic, handling in postrouting is a better choice as you use a single set of rules; if you need to treat them differently, use forward and output for selective mangling.

nagylzs · Mon Dec 14, 2020 7:15 pm

All right, here is another try. I tried to create a test that is as easy to repeat as possible.

I'm going to use these speedtest regular HTTP links for testing the speeds:

server1: http://speedtest.tele2.net/ http://speedtest.tele2.net/10GB.zip measured 42MB/sec before the test
server2: http://speedtest.novoserve.com/ http://speedtest.novoserve.com/10GB.bin measured 40MB/sec before the test

The sum of those would be 80MB/sec, that is about 640mbps. The theoretical maximum for my ISP connection is 500MB,
the actual limit was between 440MBps and 250Mbps. I measured this right before the test, before adding the queue trees.
It was fluctuating badly, but it was never below 250 and never above 440. ( https://www.speedtest.net/result/10588376043 )

This seems to be a good test because I can max out the uplink, and test the queues in a realistic situation.

Test board was RBD52G-5HacD2HnD (HAP AC2) running RouterOS 6.47.8

I have connected ether1 to my ISP uplink and ether2 to my laptop. Nothing else was connected to it.

First I reset the configuration to defaults and disabled fasttrack-connection

/system reset-configuration
/ip firewall filter
set [find action=fasttrack-connection] disabled=yes

Then I looked up the IP addresses of the test servers

[admin@MikroTik] /ip firewall filter> :put [/resolve speedtest.tele2.net]
90.130.70.73
[admin@MikroTik] /ip firewall filter>> :put [/resolve speedtest.novoserve.com]
185.80.233.178

I was following the advise, and used very very simple mangle rules. I did not use connection marks. Neither conditions on interfaces.
I was not concerned about router CPU time. The main point was to make sure that all packets are marked correctly.

/ip firewall mangle
add chain=postrouting packet-mark=no-mark  proto=tcp src-port=80 src-address=90.130.70.73 action=mark-packet new-packet-mark=server1
add chain=postrouting packet-mark=no-mark  proto=tcp src-port=80 src-address=185.80.233.178 action=mark-packet new-packet-mark=server2

Then here is the queue tree, as simple as possible:

/queue tree
add max-limit=500M name=local_out parent=bridge
add limit-at=10k max-limit=500M name=server1 packet-mark=server1 parent=local_out priority=1
add limit-at=10k max-limit=500M name=server2 packet-mark=server2 parent=local_out priority=3

Test results:

* Server1 got 343 Mbps
* Server2 got 4.6Mbps
* Meanwhile, CPU was 28% occupied. This board has 2 CPUs, so even the CPU power was more than enough.

Queue tree details and CPU profile here: https://imgur.com/a/33MPeUb

Results

* The full bandwidth goal was reached.
* Traffic was not distributed correctly. As far as I understand, the speed should have
been distributed between the test servers, ratio=1/3.

I tried to make this test as easy to repeat as possible. If you are willing to reset config
on a test board, you can do the setup with 10 commands at most.

Maybe I'm still doing something fundamentally wrong, but I don't know what.
Can you please point me to the right direction? I'm lost.

mducharme · Mon Dec 14, 2020 7:37 pm

Test results:

* Server1 got 343 Mbps
* Server2 got 4.6Mbps
* Meanwhile, CPU was 28% occupied. This board has 2 CPUs, so even the CPU power was more than enough.

Those results are absolutely correct given your settings and are what you should have reached.

I'm afraid you have a misunderstanding of the priority values as being ratio based (1/3) - they aren't. First of all, it gives all the bandwidth it can to the priority 1 queue, once that is all allocated, it gives whatever is left to the priority 2 queue, once that is allocated, it gives what is left to the priority 3 queue, etc. In this case, the priority 1 queue is full and has a max-limit equal to the parent, so what should happen is the priority 1 queue should get basically all of the bandwidth and the priority 3 (or 2, it doesn't really make any difference) queue would get basically none because priority 1 wins for all the bandwidth available in total in the parent. This is what happened.

nagylzs · Mon Dec 14, 2020 7:54 pm

I'm afraid you have a misunderstanding of the priority values as being ratio based (1/3) - they aren't. First of all, it gives all the bandwidth it can to the priority 1 queue, once that is all allocated, it gives whatever is left to the priority 2 queue, once that is allocated, it gives what is left to the priority 3 queue, etc. In this case, the priority 1 queue is full and has a max-limit equal to the parent, so what should happen is the priority 1 queue should get basically all of the bandwidth and the priority 3 (or 2, it doesn't really make any difference) queue would get basically none because priority 1 wins for all the bandwidth available in total in the parent. This is what happened.

Well, no problem at all. I came here to learn. But can you please help me? How can I reach my end goal?

My end goal is very simple:

1. allow full available bandwidth flow through the queue tree
2. when there is not enough bandwidth, then distribute it between leaf nodes with given ratios

I just don't know how to express this with queue trees.

mducharme · Mon Dec 14, 2020 8:04 pm

1. allow full available bandwidth flow through the queue tree
2. when there is not enough bandwidth, then distribute it between leaf nodes with given ratios

I just don't know how to express this with queue trees.

For #2, that is what limit-at is for, basically. You will get closer to the results you want in the previous test you did if you set the priority=3 queue for limit-at=167M. However you'll want to make sure all bandwidth is accounted for in the queueing. The issue with your queue tree setup is all traffic that is part of neither speed test is not being included in the 500M at all, so the queue does not have an accurate accounting of all the traffic on the interface. As a result if you have other traffic at the same time as the speed tests are running, it will think it has bandwidth left when it is already at the limit.

Also, the prioritization will not work entirely correctly if your actual limit at the time is less than the theoretical. You will therefore want to set your outer queue for the slowest rate that you might experience at peak times instead of setting it for the full theoretical hoped-for 500Mbps.

nagylzs · Mon Dec 14, 2020 8:18 pm

For #2, that is what limit-at is for, basically. You will get closer to the results you want in the previous test you did if you set the priority=3 queue for limit-at=167M. However you'll want to make sure all bandwidth is accounted for in the queueing. The issue with your queue tree setup is all traffic that is part of neither speed test is not being included in the 500M at all, so the queue does not have an accurate accounting of all the traffic on the interface. As a result if you have other traffic at the same time as the speed tests are running, it will think it has bandwidth left when it is already at the limit.

There was no other traffic. Only my laptop was connected to the LAN part. All right, there could have been 10kbps unaccounted traffic at max, because I did not stop all services on my laptop. But that basically has no effect on the end result.

Also, the prioritization will not work entirely correctly if your actual limit at the time is less than the theoretical. You will therefore want to set your outer queue for the slowest rate that you might experience at peak times instead of setting it for the full theoretical hoped-for 500Mbps.

Well, the actual limit will always be unknown. And it is not because I have a special kind of network. It is because it is a WAN connection, and all WAN connections are like that. (Unless you have full controll over the route from the starting point to the endpoint.) This is a fact.

Let me put it together the two facts:

* Prioritization will not work correctly if your actual limit is less than the theorietical (given in limit-at)
* The actual limit is and will always be different from the theoretical limit.

Here is the conclusion: prioritization does not work with WAN connections.

After putting so much work into this, I feel very disappointed.

mducharme · Mon Dec 14, 2020 8:21 pm

* Prioritization will not work correctly if your actual limit is less than the theorietical (given in limit-at)
* The actual limit is and will always be different from the theoretical limit.

Here is the conclusion: prioritization does not work with WAN connections.

After putting so much work into this, I feel very disappointed.

Prioritization does work with WAN connections, but you have to lower the maximum limit to the slowest rate that you might actually get at peak times. If you find that is 350Mbps then your queue tree structure should be limited to 350Mbps. Of course then you will never end up being able to use the extra 150Mbps when it is available.

nagylzs · Mon Dec 14, 2020 8:27 pm

If you find that is 350Mbps then your queue tree structure should be limited to 350Mbps. Of course then you will never end up being able to use the extra 150Mbps when it is available.

Well, I had two simple main goals. Being able to use the full bandwidth was one of them.

mikeeg02 · Mon Dec 14, 2020 8:34 pm

Why do you keep only guaranteeing 10kb/s on these two queues anyways?

Is that all thats truly desired to have gauranteed?

Or maybe the whole goal is to learn service based limiting?

mducharme · Mon Dec 14, 2020 8:36 pm

Well, I had two simple main goals. Being able to use the full bandwidth was one of them.

In my case, my home service ISP (a large cable provider) always provides a bit more than the maximum, and doing speed tests at peak times has never given me less than the maximum. As a result, in my case, I'm able to do QoS for the full package speed, albeit sacrificing the extra 10-15 Mbps that they give as a buffer above my max limit. However, not all ISPs will be the same in that regard.

nagylzs · Mon Dec 14, 2020 8:40 pm

Why do you keep only guaranteeing 10kb/s on these two queues anyways?

Is that all thats truly desired to have gauranteed?

As I told earlier, my end goal is to prioritize all TCP traffic. Anything that I put into limit-at will be given. Even if it is not used. If you try to put all TCP traffic into a tree, then it is easy to see that the more limit-at values you give, the more bandwidth you will loose for the leaf queues that are not used at the moment.

So the explanation is this: I have limit-at=10k values to lower the unused (in fact, wasted!) bandwidth. I think that giving higher values would make things worse.

I might be wrong though, I just started learning these queues a few days ago.

mducharme · Mon Dec 14, 2020 8:44 pm

As I told earlier, my end goal is to prioritize all TCP traffic. Anything that I put into limit-at will be given. Even if it is not used. If you try to put all TCP traffic into a tree, then it is easy to see that the more limit-at values you give, the more bandwidth you will loose for the leaf queues that are not used at the moment.

So the explanation is this: I have limit-at=10k values to lower the unused (in fact, wasted!) bandwidth. I think that giving higher values would make things worse.

This isn't exactly the case. Although limit-at sets a minimum guarantee, if the traffic level is low enough that the limit-at is not reached, that bandwidth from the parent will be available for other child queues. As a result, it isn't really "wasted".

You may find the new queue types in RouterOS 7 will meet your requirements better since they are more advanced and can deal with fluctuating total bandwidth amounts.

nagylzs · Mon Dec 14, 2020 8:50 pm

You may find the new queue types in RouterOS 7 will meet your requirements better since they are more advanced and can deal with fluctuating total bandwidth amounts.

Great, I would like to try it. Can I install on HAP AC2? Is it still beta?

nagylzs · Mon Dec 14, 2020 8:56 pm

It was a dumb question, sorry. I can see it at https://mikrotik.com/download

I'm going to try that now.

Who is online