Hi, I have made some large investigation around HTB and I found a weird thing that I would like to discuss here:

The configuration is this:
- I have RB433 with one miniPCI card (R52H), wireless interface set to ap-bridge and one client (RB711) connected to it.
- I have two firewall mark rules: chain output -> mark paket "OutputMark" and chain prerouting -> mark paket "OutputMark"
- I have one queue in Queue tree called WLan1Queue, type is PCQ (classifier all four checkboxes - src port, src address, dst port, dst address), limit-at=1 max-limit=100M, priority=8

Now the problem:
- I start ping from AP to the client (300ms interval, 50bytes), it's avg. 1ms
- Then I start bandwidth test (send UDP to client, no limit)
- As bandwidth test reaches capacity of the wireless link (in a few seconds), the problem appears:

State:
- There are 3 PCQ queues created (according to the classifier - one for ping, one for bandwidth test and one for bandwidth test control - that is what you can see from Torch tool)
- So the outgoing traffic is pretty divided and the round robin algorithm works well
- The problem is, that ping to client shows avg. 62ms

Also:
- Wireless link is without any interference, it is indoors, bandwidth test shows almost 30Mbps, ccq is 95-100, data-rate 54Mbps all the time.
- I was thinking that this is still some weird wireless problem, so I connected these devices using UTP cable (ether1 to ether1) and tried the same test, but with Max-limit=200M (so that HTB does not limit the traffic). Similar thing occured:
The ping was 128ms.
- I also tried to switch to NStreme, 10Mb Ethernet, 802.11N, used different wireless cards, everything behave the same, except the latency and available bandwidth - it varies from interface "type".

- If you set Max-limit lower than the capacity of the link (either wireless or wired), then the HTB will provide limitations and no delay will occur.

- I have read HTB manual hundred times and the interesting sentence is this (refman 3.0):

When there is a possibility to send out a packet, HTB queries all its self slots in order of priority, starting with highest priority on the lowest level, till lowest priority on highest level.

Please note this part: "When there is a possibility to send out a packet..."
What exactly does this mean? Well, I thought that the interface (let's say wireless card) can say to the OS that it is able to send. Or OS can ask the wireless card "Are you able to send?" and it would reply "Yes". So, if it worked that way, there couldn't be any delay! Where would it come from? If the card is able to send 30Mbps (using 1500 UDP pakets) then it means 2500 packets per second, resulting in 0.4ms delay (one way). But constant 62ms delay?
And the ethernet interface, 97Mbps of UDP traffic = 8000 pps = 0.125ms per packet. But 128ms constant delay?

So it seams that THERE IS A HIDDEN BUFFER (FIFO) on each real interface!

And it's size can be calculated:
802.11 interface: 155 packets (1500 bytes)
NStreme interface: 73 packets (1500 bytes)
Ethernet interface: 1000 packets (1500 bytes)


This presents a huge problem for wireless links! For 100Mb Ethernet you can easily set up an HTB limitation to 95Mbps and everything will work fine (it's full duplex and the data rate is fixed). But if you have a wireless link, you never know how much bandwidth you have available. So if you set limitations to HTB (say 20Mbps), it will work perfectly if you have at least 20Mbps available, but what will happen if an interference occures (whether from other wireless protocols, or "interference" from 802.11 network operating on the same frequency)? HTB will not limit the traffic and try to put it into real interface through the "hidden buffer" and as the bandwidth loweres, the ping increases, since it takes more time now to transmit these 155 packets (for 802.11a).

Anybody has come to this problem? Why is there this buffer and what determines it size? Can it be set in ROS? (for example to 10 pakets instead of 155). Thank you for your answers.

Allright, I've looked into this book: Cisco - routing and switching (http://www.ciscopress.com/bookstore/pro ... 1587201968) and I found a page where they say that devices use also a hardware queue, and the description fits exactly. It is FIFO and packets (data frames) from software queues go into them (so if HTB output exceeds posibilities of wireless link, packets start to queue in a hardware queue). Well, although there is certainly a reason for that, Cisco at least lets user to change the size of hardware queue, so user can adjust things to his needs.

Is Mikrotik also able to change the hardware queue size? I believe RouterOS does that somehow, because the queue seems to be shorter for NStreme links than for 802.11a links. Is there some command to do this?

Allright, I've looked into this book: Cisco - routing and switching (http://www.ciscopress.com/bookstore/pro ... 1587201968) and I found a page where they say that devices use also a hardware queue, and the description fits exactly.

What your experiencing is Bufferbloat (What is bufferbloat anyway?). It would be nice if ROS would let you modify the hardware TX queues, which is possible in Linux, so people can attempt to reduce bufferbloat in ROS devices.

Developers?

Well, after reading and listening to Jim Gettys, it actually is a kind of bufferbloat. However what Jim Gettys was talking about are unmanaged queues (FIFO) which are too big (unnecesarily). If you use HTB with one output queue as I did and the type of the queue will be FIFO, size let say 1000 packets, then of course you will have a bufferbloat there. Packets will be dropped only after the whole 1000 packet queue will fill and the queue will remain full (supposing there are enough flows to fill them). The bufferbloat will occur in the HTB queues.

But what I'm talking about is a kind of hardware queue, which is after the HTB queues. Even if you use FIFO in HTB and set it's size to 1 packet (dont ever set it to zero, or you will be screwed ), the packet will then go to the hardware queue. The size of hardware queue for 802.11a is approximately (according to my measurements) 155 packets. And there is the problem - if the packets from HTB queue go into the hardware queue faster than the physical interface is able to send them, the hardware queue will fill and the bufferbloat will occur there.

What is interesting about this hardware queue is that it never drops packets! It just fills up, stays full. RouterOS does not put any packets into it until some packet is send from the physical interface and therefore makes a space for another packet in the hardware queue.

So if you set a speed limit in HTB that is lower than the speed of physical interface, the hardware buffer will always contain only one packet (unless the RouterOS sends packets into hardware queue in chunks which I doubt). And you can easily avoid bufferbloat in your software queues (for example using short PFIFO, RED or using properly set PCQ).

However if you set speed limit in HTB higher than the speed of physical interface, then the hardware queue will fill and bufferbloat will occur, no matter what software queue you use. This is not the problem of ethernet links, since you know exactly what the physical speed is, but is a big problem for wireless links where you don't know the exact bandwidth, especially when there is an interference with another wireless networks. The available bandwidth can change dynamically.

So what you can do is to set the speed limit in HTB to very low level - the minimum speed you know you will get from the link no matter what - and avoid the bufferbloat in hardware queue, which will cost you a lot, since the guaranteed bandwidth will be very low. Or you can set the speed limit high and experience bufferbloat in hardware queue.

This is a real pain in the ass. It seems that you actually cannot do speed limitation and QoS for wireless links on RouterOS that would really work. Or is there a way to change the size of those buffers?

I understand you are experiencing delays due to hardware queues. This is still bufferbloat. Lowering the size of the hardware TX queues is a solution for Ethernet interfaces.

The bufferbloat devs are hard at work on various 802.11 drivers to help reduce the effect of bufferbloat on those types of interfaces.

This is still bufferbloat.

Yes, I agree.

Lowering the size of the hardware TX queues is a solution for Ethernet interfaces.

I think that the right solution is queue management where the bottleneck occurs - use of software queues with minimal hardware queue (1 packet would be the best). Queue sizes at network devices where the bottleneck never occurs are irrelevant.

The problem of bufferbloat occurs only on devices which are the bottleneck, that I would like to emphasise. So if you know where the bottleneck wil occur, you surely have to implement software queue management there (for example at your Internet gateway if you are an ISP, or at the AP, at some slow backbone link etc.). Just lowering hardware queue sizes won't solve that problem well. In fact hw queues are FIFO and therefore are not fair to all traffic flows (some of them may starve). So I think hardware developers can't do anything about it. Maybe keep hardware queue size low for "not so inteligent" devices that don't use software queues - like home wifi routers, whose owners experience congestion on them.

Lowering hardware queue size would solve a bufferbloat problem, but would surely not solve congestion problem (at least not in a good way), right?

Right. I was talking about about lowering tx queue length as being the solution the specific problem you're seeing with packets being queued below ROS when the interface is saturated.

I agree other algorithms are still needed at layer 3 to signal early network congestion (RED w/ ECN, SFB, etc).

Still no change in RouterOS 5.6. Seems like wireless QoS is really not a point of interest? Just a single parameter would be nice to add: hardware-queue-size.

Still no change in RouterOS 5.6. Seems like wireless QoS is really not a point of interest? Just a single parameter would be nice to add: hardware-queue-size.

I have been struggling allot with qos over wifi and in my opinion htb is a wast of time. It relays on the principle that you know the Max-limit, and this is unpredictable in a wireless environment.

On the meanwhile I'm having very good results with NV2 qos. My topology is (bridge<->station bridge) with NV2, queue count=4 and qos= frame priority. The data is going inside vlans. Just one bridge filter rule to extract priority from vlan and set it to the right queue inside NV2.

I hope this helps.

Kindly regards,

I think you need to examine this:
http://www.tiktube.com/index.php?video= ... vnlIomlpG=

I think the main confusion will be resolved by this:
(slide 28): "HTB priority doesn't rearrange packet sequence – it doesn't put some packets before others"

Every interface have a queue type on them after all HTBs - check /queue interface menu

On the meanwhile I'm having very good results with NV2 qos. My topology is (bridge<->station bridge) with NV2, queue count=4 and qos= frame priority. The data is going inside vlans. Just one bridge filter rule to extract priority from vlan and set it to the right queue inside NV2.

Well, when using 802.11a you can use WMM to split outgoing traffic int o 4 hardware queues. But this still does not solve the problem of too big hardware queue. How big the queue is when using NV2? I mean if you run ping to the other side and also run Bandwidth UDP test (only one way), which exceedess maximum bandwidth of the link, how big the latency is? and what is the real bandwidth?

I think you need to examine this:
http://www.tiktube.com/index.php?video= ... vnlIomlpG=

I dont see any new information in it...

I think the main confusion will be resolved by this:
(slide 28): "HTB priority doesn't rearrange packet sequence – it doesn't put some packets before others"

That is not entirely truth. HTB does put some packets before others if it is set so. If you split traffic to two queues with different priority, then at some circumstances (e.g. max-limit or parent reached) packets from queue with higher priority (lower number) will get to the hardware queue before packets from the other queue.

Every interface have a queue type on them after all HTBs - check /queue interface menu

If you create a queue in HTB and set its parent as a real interface, then the setting in /queue interface does no longer apply. /queue interface applies only if no queue tree is set or if you set parent interface as "global-out" instead of a real interface.


Anyway, this problem exists even without any HTB (tree queue). If you use only /queue interface then still packets that pass these queues will get into hardware queue. Set for example interface queue type as PCQ and see results. Run some terrible UDP traffic throught the router (one port) and try to ping to other side. Ping should be different PCQ queue (dynamically created). It really is. But as your hardware queue gets full, all ICMP ping packets that leave PCQ queue will have to wait in the hardware queue and therefore you will see increased latency.

That is not entirely truth. HTB does put some packets before others if it is set so. If you split traffic to two queues with different priority, then at some circumstances (e.g. max-limit or parent reached) packets from queue with higher priority (lower number) will get to the hardware queue before packets from the other queue.

Maybe because of queue types, but not because of the priority. Priority is just a option that indicates when to start dropping packets

Anyway, this problem exists even without any HTB (tree queue). If you use only /queue interface then still packets that pass these queues will get into hardware queue. Set for example interface queue type as PCQ and see results. Run some terrible UDP traffic throught the router (one port) and try to ping to other side. Ping should be different PCQ queue (dynamically created). It really is. But as your hardware queue gets full, all ICMP ping packets that leave PCQ queue will have to wait in the hardware queue and therefore you will see increased latency.

What is your CPU load when this happen? Hardware queue in this case is also known as interface driver and have little to do with RouterOS QoS. That is why you need to place your limits in QoS as close as possible to realistic amounts of traffic that you have available. (putting 200Mbps limit on 100Mbps interface is illogical)

On the meanwhile I'm having very good results with NV2 qos. My topology is (bridge<->station bridge) with NV2, queue count=4 and qos= frame priority. The data is going inside vlans. Just one bridge filter rule to extract priority from vlan and set it to the right queue inside NV2.

Well, when using 802.11a you can use WMM to split outgoing traffic int o 4 hardware queues. But this still does not solve the problem of too big hardware queue. How big the queue is when using NV2? I mean if you run ping to the other side and also run Bandwidth UDP test (only one way), which exceedess maximum bandwidth of the link, how big the latency is? and what is the real bandwidth?

WMM if different from NV2 qos.

While wmm is for accessing the media first (reducing the contention window), NV2 qos has specific queues, and priority will let you choose which data goes to each queue. High priority queue will be transmitted first.

Please take a look at the NV2 qos mechanism: http://wiki.mikrotik.com/wiki/Manual:Nv ... v2_network

RTT in NV2 varies allot, but it my tests i cannot see the hardware queue effect, and the priority works well.

My pings remain in 5~7ms while the link is saturated.

By the way, in v5.8 the default Ethernet interface queue has changed, now is only hardware queue "none".

Kindly regards,

Maybe because of queue types, but not because of the priority. Priority is just a option that indicates when to start dropping packets

No....

1) HTB itself does not drop any packet!
2) Packets can be dropped only in queue. (maybe there is a special case when packets are dropped if CPU cant handle it)
3) HTB does rearange packets according to priority if it is set so and such a situation happens

Let me explain how it works - each end class in HTB has it's max-limit value. If packets that fit to a certain queue arrive to router faster than max-limit value (in a certain interval), then they are put in queue attached to this class. If the queue is for example FIFO, then there can be packet drop, if such packet does not fit into the queue (queue is full).

Every leaf class of HTB has this queue. Whenever there is possibility to send packet (parent classes allow this, hardware queue is empty, max-limit is not reached), the HTB takes some packets from this queue and puts them into hardware queue.

If you have more of these HTB classes, the decision of how many packets should be taken from which class is dependent on priority and the state of the class.

Lets make an example:
Suppose you have 2 sources generating UDP traffic that should go through the router, each 700kbps. When these packets arrive at the router, they should be sent throught one output interface. Lets add a root class to queue tree called Upload, parent=output interface, limit-at=100k, max-limit=1Mb. Then let's add two subclasses, each for its source generator. Class A will get limit-at=50k, max-limit=1M, priority=8. Class B will get limit-at=50k, max-limit=1M, priority=7.

With this setting you will see that Class A gets 300kbps and Class B gets 700kbps on the output. Packets that arrive to Class A end up in a queue for Class A, because they cant be sent (of course there will be packet loss after the queue fets full).

However what I'm trying to say is that packets from data generator from ClassB will "overtake" packets from generator for Class A, because they will hang in the queue attached to Class A. So there will be packet reordering by HTB. Agree? And all packets that will be dropped will be dropped only because they cant fit into queue (queue is full), agree?

WMM if different from NV2 qos.

While wmm is for accessing the media first (reducing the contention window), NV2 qos has specific queues, and priority will let you choose which data goes to each queue. High priority queue will be transmitted first.

Yes, I agree. WMM is different. It does what you say, but also it has additional effect. If you fully saturate hardware queue for Normal priority, then the queues for Video and Voice will be still empty and you can send for example ping packets through them and see minimal delay. As well it works if the Video queue is full and you have empty Normal priority queue - if you send packet throught the normal priority queue you will see minimal delay. So you can "prioritize" packets to Normal queue instead of Video or Voice queue.

RTT in NV2 varies allot, but it my tests i cannot see the hardware queue effect, and the priority works well.

My pings remain in 5~7ms while the link is saturated.

If you saturate the link fully (using UDP), then you have to set some congestion mechanism anyway. If you saturate one NV2 queue fully and use other for pings, then I belive you get nice pings. But what about if you send ping throught the saturated queue? Then of course you get very bad pings.

What I mean is that I want for example to use PCQ for these cases. If a link is saturated, then lets split the speed among all users. Dont leave anybody starving or having high pings just because of anybody else. My vision of QoS is not about types of traffic, but about providing quallity to all users regardless what other users in the network are doing. That's why 8 queues in NV2 does not help to solve my problem. PCQ solves it very well, but then there is the problem with the hardware queue.....

Do you use any specific cogestion mechanism? Like special queue, HTB or something?

By the way, in v5.8 the default Ethernet interface queue has changed, now is only hardware queue "none".

I see that, just updated routers last week.... My settings remained ethernet-default, PFIFO 50packets... but I can see that I can set queue type None... Will look at that closer, maybe its something usefull. Thank you.

What is your CPU load when this happen? Hardware queue in this case is also known as interface driver and have little to do with RouterOS QoS. That is why you need to place your limits in QoS as close as possible to realistic amounts of traffic that you have available. (putting 200Mbps limit on 100Mbps interface is illogical)

CPU load can be as low as 20%. Well, I really think that hardware queue has much to do with any QoS. Especially in wireless networking. In wireless networking it is hard to tell what is "realistic amount of traffic", because you never know how much bandwidth you have. You can assume you have 1Mbps on link that 99% provides 20Mbps (and 1% only 1Mbps). But that is a nonsence and waste of money, time and energy.

Putting 200Mbps limit on 100Mbps interface is illogical. But is logical to put 15Mbps limit on a wireless AP? And what about if you have 1client connected at 54Mbps and one connected at 6Mbps? How do you solve that? Will you decrease the limit to 4Mbps? I guess not.

3) HTB does rearange packets according to priority if it is set so and such a situation happens

This is not true, our HTB doesn't rearange packets according to priority, it discributes available amounts of traffic according to priority, this way deciding what packets will be dropped.

So MacGaiver is right

Maybe because of queue types, but not because of the priority. Priority is just a option that indicates when to start dropping packets

No....

However what I'm trying to say is that packets from data generator from ClassB will "overtake" packets from generator for Class A, because they will hang in the queue attached to Class A. So there will be packet reordering by HTB. Agree? And all packets that will be dropped will be dropped only because they cant fit into queue (queue is full), agree?[/i]

Why no?? Your example just perfectly illustrated what i'm saying - it all depends on queue type. Queue type is only thing that can delay a packet and change order, if you will put queue-size=0 (theoretically) for all queue types you will be able to see, that priority have no effect on order.

This is not true, our HTB doesn't rearange packets according to priority, it discributes available amounts of traffic according to priority, this way deciding what packets will be dropped.

So MacGaiver is right

Maybe it's just about seeing it from different angle? Like "what is the HTB"? Like I say, if packets are queued by HTB in a queue due to lack of "distributed amout of traffic", other packet arriving after them can get to the real output interface sooner (not queued in HTB queue) and therefore there will appear packet reordering. Or am I seeing this somehow still wrong?

"deciding what packets will be dropped" - HTB drops packets only if the queue for the class is full. Right? So I say "the queue drops packets" you say "HTB drops packets", but I think we mean the same thing. Right?

Why no?? Your example just perfectly illustrated what i'm saying - it all depends on queue type. Queue type is only thing that can delay a packet and change order, if you will put queue-size=0 (theoretically) for all queue types you will be able to see, that priority have no effect on order.

I would never set queue-size=0 again then the router wont respond anymore (at least thats what happened to me on RouterOS v4.x). If you however theoretically set the queue-size=0 then yes, priority wont cause any packet reordering. However if you have some queue-size>0, then HTB can hold packets in it and can put packets from other queues to output interface before these queued. Even if the non queued packets came later. That I call reordering by HTB. Not by queue.

Moreover, normally, if you have queue-size big enough there will be no packet drop for TCP traffic at all! The limit is set by Congestion window of the TCP protocol. The maximum size is standardly 64kB, so you will see something like max. 42 queued packets. The speed limit for this transmission is not driven by packet dropping, but rather by delaying the packets in queue.

Ah there is the problem - v4.x , If you have PCQ you must have at least v5.4 (or v5.8 - because of some crash fix in PCQ), before that PCQ was slow on high speed or high packet connections.

Hi, I have made some large investigation around HTB and I found a weird thing that I would like to discuss here:

The configuration is this:
- I have RB433 with one miniPCI card (R52H), wireless interface set to ap-bridge and one client (RB711) connected to it.
- I have two firewall mark rules: chain output -> mark paket "OutputMark" and chain prerouting -> mark paket "OutputMark"
- I have one queue in Queue tree called WLan1Queue, type is PCQ (classifier all four checkboxes - src port, src address, dst port, dst address), limit-at=1 max-limit=100M, priority=8

Now the problem:
- I start ping from AP to the client (300ms interval, 50bytes), it's avg. 1ms
- Then I start bandwidth test (send UDP to client, no limit)
- As bandwidth test reaches capacity of the wireless link (in a few seconds), the problem appears:

State:
- There are 3 PCQ queues created (according to the classifier - one for ping, one for bandwidth test and one for bandwidth test control - that is what you can see from Torch tool)
- So the outgoing traffic is pretty divided and the round robin algorithm works well
- The problem is, that ping to client shows avg. 62ms

Also:
- Wireless link is without any interference, it is indoors, bandwidth test shows almost 30Mbps, ccq is 95-100, data-rate 54Mbps all the time.
- I was thinking that this is still some weird wireless problem, so I connected these devices using UTP cable (ether1 to ether1) and tried the same test, but with Max-limit=200M (so that HTB does not limit the traffic). Similar thing occured:
The ping was 128ms.
- I also tried to switch to NStreme, 10Mb Ethernet, 802.11N, used different wireless cards, everything behave the same, except the latency and available bandwidth - it varies from interface "type".

- If you set Max-limit lower than the capacity of the link (either wireless or wired), then the HTB will provide limitations and no delay will occur.

- I have read HTB manual hundred times and the interesting sentence is this (refman 3.0):

When there is a possibility to send out a packet, HTB queries all its self slots in order of priority, starting with highest priority on the lowest level, till lowest priority on highest level.

Please note this part: "When there is a possibility to send out a packet..."
What exactly does this mean? Well, I thought that the interface (let's say wireless card) can say to the OS that it is able to send. Or OS can ask the wireless card "Are you able to send?" and it would reply "Yes". So, if it worked that way, there couldn't be any delay! Where would it come from? If the card is able to send 30Mbps (using 1500 UDP pakets) then it means 2500 packets per second, resulting in 0.4ms delay (one way). But constant 62ms delay?
And the ethernet interface, 97Mbps of UDP traffic = 8000 pps = 0.125ms per packet. But 128ms constant delay?

So it seams that THERE IS A HIDDEN BUFFER (FIFO) on each real interface!

And it's size can be calculated:
802.11 interface: 155 packets (1500 bytes)
NStreme interface: 73 packets (1500 bytes)
Ethernet interface: 1000 packets (1500 bytes)


This presents a huge problem for wireless links! For 100Mb Ethernet you can easily set up an HTB limitation to 95Mbps and everything will work fine (it's full duplex and the data rate is fixed). But if you have a wireless link, you never know how much bandwidth you have available. So if you set limitations to HTB (say 20Mbps), it will work perfectly if you have at least 20Mbps available, but what will happen if an interference occures (whether from other wireless protocols, or "interference" from 802.11 network operating on the same frequency)? HTB will not limit the traffic and try to put it into real interface through the "hidden buffer" and as the bandwidth loweres, the ping increases, since it takes more time now to transmit these 155 packets (for 802.11a).

Anybody has come to this problem? Why is there this buffer and what determines it size? Can it be set in ROS? (for example to 10 pakets instead of 155). Thank you for your answers.

Look at my testbed scenarios. On the second and third you will have the 128ms delay (windows 7 interface 10/100) and 20ms if doing same test with ubuntu 11.04.
This is my testbed.


This was done without any queue on RB433UAH, the idea was just to discover where was the delay coming from.

As far as i can see this might be related to Ethernet receive buffer more than anything else.

Ah there is the problem - v4.x , If you have PCQ you must have at least v5.4 (or v5.8 - because of some crash fix in PCQ), before that PCQ was slow on high speed or high packet connections.

Is 4Mbps using 1500B packets a high speed connection? I will try the behavior using v5.8, but I'm pretty sure this will be the same.

Look at my testbed scenarios. On the second and third you will have the 128ms delay (windows 7 interface 10/100) and 20ms if doing same test with ubuntu 11.04.
This is my testbed.

This was done without any queue on RB433UAH, the idea was just to discover where was the delay coming from.

1) In the first test you are sending 100M UDP from 192.168.1.2 and ping packets from 192.168.1.3. If you are using Fast Ethernet on every link, then there doesnt have to be full link saturation. Iperf can generate slightly less than 100M traffic (enough for ping packets to fit). Try at least 110M, where these 10M will be dropped in outgoing queue on PC 192.168.1.2.

If there is output queue set to PFIFO, size=50packets on the eth1 of RB433UAH, and PC 192.168.1.2 fully saturates the eth2 link, then the delay of ping packets will be around 128ms. This is caused by going through full queues - 50packets software queue + approx.1000 packets hardware queue, packet size 1500 (output queue of RB433UAH on eth1). You can also experience some packet loss if the software queue gets full. If the speed is slightly under 100M, then there will always be enought space for ping traffic from eth3 and the delay will be minimal.

2) In the second and third test you may experience increased ping delay due to 192.168.1.2 PC output queue as you say. The reason is in the way that Iperf (or Mikrotik BWTest) generate UDP traffic. You can also see increased ping delay even if using only 50Mb output UDP. The data generator generates several packets (a packet burst) and sends it to the network card hadrware queue (or to the driver). Then waits for a certain time and do this again. If you ping program tries to send ping packet, there may be situation that the hardware queue contains some packets (the burst has not been sent yet, because it can send only 100Mbps) and therefore the ping packet has to wait in the queue - there comes the delay. However, the output queue can be also empty (if you send less than 100M, the burst was already sent and new burst has not been generated yet) - and therefore no delay.

The difference between Windows7 and UBNT can be in different Iperf function or in different hardware queue size - if the queue is the reason, you can also experience some packet drops.

When I met Dell representative in London, I criticized the switches they were presenting, that use Marvell chips with advertised QoS capabilities. My critique was that we are not able to control the queue properly to achieve usable results. The issue we also discussed with A****a engineers during their visit in Plovdiv, Bulgaria (some Marvell chips in their products). I also did lab tests with Broadcom and Realtek chips - same issue - can't control the Queue = no usable QoS. Therefore someone should fine them for false advertising?

I don't know if it was discussed. But coming from kernel development. Packets that are queued in kernel ARE processed in chunks. That is the TICK rate of the kernel. Low speed CPU's in a non-RTOS scenario are really only useable (performance wise) when this tick is at a low rate, while better computers can handle higher TICK rate. So if TICK rate is 1000 times a second (which is on the high end) it will probably try to pick up those queued packets every millisecond (actually, every few milliseconds due to scheduling have a different process/context ready to run first). But as i wrote, low speed CPU's can only handle a relatively low tick rate, somewhere around 150 interruptions/second (roughly 7 ms). So at every opportunity the kernel thread that drains queues actually do get to run, it drains as many packets as it can (bursts), until kernel queue is empty, driver output queue is full (hw queue usually), the queue bandwidth restriction is met or it gets interrupted.
Which means the OS has to keep the run-queue list as short as possible for the highest possible chance that the kernel queues are drained.
I mean, after all, this is not an RealTime OS that handles events/queues at very exact times and at an allocated cpu share.

It would be nice if MikroTik guys could tell us now that they have taken care of these performance potential issues so RouterOS is much better and would not suffer from such I think hardware-only queue on RouterBOARDS is a step towards the right direction

Oh most definitely. 128 ms is a loong time when i need a network that is below 60 ms. HTB queues, as suggested, can't even help me as MPLS frames as HTB only affects IP.
What would be *really* nice is a more unified and adaptive queue system.

A queuing system that where you specified a queue length in Milliseconds and the system would adjust this queue size depending on link data rate (or Modulation-TX-Rate * Overall-tx-CCQ) at say 0.5 second intervals.

What would also be nice is a queue with X number of sub queues for each packet Priority (64 or that would automatically adjust it's Total-Queue-Size based on data rate (as described above). This would save a LOT of mundane updates on simple queues (using max-limit), queue trees (max-limit), wireless (Modulation-TX-Rate * Overall-TX-CCQ) and on ethernet interfaces.

Additionally it would be very nice if those queues took hardware queues into consideration, adjusting those as well (but i realize sometimes that is much harder to implement).

Just imagine how much effort and time we could save;
* Administrative effort (adding a new customer doesn't require adding lots of HTB queues.
* Debugging time (readjusting queues due to signal degredation or interference).
* No longer have to maintain lots of static simple queues .
(because dynamic queues only use default-small, which doesn't work all that well when you have different speeds or asymmetric speeds)

Or the equivalent multi-queue in combination with packet Priority.

That would bring RouterOS much closer to the administrative ease level of the higher end routers.

sten, your post is gold with platinum ontop. Because this way Operators using MikroTik could offer much better bandwidth+SLA contracts! If MikroTik solutions could do this, some people would not even look at optic and more expensive solutions that go around such issues by simply having HUGE available bandwidth.

Can we compare this to an existing technology? That way it would be easier to present it to devs. Although your explanation is very clear.

sten, I'm very glad there is other person interested in real QoS on Mikrotik Now we are 3 and still growing!

Your post about TICK rate is quite interesting. It can be like that, it would be logical. But if the tick rate is 150 ticks per second, that means it collects packets every 7 ms, the average delay should be 3.5ms. But on empty wireless link you get 1ms. So are there more ticks per second? 1000?


About the second post

A queuing system that where you specified a queue length in Milliseconds and the system would adjust this queue size depending on link data rate (or Modulation-TX-Rate * Overall-tx-CCQ) at say 0.5 second intervals.

:

I dont belive it is so easy to say "give me queue with 40ms" and make it work. If this mechanism would change queue length according to the previous 0.5s bandwidth history, the queue would still be the very same queue. And it would drop packets when full. What good it is to have 40ms delay, when you loose packets that shouldnt be lost?

Especially for TCP connections losing a packet is quite a disaster. You lose one, and the rest 40 packets are waste. So you lose 41 You would get nice ping (some of them maybe lost), but terrible TCP bandwidth I guess. For TCP connections you need to hold the traffic in queues and regulate the speed by that. Not by dropping packets. So for TCP you need big queue.

I think the only thing we need is to have high TICK rate (if it works so), very small hardware queues and software HTB queues. With this you can do QoS totally according to your will. The only question is "Is it possible? Is the hardware powerfull enough to shorten the hardware queue and raise the TICK?". I would love to have control over this (hardware queue size), even if the bandwidth would be half.

Your post about TICK rate is quite interesting. It can be like that, it would be logical. But if the tick rate is 150 ticks per second, that means it collects packets every 7 ms, the average delay should be 3.5ms. But on empty wireless link you get 1ms. So are there more ticks per second? 1000?

Not at all, you have to consider tasks often just sit and wait for input on a socket or other event. Those tasks only poll to check if any event has happened and then immediately yields CPU back to the scheduler, which continues to the next task. When a NIC has received packets (with or without interrupt moderation) it will interrupt the CPU to grab those packets and put them in memory and a task is added to the scheduler task queue to handle those packets at earliest convenience. That might be several tasks later, but very often kernel gets to run before other tasks, but not always. Usually kernel tasks would get would get a very high priority to execute to "feed" waiting applications/threads and to forward packets. Packets are handled in bulk every time they are processed to make handling as fast as possible. But once packets land in the kernel packet queuing queues (HTB/pfifo/bfifo/sfq/pcq etc) and scheduler task item is added and run at earliest convenience according to scheduler but since there is no external interrupt event like an IRQ so they may sit there until scheduler runs the task that drains the queuing queues. Those packets that do not get drained in that batch (and have to wait, perhaps because of drain rate/speed constraints) will have a new task queue handler added but since there is a high likelihood of queuing packets not being drained fast enough (drain speed constraints) those handlers may not get highest CPU run-time priority as that would put the kernel in a loop spinning for packets that are not allowed to be drained queuing queue any faster than the drain constraints. You also have to take into account a task running in between previous queuing drain queue and the next might not yield it's processing time until it is forcefully interrupted at CPU TICK time. Of course there are more variables to this making it a bit more complex.
A higher CPU TICK rate would help on maximum possible wait time for packets in queue but the context switches per second alone could, and probably would, make the smaller boards useless. Often these things are adjusted at boot time depending on the CPU speed, but not automatically, someone has to program that logic (which i believe is there but i still believe the TICK rate is too low on some boards). It is not uncommon for PC routers running time-sharing systems to have a TICK rate of a few thousand hertz. Especially for those that use polling instead of IRQ's to pull in packets from the NIC's.

About the second post

A queuing system that where you specified a queue length in Milliseconds and the system would adjust this queue size depending on link data rate (or Modulation-TX-Rate * Overall-tx-CCQ) at say 0.5 second intervals.

:

I dont belive it is so easy to say "give me queue with 40ms" and make it work. If this mechanism would change queue length according to the previous 0.5s bandwidth history, the queue would still be the very same queue. And it would drop packets when full. What good it is to have 40ms delay, when you loose packets that shouldnt be lost?

Especially for TCP connections losing a packet is quite a disaster. You lose one, and the rest 40 packets are waste. So you lose 41 You would get nice ping (some of them maybe lost), but terrible TCP bandwidth I guess. For TCP connections you need to hold the traffic in queues and regulate the speed by that. Not by dropping packets. So for TCP you need big queue.

I think the only thing we need is to have high TICK rate (if it works so), very small hardware queues and software HTB queues. With this you can do QoS totally according to your will. The only question is "Is it possible? Is the hardware powerfull enough to shorten the hardware queue and raise the TICK?". I would love to have control over this (hardware queue size), even if the bandwidth would be half.

Actually, as counter intuitive as it may sound, you want packets to be dropped at some point.
TCP was designed to back off at congestion and it "detects" congestion by packet loss. Yes TCP speed will suffer when queues are smaller and packet loss occurs, yes there will be, as in your case 41 packets that needs retransmission, but also reduce it's TCP bulk transmission size by a bit in the case of TCP fast-restart (or a lot in TCP slow-restart) but this is exactly what you want to happen, otherwise every TCP sender would just blindly push more packets to their doom.

QoS is not all about giving TCP a good chance at very high transmission performance, it is by giving different types of traffic a more fair chance at performance. A lot of applications/protocols, such as DNS or when TCP is initiating connection, do not send massive amounts of traffic but instead their request/response time is the most important factor for good performance and in my opinion it would not be a very high quality of service to make them wait several hundred or even thousands of milliseconds so that someone might have their download bulk transmission a tiny bit faster. You might say it is often preferred to drop TCP data packets over TCP SYN, SYN/ACK, "empty" ACK packets and connection less packets as not every application is so generous when their packets are dropped or delayed too much and an infinite queue means infinite delay. And since we cannot single out those small connection-establishing TCP packets or TCP ACK packets and prioritize those (we don't usually want TCP packet reordering since that would be far worse than in terms of performance, it may even break connections). And you might say those latency sensitive applications/protocols are more important to me than that youtube download for my users perception of performance.

I would certainly set my queues to a size based on how many milliseconds it may buffer, to not cause situations where the latency across a congested link is in the thousands of milliseconds, TCP bulk transfer performance be damned if that can't be achieved without dropping thousands of packets. Unfortunately this is not very granular in a time-sharing (non realtime) operating system and unfortunately that is only made worse by those unconfigurable hardware queues that in my opinion are way too big by default, that makes it practically impossible to drain quickly enough, making even the best effort of attempting to have sub 100 millisecond response time when congestion occurs. Hardware queues with packet counts close to a hundred is fitting for an end node like a server or a laptop.

We would certainly desire to have queue size determined by time. As it stands right now, we have to calculate how many bytes 100ms of link rate X is in terms of bytes (or packets) and set interface to that queue size, and that will only be good enough for as long as link rate remains at X. It will no longer be queuing at the required Quality of Service level when that link rate changes or the CCQ is lowered.

But since there are hardware queues, with queue sizes that in my opinion are too large and with double queued packets that a somewhat high potential for added latency, with that that cannot be adjusted to fit within the Quality of Service levels that modern networks require. What is even more exasperating is that the router itself has a much better chance at continuously adapting to expected link rates than any script/api/other means that is within the administrators reach, especially those that do not require constant flash rewrites and wastes lots of time handling adjustment updates.
I think we can (have to) live with the fact that it doesn't always guess 100% right all the time, it cannot see into the future, but it can make a pretty good educated guess based on the last few seconds of history. It is also a time-sharing system that does not handle packets individually but in bulk. Therefore i don't think we can demand that it does it perfectly according to our time constraints but close enough considering the problems at hand. But even a small application process running in the background, gathering data, making guesses and readjusting queue accordingly at a certain interval. Even one that runs only once per second, although sub second would give better granularity, would give an exceptional performance improvement in terms of network performance (speed vs latency) and in terms of the high very high "round-trip time" of the administrator, also the hardware cost in terms of flash writes, when making updates to the queues.

What MAY prove to be a bit tricky, based on my experience in other time sharing systems, is to adjust the kernel queuing queue size without dropping all packets in the queuing queue, depending on how that is already handled. Perhaps one has to make adjustments there. Hardware queue sizes could stay relatively constant after initial configuration and but should be relatively small to decrease variations in latency but large enough to give a reasonable performance, but i agree, it is very desirable (one might say necessary) to have this under administrative control. I am sure there are other minor aspects to it. But if a program can do it remotely (albeit extremely inefficiently and with hardware wear and tear) then a program can do it locally without that overhead and inefficiency.

Perhaps one day, there will be a realtime component that would process these things at a reserved CPU time or even on reserved CPU core, and at per packet granularity, but until then i would welcome a small background that helps reduce the gap between the two architectures.

I'd like to point you to this Wiki article to help clarify the case.
http://wiki.mikrotik.com/wiki/Queue_Size and these external references:
http://en.wikipedia.org/wiki/Bufferbloat
http://www.bufferbloat.net/projects/blo ... nicalIntro
http://www.cringely.com/2011/01/04/2011 ... two-words/
http://www.readwriteweb.com/enterprise/ ... lution.php

When a NIC has received packets (with or without interrupt moderation) it will interrupt the CPU to grab those packets and put them in memory and a task is added to the scheduler task queue to handle those packets at earliest convenience.

So... when packet arrives at interface, it goes to the memory and a task is added (to handle this packet later). And it is done as soon as possible, but in the worst case every 1000/TICKS ms (7ms). And if CPU is idle, then it's done almost immediatelly. Is that right? If it is, 7ms is not so bad time for me, isn't it?

I'd like to point you to this Wiki article to help clarify the case.

Yes, Jim Gettys' Bufferbloat, have read all about it last year:-) And I agree that hardware queues in many many devices are very huge and shouldnt be. As I investigated (and mentioned in posts before) on Mikrotik it is about this:

802.11 interface: 155 packets (1500 bytes)
NStreme interface: 73 packets (1500 bytes)
Ethernet interface: 1000 packets (1500 bytes)

Which gives you quite high latency when link gets fully saturated. Thats the biggest problem from my point of view.

QoS is not all about giving TCP a good chance at very high transmission performance

That is right. But I must dissagree about the packet drop. Standard TCP has maximum contention window 64kB, that is about 43 packets of 1500B size. No more packets (unconformed) from one TCP connection will ever occur in the network! And since you have to deliver all of these packets eventually (if not now, you will have to do it later, if you drop it, it will be resent), and they are not latency sensitive, you can put them in queue. That is better than dropping them. To make myself clear, I'm talkin about only ONE TCP connection, not about many TCP connections.

If you have more TCP connections, I think you should use PCQ for TCP connections, so that none of these connections starve. If you do so, you get good performance of SYN/ACK packets (new connections will have new free subqueue, leaving old queues with lots of packets) and still you dont have to drop a single packet. If you delay TCP packets in queue, there will be no ACKs and therefore no more data (situation with full contention window). The TCP will backoff by this mean. Let running TCP connections have 3000ms latency. That is no problem. The data will be send as soon as possible. You can't make them go faster. You have only to take care about new connections and latency sensitive data.

Moreover, if you drop TCP packets, you may degrade the transmission pretty bad. If you use ONLY FIFO queue, there can occur SYN packets loss, ACK packets loss, and even normal data packet loss means retransmission of 40 packets. If you use PCQ for TCP (subqueue size can be 50, which is fine for 43 max TCP packets), there is not such a problem (allright, if you have 10k+ of connections and still increasing - attack?, you might want to drop some packets so your memory does not jump out of the board. Still you can do this in PCQ total size).

And you might say those latency sensitive applications/protocols are more important to me than that youtube download for my users perception of performance.

Yes, agree. Lets put TCP packets into PCQ and lets send more important data before them almost without queueing. Thats good QoS. And you need small hardware queues for that. Because if your hardware queue if full of packets (155 for 802.11), then you can have super duper software settings that will send important, latency sensitive packets immediatelly to the hardware queue (lets say within 0.1ms), but.... there it has to wait before aaaaaallll the previous packets are sent (approx 70ms for 25Mbit link, 118ms for 15Mbit link and so on). The software queue can be as short as possible, but with large hardware queue, this will never work.

I would certainly set my queues to a size based on how many milliseconds it may buffer

As you wrote about 64 priority subqueues, that might work. For latency sensitive data. But still you have to handle packet dropping. Or would you drop packets just randomly as they can't fit into full queue?

I strive to achieve 0ms latency (less than 1 or 1ms) end-to-end in any network I operate. By using 1Gbps+ etc.

I feel bad if a network I operate is more than 3 ms end-to-end and I don't want to have anything to do with such network.

Its just how I am.

I strive to achieve 0ms latency (less than 1 or 1ms) end-to-end in any network I operate. By using 1Gbps+ etc.

Lucky boy, you use ethernet wired links. Set HTB limit under 100M or 1000M for gigabit and enjoy. We have to operate wireless links, which are much more trouble 30ms through the network is a good state.

The proper saying is "lucky man" not lucky boy. I wouldnt say this is luck. If you switch to nstreme on wirelss don't you get up to 3ms latency per link without any aggregation ?

The proper saying is "lucky man" not lucky boy. I wouldnt say this is luck. If you switch to nstreme on wirelss don't you get up to 3ms latency per link without any aggregation ?

Im' sorry... Lucky man NStreme? Of course you get up to 3ms latency as well as on empty 802.11a. Problem starts when the link became slightly unstable and change its data rate for example from 54Mbps to 12Mbps due to interference with other networks. Since the networks never use their links at their full potential and send less data through it, the interference will not be noticable (the chance of collision is small) until the evening comes and the data flow increases (and therefore there is bigger chance of data frame collision).

So you are expecting your link (nstreme or 802.11 or even NV2) to give you lets say 20Mbps. That woks during the day. But then as the interference comes, data rate drops and you have only 8Mbps link. So if you have HTB limit set to 20Mbps, it works during the day but when link is 8Mbps, the HTB passes packets to the hardware queue and it gets full. The link is doing its 8Mbps, but the latency there rises to as much as 150ms. And that is a pretty big problem. If you set HTB limit to 8Mbit, you are wasting bandwidth during the day. So what is the way out?

I know ethernet links have their problems too and if they have them, they are maybe even worse than wireless. But I think the HTB/queue/QoS setting is far not that hard, since you know everytime your link is doing 100Mbps.

A simple way is create a script which checks the throughput of link each x seconds and changes the HTB limit according to it .

another way is http://forum.mikrotik.com/viewtopic.php ... nt#p139817

A simple way is create a script which checks the throughput of link each x seconds and changes the HTB limit according to it .

Wouldn't that write into flash everytime? I think the flash has limited write count. I wouldn't like to damage my boards in one month But yes, that could be a way out. I will try to work around a little bit.

another way is viewtopic.php?f=2&t=28837&p=139817&hilit=percent#p139817

What is that way? I haven't figured that out from that thread...? Percent base shaping?

I think with HTB you can’t control the real-time speed.

If you want CoDel support you should use OpenWrt instead of Mikrotik OS on your Routerboard, it works great!

Yeah man. Please share some test results though.

1. OpenWRT it needs to be Routing mode instead of bridging
2. have you compared to latest RouterOS with hardware-only-queue set for the interface ?

OpenWRT/latest (has Codel) sounds REALLY COOL and totally awesome - everyone should try it

Does anyone have MirkoTik's official plans on Codel/their own better queue? I haven't e-mailed them yet about this...

OndrejSkipala, can you find a solution to lower the latency of your setup to a usable level in my WiKi article: http://wiki.mikrotik.com/wiki/NetworkPr ... of_Service ?

Regarding Codel support please post/or follow this thread guys: http://forum.mikrotik.com/viewtopic.php?f=2&t=63594