Interface HTB - hidden buffer

RouterOS Wireless Networking
1

Hi, I have made some large investigation around HTB and I found a weird thing that I would like to discuss here:

The configuration is this:

  • I have RB433 with one miniPCI card (R52H), wireless interface set to ap-bridge and one client (RB711) connected to it.
  • I have two firewall mark rules: chain output → mark paket “OutputMark” and chain prerouting → mark paket “OutputMark”
  • I have one queue in Queue tree called WLan1Queue, type is PCQ (classifier all four checkboxes - src port, src address, dst port, dst address), limit-at=1 max-limit=100M, priority=8

Now the problem:

  • I start ping from AP to the client (300ms interval, 50bytes), it’s avg. 1ms
  • Then I start bandwidth test (send UDP to client, no limit)
  • As bandwidth test reaches capacity of the wireless link (in a few seconds), the problem appears:

State:

  • There are 3 PCQ queues created (according to the classifier - one for ping, one for bandwidth test and one for bandwidth test control - that is what you can see from Torch tool)
  • So the outgoing traffic is pretty divided and the round robin algorithm works well
  • The problem is, that ping to client shows avg. 62ms

Also:

  • Wireless link is without any interference, it is indoors, bandwidth test shows almost 30Mbps, ccq is 95-100, data-rate 54Mbps all the time.

  • I was thinking that this is still some weird wireless problem, so I connected these devices using UTP cable (ether1 to ether1) and tried the same test, but with Max-limit=200M (so that HTB does not limit the traffic). Similar thing occured:
    The ping was 128ms.

  • I also tried to switch to NStreme, 10Mb Ethernet, 802.11N, used different wireless cards, everything behave the same, except the latency and available bandwidth - it varies from interface “type”.

  • If you set Max-limit lower than the capacity of the link (either wireless or wired), then the HTB will provide limitations and no delay will occur.

  • I have read HTB manual hundred times and the interesting sentence is this (refman 3.0):

When there is a possibility to send out a packet, HTB queries all its self slots in order of priority, starting with highest priority on the lowest level, till lowest priority on highest level.

Please note this part: “When there is a possibility to send out a packet…”
What exactly does this mean? Well, I thought that the interface (let’s say wireless card) can say to the OS that it is able to send. Or OS can ask the wireless card “Are you able to send?” and it would reply “Yes”. So, if it worked that way, there couldn’t be any delay! Where would it come from? If the card is able to send 30Mbps (using 1500 UDP pakets) then it means 2500 packets per second, resulting in 0.4ms delay (one way). But constant 62ms delay?
And the ethernet interface, 97Mbps of UDP traffic = 8000 pps = 0.125ms per packet. But 128ms constant delay?

So it seams that THERE IS A HIDDEN BUFFER (FIFO) on each real interface!

And it’s size can be calculated:
802.11 interface: 155 packets (1500 bytes)
NStreme interface: 73 packets (1500 bytes)
Ethernet interface: 1000 packets (1500 bytes)


This presents a huge problem for wireless links! For 100Mb Ethernet you can easily set up an HTB limitation to 95Mbps and everything will work fine (it’s full duplex and the data rate is fixed). But if you have a wireless link, you never know how much bandwidth you have available. So if you set limitations to HTB (say 20Mbps), it will work perfectly if you have at least 20Mbps available, but what will happen if an interference occures (whether from other wireless protocols, or “interference” from 802.11 network operating on the same frequency)? HTB will not limit the traffic and try to put it into real interface through the “hidden buffer” and as the bandwidth loweres, the ping increases, since it takes more time now to transmit these 155 packets (for 802.11a).

Anybody has come to this problem? Why is there this buffer and what determines it size? Can it be set in ROS? (for example to 10 pakets instead of 155). Thank you for your answers.

2

Allright, I’ve looked into this book: Cisco - routing and switching (http://www.ciscopress.com/bookstore/product.asp?isbn=1587201968) and I found a page where they say that devices use also a hardware queue, and the description fits exactly. It is FIFO and packets (data frames) from software queues go into them (so if HTB output exceeds posibilities of wireless link, packets start to queue in a hardware queue). Well, although there is certainly a reason for that, Cisco at least lets user to change the size of hardware queue, so user can adjust things to his needs.

Is Mikrotik also able to change the hardware queue size? I believe RouterOS does that somehow, because the queue seems to be shorter for NStreme links than for 802.11a links. Is there some command to do this?

3

What your experiencing is Bufferbloat (What is bufferbloat anyway?). It would be nice if ROS would let you modify the hardware TX queues, which is possible in Linux, so people can attempt to reduce bufferbloat in ROS devices.

Developers?

4

Well, after reading and listening to Jim Gettys, it actually is a kind of bufferbloat. However what Jim Gettys was talking about are unmanaged queues (FIFO) which are too big (unnecesarily). If you use HTB with one output queue as I did and the type of the queue will be FIFO, size let say 1000 packets, then of course you will have a bufferbloat there. Packets will be dropped only after the whole 1000 packet queue will fill and the queue will remain full (supposing there are enough flows to fill them). The bufferbloat will occur in the HTB queues.

But what I’m talking about is a kind of hardware queue, which is after the HTB queues. Even if you use FIFO in HTB and set it’s size to 1 packet (dont ever set it to zero, or you will be screwed :smiley:), the packet will then go to the hardware queue. The size of hardware queue for 802.11a is approximately (according to my measurements) 155 packets. And there is the problem - if the packets from HTB queue go into the hardware queue faster than the physical interface is able to send them, the hardware queue will fill and the bufferbloat will occur there.

What is interesting about this hardware queue is that it never drops packets! It just fills up, stays full. RouterOS does not put any packets into it until some packet is send from the physical interface and therefore makes a space for another packet in the hardware queue.

So if you set a speed limit in HTB that is lower than the speed of physical interface, the hardware buffer will always contain only one packet (unless the RouterOS sends packets into hardware queue in chunks which I doubt). And you can easily avoid bufferbloat in your software queues (for example using short PFIFO, RED or using properly set PCQ).

However if you set speed limit in HTB higher than the speed of physical interface, then the hardware queue will fill and bufferbloat will occur, no matter what software queue you use. This is not the problem of ethernet links, since you know exactly what the physical speed is, but is a big problem for wireless links where you don’t know the exact bandwidth, especially when there is an interference with another wireless networks. The available bandwidth can change dynamically.

So what you can do is to set the speed limit in HTB to very low level - the minimum speed you know you will get from the link no matter what - and avoid the bufferbloat in hardware queue, which will cost you a lot, since the guaranteed bandwidth will be very low. Or you can set the speed limit high and experience bufferbloat in hardware queue.

This is a real pain in the ass. It seems that you actually cannot do speed limitation and QoS for wireless links on RouterOS that would really work. Or is there a way to change the size of those buffers?

5

I understand you are experiencing delays due to hardware queues. This is still bufferbloat. Lowering the size of the hardware TX queues is a solution for Ethernet interfaces.

The bufferbloat devs are hard at work on various 802.11 drivers to help reduce the effect of bufferbloat on those types of interfaces.

6

This is still bufferbloat.

Yes, I agree.

Lowering the size of the hardware TX queues is a solution for Ethernet interfaces.

I think that the right solution is queue management where the bottleneck occurs - use of software queues with minimal hardware queue (1 packet would be the best). Queue sizes at network devices where the bottleneck never occurs are irrelevant.

The problem of bufferbloat occurs only on devices which are the bottleneck, that I would like to emphasise. So if you know where the bottleneck wil occur, you surely have to implement software queue management there (for example at your Internet gateway if you are an ISP, or at the AP, at some slow backbone link etc.). Just lowering hardware queue sizes won’t solve that problem well. In fact hw queues are FIFO and therefore are not fair to all traffic flows (some of them may starve). So I think hardware developers can’t do anything about it. Maybe keep hardware queue size low for “not so inteligent” devices that don’t use software queues - like home wifi routers, whose owners experience congestion on them.

Lowering hardware queue size would solve a bufferbloat problem, but would surely not solve congestion problem (at least not in a good way), right?

7

Right. I was talking about about lowering tx queue length as being the solution the specific problem you’re seeing with packets being queued below ROS when the interface is saturated.

I agree other algorithms are still needed at layer 3 to signal early network congestion (RED w/ ECN, SFB, etc).

8

Still no change in RouterOS 5.6. Seems like wireless QoS is really not a point of interest? Just a single parameter would be nice to add: hardware-queue-size.

9

I have been struggling allot with qos over wifi and in my opinion htb is a wast of time. It relays on the principle that you know the Max-limit, and this is unpredictable in a wireless environment.

On the meanwhile I’m having very good results with NV2 qos. My topology is (bridge<->station bridge) with NV2, queue count=4 and qos= frame priority. The data is going inside vlans. Just one bridge filter rule to extract priority from vlan and set it to the right queue inside NV2.

I hope this helps.

Kindly regards,

10

I think you need to examine this:
http://www.tiktube.com/index.php?video=mJeK3iHGhLKLIKImpnCsFrHvnlIomlpG=

I think the main confusion will be resolved by this:
(slide 28): “HTB priority doesn’t rearrange packet sequence – it doesn’t put some packets before others”

Every interface have a queue type on them after all HTBs - check /queue interface menu

11

On the meanwhile I’m having very good results with NV2 qos. My topology is (bridge<->station bridge) with NV2, queue count=4 and qos= frame priority. The data is going inside vlans. Just one bridge filter rule to extract priority from vlan and set it to the right queue inside NV2.

Well, when using 802.11a you can use WMM to split outgoing traffic int o 4 hardware queues. But this still does not solve the problem of too big hardware queue. How big the queue is when using NV2? I mean if you run ping to the other side and also run Bandwidth UDP test (only one way), which exceedess maximum bandwidth of the link, how big the latency is? and what is the real bandwidth?

12

I dont see any new information in it…

That is not entirely truth. HTB does put some packets before others if it is set so. If you split traffic to two queues with different priority, then at some circumstances (e.g. max-limit or parent reached) packets from queue with higher priority (lower number) will get to the hardware queue before packets from the other queue.

If you create a queue in HTB and set its parent as a real interface, then the setting in /queue interface does no longer apply. /queue interface applies only if no queue tree is set or if you set parent interface as “global-out” instead of a real interface.


Anyway, this problem exists even without any HTB (tree queue). If you use only /queue interface then still packets that pass these queues will get into hardware queue. Set for example interface queue type as PCQ and see results. Run some terrible UDP traffic throught the router (one port) and try to ping to other side. Ping should be different PCQ queue (dynamically created). It really is. But as your hardware queue gets full, all ICMP ping packets that leave PCQ queue will have to wait in the hardware queue and therefore you will see increased latency.

13

Maybe because of queue types, but not because of the priority. Priority is just a option that indicates when to start dropping packets

What is your CPU load when this happen? Hardware queue in this case is also known as interface driver and have little to do with RouterOS QoS. That is why you need to place your limits in QoS as close as possible to realistic amounts of traffic that you have available. (putting 200Mbps limit on 100Mbps interface is illogical)

14

WMM if different from NV2 qos.

While wmm is for accessing the media first (reducing the contention window), NV2 qos has specific queues, and priority will let you choose which data goes to each queue. High priority queue will be transmitted first.

Please take a look at the NV2 qos mechanism: http://wiki.mikrotik.com/wiki/Manual:Nv2#QoS_in_Nv2_network

RTT in NV2 varies allot, but it my tests i cannot see the hardware queue effect, and the priority works well.

My pings remain in 5~7ms while the link is saturated.

By the way, in v5.8 the default Ethernet interface queue has changed, now is only hardware queue “none”.

Kindly regards,

15

Maybe because of queue types, but not because of the priority. Priority is just a option that indicates when to start dropping packets

No…

  1. HTB itself does not drop any packet!
  2. Packets can be dropped only in queue. (maybe there is a special case when packets are dropped if CPU cant handle it)
  3. HTB does rearange packets according to priority if it is set so and such a situation happens

Let me explain how it works - each end class in HTB has it’s max-limit value. If packets that fit to a certain queue arrive to router faster than max-limit value (in a certain interval), then they are put in queue attached to this class. If the queue is for example FIFO, then there can be packet drop, if such packet does not fit into the queue (queue is full).

Every leaf class of HTB has this queue. Whenever there is possibility to send packet (parent classes allow this, hardware queue is empty, max-limit is not reached), the HTB takes some packets from this queue and puts them into hardware queue.

If you have more of these HTB classes, the decision of how many packets should be taken from which class is dependent on priority and the state of the class.

Lets make an example:
Suppose you have 2 sources generating UDP traffic that should go through the router, each 700kbps. When these packets arrive at the router, they should be sent throught one output interface. Lets add a root class to queue tree called Upload, parent=output interface, limit-at=100k, max-limit=1Mb. Then let’s add two subclasses, each for its source generator. Class A will get limit-at=50k, max-limit=1M, priority=8. Class B will get limit-at=50k, max-limit=1M, priority=7.

With this setting you will see that Class A gets 300kbps and Class B gets 700kbps on the output. Packets that arrive to Class A end up in a queue for Class A, because they cant be sent (of course there will be packet loss after the queue fets full).

However what I’m trying to say is that packets from data generator from ClassB will “overtake” packets from generator for Class A, because they will hang in the queue attached to Class A. So there will be packet reordering by HTB. Agree? And all packets that will be dropped will be dropped only because they cant fit into queue (queue is full), agree?

16

WMM if different from NV2 qos.

While wmm is for accessing the media first (reducing the contention window), NV2 qos has specific queues, and priority will let you choose which data goes to each queue. High priority queue will be transmitted first.

Yes, I agree. WMM is different. It does what you say, but also it has additional effect. If you fully saturate hardware queue for Normal priority, then the queues for Video and Voice will be still empty and you can send for example ping packets through them and see minimal delay. As well it works if the Video queue is full and you have empty Normal priority queue - if you send packet throught the normal priority queue you will see minimal delay. So you can “prioritize” packets to Normal queue instead of Video or Voice queue.


RTT in NV2 varies allot, but it my tests i cannot see the hardware queue effect, and the priority works well.

My pings remain in 5~7ms while the link is saturated.

If you saturate the link fully (using UDP), then you have to set some congestion mechanism anyway. If you saturate one NV2 queue fully and use other for pings, then I belive you get nice pings. But what about if you send ping throught the saturated queue? Then of course you get very bad pings.

What I mean is that I want for example to use PCQ for these cases. If a link is saturated, then lets split the speed among all users. Dont leave anybody starving or having high pings just because of anybody else. My vision of QoS is not about types of traffic, but about providing quallity to all users regardless what other users in the network are doing. That’s why 8 queues in NV2 does not help to solve my problem. PCQ solves it very well, but then there is the problem with the hardware queue…

Do you use any specific cogestion mechanism? Like special queue, HTB or something?

By the way, in v5.8 the default Ethernet interface queue has changed, now is only hardware queue “none”.

I see that, just updated routers last week… My settings remained ethernet-default, PFIFO 50packets… but I can see that I can set queue type None… Will look at that closer, maybe its something usefull. Thank you.

17

What is your CPU load when this happen? Hardware queue in this case is also known as interface driver and have little to do with RouterOS QoS. That is why you need to place your limits in QoS as close as possible to realistic amounts of traffic that you have available. (putting 200Mbps limit on 100Mbps interface is illogical)

CPU load can be as low as 20%. Well, I really think that hardware queue has much to do with any QoS. Especially in wireless networking. In wireless networking it is hard to tell what is “realistic amount of traffic”, because you never know how much bandwidth you have. You can assume you have 1Mbps on link that 99% provides 20Mbps (and 1% only 1Mbps). But that is a nonsence and waste of money, time and energy.

Putting 200Mbps limit on 100Mbps interface is illogical. But is logical to put 15Mbps limit on a wireless AP? And what about if you have 1client connected at 54Mbps and one connected at 6Mbps? How do you solve that? Will you decrease the limit to 4Mbps? I guess not.

18

This is not true, our HTB doesn’t rearange packets according to priority, it discributes available amounts of traffic according to priority, this way deciding what packets will be dropped.

So MacGaiver is right

19

Why no?? Your example just perfectly illustrated what i’m saying - it all depends on queue type. Queue type is only thing that can delay a packet and change order, if you will put queue-size=0 (theoretically) for all queue types you will be able to see, that priority have no effect on order.

20

This is not true, our HTB doesn’t rearange packets according to priority, it discributes available amounts of traffic according to priority, this way deciding what packets will be dropped.

So MacGaiver is right

Maybe it’s just about seeing it from different angle? Like “what is the HTB”? Like I say, if packets are queued by HTB in a queue due to lack of “distributed amout of traffic”, other packet arriving after them can get to the real output interface sooner (not queued in HTB queue) and therefore there will appear packet reordering. Or am I seeing this somehow still wrong?

“deciding what packets will be dropped” - HTB drops packets only if the queue for the class is full. Right? So I say “the queue drops packets” you say “HTB drops packets”, but I think we mean the same thing. Right?