LLQ required

engineertote · Sun May 13, 2012 1:12 pm

Hi

i hope to see LLQ in the coming ROS6 .

Thanks

hel · Tue May 22, 2012 7:32 am

Strict prioritization is a must have feature.

Subscribing...

Takv2011 · Wed Jul 04, 2012 11:56 pm

I agree, suscribe and hold hope!

void · Thu Jul 05, 2012 6:32 pm

Yes !!! We need this for voip !

Sanity · Thu Jul 05, 2012 8:32 pm

I agree.

I dont caer about VOIP a lot, but I have a constant stream of financial data that goes over my network that has to have absoluet priority

AsciiWolf · Sat Jul 14, 2012 6:39 pm

This is a must-have! Subscribing.

rjickity · Mon Aug 13, 2012 3:35 pm

+1 here

Mon Aug 13, 2012 3:39 pm

Voice and "financial data" prioritisation can already be achieved, I suggest you to watch some presentations by Janis Megis:
http://www.tiktube.com/video/mJeK3iHGhL ... vnlIomlpG=

LLQ is just another feature, but you can do it already.

hel · Wed Aug 15, 2012 2:54 am

normis:
Can you answer, is it possible to make qos working with pcq?

And how "LLC" is possible?

Wed Aug 15, 2012 9:08 am

If you can't do it now in HTB (currently implemented), you most probably will not be able to do it in LLQ also. Both will have degree of complexity in configuration and acheiving goals

Both have very similar set of features, in one you can do some things easier than in other, but i don't really see the need for wasting time and implement feature that is not necessary. Let MT work on multi-threading more - that is feature we really need, especially with upcoming CCR

hel · Wed Aug 15, 2012 5:32 pm

If you can't do it now in HTB (currently implemented), you most probably will not be able to do it in LLQ also.

I don't have any visible error in my QoS configs. I checked it many times, rechecked it with others qos implementations, with mikrotik wiki, and etc etc.

Even Traffic Inspector on windows provides more useful (if not ideal) qos, but with bad throughput (no more than 1000 pps).

Most better way to check if qos working ideally - ping running at lower latencies with torrents running at max speed.
If ping goes nicely, then voice, games, dns requests will do the same for sure.

In our network there is many gamers, while their os or antivirus downloading updates - their online games becoming a slideshow.

It seems that it's not only my problem.

Thu Sep 06, 2012 3:20 pm

If you can't do it now in HTB (currently implemented), you most probably will not be able to do it in LLQ also.
I don't have any visible error in my QoS configs. I checked it many times, rechecked it with others qos implementations, with mikrotik wiki, and etc etc.

Even Traffic Inspector on windows provides more useful (if not ideal) qos, but with bad throughput (no more than 1000 pps).

Most better way to check if qos working ideally - ping running at lower latencies with torrents running at max speed.
If ping goes nicely, then voice, games, dns requests will do the same for sure.

In our network there is many gamers, while their os or antivirus downloading updates - their online games becoming a slideshow.

It seems that it's not only my problem.

Well, then you and others are doing something wrong. I have no such problems. it just take a little bit of magic in mangle with packet-marks - queues are simple afterwards.

hel · Sun Sep 09, 2012 4:01 pm

Well, then you and others are doing something wrong. I have no such problems. it just take a little bit of magic in mangle with packet-marks - queues are simple afterwards.

Do you even read my post?

I don't have any visible error in my QoS configs. I checked it many times, rechecked it with others qos implementations, with mikrotik wiki, and etc etc.

Mon Sep 10, 2012 10:21 am

post your config please. it may look correct to you, but it may be in fact wrong.

hel · Mon Sep 10, 2012 8:19 pm

My current config (I've cut it for better reading):

ros code

# jan/06/1970 05:42:18 by RouterOS 5.20
#
/queue tree
add max-limit=9400k name=bandwidth-control parent=global-out
add limit-at=8M max-limit=9400k name=users-down parent=bandwidth-control
add limit-at=1400k max-limit=9400k name=users-up parent=bandwidth-control
add max-limit=9400k name=qos-total packet-mark="" parent=global-in
add limit-at=8M max-limit=9400k name=qos-down parent=qos-total
add limit-at=1400k max-limit=9400k name=qos-up parent=qos-total
add limit-at=2M max-limit=9400k name=qos-http-down packet-mark=qos-http-down parent=qos-down priority=2
add max-limit=9400k name=qos-other-down packet-mark=qos-other-down parent=qos-down
add limit-at=256k max-limit=9400k name=qos-http-up packet-mark=qos-http-up parent=qos-up priority=2
add max-limit=9400k name=qos-other-up packet-mark=qos-other-up parent=qos-up
add limit-at=1M max-limit=9400k name=qos-cs16-down packet-mark=qos-cs16-down parent=qos-down priority=1
add limit-at=1M max-limit=9400k name=qos-cs16-up packet-mark=qos-cs16-up parent=qos-up priority=1
/queue type
set 0 pfifo-limit=45
add kind=pcq name=pcq-unlim256-down pcq-burst-time=5s pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=512k pcq-src-address6-mask=64
add kind=pcq name=pcq-unlim256-up pcq-burst-time=5s pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-limit=30 pcq-rate=192k pcq-src-address6-mask=64
add kind=pcq name=pcq-unlim512-down pcq-burst-time=5s pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=1024k pcq-src-address6-mask=64
add kind=pcq name=pcq-unlim512-up pcq-burst-time=5s pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=256k pcq-src-address6-mask=64
add kind=pcq name=pcq-workers-down pcq-burst-time=4s pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-limit=30 pcq-rate=4M pcq-src-address6-mask=64 pcq-total-limit=1000
add kind=pcq name=pcq-workers-up pcq-burst-time=4s pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-limit=30 pcq-rate=6M pcq-src-address6-mask=64 pcq-total-limit=1000
add kind=pcq name=pcq-unlim1024-up pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=512k pcq-src-address6-mask=64
add kind=pcq name=pcq-unlim1024-down pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=2048k pcq-src-address6-mask=64
/queue tree
add max-limit=9400k name=unlim256-down packet-mark=unlim256_traffic_down parent=users-down queue=pcq-unlim256-down
add max-limit=9400k name=unlim512-down packet-mark=unlim512_traffic_down parent=users-down queue=pcq-unlim512-down
add max-limit=9400k name=workers-down packet-mark=workers_traffic_down parent=users-down queue=pcq-workers-down
add max-limit=9400k name=unlim256-up packet-mark=unlim256_traffic_up parent=users-up queue=pcq-unlim256-up
add max-limit=9400k name=unlim512-up packet-mark=unlim512_traffic_up parent=users-up queue=pcq-unlim512-up
add max-limit=9400k name=workers-up packet-mark=workers_traffic_up parent=users-up queue=pcq-workers-up
add max-limit=9400k name=unlim1024-down packet-mark=unlim1024-traffic-down parent=users-down queue=pcq-unlim1024-down
add max-limit=9400k name=unlim1024-up packet-mark=unlim1024-traffic-up parent=users-up queue=pcq-unlim1024-up
/ip firewall address-list
add address=00.00.49.233 comment="{-5BAD-4C96-BC65-}" list=unlim1024_clients
add address=00.00.49.132 comment="{-7387-4EDC-928B-}" list=unlim256_clients
add address=00.00.49.148 comment="{-C3CC-4FEA-BDE2-}" list=unlim256_clients
add address=00.00.49.147 comment="{-52AD-4139-9758-}" list=unlim256_clients
add address=00.00.49.199 comment="{-A7C0-4C09-A870-}" list=unlim512_clients
add address=00.00.49.20 comment="{-BD9D-4BE5-8E14-}" list=unlim256_clients
add address=00.00.49.186 comment="{-CCDD-4C5E-83D1-}" list=unlim256_clients
add address=00.00.49.37 comment="{-455C-490F-9992-}" list=unlim256_clients
add address=00.00.49.26 comment="{-7B2B-4678-B26F-}" list=workers_clients
add address=00.00.49.134 comment="{-7B82-4E33-A612-}" list=unlim256_clients
add address=00.00.49.188 comment="{-46B3-4A1A-933F-}" list=unlim256_clients
/ip firewall filter
add chain=input protocol=icmp
add chain=input connection-state=established in-interface=WAN
add chain=input connection-state=related in-interface=WAN
add chain=forward comment="Authorize users" out-interface=WAN src-address-list=unlim256_clients
add chain=forward out-interface=WAN src-address-list=unlim512_clients
add chain=forward out-interface=WAN src-address-list=unlim1024_clients
add chain=forward out-interface=WAN src-address-list=workers_clients
add action=drop chain=forward out-interface=WAN
add chain=forward dst-address-list=unlim256_clients in-interface=WAN
add chain=forward dst-address-list=unlim512_clients in-interface=WAN
add chain=forward dst-address-list=unlim1024_clients in-interface=WAN
add chain=forward dst-address-list=workers_clients in-interface=WAN
add action=drop chain=forward in-interface=WAN
add action=drop chain=input comment="Block all another traffic from internet" in-interface=WAN
/ip firewall mangle
add action=mark-packet chain=forward comment=unlim256 dst-address-list=unlim256_clients in-interface=WAN new-packet-mark=unlim256_traffic_down out-interface=LAN passthrough=no src-address-list=!local_subnet
add action=mark-packet chain=forward dst-address-list=!local_subnet in-interface=LAN new-packet-mark=unlim256_traffic_up out-interface=WAN passthrough=no src-address-list=unlim256_clients
add action=mark-packet chain=forward comment=unlim512 dst-address-list=unlim512_clients in-interface=WAN new-packet-mark=unlim512_traffic_down out-interface=LAN passthrough=no src-address-list=!local_subnet
add action=mark-packet chain=forward dst-address-list=!local_subnet in-interface=LAN new-packet-mark=unlim512_traffic_up out-interface=WAN passthrough=no src-address-list=unlim512_clients
add action=mark-packet chain=forward comment=unlim1024 dst-address-list=unlim1024_clients in-interface=WAN new-packet-mark=unlim1024-traffic-down out-interface=LAN passthrough=no src-address-list=!local_subnet
add action=mark-packet chain=forward dst-address-list=!local_subnet in-interface=LAN new-packet-mark=unlim1024-traffic-up out-interface=WAN passthrough=no src-address-list=unlim1024_clients
add action=mark-packet chain=forward comment=workers dst-address-list=workers_clients in-interface=WAN new-packet-mark=workers_traffic_down out-interface=LAN passthrough=no src-address-list=!local_subnet
add action=mark-packet chain=forward dst-address-list=!local_subnet in-interface=LAN new-packet-mark=workers_traffic_up out-interface=WAN passthrough=no src-address-list=workers_clients
add action=mark-packet chain=prerouting comment=QoS-HTTP dst-address-list=local_subnet in-interface=WAN new-packet-mark=qos-http-down passthrough=no protocol=tcp src-address-list=!local_subnet src-port=80
add action=mark-packet chain=prerouting dst-address-list=!local_subnet dst-port=80 in-interface=LAN new-packet-mark=qos-http-up passthrough=no protocol=tcp src-address-list=local_subnet
add action=mark-packet chain=prerouting comment=QoS-CS1.6 dst-address-list=local_subnet in-interface=WAN new-packet-mark=qos-cs16-down passthrough=no protocol=tcp src-address-list=!local_subnet src-port=27000-27150
add action=mark-packet chain=prerouting dst-address-list=!local_subnet dst-port=27000-27150 in-interface=LAN new-packet-mark=qos-cs16-up passthrough=no protocol=tcp src-address-list=local_subnet
add action=mark-packet chain=prerouting comment=QoS-Other dst-address-list=local_subnet in-interface=WAN new-packet-mark=qos-other-down passthrough=no src-address-list=!local_subnet
add action=mark-packet chain=prerouting dst-address-list=!local_subnet in-interface=LAN new-packet-mark=qos-other-up passthrough=no src-address-list=local_subnet

What can I say... I've posted similiar configs on the forum, and no one found any problems.
Using this config http packets can't take over the torrents even if http's limit-at is high and its priority is highest.

hel · Tue Sep 11, 2012 4:02 pm

Is the silence a proof that my config correct?

Tue Sep 11, 2012 6:31 pm

Is the silence a proof that my config correct?

I think, no. Not at all.

I have not fully analyzed the config snippet you've posted since it's quite time-consuming task, but I do think there's something I should point you at.

From the snippet you've posted it is not cleat what local_subnet address list is. I assume that all entries of the unlim256_clients, unlim512_clients, unlim1024_clients and workers_clients address lists belong to the local_subnet as well. Please correct me if I'm wrong.

Then as far as I understand any packet can be marked with only a single packet mark at a time, so any time you mark the packet with a new packet mark the previous packet mark gets lost.

Now looking at the Packet Flow Diagram you can see that Mangle Prerouting happens before Mangle Forward and thus all your 'qos-*' packet marks get overwritten in Mangle Forward. And specifying 'passthrough=no' in one chain does not make any difference for the rules from another chain.

hel · Tue Sep 11, 2012 8:03 pm

From the snippet you've posted it is not cleat what local_subnet address list is. I assume that all entries of the unlim256_clients, unlim512_clients, unlim1024_clients and workers_clients address lists belong to the local_subnet as well. Please correct me if I'm wrong.

Yes, all these clients belong to the local_subnet.

Then as far as I understand any packet can be marked with only a single packet mark at a time, so any time you mark the packet with a new packet mark the previous packet mark gets lost.

Why do you think that the previous packet mark gets lost? I don't get your point, please explain.

Mangle Forward and thus all your 'qos-*' packet marks get overwritten in Mangle Forward.

By what? Why?

And specifying 'passthrough=no' in one chain does not make any difference for the rules from another chain.

I don't understand your point there either. passthrough=no used to ignore all next mangle rules for this marked packets in a current chain.

Tue Sep 11, 2012 10:33 pm

Then as far as I understand any packet can be marked with only a single packet mark at a time, so any time you mark the packet with a new packet mark the previous packet mark gets lost.
Why do you think that the previous packet mark gets lost? I don't get your point, please explain.

That's how it works. I did not find it documented anywhere, so I found it out by experiment. As soon as you mark a packet with a new packet mark, previous packet mark (if any) gets discarded.

Mangle Forward and thus all your 'qos-*' packet marks get overwritten in Mangle Forward.
By what? Why?

Lets take, for example, the 0.0.49.20 IP address from your config. It falls into the unlim256_clients address list as well as into the local_subnet address list. When packet from 0.0.49.20 to (say) 1.2.3.4 port 80/tcp passes you router it first gets into the prerouting mangle chain, hits the rule

add action=mark-packet chain=prerouting comment=QoS-HTTP dst-address-list=local_subnet in-interface=WAN new-packet-mark=qos-http-down passthrough=no protocol=tcp src-address-list=!local_subnet src-port=80

at which point the qos-http-down packet mark is assigned to the packet. But while being further processed by the router the packet then gets into the forward mangle chain, where it hits the following rule

add action=mark-packet chain=forward comment=unlim256 dst-address-list=unlim256_clients in-interface=WAN new-packet-mark=unlim256_traffic_down out-interface=LAN passthrough=no src-address-list=!local_subnet

and at this point the unlim256_traffic_down is assigned to the packet while the previous packet mark (qos-http-down) is discarded.

hel · Wed Sep 12, 2012 12:25 am

Lets take, for example, the 0.0.49.20 IP address from your config. It falls into the unlim256_clients address list as well as into the local_subnet address list. When packet from 0.0.49.20 to (say) 1.2.3.4 port 80/tcp passes you router it first gets into the prerouting mangle chain, hits the rule
Code: Select all
add action=mark-packet chain=prerouting comment=QoS-HTTP dst-address-list=local_subnet in-interface=WAN new-packet-mark=qos-http-down passthrough=no protocol=tcp src-address-list=!local_subnet src-port=80
at which point the qos-http-down packet mark is assigned to the packet. But while being further processed by the router the packet then gets into the forward mangle chain, where it hits the following rule
Code: Select all
add action=mark-packet chain=forward comment=unlim256 dst-address-list=unlim256_clients in-interface=WAN new-packet-mark=unlim256_traffic_down out-interface=LAN passthrough=no src-address-list=!local_subnet
and at this point the unlim256_traffic_down is assigned to the packet while the previous packet mark (qos-http-down) is discarded.

First packets gets caught by mangle prerouting, 'passthrough=no' sends packet directly to HTB, where it get prioritizes in global-in queue. Next 'shaped unmarked packets' gets caught by mangle forward (or postrouting), after that it goes into HTB where they get shaped according to user bandwidth limit in the global-out queue.

After shaping these packets will not being qos'ed, because qos queues is in global-in, as packet flow diagram says it isn't after global-out for sure.

Wed Sep 12, 2012 9:20 am

1) do you use src-nat (private IP addresses for clients)? - if yes, prerouting is not aware of them.
2) Do you have all the clients in the same IP network subnet? - if no, what happens when one client from one local subnet is communicated with other client in other local subnet?
3) Are you aware that "priority" does not rearrange packet sequence - it just decide what packets to drop. So packets come out from global-in in the same order as they came in. (only some of them might be dropped by global-in)

Regarding the setup itself:
1) use a lot more of "mark-connection", and use "mark packet" only with connection-mark matcher - this will increase performance a lot as main checks will be done only once per connection, not per every packet.
2) you have relatively small amount of possible marks 3x4=12. make all marking in the same place and have 12 different marks (without remarking every packet 2x. Then in queues you can apply multiple marks to the same queue.

next time also describe what doesn't work in queues - you can't see traffic or your latency to CS1,6 servers are high. etc.

Wed Sep 12, 2012 10:17 am

First packets gets caught by mangle prerouting, 'passthrough=no' sends packet directly to HTB

Well, you are right here, the packet gets into the global-in HTB before it reaches Forward.

where it get prioritizes in global-in queue.

Wrong. In the definition of a queue 'priority' has nothing to do with prioritization of packets- it prioritizes child queues in case there's not enough bandwidth to satisfy limit-at for all of them.

Next 'shaped unmarked packets' gets caught by mangle forward (or postrouting)

Wrong. Actually all packets gets caught by your mangle forward rules, regardless of whether they are marked or not. You have to specify 'packet-mark=no-mark' in a rule to match unmarked packets only.

after that it goes into HTB where they get shaped according to user bandwidth limit in the global-out queue. After shaping these packets will not being qos'ed, because qos queues is in global-in, as packet flow diagram says it isn't after global-out for sure.

That's right. So, what do you expect to achieve?

hel · Wed Sep 12, 2012 10:31 am

1) do you use src-nat (private IP addresses for clients)? - if yes, prerouting is not aware of them.

I'm not using any kind of nat.

I'm also not using wifi at all.

2) Do you have all the clients in the same IP network subnet? - if no, what happens when one client from one local subnet is communicated with other client in other local subnet?

In case of client communication with another client - mangle rules doesn't capture packets on these connections.

3) Are you aware that "priority" does not rearrange packet sequence - it just decide what packets to drop. So packets come out from global-in in the same order as they came in. (only some of them might be dropped by global-in)

Sure. Even in that case, for example - http pages loads very slowly.

Regarding the setup itself:
1) use a lot more of "mark-connection", and use "mark packet" only with connection-mark matcher - this will increase performance a lot as main checks will be done only once per connection, not per every packet.
2) you have relatively small amount of possible marks 3x4=12. make all marking in the same place and have 12 different marks (without remarking every packet 2x. Then in queues you can apply multiple marks to the same queue.

I does make connection marking before. With absolutely no changes on the problem.
On the 2nd tip please explain.

next time also describe what doesn't work in queues - you can't see traffic or your latency to CS1,6 servers are high. etc.

If not CS1.6, which is a good indicator of low latency (voip have very same latencies).
Let's see. For example my client have 512kbit line. He run torrent at no limit. Pings goes up to 500-800ms (from 33ms in normal). Then he wants to open some page (after browser cache clear), and page loads fully only in 8 seconds. He stop torrents and web page (after browser cache clear) loads in 1-2 seconds.

Torrent client shows that speed did not fall below 100kbytes/sec (at full speed it shows 119,9kbytes/sec), so, only 19kbytes for web page? [sarcastic]Good QoS indeed.[/sarcastic]

Not only priority - limit-at also not working. Even 2 mbit limit-at line for http not helping to get over torrents, which haven't any limit-at.

Wed Sep 12, 2012 10:42 am

Let's see. For example my client have 512kbit line. He run torrent at no limit. Pings goes up to 500-800ms (from 33ms in normal). Then he wants to open some page (after browser cache clear), and page loads fully only in 8 seconds. He stop torrents and web page (after browser cache clear) loads in 1-2 seconds.

Well, that's expected behavior with your current setup.

hel · Wed Sep 12, 2012 10:47 am

That's right. So, what do you expect to achieve?

I watched presentations, forums and from them I understand that mikrotik is a powerful thing for QoS and bandwidth limitation. VOIP, games - they can be prioritized to make users feel more comfortable.

And now I'm feeling that I've been fooled.

Now I'm just trying to say, that strict prioritization is needed.
Seems my configs are correct anyway.

So I vote for LLQ.

Well, that's expected behavior with your current setup.

Could you point where is a problem in my config or at least give some hints?

Wed Sep 12, 2012 4:11 pm

you need to analyze it step by step.
Lets take a look at queues and imagine that we have only one very active customer that without shaping would use
a)10Mbps for torrents
b)200kbps for cs1.6 -
c)100kbps for ping something
of download.

1) mangle prerouting you mark everything into streams a) b) c)
2) in global-in you apply your limits and a) and b) goes through without loses, but c) is shaped to 7700kbps (if upload uses its part of traffic)
3) mangle forward you remark all traffic to a single mark D) cause it is all from single customer
4) in global-out all that traffic that got out from global-in is fighting for single PCQ rate-limit (256-1024kbps) as it is PCQ substream in effect is FIFO queue - more aggressive traffic (with more pps) will win. - results in bigger ping times and bigger latency for games

Your setup is made on wrong assumption that a) and b) traffic left global-in "first" and that global-out will drop only "last" c) traffic.
Priority is used only for drop decision inside specific HTB, packets are leaving in same order they entered (except those that was dropped)

Suggestions:
1) for PCQ use out-interface HTBs not global-out - have separate queue tree for upload and download
2) If you use one max-limit for both (upload and download) (9400k) in global-in there is no point of mark upload and download separately, you are only shooting yourself in the leg.
3) apply a PCQ with src-address.dst-address classifier with pcq-rate=0 on "other-up" and "other-down" queues.
4) P2P traffic can pretend to be HTTP - i suggest to prioritize only first few Mb of any port 80 connection (use "connection-bytes" option in mangle)

Wed Sep 12, 2012 4:31 pm

Ok, hel, I see what you mean. Prioritizing traffic withing a single PCQ-substream does not seem to be possible. The macgaiver's suggestions should work, I think, but, well, they feel more like a workarounds then like a real solution.

hel · Fri Sep 14, 2012 10:52 am

Suggestions:

Yeah, it's not helping on <100% load link, which I expected per-pcq-subqueue QoS, as andriys correctly noted.

And yes, only per-pcq subqueue QoS can make dual shaper working as it should.

Mikrotik priority and limit-at working as it should only on dedicated link without user bandwidth limitation.
There is no per-user prioritization.

But if not, why so many people states, that QoS there is as good that it improves VOIP and games connections in PCQ+QoS scheme?
It's definitelly not, in the current logic of mikrotik's prioritization.

Or, perhaps, they doing it with some magic.

cmoegele · Tue Sep 18, 2012 7:55 pm

And now I'm feeling that I've been fooled.

So I vote for LLQ.

+1 for LLQ

Since December I´m trying many different QOS setups on many different RB Hardware in order to get Prio1 for gaming, even took professional help , but with no satisfying result,.....
Now I think at the moment there seems to be no real solution for low latency from RouterOS!?
Or is there any Guru which can show us the Golden QOS ?

best regards

cm

PS: since PM is not working you can contact me via skype ( cmoegele)

Wed Sep 19, 2012 2:54 pm

OK, could any of "LLQ needers" can link me some doc or standard, that shows that you can get out of it something different than HTB? I spend some time and examined some Cisco documents, biggest change that i can see is use of percentage instead of bandwidth units. And percentage of what - of interface speed, so if you have 10Mbps up/down connection on Gigabit ethernet - it is useless.

NetworkPro · Mon Sep 24, 2012 10:59 am

LLQ is a marketing word nowadays, that Ethernet best-effort is so awesome.

If you experience some problem it probably is elsewhere - for example - upstream.

Bang!

hel · Mon Sep 24, 2012 1:33 pm

LLQ does some basic packet rearrange, low latency packets sends first, and other packets sends after, getting shaped as non-LLQ queues (as I understand - it's main difference of LLQ and PQ - in PQ there is only strict prioritization, but in LLQ there is 2 types of prioritization - low latency and fair weighted, first for high-priority traffic, and second for any other traffic)

Maybe I incorrectly understand it.

In current logic of dual shaper - it's not important what does first shaper. Second one will ruin it for sure, because it doesn't care for traffic priority anymore and will drop high priority packets with very same chance as low priority one.

Yeah, I agree, LLQ seems is not an option here.

PS: After I wrote this message, I've got an idea, how to make some kind of LLQ without LLQ, will check it before I forget.

hel · Mon Sep 24, 2012 2:01 pm

Yeah, just as I expected. Some kind of LLQ is easy to get to work in routerOS.
Torrents at full speed, pings are at lower latencies.

Not an elegant solution, but...

NetworkPro · Mon Sep 24, 2012 2:11 pm

not an elegant solution ?

Paint the router in some colour then it will be elegant for you

hel · Mon Sep 24, 2012 2:21 pm

not an elegant solution ? Paint the router in some colour then it will be elegant for you

By the way, I prioritized packets in the second shaper, by not elegant I mean that those packets doesn't get limited for the users according to their bandwidth limits. Will think more deeply, maybe I will find a solution.

NetworkPro · Mon Sep 24, 2012 2:42 pm

I understand. Have you considered Dual QoS (in one MT router) or two hop solution (use gigabit and powerful machines plz) one for client max limit and one for overall QoS selective drop?

hel · Mon Sep 24, 2012 2:47 pm

I understand. Have you considered Dual QoS (in one MT router) or two hop solution (use gigabit and powerful machines plz) one for client max limit and one for overall QoS selective drop?

I have one RB1100, so I using Dual QoS in one router.

NetworkPro · Mon Oct 08, 2012 7:43 am

Hello guys.

If we think about it in a simple way, the strict prio/LLQ will make a sub-millisecond difference some of the time. Do we really need something like this?

If you read this http://wiki.mikrotik.com/wiki/NetworkPr ... of_Service do you have any further questions?

Thank you.

Mon Oct 08, 2012 11:51 am

Hi NetworkPro,

If we think about it in a simple way, the strict prio/LLQ will make a sub-millisecond difference some of the time. Do we really need something like this?

If you read this http://wiki.mikrotik.com/wiki/NetworkPr ... of_Service do you have any further questions?

You've missed the whole point. Nobody wants LLQ as such. What people really want is a way to solve their real-life needs, and many see LLQ (i.e. packet reordering) as a possible solution.

Lets have a closer look on a problem that hel rised earlier in this topic. Let's say you have many customers and are using PCQ to limit bandwidth to each of them individually. Now lets say you want to make your customers happier by giving a higher priority to their low-volume latency-critical traffic, so they can enjoy online gaming, use VoIP services etc. at the time they use torrents that utilize all the bandwidth available to them.

There is currently no way you can do it in RouterOS. What you and Mikrotik guys are saying should work does not work. There are currently no way to prioritize traffic inside a single PCQ substream. Prioritizing traffic in global-in helps when your upstream link in congested, but it makes no difference when your client's PCQ substream in congested. And that is exactly what many people want- they want a way to prioritize traffic within a single PCQ substream.

The whole thing is about PCQ. If PCQ is not needed, everything described above is easily doable with queue trees.

If some of this is not what you believe - please e-mail me and explain. I can reply in a simple and logical way that relieves any concerns. I can also correct the article to reflect the actual truth by choosing better words. Thank you.

Ditto.

NetworkPro · Mon Oct 08, 2012 12:24 pm

Thanks for the reply.

Let's see if we can not miss each other's point so that we all get what is the truth rather than something we think we can imagine.

The VoIP/gaming/ packets arrive in between other packets as they were put on the wire by customers PC.

Before we can reorder packets in the router, we process the ones we get from the interface.

This leaves no time for putting a packet in a queue for it to wait for the next packet.

The packets leave the router before it is possible to rearrange them.

If we decide we want to delay packets and rearrange them - we will surely be damaging everything with the extra latency for all packets as well as with the actual rearrangement.

To solve real life problem customer should look at where exactly the problem is.
MirkoTik do not "own" an issue that is outside of Ethernet's domain such as latency issue upstream on legacy networks.

We have to reach an agreement on the Theory here and now, so that no other person gets infected with packet reorder virus in their head and feel bad about it.

OK?

Mon Oct 08, 2012 2:28 pm

Thanks for the reply.

Welcome.

The VoIP/gaming/ packets arrive in between other packets as they were put on the wire by customers PC.

Before we can reorder packets in the router, we process the ones we get from the interface.

This leaves no time for putting a packet in a queue for it to wait for the next packet.

The packets leave the router before it is possible to rearrange them.

Wrong. Please read what I wrote in the previous message once again. We are talking about the situation when a client's PCQ sub-stream is out of bandwidth (i.e. the corresponding sub-queue is full and client's link is effectively congested). The packets are not processed immediately in this case, but rather wait in a queue for some time so that the overall speed was lower then the configured limit, or get dropped if they are not in the queue yet, and there are no space left in the queue.

What people want is the ability to assign different priorities to the packets (not queues), so that if the queue is full of low-priority packets and a higher priority packet arrives, some low priority packets are purged from the queue to make some room where the high-priority packets can be placed.

If we decide we want to delay packets and rearrange them - we will surely be damaging everything with the extra latency for all packets

Exactly. That is what traffic shaping is all about after all. And while we add some extra latency to the packets en masse, there are the necessity sometimes to process some higher-priority packets with lower latency then all others.

We have to reach an agreement on the Theory here and now, so that no other person gets infected with packet reorder virus in their head and feel bad about it.

Let me say it once again. Packet reordering (or LLQ, or whatever whoever names it) is not an end in itself. What is required is the ability to prioritize packets within a single PCQ sub-stream. The implementation details do not really matter. Packet rearrangement is just one possible way. Adding the possibility for a PCQ sub-queue to be a parent of another (sub)queues is another possible way. I'm sure there are others.

NetworkPro · Mon Oct 08, 2012 2:46 pm

I understand your two posts now. I am able to put the PCQ queueing in slightly different perspective now - it's like "bufferbloat".

In this case we have to drop rather than queue stuff in PCQ, right ?

Let's put the priority packets in another queue which has nice limit-at setting. Is this not going to solve the problem?

Mon Oct 08, 2012 3:34 pm

I understand your two posts now. I am able to put the PCQ queueing in slightly different perspective now - it's like "bufferbloat".
In this case we have to drop rather than queue stuff in PCQ, right ?

Well, PCQ should not be radically different from an ordinary child queue. The only major difference is that PCQ creates sub-queues base on the configured aggregation criteria (usually source or destination address). Ordinary child queues do queue packets, do delay them to satisfy max-limit of itself and limit-at of the others, and do drop packets when there are no more space left in the queue.

Let's put the priority packets in another queue which has nice limit-at setting. Is this not going to solve the problem?

No, it is not. It can solve the problem of congested uplink. In can not solve problem of congested client's PCQ substream.

Let me explain once again. You have a set of customers with a particular bandwidth limits. A natural approach is to configure a couple of PCQ queues (one for upload and one for download) to enforce the limit. Now you need to allow high priority packets first while still honoring the limit, and this is what's not possible with PCQ.

NetworkPro · Mon Oct 08, 2012 3:58 pm

We drop low priority packets before they get to the PCQ to cause queueing and dropping of the high priority ones. This is the two setp QoS that Janis Megis gave us.

Mon Oct 08, 2012 4:11 pm

We drop low priority packets before they get to the PCQ to cause queueing and dropping of the high priority ones. This is the two setp QoS that Janis Megis gave us.

No, it does not work that way. And it is actually not possibly to do it that way, because only a particular PCQ sub-stream "knows" if and when a low-priority packets need to be dropped. In other words, we need to drop low-priority packets that are already queued in favor of a higher priority packets only in case PCQ sub-stream is congested, and the congestion can not be detected until we actually try to put the packet in a particular sub-stream.

NetworkPro · Mon Oct 08, 2012 4:30 pm

Wait - you mean when we put in a Rate limit in the PCQ ?
I remember back in the day when the dual QoS first came out I just put the PCQ as the queue in the first step solving this problem and shifting it to a problem of not specific user limit

.

Sorry I had forgotten about this

Thanks for bringing the attention to this problem though. It is interesting indeed.

Before the two-step with PCQ there was the massive Queue Tree ^_^

brandonrossl · Wed Oct 17, 2012 5:11 pm

My first experience with QoS was on a Dlink DGL-4300, it was upload only, but it had enough options to be dangerous.

It also overheated when applying QoS, which I had to fix with a heat-pipe chipset cooler.

It had one option that drastically improved performance of small-packet outgoing traffic, packet fragmentation.

Some things did not like having this option enabled (Vonage for one, would sound bad when enabled), but performance of games and other traffic would increase noticeably. Maybe packet fragmentation would help chop large packets up to insert some smaller packets into the Queue to reduce latency rather than transmitting a max MTU sized packet when there is a buffer forming?

Just a toss of an idea with ZERO idea how to implement, or if Mikrotik supports it.

NetworkPro · Wed Oct 17, 2012 5:22 pm

rarely a solution - very custom I would say No to this Thank you for bringing it up.

brandonrossl · Wed Oct 17, 2012 6:12 pm

I figured as much, it was useful for torrents, voip, gaming etc on a VERY asynchronous cable connection (6MB down, <1 up)

This is THE SINGLE MOST USEFUL THING I've read yet on how to configure QoS.

Control is Outbound
Router Inbound traffic - traffic that is received by the router from any side. Any frames serialised by someone else on the wire and travelling towards the router.
Router Outbound traffic - traffic that goes out of a router's interface, towards the world or towards our clients. This is where we set up limits and manage the bandwidth.

RuterOS HTB allows us to work with the Outgoing traffic - the traffic that is leaving the router via any interface.

There are 4 ways we can look at traffic:
1) client's upload that the router receives on the local interface
2) client's upload that the router sends to the Internet
3) client's download that the router receives on the public interface
4) client's download that the router sends to the customer
1) and 3) - is Inbound traffic
2) and 4) - is Outbound traffic

We control 1) when its sent out as 2), and 3) when its sent out as 4)

It helps to understand why/how we "stamp" traffic and where that "stamp" (mark) is read.

I have 50down/25up connection at home that's rarely at capacity, but in trying to reduce latency I find myself turning to QoS to attempt to get the important packets out unhindered. It seems as though packet reordering would help someone like me quite a bit since connection saturation is not the problem, but the nature of FIFO of the router.

NetworkPro · Wed Oct 17, 2012 7:37 pm

We do not reorder. We discard. We do not have conditions for reordering. Ethernet is a serial technology - one frame at a time, with decently low latency on Gigabit and up.

MikroTik RouterBOARDs and x86 routers send out packets at Ethernet speeds.

If outgoing is 100Mbps and incoming is a flood of 1000Mbps - then the router should have received 10 frames while sending 1.

MikroTik claim they do not reorder therefore I conclude that the hardware+software do not do the extra resources to even try to reorder these 10 frames before sending them out.

I can imagine a reason for this being that the router would need to have 10 more ultra-expensive CPUs on a fast bus to calculate what, when, how, why to reorder.

Unless Atheros come up with a solution to this specific scenario what we can do is - have (multiple) 1000Mbps with our provider so that any such problem would be eliminated from existence.

I wonder if my conclusion here can be confirmed from Latvia? So that no one else asks about reordering.

brandonrossl · Wed Oct 17, 2012 9:28 pm

So to be clear and since adding to an on-topic thread is better than making a new one, and we don't have PM:
*assuming NAT=masquerade home router style, wan on eth1, bridging and using ip firewall*
Users outgoing packets TO internet are marked in mangle and shaped in what?
Packets from the internet to users are marked in mangle and shaped in what?

I'm still trying to wrap my head around it. My goal is to shape traffic by type, and not by user since 'users' have multiple devices that can access it all at once.

Something like:
Gaming
Streaming Video (netflix)
Browsing-High (first Xmb of connection)
The rest/torrents

Similar to what you said here:

So the plan is to mark by traffic type in prerouting and limit by traffic type in global-in.
1) we mark all traffic, that would be managed by one particular Queue, at the same place (prerouting)
2) we must mark upload and download for every type of traffic separately
5) we need 2 sets of queues - one for upload, one for download

and i cut out the sections that limit by IP since I have no need for that since my users don't have per-device or per-user priority, just traffic type.

NetworkPro · Thu Oct 18, 2012 12:49 am

Queue Tree max=-limit=the max limit of the WAN connection
-Gaming / limit-at=3Mbps / queue type: default-small
-Streaming Video (netflix) /limit-at=1-2 streams at the same time/ queue type: default / priority=1
-Browsing-High (first Xmb of connection) / queue type: PCQ /priority=2
-HTTP downloads / queue type: PCQ / priority=3
-The rest/torrents queue type: PCQ

The mentioned PCQ should have limit=5 total-limit=200 rate=0

Can you test with this in the router - one per direction upload/download ?

NetworkPro · Sat Oct 20, 2012 3:31 am

I figured out the answer to my question two posts above http://forum.mikrotik.com/viewtopic.php ... 74#p338374 .

The router forwards at the gigabit speed and sends the frames to the 100 megabit NIC. The NIC buffer fills and tail-drop occurs there. We prevent this by having a HTB that will drop 9 out of these 10 frames so that we do not have buffering in the NIC buffer.

P.S. if we put each of these 10 into separate leafs - then low level HTB theory applies and this may be still inherited by the Linux HTB (source and docs available on the web)

tenenbaum · Fri Apr 12, 2013 5:11 am

I vote for LLQ

tenenbaum · Fri Apr 12, 2013 5:45 am

NetworkPro,

Thank you for your input here and the article. Let me try and explain why I think LLQ is needed, and please correct me if I'm wrong.

We have an extensive network of relays that connects residential and business customers. During the day we see only about 50% utilization of our network, sometimes as low as 25%. At night we see near 90% and we are working to add more bandwidth.

I implemented QoS based on your article and it does seem to help, but, and I'll take a specific example I'm working on. So we have a business customer that has a voip pbx server on site. During the night, when the network is super congested their VoIP works perfectly and I can see their packets winning over everyone else in the queues.

Yet, during the day, even though the total bandwidth consumed is not high at all, the sheer number of packets processed on our (x86 Maxxwave) routers is like x10 the night.

Problem is that with packet priority using RouterOS, the priority only comes to effect when the queues are saturated. Well, in this case the queues are not saturated but the number of packets going through increases the latency a fluctuation a lot, and from what I read about LLQ, it would address this particular problem by allowing marked packets to simply travel in the router ahead of other packets regardless of whether the queue is congested or not.

True, we can go to even faster and faster routers and we will probably do that (our network is all solar+wind based so need to consider the limitations) but if there was a way to tell the router to process certain packets ahead of others, wouldn't that solve our problem?

And if it's possible to do in Mikrotik, how would I do it? How can I give packets priority without the queues reaching limits? And mind you, I tested our routers with mikrotik bandwidth generator mid day and they have a lot more capacity to handle, voip suffers greatly.

We of course checked signal levels during the day, capacity, cpu and other factors on the network.

Your input is appreciated.
Ofer

PS. Our network is fully routed. No NAT no private IPs, each customer is publicly routed with a single or /29 subnet.

NetworkPro · Fri Apr 12, 2013 8:21 am

interesting situation - this is a good question

so you have a radio network with high pps rates ? If you test the same setup on Gigabit Ethernet what would be the result ?

- problem is elsewhere
- LLQ is a silly thing pushed by Cl$c0 to increase sales (sleazy weasels can put their marketing up their a$$). No QoS should add latency. I challenge them to tell the truth what exactly happens inside their routers when you turn on the thing they call "LLQ"?

Its early morning , I may take a look at the post again later. (yawn)

tenenbaum · Fri Apr 12, 2013 4:14 pm

interesting situation - this is a good question

so you have a radio network with high pps rates ? If you test the same setup on Gigabit Ethernet what would be the result ?

Thank you for the reply. The problem is increased number of packets during the work day with low bandwidth utilization. We are a wireless ISP and yes, our equipment is gigabit ethernet in the relay stations but the gear we use Ubiquiti is not and I can't change that.

Thing is, it does work well during peak hours when bandwidth is maxed, so what I need is to be able to prioritize the packets based on the nature of the packet and not based on whether a queue is saturated or not. Any ideas on how I can do it with Mikrotik?

I understand what you are saying about LLQ and I don't have the knowledge to argue if it's true or not, so let's assume what you are saying is 100% true. I can't also change the wireless gear I'm using. Yet I know voip works great when the number of packets is under control even at 90% bandwidth utilization and that's 90mbit on that leg. But even when the bandwidth is just 25mbit during the day, the voip goes to hell because the pps is increasing and I have no way to tell the packets "you wait" and let the voip pass first...

Any help would be appreciated.

Sun Apr 14, 2013 4:48 pm

We are a wireless ISP and yes, our equipment is gigabit ethernet in the relay stations but the gear we use Ubiquiti is not and I can't change that.

Thing is, it does work well during peak hours when bandwidth is maxed, so what I need is to be able to prioritize the packets based on the nature of the packet and not based on whether a queue is saturated or not.

I think, in your particular case, it is your wireless equipment that should do packet prioritization. It should be doing it based on the DSCP, which you can set/adjust using some mangle rules on you Mikrotik router.

tenenbaum · Mon Apr 15, 2013 6:30 am

andriys and network pro,

I thought our system was tagging things properly but I found out that the packets coming out of the pbx system were losing their tags in the router (I am still researching why.) So I made the router re-tag in postrouting and sure enough that addressed the issue. Mid day and things sound more than great!

I guess I'll take my vote off LLQ

Thank you both!
Ofer

omidkosari · Wed Jun 12, 2013 10:23 am

@macgaiver

I see that you have good dominance on QOS and reply to user questions in forum . but reading and understanding a multipage forum topic is boring . Why you don't create some wiki documents with full examples (it is better with pictures) and share your knowledge with other users ?

Your efforts will be appreciated .

Wed Aug 14, 2013 3:33 pm

@macgaiver

I see that you have good dominance on QOS and reply to user questions in forum . but reading and understanding a multipage forum topic is boring . Why you don't create some wiki documents with full examples (it is better with pictures) and share your knowledge with other users ?

Your efforts will be appreciated .

Been there, done that!

Full examples make people lazy - they just copy/paste and do not think, only complain.
Complain that example presented is different from situation they have,

There is no point of making another page of theoretical stuff - nobody will read it - i know that just because it looks like nobody bothers to read the manual

i will stick to this forum - less effort, more results.

omidkosari · Wed Aug 14, 2013 10:19 pm

Thanks for reply .
I don't mean copy/paste examples . I mean schematic examples with description . I am one of your fans

in forum and i search for your posts and learn from them . Learn something that is not in wiki,docs etc . but as i said it is hard to use them as reference .

I mean you put compact part of each useful topic as a wiki article and complete it time to time .

It's just a suggestion . Your efforts will be appreciated .

efaden · Wed Aug 14, 2013 11:14 pm

Thanks for a really interesting thread guys.

One theoretical question, is it possible to do qos without knowing a maximal bandwidth. For example of the isp allows bursting, could qos be implemented without losing the bursting?

My assumption would be no. But curious if there is another answer.

Sent from my SCH-I545 using Tapatalk 2

skynets · Sat Dec 14, 2013 3:48 am

i vote for llq

i have a similar problem, as is tenenbaum.

Network based on satelite. Consist of micro networks (20-25 users on wifi).
In micro networks wifi mikrotik. If i use router cisco 1801 then voip is good.
If use mikrotik rb2011 voip is very bad. Talking time 1,5 - 3 minutes and disconnect.

Users voip gateway market paket dscp 46.

rebeen · Fri Mar 11, 2016 10:01 pm

Hello,

Guys I really need to implement LLQ in my MikroTik router but I was struggling to do so, is there to implement LLQ in MikroTik routers? Thanks

NetworkPro · Sat Mar 12, 2016 9:09 am

If you need to send out packets without delay or drop, you will have to configure packet queue settings that are already present.

There is no such thing as LLQ this is made up and does not exist. It's a lie.

For more info check the link in my signature.

vortex · Mon Mar 14, 2016 3:47 am

@macgaiver

I see that you have good dominance on QOS and reply to user questions in forum . but reading and understanding a multipage forum topic is boring . Why you don't create some wiki documents with full examples (it is better with pictures) and share your knowledge with other users ?

Your efforts will be appreciated .
Been there, done that!

Full examples make people lazy - they just copy/paste and do not think, only complain.
Complain that example presented is different from situation they have,

There is no point of making another page of theoretical stuff - nobody will read it - i know that just because it looks like nobody bothers to read the manual

i will stick to this forum - less effort, more results.

When I need to do something new, first I websearch into the wiki.

If that is unsatisfactory, then I try to find the answer in the forum.

ros code

Who is online