Working QoS configuration

negge · **Posted:** Fri Jul 10, 2009 12:14 pm

Hi everybody,

I've been struggling for some time to create a working QoS configuration, reading through countless manual pages, wikis, forum topics etc. I've noticed two things; in order to do something great with a Mikrotik router, you have to be a network guru. I thought I knew relatively much about networking when I got my router, but it seems I was wrong. Second, it seems like there's a tendency among those who actually do know something and who actually have working configurations to be pretty secretive about them, which is understandable, but very frustrating for novices. That's why I thought I'd post my QoS configuration here as well as explain what it does (or at least what I want it to do).

1. My network layout

WAN connection: 110/5 Mbit (up/down). The actual downlink speed is limited to 100 Mbit by the fact that my RouterBOARD only has 100 Mbit Ethernet ports.

My stuff is hooked up like this:
Cable modem <-- [bridge] --> Routerboard --> Switch(es) --> computers.

2. What I want to accomplish

I'm a heavy BitTorrent user, which means I'm pretty much maxing out my upstream bandwidth all the time. I also have my own FTP server which some of my friends use to download some stuff from me, a HTTP server from which stream music from my collection when I'm at other peoples places (I have my own home-made system for this) and finally I'm always connected to my computer at home using SSH.

What happens with no QoS at all is that I constantly have to limit the torrent clients upload speed everytime I come somewhere and want to stream something, or a friend needs to get something fast over FTP. It also means that unless I cap the upload speed in the torrent client I get incredible lag when using SSH to chat on IRC (for example). Of course web browsing is a PITA too when seeding at full speed...

3. The solution

The solution to all this is to mark traffic from the most critical to the non-critical. I got most of my rules from this guide. This is my current mangle config and my queue tree config. Just to clearify: "Public" is the interface connected to the modem and "Local" is the interface connected to the switch. The key seems to be to mark the ACK packets correctly (only marking the ones with a size of 0-80 bytes like most people suggest is not sufficient as the ACK flag can be set on bigger packets as well).

Mangle rules:

Code:

[xxx@xxx] /ip firewall mangle> print
Flags: X - disabled, I - invalid, D - dynamic 
 0   ;;; Link-critical traffic (ARP, DHCP)
     chain=postrouting action=mark-packet new-packet-mark=link_critical passthrough=no 
     protocol=udp out-interface=Public src-port=68 dst-port=67 

 1   ;;; Time-critical traffic (DNS, control packets)
     chain=postrouting action=mark-packet new-packet-mark=time_critical passthrough=no 
     protocol=udp out-interface=Public dst-port=53 

 2   chain=postrouting action=mark-packet new-packet-mark=time_critical passthrough=no 
     tcp-flags=fin,syn,rst protocol=tcp out-interface=Public 

 3   chain=postrouting action=mark-packet new-packet-mark=time_critical passthrough=no 
     tcp-flags=ack protocol=tcp out-interface=Public packet-size=40-89 

 4   chain=postrouting action=mark-packet new-packet-mark=time_critical passthrough=no 
     connection-state=new protocol=tcp out-interface=Public 

 5   ;;; Critical traffic
     chain=postrouting action=mark-packet new-packet-mark=critical passthrough=no 
     tcp-flags=ack protocol=tcp out-interface=Public packet-size=90-159 

 6   ;;; High-priority interactive traffic (SSH)
     chain=postrouting action=mark-packet new-packet-mark=high_pri_interactive 
     passthrough=no protocol=tcp out-interface=Public port=22,2200 

 7   chain=postrouting action=mark-packet new-packet-mark=high_pri_interactive 
     passthrough=no tcp-flags=ack protocol=tcp out-interface=Public packet-size=160-249 

 8   chain=postrouting action=mark-packet new-packet-mark=high_pri_interactive 
     passthrough=no protocol=tcp out-interface=Public port=8291 

 9   ;;; Low-priority interactive traffic (HTTP, HTTPS, DelugeWebUI)
     chain=postrouting action=mark-packet new-packet-mark=low_pri_interactive 
     passthrough=no protocol=tcp out-interface=Public port=80,443,8112 

10   chain=postrouting action=mark-packet new-packet-mark=low_pri_interactive 
     passthrough=no tcp-flags=ack protocol=tcp out-interface=Public packet-size=250-359 

11   ;;; High-priority non-interactive traffic (FTP)
     chain=postrouting action=mark-packet new-packet-mark=high_pri_non_interactive 
     passthrough=no protocol=tcp out-interface=Public connection-type=ftp 

12   chain=postrouting action=mark-packet new-packet-mark=high_pri_non_interactive 
     passthrough=no tcp-flags=ack protocol=tcp out-interface=Public packet-size=360-489 

13   ;;; Low-priority non-interactive traffic (POP, SMTP)
     chain=postrouting action=mark-packet new-packet-mark=low_pri_non_interactive 
     passthrough=no protocol=tcp out-interface=Public port=25,110 

14   chain=postrouting action=mark-packet new-packet-mark=low_pri_non_interactive 
     passthrough=no tcp-flags=ack protocol=tcp out-interface=Public packet-size=490-639 

15   ;;; Non-critical traffic (P2P)
     chain=postrouting action=mark-packet new-packet-mark=non_critical passthrough=no 
     tcp-flags=ack protocol=tcp out-interface=Public packet-size=640-809 

16   chain=postrouting action=mark-packet new-packet-mark=non_critical passthrough=no 
     protocol=tcp out-interface=Public

Queue tree:

Code:

[admin@Neggelandia] /queue tree> print
Flags: X - disabled, I - invalid 
 0   name="Link-critical" parent=Outgoing queue packet-mark=link_critical limit-at=0 
     queue=default priority=1 max-limit=4350000 burst-limit=0 burst-threshold=0 
     burst-time=0s 

 1   name="Time-critical" parent=Outgoing queue packet-mark=time_critical limit-at=0 
     queue=default priority=2 max-limit=4350000 burst-limit=0 burst-threshold=0 
     burst-time=0s 

 2   name="Critical" parent=Outgoing queue packet-mark=critical limit-at=0 queue=default 
     priority=3 max-limit=4350000 burst-limit=0 burst-threshold=0 burst-time=0s 

 3   name="High-pri interactive" parent=Outgoing queue packet-mark=high_pri_interactive 
     limit-at=0 queue=default priority=4 max-limit=4350000 burst-limit=0 
     burst-threshold=0 burst-time=0s 

 4   name="Low-pri interactive" parent=Outgoing queue packet-mark=low_pri_interactive 
     limit-at=0 queue=default priority=5 max-limit=4350000 burst-limit=0 
     burst-threshold=0 burst-time=0s 

 5   name="High-pri non-interactive" parent=Outgoing queue 
     packet-mark=high_pri_non_interactive limit-at=0 queue=default priority=6 
     max-limit=4350000 burst-limit=0 burst-threshold=0 burst-time=0s 

 6   name="Outgoing queue" parent=Public packet-mark="" limit-at=0 queue=default 
     priority=8 max-limit=4350000 burst-limit=0 burst-threshold=0 burst-time=0s 

 7   name="Low-pri non-interactive" parent=Outgoing queue 
     packet-mark=low_pri_non_interactive limit-at=0 queue=default priority=7 
     max-limit=4350000 burst-limit=0 burst-threshold=0 burst-time=0s 

 8   name="Non-critical" parent=Outgoing queue packet-mark=non_critical limit-at=0 
     queue=default priority=8 max-limit=4350000 burst-limit=0 burst-threshold=0 
     burst-time=0s

Queue types:

Code:

[xxx@xxx] /queue type> print
 0 name="default" kind=pfifo pfifo-limit=200 

 1 name="ethernet-default" kind=pfifo pfifo-limit=200

4. Result

With the current settings, if I'm seeding at full speed and somebody starts downloading something over FTP, the FTP transfer gets all upload bandwidth while the torrents almost stop. If I'm connected with SSH, there's absolutely no lag when I'm writing in the console. When surfing the web, DNS lookups happen instantly and pages load just like if I wasn't seeding at all.

Feel free to comment on my config if you have any suggestions that would make it better.

Edit: Forgot to mention that I changed the queue size to 200 packets (default is 50)

Radius · **Posted:** Wed Aug 19, 2009 12:21 am

seems that its working, is possible do this for download too

wan to lan qos

this is for
lan to wan qos

but p2p uploading is < 50% of free bandwith, when I start uploading something else (vpn), p2p is stil so low and vpn works 50KB (where is free bandwith +-500KB)... better will be p2p 50KB, vpn 600KB

pings are very high +- 50ms average... maybe qos download is the solution :shock:

negge · **Posted:** Wed Aug 19, 2009 10:08 am

You cannot apply QoS on the downlink because you have no way of controlling how much data is being sent to you. If you want to prioritize VPN just mark packets with IPsec traffic and give them higher priority in the queue.

lukkes · **Posted:** Thu Jan 14, 2010 1:41 am

Negge, first congratulation, it works good..

i'm trying to modify it to work in my scenario, i just want to know why you use thats packet size for ack? why you did no use 0-80 too? i've read the site that you told but they dont talk about the packets size you used, can you help me? where did you find that, thanks

JJOliver998 · **Posted:** Thu Jan 14, 2010 3:17 am

Hi Negge.

Thanks for this. I'm newish to MT and know little about routing. Thanks for writing down your config in this forum. Can I make a suggestion that you also write a wiki on the MT wiki page as I think that will be longer lasting. People, like you yourself mentioned check out the wiki first.

I'm glad you got it working well.

Josh

rmichael · **Posted:** Thu Jan 14, 2010 5:35 am

lukkes wrote:

Negge, first congratulation, it works good..

i'm trying to modify it to work in my scenario, i just want to know why you use thats packet size for ack? why you did no use 0-80 too? i've read the site that you told but they dont talk about the packets size you used, can you help me? where did you find that, thanks

Quote from the link Negge reffered to (length highlighted in red):

CLASSIFY udp -- anywhere anywhere length 490:639 CLASSIFY set 1:80
CLASSIFY tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,ACK/ACK length 640:809 CLASSIFY set 1:80
CLASSIFY udp -- anywhere anywhere length 360:489 CLASSIFY set 1:70
CLASSIFY tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,ACK/ACK length 490:639 CLASSIFY set 1:70
CLASSIFY udp -- anywhere anywhere length 250:359 CLASSIFY set 1:60
CLASSIFY tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,ACK/ACK length 360:489 CLASSIFY set 1:60
CLASSIFY udp -- anywhere anywhere length 160:249 CLASSIFY set 1:50

I'm still trying to figure out why this is needed...

...hm it's actually explained in the guide as well:

Quote:

The generalisation of the above is quite effective since we can assume that the larger the size of a TCP ACK (or UDP) packet, the more likely it is to contain payload data. The more data it has, the further it sinks in priority of being sent out our broadband connection first leaving all the real control ACK packets (with little or no payload) to exit as quickly as possible

EDIT: added content

PS: I think this setup makes a lot of packets arrive out of order.

netrat · **Posted:** Thu Jan 14, 2010 5:50 am

negge would you mind doing an export of your ip mangle and queue settings? I'd like to easily import them on a router and give your setup a try.

lukkes · **Posted:** Thu Jan 14, 2010 7:18 am

hmmm, ok understod, thanks, maybe i didn't read well the link, somebody know if i's really needed another lenght more tha 0-80? with the others lenght it catch much traffic, including p2p traffic
thanks

negge · **Posted:** Thu Jan 14, 2010 9:42 am

First of all, it's nice to see that it's working for other people too! It's been some time since I wrote this but the configuration has remained the same since then, so here it is:

Quote:

/ip firewall mangle
add action=mark-packet chain=postrouting comment="Link-critical traffic (DHCP)" disabled=no dst-port=67 new-packet-mark=link_critical out-interface=WAN passthrough=no protocol=udp src-port=68
add action=mark-packet chain=postrouting comment="IPSec VPN (same priority as link critical)" disabled=no new-packet-mark=link_critical out-interface=WAN passthrough=no protocol=ipsec-esp
add action=mark-packet chain=postrouting comment="Time-critical traffic (DNS, TCP control packets, certain ACK packets, new connections)" disabled=no dst-port=53 new-packet-mark=time_critical out-interface=WAN passthrough=no protocol=udp
add action=mark-packet chain=postrouting comment="" disabled=no new-packet-mark=time_critical out-interface=WAN passthrough=no protocol=tcp tcp-flags=fin,syn,rst
add action=mark-packet chain=postrouting comment="" disabled=no new-packet-mark=time_critical out-interface=WAN packet-size=40-89 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting comment="" connection-state=new disabled=no new-packet-mark=time_critical out-interface=WAN passthrough=no protocol=tcp
add action=mark-packet chain=postrouting comment="Critical traffic (just some ACK packets)" disabled=no new-packet-mark=critical out-interface=WAN packet-size=90-159 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting comment="High-priority interactive traffic (SSH, WinBox, certain ACK packets)" disabled=no new-packet-mark=high_pri_interactive out-interface=WAN passthrough=no port=22,2200 protocol=tcp
add action=mark-packet chain=postrouting comment="" disabled=no new-packet-mark=high_pri_interactive out-interface=WAN passthrough=no port=8291 protocol=tcp
add action=mark-packet chain=postrouting comment="" disabled=no new-packet-mark=high_pri_interactive out-interface=WAN packet-size=160-249 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting comment="Low-priority interactive traffic (HTTP, HTTPS, DelugeWebUI)" disabled=no new-packet-mark=low_pri_interactive out-interface=WAN passthrough=no port=80,443,8112 protocol=tcp
add action=mark-packet chain=postrouting comment="" disabled=no new-packet-mark=low_pri_interactive out-interface=WAN packet-size=250-359 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting comment="High-priority non-interactive traffic (FTP)" connection-type=ftp disabled=no new-packet-mark=high_pri_non_interactive out-interface=WAN passthrough=no protocol=tcp
add action=mark-packet chain=postrouting comment="" disabled=no new-packet-mark=high_pri_non_interactive out-interface=WAN packet-size=360-489 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting comment="Low-priority non-interactive traffic (POP, SMTP)" disabled=no new-packet-mark=low_pri_non_interactive out-interface=WAN passthrough=no port=25,110 protocol=tcp
add action=mark-packet chain=postrouting comment="" disabled=no new-packet-mark=low_pri_non_interactive out-interface=WAN packet-size=490-639 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting comment="Non-critical traffic (P2P)" disabled=no new-packet-mark=non_critical out-interface=WAN packet-size=640-809 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=postrouting comment="" disabled=no new-packet-mark=non_critical out-interface=WAN passthrough=no protocol=tcp

Quote:

/queue tree
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=2M name="Outgoing queue" packet-mark="" parent=WAN priority=8
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=2M name="Low-pri non-interactive" packet-mark=low_pri_non_interactive parent="Outgoing queue" priority=7 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=2M name=Non-critical packet-mark=non_critical parent="Outgoing queue" priority=8 queue=non_critical_queue
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=2M name=Link-critical packet-mark=link_critical parent="Outgoing queue" priority=1 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=2M name=Time-critical packet-mark=time_critical parent="Outgoing queue" priority=2 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=2M name=Critical packet-mark=critical parent="Outgoing queue" priority=3 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=2M name="High-pri interactive" packet-mark=high_pri_interactive parent="Outgoing queue" priority=4 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=2M name="Low-pri interactive" packet-mark=low_pri_interactive parent="Outgoing queue" priority=5 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=2M name="High-pri non-interactive" packet-mark=high_pri_non_interactive parent="Outgoing queue" priority=6 queue=default

You may need to edit the commands to reflect your own interface names. I don't use "Public" and "Local" anymore like I did in my first post, nowadays it's simply "WAN" and "LAN".

Quote:

i'm trying to modify it to work in my scenario, i just want to know why you use thats packet size for ack? why you did no use 0-80 too? i've read the site that you told but they dont talk about the packets size you used, can you help me? where did you find that, thanks

The reason why I only mangle ACK packets with size between 40-89 and not 0-89 is because a TCP packet can't be smaller than 40 bytes.

Quote:

Thanks for this. I'm newish to MT and know little about routing. Thanks for writing down your config in this forum. Can I make a suggestion that you also write a wiki on the MT wiki page as I think that will be longer lasting. People, like you yourself mentioned check out the wiki first.

I might just do that, if there's enough interest and if I find some spare time.

Quote:

PS: I think this setup makes a lot of packets arrive out of order.

Why would it do that? This is TCP.

Pilgrim · **Posted:** Thu Jan 14, 2010 12:25 pm

Thanks Negge, Great piece of work. Just want to add a question concerning your comments on the download.

Quote:

You cannot apply QoS on the downlink because you have no way of controlling how much data is being sent to you. If you want to prioritize VPN just mark packets with IPsec traffic and give them higher priority in the queue.

I have seen many examples on prioritizing downloads and have also set up a QoS for my download. Maybe I misunderstood you, but did you mean to say that it is not possible to prioritize downloads. I am just wondering if your rules could not be applied to download also - or would it spoil / overlap the purpose "Maximise Download Speed via Outbound Traffic Shaping"?

rgs Pilgrim

rmichael · **Posted:** Thu Jan 14, 2010 1:39 pm

Quote:

Quote:
PS: I think this setup makes a lot of packets arrive out of order.

Why would it do that? This is TCP.

Packets will eventually be reordered since -as you point out it's TCP -however during transmission dupack will be sent and in some cases (depending on dupack limit) can trigger retransmission. I don't know how badly this would affect the speed though. Myself I try not to rearange packets within the same stream.

negge · **Posted:** Fri Jan 15, 2010 8:30 am

@Pilgrim: What I meant to say is you can't directly shape your incoming connections, but you can get the same effect by prioritizing certain ACK packets instead. That's basically what my configuration does. If all the ACK rules wouldn't be there, the QoS would only work on a server where no one rarely downloads anything at all.

Pilgrim · **Posted:** Fri Jan 15, 2010 1:32 pm

Thanks negge, That is really interesting. I also read the article and to me this is completely new angle on QoS.

Great job in implementing this on the mikrotik platform and like it was already mentioned it would be really nice if this could be turned into an article on the wiki.

rgs Pilgrim

negge · **Posted:** Fri Jan 15, 2010 1:52 pm

I'll see if I can put together a decent wiki page in the next couple of weeks!

phix · **Posted:** Sat Jan 16, 2010 1:20 pm

rmichael wrote:

Quote:

Quote:
PS: I think this setup makes a lot of packets arrive out of order.

Why would it do that? This is TCP.

Packets will eventually be reordered since -as you point out it's TCP -however during transmission dupack will be sent and in some cases (depending on dupack limit) can trigger retransmission. I don't know how badly this would affect the speed though. Myself I try not to rearange packets within the same stream.

Hi Rmichael,

I'm Mark, I wrote the HOWTO that Negge has based his MikroTek configuration on. I just wanted to make a quick comment about your statement here (I have updated my HOWTO accordingly).

I agree with your statement about dupACKs, but, only where TCP packets have not been classified into a particular queue based on something consitent like a client or peer port number. In this instance, they will be classified on size instead and the shaping rules I've described do this in a banded way based on size. Again, in this instance, the only possible way out of ordered packets could then occur is if BOTH the client and peer are sending payload (data) ACK packets to each other, within the same TCP session. When I wrote that HOWTO (some years ago now!), I was observing how BitTorrent exchanges data across TCP. Most of the time, I observed that BitTorrent TCP streams could be classified into one of two forms:

1) TCP sessions between you and peer where YOU are sending large payload ACKs out, peers sending small ACKs back
2) TCP session between you and peer where THEY are sending large payload ACKs in, you sending small ACKs back

If either of the above hold true, the chances of out of ordered packets reaching you or your peers are fairly remote. If however, you and your peers are doing BOTH of the above, within the SAME TCP session, then yes, out of ordered packets could occur.

BitTorrent has since moved onto uTP anyway (which uses UDP) - if anyone has any further information about this protocol, such as its frame format, it is probably very possible to use iptables to match on particular bytes within such packets to determine if they are uTP, and thus place such traffic in a low priority class, rather than use the crude method of size. This would be preferable.

I'm glad everyone here has found Negge's configuration and my HOWTO useful - when I originally came up with the ideas in it, it dramatically improved the usability and speed of my connection when I had a 1Mbit broadband line (things have now moved on though, now I am on 50Mbit) :-)

negge · **Posted:** Tue Apr 06, 2010 10:15 pm

I'm sorry I haven't put together the wiki article yet, I simply haven't had that much spare time lately. Don't give up the hope though, it will come sooner or later!

phix · **Posted:** Sun Jul 18, 2010 11:36 am

Just a heads up for anyone using these rules. Due to DNSSEC being switched on in the root name servers, with the rest of the world following shortly:

http://tech.slashdot.org/story/10/07/17 ... SEC-Signed

there is a legacy DNS querk where if a reply from a DNS server exceeds 512 bytes (which is now the case from the root servers), the protocol switches to using TCP. So you'll want to also include TCP destination port 53 traffic in these classification rules as well as UDP (just repeat the UDP port 53 rule but for TCP instead). If you miss this out, it -could- affect the speed at which you resolve host names, and hence will affect the speed at which you can browse.

Although, this issue does not appear to have affected me because I use the 'unbound' ( http://www.unbound.net/ ) resolver daemon which, upon initial checks, ignores this limit and continues to use UDP regardless.

Working QoS configuration

Who is online