QoS setup

quickgen · September 22, 2007, 11:58am

Dear all,
I have 2 ISPs and 300 clients. I am using mangle to separate local and overseas traffic for each client, because we assign different rates for each kind of traffic. So for one client i have in Mangle 4 rules:

Download Overseas Traffic
Upload Overseas Traffic
Download Local Traffic
Upload Local Traffic

In Qtree I shape each kind of mangle rule, so again I have 4 shapers.

My Mikrotik setup is as follows:

Mangle

8 ;;; 192.168.0.2
chain=forward dst-address=192.168.0.2 src-address-list=!local action=mark-packet new-packet-mark=192.168.0.2-overseas-dl passthrough=no

9 chain=forward src-address=192.168.0.2 dst-address-list=!local action=mark-packet new-packet-mark=192.168.0.2-overseas-ul passthrough=no

10 chain=forward dst-address=192.168.0.2 src-address-list=local action=mark-packet new-packet-mark=192.168.0.2-local-dl passthrough=no

11 chain=forward src-address=192.168.0.2 dst-address-list=local action=mark-packet new-packet-mark=192.168.0.2-local-ul passthrough=no

Qtree

0 name=“Overseas-Download” parent=Lan packet-mark=“” limit-at=0
queue=PCQ_Download priority=8 max-limit=0 burst-limit=0
burst-threshold=0 burst-time=0s

1 name=“Local-Upload” parent=global-out packet-mark=“” limit-at=0
queue=PCQ_Upload priority=8 max-limit=0 burst-limit=0
burst-threshold=0 burst-time=0s

2 name=“Local-Download” parent=Lan packet-mark=“” limit-at=0
queue=PCQ_Download priority=8 max-limit=25000000 burst-limit=0
burst-threshold=0 burst-time=0s

3 name=“Overseas-Upload” parent=global-out packet-mark=“” limit-at=0
queue=PCQ_Upload priority=8 max-limit=0 burst-limit=0 burst-threshold=0
burst-time=0s

4 name=“192.168.0.2-local-dl” parent=Local-Download
packet-mark=192.168.0.2-local-dl limit-at=0 queue=default priority=8
max-limit=4000000 burst-limit=0 burst-threshold=0 burst-time=0s

5 name=“192.168.0.2-local-ul” parent=Local-Upload
packet-mark=192.168.0.2-local-ul limit-at=0 queue=default priority=8
max-limit=2000000 burst-limit=0 burst-threshold=0 burst-time=0s

6 name=“192.168.0.2-overseas-dl” parent=Overseas-Download
packet-mark=192.168.0.2-overseas-dl limit-at=0 queue=default priority=8
max-limit=512000 burst-limit=0 burst-threshold=0 burst-time=0s

7 name=“192.168.0.2-overseas-ul” parent=Overseas-Upload
packet-mark=192.168.0.2-overseas-ul limit-at=0 queue=default priority=8
max-limit=512000 burst-limit=0 burst-threshold=0 burst-time=0s

My questions are:

This setup takes too much CPU resources, is there possible any other kind of setup to achieve what I have done.
I want to give priority to traffic as it was disscused here http://forum.mikrotik.com/t/burst-for-each-tcp-connection/10713/1 so each client can browse faster. The only way I made it is mangling each kind of traffic for each ip, but it’s 20 !!! mangle rules for IP. My idea is to give priority Globaly. As I understand with my setup I’m unable to achieve that because the packets will be remarked.
If not, how to accomplish this?
(I’m thinking of 2 Mikrotiks. One for priority and another for shaping each ip address).

Thank you for every answer.

Alex

quickgen · September 24, 2007, 10:56am

Anybody?

sergejs · September 24, 2007, 12:54pm

For 300 clients you may thing about PCQ implementation, as it is possible to create separate groups,

specify amount of bandwidth, that will be equaly divided between the members of group;
specify amout of bandwidth, that will be assigned for each member of the group.

About priority, it is possible to mark entire traffic for the specific service with one rule, it is not necessary to create priority rule for each IP separately.

quickgen · September 24, 2007, 1:11pm

sergejs, thank you for reply. I have earlier tested PCQ and indeed it does limit client to the rate specified, but with PCQ rate specified in Qtree I can’t apply burst for each tcp connection disscused here: http://forum.mikrotik.com/t/burst-for-each-tcp-connection/10713/1 and it’s the only real implementation that worked as expected and throlled down p2p.

About priority:
If I mark-packet(passthrough=no) globally services i want to give priority, then i can’t apply packet-mark for each ip (client). If passthrough=yes than packets get remarked and priority in Qtree is not working.

Maybe I misunderstand something?
Please give me the right direction.
Thank you.

sergejs · September 24, 2007, 1:21pm

You are correct about passtrough.
It is very complicated configuration to join prioritization with bandwidth limiting.
You may use the following configuration tips to accomplish it, combine ‘queue tree’ + ‘mangle’.

Firstly let’s ensure prioritization, add mangle marks to chain=prerouting and use ‘queue tree’ with global-in/globabl-out interfaces to set the priority for the specific traffic.
Then let’s ensure bandwidth limiting, add rules to chain=forward and use ‘remote’/‘local’ as interfaces at ‘queue tree’ configuration. Additionaly you may use ‘address-list’ to decrease mangle rule number (address-list is feature that allows to create different groups of IP addresses).

quickgen · September 24, 2007, 2:22pm

sergejs thanks for fast reply.
I will consider your advise.
Still my configuration is tricky:
1. I have 2 ISPs, so I can’t get use of ‘remote’/‘local’ as interfaces, because I’ll need to make rules for each ISP(taking into account Local County/Overseas traffic) per client. Traffic already gets divided for every client as Local County/Overseas so it’s 4 rules per client.
In Qtree i’m using parent=Lan(It’s the only Client interface) for Download Traffic and parent=global-out for Upload Traffic, to avoid shaping per interface.

1 name=“Local-Upload” parent=global-out packet-mark=“” limit-at=0
queue=PCQ_Upload priority=8 max-limit=0 burst-limit=0
burst-threshold=0 burst-time=0s

2 name=“Local-Download” parent=Lan packet-mark=“” limit-at=0
queue=PCQ_Download priority=8 max-limit=25000000 burst-limit=0
burst-threshold=0 burst-time=0s

2. You advised to use Adress-List for mangling. Do you mean that using 4 mangle rules only and specifying Src-address-list/Dst-address-list I will be able in Qtree to limit each IP? Maybe I’m wrong, but as expected it will limit ALL Clients.
Still I have to limit Upload/Download for both Local Country Traffic and Overseas Traffic, so its 4 shapers.

I’m thinking of testing 2 Mikrotiks:
First will deal with 2 ISPs and Routes and second will be a transparent traffic shaper for clients. They’ll comunnicate through separate Interfaces. Do you think it’s a better way to achieve my goal?

Thank you.

sergejs · September 25, 2007, 12:28pm

You have to consider about the prioritization then. Probably it is more useful to route important data over fastest connection and non-important to second gateway.
As there is no sense to prioritize traffic, that will be divided then to two parts.
Use only bandwidth limiting then use ‘global’ interface to specify the total upload, instead of specified different itnerfaces.
PCQ queue and address-list allow you to do this. As PCQ queue is applied for group not for separate user.

quickgen · September 25, 2007, 1:47pm

sergejs thank you. I clearly see your point of view, because I have already tested PCQ.
But still remains one unsolved question for me. How to combine PCQ with what is described here http://forum.mikrotik.com/t/burst-for-each-tcp-connection/10713/1 ?
This setup was tested by me and it works like a charm. Clients there very satisfied with http-browsing even at the full load of the channel. But I have to limit their rate anyway, because they paid for that rate only.
My PCQ setup will have pcq-rate=‘for example 128k’ because the client is paying for no more than 128k. PCQ rate will not allow him more than 128k. My queston is how to combine http://forum.mikrotik.com/t/burst-for-each-tcp-connection/10713/1 with fixed pcq-rate.
I was not able to achieve the proper combination.

Thank you.
P.S.This topic will be very useful for the MT community =)

quickgen · September 27, 2007, 2:21pm

sergejs? Is there any way to combine fixed PCQ-rate with Prioritization as described in my last post?

quickgen · November 2, 2007, 1:40am

Solved traffic priority and queue tree limmiting by using 2 mikrotiks in line. One is just mangling clients and shaping to the correct speed. The other one is doing prioritization. Seems good for 1 month =)

titius · November 7, 2007, 12:03pm

@quickgen

Hi can you post your mangle and queue configuration.

From both routers.

Please .

quickgen · January 17, 2008, 1:38pm

Titius, sorry for not checking this topic.

Here is my configuration for the border router:

/ ip firewall mangle

add chain=forward src-address-list=Local-traffic dst-address-list=“Local IpS”
action=mark-connection new-connection-mark=Local-DL passthrough=yes
comment=“Mark All Local Connections” disabled=no
add chain=forward src-address-list=“Local IpS” dst-address-list=Local-traffic
action=mark-connection new-connection-mark=Local-UL passthrough=yes comment=“”
disabled=no

add chain=forward src-address-list=!Local-traffic dst-address-list=“Local IpS”
action=mark-connection new-connection-mark=World-DL passthrough=yes
comment=“Mark All World Connections” disabled=no
add chain=forward src-address-list=“Local IpS” dst-address-list=!Local-traffic
action=mark-connection new-connection-mark=World-UL passthrough=yes comment=“”
disabled=no

add chain=forward protocol=tcp connection-mark=World-DL connection-bytes=0-200000
action=mark-packet new-packet-mark=World-tcp-browse-dl passthrough=no
comment=“World Download” disabled=no
add chain=forward protocol=tcp connection-mark=World-DL action=mark-packet
new-packet-mark=World-tcp-other-dl passthrough=no comment=“” disabled=no
add chain=forward protocol=icmp connection-mark=World-DL action=mark-packet
new-packet-mark=World-icmp-dl passthrough=no comment=“” disabled=no
add chain=forward protocol=udp connection-mark=World-DL action=mark-packet
new-packet-mark=World-udp-dl passthrough=no comment=“” disabled=no
add chain=forward connection-mark=World-DL action=mark-packet
new-packet-mark=World-other-dl passthrough=no comment=“” disabled=no

add chain=forward protocol=tcp connection-mark=World-UL connection-bytes=0-200000
action=mark-packet new-packet-mark=World-tcp-browse-ul passthrough=no
comment=“World Upload” disabled=no
add chain=forward protocol=tcp connection-mark=World-UL action=mark-packet
new-packet-mark=World-tcp-other-ul passthrough=no comment=“” disabled=no
add chain=forward protocol=icmp connection-mark=World-UL action=mark-packet
new-packet-mark=World-icmp-ul passthrough=no comment=“” disabled=no
add chain=forward protocol=udp connection-mark=World-UL action=mark-packet
new-packet-mark=World-udp-ul passthrough=no comment=“” disabled=no
add chain=forward connection-mark=World-UL action=mark-packet
new-packet-mark=World-other-ul passthrough=no comment=“” disabled=no

add chain=forward protocol=tcp connection-mark=Local-DL connection-bytes=0-200000
action=mark-packet new-packet-mark=Local-tcp-browse-dl passthrough=no
comment=“Local Download” disabled=no
add chain=forward protocol=tcp connection-mark=Local-DL action=mark-packet
new-packet-mark=Local-tcp-other-dl passthrough=no comment=“” disabled=no
add chain=forward protocol=icmp connection-mark=Local-DL action=mark-packet
new-packet-mark=Local-icmp-dl passthrough=no comment=“” disabled=no
add chain=forward protocol=udp connection-mark=Local-DL action=mark-packet
new-packet-mark=Local-udp-dl passthrough=no comment=“” disabled=no
add chain=forward connection-mark=Local-DL action=mark-packet
new-packet-mark=Local-other-dl passthrough=no comment=“” disabled=no

add chain=forward protocol=tcp connection-mark=Local-UL connection-bytes=0-200000
action=mark-packet new-packet-mark=Local-tcp-browse-ul passthrough=no
comment=“Local Upload” disabled=no
add chain=forward protocol=tcp connection-mark=Local-UL action=mark-packet
new-packet-mark=Local-tcp-other-ul passthrough=no comment=“” disabled=no
add chain=forward protocol=icmp connection-mark=Local-UL action=mark-packet
new-packet-mark=Local-icmp-ul passthrough=no comment=“” disabled=no
add chain=forward protocol=udp connection-mark=Local-UL action=mark-packet
new-packet-mark=Local-udp-ul passthrough=no comment=“” disabled=no
add chain=forward connection-mark=Local-UL action=mark-packet
new-packet-mark=Local-other-ul passthrough=no comment=“” disabled=no

/ queue tree

add name=“World-Download” parent=global-out packet-mark=“” limit-at=0 queue=default priority=2 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“1.World-tcp-browse-dl” parent=World-Download packet-mark=World-tcp-browse-dl limit-at=0 queue=default priority=2 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“2.World-tcp-other-dl” parent=World-Download packet-mark=World-tcp-other-dl limit-at=0 queue=default priority=7 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“3.World-icmp-dl” parent=World-Download packet-mark=World-icmp-dl limit-at=0 queue=default priority=1 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“4.World-udp-dl” parent=World-Download packet-mark=World-udp-dl limit-at=0 queue=default priority=6 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“5.World-other-dl” parent=World-Download packet-mark=World-other-dl limit-at=0 queue=default priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no

add name=“World-Upload” parent=global-out packet-mark=“” limit-at=0 queue=default priority=1 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“1.World-tcp-browse-ul” parent=World-Upload packet-mark=World-tcp-browse-ul limit-at=0 queue=default priority=2 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“2.World-tcp-other-ul” parent=World-Upload packet-mark=World-tcp-other-ul limit-at=0 queue=default priority=7 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“3.World-icmp-ul” parent=World-Upload packet-mark=World-icmp-ul limit-at=0 queue=default priority=1 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“5.World-other-ul” parent=World-Upload packet-mark=World-other-ul limit-at=0 queue=default priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“4.World-udp-ul” parent=World-Upload packet-mark=World-udp-ul limit-at=0 queue=default priority=6 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no

add name=“Local-Download” parent=global-out packet-mark=“” limit-at=0 queue=default priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“1.Local-tcp-browse-dl” parent=Local-Download packet-mark=Local-tcp-browse-dl limit-at=0 queue=default priority=2 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“2.Local-tcp-other-dl” parent=Local-Download packet-mark=Local-tcp-other-dl limit-at=0 queue=default priority=7 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“3.Local-icmp-dl” parent=Local-Download packet-mark=Local-icmp-dl limit-at=0 queue=default priority=1 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“4.Local-udp-dl” parent=Local-Download packet-mark=Local-udp-dl limit-at=0 queue=default priority=6 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“5.Local-other-dl” parent=Local-Download packet-mark=Local-other-dl limit-at=0 queue=default priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no

add name=“Local-Upload” parent=global-out packet-mark=“” limit-at=0 queue=default priority=7 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“1.Local-tcp-browse-ul” parent=Local-Upload packet-mark=Local-tcp-browse-ul limit-at=0 queue=default priority=2 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“2.Local-tcp-other-ul” parent=Local-Upload packet-mark=Local-tcp-other-ul limit-at=0 queue=default priority=7 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“3.Local-icmp-ul” parent=Local-Upload packet-mark=Local-icmp-ul limit-at=0 queue=default priority=1 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“4.Local-udp-ul” parent=Local-Upload packet-mark=Local-udp-ul limit-at=0 queue=default priority=6 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no
add name=“5.Local-other-ul” parent=Local-Upload packet-mark=Local-other-ul limit-at=0 queue=default priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no

quickgen · January 17, 2008, 2:05pm

This is the configuration of the MT that deals directly with the clients:

/ ip firewall mangle

add chain=forward dst-address=192.168.0.2 src-address-list=!Local-traffic
action=mark-packet new-packet-mark=192.168.0.2-world-dl passthrough=no
comment=“192.168.0.2” disabled=no
add chain=forward src-address=192.168.0.2 dst-address-list=!Local-traffic
action=mark-packet new-packet-mark=192.168.0.2-world-ul passthrough=no
comment=“” disabled=no

add chain=forward dst-address=192.168.0.2 src-address-list=Local-traffic
action=mark-packet new-packet-mark=192.168.0.2-local-dl passthrough=no
comment=“” disabled=no
add chain=forward src-address=192.168.0.2 dst-address-list=Local-traffic
action=mark-packet new-packet-mark=192.168.0.2-local-ul passthrough=no
comment=“” disabled=no

/ queue tree

add name=“1.World-DL” parent=global-out packet-mark=“” limit-at=0 queue=default
priority=2 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s
disabled=no
add name=“192.168.0.2-world-dl” parent=1.World-DL packet-mark=192.168.0.2-world-dl
limit-at=0 queue=default priority=8 max-limit=10000000 burst-limit=0
burst-threshold=0 burst-time=0s disabled=no

add name=“2.World-UL” parent=global-out packet-mark=“” limit-at=0 queue=default
priority=1 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s
disabled=no
add name=“192.168.0.2-world-ul” parent=2.World-UL packet-mark=192.168.0.2-world-ul
limit-at=0 queue=default priority=8 max-limit=10000000 burst-limit=0
burst-threshold=0 burst-time=0s disabled=no

add name=“3.Local-DL” parent=global-out packet-mark=“” limit-at=0 queue=default
priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s
disabled=no
add name=“192.168.0.2-local-dl” parent=3.Local-DL packet-mark=192.168.0.2-local-dl
limit-at=0 queue=default priority=1 max-limit=50000000 burst-limit=0
burst-threshold=0 burst-time=0s disabled=no

add name=“4.Local-UL” parent=global-out packet-mark=“” limit-at=0 queue=default
priority=7 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s
disabled=no
add name=“192.168.0.2-local-ul” parent=4.Local-UL packet-mark=192.168.0.2-local-ul
limit-at=0 queue=default priority=8 max-limit=10000000 burst-limit=0
burst-threshold=0 burst-time=8s disabled=no

quickgen · January 17, 2008, 2:11pm

If something will be not so clear with this setup - ask questions and I’ll be glad to answer them. If you know something about optimizing this setup, please let me know.
Thanks.

titius · January 20, 2008, 10:12pm

thx

quickgen · January 21, 2008, 5:29pm

Titius, This config is very CPU hungry. I have applied http://wiki.mikrotik.com/wiki/Different_limits_for_Local/Overseas_traffic_for_3_bandwitch_rates_using_pcq_and_queue_tree to the client side routers and I can say that it discreased the cpu load from 80-90% to 10-20% on a Pentium4 3.0 GHz , so I really recommend it. Although you cannot monitor each of the 4 queues for each client and you can’t know how much bandwidth each IP is using, but this is really great compensated by the low CPU load.
So maybe someone knows other ways to achieve the same goal? Feel Free to improve.

KirillTs · May 9, 2008, 1:10pm

192.168.0.2 is address of mikrotik router?

quickgen · May 10, 2008, 7:19am

KirillTs no. It is a client ip address.