Burst for each TCP connection

janisk · **Posted:** Thu Dec 21, 2006 3:01 pm

This is little how-to create manual burst using queue tree.

As it is bandwidth control using queue tree first we need to mangle traffic

first i mangle all connections, then i mark first 2Mbytes then i mark the rest of packets

Code:

/ip firewall mangle add chain=forward protocol=tcp action=mark-connection new-connection-mark=new_conn passthrough=yes comment="mark all new connections" disabled=no 
/ip firewall mangle add chain=forward protocol=tcp  connection-mark=new_conn connection-bytes=0-2000000 action=mark-packet new-packet-mark=new_packet passthrough=no comment="mark packets" disabled=no 
/ip firewall mangle add chain=forward protocol=tcp connection-mark=new_conn action=mark-packet new-packet-mark=old_packets passthrough=no comment="marking old packets" disabled=no 

now, when packets has been marked i need queue types that allows me limit traffic per-user basis, so i have to create 2 queue types.

Code:

/queue type add name="PCQ_Upload" kind=pcq pcq-rate=0 pcq-limit=50 pcq-classifier=src-address pcq-total-limit=2000 
/queue type add name="PCQ_Download" kind=pcq pcq-rate=0 pcq-limit=50 pcq-classifier=dst-address pcq-total-limit=2000 

now i have to limit all the rates using queue tree, i will limit it that way - that first 2Mbits (as in mangle) will have higher priority over rest of traffic where connection bytes exceed 2Mbits.

i assume that connection that is available have 5Mbits full-duplex

Code:

/queue tree add name="Main_Upload" parent=Public packet-mark="" limit-at=0 queue=default priority=8 max-limit=5000000 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no 

/ queue tree add name="Up First 2Mbyte" parent=Main_Upload packet-mark=new_packet limit-at=4000000 queue=PCQ_Upload priority=1 max-limit=5000000 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no 

/ queue tree add name="Up Rest Mbytes" parent=Main_Upload packet-mark=old_packets limit-at=1000000 queue=PCQ_Upload priority=8 max-limit=5000000 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no 

/ queue tree add name="Main_Download" parent=Local packet-mark="" limit-at=0 queue=default priority=8 max-limit=5000000 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no 

/ queue tree add name="Down First 2Mbyte" parent=Main_Download packet-mark=new_packet limit-at=4000000 queue=PCQ_Download priority=1 max-limit=5000000 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no 

/ queue tree add name="Down Rest Mbytes" parent=Main_Download packet-mark=old_packets limit-at=1000000 queue=PCQ_Download priority=8 max-limit=5000000 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no

any suggestions? comments?

macgaiver · **Posted:** Thu Dec 21, 2006 4:25 pm

I was 1day from doing the same thing (make some configuration and paste it in here) - Good work Janis

And what about UDP???

janisk · **Posted:** Thu Dec 21, 2006 4:36 pm

maybe someone could do that?

mmx · **Posted:** Thu Dec 21, 2006 4:56 pm

hi,

i'm also very interesting in this advances QoS. It will be gret to find a right solution and to put it like an example in one of manual.

I'll try also to post some scripts

raivis-v · **Posted:** Thu Dec 21, 2006 5:42 pm

mmx wrote:

hi,

i'm also very interesting in this advances QoS. It will be gret to find a right solution and to put it like an example in one of manual.

I'll try also to post some scripts

WIKIPEDIA is for examples. If you make some good example script, better place it here: http://wiki.mikrotik.com/wiki/Main_Page

jaafarkhalaf · **Posted:** Mon Feb 05, 2007 7:31 pm

this is very useful and powerful procedure.....thank u janisk....i applied it to my MT and it works fine :lol:

samsoft08 · **Posted:** Mon Feb 05, 2007 7:56 pm

does the marking rules mark the PROXY packets ??
i think it doesnt ..

samsoft08 · **Posted:** Mon Feb 05, 2007 10:58 pm

very important :

from the manual :

Quote:

connection-bytes=2000000-0 means that the rule matches if more than 2MB has been transfered through the relevant connection

2000000 is 2MByte not 2Mbit...

janisk · **Posted:** Tue Feb 06, 2007 11:28 am

samsoft08 wrote:

very important :

from the manual :

Quote:

connection-bytes=2000000-0 means that the rule matches if more than 2MB has been transfered through the relevant connection

2000000 is 2MByte not 2Mbit...

in that field you enter value of - how many bits to count.

byte is 8 bits

kbyte is 1024 bytes and 8192 bits

so do the math how much 2000000 bits are in Bytes or MBytes

samsoft08 · **Posted:** Tue Feb 06, 2007 2:22 pm

thats right .. but in your post you said 2Mbits which is wrong ..
its 2Mbytes according to the manual..

janisk · **Posted:** Tue Feb 06, 2007 3:15 pm

corrected error in my post, thx for pointing that out.

these really are bytes not bits

then do not mess with M is for 1000000 not 1024*1024 (like in hard-drives)

menfung · **Posted:** Tue Jun 19, 2007 3:02 am

janisk wrote:

This is little how-to create manual burst using queue tree.

As it is bandwidth control using queue tree first we need to mangle traffic

first i mangle all connections, then i mark first 2Mbytes then i mark the rest of packets

Code:

/ip firewall mangle add chain=forward protocol=tcp action=mark-connection new-connection-mark=new_conn passthrough=yes comment="mark all new connections" disabled=no 
/ip firewall mangle add chain=forward protocol=tcp  connection-mark=new_conn connection-bytes=0-2000000 action=mark-packet new-packet-mark=new_packet passthrough=no comment="mark packets" disabled=no 
/ip firewall mangle add chain=forward protocol=tcp connection-mark=new_conn action=mark-packet new-packet-mark=old_packets passthrough=no comment="marking old packets" disabled=no 

any suggestions? comments?

Hi Janisk,

Thanks for this good configuration, it's so cool.

But how to target those mangle markings above to some IP addresses or some Group of IP addresses?? Hopefully you won't mind to teach a newbie like me.

Thks n Rgds,

Men Fung

menfung · **Posted:** Tue Jun 19, 2007 12:20 pm

Hi Janisk,

I did it well... I was confous before, I shared the internet connection to 2 networks, LAN and WLAN, but everything is OK now, I also solve my client's http download problem that caused all the other clients can not browse. Your configuration is really good... once more thks..

And if I have another problem in future, pls give a hand.

Thks n Rgds,

Men Fung

trottolino1970 · **Posted:** Tue Sep 25, 2007 2:39 pm

menfung wrote:

Hi Janisk,

I did it well... I was confous before, I shared the internet connection to 2 networks, LAN and WLAN, but everything is OK now, I also solve my client's http download problem that caused all the other clients can not browse. Your configuration is really good... once more thks..

And if I have another problem in future, pls give a hand.

Thks n Rgds,

Men Fung

what mean pubblic in configuration of queue tree

Antixx · **Posted:** Mon Jan 21, 2008 8:16 pm

what should be in parent if we use PPPOE client instead? because after that we would limit the user in simple queue.

carcara · **Posted:** Tue Jan 22, 2008 7:42 pm

i have the same problem... probably the default config. All simple is in this.

menfung · **Posted:** Tue Jan 22, 2008 9:12 pm

trottolino1970 wrote:

what mean pubblic in configuration of queue tree

Sorry.. just read this message...

Public is the name I gave to the interface that connected to the ISP.

thks,

pospanko · **Posted:** Mon Feb 04, 2008 1:30 pm

Why is on Parent Upload and Download queue queue type set to default and not to PCQ_UP/PCQ_DL?

pospanko · **Posted:** Tue Feb 12, 2008 11:18 am

Anyone...?

Chupaka · **Posted:** Thu Feb 14, 2008 3:54 am

pospanko wrote:

Why is on Parent Upload and Download queue queue type set to default and not to PCQ_UP/PCQ_DL?

because PCQ do not affect child queues, so it really makes no matter, what type you set for parent: parent is just to see total traffic

pospanko · **Posted:** Thu Feb 14, 2008 2:01 pm

Ok. Thx mate!

Closed_1 · **Posted:** Thu Feb 28, 2008 10:56 am

cool code dude....

but if i had 3 different client group in my office.... each group the ip listed in adrress list.
ie:
ip1 192.168.1.1
ip1 192.168.1.5
ip1 192.168.1.7
ip2 192.168.1.4
ip2 192.168.1.12
ip2 192.168.1.2
ip3 192.168.1.10
ip3 192.168.1.11
group 1 limit traffic by 96 kbps download 96 kbps upload
group 2 limit traffic by 128 kbps download 64 upload
group 3 limit traffic by 200 kbps download 128 upload

each group using PCQ Equal Share with mangle and queue tree writen in wiki
how to implemented those script to my script?

so i must write 2 connection mark? and 2 queue tree for different group, is that right?

another question.
i had idea to traffic prioritizing in each group, which email traffic (pop3)to be higher priority than http traffic. how to implemented like these to PCQ Equal Share in wiki???

Antixx · **Posted:** Thu Feb 28, 2008 9:34 pm

Dear Friends,

I have tested in my network and there is a nice problem

that may be reason you to dont use this code or you may find the way to solve this problem.

In my situation which i have set the 1024kbps for first 500k and then fall down to 512kbps for special users when a user download a big file after 2 or 3 MB fall down to 512kbps BUT if that person use software like IDM (Internet Download Manager) each time he/she click on STOP and then again START he/she will get 1024kbps speed

ofcouse no one call me to tell me because i didnt use this code generally just for test only and I found this :shock:

Please tell me your opinion about this. it wil be nice if we could find the way to block this kind of trick which users may (actually) will use.

hulk-bd · **Posted:** Thu Feb 28, 2008 11:05 pm

Dear friend Antixx,

Yes there is a problem with download managers, cause if clients stops and starts his/her download after every 1mb then it's starts as a freash (1st rule) rate ratio. and with download manager there has few options like 5 connections for a file download that's why the download bandwidth fluctuate very time between 1st and 2nd pcq rule. you can close per IP connection limites with firewall.

Hope you understand that

Thanks

Antixx · **Posted:** Fri Feb 29, 2008 6:05 pm

hulk-bd wrote:

Dear friend Antixx,

Yes there is a problem with download managers, cause if clients stops and starts his/her download after every 1mb then it's starts as a freash (1st rule) rate ratio. and with download manager there has few options like 5 connections for a file download that's why the download bandwidth fluctuate very time between 1st and 2nd pcq rule. you can close per IP connection limites with firewall.

Hope you understand that

Thanks

I think i understand the problem and the reason. but in my view if i limit the connected to each dst ip to 5 connection this problem will occure again? isnt it?

titius · **Posted:** Sun Mar 09, 2008 10:37 pm

if there is wackos to stop start download every minute or so, then let them do that . . .

whyaskgh · **Posted:** Mon Mar 10, 2008 2:14 am

Please how can this be applied on a mt with load balancing?
Thanks

janisk · **Posted:** Mon Mar 10, 2008 1:01 pm

first i would see that user that stops download after each X Mbit you have set in a download.

please read manual about simple queues and queue tree. if you have several groups, then just add tree nodes to the queue tree and set up configuration for them, so, they function properly, you will have to mark connections and then packets for each type of users, you just add dst-address-list for download and src-address-list for upload for each group.

after that you simple follow the example

Antixx · **Posted:** Fri Mar 14, 2008 1:51 pm

i think it is misunderstanding, with this setup usercan achive more bandwidth by doing stop and start the download manager software, anyway to solve this problem?

it will be great doing this...

Thank you very much in did/

janisk wrote:

first i would see that user that stops download after each X Mbit you have set in a download.

please read manual about simple queues and queue tree. if you have several groups, then just add tree nodes to the queue tree and set up configuration for them, so, they function properly, you will have to mark connections and then packets for each type of users, you just add dst-address-list for download and src-address-list for upload for each group.

after that you simple follow the example

erjon · **Posted:** Mon Mar 31, 2008 3:34 pm

Hi to all. I've tested this configuration on my MT 2.9.51 at work. And i must say is wonderful.But the problem is if i start to download with flashget, configured to download with only 1 connections, this setup is perfect.Sow when i configured flashget to download with 5-10 connections, the download speed vary from 1st rule to the 2nd rule after 1 min and after 3 min vary from 2nd rule to the 1st, and so on.So my opinion is to limit at firewall filter to 1/32 connections for the specific packet_mark old.At this point i see another problem, at firwall filter we can drop only syn connections with packet_mark old, what about those with estabilished type, cause if i start to download with flashget with configuration of 10 connections, at this moment in connections tab of MT is see 10 connections estabilished. How can i close 9 of these?
P.s. sorry for my bad english. my configuration is excast as descripted in this thread.
Best regards to all from Albania.

erjon · **Posted:** Fri Apr 11, 2008 1:23 pm

No one can help at this point?

titius · **Posted:** Wed May 14, 2008 2:59 am

Where have all posts gone??!!

0ldman · **Posted:** Wed Aug 13, 2008 8:34 am

Bump.

I've been using this, works great, however, does TCP have to be identified in the mangle rules?

I'm having a hard time shaping other traffic separately. If I try to use mangle to prioritize port 80 and give other ports a 256k limit, PCQ just stops.

In order to shape P2P, I have to use another RB. I've been doing it on the client side, works well as I only have two problem clients, but I'd like to have a blanket rule so some new customer that discovers P2P won't go completely nuts on the network.

edit: tinkering with my 2nd AP in line, first doing this as the primary, it appears that choosing the TCP protocol is not required. Any input from those more experienced would be greatly appreciated.

dunga · **Posted:** Sun Oct 11, 2009 10:22 pm

What of for hotspot users.

I have 3 interfaces,
ether1 = wan- isp
ether2 = hotspot ip range: 172.16.0.0/24
ether3 = local users = ip range : 192.168.20.0/24

How do i go about it

lukkes · **Posted:** Fri Jan 15, 2010 8:27 am

0ldman wrote:

Bump.

I've been using this, works great, however, does TCP have to be identified in the mangle rules?

I'm having a hard time shaping other traffic separately. If I try to use mangle to prioritize port 80 and give other ports a 256k limit, PCQ just stops.

In order to shape P2P, I have to use another RB. I've been doing it on the client side, works well as I only have two problem clients, but I'd like to have a blanket rule so some new customer that discovers P2P won't go completely nuts on the network.

edit: tinkering with my 2nd AP in line, first doing this as the primary, it appears that choosing the TCP protocol is not required. Any input from those more experienced would be greatly appreciated.

I have the same problem, i make a lot of rules to mark, ssh,p2p, pop, etc. and make a queue for prioritize it, then mark it in postrouting and prerouting, works perfect, but when i setup this or the Janis mangle example from the mum that marks connections in forward the apply PCQ, the pcq simply doesnt work.. somebody can help us? thanks

rmichael · **Posted:** Fri Jan 15, 2010 8:42 pm

Maybe you are overwriting forward chain marks with postrouting marks?

lukkes · **Posted:** Sat Jan 16, 2010 12:11 am

well, the forward chain marks connection based on src-adress-list but the prerouting chains marks packets based on src and dst ports. alos mark some connection to use the connection-bytes but its marked also by src port 80.. any idea?

wifi442 · **Posted:** Fri Mar 26, 2010 12:04 am

erjon wrote:

Hi to all. I've tested this configuration on my MT 2.9.51 at work. And i must say is wonderful.But the problem is if i start to download with flashget, configured to download with only 1 connections, this setup is perfect.Sow when i configured flashget to download with 5-10 connections, the download speed vary from 1st rule to the 2nd rule after 1 min and after 3 min vary from 2nd rule to the 1st, and so on.So my opinion is to limit at firewall filter to 1/32 connections for the specific packet_mark old.At this point i see another problem, at firwall filter we can drop only syn connections with packet_mark old, what about those with estabilished type, cause if i start to download with flashget with configuration of 10 connections, at this moment in connections tab of MT is see 10 connections estabilished. How can i close 9 of these?
P.s. sorry for my bad english. my configuration is excast as descripted in this thread.
Best regards to all from Albania.

I was having the same problem as mentioned here until I used our new friend connection-rate :-)

I set the connection rate limit for each connection to be just over the connection speed of the non burst queue. It seems to be working so far. If multiple downloads are started with a download manager, each individual connection never reaches the speed specified in connection-rate so they stay in the non-burst queue. Its working beautifully so far.

Here is a small preview :-)

Code:

add action=mark-connection chain=forward comment="" connection-bytes=\
    0-2000000 connection-mark=JetWire_Conn_1m connection-rate=2M-100M \
    disabled=no new-connection-mark=JetWire_Conn_1m_Burst passthrough=yes \
    protocol=tcp src-address-list="PPPoE Private 1M Service"

Burst for each TCP connection

Who is online