Community discussions

MikroTik App
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

How to handle "download managers"?

Sun Mar 14, 2010 6:19 am

Is there any way to detect and QoS download managers, such as "Free Download Manager"?
They can easily open tons of connections with pretty impressive download rate, and hog as much bandwidth as they can.

Is it possible to detect and queue this sort of traffic?
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Sun Mar 14, 2010 5:55 pm

Have you checked the layer7 protocols? Maybe you can find sth there

http://l7-filter.sourceforge.net/protocols
-Toni-
Don't crash the ambulance, whatever you do
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Sun Mar 14, 2010 7:43 pm

No, nothing in there.

I was under impression that L7 is not a good idea for smaller routerboards (mine is 450G), as it will hog CPU and affect the overall performance - am I wrong here?
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Sun Mar 14, 2010 11:27 pm

I think you are wrong here :)
Of course it will use more CPU, but that doesn't mean it will hog it.

is this "free download manager" using a given port?
If yes, you can capture it's traffic using the port, other wise I will
suggest to have a look at the "conection rate"

http://wiki.mikrotik.com/wiki/Connection_Rate
-Toni-
Don't crash the ambulance, whatever you do
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: How to handle "download managers"?

Mon Mar 15, 2010 12:16 am

You can also use connection-limit to set a limit on how many connections each user can have.
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Mon Mar 15, 2010 12:32 am

is this "free download manager" using a given port?
If yes, you can capture it's traffic using the port, other wise I will
suggest to have a look at the "conection rate"
There are quite a few such tools, but this one is one of more popular.
http://www.freedownloadmanager.org

I suppose it does regular HTTP downloads, but it opens many connections on the same file loading different parts simultaneously. And it usually works with few files at the same time - total is pretty impressive.

I don't really want to limit it permanently, but rather to slow it down dynamically when necessary.
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Mon Mar 15, 2010 12:56 am

Than, the example of wiki about the connection-rate is definitly for this case
-Toni-
Don't crash the ambulance, whatever you do
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24944
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: How to handle "download managers"?

Mon Mar 15, 2010 8:24 am

I don't understand how is download manager different from any other web activity? Either limit your customer to certain total speed, certain max download limit, or certain connection number.
No answer to your question? How to write posts
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Mon Mar 15, 2010 1:48 pm

In my opinion there is a difference between normal web browsing and web downloading.
Web browsing is short in time and not that big, that means it's connection are short,
from time to time, with small quantities of packets.
Web downloading instead, lasts as long as it needs to download the file, at it will grab the
whole bandwidth available.

Now, in a situation when you give priorities, high priority to web browsing and lower to
download (using connection-rate) you would expect that the webpage will open as soon as
you click on it, even if in the meantime there is a download going on. I have seen this is not true.
This is a situation like when you try to pass through a crowded door. When there are a lot of people
trying to pass through, even if the doorkeeper is a "friend" of yours, he still needs to see you
so that he can pass you before the others.

Let me explain with to different configuration of queue tree:
the first one
1 name="download" parent=ether1 limit-at=0 priority=8 max-limit=1M
burst-limit=0 burst-threshold=0 burst-time=0s

2 name="other_download" parent=download packet-mark=other_traffic
limit-at=768k queue=PcqDown priority=1 max-limit=1M burst-limit=0
burst-threshold=0 burst-time=0s

3 name="heavy_download" parent=download packet-mark=heavy_traffic
limit-at=128k queue=PcqDown priority=8 max-limit=1M burst-limit=0
burst-threshold=0 burst-time=0s
the second one

1 name="download" parent=ether1 limit-at=0 priority=8 max-limit=1M
burst-limit=0 burst-threshold=0 burst-time=0s

2 name="other_download" parent=download packet-mark=other_traffic
limit-at=768k queue=PcqDown priority=1 max-limit=1M burst-limit=0
burst-threshold=0 burst-time=0s

3 name="heavy_download" parent=download packet-mark=heavy_traffic
limit-at=128k queue=PcqDown priority=8 max-limit=896k burst-limit=0
burst-threshold=0 burst-time=0s

As you can see, in the second configuration, I leave always a little room for packets
to pass, although there could be a heavy download going on. Untill now this is working better.
-Toni-
Don't crash the ambulance, whatever you do
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Mon Mar 15, 2010 3:48 pm

@normis
Download managers use different strategy comparing to browsing or "manual" download - their very purpose is to make max use of all available bandwidth to minimize the download time. Take into account a typical use case: some fifty 200kB files queued, 6-8 files at a time, each of then accessed by 6-10 connections at max speed possible. I did some tests and saw 30-35mbps on WAN interface just from that, not even sure it is the maximum - on our 50mbps link. And usually the person has no idea what he's doing to other users.

@Caci99
Let me see if I understand it right - in the second example you decreased the queue buffer size, thus decreasing the time to free the way for higher priority traffic - is it correct?
I would really appreciate if you can type a couple of words on limit-at and max-limit - what they really affect. Is there a downside of minimizing the buffer?

Couldn't you use Bursts to jump start http pages?
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24944
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: How to handle "download managers"?

Mon Mar 15, 2010 3:51 pm

edmidor, if you limit this guy at, say, 2mbps. what is the difference if he uses one or 10 connections?
No answer to your question? How to write posts
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Mon Mar 15, 2010 5:11 pm

That probably would do, if I can detect his activity, slow him down to 2mbps, and after he finished 'abusing' the connection remove that limit - that what I'm looking for. The only question is "how".
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Mon Mar 15, 2010 5:52 pm

What I posted was just a snip of the "Connection rate" configuration.
This configuration is very good in situation where heavy download disturbs
normal internet operation. I strongly suggest you to read the wiki about connection rate.

With this configuration, you will be able to divide the heavy download, whatever it is, P2P,
download manager, youtube etc., from the "normal" web browsing, Voip etc.

In what I have posted, you will see that there is a queue that acts as parent with limit to 1MB,
and there are two other child queues with the same limit, but with different priorities.
Later, I changed a little bit this configuration, living always a space of 128kbps to the child queue
with the highest priority.

If you read the wiki you will understand me better.
-Toni-
Don't crash the ambulance, whatever you do
 
Steve0
newbie
Posts: 37
Joined: Mon Feb 08, 2010 8:34 pm
Location: Somewhere between the Bit and the Byte

Re: How to handle "download managers"?

Mon Mar 15, 2010 7:08 pm

http://mum.mikrotik.com/presentations/C ... _Megis.pdf

Just put the user in a class of "abusers"
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Tue Mar 16, 2010 5:49 am

What I posted was just a snip of the "Connection rate" configuration.
This configuration is very good in situation where heavy download disturbs
normal internet operation. I strongly suggest you to read the wiki about connection rate.

With this configuration, you will be able to divide the heavy download, whatever it is, P2P,
download manager, youtube etc., from the "normal" web browsing, Voip etc.

In what I have posted, you will see that there is a queue that acts as parent with limit to 1MB,
and there are two other child queues with the same limit, but with different priorities.
Later, I changed a little bit this configuration, living always a space of 128kbps to the child queue
with the highest priority.

If you read the wiki you will understand me better.
I read wiki. Now, given my connection numbers 50mbps down and 1 up - is it effective applied on download at all?
Should I create queues for upload only, or for both?
At my worth tests I never seen download traffic reaching 50mbps, but I think 1mbps upstream is easily saturated.
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Tue Mar 16, 2010 12:37 pm

Your 1Mbps upload related to the 30Mbps real download is terribly low.
This internet line is so asymmetric! But this is not the issue we are discussing right now.

I think you should really implement that configuration, so that no one could grab anymore the
whole bandwidth available as long as you are sharing it for the all users.
The other way is to specify simple queues for each user. You choose.

My advice:
Always backup current configuration before any change.

/ip firewall mangle
add action=mark-connection chain=forward comment="Connection rate" \
connection-mark=!heavy_traffic disabled=no new-connection-mark=all_con \
passthrough=yes protocol=tcp
add action=mark-connection chain=forward comment="" connection-bytes=500000-0 \
connection-mark=all_con connection-rate=50k-100M disabled=no \
new-connection-mark=heavy_traffic passthrough=yes protocol=tcp
add action=mark-connection chain=forward comment="" connection-bytes=500000-0 \
connection-mark=all_con connection-rate=80k-100M disabled=no \
new-connection-mark=heavy_traffic passthrough=yes protocol=udp
add action=mark-packet chain=forward comment="" connection-mark=heavy_traffic \
disabled=no new-packet-mark=heavy_traffic passthrough=no
add action=mark-packet chain=forward comment="" connection-mark=\
!heavy_traffic disabled=no new-packet-mark=other_traffic passthrough=no

/queue type
add name="PcqDown" kind=pcq pcq-rate=0 pcq-limit=50 pcq-classifier=dst-address
pcq-total-limit=2000

add name="PcqUp" kind=pcq pcq-rate=0 pcq-limit=50 pcq-classifier=src-address
pcq-total-limit=2000

/queue tree
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
max-limit=1M name=upload parent=wlan1 priority=6
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=512k \
max-limit=1M name=other_upload packet-mark=other_traffic parent=upload \
priority=1 queue=PcqUp
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=512k \
max-limit=1M name=heavy_upload packet-mark=heavy_traffic parent=upload \
priority=6 queue=PcqUp

add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
max-limit=50M name=download parent=ether1 priority=8
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=30M \
max-limit=50M name=other_download packet-mark=other_traffic parent=\
download priority=1 queue=PcqDown
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=20M \
max-limit=28M name=heavy_download packet-mark=heavy_traffic parent=\
download priority=8 queue=PcqDown

Note: wlan1 is the out interface and ether1 is the LAN interface. I have left default values
por the pcq queues in /queue type. You should correct them as your network requests it.
You should also be sure if your download is 30M or 50M and change the values in the
queue limits as needed.
-Toni-
Don't crash the ambulance, whatever you do
 
Ozelo
Member
Member
Posts: 338
Joined: Fri Jun 02, 2006 3:56 am

Re: How to handle "download managers"?

Tue Mar 16, 2010 10:18 pm

Something seems not good on wireless when some programs (i.e. torrent) start downloading files through many connections that transport a very small part of it with a lot of packets. That obviously make wireless more busy (more overhead than data throughput) than any other common connection like web surfing. There is a game server I know that keep about 40 packets @ 100 bytes each per second per user connected, quiet troublesome on wireless Ive seen.

Still didn't know a better way to handle this on wireless than moving such kind of user from 2.4g to 5.8g. Then web surfing users and gamers stop complaining were both of them was together on 2.4g... Any other method we tried just caused one or another to get its connectivity and performance bad. Also, didn't ever experience any of these problems on a 10/100 ethernet medium user.
MTCRE - 1104RE006
MTCINE - 1104INE001
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Wed Mar 17, 2010 3:20 am

Thanks Toni!
How can I make sure outbound VPN connection (Cisco VPN client) won't fall under 'heavy traffic'?
I'm not sure about the rate used by VPN, but I suppose it can be fast, and it lasts long...
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Wed Mar 17, 2010 12:41 pm

Unfortunately I have not worked that much with VPN, so I am not sure about their behaviour,
But i guess you can put a mangle rule at the top of the others with action "accept" sth like:

/ip firewall mangle
add chain=prerouting src-addres=VPN-IP action=accept

this way it's traffic will not be processed any more.
-Toni-
Don't crash the ambulance, whatever you do
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Thu Mar 25, 2010 4:44 am

Running with your config for few days - VoIP works even with heavy download - thanks! :)

Question - how do you decide on limit-at value? Any general rules?
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Thu Mar 25, 2010 11:24 am

Take a look at this topic:
http://forum.mikrotik.com/viewtopic.php?f=2&t=39998

The limit-at is about priority and QoS
and, I'm glad it is working for you :)
-Toni-
Don't crash the ambulance, whatever you do
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Sat Mar 27, 2010 6:03 am

Should've read it from the start. Much better now :)

The remaining question is - theoretical max limit vs. actual throughput: if I have two upload queues with limit-at 500K each, on 1Mbps max at parent, and the actual throughput falls from 1M to 500K... how router would handle it? It guaranteed 500K to each, but connection can only transfer 500K in total...
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Sat Mar 27, 2010 10:23 am

"... You can't always get what you want..." :wink:
-Toni-
Don't crash the ambulance, whatever you do
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Sat Apr 03, 2010 5:39 am

I monitored my qos setup over last week, what I noticed is that marking connections based on connection-bytes with connection-rate isn't good enough when connections are short lived - download manager opens dozens of connections for every file, and there are many of them (think 50-80 100M files in a queue, which goes pretty fast). To mark a new connection as "heavy" it takes some 500K download with regular priority, and as new connections created often and in large quantities all those first 500Ks slow down regular HTTP traffic noticeably.

Need some more ideas on how to catch multiple connections heavy stuff earlier...
 
kazanova
Member
Member
Posts: 407
Joined: Tue Sep 06, 2005 11:52 am

Re: How to handle "download managers"?

Sat Apr 03, 2010 12:01 pm

u need to drop each programme u know it :D it hard but it only way
and cloth p2p conection
انا انزلنا التوراه فيها هدى ونور يحكم بها النبيون الذين اسلموا للذين هادوا والربانيون والاحبار بما استحفظوا من كتاب الله وكانوا عليه شهداء فلا تخشوا الناس واخشون ولا تشتروا باياتي ثمنا قليلا ومن لم يحكم بما انزل الله فاولئك هم الكافرون
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Sat Apr 03, 2010 12:26 pm

You can try to introduce connection-limit also.

I am not a fan of dropping some application, first of all that's not 100% accurate,
and soon you will find yourself in a deeper swamp of configuration not acting as expected.
-Toni-
Don't crash the ambulance, whatever you do
 
WirelessRudy
Forum Guru
Forum Guru
Posts: 3111
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: How to handle "download managers"?

Sat Apr 03, 2010 5:26 pm

Your 1Mbps upload related to the 30Mbps real download is terribly low.
This internet line is so asymmetric! But this is not the issue we are discussing right now.
1/30 is a very real ADSL line. I can't get anything else in Spain.
I have different providers and BEST I get is 1/30. Real througput is even worse. (1/35)
 
WirelessRudy
Forum Guru
Forum Guru
Posts: 3111
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: How to handle "download managers"?

Sat Apr 03, 2010 8:31 pm

I monitored my qos setup over last week, what I noticed is that marking connections based on connection-bytes with connection-rate isn't good enough when connections are short lived - download manager opens dozens of connections for every file, and there are many of them (think 50-80 100M files in a queue, which goes pretty fast). To mark a new connection as "heavy" it takes some 500K download with regular priority, and as new connections created often and in large quantities all those first 500Ks slow down regular HTTP traffic noticeably.

Need some more ideas on how to catch multiple connections heavy stuff earlier...
Hmmm, I don't think "earlier" is possible. When connections are started router has no idea how long or how big these are going to be.

Another approach maybe:
Normally we use pcq's with classifier dst or src address only. So the queue is grouping all srcIP/port-dstIP/port connections in one queue as long as the src IP is the same. (And thus all coming from one client.)
If now the mange filters all port 80 traffic to give it a conn.marker and a package marker the queue tree then uses a queue type to put all these connections in. Normally we have pcq-type-rate set at "0" to allow all connections to balance their load inside that queue and we set max.rate in the child of the queue tree (and prioritise) to limit ALL similar (= port 80) connections from ALL clients.

So if indeed already plenty connections from one client are made and put in that queue belonging that client new connections have a hard time to enter.

What happens now if we would put src-port as an extra classifier? So we make a new queue type with src-port and src-IP each time a new connection is made? Now we create multiple pcq's from one client each time with a different src.port. Port 80 connections from client if made by download manager will only differ in src-port.

By also separating queues based on src port, each new connection doesn't have to compete with other already existing queues, but only with other queues (from other clients) in the queue tree. And here we can rate limit and prioritize with the help of mangle connection rate and size matcher?

How will it work:
Download manager of client opens several connections.
Conn.tracker creates connections based on src-IP/Port-dst-IP/Port combinations. Since each connection from download manager has different src-port different connections are registred by conn.tracker.
Mangle ´sees´ these all and since they are all port 80 gives them "http" conn.mark.
Now several streams of packages labeled with conn.mark "http" is created for that client.
All these streams are now matched by package marker with matcher "conn.mark" and given package mark "HTTP"
So now we have several streams (from each client, and from many clients) all with a package mark "HTTP".
So far normal mangle for QoS setup.

But now we want to distinguish within this group of connections the slow or short lived ones from the fast and/or lasting onces.
So we have to mangle again the stream of “HTTP” labeled packages and with the help of matcher "connection-bytes" and "connection-rate" we give different new package marks. Like "HTTP_normal" and HTTP-Heavy".

Now we set queue tree.
We have parent, limiting to total physical available bandwith.
Then we create childs:

First child will put all port 80 traffic in a queue that has package marker "HTTP_normal" and give it a normal priority (5) and we set limit-at rate for bandwidth guarantee and max. limit for max. throughput limit. (Set it to same max.limit as parent. See other topic for this)
Now we also make a new child for traffic that in mangle has been given a package mark "HTTP-Heavy". Since we want to give this traffic lower priority over "Http-normal" we give it a lower priority. We can also give it some lower guaranteed bandwidth so it has to use priority in competition with other traffic sooner. We can even give lower max. rate then other so it will never completely fill the pipe.

regarding the queue types we use:
For normal traffic we use normal pcq type with src.address classifier only. For ALL port 80 connections (new for browsing or new for download) it guarantees that all these connections share the total bandwidth of this child equally with unlimited speed (rate=0). (The total limiting is done in the child for all these connections together and also by client shaping for each client individually.)

Now, if some of these connections hit connection-rate and size matcher in mangle their packages suddenly get a different label "HTTP-Heavy".
Since the normal child made for normal browsing looks only to package marked "HTTP-normal" they disappear now from this child. So we have to make new child.
This one now has to look for the label "HTTP-Heavy". We also change the queue type. Here we make a new queue type, and we give it the classifiers src.address AND port so each connection now suddenly gets its own queue.
This child gets a lower priority so new “same kind of traffic but not heavy yet” (0HTTP-normal) get preference over these.
We can now also further limit the speed of these existing but newly labeled package streams by setting a rate limit in the queue. Since each different src.port even from same client, now has its own connection, and thus queue, we can limit this queue. We can set it to any limit we can as long as it is lower then the total available for all port 80 traffic for that client. (If not only priority would make a difference.)

I think this way you can even make several steps in limiting long living connections.


I don’t know if I am right in all this. Maybe I am overlooking something and I also did not try this setup yet.
So any input, even if it make me look stupid, is still appreciated!
 
User avatar
MCT
Member Candidate
Member Candidate
Posts: 157
Joined: Wed Mar 03, 2010 5:53 pm

Re: How to handle "download managers"?

Sun Apr 04, 2010 1:35 am

I'm a rather strong believer in net neutrality. If someone is paying me for a certain download speed then they have every right to do whatever they want with the speed that they payed for. It's also a matter of professional ethics. If someone is paying you their money for a certain download rate then you should give it to them and not say 'oh you have a 1mb connection except when you try to download stuff then I'm going to throttle you down.' The only time you really get to use the bandwidth you pay for is downloading. If you're overselling your bandwidth that's just a bad business practice.

The way around that is to say you have a 256kb connection burstable to 6mb so web pages will download fast but extended downloads will drop down to their payed bandwidth. That way you are not misrepresenting your service.
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Tue Apr 06, 2010 5:00 am

I don't know how many times I've read it... took a while to digest :)

The problem seems to be still there - there will be many heavy connections marked as regular and waiting to be remarked as heavy. But while all of them are waiting the regular queue will be overloaded, which directly translates into slowness of browsing.

Said that, in default PCQ queue config packets compete within the same queue. When done like you suggested there will be a bunch of queues competing for the same bandwidth. What's the advantages of second way? You can hard limit the rate, but I guess you don't want to, because if no one is browsing at some moment what's the point to limit downloads and waste bandwidth? Heavy stuff should be limited only when necessary, not always...
 
WirelessRudy
Forum Guru
Forum Guru
Posts: 3111
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: How to handle "download managers"?

Tue Apr 06, 2010 5:49 pm

I'm a rather strong believer in net neutrality. If someone is paying me for a certain download speed then they have every right to do whatever they want with the speed that they payed for. It's also a matter of professional ethics. If someone is paying you their money for a certain download rate then you should give it to them and not say 'oh you have a 1mb connection except when you try to download stuff then I'm going to throttle you down.' The only time you really get to use the bandwidth you pay for is downloading. If you're overselling your bandwidth that's just a bad business practice.

The way around that is to say you have a 256kb connection burstable to 6mb so web pages will download fast but extended downloads will drop down to their paid bandwidth. That way you are not misrepresenting your service.
Well, everyone can buy a car and can do with it what he wants..... as long as you stay within the laws on driving cars. Why these laws? Well, without them it will become a massacre on the roads.
Same with networks. If everybody is allowed to do what he wants it will ruin others usage on the net, and as provider, your income.

Internet usage is almost ALWAYS a shared one. Providers selling 1:1 while connected 1:1 to the backbone don't exist. That would require a business model impossible to budget for.
I sell my clients 3Mb download. I have 200 clients. Do I have 600Mb download capacity? No, I buy 30Mb from the national provider. Does he sell 6Mb (what they promote as normal client speed) and deliver that to all their clients? I believe they have over 15 million clients. Times 6Mb... how much would that work out to be?
No, everybody works with a contention rate. And to have your network running as smooth as possible within its boundaries of capacities while clients will have the best possible service you have to do QoS on it. Which comes with limits and preferences.
Just like the police car and driving rules. It might slow you down at times but in general they are a benefit to us all.

If you sell a bandwidth to a client it is not the same as a guarantee he will get that speed. It is always a max. limit.
In your opinion a client could even complain about slow download to you as contracted provider for not delivering while actually the hosting file server is a slow one. What can YOU do about that?
 
WirelessRudy
Forum Guru
Forum Guru
Posts: 3111
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: How to handle "download managers"?

Tue Apr 06, 2010 6:56 pm

I don't know how many times I've read it... took a while to digest :)

The problem seems to be still there - there will be many heavy connections marked as regular and waiting to be remarked as heavy. But while all of them are waiting the regular queue will be overloaded, which directly translates into slowness of browsing.

Said that, in default PCQ queue config packets compete within the same queue. When done like you suggested there will be a bunch of queues competing for the same bandwidth. What's the advantages of second way? You can hard limit the rate, but I guess you don't want to, because if no one is browsing at some moment what's the point to limit downloads and waste bandwidth? Heavy stuff should be limited only when necessary, not always...
We actually talking about two types of queues here:
1. The traffic that is put in a child making part of the queue tree. this is traffic (depending on given mark) from ALL users together. So you will find queues from all clients in this child-queue.
2. In this child queue traffic is also separated by their src and dst address and port in the queue types. This is normally spoken for EACH of the client.

Normally we limit (max.limit) TOTAL throughput for certain service of all clients together in the queue tree while we CAN limit each underlying client attached queue (queue type). Normal we use standard PCQ with rate limit "0" which then just guarantees that all available rate for that child queue gets distributed over AL underlying queues. So over all underlying clients data streams.
We can now also limit these underlying queues for each IP/port-IP/port combination (=connection) by setting their max. rate.

What we establish now is that traffic is placed in the normal child queue for port 80 traffic and it get the standard PCQ and traffic is balanced over all users.
Then, if one connection becomes "heavy" over time (we decide when in mangle) it receives a new package marker, and the queue tree then places this new connection in another child queue.
This last child queue now has queue type set for each individual connection based on dst-port as classifier as well.
So this child will be filled now with more, but smaller, queues then first. If max. limit for this child is set same as max. limit for the original child queue it still receives the same amount of bandwidth.
If we now would maintain same priority setting for this child as the previous one there would be not so much difference.
But if we now give this new child a lower priority then the previous one, we guarantee that newly made port 80 connections start with higher priority then already running "heavy" ones.

On the other hand I do realise that if a download manager would open many, many connections at once and then another normal browsing connection at the same time will suffer from it. I'll guess you have to set the time elapse for matching "heavy" traffic will have to be set as small as possible. The side effect that will have is that some big web pages will probably also fall in that matcher now.
Also, some web pages are containing so much information that some of its opening connections bear more traffic then a download manager will receive on its connections while trying to download a small file.....
So it will still have to be balanced out what the best settings will be...
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1073
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: How to handle "download managers"?

Tue Apr 06, 2010 9:44 pm

I'm a rather strong believer in net neutrality. If someone is paying me for a certain download speed then they have every right to do whatever they want with the speed that they payed for. It's also a matter of professional ethics. If someone is paying you their money for a certain download rate then you should give it to them and not say 'oh you have a 1mb connection except when you try to download stuff then I'm going to throttle you down.' The only time you really get to use the bandwidth you pay for is downloading. If you're overselling your bandwidth that's just a bad business practice.

The way around that is to say you have a 256kb connection burstable to 6mb so web pages will download fast but extended downloads will drop down to their payed bandwidth. That way you are not misrepresenting your service.
I agree with you, somehow. This topic is about making life easier to the client whatever his limits are. For example, this configurations are optimal
for companies which have 10 or 100 of PC inside no matter if the ISP has guaranteed his Down/Up limits.
Also as wirelesrudy mentioned, this is difficult from the point of view of ISP. In my country their is one ISP who actually operates this way
and they are selling 1Mbit at (hold your breath) 300Euros/month
But I agree that if you guarantee to the user Down/Up limit, then it is all up to him how he manages that bandwidth.
-Toni-
Don't crash the ambulance, whatever you do
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Wed Apr 07, 2010 12:19 am

Most support net neutrality. But it usually presumes going against shaping for business/political reasons (p2p shaping by Comcast is a good example); technical traffic shaping always was there, and usually isn't really felt by users as it merely balances traffic.
 
WirelessRudy
Forum Guru
Forum Guru
Posts: 3111
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: How to handle "download managers"?

Wed Apr 07, 2010 12:51 am

Most support net neutrality. But it usually presumes going against shaping for business/political reasons (p2p shaping by Comcast is a good example); technical traffic shaping always was there, and usually isn't really felt by users as it merely balances traffic.
Well, if that is what we name it "technical traffic shaping" then that is what we do.
I have no objections on what my users do on my network. As long as it does not bring me in conflict with my government and as long as I am able to ´shape´ my network to my best so the users can expect the best I can offer.

We inform new clients that P2P can give problems on a wireless network and that we shape the upload mostly. On the download we try to treat it like any other download. The result is that we have several P2P users that are happy and they don't hamper my network because they are pretty much under control.
But I also had some complaints from user that said their torrents are not coming in any more because they have been downloading too much in comparison to their seeding. Well, to bad for them but I am not giving them better upload and put my limited upload capacity in jeopardy only because some guys want to watch otherwise illegal content. (Usually UK citizens watching TV through proxies, which is not allowed according their own country law.)
I don't need to know what they exactly do, but they can't force me to give them any assistance in what they want.
 
edmidor
Member Candidate
Member Candidate
Topic Author
Posts: 126
Joined: Fri Mar 05, 2010 12:06 am
Location: Canada
Contact:

Re: How to handle "download managers"?

Wed Apr 07, 2010 2:57 am

So this child will be filled now with more, but smaller, queues then first
I really don't get what's the point of doing that
But if we now give this new child a lower priority then the previous one, we guarantee that newly made port 80 connections start with higher priority then already running "heavy" ones.
That's what I had with the initial setup - all connections created with medium priority, and after some conn-bytes at conn-rate some of them get downgraded to to low priority. I'm not sure I understand what we can gain by creating individual queues per src and dest combinations though - overall bandwidth distribution remains the same, doesn't it? And main problem is with very fact that all of them are initially in medium priority bucket, screaming and kicking at each other.
 
hendra
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Wed Nov 22, 2017 2:21 am

Re: How to handle "download managers"?

Sat Jun 19, 2021 10:22 am

I monitored my qos setup over last week, what I noticed is that marking connections based on connection-bytes with connection-rate isn't good enough when connections are short lived - download manager opens dozens of connections for every file, and there are many of them (think 50-80 100M files in a queue, which goes pretty fast). To mark a new connection as "heavy" it takes some 500K download with regular priority, and as new connections created often and in large quantities all those first 500Ks slow down regular HTTP traffic noticeably.

Need some more ideas on how to catch multiple connections heavy stuff earlier...
Hmmm, I don't think "earlier" is possible. When connections are started router has no idea how long or how big these are going to be.

Another approach maybe:
Normally we use pcq's with classifier dst or src address only. So the queue is grouping all srcIP/port-dstIP/port connections in one queue as long as the src IP is the same. (And thus all coming from one client.)
If now the mange filters all port 80 traffic to give it a conn.marker and a package marker the queue tree then uses a queue type to put all these connections in. Normally we have pcq-type-rate set at "0" to allow all connections to balance their load inside that queue and we set max.rate in the child of the queue tree (and prioritise) to limit ALL similar (= port 80) connections from ALL clients.

So if indeed already plenty connections from one client are made and put in that queue belonging that client new connections have a hard time to enter.

What happens now if we would put src-port as an extra classifier? So we make a new queue type with src-port and src-IP each time a new connection is made? Now we create multiple pcq's from one client each time with a different src.port. Port 80 connections from client if made by download manager will only differ in src-port.

By also separating queues based on src port, each new connection doesn't have to compete with other already existing queues, but only with other queues (from other clients) in the queue tree. And here we can rate limit and prioritize with the help of mangle connection rate and size matcher?

How will it work:
Download manager of client opens several connections.
Conn.tracker creates connections based on src-IP/Port-dst-IP/Port combinations. Since each connection from download manager has different src-port different connections are registred by conn.tracker.
Mangle ´sees´ these all and since they are all port 80 gives them "http" conn.mark.
Now several streams of packages labeled with conn.mark "http" is created for that client.
All these streams are now matched by package marker with matcher "conn.mark" and given package mark "HTTP"
So now we have several streams (from each client, and from many clients) all with a package mark "HTTP".
So far normal mangle for QoS setup.

But now we want to distinguish within this group of connections the slow or short lived ones from the fast and/or lasting onces.
So we have to mangle again the stream of “HTTP” labeled packages and with the help of matcher "connection-bytes" and "connection-rate" we give different new package marks. Like "HTTP_normal" and HTTP-Heavy".

Now we set queue tree.
We have parent, limiting to total physical available bandwith.
Then we create childs:

First child will put all port 80 traffic in a queue that has package marker "HTTP_normal" and give it a normal priority (5) and we set limit-at rate for bandwidth guarantee and max. limit for max. throughput limit. (Set it to same max.limit as parent. See other topic for this)
Now we also make a new child for traffic that in mangle has been given a package mark "HTTP-Heavy". Since we want to give this traffic lower priority over "Http-normal" we give it a lower priority. We can also give it some lower guaranteed bandwidth so it has to use priority in competition with other traffic sooner. We can even give lower max. rate then other so it will never completely fill the pipe.

regarding the queue types we use:
For normal traffic we use normal pcq type with src.address classifier only. For ALL port 80 connections (new for browsing or new for download) it guarantees that all these connections share the total bandwidth of this child equally with unlimited speed (rate=0). (The total limiting is done in the child for all these connections together and also by client shaping for each client individually.)

Now, if some of these connections hit connection-rate and size matcher in mangle their packages suddenly get a different label "HTTP-Heavy".
Since the normal child made for normal browsing looks only to package marked "HTTP-normal" they disappear now from this child. So we have to make new child.
This one now has to look for the label "HTTP-Heavy". We also change the queue type. Here we make a new queue type, and we give it the classifiers src.address AND port so each connection now suddenly gets its own queue.
This child gets a lower priority so new “same kind of traffic but not heavy yet” (0HTTP-normal) get preference over these.
We can now also further limit the speed of these existing but newly labeled package streams by setting a rate limit in the queue. Since each different src.port even from same client, now has its own connection, and thus queue, we can limit this queue. We can set it to any limit we can as long as it is lower then the total available for all port 80 traffic for that client. (If not only priority would make a difference.)

I think this way you can even make several steps in limiting long living connections.


I don’t know if I am right in all this. Maybe I am overlooking something and I also did not try this setup yet.
So any input, even if it make me look stupid, is still appreciated!
i think this is good thanks man

Who is online

Users browsing this forum: No registered users and 75 guests