Community discussions

MikroTik App
 
1littlewisp
newbie
Topic Author
Posts: 36
Joined: Wed Jun 10, 2009 6:23 pm

[SOLVED]An Interesting Challenge... ~or~ Choking Youtube...

Thu Jun 02, 2011 11:05 pm

So I've got this client with a seemingly innocent request. He doesn't want to block Youtube outright, he just wants to queue traffic down so that the video streams don't kill his 3Mbps connection. Simple enough, right? Here's the solution I had come up with:

1) We can't throttle Youtube by name, but we can throttle the IP. First, we set up a script to resolve hostnames and put them in an address list:
# define variables
:local list
:local hosts
:local newip

# Loop through each entry in the address list.
:foreach i in=[/ip firewall address-list find] do={

# Get the first four characters of list name
  :set list [:pick [/ip firewall address-list get $i list] 0 4]

# Condition: if the value of $list is "dns_" process it.
  :if ($list = "dns_") do={

# Of the items being processed, store their "comment" fields as the variable "hosts"
    :set hosts [/ip firewall address-list get $i comment]

# Resolve it and set the address list entry accordingly.
    :set newip [:resolve $hosts]
    /ip firewall address-list set $i address=$newip
    }
  }
*Credit goes to Paul Gu for this idea: http://wiki.mikrotik.com/wiki/Use_host_ ... wall_rules

2) Now we make the list and put an entry in it for Youtube. The way that script works is that you just put the FQDN in the comment field of the list entry, and script does dns lookups based on that.
/ip firewall address-list add address=0.0.0.0 comment=youtube.com list=dns_choke
Now we invoke the script from before, it will resolve and update the IP for Youtube.

3) The next step is to make some firewall rules to mark all packets with .flv or .mp4 content coming from the IP we have in the address list:
/ip firewall mangle add chain=forward protocol=tcp src-address-list=dns_choke content=.flv action=mark-packet new-packet-mark=choked_video comment="Mark .flv content from address list for queuing"

/ip firewall mangle add chain=forward protocol=tcp src-address-list=dns_choke content=.mp4 action=mark-packet new-packet-mark=choked_video comment="Mark .mp4 content from address list for queuing"
4) Finally, we create the queue:
/queue simple add name="Youtube Video Content" max-limit=64k/64k packet-marks=choked_video
I was psyched about this solution, but it does not work. The mangle rule hasn't processed any packets whatsoever. I noticed that when I ran the dns lookup script multiple times in a row, it kept resolving a different last octet for Youtube.

...of course. Youtube uses load distribution through DNS (such as the round-robin technique.) Since the IP could be any of a number of servers, the computers resolving on the LAN side aren't likely to hit the same IP that's in the address list, so the packets aren't being processed.

Anyone know a way to accomplish what I'm attempting here?
Last edited by 1littlewisp on Mon Jun 06, 2011 6:26 pm, edited 1 time in total.
 
ayufan
Member
Member
Posts: 334
Joined: Sun Jun 03, 2007 9:35 pm
Contact:

Re: An Interesting Challenge... ~or~ Choking Youtube Videos

Fri Jun 03, 2011 12:19 am

Find packets with content-type: video by using layer7 processing. Then mark connections as "video" and shape them ;)
 
User avatar
mramos
Member Candidate
Member Candidate
Posts: 231
Joined: Sun Nov 23, 2008 1:05 am
Location: S. B do Campo - SP - Brazil

Re: An Interesting Challenge... ~or~ Choking Youtube Videos

Fri Jun 03, 2011 1:39 am

This is working for me:

Ip Firewall Mangle: to add Youtube server to address list for 5 minutes.
;;; Youtube Address List
     chain=prerouting action=add-dst-to-address-list protocol=tcp address-list=Youtube 
     address-list-timeout=5m in-interface=!(PUBLIC) dst-port=80 content=youtube.com
Ip Firewall Mangle: To mark youtube packets (web-proxy enabled):
 ;;; Youtube
     chain=prerouting action=mark-connection new-connection-mark=youtubeconn passthrough=yes dst-address-list=Youtube 

     chain=output action=mark-packet new-packet-mark=youtube_pct passthrough=yes connection-mark=youtubeconn
Queue Types (to limit each user to 512K stream, 240p needs only 384k):
name="Youtube" kind=pcq pcq-rate=512k pcq-limit=50 pcq-classifier=dst-address pcq-total-limit=2000 pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 pcq-src-address6-mask=128 pcq-dst-address6-mask=128
Queue Tree (use 512K youtube PCQ & set priority = 8 )
name="Youtube" parent=global-out packet-mark=youtube_pct limit-at=0 queue=Youtube priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s 
Regards;
 
1littlewisp
newbie
Topic Author
Posts: 36
Joined: Wed Jun 10, 2009 6:23 pm

Re: An Interesting Challenge... ~or~ Choking Youtube Videos

Fri Jun 03, 2011 9:46 pm

This is working for me:

Ip Firewall Mangle: to add Youtube server to address list for 5 minutes.
;;; Youtube Address List
     chain=prerouting action=add-dst-to-address-list protocol=tcp address-list=Youtube 
     address-list-timeout=5m in-interface=!(PUBLIC) dst-port=80 content=youtube.com
Ip Firewall Mangle: To mark youtube packets (web-proxy enabled):
 ;;; Youtube
     chain=prerouting action=mark-connection new-connection-mark=youtubeconn passthrough=yes dst-address-list=Youtube 

     chain=output action=mark-packet new-packet-mark=youtube_pct passthrough=yes connection-mark=youtubeconn
Queue Types (to limit each user to 512K stream, 240p needs only 384k):
name="Youtube" kind=pcq pcq-rate=512k pcq-limit=50 pcq-classifier=dst-address pcq-total-limit=2000 pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 pcq-src-address6-mask=128 pcq-dst-address6-mask=128
Queue Tree (use 512K youtube PCQ & set priority = 8 )
name="Youtube" parent=global-out packet-mark=youtube_pct limit-at=0 queue=Youtube priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s 
Regards;
Thank you!

This did pretty much what I wanted. How would you configure it just to add .flv and .mp4 content to the list, though? With this, if someone googles "youtube.com" Google will be throttled for 5 minutes.
 
User avatar
mramos
Member Candidate
Member Candidate
Posts: 231
Joined: Sun Nov 23, 2008 1:05 am
Location: S. B do Campo - SP - Brazil

Re: An Interesting Challenge... ~or~ Choking Youtube Videos

Sat Jun 04, 2011 3:14 am

This did pretty much what I wanted. How would you configure it just to add .flv and .mp4 content to the list, though? With this, if someone googles "youtube.com" Google will be throttled for 5 minutes.
Nice to hear that helped you.

Abt flv & mp4 ... well, I guess it's necessary then to use layer 7. Too much for my knowledge :D

And yes, if someone uses any search tool to find http://www.youtube.com the search tool itself will be added to address list (google, bing, yahoo etc).

But ... 512K is not enough to make a search tool smooth? :D
Anyway, you can reduce the time of address list from 5 minutes to some seconds (I think) because the session for youtube will be already established so there is no reason to keep it at the adress list anyway. At least you'll avoid other google.com user to share those youtube packet/connection marks & queue type / priority.

(forget it: if this addresses are not at the addr list packages / connections will not be marked and then queue tree will not be used!
Another aproach ... if you left more bandwidht for youtube ... at least this user will release the necessary bw earlier, lefting it avialable for other users. Sometimes is better to store content at user PC asap than keep it using 512K for 10 minutes. Need some statistical analysis.

Regards;
 
1littlewisp
newbie
Topic Author
Posts: 36
Joined: Wed Jun 10, 2009 6:23 pm

Re: An Interesting Challenge... ~or~ Choking Youtube Videos

Mon Jun 06, 2011 5:36 pm

This did pretty much what I wanted. How would you configure it just to add .flv and .mp4 content to the list, though? With this, if someone googles "youtube.com" Google will be throttled for 5 minutes.
Nice to hear that helped you.

Abt flv & mp4 ... well, I guess it's necessary then to use layer 7. Too much for my knowledge :D

And yes, if someone uses any search tool to find http://www.youtube.com the search tool itself will be added to address list (google, bing, yahoo etc).

But ... 512K is not enough to make a search tool smooth? :D
Anyway, you can reduce the time of address list from 5 minutes to some seconds (I think) because the session for youtube will be already established so there is no reason to keep it at the adress list anyway. At least you'll avoid other google.com user to share those youtube packet/connection marks & queue type / priority.

(forget it: if this addresses are not at the addr list packages / connections will not be marked and then queue tree will not be used!
Another aproach ... if you left more bandwidht for youtube ... at least this user will release the necessary bw earlier, lefting it avialable for other users. Sometimes is better to store content at user PC asap than keep it using 512K for 10 minutes. Need some statistical analysis.

Regards;
Thanks for the reply. In the implementation I'm planning for this solution has only 3M down from their ISP so 512k is still going to be too generous. They've got many hosts on the LAN side. I think I may try adding another rule to filter content matching .flv or .mp4 first, then have the rules you defined above use my list as a source. Search for youtube.com from a smaller subset of rules. It's sorta sloppy, but I don't know regular expressions, either so L7 is going to be a little much. On a related note, http://xkcd.com/208/ <= this.

Anyway, I'll mark the thread as solved. Thanks again!
 
cylent
Member
Member
Posts: 383
Joined: Sun May 28, 2006 10:30 am

Re: [SOLVED]An Interesting Challenge... ~or~ Choking Youtube

Tue Apr 03, 2012 3:27 pm

this is a wonderful solution...

dont know how people wont start abusing it because you maybe giving priority to lets say facebook.com and all of a sudden they are feeding crap from that.

it would also be nice to unify or lessen the rules?

Who is online

Users browsing this forum: actomobile and 44 guests