Fight against rapidshare

sergejs · August 15, 2007, 5:40am

How do you imagine this automatically ?
MikroTik should create rules for specific resource rapidshare ? In my opinion not all the users will find this option useful (as resources might be different), unless you have the opportunity to create rules with queues and address-list.
Do you have any problems with address-lists and queues configuration (indeed address-list is already automation tool, as you do not need to create multiple mangle/firewall rules to mark all rapidshare data, but just put one rule to mangle and use address-list with rapidshare addresses).

janisk · August 15, 2007, 7:56am

create script, tool that recovers rapdishare ip addresses, and then add them to address-list thats all the automation you need

kev23m · August 15, 2007, 8:13am

I just tried this by collecting the ip addresses of the mirrors by d’loading a file.
Here are the ips i got.

195.122.131.88
212.162.63.88
62.67.57.88
207.138.168.88
80.239.151.88
62.67.46.88
64.215.245.88
195.219.1.88
82.129.39.88
80.129.35.88
80.239.236.88
82.129.36.18
80.239.159.18

Now i added this in mangle n marked the conn as rapidshare and all packets as rapidpackets.
Made a simple queue and tried to limit the speeds but it is not working, even tried dropping these packets but its not catching the d’load at all as i dont see any packets increasing in the mangle rule.
What could be the problem?

Kev

janisk · August 15, 2007, 8:21am

read here about mangle:
http://www.mikrotik.com/testdocs/ros/2.9/ip/mangle.php

about queues
http://www.mikrotik.com/testdocs/ros/2.9/root/queue.php

and do not forget that src address list should be used, set protocol tcp port 80

kev23m · August 15, 2007, 8:29am

I am using Mikrotik in bridge mode.

Did try setting tcp dest port to 80 , also tried giving the ip pool assigned , no luck.
It just goes thru as before.

janisk · August 15, 2007, 9:04am

address-list is under /ip firewall address-list not /ip pool these are 2 different things

and check wiki, there was example how to limit that if you use bridge

Henrik · August 15, 2007, 2:23pm

Hi

It should is possible to mark http connections exceeding a certain amount of B and classify it with a low priority in ques. And mark http traffic with low amount of Bytes, as surfing, and put it in front of the ques with a high priority.

Then surfing would be higher priority than http Download. And would work on any server, and any http download, even from “legal” sites. We use it, and it works fine.

Sergejs mentioned it at the top of this tread.

Isn’t this the solution or did I miss anything.?

Best regards
Henrik

GWISA-Kroonstad · August 15, 2007, 5:09pm

Agree with you Normunds. Users are confusing multiple mirror sites with P2P. Yet, for all those who use FileTopia… Have you seen the extensive P2P-like downloads on HTTPS? Several times realized the download goes straight between the clients. That is definitely P2P! And on HTTPS! What now, limit HTTPs? LOL

cpresto · August 16, 2007, 8:09am

Thank you Sergej/Janisk,
but what you suggest is what I’ve written at the beginning of this post (if I correctly understood): I’m using MT DNS cache to identify Rapidshare IP addresses and put them into a dedicated access list, let’s call it Rapid_list. Http traffic to/from Rapid_list will be considered as P2P, so P2P queues are applied to this (known process).
At the moment I copy manually these IPs into Rapid_list, what I would like to have is a script that’s able to do this for me, this should be the argument of this post.

sergejs · August 16, 2007, 8:50am

No, this traffic will not be considered as P2P, it will be considered as rapidshare traffic, that is being marked with mangle by ‘address-list’ option.

Your argument is clear, the best way to do this, find out all addresses used by rapidshare, create mangle+queues, and create export for this configuration, then copy to all routers.
There is no automated option as far as I know, there is not automatic configuration for this, because other user might want to block/limit another resource.

cpresto · August 16, 2007, 10:23am

Thank you sergejs,
yes, it is not P2P, but I “shape” this traffic as it is, in order to limit its download.
This is because I already have a queue for P2P traffic but, once identified, another queue might be created and used specifically for Rapidshare. The problem remains, on how to do this automatically…
I think that the problem is due to the fact that MT does not store DNS cache entries in a file, but simply into its memory. Probably using an external DNS server that stores its entries in a file (better in a mySQL table database) will give better results, because search operation could be done on this file and actions (insert into MT address list) performed by external scheduler (linux chron). Unfortunately I do no have these sw knowledge so I have to look for someone that can do this for me…

cmit · August 16, 2007, 1:15pm

cpresto,

here is your script:

:foreach i in=[/ip dns cache find] do={
  :if ([:find [/ip dns cache get $i name] "rapidshare"] > 0) do={
    :log info ("rapidshare: " . [/ip dns cache get $i name] . " (ip address " . [/ip dns cache get $i address] . ")")
    /ip firewall address-list add address=[/ip dns cache get $i address] list=rapidshare disabled=no
  }
}

Run this regularly using the scheduler, and it will scan the DNS cache of your MikroTik, and add all addresses the have the phrase “rapidshare” in the DNS name to an address-list named “rapidshare”.

As I read your original post, you know how to apply your traffic shaping to addresses in that list, right?

This script will only ADD to the address-list, and as the scripting language does not allow to set a life-time for address-list entries (like you can do from a firewall rule), the addresses will stay there forever (or until manually deleted).
If you want to have a smaller/cleaner/more recent address list, you could add the line

/ip firewall address-list remove [/ip firewall address-list find list=rapidshare]

to the beginning of the script, to clear the address list every time the script is run. Then only hosts which have recently been used (i.e. are in the DNS cache) will be put on the address list.

Does that help?

Best regards,
Christian Meis

cpresto · August 16, 2007, 1:35pm

Great cmit,
I’ll try and let you know

Thank you,
Carlo

cpresto · August 16, 2007, 5:14pm

Hi cmit,
script works almost fine: IPs whose name is “xxx.rapidshare.com” are added to the list, while IPs that correspond to “rapidshare.com” are not inserted into the list. Please have a look at addresses here below.
Probably something have to be changed into script search criteria…

[admin@AdiesselleP2K] > ip dns cache print
Flags: S - static

NAME ADDRESS TTL

…
…
12 http:/ 13 images.rapi 14 rs144l3.ra 15 rs178cg.ra 16 rapidshare.com 17 rapidshare.com 18 rapidshare.com 19 rapidshare.com 20 rapidshare.com 21 rapidshare.com 22 rapidshare.com 23 rapidshare.com 24 rapidshare.com 25 rapidshare.com 26 rapidshare.com 27 rapidshare.com 28 rapidshare.com 29 rapidshare.com 30 rapidshare.com 31 rapidshare.com 32 rapidshare.com 33 rs148cg.rapidshare.com 82.129.39.149 14m22s

[admin@AdiesselleP2K] ip firewall> address-list
print
Flags: X - disabled, D - dynamic

LIST ADDRESS

…
…
53 rapidshare 195.122.131.250
54 rapidshare 195.122.131.251
55 rapidshare 195.122.131.145
56 rapidshare 82.129.39.179
57 rapidshare 82.129.39.149

[admin@AdiesselleP2K] ip firewall>

Rgds

cmit · August 16, 2007, 7:13pm

make that “>0” a “>= 0” in the second line and try again…

Best regards,
Christian Meis

cpresto · August 17, 2007, 10:35am

Almost done Christian…
it stops when an already inserted entry is found into DNS cache address list, please have a look here below:

[admin@AdiesselleP2K] > ip dns cache print
Flags: S - static

NAME ADDRESS TTL

…
55 ns1.rapidshare.com 195.122.131.250 1d19h59m
56 ns2.rapidshare.com 80.237.244.50 1d19h59m
81 rapidshare.com 195.122.131.2 13m10s
82 rapidshare.com 195.122.131.3 13m10s
83 rapidshare.com 195.122.131.4 13m10s
84 rapidshare.com 195.122.131.5 13m10s
85 rapidshare.com 195.122.131.6 13m10s
86 rapidshare.com 195.122.131.7 13m10s
87 rapidshare.com 195.122.131.8 13m10s
88 rapidshare.com 195.122.131.9 13m10s
89 rapidshare.com 195.122.131.10 13m10s
90 rapidshare.com 195.122.131.11 13m10s
91 rapidshare.com 195.122.131.12 13m10s
92 rapidshare.com 195.122.131.13 13m10s
93 rapidshare.com 195.122.131.14 13m10s
94 rapidshare.com 195.122.131.15 13m8s
95 rapidshare.com 195.122.131.250 13m8s
96 rs144cg.rapidshare.com 82.129.39.145 13m8s
97 rs26cg.rapidshare.com 82.129.39.27 13m18s
98 rs91cg.rapidshare.com 82.129.39.92 13m29s
99 rs67cg.rapidshare.com 82.129.39.68 13m50s
100 rs140cg.rapidshare.com 82.129.39.141 13m58s
…

[admin@AdiesselleP2K] > ip firewall address-list print
Flags: X - disabled, D - dynamic

LIST ADDRESS

…
53 rapidshare 195.122.131.250
54 rapidshare 80.237.244.50
55 rapidshare 195.122.131.2
56 rapidshare 195.122.131.3
57 rapidshare 195.122.131.4
58 rapidshare 195.122.131.5
59 rapidshare 195.122.131.6
60 rapidshare 195.122.131.7
61 rapidshare 195.122.131.8
62 rapidshare 195.122.131.9
63 rapidshare 195.122.131.10
64 rapidshare 195.122.131.11
65 rapidshare 195.122.131.12
66 rapidshare 195.122.131.13
67 rapidshare 195.122.131.14
68 rapidshare 195.122.131.15

Closed_1 · August 18, 2007, 8:18pm

maybe we can add rule checking if the address list contained the same ip from dns cache it will bypass adding ip to addresslist, so next ip will added.

:foreach i in=[/ip dns cache find] do={
  :if ([:find [/ip dns cache get $i name] "rapidshare"] > 0) do={
    :log info ("rapidshare: " . [/ip dns cache get $i name] . " (ip address " . [/ip dns cache get $i address] . ")")
(......add ip rule checking in addresslist here: w/ foreach and if)
    /ip firewall address-list add address=[/ip dns cache get $i address] list=rapidshare disabled=no
(......end of add ip rule checking in addressllist here: w/ foreach and if)
  }
}

thats my suggestion, CMIIW

radocicala · August 19, 2007, 2:59pm

that new script doesn´t work, could you repair it, it is good idea to check if ip of searched word isn´t allready in address list and if it is there it won´t be written

cpresto · August 20, 2007, 8:30am

Yes,
it should be something similar, new part in red. It doesn’t work, I’m not good enough with scripts …anyone may have a look and correct it?

:foreach i in=[/ip dns cache find] do={
:if ([:find [/ip dns cache get $i name] “rapidshare”] != 0) do={
:log info ("rapidshare: " . [/ip dns cache get $i name] . " (ip
address " . [/ip dns cache get $i address] . “)”)

a new for cicle is need to search for already inserted IP address

into rapidshare address list

only new IPs will be added to address list

:foreach f in=[/ip firewall address-list find] do={
:if ([:find [/ip firewall address-list get $f address]] != [:find [/ip
dns cache get $i address]]) do={
/ip firewall address-list add address=[/ip dns cache get $i
address] list=rapidshare disabled=no
}
}
}
}

mrz · August 20, 2007, 9:57am

replace your red code with this

:foreach f in=[/ip firewall address-list find] do={
   :if ([/ip firewall address-list get $f address ] != [/ip dns cache get $i address] ) do={ 
...