Community discussions

MikroTik App
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Fight against rapidshare

Thu Aug 09, 2007 11:22 pm

Hi all,
as a WISP, we are fightng our battle against P2P every day... :?
P2P is identified with mangle for port different than well known ports (>1024) and redirected through dedicated connection (MT default gateway).
A big problem is represented by P2P on port 80, like Rapidshare. This traffic is not identified as P2P, but is considered as normal web traffic... a disaster !!! :shock:
Rapidshare has many servers around the Internet and newer are added avery day, so it is almost impossible have an updated list of its servers IP address to limits traffic to/from these IP addresses.
With MT DNS client & cache http://www.mikrotik.com/testdocs/ros/2. ... scache.php Rapidshare IP addresses will be stored (in cache), every time they are requested by customers.
If it might be possible to add these IP address to a MT address list dinamically (with a script), traffic to/from this address list will be mangled as P2P and problem will be solved.
Could it be possible to write a script to do this? Any advice?

Rgds
 
changeip
Forum Guru
Forum Guru
Posts: 3830
Joined: Fri May 28, 2004 5:22 pm

Re: Fight against P2P on port 80

Fri Aug 10, 2007 12:12 am

if they end up in the cache then you can write a script to make a address-list from them probably. send me an example of them and I'll help you:

/ip dns cache print detail

Sam
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Fri Aug 10, 2007 11:15 am

Thank you Sam,
please find here below cache content:

Flags: S - static
# NAME ADDRESS TTL
0 za.akadns.org 195.219.3.169 11h13m58s
1 zb.akadns.org 206.132.100.105 11h13m58s
2 zc.akadns.org 61.200.81.111 11h13m58s
......
......
56 http://www.rapidshare.com 195.122.131.250 14m22s
57 images.rapidshare.com 195.122.131.251 2m31s
58 rs181l3.rapidshare.com 195.122.131.182 14m24s
59 g.msn.com 207.68.179.219 12m40s

I think that writing a script is quite difficult, because these information are not stored in a file, but simply written in cache...

Rgds,
Carlo
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26368
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Fight against P2P on port 80

Fri Aug 10, 2007 3:45 pm

rapidshare is not p2p, it's just a regular file hosting site.
 
titius
Member
Member
Posts: 338
Joined: Mon Oct 17, 2005 11:43 am
Location: Titel Serbia

Re: Fight against P2P on port 80

Fri Aug 10, 2007 4:00 pm

@normis

Cpresto is aware of that, but many clients use rapidshare, and downloadind all day long. So it is like p2p bandwith is 100% used.

@cpresto

Cmon man, what do you want? buy more bandwith.

Do you expect from your customers to read&send mail. Yeah right. :roll:
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6695
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: Fight against P2P on port 80

Fri Aug 10, 2007 4:20 pm

Very interesting option is 'connection-bytes' in firewall/mangle, it allows you to filter/mark/limit connections that are exceeding specific limit (somehow determine which request is to open web-page and which is download file).
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Fri Aug 10, 2007 4:50 pm

Yes sergejs,
this might be a solution, but to mangle a connection and establish limits for this (if port 80 is used), I do have to know its IP address to differentiate it from all the others, so DNS it's back again... :?
I don't want to limit users that download software from web site (antivirus trials, updates, etc...) but I don't want to give them the opportunity to download P2P files all day long from hosting sites like Rapidshare. Since know a small number of them have discovered this "Eldorado", but as this voice goes around our Internet connection will be saturated by this type of downloads. Unfortunately for WISPs bandwidth is a very precious resource and not be wasted...

I suppose that a script will be easier be written if DNS cache information are available in a file (.txt o .csv), could this be possible?

Rgds,
Carlo
 
abab_rafiq
Member Candidate
Member Candidate
Posts: 118
Joined: Thu Aug 24, 2006 12:47 pm
Location: Dhaka

Re: Fight against P2P on port 80

Sat Aug 11, 2007 1:54 pm

Use proxy to drop or down priority of downloading from rapidshare.com

Actually for stop or make priority for file transferring on port 80 it is the way to use layer 7 classifier.
Regular expression, good knowledge in C and some other types of packet marking is needed to do the better.

Rafiq...
 
User avatar
tgrand
Long time Member
Long time Member
Posts: 667
Joined: Mon Aug 21, 2006 2:57 am
Location: Winnipeg, Manitoba, Canada

Re: Fight against P2P on port 80

Sat Aug 11, 2007 3:47 pm

As opposed to trying to identify P2P would it not be better to identify known valid traffic.
pop3, snmp, http, https, sip, irc, ftp, etc.
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Mon Aug 13, 2007 5:57 pm

Hi tgrand,
this is already done (port < 1024) as described in my post.
The problem is that P2P on port 80 (file sharing hosting as Rapidshare) is identified (based on its port traffic) as normal http traffic instead of P2P...
 
User avatar
tgrand
Long time Member
Long time Member
Posts: 667
Joined: Mon Aug 21, 2006 2:57 am
Location: Winnipeg, Manitoba, Canada

Re: Fight against P2P on port 80

Mon Aug 13, 2007 9:22 pm

Yes but http is very easy to identify if you do deep packet inspection
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Mon Aug 13, 2007 9:49 pm

Please note that Rapidshare traffic is normal http traffic on port 80, the same "normal browsing" traffic.
The only difference is that on RapidShare servers are stored files to be shared.
Definetly, it is not P2P traffic but simply "P2P content".
 
User avatar
tgrand
Long time Member
Long time Member
Posts: 667
Joined: Mon Aug 21, 2006 2:57 am
Location: Winnipeg, Manitoba, Canada

Re: Fight against P2P on port 80

Tue Aug 14, 2007 6:23 am

Then mark and queue this range: 195.0.0.0/8
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26368
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Fight against P2P on port 80

Tue Aug 14, 2007 9:34 am

it doesn't seem that cpresto understands what p2p is.

rapidshare is recognized as http precisely for the reason that it is http. p2p content? what's that? I can put my work documents on rapidshare. it doesn't mean that everything on rapidshare is pirated programs.

rapidshare is regular file download from www page over http. the same as you download netinstall from http://www.mikrotik.com !
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Tue Aug 14, 2007 11:27 am

Probably I've not been clear enough... :?
From my post it should be clear that I perfectly understood that Rapidshare is normal http traffic, and this is exactly the problem !!! :(

I do refer to Rapidshare as P2P because it is used like this: with P2P you leave your PC on and download files 24h a day. This is ok if a "normal" P2P program is used, because trafficic on these ports (> 1024) can be mangled and putted into queues to be filtered.
With Rapidshare (and similars) this is not possible, because it is not P2P traffic, but it is simple http traffic... 24h a day at maximum rate allowed for the customer that is using it: a disaster!!! :shock:
The only way (in my understanding) to identify this type of traffic is to identify Rapidshare servers, using DNS request from customers: if a customer looks for "rs181l3.rapidshare.com" (please look @ my second reply to this post), MT DNS proxy replies "195.122.131.182" and store this entry into its cache.
Now I simply check DNS entry cache every two/three days, and add Rapidshare address to MT address list named "Rapidshare": traffic to/from this address list is queued together with P2P traffic. I'm trying to understand if this process might be automated using a script.

Unfortunately it is not possible to identify Rapidshare traffic with the whole 195.0.0.0/8 network, 2^24-2 are too many servers also for Rapidshare and for sure in this huge range other normal web sites are hosted.

Rgds
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26368
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Fight against P2P on port 80

Tue Aug 14, 2007 11:37 am

what do you mean by 24h a day for rapidshare? rapidshare hosts files one by one, you can't download all day from it.

maybe you simply need some burst limitations (user can download fast for some time, then slows down), or use webproxy and block rapidshare entirely?
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Tue Aug 14, 2007 11:58 am

Yes,
you can download files one by one from Rapidshare but, due to their nature (software, porno video,...) customers spend ours a day downloading files.
Burst limitations might be a solution, but IP server identification is necessary as well, otherwise this will be applied also for other web downloads. Blocking it entirelly is not a (polite) solution.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26368
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Fight against P2P on port 80

Tue Aug 14, 2007 1:19 pm

OK, let's just concentrate on "slowing down rapidshare, and not affecting other http". your subject is very misleading.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6263
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: Fight against P2P on port 80

Tue Aug 14, 2007 4:18 pm

you can create address list that will hold all the rapidshare ip addresses, and for that list create queue that will limit speed available.
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Tue Aug 14, 2007 6:29 pm

Thank janisk,
but, as you can read in my posts... this is exactly what I'm doing :D
The objective is to let MT do this automatically...
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6695
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: Fight against P2P on port 80

Wed Aug 15, 2007 8:40 am

How do you imagine this automatically ?
MikroTik should create rules for specific resource rapidshare ? In my opinion not all the users will find this option useful (as resources might be different), unless you have the opportunity to create rules with queues and address-list.
Do you have any problems with address-lists and queues configuration (indeed address-list is already automation tool, as you do not need to create multiple mangle/firewall rules to mark all rapidshare data, but just put one rule to mangle and use address-list with rapidshare addresses).
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6263
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: Fight against P2P on port 80

Wed Aug 15, 2007 10:56 am

create script, tool that recovers rapdishare ip addresses, and then add them to address-list thats all the automation you need
 
kev23m
just joined
Posts: 12
Joined: Tue Jul 17, 2007 2:23 pm

Re: Fight against P2P on port 80

Wed Aug 15, 2007 11:13 am

I just tried this by collecting the ip addresses of the mirrors by d'loading a file.
Here are the ips i got.

195.122.131.88
212.162.63.88
62.67.57.88
207.138.168.88
80.239.151.88
62.67.46.88
64.215.245.88
195.219.1.88
82.129.39.88
80.129.35.88
80.239.236.88
82.129.36.18
80.239.159.18

Now i added this in mangle n marked the conn as rapidshare and all packets as rapidpackets.
Made a simple queue and tried to limit the speeds but it is not working, even tried dropping these packets but its not catching the d'load at all as i dont see any packets increasing in the mangle rule.
What could be the problem?

Kev
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6263
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: Fight against P2P on port 80

Wed Aug 15, 2007 11:21 am

read here about mangle:
http://www.mikrotik.com/testdocs/ros/2.9/ip/mangle.php

about queues
http://www.mikrotik.com/testdocs/ros/2.9/root/queue.php

and do not forget that src address list should be used, set protocol tcp port 80
 
kev23m
just joined
Posts: 12
Joined: Tue Jul 17, 2007 2:23 pm

Re: Fight against P2P on port 80

Wed Aug 15, 2007 11:29 am

I am using Mikrotik in bridge mode.

Did try setting tcp dest port to 80 , also tried giving the ip pool assigned , no luck.
It just goes thru as before.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6263
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: Fight against P2P on port 80

Wed Aug 15, 2007 12:04 pm

address-list is under /ip firewall address-list not /ip pool these are 2 different things

and check wiki, there was example how to limit that if you use bridge
 
Henrik
just joined
Posts: 16
Joined: Tue Jun 20, 2006 11:12 pm

Re: Fight against P2P on port 80

Wed Aug 15, 2007 5:23 pm

Hi

It should is possible to mark http connections exceeding a certain amount of B and classify it with a low priority in ques. And mark http traffic with low amount of Bytes, as surfing, and put it in front of the ques with a high priority.

Then surfing would be higher priority than http Download. And would work on any server, and any http download, even from “legal” sites. We use it, and it works fine.

Sergejs mentioned it at the top of this tread.

Isn’t this the solution or did I miss anything.?
:shock:
Best regards
Henrik
 
User avatar
GWISA-Kroonstad
Member Candidate
Member Candidate
Posts: 111
Joined: Fri Nov 10, 2006 3:34 pm

Re: Fight against P2P on port 80

Wed Aug 15, 2007 8:09 pm

rapidshare is not p2p, it's just a regular file hosting site.
Agree with you Normunds. Users are confusing multiple mirror sites with P2P. Yet, for all those who use FileTopia... Have you seen the extensive P2P-like downloads on HTTPS? Several times realized the download goes straight between the clients. That is definitely P2P! And on HTTPS! What now, limit HTTPs? LOL
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Thu Aug 16, 2007 11:09 am

Thank you Sergej/Janisk,
but what you suggest is what I've written at the beginning of this post (if I correctly understood): I'm using MT DNS cache to identify Rapidshare IP addresses and put them into a dedicated access list, let's call it Rapid_list. Http traffic to/from Rapid_list will be considered as P2P, so P2P queues are applied to this (known process).
At the moment I copy manually these IPs into Rapid_list, what I would like to have is a script that's able to do this for me, this should be the argument of this post.
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6695
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: Fight against P2P on port 80

Thu Aug 16, 2007 11:50 am

No, this traffic will not be considered as P2P, it will be considered as rapidshare traffic, that is being marked with mangle by 'address-list' option.

Your argument is clear, the best way to do this, find out all addresses used by rapidshare, create mangle+queues, and create export for this configuration, then copy to all routers.
There is no automated option as far as I know, there is not automatic configuration for this, because other user might want to block/limit another resource.
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Thu Aug 16, 2007 1:23 pm

Thank you sergejs,
yes, it is not P2P, but I "shape" this traffic as it is, in order to limit its download.
This is because I already have a queue for P2P traffic but, once identified, another queue might be created and used specifically for Rapidshare. The problem remains, on how to do this automatically...
I think that the problem is due to the fact that MT does not store DNS cache entries in a file, but simply into its memory. Probably using an external DNS server that stores its entries in a file (better in a mySQL table database) will give better results, because search operation could be done on this file and actions (insert into MT address list) performed by external scheduler (linux chron). Unfortunately I do no have these sw knowledge so I have to look for someone that can do this for me...
 
cmit
Forum Guru
Forum Guru
Posts: 1547
Joined: Fri May 28, 2004 12:49 pm
Location: Germany

Re: Fight against P2P on port 80

Thu Aug 16, 2007 4:15 pm

cpresto,

here is your script:
:foreach i in=[/ip dns cache find] do={
  :if ([:find [/ip dns cache get $i name] "rapidshare"] > 0) do={
    :log info ("rapidshare: " . [/ip dns cache get $i name] . " (ip address " . [/ip dns cache get $i address] . ")")
    /ip firewall address-list add address=[/ip dns cache get $i address] list=rapidshare disabled=no
  }
}
Run this regularly using the scheduler, and it will scan the DNS cache of your MikroTik, and add all addresses the have the phrase "rapidshare" in the DNS name to an address-list named "rapidshare".

As I read your original post, you know how to apply your traffic shaping to addresses in that list, right?

This script will only ADD to the address-list, and as the scripting language does not allow to set a life-time for address-list entries (like you can do from a firewall rule), the addresses will stay there forever (or until manually deleted).
If you want to have a smaller/cleaner/more recent address list, you could add the line
/ip firewall address-list remove [/ip firewall address-list find list=rapidshare]
to the beginning of the script, to clear the address list every time the script is run. Then only hosts which have recently been used (i.e. are in the DNS cache) will be put on the address list.

Does that help?

Best regards,
Christian Meis
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Thu Aug 16, 2007 4:35 pm

Great cmit,
I'll try and let you know

Thank you,
Carlo
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Thu Aug 16, 2007 8:14 pm

Hi cmit,
script works almost fine: IPs whose name is "xxx.rapidshare.com" are added to the list, while IPs that correspond to "rapidshare.com" are not inserted into the list. Please have a look at addresses here below.
Probably something have to be changed into script search criteria...

[admin@AdiesselleP2K] > ip dns cache print
Flags: S - static
# NAME ADDRESS TTL
...........
...........
12 http://www.rapidshare.com 195.122.131.250 8m55s
13 images.rapidshare.com 195.122.131.251 7m7s
14 rs144l3.rapidshare.com 195.122.131.145 8m56s
15 rs178cg.rapidshare.com 82.129.39.179 9m45s

16 rapidshare.com 195.122.131.250 6m44s
17 rapidshare.com 195.122.131.2 6m44s
18 rapidshare.com 195.122.131.3 6m44s
19 rapidshare.com 195.122.131.4 6m44s
20 rapidshare.com 195.122.131.5 6m44s
21 rapidshare.com 195.122.131.6 6m44s
22 rapidshare.com 195.122.131.7 6m43s
23 rapidshare.com 195.122.131.8 6m43s
24 rapidshare.com 195.122.131.9 6m43s
25 rapidshare.com 195.122.131.10 6m43s
26 rapidshare.com 195.122.131.11 6m43s
27 rapidshare.com 195.122.131.12 6m43s
28 rapidshare.com 195.122.131.13 6m43s
29 rapidshare.com 195.122.131.14 6m43s
30 rapidshare.com 195.122.131.15 6m43s
31 rapidshare.com 195.122.131.16 6m43s
32 rapidshare.com 195.122.131.17 6m43s
33 rs148cg.rapidshare.com 82.129.39.149 14m22s

[admin@AdiesselleP2K] ip firewall> address-list
<s-list> print
Flags: X - disabled, D - dynamic
# LIST ADDRESS
......
......
53 rapidshare 195.122.131.250
54 rapidshare 195.122.131.251
55 rapidshare 195.122.131.145
56 rapidshare 82.129.39.179
57 rapidshare 82.129.39.149

[admin@AdiesselleP2K] ip firewall>

Rgds
 
cmit
Forum Guru
Forum Guru
Posts: 1547
Joined: Fri May 28, 2004 12:49 pm
Location: Germany

Re: Fight against P2P on port 80

Thu Aug 16, 2007 10:13 pm

make that ">0" a ">= 0" in the second line and try again...

Best regards,
Christian Meis
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Fri Aug 17, 2007 1:35 pm

Almost done Christian...
it stops when an already inserted entry is found into DNS cache address list, please have a look here below:

[admin@AdiesselleP2K] > ip dns cache print
Flags: S - static
# NAME ADDRESS TTL

.........
55 ns1.rapidshare.com 195.122.131.250 1d19h59m
56 ns2.rapidshare.com 80.237.244.50 1d19h59m
81 rapidshare.com 195.122.131.2 13m10s
82 rapidshare.com 195.122.131.3 13m10s
83 rapidshare.com 195.122.131.4 13m10s
84 rapidshare.com 195.122.131.5 13m10s
85 rapidshare.com 195.122.131.6 13m10s
86 rapidshare.com 195.122.131.7 13m10s
87 rapidshare.com 195.122.131.8 13m10s
88 rapidshare.com 195.122.131.9 13m10s
89 rapidshare.com 195.122.131.10 13m10s
90 rapidshare.com 195.122.131.11 13m10s
91 rapidshare.com 195.122.131.12 13m10s
92 rapidshare.com 195.122.131.13 13m10s
93 rapidshare.com 195.122.131.14 13m10s
94 rapidshare.com 195.122.131.15 13m8s
95 rapidshare.com 195.122.131.250 13m8s :!:
96 rs144cg.rapidshare.com 82.129.39.145 13m8s
97 rs26cg.rapidshare.com 82.129.39.27 13m18s
98 rs91cg.rapidshare.com 82.129.39.92 13m29s
99 rs67cg.rapidshare.com 82.129.39.68 13m50s
100 rs140cg.rapidshare.com 82.129.39.141 13m58s
......

[admin@AdiesselleP2K] > ip firewall address-list print
Flags: X - disabled, D - dynamic
# LIST ADDRESS
...........
53 rapidshare 195.122.131.250
54 rapidshare 80.237.244.50
55 rapidshare 195.122.131.2
56 rapidshare 195.122.131.3
57 rapidshare 195.122.131.4
58 rapidshare 195.122.131.5
59 rapidshare 195.122.131.6
60 rapidshare 195.122.131.7
61 rapidshare 195.122.131.8
62 rapidshare 195.122.131.9
63 rapidshare 195.122.131.10
64 rapidshare 195.122.131.11
65 rapidshare 195.122.131.12
66 rapidshare 195.122.131.13
67 rapidshare 195.122.131.14
68 rapidshare 195.122.131.15 :!:
 
Closed_1
just joined
Posts: 7
Joined: Fri Dec 08, 2006 11:40 am

Re: Fight against P2P on port 80

Sat Aug 18, 2007 11:18 pm

maybe we can add rule checking if the address list contained the same ip from dns cache it will bypass adding ip to addresslist, so next ip will added.
:foreach i in=[/ip dns cache find] do={
  :if ([:find [/ip dns cache get $i name] "rapidshare"] > 0) do={
    :log info ("rapidshare: " . [/ip dns cache get $i name] . " (ip address " . [/ip dns cache get $i address] . ")")
(......add ip rule checking in addresslist here: w/ foreach and if)
    /ip firewall address-list add address=[/ip dns cache get $i address] list=rapidshare disabled=no
(......end of add ip rule checking in addressllist here: w/ foreach and if)
  }
}
thats my suggestion, CMIIW
 
radocicala
Member Candidate
Member Candidate
Posts: 136
Joined: Fri Aug 10, 2007 6:56 pm

Re: Fight against P2P on port 80

Sun Aug 19, 2007 5:59 pm

that new script doesn´t work, could you repair it, it is good idea to check if ip of searched word isn´t allready in address list and if it is there it won´t be written
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Mon Aug 20, 2007 11:30 am

Yes,
it should be something similar, new part in red. It doesn't work, I'm not good enough with scripts :? ...anyone may have a look and correct it?

:foreach i in=[/ip dns cache find] do={
:if ([:find [/ip dns cache get $i name] "rapidshare"] != 0) do={
:log info ("rapidshare: " . [/ip dns cache get $i name] . " (ip
address " . [/ip dns cache get $i address] . ")")
# a new for cicle is need to search for already inserted IP address
# into rapidshare address list
# only new IPs will be added to address list
:foreach f in=[/ip firewall address-list find] do={
:if ([:find [/ip firewall address-list get $f address]] != [:find [/ip
dns cache get $i address]]) do={

/ip firewall address-list add address=[/ip dns cache get $i
address] list=rapidshare disabled=no
}
}
}
}
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7053
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Fight against P2P on port 80

Mon Aug 20, 2007 12:57 pm

replace your red code with this
:foreach f in=[/ip firewall address-list find] do={
   :if ([/ip firewall address-list get $f address ] != [/ip dns cache get $i address] ) do={ 
...
 
radocicala
Member Candidate
Member Candidate
Posts: 136
Joined: Fri Aug 10, 2007 6:56 pm

Re: Fight against P2P on port 80

Mon Aug 20, 2007 2:48 pm

Still doesn´t work, could you make it right? and write it complet, not in parts
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7053
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Fight against P2P on port 80

Mon Aug 20, 2007 4:02 pm

so this is complete WORKING script:
I added comments so that guys, who don't know what each line do, could learn.
# check every dns entry
:foreach i in=[/ip dns cache find] do={
    :local bNew "true";
#  check if dns name contains rapidshare
    :if ([:find [/ip dns cache get $i name] "rapidshare"] != 0) do={
        :local tmpAddress [/ip dns cache get $i address] ;
#---- if address list is empty do not check ( add address directly )
        :if ( [/ip firewall address-list find ] = "") do={
            /ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no;
        } else={
#------- check every address list entry
            :foreach j in=[/ip firewall address-list find ] do={
#---------- set bNew variable to false if address exists in address list
                :if ( [/ip firewall address-list get $j address] = $tmpAddress ) do={
                    :set bNew "false";
                }
            }
#------- if address is new then add to address list
            :if ( $bNew = "true" ) do={
                /ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no
            }
        }
    }
}

# [ THE END ]
 
radocicala
Member Candidate
Member Candidate
Posts: 136
Joined: Fri Aug 10, 2007 6:56 pm

Re: Fight against P2P on port 80

Mon Aug 20, 2007 7:24 pm

Still doesn´t work like it is supposed. It writes every ip saved in cache in DNS. But it is finding only rapidshare when you change
!= 0
for
>= 0
.
But there is still some problem: if there is subnet´s ip in address list(237.138.168.0/24) it is also writes the ip´s that belong that subnet(207.138.168.61). Could it be possible to edit it not to write ips of subnet if the subnet is allready in address list?
 
User avatar
cpresto
Member Candidate
Member Candidate
Topic Author
Posts: 177
Joined: Tue Jul 18, 2006 3:12 pm

Re: Fight against P2P on port 80

Mon Aug 20, 2007 9:24 pm

If I use
>=0, scripts stops if an already addeded IP address is found into address list,
With !=0 it seems ok...
 
radocicala
Member Candidate
Member Candidate
Posts: 136
Joined: Fri Aug 10, 2007 6:56 pm

Re: Fight against P2P on port 80

Mon Aug 20, 2007 11:44 pm

no if you use != 0, all ip written in dns cache are written to address list, check some ip from address list to put it in you browser, if it is rapidshare, the official rapidshare site will open if not no rapidshare site will be shown. Or check dns cache you will see that all ips from there are written to ip address. I am 100% sure, I confirmed it clearing DNS cache, running the script with !=0, no rapidshare was not in DNS cache and it wrote me all ips from DNS cache.(NO RAPIDSHARE).
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7053
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Fight against P2P on port 80

Tue Aug 21, 2007 10:00 am

Still doesn´t work like it is supposed. It writes every ip saved in cache in DNS.
It's not possible ( TESTED, WORKING ). Make sure you typed code correctly.
But there is still some problem: if there is subnet´s ip in address list(237.138.168.0/24) it is also writes the ip´s that belong that subnet(207.138.168.61). Could it be possible to edit it not to write ips of subnet if the subnet is allready in address list?
Yes it is possible. Calculate subnets address range and check if IP is in that range. It's quite easy, but you will have to do it for yourself. If you want to use scripts then learn how to script.
 
sgsmc
just joined
Posts: 9
Joined: Wed Jul 25, 2007 5:01 pm

Re: Fight against P2P on port 80

Thu Aug 23, 2007 2:13 pm

Calculate subnets address range and check if IP is in that range. It's quite easy, but you will have to do it for yourself. If you want to use scripts then learn how to script.

mrz i am newbie

tried to learn script and tried to do it but could naot mange
now need someone. can you give it out how to do
 
User avatar
jdejansb
Frequent Visitor
Frequent Visitor
Posts: 69
Joined: Thu Jul 13, 2006 1:35 pm
Location: Srbija
Contact:

Re: Fight against P2P on port 80

Fri Aug 31, 2007 11:38 am

what do you mean by 24h a day for rapidshare? rapidshare hosts files one by one, you can't download all day from it.

maybe you simply need some burst limitations (user can download fast for some time, then slows down), or use webproxy and block rapidshare entirely?
Hi all, long time since my last post, but here is something that is interesting - using BURSTS. While I had users in MT's internal "base" I used profiles with bursts, everyting worked just fine. Ppl had great speeds for surf and something less for long (large) downloads. And then I switched to a Radius server and sql database and couldn't make bursts to work ......

Is it possible to use BURST when authorize thru RADIUS server? I "see" the INTEGER value for tx/rx speeds, where (if I'd like to use bursts) should be a TEXT string for bursts :( (512k/256k ccc/ddd etc...). Any help on this??

Dejan
 
sgsmc
just joined
Posts: 9
Joined: Wed Jul 25, 2007 5:01 pm

Re: Fight against P2P on port 80

Wed Oct 24, 2007 10:17 pm

Yes it is possible. Calculate subnets address range and check if IP is in that range. It's quite easy, but you will have to do it for yourself. If you want to use scripts then learn how to script.
mrz please!!!!

I am newbie and tried quite a lot but there is some small error and it is not working.
 
wildbill442
Forum Guru
Forum Guru
Posts: 1055
Joined: Wed Dec 08, 2004 7:29 am
Location: Sacramento, CA

Re: Fight against P2P on port 80

Thu Oct 25, 2007 1:20 am

Use OpenDNS as your forwarding servers and just block access to Rapidshare.com :)

I didn't feel like reading through all the replies.... but I agree with one of the other users who mentioned purchasing more bandwidth ;)

either that or setup a queue to limit bandwidth going to the rapid share webservers..

do an nslookup on their content servers domain name and it will give you all the IP's associated with that domain.

ex:
C:\>nslookup
*** Can't find server name for address 192.168.0.2: Non-existent domain
*** Default servers are not available
Default Server:  UnKnown
Address:  192.168.0.2

> rapidshare.com
Server:  UnKnown
Address:  192.168.0.2

Non-authoritative answer:
Name:    rapidshare.com
Addresses:  195.122.131.14, 195.122.131.15, 195.122.131.250, 195.122.131.2
          195.122.131.3, 195.122.131.4, 195.122.131.5, 195.122.131.6, 195.122.13
1.7
          195.122.131.8, 195.122.131.9, 195.122.131.10, 195.122.131.11, 195.122.
131.12
          195.122.131.13

>
 
titius
Member
Member
Posts: 338
Joined: Mon Oct 17, 2005 11:43 am
Location: Titel Serbia

Re: Fight against P2P on port 80

Thu Oct 25, 2007 2:12 am

0   ;;; Upload_ALL
     chain=prerouting src-address-list=twi dst-address-list=!twi 
     action=mark-connection new-connection-mark=UPLOAD passthrough=yes 

 1   chain=prerouting connection-mark=UPLOAD src-address-list="" 
     dst-address-list=!twi action=change-tos new-tos=normal 

 2   chain=prerouting connection-mark=UPLOAD src-address-list=twi 
     dst-address-list=!twi action=mark-packet new-packet-mark=UPLOAD_P 
     passthrough=yes 

 3   ;;; Down_ALL
     chain=postrouting dst-address-list=twi action=mark-connection 
     new-connection-mark=DOWNLOAD passthrough=yes 

 4   chain=postrouting connection-mark=DOWNLOAD action=change-tos 
     new-tos=normal 

 5   chain=postrouting connection-mark=DOWNLOAD action=mark-packet 
     new-packet-mark=DOWNLOAD_P passthrough=yes 

 6   ;;; DNS
     chain=prerouting protocol=udp dst-port=53 src-address-list=twi 
     dst-address-list=!twi action=mark-connection new-connection-mark=DNS_UP 
     passthrough=yes 

 7   chain=prerouting protocol=udp dst-port=53 connection-mark=DNS_UP 
     src-address-list=twi dst-address-list=!twi action=change-tos 
     new-tos=min-delay 

 8   chain=prerouting protocol=udp dst-port=53 connection-mark=DNS_UP 
     src-address-list=twi dst-address-list=!twi action=mark-packet 
     new-packet-mark=DNS_UP passthrough=no 

 9   chain=postrouting protocol=udp src-port=53 dst-address-list=twi 
     action=mark-connection new-connection-mark=DNS_DOWN passthrough=yes 

10   chain=postrouting protocol=udp src-port=53 connection-mark=DNS_DOWN 
     dst-address-list=twi action=change-tos new-tos=min-delay

11   chain=postrouting protocol=udp src-port=53 connection-mark=DNS_DOWN dst-address-list=twi action=mark-packet 
     new-packet-mark=DNS_DOWN_P passthrough=no 

12   ;;; HTTP_UP
     chain=prerouting protocol=tcp dst-port=80 src-address-list=twi dst-address-list=!twi action=mark-connection 
     new-connection-mark=HTTP_UP passthrough=yes 

13   chain=prerouting protocol=tcp dst-port=80 connection-mark=HTTP_UP src-address-list=twi dst-address-list=!twi 
     action=change-tos new-tos=max-throughput 

14   ;;; first_512kB_UP
     chain=prerouting protocol=tcp dst-port=80 connection-mark=HTTP_UP connection-bytes=0-512000 src-address-list=twi 
     dst-address-list=!twi action=mark-packet new-packet-mark=PRVIH_512_UP passthrough=no 

15 ;;; rest_UP  
    chain=prerouting protocol=tcp dst-port=80 connection-mark=HTTP_UP src-address-list=twi dst-address-list=!twi 
     action=mark-packet new-packet-mark=HTTP_UP_P passthrough=no 

16   ;;; HTTP_DOWN
     chain=postrouting protocol=tcp src-port=80 src-address-list=!twi dst-address-list=twi action=mark-connection 
     new-connection-mark=HTTP_DOWN passthrough=yes 

17   chain=postrouting protocol=tcp src-port=80 connection-mark=HTTP_DOWN dst-address-list=twi action=change-tos 
     new-tos=max-throughput 

18   ;;; First_MB_Down
     chain=postrouting protocol=tcp src-port=80 connection-mark=HTTP_DOWN connection-bytes=0-1024000 
     src-address-list=!twi dst-address-list=twi action=mark-packet new-packet-mark=HTTP_DOWN_1MB passthrough=no 

19   ;;; rest
     chain=postrouting protocol=tcp src-port=80 connection-mark=HTTP_DOWN src-address-list=!twi dst-address-list=twi 
     action=mark-packet new-packet-mark=HTTP_DOWN_P passthrough=no 

Then you can use queues or queue trees to limit this traffic the way you like it.
It works for now.

I would like to hear you opinion about this mangle rules.

I almost forgot.

Thanks to JANISK for right road to follow :) .
 
sgsmc
just joined
Posts: 9
Joined: Wed Jul 25, 2007 5:01 pm

Re: Fight against P2P on port 80

Fri Oct 26, 2007 3:05 pm

Yes it is possible. Calculate subnets address range and check if IP is in that range. It's quite easy, but you will have to do it for yourself. If you want to use scripts then learn how to script.
mrz's script is working perfect. But what's the correct code to be added to the script to calculate sunet as above?
 
sgsmc
just joined
Posts: 9
Joined: Wed Jul 25, 2007 5:01 pm

Re: Fight against P2P on port 80

Fri Nov 30, 2007 5:09 pm

# check every dns entry
:foreach i in=[/ip dns cache find] do={
    :local bNew "true";
#  check if dns name contains rapidshare
    :if ([:find [/ip dns cache get $i name] "rapidshare"] != 0) do={
        :local tmpAddress [/ip dns cache get $i address] ;
            :for k from=0 to=255 do={
            :put ("IP: ".$i address.".".$i address.".".$i address".".$k."");
#---- if address list is empty do not check ( add address directly )
        :if ( [/ip firewall address-list find ] = "") do={
            /ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no;
        } else={
#------- check every address list entry
            :foreach j in=[/ip firewall address-list find ] do={
#---------- set bNew variable to false if address exists in address list
                :if ( [/ip firewall address-list get $j address] = $tmpAddress ) do={
                    :set bNew "false";
                }
            }
#------- if address is new then add to address list
            :if ( $bNew = "true" ) do={
                /ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no
                }
            }
        }
    }
}

# [ THE END ]

I have added for loop as above to calculate the subnet. is this correct and working?
 
tomtom80
newbie
Posts: 43
Joined: Sun Jan 06, 2008 12:48 am

Re: Fight against P2P on port 80

Mon Feb 11, 2008 1:00 am

mrz's script doesnt work at my MT..
like radocicala wrote .. It writes every ip saved in cache in DNS !
same at my MT!
I copied and pasted the code. So no type error possible!
i have version 2.9.49 .. maybe thats the reason why it doesnt work?!?

here ones more mrz's code ..
# check every dns entry
:foreach i in=[/ip dns cache find] do={
    :local bNew "true";
#  check if dns name contains rapidshare
    :if ([:find [/ip dns cache get $i name] "rapidshare"] != 0) do={
        :local tmpAddress [/ip dns cache get $i address] ;
#---- if address list is empty do not check ( add address directly )
        :if ( [/ip firewall address-list find ] = "") do={
            /ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no;
        } else={
#------- check every address list entry
            :foreach j in=[/ip firewall address-list find ] do={
#---------- set bNew variable to false if address exists in address list
                :if ( [/ip firewall address-list get $j address] = $tmpAddress ) do={
                    :set bNew "false";
                }
            }
#------- if address is new then add to address list
            :if ( $bNew = "true" ) do={
                /ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no
            }
        }
    }
}

# [ THE END ]
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7053
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Fight against P2P on port 80

Mon Feb 11, 2008 11:59 am

mrz's script doesnt work at my MT..
like radocicala wrote .. It writes every ip saved in cache in DNS !
same at my MT!
I copied and pasted the code. So no type error possible!
i have version 2.9.49 .. maybe thats the reason why it doesnt work?!?
This script was written initially on version 3.0rc. It was working fine on all 3.x versions since. I haven't tested it on 2.9. Probably it isn't working on 2.9 because you have to use nil or something else to compare:
:if ([:find $cacheName "ss.lv"] != nil ) do={ .....
 
tomtom80
newbie
Posts: 43
Joined: Sun Jan 06, 2008 12:48 am

Re: Fight against P2P on port 80

Tue Feb 12, 2008 11:56 am

Thank You MRZ!!

I upgraded to v3.2 an it works fine!! :D

Is it possible that these addresses has got a Timeout of for example one week ? (like in Firewall - Mangle, there you can define a Timeout)
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7053
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Fight against P2P on port 80

Tue Feb 12, 2008 12:13 pm

No you can't set timeout because these are static entries. Probably you can write a script that will remove all entries in this list
 
pospanko
Member Candidate
Member Candidate
Posts: 279
Joined: Sun Dec 18, 2005 4:23 pm

Re: Fight against P2P on port 80

Tue Feb 19, 2008 7:34 pm

This script adds /24 type addresses to address-list in mrz's code. I'm newbie in scripting so maybe ther is more cleaner way to do that but...
# check every dns entry
:foreach i in=[/ip dns cache find] do={
    :local bNew "true";
#  check if dns name contains rapidshare
    :if ([:find [/ip dns cache get $i name] "rapidshare"] != 0) do={
        :local tmpAddress [/ip dns cache get $i address] ;

# convert IP to "/24" type address
        :local mjesto ([:find $tmpAddress "."]);
        :set mjesto ([:find $tmpAddress "." $mjesto]);
        :set mjesto ([:find $tmpAddress "." $mjesto]);
        :local tmpAddress ([:pick $tmpAddress 0 $mjesto]);
        :set tmpAddress ($tmpAddress . ".0/24");

#---- if address list is empty do not check ( add address directly )
        :if ( [/ip firewall address-list find ] = "") do={
            /ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no;
        } else={
#------- check every address list entry
            :foreach j in=[/ip firewall address-list find ] do={
#---------- set bNew variable to false if address exists in address list
                :if ( [/ip firewall address-list get $j address] = $tmpAddress ) do={
                    :set bNew "false";
                }
            }
#------- if address is new then add to address list
            :if ( $bNew = "true" ) do={
                /ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no
            }
        }
    }
}
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7053
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Fight against P2P on port 80

Mon Mar 17, 2008 12:13 am

You added some lines to script that changes /32 addresses to /24 network address.

Remove these lines:
# convert IP to "/24" type address
        :local mjesto ([:find $tmpAddress "."]);
        :set mjesto ([:find $tmpAddress "." $mjesto]);
        :set mjesto ([:find $tmpAddress "." $mjesto]);
        :local tmpAddress ([:pick $tmpAddress 0 $mjesto]);
        :set tmpAddress ($tmpAddress . ".0/24");
 
sgsmc
just joined
Posts: 9
Joined: Wed Jul 25, 2007 5:01 pm

Re: Fight against P2P on port 80

Wed Mar 19, 2008 11:52 am

mrz

The script works with v3.2 fine.
The problem is back again with v3.4. It writes all the addresses from cache in v3.4.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26368
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Fight against P2P on port 80

Wed Mar 19, 2008 11:56 am

mrz

The script works with v3.2 fine.
The problem is back again with v3.4. It writes all the addresses from cache in v3.4.
there is a bug in 3.4 scripting, it will be fixed in 3.5 which should be out sometime this week
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7053
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Fight against P2P on port 80

Wed Mar 19, 2008 12:10 pm

There are some console bugs in 3.4, downgrade to 3.2 or 3.3 and wait for 3.5 release.
 
sgsmc
just joined
Posts: 9
Joined: Wed Jul 25, 2007 5:01 pm

Re: Fight against P2P on port 80

Thu Mar 20, 2008 3:58 pm

Thanks mrz and Thanks Normis

Mikrotik was kind to release v3.5 very soon. I have loaded v3.5 and now it works fine.

mrz one problem still remains unresolved with us:

if there is subnet´s ip in address list(237.138.168.0/24) it is also writes the ip´s that belong that subnet(207.138.168.61)

I promise you we tried writing various codes over midnight cofee cups but none seems to be working.

I am a Newbie to scripting and still on a learnig curve
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7053
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Fight against P2P on port 80

Thu Mar 20, 2008 4:44 pm

This is a feature address list allows you to write 207.138.168.61 even if 207.138.168.0/24 already exist.
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8709
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Fight against P2P on port 80

Fri Mar 21, 2008 2:27 am

even more: address list allows you to write 207.138.168.61 even if 207.138.168.61 already exist =)
and if you delete second 207.138.168.61, ROS will act as you deleted both of them, until you disable and then enable residual item. it's not a feature, i'ts a BUG, but i write to support when v3.3 was the latest, still not resloved.

Normis?..
 
radocicala
Member Candidate
Member Candidate
Posts: 136
Joined: Fri Aug 10, 2007 6:56 pm

Re: Fight against P2P on port 80

Tue Apr 08, 2008 10:38 am

I tried this script to find rapidshare and other servers in cache, but problem is that script works well just under the routeros v3.
Could you edit it to work well under v2.9?

# check every dns entry
:foreach i in=[/ip dns cache find] do={
:local bNew "true";
# check if dns name contains rapidshare
:if ([:find [/ip dns cache get $i name] "rapidshare"] != 0) do={
:local tmpAddress [/ip dns cache get $i address] ;

#---- if address list is empty do not check ( add address directly )
:if ( [/ip firewall address-list find ] = "") do={
/ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no;
} else={
#------- check every address list entry
:foreach j in=[/ip firewall address-list find ] do={
#---------- set bNew variable to false if address exists in address list
:if ( [/ip firewall address-list get $j address] = $tmpAddress ) do={
:set bNew "false";
}
}
#------- if address is new then add to address list
:if ( $bNew = "true" ) do={
/ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no
}
}
}
}
}

# [ THE END ]
 
User avatar
kolorasta
Member
Member
Posts: 304
Joined: Sun Jun 25, 2006 11:55 pm
Location: Argentina

Re: Fight against P2P on port 80

Thu Apr 10, 2008 9:25 pm

usefull script.
how can I add an "OR" condition, something like this:

rapidshare OR megaupload OR gigasize OR ..... OR ....

???

any suggestions?
 
User avatar
kolorasta
Member
Member
Posts: 304
Joined: Sun Jun 25, 2006 11:55 pm
Location: Argentina

Re: Fight against P2P on port 80

Fri Apr 11, 2008 4:12 pm

@normis

Cpresto is aware of that, but many clients use rapidshare, and downloadind all day long. So it is like p2p bandwith is 100% used.

@cpresto

Cmon man, what do you want? buy more bandwith.

Do you expect from your customers to read&send mail. Yeah right. :roll:
Buy more bandwidth!!! it's easy to say where 1mbps cost 30-40 dollars or something like that... in my country (w)isp pay more than u$s500 for 1mbps.
1mbps cost from u$s500 to u$s2000 in my country. Our clients pay for a 512kbps access u$s20... rapidshare, megaupload, gigasize, all-p2p, are really a headache to us.
 
User avatar
kolorasta
Member
Member
Posts: 304
Joined: Sun Jun 25, 2006 11:55 pm
Location: Argentina

Re: Fight against P2P on port 80

Fri Apr 11, 2008 4:21 pm

I tried this script to find rapidshare and other servers in cache, but problem is that script works well just under the routeros v3.
Could you edit it to work well under v2.9?

# check every dns entry
:foreach i in=[/ip dns cache find] do={
:local bNew "true";
# check if dns name contains rapidshare
:if ([:find [/ip dns cache get $i name] "rapidshare"] != 0) do={
:local tmpAddress [/ip dns cache get $i address] ;

#---- if address list is empty do not check ( add address directly )
:if ( [/ip firewall address-list find ] = "") do={
/ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no;
} else={
#------- check every address list entry
:foreach j in=[/ip firewall address-list find ] do={
#---------- set bNew variable to false if address exists in address list
:if ( [/ip firewall address-list get $j address] = $tmpAddress ) do={
:set bNew "false";
}
}
#------- if address is new then add to address list
:if ( $bNew = "true" ) do={
/ip firewall address-list add address=$tmpAddress list=rapidshare disabled=no
}
}
}
}
}

# [ THE END ]
how often do you execute this script? (schedule)
 
jcremin
Member
Member
Posts: 360
Joined: Fri May 25, 2007 7:57 am

Re: Fight against P2P on port 80

Fri Apr 11, 2008 11:44 pm

1mbps cost from u$s500 to u$s2000 in my country. Our clients pay for a 512kbps access u$s20...
Sounds like you maybe aren't charging enough if your bandwidth costs that much..
 
User avatar
kolorasta
Member
Member
Posts: 304
Joined: Sun Jun 25, 2006 11:55 pm
Location: Argentina

Re: Fight against P2P on port 80

Tue Apr 15, 2008 7:55 am

1mbps cost from u$s500 to u$s2000 in my country. Our clients pay for a 512kbps access u$s20...
Sounds like you maybe aren't charging enough if your bandwidth costs that much..
i know that... :(
 
User avatar
hulk-bd
Member Candidate
Member Candidate
Posts: 231
Joined: Mon Sep 03, 2007 7:19 pm
Location: Uttara, Dhaka, Bangladesh

Re: Fight against P2P on port 80

Tue Apr 15, 2008 9:05 am

Dude you are talking about bandwidth price! in our country we had to pay near 2000 $ US for 1 mb and from few days the cost is bearable now we have to pay 1000$ US per mb, so this is costly than them who pay nearly 50 to 80 $ US per mb. I'm sorry cause this is now going away from the main topic but can't help writing this. :)

Peace
 
nitrium
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Wed Apr 04, 2007 9:00 pm

Re: Fight against P2P on port 80

Thu May 15, 2008 3:09 am

Hi guys, well.. I own an ISP and i have to deal with this stuff also! My workaround was inside the queues. I just made good use of Limit-at and Max-limit. Set Max-Limit of your client as the maximum bandwidth he can use, and Limit-at you can set at 15% or 30% of the maximum bandwidth. The most important thing is that in hierarchy of queues, the top most important must not exceed your total bandwidth with the sum of Limit-At rules! I hope it works for you as it worked for me... Since users with Download Managers and Rapidshare accounts Premium get lots of bandwidth available, they normally consume all of your bandwidth, but with this rules, they will drop if other users need the bandwidth too! And make a GOOD CONFIGURED PROXY working, this will help A LOT!!! Cya around!!!
 
BeNBeN
just joined
Posts: 18
Joined: Fri Oct 09, 2009 3:13 pm

Re: Fight against P2P on port 80

Thu Nov 05, 2009 2:12 pm

Write "rapidshare" in content field in the filter rule that you created for rapidshare. it works for me. it block also web pages which has rapidshare links in it.
 
ManyX
Member Candidate
Member Candidate
Posts: 111
Joined: Sat Jan 07, 2006 12:48 pm
Location: PolanD

Re: Fight against P2P on port 80

Wed Jan 06, 2010 10:31 pm

hello

http://iblocklist.com/list.php?list=zfu ... kalytktyiw

list of rapidshare IP

I am looking for megaupload, megavideo list
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 2101
Joined: Mon Jan 14, 2008 1:53 pm
Location: Over the Rainbow
Contact:

Re: Fight against rapidshare

Thu Jan 07, 2010 11:17 pm

This guy creates a nightly list of RapidShare IP addresses, and even published a .rsc that can be imported to Mikrotik.

http://www.uebi.net/howtos/rapidshare-networks.htm
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 2101
Joined: Mon Jan 14, 2008 1:53 pm
Location: Over the Rainbow
Contact:

Re: Fight against rapidshare

Fri Jan 08, 2010 12:01 am

I had an idea of a more accurate way to do this, by finding RapidShare's AS and then pushing all associated prefixes into an address list. This was a great theory, then I discovered they seem to be using their upstream providers ASN of 3356 :(
 
User avatar
kolorasta
Member
Member
Posts: 304
Joined: Sun Jun 25, 2006 11:55 pm
Location: Argentina

Re: Fight against rapidshare

Fri Jan 08, 2010 12:27 am

i use a script that search in the dns cache for all ips with "rapidshare" in the dns and store those ips in a address-list

then with that address-list you do whatever you want

sorry for my poor english
 
roadracer96
Forum Veteran
Forum Veteran
Posts: 733
Joined: Tue Aug 25, 2009 12:01 am

Re: Fight against rapidshare

Fri Jan 08, 2010 4:18 am

Rapidshare isnt p2p. Rapidshare is simply downloading a file from the net. Lets say I have 100mb of pictures I want to email you. But emailing them would be dumb. I zip them, put them on rapidshare and send you the link to download it.

By blocking it, you could be blocking legit computer use and even business related use.

I dont know why people get so butt-hurt over p2p anyways. Just shape it down.
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 2101
Joined: Mon Jan 14, 2008 1:53 pm
Location: Over the Rainbow
Contact:

Re: Fight against rapidshare

Fri Jan 08, 2010 12:18 pm

Lol, thats what this entire thread is about, being able to accurately identify traffic to RapidShare IP addresses, mark it and then shape it...
 
rodolfo
Long time Member
Long time Member
Posts: 553
Joined: Sat Jul 05, 2008 11:50 am

Re: Fight against rapidshare

Fri Jan 08, 2010 3:23 pm

In my opinion this approach is incorrect.
You need to run this script for a lot of content providers.

The best approach was indicated two times in this post: mangle with different packet marks http connection with less than 5MBytes and http connection with >= 5MBytes.

To the first give high priority (is browsing or download of small files); to the second give lower priority (you can use also pcq queue!).

This method will funcionction for all http downloads and needs no maintenance.
 
rmichael
Forum Veteran
Forum Veteran
Posts: 718
Joined: Sun Mar 08, 2009 11:00 pm

Re: Fight against rapidshare

Fri Jan 08, 2010 4:06 pm

I tend to agree. However it's worth to note that connection size approach will "waste" 5GB of perfectly good bandwidth where ID by IP will "waste" none. Hence knowing that traffic is of certain kind immediatelly has it's value especially for slower links where 5GB download takes more than few seconds.
 
User avatar
m4rk0
Member Candidate
Member Candidate
Posts: 196
Joined: Sat Feb 16, 2008 8:30 pm
Location: BA
Contact:

Re: Fight against rapidshare

Sat Aug 28, 2010 10:58 pm

Hotfile.com IP ranges:

74.120.8.0/22
199.7.176.0/21
 
sewlist
Frequent Visitor
Frequent Visitor
Posts: 70
Joined: Fri Jun 02, 2006 3:48 pm

Re: Fight against rapidshare

Sun Aug 29, 2010 10:18 am

Not sure if this might help, we also quite a large WISP, we push heavy traffic, found that using a squid proxy server can save alot of data and u can manage certain sites

my problem was windowsupdates during the day as we have a 100mb internet breakout, so I slow it down to very small speed during the day and at night open it up

You can just add rapidshare there or make a new acl for it

acl winupdate dstdomain .windowsupdate.com
acl peakperiod time 06:00-23:00
delay_pools 1
delay_class 1 1
# 128 Kbit/s
delay_parameters 1 128000/128000
delay_access 1 allow winupdate peakperiod


S
 
ruhanda
just joined
Posts: 9
Joined: Wed May 12, 2010 12:02 pm
Location: Balikpapan, Indonesia

Re: Fight against rapidshare

Wed Sep 01, 2010 1:47 pm

Hi All,

Actually rapidshare is only web site that provide space so user can store and retrieve file by uploading or downloading files.
the problem is downloading and uploading process using same port 80 with HTTP.
In my internet cafe I'm using layer 7 protocol,firewall and queue to separate regular browsing and downloading files by set download priority lower then HTTP (Browsing).
Sometimes filter using packet-size is not effective.

Due to user doing Download more intense than upload so the code is not separate it when upload.
Please find the firewall code below,
# sep/01/2010 17:19:23 by RouterOS 4.9
/ip firewall layer7-protocol
add comment="" name=EXE regexp="^.*get.+\\.exe.*\$"
add comment="" name=RAR regexp="^.*get.+\\.rar.*\$"
add comment="" name=ZIP regexp="^.*get.+\\.zip.*\$"
add comment="" name=7z regexp="^.*get.+\\.7z.*\$"
add comment="" name=FLV regexp="^.*get.+\\.flv.*\$"
add comment="" name=WMV regexp="^.*get.+\\.wmv.*\$"
add comment="" name=MP3 regexp="^.*get.+\\.mp3.*\$"
add comment="" name=MP4 regexp="^.*get.+\\.mp4.*\$"
add comment="" name=3GP regexp="^.*get.+\\.3gp.*\$"

/ip firewall address-list
add address=192.168.1.0/24 comment="" disabled=no list=Local_Networks
add address=192.168.2.0/24 comment="" disabled=no list=Local_Networks
add address=10.5.50.0/24 comment="" disabled=no list=Local_Networks
add address=10.5.51.0/24 comment="" disabled=no list=Local_Networks

/ip firewall filter
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download EXE" disabled=no \
    layer7-protocol=EXE protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download 7z" disabled=no \
    layer7-protocol=7z protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download ISO" content=.iso \
    disabled=no protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download ZIP" disabled=no \
    layer7-protocol=ZIP protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download MPEG" content=\
    .mpeg disabled=no protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download MPG" content=.mpg \
    disabled=no protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download FLV" disabled=no \
    layer7-protocol=FLV protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download 3GP" disabled=no \
    layer7-protocol=3GP protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download RM" content=.rm \
    disabled=no protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download AVI" content=.avi \
    disabled=no protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download RAR" disabled=no \
    layer7-protocol=RAR protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download MP4" disabled=no \
    layer7-protocol=MP4 protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download MKV" content=.mkv \
    disabled=no protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download MOV" content=.mov \
    disabled=no protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download MSI" content=.msi \
    disabled=no protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download WAV" content=.wav \
    disabled=no protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download WMV" disabled=no \
    layer7-protocol=WMV protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download WMA" content=.wma \
    disabled=no protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download MP3" disabled=no \
    layer7-protocol=MP3 protocol=tcp src-address-list=Local_Networks
add action=add-dst-to-address-list address-list=DOWNLOAD \
    address-list-timeout=1h chain=forward comment="Download Videoplayback" \
    content=videoplayback disabled=no protocol=tcp src-address-list=\
    Local_Networks

/ip firewall mangle
add action=mark-connection chain=prerouting comment=DOWNLOAD disabled=no \
    dst-address-list=DOWNLOAD dst-port=80,8080,3128 in-interface=LAN \
    new-connection-mark=QoS_5_con_downld passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting comment="" connection-mark=\
    QoS_5_con_downld disabled=no new-packet-mark=QoS_5 passthrough=no
add action=mark-connection chain=prerouting comment="" disabled=no \
    dst-address-list=DOWNLOAD dst-port=110,995,143,993,25,20,21 in-interface=\
    LAN new-connection-mark=QoS_5_con_downld passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting comment="" connection-mark=\
    QoS_5_con_downld disabled=no new-packet-mark=QoS_5 passthrough=no
add action=mark-connection chain=prerouting comment=BROWSING disabled=no \
    dst-port=80,8080,3128,443,7778 in-interface=LAN new-connection-mark=\
    QoS_4_con packet-size=0-666 passthrough=yes protocol=tcp tcp-flags=syn
add action=mark-packet chain=prerouting comment="" connection-mark=QoS_4_con \
    disabled=no new-packet-mark=QoS_4 passthrough=no
add action=mark-connection chain=prerouting comment="" connection-bytes=\
    0-1000000 disabled=no dst-port=80,8080,3128,443,7778 in-interface=LAN \
    new-connection-mark=QoS_4_con passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting comment="" connection-mark=QoS_4_con \
    disabled=no new-packet-mark=QoS_4 passthrough=no
add action=mark-connection chain=prerouting comment="" disabled=no dst-port=\
    110,995,143,993,25,20,21 in-interface=LAN new-connection-mark=QoS_4_con \
    packet-size=0-666 passthrough=yes protocol=tcp tcp-flags=syn
add action=mark-packet chain=prerouting comment="" connection-mark=QoS_4_con \
    disabled=no new-packet-mark=QoS_4 passthrough=no
add action=mark-connection chain=prerouting comment="" connection-bytes=\
    0-1000000 disabled=no dst-port=110,995,143,993,25,20,21 in-interface=LAN \
    new-connection-mark=QoS_4_con passthrough=yes protocol=tcp
add action=mark-packet chain=prerouting comment="" connection-mark=QoS_4_con \
    disabled=no new-packet-mark=QoS_4 passthrough=no
Simple Queue code
# sep/01/2010 17:19:54 by RouterOS 4.9
# 
#
/queue type
set default-small kind=pfifo name=default-small pfifo-limit=10

/queue simple
add burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s comment="" \
    direction=both disabled=no dst-address=0.0.0.0/0 interface=all limit-at=\
    256k/2M max-limit=500k/3M name=TOTAL parent=none priority=5 queue=\
    default-small/default-small target-addresses=\
    192.168.2.0/24,10.5.50.0/24,10.5.51.0/24 total-queue=default-small
add burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s comment="" \
    direction=both disabled=no dst-address=0.0.0.0/0 interface=all limit-at=\
    32k/128k max-limit=256k/2M name=QoS_4-Browsing packet-marks=QoS_4 parent=\
    TOTAL priority=4 queue=default-small/default-small target-addresses=\
    192.168.2.0/24,10.5.50.0/24,10.5.51.0/24 total-queue=default-small
add burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s comment="" \
    direction=both disabled=no dst-address=0.0.0.0/0 interface=all limit-at=\
    16k/32k max-limit=128k/2M name=QoS_5-Download packet-marks=QoS_5 parent=\
    TOTAL priority=5 queue=default-small/default-small target-addresses=\
    192.168.2.0/24,10.5.50.0/24,10.5.51.0/24 total-queue=default-small
For best result you can use PCQ and Queue Tree. but for me simple queue is enough for my internet cafe and all my customer satisfied with current internet cafe network speed.

Adding and modify the code are welcome.

Thanks,
Ruhanda

Who is online

Users browsing this forum: No registered users and 52 guests