Need Help In Layer7 || isolation between Browsing , download

RouterOS Scripting
1

i used layer 7 for identifying the context of web pages

and i want to do something that’s important

and want to make Mikrotik sense and make difference between download traffic and browsing traffic

To make it isolate between Browsing and download

and better thing i thought about was

extension of files

like
exe , rar , zip , iso , daa , avi and etc…

and iam doing this by putting for example exe word as Regular Expression in Regexp Textbox in layer 7 filter

and make rule in Firewall Mangle to mark packet that contain layer 7 condition
as download packet

and in the Queue what ever simple Queue or Queue Tree i shape the traffic with the nice speed i want to.

the big problem i just found

that the layer 7 search about the word exe in the entire code of the HTML page

Head and Body

not only the Head Code

and any one know that we want layer 7 search just in the Head code “Search Between just Head Tags” because The URL in contained in between Head Tags

and if it searched about the word “flv” for example in the body code of the web page there is flashes and many web pages contain small flv files in the body code and by this layer 7 when find flv work in the body code of the page filter it as download

and take for another example if some one explain in a tutorial or topic the extensions or just mentioned any of the extensions i shape in the body of the page

the page then is shaped

so any one know the Regular Expression code that make layer 7 search just in only the Head of the HTML Code

and make it not to search in the Body " Body Code "of the page please tell us

2

you have to realize that downloading = browsing only thing that is different is size of objects transferred. Wile you can find some patterns using layer-7, use of it for purpose you want to would not be good solution due to resource use of layer-7 and quite substantial chance of false positives.

Focus on connection bytes, as objects and pictures of web page will be considerably smaller than say.

3

add tags to your regexp, but as janisk said searching those patterns will take a lot of rotuer’s resources.

4

ok not matter the resources that it consume if it works

Specifications of my server " Quad Core Processor and 4 GB of RAM "

So not matter

but how i add to the word exe for example in regexp

loke this “exe”
or what

i want to make layer7 just earch exclusively between

............href="[www.example.com/file](http://www.example.com/file).**exe**"......... ...........This page is to learn extensions and how it work for example **exe** is the extension of the executable files ...... bla bla bla

if we notice here the exe word just came twice , one time between head tags and the other time between body tags

and any one know that body is the context of the entire web page
and head tags just contain the title and URL and some fewer tags
so we

5

At first you have to learn how to write regular expressions.
Here is good starting point: http://www.regular-expressions.info/

Then write regular expression that matches “.exe” inside … tags.

6

ok i read this topic many days ago very carefully , but i can not till now write the right code

to make the layer7 search just in between Head Tags

Head Code

any one know the code just make it easier to me and write the code here

and take exe for example.

7

ok making it much easier to any one want to help

this code of Regular Expressions take any extension , not specific extension

.[^.\/:*?"<>|\r\n]+$


and this code take the entire HTML web Page file

.*?.*?.*?.*?.*?]*>.*?.*?

and this one can capture exe extension if it came between two tags

(<head[^>]*>)(exe)()

it will take exe if the page code was like this

exe

but will not take affect if the code of the page was like this

any other words here besides exe \ \ with these codes any one can help us ??

these are last thing i could reach

and i think with more customize to one of the last codes we can get the code
of search for specific extension just between Head Code Tags

8

(<head[^>]*>)(.*exe)()

9

First of all thanks for paying attention to my question

This worked true

but for just if the example applied on this code

\

any words here before exe \ \ and no space between exe or even any other character

if it just take spaces of characters no code will be applied

and you can see yourself in layer7 as a test or using RegexBuddy Programs

i added .* before exe and after it and it works

(<head[^>]*>)(.exe.)()


and i have tested it , it worked successfully on like this HTML code

Download Page

but still one problem

if the HTML code was like previous one but the tags close came in another line not in the same line RE code never take effect on HTML code

so need a simple add-in to the last RE code

(<head[^>]*>)(.exe.)()

to make it what ever () came in the same line or in another line will take effect also.

Thanks in advanced to anyone pay attention.

10

No one can give us any hope ???

11

what is RE code?.. :confused:

maybe try <head[^>]>.exe.(\r\n).* or something like that…

12

Ok fine

thanks for paying attention to me

RE i mean with it Regular Expressions

the last code worked for me fine

but i noticed that if we have a line such this

http://download.microsoft.com/download/C/C/0/CC0BD555-33DD-411E-936B-73AC6F95AE11/IE8-WindowsXP-x86-ENU.exe

and it not included in the head tag the code will not work

so we need another simple code that say that if http word & exe word came in the same line catch it

so we can catch any http traffic contain the executable files “exe files”

thanks in advance.

13

L7 inspects only first 2KB of data… your detecting may fail because of that…

p.s. if the page is linked to some .exe file, it doesn’t mean the file will be downloaded…

14

that is why use connection-bytes in the first place, so if connection takes longer than say, 2MB (for flash page to load) the throttle it down a notch or two.

Only difference between downloading and browsing is that downloading transfers more bytes per connection.

15

ok fine i know that very well

but i want just to add something

if i just added the word exe in the layer7 regexp
and if the link of the file contain the word exe

for example the extension was .exe

the rule of mangle applied very well and the shaping of Queue work very well
the only wrong thing happening if the HTML page contain in it’s body code the word exe
for example someone mentioned the word executable and the exe part of it and etc…

so the only problem here
we want to add condition to the word exe in regexp textbox so we can say with this codition

if the word exe came in the same line with the word http

capture it ok i think you got it now
for example

http://anydomain.com/anyurl.exe

here http and exe just come in the same line if this line if download link ok

i want a code for this

to say if the http & exe came in the same line capture it

thanks for who will help in advanced.

16 
https?://.*\.exe

?..

17

when i used this in RegexBuddy Software it worked correctly
but when using it in layer7 it did not work

and i will give you the code and you can try it yourself


/ip firewall layer7-protocol
add comment=“” name=exe regexp=“https?://.*\.exe”

/ip firewall mangle
add action=mark-packet chain=forward comment=“” disabled=no dst-port=80 layer7-protocol=exe new-packet-mark=
“Exe Download” passthrough=yes protocol=tcp src-address=192.168.0.0/24


the same code when i replace “https?://.*\.exe” with just exe

the mangle capture the traffic but with the code you gave me it did not work

and of course we can not use just exe word due to the reason i mentioned before
if the word exe came in the body code of the HTML Page it will be captured

so we want to make our code search Exclusively in between URL’s “http://…words…exe”

and you can take a download link for example contains exe file to try on it


http://download.microsoft.com/download/C/C/0/CC0BD555-33DD-411E-936B-73AC6F95AE11/IE8-WindowsXP-x86-ENU.exe

and finally i want to add that you are the most one who was active with me
and most helpfull one for me so if this problem has been solved or not
thanks at all.

18

I see the main reason:

/ip firewall mangle
add action=mark-packet chain=forward comment=“” disabled=no dst-port=80 layer7-protocol=exe new-packet-mark=
“Exe Download” passthrough=yes protocol=tcp src-address=192.168.0.0/24

you check only upload traffic, not traffic from the website. in upload (requests), it looks like

GET /path/to.exe HTTP/1.1
Host: http://www.example.com
<bla-bla-bla>

there’s no “http://some/thing.exe” string in those packets

19

At all as i said to you before thank you very much

I have reached the method which enables me to shape download traffic Via extensions of files
and i will give you the Entire code right here soon to you Allah willing.
It worked for me 100 %

if i download such a exe , flv, rar , zip…bla..bla..bla files
the traffic is shaped if any else browsing traffic
traffic is not shaped

and thanks very much for you effort with me.

20

As i promised that i will give you the code
here is it

/ip firewall layer7-protocol
add comment=“” name=“Extension " .exe "” regexp=“\.(exe)”
add comment=“” name=“Extension " .rar "” regexp=“\.(rar)”
add comment=“” name=“Extension " .zip "” regexp=“\.(zip)”
add comment=“” name=“Extension " .7z "” regexp=“\.(7z)”
add comment=“” name=“Extension " .cab "” regexp=“\.(cab)”
add comment=“” name=“Extension " .asf "” regexp=“\.(asf)”
add comment=“” name=“Extension " .mov "” regexp=“\.(mov)”
add comment=“” name=“Extension " .wmv "” regexp=“\.(wmv)”
add comment=“” name=“Extension " .mpg "” regexp=“\.(mpg)”
add comment=“” name=“Extension " .mpeg "” regexp=“\.(mpeg)”
add comment=“” name=“Extension " .mkv "” regexp=“\.(mkv)”
add comment=“” name=“Extension " .avi "” regexp=“\.(avi)”
add comment=“” name=“Extension " .flv "” regexp=“\.(flv)”
add comment=“” name=“Extension " .pdf "” regexp=“\.(pdf)”
add comment=“” name=“Extension " .wav "” regexp=“\.(wav)”
add comment=“” name=“Extension " .rm "” regexp=“\.(rm)”
add comment=“” name=“Extension " .mp3 "” regexp=“\.(mp3)”
add comment=“” name=“Extension " .mp4 "” regexp=“\.(mp4)”
add comment=“” name=“Extension " .ram "” regexp=“\.(ram)”
add comment=“” name=“Extension " .rmvb "” regexp=“\.(rmvb)”
add comment=“” name=“Extension " .dat "” regexp=“\.(dat)”
add comment=“” name=“Extension " .daa "” regexp=“\.(daa)”
add comment=“” name=“Extension " .iso "” regexp=“\.(iso)”
add comment=“” name=“Extension " .nrg "” regexp=“\.(nrg)”
add comment=“” name=“Extension " .bin "” regexp=“\.(bin)”
add comment=“” name=“Extension " .vcd "” regexp=“\.(vcd)”

/ip firewall mangle
add action=mark-connection chain=prerouting comment=“7z DOWNS” disabled=no layer7-protocol=“Extension " .7z "” new-connection-mark=“7z DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“7z DOWNS” disabled=no new-packet-mark=7z passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“asf DOWNS” disabled=no layer7-protocol=“Extension " .asf "” new-connection-mark=“asf DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“asf DOWNS” disabled=no new-packet-mark=asf passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“avi DOWNS” disabled=no layer7-protocol=“Extension " .avi "” new-connection-mark=“avi DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“avi DOWNS” disabled=no new-packet-mark=avi passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“bin DOWNS” disabled=no layer7-protocol=“Extension " .bin "” new-connection-mark=“bin DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“bin DOWNS” disabled=no new-packet-mark=bin passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“flv DOWNS” disabled=no layer7-protocol=“Extension " .flv "” new-connection-mark=“flv DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“flv DOWNS” disabled=no new-packet-mark=flv passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“iso DOWNS” disabled=no layer7-protocol=“Extension " .iso "” new-connection-mark=“iso DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark= “iso DOWNS” disabled=no new-packet-mark=iso passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“mkv DOWNS” disabled=no layer7-protocol=“Extension " .mkv "” new-connection-mark=“mkv DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“mkv DOWNS” disabled=no new-packet-mark=mkv passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“exe DOWNS” disabled=no layer7-protocol=“Extension " .exe "” new-connection-mark=“exe DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“exe DOWNS” disabled=no new-packet-mark=exe passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“mov DOWNS” disabled=no layer7-protocol=“Extension " .mov "” new-connection-mark=“mov DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“mov DOWNS” disabled=no new-packet-mark=mov passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“mp3 DOWNS” disabled=no layer7-protocol=“Extension " .mp3 "” new-connection-mark=“mp3 DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“mp3 DOWNS” disabled=no new-packet-mark=mp3 passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“mp4 DOWNS” disabled=no layer7-protocol=“Extension " .mp4 "” new-connection-mark=“mp4 DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“mp4 DOWNS” disabled=no new-packet-mark=mp4 passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“mpeg DOWNS” disabled=no layer7-protocol=“Extension " .mpeg "” new-connection-mark=“mpeg DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“mpeg DOWNS” disabled=no new-packet-mark=mpeg passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“mpg DOWNS” disabled=no layer7-protocol=“Extension " .mpg "” new-connection-mark=“mpg DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“mpg DOWNS” disabled=no new-packet-mark=mpg passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“nrg DOWNS” disabled=no layer7-protocol=“Extension " .nrg "” new-connection-mark=“nrg DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“nrg DOWNS” disabled=no new-packet-mark=nrg passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“pdf DOWNS” disabled=no layer7-protocol=“Extension " .pdf "” new-connection-mark=“pdf DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“pdf DOWNS” disabled=no new-packet-mark=pdf passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“ram DOWNS” disabled=no layer7-protocol=“Extension " .ram "” new-connection-mark=“ram DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“ram DOWNS” disabled=no new-packet-mark=ram passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“rar DOWNS” disabled=no layer7-protocol=“Extension " .rar "” new-connection-mark=“rar DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“rar DOWNS” disabled=no new-packet-mark=rar passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“rm DOWNS” disabled=no layer7-protocol=“Extension " .rm "” new-connection-mark=“rm DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“rm DOWNS” disabled=no new-packet-mark=rm passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“rmvb DOWNS” disabled=no layer7-protocol=“Extension " .rmvb "” new-connection-mark=“rmvb DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“rmvb DOWNS” disabled=no new-packet-mark=rmvb passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“wav DOWNS” disabled=no layer7-protocol=“Extension " .wav "” new-connection-mark=“wav DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“wav DOWNS” disabled=no new-packet-mark=wav passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“wma DOWNS” disabled=no layer7-protocol=“Extension " .wma "” new-connection-mark=“wma DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“wma DOWNS” disabled=no new-packet-mark=wma passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“wmv DOWNS” disabled=no layer7-protocol=“Extension " .wmv "” new-connection-mark=“wmv DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“wmv DOWNS” disabled=no new-packet-mark=wmv passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“zip DOWNS” disabled=no layer7-protocol=“Extension " .zip "” new-connection-mark=“zip DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“zip DOWNS” disabled=no new-packet-mark=zip passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“youtube DOWNS” disabled=no layer7-protocol=“YouTube " new-connection-mark=“youtube DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=”" connection-mark=“youtube DOWNS” disabled=no new-packet-mark=youtube passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“daa DOWNS” disabled=no layer7-protocol=“Extension " .daa "” new-connection-mark=“daa DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“daa DOWNS” disabled=no new-packet-mark=daa passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“dat DOWNS” disabled=no layer7-protocol=“Extension " .dat "” new-connection-mark=“dat DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“dat DOWNS” disabled=no new-packet-mark=dat passthrough=no protocol=tcp


add action=mark-connection chain=prerouting comment=“vcd DOWNS” disabled=no layer7-protocol=“Extension " .vcd "” new-connection-mark=“vcd DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“vcd DOWNS” disabled=no new-packet-mark=vcd passthrough=no protocol=tcp

add action=mark-connection chain=prerouting comment=“cab DOWNS” disabled=no layer7-protocol=“Extension " .cab "” new-connection-mark=“cab DOWNS” passthrough=yes protocol=tcp
add action=mark-packet chain=postrouting comment=“” connection-mark=“cab DOWNS” disabled=no new-packet-mark=cab passthrough=no protocol=tcp






This shape download + upload with only 1 Mb/sec to the whole Network



if you want just to shape download you can use in layer7 regexp this rule for example

/ip firewall layer7-protocol
add comment=“” name=“Extension " .exe "” regexp=“^.get.+\.exe.$”

and apply it to the rest of the extensions


if you want to shape upload only use this code in layer7


/ip firewall layer7-protocol
add comment=“” name=“Extension " .exe "” regexp=“^.set.+\.exe.$”

and you can shape Download Traffic by 64 kb on one IP like 192.168.0.2 using simple Queue instead of Queue Tree like this


/queue simple
add burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s comment=“” direction=both disabled=no dst-address=0.0.0.0/0 interface=all limit-at=1M/1M max-limit=1M/1M name=“download max” packet-marks=exe,flv,rar,zip,7z,bin,mp3,mp4,wav,pdf,asf,avi,mov,wmv,mpg,mpeg,rm,iso,nrg,daa,vcd,cab,3gp,ram,rmvb,mkv,wma,youtube parent=none priority=8 queue=default-small/default-small total-queue=default-small

and then make 64 Kb a speed for the Specified user in this parent group and refer to it’s parent the first rule like this

add burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s comment=“” direction=both disabled=no dst-address=0.0.0.0/0 interface=all limit-at=64k/64k max-limit=64k/64k name=“User " 192.168.0.2 "” parent=“download max” priority=8 queue=default-small/default-small target-addresses=192.168.0.2/32 total-queue=default-small


using this we can Shape IP 192.168.0.2 Download Traffic By 64 Kb
or what ever the speed we want to do it for the client.



the only problem this code is not working with loadbalancing ADSL Lines
is there any way to make it work ???
any one know how to do both Load Balancing + Shape download Traffic please tell us .


Notice : All code above applied on 3.3 Mikrotik OS Version