Page 1 of 1

cache all http downloads

Posted: Fri Jul 29, 2005 11:49 pm
by Hellbound
hi ppl
how can I cache all EXE.ZIP.RAR etc etc file downloading so ppl dont have to redownload second time?

I have around 800gb hard disk (4x200gb) for doing so...

mikrotik can detect that? I haven't tried to put it together.

Posted: Sat Jul 30, 2005 12:53 pm
by hzeid
well i am also intersted in this topic. if it could be done, can i let the costumers use full speed for such downloads cause i limit the connection using pcq rules, if that works it would be great cause i will stop using microsoft isa for caching. hoping to see some answers

Posted: Sat Jul 30, 2005 4:20 pm
by franco
That would be cool if it would function

Posted: Sun Jul 31, 2005 10:24 pm
by rickard
Yes it works... we do that on our system but i dont think MT support 2 drives. i use one 120 Gb SATA disk , and that work.
from the Main router we use transparent proxy with parentproxy, to a second box with proxy enabled.

Posted: Sun Jul 31, 2005 10:34 pm
by wildbill442
Actually.. I think they do support up to two drives, I was just looking through the manual the other day and I remember seeing something about that.. however it may have been the 2.9 manual i was looking at..

Posted: Mon Aug 01, 2005 12:05 am
by Hellbound
but how can I cache all EXE.ZIP.RAR etc etc file downloading?

Posted: Mon Aug 01, 2005 9:55 pm
by hzeid
Yes it works... we do that on our system but i dont think MT support 2 drives. i use one 120 Gb SATA disk , and that work.
from the Main router we use transparent proxy with parentproxy, to a second box with proxy enabled.

Pls Can u Tell how??????

Posted: Thu Aug 04, 2005 1:24 pm
by hzeid
no one interested in this topic or can provide detailed help????

Posted: Sun Aug 21, 2005 3:56 am
by goldclick
but how can I cache all EXE.ZIP.RAR etc etc file downloading?
All you need to cache EXE.ZIP.RAR should be all setup once you specify a cache drive, appropriate disk space to use and enable the web proxy. Objects/urls to be excluded from cache can be specified in the /ip web-proxy cache command. You need to decide and make a choice on maximum size of object (determines max size of EXE.ZIP.RAR) that the web-proxy keeps in cache. its a choice on speed vs bandwidth saving (more hits).
well i am also intersted in this topic. if it could be done, can i let the costumers use full speed for such downloads cause i limit the connection using pcq rules, if that works it would be great cause i will stop using microsoft isa for caching. hoping to see some answers
What I do is run a version of squid patched with TOS option marking for cached hits (see http://www.it-academy.bg/zph/) in another box. You can mark the cache hits with any TOS value of choice and then have a mangle rule in Mikrotik further mark packets with this TOS byte. The cache mangle rule stays above every other rule. You can then use Queue trees (2.8 and above) or simple Queues (2.9 and above) to decide how fast a user downloads the cached object. My config here has a Mikrotik as a bridge and bandwidth manager while the second box is Redhat 9 with transparent squid proxy patched with ZPH TOS marking. Been working great for me.

It will be nice to have mikrotik include this patch (as optional web-proxy-tos.NPK or so) and provide a configurable option to specify TOS of choice for cached hits. will be glad to throw away my second box and stick to Mikrotik's web-proxy.

Anybody doing this in another way?

Sonny

My settings...

Posted: Mon Aug 22, 2005 1:29 am
by rickard
This is my settings. we are running Hotspot on 192.168.254.1 on port 80 and the proxy is on IP 192.168.254.4 and port 8080.
And this setting works for like Http downloads from Microsoft etc..

******On the Main GW MT Router*****

/ ip firewall dst-nat

add src-address=192.168.0.0/16 in-interface=Ether2 dst-address=!192.168.254.1/32:80 protocol=tcp \
connection=!hs-auth action=redirect to-dst-port=8080 comment="Transparent Proxy" disabled=no

/ ip web-proxy
set enabled=yes src-address=0.0.0.0 port=8080 hostname="proxy" transparent-proxy=yes \
parent-proxy=192.168.254.4:8080 cache-administrator="Email adress" max-object-size="0 kB" \
cache-drive=system max-cache-size=none
/ ip web-proxy access
add dst-port=!443 method=connect action=deny comment="allow CONNECT only to SSL ports 443 \
\[https\] and 563 \[snews\]" disabled=no
add src-address=192.168.0.0/16 action=allow comment="Our Customers" disabled=no
add dst-port=23-25 action=deny comment="Deny using us as telnet ans SNMP relay" disabled=no
add action=deny comment="Deny everything else" disabled=no
/ ip web-proxy cache
add url="cgi-bin \\?" action=deny comment="don't cache dynamic http pages" disabled=no



*****On the Proxy MT router****

/ ip web-proxy
set enabled=yes src-address=0.0.0.0 port=8080 hostname="192.168.254.4" transparent-proxy=no \
parent-proxy=0.0.0.0:0 cache-administrator="email adress" max-object-size="200000 kB" \
cache-drive=secondary-master max-cache-size=unlimited
/ ip web-proxy access
add dst-port=!443 method=connect action=deny comment="allow CONNECT only to SSL ports 443 \
\[https\] and 563 \[snews\]" disabled=no
add src-address=192.168.0.0/16 action=allow comment="" disabled=no
add action=deny comment="" disabled=no
/ ip web-proxy cache
add url="cgi-bin \\?" action=deny comment="" disabled=no
add url="\\.exe\$" action=allow comment="" disabled=no
add url="\\.zip\$" action=allow comment="" disabled=no
add url="\\.mpeg\$" action=allow comment="" disabled=no
add url="\\.mp3\$" action=allow comment="" disabled=no
add url="\\.avi\$" action=allow comment="" disabled=no
add url="\\.pdf\$" action=allow comment="" disabled=no
add url="\\.rar\$" action=allow comment="" disabled=no
add url="\\.mov\$" action=allow comment="" disabled=no
add url="\\.mpg\$" action=allow comment="" disabled=no
add url="\\.dat\$" action=allow comment="" disabled=no
add action=deny comment="" disabled=no

Posted: Mon Aug 22, 2005 5:38 am
by Hellbound
I must have two mt box for doing this?

Posted: Mon Aug 22, 2005 10:09 am
by rickard
Yes

Posted: Tue Aug 23, 2005 7:20 am
by Hellbound
I tried that on one mt box and it was working,
thanks

Posted: Tue Aug 23, 2005 1:06 pm
by rickard
Im sorry yes it will work on one MT box but i dont whant to have the main router and the proxy on the same box. We have 2 main routers with fail over and one proxy server.

//Rickard

Posted: Fri Sep 30, 2005 11:09 pm
by hzeid
hi Rickard

well pls can u post ur full configuration on how it worked with u and if u found it good. i am a newbie and liked the idea but i was un able to do it without the help u can provide

Posted: Fri Sep 30, 2005 11:11 pm
by proxy
I tried that on one mt box and it was working,
thanks
which version are you using?

Posted: Sat Oct 01, 2005 1:29 am
by Hellbound
2.8

Posted: Tue Oct 04, 2005 1:07 pm
by hzeid
I tried that on one mt box and it was working,
thanks

pls can u post ur setup?

Posted: Tue Oct 04, 2005 1:13 pm
by Hellbound
setup is posted above, just remove transparent proxy IP and add the parent proxy setting to the same box, thats all