Cache Server - Plan and Design

neandero · Sat Nov 26, 2011 6:13 pm

Hello There,

Anyone interested in a Cache Server? I believe its an important aspect for any network.

Question 1:
The wiki posted all reference a box to have 2 NIC cards so is it possible with a box of 1 NIC?

Scenario -
A Routerboard with 3 LANs
LAN A = ( out ) internet
LAN B = ( server ) cache server
LAN C = ( in ) users

Here a user connected to C will make a request say a video or image
Before getting to A Lan B will look at the cache server to see if the object is available
if yes, the object is delivered but if not the request will be elevated to A and out to internet.

can we run a cache server with one nic?

Thanks,

jtroybailey · Sat Nov 26, 2011 6:39 pm

Yep, I use squid3. make sure that the cache server is on a different subnet, so you dont get issues like
"hairpin NAT". make sure that squid is in transparent mode. and than you just need to add:

/ip firewall nat
add chain=dstnat action=dst-nat to-addresses=172.19.65.250 to-ports=3128 protocol=tcp dst-address=!172.16.0.0/12 in-interface=ether3 dst-port=80

So I direct port 80, everything that isnt to a local address from ether3 to 172.19.65.250:3128

aacable · Sun Nov 27, 2011 9:02 am

The above rule is the simplest solution to redirect all HTTP request to External Proxy Server like Squid & it works fine.

But In my Opinion, Marking HTTP request and Route them to SQUID is much better. I always use packet marking to route HTTP request to squid, this way you will be able to log users ip's for record purposes , usage reports , etc , & also it will prevent double NAT. Although double natting doesn't prevent anything from being working, but routing is better then using double NAT

If you want some ideas on this topology, you can read the Following . . .
http://aacable.wordpress.com/2011/07/21 ... client-ip/

neandero · Mon Nov 28, 2011 12:28 pm

Fighting to run it on win2k

jtroybailey · Mon Nov 28, 2011 12:40 pm

Fighting to run it on win2k

run what on w2k?

neandero · Mon Nov 28, 2011 1:00 pm

squid3

jtroybailey · Mon Nov 28, 2011 1:03 pm

kind of off topic, but why would you be fighting to run squid on windows?

neandero · Mon Nov 28, 2011 1:13 pm

I always wanted Linux Ubuntu, but their last version is much graphic sometimes I don't even know what I'm doing.

WIN2K is way out dated, but its still beloved to me. I'm gonna do my best to have worked else I will go with previous versions of Ubuntu.

neandero · Mon Nov 28, 2011 1:16 pm

You know getting a newer system makes drivers impossible to find for older OS. so yes, a full day just to make win2k fully worked.

jtroybailey · Mon Nov 28, 2011 1:24 pm

Well, each to their own, but bad move in my opinion. I would use either Ubuntu 11.10 Server Edition or Debian Server, if i were you.

mwheeler · Mon Nov 28, 2011 1:27 pm

Windows 2000 is over 10 years old now and support from Microsoft ended on July 13, 2010 (http://support.microsoft.com/ph/1131). I suggest you move to something like Windows Server 2008 or as you said Ubuntu. Don't forget Debian is also an option. Newer operating systems have a lot better hardware support for older computer. Squid was designed for Unix based systems, so it's a bet easier to get up an running on Ubuntu and Debian.

With Debian, you can install it without the GUI and just do everything from the command line.

neandero · Mon Nov 28, 2011 2:19 pm

Lets start with Squid config file

In my network there is a master router - UM.
Lan 1 = Out - Internet
Lan 2 = (switch) connected to multiple routers across town and each router runs Hotspot.
Lan 3 = Server ( http and Squid)

Lan 2 = each user will get this range if IP (10.20.0.0/16)
Lan 3 = Server IP (10.10.10.1)

what line of squid.conf need to be changed?

thanks,

neandero · Tue Dec 06, 2011 8:47 pm

Come on guys, just little help making my transparent proxy work yo.

i just need the lines i must change in the squid config file according to the above setup so it can start the cache?

neandero · Tue Dec 06, 2011 11:03 pm

Well, I got it working... BUT little confused!!

/ip firewall nat
add chain=dstnat action=dst-nat to-addresses=172.19.65.250 to-ports=3128 protocol=tcp dst-address=!172.16.0.0/12 in-interface=ether3 dst-port=80

ip 172.19.65.250 is the proxy pc running transparent on 3128

my question here is on
1- ether3 ?
2- !172.16.0.0/12 ?

can you please clear these points.

example of my network:
ether1 = connected to internet (public)
ether2 = provides addresses to local users
ether3 = the proxy server

and !172.16.0.0/12 can you please explain if these are users ip or what?

i'm little lost here..

fewi · Tue Dec 06, 2011 11:31 pm

How can we know what 172.16.0.0/12 is on your network? It's private IP space. Nothing in this thread mentions it before. The rule means, literally: take all traffic to tcp/80 that comes in via ether3 and isn't going to 172.16.0.0/12, and send it to 172.19.65.250 on port tcp/3128 instead.

What that means on your network is up to you to determine.

neandero · Tue Dec 06, 2011 11:49 pm

I know its up to me to determine, but what i want is the idea of handling in and out.

can you make this rule simple to me to understand:

/ip firewall nat
add chain=dstnat
action=dst-nat 
to-addresses=10.1.1.1 //this is the ip of the cache server
to-ports=3128 //port of the cache server
protocol=tcp
dst-address=!172.16.0.0/12 //my user ip's are in the range of 10.10.*.* so should this ip be (dst-address=!10.10.0.0/16) ?
in-interface=ether3 //is this ether the public or cache or the user's ether ?
dst-port=80

Thank you

neandero · Mon Dec 12, 2011 1:18 pm

I guess I made it working.. just need to get a good php script that can also cache youtube videos.

Now what firewall rule i can use to bypass user's simple queue if the transmitted file already existed in the cache and was downloaded from the cache server to the user?

thanks,

aacable · Mon Dec 12, 2011 6:29 pm

I guess I made it working.. just need to get a good php script that can also cache youtube videos. Now what firewall rule i can use to bypass user's simple queue if the transmitted file already existed in the cache and was downloaded from the cache server to the user?
thanks,

You can cache youtube like content via using 3rd party PAID Addon for SQUID name VIDEO CACHE .
I will not recommend you to Cache Youtube , it's not worth it and you can't actually cache even 1% of YOUTUBE, imagine how much space you will be needing , grrrrrrrr :p,
Also there are very slight chances that user will HIT the same video because of CDN.

To bypass rate limit for cached content, Using Squid with ZPH, and Mikotik Ver above 3+ , Use the following.

Add these Lines in SQUID.CONF

#==============
#ZPH 
#=======================
tcp_outgoing_tos 0x30 lanuser
zph_mode tos
zph_local 0x30
zph_parent 0
zph_option 136
#[lanuser is ACL for local network, modify at as per your network]

Make sure squid is marking cache content, you can do it via TCPDUMP command. i.e

# tcpdump -vni eth0 | grep ‘tos 0×30′
(eht0 = LAN connected interface)

At Mikrotik, create one mangle rule, which will marks cache contents coming from SQUID,
Second , QUEUE tree, which will enable cache content to deliver cache content to user at LAN speed.

/ip firewall mangle add action=mark-packet chain=prerouting disabled=no dscp=12 new-packet-mark=proxy-hit passthrough=no 

/queue tree add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=0 name=pmark packet-mark=proxy-hit parent=global-out priority=8 queue=default

Or Read more at:
http://aacable.wordpress.com/2011/07/21 ... t-traffic/

pospanko · Tue Dec 13, 2011 9:59 am

You can cache youtube like content via using 3rd party PAID Addon for SQUID name VIDEO CACHE .
I will not recommend you to Cache Youtube , it's not worth it and you can't actually cache even 1% of YOUTUBE, imagine how much space you will be needing , grrrrrrrr :p,
Also there are very slight chances that user will HIT the same video because of CDN.

I must disagree on this. I have 25-30% of all trafiic coming from my proxies with VideoCache on squid ( 2TB disk space which is very cheap this days ). Peoples usually watch simillar videos. I put something on facebook, and all my friend want to see what it is and video gets viewed 20-30 times in few hours.

neandero · Tue Dec 13, 2011 10:34 am

You can cache youtube like content via using 3rd party PAID Addon for SQUID name VIDEO CACHE .

has anyone tried out this addon? if its working, can we get a free copy same or similar?
i followed the script posted on squid wiki (partial solution), but it didn't work. although, the post did mention it may not work for everyone.

neandero · Wed Dec 14, 2011 1:39 pm

Dear aacable

I am back on Ubuntu Server Edition again and squid 3 is fully functional.

I followed your guide with TOS, TCPDUMP, MK Mangle, and Queue Packet Mark
for some reason the queue limit are completely bypasses and users are receiving the full speed as there is no queue at all?

thanks,

aacable · Thu Jan 12, 2012 9:03 am

You can cache youtube like content via using 3rd party PAID Addon for SQUID name VIDEO CACHE .
has anyone tried out this addon? if its working, can we get a free copy same or similar?
i followed the script posted on squid wiki (partial solution), but it didn't work. although, the post did mention it may not work for everyone.

Try this
http://aacable.wordpress.com/2012/01/11 ... tik-queue/

bax · Wed Jan 25, 2012 4:43 pm

For caching youtube try with my vmware host here: http://rapidshare.com/files/2933856136/ ... .1.2011.7z
Original post is here: http://forum.wisp.hr/viewtopic.php?f=3&t=559

neandero · Tue Feb 14, 2012 9:51 pm

You can cache youtube like content via using 3rd party PAID Addon for SQUID name VIDEO CACHE .
has anyone tried out this addon? if its working, can we get a free copy same or similar?
i followed the script posted on squid wiki (partial solution), but it didn't work. although, the post did mention it may not work for everyone.
Try this
http://aacable.wordpress.com/2012/01/11 ... tik-queue/

this SQUID/LUSCA mentioned at the above link, discusses a network with 2 Lans. My ubuntu server is built with only 1 Lan so I wonder if it can still be used to cache videos and dynamic contents?

Thanks brother...

aacable · Wed Feb 15, 2012 11:54 am

this SQUID/LUSCA mentioned at the above link, discusses a network with 2 Lans. My ubuntu server is built with only 1 Lan so I wonder if it can still be used to cache videos and dynamic contents?
Thanks brother...

Yes it can work with single LAN interface too

SQUID/Linux is highly customizable and it can fit in any environment easily.
Create Reverse route in squid pointing to Mikrotik, and create corresponding NAT rule.

neandero · Thu Feb 16, 2012 12:58 am

I get many TCP_MEM_HIT:NONE and TCP_REFRESH_HIT:DIRECT and TCP_NEGATIVE_HIT:NONE

but I get no TCP_HIT ?????????

also i get this error: clientNatLookup: NF getsockopt(SO_ORIGINAL_DST) failed: (92) Protocol not available

Running now with LUSCA.

thank you,

Cache Server - Plan and Design

Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Re: Cache Server - Plan and Design

Who is online