Community discussions

MikroTik App
 
welan
newbie
Topic Author
Posts: 37
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

RB1000, web proxy and high CPU usage

Mon Nov 23, 2009 11:09 pm

Hi,
I have problems with RB1000 and webproxy, I see that the I/O in the compact flash
produces 100% CPU. Anyone knows how to minimize this?
I have 150 users max, and the CPU goes 100% in about 30 minutes with 20 users.
Many thanks.
 
User avatar
gmsmstr
Trainer
Trainer
Posts: 942
Joined: Fri Jun 04, 2004 2:22 am
Location: St. Louis, MO
Contact:

Re: RB1000, web proxy and high CPU usage

Wed Nov 25, 2009 8:33 pm

What occurs with slow CF cards, and small processors . IN some cases WebProxy will work fine on these, it just depends on the size of the CF etc. With the CF, as soon as you boot, it does a read on all of the files to rebuild the cache etc. SOOO. Thats why it takes a long time on reboot.
Dennis Burgess, MCTCE, MTCNA, MCTCTE, MTCWE, MTCNIE, A+, N+, MCP, MTCSE Mikrotik Certified Consultant / Trainer
Need Mikrotik Support: http://www.linktechs.net -- Link Technologies, Inc.
-- Author of "Learn RouterOS: Second Edition"
 
reinerotto
Member
Member
Posts: 441
Joined: Thu Dec 04, 2008 2:35 am

Re: RB1000, web proxy and high CPU usage

Fri Nov 27, 2009 3:07 pm

Hi,
I have problems with RB1000 and webproxy, I see that the I/O in the compact flash
produces 100% CPU. Anyone knows how to minimize this?
I have 150 users max, and the CPU goes 100% in about 30 minutes with 20 users.
Many thanks.
You might think about a LINUX-box with squid and HDD as an upstream (parent) cache for your RB, and use RAM-based caching on your RB only. Howver, still a pending bug with this one.
Refere to
http://forum.mikrotik.com/viewtopic.php?f=2&t=36002
 
welan
newbie
Topic Author
Posts: 37
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: RB1000, web proxy and high CPU usage

Sat Nov 28, 2009 12:43 pm

Many thanks for the reply.
Actually, I have a LINUX-box with squid and HDD as an upstream (parent) cache for my RB, and use RAM-based caching on my RB only (sorry for copy and paste, but your english is better).
However when there are online a lot of clients (14-15Mbit of traffic), the CPU of RB1000 is over 80-90%. Is it normal? I think the it could be a bottleneck.
Many thanks.

P.S.: I don't have problem with the website of the topic
 
reinerotto
Member
Member
Posts: 441
Joined: Thu Dec 04, 2008 2:35 am

Re: RB1000, web proxy and high CPU usage

Sun Nov 29, 2009 3:18 pm

Sorry, on my hotspot I do not have such a high load.

Regarding my reported problem: It is interesting, that you have no problems. I re-tested, and found same problem under various conditions always to appear, in case either hotspot is activated, or web-proxy. So it might be something regarding the "universal proxy" scheme, MT uses.

To be shure, that you really NOT have this problem, too, can you please do following test:
From hotspot-client,
(having web-proxy/upstream/transparent proxy enabled on RB), go to
http://messages.finance.yahoo.com/Stock ... of=1&frt=2

Click "Reply"

type a message, then click "Post Message"

On my system, error from squid shows up.


In case, you still do not encounter my problem, please post your version of squid, you are using. And version of RoS. That might be an issue,.

Thanx in advance.
 
welan
newbie
Topic Author
Posts: 37
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: RB1000, web proxy and high CPU usage

Sun Nov 29, 2009 9:34 pm

Hello, I tried to post and my squid is working without any problem.

http://messages.finance.yahoo.com/Stock ... rt=2&off=1

However, I don't use proxy in hotspot, I use squid with my residential clients, and the CPU goes up when there are 15-16 Mbit of traffic on my network. I never had any type of problem with squid, however sometimes (with certain of website, like mail website) I exclude them to be proxed (I have an address list on mikrotik).
I think that the high CPU load is concerned to the number of the on-line clients (so the number of connections) and not to the traffic load.
My version of ros is 3.30 on the rb1000, and my parent proxy is Squid 2.7.STABLE6.

Some example:
4.8Mbps of traffic | CPU 37%
9.2Mbps of traffic | CPU 63%
17.2Mbps of traffic | CPU 90-100%

I have mangle rules, however if i disable the rules, the situation is the same.
Instead if I disable webproxy and activate the mangle rules the CPU usage is about 3-5%.
Thanks for the reply.
 
reinerotto
Member
Member
Posts: 441
Joined: Thu Dec 04, 2008 2:35 am

Re: RB1000, web proxy and high CPU usage

Sun Nov 29, 2009 11:38 pm

@welan: (I have an address list on mikrotik).
Can you give a link, please ?
I will test your prblem sites, too.
In worst case, I have an exception list to start with :-)
Will also try to use an older squid for my problem; RoS version should not be an issue, I guess.

Thanx for your work, anyway.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: RB1000, web proxy and high CPU usage

Mon Nov 30, 2009 9:35 am

note that hotspot uses web-proxy, and uses settings set up in the proxy settings, except, it is enabled if you are running hotspot. You should configure proxy to reflect that no caching is required, or set up parent proxy feature.
 
reinerotto
Member
Member
Posts: 441
Joined: Thu Dec 04, 2008 2:35 am

Re: RB1000, web proxy and high CPU usage

Mon Nov 30, 2009 10:52 am

note that hotspot uses web-proxy, and uses settings set up in the proxy settings, except, it is enabled if you are running hotspot.
Yes, I realized that.
>You should configure proxy to reflect that no caching is required, or set up parent proxy feature<
No caching ???? Why ?
Actually I am using both: hotspot, web-proxy with RAM-cache only, parent proxy running squid. And usually it works. But again, one special problem site I have. So there is a bug, which I reported already some time ago. Still pending.
 
welan
newbie
Topic Author
Posts: 37
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: RB1000, web proxy and high CPU usage

Mon Nov 30, 2009 6:12 pm

Ok, I think that it's not clear my situation.
I'm running the RB1000 as a router in my network:

Clients 150+ ---> |RB1000 - ROS3.30| ---> INTERNET
*********************| /
*********************| /
*********************|SQUID2.6STABLE7| ------ /

The * is for align the picture, sorry.

In the RB1000 is running a web proxy server, with this settings:

enabled: yes
src-address: 0.0.0.0
port: 3128
parent-proxy: 172.16.28.100
parent-proxy-port: 800
cache-administrator: "pippo@pippo.com"
max-cache-size: none
cache-on-disk: no
max-client-connections: 4000
max-server-connections: 4000
max-fresh-time: 3d
serialize-connections: no
always-from-cache: no
cache-hit-dscp: 4
cache-drive: CF1

My parent proxy is running squid. The clients are surfing very well, but I think that the CPU usage on RB1000 is too high when traffic is heavy.
I attached a screenshot.
Many thanks.
You do not have the required permissions to view the files attached to this post.
 
User avatar
gmsmstr
Trainer
Trainer
Posts: 942
Joined: Fri Jun 04, 2004 2:22 am
Location: St. Louis, MO
Contact:

Re: RB1000, web proxy and high CPU usage

Mon Nov 30, 2009 6:31 pm

You are still proxying traffic. Hence tracking it, etc. So, hence, you have the high cpu count. Think of this. What happens is the MT gets the traffic, and then sends it out, as a client to your parent proxy, and then returns that information to the client. So, yes, this will eat up CPU as it should.

What you SHOULD be doing, is the transparent proxy rule, but direct it stright to your parent proxy server. Then there is NO proxy that should be even running on your 1000. So, simple configuration issue!
Dennis Burgess, MCTCE, MTCNA, MCTCTE, MTCWE, MTCNIE, A+, N+, MCP, MTCSE Mikrotik Certified Consultant / Trainer
Need Mikrotik Support: http://www.linktechs.net -- Link Technologies, Inc.
-- Author of "Learn RouterOS: Second Edition"
 
welan
newbie
Topic Author
Posts: 37
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: RB1000, web proxy and high CPU usage

Mon Nov 30, 2009 6:35 pm

Ok, I tried to do this, with a simple dst-nat like this:

chain=dstnat action=dst-nat to-addresses=172.16.28.49 to-ports=80
protocol=tcp src-address=172.16.28.0/24 in-interface=ether1 dst-port=800

but I cannot surf. What is wrong?
Thanks

PS: I bought your book, very interesting
 
User avatar
gmsmstr
Trainer
Trainer
Posts: 942
Joined: Fri Jun 04, 2004 2:22 am
Location: St. Louis, MO
Contact:

Re: RB1000, web proxy and high CPU usage

Mon Nov 30, 2009 6:47 pm

I would have to look at it, but I am sure its a NAT rule. You would have to also accept traffic from your web proxy.
Dennis Burgess, MCTCE, MTCNA, MCTCTE, MTCWE, MTCNIE, A+, N+, MCP, MTCSE Mikrotik Certified Consultant / Trainer
Need Mikrotik Support: http://www.linktechs.net -- Link Technologies, Inc.
-- Author of "Learn RouterOS: Second Edition"
 
welan
newbie
Topic Author
Posts: 37
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: RB1000, web proxy and high CPU usage

Mon Nov 30, 2009 8:33 pm

Tried, but not surfing.
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: RB1000, web proxy and high CPU usage

Mon Nov 30, 2009 8:40 pm

chain=dstnat action=dst-nat to-addresses=172.16.28.49 to-ports=80 protocol=tcp src-address=172.16.28.0/24 in-interface=ether1 dst-port=800 
You're dst-nat'ing back into your own network. That's problematic since the client is sending to the router, and is expecting an answer back from the router. But the router is forwarding the request to a host on the same subnet, which sends back directly to the client, which then discards the traffic as "what the hell is this".

Try a parent proxy outside of the client network and a similar rule should work.

Edit: alternatively you could also src-nat the forwarded request to the router IP. At that point traffic would flow as expected (client -> router, router -> proxy, proxy -> router, router -> client) - but the proxy would only see the router IP address instead of the individual clients. All in all it's cleaner to move the proxy to a different network.
 
welan
newbie
Topic Author
Posts: 37
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: RB1000, web proxy and high CPU usage

Tue Dec 01, 2009 9:21 pm

Ok, thanks, but if I move my proxy over another subnet, this is the response of mikrotik when I try to surf:

ERROR: Gateway Timeout

--------------------------------------------------------------------------------

While trying to retrieve the URL http://www.google.it/:

Connection refused
Your cache administrator is info@pippo.com.

--------------------------------------------------------------------------------

Generated Tue, 01 Dec 2009 19:15:03 GMT by 172.16.28.251 (Mikrotik HttpProxy)

I have no idea.
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: RB1000, web proxy and high CPU usage

Tue Dec 01, 2009 9:24 pm

If you're proxying directly to the external proxy now moved to a different network, why is the Mikrotik proxy still active and intercepting the connection?
 
welan
newbie
Topic Author
Posts: 37
Joined: Thu Jul 10, 2008 12:06 am
Location: Italy
Contact:

Re: RB1000, web proxy and high CPU usage

Tue Dec 01, 2009 9:29 pm

The message above is squid as parent proxy over another subnet, but I tried your solution too:

Assume that:
172.16.28.99 is my PC
172.16.27.251 is squid

chain=dstnat action=dst-nat to-addresses=172.16.27.251 to-ports=800
protocol=tcp src-address=172.16.28.99 in-interface=ether1 dst-port=80

172.16.27.251 is on ether4, this is the filter rule of firewall:

chain=input action=accept in-interface=ether4

What is wrong?
 
tombee79
Member Candidate
Member Candidate
Posts: 247
Joined: Sun May 09, 2010 2:28 am

Re: RB1000, web proxy and high CPU usage

Thu Jan 08, 2015 9:59 am

HI

I HAVE EXACTLLY the same problem with rb750g when downloading around 15mbs the cpu uses 100%. i use the proxy to redirect to parent squid but still is the same.

i tried to redirect in NAT Firewall but it doesent work, proxy err page in browser


Did u get this resolve?
 
User avatar
hossain2004a
Member Candidate
Member Candidate
Posts: 247
Joined: Mon Dec 22, 2014 7:34 pm
Location: Iran

Re: RB1000, web proxy and high CPU usage

Thu Jan 08, 2015 1:25 pm

I'm using 450G in our internet cafe, I've enable Hotspot and Webproxy.
first time I've enabled cache on disk, and always my CPU was %90-%100. Then I disable that and delete caches file from "Files"
now it's CPU usage is better.
Seems MT webproxy cant handle too much traffic.

Who is online

Users browsing this forum: Lifz, Voisanats, ysha and 121 guests