Community discussions

MikroTik App
 
User avatar
Ibersystems
Forum Guru
Forum Guru
Topic Author
Posts: 1685
Joined: Wed Apr 12, 2006 12:29 am
Location: Cabrils, Barcelona - Spain
Contact:

SLOW Webproxy cache with CompactFlash

Fri Aug 14, 2009 3:49 pm

Hi everybody,

I have problems with a RB1000 and transparent webproxy cache in Compact Flash Extreme3. The LAN goes working slow when cache is filling up (4GB of 8GB total) and the CPU of the RB1000 goes to 100% too.

Uldis said to me that on every write on the CFlash, the CPU increases and it's normal that the CPU stays always at 100% when using webproxy cache. He said that the system could work fine with the CPU at this very high % of use too.

My problem is that when I activate the webproxy and the CPU goeas to 100%, all the web browsing goes too slow. I use DNS cache too, but it doesn't speed up my network.

Anyone of you has installed the RB1000 with webproxy and has a speed web browsing without speed problems? What can I do to make it faster? I need to use more than the 1Gb of RAM installed to make cache.

I usually have "only" 60 users at the same time, browsing internet.

I use ROS 3.27.


Thanks,
Martín Ruiz
martinruiz at ibersystems.es
Experto en redes WiFi y enlaces WiFi.

Facebook: @Ibersystems
Twitter: @Ibersystems

Certified in:
MTCNA - MTCTCE - MTCWE - MTCINE - MTCUME - MTCRE - MTCSE
 
noam_chom
newbie
Posts: 37
Joined: Wed May 03, 2006 5:55 am

Re: SLOW Webproxy cache with CompactFlash

Sat Aug 15, 2009 10:45 am

I have the same problem , but with ROS v3.23 !
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1370
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: SLOW Webproxy cache with CompactFlash

Sun Aug 16, 2009 8:44 am

Some websites do not work with the MikroTik Proxy in the first place. I think it is a cookies problem, someone reported google mail not working OK and I know some of our Bulgarian clients cant login into their dating sites for example http://aha.bg - through the MT Proxy.

I've gone through similar trouble with microSD, check it and see if you have checked every possible cause: http://forum.mikrotik.com/viewtopic.php?f=2&t=31232

Basicly you should check these : http://wiki.mikrotik.com/wiki/Image:Pro ... torage.gif and if the cache is OK, probably you can try to rebuld the cache :)

Please keep in mind that Web Proxy needs time to rebuild the cache sometimes after it being started... you can see what the Proxy is up to in its settings window in WinBox on the bottom...

When dealing with Proxy storage - first Stop the Proxy - then make changes to the storage ...

P.S. another thing that annoys me is that the MT Proxy does not use RAM to further speed up its process.

So soon I am starting to deploy fast ClarkConnect boxes with 1gig ethernet cards that have Squid running.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
noam_chom
newbie
Posts: 37
Joined: Wed May 03, 2006 5:55 am

Re: SLOW Webproxy cache with CompactFlash

Mon Aug 17, 2009 5:12 pm

I have tried to upgrade to ROS 3.28 and the problem still exist , processor is on 100% almost all the time.
It works great about 15 minutes from turning on , and after that it goes to 100% CPU utilization.
I didn't have this kind of problems with 2.9.xx version (squid version) on x86, so I thought that it shouldn't have a problem running on RB1000 with CF on ROS 3.xx.

As NetworkPro suggested that cookie's could make a problem I explicitly make not to cache cookies and still the problem persist.
I think that this is serious problem, I have not read comment from anyone who don't have this kind or similar problem with new implementation of MT Proxy.
If the problem is with CF card I suppose that It would not work at all (anyway Dude is on the same CF and it works flawless).

regards,
Ilija
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1370
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: SLOW Webproxy cache with CompactFlash

Mon Aug 17, 2009 5:47 pm

...I explicitly make not to cache cookies...

Show me the proxy rule that does this!


Try turning off the option Cache on Disk - so the proxy will be tested in RAM. Just don't power cycle it or anything - to keep the contents of the cache.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
noam_chom
newbie
Posts: 37
Joined: Wed May 03, 2006 5:55 am

Re: SLOW Webproxy cache with CompactFlash

Mon Aug 17, 2009 8:21 pm

add action=deny comment=cookie disabled=no dst-host=*@*txt*

This should stop cookies from storing into web proxy.

I've tried turning off save on disk and the CPU utilization dropped to 70-80% but it is still unusable. Router response time is desperate. :(
Any other recommend ?
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1370
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: SLOW Webproxy cache with CompactFlash

Mon Aug 17, 2009 8:36 pm

The MT Proxy usually is usable on (nice new) x86, it can have more CPU when it needs.

How many users are behind this proxy? Itsa powerful router, so 60 users should not be a problem, unsless some of them are infected, malicous etc.

Are the stats of the proxy very high? Probably - too many connections mate !
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
noam_chom
newbie
Posts: 37
Joined: Wed May 03, 2006 5:55 am

Re: SLOW Webproxy cache with CompactFlash

Mon Aug 17, 2009 10:12 pm

NetworkPro, I suppose that RB1000 should be more powerful than a standard PC (it's PowerPC), that's why I am using it.
I have about 80-90 online users regularly per RB1000, but the tests were done with about 20-30 online users.
Nevertheless I suppose it should fine with up to 200-300 users, about 1500-2000 connections.

BTW, I am now trying to simulate nearly same situation on my RB433 ,with system as a caching disk and results are disaster. It goes up to 40% on 2 clients.

Something is relay wrong with proxy.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1370
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: SLOW Webproxy cache with CompactFlash

Mon Aug 17, 2009 10:48 pm

...RB1000 should be more powerful than a standard PC (it's PowerPC)...
When I am alking about a x86 machine more powerful than the RB1000 - I mean a x86 machine more powerful than the RB1000.

Haven't you heard of multiprocessor systems, Industrial boards, etc etc?! x86 FTW. ;)
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
noam_chom
newbie
Posts: 37
Joined: Wed May 03, 2006 5:55 am

Re: SLOW Webproxy cache with CompactFlash

Tue Aug 18, 2009 2:07 am

So the conclusion is we don't need RB1000 :-).
I don't think so, I think that there must be some solution for this problem and it is software based.

Ilija
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24605
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: SLOW Webproxy cache with CompactFlash

Tue Aug 18, 2009 12:58 pm

For very active proxy caches, where many files are stored and retrieved at once, the RB1000 is not so optimized, as it doesn't have native IDE controller on it (speed is limited). However, the CF slots work perfectly fine on regular load cache, for user manager, for logs, for Dude etc.
No answer to your question? How to write posts
 
noam_chom
newbie
Posts: 37
Joined: Wed May 03, 2006 5:55 am

Re: SLOW Webproxy cache with CompactFlash

Tue Aug 18, 2009 3:16 pm

So Normis, what would be very active proxy , 1000-2000 connections ?
As I recall previous implementation (2.9.xx) had combination of ram proxy and disk proxy , so this kind of implementation was not very (disk speed) demanding and could satisfy our needs. Anyway I think that high-speed CF could get up to 18MB/s roughly about 140Mbit/s of communication, so this could not be a bottleneck of the system (if there is no some glitch in the file system).
Now, the question is will Mikrotik make some improvement in proxy, or you think that this is what you wanted to achieve with this part, maybe there is some trick to make it work with some configuration magic ?

Regards,
Ilija
 
jasejames
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Fri Jun 26, 2009 11:04 am

Re: SLOW Webproxy cache with CompactFlash

Tue Aug 18, 2009 5:00 pm

Further to what noam_chom has said, might I make a suggestion?

It seems to me that the RB1000, and even the RB450G, are more than capable of keeping up with a caching proxy in CPU performance terms.

If the issue is that the CPU is being eaten alive by PIO calls, then how difficult would it be to issue a version of either system which has a SATA port built-in?

There are already SATA Mini-PCI cards available, but these don't work because they require a 5V source. Again, how hard would it be to provide this 5V line to the card?

Then you could produce an oversized case that could take a 3.5" SATA drive (3.5" because they are more robust than the 2.5s). The 1U version of the RB1000 could be easily modified for this purpose.

This seems to be another one of the occasions when an otherwise excellent Mikrotik solution is spoiled by little silly things.

* If you are going to include a cache, you should enable a more robust way of using it on your boards.
* If you are going to include a CF shot, it should be IDE enabled -- especially when you advertise TrueIDE on your site.
* If you are going to include an OpenVPN implementation, give us enough options such that we can do things like push routes down.
* A connected route that stays up when the link is down is bad practice, pure and simple.

All frustrating bugs that really shouldn't be there! And I've only been using the product for a short time. I asked about the CF/SD performance issue a few weeks back, and Mikrotik assured that there wasn't a problem...
 
User avatar
Ibersystems
Forum Guru
Forum Guru
Topic Author
Posts: 1685
Joined: Wed Apr 12, 2006 12:29 am
Location: Cabrils, Barcelona - Spain
Contact:

Re: SLOW Webproxy cache with CompactFlash

Wed Aug 26, 2009 2:44 pm

Hi,

Normis, I have problems at about 20-30 minutes of restart. And the number of users are about 50-60, but problem exists with 30 users too.

It seams like we can't use RB1000 as a webproxy for more than 20-25 users. This makes me a big problem with a costumer installed at 500 KMs..

unf.
Martín Ruiz
martinruiz at ibersystems.es
Experto en redes WiFi y enlaces WiFi.

Facebook: @Ibersystems
Twitter: @Ibersystems

Certified in:
MTCNA - MTCTCE - MTCWE - MTCINE - MTCUME - MTCRE - MTCSE
 
noam_chom
newbie
Posts: 37
Joined: Wed May 03, 2006 5:55 am

Re: SLOW Webproxy cache with CompactFlash

Sun Sep 13, 2009 3:10 pm

I have done extensive testing of the WebProxy package and so far I found that the it works "as it should" (not too bad) just on x86. So my guess is that powerpc WebProxy package is not optimized for routerboard, so i tried to make a parent proxy on RB1000 and proxy on HP Proliant ML115. But when the number of users that use Proxy become over 20 users RB1000 started to jump to 90-100% cpu utilization, when redirecting http requests to x86 Proxy.

Conclusion:
Something is really wrong with proxy package.


Please someone from Mikrotik confirm that this is noted, and either you are working on the solution (so we need to wait for solution) or should we work to find alternative non-mikrotik webproxy solution.

Regards,
Ilija
 
eneimi
Member
Member
Posts: 388
Joined: Sun Sep 09, 2007 12:55 pm

Re: SLOW Webproxy cache with CompactFlash

Sun Sep 13, 2009 6:51 pm

Just to lend a voice!
I am also having performance issues with mikrotik webproxy on compact flash+RB1000. I have had to dump it..
For very active proxy caches, where many files are stored and retrieved at once, the RB1000 is not so optimized, as it doesn't have native IDE controller on it (speed is limited). However, the CF slots work perfectly fine on regular load cache, for user manager, for logs, for Dude etc.
So for more than 'regular load', if we must use routeros it has to be x86?
* If you are going to include a cache, you should enable a more robust way of using it on your boards.
I agree. The RB1000 is an otherwise excellent product - price, power, performance, features. If you can up it's specs to handle heavy webproxy demands, you win..we all win!
 
User avatar
roadrunner
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Thu May 18, 2006 11:22 pm
Location: US

Re: SLOW Webproxy cache with CompactFlash

Mon Sep 14, 2009 12:22 pm

I have done extensive testing of the WebProxy package and so far I found that the it works "as it should" (not too bad) just on x86. So my guess is that powerpc WebProxy package is not optimized for routerboard, so i tried to make a parent proxy on RB1000 and proxy on HP Proliant ML115. But when the number of users that use Proxy become over 20 users RB1000 started to jump to 90-100% cpu utilization, when redirecting http requests to x86 Proxy.
Were you redirecting http requests to the x86 proxy using the redirect rule or parent proxy setting?
Was Cache On Disk disabled?

Are you also using the SanDisk Extreme III like Martin at Ibersystems?
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: SLOW Webproxy cache with CompactFlash

Mon Sep 14, 2009 2:19 pm

I have done extensive testing of the WebProxy package and so far I found that the it works "as it should" (not too bad) just on x86. So my guess is that powerpc WebProxy package is not optimized for routerboard, so i tried to make a parent proxy on RB1000 and proxy on HP Proliant ML115. But when the number of users that use Proxy become over 20 users RB1000 started to jump to 90-100% cpu utilization, when redirecting http requests to x86 Proxy.
as normis wrote before - no IDE = a lot of PIO to handle by CPU, all x86 boxes i ahve seen comes with some sorts of IDE controller, hence, decent operation, since all that PIO from disk writes/reads are handled by adapter not the CPU.


as a solution in larger network you can add 1 x86 box for proxy, and point routerboards to it as parent proxy.

if you have some free resources (RAM) on your RB1000 give some RAM for proxying and set up parent proxy. That way all the requests will be kept inside your network and proxy will work better as more requests and more proxed pages inside your network.
 
noam_chom
newbie
Posts: 37
Joined: Wed May 03, 2006 5:55 am

Re: SLOW Webproxy cache with CompactFlash

Thu Sep 17, 2009 12:13 pm

Janis, nice to hear from you :).
as a solution in larger network you can add 1 x86 box for proxy, and point routerboards to it as parent proxy.
I am pretty concerned that when I used RB1000 just for parent proxy-ing (port 80 - to x86 box) made RB1000 processor too utilized (90-100%).
I have turned off cache on disk (roadrunner), and that didn't helped.
So any solution including RouterBoard and Proxy right now is not possible for a larger network (except using x86 as a Router and other x86 as a Proxy).
As far as I know parent proxy-ing should not utilize RB1000 processor so much.
Were you redirecting http requests to the x86 proxy using the redirect rule or parent proxy setting?
Was Cache On Disk disabled?

Are you also using the SanDisk Extreme III like Martin at Ibersystems?
roadrunner, yes I've tried every CF 16gb that I could find on our market, and I can say MT has a great compatibility list concerning CF cards.
I can't say that any of the CF I've tested worked better than other, they worked equally good.

Janis as you are saying there are too much requests (YES) , but i suppouse speed 18mb/s ~ 144 Mbit/s access to CF should be enough so the cache is usable.I think that if you have a cache filesystem table in the RB memory (with file physical address and file crc32) and access to CF read only on hit would make processor less utilized. So nevertheless multithreaded solution could be also helpful if the problem is access to CF because RB doesn't need any CF access to manage the rest of the tasks it should accomplish.


It is not for me to judge but don't you think that this is a problem worth of solving ?


Regards,
Ilija
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1370
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: SLOW Webproxy cache with CompactFlash

Mon Oct 11, 2010 11:52 pm

Well well well what do you know MirkoTik proxy works perfectly with Facebook wheres Squid 2.6-3.1 fails miserably. Too bad MikroTik proxy can't be anonymous. I mean stealth. Now it's anonymous.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service

Who is online

Users browsing this forum: ChristopherFS, marypoppins, olivier2831 and 72 guests