Community discussions

MUM Europe 2020
 
gammy69er
newbie
Topic Author
Posts: 46
Joined: Sun May 18, 2014 3:01 am

Hotspot, HTTPS and CPU performance

Mon Nov 03, 2014 1:38 am

Now Before you start asking me for Supouts... I understand that I can send these, and my issue will get resolved... However, I am wanting to get a bit more troubleshooting done, prior to Supout

The Issue : www in the /tools Profile is Eating as much CPU as it can

The Cause : Hotspot HTTPS Login system (/ip hotspot profile set login-by=https)

The Resolution : Disabling HTTPS as a login option

The New Issue : a lot of people have HTTPS homepages (cheers Google), breaking the redirect to the login page due to no HTTPS.

Workaround : tell the Client to go to a Non HTTPS Page - not ideal when dealing with up to 100 clients pre radio per day (when we have over 90 hotspot systems.

Question : is there some known causes to the https pulling such a High CPU - maybe a few different options I can try and report back with solutions - or is this a bug (apparently has been before and has been fixed and broken several times)

as a background - we have Full certified SSL Certs. Some Hotspots do not Suffer this issue - even larger ones, other take 50-90+ clients to go bad, others take 5+ clients to poop out.

also, I am not using Web Cache, however am using DNS cache. Some have Multiple Hotspots, others have one.

Disabling HTTPS drops www usage from Max (whatever it can grab - between 30%-70%) to NONE - like 0%

Having it disabled is a Workaround, however not an acceptable one - due to the redirect errors.

Also, before flaming me for not adding to another post - I did my absolute best effort to try and find solutions in the other posts, but everyone is cracking on about supouts, then there is never any more updates. I am just asking for a list of Possible issues... Sometimes trying these things out can lead to better knowledge of a system.
 
SystemErrorMessage
Member
Member
Posts: 378
Joined: Sat Dec 22, 2012 9:04 pm

Re: Hotspot, HTTPS and CPU performance

Mon Nov 03, 2014 10:41 am

HTTPS uses CPU to do encryption such that even a brute force attempt on SSH can be CPU consuming unless the CPU has hardware encryption support which routerOS uses. What routerboard are you using? How many clients do you serve at a time and how often is HTTPS used on your network in terms of bandwidth and packets per second.

might also be helpful to show your hotspot rules too.
 
gammy69er
newbie
Topic Author
Posts: 46
Joined: Sun May 18, 2014 3:01 am

Re: Hotspot, HTTPS and CPU performance

Thu Nov 06, 2014 3:33 am

HTTPS uses CPU to do encryption such that even a brute force attempt on SSH can be CPU consuming unless the CPU has hardware encryption support which routerOS uses. What routerboard are you using? How many clients do you serve at a time and how often is HTTPS used on your network in terms of bandwidth and packets per second.

might also be helpful to show your hotspot rules too.
TY For the Reply - Fair Call On the Info - Was just getting a general idea of the Workings and what could be causing the Issue.

The RB's Affected are RB411's, RB532's, RB751U and RB600.

RB600 is a Major Node - and has been for many years - Tweaks are made constantly - up to 150 users via multiple nodes (maybe 80 users direct - rest via other RB's)

RB751U's are Newer (last 12 Months) - Most are Direct to Web, only a couple of them have had the issue - Around 20+ users

RB532's - have a lot of these still in service - yet only 1 so far has been affected

RB411's. Most of these are gone now - but the last one I pulled was due to this issue.

So, we have Mipsbe and Mipsle CPUs of varied Speeds. Different Ram Levels, yet a fairly universal config for the Hotspot - apart from the RB600 - which is running 3 Hotspots on itself, and one of the 751's which has 2.

The RB532 was emergency replaced with a Groove A-52HPn (this covers 20 Clients) - We are continuing to Monitor that, and a 751 has been replaced with a 951Ui (this covers up to 100 Clients) - checking if power and ram being thrown at the issue will help to resolve.

A 751 came up with the error today, so have rebooted and am monitoring him... but the thing that gets me, especially with the 751's is that we have a Standard Config... I had an Old 951 (the 233mhz, 32mb ones) Running a hotspot with anywhere from 60 to 180 clients connected at one time - pumping through up to 65 Mbps across a UBNT backbone. This was configured after the 751 I am now monitoring and a few weeks before the 751 I just replaced with a 951U. No Config changes were made to the base script.

The only change that has been made is that we have installed new SSL Certs - But that has happened across the board and I have another site with a 532a that is still cracking along fine - up to 50 clients.

Either way - here is our Default HS Config (I take it this is what you meant from Rules) -

# nov/06/2014 13:33:57 by RouterOS 6.13
# software id = D4Y0-VID3
#
/ip hotspot profile
add dns-name=XXXXXXX.co.nz hotspot-address=10.5.XXX.XXX html-directory=hotspot4.4_t2 login-by=mac,cookie,http-chap,https,http-pap,trial name=hsprof1 \
radius-interim-update=2m trial-uptime=2m/1d use-radius=yes
add dns-name=XXXXXXX.co.nz hotspot-address=10.5.XXX.XXX html-directory=hotspot4.4_t2 name=hs-trial rate-limit=\
"256k/1M 512k/2M 384k/1500k 20/20 6"
/ip hotspot
add address-pool=hs-pool-1 disabled=no idle-timeout=8h interface=hotspot name=hs-XXXX profile=hsprof1
/ip hotspot user profile
set [ find default=yes ] idle-timeout=none keepalive-timeout=8h mac-cookie-timeout=3d shared-users=25

This is from a 751 - the ROS Versions that we have the issue with vary from 5.18 through 6.19

in saying that, the idle timeout and Keepalive were also recently updated fro 2 mins and 5 mins to 8 hours - we have done away with the time card system, and were getting complaints about being logged out from client guests.
... and how often is HTTPS used on your network in terms of bandwidth and packets per second.
not too sure where to find this info bud - I bet it's in the Supout huh... lol.

Either way - TY for the info. Because of the varied range of units affected, it appears it is one of 3 things:

1: Freak Coincidental Hardware Failure... Possible and not unheard of, but unlikely
2: Cert issue - Possible - yet due to the changes being made on around 100 radios of numerous capacity - once again, unlikely unless we really screwed something up on these ones - however they do as intended after reboot - until the blow out the CPU/Ram again
3: Hotspot Config error - somewhat likely - however this has been changed across the board, so theoretically would cause that same issue on most of my routers.

I guess the only real question I have for SystemErrorMessage is that is there a way that this hardware encryption could have been disabled - and if so, where would that be. An accidental switch off here does seem to be a good place to look.

Cheers again for the info bud - will keep digging.
 
SystemErrorMessage
Member
Member
Posts: 378
Joined: Sat Dec 22, 2012 9:04 pm

Re: Hotspot, HTTPS and CPU performance

Thu Nov 06, 2014 3:47 am

You can find the bandwidth used calculatedly. Create a test user and look at your packets and packets per second. Now login as the test user and take note of the amount of traffic and CPU used for the login.

MIPS based have limited performance when it comes to encryption, this is why PPC is preferred over MIPS however all the CPUs you listed have a very low frequency which explains your slow HTTPS, they are meant to be used as an AP only and not to perform hotspot by themselves for as many clients as you push to each of them. MIPS based devices tend to reboot after having high CPU loads or overheating. Even with low frequency and hardware acceleration (that PPC and TILE have) it will still be slow

Assuming all your APs are connected to some central router or such i strongly suggest you perform hotspot there if possible on a much faster system. This should solve your problem. If you are a WISP operating access points all over the place you may want to tunnel everything like a huge switch to some central place to do all the hotspot and routing before going online.

You may not need HTTPS hotspot to be secure if you are using encryption on wireless such as WPA auto but if your users are paying to use your hotspot from the login than you would need to use HTTPS.

Could also be possible some guest who didnt login is using some form of DoS.

Hardware acceleration is enabled by compilation flags when compiling the OS which i would assume mikrotik does. It works by the CPU having the logic and instruction so for example it would look like encrypt [a] [key] [destination]. WIthout hardware acceleration it would have to use the equivalent of other instructions to achieve it.
 
gammy69er
newbie
Topic Author
Posts: 46
Joined: Sun May 18, 2014 3:01 am

Re: Hotspot, HTTPS and CPU performance

Thu Nov 06, 2014 10:20 pm

Well, Thank you for your points - I will look into them further and see if there is something specific to be drawn from this.

However, as I had said previously - I have 532's that have been going for years with dozens/hundreds of connections per day - no issues. It may be due to a lot more websites running in https - with more people setting their homepages to such sites that could be bring this limitation to light - but I would have expected a more blanket fail - not a here and there.

HTTPS Speed has never been an issue - it only becomes an issue when these units begin hanging on whatever it is they are hanging on, and then it becomes a blanket issue for all connected units. The HTTPS is only on so the Hotspot Login can redirect HTTPS homepage'd Clients, without them having a Certificate error come up, to our RADUIS Authentication Server. Turning it off does not appear to affect the usability once logged in (all web, including https, is accessible) - so this is the workaround we have for particularly troublesome sites.

Also, just as an update - my 951Ui (600MHz - 128mb) has just started to do it too - the one I replaced 2 days ago - profiler shows maxing out of 'www' process and a quick removal of HTTPS as a login option on the Hotspots has reduced that back down to little - DNS is holding highish (between 15% and 35%) - may be related.

Assumedly, 'www' in the profile tool would be any and all internet traffic - yet it only shows up normally for short bursts - 0.5% and then disappears - even when there is sustained traffic. http://wiki.mikrotik.com/wiki/Manual:Tools/Profiler lists the fact that it is a shown function, however does not mention what exactly it is related to... this would be nice to know

After a few minutes of watching, the DNS CPU usage has dropped back down to lower levels again, and now the router is sitting happily @ 10-25% - this is with the https re-activated. Could be burst load, then an inability to rectify... IDK...

As for the Hardware acceleration - cool to know how it does it, so I guess that the only way for that to bug is a bad upgrade - I assume a netinstall would be the best way to ensure that the software was installed correctly

Looks like i'll need a bit of coffee over the next few days of monitoring :) - next time the 951Ui goes - i'll pull a supout and have a look at it (yay just found the supout viewer). Again - Cheers for all the Knowledge. Will keep everyone appraised of any outcomes.
 
SystemErrorMessage
Member
Member
Posts: 378
Joined: Sat Dec 22, 2012 9:04 pm

Re: Hotspot, HTTPS and CPU performance

Fri Nov 07, 2014 9:37 am

google uses https by default, so you can expect every user to be putting load on that. It is also very likely that they may have other tabs opened that could have many using https which could cause hang.

I really dont know your network architecture whether the APs are connected to a central gateway or a gateway for every AP. From the looks of it you are having each AP to act as a gateway. Perhaps if you showed your network architecture diagram it would help a lot.

You dont need to run multiple hotspots, you can run 1 hotspot for your entire network with different configs for static hosts and non-static.
 
gammy69er
newbie
Topic Author
Posts: 46
Joined: Sun May 18, 2014 3:01 am

Re: Hotspot, HTTPS and CPU performance

Tue Nov 11, 2014 8:14 am

google uses https by default, so you can expect every user to be putting load on that. It is also very likely that they may have other tabs opened that could have many using https which could cause hang.

I really dont know your network architecture whether the APs are connected to a central gateway or a gateway for every AP. From the looks of it you are having each AP to act as a gateway. Perhaps if you showed your network architecture diagram it would help a lot.

You dont need to run multiple hotspots, you can run 1 hotspot for your entire network with different configs for static hosts and non-static.
Again - Ty for your input.

I know Google now uses https by default. and as i say, this is possibly why this is really beginning to rear it's head - because i know my Homepage wasn't https until around 18 months ago. have been experienceing this problem for a while apparent, but with things moving so fast - no one had a chance to look at it, just put the work around in place... when we renewed the certs - we enabled it all again - and the fun began (2 weeks later mind you)

We tend to like to spread the load - General Architecture is "Primary Site - Sub Site - AP - Client" - Primary Site can also have Station - AP - Client run by Primary Site

And as I said previously - Just fishing for possible ideas on the underlying issue.

1st Primary Site has 3 Hotspots (5 Stations - 2 of which have customised Splash pages - therefore multiple HS is required in the current config) - and another 8 Sub Sites, each with their own RB/Hotspot (so no load on main unit for logins - just traffic) - This is on an RB600 - and is one that we are planning on Upgrading soon(ish) - had never hit above 50% CPU unless an issue was occurring

2nd Site is on a 951Ui - 2 Hotspots - 1 Link to another node which has 5 Sub Sites.

3rd Site which has had this issue (others have as well, however similar to this) is Stand alone - 751U.

Now, Sites 2 and 3 are new configs, on Fresh HW - site 1 is Long Standing (4+ Years). All, have however been upgraded through. When i started , site 1 was on 5.24. this was just after 6.7 was released. We did an upgrade to 6.7 because there was another issue beginning to rear it's head - and performance was being affected - the issue of the time dissappeared.

now, in saying all this - the issue at site 2 appears to have gone away...

Changes Made - I selected the HS Page from the drop down list on the server profile, as opposed to the scripted Setup.

Why i did this - was setting up an MaP and when i copied the HS files to the unit, in the root of the files folder (as is with all other RB's i have come across so far) - i rebooted and they were gone - MaP has a "flash/" directory under the root - so it seems they need to go there. so i copied again, selected the file required and ... Wha Wha, no HS page at all... spent a bit of time fluffing round - finally deleted the "flash/" from the selected dropdown and Boom, Hotspot up and running - however, NO Warning for redirect on my PC - now i usually ignore this, due to the fact my lappy is due for a clean... and that my other testing device, Windows Phone Auto pops Login, unless there is a cert error.

This made me think of site 2, as it is the 2nd most loaded of the problem sites, so i went there - changed just that (HS Page selection) and... wha wha... still getting the redirect error - however the site has been stable now for 5 days and holding, and considering it was only 1.5 days last time, i think that is looking good. Thinking that this particular error may relate to how the later versions of RoS are scripting, vs the last setup script we created - which was well over 15 months ago - not including tweaks. maybe missing a vital add in, maybe it's just changed up a little, idk, the only major difference i can see is that all of our script has "Quotes" around Variables - whereas an Export from said site does not... this may be the difference (when running the script getting no errors on the load - I ALWAYS check through when done)

At the end of the day - even though everything is in it's place - it looks to be a config error - possible incompatibility between scripts used and current FW - Continuing Investigation.

I will be looking to try this out on the next problem site i have. Unfortunately this is not a Go or No Go problem... so we have to wait for the results. :) Will keep you posted
 
gammy69er
newbie
Topic Author
Posts: 46
Joined: Sun May 18, 2014 3:01 am

Re: Hotspot, HTTPS and CPU performance

Thu Dec 11, 2014 11:22 pm

To any who have been Following this - here is an update... Finally - But I wanted to be Sure.

SO.... Did some Live Testing on Site 2. Upgraded to 6.21.1 Let him Run. 1 day Later... Wha whaa... he was doing it again Disabled Https and waited till I could get on site.

By the time I got there - (2 days later) 6.22 was out.

Upgraded again (was considering the downgrade, but decided for testing I'd keep it New). Now, there was no apparent HS upgrades in this FW - so I also did the Other thing I was going to do - I Blew away the Scripted Version of the Hotspot and manually configured it via Winbox.

3 weeks now, No issues. Also had upgraded the link between the 5 Nodes that come off it during that time - and everyone is now "NOT" noticing the internet (noticing denotes issues :D)

So... Recommendation... If you have a Script from a while back that has been solid for years... Time to hand Build a radio and pull a fresh script. As I may have mentioned - I noticed that our old scripts had a lot of ""s - and the current exports from 6+ appear to not have these, and I know that in theory, because these are string values, this should not matter in the slightest... However, in my case, it appears to have. I have tested this at another site which had an old RB450, which was having another issue - but was on old FW. The issue they had was unrelated, so going to a 450g solved that, but there is anywhere between 40 and 100 people online there at one time (so load is high enough for this issue to occur fast), and (I believe) because I hand built the router (no Script) - it hasn't skipped a beat so far - 2 weeks in.

Will update you guys again if I do get the same error again - happy to prove my S*** wrong in the name of progress. And again, Thank you so much to SystemErrorMessage - for using a forum as it is meant to be used - you gave me lots of insights and help - so thank you.
 
User avatar
awacenter
Member Candidate
Member Candidate
Posts: 200
Joined: Thu Dec 09, 2004 12:58 pm
Location: Castellón
Contact:

Re: Hotspot, HTTPS and CPU performance

Tue Feb 24, 2015 10:07 am

You post was very helpfull, gammy69er.
I notice this behaviour in several times in mu devices.
May it be that we do not put the correct (or powerfull enough) device for our needs?
ImageImage
 
gammy69er
newbie
Topic Author
Posts: 46
Joined: Sun May 18, 2014 3:01 am

Re: Hotspot, HTTPS and CPU performance

Sun Mar 15, 2015 10:47 pm

You post was very helpfull, gammy69er.
I notice this behaviour in several times in mu devices.
May it be that we do not put the correct (or powerfull enough) device for our needs?
Cheers Awa - good to know peeps are getting informed from this.

Have actually Posted a reply in another post due to some discoveries I have recently made

Post Is:
http://forum.mikrotik.com/viewtopic.php?f=2&t=89820

Have actually discovered that it appears to be an issue in RoS v6 'vs' Routeros HW that has 32mb ram or less (to my regular PC hardware troubleshooting Skills anyway - I know it's different - but not totally).

I am seeing units that have a Lower RAM amount tend to struggle with the HS Clients - each one coming through slower and slower - until it finally just reboots from watchdog.

This was tested - and currently still proving in 2 x rb751U's - 1 has 64mb - stable for months - one has 32mb - Different location - Similar foot traffic - same config (well apart from name strings - for 2 library's - free Wifi) - 32mb falls over up to 5 times a day - have had a couple of good 4-5 day runs though - but not often.

Units with higher RAM seem to handle this a lot better. It's Just a Fact.

So recommendations from my non official background - Upgrade your HW if you are required to upgrade to RoS 6 (RB951U is cheaper than RB751U was when I originally posted - and has 128mb, and 600MHz - has served me well) - if security is handled elsewhere - then your router probably isn't doing much anywho - so RoS 5 is good... however for everything, I am a technician - therefore recommend the upgrade to the latest features/security.
 
gammy69er
newbie
Topic Author
Posts: 46
Joined: Sun May 18, 2014 3:01 am

Re: Hotspot, HTTPS and CPU performance

Thu Apr 09, 2015 9:08 am

Update...

Finally got into the Library and Upgrade the H/W - Upgraded to 951G (i usually get 951U - but out of stock)

Either way - 7 days and counting - absolutely zero reported issues so far (touch wood :P) I did however notice a drop in ppp 2 days ago - but it came straight back up.

As of Late Night open Thursday @ 6pm - CPU Floating up to ~20% on login just witnessed, and with 90.1mb RAM Free (ooo, 89.8 now) - that is certainly more than 32mb :D

Either way, when I remember to check again - Will update here as to Uptimes etc.

Anywho - keep well all :)
 
gammy69er
newbie
Topic Author
Posts: 46
Joined: Sun May 18, 2014 3:01 am

Re: Hotspot, HTTPS and CPU performance

Fri May 22, 2015 3:26 am

Potentially Final Update for this one - if anyone gives a Toss Now :)

25 Days uptime (see a log for a while back that it got rebooted - unsure as to why - could be power outage)

Zero Support Calls since upgrade (was getting 3+ a week)

Just Logged in to check him before this update. As per what i have noticed with the Hotspot's, it was cracking 100% CPU on login - but smashing through that very quickly and logging guests in (100% CPU is WWW - Presumably RB telling PC to go to Splash Page) After which it falls quickly to below 20% overall usage (~10 Users - so not excessive load)

Either way - a Win in my Books. Anyone has any comments, Queries - Feel free to leave them here - i will check in from time to time :)
 
User avatar
awacenter
Member Candidate
Member Candidate
Posts: 200
Joined: Thu Dec 09, 2004 12:58 pm
Location: Castellón
Contact:

Re: Hotspot, HTTPS and CPU performance

Tue Aug 04, 2015 3:45 pm

Potentially Final Update for this one - if anyone gives a Toss Now :)

25 Days uptime (see a log for a while back that it got rebooted - unsure as to why - could be power outage)

Zero Support Calls since upgrade (was getting 3+ a week)

Just Logged in to check him before this update. As per what i have noticed with the Hotspot's, it was cracking 100% CPU on login - but smashing through that very quickly and logging guests in (100% CPU is WWW - Presumably RB telling PC to go to Splash Page) After which it falls quickly to below 20% overall usage (~10 Users - so not excessive load)

Either way - a Win in my Books. Anyone has any comments, Queries - Feel free to leave them here - i will check in from time to time :)
I confirm your supositions. With low cost HW (32 MB RAM) and ros v6 the behaviour is really bad. And with a RB951G the CPU raises until 100% some secons (4-5 secs, I think too much time) and later CPU falls to 10-15%.

Anyway, It is necessary to verify the hotspot service implementation when https login method is used in hotspot profile.
I think it is a bug.
ImageImage
 
User avatar
awacenter
Member Candidate
Member Candidate
Posts: 200
Joined: Thu Dec 09, 2004 12:58 pm
Location: Castellón
Contact:

Re: Hotspot, HTTPS and CPU performance

Thu Aug 27, 2015 1:27 pm

Any news about why CPU usage arise to 100% when hotspot authenticacion is https?
ImageImage
 
gammy69er
newbie
Topic Author
Posts: 46
Joined: Sun May 18, 2014 3:01 am

Re: Hotspot, HTTPS and CPU performance

Fri Aug 28, 2015 1:47 am

Any news about why CPU usage arise to 100% when hotspot authenticacion is https?
I Gots Nothing.... It happens with Every Version of the H/W I have used for H/S (Haven't used Cloud Core Yet for HS - but that is kinda like taking a Monster Truck to a Go Cart race)

I will have to Defer to any Code Monkeys (:P) out there that understand what the HTTPS calls are trying to do to the router to make it crank up to 100% cpu. All I know is the Better the H/W, the less time the CPU is at 100%, therefore alleviating, if not eliminating the problem (almost non existent on an 850Gx2)

My Findings...

RB751U or equivalent - 400mhz - 32mb RAM - Mipsbe - 5 - 20 seconds CPU Perf (unless it falls over)
RB751U or equivalent - 400mhz - 64mb RAM - Mipsbe - 5 - 20 seconds CPU Perf (unlikely to fall over)
RB532A - 399mhz (upclocked) - 64mb RAM - Mipsle - 5-20 second CPU Perf (unlikely to fall over)
RB951U or Equivalent - 600mhz - 128mb RAM - Mipsbe - 1-5 seconds CPU Perf - no fall overs noted due to Hotspot
RB450G - 680mhz - 256MB RAM - Mipsbe - 1-5 seconds CPU perf - no Fall overs at all due to HS
RB600 - 400Mhz - 128MB RAM - PPC - 5-20 seconds Perf - not often to fall over - only under multiple logins
RB850Gx2 - 533mhz Dual - 512MB RAM - PPC - Almost Unoticable Perf - Hasn't fallen over due to Hotspot so far.

So i guess what i am saying here, that in regards to H/W - from what I have seen, Mipsbe Chips seem to have the most issues - but the Mipsle is right there beside it, most of my lower spec "LE's" were thrown away not long after I finally understood how things worked. in Saying all that - the 600Mhz+ ones are fairly boss, handling things rather well.

In saying that PPC RB600 didn't fair much better. I have had to remove HTTPS to actually get this unit to continue to function after the 6.XX upgrades. In saying that it did have 2 HS and 10+ Stations running (some via HS, most just NAT'd Private IP with their own HS)

As for the RB850Gx2 - it is a Monster.... 200+ logins per day (mostly between 5-8pm) - Never skips a Beat.... For a Big site (but not massive), I would totally recommend these. When Summer comes back to the Southern hemisphere, I will be Making my RB600 onto a RB850Gx2 with Ubnt BH. for Massive sites (1000+) - you would probably need Multiple H/S, or CCR (in saying that an RB1100 would be very good up to a point too).

Who is online

Users browsing this forum: Bing [Bot], harryurgreat and 120 guests