Received a call from a customer that they suddenly couldn’t access our SSL secured payment pages, which are in the Hotspot Walled Garden list. They are using the RB1100 and the Walled Garden entry is simply like “*.my-domain-name.com”.
While this seemed to have worked fine for a few months, they are now frequently experiencing the problem that they are unable to access the SSL payment pages. I then deleted the wildcard domain and added the domain as FQDN with each hostname and protocol (80/443), but still the same problem. We then rebooted the RB1100, but still no luck. I had to add the IP address into the Walled Garden IP list and then it worked. But I prefer using domain names as it is getting complicated when using PayPal with their multiple, regional different IP addresses.
I then run a test on my RB450G and RB750 and got the same problems … but not all the times.
It seems to be related to the SSL cert. When using Godaddy’s Standard SSL cert and Comodo, I got the problem, while using other certs I did not. Google’s Chrome reported “Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL Protocol error.” Firefox reports “The connection was interrupted”.
Well, it seems these same errors occur for every https page not included in the Walled Garden as well, instead of being redirected to the login page.
Once the page cannot be reached, the page remains unreachable despite many page reload attempts. When I then go to a different SSL site, secured with e.g. a Godaddy cert, the page loads and then the first ‘unloadable’ page can also be loaded successfully and the problem disappears. I have to restart the browser in order to reproduce the page load problem.
This is a big problem as there is no indication that customers cannot use the payment pages. The amount of payments simply drop.
I have tried this with 3 different RBx and ROS 4.11.
I did not experience any problems with the more expensive SSL certs, but with the entry level certs.
For PayPal payment pages I would have to know all the different IP addresses in different regions of the world and enter them into the ‘Walled Garden IP List’. Time as in timeout also seems to be a factor as the problem reoccurs after some time. I am trying to get a reproducible test scenario, but it is simply intermittent and difficult to catch.
Here is another example with a Godaddy Standard SSL cert:
I add *.onlinewellnessassociation.com to the Walled Garden
I visit www.onlinewellnessassociation.com and click on the Login button, which redirects me to the https page and I get the same error … I am unable to reach the login page via the Hotspot Walled Garden entry.
Well, my post has been online for a few weeks and it seems nobody else is experiencing the SSL problems I get.
I just tested it again after I deleted all Walled Garden entries and just added *.onlinewellnessassociation.com.
When I click on the ‘Login’ button on the http://www.onlinewellnessassociation.com I immediately get an error message when going to the https site.
Firefox reports “The connection was interrupted” and Safari cannot establish a secure connection.
I selected the http://www.onlinewellnessassociation.com site because they also use the low cost Godaddy ssl cert.
It seems I do not get the problems with other certs … except Comodo.
Problem seen with ROS 4.11 and RB450G, RB750 and RB1100.
Anybody willing to go for a test? Thanks!
Insanity: doing the same thing over and over again and expecting different results. - Albert Einstein
I have a challenge with the SSL portion of the walled garden. It does fine for a while, then it stops letting the connections through. I have a request in to mikrotik support about how the walled garden determines whether to bypass an ip with a SSL domain name in the walled garden. I will post the results here for you.
As I understand the SSL protocol, the browser resolves the domain to an ip, and gets a certificate. After that, the URL is encrypted on a SSL connection packet, so it can’t be retrieved by the router. Presently, I use ips in “/ip hotspot walled-garden ip” to avoid the connection drop. ADD: Using the ips works well! No problems even after 2 years.
I only use the IP for SSL sites as a workaround. The Hotspot servlet sometimes seems to have issues to correlate previous DNS requests with HTTPS requests, and as you said it cannot possibly see the requested domain in the HTTP header as that is encrypted.
Thanks! Looking forward seeing the reply from Mikrotik.
Well, from my understanding, the domain name is never in the IP header, as the IP header only knows about IP addresses. The translation from domain name to IP address is done by the good old DNS before sending any packets.
SSL/TLS encrypts on the Transport Layer and the IP header (Network Layer) should not be affected.
Yes, I know that adding the IP addresses in the ‘Walled Garden IP List’ works fine, but what you gonna do when you are using e.g. PayPal and their paypalobjects.com domain resolves into all kind of different IP addresses, which seem to be different in different regions of the world.
Did some further tests and it seems that it is not just the Godadddy cert. Had the same problem with the Verisign cert, but not with digicert.
Well, it seems I have to upgrade my PayPal account to the Pro version and spend extra 30 bucks a month to avoid this problem.
I have the same challenge with authorize.net, so paying more is not always the best thing to do. You might want to wait a day or so to see what the MT team says.
ADD: As I suspected…with a little experimentation…using nslookup on www.paypal.com returns 4 servers. Every time you repeat the nslookup, the order of the servers change. Your odds are only one in four that the top server is the one you are using.
And wouldn’t it be nice to do a “/ip hotspot walled-garden ip print all” and see the ip addresses that are being used (bypassed) for each domain name in the walled garden?
Well, with authorize.net, just like with PayPal Website Payments Pro, the buyer does not have to leave your own payment pages. Therefore your own site with SSL will most likely have always the same IP address and can be added to the Walled Garden IP List.
Yes, nslookup will turn up with the same IP addresses on consecutive lookups for paypal.com. But first, PayPal uses several domain names for their payment page and secondly, they have different IP addresses depending on your location (e.g. country). And it seems every so often these addresses change, making it difficult using PayPal Website Payments Standard for ‘walled Garden’ type sites (hotspots).
Yes, A “/ip hotspot walled-garden ip print all” would be nice.
I see where you are going. You could let your server handle the connection with PayPal, and all you need is the “yes or no” from your server.
I use authorize.net to handle my payment pages (SIM).
But this does not correct the challenge with the walled garden. I am still interested on how the hotspot knows to allow an ip through (bypass) using a domain name in the walled-garden. Is it a reverse nslookup on the ip? Another nslookup on the domain names in the walled-garden? The dns cache? Some or all of those?
This is not new. If you search previous posts, you will see my first report of this problem in July 2008.
ADD: If it uses dns cache, bad choice. Like I said, PayPal uses the dns to rotate the server ips. The same ips show up, but in a different order! The dns cache only stores one ip of the four available.
This is purely conjecture on my part. I would love to hear a developer weigh in.
I think the walled garden decides normal HTTP transactions by just listening to the HTTP request. All Hotspot HTTP connections, authenticated and unauthenticated, by default get proxied to the Hotspot servlet. Because it’s a destination NAT redirect the client thinks it’s really talking to the server, and would issue a normal HTTP transactions. All clients nowadays enumerate the host in the HTTP request, and the path of course is part of it, so the Hotspot servlet can simply look at the request and determine the host and path from there, and then either (for unauthenticated users) fetch the real site as a proxy and send it back to the client, or throw a redirect to the login page.
Of course that doesn’t work for HTTPS, so I think it would do DNS snooping (all DNS also gets redirected to the Hotspot servlet by default) and remember that a host just did a lookup for a name that resolved to a certain IP, and now is trying to connect to TCP/443 on the IP address that was listed in the DNS reply. At that point you can no longer match by path, I can’t see how that would ever be possible for HTTPS as the path is part of the encrypted request. You also have a problem when the servlet only snoops one of multiple IP addresses and the client picks a different one, or if the client still has the DNS name cached from before attaching to the Hotspot, or when the SSL site is accessed by IP address (rare because certs can’t match IPs), or a number of other different scenarios.
I always bypass the DNS redirect anyway, so I always list SSL servers that need to be reachable in the walled garden by IP address rather than host name. All SSL transactions for me are brokered by a server I own (credit card transactions etc.) so that one server is very easy to list by IP, just like poinths described. This isn’t too hard to do and enables a few other neat things, like being able to use that server for any number of Hotspots that are all configured basically the same with any changes to the Hotspot infrastructure being completely centralized.
I’m hoping a dev weighs in, but they might consider the inner workings of the Hotspot something they don’t want to post about.
I just received an email from Sergejs requesting more info about the problem. I explained it as well as I could. It is “quitting time” there, so I expect it will be Monday before I hear anything else. Good with me.
BTW, Sergejs verified that “/ip dns” is how the hotspot decides. I will assume (OMG! ) that means an ip must be in “/ip dns cache all”. No?
ADD: I just checked “/ip dns cache” with PayPal. http://www.paypal.com has 4 ips each with a TTL of 4 minutes. That is all you have until the ips will not go through, unless someone else does another dns request for paypal after that.
ADD2: After another check, the www.paypal.com TTL is less than 4 minutes. I have gotten TTLs as short as 16 seconds!
If you would like to help me troubleshoot this, you can run this test locally. Maybe it is only certain areas that will have this trouble. The www.paypal.com dns TTL should be very short, so you gotta look quick.
/ip dns cache
print
:put [:resolve www.paypal.com];
print
After the first print, note if www.paypal.com has any entries. Then do the resolve. Another print and look for www.paypal.com entries and note the TTL. Mine are 4 minutes or less. When the TTL reaches 0, the entry disappears, and no more walled garden for paypal.
If you have trouble with other domains, try those with this also. They may have the same type dynamic dns servers.
ADD: Any daring PayPal users want to try a script? It resolves www.paypal.com, removes old www.paypal.com entries from “/ip hotspot walled-garden ip”, then adds new ips. I have tested this on the main pages. I did not make a payment, but there seems to be nothing stopping it now.
:resolve www.paypal.com;
:global paypalips [/ip dns cache find name=www.paypal.com];
:global oldips [/ip hotspot walled-garden ip find dst-host=www.paypal.com];
:local thisip none;
:foreach x in=$oldips do={
/ip hotspot walled-garden ip remove $x;
}
:foreach i in=$paypalips do={
:set thisip [/ip dns cache get $i address];
/ip hotspot walled-garden ip add dst-host=www.paypal.com dst-address=$thisip;
}
I wrote a script a few weeks ago that basically searches the DNS cache and adds an address list including all matched items. It could probably be extended to the hotspot walled garden entries as well. I’ll do some tests and see what I can come up with.
Edit: On second thought, you might be able to use the straight script as it is with address lists. Then create a pre-hotspot rule in NAT that says any traffic going to the IPs in the address list action=accept. This way, they’re not even handled by the hotspot at all.
This might actually save a lot on cpu load as well, as the hotspot isn’t doing so much work.
Now that is interesting. I’ll have to play with that. If this is the “hack” for dns cache short TTLs, then that would probably be more efficient.
ADD: But with the way the script is now, I get what I wanted above. I wanted to be able to do a “/ip hotspot walled-garden ip print” and see what is being bypassed.
On third thought… I was looking into the walled-garden, and it looks like it’s NOT part of the hotspot process itself, but rather just a set of firewall rules (filter and nat).
Does this seem correct?
@SurferTim,
Yuup, your script looks like it’ll work good.
And I got more or less always the same 4-8 PayPal addresses. I tested this in 4 different countries.
The TTL was always quite short.
Anyhow, I do not really see that the TTL life time is related to my actual problem.
The http://www.paypalobjects.com might be of more interest in this regard as the TTL is about 30 seconds and I get one IP address, but different ones with a few retries after the IP is cleared from the cache.
E.g.
:put [:resolve http://www.paypalobjects.com];
4 e120.g.akamaiedge.net 184.84.64.146 0s
4 e120.g.akamaiedge.net 69.192.160.146 19s
7 e120.g.akamaiedge.net 184.86.160.146 17s
3 e120.g.akamaiedge.net 184.85.16.146 12s
As soon as I try to visit a site, a DNS request is send and then cached in DNS cache.
Despite that the https domain I try to visit is correctly listed in DNS cache, I am unable to reach the site.
As soon as I disable the hotspot server, I can reach the https site without any problems.
Here again my simply test setup:
Configure a hotspot setup (I use a remote RADIUS server, but I think it doesn’t matter)
Add e.g. *.onlinewellnessassociation.com to the Walled Garden List
Visit http://www.onlinewellnessassociation.com and click on the Login button (redirecting to https:…)
As soon as I try to reach the https page, I get an error message and as soon as I disable the hotspot server, I get through.
DNS cache:
4 onlinewell… 67.228.27.22 3h50m33s
After I add 67.228.27.22 to the Walled Garden IP List I am able to reach the https pages.
When I disable the Walled Garden IP List entry, I get again an error.
ADD: I am thinking about the paypalobjects challenge. Like I said, I only did basic webpage nav there. Here is a script that picks up all www.paypalobjects.com ips. Schedule it to run every 20 seconds.
:global ppobjip [:resolve www.paypalobjects.com];
:local paypalobject [/ip hotspot walled-garden ip find dst-host=www.paypalobjects.com];
:local thisip none;
:local noip true;
:foreach i in=$paypalobject do={
:set thisip [/ip hotspot walled-garden ip get $i dst-address];
:if ( $thisip = $ppobjip ) do={
:set noip false;
}
}
:if ($noip) do={
:log info "paypalobj script adding $ppobjip";
/ip hotspot walled-garden ip add dst-host=www.paypalobjects.com dst-address=$ppobjip;
}
Within a minute or so, you should see your current www.paypalobjects.com ips in the “/ip hotspot walled-garden ip”. It also inserts a log entry every time it adds an ip. Actually, it is both ips for me. There are only two it is issuing here.
OK, after several minutes, I picked up two more…and two more
I am using winbox and I cannot use https://onlinewellnessassociation.com in the Walled Garden List as Dst-Host (it’s host, not URL). While it allows this entry, it simply does not work. I added *.onlinewellnessassociation.com (or http://www.onlinewellnessassociation.com) and this works for the http pages of the site.
I can add the Dst-Port 80 and 443, but that does not change anything.
If I have another site not using the Godaddy standard SSL cert, it all works fine most of the time.
If I e.g. add *.digicert.com into the Walled Garden list, I can visit http://www.digicert.com and https://www.digicert.com without any problems. I only use http://www.onlinewellnessassociation.com as an example because they use the Godaddy Std cert. and I am using the Godaddy Std SSL cert for my site too.
I still think that it is SSL certificate related. But again, once the SSL site has a single IP address and I add this IP to the ‘Walled Garden IP’ list all works fine. But as mentioned, PayPal is using a whole bunch if changing IP addresses for a number of domains just for the payment page (e.g. http://www.paypal.com, http://www.paypalobjects.com, paypal.112.2o7.net, securepics.ebaystatic.com … and occasionally other domains for e.g. surveys like secure.opinionlab.com, …mediaplex.com)
Is there anybody who can confirm my results? … Simply add Dst.-Host *.onlinewellnessassociation.com to the hotspot ‘Walled Garden’ and then go to http://www.onlinewellnessassociation.com (should be successful) and then click on the Login button on that page … which should redirect you to the https: page, which results in an error on all RouterOS boxes I have tested so far.
Well, I use the wildcard character * in front of the domain name, e.g. *.onlinewellnessassociation.com, which should cater for www and all other hostnames.
The domain onlinewellnessassociation.com is not my domain. I only use this domain as an example since they use the godaddy standard SSL cert, the same cert I use for some of my sites. http://www.onlinewellnessassociation.com works fine with my Walled Garden when using *.onlinewellnessassociation.com
Anyhow, I think you are somehow on the right track. The problem might be CNAME related …
I abandoned the use of the wildcard character and added the following in the Walled Garden:
And it seems to work (most of the times) at least from my site here…
I presume I have to add some more hosts/domains for PayPal, but the basic host/domain combination seems to work now.
Will dig a bit more …
That last part is good to know. Both entries together did it.
I have only heard questions from support. When I get a statement from them, I’ll post it.
) that means an ip must be in “/ip dns cache all”. No?
I’ll have to play with that. If this is the “hack” for dns cache short TTLs, then that would probably be more efficient.