Your findings are fully consistent with what I am seeing; however, I can report that more than websites are affected. We are seeing the issues with other non browser web applications, such as connecting to iCloud and AppStore (where it says it cannot connect), email using Exchange and iMap (where we get server “errors” at the botton of the screen and “updates” are as much as an hour old), streaming with Netflix, CNN, BBC and even SPEEDTEST. With regards to SPEEDTEST, we have seen it timeout looking for servers (local and remote) on numerous occasions. You should try SPEEDTEST at random a few more times. The problem is transient, and things seems to work on/off/on/off, etc… It’s strange that connectivity and other tests (such as Ping) seem to work on our local LAN without issue, although I have not spent a lot of time testing this nuance.
It would be really nice if this got fixed!
I was asked by support person to check if it is an AC or 80MHz wide channel deciding factor. I did test it and in my case it seems the problem exists also on AC 40MHz wide channel.
The only common factor for me is ipv6. With it off problem goes away. Currently running ac at 80mhz and all fine. As soon as I unable ipv6 it dies. If you disable ipv6 also make sure your clients are not being allocated and don’t have an ipv6 address too.
I tried the same thing… no change with 40 MHZ wide channel… I still have all AC disabled.
I origionally thought it might have something to do with IPv6; however, I am not sure I can easily disable IPv6. We have an assigned static prefix and a number of our devices (such as disk arrays, etc.) are passing a lot of native traffic over IPv6. The general PC/Macs are also.
Our neighbor has a full commercial UBNT deployment. This problem is not happening over there, which I have personally verified. The suggestion came up last week that in the absence of any comment from Mikrotik (and they have stopped responding to inquiries related to my support ticket), we should consider changing platform away from Mikrotik. Who is at fault is irrelevant; we just need our networks to work. I have been tasked to pull some quotes from vendors to do this, but I really would rather not. We have a holiday shutdown planned where we could do the conversion, but that would mean no holiday for my staff… 
.
****** Mikrotik, PLEASE COMMENT…
By ‘off’ you mean complete disabling of ‘ipv6’ package or just disabling IPv6 DHCP Server / ND so that devices don’t get ipv6 routable addresses?
I’m using 6to4 so I disable the sit interface, disable the ip address on the wan and lan including the advertise. I then force the client to loose its ipv6 address and that’s enough. Ipv6 package is still active.
Interesting workaround; however, not a good solution for us. Disabling A/N/AC and going with just A/N @ 40Mhz works, although we don’t get the performance we would like…
Did anyone tried to debug the problem a bit deeper? Like sniffing phone’s packets to see what goes wrong.
I have tried, but I found nothing conclusive. I have also supplied logs to Apple.
Somebody please set my expectations. I have opened a ticket with Mikrotik (Ticket#2018101122002554), but I have not seen any reasonable responses. I have collaborated with folks in this topic thread and even brought up the subject in the "v6.44beta [testing] is released! (http://forum.mikrotik.com/t/v6-44beta-testing-is-released/123205/1).
It seems like this issue is not being taken very seriously by Mikrotik… should I assume that this problem is not going to get fixed?
Frustrated…
By saying the local traffic works - it works with multiple hops inside your network (no NAT/Masquerade)? So as soon as the traffic goes over the NAT/Masquerade you start seeing the problem?
It is possible the problem related to the TCP traffic as the L2TP VPN uses UDP protocol.
We are now trying to reproduce the problem with the minimal configuration to make the problem reproduce all the time not just random time.
What is the best website or app to see the problem more frequently to appear?
Could you help us by telling the below info which works or doesn’t work for you:
- Does it happen with all the center channel frequencies?
- AC with one chain enabled and 20mhz width
- AC with one chain enabled and 20/40 width
- AC with one chain enabled and 20/4080 width
- AC with two chain enabled and 20mhz width
- AC with two chain enabled and 20/40 width
- AC with two chain enabled and 20/4080 width
Thanks for responding!
I just observed that pinging local devices seems to work, but nothing analytical has been tested to prove that local traffic is working correctly. I agree that it might be a TCP problem. Things are broken for most websites and applications (e.g. imap/pop mail, google.com, yahoo.com, iCloud login, Apple App store, etc). Sometimes these site/apps timeout, other times they partially load, and occasionally they go through; however, SpeedTest does work but it can take a long time to find a server, if not just time out like the others. Once it does find a server and starts, it functions properly. The actual SpeedTest packets may be going over UDP. So, the problem is not specific to any one website or application.
I am seeing this on all of the configs you have listed above:
1: yes… broken at all center channels, including XXXX
2-4: (chain 0 only): Broken at all widths (20 20/40 and 20/40/80)
5-8: (chain 0, chain 1): Broken at all widths (20 20/40 and 20/40/80)
Extra: (chain 0, chain 1, chain 2): Broken at all widths (20 20/40 and 20/40/80)
Extra2: Everything works fine if you turn off AC and just use 5GHz A/N (at all channel and chain configurations)
I am sure that if you put one of these Apple Xs devices on your net, you should be able to quickly see the problem I have done all of the above tests on a hAP 962UiGS-5HacT2HnT running 6.43.4, but my other MT models show the same issue. The test device is just an acting as an AP bridge, with no routing or NAT. I set-up this unit specifically for this test. We have a core router that is not broadcasting any WiFI, and the configuration on that unit has remained unchanged throughout this process. Any NAT/Masquerade is upstream from the APs on the core router. So, with this test unit in AC mode we see the problem; however, the other APs with AC disabled are working fine.
netispguy, thank you for sharing us the information. For now we are unable to reproduce the issue, the websites opens ok, the, speedtest is ok as well.
Do you have ipv6 also running or just IPv4?
Which DNS server you are using - some local one from the Router our some other public DNS?
If you try to access some local lan - routed network services, do they work ok? Or problem happens when you go over the NAT?
Have you noticed some specific websites that fails to load?
Uldis, couple facts that I have observed. Just FYI I’m using “cAP AC” running 6.43.2.
- The problem persists even after yesterdays update to iOS 12.1
- I am convinced the problem is in DNS communication (!) - somehow suppressed by Mikrotik cAP AC 802.11ac link - why I think so? - you will read below.
- I am using manually configured DNS on my iPhone XS - set to 8.8.8.8
- The DNS and whole “networking” works fine for short time after initial connection to “Mikrotik cAP AC 5GHz 802.11ac” wireless network interface.
- On the iPhone XS initially there is only IPv4 from DHCP Server which is main router - ASUS RT-AC86U
- After some time on my iPhone XS I can see two IPv6 addresses assigned with one IPv6 gateway - which I have no clue where they come from, as my “cAP AC” doesn’t have IPv6 module/package enabled nor my ASUS RT-AC86U has IPv6 solicitation configured, nor my ISP is using IPv6 - and it seems after that IPv6 the web browsing stops working, but may be not related.
- I have discovered that when the issue occurs, I can still browse through websites that I have recently opened - like the IP address of the domain is in iPhone’s XS cache and I can open new pages of the same website but not new websites or HTTP based applications - IMHO this points to DNS connectivity issues.
- IPv6 addresses also appears after connecting to 2.4GHz interface of the same “cAP AC” and I have zero problems with connectivity or DNS
- Zero issues observed on the iPad Pro when connected to the 802.11ac interface of “cAP AC” - so this is exclusively issue with iPhone XS
I believe somehow the DNS queries or replies from the DNS servers are dropped by Mikrotik when connected to wireless interface configured to 802.11ac.
If my Speedtest app has domains cached and I can use it on my iPhone XS connected to 802.11ac Mikrotik’s interface, I can get high performance test results - like 300Mbps down and 95Mbps up, which is my ISPs service that I have.
For other members of this forum. I don’t have time right know to set this up, but maybe try to mirror traffic on Mikrotik to external IP address with Wireshark and capture traffic that is generated when the issue occurs.
best regards,
Greg
I am shocked you do not see this problem. I have a couple of questions for you:
- Are you testing on an iPhone Xs or XsMax running at least 12.0.1?
- Are you using a “US” version? My exact version number for my test AP is: RouterBOARD 962UiGS-5HacT2HnT-US. All of my current AP devices are “US” versions.
With regards to your questions:
- For the latest test, the 962UiGS-5HacT2HnT has IPv6 was totally disabled from the package list. Even with it disabled, I still get the same issues.
- I have tried numerous DNS servers, all public. 8.8.8.8 (Google) and 75.75.75.[75-76] (Comcast) are the most common ones I use. I do NOT think this is a DNS problem since with I turn AC off, everything works normally, even with the same DNS configuration.
- All of my other devices that are NOT iPhone Xs variants work normally using all AC configurations, either locally or remote.
- YES!!! I just tested to 4 servers on my local LAN. It appears that I can get to all of my local servers without any issue. It seems that when I NAT/masquerade out to the Internet the problem happens. The error is generally one of a) for websites: “… could not open page because the server stopped responding”, or b) for applications: “…failed to connect”.
- When going to the Internet, it is not site specific. I am seeing the issue with most website and applications.
- I submitted a Supout.rif to Rudolfs via ticket Ticket#2018101122002554 last night.
- My core router is: 493G (with a 5Ghz radio card added in) running 6.43.4. We have not used the radios in this unit for some time…it is just being used as the edge router. We have both IPv4 and IPv6 static IP addresses.
-100% of everything works on the iPhone Xs[Max] when I turn AC off!
*** Thanks for your continued support!
Hello,
I have two iPhone XS units and I’m seeing the same issues result with random DNS failure especially while refreshing from the phone.
Things has gotten better with 12.1 but I had the issues since initial IOS version (v12) and Mikrotik was at v6.43.2 (now upgraded to 6.43.4)
I have 3xHap AC units and IPv6 package is uninstalled I can still see on the phone that the status of IPv6 addresses is not consistent
And it’s not IPv6 addresses that are local (end in fe80)
-Ofer
I have noticed a few differences than you are seeing:
- When I go to a site that has already been loaded, I do get cached version; however, if I force a “reload” of the site, the problem occurs. I also have the same issue going to other pages on the same website regardless of whether or not I have previously loaded the site.
- I am not at all convinced it is a DNS problem. Why would DNS be an issue using 5GHz A/N/AC and NOT be an issue using 5GHz A/N? Also, I have tried going to websites using their IP address (where DNS is mostly not involved,) and I get the same issue.
- With regards to IPv6 keep the following in mind (look for an IPv6 addressing primer online for more in-depth explanations)
- The IPv6 address protocol is designed to allow “self assignment.” Unlike IPv4 where self assignment is really an error when no DHCP server supplies a “lease”, IPv6 allows devices to get information (from other IPv6 devices on your network) to set it own unique address. IPv6 also heavily uses “local link” addresses with fe80:: as the prefix. These are “private” IP addresses much like IPv4 uses network 10 or 192.168.X.X. If a device fails to see a valid IPv6 router to the Internet (and thus a public prefix), it will use a local-link address. Most devices (IOS devices included) will shape two addresses. In accordance with IPv6 protocols, first address will be generated based on the unique MAC address of you iPhone. The second address is a “masking” address that sort of scrambles the first address so that your true MAC address is not exposed out on the Internet. The method by which this “masking” is done is documented with the IPv6 addressing protocol.
- So, you will see three addresses on your iPhone: 1) What your iPhone believes to be the IPv6 router (default route), 2) An IPv6 address based on the unique MAC address of your WiFi radio, and 3) a “masked” IP address to use as more secure address when going to a public network.
- Apple IOS does NOT allow you to manually set IPv6 parameters, which pisses me off on a daily basis. (Note: Mac OS X does allow manual IPv6 address configurations).
-Scott
I can also confirm that IOS 12.1 does not fix the problem.
Netispguy,
thanks for the lecture about IPv6 but I already know that. I’m a system engineer for a large network equipment vendor and I work close with developers - I’ve seen things people didn’t even dream off, so I know what “magic” things can happen under the hood when processing packets (usually these are bugs) - that’s why I still believe this is processing of DNS requests/replies related issue.
You’ve mentioned even cached sites doesn’t work for you. Bare in mind that almost every regular site is using different domains when loading web page as sources for ads, images, scripts etc so even previously opened one can be stuck when you loose proper DNS connectivity. Same thing if you’re using IP address of web server instead of FQDN. You need very simple web page to test it.
Try using my website http://109.74.108.40/ , you should see a text “:)” even other sites won’t open on your iPhone XS.
Best regards,
Greg