hap ac^2 - Group Key Exchange timeout / No Reconnect possible

Kerbia · Tue May 22, 2018 1:59 pm

Hello,
since ~1 week, i have wifi problems with my MT hap ac^2.

All devices get irregular disconnects and no device is able to reconnect to the access point any more. Ethernet connection is still possible.

I'm using Firware 6.42.2 - but I have the same issue with the latest RC version.

The first time it happened the Log showed the following messages:

disconnected, group key exchange timeout
disconnected, unicast key exchange timeout

Since I read about some NTP issues, I switched to a different NTP server. The unicast key problem seems to be gone since then.
With my hap lite, i didn't have any issues like that for more than 2 years.

However, the group key exchange remains.
Update interval is 5 minutes. Using WPA with AES. WPA Passwort consists of letters and numbers only.

aidan · Tue May 22, 2018 3:26 pm

A group update interval of 5 minutes is rather low. What happens if you set it to an hour?

Kerbia · Wed May 23, 2018 4:43 pm

Hey!
I made the changes you suggested. Exact 25 hours after I made the changes and restarted the router, all clients disconnect again and can't reconnect.

Any more ideas?

Sparxx · Wed May 23, 2018 5:07 pm

Do you have any Virtual AP interface binded to any physical interface ?

Kerbia · Wed May 23, 2018 5:19 pm

yes i do - i have a virtual AP on 2.4 ghz which is used for the guest network

Kerbia · Sat May 26, 2018 11:45 am

I checked my other devices, I see the same problem on a hap lite, cap ac and the already mentioned hap ac^2.

The following firmwares were tested on all of those devices so far:
- 6.40.8
- 6.42.2
- 6.42.3

Mon May 28, 2018 10:38 am

Does this problem happens also on the RouterOS v6.43rc version?
It happens on both 2.4ghz and 5ghz wifi?
Only disable/enable helps?
Could you make support output file at that time and send it to support@mikrotik.com

Kerbia · Mon May 28, 2018 8:14 pm

Since i have these issues, i tried out every new RC version. It didn't help.

Enabling/Disabling helps, but not always. When i couldn't reconnect at all any more i just switched between firmwares to get it going again.

Support output file was sent to support last week. I'm waiting for a response.

I am unsure if it happens only on 2.4ghz or also on 5ghz. I will check and provide this information in 2-3 days, when I am at home again.

I also noticed group exchange timeouts on an hap lite, after updating it. Before, i didn't have any kind of such messages in the log.

Kerbia · Fri Jun 01, 2018 2:57 am

6 days ago they just asked me if i already tried out the RC versions, which was already mentioned in the ticket itself.

since that time, i didn't get any response from the support.

Hello,

Thank you for writing to MikroTik Support.
This is an automated reply.

We will try to help you as soon as possible (reply might take up to 3 business days).

3 business days....

Every day I am getting more and more fed up with MT. The last 2 months were a pain. Let's see how long my patience will last.

Fri Jun 01, 2018 9:47 am

Hello, we replied to your ticket within three business days stating a question about whether the issue is still relevant in the latest RC version at the time, but never got back a reply from you. Since 6.42.2 there have been quite a few improvements for hAP ac^2 and we hope that your issue is fixed within these releases. If it is not, please reply to the ticket you submitted.

Kerbia · Fri Jun 01, 2018 1:30 pm

I answered two times on the ticket:
- 1st time: 25.05.2018 at 11:54

Hello,
it does not appear on 6.41.4 or 60.40.8.

However, running the 60.40.8 isn't a real alternative to me, since the wireless performance of the device isn't as good as with the current or RC firmware.

Regards

- 2nd time: 28.05.2018 at 19:20

Hello,
after some more testing, i have to correct myself:

The issue appears on the following firmware Versions: 6.40.8, 6.42.2, 6.42.3.

More testing is going on here, on multiple devices.

Did you already make investigations in my support file?

Regards

I didn't get any error message from the mailservers or your ticket system. Messages to other people also arrived. So I assume they got delivered successfully.

brg3466 · Sat Jun 02, 2018 8:47 pm

I have reported the same issue on other thread. But I can add here that I have same issues on cAP ac as well. My firmware is 6.42.3. The WiFi will drop and if you check the log, it was “ group key timeout”. And my iPhone goes to LTE until you notice it. It won’t automatically reconnect to WiFi until you manually do it.

Kerbia · Tue Jun 05, 2018 2:18 am

Hello, we replied to your ticket within three business days stating a question about whether the issue is still relevant in the latest RC version at the time, but never got back a reply from you. Since 6.42.2 there have been quite a few improvements for hAP ac^2 and we hope that your issue is fixed within these releases. If it is not, please reply to the ticket you submitted.

I just dropped a reminder by Email two minutes ago, that I still didn't get any response 1.5 weeks after the last message.

If my messages really don't appear in your ticket system, something is wrong with it. However, I don't believe that you didn't receive the Emails. I feel that's how Mikrotik support looks like these days. I expected something else. Won't buy any products any more, irregardless of how this issue will be solved or not solved from now on. Maybe even leaving some warnings for potential future customers on Amazon etc.

Kerbia · Thu Jun 07, 2018 1:02 pm

Thats's a message I received from support:

i see that this device is passing VLANs through, what is the lease time on DHCP server?
Does disconnects happen on intervals at half of lease time?
Can You increase it for a test?

My lease time was 3 days. I increased it to 5 days and didn't see any improvement.
brg3466, can you test this as well and share your experience?

Disconnects do not happen on a reproducable intervall - that means a NO to question no. 2 of the support.

gm2066 · Fri Jun 08, 2018 12:05 am

it seams that I have the same isue:
Time Jun/07/2018 22:50:36
Buffer memory
Topics
wireless
info
Message 5C:CF:7F:B1:44:7C@wlan1: disconnected, group key exchange timeout

any help?

brg3466 · Mon Jun 11, 2018 2:46 am

I used to have lease time for 5 minutes and later I changed to 1hr, but no improvement on the disconnection issues. I changed to 1 day the other day but the disconnection still happened.
My firmware is 6.42.3. My wireless setting is very simple , CCR1009 + 3 cAP ac and the CAPsMAN is on CCR1009. I tried to make static address on dhcp server for my apple stuff, iphone, ipad, etc , but it doesn't work. From time to time, the iphone lost connection due to the " group key exchange timeout" and couldn't connect to wifi automatically, I need to do it manually.

Kerbia · Tue Jun 12, 2018 7:36 pm

After I changed the DHCP Lease Time and told them that this did not have any positive or negative effect at all, they wrote me a new Email few hours ago.

They ask now if i collected some more information which would help them to understand the problem. Unfortunately I am not able to deliver more information, since I have absolutely no clue where else to look at. I already did change every potential relevant setting I can imagine.

They also didn't loose a single word about the support file I send them. It would be really nice to know if they find any "bad" setting in there. That's what I asked them now.

I don't have the feeling they even had a look at the file I send to them. All these one line responses just give me the feeling they just write back to me, to have an update in the ticket...

Let's see what there answer regarding my support file will be. They had enough time to look at it, to take that into account. I'm doing well with 6.41.4 now on my devices. All up-to-date versions of current / RC / bugfix channels are pure cancer when i have them on my devices.

Tue Jun 12, 2018 8:50 pm

What is your group key update time interval? What happens when you change it to 1hour?
Oh. You already did it...it helps to me always...
I have no other idea.

brg3466 · Wed Jun 13, 2018 7:34 pm

@Kerbia. do you get stable wifi under 6.41.4 ? If so, I probably want to switch back as well. The drop of Wifi due to this 'groupkey timeout" really bothered me. Hope that Mikrotik can do something about it and fix it.

Kerbia · Sat Jun 16, 2018 11:52 am

Yeah, 6.41.4 runs stable for me. However, there is still the kinda weak throughput with this firmware. But at least it is quite stable.

Ping: Lenovo Y510 Notebook --> Mikrotik hap ac ^2 (6.42.3)

mikrotik_2.png

With 6.41.4 Firmware I have no timeouts, and the ping is between <1 and 1 ms.

Everything can be reproduced on cap ac.

Also: I believe more and more and this happens only at 2.4ghz.

@brg3466, Did you see any disconnects / group exchange timeouts on 5ghz?
Also: What wireless protocoll is selected in your settings? - Is it "any"?

Kerbia · Sat Jun 16, 2018 12:02 pm

Small impression of the router log, while having a phone connected (or disconnected!).

Kerbia · Sat Jun 16, 2018 12:13 pm

Having fun in the last 6 minutes:

sjoukes · Wed Jun 20, 2018 12:29 pm

Hi I've got multiple clients with Capsman setups which display similar behaviour with different hardware.
RB2011RM + wAP AC + Capsman
RB3011 + wAP AC + Capsman
RB2011UiAS-2HnD-IN + Capsman
RB2011UiAS-2HnD-IN + cAP 2n + Capsman
In all situations one or more AP's show groupkey timeouts for 90+% of the clients and the clients give a verification error.
Other AP's seem to keep running fine in the same capsman setup.
This happened out of the blue starting a few weeks ago, it happens with multiple routerOS firmware versions including 6.42.3, 6.42.1 and 6.41.x.
Sometimes it can be resolved by rebooting the hardware and in other situations it resolved itself after a few hours.
I'm starting to think that the issue is related to a firmware/software update of client hardware which is incompatible with the mikrotik Wi-Fi implementation and it can take out an AP.
The screenshot shows one situation where this error occured last week.
I've seen this both on 2.4GHz and 5GHz.

What I have to note is that all sites where this has accured so fare are using lots of streaming devices.
Sonos, Chromecast, Apple TV, Heos, LG and Samsung smartphones and tablets.

Kerbia · Mon Jun 25, 2018 4:49 pm

Last email to support was sent on June 16th. Again no response yet. Not even a "sorry, it might take a bit longer" notification.

6.42.4 tests are going on. First impression is that exchange key problems got less but wifi has ping spikes. I even tested that on a location, where there is no other 2.4 ghz wifi around.

Kerbia · Sun Jul 01, 2018 10:29 pm

After two weeks still no new response from the "support".

Mon Jul 02, 2018 3:57 pm

I don's see any such emails in our system, we usually respond in a few work days. Tell me the ticket number that was assigned and I will check where the email was lost.

anuser · Mon Jul 02, 2018 6:09 pm

Try the following:
- On hap ac2:

--delete everything. reset with no defaults.
--use 6.42.5 firmware

/interface wireless cap set enabled=no
/interface wireless set adaptive-noise-immunity=ap-and-client-mode wlan2
/interface wireless set adaptive-noise-immunity=ap-and-client-mode wlan1 
/interface wireless set amsdu-limit=4096 amsdu-threshold=4096 mode=ap-bridge wps-mode=disabled wlan2
/interface wireless set amsdu-limit=4096 amsdu-threshold=4096 mode=ap-bridge wps-mode=disabled wlan1
/interface wireless set wireless-protocol=802.11 wlan2
/interface wireless set wireless-protocol=802.11 wlan1 
/interface wireless set wmm-support=enabled wlan2
/interface wireless set wmm-support=enabled wlan1

Reenable CAPSMAN afterwards

...
/interface wireless cap set enabled=yes

and set group key update time to 5 minutes, i.e.: 00:05:00 on CAPSMAN controller

Kerbia · Tue Jul 03, 2018 1:09 pm

I don's see any such emails in our system, we usually respond in a few work days. Tell me the ticket number that was assigned and I will check where the email was lost.

Ticket#2018052222003846

Last Email was sent at June 16, 10.59 am CET
I'm sending 50-100 Emails every day. It's interesting that they all arrive, except the ones that are for you guys...

aaronvonawesome · Wed Jul 04, 2018 2:24 pm

I have three devices set up with CAPsMAN (one of course is the "manager"). The devices are 951G-2HnD devices. I was having similar issues, but I recently updated my RouterOS and RouterBOARD firmware on the devices to 6.42.5, and I'm not receiving any more messages like "disconnected, group key timeout". The maximum number of clients on my setup so for has been 37; ranging from Android devices to iOS devices to Desktops and Laptop. On prior versions I was getting "disconnected, group key timeout" all the time.

Here are my CAPsMAN settings:

/caps-man security
add authentication-types=wpa2-psk encryption=aes-ccm group-encryption=aes-ccm group-key-update=5m name=My-Awesome-WiFi-Security passphrase=battle-of-wits
/caps-man configuration
add channel.tx-power=30 country="united states" datapath=datapath-awesome disconnect-timeout=3s distance=indoors frame-lifetime=0ms guard-interval=any hw-protection-mode=rts-cts hw-retries=7 keepalive-frames=enabled max-sta-count=150 mode=ap multicast-helper=default name=config-awesome-WiFi rx-chains=0,1 security=My-Awesome-WiFi-Security ssid=vonAwesome tx-chains=0,1

For the full configuration, you can see my post here: https://www.reddit.com/r/mikrotik/comme ... accepting/

I hope this is helpful B-)

brg3466 · Mon Jul 09, 2018 10:12 pm

I do hope that the new firmware can solve this annoying issues. Since months ago, I was trying different configurations to fight this " group key timeout" issue but failed. I will go update the firmware the first thing when I am back home today and see if it happens again. Have to say, it is really a headache !

Kerbia · Tue Jul 10, 2018 5:27 pm

I don's see any such emails in our system, we usually respond in a few work days. Tell me the ticket number that was assigned and I will check where the email was lost.

If this statement would have been true, how is it possible that I received a poor one line response today (after 3.5 weeks - not 3 days!) where they suggest using b/g/n and a different frequency.
That's not the kind of question anyone wants to hear after numerous responses and almost 2 months after the intial email.

If your companys support has some capacity problems atm, just be honest and stop lying.

6.42.5 is running fine now with my original settings. thank god, however all trust is gone after seeing this clown-fiesta with the support.

However, thanks for that detailled impression of the after sale service. My next order will be placed at UBNT.

brg3466 · Tue Jul 17, 2018 10:16 pm

I upgraded to 6.42.6 and it ran smoothly until this morning. I found my iPhone was using LTE....

I checked the log and it was again the 'group key time out".

( I don't know how to insert the screen shot , didn't find the button in the toolbar)

brg3466 · Tue Jul 17, 2018 11:11 pm

@Kerbia, I sent the suport.rif file together with the screenshot to support @mikrotik.com , let's see if they can receive it and reply.

brg3466 · Fri Jul 20, 2018 10:23 pm

Just some update, the support asked me to try to set the CAPsMAN data rates to default. ( because I set the basic rates at 6M).

Anyone who has experienced the group-key time out with the default data rates ?

brg3466 · Tue Jul 31, 2018 9:40 pm

anyone who can tell me how to upload photo ? I found a button 'insert image' but cannot upload the photos.

I have some interesting photos shows that both registration table and the log indicate that my iPhone connected to the AP but the iPhone home screen only shows LTE.

Kerbia · Thu Aug 02, 2018 2:56 pm

@Kerbia, I sent the suport.rif file together with the screenshot to support @mikrotik.com , let's see if they can receive it and reply.

Advice:

1.) Read
2.) Post

Not the other way around.

Support didn't help at all. They're lying about not receiving Emails, but then answer to them after weeks of waiting.

Kerbia · Tue Aug 14, 2018 1:45 pm

6.42.6 --> say hello to group exchange timeouts again.

don't update from 6.42.5..

damboor · Sun Aug 19, 2018 11:14 pm

Have the same issue with samsung smart tv .
today i update to rc version and in dhcp tick add arp for leases
now i am testing and its look good no dc can watch my netflix ...

opkky · Wed Aug 29, 2018 2:21 am

6.42.6 --> say hello to group exchange timeouts again.

don't update from 6.42.5..

Same behaviour with 6.42.7
I am using 922UAGS-5HPacT and some mobile users can not login at all because of group exchange timeout.

I resolved my issue by changing wireless basic and supported rates. I do not understand how these 2 things relates to each other but it helped me

I am back to 6.42.7. Everything works fine.

brg3466 · Thu Aug 30, 2018 8:04 am

@opkky can you elaborate a bit ? You change basic rate and support rate to what level ?

I am still suffering from this issue.

Thank you !

Kerbia · Mon Sep 24, 2018 3:13 pm

6.42.7 runs quite fine - but webserver is broken there.

6.43.2 intense group key exchange problem as usual. it's 4 months since this issue has not been fixed. i just plugged in my mikrotik for a week of testing, not using them anywhere in any productive environment.

sjoukes · Sun Dec 30, 2018 5:26 pm

Anyone experiencing this issue please update to 6.43.8

I've been debugging this and comparing the Group Key Handshake (GTK update) between 6.43.7 and 6.43.8 during the weekend, since I have a lot of clients with the famous group-key timeout issue.
I came to the following conclusion: if you set the group-key-update timer to 1:00:00 (1 Hour) on versions 6.43.7 and older the group-key is updated every 36 seconds instead of 3600 ( I believe the minimum interval should be 5 minutes (300 seconds)).
After you apply the 6.43.8 update the Group Key Handshake is performed every 3600 seconds just like configured. I did not test any value lower than 1 hour, so I don't know what it did when set to 30 minutes or 5 minutes.
The GTK is shared on establishing a new connection to a Wi-Fi ap during the WPA(2) 4 way handshake. After that it is updated every configured interval with an advised minimum of 300 seconds via a 2 way handshake. You can imagen what happens when a client misses this update and is unable to communicate (unicast/multicast) after it expired. I did not fully read the 802.11 spec. I believe an expired key can be used up to 60 seconds after it expired after that a client needs to re-establish its connection, probably by performing a full 4 way handshake effectively losing its Wi-Fi connection.
Long story short, the Group Key Update feature is working now like it should and did not work before as expected and was probably causing a lot of issues on busy networks or on devices in battery save mode since it had a a lot of opportunities for missing an update of the GTK.

brg3466 · Sun Dec 30, 2018 10:43 pm

@sjoukes Thank you very much for the info ! Really appreciate you dig into it and had some conclusions. Yes, this GKT issue bother me for a long time and Mikrotik actually didn't explain anything about it, neither told us how to solve it. You really did a good job on it and hopefully 6.43.8 solved it. Will upgrade it right now and see the outcome.
Thank you again and happy new year of 2019 !

Kerbia · Sun Jan 06, 2019 3:33 pm

@sjoukes would you be so kind and let me have a look at your "fixed" settings where the group exchange timeout doesn't happen any more?

While running 6.42.9 I didn't have any issues the last months on a hap ac^2, while shit is driving my nuts on a cap ac.
Both access points are connected to the same router still.

I'm so disapointed of MT after I bought the new ARM devices whole router os on wifi devices is just sh***. While it was rock solid over all the last years.

Kerbia · Wed Jan 09, 2019 7:44 pm

Okay, so with 6.43.8 it's still not fixed.

Group exchange timeouts happen exactly in the intervall that is set in the security profile. For me that's atm one hour.

Good job, Mikrotik. NOT!

sjoukes · Thu Jan 10, 2019 10:05 am

@sjoukes would you be so kind and let me have a look at your "fixed" settings where the group exchange timeout doesn't happen any more?

While running 6.42.9 I didn't have any issues the last months on a hap ac^2, while shit is driving my nuts on a cap ac.
Both access points are connected to the same router still.

I'm so disapointed of MT after I bought the new ARM devices whole router os on wifi devices is just sh***. While it was rock solid over all the last years.

All my setups are based on Capsman implementations.

We use the following combinations:
RB2011RM + wAP AC + Capsman
RB3011 + wAP AC + Capsman
RB2011UiAS-2HnD-IN + Capsman
RB2011UiAS-2HnD-IN + cAP 2n + Capsman
RB2011UiAS-2HnD-IN + wAP AC + Capsman
hap ac^2 + Capsman
hap ac^2 + wAP AC + Capsman

After the upgrade to 6.43.8 I did not see any structural group key time-outs on updated clients.
I double checked our central monitoring system and I only see massive group key timeouts at clients with software below 6.43.8.
Some incidental group-key and 4-way handshake timeouts on 6.43.8. but they could all be caused by low signal strengt and or people walking in and out of buildings.
So all has been stable since the update.

I use the following basic setup ( I left out some parts like guest networks, VLAN tagging, multiple bridges etc.)

/caps-man manager
set ca-certificate=auto certificate=auto enabled=yes package-path=/firmware upgrade-policy=suggest-same-version

/caps-man channel
add band=2ghz-b/g/n control-channel-width=20mhz extension-channel=disabled frequency=2462 name=2GHz-11 save-selected=yes skip-dfs-channels=no tx-power=20
add band=5ghz-a/n/ac control-channel-width=20mhz extension-channel=disabled name=5GHz save-selected=yes skip-dfs-channels=yes tx-power=25
add band=2ghz-b/g/n control-channel-width=20mhz extension-channel=disabled frequency=2412 name=2GHz-01 save-selected=yes skip-dfs-channels=no tx-power=20
add band=2ghz-b/g/n control-channel-width=20mhz extension-channel=disabled frequency=2437 name=2GHz-06 save-selected=yes tx-power=20

/caps-man security
add authentication-types=wpa2-psk encryption=aes-ccm group-encryption=aes-ccm group-key-update=1h name=Customername passphrase=CustomerPassword

#####
## Guest network disabled client-to-client forwarding
#####
/caps-man datapath
add bridge=Customerbridge client-to-client-forwarding=yes local-forwarding=no name=Customername

/caps-man configuration
add channel=2GHz-01 country=netherlands datapath=Customername  mode=ap name=2.4GHz-01 security=Customername ssid=Customername-2G 
add channel=2GHz-06 country=netherlands datapath=Customername  mode=ap name=2.4GHz-06 security=Customername ssid=Customername-2G
add channel=2GHz-11 country=netherlands datapath=Customername  mode=ap name=2.4GHz-11 security=Customername ssid=Customername-2G
add channel=5GHz country=netherlands datapath=Customername mode=ap name=5GHz-Only security=Customername ssid=Customername
#####
## Create a profile per MAC address with fixed 2.4GHz channel, 5GHz does not need this.
#####
/caps-man provisioning
add action=create-dynamic-enabled comment="Generic 5GHz" hw-supported-modes=ac master-configuration=5GHz name-format=prefix-identity name-prefix=5G
add action=create-dynamic-enabled comment="Generic 2.4GHz" hw-supported-modes=gn master-configuration=2.4GHz-06 name-format=prefix-identity name-prefix=2G
#####

##### Kick devices with low SNR in high density environments
##/caps-man access-list
##add action=accept disabled=yes signal-range=-80..120 ssid-regexp=""
##add action=reject disabled=yes signal-range=-120..-81 ssid-regexp=""
##### Apple support wmm / Wi-Fi Calling
##/ip firewall mangle add action=set-priority chain=postrouting comment="Set priority for WMM" new-priority=from-dscp-high-3-bits passthrough=yes

@Kerbia Would you please share your configration?
What type of clients are disconnecting and what is theire average signal strenght while this happens?

Kerbia · Fri Jan 11, 2019 2:27 pm

It's multiple devices from different manufacturers.

Playstation4
Wireless Printer from HP
Various different kind of Samsung Smartphones
Iphone
etc.

Signal strenght across all devices is very good, since they're all located in the same small flat. Disconnect happens exactly when the exchange intervall is due.

sjoukes · Fri Jan 11, 2019 2:43 pm

It's multiple devices from different manufacturers.

Playstation4
Wireless Printer from HP
Various different kind of Samsung Smartphones
Iphone
etc.

Signal strenght across all devices is very good, since they're all located in the same small flat. Disconnect happens exactly when the exchange intervall is due.

Does it happen on both 5Ghz and 2.4?
It would be interesting to have the RAW WiFi traffic captured from your network to see what is happening, but that is not simple to setup.
Are you willing to share your configuration?

Who is online