I am still struggling with disconnecting Ecobee thermostats.
Can someone please let me know what might be going on with these frames?
The Ecobees are clearly sending frames to the Ubiquiti APs (ether 3 and ether 4 on the hEX), the hEX is seeing these frames (sent to ff:ff:ff:ff:ff:ff) but no responses are sent or heard.
Could these simply be DHCP requests that, for some reason, the hEX is ignoring?
The hEX provides DHCP addresses for lots of other clients, including some Ecobees.
Before sniffing again, add interface=bridge or interface=ether3 to the filtering conditions (for this particular case, in other cases the need may be different). Once you sniff enough packets, run /tool/sniffer/save file=something.pcap (or maybe there is a button in Winbox?). Then download the file and open it using Wireshark, it should show you the actual contents and purpose of those packets.
From this screenshot, one can say that many yellow cars drove by the camera, but not how many of them carried teenagers with purple glasses. Wireshark shows you the latter type of information.
Totally brilliant approach!
Unfortunately, I don’t see anything identifyable and useful in these packets.
.
I should point out that (to me) this is completely bizzarre because the mac addresses on these packets are coming from (Ecobee) devices that are NOT connected (associated) with APs, as shown below.
The lack of connection is what brings me here to begin with. Then I sniffed for those mac addresses and found frames. It’s almost as if the APs here the frames via RF and pass them along to the hEX, but don’t actually establish a connection.
.
The useful part is that these are definitely not DHCP packets - actually, not even IP ones.
Another useful bit of information is that the wireless channel as such must be established (as in, the Ecobees did authenticate as STAtions to the AP, so you should see them in the wireless||wifi||capsman registration table if the AP is a Mikrotik one), otherwise you wouldn’t see Ethernet frames with their source MAC address in the sniff.
Wireshark has dissectors for pretty much every standard protocol, so this stuff seems like some kind of controller discovery proprietary to Ecobee.
I’d say start sniffing, reboot them, and see how the DHCPDISCOVER and DHCPREQUEST from them look like. I’ve got a couple of devices that never renew their DHCP lease (but neither they let it expire internaly), so maybe giving these Ecobees selectively a very long lease might reduce the headache? It could also be that they did try at some point where you were not sniffing but as they could not succeed for whatever reason, they gave up forever. All these thoughts are related to “easy to work around” bugs, there is no “forever ban” message a DHCP server could send to a client to make it shut up forever.
In the ideal case, you would keep sniffing on one MAC address into a file until it loses contact to its cloud and then see what was going on during normal operation and before it died off. There may be some failed attempts that can suggest what went wrong - DHCP renewal, DNS queries, just SYN packets towards a server that disappeared… So many things can go wrong in the life of a poor dumb IoT device.
This has got to be the quote of the month: “So many things can go wrong in the life of a poor dumb IoT device.”
That is another great suggestion: reboot the ecobees, confirm all is working, then keep sniffing until it fails.
Thank you, as always!
Since these frames are some kind of broadcasts, you may want to set multicast-enhance=enabled on wifi interface of your AP … it may or may not help with the problem.
I believe the multicast-enhance is a setting for the AP, which in this case is a Ubiquity Unifi AP and is set in the WIFI config settings. I just enabled it (checkbox).
The packets being captured by the hEX have not changed.
Since packets are broadcast, you’ll always see them on hEX, passing in any possible direction … whether they are getting received somewhere or not. And it’s up to anyone’s guess as to what’s their purpose (and if they would cease to flow if all IoT gadgets would be happy with their connectivity to controller).
I think I understand.
What I don’t understand is how these IoT devices’ packets are being heard, received, and repeated by the AP onto the wired network when there is no established wireless connection between the IoT device and the AP.
You said that APs are Unifi … so the question is for Ubiquiti support … to verify if devices are connected to AP or not. If devices don’t need IP to communicate, then they might simply connect to AP (and you should see them in list of wireless stations) and don’t even perform DHCP handshake to get IP address. And they would still be able to send out broadcast ethernet frames (see, no IP mentioned in sentence up to the brace).
They show up in the Ubiquiti management page as NOT connected. That means, there is no way to communicate with them via Ubiquiti’s management infrastructure.
Three possibilities:
To me, the first one seems the most likely to me, and the last one the least likely.
The vernacular of WiFi is confusing, so the STAtions may be “authenticated” and “associated” to the AP but they may not have passed the WPA(2) authentication yet; this normally means that the AP does not forward the traffic between its Ethernet side and the STA, but it could be that in such a state, the AP lets through those specific frames you can see due to some bug or even intentionally.
I set up a separate SSID (called a “wifi network” in Ubiquiti wording) associated with a single AP (no roaming) and only 2.4ghz.
I have the hex sniffing all 44:61 mac addresses (all 8 of the Ecobees at this one location).
Hopefully, they will stay connected.
If they don’t, and the hEX doesn’t fill up and crash or stop sniffing, I’ll have the sniffed packets file to see what happened.
One thing that is surprising is that the same broadcast packets are showing up at the hEX from the AP and from the Ecobees. No idea what they are, but I thought for sure they were a sign or symptom of the failure to connect correctly. That is, they are there even with properly connected Ecobees.
Thank you!
It’s been an hour and so far so good.
All 8 Ecobee’s remain connected to the single AP via ssid “ecobee”, and the hEX still hears them.
.
.
A morning status update: All is working. All Ecobees are still connected.
Had sniifer running all night, file size is 11MB; downloaded for future investigation; stopped and started sniffer again.
In case anyone is interested, here’s what this morning looks like:
.
.
.
Ecobee, totally unknown to me, just as all the specific Apple protocols are.
Ecobee points to Homekit/HAP (HomeKit Accessory Protocol) … so maybe this document explains somewhere the packets … https://forum.iobroker.net/assets/uploads/files/1634848447889-apple-spezifikation-homekit.pdf
TLDR;
“Bonjour” rings a bell, just as “mDNS”.
Ecobee is a smart-thermostat manufacturer. It’s a completely independant company from Apple.
I don’t know the connection between Ecobee and Homekit/HAP. I looked at the document and there is certainly a lot of Apple-related topics, but none for Ecobee. Perhaps Ecobee borrows some of Apple’s integration processes?
I see one Ecobee device sending ARP requests asking who knows – I don’t know why 1 ecobee needs to know how to talk to another ecobee. I thought they only talk to the Ecobee cloud.
And they are still sending broadcast packets using XID protocol.
And MDNS reqests to 224.0.0.251.
But, none of the Ecobee devices have lost wireless connection with the AP so far.
.
I think if you want to debug issues with wireless connection specifically, you need to sniff wireless packets. For Wireshark take a look at WLAN (IEEE 802.11) capture setup.
FWIW try to set a static channel for ecobee AP and adjust the DHCP server to give long leases (e.g. 1 week).
That is a great point – sniffing the wireless packets might be very interesting.
Unfortunately, that would require runnign Wireshark on a device with it’s own wifi access – something I don’t have set up at that location.
There is a way to run tcpdump on the Unifi access points:
I SSH into a nearby AP and ran “ip addr” to show all the network interfaces. A little bit of googling shows that ra0 is the 2.4ghz wifi interface.
I then ran “tcpdump -i ra0 | grep 44:61” to confirm it was working.
Then “tcpdump -i ra0 -w captured” to save a .pcac file of the captured packets on 2.4ghz.
Then downloaded and opened in Wireshark, filtered on ether.add[0:2] == 44:61 and here is just one out of many frames. No idea what do to with this now.
.
