wAP ax / CAPsMAN: multiple device onboarding flows fail, work immediately via phone hotspot

RouterOS Wireless Networking
1

I’m having a reproducible problem with device onboarding/commissioning over my MikroTik WiFi setup.

Network setup

  • 3 × MikroTik wAP ax, one on each floor
  • Managed centrally using CAPsMAN
  • wifi-qcom
  • traffic-processing=on-cap
  • VLAN-aware network
  • OPNsense as router/firewall
  • AdGuard Home for client DNS, forwarding to Unbound on OPNsense
  • All MikroTik devices/CAPsMAN are running the latest stable RouterOS
  • WPA2-PSK used during compatibility testing

Normal WiFi operation appears to be fine.

The problem occurs specifically during the initial onboarding/commissioning of devices through their vendor apps.

I have reproduced the same type of failure with several unrelated devices/vendors:

  • Bosch ovens / Home Connect
  • Bosch Cookit
  • Sony soundbar
  • Canon printer

The typical sequence is:

  1. Device connects to the wAP ax WiFi.
  2. Device receives a DHCP lease.
  3. DNS and Internet connectivity work.
  4. The vendor app progresses almost completely through setup.
  5. The final onboarding/registration step stalls or fails.

Strongest A/B test

What makes me suspect the MikroTik WiFi very strongly is the following test.

I turn all wAP ax WiFi off and create a mobile hotspot using the same SSID and WPA2 password as my normal home WLAN.

Without changing the appliance configuration otherwise, the affected devices then complete onboarding immediately.

After successful onboarding I can:

  1. disable the phone hotspot,
  2. turn the wAP ax WiFi back on,
  3. and the already-configured devices reconnect to the MikroTik WLAN and generally operate normally.

So the wAP ax WLAN appears perfectly capable of carrying ordinary traffic once the device has been commissioned. It is specifically the commissioning/onboarding phase that fails.

I have also tested with multiple phones/tablets, so this does not appear to be caused by one particular Android phone.

Additional WiFi tests

I also tried to simplify the RF side as much as possible:

  • only one wAP ax active
  • only 2.4 GHz enabled
  • only 5 GHz enabled
  • both bands tested separately
  • WPA2-only
  • 20 MHz channel width on 2.4 GHz
  • roaming-related features disabled for testing

The onboarding issue remained unchanged in all of these tests.

So the failure does not appear to depend on roaming between APs, multi-AP steering, or on the client choosing the wrong band.

Packet observations

With Bosch Home Connect, packet captures show that the appliance:

  • obtains DHCP
  • advertises itself using mDNS
  • resolves the Home Connect cloud endpoint
  • establishes TCP/443 connections
  • begins TLS negotiation

During failed onboarding I repeatedly observed the appliance sending a TLS 1.2 fatal:

Certificate Expired

Because exactly the same appliance can be commissioned successfully through the phone hotspot, I suspect this may be a consequence of the failed commissioning process rather than an actually expired certificate.

With the Sony soundbar I confirmed that it:

  • obtains DHCP
  • is present in ARP
  • sends mDNS traffic to 224.0.0.251:5353
  • sends other multicast discovery traffic
  • has Internet connectivity
  • but still fails at the final setup step

This makes me suspect something around local WiFi traffic during commissioning, especially:

  • WiFi client-to-client forwarding
  • multicast delivery
  • mDNS / SSDP
  • CAPsMAN datapath handling
  • VLAN-aware WiFi forwarding
  • wifi-qcom

Questions

Has anyone seen similar behavior with wAP ax / wifi-qcom / CAPsMAN, where ordinary connectivity works but device commissioning fails?

In particular, I would be interested in whether there are known issues or recommended settings concerning:

  • multicast/mDNS/SSDP between wireless stations
  • client-to-client forwarding
  • traffic-processing=on-cap
  • VLAN-aware CAPsMAN datapaths
  • multicast-enhance
  • testing traffic-processing=on-capsman as a comparison
2

My first suspect would be AdGuard . . . I have a number of things that just won't work with it on, and have not been able to figure out the specific rule that is problematic. The registration may well be hittingnone of these, which using your phone bypasses.

3

I had my lad come to me last week with a brand new Amazon kindle he could not get working.
I thought it might of been me being on 7.24RC so wound the router back to 7.23.2 which still gave the same problem. At that point I asked him to make a hotspot on his phone get it registered and we go from there. After doing all that and finding his MAC address I then removed that and he got connected. Hap ax2 by the way. Also I use Adguard public servers via Rpi/Technitium with hagezi add blocking if that helps in any way. I didn't think too much more about it tbh. I just thought it was a DHCP problem.

Edit:
As a secondary in the same week I had a new leccy/gas meter installed which had the cloud device, this time I didn't mess about I grabbed the Mac address and stuffed it into DHCP with an address outside my pool and it connected straight away and grabbed the address I fed it. Same deal APP registration.

4

The AdGuard public servers don't give me issues - just AdGuard home (running on my RB5009). If I set the device in quetstion to use "dns.adguard.com", all works.

(Specifically, the issue is weather radar display in AccuWeather. The radar display comes up blank when using AdGuard home, and I suspect that they are being "pissy" when thier data gathering attempts are denied . . . and yes, I do use more than the default filters in AGHome, but have also seen it sometimes block necessary stuff, and considering that most appliances are loaded with information gathering "features", I still suspect this - It would seem like a logical point in setup to configure the spyware.

Try setting your DHCP served DNS server IP to something public for a brief period, and try again.

5

Yeah, your mobile phone hotspot does not have the following:

  • OPNsense routing
  • OPNsense firewalling
  • Adguard DNS filtering
  • Unbound DNS resolving
  • Broadband ISP forwarding traffic

And your first instinct is to blame the WiFi access points.

6

Hi tadawson,

thanks — that’s a good point.

I’ve probably spent 50+ hours on this over the last couple of months since moving into the new house, because one device after another needed to be set up. At this point I’ve tried so many combinations that I honestly can’t remember whether I ever tested with AdGuard completely disabled.

It would definitely be one of those “oh, for fuck’s sake…” moments if that turns out to be the whole issue after all this. :grinning_face_with_smiling_eyes:

I’ll disable AdGuard entirely and retry the onboarding with one of the affected devices. Thanks for the suggestion — definitely worth ruling out properly.

7

Thanks — that does sound similar at first glance, especially the fact that using a phone hotspot got the registration through.

One important difference in my case is that DHCP itself appears to work normally. The affected devices associate with the wAP ax, get an IP address, show up in DHCP/ARP, and in several cases already have working Internet access.

A good example is my Sony TV/soundbar setup: both devices can be connected to the WLAN, get an IP and have Internet access, but the onboarding still never completes, so they never show up properly in the Bravia Connect app/account. In some cases the device then disconnects again afterward simply because the setup process was never fully completed.

So the failure seems to happen later than DHCP — somewhere in the actual vendor registration/commissioning step.

That said, your hotspot workaround is very interesting because that is exactly what works reliably for me too. I’ll still keep the DHCP/static lease angle in mind and may test it with one of the affected devices just to rule it out properly. Thanks for sharing the details.

8

And a wifi issue would really surprise me there. Unless you hare firewall filters, to the AP, a packet is a packet . . . it has no idea what the purpose, stage of a process, etc. It passed it, or it does not . . .

9

I’m still looking at the WiFi/L2 side because the affected appliances do get onto the WLAN successfully, receive DHCP, and can reach the Internet/vendor cloud, but the onboarding still fails at the final stage.

My suspicion is that the missing piece is the local communication after the appliance joins the home WLAN. Typically the phone first provisions the appliance with the SSID/password, then the appliance connects to the normal WiFi, and the app has to find/reach it again locally to finish commissioning. Depending on the vendor, that can involve mDNS, SSDP, multicast/broadcast, or direct client-to-client traffic.

So I agree that for normal unicast Internet traffic, “a packet is a packet.” But the fact that cloud access works while commissioning still fails is exactly why I’m wondering about L2/local discovery or station-to-station forwarding rather than basic routing/firewalling. That’s the part I’m trying to isolate.

10

Ok. Working so far.

"the final step" or variant "final setup step". What does that actually mean?

11

Hmmm . . . Perhaps our terminology is different. To me, "WiFi" as used here, describes the radio side, and nothing more. L2/L3 issues (to me at least) are routing, not WiFi . . . (and kind of get back to my thought about AdGuard - somthing is borking traffic from flowing somewhere, and I am still of the opinion that the "Wifi" (IE radio side) of the AP doesn't know how to do that . . .)

If by "WiFi" you are referring to the entire MikroTik device, then I can't argue that as a possibility . . . It def seems to be something in the environent blocking traffic.

If you look at the firewall page, do you see any of the packet counters for your drop rules climbing? It might give you a tad of insight . . . Or perhaps run torch while trying an onboard, and see if you can see what it is dying on.

You might also review AdGuard to see what it is blocking for those devices.

Hopefully I have not confused anyone . . .

12

This is a phenomenon I observed over the years I am here in the forum: People's first suspect when something not working is Mikrotik component.

Opnsense, unbound, adguard, sony, bosch, and so on: no way these are at fault.

These "smart home" devices, when used with their own cloud access points, are not just phoning home all the time and storing all your data "anonymously" in "cloud databases". No, they have built in all kind of telemetry, logging, crash reporting, and what not. If just one of these components can't communicate home because of a blocked domain by adguard - the whole cardhouse collapses.