Hi Gang

Anyone else experiencing RouterOS Crash when Bridging a VXLAN Interface with a WifiWave2 Interface?

Set-Up

Side 1: CCR1009
PPPoE default route.
RouterOS 7.11.0 multiple ethernet interfaces in 'bridge1'.
VX-Lan interface (to Side 2) in 'bridge1'
Multiple CAPS via CAPSMAN (not CapsMan2).

Side 2: C52iG-5HaxD2HaxD
RouterOS 7.11.2
ethernet 1, uplink, default route.
ethernet 2-4 in 'bridge1'
VX-Lan interface (to Side 1) in 'bridge1'

Clients on both LAN are on same Layer2 interface, works without issue.

On Side 2 I now add an WifiWave2 Interface to 'bridge2'

Clients connecting via WLAN are on same Layer2, and this works for a couple of seconds to a couple of minutes, until the CCR1009 on Side 1 crashes!

Crash does not occur, if WifiWave2 Interfaces are not bridged on Side 2

I have attemted to observe the FDB and Bridge Hosts on Side 2.

What I noticed is that clients connecting via WifiWave2 on Side 2 show up with 'unknown' Bridge Interface.

I also have attempted all different ways to assign the WifiWave2 Interfaces to the Bridge (adding them as static bridge, adding them via WifiWave2 Config Template, adding them via Datapath, etc... I can 100% reproduce the CCR1009 crashing when an active (WiFi Clients Connected) Interface is Bridged to the Bridge containing an VXLAN Interface.

Any suggestions?

I will re-try this set-up with an older, non WifiWave2 AP when I find the time, but usually this is not needed as I then use remote CAP via CAPSMAN.

-Benoit-

I'd make sure they both run the same version. There were numerous bugs in bridging in 7.11.0. So mixing 7.11.0 and 7.11.2 seems like a bad idea.

You might want to use 7.12beta since that has MTU changes so you can set a higher MTU and L2MTU on the wifiwave2 interface briding (e.g. 1600 to be safe) & set VXLAN's MTU to 1500 on both. But MTU fixes are only in 7.12.

Thanks for your reply

VXLAN MTU is 1500. Without L2 LAN would not work

I have updated the WifiWave2 side to 7.12beta3. Still same issue with 7.11 crashing some random time after a Client joins the WifiWave2 AP on the other side of the VXLAN Link.

Will update the other side to 7.12beta3 too, as soon as I have physical access. I experienced too many times to need a Netboot and restore on the CCR1009 after upgrades. So maybe that box is not very healthy anymore.

-Benoit-

I think 7.11.0 was for sure not a good release... In 7.12 the MTU setting will actually do something – before even if you set wifiwave interface to higher MTU, it was fragmenting the VXLAN traffic over-the-air – before 7.12 MTU fix.

I don't route VXLAN traffic over the air.

My goal is to build something like CAP over a Layer3 Network, but without using capsman as I'm still on old Capsman as I only have one WiFiWave2 AP at the moment and all other ones are older.

So it for sure is not a fragmentation issue on the air side. I more fear, there is something broken with the bridge code causing the 'remote' side of the VXLAN Link to crash. The 'remote' side has another VXLAN Link which runs rock solid but is connected to a Linux box not a Mikrotik.

What I find surprising is that the remote crash can be provoked by adding a WifiWave2 Interface to the bridge also containing the VXLAN interface.

I'll upgrade my router to 7.12beta3 as soon as I can and re-test. Maybe it's also not WifiWave2 related as I only ever had one 'client' on the also bridged ethernet interface on the local side. Maybe the crash happens if more than 2 clients connect (making it 4 MAC addresses on the local bridge: Bridge MAC, Client MAC, WifiWave2 MAC Address, WifiWave2 Client).

Upgraded both sides to 7.12beta3

Kernel Crash still occures on the 'tile' CPU after some random time, usually within about 10 minutes, as soon as WiFiWave2 Interfaces become active on the bridge also containing a VXLAN interface.

When I find time, I will try to terminate the VXLAN interface on a Mikrotik with ARM cpu.

As the crash occurs on the remote side of the VXLAN link, not the side with the bridged WiFiWave2 interface, I suppose the mikrotik with the bridge in question generates some ethernet frame, which when sent over the VXLAN link, causes the other side to crash.

PS: I made a short attempt to replace VXLAN with EoIP. It also crashed, but I did not try to repeat as I much prefer VXLAN over EoIP.

I'd be wary of running wifiwave2 on any platform other than ARM ... since it's available on those only for running CapsMan2, I'd expect it to be much less tested (and debugged) than on ARM (where it's used also to drive wireless hardware).

I just read the release notes of 7.12beat7 where they 'increased stability for VXLAN on tile devices'.

Will test again if my kernel crashes were fixed by this.

If it still crashes, best to provide supout.rif to support.
(or autosupout if that was created)

It looks like 7.12beta7 fixed the VXLAN WifiWave2 related crash.

I had the CCR1009 crash once with beta7, but that was probably my fault as inadvertently created a loop via VXLAN

Since it is running stable in the scenario which caused it to crash before.

You have found a way to bypass capsman............ that is pure gold.
Please provide the steps on how to do so successfully.

Capsman has many advantages like a centralised configuration point and hopefully with WifiWave2 sometime centralised key management to allow smoth roaming without re-keying.

So I would not renounce to use Capsman if I would mix WifiWave2 with older AP.

But ok, I give you some hints

Basically you want a L2 Network to all your AP, no matter where they are.

So how to bridge L2 via L3? Link them together with VXLAN! This REQUIRES a static IP on both sides and NO NAT (VXLAN seems to use random ports for each packet).

So what if one side is behind NAT and has a dynamic IP?

Use Wireguard!

So you end up with point to point WireGuard (via NAT and dynamic IP) from each AP to your central router which needs to be reachable via static public IP.

On top of each Wireguard point to point link, you can then establish the VXLAN L2 Link and bridge the VXLAN Interfaces with the WiFiWave2 Interfaces on the AP and with your basic Bridge on your central router.

I'm sure it's not very performing and wastes a lot by encapsulating the packets twice. But hey! It works!

Would I recommend this set-up? Probably not

Beware of creating loops! They are sometimes not obvious.

Also roaming between AP connected this way does not work very well. I guess ARP entries stick around.

-Benoît-