Hi Gang
In the past I made a post, that I can cause a kernel crash on my CCR1009 (tilegx) when using WireGuard and VXLAN on that same device. The devs apparently could not reproduce.
But I still did not give up in trying to find a way to l2 bridge a client with a dynamic IP into the LAN. So I moved VXLAN interfaces to a Mikrotik behind my CCR1009 Router and WG Server. A Mikrotik with a static IP of course.
So the set-up looks like this:
MT-Client (dynamic IP) <-> (Internet) <-> vdsl MTU-1492 <-> CCR1009 (terminating WG) <-> MT-VXLAN (bridged to the LAN).
WireGuard MTU is set to 1412 to perform over VDSL
VXLAN MTU is set to 1500 and DF is set to 'disable'.
But often, there are tcp sessions getting stuck. I suspected and MTU issue and I was right, but not in the way I assumed that packets bigger than some specific size start to cause issues. No that is not the case. Packets with Size 1500 get through, no problem!
So on a client attached to the MT-Client I started testing packet size towards another client within my LAN:
for i in `seq 1000 1500`; do ping LAN-Client-IP -M do -s $i -c 1 -W 1 > /dev/null && echo "$i OK" || echo "$i FAILED"; done
I noticed that a ranges of 8 specific packet sizes from if I am not mistaking 1283 to 1289 do NOT get through!
Sniffing both sides is showing: it's the replies LAN => Client which don't make it through.
Sniffing the VXLAN interface on the MT-LAN shows the replies being present there.
Changing the MTU for the WireGuard Tunnel or the VXLAN Interface do not seem to have any effect.
Interestingly, changing the MTU of the Ethernet Interface on which the IP address used by VXLAN (MT-VXLAN side) does make the windows of packets sizes affected move around. Setting the MTU to 1000 shifts the window to packetsizes not getting throught to 902 to maybe 905 (not 8, but about 4 sizes affected).
I start to wonder especially when using 'reasonable' MTU which should work, and observing exactly 8 different packet sized being affected. Is somwhere in the code of WireGuard or VXLAN and hardcoded size of 1500 which collides wen running over VDSL 1492 MTU?
-Benoît-