Community discussions

MikroTik App
 
woodych
just joined
Topic Author
Posts: 20
Joined: Fri Nov 12, 2021 7:09 pm

VXLAN over WireGuard weird packet size issue over VDSL - Have I found a bug?

Sun Dec 29, 2024 5:14 pm

Hi Gang

In the past I made a post, that I can cause a kernel crash on my CCR1009 (tilegx) when using WireGuard and VXLAN on that same device. The devs apparently could not reproduce.

But I still did not give up in trying to find a way to l2 bridge a client with a dynamic IP into the LAN. So I moved VXLAN interfaces to a Mikrotik behind my CCR1009 Router and WG Server. A Mikrotik with a static IP of course.

So the set-up looks like this:

MT-Client (dynamic IP) <-> (Internet) <-> vdsl MTU-1492 <-> CCR1009 (terminating WG) <-> MT-VXLAN (bridged to the LAN).

WireGuard MTU is set to 1412 to perform over VDSL
VXLAN MTU is set to 1500 and DF is set to 'disable'.

But often, there are tcp sessions getting stuck. I suspected and MTU issue and I was right, but not in the way I assumed that packets bigger than some specific size start to cause issues. No that is not the case. Packets with Size 1500 get through, no problem!

So on a client attached to the MT-Client I started testing packet size towards another client within my LAN:

for i in `seq 1000 1500`; do ping LAN-Client-IP -M do -s $i -c 1 -W 1 > /dev/null && echo "$i OK" || echo "$i FAILED"; done

I noticed that a ranges of 8 specific packet sizes from if I am not mistaking 1283 to 1289 do NOT get through!

Sniffing both sides is showing: it's the replies LAN => Client which don't make it through.
Sniffing the VXLAN interface on the MT-LAN shows the replies being present there.

Changing the MTU for the WireGuard Tunnel or the VXLAN Interface do not seem to have any effect.

Interestingly, changing the MTU of the Ethernet Interface on which the IP address used by VXLAN (MT-VXLAN side) does make the windows of packets sizes affected move around. Setting the MTU to 1000 shifts the window to packetsizes not getting throught to 902 to maybe 905 (not 8, but about 4 sizes affected).

I start to wonder especially when using 'reasonable' MTU which should work, and observing exactly 8 different packet sized being affected. Is somwhere in the code of WireGuard or VXLAN and hardcoded size of 1500 which collides wen running over VDSL 1492 MTU?

-Benoît-
 
rplant
Long time Member
Long time Member
Posts: 559
Joined: Fri Sep 29, 2017 11:42 am

Re: VXLAN over WireGuard weird packet size issue over VDSL - Have I found a bug?

Mon Dec 30, 2024 2:16 am

Not sure.

You may have already tried this.

I would try using a separate ethernet interface for the wireguard connection into the 2nd router from the CCR.
With an mtu of 1492, and different IP address range to the lan.

You may need a routing rule to get the wireguard traffic to exit via this separate interface.
or perhaps just set the default gateway for this 2nd router to be via this separate interface.
or if the remote end is static, a route to that via separate interface.