Community discussions

MikroTik App
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

IPIP over WG performance

Mon Mar 20, 2023 10:16 am

I'm running mesh WG "intranet" between several locations. It runs great, speed tests show WG performs within the links' limits, no complaints.

I want to run IPIP on top of that mesh intranet because it makes number of things significantly easier for me. I've done some experiments and it looks IPIP tunnel has 50-80% of the underlying WG link. Should I be looking into MTU, fragmentation or some other potential issue?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11433
Joined: Thu Mar 03, 2016 10:23 pm

Re: IPIP over WG performance

Mon Mar 20, 2023 10:59 am

It's MTU+fragmentation on top of additional overhead (both computational as well as data volume).

I wonder what IPIP functionality is missing from WG which makes you run IPIP on top of WG?
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

Re: IPIP over WG performance

Mon Mar 20, 2023 11:20 am

It's MTU+fragmentation on top of additional overhead (both computational as well as data volume).

I wonder what IPIP functionality is missing from WG which makes you run IPIP on top of WG?
did fresh test, 30Mbps line, 27Mbps effective TCP. I can totally live with that. Let's see if it holds together with multiple connections.

WG is great for backhaul, but it's routing is very annoying. I want to have failover or dynamic VPN routing. If one direct hop fails, go via some neighbour. With WG I can't do that because I can't have the same subnet "allowed" in two peers. Also, with WG I need to list each and every routed subnet and that's super annoying. IPIP + OSPF so far make my life significantly easier.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3250
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: IPIP over WG performance

Mon Mar 20, 2023 11:33 am

MTU is the right question, and given WG is generally 1420, and IPIP is 1480... MTU is handled complex in RouterOS, but IPIP MTU but would need to be lower WG (e.g. the GRE packet size). Also if there is no IPv6 is involved, the WG MTU perhaps could be higher, but not the expert in WG to say for sure (and the WAN MTU also be involved if you want to be more exact).

Also IPIP would do MSS clamping for TCP by default, but it likely use the IPIP MTU value for that, not the lower WG+IPIP. This may explain the variation in result actually, if the TCP stack on your device figures out the MTU (thus MSS) quickly, it be speed be quick. If it's resulting to using the the minimum TCP packet size (~576 bytes) based on retransmissions, it be much slower.
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

Re: IPIP over WG performance

Mon Mar 20, 2023 12:07 pm

that escalated quickly :) thanks, this is what I asked for.

to makes matter more interesting — some nodes are on fiber, others on LTE/5G
I'll go through all that and see what comes out of it.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3250
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: IPIP over WG performance

Mon Mar 20, 2023 12:31 pm

to makes matter more interesting — some nodes are on fiber, others on LTE/5G
Oh boy. Then yeah WG MTU is actually too high for the LTE in most cases. Typically LTE has been ~1430 MTU for me in US. Mikrotik defaults LTE interface to 1500, but MBIM reports lower at least for me BUT others report higher MTU (and it would be carrier specific so this make sense), I'd look at the logs for the message described here: see viewtopic.php?t=171061.

If one presumes fragmentation is problem... I'm guessing the LTE MTU likely what all the IPIP/WG be based on. And you likely don't need the MSS clamping. You'd be given up some space over the fiber. Then again stuff like ZeroTier uses 2800 MTU, presumable to just accept fragmentation but do it big. So maybe I worry too much about the MTUs.

But it actually hard to test this since various things will remember the MTU in one way or another. So restarting browser/etc, clearing connections on router, before a test may be a good idea. But tool sniffer will show the packet size, so you can see what's happening. If you use /tool/ping with "don't fragment" option with various packet size and your far end IPs, you can generally figure out the MTU that way too.

If I had to guess, WG should be 1370, IPIP should be 1350. I'd leave MSS clamping turned off if you lower the MTU. If your LTE MTU >1420, that could be higher. But just a guess (and using Mikroitk rounding so they end in 0)
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

Re: IPIP over WG performance

Mon Mar 20, 2023 2:39 pm

my WG/IPIP MTUs were 1420/1400 for the LTE device. It's ISP-provided oldish huawei in bridge mode by the way. Can't figure out from the UI the "base" MTU. Using ping + don't fragment shows 1400 is OK, 1401 not. So I guess it's correctly set.
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

Re: IPIP over WG performance

Mon Mar 20, 2023 2:55 pm

my WG/IPIP MTUs were 1420/1400 for the LTE device. It's ISP-provided oldish huawei in bridge mode by the way. Can't figure out from the UI the "base" MTU. Using ping + don't fragment shows 1400 is OK, 1401 not. So I guess it's correctly set.
but in one of the directions the bandwidth test shows practically zero unless I set the remote udp mtu to 1300 :) LOL fun times ahead
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3250
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: IPIP over WG performance

Mon Mar 20, 2023 5:21 pm

but in one of the directions the bandwidth test shows practically zero unless I set the remote udp mtu to 1300 :) LOL fun times ahead
If you want to test ping, increase the MTU so you it doesn't get blocked by your setting. There may be more MTU available, but if you're already set to 1400, that be what ping finds.

But to be honest I'm not sure what the right combo be – the LTE part makes this especially tricky. And some UDP things don't like a lower MTU, including bandwidth test...

Who is online

Users browsing this forum: ivicask, onnyloh, tim427 and 99 guests