I’ve been experimenting with configuring RB7601GS’s (hEX S) to create Layer 2 Tunnels for my network. I used this document https://mum.mikrotik.com/presentations/KH17/presentation_4192_1493374138.pdf to get through the initial configuration. I’m using L2TP w/IPSEC using the hEX S’s default which ends up with AES 256 and SHA1. The hEX S is running the lastest stable firmware, 6.49.2.
Upon first configuring the tunnels I was able to get things functioning, but was not getting the throughput I was hoping for. I tried creating mangle rules in the firewall to change the MSS, but have been unable to get them to take effect. I was ultimately successful in getting good bandwidth (it saturates the slowest Inet connection in both directions) by reducing the MRU on the L2TP configuration to 1380 (MRU only, I left MTU at the default 1450).
In checking packet captures to try to understand things better I see in my Syn packets that my TCP sessions between endpoints running through the tunnel are running with MSS at 1460 bytes.
So, all this leaves me with a few questions:
Why did changing the MRU on the L2TP configuration have the effect of increasing throughput?
Why doesn’t changing the MTU at any level have any effect on throughput?
Should I care that the packets between endpoints through the tunnel need to be fragmented at some level?
How can I effect the MSS between endpoints through the tunnel to eliminate any fragmentation?
Hope that all makes sense. I’m getting as much bandwidth as is possible, so it’s working great. I’d just like to have a better understanding, and make things work as efficiently as possible.
Fragmentation can occur in several places. If the packet is greater than the tunnel negotiated MTU each chunk of MTU (or the remainder for the final chunk) is fed into the tunnel encapsulation, if the encapsulated chunk is now greater than the MTU of the WAN connection it will undergo IP fragmentation. The default of 1450 is not really appropriate for any IPsec encryption, it is only sufficient for the ancient and insecure MPPE encryption, so you typically end up with both types of fragmentation and three packets transmitted without tuning.
Whilst IP fragmentation of an encapsulated packet uses less bytes and processing than multiple non-fragmented encapsulated packets they may not be treated as well in transit, so adjust the MTU/MRU to acommodate a chunk which will be less than the WAN MTU. AES operates on 16-byte blocks of data so if the actual data payload (this is not the same as the MTU/MRU) may have padding added.
The negotiated MTU at each end is the lesser of local MTU and remote MRU.
If you are tunneling layer2 frames there will always be fragmentation for larger packets when the size of ethernet frame + tunnel encapsulation + encryption > WAN MTU.
The BCP + MRRU hides the fragmentation, it transparently chops up and reassembles layer2 frames. The ip firewall doesn’t see the IP data, offhand I can’t remember if the bridge firewall has enough functionality. Normally if you want layer2 connectivity you put up with the fragmentation as the tunnel is acting as a long ethernet cable, otherwise use layer3 (i.e. no BCP) with different subnets at each end, and clamp MSS during routing so that TCP connections do not generate any fragmentation.
I’m bridging multiple tagged VLANs through this tunnel, and the way the network is architected I need it to “act as a long ethernet cable”. Different IP segments at each end routed is not an option. With the change of the L2TP MRU, and what you’re saying, I’m probably getting the best I can without the option to do a mangle rule.
Here’s a remaining question. When the MRU on the L2TP was 1450, do you think that was forcing fragmentation in the IPSEC layer which couldn’t manage the fragmentation well, and by reducing the MRU to 1380 it was enough to keep the fragmentation from having to be done by IPSEC? Just really trying to wrap my head around the massive change in throughput by going from 1450 to 1380 with the L2TP config. I’ve done a bunch of packet capturing at different levels to try to see exactly what’s happening, but as soon as you get into capturing the encryption (on the WAN interface) it’s really hard to know what’s happening.
Thanks again. I’m quite impressed with these Mikrotik devices and their capabilities. A lot of bang for the buck there!
Tuning the MRU/MTU to make best use of the WAN MTU, i.e. ensuring there is no IPsec padding and getting the packet as close to the MTU as possible, is definitely the starting point. Even with encryption you can see the the number and size of the packets sent via the WAN interface using the packet sniffer.
I know that L2TP/IPsec AES with SHA-256 and no NAT-T is 80 bytes (so an MRU/MTU of 1420 with a WAN MTU of 1500), SHA-1 will be slightly less but MLPPP increases it and there is an ethernet header as well so it may take some testing to pin down the optimal size. Certainly with an MTU of 1450 each large source packet will generate three packets across the WAN, the MLPPP will be breaking the source packet into chunks less than the MTU, after IPsec the larger of these chunks will be greater than the WAN MTU and subjected to IP fragmentation.
The Mikrotik BCP implementation has limitations with VLAN-aware bridges in that VLANs don’t work as there is no mechanism to configure them on the BCP link. Regular bridges, which act like unmanged switches and pass any untagged or tagged ethernet frames on all ports, are fine but this may limit what else you can set up on the Mikrotik. EoIP is an alternative to BCP but with more overhead, you could do either EoIP/IPsec or unecrypted EoIP over L2TP/IPsec if you need to use VLAN-aware bridges.
Having optimised the MRU/MTU to avoid IP fragmentation you could in addition look at MSS clamping to optimise TCP packet size and boost performance further. As the bridge filter doesn’t have the appropriate matchers and actions it would be one of the few cases where the bridge settings use-ip-firewall and use-ip-firewall-for-vlan are used to force the layer 2 traffic between bridge ports, and including the BCP link, through some of the ip firewall chains to spot new TCP connections and clamp the MSS - see https://help.mikrotik.com/docs/display/ROS/Packet+Flow+in+RouterOS.
Mikrotiks are excellent value for money, very configurable but as with many products do have some annoying omissions and foibles.