I am currently running a setup between 2 Mikrotik devices (router A and router B) and would like to confirm whether MTU are set for optimal performance.
Network diagram enclosed.
Devices behind the 2 Mikrotik devices are seggregated in VLAN and can communicate inter-sites on devices within same VLAN (thanks to EoIP).
The setup works with devices operating in both IPv4 and IPv6 addresses.
The question here is regarding MTU: is this optimized for speed - or if there's some MTU adjustments I missed?
- The Mikrotik router A and B are connectected via an ethernet cable 1Gbps (and inter-site cable) - Ethernet ports Actual MTU 1500
- The ethernet ports of the router A and B are part of the same inter-site VLAN - inter-site Vlan Actual MTU 1500
- Within this VLAN, the mikrotik establish a wireguard tunnel to encrypt the traffic - Wireguard actual MTU 1420
- Inside this Wireguard tunnel, an EoIP tunnel is established - EoIP Actual MTU 1380
- inside the EoIP tunnel, multiple VLAN are trunk'd - the routers' ports are allocated VLAN - Devices' VLAN actual MTU 1500
The WireGuard MTU can be increased to 1440, if only IPv4 is used outside the tunnel to transport the WG packets. The usual 1420 default value was choosen for the case where IPv6 is used to transport the WG packet over an IP MTU of 1500. Using IPv4 reduces the header overhead by 20 bytes.
Why WG to encrypt traffic?
If it is direct connection with cable what are you afraid of according data?
Why, wanting to encypt traffic you do not establish encrypted EIP? It one password to be added to EOIP configuration. Then you bind VLANS to EOIP interface and ... voila ... done. All traffic is encrypted trafvelling over EOIP.
Does WG start so much faster? On the other hand why not to change ethernet cable to FO?
On the third other hand where that eth cable is installed (max 100m) if you are aware of compromising so much?
Do not underestimating the problem, just asking.
Unless the EoIP tunnel MTU is set to 1500 you will have connectivity issues for any untagged traffic as devices usually do not perform PMTUD on L2 connections - they expect ethernet to have a minimum MTU of 1500 as mandated by the standards.
Any VLANs attached to the EoIP interface will generate packets with the VLAN and EoIP encapsulation overheads, the maximum size of the packet is only limited by the EoIP L2MTU so the EoIP packet will be split into two fragments by the WG interface.
You should set EoIP MTU to at least 1504 to accomodate 802.1Q header used by VLANs. But IMO there's no harm in setting it even a bit higher. IP packets transported through layered tunnel will be 1500 bytes (gross size with headers included) ... which is "plain ethernet" L2MTU. And that's already larger tgan what your layered tunnel supports without fragmentation occuring. But when fragmentatiin does happen, it'll be two frames (one maximum size and one the remaining) and as long as EoIP L2MTU doesn't go higher than double the maximum payload size of tunnel, it'll be two packets in any case (with same overhead, both in terms of additional bits and in terms of processing).
But setting L2MTU higher than bare minimum may save you some headache. It's actually similar to seeing L2MTU of physical interfaces (ether, wifi, etc.) somewhere near 1600 bytes ... which doesn't mean that all frames are of that size.
OTOH if that intersite ethernet (the outer layer of your "onion tunnel") can transport jumbo frames (at least 1700 bytes), then you should be able to setup the layered tunnel so that EoIP could provide MTU of 1504 bytes (or higher) without need to fragment frames. And that would be the best setup performance-wise.
The points mentioned above are quite correct — providing an MTU smaller than 1500 on a Layer 2 network can indeed cause various issues.
If you really need to configure it this way, it’s best to also apply the correct MTU settings on the client network interfaces.
Handling fragmentation places a significant load on the router’s CPU, as it effectively doubles the packets per second (PPS) to process.
If you’re connected directly via Ethernet cable or Dark Fiber (or if your provider offers SDN or overlay/MPLS solutions that natively support higher MTUs), it’s better to increase the outer-layer MTU instead.
Additionally, some switches or NICs support MACsec with hardware-accelerated line-rate forwarding. This encryption is specifically designed for Layer 2 networks and is generally a better fit. (Granted, Mikrotik might not yet support hardware acceleration for MACsec, but if you’re running something like CHR in a VM, you can implement it on the host system — some NICs support full hardware offload for MACsec, for example: NVIDIA MACsec Full Offload.)
Regarding the table: you can calculate required MTU and L2MTU values for each layer by going "outwards" (or in your table from bottom upwards). Values for "devices VLAN" and "EoIP" are fine. Since EoIP adds 42 bytes overhead, you need MTU of at least 1546 bytes on wireguard. Wireguard adds 60 bytes (when transported over IPv4) or 80 bytes (when transported over IPv6), which means inter-site VLAN MTU of 1606 or 1626 (depends on IP variety). And accordingly higher (by 4 bytes) MTU values for physical interconnect port.
As I already wrote, most of time it doesn't hurt to have all of these MTUs set even higher ... as long as you keep "devices VLAN" at 1500 bytes, frames won't get fragmented and packers/frames on each layer will be of the same size regardless of L2MTU and MTU settings ("M" stands for Maximum and it's perfectly normal to transport smaller packets/frames without any additional overhead).
Just don't overdo the sizes. Devices (switches in particular) tend to keep port buffers aligned to certain boundaries and if you set L2MTU slightly higher, buffer space effectively halves (even though the new L2MTU is not double the size) even if no frame transported will ever be larger than that threshold.
does this larger frame settings would need updated setting on clients devices TCP/IP stack (desktops, laptops, smartphones ….)? - I woudl suppose the answer is ‘no’: as we changed the onion outer layer to retain a standard 1500 MTU size facing clients devices.
Is there a way to test the MTU on each layer? Is ping with no fragment the right tool? For example with wireguard layer, ping both sides without fragment with packet size (example: ping address=ip.addr.to.ping size=1546 do-not-fragment)
You're right in need to adjust L2MTUs so that L2 interfaces support required (L3) MTU. As I already wrote, L2MTU can be set much higher than corresponding (L3) MTU, the only gotcha is the one I mentioned previously (granularity of size of port buffers).
indeed you don't have to change anything on clients devices ... they will be getting standard 1500 byte MTU and that's the only thing which matters to them
you can test MTU on L3 layers ... because you'll have some IP connectivity and you can ping the peer device. And yes, ping is one option. Linux ping knows command line option "-M" and if you want to test maximum packet size which passes L3 connection, you definitely don't want fragmentation to happen ... so you want to use option "-M do" .
But it's hard to do it on L2 layers ... because L2 layers typically don't support similar function to ICMP echo request/reply.
