I guarantee you that if you have a 1500 IP MTU on a tunnel interface, then something somewhere is doing fragmentation / reassembly because something’s gotta give.
Native un-tunneled IP packets typically expect to have 1500 bytes to play with - tax, tag, and title. This means that the L2 MTU needs to have the extra 18 bytes needed for the ethernet header (in the case of ethernet layer2 links).
so if you’re doing tunneling, then the outer IP packet has the normal 1500 bytes of MTU space, but the INNER interface loses X bytes - X being the header size of your particular tunnel protocol’s header. In the case of EoIP, the header size is 42 bytes.
It’s got to come from SOMEWHERE.
If you set the inner interface’s MTU to 1500, then that means the space goes “outward” - i.e. the outer IP MTU must be 1542 in order to carry this packet w/o fragmentation, and then add another 18 bytes for ethernet links - so the ethernet interface’s L2MTU needs to be at LEAST 1560 to carry your 1500-byte inner packet without fragmentation. If any link in the chain between B and C cannot carry the associated frame size (1560) or packet size (1542) then that device will do something based on the situation.
A: If it’s a switch w/ a l2mtu less than 1560, it’s going to silently discard the frame and bump its giants counter on the interface.
B: If it’s a router (or L3 switch) with the next-hop interface’s IP MTU being less than 1542 then one of two things will happen:
-
- if the EoIP packet has the DF bit set, then the router will discard the oversized packet and return an ICMP mtu exceeded message to your tunnel Mikrotik which sent the EoIP packet
-
- if the EoIP packet has no DF bit set, then the router will fragment the EoIP packet and send the chunks along their way. Your receiving tunnel router will get the two fragments (hopefully) and re-assemble them into the original 1542-byte packet, decapsulate the inner payload, which is an ethernet frame containing an IP packet - which will be 1518 bytes - decode THAT and then handle the resulting 1500 byte inner IP packet.
Note that situation A looks exactly like your situation.
Note that situation B1 looks exactly like yours as well, except that you would be able to detect ICMP packet too large messages coming from wherever it is bottle-necking. (assuming that nobody’s filtering the ICMP messages on their path back to you)
Probably in your case, the EoIP packets are getting fragmented/reassembled on “all the other tunnels” but getting discarded somewhere along the path from B to C.
The clean solution is to determine the true path MTU of the inner packets, and set your EoIP interfaces’ IP MTUs to that value. Your own network will be able to use path mtu discovery to properly utilize all of the links. (assuming that you’re not being daft and filtering ICMP messages)
