Here is some background and comparison between Mikrotik L2-MTU and switches hardware MTU.
There is a lot of confusion here because most manufacturers (except Mikrotik) do not explicitly describe how the hardware MTU is calculated.
Some background :
http://thenetworksherpa.com/ospf-master-the-mtu-madness/
https://h20566.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=1827689&docLocale=en_US&docId=emr_na-c02587751
And my contribution to try to make things clearer :
usually frame checksum is not counted into port L2 MTU. the regular math (in case of other vendors, like cisco & juniper) the untagged frame size is said to be 1514 bytes and the single tagged one is 1518 bytes.
all in all, L2 MTU mismatch is not an issue, if the intended ethernet payload fits into the smaller one. so if both (read: all) ends of the L3 connection share the same IP MTU, and every L2 device in the path has large enough L2 payload to accommodate the whole IP datagram, it will be just fine. Of course frames found to be bigger than the egress /ingress port’s MTU will be silently discarded.
bottom line: if you use the “other” vendor’s way of calculation (that produces bigger numbers), and use the same value for the mikrotik end (or path), it will not do no harm. just make sure that the L3/IP MTU is calculated according to this.
I normally assume an IP MTU of 1500 bytes and the extra bytes for ethernet header, VLAN tag etc to fit in the L2 MTU limit.
Most MikroTik have a default L2 MTU around 1600 bytes so there is ample room in practice and there is no need to trim
this value down to the “exact” size.
Of course one sometimes encounters hardware that will not accomodate more than one VLAN tag, but I think the appropriate
action is to not use nested VLAN with such hardware, instead of shrinking the IP MTU to make it fit.
I use smaller IP MTU only with tunnels like GRE.
That’s true, but getting the right knowledge about L2-MTU, MPLS-MTU and IP-MTU is a good start as soon as you get problems, so that you know where you need to look for and so that you can design simple tests to diagnose.
For example i did start this basic MTU study because of a problem with a static VPLS tunnel going to a Mikrotik wifi client in station mode. Strangely i have an MTU problem when the control word is enabled. I solved it disabling the control word and rising the L2-MTU on the path, but it’s always interesting to know the root cause of such a problem. In this case it is important to have an exact knowledge of the involved Ethernet frames size, and how they fit in the hardware on the path, so that you can diagnose more easily where the problem is located.
So in the end encapsulating, tunneling or labeling is always a source of MTU problems when the environment is not fully prepared for it.
HP Procurve for example do take into account the FCS CRC Ethernet trailer (4 bytes) into the L2-MTU calculation for there switch. This mean that their advertised hardware MTU of 1522 is in fact equivalent to a Mikrotik L2-MTU of 1504, quite a small value…
A non educated client could think he could do MPLS over such a 1522 hardware MTU, but it is not the case, because it is in fact a 1504 L2-MTU, just enough for a single VLAN tag.
More, there is often no possibility to rise this 1522 limit ( = 1504 limit in Mikrotik L2-MTU terminology) to something bigger - at least for 100 Mbps ports switches. Jumbo framing is most of the time supported only on 1000 Mbps switches.
As soon as you start to use MPLS, VPLS tunnels, VPLS + VLAN or even QinQ or Mac in Mac, you will go well over this limit.
So choosing some hardware for example, it is important to have a good understanding of L2-MTU and hardware MTU, because each manufacturer can have a different way of calculating it. Even worse, Juniper for example do have two different ways of calculating the level2 MTU according to the product line. See my previous post. This add to the confusion.
Then according to what we see in the market, we could differentiate approximately between two way of calculating the low level MTU :
Hardware MTU (do take into account the 14 bytes Mac header, and sometimes the 4 bytes CRC trailer)
(even this hardware MTU is not the true real hardware MTU on the wire (it miss the Ethernet sync preambule for example).
L2-MTU (do not take the Mac header neither the CRC trailer into account)
L2-MTU seems simpler to understand for the end user.
Finally, it’s perhaps not a good idea to put the L2-MTU to a bigger value than what is really needed, it could be an entry point for attacks specially for newer not widely tested protocols. For example :
CVE-2007-4567: The ipv6_hop_jumbo function in net/ipv6/exthdrs.c in the Linux kernel before 2.6.22 does not properly validate the hop-by-hop IPv6 extended header, which allows remote attackers to cause a denial of service (NULL pointer dereference and kernel panic) via a crafted IPv6 packet.
CVE-2008-0352: The Linux kernel 2.6.20 through 2.6.21.1 allows remote attackers to cause a denial of service (panic) via a certain IPv6 packet, possibly involving the Jumbo Payload hop-by-hop option (jumbogram).
CVE-2010-0006: The ipv6_hop_jumbo function in net/ipv6/exthdrs.c in the Linux kernel before 2.6.32.4, when network namespaces are enabled, allows remote attackers to cause a denial of service (NULL pointer dereference) via an invalid IPv6 jumbogram, a related issue to CVE-2007-4567.
I fully agree with you that it is good to fully understand how these things work and what pitfalls you can
run into. I once attempted to send nested VLAN over a MoCA link and hit the L2MTU limit, which is of course
what I tested when doing this. With MikroTik it is not much of an issue but with other manufacturers it can be.
Thanks for the article and nice graphics.
To complete this discussion :
Hardware MTU = Ethernet Mac Header (14 bytes) + L2-MTU
or for some manufacturers :
Hardware MTU = Ethernet Mac Header (14 bytes) + L2-MTU + FCS CRC trailer (4 bytes)
And the true level 1 hardware MTU is something like this :
True Hardware L1 MTU = Ethernet Sync Preamble (8 bytes) + L2-MTU + Ethernet Mac Header (14 bytes) + CRC trailer (4 bytes)
So the Hardware L1 MTU is 8 or 12 bytes more compared to the hardware MTU advertised by manufacturers.
For 1000 Base-T ports the true hardware L1 frame length is never smaller than 520 bytes to enhance synchronization (416 bytes for 1000 Base-X). This is permitted through a Carrier extension filling the frame to a minimum of 416 or 520 bytes depending about the fiber or copper nature of the port.
This allow for frames as small as 64 bytes. This bloc is discarded at the destination, like the preamble header and the FCS CRC trailer.
In the end, for simplicity, the user can rely on L2-MTU (the one defined by Mikrotik) for path maximum size calculations, because all other blocs in the frame (Preamble, FCS CRC, Carrier Extension) are purely hardware blocs, automatically added at the transmitter end, and discarded at the receiver end.
Hardware MTU is probably used by manufacturers for marketing reasons, because its value is higher, but it is not useful and even very confusing for the end user that should concentrate about L2-MTU, like defined by Mikrotik.
Even worse, Hardware MTU as advertised by some manufacturers is in accordance with nothing. It’s smaller than the true L1 frame length, and bigger than the L2-MTU. This add to the confusion and is a perfect tool to make the MTU learning curve very hard for beginners.
To make things clearer i did this :
Carrier extension is only present for 1000 base-T.
Up to 448 padding bytes are added for small packets. This mean that for a large number of small packets, the throughput is only marginally better than Fast Ethernet 100 Base-T !!
So to get a good throughput on a 1000 base-T link, small packets should be avoided. Minimum IP packet size around 400 bytes is needed to fill the link correctly.
In some cases (point to point links) it is possible to use packet bursting to send small packets almost back to back with reduced Carrier Extensions added in between.
Nevertheless i do not see this option on standard 1000 base-T hardware.
10G base Ethernet seems better for small packets, as it does not seem to have the frame carrier extension system. Certainly because it does not keep the compatibility with half-duplex operation.
Here you can see that even if IP-MTU, MPLS-MTU and L2-MTU are the only MTU values useful to diagnose MTU problems, with some more knowledge about the physical L1 layer you can eventually avoid wasting throughput on your costly links with small packets (at least for 1000base-T or 1000base-X links) 