I have a network backbone with several core switches, servers and endpoint switches connected. An MTU of 9000 is set on the backbone ports, on endpoint switches to 1500.
I want to optimize the connection to our router (CCR1009-7G-1C-1S+), where the LAN connection on the router presently is set to an MTU of 1500.
My question is:
Is it more efficient to set the MTU to 1500 on the switch port where the router is connected, or feed the router an MTU of 9000 from the switch, and let the router handle fragmentation of jumbo frames? Or leave it as is, where the port connecting the router to the backbone is set to MTU 1500, and the feeding switch port is set to 9000?
Instinctively, I would assume that fragmentation from jumbo frames to standard frames, should be done on the device with the highest processing capacity. Which naturally is the core switch.
I haven’t got any problems with the current setup, but I’m curious about what should be considered best practices here.
Before you delve into technicalities of where and how to change MTU consider this: MTU should be the same on all hosts, members of same IP subnet. If it’s not, then large packets will be dropped when sent from jumbo-frame device towards standard-frame device. The reason being that frames don’t get fragmented by normal hosts on same L3 subnet, they only get fragmented by gateways if gateway has different MTU sizes on different interfaces (connecting to different L3 subnets). And consequently, switches (L2 devices) don’t fragment packets (as these are L3 entities) … they either pass frames or not depending on L2MTU limits… and L2MTU doesn’t directly relate to (L3) MTU.
I have got about 10 VLANs, each with it’s own IP range. So where should I set the MTU value in the router? On each VLAN interface in the bridge? Or on the entry interface?
I assume the L2MTU values should be set to their maximum values everywhere.
I have got one problematic issue, and that is what jumbo frame means in the terminology for different switch manufacturers. I have got a couple of HPE 1950 switches, and they have got a setting of 9216 as max size. It’s not at all clear if that value is L2MTU or the L3 MTU. I’m bound to believe that the setting is L2MTU, but not at all sure.
Switches always work on L2 … and either accept frame (if its size doesn’t exceed L2MTU or they drop it. And they usually have same L2MTU on all ports.
When it comes to routers, they work on L3 and they have MTU set on each L3 interface (that’s interface with IP address set). In your case these would be vlan interfaces. Due to layered structure there are some constraints about MTU sizes - each higher layer has maximum MTU limited by MTU of one layer lower plus layer overhead. E.g. VLAN interface can have maximum MTU set to MTU of underlying interface (e.g. L2MTU of bridge interface) reduced by 4 (VLAN overhead). So if bridge interface L2MTU is 1500 bytes (usually its higher), then VLAN interface MTU is limited to 1496.
Another example: if ethernet ports have L2MTU of 9216, then bridge interface will have the same L2MTU (bridge interface automatically assumes lowest L2MTU of all member ports) and VLAN interface L2MTU will be 9216-4=9212 bytes.
Or in the opposite direction: if you want to use standard 1500 bytes IP packets over PPPoE tunnel (with overhead of 8 bytes) over VLAN (4 bytes overhead) over ethernet, then this requires 1500+8+4=1512 bytes L2MTU on ethernet layer.
But: it’s often that different devices will have different L2MTUs (e.g. some older Gbps ethernet adapters have L2MTU limited to 4kB). And as I wrote, in L3 network (IP subnet) all devices must have same (L3) MTU … which is then limited with lowest L2 MTU … and can get reduced if a device with lower max L2MTU gets added into network. Since MTU is “locally configured property” (i.e. not provisioned by e.g. DHCP server), it’s best to stick to industry standard 1500 bytes unless one knows much better.
BTW, on modern machines, unless one wants to squeeze out every last bit of throughput on underpowered devices, using jumbo frames in a switched environment is overrated, most devices can cope with very high PPS rates.
Back to routers: each L3 interface can have different MTU setting and router can in this case fragment IP packet. Note that switch (being L2 entity) can not fragment packet because fragmenting a packet means changing/adding IP headers. In IPv6 fragmentation is not allowed so router instead drops oversize packet and replies to sender with ICMP packet too big message which includes information about MTU limitation. Increasingly large number of routers do the same for IPv4 (fragmentation is possible, it’s not required) because fragmentation is very CPU-intensive operation and adds to overall overhead. The whole process is known as PMTUD and if done correctly then each connection will use optimally sized packets (but possibly way smaller than jumbo frames).
So again, jumbo frames are only useful in switched connections and even then the benefits are mostly negligible. And thus IMO not worth the hassle.
Many thanks for taking your time, and the deeper explanation.
I agree with you, that the benefits normally are not worth the trouble. But some implementations are not available, if you cannot set a higher MTU. I have got a Dell PowerStore where there’s only one MTU setting for the whole appliance. At the moment, it’s not possible to set MTU on different VLANs. Backups would benefit quite a lot using a higher MTU, and also iSCSI. As it also serves users through SMB shared folders, the MTU needs to be kept at standard 1500. Raising the MTU would require a complete restructuring of most of the networks, and is probably not feasible. The workstations are not the problem here, but there are several classes of devices that wouldn’t work afterward (special printers, WiFi equipment, IoT devices, NC-machines, etc.).
The switches are transparent (or should be), so the maximum MTU will remain on those. All have MTU capabilities above 9000. I just need to make sure that devices supplying or consuming data use the standard 1500.