I have a NAS and a PC connected over 2.5gbe on a switch which connects to an hAP ax3. PC and NAS are in the 2.5gbit ports and I am able to do 2.5gbit between them. The PC, switch and NAS support jumbo frames up to 9014 bytes.
Eventually I want to get another NIC for the PC so I can get a direct link between the PC and NAS with jumbo frames so I can leave the other link at 1500 and avoid all this headache.
But for now… I would like to understand why when I enable jumbo frames on my PC, I cant ping LAN devices with a larger payload than the default MTU (im NOT setting the dont fragment flag). When I ping WAN addresses with a payload larger than the default MTU it properly gets fragmented and I get a response. When I ping the router itself it gets properly fragmented and I get a response. Just not any other device on the LAN (except for the NAS) will respond when I ping with larger payloads. However, when I disable jumbo frames on my PC, larger payloads get fragmented properly when pinging LAN devices.
I would like to know if there’s a away I can ensure that packets between LAN devices get fragmented properly while still maintaining 9k MTU on the PC ↔ switch ↔ NAS link?
Note that the PC and NAS are the only two devices on this switch. Every other device on the LAN is connected via the router (wired and wireless)
L2 devices (switches) don’t fragment large frames, they silently discard them. And all devices inside same broadcast domain (most commonly this means same IP subnet) have to be set up with same MTU (traffic from small MTU towards large MTU is fine, traffic in opposite direction will get dropped). The only place where MTU change may hapoen, is on router … which has multiple L3 interfaces and in case of MTU reduction it can fragment packet (in IPv4 and if router is configured so; IPv6 forbids fragmentation and most big routers don’t fragment because they’re HW offloading routing and underlying switch chips don’t support fragmentation) or drop packet and send ICMP message back (this behaviour is basis of PMTUD).
IMO using large frames is nowdays overrated. This was in times when NICs were almost only modems and all the real frame/packet processing was done by slow (MegaHertz vs. modern GigaHertz clocks) CPUs where processing smaller number of larger frames did help a lot. Say in times of FDDI (whooping 100Mbps when 10Mbps coax ethernet was taking up) and early Intel 80386 processors (pieces running at 40MHz were “the fast ones”).
Yes, it may make sense to run network with MTU larger than industry-standard 1500 (or smaller than that) but it comes with lots of headache, so don’t do it if you don’t have a pack of aspirines handy.