Can someone explain why MTU discovery doesn't work?

I have a RB5009 connected via a Wireguard tunnel to a Ubiquiti EdgeRouter. I have full bidirectional connectivity between the two LANs behind each router. I had been experiencing an issue where if I tried to access the web management UI of a smart switch at one location over the VPN, it would hang. Looking at tcpdump output showed ICMP “need to fragment” packets. The MTU of the tunnel is 1420, and the MTUs of the two LANs are the usual 1500.

I added a MSS-clamp-to-pmtu firewall rule on the tunnel in the RB5009 and everything started working. My question is why don’t the web browser and smart switch adjust the MTU like they’re supposed to? ICMP is not blocked anywhere along the connection