TLDR;; The blues of a tricky Movistar setup + a tricky default setting in Mikrotik, and how
I have spent a couple of years with a broken internet connection due to it
My internet provider (Movistar, the “new” branding of the old monopoly Telefonica de España)
has PMTU discovery broken. This is bad. But it is rarely seen as a problem, because
they offer a mostly sound workaround: all their offlcial ONT and fiber
“local loop” infrastructure support L2MTU up to 1600,
which makes easy a negotiation of, say, an ethernet l2mtu of 1520 and
thus be able to have PPPOE with 1500 MTU inside it.
Why do they have PMTU broken? This one is easy to answer, though it took
me a long time to understand: they use PPPOE setup, and their
pppoe server side addres is a RFC 1918 address: 192.168.144.1
for all their network. i.e., they offer my machine a pppoe tunnel, with my public ipv4 address
for me, and 192.168.144.1/32 at the other side.
This means that when their PPPOE tunnel server receives an incoming packet
with size greater than the MTU and with DF, the (ICMP)
Fragmentation Needed (Type 3, Code 4) message containing its MTU
will be sent with a 192.168.144.1 source and thus dropped as it is unroutable.
They could fix it in several ways, from sending a spoofed packet from my address to using
a previous node for it but for whatever reasons they won’t.
Maybe their PPPOE terminators have some bug, or the fix would require
lots of public addresses. Whatever. And, in addition, most modern
network equipment, including by the way Mikrotik routers, support
modest “oversize” L2MTU big enough to absorb the PPPOE overhead.
Except that, by default, if you leave max-mtu and mrru as “auto”,
Mikrotik will negotiate 1480 bytes ( see pe1chl here )
even if the L2MTU is 1520. This means that when you find the
problem and try to fix it by forcing a 1520 MTU
/interface ethernet set ether1 mtu=1520
Mikrotik will still negotiate a MRRU of 1480 and leave you with
a broken PMTU… 
To fix the problem, and in addition of the previous mtu setting, the
pppoe client has to be set up with
max-mru=1500 max-mtu=1500
.
And then everything works like a charm.
I wanted to tell the story because some othe people might
fall into the trap of thinking RouterOS will compute its
max-mtu and max-mru parameters depending on the l2mtu
of the underlying interface.
Can it be fixed easily? I think Routerboard could propose
- instead of proposing a fixed
1480 MTU / 1480 MRRU if you specify auto in both parameters. But
this could break in other setups, like when the MTU bottleneck
is further links away.
Now that I understand the issue, I don’t care much about Mikrotik changing RouterOS, as it
can be easily fixed with a bit of extra configuration. But I have spent
a few hours fighting with it, and this is why I try to document it here.