PPPOE MTU ALWAYS DEFAULTS TO 1480 INSTEAD OF 1492

[himurae]

hello people


i have ros 17.7 x86 on a bare metal mini pc ,my issue is my isp provides mtu 1492 (i have tested on VYOS/OPNSENSE it is indeed this and it works without any issues )but on mk ros it always defaults to 1480 .
I tried changing parent interface mtu to 1508/1504 also pppoe interface max mtu to 1492/1500 but actual mtu goes to 1492 after any change for 5 secs then reverts back to 1480.
why this strange behavior

Edit I tested also on promomox CHR version results are absolutely the same after help from community i have been able to bump mtu to 1488 ,but still not 1492

[mkx]

Yes, MT seems to think it needs extra 12 bytes of PPPoE overhead. And since PPPoE rarely allows for full standard 1500 byte MTU, there will be fragmentation either way. With MT’s lower MTU of 1480 (versus more generous 1492) there will be a tiny bit more fragmentation, but not much more (IMO only a few packets are between 1481 and 1492 bytes in length which would make a difference here).
If you implenent “MTU clamping” or PMTUD works out as intended, then you may see less than 1% of reduction of throughput. If neither “MTU clamping” nor PMTUD work, then you’ll eiher see massive fragmentation (but in most cases packet will get fragmented regardless the PPPoE MTU) or you’ll see massive packet loss (again the magnitude will be the same regardless the PPPoE MTU.

So I wouldn’t sweat over MTU issue too much.

[neelc]

While this article is aimed at ISPs running PPPoE servers, it could also be useful for a PPPoE client: https://blog.apnic.net/2021/06/24/how-to-edge-router-and-bng-optimization/

You could try this:

/interface ethernet set [ find default-name=ether1_for_customer_delegation ] mtu=1520

Then you could increase the MRU to 1492. It seems MikroTik likes a bit more overhead versus other NOS’.

I haven’t tested it though since I haven’t had PPPoE for almost a year (homelab networks aside) when I moved eastwards and have exclusively used DHCP. Well, I’m moving again to a house where it’s basically T-Mobile 5G+BuyVM L2TP. I’ll try to get Verizon to add fiber to our townhome, but if not it’s 5G until Spectrum does DOCSIS 4.0 (I get 60 Mbps uploads vs 35 on Spectrum as I already have the router, but we’re awaiting some home additions).

[tdw]

The underlying layer 3 mtu setting is irrelevent for PPPoE, only the l2mtu has to be sufficiently large (greater than PPPoE mtu + . Not sure why the value is showing up as zero in the OP, certainly on MIkrotik hardware rather than bare-metal x86 it is typically around 1600 and setting the PPPoE client mtu and mru to 1492 just works.

[rplant]

For me with v7 it just chose 1492 with v6 it always chose 1480.

However as @tdw mentioned, you need to set both mru and mtu to 1492.

[himurae]

I tried setting max mtu and mru to 1492 didn’t help ,maybe l2mtu is disabled for x86 WHY CANT MIKROTIK guys fix this important basic feature which every-other firewall/router platform is doing ( i checked OPNSENSE/VYOS/IPFIRE/SOPHOS) all default to 1492 mtu is it too much to ask ???the isp wont change…

[rextended]

The forum supports attaching images without forcing users to see your advertising banners.

[himurae]

The forum supports attaching images without forcing users to see your advertising banners.

i am a newbie still learning around can you comment on my issues since you seem a expert in here.

[qatar2022]

I have seam issue
http://forum.mikrotik.com/t/invalid-mtu-1492-on-pppoe-out1/175450/1

[CGGXANNX]

Do you have a VLAN interface underneath the pppoe-out interface? Maybe you can temporarily turn on logging for PPPoE by adding a new rule for that topic under System → Logging:

You should see in the log that, when initiating the PPPoE connection, your router broadcasts PADI messages, with ppp-max-payload=xxxx (1492 if you didn’t set Max MTU and Max MRU, otherwise their value will be here).

The other side will send back a PADO message. Does it message also include a ppp-max-payload parameter (it would if the PPPoE server supports RFC 4638)? If yes, is the number smaller than 1492?

After that your router will send out PADR message, and get back PADS message. Then there will be several LCP messages being sent back and forth. Do they specify the mru value (such as 1492)? With the LCP messages, the two sides will also agree on the MRU value (MRU of the other side is MTU for your side). Is the agreed MRU value still 1492 or has it been lowered after the LCP exchanges?

Then authentication will be performed. Once this is done, your router will do something that I think is relevant to your issue: It will try to send a LCP echo message with a very large payload, the payload will be chosen so that the packet reaches the MTU/MRU agreed above (in this example below 1484 + 8 = 1492):

If the PPPoE server sends back a EchoRep reply with the same payload length, then the MTU/MRU value is valid and will be displayed in Actual MTU in WinBox. If, however, the underlying ethernet layer could not send a packet that big, no EchoRep will arrive. After a timeout RouterOS will reduce the MTU/MRU of the connection to 1480.

Because my ISP supports RFC 4638, I can replicated this issue by making a PPPoE connection with Max MTU and Max MRU set to something that will later cause the EchoReq message to be larger than my L2MTU. My L2MTU is 1510 and I’ll try to create a PPPoE connection with Max MTU = Max MRU = 1506:

Red arrow is my router setting ppp-max-payload in PADI and blue arrow is my ISP saying that that’s ok because it supports RFC 4638. Next is the agreement on MRU and PAP as authentication method:

Authentication succeeded, and here you can see my router sending out the EchoReq message to test whether a packet with size 1506 could really be sent (1498 + 8 = 1506).

This packet will require the ethernet layer to be able to send 1514 bytes payload (1506 + 8 = 1514), but my L2MTU is only 1510:

So it could not be sent, and no reply will be received. After a few more tries RouterOS will drop the Actual MTU to 1480!

I suspect this is what happened with your configuration, it also explains the 5-second delay before the MTU drop (it’s the wait for the EchoReq tries). And I have seen others observing the same problem, and one thing they have in common with you is that they were also running RouterOS on x86.

I suspect that your network adapter has a low upper limit for the max possible MTU (what MikroTik normally shows with L2MTU). Maybe the adapter supports ethernet frame just large enough for the usual ethernet header and trailer for a 1500-byte IP packet. That’s why I asked above if you have VLAN set up. Normally with VLAN the ethernet frame must be able to grow by 4 more bytes, if you still want to be able to send an IP packet of 1500 bytes. MikroTik hardwares all have L2MTU > 1500 + 4 so you can always enable VLAN and have no problem with MTU 1500. But that’s not necessarily true of the network adapter in your x86 build. Does the adapter have a setting toggle for VLAN support?

[himurae]

Do you have a VLAN interface underneath the pppoe-out interface? Maybe you can temporarily turn on logging for PPPoE by adding a new rule for that topic under System → Logging:


pppoe-log.png


You should see in the log that, when initiating the PPPoE connection, your router broadcasts PADI messages, with ppp-max-payload=xxxx (1492 if you didn’t set Max MTU and Max MRU, otherwise their value will be here).

The other side will send back a PADO message. Does it message also include a ppp-max-payload parameter (it would if the PPPoE server supports RFC 4638)? If yes, is the number smaller than 1492?

After that your router will send out PADR message, and get back PADS message. Then there will be several LCP messages being sent back and forth. Do they specify the mru value (such as 1492)? With the LCP messages, the two sides will also agree on the MRU value (MRU of the other side is MTU for your side). Is the agreed MRU value still 1492 or has it been lowered after the LCP exchanges?

Then authentication will be performed. Once this is done, your router will do something that I think is relevant to your issue: It will try to send a LCP echo message with a very large payload, the payload will be chosen so that the packet reaches the MTU/MRU agreed above (in this example below 1484 + 8 = 1492):


pppoe-mru-6.png


If the PPPoE server sends back a EchoRep reply with the same payload length, then the MTU/MRU value is valid and will be displayed in Actual MTU in WinBox. If, however, the underlying ethernet layer could not send a packet that big, no EchoRep will arrive. After a timeout RouterOS will reduce the MTU/MRU of the connection to 1480.

Because my ISP supports RFC 4638, I can replicated this issue by making a PPPoE connection with Max MTU and Max MRU set to something that will later cause the EchoReq message to be larger than my L2MTU. My L2MTU is 1510 and I’ll try to create a PPPoE connection with Max MTU = Max MRU = 1506:


pppoe-mru-9.png


Red arrow is my router setting ppp-max-payload in PADI and blue arrow is my ISP saying that that’s ok because it supports RFC 4638. Next is the agreement on MRU and PAP as authentication method:


pppoe-mru-10.png


Authentication succeeded, and here you can see my router sending out the EchoReq message to test whether a packet with size 1506 could really be sent (1498 + 8 = 1506).


pppoe-mru-11.png


This packet will require the ethernet layer to be able to send 1514 bytes payload (1506 + 8 = 1514), but my L2MTU is only 1510:


pppoe-mru-12.png


So it could not be sent, and no reply will be received. After a few more tries RouterOS will drop the Actual MTU to 1480!


pppoe-mru-13.png


I suspect this is what happened with your configuration, it also explains the 5-second delay before the MTU drop (it’s the wait for the EchoReq tries). And I have seen others observing the same problem, and one thing they have in common with you is that they were also running RouterOS on x86.

I suspect that your network adapter has a low upper limit for the max possible MTU (what MikroTik normally shows with L2MTU). Maybe the adapter supports ethernet frame just large enough for the usual ethernet header and trailer for a 1500-byte IP packet. That’s why I asked above if you have VLAN set up. Normally with VLAN the ethernet frame must be able to grow by 4 more bytes, if you still want to be able to send an IP packet of 1500 bytes. MikroTik hardwares all have L2MTU > 1500 + 4 so you can always enable VLAN and have no problem with MTU 1500. But that’s not necessarily true of the network adapter in your x86 build. Does the adapter have a setting toggle for VLAN support?

Thanks for this detailed reply

This same box has run Vyos/opnsense/ipfire/pfsense all with pppoe mtu of 1492 nic are intel

please check my logs no matter what i change it defaults to 1480 mtu

[CGGXANNX]

Can you look further down in the log, after the authentication has been performed? There should be a relevant section where your router sent LCP EchoReq messages with . Are there any rcvd LCP EchoRep lines after that?

Looking at your screenshots it looks like the server side does support RFC 4638 and it let your router know that ppp-max-payload can be 1500 bytes. According to the provided log, it looks like your router and the PPPoE server has negotiated a (for your side) MRU of 1500 and MTU of 1492 (which means for the server side, the negotiated MRU is 1492 and MTU is 1500). Your router will send packet with payload up to 1492 bytes, and the server will send packet with payload up to 1500 bytes. Probably this later part will fail, but we don’t see that part of the log from your screenshots.

When you wrote that you did set the MTU to 1492, did you set both values of Max MRU and Max MTU in the PPPoE connection property dialog to 1492? or only the Max MTU part? According to the log, your router still says that it can accept MRU up to 1500, which means you probably didn’t set Max MRU = 1492. Can you try setting both parameters Max MRU = 1492 and Max MTU = 1492?

[himurae]

Hope you can see the bug…

[CGGXANNX]

Yes, from the screenshots of the log, it looks like it happened like I described above. In the new log screenshots both ends of the tunnel now has agreed on MRU = 1492, after successful authentication your router send a test packet with 1484 bytes payload. Together with the LCP header and the Magic number (total 8 bytes) it forms a 1492 bytes PPP payload packet (equals MTU and MRU). But either this packet cannot be sent to the other side, or the reply message (LCP EchoRep, which will be of the same size) does not arrive from the other side, hence the “missed echo reply” log message. RouterOS then drops Actual MTU to 1480 as the result.

Can you try to gradually reduce Max MTU and Max MRU 2 bytes at a time (1490,1488, 1486, etc…) to see at which size the LCP EchoReq packet can be successfully sent, and LCP EchoRep successfully received?

[himurae]

Yes, from the screenshots of the log, it looks like it happened like I described above. In the new log screenshots both ends of the tunnel now has agreed on MRU = 1492, after successful authentication your router send a test packet with 1484 bytes payload. Together with the LCP header and the Magic number (total 8 bytes) it forms a 1492 bytes PPP payload packet (equals MTU and MRU). But either this packet cannot be sent to the other side, or the reply message (LCP EchoRep, which will be of the same size) does not arrive from the other side, hence the “missed echo reply” log message. RouterOS then drops Actual MTU to 1480 as the result.

Can you try to gradually reduce Max MTU and Max MRU 2 bytes at a time (1490,1488, 1486, etc…) to see at which size the LCP EchoReq packet can be successfully sent, and LCP EchoRep successfully received?

i will report back this evening thanks for your inputs hope mikrotik guys are reading this and making some changes to their PPPOE stack.

[himurae]

Yes, from the screenshots of the log, it looks like it happened like I described above. In the new log screenshots both ends of the tunnel now has agreed on MRU = 1492, after successful authentication your router send a test packet with 1484 bytes payload. Together with the LCP header and the Magic number (total 8 bytes) it forms a 1492 bytes PPP payload packet (equals MTU and MRU). But either this packet cannot be sent to the other side, or the reply message (LCP EchoRep, which will be of the same size) does not arrive from the other side, hence the “missed echo reply” log message. RouterOS then drops Actual MTU to 1480 as the result.

Can you try to gradually reduce Max MTU and Max MRU 2 bytes at a time (1490,1488, 1486, etc…) to see at which size the LCP EchoReq packet can be successfully sent, and LCP EchoRep successfully received?

Brilliant analysis!!! as you told mtu/mru 1488 is accepted finally and now i have 1488 as actual mtu ,Wish we could make it 1492 any chance i already see perceived improved latency in my system now…Thanks

[rplant]

It sort of looks like there are 4 bytes required.

Perhaps the pppoe comes on a vlan (quite common)
Or perhaps they are adding a priority vlan header.

Can you manually set the l2mtu of the interface to something higher than 0
eg. 1504

Perhaps via webfig/winbox/cli if you can’t do it from the mobile.

[himurae]

It sort of looks like there are 4 bytes required.

Perhaps the pppoe comes on a vlan (quite common)
Or perhaps they are adding a priority vlan header.

Can you manually set the l2mtu of the interface to something higher than 0
eg. 1504

Perhaps via webfig/winbox/cli if you can’t do it from the mobile.

There are no options through winbox to change L2mtu of pppoe parent interface perhaps through cli can you tell me the command to do so??

[CGGXANNX]

You can only change L2MTU on MikroTik made devices. You have a x86 system with network adapters not in control of MikroTik so that is not possible. Do you run RouterOS directly on the mini PC or do you have a hypervisor in between?

[himurae]

You can only change L2MTU on MikroTik made devices. You have a x86 system with network adapters not in control of MikroTik so that is not possible. Do you run RouterOS directly on the mini PC or do you have a hypervisor in between?

Yes presently i am running on proxmoxx hypervisor on x86 boxes with intel nic and i have in past run direct on this box also result is same no difference