Hi all,
I have recently setup a PPPoE connection over a FTTH fiber link.
The ISP informed that the interface was set MTU to 1492 at its side and provided also the VLAN number to establish the connectivity.
I have setup the VLAN interface with a MTU of 1512 so that the PPPoE connection got 1492 on my side.
Since the VLAN interface is associated to one SFP port, was I suppose to set also such SFP interface with MTU 1512 ?
Thanks in advance.
Armando
Re: Setting MTU correctly on vlan-if for pppoe connection. [SOLVED]
The setting that matters is the Max.MTU/Max.MRU setting in the PPPoE client. Set that to 1492 if that is the max that your ISP supports (and inform them that you would be happier when they supported 1500 and RFC4638).
The MTU on the VLAN and the SFP does not really matter, you can set that to anything >= 1500 for the above setting. But 1512 is fine as well. I have it at 1598 here (the maximum allowed in my case).
The MTU on the VLAN and the SFP does not really matter, you can set that to anything >= 1500 for the above setting. But 1512 is fine as well. I have it at 1598 here (the maximum allowed in my case).
Re: Setting MTU correctly on vlan-if for pppoe connection.
Thank you for clarifying this for me.
Actually I spoke already with the ISP, but they are using the fiber provided by a national company and they have got such limitation as well, so 1492 was the maximum that they could give us.
Initially with VLAN if at MTU=1500 I have got the PPPoE at 1480, therefore I thought to increase the size of MTU within VLAN to raise the MTU seen with PPPoE, but now I have set those values in the PPPoE interface directly.
I did some speedtest to a CHR router that I have in a datacenter and by setting the bandwidth test to send/receive UDP with 1492 packet size I was able to reach almost 980Mbits down and 287Mbits up which is in line with the target speed that the ISP communicated of 1000/300... as long as not so many users will show up, as we are on a split 1:16 and currently I was told there are only 7 users on this branch.
Thanks again for your help.
Armando.
Actually I spoke already with the ISP, but they are using the fiber provided by a national company and they have got such limitation as well, so 1492 was the maximum that they could give us.
Initially with VLAN if at MTU=1500 I have got the PPPoE at 1480, therefore I thought to increase the size of MTU within VLAN to raise the MTU seen with PPPoE, but now I have set those values in the PPPoE interface directly.
I did some speedtest to a CHR router that I have in a datacenter and by setting the bandwidth test to send/receive UDP with 1492 packet size I was able to reach almost 980Mbits down and 287Mbits up which is in line with the target speed that the ISP communicated of 1000/300... as long as not so many users will show up, as we are on a split 1:16 and currently I was told there are only 7 users on this branch.
Thanks again for your help.
Armando.
Re: Setting MTU correctly on vlan-if for pppoe connection.
That is a known problem with MikroTik. When they auto-calculate an inner max MTU/MRU from the outer MTU they assume a worst-case header size of 20 for PPPoE while in reality it is almost always 8.Initially with VLAN if at MTU=1500 I have got the PPPoE at 1480
When you overrule the auto-calculated value in the PPPoE interface it just works, even with PPPoE MRU/MTU of 1492 and outer MTU of 1500.
But, when you set an outer MTU of 1508 or more and the Max MTU/MRU of 1500 it will again fall back to 1480 when the 1500 is refused.
Re: Setting MTU correctly on vlan-if for pppoe connection.
Thank you for this too.
Yes I was reading in the past another post about these 20bytes for MPLS enablement.
Currently the guy at provider told me that they are using MPLS so that the fiber's provider brings the connection to the internet exchange where they are hosting their ISP, so this was part of the reason of 1492 max MTU setting that they provided.
Anyway I understand that with this connection there will be fragmentation unless I will setup some specific client to run MTU=1492 to match to the internet pipe, but the company behind this connection does not do much heavy load traffic. However they will deal with several UDP traffic as they provide some radio services for public safety and other kind of those things with DMR and the like.
I guess I will see when we will setup some of these repeaters if we need to adjust something, even though those devices don't have a lot to set on their network interface side.
Armando
Yes I was reading in the past another post about these 20bytes for MPLS enablement.
Currently the guy at provider told me that they are using MPLS so that the fiber's provider brings the connection to the internet exchange where they are hosting their ISP, so this was part of the reason of 1492 max MTU setting that they provided.
Anyway I understand that with this connection there will be fragmentation unless I will setup some specific client to run MTU=1492 to match to the internet pipe, but the company behind this connection does not do much heavy load traffic. However they will deal with several UDP traffic as they provide some radio services for public safety and other kind of those things with DMR and the like.
I guess I will see when we will setup some of these repeaters if we need to adjust something, even though those devices don't have a lot to set on their network interface side.
Armando
Re: Setting MTU correctly on vlan-if for pppoe connection.
Well of course to enable 1500 byte MTU on PPPoE the ethernet transport network between your router and the ISP PPPoE server has to allow 1508 byte packets.
That means the MTU on that network has to be configured at 1508 everywhere (usually there is a VLAN so 1512 bytes for that) and the hardware has to be able to do that, and the network provider prepared to configure it.
When they use MPLS their hardware probably can do it (because MPLS also adds headers and they still provide 1500 byte MTU to the customer), but the big problem is if the company want to go through the effort of doing it. In your case, apparently not.
Here in the Netherlands my ISP does offer 1500 byte MTU on PPPoE using RFC4638 and apparently they have convinced the network company to implement it.
(maybe it helps that the ISP is owned by the network company)
The issues will not be fragmentation overhead. The UDP packets for such voice systems usually are quite a bit smaller than 1500 bytes and do not need to be fragmented for a slightly smaller MTU.
What you mostly encounter is problems with Path MTU discovery in TCP connections. To avoid fragmentation overhead in intermediate routers, most modern systems use this, but there still are a lot of "Steve Gibson followers" on the internet which break it.
TCP does negotiate an MSS value during setup of the connection, which determines the maximal size of packets (20 more than the MSS).
To solve a lof of problems it is best to put this mangle rule in your router:
This will "peek" in each TCP SYN packet and adjust the MSS value downward to match the lower MTU of your PPPoE link.
As a result, TCP packets will automatically be a bit smaller and there will be no fragmentation. That works much better than the "Path MTU discovery" method.
Also, when you are using IPv6, make sure you set the MTU of your link (1492) in the MTU field in IPv6->ND. That tells ND to broadcast that MTU value to the clients and the clients will put this in their tables as the MTU to be used on all outgoing traffic, with a similar effect as that mangle rule in IPv4.
Many consumer routers do this automatically but RouterOS does not, you have to configure that yourself.
That means the MTU on that network has to be configured at 1508 everywhere (usually there is a VLAN so 1512 bytes for that) and the hardware has to be able to do that, and the network provider prepared to configure it.
When they use MPLS their hardware probably can do it (because MPLS also adds headers and they still provide 1500 byte MTU to the customer), but the big problem is if the company want to go through the effort of doing it. In your case, apparently not.
Here in the Netherlands my ISP does offer 1500 byte MTU on PPPoE using RFC4638 and apparently they have convinced the network company to implement it.
(maybe it helps that the ISP is owned by the network company)
The issues will not be fragmentation overhead. The UDP packets for such voice systems usually are quite a bit smaller than 1500 bytes and do not need to be fragmented for a slightly smaller MTU.
What you mostly encounter is problems with Path MTU discovery in TCP connections. To avoid fragmentation overhead in intermediate routers, most modern systems use this, but there still are a lot of "Steve Gibson followers" on the internet which break it.
TCP does negotiate an MSS value during setup of the connection, which determines the maximal size of packets (20 more than the MSS).
To solve a lof of problems it is best to put this mangle rule in your router:
Code: Select all
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes \
protocol=tcp tcp-flags=syn
As a result, TCP packets will automatically be a bit smaller and there will be no fragmentation. That works much better than the "Path MTU discovery" method.
Also, when you are using IPv6, make sure you set the MTU of your link (1492) in the MTU field in IPv6->ND. That tells ND to broadcast that MTU value to the clients and the clients will put this in their tables as the MTU to be used on all outgoing traffic, with a similar effect as that mangle rule in IPv4.
Many consumer routers do this automatically but RouterOS does not, you have to configure that yourself.
Re: Setting MTU correctly on vlan-if for pppoe connection.
Thank you for this hint about TCP MSS.
I have added this mangle rule now.
Concerning TCP MSS, in the PPPoE profile I was keeping the default value for "Change TCP MSS"; will this kind of applying "a double MSS change" or it won't interfere ?
Otherwise I guess I should set it to no and let the mangle do all the work on this.
About the ISP, it's a relatively small one (compared to others that we have here in Italy) and I know they use MikroTik, but not sure for which role in their networks and if it's all MK or not. But I will try to check with them if any plan to adjust that situation in the medium/long term.
At the moment this customer has 24 fibers coming in their premise, so we might activate another link with a different ISP (this time with one of the big names) for redundancy, so in that case we will see what their config will be.
Armando
I have added this mangle rule now.
Concerning TCP MSS, in the PPPoE profile I was keeping the default value for "Change TCP MSS"; will this kind of applying "a double MSS change" or it won't interfere ?
Otherwise I guess I should set it to no and let the mangle do all the work on this.
About the ISP, it's a relatively small one (compared to others that we have here in Italy) and I know they use MikroTik, but not sure for which role in their networks and if it's all MK or not. But I will try to check with them if any plan to adjust that situation in the medium/long term.
At the moment this customer has 24 fibers coming in their premise, so we might activate another link with a different ISP (this time with one of the big names) for redundancy, so in that case we will see what their config will be.
Armando
Re: Setting MTU correctly on vlan-if for pppoe connection.
Ok I normally do not use the "change TCP MSS" checkmark in the profile as I prefer to see what is happening and have some more control... also that option has sometimes malfunctioned in the past so then I went with the mangle rule. I also sometimes set the MSS explicitly to a lower value like 1280 when there are other bottlenecks in the path like a VPN. Setting a lower MSS once and for all resolves all MTU issues on a mixed-MTU path, at least for TCP.
But maybe the profile option is a little less CPU intensive so it may help when your router is overloaded already. There should not be any conflict.
The reason not to support MTU 1500 often is not the actual equipment at the ISP itself, but the transport network leased from some telecom company.
(the network that transports the PPPoE frames from the customer to the ISP)
This has to allow 1508 or higher MTU, and when it doesn't there is no other option than to offer lower MTU on PPPoE lines.
But maybe the profile option is a little less CPU intensive so it may help when your router is overloaded already. There should not be any conflict.
The reason not to support MTU 1500 often is not the actual equipment at the ISP itself, but the transport network leased from some telecom company.
(the network that transports the PPPoE frames from the customer to the ISP)
This has to allow 1508 or higher MTU, and when it doesn't there is no other option than to offer lower MTU on PPPoE lines.
Re: Setting MTU correctly on vlan-if for pppoe connection.
Ok I will leave that setting in profile as is.
Well about the max MTU, then we are not going to see much differences with another ISP, as the transport network is managed by the same telco company and all the remaining 23 fibers are coming from the same place. It will be mostly for redundancy at this point, knowing that we will have to deal with 1492 max MTU in any case.
Again thanks for having shared all these opinions.
Armando
Well about the max MTU, then we are not going to see much differences with another ISP, as the transport network is managed by the same telco company and all the remaining 23 fibers are coming from the same place. It will be mostly for redundancy at this point, knowing that we will have to deal with 1492 max MTU in any case.
Again thanks for having shared all these opinions.
Armando