Community discussions

 
Lynxta
just joined
Topic Author
Posts: 3
Joined: Thu Aug 23, 2018 4:10 pm

PPPoE MTU problem

Thu Aug 23, 2018 4:30 pm

Good afternoon, i'm having a problem with MTU with my PPPoE Clients.

If i try to ping them with size=1500, i get a "Timeout" reply.
If i try with size=1600 i get "fragmentation needed and DF set".
If i try with size=1300 i can ping without problem.

There seems to be a blackhole problem, doing tests with a Windows PC i discoveder that the "Timeout" reply appear only in the range of 1453-1472 Bytes. If it's greater i get "fragmentation needed", if it is inferior i can ping.

Also, i see that the error presents only if the PPPoE Server is configured outside my BGP router...

Example:

OK) [BGP] --------- [CCR + PPPoE Server]----- [PPPoE Client]

Always get a reply (frag needed or ping).


NOT OK) [BGP] -------- [CCR] ----- [Another CCR + PPPoE Server] ----- [PPPoE Client]

I get "timeout" in the 1453-1472 range like i described above.

Doesn't matter if i change MTU/MRU values on PPPoE server, i alway get the problem.
Every link is ethernet with default MTU 1500.

I tried with ROS version 6.34.6, 6.38.7, 6.40.9 and 6.42.7, nothing change...

I don't understand where is the problem. Help pls!
 
mducharme
Trainer
Trainer
Posts: 868
Joined: Tue Jul 19, 2016 6:45 pm

Re: PPPoE MTU problem

Thu Aug 23, 2018 11:28 pm

There seems to be a blackhole problem, doing tests with a Windows PC i discoveder that the "Timeout" reply appear only in the range of 1453-1472 Bytes. If it's greater i get "fragmentation needed", if it is inferior i can ping.

Also, i see that the error presents only if the PPPoE Server is configured outside my BGP router...
In Windows ping when you specify the size it is of the ICMP packet itself, different from MikroTik ping where it is the size of the IP packet. When using Windows ping, you have to subtract 28 bytes from the IP packet size you want to test. 1500 - 28 = 1472. Put another way, pinging with size 1472 in Windows is actually testing 1500 bytes. Your timeout reply is therefore in the range 1481-1500; greater than 1500 -> fragmentation needed, less than 1481 -> you get a reply. This makes sense because the default negotiated MTU with MikroTik PPPoE is 1480.
I don't understand where is the problem. Help pls!
First, make sure that you are clamping the TCP MSS, this setting is in the General tab of the PPP profile.
Second, verify that your routers are configured to allow ICMP in their firewalls, since blocking it will break path MTU discovery.

If your PPPoE clients all support RFC4638, it is possible to remove MTU related issues by setting max MTU and max MRU on both PPPoE client and server to 1500. For this to work, you need layer 2 MTU between PPPoE server and PPPoE client of at least 1508 bytes. Not all PPPoE clients support this RFC4638. If you are using MikroTiks as CPE's they would support it.
 
Lynxta
just joined
Topic Author
Posts: 3
Joined: Thu Aug 23, 2018 4:10 pm

Re: PPPoE MTU problem

Fri Aug 24, 2018 12:40 am

There seems to be a blackhole problem, doing tests with a Windows PC i discoveder that the "Timeout" reply appear only in the range of 1453-1472 Bytes. If it's greater i get "fragmentation needed", if it is inferior i can ping.

Also, i see that the error presents only if the PPPoE Server is configured outside my BGP router...
In Windows ping when you specify the size it is of the ICMP packet itself, different from MikroTik ping where it is the size of the IP packet. When using Windows ping, you have to subtract 28 bytes from the IP packet size you want to test. 1500 - 28 - 1472. Put another way, pinging with size 1472 in Windows is actually testing 1500 bytes. Your timeout reply is therefore in the range 1481-1500, greater than 1500 fragmentation needed, less than you get a reply. This makes sense because the default negotiated MTU with MikroTik PPPoE is 1480.
I don't understand where is the problem. Help pls!
First, make sure that you are clamping the TCP MSS, this setting is in the General tab of the PPP profile.
Second, verify that your routers are configured to allow ICMP in their firewalls, since blocking it will break path MTU discovery.

If your PPPoE clients all support RFC4638, it is possible to remove MTU related issues by setting max MTU and max MRU on both PPPoE client and server to 1500. For this to work, you need layer 2 MTU between PPPoE server and PPPoE client of at least 1508 bytes. Not all PPPoE clients support this RFC4638. If you are using MikroTiks as CPE's they would support it.
Thank you for your answer!

Usually i use Mikrotik equipement for this test, so i get the "timeout" and "frag needed" reply using MTik on MTik.
I tried with no firewall at all, so i think i'm not blocking ICMP...

Unfortunately my clients use different vendors, so i can't control them.

I will try with TCP MSS clamping, it is sufficient to set this value only on the server or i have to configure (where possibile) also to the clients?

PS: the other question that makes me crazy is "why, if i set up PPPoE server on first router (the one with BGP) with same configuration, it works?"

Thank you again.
 
mducharme
Trainer
Trainer
Posts: 868
Joined: Tue Jul 19, 2016 6:45 pm

Re: PPPoE MTU problem

Fri Aug 24, 2018 1:13 am

Usually i use Mikrotik equipement for this test, so i get the "timeout" and "frag needed" reply using MTik on MTik.
I tried with no firewall at all, so i think i'm not blocking ICMP...
If I were you I would double check that. Make sure that all three routers in this setup allow ICMP on both input and forward chain from everywhere: NOT OK) [BGP] -------- [CCR] ----- [Another CCR + PPPoE Server]
Unfortunately my clients use different vendors, so i can't control them.
It is still possible for you to enable RFC4638 support on the PPPoE server by setting max MTU and max MRU both to 1500, provided that the path between supports the slightly larger layer 2 MTU of 1508. If a client supports RFC4638, they can set theirs to 1500. This setting should not cause problems with clients that do not support RFC4638, they should continue to get the lower MTU that they get right now and the server's higher max setting of 1500 should not cause a problem. We do this in production and have many clients that do not support RFC4638, they have no issues.
I will try with TCP MSS clamping, it is sufficient to set this value only on the server or I have to configure (where possibile) also to the clients?
Both, but most clients already have this function enabled by default.
PS: the other question that makes me crazy is "why, if i set up PPPoE server on first router (the one with BGP) with same configuration, it works?"
There are a few possibilities that I can think of that would cause that to be the case:
  • The BGP router may be configured to block ICMP in the forward chain
  • The CCR between your PPPoE server CCR and the BGP router may be configured to block ICMP in the forward chain
  • The PPPoE server CCR router may be configured to block ICMP in the input chain
  • The BGP router may be running a different RouterOS version where there is different behavior
  • If you are doing NAT somewhere, there may be a configuration issue with your NAT that is preventing receipt of the packet-too-big message.
  • There may be a missing route on one of the devices that is preventing proper delivery of the packet-too-big messages depending on where they originate.
 
Lynxta
just joined
Topic Author
Posts: 3
Joined: Thu Aug 23, 2018 4:10 pm

Re: PPPoE MTU problem

Thu Sep 06, 2018 9:57 am

Usually i use Mikrotik equipement for this test, so i get the "timeout" and "frag needed" reply using MTik on MTik.
I tried with no firewall at all, so i think i'm not blocking ICMP...
If I were you I would double check that. Make sure that all three routers in this setup allow ICMP on both input and forward chain from everywhere: NOT OK) [BGP] -------- [CCR] ----- [Another CCR + PPPoE Server]
Unfortunately my clients use different vendors, so i can't control them.
It is still possible for you to enable RFC4638 support on the PPPoE server by setting max MTU and max MRU both to 1500, provided that the path between supports the slightly larger layer 2 MTU of 1508. If a client supports RFC4638, they can set theirs to 1500. This setting should not cause problems with clients that do not support RFC4638, they should continue to get the lower MTU that they get right now and the server's higher max setting of 1500 should not cause a problem. We do this in production and have many clients that do not support RFC4638, they have no issues.
I will try with TCP MSS clamping, it is sufficient to set this value only on the server or I have to configure (where possibile) also to the clients?
Both, but most clients already have this function enabled by default.
PS: the other question that makes me crazy is "why, if i set up PPPoE server on first router (the one with BGP) with same configuration, it works?"
There are a few possibilities that I can think of that would cause that to be the case:
  • The BGP router may be configured to block ICMP in the forward chain
  • The CCR between your PPPoE server CCR and the BGP router may be configured to block ICMP in the forward chain
  • The PPPoE server CCR router may be configured to block ICMP in the input chain
  • The BGP router may be running a different RouterOS version where there is different behavior
  • If you are doing NAT somewhere, there may be a configuration issue with your NAT that is preventing receipt of the packet-too-big message.
  • There may be a missing route on one of the devices that is preventing proper delivery of the packet-too-big messages depending on where they originate.
Sorry for late response, i explained myself in a bad way...

the BGP router IS the first CCR with PPPoE:

[ Router A = CCR+BGP+PPPOE]-------[Router B = CCR+PPPoE]

if i try to ping a client logged in router A everything is ok. At default PPPoE values (MTU/MRU 1480) i can ping the client, from another connection, without problem: if size is <= 1480 i get my response, if size is >=1481 i get fragmentation needed.

Instead if i try to ping a client logged in router B i get response with size <=1480, BUT if the size is betweeen 1481 and 1500 i get "timeout", if >1500 i get "fragmentation needed".

I checked all the firewall rules but i'm not blocking anything (except some TCP but not in forward).
Every router has the same ROS version (i tried with different ones).

Note: if i set a rule in router A with action=clear df i get corret answer.
 
mducharme
Trainer
Trainer
Posts: 868
Joined: Tue Jul 19, 2016 6:45 pm

Re: PPPoE MTU problem

Thu Sep 06, 2018 8:37 pm

if i try to ping a client logged in router A everything is ok. At default PPPoE values (MTU/MRU 1480) i can ping the client, from another connection, without problem: if size is <= 1480 i get my response, if size is >=1481 i get fragmentation needed.

Instead if i try to ping a client logged in router B i get response with size <=1480, BUT if the size is betweeen 1481 and 1500 i get "timeout", if >1500 i get "fragmentation needed".
Where are you pinging from? Have you tried a packet capture to see what is going on?

You are using no NAT at all?

You have no missing routes that would prevent proper delivery of the packet-too-big?
 
idlemind
Forum Guru
Forum Guru
Posts: 1102
Joined: Fri Mar 24, 2017 11:15 pm
Location: USA

Re: PPPoE MTU problem

Sun Sep 09, 2018 6:56 pm

There seems to be a blackhole problem, doing tests with a Windows PC i discoveder that the "Timeout" reply appear only in the range of 1453-1472 Bytes. If it's greater i get "fragmentation needed", if it is inferior i can ping.

Also, i see that the error presents only if the PPPoE Server is configured outside my BGP router...
In Windows ping when you specify the size it is of the ICMP packet itself, different from MikroTik ping where it is the size of the IP packet. When using Windows ping, you have to subtract 28 bytes from the IP packet size you want to test. 1500 - 28 = 1472. Put another way, pinging with size 1472 in Windows is actually testing 1500 bytes. Your timeout reply is therefore in the range 1481-1500; greater than 1500 -> fragmentation needed, less than 1481 -> you get a reply. This makes sense because the default negotiated MTU with MikroTik PPPoE is 1480.
I don't understand where is the problem. Help pls!
First, make sure that you are clamping the TCP MSS, this setting is in the General tab of the PPP profile.
Second, verify that your routers are configured to allow ICMP in their firewalls, since blocking it will break path MTU discovery.

If your PPPoE clients all support RFC4638, it is possible to remove MTU related issues by setting max MTU and max MRU on both PPPoE client and server to 1500. For this to work, you need layer 2 MTU between PPPoE server and PPPoE client of at least 1508 bytes. Not all PPPoE clients support this RFC4638. If you are using MikroTiks as CPE's they would support it.
TCP MSS clamping will do absolutely nothing for ICMP. It will only affect TCP. Personally you should not actually need TCP MSS clamping at all. It's a crutch.

The answer is to fix the MTU along the path to reflect the media and encapsulation methods being used.

Who is online

Users browsing this forum: No registered users and 18 guests