Community discussions

 
en1gm4
newbie
Topic Author
Posts: 33
Joined: Sun Oct 02, 2016 6:27 pm
Location: UK

understanding and fixing MTU/MSS/PMTU with IPsec

Sun Jan 13, 2019 2:55 pm

We have a in issue with our office connection to AWS via an IPSEC tunnel in that anything session oriented (http, ssh) will not work properly,
We discovered however that reducing the MTU on the ethernet interface on one of the computers in the office to 1400 appears to solve the problem and both SSH and HTTP work fine.
I am presuming that changing to 1400 lowers the MTU to the lowest in the path and messages are being correctly passed to web servers (SYN?) on AWS to make that work... but at 1500 there is something not working in the path as the web servers are not ever "hearing" that 1500 bytes is too big
From this it appears that PMTU is not working and devices on AWS are not reducing packet sizes correctly.

It is a relatively simple setup
Office connected to internet via PPPoE DSL (RB4011)
CHR on AWS connecting directly to AWS internet connection and creating an IPsec tunnel back to the office (ESP)

I've used ping with DF set to ascertain that the largest ping between the office and our AWS vpc is 1378. Add 28 bytes for IP and ICMP should make the MTU 1406. This traffic goes over the ipsec vpn
Pinging from the same host in the office to 8.8.8.8 gives a 1452 ping + 28 = 1480 MTU which is consistent with the MTU setting on the office router (4011) connected via ADSL modems to our ISP (Plusnet in the UK). Internet traffic works fine.
I get the same maximum ping size the other way (from AWS vpc to the office)

I'm not clear on what to configure to fix this ... despite much reaching on MTU. MSS, clamping, SYN etc. (as well as lots of posts about VPN's causing problems like this)
Lowering the MTU on the Ethernet to the modem or on the CHR does not seem right as it is working fine for all our web traffic in what is pretty much a standard configuration.
It would seem that focusing on the the IPsec tunnel and ensuring that both ends know that MTU/MSS would be sensible but I cannot find any setting in the actual tunnel config.
That seems to leave using a mangle ? But how?

I keep seeing something like this mentioned... but what PMTU size is it clamping to? Will it reduce mss on traffic that is already working fine to the internet rather than over the tunnel? on some of the traffic will go to AWS. Most just goes to the internet directly
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes \
protocol=tcp tcp-flags=sy
sorry for the newbie questions... I have tried to do my homework first.. honest.
 
idlemind
Forum Guru
Forum Guru
Posts: 1057
Joined: Fri Mar 24, 2017 11:15 pm
Location: USA

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Jan 14, 2019 12:07 am

You have to points of concern actually. Traffic inside of the VPN and traffic outside of the tunnel.

You'll want to make sure you are allowing the ICMP too big and fragmentation needed messages on input and forward (outside of tunnel and inside of tunnel).

MSS clamping is technically not required if MTU and path MTU discovery is working correctly. Relying on MSS clamping only corrects TCP traffic.

I'm on mobile right now so I can't check the policy based VPN options around these features. I usually use tunnel interfaces with IPSEC securing the tunnel over tunnel mode IPSEC but I'm weird. I can try to update more later.
 
en1gm4
newbie
Topic Author
Posts: 33
Joined: Sun Oct 02, 2016 6:27 pm
Location: UK

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Jan 14, 2019 12:53 am

Thanks.
At the moment all traffic outside of the tunnel (to the internet from the office or from our vpc) works fine (although it may be worth checking if things are getting fragmented thet shouldn't)
I'll have a look at what might be blocking MTU discovery. I think all ICMP are allowed between any devices in our internal 10.x.x.x network. Any other known suspects? It's not something I know much about.
I'd certainly prefer that MTU discovery was automatic.. given the issue is only on our vpn it does not seem likely that is any other device.... But I'll also double check for any possible aws security group issue.
 
en1gm4
newbie
Topic Author
Posts: 33
Joined: Sun Oct 02, 2016 6:27 pm
Location: UK

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Jan 14, 2019 12:16 pm

for the record, altering the MTU on the ethernet interface of our AWS instance to the same value worked out using ping testing (1406) fixes the problem so it seems clear that PMUD is not working

this doc helped
https://community.cisco.com/t5/collabor ... -p/3115561

... but I have not yet worked out how to track down where the ICMP messages that control this are not working
Presumably I could also use the router to change the MSS manually instead of changing MTU in AWS??
(changing that MTU lowers performance for access coming from the internet and even more so for internal to AWS where Amazon us using 9000 byte frames)
 
en1gm4
newbie
Topic Author
Posts: 33
Joined: Sun Oct 02, 2016 6:27 pm
Location: UK

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Mon Jan 14, 2019 6:16 pm

I feel like I am in a conversation with myself here but doing it (briefly) anyway in hope it will help someone in future.
Adding a mangle to rewrite the mss on syn packets going from our office to our AWS VPC seems to have done the trick. The VPC hosts then see a 1364 mss which is small enough to cross the ipsec tunnel without fragmentation.

Figuring out why ICMP is not allowing this to be automatic is a task for another day
 
Trackboy
Member Candidate
Member Candidate
Posts: 206
Joined: Mon Oct 31, 2011 11:19 am
Location: Hungary

Re: understanding and fixing MTU/MSS/PMTU with IPsec

Thu Jan 17, 2019 2:55 pm

Hello guys! Sorry for my english first.....
I have got similar problem or i think so. I have got a Mikrotik IKEv2 road warrior VPN with RSA authentication. I have got PPPoE connection at home. MTU and MRU is 1480.
When i connect with Strongswan client on Linux Mint, i experience that, ping is fine, there is not packet loss at all, but the browsing is not working well.
I tried with WIN 10, but that is ok, working everything fine, so this is really weird for me.

Who is online

Users browsing this forum: No registered users and 122 guests