MTU / MSS problem on IPSec tunnel

RouterOS General
1

I have a MT to MT IPSec tunnel and I’ve found some serious issues on it related to MTU.

Without going into a bunch of detail, I found that print jobs from a Windows based PC on one end of the tunnel was having lost print jobs to a printer on the other end of the tunnel.

[There was a lot of confounding factors that vastly complicated troubleshooting it…but eventually I started looking at packet size.]

And if I set the MSS in mangle at both ends to around 1350, the problem vanishes.
Like so:

/ip firewall mangle add
chain=forward action=change-mss new-mss=1350 passthrough=yes tcp-flags=syn
protocol=tcp src-address=1.2.3.0/24 dst-address=1.2.4.0/24
tcp-mss=!0-1350

So, the real question is: Does anyone have any way to more “correctly” calculate [deterministic method] the MTU/MSS required to help larger packets survive over the IPSec tunnel?

Again: How does one go about calculating the MSS/MTU required on an IPSec tunnel. [Straight IPSec, not over any other tunnels and no tunneling inside either.]

-Greg

2

Depends on whether it is tunnel or transport mode and if ESP or AH or both are used.

For example in tunnel mode with ESP used packet size would be Mac-header + IP-header + ESP header + Data

3

Would you mind showing me the calculations themselves?
Assume IPSec tunnel with AH and ESP.

4

It is not hard to calculate if you look at standards and what size of headers each of protocols have.

For example
ispec tunnel mode ESP with sha1 authentication and aes encryption
will be able to pass 1438 bytes of data without encryption

IP (20) + ESP [(SPI+sequence 8bytes) + Payload [IP (20) + data (1418)] + (padding and authentication, in this case it is in total 34 bytes)]

So in this example TCP MSS will be 1418 - TCP header size.

Here is even illustration what and where is located
http://www.unixwiz.net/techtips/iguide-ipsec.html#esp


Easiest way to determine exact size f data that you can pass is to set up ipsec connection and run ping over it with do-not-fragment bit and check what is the maximum packet size that can be forwarded without fragmentation needed.

You can even use packet sniffer and decode ipsec packet with wireshark and see exact sizes of each header and payload size.

5

Sheesh, since it's "not hard" it would be nice if you'd do it.
[I can probably dig around to find the data, but I assumed someone there would have it at their finger tips and save me a lot of guessing and many minutes digging.]

Since that isn't happening, [thanks for nothing MikroTik] here is what I have found after a lot of digging, and it doesn't agree with the numbers from Mikrotik.

According to Cisco's docs, depending on the type of encryption, a different number of bytes will be added to the packet.

Here's what they list:
esp-aes-(256 or 192 or 128) esp-sha-hmac or md5: 73
esp-aes (256 or 192 or 128): 61
esp-3des, esp-des: 45
esp-(des or 3des) esp-sha-hmac or md5: 57
esp-null esp-sha-hmac or md5: 45
ah-sha-hmac or md5: 44

So, figure you start with 1500 bytes maximum packet size.

1500
Subtract 20 bytes for the IP Header
Then the packet will get wrapped by IPSec
Subtract the varying number of bytes above. In my case, with AES-256+SHA1 that would be 73 bytes. [It's the same size and combination of AES25 or AES or AES128 and, SHA1 or MD5]
So, provided my calculations are right, that leaves a internal packet size of 1407 bytes.

Let me show that again:
1500
-20 (header)
-73 (IPSec header+padding, AES256|AES192|AES128 + SHA1|MD5)

1407 Total remaining
[My test show 1410 byte pings pass without fragmentation.]

If you're using 3DES + SHA1|MD5 it would be


1500
-20 (header)
-57 (IPSec header+padding, 3DES + SHA1|MD5)

1423 Total remaining

[If you're using a GRE tunnel, take out 31 more bytes - that's "non tunnel mode" in RoS, I think.]

However this size didn't help me, so I'd guess there's something missing here.
I set my MSS to 1400 and it would still fail even when 1410 bytes should not get fragmented.

Setting it to 1350 resolved the problem.
I simply didn't have time to test another half a dozen MSS sizes until I got just the "right" one. While I'd love to know the exact size, and 1350 is slightly less efficient than 1400, I will live with it this way.

I'd guess there's some calculation here that's in error, though I don't know what.

I'm glad to have anyone correct these, but from my searching this is the best data I was able to obtain.

It should be obvious, but for those for whom it's not...The above only applies to IPv4, not IPv6.

Keywords:
IPSec MTU MSS fragmentation packet size MRU AES256 AES-256 SHA1 MD5 IPSec Tunnel

6

Oh, and setting the MSS is done like this:

/ip firewall mangle add chain=forward
action=change-mss new-mss=1350 passthrough=yes
tcp-flags=syn protocol=tcp
src-address=10.1.1.0/24
dst-address=10.1.2.0/24
tcp-mss=!0-1350

Change your source and destination ranges to be the local and remote end of your IPSec tunnels
[The internal IP, not WAN IP]

If you use a different MSS size, change both the tcp-mss and new-mss sizes.
The first makes sure you don’t modify MSS’s that are already smaller or the same. The second is what to change it to.

-Greg

7

Those cisco numbers are worst case scenario. And my numbers for specific configuration are valid.
As I mentioned you can always verify MAX packet size by ping and do-not-fragment flag.

8

  1. While the cisco numbers might be the worst possible, it would be FAR better to be a few bytes too small than a few too large and incurr the overhead of fragmentation and re-assembly. Right? So, why not make that clear and use less optimistic numbers? Or at minimum, explain that the bytes of overhead can vary some.

  2. Why not do a table like Cisco did, and give the range of options. Then I can calculate what it should be.

This is a primary beef with MikroTik. Explanations are woefully short, incredibly terse and are the equivalent of “give a man a fish…” vs “teach a man to fish…”

Give me a table so I and other customers can calculate the sizes themselves. [Better yet, update your documentation so it’s in there and one can find the data where they should - in the docs - instead of having to search a morass of forum posts.]

You don’t even really explain your calculations much and it’s really unclear what’s going on.

I post and ask for help, and really the help you give was mostly useless. The only helpful thing you gave was using a ping to determine the size - but that only helps once the tunnel is built and not for pre-planning. If one was trying to see in a theoretical situation what the max MTU was it wouldn’t help at all.

Can you see how the data I’ve left for others to see in the future is a whole lot more useful than yours?
I’m saving you time having to answer this question again. But you didn’t even answer my question with much that was useful, much less others in the future.

-Greg

9

Hi Greg,

Thanks for posting on this subject. I’m wrestling with this this MTU stuff currently and found some worthwhile info here.

If I may say so, though, I think you are being a bit harsh with your comments about the support responses from Mikrotik.

regards, Alan

10

I’m glad the information is helpful.

As for harsh-ness on Mikrotik responses…I guess we’ll have to agree to disagree. My experience with 'Tik’s support is pretty lousy. The typical response is almost always “You’re holding it wrong” and that really doesn’t make me a happy camper.

Again, showing your work, and how you get to the specific MTU size would be a lot more valuable than just saying “1418.” IMO, having this in the docs would seem to be a no-brainer - but as is again, typical for 'Tik, it’s not. Just like OpenVPN docs are a horrible mess and practically unusable, etc.

'Tik has some interesting and neat stuff going, but the RoS bugs, alpha/beta-quality RoS releases, bad docs, [and more] not so much.

-Greg