IPSec restransmissions

[pakjebakmeel]

Hiya, I’m having an issue with my IPSec tunnel and I cannot figure out what the issue is. I have a fairly simple PSK + Xauth setup:

[rbadmin@MikroTik] /ip pool> print where name="IPSec Clients"
 # NAME                RANGES
 0 IPSec Clients      10.7.0.100-10.7.0.254

[rbadmin@MikroTik] /ip ipsec> export hide-sensitive verbose compact
# jun/04/2014 10:27:28 by RouterOS 6.13
# software id = 3192-5XAK
#
/ip ipsec mode-config
set (unknown) name=request-only send-dns=yes
add address-pool="IPSec Clients" address-prefix-length=24 name=MobileClients send-dns=yes split-include=192.168.10.0/24
/ip ipsec policy group
add name=MobileClients
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha1 disabled=no enc-algorithms=aes-256-cbc lifetime=8h name=default pfs-group=none
/ip ipsec peer
add address=0.0.0.0/0 auth-method=pre-shared-key-xauth dh-group=modp1024 disabled=no dpd-interval=2m dpd-maximum-failures=5 enc-algorithm=aes-256 exchange-mode=main \
    generate-policy=port-strict hash-algorithm=sha1 lifebytes=0 lifetime=1d local-address=192.168.10.1 mode-config=MobileClients nat-traversal=yes passive=yes policy-group=\
    MobileClients port=500 proposal-check=obey send-initial-contact=no
/ip ipsec policy
add disabled=no dst-address=0.0.0.0/0 group=MobileClients proposal=default protocol=all src-address=10.7.0.0/24 template=yes

First thing I notice is the line that says set (unknown) name=request-only send-dns=yes. Is it supposed to say unknown? Anyhow, I have configured both my android device and my Windows7 laptop with Shrewsoft to connect and that works fine. The tunnel is established and it passes data. I can ping internal resources and internal resources can ping the mobile client.

But when I copy data in bigger packets such as CIFS towards my samba server of SCP to the routerboard itself there are a lot of retransmissions and duplicate ACK’s and the transfer stalls almost immediately. I have attached a screenshot to this post, this is what the packet stream from an IPSec client doing SCP to the routerboard looks like.

The internet connection is fine, no retransmissions or packet loss there.
The remote internet connection is fine, no retransmissions or packet loss there.
Tested from both Android and Windows client, same behaviour.
Only happens on IPSec, running OpenVPN through the routerboard (terminated inside the LAN) works fine.

I have firewall rules that allow 10.7.0.0/24 to anywhere in the forward chain and the default accept established. I also added an allow rule for 192.168.10.0/24 (LAN) to 10.7.0.0/24 on any interface.

This almost looks like a bug to me? Any suggestions? The capture is done on the routerboard itself.

[cdiedrich]

Everything looks fine on first sight.
Out of a gut feeling, try generate-policy=port-override instead of port-strict in the peer.

Good luck!
-Chris

[pakjebakmeel]

@cdiedrich

Everything looks fine on first sight.
Out of a gut feeling, try generate-policy=port-override instead of port-strict in the peer.

Good luck!
-Chris

Hi Chris,

Thanks for the reply, I have tried that too yes. Same result. I have done a lot of trial and error but nothing seems to make a difference. It almost looks like a bug because it was working fine a few days ago.

I think that if the policy generation is incorrect it should not create a policy at all? It seems the policy gets created. Strange though is that on the command line it looks fine:

 2  D src-address=10.7.0.236/32 src-port=any dst-address=0.0.0.0/0 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes
      sa-src-address=178.250.98.141 sa-dst-address=94.210.71.194 priority=2

and data is passing too, I can ping fine and the transfers appear to start but then freeze because of retransmissions. When you look in Winbox something strange happens. On the dynamic policy it ALWAYS uses the “default” proposal. Even if I create a custom proposal and name it “RoadWarrior” or “MobileClient” and set that on the template it shows “default” on the dynamically created policy. I have flushed SA’s and killed client connection but if I reconnect it always shows “default” even if I specify a different proposal on the template.

Also, it says no PH2 is established?

[rbadmin@MikroTik] /ip ipsec policy> print stats
 0

 1 ph2-state=no-phase2

 2 ph2-state=no-phase2

Is that normal? The tunnel is passing data, this should not be possible if no Phase2 is established??

A copy from an internal samba server towards the mobile client (initiated by the client) seems to be ok. Attached image of transfer in progress. When copying the other direction, from client to the internal server things go belly up and freeze.

Although the copy/transfer speed looks allright it actually has quite a bit of TCP issues when looking at the packets. Mainly from the Mobile client to the RouterBoard. This probably still works ‘allrightish’ because the client only sends small ack packets. But when copying from the client to the server it sends many large packets which completely breaks things. Attached also a screenshot with the packet capture during transfer to the mobile client.

The problem only occurs with IPSec Any other protocol going through the pipes is fine, FTP, SSH, OpenVPN, BitTorrent no issues. Just IPSec.

[cdiedrich]

Strange indeed.
I just checked my VPN gateway with 20 active IPsec tunnels - for each tunnel it also reports no-phase2, but the tunnels (site2site and roadVPN) are up and running, so we can consider this normal.

Could your issue be a MTU problem?
I’ve found this thread which makes sense to me.

Try to lower the MTU to 1420.

For the default proposal I have no clue… But it somehow seems to work, maybe you should first get that copy problem fixed and then approach the proposal issue (ahich I don’t consider to be related).

Cheers
-Chris

[pakjebakmeel]

@cdiedrich

Strange indeed.
I just checked my VPN gateway with 20 active IPsec tunnels - for each tunnel it also reports no-phase2, but the tunnels (site2site and roadVPN) are up and running, so we can consider this normal.

Could your issue be a MTU problem?
I’ve found this thread which makes sense to me.

Try to lower the MTU to 1420.

For the default proposal I have no clue… But it somehow seems to work, maybe you should first get that copy problem fixed and then approach the proposal issue (ahich I don’t consider to be related).

Cheers
-Chris

Hmm, ok so that’s normal then. Maybe just a GUI glitch.

On Android I cannot change the MTU but on Windows I am already running 1380 by default and I went as far as 1300 which should give plenty of headroom for the overhead.

I have changed the default proposal to exactly match my custom one so it shouldn’t matter which one it uses but I find 2 things VERY strange.

1. Even if I specify a different one in the template it always uses ‘default’ (should work though as I ammended that one to match)
2. That (unknown) parameter in the mode-config. This is the default request-only mode config that cannot be modified. Do you have that aswell when you do an export command?

/ip ipsec mode-config
set (unknown) name=request-only send-dns=yes

To be honest, I think I’m facing some nasty bug here. I think things are properly setup and the WAN links are fine. The SA’s and proposals are working and the link is passing data. It was working before and no matter what I try, even using different clients and WAN links or even a mobile 4G connection and lowering MTU, the problem persists. I would guess some bug in the IPSec engine>??

[cdiedrich]

Sigh…

Maybe a bug in RouterOS or in our both brains…
Did you try this on ROS 6.12?My VPN gateway is running 6.12 and works like a charm…

-Chris, out of ideas and now out for lunch

[pakjebakmeel]

@cdiedrich

Sigh…

Maybe a bug in RouterOS or in ur both brains…
Did you try this on ROS 6.12?My VPN gateway is running 6.12 and works like a charm…

-Chris, out of ideas and now out for lunch

Enjoy

thanks

[andriys]

Looks like a typical PMTUD problem. What’s in your firewall config? Do you happen to block ICMP packets unconditionally?

[pakjebakmeel]

[rbadmin@MikroTik] /ip firewall filter> print chain=icmp
Flags: X - disabled, I - invalid, D - dynamic
0 ;;; 0:0 and limit for 20p/s
chain=icmp action=accept protocol=icmp icmp-options=0:0-255 limit=20,1

1 ;;; 3:3 and limit for 20p/s
chain=icmp action=accept protocol=icmp icmp-options=3:3 limit=20,1

2 ;;; 3:4 and limit for 20p/s
chain=icmp action=accept protocol=icmp icmp-options=3:4 limit=20,1

3 ;;; 8:0 and limit for 20p/s
chain=icmp action=accept protocol=icmp icmp-options=8:0-255 limit=20,1

4 ;;; 11:0 and limit for 20p/s
chain=icmp action=accept protocol=icmp icmp-options=11:0-255 limit=20,1

5 ;;; Drop everything else
chain=icmp action=drop protocol=icmpI have done this now:
[rbadmin@MikroTik] /ip firewall mangle> print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=forward action=change-mss new-mss=1350 passthrough=yes tcp-flags=syn
protocol=tcp src-address=192.168.7.0/24 tcp-mss=!0-1350

1 chain=forward action=change-mss new-mss=1350 passthrough=yes tcp-flags=syn
protocol=tcp dst-address=192.168.7.0/24 tcp-mss=!0-1350And the problem has disappeared, so your MTU theory is spot on. What would be a proper was to fix this issue? I have also setup a site-to-site IPSec tunnel against a PfSense box and that does not have this issue. Maybe they set MSS by default.

Why does shrewsoft for windows and the Android VPN client not set the proper MTU. The ICMP rules should allow for PMTU discovery? So how would I prevent oversize packets in the first place? Isn’t this mangle rule a rather crude way of doing this?

[andriys]

Looks good. Type 3 code 4 is allowed, so my guess appears to be wrong.

[pakjebakmeel]

@andriys

Looks good. Type 3 code 4 is allowed, so my guess appears to be wrong.

Is it? Because when I use the mangle rules to force the MSS values, which should decrease the maximum packet size effectively the problem goes away. It should work without these rules too if PMTUD would just discover the proper MTU. So the fact that limiting the packet size with MSS tells me that the problem is indeed MTU size related and solved by avoiding larger packets. The fact that is does not work without those mangle rules is that the packets sent are too big and PMTU isn’t kicking in on a smaller MTU size.

If I look at the data I would say your theory is correct that and PMTUD isn’t working properly. The ping rules are there so that cannot be the cause. Also it works fine on a Site-to-Site link but this RoadWarrior setup is having issues..

hmm.

[andriys]

@pakjebakmeel

What would be a proper was to fix this issue?

Allow ICMP type 3 code 4. But you seem to have them allowed. Still they may be dropped by some other rule in your FW.

Instead of manually allowing some types of ICMP packets, I usually place the following rules at the very beginning of my firewall config (please note the entries with ‘connection-state=related’- that’s what allows ICMP packets related to active connections). I also consider this approach more secure and more efficient (in terms of performance).
/ip firewall filter
add chain=input comment=“Handle already established connections” connection-state=established
add chain=input connection-state=related
add chain=forward connection-state=established
add chain=forward connection-state=related
add chain=output connection-state=established
add chain=output connection-state=related

@pakjebakmeel

Why does shrewsoft for windows and the Android VPN client not set the proper MTU.

ShrewSoft lets you specify MTU for the virtual interface it creates, and sets it to 1380 by default.

@pakjebakmeel

The ICMP rules should allow for PMTU discovery?

That’s correct.

@pakjebakmeel

So how would I prevent oversize packets in the first place?

In road-warrior scenario- by setting proper MTU on your client device if possible. But you don’t have to do it in general.

@pakjebakmeel

Isn’t this mangle rule a rather crude way of doing this?

It is. But it became a common practice nowadays, since your FW is not the only place where the proper ICMP packets may get blocked.

[pakjebakmeel]

Yeah the other FW platforms I have worked with use to create a virtual interface on which you can set the MTU size so PMTU works. As RoS does not create a virtual interface, how should PMTUD work then? How would it figure out the maximum payload size? Does RoS send the fragment ICMP packet back if fragment is required?

Is there a reason not to have a virtual interface because it really eases the firewall rule creation.. Now it’s hard to differenciate between IPSec and WAN traffic because they share the same interface. I can only match using IP.

Is it safe to just bluntly allow related connections in general? Is this not exploitable>?

I tried setting the MTU for shrewsoft to 1350 and still the problem occurs.. Only the MSS mangle rules solve it. Traffice from the server to the client works. Traffic (with bigger packets) from the client to the server stall which is strange because the local MTU of the client should have limited that regardless of PMTUD?

So setting the MTU on the client shoudl prevent it from sending larger packets than the local interface, unfortunately I cannot set this on Android so I presume that relies on PMTUD or it sets a standard ‘safe’ MTU I think.

The MSS rules are more of a workaround rather than a proper solution which makes me wonder why PMTUD doesn’t work properly. Also, My site-to-site is fine without the rules. But again, I think PfSense applies MSS by default so that could explain.

[andriys]

@pakjebakmeel

As RoS does not create a virtual interface, how should PMTUD work then? How would it figure out the maximum payload size? Does RoS send the fragment ICMP packet back if fragment is required?

PMTUD have no relation to virtual interfaces. What it does is, it tries to send a packet of the maximum size possible and with don’t fragment (DF) bit set. Any intermediate router that is not capable of forwarding this packet without fragmenting it first drops the packet and replies to the sender with ICMP type 3 code 4. Sender then tries sending smaller packets.

@pakjebakmeel

Is there a reason not to have a virtual interface because it really eases the firewall rule creation.

That’s how “classic” policy based IPsec works. For IPsec itself virtual interface is not strictly required on the client side either. Clients create virtual interface so that they could assign an IP address (supplied via mode-cfg) to that interface.

@pakjebakmeel

Now it’s hard to differenciate between IPSec and WAN traffic because they share the same interface.

It is relatively easy, in fact. First, you use mangle to mark encrypted packets, then you use that packet marks to match inner-tunnel packets in you filter rules. This approach is based on the fact that if you mark an encrypted (ESP) packet upon arrival, the inner (decapsulated) packet will still have the same packet mark assigned after decryption. You can find an example config on the wiki here.

@pakjebakmeel

Is it safe to just bluntly allow related connections in general?

It is. In any case it is safer to allow only packets related to some existing connection, rather then allowing any ICMP packets of the given type/code.

[pakjebakmeel]

@andriys

PMTUD have no relation to virtual interfaces. What it does is, it tries to send a packet of the maximum size possible and with don’t fragment (DF) bit set. Any intermediate router that is not capable of forwarding this packet without fragmenting it first drops the packet and replies to the sender with ICMP type 3 code 4. Sender then tries sending smaller packets.

Yes I understand that but I thought the router figures out whether it can or cannot forward it by looking at the MTU on the interface to the next hop. But apparently RoS calculates the maximum payload of the inner tunnel and sends an ICMP 3:4 when it thinks it does not fit.

@andriys

That’s how “classic” policy based IPsec works. For IPsec itself virtual interface is not strictly required on the client side either. Clients create virtual interface so that they could assign an IP address (supplied via mode-cfg) to that interface.

Understood.

@andriys

It is relatively easy, in fact. First, you use mangle to mark encrypted packets, then you use that packet marks to match inner-tunnel packets in you filter rules. This approach is based on the fact that if you mark an encrypted (ESP) packet upon arrival, the inner (decapsulated) packet will still have the same packet mark assigned after decryption. You can find an example config on the wiki here.

Riiight, ok so the packet mark survives de-capsulation.. That makes things easier. The only drawback is that mangling the packets potentially consumes more CPU than just firewalling against an incoming (virtual) interface. Thanks for the article, I think I’ll do something like that.

@andriys

It is. In any case it is safer to allow only packets related to some existing connection, rather then allowing any ICMP packets of the given type/code.

makes sense.

So we have determined that the required ICMP protocols can pass through the firewall (they are not dropped elsewhere) so PMTUD should theoratically work.

Packets from LAN → RoS → RoadWarrior pass without issue.
Packets from RoadWarrior → RoS → LAN are (apparently) too big and get dropped.

As ping commands with a too big payload and do not fragment result in a “Needs fragment but DF set” return code we also know that this PMTUD should work. So WHY does it then still send packets that are too large? I mean, the MTU on the virtual adapter in shrewsoft is set to 1350 and even then the problem occurs.. When the MTU on the virtual adapter is set to 1350 how can it then send something bigger? It seems that that is the case because the mangle rule that sets MSS immediately resolves the issue.

Thanks for all the responses, I’m learning here. (Sysadmin, no networks guy)

Last question; I never see any ESP and/or AH protocols. All traffic is UDP/500 and UDP/4500, when does IPSec actually use those protocols instead of encapsulation in UDP? How would I mark those incoming packets in my case?

Update, I found this : https://supportforums.cisco.com/document/64281/how-does-nat-t-work-ipsec

That is probably why, so I just tag UDP4500 instead of ESP

Update2, this works indeed.. very nice!

I have 1 remaining issue, whichever proposal I set on the policy template doesn’t seem to matter. Whenever the dynamic policy is created based on my template it always uses the ‘default’ proposal instead of the one I specify on the template. Any clues?

[andriys]

@pakjebakmeel

I have 1 remaining issue, whichever proposal I set on the policy template doesn’t seem to matter. Whenever the dynamic policy is created based on my template it always uses the ‘default’ proposal instead of the one I specify on the template. Any clues?

No ideas, sorry. I have just a single Mikrotik router where I have road-warrior IPsec VPN configured, and I have never seen similar problems there. Though that particular device is still running RouterOS v6.6, and I don’t plan on upgrading anytime soon.