Traffic Loss on IPSEC Site to Site VPN

Hi

I have a site to site IPSEC VPN configured between a Unifi Gateway on Starlink and a Mikrotik CCR2004.

This was previously stable on a CCR1009 - but we have upgraded the device to the CCR2004 which has much newer firmware.

Previously, the VPN worked perfectly - however after the upgrade traffic fails on the VPN after about 5 or 6 hours.

The VPN remains connected but no traffic passes. If we pause the VPN from the unifi gateway and then resume it, then traffic resumes - however stops again after a while.

Any thoughts welcome on what may be happening.

Paul

Since there have been quite a few IPsec changes and issues with certain RouterOS versions over the past year, the ROS version probably matters more than the hardware you moved to.

Which version was the CCR1009 running and which one is on the CCR2004?

A few other things that could help narrow it down:

  • Is it IKEv1 or IKEv2 and what are your phase 1 and phase 2 lifetimes? Traffic dying after 5-6 hours could point to a rekey going wrong.
  • When traffic stops, does the peer still show up in "/ip ipsec active-peers" and are there still SAs in "/ip ipsec installed-sa"?
  • Does anything show up in the log when it stops working? Tip: adding ipsec to logging makes it easier to catch (you can leave out the packet logging though, it just floods the log).

Quick side note: since Starlink uses CGNAT, it could be the NAT mapping changing in the background while the tunnel still looks up. It worked fine on the CCR1009 though, so that's just a wild guess.

If you can post your ipsec config (/ip ipsec export with the secrets removed), it'll be easier to see what's going on. And if anything odd shows up in the log, feel free to include that too.

Thanks for the reply!

The CCR1009 is on 6.43.1 - we inherited this with a very complex configuration and have never felt confident about upgrading it!

The CCR2004 is on 7.24.4

It's IKEv2

Lifetimes seem to be 1 hour and 8 hours

When the data stops the peer is still active - I'm not sure about the SA but I can check that when it next happens.

Config is as follows

/ip ipsec policy group
add name=unifi
/ip ipsec profile
add dh-group=modp2048 dpd-interval=10s dpd-maximum-failures=5 enc-algorithm=
aes-256 hash-algorithm=sha256 lifetime=8h name=unifi-profile
/ip ipsec peer
add exchange-mode=ike2 local-address=x.x.x.x name=peer1 passive=yes
profile=unifi-profile send-initial-contact=no
/ip ipsec proposal
add auth-algorithms=sha256 enc-algorithms=aes-256-cbc,aes-256-ctr,aes-256-gcm
lifetime=1h name=unifi-proposal pfs-group=modp2048
/ip ipsec identity
add peer=peer1 policy-template-group=unifi
/ip ipsec policy
add dst-address=x.x.x.x/24 peer=peer1 proposal=unifi-proposal
src-address=x.x.x.x/24 tunnel=yes

Thanks!

That info was actually pretty useful!

The peer staying active is actually a good clue. With DPD at 10s it would drop the peer if IKE traffic stopped getting through, so IKE looks fine and it's the ESP side (phase 2) that dies. That also makes my Starlink NAT theory less likely, so we can probably forget about that one.

With a 1h phase 2 lifetime, 5-6 hours lines up with roughly the 5th or 6th child SA rekey. Since you've got PFS enabled, every rekey does a fresh key exchange, and 7.24 (changelog) changed quite a bit in exactly that area:

*) ipsec,ike2 - fixed ppk child key generation during rekey;
*) ipsec,ike2 - improved KE generation validation during initial setup and child SA creation;
*) ipsec,ike2 - use first child KE selection only during IKE_AUTH exchange;

Worth noting too: with passive=yes the MikroTik never initiates anything on its own, so if a rekey fails it just sits there waiting for the Unifi to do something. That would explain why pausing and resuming on the Unifi side sorts it out.

A few other things worth checking too:

  • Do the lifetimes and PFS group on the Unifi side match (1h/8h and modp2048)?
  • Next time it stops, run "/ip ipsec installed-sa print" and see if the SAs are still there and when they were created.
  • Turn on ipsec logging (use !packet) so we can see what happens at the last rekey.

If it does turn out to be the rekey, a quick test would be to turn off PFS on both sides and see if the problem goes away. If it does, it's probably worth reporting to MikroTik support.