Community discussions

MUM Europe 2020
 
vladop
just joined
Topic Author
Posts: 9
Joined: Mon Sep 08, 2014 11:14 am

all IPsec tunnels stops after few days

Mon Sep 08, 2014 11:53 am

I have a router with 3 IPsec tunnels configured. Normally everything works as expected, but after few days all 3 tunnels stop at the same time. The only way I can correct the situation is to reboot the router. After the reboot all 3 tunnels are fully functional again.

I can't find any reason for this. My main questions is: how can I find what went wrong when it happens?

On the remote hosts there is nothing unusual in logs. Only entries about expired SAs and new SAs.

The endpoints can ping each other.

There are vaild phase1 (ISAKMP) associations and phase2 (SA) associations as well. They match on both sides. If I reset any of these on the router, new associations will be negotiated within seconds, but it does not help.

Remote osts are sending AH/ESP packets to the router, but no packets are coming from the router.

HW, SW is RB2011UiAS, 6.18. I have just upgraded to 6.19 and I wait for this error to occur again. This may take a week.
 
jarda
Forum Guru
Forum Guru
Posts: 7602
Joined: Mon Oct 22, 2012 4:46 pm

Re: all IPsec tunnels stops after few days

Mon Sep 08, 2014 3:06 pm

Check by netwatch if tunnel works. When it stops responding, remove SAs. It will link again.
 
User avatar
cdiedrich
Forum Veteran
Forum Veteran
Posts: 937
Joined: Thu Feb 13, 2014 2:03 pm
Location: Basel, Switzerland // Bremen, Germany
Contact:

Re: all IPsec tunnels stops after few days

Mon Sep 08, 2014 3:54 pm

Do both ends use the very same NTP server?
I had exactly this issue a year ago when my VPN RB was syncing with a different NTP server than the remote peer.
Bringing both on the same NTP server instantly solved the problem.
-Chris
Christopher Diedrich
MTCNA, MTCUME, MTCWE
Basel, Switzerland
Bremen, Germany

There are 10 types of people: Those who understand binary and those who don't.
There are two types of people: Those who can extrapolate from incomplete data
 
vladop
just joined
Topic Author
Posts: 9
Joined: Mon Sep 08, 2014 11:14 am

Re: all IPsec tunnels stops after few days

Mon Sep 08, 2014 4:25 pm

Check by netwatch if tunnel works. When it stops responding, remove SAs. It will link again.
As I wrote, if I remove SAs, new SAs will be installed, but the connections remain unusable.

It happens to all connections in the same time. I guess there is nothing wrong with individual SAs, individual peers, and policies. Something seems to affect the whole IPsec function.

If it happens again, I will try to check if packets entering the router from LAN are leaving the router encapsulated on WAN. I have no ideas what else to check.

Do both ends use the very same NTP server?
I had exactly this issue a year ago when my VPN RB was syncing with a different NTP server than the remote peer.
Bringing both on the same NTP server instantly solved the problem.
Very strange. Is there an explanation for this? IMHO, IPsec should work even if they were not sychronized at all.

To answer your question: both ends use NTP, but not the same server. Remote hosts use a random server from a NTP pool for Europe. The router gets its time from the central server on the LAN which in turn is synchronized using the mentioned pool.
 
jarda
Forum Guru
Forum Guru
Posts: 7602
Joined: Mon Oct 22, 2012 4:46 pm

Re: all IPsec tunnels stops after few days

Mon Sep 08, 2014 4:49 pm

When removing sas it has to be done on both sides simultaneously. The synchronised time is really necessary.
 
vladop
just joined
Topic Author
Posts: 9
Joined: Mon Sep 08, 2014 11:14 am

Re: all IPsec tunnels stops after few days

Mon Sep 08, 2014 5:41 pm

When removing sas it has to be done on both sides simultaneously.
This is one of the things I have tried. It did not help.

OTOH, If I reboot only the router, the problem is gone. No action on the remote side necessary.

That's why I must disagree in this particular case.
The synchronised time is really necessary.
All hosts have exact NTP time from public stratum 2 or stratum 3 servers.
 
jarda
Forum Guru
Forum Guru
Posts: 7602
Joined: Mon Oct 22, 2012 4:46 pm

Re: all IPsec tunnels stops after few days

Mon Sep 08, 2014 7:16 pm

I just wrote what helped to me. Before I stopped to use ipsec I used such scripts that worked. Maybe your problem is different...
 
vladop
just joined
Topic Author
Posts: 9
Joined: Mon Sep 08, 2014 11:14 am

Re: all IPsec tunnels stops after few days

Tue Sep 16, 2014 12:06 am

It took 8 days this time.

I tried it again and I have to repeat myself that none of these helps:
 /ip ipsec remote-peers kill-connections
 /ip ipsec installed-sa flush sa-type=all
Even if combined with a similar action on the remote site, it does not help. All that happens is that new phase1 and phase2 SAs appear quickly as they should.

I tried a ping from a remote site over the IPsec tunnel. The packet sniffer shows that:
- AH/ESP packets do arrive from WAN,
- decapsulated ping requests are sent to LAN,
- ping replies are coming back
,
- but routerboard does not send any encapsulated AH/ESP packets.

I checked CPU usage and free memory. No problems there.

I have no idea what else to check or to reset. On a Linux host I would check the current state of SPD (Security Policy Database) in the kernel with setkey -DP.

After 45 minutes I rebooted the router to restore normal operation.
 
vladop
just joined
Topic Author
Posts: 9
Joined: Mon Sep 08, 2014 11:14 am

Re: all IPsec tunnels stops after few days

Mon Sep 22, 2014 11:23 am

IPsec down again. I think I will write an automatic reboot script.

It looks to me like a RouterOS issue, but I got no "me too" replies. If there is anybody reading this and having multiple IPsec tunnels configured, I would appreciate a short note.
 
phase42
just joined
Posts: 1
Joined: Wed Sep 10, 2014 5:49 am

Re: all IPsec tunnels stops after few days

Wed Oct 01, 2014 12:14 am

Hi vladop,
We are having an annoying issue like this too.
Slightly different to you, we have an IPsec between a pfsense(SiteA) and a Mikrotik(SiteB) and another IPsec between pfsense(SiteA) and pfSense(SiteC).
It works fine for hours, and then randomly stops. Usually when the Mikrotik(SiteB end aDSL drops for a minute or so, but not always.
When it happens both ends complain about time outs waiting for phase1.

If I reboot the Mikrotik(siteB) the IPsec comes back fine.
Nothing ever seems to go wrong with the SiteA <--> SiteC IPsec

The odd thing is that we replaced the SiteB pfsense with the Mikrotik (because we thought it a problem with SiteB) and it's not made much difference at all.

Reading below, I wonder if it is is some sort of timing problem......
The SiteA logs have(note I have edited them and changed the IPs and spi values etc):
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: [218.214.147.172] INFO: DPD: remote (ISAKMP-SA spi=0d35yyyyye0b9d5b0:1cb0yyyy46cb1) seems to be dead.
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: INFO: purging ISAKMP-SA spi=0d35a5byyyy5b0:1cb039yyyy46cb1.
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: INFO: purged IPsec-SA spi=510556195.
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: INFO: purged IPsec-SA spi=1307544530.
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: INFO: purged IPsec-SA spi=2888775497.
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: INFO: purged IPsec-SA spi=147348029.
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: INFO: purged ISAKMP-SA spi=0d35a5byyyy5b0:1cb039yyyy46cb1.
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: INFO: ISAKMP-SA deleted yy.yy.yy.yy[500]-xx.xx.xx.xx[500] spi:0d35a5b2yyyyy5b0:1cb0yyyyy346cb1
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: INFO: IPsec-SA request for xx.xx.xx.xx queued due to no phase1 found.
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: INFO: initiate new phase 1 negotiation: yy.yy.yy.yy[500]<=>xx.xx.xx.xx[500]
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: INFO: begin Identity Protection mode.

Any ideas?

Rgds Ben
 
vladop
just joined
Topic Author
Posts: 9
Joined: Mon Sep 08, 2014 11:14 am

Re: all IPsec tunnels stops after few days

Thu Oct 02, 2014 9:54 am

The SiteA logs have(note I have edited them and changed the IPs and spi values etc):
2014.09.30-12:46:26 <yy.yy.yy.yy>: <30>Sep 30 12:46:26 racoon: [218.214.147.172] INFO: DPD: remote (ISAKMP-SA spi=0d35yyyyye0b9d5b0:1cb0yyyy46cb1) seems to be dead.
.....
Thank you for your note. I do not see anything unusual in the logs you have posted. I have disabled DPD, but our racoon logs look normal too.

Did you tried anything else except rebooting the mikrotik router?

We have two Internet connections. I have reconfigured one of the IPsec tunnels to other provider's link. I will wait what happens. If the problem is triggerred by a link outage, I will get 2 IPsec tunnels unusable instead of all 3.

For the record: during the last problem occurence, I had two simultaneous session between same hosts opened. One becomes dead and the other was fine for a while (less than a minute). Then it froze too.
 
sanitycheck
newbie
Posts: 47
Joined: Wed Nov 16, 2011 6:03 am
Location: USA

Re: all IPsec tunnels stops after few days

Thu Oct 02, 2014 8:26 pm

I had similar problems but just with SIP (or certain UDP connections) over IPSEC. It was triggered by a port flap on an Internet modem or other Internet outage. The problem seems to have been addressed in 6.18/6.19.

However, before the firmware updates were released I noticed that fixing the router port speed of the modem dramatically reduced the problem (e.g. if the port shows the modem auto-negotiates at 100M-full, turn off auto-negotiation and fix the router port speed to 100M-full). I never turned the port speeds back to auto-negotiate after the firmware update, so it's possible the combination of firmware update to 6.18/6.19 and fixing the port speed were required to fix the problem.
 
vladop
just joined
Topic Author
Posts: 9
Joined: Mon Sep 08, 2014 11:14 am

Re: all IPsec tunnels stops after few days

Thu Oct 16, 2014 12:11 am

We have two Internet connections. I have reconfigured one of the IPsec tunnels to other provider's link. I will wait what happens. If the problem is triggerred by a link outage, I will get 2 IPsec tunnels unusable instead of all 3.
It did break down again few days ago. All IPsec's were down no matter what interfaces were used for the communication.

I'm happy I don't have to explain it to a customer. :evil:
 
vladop
just joined
Topic Author
Posts: 9
Joined: Mon Sep 08, 2014 11:14 am

Re: all IPsec tunnels stops after few days

Fri Jan 09, 2015 12:49 pm

Small update: We did some upgrades. The router is running 6.23 and one of the ipsec endpoints is now using libreswan instead of racoon.

Unfortunately it did not help. I had to reboot this morning. The uptime was 2w4d but there was no significant traffic during Xmas and new year holliday time.
 
zolti
just joined
Posts: 6
Joined: Thu Jun 05, 2014 4:34 pm

Re: all IPsec tunnels stops after few days

Wed Mar 04, 2015 2:01 pm

Hi. Have this error on 6.27. Anyone have solution?
 
vladop
just joined
Topic Author
Posts: 9
Joined: Mon Sep 08, 2014 11:14 am

Re: all IPsec tunnels stops after few days

Tue Mar 24, 2015 4:52 pm

I started this topic.

I hope I'm not wrong, but it seems that the problem is gone. I have 29 days uptime without ipsec problems.

I was upgrading often and some of the fixes in the RouterOS did it.
 
martr84
just joined
Posts: 19
Joined: Sun Feb 12, 2012 1:17 am

Re: all IPsec tunnels stops after few days

Wed May 11, 2016 1:42 pm

sorry to resurrect an old thread but I'm seeing what seems to be this issue on 6.35.2 on a CCR1016-12G.

i'm running l2tp with ipsec between the 2 local networks at each site. the l2tp tunnels stay up and the ipsec tunnel seems to establish with no errors but if you look at current bytes under installed SAs there is no traffic in one direction. as soon as i reboot the router traffic starts to flow in both directions.

Any help would be appreciated?

Thanks
 
mattstephenson
newbie
Posts: 41
Joined: Wed Feb 01, 2017 1:03 am
Location: UK

Re: all IPsec tunnels stops after few days

Wed Mar 29, 2017 12:36 pm

I too have this problem, if the internet connection (PPPoE in my case) goes down and up quickly, it stalls on PH2 state "no phase 2", rebooting the router again, or removing the 'remote peers' re-establishes the links.

We are experiencing this in 5 different site to site connections.

All are RB3011 running either 6.38.1 and 6.38.5.

Who is online

Users browsing this forum: No registered users and 107 guests