Hello everyone,
The situation is the following (If you need the network diagram I will provide you one).
HQ: two WANs
BO: two or more WANs.
In order to provide VPN connection with failover, for every WAN connection in the BO there are two GRE tunnels with the HQ. In this specific case, we have 3 WANs in the BO, so 6 GRE tunnels. Every tunnel has a static route with a different distance.
Everything works fine, until the PPPoE interface of the WAN in the BO flaps. Since this moment, the GRE tunnel is down. I am not able even to ping the remote address used to make the GRE tunnel, forcing the correct source address.
Gre Interfaces - BO side
Tunnel configuration - BO side
Gre interfaces on HQ (interfaces names are the same on both sides)
Same Tunnel configuration - HQ side
I started the tests from the IPSec phase one: the state is established, but the RX value is 0. Phase 2 is established too. IPSec is configured with tunnel mode.
If I try to ping the remote address with a src address, like is defined in the Phase2 policy, the ping doesn’t work. This makes me think about an issue with IPSec, but the next steps make me even more confused.
Ping from BO → HQ on the loopback interfaces
In the screenshot above, in the connection tracking, a GRE connection is shown. The timeout on the GRE connection is 3 minutes.
Now if I keep the GRE interfaces disabled for three minutes on both sites, the timeout expires. Since this moment, I am able to ping the loopback interfaces on the other side.
So, at this moment, I bring up the GRE Interfaces on both sides, and everything start working well again.
The manual troubleshooting works everytime, but I would like to understand what causes this issue. I have a big number of GRE Tunnel and every morning I have to check if this problem is active and fix it.
I am very confused. In the first moment I thought it could be a problem of IPSec when the PPPoE interface flaps. Now I would say that in the flap moment something happens in the routing table, and in some way the GRE keepalive packets go through another route and remain active for some reason.
I would really appreciate to resolve this problem, I can provide any type of information or test if needed.
Thank you
Lorenzo
Would it be too painful for you to change the tunnels from GRE to IPIP?
The thing is that at least since a fix of some GRE-related vulnerability somewhere in 6.45.x, the issue you describe exists, plus only on some CPU architectures to make it even more entertaining. I’ve migrated all my affected tunnels to IPIP (enable ipencap, not ipip in the firewall rules!) and it’s been working fine ever since. As a bonus point, you’ll win a few bytes of MTU - the GRE headers are larger than the IPIP ones, but Mikrotik cannot make use of them, so you cannot set up multiple GRE tunnels between two peers, differentiated by Tunnel-ID.
I was never able to collect enough evidence to open a support case with Mikrotik because it could never reproduce it on my set of lab machines, and I’m not going to send supout.rif from production machines with PSK authentication on IPsec even though Mikrotik states the PSKs are not saved into the supout.
Hello Sindy!
Thanks for the answer.
Well, at least I know that If I migrate from Gre to IPIP I should resolve this problem! For sure it’s not easy, because we are speaking of more than 200 tunnels… But I can try on one site and see…
But anyway it should be interesting resolve this issue with GRE.
I wasn’t able to reproduce it in lab too..(which made me think I could have an issue with some firewall rules on the HQ?), but I have a production environment where I am able to reproduce it every time I want.
It could be interesting if the support could connect with a remote session and see what is going on…
Meanwhile I opened a case with Mikrotik and I sent this thread… let’s see what happen
I am using such a config and I do not see any issues. Watch out for:
firewall errors (as sindy mentioned, there is a bug in RouterOS for the past couple of versions. incoming GRE traffic is marked “invalid” instead of “established” or “new”, when you drop invalid traffic before accepting GRE it fails
NAT issues in other routers. when IPsec is sent via NAT and the session is interrupted, it can get into an unrecoverable state. This happens when your router is behind some other product like AVM Fritzbox. It often helps to “forward” the traffic, in this case UDP port 500 and 4500 to the router.
@pe1chl, unfortunately the PITA the OP has described exists in addition to the two you’ve mentioned. I’ve done all my homework to work these around (exemption of GRE from “drop invalid”, measures to make sure that IPsec recovers from an interruption/restart of a mid-path router properly, filter rule allowing forwarding of GRE packets emerging from GRE tunnel (you favourite keepalive theme) - this one is only necessary with paranoid firewalls) and still there’s a problem sometimes, where after an outage or after initial configuration of the GRE tunnel, without touching anything else, you have to just disable the GRE for 10 minutes at both ends and then re-enable it to make it work again. If you don’t, it stays broken forever, so not related even to SA rekeying.
I recognize that only when there is another NAT router inbetween. The NAT router has translated the original session to a different address but the same port numbers (500 and 4500), but after the outage the NAT router thinks there is a new connection and has not yet deleted the old one, and decides to translate the port number because it sees a session to the same IP/port (500). So the NAT rule translates to some random port number and the two sides never meet again.
Disabling for 10 minutes deletes this entry from the NAT table and sessions are possible again.
When I do a “port forwarding” in the NAT router this usually does not happen because the router knows to translate only the address and not the port number.
On routers that are directly on internet without extra NAT router inbetween I never see this problem, not even when one of them uses PPPoE.
I do see another issue with PPPoE sometimes: when there is a reset in the backbone network and the PPPoE is not cleanly closed and reopened, it sometimes occurs that the PPPoE interface does not get an IPv4 address. It does get the IPv6 address. As I make 2 tunnels, one GRE and one GRE6, between each location, I then see that the GRE6 works normally and the GRE does not.
I have a script to detect this problem and disable/enable the PPPoE interface, which recovers both tunnels without requiring further action on the tunnel interfaces.
I tried on one site but I have the same issue! Could it mean that I have some issue on the HQ firewall?
I have a CCR1036 in the HQ and a RB3011 in the BO…
I read your nice considerations about the thread and what I can say is that I always have a public IP over the PPPoE interfaces
I also have other environments with other Mikrotik models and there the issue never happens.
The only difference is that in the other enviroments the GRE-IPIP tunnel is not the default gateway in the BO: is it possible that cause the issue?
But still, I am not able to reproduce the issue in the lab environment.
About what @pe1chl said, this is quite interesting but I will focus on one thing:
IPsec is in tunnel mode, with phase2 established.
The phase two policy is:
src : x.x.x.x
dst: y.y.y.y
Where these two are the remote and local address used for initializate the GRE interface.
I leave a CLI session opened with a ping from x.x.x.x to y.y.y.y
CASE 1: GRE tunnel is running and enabled
I voluntary disable and enable the PPPoE interface
The ping stops working and it won’t work anymore until I disable the two GRE interfaces (one in BO and one in HQ)
CASE 2: GRE tunnel is disabled
I voluntary disable and enable the PPPoE interface
The ping stops working but when the PPPoE comes up the ping works correctly.
What is seems to me, is that the GRE invalid session break the IPSec somehow..
But still, I should say it’s maybe a firewall issue on the HQ?
Hopefully one of these is the reason, otherwise it would mean that the IPIP handling would be flawed too.
So please post the anonymized configuration of both devices (the BO one currently running IPIP and the HQ one), see my automatic signature below for a hint.
When it is the same issue as what I see, it is not a GRE or other tunnel issue but an IPsec issue.
Any stateful firewall (including the NAT example I gave but also the firewall you have) can cause IPsec issues that go away when the session is silenced for a while.
I even wrote a script for this I run on a router that sometimes has those issues:
/system script
add dont-require-permissions=no name=ipsecerrorhandler owner=admin policy=\
ftp,read,write,policy,test source="# scan log buffer for ipsec error messa\
ges\r\
\n# when error is \"phase1 negotiation failed due to time up\", temporaril\
y block the sender\r\
\n# this is done to work around problems with some NAT routers\r\
\n\r\
\n:global lastTime;\r\
\n\r\
\n:local currentBuf [ :toarray [ /log find topics=ipsec,error and message~\
\"phase1 negotiation failed due to\" ] ] ;\r\
\n:local currentLineCount [ :len \$currentBuf ] ;\r\
\n\r\
\n:if (\$currentLineCount > 0) do={\r\
\n :local currentTime [/log get [ :pick \$currentBuf (\$currentLineCoun\
t -1) ] time ] ;\r\
\n if (\$currentTime != \$lastTime) do={\r\
\n :set lastTime \$currentTime ;\r\
\n :local currentMessage [/log get [ :pick \$currentBuf (\$currentL\
ineCount -1) ] message ] ;\r\
\n :if (\$currentMessage~\"due to time up\") do={\r\
\n :local ipaddress [:pick \$currentMessage ([:find \$currentMe\
ssage \"<=>\" ]+3) 99 ] ;\r\
\n :set ipaddress [:pick \$ipaddress 0 [:find \$ipaddress \"[\"\
\_] ];\r\
\n :local activepeers [ /ip ipsec active-peers find where remot\
e-address=\$ipaddress and state=established ] ;\r\
\n :if ( [ :len \$activepeers ] = 0 ) do={\r\
\n :log info \"Temporarily blocking \$ipaddress due to erro\
rs\" ;\r\
\n /ip firewall address-list add list=blocked address=\$ipa\
ddress timeout=\"00:05:00\" ;\r\
\n }\r\
\n }\r\
\n }\r\
\n}"
The script is scheduled to run every 2 minutes, and when it finds a remote that is
having problems establishing the connection it puts its address in a list and in the
firewall you should block the packets from source address in that list. After 5
minutes (you can use 10 minutes when required) it expires and the connection is
established again.
I really appreciate the help but in this moment the HQ configuration is a disaster, we are fixing it but I think we need some weeks before having a precise configuration. If the issue will happen after the configuration will be corrected, I will send you the configuration
Anyway, I have other news:
@pe1chl
I made other tests with some colleagues and probably you got the point.
What we made is establishing a new IPIP-TUNNEL from the same BO to another Mikrotik (let’s call it HQ-TEST), which is not the HQ.
So, we should exclude any problem due to the default route in the BO (so the IPIP tunnel itself towards HQ)
The issue still happens.
So I kept the new IPIP-TUNNEL with HQ-TEST disabled, I disabled and enabled the PPPoE interfaces, and the phase2 doesn’t work.
For the BO and HQ-TEST mikrotiks the state of the policy is “established”, but a ping from the source to the destination doesn’t work.
My colleague tried to decrease the DPD timeout on phase1 (something very strict: like 1-1), and in this way, the phase2 always comes back up after the PPPoE flap.
So I am starting to think it’s something more related to IPsec… but still no idea…
Which DPD configuration would you suggest?
EDIT
Seems that if there is no traffic between BO and HQ-TEST, when the PPPoE flaps the IPsec phase 2 works
If there is traffic between BO and HQ-TEST, like IPIP keep alive or just a ping, when the PPPoE flaps, the IPSEC phase 2 remains established but stop working.
As @pe1chl wrote, there may be firewall/NAT issues associated with the PPPoE flap. If there is NAT somewhere between the peers, both IKE (or IKEv2) and the transport packets use the same UDP stream, and either Mikrotik’s own NATs or those on the ISP’s devices may behave in an unexpected way when the address and port at one end changes whilst the other end keeps remembering the old ones.
With multiple WANs, there’s one more complication - I had cases where the SAs chose a wrong peer locally after restart or path glitch, so the remote peer was ignoring the transport packets as they came in via a wrong SA.
To suggest some analysis steps, the anonymized configuration is necessary. Maybe start from posting the one from the BO side running the IPIP tunnel (which is not a “disaster” like the HQ one), and stating whether the WANs get the same IP each time the PPPoE connects and whether that IP is a public one or not.
Hello Sindy,
in this moment I have two test WANs in the datacenter, so I took a RB2011 for tests, I put a simple configuration, and I simulated that it’s the TEST-HQ.
So I connected to the production BO with a single IPSEC tunnel.
If I disable the PPPoE on TEST-HQ site, without any tunnel (IPIP / GRE or what else) configured, the flap doesn’t make any problem on the IPsec Phase2 status.
If I configure a IPIP or even a GRE tunnel between the TEST-HQ and the BO, if I disable and enable the PPPoE on HQ site, the IPsec phase 2 stop working.
I attached the configuration of the TEST-HQ, which is pretty clear and easy, compared to the production HQ.
Could you please take a look and tell me if there is something wrong?
I am working on the BO configuration but I won’t be ready until the next days. HQ-TEST.rsc (6.12 KB)
Meanwhile I realized a test environment with one RB2011 and one RB3011 and a internet connectivity for each routerboard, provided by the same provider.
One WAN connection on the RB2011, made with a PPPoE interface.
One WAN connection on the RB3011, made with a PPPoE interface.
If I disable and re-enable the PPPoE interface on the RB2011, the IPSec works properly.
If I disable and re-enable the PPPoE interface on the RB3011, the IPSec stops and I need to reset it manually.
I attached the two configurations. It’s a very simple configuration, could you please take a look?
The only difference between the two setup is:
on the RB2011 the internet connection is made with a wireless link, on the RB3011 the internet connection is made with a FTTC Modem in bridge mode. But for the routerboards there should not be any difference, it should be transparent… HQ-2011.rsc (4.86 KB) BO.rsc (4.87 KB)
Not helpful but you’re not alone having IPSec issues with PPPoE flaps
I’m seeing IPSec break on a RB1100AH4 every time PPPoE flaps and the installed SA’s clear following this ipsec is broken until reboot, or if you hold the PPPoE down for like 3~5 min, I’ve a case with Tik Support going on about it. Might be worth reaching out to support also,
If I make any progress I’ll share here
Cheers
Mark
I’ve mixed success with going through and disabling all elements of the ipsec config peer, policy etc and leaving them disabled for a while but this doesn’t always work.
The only sure fire ways that work for me are holding PPPoE down for 3-5 min once the issue presents or just rebooting the device.
This is a real pain and I hope they can get it fixed soon, It’s really strange for me it’s only impacting certain devices I’m seeing it on a RB1100AH4 but i’ve a pair of these side by side and only one is having the issue even after a full factory reset and only basic config applied
Are these devices all connected to internet the same way? I.e. is it always a plain PPPoE connection to an ISP that offers a real and static external IP, not using cg-nat?
I ask that because the behavior may very well depend on NAT occurring external to the MikroTik routers, or depend on the IP changing after the PPPoE interface flap.
That would explain why certain people see it and others don’t, or why you see it on some router and not on another.
