I have 2 Routers, Router A, Router B
Router A has 4 WAN Connections with Routing Marks for different Traffic
Router B has 2 WAN Connections with Routing Marks.
This works as expected and i can send different traffic over the different ISP’s depending on what is required.
I have 2 IPIP Tunnels between Router A and Router B
I have the Routing Mark sending the traffic out of the correct ISP for the IPIP tunnel.
I have IPsec Enabled on the Tunnels, and fastpath disabled.
I have a /30 assigned to each tunnel, lets say
Tunnel A:
Router A: 10.10.10.1/30
Router B: 10.10.10.2/30
Tunnel B:
Router A: 10.10.10.5/30
Router B: 10.10.10.6/30
I can see the IPsec comes up and goes established and the tunnel works, from Router A, i can ping 10.10.10.2 with source interface of the Tunnel A interface.
A few days later, i get a alert, to say this tunnel is down.
I log into Router A, and I notice the IPSec is still established. I ping 10.10.10.2, i get a timeout.
I flush the Ipsec, and it re-establishes fine, right away, but still can’t ping 10.10.10.2
I did various troubleshooting, including checking the Ipsec Secret, changing the Subnet to a different /30, flapping connections on both sides etc, i get no response over the tunnel.
I also removed the IPsec Encryption off the Tunnel to see if it will work without any encryption, still nothing.
Out of desperation, I created a new GRE tunnel, with the exact same parameters.(source ip, destination ip, random key) Tunnel comes up.
I moved the 10.10.10.0/30 Ip’s over from the IPIP tunnel to the GRE tunnel, and it worked fine right away.
I left it like that for a few days, as I had other things to attend to, and a few days later, i get a alert again, same thing, tunnel is down. Same Symptoms.
This time i try to change it back to IPIP (which was still sitting there disabled), no good.
I rebooted Router B, no good, IPsec comes up as established, but no traffic goes over the tunnel.
I then rebooted Router A, and as soon as it came up, both the IPIP tunnel and the GRE tunnel that wasn’t working is working again.
While it was ‘Not Working’ I torched the IPIP interface, and i could see on the source side the ICMP traffic, but on the dst side, there is no traffic on the interface at all.
When it works, it comes up right away, it is stable, every time you disable and re-enable it comes up as expected etc.
The fact that a reboot fixed it tells me there is something else going on here that likely isn’t configuration related. What will be the best steps to take next time this happens so that I can get to the bottom of what is going on here?
Thanks