After updating the production environment CCR1009-7G-1C-1S+ from ROS version 7.11.2 to ROS version 7.14.3, we encountered an issue where the GRE tunnel would intermittently go down for approximately 25 minutes, occurring 1-4 times within a span of 2 days. We conducted tests and replicated the issue in a controlled environment.
During the downtime of the GRE tunnel, the status of the IPsec peers and IPsec policies remained up. Upon enabling debug for IPsec, we observed that the GRE tunnel went down when the messages “ipsec delete ESP SA” and “ipsec IPsec-SA killing” appeared, and it was restored when a new IPsec-SA was initiated.
After troubleshooting, we set up a test scenario where Router 3 (R3) was connected to Router 1 (R1) and Router 2 (R2). R1 was running ROS version 6.49.15, while R2 was running ROS version 7.14.3. The GRE tunnel was observed to flap only when connected to R1. Interestingly, when we downgraded R3 to ROS version 7.11.2, the GRE tunnel stability improved, and the flapping ceased.
short logs:
08:15:17 ipsec,debug ===== sending 124 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:15:17 ipsec,debug ===== received 156 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:15:17 ipsec,debug ===== sending 140 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:15:17 ipsec,debug ===== received 140 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:17:17 ipsec,debug ===== received 140 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:17:17 ipsec,debug ===== sending 140 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:19:17 ipsec,debug ===== sending 124 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:19:17 ipsec,debug ===== received 156 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:21:17 ipsec,debug ===== sending 92 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:21:17 ipsec,debug ===== received 124 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:21:17 ipsec,debug ===== sending 92 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:21:17 ipsec,debug ===== received 140 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:21:25 ipsec,debug ===== received 572 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:21:25 ipsec checking: 10.0.4.249 ip-proto:47 <=> 10.0.20.34 ip-proto:47
08:21:25 ipsec,debug ===== sending 636 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:21:25 ipsec IPsec-SA established: 10.0.20.34[4500]->10.0.4.249[4500] spi=0x92161d3
08:21:29 ipsec IPsec-SA expired: ESP/Transport 10.0.20.34[500]->10.0.4.249[500] spi=0x1fec7f7
08:21:30 ipsec IPsec-SA established: 10.0.4.249[4500]->10.0.20.34[4500] spi=0x2f991c6
08:21:33 ipsec,debug ===== received 76 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:21:33 ipsec,debug ===== sending 108 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:21:33 ipsec processing payloads: DELETE
08:21:33 ipsec delete ESP SA
08:21:33 ipsec delete spi: 0xb42a442
08:21:33 ipsec IPsec-SA killing: 10.0.20.34[4500]->10.0.4.249[4500] spi=0x1fec7f7
08:21:33 ipsec IPsec-SA killing: 10.0.4.249[4500]->10.0.20.34[4500] spi=0xb42a442
08:21:34 ipsec sending dpd packet
08:23:33 ipsec,debug ===== sending 140 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:23:33 ipsec,debug ===== received 92 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:23:33 ipsec,debug ===== sending 156 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:23:33 ipsec,debug ===== received 92 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:25:33 ipsec,debug ===== sending 108 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
…
08:41:33 ipsec,debug ===== received 108 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:43:33 ipsec,debug ===== sending 140 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:43:33 ipsec,debug ===== received 92 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:43:33 ipsec,debug ===== sending 92 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:43:33 ipsec,debug ===== received 92 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:45:33 ipsec,debug ===== received 124 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:45:33 ipsec,debug ===== sending 124 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:45:40 ipsec,debug ===== received 588 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:45:40 ipsec checking: 10.0.4.249 ip-proto:47 <=> 10.0.20.34 ip-proto:47
08:45:41 ipsec,debug ===== sending 620 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:45:41 ipsec IPsec-SA established: 10.0.20.34[4500]->10.0.4.249[4500] spi=0xf111d8d
08:45:45 ipsec IPsec-SA expired: ESP/Transport 10.0.20.34[500]->10.0.4.249[500] spi=0x92161d3
08:45:46 ipsec IPsec-SA established: 10.0.4.249[4500]->10.0.20.34[4500] spi=0x27ba497
08:45:48 ipsec,debug ===== received 316 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:45:48 ipsec,debug ===== sending 124 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:45:48 ipsec processing payloads: DELETE
08:45:48 ipsec delete ESP SA
08:45:48 ipsec delete spi: 0x2f991c6
08:45:48 ipsec IPsec-SA killing: 10.0.20.34[4500]->10.0.4.249[4500] spi=0x92161d3
08:45:48 ipsec IPsec-SA killing: 10.0.4.249[4500]->10.0.20.34[4500] spi=0x2f991c6
08:45:50 interface,info gre-dts-r1-vbc1-gk link down <------------------------------------------------Incident
08:47:48 ipsec,debug ===== sending 140 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:47:48 ipsec,debug ===== received 156 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:49:48 ipsec,debug ===== sending 108 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:49:48 ipsec,debug ===== received 124 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
08:49:48 ipsec,debug ===== sending 124 bytes from 10.0.4.249[4500] to 10.0.20.34[4500]
08:49:48 ipsec,debug ===== received 156 bytes from 10.0.20.34[4500] to 10.0.4.249[4500]
09:09:58 interface,info gre-dts-r1-vbc1-gk link up
we’ve also tried decrease Ipsec hard Lifetime 30m → 3m, but flap frequency increases