Hello. I have two CCRs and I have configured an IPIP tunnel with IPsec in transport mode. It usually works, but there are times when IPsec stops being established.
When not set correctly, router A only transmits packets to router B. Router B receives and transmits packets to router A. ping between router A and B works correctly in both directions.
What can be?
What you mean by “stop being established”? Do you start the tunnels “once forever” but then they break, or are you shutting the tunnels down and then re-enabling them and sometimes they don’t come up the way you have described after you re-enable them?
As your sniffs show packets in both directions on one device (B) and only packets in one direction on the other device (A), it rather seems to me that the packets get lost somewhere on the way, so as an issue of some firewall or a routing issue somewhere between the Mikrotiks.
Do both devices have a public IP address directly on themselves, or one or both are behind a NAT? What is the network structure between them?
The IPIP tunnel is always established. The problem is IPsec in transport mode. Sometimes it does not establish the SA. As seen in the previous sniffer (and torch), the packets do not reach Router A.
If I disable the policies and the peers, the tunnel works correctly, but without IPsec. With IPsec enabled, after a while, it manages to establish the SA. That is why he said that sometimes it is not established.
Both routers have public IP, and they are not behind NAT.
The connections are: Office 1 - Router A - Internet - Router B - Office 2
While the sniffer does not receive the packets, if I ping between Router A and B, it is received in both directions.
Did you sniff simultaneously at both A and B? I mean, is it really so that sniffer running at WAN interface of B could see the IPsec transport packet towards A to be sent but the sniffer at WAN interface of A could not see it arrive? And if you ping from A to B at that time, the sniffer shows both ping requests and ping responses?
What languages do you support without Google Translate?
Yes, sniffer screenshots are simultaneous on both routers. Yes, on router B, I can see Rx and Tx packets, but on router A I only see UDP 4500 Tx packets (IPsec).
If I ping, in both routers snnifer I see Tx and Rx, and as I said, if I disable IPsec, the tunnel works correctly.
I do not understand where the UDP 4500 packets from B to A.
In Firewall - Connections of router A, I could see a connection:
Src: Router.A.IP: 4500
Dst: Router.B.IP: 4500
Reply Src: Router.B.IP: 4500
Reply Dst: Router.A.IP: 4500
Protocol: UDP
But they only increased the Tx counters. Rx was 0.
In Firewall - Connections of router B, I could see a connection:
Src: Router.A.IP: 4500
Dst: Router.B.IP: 4500
Reply Src: Router.B.IP: 4500
Reply Dst: Router.A.IP: 4500
Protocol: UDP
True in most cases (and most likely also here), but Mikrotik in particular seems to initiate IKEv2 connections towards 4500 straight away. I haven’t tested after a reboot, but after disabling and re-enabling a peer and where the NAT is not done on the same device where the peer is configured, the IKEv2 initiator request goes from 4500 to 4500 right away, no preceding exchange on port 500.
I have a lab with three clean CHR (simulate router A - ISP - router B) with very basic configuration (LAN/WAN masquerade and IPsec on router A/B. Only routing on “ISP router”)
If I configure IPsec IKEv2 between router A and B, they connects at port UDP 4500. When change configuration to “main echange mode” they connects at port UDP 500.
As there is PPPoE involved, the first thought would be that there is some issue with MTU, except that the sniff shows just 328 byte packets so this is out of question at least at this stage of the process.
As router B’s connection tracker shows both originating and reply packets while router A’s connection tracker shows only the originating ones, and as at router B also the sniffer shows both directions, my conclusion is that something on the path between the two devices doesn’t like the B->A IPsec packets and drops them, while it doesn’t have any problem with icmp and plain ipip packets.
The chain looks as follows:
peer A | connection tracker | pppoe-client interface | isp’s pppoe server | internet | isp’s pppoe server | pppoe-client interface | connection tracker | peer B
So as the interface at side B can see packets in both directions, and the sent ones have a source address matching the one of that interface, and already the interface at side A cannot see the incoming packets from B, something between the interfaces must be guilty. If the firewall on one of the Mikrotiks was dropping the packets, the B->A packets would have to be seen either in both sniffs (if firewall at A was dropping them) or in none of them (if firewall at B was dropping them).
Operators in some countries may block IPsec, but I would expect them to do that systematically, not just from time to time.
So we are getting to urban legend scenarios where poor driver or silicon implementations are believed to dislike packets with particular bit patterns in them. Given that some fields in the IKE initial requests are filled with random values generated for each new connection, this could be relevant. Another case could be poor calculation of Ethernet FCS or IP/UDP checksum for particular packet contents already in Mikrotik; to exclude this, I would recommend to sniff into file at the underlying interface of the PPPoE one at Mikrotik B and check using Wireshark whether the checksum fields are OK; unfortunately, this won’t show the Ethernet FCS, only the IP and UDP checksums.
