Community discussions

MikroTik App
 
atakacs
Member Candidate
Member Candidate
Topic Author
Posts: 121
Joined: Mon Mar 07, 2016 5:39 pm

Site to Site IPsec - muti subnet routing & capturing

Mon Jul 26, 2021 1:05 pm

Hello

I need to setup a site to site VPN where the Mikrotik router holds multiple subnets. The remote site is managed via a Ubiquiti USG.

Image

Overall I have this setup on the mikrotik

Image

The tunnel is established but I have a hard time reaching all subnets from the remote site. More worrying / strange it seems to be relatively random - ie I can reach a given subnet at some point and not latter…

Also seeing this kind of “drops” in the connection (pinging from 172.16.101.0/24).

C:\Users\admin>ping 192.168.0.254 -t
Pinging 192.168.0.254 with 32 bytes of data:
Request timed out.
Request timed out.
Reply from 192.168.0.254: bytes=32 time=9ms TTL=63
Reply from 192.168.0.254: bytes=32 time=9ms TTL=63
Reply from 192.168.0.254: bytes=32 time=9ms TTL=63
Reply from 192.168.0.254: bytes=32 time=9ms TTL=63
Reply from 192.168.0.254: bytes=32 time=9ms TTL=63
Reply from 192.168.0.254: bytes=32 time=9ms TTL=63
Request timed out.
Reply from 192.168.0.254: bytes=32 time=9ms TTL=63
Reply from 192.168.0.254: bytes=32 time=9ms TTL=63
Clearly something is wrong…

On the onset do you see something fundamentaly wrong with my setup ? Did anyone have an experience in such a setup ?

Next, how I can on the mikrotik capture packet egressing to the remote site, respectively coming in on the VPN link ?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Site to Site IPsec - muti subnet routing & capturing

Mon Jul 26, 2021 2:25 pm

Payload packets being sent via an IPsec SA cannot be captured using /tool sniffer, whereas received payload packets decapsulated from IPsec SA transport packets can. They used to appear in the capture earlier than the transport packets that brought them, which was rather confusing; I don't know whether it's still the case.

You can use action=sniff-tzsp rules in mangle to overcome this, but there used to be bugs that needed workaround settings in Wireshark to decode the sniffed packets correctly.

As you've got public addresses at both peers, and thus ESP is used as a transport protocol and no keepalives are sent every 20 seconds, you can just watch the transport packets - use ping count=1 while sniffing on protocol=ipsec-esp to see which end causes the trouble. Ideally you would sniff at both ends to see whether there is a packet loss, but I'm not sure what are the sniffing capabilities of Ubiquiti.

Nothing seems wrong in the policies you've shown, so I'd think of packet loss first.
 
atakacs
Member Candidate
Member Candidate
Topic Author
Posts: 121
Joined: Mon Mar 07, 2016 5:39 pm

Re: Site to Site IPsec - muti subnet routing & capturing

Mon Jul 26, 2021 8:37 pm

Thanks

I'm afraid this is getting somewhat above my "paygrade" (or more to the point my level of competency).
  1. I dont think there is any issue with the WAN link between the two sites - both are on a Gb symmetric fiber with high level SLA. Ping is stable between both sites and absolutely "clean"
  2. I have started to look into the sniffer tool and captured some traffic (limiting it to IPsec protocol) but I am not sure there is anything useful here
    154  28.259 et b.b.b.b                              a.a.a.a                              ips>
    155  28.305 et a.a.a.a                               b.b.b.b                             ips>
    156  28.523 et b.b.b.b                              a.a.a.a                              ips>
    157  28.543 et a.a.a.a                               b.b.b.b                             ips>
    158  28.712 et b.b.b.b                              a.a.a.a                              ips>
    probably not capturing the right "material".
  3. There is a TCPDUMP tool on the Ubiquiti router so I guess I can capture on that side too but again note sure what i am looking for

All in all I think I will need some further guidance
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Site to Site IPsec - muti subnet routing & capturing

Mon Jul 26, 2021 9:36 pm

What you are essentially looking for is the "culprit" of the issue. If everything works properly:
  • an icmp echo request packet is sent by the PC
  • it reaches router A whose firewall lets it in
  • the packet gets routed using the regular routing at router A via some interface
  • the header of this packet matches the traffic selector of the IPsec policy at router A
  • the policy diverts that packet into an SA, which encrypts it and encapsulates it into an ESP transport packet
  • the transport packet, sent from Router A's own address, is routed towards router B using regular routing
  • the packet is delivered to Router B
  • router B's firewall lets the ESP packet in
  • router B's IPsec stack finds the ESP packet to match an existing SA and the packet's sequence number successfully passes replay attack checks
  • the IPsec stack decapsulates and decrypts the payload packet and hands it over to the IP stack as if it was a newly received one
  • the IP stack determines it is a packet for the router itself or for some device behind it and lets the corresponding firewall chain (input or forward) process it
  • the packet is delivered to the destination
  • the destination host responds it with an icmp echo response packet
  • the response packet is handled the same way like the request, except that the roles of the routers are swapped
And the ultimate task is to find out which of the steps breaks in those cases where you don't get the response and fix that. For start, you need to use ping with count=1 and sniff for ipsec-esp packets. If everything works, you should see an r1.r1.r1.r1 -> r2.r2.r2.r2 ESP packet at r1 and then at r2, carrying the icmp echo request, and then an r2.r2.r2.r2->r1.r1.r1.r1 ESP packet at r2 and then at r1, carrying the icmp echo response.

You need to make sure that no other traffic than those test pings is trying to pass through the tunnel, as you wouldn't be able to distinguish the ESP packets carrying that other traffic from those carrying the ping requests and responses by any property except size.

The request may not leave r1, or it may not reach r2, or the response may not leave r2, or it may not reach r1, or it may reach r1 but not be shown as coming. This should tell you what went wrong. If the encrypted request doesn't even leave r1, or if the response arrives to r1 but doesn't reach the PC, the issue is at the r1 site; if the request leaves r1 but the response doesn't come, the issue is in the network between them or at the r2 site.

There's no magic, just the boring job of splitting the path into halves, quarters etc. until you isolate the element that drops the packet.

For starters, you can sniff at the Mikrotik alone, and see wh
 
User avatar
jprietove
Trainer
Trainer
Posts: 212
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: Site to Site IPsec - muti subnet routing & capturing

Tue Jul 27, 2021 12:57 am

(Edited) For debugging, maybe it can help if you change ESP to AH at IPsec policy.
If the packet lose persist, you can at least use Wireshark to look into the packet.

Who is online

Users browsing this forum: No registered users and 75 guests