Strange routing behaviour

RouterOS General
1

I have a VPN client with LAN address 192.168.19.254/24 and VPN server with remote LAN address 192.168.14.254/24.

Let’s suppose that the VPN connection is established, but the server does not respond to ICMP requests.

Here is the actual (active) ipsec policy:

[admin@client] /ip ipsec policy> print
Flags: T - template, B - backup, X - disabled, D - dynamic, I - invalid, A - active, * - default
 #      PEER TUN SRC-ADDRESS                                    DST-ADDRESS                                    PROTOCOL   ACTION
 0 T  *          0.0.0.0/0                                      0.0.0.0/0                                      all
 1   A  la.. yes 192.168.19.0/24                                192.168.14.0/24                                all        encrypt

The connection is stable:

[admin@client] /ip ipsec active-peers> print
Flags: R - responder, N - natt-peer
 #    ID                   STATE              UPTIME          PH2-TOTAL REMOTE-ADDRESS                           DYNAMIC-ADDRESS
 0    not.telling.com         established        32m57s                  1 89.134.111.123

When I try to do traceroute from client to server then I see this most of the time:

 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1                                  100%    4 timeout
 2                                  100%    4 timeout
 3                                  100%    4 timeout
 4                                  100%    4 timeout
 5                                  100%    4 timeout

But sometimes this happens:

[admin@client] /ip route> /tool traceroute 192.168.14.254
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 89.134.243.1                    56..   16 timeout     0.1       0     0.1       0 host unreachable from 89.134.243.1
 2 89.134.243.1                    53..   16     0ms       0       0       0       0 host unreachable from 89.134.243.1
 3 89.134.243.1                    13..   15     0ms    77.1       0   986.2   262.4 host unreachable from 89.134.243.1
 4                                    0%    0     0ms

E.g. sometimes I get “host unreachable from ” instead of a plain timeout.

The routing table on the client was created with:

/ip route
add comment="Prevent package leak RFC1918 class A" distance=1 dst-address=10.0.0.0/8 type=unreachable
add comment="Prevent package leak RFC1918 class B" distance=1 dst-address=172.16.0.0/12 type=unreachable
add comment="Prevent package leak RFC1918 class C" distance=1 dst-address=192.168.0.0/16 type=unreachable
add comment="VPN to server" distance=1 dst-address=192.168.14.0/24 gateway=ipsec pref-src=192.168.19.254

The actual routing table is more difficult:

 /ip route> print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          89.134.255.254            1
 1 A SU ;;; Prevent package leak RFC1918 class A
        10.0.0.0/8                                                   1
 2 ADC  10.19.10.0/24      10.19.10.1      BLUE_VLAN                 0
 3 ADC  10.19.20.0/24      10.19.20.1      GREEN_VLAN                0
 4 ADC  10.19.30.0/24      10.19.30.1      RED_VLAN                  0
 5 ADC  89.134.240.0/20    89.134.243.1   ether5-wan                0
 6 A SU ;;; Prevent package leak RFC1918 class B
        172.16.0.0/12                                                1
 7 A SU ;;; Prevent package leak RFC1918 class C
        192.168.0.0/16                                               1
 8 A S  ;;; VPN to server
        192.168.14.0/24    192.168.19.254  ipsec                     1
 9 ADC  192.168.19.0/24    192.168.19.254  BASE_VLAN                 0

The “ipsec” bridge was specifically created to allow routing of to-be-esp-encapsulated packages:

/interface bridge
add name=ipsec protocol-mode=none

The most specific route is the one that goes to the “ipsec” bridge:

 8 A S  ;;; VPN to server
        192.168.14.0/24    192.168.19.254  ipsec                     1

If that fails, then the next most specific should be the unreachable static route - that was specifically created to prevent leaking such packages:

 7 A SU ;;; Prevent package leak RFC1918 class C
        192.168.0.0/16                                               1

This route should always catch packages and prevent them from leaking out on the default gateway.

As far as I understand packet flow, the packets with 192.168.14.0/24 destination should leave the routing process towards the ipsec bridge, and the ESP encapsulation should happen after that.

So how does my router responds with “host unreachable” messages? And why it is undeterministic? (E.g. 90% of the time I get a timeout, and only in 10% I get a “host unreachable” message)?

2

The above is only true if also the source address of such packets matches the IPsec policy, i.e. falls into 192.168.19.0/24. You have configured a corresponding value of the pref-src parameter of the route with gateway=ipsec, 192.168.19.254, but some rule in srcnat may change the source address of these packets, making them “invisible” for the policy.


In my case (ROS 6.47.9 on a CHR), it is not 9:1 but every 3 seconds no matter how frequently the ping is sent, however the host unreachable response is not caused by the type=blackhole route, as you can easily check by changing type=blackhole to type=prohibit. Try pinging with interval=5s, you’ll get only those host unreachable messages.

When the gateway of a route is set to an L2 interface name, RouterOS sends an ARP request for the destination IP address down that interface; if it doesn’t get a response to that ARP request within three seconds, it sends an ICMP “host unreachable” message back to the original sender.


BTW, this whole “blackhole bridge” trick is only necessary when the IPsec policy is generated dynamically. If a policy with action=encrypt exists, it always intercepts packets matching its traffic selector. If a security association is currently linked to that policy, the packets are sent via that SA; if no security association exists, the packets are silently dropped.

3

I see, so the “host unreachable” comes in every 3s because there was no answer to the ARP requests. And the normal timeout comes because there was no ICMP answer.

But I still don’t understand the source ip for “host unreachable”. Why is it coming from my WAN/public IP?

4

I did not know about this, thank you.