ProtonVPN Wireguard works well except the traceroute

RouterOS General
1

Hi! I set up the ProtonVPN wireguard on the 7.3.1 RouterOS (map 2nD router). The internet works well, pings run but traceroute does not work.

Wlan-client interface (by dhcp: 192.168.2.111/24, gw 192.168.2.1) works as WAN.
LAN address is 192.168.19.1/26.
Mikrotik wg address is 10.2.0.2/32, ProtonVPN wg host is 10.2.0.1, pingable and traced fine.

When I do traceroute for any other Internet IP then all I get is timeouts:

[user@mikrotik] > /tool/traceroute 
address: 1.1.1.1 
Columns: LOSS, SENT, LAST
#  LOSS  SENT  LAST   
1  100%     6  timeout
2  100%     5  timeout
3  100%     5  timeout
4  100%     5  timeout
5  100%     5  timeout
-- [Q quit|D dump|C-z pause]

While traceroute through the usual WAN (with disabled wireguard) works fine:

[user@mikrotik] > /tool/traceroute 
address: x.y.z.97 
Columns: ADDRESS, LOSS, SENT, LAST, AVG, BEST, WORST, STD-DEV
#  ADDRESS         LOSS  SENT  LAST    AVG   BEST  WORST  STD-DEV
1  192.168.2.1     0%      17  2.8ms   5.7   2.7   20.5   4.5    
2  a.b.c.d  0%      17  13ms    14.4  9.2   30.9   5.5    
3  e.f.g.h   0%      17  14.6ms  14.5  13.4  17.9   1.3    
4  i.g.k.l   0%      17  17.6ms  17.5  12.7  26.5   4.4    
5  195.22.214.165  0%      17  12.3ms  15    12.3  25.4   3.8    
6  195.22.214.71   0%      17  33.6ms  36.3  32.2  55.2   5.7    
7  m.n.k.194  0%      17  47.2ms  36.5  33.6  47.2   3.2    
8  x.y.z.97   0%      17  34.1ms  37.5  33.6  67.6   7.7

Same behaviour when pinging/tracing on mikrotik, linux laptop Android device behind inside of the LAN.
I guess it is something with the NAT masquerades but my knowledge is not good enough to figure out the cause.

Here are the router settings.

[user@mikrotik] > /ip/address/ print 
Flags: D - DYNAMIC
Columns: ADDRESS, NETWORK, INTERFACE
#   ADDRESS           NETWORK       INTERFACE    
;;; defconf
0   192.168.19.1/27   192.168.19.0  bridge        
1   10.2.0.2/32       10.2.0.1      wg-proton-esp
2 D 192.168.2.111/24  192.168.2.0   wlan1-client

Wireguard configs:

[user@mikrotik] > /interface/wireguard/ print 
Flags: X - disabled; R - running 
 0  R name="wg-proton-esp" mtu=1360 listen-port=13201 private-key="some key" 
      public-key="another key"

and the peers


[user@mikrotik] > /interface/wireguard/peers/ print 
Flags: X - DISABLED
Columns: INTERFACE, PUBLIC-KEY, ENDPOINT-ADDRESS, ENDPOINT-PORT, ALLOWED-ADDRESS, PERSISTENT-KEEPALIVE
#   INTERFACE      PUBLIC-KEY                                    ENDPOINT-ADDRESS  ENDPOINT-PORT  ALLOWED-ADDRESS  PER
0   wg-proton-esp  somekey                                       PROTON_VPN_IP     51820          0.0.0.0/0        10s
                                                                                                  10.2.0.1/32




[user@mikrotik] > /ip/route/ print 
Flags: D - DYNAMIC; X, I, A - ACTIVE; c, s, d, y - COPY; H - HW-OFFLOADED
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#      DST-ADDRESS         GATEWAY        DISTANCE
0  As  0.0.0.0/0           10.2.0.1              2
  D d  0.0.0.0/0           192.168.2.1           3
  DAc  10.2.0.1/32         wg-proton-esp         0
  DAc  192.168.2.0/24      wlan1-client          0
  DAc  192.168.19.0/27     bridge                0
1  As  PROTON_VPN_IP/32    192.168.2.1           1




[user@mikrotik] > /ip/firewall/nat/ print 
Flags: X - disabled, I - invalid; D - dynamic 
 0    ;;; defconf: masquerade
      chain=srcnat action=masquerade out-interface-list=WAN log=no log-prefix="" ipsec-policy=out,none 
 1    chain=srcnat action=masquerade out-interface=wg-proton-esp log=no log-prefix=""




[user@mikrotik] > /ip/firewall/filter/ print
Flags: X - disabled, I - invalid; D - dynamic 
 0  D ;;; special dummy rule to show fasttrack counters
      chain=forward action=passthrough 
 1    ;;; defconf: accept established,related,untracked
      chain=input action=accept connection-state=established,related,untracked 
 2    ;;; defconf: drop invalid
      chain=input action=drop connection-state=invalid 
 3    ;;; defconf: accept ICMP
      chain=input action=accept protocol=icmp 
 4    ;;; defconf: accept to local loopback (for CAPsMAN)
      chain=input action=accept dst-address=127.0.0.1 
 5 X  ;;; defconf: drop all not coming from LAN
      chain=input action=drop in-interface-list=!LAN 
 6 X  ;;; defconf: accept in ipsec policy
      chain=forward action=accept ipsec-policy=in,ipsec 
 7 X  ;;; defconf: accept out ipsec policy
      chain=forward action=accept ipsec-policy=out,ipsec 
 8    ;;; defconf: fasttrack
      chain=forward action=fasttrack-connection hw-offload=yes connection-state=established,related 
 9    ;;; defconf: accept established,related, untracked
      chain=forward action=accept connection-state=established,related,untracked 
10    ;;; defconf: drop invalid
      chain=forward action=drop connection-state=invalid 
11    ;;; defconf: drop all from WAN not DSTNATed
      chain=forward action=drop connection-state=new connection-nat-state=!dstnat in-interface-list=WAN 
12    ;;; DNS Amplification attack
      ;;; http://forum.mikrotik.com/t/dns-amplification-attack/68310/1
      chain=input action=drop protocol=tcp in-interface=wlan1-client dst-port=53 log=no log-prefix="" 
13    ;;; DNS Amplification attack #2
      ;;; http://forum.mikrotik.com/t/dns-amplification-attack/68310/1
      chain=input action=drop protocol=udp in-interface=wlan1-client dst-port=53 log=no log-prefix=""

Could You please help me to fix this issue?

Thank You in advance.

2

First, use /tool sniffer quick interface=wg-proton-esp ip-protocol=icmp while running /tool traceroute. If you can see the ICMP packets to leave but nothing to come back, the traceroute traffic is blocked by ProtonVPN.

3

It looks like the responses are received, aren’t they?

[user@mikrotik] > /tool/sniffer/ quick interface=wg-proton-esp ip-protocol=icmp 
Columns: INTERFACE, TIME, NUM, DIR, SRC-ADDRESS, DST-ADDRESS, PROTOCOL, SIZE, CPU
INTERFACE      TIME   NUM  DIR  SRC-ADDRESS      DST-ADDRESS   PROTOCOL  SIZE  CPU
wg-proton-esp  0.069    1  <-   195.181.167.221  10.2.0.2      ip:icmp     84    0
wg-proton-esp  0.947    2  ->   10.2.0.2         1.1.1.1       ip:icmp     56    0
wg-proton-esp  1.005    3  <-   185.229.188.86   10.2.0.2      ip:icmp     84    0
wg-proton-esp  1.244    4  ->   10.2.0.2         10.2.0.1      ip:icmp     56    0
wg-proton-esp  1.249    5  ->   10.2.0.2         192.168.9.69  ip:icmp     56    0
wg-proton-esp  1.31     6  <-   10.2.0.1         10.2.0.2      ip:icmp     56    0
wg-proton-esp  1.949    7  ->   10.2.0.2         1.1.1.1       ip:icmp     56    0
wg-proton-esp  2.006    8  <-   81.173.106.178   10.2.0.2      ip:icmp     56    0
wg-proton-esp  2.95     9  ->   10.2.0.2         1.1.1.1       ip:icmp     56    0
wg-proton-esp  3.025   10  <-   81.173.106.39    10.2.0.2      ip:icmp     56    0
wg-proton-esp  3.95    11  ->   10.2.0.2         1.1.1.1       ip:icmp     56    0
wg-proton-esp  4.948   12  ->   10.2.0.2         1.1.1.1       ip:icmp     56    0
wg-proton-esp  5.004   13  <-   195.181.167.221  10.2.0.2      ip:icmp     84    0
wg-proton-esp  5.959   14  ->   10.2.0.2         1.1.1.1       ip:icmp     56    0
wg-proton-esp  6.018   15  <-   185.229.188.86   10.2.0.2      ip:icmp     84    0
wg-proton-esp  6.96    16  ->   10.2.0.2         1.1.1.1       ip:icmp     56    0
wg-proton-esp  7.016   17  <-   81.173.106.178   10.2.0.2      ip:icmp     56    0
wg-proton-esp  7.962   18  ->   10.2.0.2         1.1.1.1       ip:icmp     56    0
wg-proton-esp  8.017   19  <-   81.173.106.39    10.2.0.2      ip:icmp     56    0

ProtonVPN usually does not block icmp, I was able to check it many times when connected directly to ProtonVPN on laptop or Android.
Also I tried to connect with the different ProtonVPN server (assumed that only this one has the issue with icmp) but the behaviour was the same.

4

are u sniffing the traffic from 192.168.19.0/27. Cant see that up on your resolute

5

Yes, they are, so there is something strange in the Mikrotik.
The rules in /ip firewall filter seem fine to me, do you have any rules in /ip firewall raw?
What does put [/ip settings get rp-filter] show?
If you run ping 1.1.1.1 ttl=3, can you see any other response than timeout?

6

nichky, probably not.

[user@mikrotik] > /ip/firewall/raw print 
Flags: X - disabled, I - invalid; D - dynamic 
 0  D ;;; special dummy rule to show fasttrack counters
      chain=prerouting action=passthrough

rp-filter is set to “No”

Pingning from the laptop (connected to the ethernet port in mikrotik)
ping -t 7 1.1.1.1 - timeouts; also with anything else lower then 7
ping -t 8 1.1.1.1: “64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=55.4 ms”

7

When doing traceroute from the Mikrotik itself, you did not reach the TTL of 8. So try it again with max-hops set to 10 to see what happens.

Independent of that, as you could see the ICMP “TTL exceeded” messages to come back, it might be interesting to sniff the traceroute into a file, open the file using Wireshark and see whether there is something unusual in those messages.

8 
/tool/traceroute address=1.1.1.1 max-hops=10 
Columns: ADDRESS, LOSS, SENT, LAST, AVG, BEST, WORST, STD-DEV
#  ADDRESS  LOSS  SENT  LAST     AVG   BEST  WORST  STD-DEV
1           100%     2  timeout                            
2           100%     2  timeout                            
3           100%     2  timeout                            
4           100%     2  timeout                            
5           100%     2  timeout                            
6           100%     1  timeout                            
7  1.1.1.1  0%       1  65.7ms   65.7  65.7  65.7         0
9

OK, so the only remaining mystery to solve is why the previous responses are handled as if they didn’t come => sniff to file.

10

TTL of reply (58) indicates that there are most probably[*] 6 hops between you and 1.1.1.1 … and none of them want to be seen (they are not constructing proper “time expired” reply).

[*] most IP stack implementations use TTL with value of power of 2 … and since most routes on internet nowdays involve less than 30 hops, it’s quite likely that 1.1.1.1 uses TTL of 64. However things can be iffy. Traceroute from my chair to 1.1.1.1 currently shows 9 hops (first one being my gateway, last one being 1.1.1.1), but pings come back with TTL of 51 (which would indicate 13 hops … pinging my gateway shows TTL of 64) and when using ping -t x with x<=8 nothing comes back.

11

Should I do traceroute on my laptop (connected to the mikrotik via ethernet) and record this with the wireshark? Do I undertand this correctly?

12

This is one possibility, but the laptop will probably use UDP to traceroute - which doesn’t matter that much but it requires a more complicated capture filter (something like host 1.1.1.1 or icmp should do). What I had in mind was to set /tool sniffer set file-name=traceroute.pcap, then run the /tool sniffer quick interface=wg-proton-esp ip-protocol=icmp, then run /tool traceroute 1.1.1.1; once it fails, stop the capture, download the traceroute.pcap file to the PC and open it using Wireshark.

From what @mkx wrote I understand that he’s already done the same in the meantime, and has provided the result of his analysis, which says that Mikrotik doesn’t understand the ICMP warnings because they are incorrectly formatted.

13

The dump is here: https://mega.nz/file/uZUXTQTJ#0TA1IXDolLDCymHD7StgCGnZpmjvzhVLr030G2bEV_A

The replies are received but “Time-to-live exceeded in transit”

14

Shouldnt the MT device wireguard interface address be 10.2.0.2/24?

15

It can be, but what is the purpose? Having wg interface address 10.2.0.2/32 and a network 10.2.0.1 acts like a p2p if I understand it correctly.

16

Sob can answer it better than I, but the interface address on the MT device is a standard that ensures all aspects of traffic flow in and out of the router and WG are handled correctly.
It may make NO difference with your issues but its how it should be setup.

17

If this is more “correct” from the network administration POV then I will agree with Your opinion, as I am not a network administrator.
Actually I started with the /30 netmask when playing with protonvpn and the behaviour of the traceroute was the same.
During my investigations I reduced the configurations as much as possible and so came to this setup.

18

I hesitate to reply not being a qualified network admin. I am an expert at hot air though!!

19

It seems I have misunderstood what @mkx wrote and he was actually only commenting the output of your traceroute, not a result of his own sniffing.

Anyway, the .pcap you’ve posted shows that the actual issue is that the source address of the original packet encapsulated in the notification is 10.2.208.158 rather than the one from which the Mikrotik actually sends them (10.2.0.2), so Mikrotik doesn’t recognize the notifications as related to the packets it has sent (which is a correct behaviour).

20

How did You find it? Did You decrypt the binary message body? Could You please tell my what exactly icmp message contains this IP?
Just to clarify - I took 1.1.1.1 as a target for traceroute only for an example, actually traceroute to none of the Internet IPs work.