Community discussions

MikroTik App
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Tue Sep 08, 2020 4:16 pm

Problem described in detail here: https://www.reddit.com/r/mikrotik/comme ... ection_to/

I have two MIkrotik devices. I am testing Wireguard on the internal (behind internet router) Mikrotik device before I send it off to my parents to use behind their ISP router using a port forward (non-Mikrotik router):

Device A - Mikrotik Chateau LTE12 which is internet facing.
Device B - Mikrotik cAP ac; sits behind Device A; both ether interfaces in the same bridge; firewall disabled, no NAT; WireGuard running on UDP port 13232.

There is a DSTNAT rule on Device A which NATs inbound traffic from internet on UDP 13232 to Device B on UDP 13232.

From a device on the internal network I can successfully make a WireGuard connection to Device B but from the internet I get the constant message, "Handshake did not complete after 5 seconds, retrying".

I have also enabled a different DSTNAT rule through to the Device B routerOS web interface and it works fine from the internet so I know the routing and NAT'ing is working. I also know WireGuard keys are working on Device B because I can connect from the internal network.

I think there is an issue with running WireGuard behind a DSTNAT rule on the Mikrotik device.

Mikrotik.png
You do not have the required permissions to view the files attached to this post.
Last edited by ilium007 on Sat Nov 07, 2020 10:07 am, edited 4 times in total.
 
User avatar
eworm
Forum Guru
Forum Guru
Posts: 1070
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Wireguard not working when behind internet facing router with DSTNAT

Tue Sep 08, 2020 4:30 pm

Ok, some questions here:

Why do you configure wireguard on device B, not device A?

What does the other side look like? Does it have a public address without NAT? If it does: Things should work without destination NAT if connection is initiated from device behind NAT.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working when behind internet facing router with DSTNAT

Tue Sep 08, 2020 4:36 pm

Why do you configure wireguard on device B, not device A?

Because as I said, I am setting this device up to send to my parents to use as a wireless access point but also to give me a WireGuard VPN for remote access. They have an ISP issued VoIP router that can't be swapped out so I need to place the cAP ac behind the ISP router and use a port forward.

What does the other side look like? Does it have a public address without NAT? If it does: Things should work without destination NAT if connection is initiated from device behind NAT.

What other side? The client? The client is using the WireGuard iOS app, on an iPhone (for testing) over an LTE interface that IS NAT'd. Its the way Telstra here in Australia does things, LTE / 4G clients get a CGNAT'd IP address, not a public routable IP address. In saying this, the same iPhone with its CGNAT IP can connect to all other WireGuard instances I use, but none of these are being DSTNAT’d to from another router.
 
User avatar
eworm
Forum Guru
Forum Guru
Posts: 1070
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Wireguard not working when behind internet facing router with DSTNAT

Tue Sep 08, 2020 5:30 pm

Ah, I misread and misunderstood some details. So both peers are behind NAT, one is supposed to be reachable via destination NAT.
Never tried that with wireguard, no idea if this should work.
 
nostromog
Member Candidate
Member Candidate
Posts: 226
Joined: Wed Jul 18, 2018 3:39 pm

Re: Wireguard not working when behind internet facing router with DSTNAT

Tue Sep 08, 2020 6:36 pm

I used to have a setup where port 51820 was mapped from a Mikrotik router to my old laptop, and I connected to it from my current laptop (both linux) using wireguard.

When I upgraded my travel router to 7.1beta2 I cloned into it my current laptop configuratiion and it works like a charm. Of course handshake can only be initiated from my travel router, that knows the IP of my office one, but one it is done it keeps going until my doubly NATted travel router using USB tethering with a mobile operator that has rfc1918 addresses times out the connection and comes out through a different IP.

In other words, it works like a charm. I don't understand why it does not work for the original poster, but I don't think it is a problem in the MT software.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working when behind internet facing router with DSTNAT

Tue Sep 08, 2020 7:10 pm

I tried quick test with RouterOS as WG server behind NAT with forwarded port, same as OP has, and it works fine. Exactly as expected, since WG shouldn't care about NAT at all.
 
roswitina
newbie
Posts: 42
Joined: Tue Mar 12, 2013 8:12 am

Re: Wireguard not working when behind internet facing router with DSTNAT

Tue Sep 08, 2020 7:29 pm

Could it be that the LTE stick has a NAT address? CGNAT.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working when behind internet facing router with DSTNAT

Tue Sep 08, 2020 11:08 pm

There’s no “LTE stick”. Internet facing router is a Mikrotik Chateau LTE12.

As I said, other port forwards work through to the cAP ac so I know DSTNAT from Device A and all routing is working. I know WireGuard works on Device B because I can connect to it locally and handshake is successful. I also know how to successfully set up WireGuard as I have it running at work and home for remote access.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working when behind internet facing router with DSTNAT

Wed Sep 09, 2020 12:23 am

Ah, I misread and misunderstood some details. So both peers are behind NAT, one is supposed to be reachable via destination NAT.
Never tried that with wireguard, no idea if this should work.
Device A has a public IP address (no CGNAT) via its LTE modem. I port forward (DSTNAT) UDP 31232 to Device B which has the WireGuard instance.

The client device that his attempting to connect has a CGNAT’d IP address. This same client connects to my other WireGuard instances without any issues.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working when behind internet facing router with DSTNAT

Wed Sep 09, 2020 11:06 am

I tried quick test with RouterOS as WG server behind NAT with forwarded port, same as OP has, and it works fine. Exactly as expected, since WG shouldn't care about NAT at all.
What routerOS version is running on the Mikrotik router doing your port forward?
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working when behind internet facing router with DSTNAT

Wed Sep 09, 2020 1:48 pm

I've done some more testing and can't make sense of this.

I can successfully DSTNAT TCP 8080 traffic from the internet to Device B on TCP port 80 (just a test to the Device B webfig app) - DSTNAT works from Device A to Device B:

http://mikrotik2020.duckdns.org

[admin@MikroTik] /ip/firewall> nat/ pr
Flags: X - disabled, I - invalid; D - dynamic
 0    ;;; defconf: masquerade
      chain=srcnat action=masquerade out-interface-list=WAN log=no log-prefix="" ipsec-policy=out,none

 1    ;;; ;;; force DNS
      chain=dstnat action=dst-nat to-addresses=192.168.10.2 protocol=udp src-address=!192.168.10.2 dst-address=!192.168.10.2 in-interface=bridge dst-port=53 log=no log-prefix=""

 2    ;;; ;;; force DNS
      chain=srcnat action=masquerade protocol=udp src-address=192.168.10.0/24 dst-address=192.168.10.2 dst-port=53 log=no log-prefix=""

 3    ;;; DSTNAT for Device B WireGuard
      chain=dstnat action=dst-nat to-addresses=192.168.10.5 to-ports=13232 protocol=udp in-interface=lte1 dst-port=13232 log=no log-prefix=""

 4    ;;; DSTNAT for Device B webfig
      chain=dstnat action=dst-nat to-addresses=192.168.10.5 to-ports=80 protocol=tcp in-interface=lte1 dst-port=8080 log=yes log-prefix="8080_dstnat"
[admin@MikroTik] /ip/firewall>

I can make a successful Wireguard connection to Device B from inside the LAN so I know the Device B WireGuard keys are working.

I still can't make a connection to the Device B WireGuard from the internet via the DSTNAT rule above. I don't understand why, if I can DSTNAT the port 8080 traffic, why I wouldn't be able to DSTNAT the WireGuard traffic.


Device A - 192.168.10.1 / Public IP on WAN:

IP Addresses:
[admin@MikroTik] > ip address/ print
Flags: D - DYNAMIC
Columns: ADDRESS, NETWORK, INTERFACE
  #     ADDRESS            NETWORK         INTERFACE
;;; defconf
  0     192.168.10.1/24    192.168.10.0    ether1
  1  D  123.209.117.65/32  123.209.117.65  lte1
[admin@MikroTik] >
Routes:
[admin@MikroTik] >  ip route/ pr
Flags: D - DYNAMIC; A - ACTIVE; C - CONNECT, m - MODEM
Columns: DST-ADDRESS, GATEWAY, DISTANCE
       DST-ADDRESS       GATEWAY     D
  DAm  0.0.0.0/0         lte1        2
  DAC  123.209.117.65    lte1        0
  DAC  192.168.10.0/24   bridge      0
[admin@MikroTik] >
Firewall Rules:
[admin@MikroTik] > /ip fire ex
# sep/09/2020 20:32:55 by RouterOS 7.1beta2
# software id = 8DD5-P647
#
# model = RBD53G-5HacD2HnD
# serial number = C8CA0CB0B626
/ip firewall address-list
add address=192.168.10.11-192.168.10.255 list=lan_clients
add address=192.168.10.100 list=support
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid log=yes log-prefix=drop
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN log=yes log-prefix=drop
add action=accept chain=forward comment="defconf: accept in ipsec policy" disabled=yes ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" disabled=yes ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=dst-nat chain=dstnat comment=";;; force DNS" dst-address=!192.168.10.2 dst-port=53 in-interface=bridge protocol=udp src-address=!192.168.10.2 to-addresses=192.168.10.2
add action=masquerade chain=srcnat comment=";;; force DNS" dst-address=192.168.10.2 dst-port=53 protocol=udp src-address=192.168.10.0/24
add action=dst-nat chain=dstnat comment="DSTNAT for Device B WireGuard" dst-port=13232 in-interface=lte1 protocol=udp to-addresses=192.168.10.5 to-ports=13232
add action=dst-nat chain=dstnat comment="DSTNAT for Device B webfig" dst-port=8080 in-interface=lte1 log=yes log-prefix=8080_dstnat protocol=tcp to-addresses=192.168.10.5 to-ports=80
[admin@MikroTik] >

Device B - 192.168.10.5:

IP Addresses:
[admin@MikroTik] > ip address/ print
Columns: ADDRESS, NETWORK, INTERFACE
  #  ADDRESS           NETWORK        INTERFACE
  0  192.168.10.5/24   192.168.10.0   ether1
  1  192.168.201.1/24  192.168.201.0  wireguard1
[admin@MikroTik] >
Routes:
[admin@MikroTik] > ip route/ pr
Flags: D - DYNAMIC; A - ACTIVE; C - CONNECT, S - STATIC, m - MODEM
Columns: DST-ADDRESS, GATEWAY, DISTANCE
  #       DST-ADDRESS       GATEWAY       D
  0   AS  0.0.0.0/0         192.168.10.1  1
     DAC  192.168.10.0/24   bridge        0
     DAC  192.168.201.0/24  wireguard1    0
[admin@MikroTik] >
Firewall Rules:
[admin@MikroTik] > /ip fire ex
# sep/09/2020 20:28:42 by RouterOS 7.1beta2
# software id = 50RA-6BBJ
#
# model = RBcAPGi-5acD2nD
# serial number = BECD0C73D111
[admin@MikroTik] >
WireGuard Config:
[admin@MikroTik] /interface/wireguard> export
# sep/09/2020 20:41:56 by RouterOS 7.1beta2
# software id = 50RA-6BBJ
#
# model = RBcAPGi-5acD2nD
# serial number = BECD0C73D111
/interface wireguard
add listen-port=13232 mtu=1420 name=wireguard1 private-key="xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
/interface wireguard peers
add allowed-address=192.168.201.10/32 endpoint=100.103.197.44:61497 interface=wireguard1 persistent-keepalive=25 preshared-key="xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" public-key=\
    "meLYM9lu4ViXIOkwB8qCre452hBwV9asGJ/2DIzuiCQ="
[admin@MikroTik] /interface/wireguard>

I am hoping this is just something stupid in the config but I feel its a bug in WireGuard in routerOS 7. I have done this type on thing for years on Ubiquiti EdgeOS without an issue. I just don't think the problem is in my config.
Last edited by ilium007 on Thu Sep 10, 2020 9:37 am, edited 2 times in total.
 
roswitina
newbie
Posts: 42
Joined: Tue Mar 12, 2013 8:12 am

Re: Wireguard not working when behind internet facing router with DSTNAT

Wed Sep 09, 2020 9:23 pm

Have you tried to access Wireguard on device A via port 8080 and set up DSTNAT to device B to the Wireguard port. Some ISPs block various ports. If port 8080 (TCP) works, it should also work on WG.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working when behind internet facing router with DSTNAT

Wed Sep 09, 2020 10:36 pm

Have you tried to access Wireguard on device A via port 8080 and set up DSTNAT to device B to the Wireguard port. Some ISPs block various ports. If port 8080 (TCP) works, it should also work on WG.
As expected it behaves exactly the same. Packets were always getting through the DSTNAT to the WireGuard instance on Device B. Something happens with the replies to the client attempting connection.
Last edited by ilium007 on Thu Sep 10, 2020 3:42 am, edited 1 time in total.
 
User avatar
eworm
Forum Guru
Forum Guru
Posts: 1070
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Wireguard not working when behind internet facing router with DSTNAT

Wed Sep 09, 2020 11:47 pm

Is there are firewall rule that does source NAT just after destination NAT for the incoming packet? Possibly that confuses wireguard...
What interfaces are in interface list "WAN"?
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working when behind internet facing router with DSTNAT

Thu Sep 10, 2020 3:58 am

Is there are firewall rule that does source NAT just after destination NAT for the incoming packet? Possibly that confuses wireguard...
What interfaces are in interface list "WAN"?
SRCNAT / masquerade rule is above the DSTNAT rule for Device B WireGuard instance:
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none \
    out-interface-list=WAN
add action=dst-nat chain=dstnat comment=";;; force DNS" disabled=yes dst-address=!192.168.10.2 \
    dst-port=53 in-interface=bridge protocol=udp src-address=!192.168.10.2 to-addresses=192.168.10.2
add action=masquerade chain=srcnat comment=";;; force DNS" disabled=yes dst-address=192.168.10.2 \
    dst-port=53 protocol=udp src-address=192.168.10.0/24
add action=dst-nat chain=dstnat comment="DSTNAT for Device B WireGuard" dst-port=13232 in-interface=\
    lte1 protocol=udp to-addresses=192.168.10.5 to-ports=13232
add action=dst-nat chain=dstnat comment="DSTNAT for Device B webfig" disabled=yes dst-port=8080 \
    in-interface=lte1 log=yes log-prefix=8080_dstnat protocol=tcp to-addresses=192.168.10.5 \
    to-ports=13232
[admin@MikroTik] > 

WAN interface list (Device A):
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=lte1 list=WAN
[admin@MikroTik] > 
Someone over in the Reddit Mikrotik group has, today, responded to my initial attempts to troubleshoot and has verified the issue when DSTNAT’ing WireGuard traffic to Device B from Device A.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working when behind internet facing router with DSTNAT

Thu Sep 10, 2020 6:40 am

What routerOS version is running on the Mikrotik router doing your port forward?
My device A has 6.45.8, but it has nothing to do with it, you can use literally any RouterOS version. Device A doesn't do anything related to WG, it just forwards one udp port to device B and doesn't care what's inside those packets.

I'd try packet sniffer on client device, to verify that response packets from server are making it back to it. There's no reason why they shouldn't, but for the lack of better ideas... Only you may need to use something other than iPhone, I'm not familiar with it, but I somehow doubt that it has packet sniffer.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working when behind internet facing router with DSTNAT

Thu Sep 10, 2020 6:47 am

But there's one thing I noticed now, you have 192.168.201.0/24 as address on wireguard1 interface. But it's not correct address, with .0 at the end it's subnet address. Change it to something else, e.g. .1.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working when behind internet facing router with DSTNAT

Thu Sep 10, 2020 9:39 am

But there's one thing I noticed now, you have 192.168.201.0/24 as address on wireguard1 interface. But it's not correct address, with .0 at the end it's subnet address. Change it to something else, e.g. .1.
That was a configuration typo during testing! I have fixed now and it behaves the same. I am away from home tonight so I can get a different client set up and perform a Wireshark capture.
[admin@MikroTik] > ip add print
Columns: ADDRESS, NETWORK, INTERFACE
  #  ADDRESS           NETWORK        INTERFACE
  0  192.168.10.5/24   192.168.10.0   ether1
  1  192.168.201.1/24  192.168.201.0  wireguard1
[admin@MikroTik] >
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working when behind internet facing router with DSTNAT

Thu Sep 10, 2020 11:47 am

After a lot of messing around I have captured the USP packet stream for both the client and Device B.

https://transfer.sh/Poqim/client_connect.pcapng

172.20.1.195 - hotel room IP address
124.19.6.93 - hotel WAN IP address
123.209.117.65 - Device A WAN IP address (LTE modem interface where DSTNAT to 192.168.10.5 is done)
192.168.10.5 - Device B LAN IP aaddress (wireguard server on UDP 13232)
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Thu Sep 10, 2020 9:13 pm

And how exactly did you capture this? It looks like strange mix. Some packets are wrapped in TZSP, from 192.168.10.5 to 192.168.200.11 (what's that?), while others look like direct capture from interface. But if it's captured using packet sniffer on 192.168.10.5, as TZSP suggests, then it shouldn't see hotel's 172.20.1.195, but it's in there.

Also, how does client's config look like? I don't think you posted it before.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Fri Sep 11, 2020 1:02 am

And how exactly did you capture this? It looks like strange mix. Some packets are wrapped in TZSP, from 192.168.10.5 to 192.168.200.11 (what's that?), while others look like direct capture from interface. But if it's captured using packet sniffer on 192.168.10.5, as TZSP suggests, then it shouldn't see hotel's 172.20.1.195, but it's in there.

Also, how does client's config look like? I don't think you posted it before.
I established a separate WireGuard connection to Device A from my laptop in the hotel, that Device A WireGuard subnet is 192.168.200.0/24 and my laptop has IP address 192.168.200.11 once connected. Device B has a static route for the 192.168.200.0/24 subnet via 192.168.10.1. I used the packet sniffer streaming feature on Device B (192.168.10.5) to stream the UDP packets back to my laptop in the hotel (192.168.200.11:37008 via WireGuard). So whilst the WireGuard VPN to Device A was established (to allow me to stream the UDP packets back from Devcie B) I then established the WireGuard connection from the laptop to Device B (using command line WireGuard client).

With Wireshark running a capture filter "(udp port 37008) or (udp port 13232)" on the hotel laptop I captured the streamed packets from Device B on UDP port 37008 and the WireGuard connection attempt on UDP port 13232 at the same time so we could see both sides in the one capture.

Client config:
[Interface]
PrivateKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Address = 192.168.201.200/32
#DNS = 192.168.10.2

[Peer]
PublicKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
#AllowedIPs = 192.168.10.0/24
AllowedIPs = 192.168.201.0/24
Endpoint = libertyrd.duckdns.org:13232
PresharedKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
PersistentKeepalive = 25
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Fri Sep 11, 2020 1:32 am

I realised that I could have just produced a .pcap file on Device B itself using the packet sniffer tool.

To avoid confusion I have produced two seperate packet captures, one from my laptop in the hotel, the other from Device B.

Laptop:
SRC IP: 172.20.1.195
Hotel WAN IP: 124.19.6.93
https://transfer.sh/gFedF/client_connect.pcap

Client Config:
[Interface]
PrivateKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Address = 192.168.201.200/32
#DNS = 192.168.10.2

[Peer]
PublicKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
#AllowedIPs = 192.168.10.0/24
AllowedIPs = 192.168.201.0/24
Endpoint = libertyrd.duckdns.org:13232
PresharedKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
PersistentKeepalive = 25

Device B:
LTE WAN IP: 123.209.117.65
https://transfer.sh/Mkloy/device_b.pcap

I’ll configure a Raspberry Pi tonight in place of Device B and confirm everything works as I would expect it to.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Fri Sep 11, 2020 7:23 am

That transfer.sh server is semi-dead for me, it was slow before, but now I managed to download only client_connect.pcap and it took two hours.

The obvious problem is that it contains only initial requests, there's not a single response. It was the same in previous combined capture as well. All non-TZSP packets are from client to server. But if you look at TZSP packets, you can see that device B is sending responses.

So one more thing you can try is to capture packets on device A's WAN interface. If you see responses there, the problem is somewhere else. If not, then they are eaten either by device A, or by something between device A and B (you could try another capture on device A's LAN interface to tell which it is).
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Fri Sep 11, 2020 6:22 pm


The obvious problem is that it contains only initial requests, there's not a single response. It was the same in previous combined capture as well. All non-TZSP packets are from client to server. But if you look at TZSP packets, you can see that device B is sending responses.

So one more thing you can try is to capture packets on device A's WAN interface. If you see responses there, the problem is somewhere else. If not, then they are eaten either by device A, or by something between device A and B (you could try another capture on device A's LAN interface to tell which it is).
I have captured the packets again.

http://ge.tt/8210P573

Client WAN IP: 49.195.16.14
Client LAN IP: 192.168.8.101

Device A WAN IP: 123.209.117.65
Device B LAN IP: 192.168.10.5

I have also built a new WireGuard server on a Raspberry Pi. I can connect to it successfully locally (within the 192.168.10.0/24 LAN) but when I DSTNAT to it from Device A I cannot connect to it from the WAN side. I will provide packet captures for this connection as well if it helps.
Last edited by ilium007 on Fri Sep 11, 2020 6:46 pm, edited 1 time in total.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Fri Sep 11, 2020 6:44 pm

Captures from the DSTNAT connection attempt to a Raspberry Pi WireGuard server running behind the Mikrotik Device A.

http://ge.tt/2jsaQ573

Client WAN IP: 49.195.16.14
Client LAN IP: 192.168.8.101

Device A WAN IP: 123.209.194.128
Device B LAN IP: 192.168.10.20
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Fri Sep 11, 2020 10:13 pm

It's still the same thing. There are requests coming from client to server, and on your routers you can see responses from server to client. But for some reason, none of them reaches the client, so it's no suprise that connection fails. It doesn't look like you can do anything on your routers. I'd blame the other side, or your ISP, but it seems you tested it from two different places, and you wrote that WG connection to device A works, so that's not it either.

I'm running out of ideas. Last one for now, although there's no reason why it should be the problem, did you try to connect only to device B and no other server at the same time?
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sat Sep 12, 2020 2:13 am

I'm running out of ideas. Last one for now, although there's no reason why it should be the problem, did you try to connect only to device B and no other server at the same time?
The last two sets of packet captures where whilst connecting to Device B only. One WireGuard connection only.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sat Sep 12, 2020 3:21 am

So this happened.....

I set up a dirt cheap TP-Link LTE 4G router with the same SIM card and gave it the same LAN IP address and set up a port forward to the Mikrotik Device B. I connected from the same iOS WireGuard client and it successfully connected and performed handshake.

The issue has to be within the DSTNAT part of the Mikrotik Device A.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sat Sep 12, 2020 9:02 pm

Not that it would be completely impossible, but dstnat is simple thing, it's there for years, everyone uses it, ... it's not likely that it would get broken and not noticed by many other people. On the other hand, it is beta version, so maybe the chance is slightly higher.

You can test another service. Not tcp, that may be different, but udp. In v7 there's udp support for OpenVPN, so you can test it with that.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sun Sep 13, 2020 2:00 am

Not that it would be completely impossible, but dstnat is simple thing, it's there for years, everyone uses it, ... it's not likely that it would get broken and not noticed by many other people. On the other hand, it is beta version, so maybe the chance is slightly higher.

You can test another service. Not tcp, that may be different, but udp. In v7 there's udp support for OpenVPN, so you can test it with that.

I agree.... its just strange that the Mikrotik cAP running WireGuard worked fine behind a non-Mikrotik cheap 4G router doing the dstnat. No changes were made to the Mikrotik cAP, just connected ethernet and set up the port forward and it worked. I can test OpenVPN but it will take some time to set it up on the Mikrotik. I will give it a go later tonight.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sun Sep 13, 2020 2:53 am

Or try dns as the first easy test. It's just one packet with request and one with response, but since with WG not even first response made it back to client, it could be enough.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sun Sep 13, 2020 12:39 pm

DNS query from Internet to Device B via Device A DSTNAT worked.


Packet capture:
http://ge.tt/4O1Xh773

IMG_3207.jpg
You do not have the required permissions to view the files attached to this post.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sun Sep 13, 2020 9:43 pm

I didn't expect that it would change anything, but I tested device A with 7.1beta2 and no problem at all, everything works.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sun Sep 13, 2020 10:07 pm

I didn't expect that it would change anything, but I tested device A with 7.1beta2 and no problem at all, everything works.
What exactly did you test??
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Mon Sep 14, 2020 3:28 am

The 'broken port forwarding' theory. So no change for device B, but I also used 7.1beta2 on device A.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Mon Sep 14, 2020 12:34 pm

The 'broken port forwarding' theory. So no change for device B, but I also used 7.1beta2 on device A.

Sorry, but I'm a little confused. Are you saying that you have put a 7.1beta2 WireGuard instance (Device B) behind a 7.1beta2 router (Device A) and its all working?
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Mon Sep 14, 2020 1:31 pm

Exactly.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Mon Sep 14, 2020 2:39 pm

Exactly.
So how do I troubleshoot my failed installation then? It works if I put a cheap router in front and port forward, it fails when I put a $365 Mikrotik Chateau LTE12 in front and dstnat the WireGuard traffic.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Mon Sep 14, 2020 4:11 pm

It's difficult. Your packet captures showed that responses are being sent by device B and they also seemed to successfully pass through device A and continue to internet and client. But with LTE connection you can't verify this last part. If it was regular ethernet, you could add another device between device A and ISP and see that it definitely sends those packets.

Although, you can sort of do it. Put SIM in TP-Link, connect Chateau behind it, and then cAP behind that, so you'll end up with double NAT. Forward port from TP-Link to Chateau and from there to cAP. If it works, then 7.1beta2 on your Chateau is not completely broken. The difference between non-working and working would be only whether you use LTE or ethernet as uplink. In that case, it would seem as possible bug, and you could try to continue with MikroTik support, send them supout from router and hopefully they would see something in it.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Wed Sep 16, 2020 5:52 am

Ok, so putting the TPLink LTE router (Device 0) in front of Device A and placing a port forward (UDP 13232) on Device 0 to Device A, and then Device A with a port forward to Device B WORKED.

So its not the dstnat that the problem it is something in the LTE side of things.

Device 0 - TPlink LTE router WAN 100.96.102.142 / LAN 192.168.6.1

Device A - Mikrotik Chateau LTE12 - WAN 192.168.6.254 / LAN 192.168.10.1

Device B - Mikrotik cAP ac - LAN 192.168.10.254

Packet capture from Device A and Device B (couldn't get packet capture from TPLink as there is no way to get a port into promiscuous mode):

http://ge.tt/2HbXTC73

So where to from here?
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Wed Sep 16, 2020 3:54 pm

I'm out of ideas.

- TP-Link with LTE works => LTE is ok
- Chateau with ethernet works => Chateau is ok
- Chateau with LTE does not work => ???, but why, when both should be ok?

It would be interesting if someone else could test it with their Chateau and LTE, but so far there isn't anyone else. I assume you did test it with default config with no modifications other than the one dstnat rule, to completely rule out any mistake you could have done, right? Then all what's left is contacting MikroTik support, but it will be "interesting" to explain it, because it would be really weird and hard to believe bug.
 
User avatar
eworm
Forum Guru
Forum Guru
Posts: 1070
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Thu Sep 17, 2020 1:36 am

Does it make a difference if you lower the mtu size on wireguard interfaces?
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Thu Sep 17, 2020 1:39 am

Does it make a difference if you lower the mtu size on wireguard interfaces?
On Device B?
 
User avatar
eworm
Forum Guru
Forum Guru
Posts: 1070
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Thu Sep 17, 2020 1:41 am

Does it make a difference if you lower the mtu size on wireguard interfaces?
On Device B?
Yes, on device B and on your client. I think the mtu should match on both sides. No idea what happens if it does not.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Thu Sep 17, 2020 1:43 am

I'm out of ideas.

- TP-Link with LTE works => LTE is ok
- Chateau with ethernet works => Chateau is ok
- Chateau with LTE does not work => ???, but why, when both should be ok?

It would be interesting if someone else could test it with their Chateau and LTE, but so far there isn't anyone else. I assume you did test it with default config with no modifications other than the one dstnat rule, to completely rule out any mistake you could have done, right? Then all what's left is contacting MikroTik support, but it will be "interesting" to explain it, because it would be really weird and hard to believe bug.

How can it be a 'hard to believe bug' - I am pretty sure I have provided more than enough information in this thread to make it believable.

There is a pinned point in this forum stating that routerOS beta 7 bug reports get made here, in this forum. I thought thats what I have been doing by spending hours dumping the info in this thread!

viewtopic.php?f=1&t=152006
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Thu Sep 17, 2020 1:59 am

Yes, on device B and on your client. I think the mtu should match on both sides. No idea what happens if it does not.

Tried lowering on both with no success. I have checked MTU on the lte1 interface and it is set at 1500. There is no packet fragmentation when doing ping tests to an internet host at MTU 1472 (+28 ICMP header) from a network client.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Thu Sep 17, 2020 2:02 am

The interface should just send and receive packets, no matter if it's LTE, ethernet, or anything else. Whether port forwarding works correctly or not, that should be decided on another level and interface type should have nothing to do with it. It's not impossible, some bugs may be weird and unexpected. That's exactly what I can say about this - weird.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Thu Sep 17, 2020 3:33 pm

I agree, but it pretty clear from all the testing that I have done that there is an issue. The dstnat works fine from another cheap router via its LTE interface, dstnat on ethernet on Device A works fin, the WireGuard instance on Device B works fine. The problem is dstnat when using the LTE WAN interface on Device A.

How do we raise a bug for routerOS 7.1beta2?
 
User avatar
antonsb
MikroTik Support
MikroTik Support
Posts: 385
Joined: Sun Jul 24, 2016 3:12 pm
Location: Riga, Latvia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Mon Oct 12, 2020 5:13 pm

we have possible fix for this issue, that will be included in upcoming version.
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Mon Oct 12, 2020 10:31 pm

we have possible fix for this issue, that will be included in upcoming version.
I tested the bug on beta3 and it’s still there
 
FIPTech
Long time Member
Long time Member
Posts: 558
Joined: Tue Dec 22, 2009 1:53 am

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sat Oct 24, 2020 11:47 pm

we have possible fix for this issue, that will be included in upcoming version.
I tested the bug on beta3 and it’s still there
Where did you get beta 3 ?
 
rplant
Member Candidate
Member Candidate
Posts: 281
Joined: Fri Sep 29, 2017 11:42 am

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sat Oct 31, 2020 10:19 am

I thought chateau was only ros v7?
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sat Nov 07, 2020 9:54 am

I thought chateau was only ros v7?
It is, my issues are all on RouterOS7, verified in RouterOS7 Beta 3
 
mmlea
just joined
Posts: 8
Joined: Sun Nov 08, 2020 12:56 am

Re: Wireguard not working when behind internet facing router with DSTNAT

Sun Nov 08, 2020 1:02 am

Ah, I misread and misunderstood some details. So both peers are behind NAT, one is supposed to be reachable via destination NAT.
Never tried that with wireguard, no idea if this should work.
yeah it works
check this article
https://www.jordanwhited.com/posts/wire ... traversal/
 
ilium007
Member Candidate
Member Candidate
Topic Author
Posts: 206
Joined: Sun Jan 31, 2010 9:58 am
Location: Newcastle, Australia

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sat Nov 14, 2020 11:09 pm

It works 100% with other hardware. It only fails for me on MikroTik RouterOS.
 
dpaiha
just joined
Posts: 1
Joined: Thu Mar 14, 2019 8:58 am
Location: Austria

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Sat Nov 28, 2020 11:57 am

I've seen the same problem in a similar scenario:
software = RouterOS 7.1beta2
model = RBD53G-5HacD2HnD
Scenario:
smartphone <--> internet <--> LTE with chateau12 <-->RP962UiGS-5Hac.. <--> raspi with wireguard on port 51820
using a different LTE-router worked without any problems (but didn't allow me to sniff packets going out). After some days and some traces I realized that the raspi sends out the packets with a dscp-mark of AF-41. I removed the mark at the source with
iptables -t mangle -A POSTROUTING -p udp --sport 51820 -j DSCP --set-dscp 0x0
and immediately after that command the connection started to work. Looks like someone on the internet throws away packets marked with AF-41...

So in case you can't or don't want to change the dscp on the wireguard server side, you can also configure everything on the mikrotik side
/ip firewall address-list
add address=<removed>.sn.mynetname.net list="MY WAN ADDRESS"

/ip firewall nat
add action=dst-nat chain=dstnat dst-address-list="MY WAN ADDRESS" dst-port=51820 protocol=udp \
to-addresses=192.168.1.23 to-ports=51820

/ip firewall mangle
add action=set-priority chain=postrouting log-prefix=dscp0 new-priority=0 passthrough=no protocol=udp \
src-address=192.168.1.23 src-port=51820
Give it a try and see if it's the same problem on your side

and with 7.1beta3 it works without the mangle-rule ;-)
Last edited by dpaiha on Sat Dec 05, 2020 12:50 pm, edited 3 times in total.
 
DuncanCT
newbie
Posts: 28
Joined: Thu May 24, 2018 1:28 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Tue Dec 01, 2020 10:10 pm

Not sure if this is will help. I've just got through a setup of cAP Lite. It's working now. Looking at the differences between what I have and yours...

Device B - 10.0.0.9:
Columns: ADDRESS, NETWORK, INTERFACE
  #     ADDRESS      NETWORK   INTERFACE 
  0  D  10.0.0.9/24  10.0.0.0  bridge1   
  1     10.1.0.1/24  10.1.0.0  wireguard1
ether 1 is on the bridge and the bridge has the IP address. Routes defined for the bridge.

Not sure if this makes any difference but I also started from no default config.
 
dcavni
Member Candidate
Member Candidate
Posts: 107
Joined: Sun Mar 31, 2013 6:02 pm

Re: Wireguard not working behind internet facing router with DSTNAT v7.1beta2

Mon Jan 04, 2021 1:26 am

Hi

I'm having some problems with Wireguard on Hap Ac2 (7.1 beta3). I somehow manage to get port forwarding working on my main router (Hap AC3) and now i would like to make a wireguard server on one of the CAPsMAN client (HapAC2).

It seems, that connection is working, since i can ping my phone's IP from Wireguard server and also i can ping Wireguard server from my phone. But that is all that is working. I can't get acces to any device in local network and also not to the internet from my phone. In allowed IP's on client i have 0.0.0.0/0.

Should i enter some aditional routes in IP/routes or maybe some firewall rules (currently there is no rules in Firewall and NAT)?
# jan/04/2021 00:26:55 by RouterOS 7.1beta3
# software id = 1W79-SWFL
#
# model = RBD52G-5HacD2HnD
# serial number = C6140BFD9F14
/interface wireguard
add listen-port=51821 mtu=1420 name=WG
/interface wireguard peers
add allowed-address=10.0.0.2/32 interface=WG public-key=\
    "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
add allowed-address=10.0.0.3/32 interface=WG public-key=\
    "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
[admin@MikroTik] /interface/wireguard> 

[admin@MikroTik] /ip/address> print
Columns: ADDRESS, NETWORK, INTERFACE
  #  ADDRESS          NETWORK      INTERFACE  
  0  192.168.3.31/24  192.168.3.0  bridgeLocal
  1  10.0.0.1/24      10.0.0.0     WG         
[admin@MikroTik] /ip/address> 
[admin@MikroTik] /ip/route> print
Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, s - STATIC, y - COPY
Columns: DST-ADDRESS, GATEWAY, DISTANCE
  #       DST-ADDRESS     GATEWAY      D
  0   As  0.0.0.0/0       192.168.3.3  1
     DAc  10.0.0.0/24     WG           0
     DAc  192.168.3.0/24  bridgeLocal  0
[admin@MikroTik] /ip/route> 

Who is online

Users browsing this forum: No registered users and 23 guests