Wireguard - "asymmetric routing"

manojlovicl · Tue Mar 07, 2023 9:40 pm

Hi!

I have a case, that router routes some traffic over VPN (over which I get public IP address) - I want to use this public IP as Wireguard endpoint. The problem is that clients try to connect to IP (I can see packets) but responses from MT Wireguard service are going back by using default routing table.
I have output mangle rule for this public IP and it works as I can connect to MikroTik by using Winbox or ping it - but Wireguard does not want to respond on the same IP/interface where the requests are originating from.

Thank you for your help,
L

anav · Tue Mar 07, 2023 10:07 pm

Diagram of network and full config
/export file=anynameyouwish ( minus router serial # and any public WANIP information keys etc. )

With this info should be quick to fix.

manojlovicl · Tue Mar 07, 2023 10:13 pm

It is a router (A) currently located as guest in a NAT-ed network. It has (private) IP in that network. From here I do a VPN to another router (at our ISP) where I have the option to route public IP to the router (A) - this part works. I can access router (A) via Winbox by using public IP so output mangle rule works. I have setup SSTP VPN on it and it also works without problems. Today I tried to setup Wireguard on it - I am trying to connect to public IP but if I create Output firewall rule (just for logging purposes) I can see that it does not "reply" from the public IP but from the IP that it has in the NAT-ed network that I mentioned at the beginning.
Funny thing is that IP/services "respond" correctly by using public IP (when I try to access it via Winbox) and also SSTP VPN works correctly - only Wireguard has chosen the wrong out interface and wrong source IP to respond.

manojlovicl · Tue Mar 07, 2023 10:26 pm

Diagram of network and full config
/export file=anynameyouwish ( minus router serial # and any public WANIP information keys etc. )

With this info should be quick to fix.

Tue Mar 07, 2023 10:29 pm

Why this ?

/ip address
add address=10.xxx.xxx.1/24 interface=br-lan network=10.xxx.xxx.0
add address=PUBLICIP interface=sstp-kate-wing network=PUBLICIPNetwork
add address=10.xxx.xxx.1/24 interface=br-lan network=10.xxx.xxx.0
add address=10.xxx.xxx.1/24 interface=wireguard1 network=10.xxx.xxx.0

Everything the same address ?? I hope not.
If you want to mask info (which BTW is NOT needed for internal addresses) at least make it clear there is a difference (which there SHOULD be).

Care to try again ?

manojlovicl · Tue Mar 07, 2023 10:34 pm

Why this ?
Code: Select all
/ip address
add address=10.xxx.xxx.1/24 interface=br-lan network=10.xxx.xxx.0
add address=PUBLICIP interface=sstp-kate-wing network=PUBLICIPNetwork
add address=10.xxx.xxx.1/24 interface=br-lan network=10.xxx.xxx.0
add address=10.xxx.xxx.1/24 interface=wireguard1 network=10.xxx.xxx.0
Everything the same address ?? I hope not.
If you want to mask info (which BTW is NOT needed for internal addresses) at least make it clear there is a difference (which there SHOULD be).

Care to try again ?

No, there are 3 different (private) subnets, do not worry - I think there is something in mangle missing to force wireguard to send packets back where they originated from...

Tue Mar 07, 2023 10:41 pm

Wireguard shouldn't need mangle.

manojlovicl · Tue Mar 07, 2023 10:46 pm

Wireguard shouldn't need mangle.

I also thought it will be simple... But look - this is output filter rule for logging only:
output: in:(unknown 0) out:br-lan, connection-state:new proto UDP, 10.x.y.z:443->PUBLICIPOFMYPHONECLIENT:7736, len 120

Instead of private IP there should be public IP as a source...

Tue Mar 07, 2023 10:49 pm

Since you have multiple wan, mangle might be needed. Read this thread from anav.
Section 9 B

viewtopic.php?t=182340

manojlovicl · Tue Mar 07, 2023 10:56 pm

Since you have multiple wan, mangle might be needed. Read this thread from anav.
Section 9 B

viewtopic.php?t=182340

Do you think - as there is UDP that I will need a combination of packet mark + routing mark? I already have a "general" output routing mark mangle rule for public IP that this router has but it is not enough ...
In fact it is really difficult to "catch" the traffic as it is already in output as you can see in LOG attached above ...

anav · Tue Mar 07, 2023 11:25 pm

I will provide one last opportunity.
Diagram and more importantly FULL CONFIG. Including firewall rules!! etc....

/export file=anynameyouwish (minus router serial number and any public WANIP information, keys etc.)

THIS DOES NOT MEAN cover up every private IP, just the info the ISP provides that is used to identify your connection. Private IPs do not!!!

manojlovicl · Tue Mar 07, 2023 11:38 pm

I will provide one last opportunity.
Diagram and more importantly FULL CONFIG. Including firewall rules!! etc....

/export file=anynameyouwish (minus router serial number and any public WANIP information, keys etc.)

THIS DOES NOT MEAN cover up every private IP, just the info the ISP provides that is used to identify your connection. Private IPs do not!!!

Here is full config - there are no firewall rules (and I removed some PPP secrets...) This router is inside an private network and makes SSTP VPN client connection out - via that connection I am "bringing" public IP on it that I can use to connect via Winbox or use SSTP VPN (Server this time) on it...

anav · Tue Mar 07, 2023 11:48 pm

A diagram would have made this much clearer.

For example, is the SSTP VPN a connection to the MT device/router? or to the Router in front of the MT and then the VPN connection is fed to the MT as a WAN port with an IP address and gateway??

On the MT device should all users, or single subnet of many, etc use this SSTPVPN for what I assume is internet access???

+++++++++++++++++++++++++++++++++++++++++++++++

If the SSTP VPN is anchored in within the MT device, this assumes port forwarding to the MT device from the in front router and thus also assumes you have an etherport getting wanip on the private LAN of the router in front correct...........

Starting to see why a diagram is necessary.................without context, no credible answers are possible

manojlovicl · Tue Mar 07, 2023 11:55 pm

A diagram would have made this much clearer.

For example, is the SSTP VPN a connection to the MT device/router? or to the Router in front of the MT and then the VPN connection is fed to the MT as a WAN port with an IP address and gateway??

This router is a standalone router connected with 1 cable in a private network - like if brought it in your home network and connected it. It received IP address in that network via DHCP client and then it has started SSTP-OUT VPN to my datacenter (ISP) - from there I have routed /30 subnet via (that SSTP VPN) to the router, so now it has also public IP reachable from anywhere.
So at the moment router is reachable worldwide via that public IP.
By using public IP I am now able to connect to the router from anywhere via Winbox (yes, later I will setup firewall rules but let's ignore this for this example).
By using public IP I am also able to connect to SSTP VPN that I have setup (with PPP secrets... and Let's encrypt certificate...) on it.
I would like to be able to connect also via Wireguard but Wireguard (I do not know why) does not choose to respond via public IP and routing mark (OverVPN) but as it looks like it is still choosing main routing table and tries to "respond" by using private network we have our router in...
Is it clearer now?

Thank you,
Luka

Tue Mar 07, 2023 11:58 pm

Old school tools still can do wonders.

Pen and paper.
Make a drawing.

manojlovicl · Wed Mar 08, 2023 12:10 am

Old school tools still can do wonders.

Pen and paper.
Make a drawing.

anav · Wed Mar 08, 2023 1:13 am

Okay so let me see if I understand........

You have an MT router that is NOT connected to the ISP modem.
You have an ISP modem/router in front of it, that may or may not get a public IP.
I suspect it does not otherwise why would you need an SSTP client to go out to the internet vice SSTP server to host.
Furthermore if the ISP MODEM router gave you a reachable public IP you would not need to go through SSTP VPN to provide a public IP to wireguard!
Furthermore if the ISP modem router gave you a reachable public IP ( and you could forward ports on it), you dont need the SSTP VPN to make
your MT device a wireguard endpoint.

Forgetting everything else, its seems what you want to do is emulate a local WAN connection (for wireguard) via your SSTP VPN connection. So the WIreguard is riding within an SSTP VPN tunnel.
Now you can see why a better diagram with better clarity is needed. As stated above, if you have reachable public IP where you are at, and one can port forward to the MT, you dont need this level of complication.

It almost sounds like a teenager is trying to circumvent their family's internet by paying for an external SSTP VPN service with access to public IPs. Since the router is attached to the home network, it looks like any other device. With SSTP client on the router, the router now can go out the normal internet of the home, establish the VPN tunnel and no one is the wiser.
What I dont understand is if you already have this SSTP VPN connection, WHY Do you need wireguard?
SSTP VPN is also two way street so you should be able to remotely connect to the SSTP VPN server and then to the MT router SSTP client. Also users behind the MT router should be able to use the SSTP VPN to go a different path for internet traffic if so desired.

You really need to fill in some missing pieces here...........

anav · Wed Mar 08, 2023 1:21 pm

NETWORK DIAGRAM APPS:
https://nulab.com/cacoo/
https://online.visual-paradigm.com/diag ... -software/
https://www.lucidchart.com/pages/
https://drawio-app.com/product/
https://www.diagrams.net/ (its older sibling soon to be discontinued https://drawio-app.com/product/)
( Other links for diagrams.net - https://www.youtube.com/watch?v=P3ieXjI7ZSk & https://www.youtube.com/watch?v=mpF1i9sfEJ0 )
https://sourceforge.net/projects/dia-installer/
https://www.yworks.com/products/yed (and icons for yed --> https://github.com/danger89/yEd_cisco_network_icons )
https://www.libreoffice.org/discover/draw/

http://kilievich.com/fpinger/ - has a simple drawing program but not its main intent.

Wed Mar 08, 2023 3:29 pm

Easiest solution: Pen and paper ... why does everyone keep forgetting about that ?

Sob · Wed Mar 08, 2023 10:02 pm

Because it was so long ago when such things were used ^*1.

-
^{*1 Individual experiences may differ for each person}

anav · Thu Mar 09, 2023 2:33 am

Send me an email sob and I will enlighten you..............

manojlovicl · Thu Mar 09, 2023 10:32 am

I would like to thank Anav who helped me solving it out. Long story short - Wireguard (service) that runs on MikroTik prefer main routing table and binds to IP address that is in the same subnet as gateway in main routing table, even if you try to use mangle rule to make it go via other routing table and other output interface/IP. I do not know if it is a bug or it is like that by design - in my quite complex setup (Anav now knows why I was not able to draw everything

) I needed such feature.

Solution was that Anav created dst-nat rule that dstnatted incoming wireguard traffic. And just after that he managed to create connection mark and routing mark mangle rule that sent the traffic out correctly (via some other interface / IP).

Thank you for your help, colleagues.
Luka

anav · Thu Mar 09, 2023 2:17 pm

Trust me I had help, but glad you got it working, in the end it was you that found the last two entries to make it work...........
I like the premise of the scenario and will be adding it to the wireguard scenario folder.
What is not clear is if the dst nat rule was needed due to a bug in WG.... more to follow.

Tanuki · Wed Jul 05, 2023 1:11 pm

anav and manojlovicl, do you have an example of the NAT you implemented please? I am hitting the same bug, where the WireGuard service is latching onto the main table, regardless of route or mangle rules.

I can see the incoming connection correctly marked in the Connection Tracking table, and its RX counters increment but a second connection with no mark is created which increments just its TX counters. I've also confirmed this via Torch and packet capture.

This doesn't appear to affect any other local services that I've tested - for example SSH, ICMP ping, FTP, Winbox, etc.... work as expected.

Has anyone logged a bug for this with Mikrotik support that I could attach my notes to?

Thanks.

manojlovicl · Wed Jul 05, 2023 1:47 pm

Hi!

It is a simple DST-NAT where DST address is my public IP that I am receiving over VPN (udp on port 443) that is DST NATed to IP I am receinving in "hosting network".

Luka

Tanuki · Wed Jul 05, 2023 2:53 pm

Any chance you could please post the relevant config snippets?

EDIT:

I've since solved this with a simple, sane but rather unnecessary SRC-NAT rule - forcing the WireGuard daemon to think the connections are coming directly from the other end of the local WAN link and thus needing to reply that way as the link is the best route. This is definitely a bug.

manojlovicl · Wed Jul 05, 2023 3:07 pm

add action=dst-nat chain=dstnat dst-address=PUBLICIPOVERVPN dst-port=443 \
protocol=udp to-addresses=PRIVATEIPININTERNALNETWORK

Tanuki · Wed Jul 05, 2023 3:10 pm

Thanks for your quick responses, much appreciated.

Tanuki · Thu Jul 06, 2023 6:09 am

(This is a cross-post of my analysis and solution from viewtopic.php?t=197545 to get more eyes on it)

--

I've reproduced this in GNS3, and I'll submit it to MikroTik support as well, but here's the bug we're discussing here.

Network topology

Screenshot 2023-07-06 at 12.14.27.png

Entire (relevant) router config

/interface ethernet
set [ find default-name=ether1 ] comment=ISP1 disable-running-check=no
set [ find default-name=ether2 ] comment=ISP2 disable-running-check=no
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1 private-key="EMfgjAes9lBy7MZaeRfHPnwqW3S3cvRFyFOCxBRVulM="
/port
set 0 name=serial0
/routing table
add fib name=RT-via-ISP1
add fib name=RT-via-ISP2
/interface wireguard peers
add allowed-address=172.16.0.2/32 comment="WNmh78bmcvq155V0Xo8npVFACkQGbcgIIDctPXIkF1A=" interface=wireguard1 public-key="wdyxvVy8ceMPXf5dH7Bfd61Wxo5wKbf/Nx6MCR3+IEM="
/ip address
add address=192.0.2.2/24 interface=ether1 network=192.0.2.0
add address=198.51.100.2/24 interface=ether2 network=198.51.100.0
add address=172.16.0.1/24 interface=wireguard1 network=172.16.0.0
/ip firewall mangle
add action=mark-connection chain=prerouting comment="Mark incoming connections via ISP1" connection-mark=no-mark in-interface=ether1 new-connection-mark=CM-via-ISP1 passthrough=yes
add action=mark-routing chain=output comment="Mark routing for marked connections via ISP1" connection-mark=CM-via-ISP1 new-routing-mark=RT-via-ISP1 passthrough=no
add action=mark-connection chain=prerouting comment="Mark incoming connections via ISP2" connection-mark=no-mark in-interface=ether2 new-connection-mark=CM-via-ISP2 passthrough=yes
add action=mark-routing chain=output comment="Mark routing for marked connections via ISP2" connection-mark=CM-via-ISP2 new-routing-mark=RT-via-ISP2 passthrough=no
/ip route
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=192.0.2.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=20 dst-address=0.0.0.0/0 gateway=198.51.100.1 routing-table=main suppress-hw-offload=no
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=192.0.2.1 pref-src="" routing-table=RT-via-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=198.51.100.1 pref-src="" routing-table=RT-via-ISP2 scope=30 suppress-hw-offload=no target-scope=10

Observed behaviour
Almost all connections that ingress on the router will be marked correctly and will use the correct interface for egress based on the connection mark.

This can be proved as working for ICMP ping, SSH and Winbox.

The WireGuard process however does not appear to take any regard for connection marks and just replies back via the interface with the best route in the main table.

This can be proved in a number of ways:
- Only the initial handshake appears on the ISP2 packet capture, all replies appear on the ISP1 packet capture
- As I'm using Linux as a client, which supports automatically switching to the last endpoint address it heard from, as the responses are via ISP1 it immediately starts communicating with the ISP1 address even when configured to use ISP2
- In the Connection Tracking table, a connection with the correct connection mark (CM-in-WAN2) is created however it never has any traffic increment the Repl. Bytes column, rather a connection via ISP1 is immediately established

Workarounds
I've found if you force the return traffic via the correct Interface with a SRC-NAT rule this will work as expected.

/ip firewall nat
add action=src-nat chain=input comment="Hack to force WG to reply via RT-via-ISP1" dst-address=192.0.2.2 dst-port=13231 log=yes protocol=udp to-addresses=192.0.2.1
add action=src-nat chain=input comment="Hack to force WG to reply via RT-via-ISP2" dst-address=198.51.100.2 dst-port=13231 log=yes protocol=udp to-addresses=198.51.100.1

What this is doing is making the WireGuard process think that packets from ISP1 are coming from the other end of the ISP1 link, and the ISP2 packets are coming from the other end of the ISP2 link, giving it no choice but to follow the best route back to those addresses, which is via the correct ISP uplink.

This doesn't necessarily have to be a real address, just something the main route table only knows a path to via that connection. You could use a static route and SRC-NAT in combination, for example:

/ip route add
dst-address=1.2.3.4/32 gateway=ether2
/ip firewall nat
add action=src-nat chain=input comment="Hack to force WG to reply via RT-via-ISP2" dst-address=198.51.100.2 dst-port=13231 log=yes protocol=udp to-addresses=1.2.3.4

anav · Thu Jul 06, 2023 4:09 pm

Glad you submitted the supout.

Did you try putting in an additional rule in-between the two rules, to check counter, Interesting to see if it got any hits.
If so then mangling part works its the routing that is broken...........

add action=mark-connection chain=prerouting comment="Mark incoming connections via ISP1" connection-mark=no-mark in-interface=ether1 new-connection-mark=CM-via-ISP1 passthrough=yes
add action=passthrough chain=output src-port=13231
add action=mark-routing chain=output comment="Mark routing for marked connections via ISP1" connection-mark=CM-via-ISP1 new-routing-mark=RT-via-ISP1 passthrough=no

anav · Thu Jul 06, 2023 8:22 pm

So if the conclusion that the mangling output rule is being ignored for some reason.

Lets introduce Routing Rules instead of mangling to see if that has the desired effect.
Easily done if the ISPs provide static IP addresses.
IF they are dynamic one will need to create a script to update the the routing rule with the new WANIP

So in your GN3 Try this.....
a. Disable the mangle rules.
b. remove your src nat hacks.
b. add two routing rules
c. keep the defined tables and associated routes as we can re-use them for this attempt!!

add src-address=192.0.2.2 action=lookup-only-in-table table=RT-via-ISP1
add src-address=198.51.100.2 action=lookup-only-in-table table=RT-via-ISP2

/routing table
add fib name=RT-via-ISP1
add fib name=RT-via-ISP2

/ip routes ( distance not required with manual table, unless you have multiple routes within the same table }
add dst-address=0.0.0.0/0 gateway=192.0.2.1 routing-table=RT-via-ISP1
add dst-address=0.0.0.0/0 gateway=198.51.100.1 routing-table=RT-via-ISP2

Wireguard - "asymmetric routing"

Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Re: Wireguard - "asymmetric routing"

Who is online