Send me an email sob and I will enlighten you…
I would like to thank Anav who helped me solving it out. Long story short - Wireguard (service) that runs on MikroTik prefer main routing table and binds to IP address that is in the same subnet as gateway in main routing table, even if you try to use mangle rule to make it go via other routing table and other output interface/IP. I do not know if it is a bug or it is like that by design - in my quite complex setup (Anav now knows why I was not able to draw everything 
) I needed such feature.
Solution was that Anav created dst-nat rule that dstnatted incoming wireguard traffic. And just after that he managed to create connection mark and routing mark mangle rule that sent the traffic out correctly (via some other interface / IP).
Thank you for your help, colleagues.
Luka
Trust me I had help, but glad you got it working, in the end it was you that found the last two entries to make it work…
I like the premise of the scenario and will be adding it to the wireguard scenario folder.
What is not clear is if the dst nat rule was needed due to a bug in WG… more to follow.
anav and manojlovicl, do you have an example of the NAT you implemented please? I am hitting the same bug, where the WireGuard service is latching onto the main table, regardless of route or mangle rules.
I can see the incoming connection correctly marked in the Connection Tracking table, and its RX counters increment but a second connection with no mark is created which increments just its TX counters. I’ve also confirmed this via Torch and packet capture.
This doesn’t appear to affect any other local services that I’ve tested - for example SSH, ICMP ping, FTP, Winbox, etc… work as expected.
Has anyone logged a bug for this with Mikrotik support that I could attach my notes to?
Thanks.
Hi!
It is a simple DST-NAT where DST address is my public IP that I am receiving over VPN (udp on port 443) that is DST NATed to IP I am receinving in “hosting network”.
Luka
Any chance you could please post the relevant config snippets?
EDIT:
I’ve since solved this with a simple, sane but rather unnecessary SRC-NAT rule - forcing the WireGuard daemon to think the connections are coming directly from the other end of the local WAN link and thus needing to reply that way as the link is the best route. This is definitely a bug.
add action=dst-nat chain=dstnat dst-address=PUBLICIPOVERVPN dst-port=443
protocol=udp to-addresses=PRIVATEIPININTERNALNETWORK
Thanks for your quick responses, much appreciated.
(This is a cross-post of my analysis and solution from http://forum.mikrotik.com/t/output-route-selection-wireguard/167827/1 to get more eyes on it)
–
I’ve reproduced this in GNS3, and I’ll submit it to MikroTik support as well, but here’s the bug we’re discussing here.
Network topology
Entire (relevant) router config
/interface ethernet
set [ find default-name=ether1 ] comment=ISP1 disable-running-check=no
set [ find default-name=ether2 ] comment=ISP2 disable-running-check=no
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1 private-key="EMfgjAes9lBy7MZaeRfHPnwqW3S3cvRFyFOCxBRVulM="
/port
set 0 name=serial0
/routing table
add fib name=RT-via-ISP1
add fib name=RT-via-ISP2
/interface wireguard peers
add allowed-address=172.16.0.2/32 comment="WNmh78bmcvq155V0Xo8npVFACkQGbcgIIDctPXIkF1A=" interface=wireguard1 public-key="wdyxvVy8ceMPXf5dH7Bfd61Wxo5wKbf/Nx6MCR3+IEM="
/ip address
add address=192.0.2.2/24 interface=ether1 network=192.0.2.0
add address=198.51.100.2/24 interface=ether2 network=198.51.100.0
add address=172.16.0.1/24 interface=wireguard1 network=172.16.0.0
/ip firewall mangle
add action=mark-connection chain=prerouting comment="Mark incoming connections via ISP1" connection-mark=no-mark in-interface=ether1 new-connection-mark=CM-via-ISP1 passthrough=yes
add action=mark-routing chain=output comment="Mark routing for marked connections via ISP1" connection-mark=CM-via-ISP1 new-routing-mark=RT-via-ISP1 passthrough=no
add action=mark-connection chain=prerouting comment="Mark incoming connections via ISP2" connection-mark=no-mark in-interface=ether2 new-connection-mark=CM-via-ISP2 passthrough=yes
add action=mark-routing chain=output comment="Mark routing for marked connections via ISP2" connection-mark=CM-via-ISP2 new-routing-mark=RT-via-ISP2 passthrough=no
/ip route
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=192.0.2.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=20 dst-address=0.0.0.0/0 gateway=198.51.100.1 routing-table=main suppress-hw-offload=no
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=192.0.2.1 pref-src="" routing-table=RT-via-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=198.51.100.1 pref-src="" routing-table=RT-via-ISP2 scope=30 suppress-hw-offload=no target-scope=10
Observed behaviour
Almost all connections that ingress on the router will be marked correctly and will use the correct interface for egress based on the connection mark.
This can be proved as working for ICMP ping, SSH and Winbox.
The WireGuard process however does not appear to take any regard for connection marks and just replies back via the interface with the best route in the main table.
This can be proved in a number of ways:
- Only the initial handshake appears on the ISP2 packet capture, all replies appear on the ISP1 packet capture
- As I’m using Linux as a client, which supports automatically switching to the last endpoint address it heard from, as the responses are via ISP1 it immediately starts communicating with the ISP1 address even when configured to use ISP2
- In the Connection Tracking table, a connection with the correct connection mark (CM-in-WAN2) is created however it never has any traffic increment the Repl. Bytes column, rather a connection via ISP1 is immediately established
Workarounds
I’ve found if you force the return traffic via the correct Interface with a SRC-NAT rule this will work as expected.
/ip firewall nat
add action=src-nat chain=input comment="Hack to force WG to reply via RT-via-ISP1" dst-address=192.0.2.2 dst-port=13231 log=yes protocol=udp to-addresses=192.0.2.1
add action=src-nat chain=input comment="Hack to force WG to reply via RT-via-ISP2" dst-address=198.51.100.2 dst-port=13231 log=yes protocol=udp to-addresses=198.51.100.1
What this is doing is making the WireGuard process think that packets from ISP1 are coming from the other end of the ISP1 link, and the ISP2 packets are coming from the other end of the ISP2 link, giving it no choice but to follow the best route back to those addresses, which is via the correct ISP uplink.
This doesn’t necessarily have to be a real address, just something the main route table only knows a path to via that connection. You could use a static route and SRC-NAT in combination, for example:
/ip route add
dst-address=1.2.3.4/32 gateway=ether2
/ip firewall nat
add action=src-nat chain=input comment="Hack to force WG to reply via RT-via-ISP2" dst-address=198.51.100.2 dst-port=13231 log=yes protocol=udp to-addresses=1.2.3.4
Glad you submitted the supout.
Did you try putting in an additional rule in-between the two rules, to check counter, Interesting to see if it got any hits.
If so then mangling part works its the routing that is broken…
add action=mark-connection chain=prerouting comment=“Mark incoming connections via ISP1” connection-mark=no-mark in-interface=ether1 new-connection-mark=CM-via-ISP1 passthrough=yes
add action=passthrough chain=output src-port=13231
add action=mark-routing chain=output comment=“Mark routing for marked connections via ISP1” connection-mark=CM-via-ISP1 new-routing-mark=RT-via-ISP1 passthrough=no
So if the conclusion that the mangling output rule is being ignored for some reason.
Lets introduce Routing Rules instead of mangling to see if that has the desired effect.
Easily done if the ISPs provide static IP addresses.
IF they are dynamic one will need to create a script to update the the routing rule with the new WANIP
So in your GN3 Try this…
a. Disable the mangle rules.
b. remove your src nat hacks.
b. add two routing rules
c. keep the defined tables and associated routes as we can re-use them for this attempt!!
add src-address=192.0.2.2 action=lookup-only-in-table table=RT-via-ISP1
add src-address=198.51.100.2 action=lookup-only-in-table table=RT-via-ISP2
/routing table
add fib name=RT-via-ISP1
add fib name=RT-via-ISP2
/ip routes ( distance not required with manual table, unless you have multiple routes within the same table }
add dst-address=0.0.0.0/0 gateway=192.0.2.1 routing-table=RT-via-ISP1
add dst-address=0.0.0.0/0 gateway=198.51.100.1 routing-table=RT-via-ISP2