(This is a cross-post of my analysis and solution from
viewtopic.php?t=197545 to get more eyes on it)
--
I've reproduced this in GNS3, and I'll submit it to MikroTik support as well, but here's the bug we're discussing here.
Network topology
Screenshot 2023-07-06 at 12.14.27.png
Entire (relevant) router config
/interface ethernet
set [ find default-name=ether1 ] comment=ISP1 disable-running-check=no
set [ find default-name=ether2 ] comment=ISP2 disable-running-check=no
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1 private-key="EMfgjAes9lBy7MZaeRfHPnwqW3S3cvRFyFOCxBRVulM="
/port
set 0 name=serial0
/routing table
add fib name=RT-via-ISP1
add fib name=RT-via-ISP2
/interface wireguard peers
add allowed-address=172.16.0.2/32 comment="WNmh78bmcvq155V0Xo8npVFACkQGbcgIIDctPXIkF1A=" interface=wireguard1 public-key="wdyxvVy8ceMPXf5dH7Bfd61Wxo5wKbf/Nx6MCR3+IEM="
/ip address
add address=192.0.2.2/24 interface=ether1 network=192.0.2.0
add address=198.51.100.2/24 interface=ether2 network=198.51.100.0
add address=172.16.0.1/24 interface=wireguard1 network=172.16.0.0
/ip firewall mangle
add action=mark-connection chain=prerouting comment="Mark incoming connections via ISP1" connection-mark=no-mark in-interface=ether1 new-connection-mark=CM-via-ISP1 passthrough=yes
add action=mark-routing chain=output comment="Mark routing for marked connections via ISP1" connection-mark=CM-via-ISP1 new-routing-mark=RT-via-ISP1 passthrough=no
add action=mark-connection chain=prerouting comment="Mark incoming connections via ISP2" connection-mark=no-mark in-interface=ether2 new-connection-mark=CM-via-ISP2 passthrough=yes
add action=mark-routing chain=output comment="Mark routing for marked connections via ISP2" connection-mark=CM-via-ISP2 new-routing-mark=RT-via-ISP2 passthrough=no
/ip route
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=192.0.2.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=20 dst-address=0.0.0.0/0 gateway=198.51.100.1 routing-table=main suppress-hw-offload=no
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=192.0.2.1 pref-src="" routing-table=RT-via-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=198.51.100.1 pref-src="" routing-table=RT-via-ISP2 scope=30 suppress-hw-offload=no target-scope=10
Observed behaviour
Almost all connections that ingress on the router will be marked correctly and will use the correct interface for egress based on the connection mark.
This can be proved as working for ICMP ping, SSH and Winbox.
The WireGuard process however does not appear to take any regard for connection marks and just replies back via the interface with the best route in the
main table.
This can be proved in a number of ways:
- Only the initial handshake appears on the ISP2 packet capture, all replies appear on the ISP1 packet capture
- As I'm using Linux as a client, which supports automatically switching to the last endpoint address it heard from, as the responses are via ISP1 it immediately starts communicating with the ISP1 address even when configured to use ISP2
- In the Connection Tracking table, a connection with the correct connection mark (CM-in-WAN2) is created however it never has any traffic increment the Repl. Bytes column, rather a connection via ISP1 is immediately established
Workarounds
I've found if you force the return traffic via the correct Interface with a SRC-NAT rule this will work as expected.
/ip firewall nat
add action=src-nat chain=input comment="Hack to force WG to reply via RT-via-ISP1" dst-address=192.0.2.2 dst-port=13231 log=yes protocol=udp to-addresses=192.0.2.1
add action=src-nat chain=input comment="Hack to force WG to reply via RT-via-ISP2" dst-address=198.51.100.2 dst-port=13231 log=yes protocol=udp to-addresses=198.51.100.1
What this is doing is making the WireGuard process think that packets from ISP1 are coming from the other end of the ISP1 link, and the ISP2 packets are coming from the other end of the ISP2 link, giving it no choice but to follow the best route back to those addresses, which is via the correct ISP uplink.
This doesn't necessarily have to be a real address, just something the main route table only knows a path to via that connection. You could use a static route and SRC-NAT in combination, for example:
/ip route add
dst-address=1.2.3.4/32 gateway=ether2
/ip firewall nat
add action=src-nat chain=input comment="Hack to force WG to reply via RT-via-ISP2" dst-address=198.51.100.2 dst-port=13231 log=yes protocol=udp to-addresses=1.2.3.4
You do not have the required permissions to view the files attached to this post.