I’m wondering how dnating to onother ip of wan1 will assure that originated from wireguard packets will have src ip of second wan? Or i’m missing something.
Thank you for the links.
Do you have a full example of the specific config to try?
If you want help with your setup post a new thread and will need your traffic requirements and current config.
This is my test setup.
/interface bridge add arp=disabled frame-types=admit-only-vlan-tagged name=dum1 protocol-mode=none vlan-filtering=yes
/routing table add disabled=no fib name=bugfix_wg
/ip address add address=172.20.215.1/31 interface=dum1 network=172.20.215.0
/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-36 address-list-timeout=1m chain=input dst-address=X.X.X.36 dst-port=13231 protocol=udp
/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-37 address-list-timeout=1m chain=input dst-address=X.X.X.37 dst-port=13231 protocol=udp
/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-50 address-list-timeout=1m chain=input dst-address=Y.Y.Y.50 dst-port=13231 protocol=udp
/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-55 address-list-timeout=1m chain=input dst-address=Y.Y.Y.55 dst-port=13231 protocol=udp
/ip firewall filter add action=accept chain=input dst-port=13231 protocol=udp src-port=13231
/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-36 address-list-timeout=1m chain=input dst-address=X.X.X.36 dst-port=13231 protocol=udp
/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-37 address-list-timeout=1m chain=input dst-address=X.X.X.37 dst-port=13231 protocol=udp
/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-50 address-list-timeout=1m chain=input dst-address=Y.Y.Y.50 dst-port=13231 protocol=udp
/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-55 address-list-timeout=1m chain=input dst-address=Y.Y.Y.55 dst-port=13231 protocol=udp
/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-55 new-connection-mark=In-Via-55 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-36 new-connection-mark=In-Via-36 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-50 new-connection-mark=In-Via-50 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-37 new-connection-mark=In-Via-37 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-36 new-routing-mark=Via-36 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-37 new-routing-mark=Via-37 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-50 new-routing-mark=Via-50 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-55 new-routing-mark=Via-55 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-50 new-routing-mark=Via-50 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-55 new-routing-mark=Via-55 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-56 new-routing-mark=Via-56 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-Y new-routing-mark=Via-Y passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-36 new-routing-mark=Via-36 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-37 new-routing-mark=Via-37 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-X new-routing-mark=Via-X passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-36 address-list-timeout=1m chain=dstnat dst-address=X.X.X.36 dst-address-type=local dst-port=13231 protocol=udp
/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-37 address-list-timeout=1m chain=dstnat dst-address=X.X.X.37 dst-address-type=local dst-port=13231 protocol=udp
/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-50 address-list-timeout=1m chain=dstnat dst-address=Y.Y.Y.50 dst-address-type=local dst-port=13231 protocol=udp
/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-55 address-list-timeout=1m chain=dstnat dst-address=Y.Y.Y.55 dst-port=13231 protocol=udp
/ip firewall nat add action=dst-nat chain=dstnat dst-address-type=local dst-port=13231 protocol=udp to-addresses=172.20.215.1
/ip firewall nat add action=accept chain=srcnat protocol=udp src-port=13231
/ip route add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=dum1 pref-src=172.20.215.1 routing-table=bugfix_wg scope=30 suppress-hw-offload=no target-scope=10
/routing rule add action=lookup-only-in-table disabled=no src-address=172.20.215.1/32 table=bugfix_wg
I can not set IP of dum1 bridge to /32 prefix as this somehow breaks BGP.
X.X.X.36 is the IP of WAN1 which is set to packets src generated by wireguard.
Client connects to Y.Y.Y.55 and at first connections the packets come back from Y.Y.Y.55 IP.
But if a ping is made from server (mikrotik) or something initiates a packet from wireguard (ie keepalive), the src IP of the initated packet is changed back to X.X.X.36.
The only real solution working is from @lurker888 with srcnat. But in this case mikrotik does not see the real IP of connected client.
Both dst-nat and src-nat (and masquerade) actions do both a translation that is specified in the rule, and a translation of the packets in the reverse direction. If you enable logging in your firewall you will see this as
NAT realsrcip->(wan2ip->wan1ip)
This means that incoming packets have their dst addr altered from wan2ip to wan1ip, but also the response packets get their src addr altered to wan2ip. What was not explained very clearly in this thread is that this works (only) in case the lookup in the main table would result in the response packet going through wan1, so when emitted has the addressing wan1ip->realsrcip, which conntrack identifies and trnasforms to (wan1ip->wan2ip)->realsrcip.
In my version the incoming transform is (realsrcip->192.168.222.2)->(wan2ip->192.168.222.1) and the reverse translation is (192.168.222.1->wan2ip)->(192.168.222.2->realsrcip)
Well, thank you. I didn’t find having to live with the real IP being masked as particularly onerous. WG packets are authenticated, and in case you want to do filtering in the firewall, the matchers have access to the pre-translation source ip.
If you meet the criteria that I’ve explained in the previous post about “main” routes and addresses not being lost, sindy’s version works absolutely fine with leaving the source address in place. Of course only if correctly implemented 
Public addresses are static, but providers may disconnect of some power outage or any other problem and reconnect back. Mostly it is constant.
I do not find any of Sindy’s post related to this problem. Could you pinpoint it?
Well there you have it, if the interface goes down, you lose the address.
Sindy’s method was discussed in a similarly long thread. I don’t think any ready-made solution/config was available beyond what was discussed here. Correctly applying these requires a better than usual understanding of several of the interrelated mechanisms at play.
I saw the address-list based approach you offer for discussion in the other thread on this forum. I don’t really get it.
@Mimiko, your only options are either routing rules or NAT, unless you’re on one of the newer ROS versions where they supposedly fixed the issue, i.e. that WireGuard’s initial handshake is going out via the default gateway instead of the inbound interface. Might’ve been mentioned in this or the other big thread on the topic. We’re running v7.15 ourselves and use routing rules for multi-WAN WireGuard.
The idea of the poster behind the address list is to mimic connection tracking for wireguard. Ie. First connection comes to IP2 then put the client address in the address list for IP2. When wireguard respond, catch that packet and route via desired outbound gateway using routing tables and put the pref-source in header. But this does not work. I tried to use srcnat to IP2 if packet originates to dst IP of client in the address list, and this changes the src IP to IP2 as desiret, but also changes the src port of the packet to some random. I tried to put also src port same port to force the src ip and port, but then no packet will leave the router. This is a bug or its how it works I don’t know. Anyway, address lists idea is to mimic connection tracking for wireguard. Setting timeout when adding src ip of client to address lists to same timeout of UDP packet in connection tracking will behave same with the timeout occurs, so a keepalive will always keep the address in the list.
@Mimiko, I found the post.
Turns out it was @lurker888 who mentioned in this thread (post #23) that the issue was fixed in v7.17. So if that’s true, you should be able to sort out your issues the usual way with mangle and routing marks.
Lurker, gone down many a rabbit hole, I cannot seem to work my way through the noise of your solution…
Context: Two WANS, WAN1 primary, and WAN2 secondary and wishing to use WAN2 as the wireguard connection.
If given a faux bridge 192.168.66.0/32 address
and given a listening port of 55555, WAN1IP = 24.X and WAN2IP = 24.y
What is your proposal for
a. NAT Rules
B. Mangle rules
C. IP routes???
We cannot interfere with other traffic on WAN2, https, DNS queries etc that may need to work as well, especially if WAN1 goes down.
I was planning on writing out an example, so just for your reading pleasure: the official “rabbit hole solution” to multi-wan wireguard hosting.
We have two WANS. For this example, the two WAN connections are actually EoIP tunnels to another router. The base configuration is:
Interfaces:
/interface eoip
add name=eoip-wan1 remote-address=[other_router] tunnel-id=10
add name=eoip-wan2 remote-address=[other_router] tunnel-id=20
Addresses:
/ip address
add address=192.168.111.10/24 interface=eoip-wan1
add address=192.168.112.10/24 interface=eoip-wan2
Routes:
/routing table
add fib name=wan2
/ip route
add distance=1 dst-address=0.0.0.0/0 gateway=192.168.111.1
add distance=20 dst-address=0.0.0.0/0 gateway=192.168.112.1
add disabled=no dst-address=0.0.0.0/0 gateway=192.168.112.1 routing-table=wan2
WAN masquerade:
add action=masquerade chain=srcnat out-interface=eoip-wan1
add action=masquerade chain=srcnat out-interface=eoip-wan2
And a wireguard “server” with a single peer: (The other side has a 10s persistent keepalive configured.)
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1
/interface wireguard peers
add allowed-address=192.168.50.0/24 interface=wireguard1 name=peer1 public-key="22SMLvYh7cQzj3/sGFeFLn7l0xVePXM59DLms3qnNUY=" responder=yes
And now the interesting bit.
Create a bridge with a /30 address:
/interface bridge
add name=br-wg protocol-mode=none
/ip address
add address=172.16.10.1/30 interface=br-wg
Mark connections according to their origins, add routing for wan2 and nat them:
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address=192.168.111.10 dst-port=13231 in-interface=eoip-wan1 new-connection-mark=wg-wan1 protocol=udp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address=192.168.112.10 dst-port=13231 in-interface=eoip-wan2 new-connection-mark=wg-wan2 protocol=udp
add action=mark-routing chain=output connection-mark=wg-wan2 new-routing-mark=wan2
/ip firewall nat
add action=dst-nat chain=dstnat connection-mark=wg-wan1 to-addresses=172.16.10.1
add action=dst-nat chain=dstnat connection-mark=wg-wan2 to-addresses=172.16.10.1
add action=src-nat chain=input connection-mark=wg-wan1 to-addresses=172.16.10.2
add action=src-nat chain=input connection-mark=wg-wan2 to-addresses=172.16.10.2
A packet capture:
(The capture is done on the opposing router where the corresponding interfaces are eoip-test1 and eoip-test2)
# TIME INTERFACE SRC-ADDRESS DST-ADDRESS IP-PROTOCOL SIZE CPU
0 3.367 eoip-test1 192.168.33.1:13231 192.168.111.10:13231 udp 190 0
1 3.373 eoip-test1 192.168.111.10:13231 192.168.33.1:13231 udp 134 1
2 3.374 eoip-test1 192.168.33.1:13231 192.168.111.10:13231 udp 74 1
3 13.542 eoip-test1 192.168.33.1:13231 192.168.111.10:13231 udp 74 1
4 23.784 eoip-test1 192.168.33.1:13231 192.168.111.10:13231 udp 74 1
5 47.754 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 190 0
6 47.761 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 134 1
7 47.761 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 2
8 58.342 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 2
This shows a connection first over wan1, then over wan2. The handshake initiations are the 190 byte packets, size 134 are the handshake responses and the 74 byte ones are keepalives. All is well.
A ping test was then conducted over the wg tunnel. The pings are 3s apart, and during the test, the wan1 connection was severed for about 10s.
9 63.476 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
10 63.476 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
11 66.484 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
12 66.485 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
13 69.479 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
14 69.48 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
15 72.479 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
16 72.48 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
17 75.486 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
18 75.487 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
19 78.486 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
20 78.487 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
21 81.495 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
22 81.496 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
23 84.498 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
24 84.499 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
25 87.502 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
26 87.503 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
27 90.499 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
28 90.5 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
29 93.506 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
30 93.507 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
31 96.512 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
32 96.513 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
33 99.51 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
34 99.511 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
35 102.512 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 2
36 102.513 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
37 112.742 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 2
38 112.742 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 2
39 122.982 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 2
40 133.222 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 2
41 143.462 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 2
42 153.702 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 2
43 163.942 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 2
44 174.182 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 2
Traffic flows without interruption. (This is the problem with dst-natting to the primary wan address, there the traffic is interrupted if the address is lost.)
If you would also want to initiate wg connections from the router side and do it over wan2, even if wan1 is available, this is the additional rule you’ll need: (Traffic may be interrupted for at most keepalive time when wan1 goes down.)
/ip firewall mangle
add action=mark-connection chain=output connection-mark=no-mark new-connection-mark=wg-wan2 protocol=udp src-port=13231
So in your example you have to manipulate both wans, not just wan2??
I don’t exactly get what you mean by “manipulate”.
In the example I treat the WANs in a symmetric manner. We don’t have to. If we remove everything that is related to the wg-wan1 mark, so these:
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address=192.168.111.10 dst-port=13231 in-interface=eoip-wan1 new-connection-mark=wg-wan1 protocol=udp
/ip firewall nat
add action=dst-nat chain=dstnat connection-mark=wg-wan1 to-addresses=172.16.10.1
add action=src-nat chain=input connection-mark=wg-wan1 to-addresses=172.16.10.2
Then the packet capture looks like this: (same scenario repeated with the above rules disabled)
# TIME INTERFACE SRC-ADDRESS DST-ADDRESS IP-PROTOCOL SIZE CPU
0 13.332 eoip-test1 192.168.33.1:13231 192.168.111.10:13231 udp 190 1
1 13.339 eoip-test2 192.168.111.10:13231 192.168.33.1:13231 udp 134 1
2 13.34 eoip-test1 192.168.33.1:13231 192.168.111.10:13231 udp 74 0
3 23.92 eoip-test1 192.168.33.1:13231 192.168.111.10:13231 udp 74 0
4 34.16 eoip-test1 192.168.33.1:13231 192.168.111.10:13231 udp 74 0
5 51.177 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 190 1
6 51.183 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 134 1
7 51.184 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 1
8 61.69 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 1
9 71.92 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 1
10 73.467 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 1
11 73.467 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
12 76.476 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 1
13 76.476 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
14 79.478 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 1
15 79.479 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
16 82.483 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 1
17 82.484 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
18 85.489 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 1
19 85.49 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
20 88.487 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 1
21 88.487 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
22 91.487 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 1
23 91.488 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
24 94.484 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 138 1
25 94.485 eoip-test2 192.168.112.10:13231 192.168.33.1:13231 udp 138 1
26 104.56 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 1
27 114.8 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 1
28 125.05 eoip-test2 192.168.33.1:13231 192.168.112.10:13231 udp 74 1
So… exactly the same.
I don’t know who those three measly lines would hurt, but if someone likes it better like this, I’m all for it.
It goes to root reason. As I stated, WAN1 being primary WAN2 secondary wanting to use WAN2 for wireguard.
We only need to mangle for WAN2 and the problem was the router was sending return traffic via WAN1…
Thus we dsnatted to fool router to send traffic back out WAN2…
You pointed out the weakness of this approach when WAN1 Fails… and by using a faux bridge vice WAN1 in dstnat.
Thus in my mind I was not expecting any work for wan1…
I prefer to keep it simple, like adding EOIP to the mix is a distraction like throwing smoke grenades into the explanation 
Your level of knowledge is far more comfortable and thus probably assume we know or see what you know or see as easily, not the case at least for me but appreciate the explanations and efforts!!
PS the peer settings were either wrong or incomplete.
If the other end is a router with a subnet,192.168.50.0/24, then one is missing the /32 address of the other ends wireguard address
OR
If the other end is simply a device, then the allowed address should be 192.168.50.2/32 ( assuming router wg ip is 192.168.50.1)
My problem is not properly understanding connection tracking, Nothing more you can do LOL.
At least I kind of grasp your use of faux bridge and how traffic gets there, its after, the response traffic and mangle and routing that eludes me completely.
In my reply I pointed out that you don’t have to do anything to wan1 traffic for everything to work out.
I often do configurations for scenarios where all participating networks are in VRFs (and some or all of them via virtual links) so I tend to orient my solutions so that everything is the same for all of them; this way there are no “special cases” to have to keep in mind. Everyone’s usual application and point of view will of course be different. (One of the nice things about Mikrotik stuff is that they are useful in an unexpected variety of situations.)
EoIP is a perfectly cromulent ethernet interface. It’s far easier to test things when you control both sides, plus it’s always repeatable.
Actually the peer settings are exactly as given. (Generated by “export”.)
How to manage the traffic inside the tunnels is outside the scope of the configuration given, but here is how it’s done in this particular example:
An address is assigned on the router discussed:
/ip address
add address=192.168.50.2/24 interface=wireguard1
On the other side:
/interface wireguard
add listen-port=13231 mtu=1420 name=wg-test
/interface wireguard peers
add allowed-address=192.168.50.0/24 endpoint-address=192.168.112.10 endpoint-port=13231 interface=wg-test name=peer4 persistent-keepalive=10s public-key="+XbZ8kxNGw2i6my6hOzSK3Ih2VMH03gjzckpr4+5cj0="
/ip address
add address=192.168.50.1/24 interface=wg-test
And the ping is done from the initiator side:
/tool/ping address=192.168.50.2 interval=3
I’ll try to explain. It may be useful, my apologies if you don’t find it so. It still might be nice for someone else encountering the same problem to have it written down somewhere.
The first thing is that actually the normal way of “apply connection mark, add a routing table, mark routing on output” thing would work just fine with wireguard. The problem is that the source address of the outgoing packet is determined using the “main” routing table, and thus conntrack can’t identify the connection (thus treating it as new, and the connection mark is therefore “not there”). If we disregard this and force the routing table, even though the table selection and the egress interface will be correct, the source address will still be incorrect.
Connection tracking (for general udp flows - which wireguard is) looks only at the src/dst addresses and ports. So the steps are the following for a connection initiated from 192.168.33.1 to 192.168.112.10, as in the packet captures.
- we use dst-nat to change the dst address to 172.16.10.1; this also means that conntrack will expect the reply packets to come from 172.16.0.1, and will change the src address on the reply packet during forwarding to the original one (for wan2 this is 192.168.112.10).
- we use input src-nat to change the src address to 172.16.10.2; this means that conntrack will expect the reply packets to be addressed to 172.16.10.2, and will change the dst address on the reply packets to the original (in our example 192.168.33.1)
Now onto wg.
- wg thus receives the incoming packets from 172.16.10.2 (as translated by the input src-nat) therefore it accepts this as “current endpoint address” for the peer and so the dst address of the outgoing packets will be this
- for determining the src address, a lookup in the “main” table is done, and a route to 172.16.10.0/30 is found (added dynamically by the address assignment), and for this the appropriate src address is 172.16.10.1 - thus this is selected
Back to conntrack:
- the packet - as seen by conntrack - is from 172.16.10.1 to 172.16.10.2
- this fits the expectation for a reply packet as discussed above and the connection is identified: state is set to established, and the connection mark is also correctly set
- the src address - as described above - is translated to the original (for wan2, 192.168.112.10)
- the dst address - as described above - is translated to the original (in our example, 192.168.33.1)
We have correct addressing, we have the connection mark, we’re happy. Packet marking for routing, routing table and egress interface selection can now proceed in the usual way.
My bad that is valid, but this is assuming the remote router is an MT router. ( client peer for handshake) … makes sense, so other peers connecting to the local router can easily re-enter the tunnel and reach the remote router via the local router, so to speak.
The local router needs
allowed IPs=192.168.50.1/32 You cant have both ends with allowed addresses at 192.168.50.0/24 LOL
+++++++++++++++++++++++++++++++++++++
Hmm interesting comment:
The first thing is that actually the normal way of “apply connection mark, add a routing table, mark routing on output” thing would work just fine with wireguard. The problem is that the source address of the outgoing packet is determined using the “main” routing table, and thus conntrack can’t identify the connection (thus treating it as new, and the connection mark is therefore “not there”). If we disregard this and force the routing table, even though the table selection and the egress interface will be correct, the source address will still be incorrect.
Makes me wonder if we shouldn’t simply
add dst-address=RemoteWANIP gateway=WAN2 table=main distance=2
add dst-address=0.0.0.0/0 gateway=WAN1 table=main distance=4
add dst-address=0.0.0.0/0 gateway=WAN2 table=main distance=6
Fix our problem LOL if we had flexibility in what entry arguments were for destination address.
++++++++++++++++++++++++
The first crux of my lack of understanding is exactly here, everything above I followed…
- for determining the src address, a lookup in the “main” table is done, and a route to 172.16.10.0/30 is found (added dynamically by the address assignment), and for this the appropriate src address is 172.16.10.1 - thus this is selected
I simply dont understand this process, where does this rule come from? Is this some basic networking stuff I would know if I actually studied IT and networking??
Why is the table or routing even in the picture for this decision???.. since the destination packets were sent to 172.16.10.1 wouldnt that automatically mean the that address would be the source address of reply packets… Since this is not simply user/device to user/device, and is ROUTER traffic OWN traffic, I do not understand how it works.
I do understand that conntrac sees 172.16.10.1 sending traffic to 172.16.10.2
What I dont understand is why the connection marks on any of this traffic was removed…
What I dont understand is how the router can magically put them back on without admin direction…
Where was conntrack involved in the process of changing the addresses>> Conntrac is not involved after DSTNAT in prerouting its before, and of course before the next block of input srcnat.
(as per packet flow diagram)
Connection tracking is only seen again in output just before mangle output which I suppose is where we move that traffic out WAN2 with the connection marks and route mark.
I do understand the concepts of un-dst and un-src so that we have a traffic flow that now resembles 192.168.112.10 to 192.168.33.1.
I do understand that the original dst port (listening port on local device) and original src port ( listening port from remote device) were maintained throughout the process and need no shenanigans.
The other thing that is not clear to me is how this fools the router at all vis-a-vis avoiding WAN1. In SINDYs rule we fool the router in the last step, by getting the router to un-dsnat out wan2 vice WAN1.
We do no such thing in your example. We piddle around a faux bridge, which is mainly to avoid the scenario if WAN1 dies. but I dont see how we avoid WAN1 in main table, which SIndys rule addresses.
As described, the side whose configuration was presented has this peer entry, and this address (and the dynamic route it adds):
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1
/interface wireguard peers
add allowed-address=192.168.50.0/24 interface=wireguard1 name=peer1 public-key="22SMLvYh7cQzj3/sGFeFLn7l0xVePXM59DLms3qnNUY=" responder=yes
/ip address
add address=192.168.50.2/24 interface=wireguard1
The other side has this:
/interface wireguard
add listen-port=13231 mtu=1420 name=wg-test
/interface wireguard peers
add allowed-address=192.168.50.0/24 endpoint-address=192.168.112.10 endpoint-port=13231 interface=wg-test name=peer4 persistent-keepalive=10s public-key="+XbZ8kxNGw2i6my6hOzSK3Ih2VMH03gjzckpr4+5cj0="
/ip address
add address=192.168.50.1/24 interface=wg-test
Nothing wrong with it and works fine. Both sides can ping each other, transit traffic, etc.
Using a /32 allowed address on one end and a wider subnet on the other is a common config. So is using /32 on both. So is using the same /30 on both.
Absolutely correct. You don’t even have to have a smaller distance for the /32 route - longer prefixes are always preferred, regardless of distance. If you know the peer’s address is advance this is actually fine to do.
For normal services (such as DNS over udp, OpenVPN over udp, etc.) you are correct and the source address is where the request was sent to. Wireguard follows a different architecture, and leaves the source address to be assigned by the initial output routing stage. (It is actually done to allow easy peer roaming.)
My post with the full description:
http://forum.mikrotik.com/t/mangle-policy-based-routing/181586/6
To quote:
Packets don’t carry connection marks (it’s not in the IP packet), they are assigned by conntrack. (There are some exceptions.) The packet output by wireguard has no marks whatsoever on it initially. The packet goes:
[emitted by wg] → initial routing decision → conntrack → mangle output → final routing decision (routing adjustment) → and so on
The packet is annotated with the connection mark in the conntrack phase. Until then, there is no associated connection mark. (On normal linux, wg interfaces have a property fwmark, which allows all packets emitted by wg to be marked on creation - this is not currently available on Mikrotik.)
Again, packets on interfaces (ethernet, virtual interfaces, all of them) come into the router “naked”. Conntrack labels them based on the conntrack entry it associates with them. When a mangle assigns a connection mark, it is noted in the conntrack entry the packets nf metadata references.
Again, dst-nat modifies the conntrack entry. That’s its primary purpose. The translation is done immediately to be in line with the (freshly modified) conntrack entry.
Un-dstnat only changes the address. The interface is decided by routing. In mangle output a routing mark “wan2” is applied based on the connection mark “wg-wan2”, and the output interface is determined in the “routing adjustment” phase based on the routing mark “wan2” to be the wan2 interface.
(I see what you are doing with wireguard just dont agree with it. There is no case where both sides of a connection need 50.0/24 that I can see.)
Regarding the contrack and wireguard and dual WAN etc… I approached it from a different angle so it makes sense to me.
The initial problem before ANY fixes is that The outbound traffic from the router, or should I say source address the router was selecting for the return traffic was the WAN1 IP. So our mangling was working but really the problem was not the mangling, it was the fact that the wrong source IP address was on the return traffic from the wireguard processing itself and thus got rejected by the peer client.
I dont really understand how the router selects source of outbound traffic from the router itself but it would appear to be based on what interfaces and their ip routes are available at the main table…
The sindy dstnat rule simply ensured that at the last minute although the router chose wan1IP for the return outbound source, it was undstnatted to wan2 IP and thus the path (route) like before was correct and this time the source address was what was expected by the peer client.
Your solution is similar, we still need the mangle to ensure the path is correct but you decided to use a nasty trick of faux address to avoid the WAN1 problem (if becomes unavailable and thus fails with the above dsntat rule )…
So We have to ensure the source address of the traffic leaving the wireguard handshake is something else, not the chosen WAN1 by the router. We don’t dstnat the traffic heading for WAN2 with WANIP1, we dstnat to an existing but unused will not go away/fail unused BRIDGE 172.16.10.1 ip address. We also ensure we sourcenat the traffic going to the wireguard process, from the WANIP of the remote router, with another address on the Bridge → 172.168.10.2, forcing the wireguard process to use a reply dst address of 172,168.10.2
Previously the router, for wireguard source address outbound, would choose the best interface ( and thus source address) based on the optimal available route on the main table, and that was WAN1 but we have now an available dynamic accurate route which is preferable for 172.16.10.2 ( not sure if its because the interface and dst-address 172.16.10.2 are related or due to prefix differenence from WAN1 ) and the router chooses 172.16.10.1 for source and off it goes… In summary, we now have traffic coming from the wireguard process with destination address of 172.16.10.2 and with source address of 172.16.10.1
Thus mission accomplished we have a source address coming from the wireguard process that is not WAN1, and the router sees it as 172.16.10.1 and un-dstnats that back to the( the original destination address, WANIP 2 of the local router ( which is now the new source address ). Now the remote router will see the correct source address on the return traffic it sees on the handshake. 172.16.10.1 becomes 192.168.112.10 ( in your examples). At the same time, the Router un-srcnats the destination traffic reply IP from wireguard process ( 172.16.10.2) to the original incoming source address of 192.168.33.1 ( the new destination address ).
Traffic leaves from WAN2 due to our mangle output routing-mark, and traffic leaves the router with source address of WAN2.
+++++++++++++++
Notes: Helpful → For normal services (such as DNS over udp, OpenVPN over udp, etc.) you are correct and the source address is where the request was sent to. Wireguard follows a different architecture, and leaves the source address to be assigned by the initial output routing stage. (It is actually done to allow easy peer roaming.)
Notes: Helpful → Something I keep forgetting too: Absolutely correct. You don’t even have to have a smaller distance for the /32 route - longer prefixes are always preferred, regardless of distance. If you know the peer’s
Notes: Helfpul, so the un-dst nat takes place in ROUTING ADJUSTMENT and the unsourcenat takes place in post-routing**??**>>
[emitted by wg] → initial routing decision → conntrack → mangle output → final routing decision (routing adjustment) → and so on