Multi-wan multi-ip wireguard setup

Hello.

After reading other’s posts I came with this test setup:

/interface bridge add arp=disabled frame-types=admit-only-vlan-tagged name=dum1 protocol-mode=none vlan-filtering=yes
/routing table add disabled=no fib name=bugfix_wg
/ip address add address=172.20.215.1/31 interface=dum1 network=172.20.215.0

/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-36 address-list-timeout=1m chain=input dst-address=X.X.X.36 dst-port=13231 protocol=udp
/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-37 address-list-timeout=1m chain=input dst-address=X.X.X.37 dst-port=13231 protocol=udp
/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-50 address-list-timeout=1m chain=input dst-address=Y.Y.Y.50 dst-port=13231 protocol=udp
/ip firewall filter add action=add-src-to-address-list address-list=WG-VIA-55 address-list-timeout=1m chain=input dst-address=Y.Y.Y.55 dst-port=13231 protocol=udp
/ip firewall filter add action=accept chain=input dst-port=13231 protocol=udp src-port=13231

/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-36 address-list-timeout=1m chain=input dst-address=X.X.X.36 dst-port=13231 protocol=udp
/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-37 address-list-timeout=1m chain=input dst-address=X.X.X.37 dst-port=13231 protocol=udp
/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-50 address-list-timeout=1m chain=input dst-address=Y.Y.Y.50 dst-port=13231 protocol=udp
/ip firewall mangle add action=add-src-to-address-list address-list=WG-VIA-55 address-list-timeout=1m chain=input dst-address=Y.Y.Y.55 dst-port=13231 protocol=udp

/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-55 new-connection-mark=In-Via-55 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-36 new-connection-mark=In-Via-36 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-50 new-connection-mark=In-Via-50 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-connection chain=output dst-address-list=WG-VIA-37 new-connection-mark=In-Via-37 passthrough=yes protocol=udp src-address=172.20.215.1 src-port=13231

/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-36 new-routing-mark=Via-36 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-37 new-routing-mark=Via-37 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-50 new-routing-mark=Via-50 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output dst-address-list=WG-VIA-55 new-routing-mark=Via-55 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-50 new-routing-mark=Via-50 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-55 new-routing-mark=Via-55 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-56 new-routing-mark=Via-56 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-Y new-routing-mark=Via-Y passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-36 new-routing-mark=Via-36 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-37 new-routing-mark=Via-37 passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231
/ip firewall mangle add action=mark-routing chain=output connection-mark=In-Via-X new-routing-mark=Via-X passthrough=no protocol=udp src-address=172.20.215.1 src-port=13231

/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-36 address-list-timeout=1m chain=dstnat dst-address=X.X.X.36 dst-address-type=local dst-port=13231 protocol=udp
/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-37 address-list-timeout=1m chain=dstnat dst-address=X.X.X.37 dst-address-type=local dst-port=13231 protocol=udp
/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-50 address-list-timeout=1m chain=dstnat dst-address=Y.Y.Y.50 dst-address-type=local dst-port=13231 protocol=udp
/ip firewall nat add action=add-src-to-address-list address-list=WG-VIA-55 address-list-timeout=1m chain=dstnat dst-address=Y.Y.Y.55 dst-port=13231 protocol=udp
/ip firewall nat add action=dst-nat chain=dstnat dst-address-type=local dst-port=13231 protocol=udp to-addresses=172.20.215.1
/ip firewall nat add action=accept chain=srcnat protocol=udp src-port=13231

/ip route add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=dum1 pref-src=172.20.215.1 routing-table=bugfix_wg scope=30 suppress-hw-offload=no target-scope=10
/routing rule add action=lookup-only-in-table disabled=no src-address=172.20.215.1/32 table=bugfix_wg

I can not set IP of dum1 bridge to /32 prefix as this somehow breaks BGP.

X.X.X.36 is the IP of WAN1 which is set to packets src generated by wireguard.
Client connects to Y.Y.Y.55 and at first connections the packets come back from Y.Y.Y.55 IP.
But if a ping is made from server (mikrotik) or something initiates a packet from wireguard (ie keepalive), the src IP of the initated packet is changed back to X.X.X.36.

The only real solution working is from @lurker888 with srcnat. But in this case mikrotik does not see the real IP of connected client.

Is there any reliable solution to this wireguard outbound problem?

Sorry no context provided, why are you mangling for example…
Do you have a network diagram

There is no much complication. Several providers which give multiple static IPs. Client expects the packet back from same ip and port to where it sent its packet (this is specially needed if client is behind some firewall). The mangling rules I tried from a post here http://forum.mikrotik.com/t/wireguard-multi-wan-policy-routing/174145/90 where there is a hack with adding clients src to an address list to be used for detecting the outgoing interface and src IP (something like a conntrack). But this does not work from that post, as the wireguard src ip is set before mangling and pref source is not used. Nor a srcnat to wan IP does not work. as it do not preserve the src port. If I srcnat with src port, then it does not work at all.

The initial discussion post from where I relate is here http://forum.mikrotik.com/t/routeros-blatantly-ignores-pref-src-can-this-really-be-a-bug/180360/66

The Mikrotik does. The Wireguard stack running on that Mikrotik doesn’t. Why is it important for you that the Wireguard stack knew the actual address of the peer?

Mimiko read this post please → http://forum.mikrotik.com/t/routeros-blatantly-ignores-pref-src-can-this-really-be-a-bug/180360/66

Well, I want to know which Ip is connected to the each peer. Implementing a banning mechanism of offenders.

If you’re trying to catch hammering offenders, it might be best to use a firewall filter with “psd” or alternatively log the handshake source IP using a script tied to interface monitoring. Just be careful not to lock out legit IPs by mistake. Always have a protected OOB management entrance.

Also, I noticed you’re active in two different threads about the same issue. I think it’s easier for everyone if we keep the discussion in just one thread.

@Mimiko: I’m not clear on what you want to do. Only authenticated users show up in the peers as “Current endpoint address” Why would you want to ban authenticated users? If you really want to ban them, why not just disable the peer?

If you’re afraid of the CPU load that failed authentication attempts create, either use some of the firewall"s limit feature (even without the multiwan stuff, that’s all you can do…) or simply don’t be afraid of it. One of the design goals of wg was to use “cheap” elliptic (well, kind of) negotiation, and even a low-end device is hard to overwhelm with these.

I must ask the question: does your banning setup work for a single wan? How effective is it then?

Thank you for your reply. Banning works for other services. Not for wg. May be you are correct that limiting the the rate is a better solution than banning. For now I do not know yet how to capture from logs or other way the auth failure of the peers. For now I use the client ip to differentiate from where it connected, ie in my example from other thread, when debugging connection I can see how the client’s ip is changed.

Anyway, I will implement your option with srcnat for the moment. But the topic is open for new if something appears in changes of mikrotik.

You probably don’t have to worry about brute forcing wg with repeated connections because:

  • WG uses asymmetric key-based auth (Noise on the 25519 “curve”) and cracking this is not based on repeated auth attempts like for passwords, but it is offline (just as it is for ssh key-based auth). Currently the best attacks are based on Shor’s algorithm.
  • WG only supports signatures based on 25519, which is very fast
  • repeated handshake attempts are only good for DoS, and if a DoS is encountered anyway, WG includes a built-in mechanism for handling it (WG white paper section 5.3 Denial of Service Mitigation & Cookies)
  • this enables even a hAP ac2 to handle 200+ Mbps of failed handshake attempts (at 100Mbps it’s around 25% CPU)

You cannot handle it in the same way as other protocols, and to do so would require modifications of the kernel, because:

  • WG does not reply to failed handshake attempts, and allocates no memory resources for them (WG white paper section 5.1 Silence is a Virtue) and in accordance with this, log entries are not created

Basically rate-limiting new connections based on source IP (to some ridiculous number such as 100/s) is the best you can do, but I think it’s unnecessary in practice.

If I understand your solution correctly, thanks to the src-nat rule in input, the current enpoint address is always 172.16.10.2 (as per http://forum.mikrotik.com/t/routeros-blatantly-ignores-pref-src-can-this-really-be-a-bug/180360/72) .

So to find a list of actual source addresses of failed handshake attempts, one should look for a list of connections that got both dst-nated to 172.16.10.1 and src-nated to 172.16.10.2 but have never seen a reply packet (and scan it twice to filter false positives).

You are absolutely correct that all endpoints will show up as 172.16.10.2.

My point here was that only correctly authenticated connections show up at all - ever - as endpoint addresses. Failed handshake attempts never do. (Even if no multiwan/nat is present.)

Yup sounds familiar and as CGX pointed out we only need to use one LO address/interface to accomplish same… no need for bridge!!
/ip address
add address=10.20.30.40 interface=lo network=10.20.30.40

In Wireguard data, this is correct. In the connection tracking, it’s different - any connection attempt to the Wireguard port that comes from a remote source creates a tracked connection, and even if unresponded (because it uses an incorrect key), it still sits there for udp-timeout (the value set by the default configurations bundled with recent RouterOS versions is 10s), so plenty of time for a scheduled script to find it. Also even if no multiwan/nat is present. I am also not sure about the actual benefits of blacklisting the source addresses of failed handshake attempts (especially in mobile networks that NAT connections from multiple MTs to the same public address from a pool, doing so may even prevent actual peers from connecting), but the possibility does exist.

And yes, it is also possible to use the firewall rules directly and a second address list instead of mining the data from connection tracking using a script:
/ip firewall raw
add chain=prerouting dst-address-type=local protocol=udp dst-port=54345 src-address-list=!wg-responded action=jump jump-target=wg-sort
add chain=wg-sort src-address-list=wg-attempted action=drop
add chain=wg-sort action=add-src-to-address-list address-list=wg-attempted address-list-timeout=1h
add chain=output protocol=udp src-port=54345 action=add-dst-to-address-list address-list=wg-responded address-list-timeout=1h

Does this even belong here? The src-nat rule in input is the key part of @lurker888’s solution, and the above alone is not an equivalent of adding an address from a /30 subnet even if you manually add a route to 10.20.30.41 (or any other address of your choice that would be the to-addresses of that src-nat rule in input) with lo as a gateway.

Because if you add an address from a /30 subnet to lo and ping the other address in that subnet, RouterOS chooses the own address from that subnet as a source; if you add a route to some /32 address via lo and ping that address, RouterOS chooses 127.0.0.1 as a source.

To clarify the source nat address part is STILL required. I think he is saying
add action=dst-nat chain=dstnat connection-mark=wg-wan2 to-addresses=10.20.30.40
add action=src-nat chain=input connection-mark=wg-wan2 to-addresses=10.20.30.40

Today MikroTik updated their WG docs and added this section:

Which seems to use the strategy in @Mimiko's opening post, with some small changes (action=add-src-to-address-list chain=prerouting), and I think the mark-connection and mark-routing rules mistakenly wrote dst-port=13231 instead of src-port=13231.

And they also had to add the disclaimer, about what @Mimiko has already noticed in the 1st post:

For these rules to work as intended you need to enable "responder" option in WireGuard peer settings, as "server" could send handshakes via incorrect interface because routing was not marked.