Any advice for further debugging handshaking failed on wireguard roadwarrior setup?

[divB]

Ok, I have a simple Wireguard road warrier setup and it drives me NUTS!! I am always getting handshake failed messages from the client and after hours I figured out at least what it has to do with: Two uplinks. But from here, nothing makes sense.

Setup: RouterOS router with 192.0.2.210/28 on vlan4 and two wireguard uplinks with IPs 192.0.2.249/31 and 192.0.2.253/31. All IPs are public. I contact the wireguard endpoint via 192.0.2.210:51820 so it doesn’t matter over which uplink the connection comes.

If I disable uplink 192.0.2.253/31, everything works as expected and I can establish a successful connection via an iPhone (and ping through the tunnel). However, if both uplinks are up, I get the handshake failed on the iPhone client. The package comes in via 192.0.2.249 interface, and, according to routing table, should go out via 192.0.2.253. Not that this vanilla IP routing with redundant links and no NAT involved! That can’t cause any headaches (and doesn’t for anything else I am running).

Now comes the frustrating part: When I start the packet sniffer (only enabling address 192.0.2.210, udp, port 51820 as filter): I get:

83 time=747.72 num=84 direction=rx interface=wg-bg1-ftth src-address=212.95.5.23:2302 dst-address=192.0.2.210:51820 protocol=ip ip-protocol=udp size=156 cpu=3 
   ip-packet-size=156 ip-header-size=20 dscp=0 identification=18658 fragment-offset=0 ttl=48

That’s it! There is NO CORRESPONDING TX!! That’s insane! Now compare this to disabling the 192.0.2.253 uplink:

75 time=665.894 num=76 direction=rx interface=wg-bg1-ftth src-address=212.95.5.23:2302 dst-address=192.0.2.210:51820 protocol=ip ip-protocol=udp size=156 cpu=2 
   ip-packet-size=156 ip-header-size=20 dscp=0 identification=14945 fragment-offset=0 ttl=48 

76 time=665.895 num=77 direction=tx interface=wg-bg1-ftth src-address=192.0.2.210:51820 dst-address=212.95.5.23:2302 protocol=ip ip-protocol=udp size=156 cpu=2 
   ip-packet-size=156 ip-header-size=20 dscp=0 identification=36904 fragment-offset=0 ttl=64

In this case, the response packet is generated. So nothing is wrong per se and nothing is wrong with my packet sniffer filter. It seems RouterOS wireguard just doesn’t send a return packet all of a sudden!!

• I have “chain=input action=accept protocol=udp dst-port=51820” firewall rule
• There is nothing in the logs
• I tcpdump for “host 192.0.2.210, udp, port 51820” on both uplink tunnel endpoints and can confirm that RouterOS does not send a return packet

What the heck could possibly go wrong here? How can I debug further? For example, why is there no log about wireguard?

[anav]

You know the drill
Post the config of the MT router ( assuming its acting as peer Server for handshake )?

Since you have a public IP this should be fairly easy to fix.
If you have multiple WANs, then provide a detailed diagram for clarity as well as config.

/export file=anynameyouwish (minus router serial number, any public WANIP information, keys etc.)

[rplant]

Hi,
I am finding it difficult to understand what you are saying, but I will assume the mikrotik has 3 public ip addresses on it.
And the incoming wg packets all come into the mikrotik via either the 2.249 or 2.253 interfaces but directed at the .210 address.

Normally for this, I would use routing rules to send the return packets out the correct port based on source address.
But in this case the source address is always .210

I think in this case you can use a dst-nat rule that should (maybe) work.

From: Wireguard in 2nd WAN - #18 by sciensys

/ip firewall nat
chain=dstnat dst-address-type=local in-interface=WAN2 protocol=udp dst-port=wg-port action=dst-nat to-addresses=ip.of.wan.1

WAN2 has the higher route metric.

and from

WireGuard Multi-WAN Policy Routing

I have a conceptual question regarding WireGuard in a multi-WAN environment using dynamic addresses. Problem: in ROS, when a passive WireGuard peer receives its initial handshake (i.e., when connection-state = new), the state machine doesn’t keep track of either the destination address or the inbound interface. Consequently, the peer sends its handshake reply through the default gateway which breaks functionality in a multi-WAN environment where the ingress and egress addresses must match. This…

Perhaps the following, (sorry not tested).

/ip firewall nat
chain=dstnat dst-address-type=local in-interface=WAN2 protocol=udp dst-port=wg-port action=dst-nat to-addresses=ip.of.wan.2

Then some routing rules.

/routing rule
add action=lookup comment="min-prefix=0, all except 0.0.0.0/0" disabled=no min-prefix=0 table=main
add action=lookup comment="return path for wg/openvpn into WAN2" disabled=no src-address=ip.of.wan.2 table=viawan2

rest fall through to default behaviour.

Then a route for viawan2.

/route
add comment="viawan2" disabled=no distance=50 dst-address=0.0.0.0/0 gateway=192.?.?.? routing-table=viawan2

[wiseroute]

@rplant

If I disable uplink 192.0.2.253/31, everything works as expected and I can establish a successful connection via an iPhone (and ping through the tunnel). However, if both uplinks are up, I get the handshake failed on the iPhone client. The package comes in via 192.0.2.249 interface, and, according to routing table, should go out via 192.0.2.253. Not that this vanilla IP routing with redundant links and no NAT involved! That can’t cause any headaches (and doesn’t for anything else I am running).

that is a correct behavior from the wireguard to do rpf check (anti spoofing) before sending you handshake reply - as any other session based connection.

try having lamp stack with ssl, or build ipsec using your scenario - and see what happens.

that is why it is not recommended to put any session based servers in a multi homing nic - just to make us easier to track the session.

if you insist in multi homing Environment - then get yourself ready to make some ip route rule marking etc.

[divB]

@anav: I replaced my public /24 with 192.0.2/24 and keys/other public IPs/secrets with asterisks.

Not sure how easy to digest this full blown config is but as I said, the gist is:

• ISP (with dynamic IP) is vlan2/FTTH
• Two wireguard tunnels wg-bg1-ftth and wg-bg2-ftth which establish over this FTTH connection over public /31 networks and run OSPF and iBGP.
• Routing rule that handles everything with source address 192.0.2.0/24 in table default_myas. Those default routes are dynamically inserted via iBGP.
• vlan44 with public IP 192.0.2.210/28. This is used as “Wireguard Server”
• The wireguard road warrier interface/peer is wg-mobile/miPhone
• My main issue is that I just don’t get additional debug info (like log entries) but I can assert that ROS does not send wireguard TX packets for wg-mobile at all once both wg-bg1-ftth and wg-bg2-ftth are up (but does so, if wg-bg-ftth is down)
  
  
  

# 2024-11-25 10:54:47 by RouterOS 7.15.3
# software id = 
#
/interface bridge
add name=br-main vlan-filtering=yes
add arp=disabled fast-forward=no name=dum0 protocol-mode=none
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
/interface wireguard
add listen-port=13231 mtu=1420 name=wg-bg1-ftth
add listen-port=13232 mtu=1420 name=wg-bg2-ftth
add listen-port=51820 mtu=1420 name=wg-mobile
/interface vlan
add comment=ADM interface=br-main name=vlan1 vlan-id=1
add comment=FTTH interface=br-main name=vlan2 vlan-id=2
add comment=LAN interface=br-main name=vlan3 vlan-id=3
add comment=SRV interface=br-main name=vlan4 vlan-id=4
add comment=ADU interface=br-main name=vlan10 vlan-id=10
add comment=DEV interface=br-main name=vlan11 vlan-id=11
add comment=DEVP interface=br-main name=vlan12 vlan-id=12
add comment=DMZ44 interface=br-main name=vlan44 vlan-id=44
/interface vrrp
add interface=vlan1 name=vrrp1
add interface=vlan3 name=vrrp3 sync-connection-tracking=yes vrid=3
add interface=vlan4 name=vrrp4 vrid=4
add interface=vlan11 name=vrrp11 vrid=11
add interface=vlan12 name=vrrp12 vrid=12
add interface=vlan44 name=vrrp44 vrid=44
/interface list
add name=LAN
add comment="Admin Interfaces" name=ADM
add name=DMZ
add name=SRV
add name=DEV
add name=DEVP
add name=WAN
add include=ADM,DEV,DEVP,DMZ,LAN,SRV name=trusted
add name=peers
add name=WANMYAS
add name=ISP
add name=MACHINES
/routing ospf instance
add disabled=no name=ospf-instance-1 redistribute=static router-id=192.0.2.210
/routing ospf area
add disabled=no instance=ospf-instance-1 name=ospf-area-1
/routing table
add disabled=no fib name=default_myas
add disabled=no fib name=default_isp
add comment="Dummy table, acts only as a routing mark" disabled=no fib name=default_isp_only
/interface bridge port
add bridge=br-main interface=ether1
add bridge=br-main interface=veth1 pvid=3
/ip firewall connection tracking
set enabled=yes
/ip neighbor discovery-settings
set discover-interface-list=none lldp-med-net-policy-vlan=1
/interface bridge vlan
add bridge=br-main tagged=br-main,ether1 vlan-ids=2
add bridge=br-main tagged=br-main,ether1 vlan-ids=3
add bridge=br-main tagged=br-main,ether1 vlan-ids=4
add bridge=br-main tagged=br-main,ether1 vlan-ids=10
add bridge=br-main tagged=br-main,ether1 vlan-ids=11
add bridge=br-main tagged=br-main,ether1 vlan-ids=12
add bridge=br-main tagged=br-main,ether1 vlan-ids=44
add bridge=br-main tagged=br-main untagged=ether1 vlan-ids=1
add bridge=br-main tagged=br-main,ether1 vlan-ids=33
/interface detect-internet
set internet-interface-list=static lan-interface-list=static wan-interface-list=static
/interface list member
add interface=vlan3 list=LAN
add interface=vlan1 list=ADM
add interface=vlan2 list=WAN
add interface=vlan44 list=DMZ
add interface=wg-bg1-ftth list=WAN
add interface=wg-bg2-ftth list=WAN
add interface=vrrp3 list=LAN
add interface=vlan4 list=SRV
add interface=wg-bg1-ftth list=peers
add interface=wg-bg2-ftth list=peers
add interface=vlan11 list=DEV
add interface=vlan12 list=DEVP
add interface=wg-bg1-ftth list=WANMYAS
add interface=wg-bg2-ftth list=WANMYAS
add interface=vlan2 list=ISP
add interface=vlan33 list=ISP
add interface=vlan11 list=MACHINES
add interface=vlan12 list=MACHINES
add interface=vrrp44 list=DMZ
add interface=vrrp1 list=ADM
add interface=vrrp12 list=DEVP
add interface=vrrp4 list=SRV
add interface=vrrp11 list=DEV
add interface=vrrp11 list=MACHINES
add interface=vrrp12 list=MACHINES
/interface wireguard peers
add allowed-address=0.0.0.0/0,::/0 endpoint-address=*.*.*.* endpoint-port=51821 interface=wg-bg2-ftth name=bgate2-ftth persistent-keepalive=5s public-key=“********************************************”
add allowed-address=0.0.0.0/0,::/0 endpoint-address=*.*.*.* endpoint-port=51821 interface=wg-bg1-ftth name=bgate1-ftth persistent-keepalive=5s public-key=“********************************************”
add allowed-address=10.2.33.10/32 client-address=10.2.33.10/32 interface=wg-mobile name=miPhone public-key=“********************************************”
/ip address
add address=10.2.1.2/24 interface=vlan1 network=10.2.1.0
add address=10.2.4.2/24 interface=vlan4 network=10.2.4.0
add address=10.2.79.2/24 interface=vlan3 network=10.2.79.0
add address=192.0.2.177/31 interface=wg-bg2-ftth network=192.0.2.176
add address=192.0.2.186/29 interface=wg-bg1-ftth network=192.0.2.184
add address=192.0.2.210/28 interface=vlan44 network=192.0.2.208
add address=172.20.215.132 interface=dum0 network=172.20.215.132
add address=10.2.80.2/24 interface=vlan10 network=10.2.80.0
add address=192.168.222.2/24 interface=vlan11 network=192.168.222.0
add address=192.168.223.2/24 interface=vlan12 network=192.168.223.0
add address=10.2.79.254 interface=vrrp3 network=10.2.79.254
add address=10.2.33.1/24 interface=wg-mobile network=10.2.33.0
add address=192.168.223.254 interface=vrrp12 network=192.168.223.254
add address=192.168.222.254 interface=vrrp11 network=192.168.222.254
add address=10.2.1.254 interface=vrrp1 network=10.2.1.254
add address=10.2.4.254 interface=vrrp4 network=10.2.4.254
add address=192.168.5.102/24 interface=vlan33 network=192.168.5.0
add address=192.0.2.222 interface=vrrp44 network=192.0.2.222
/ip dhcp-client
add add-default-route=no interface=vlan2 script=":if (\$bound=1) do={\
    \n    /ip route set [find where comment=\"default_isp\"] gateway=\$\"gateway-address\" disabled=no\
    \n    } else={\
    \n    /ip route set [find where comment=\"default_isp\"] disabled=yes\
    \n    }\r\
    \n\r\
    \n" use-peer-ntp=no
/ip firewall address-list
add address=10.2.0.0/16 list=own_hosts
add address=192.0.2.0/24 list=own_hosts
add address=*.*.*.* list=border_gates
add address=*.*.*.* list=border_gates
add address=*.*.*.* list=own_hosts
add address=*.*.*.* list=own_hosts
add address=192.168.222.0/24 list=own_hosts
add address=192.168.223.0/24 list=own_hosts
add address=172.20.215.128/27 list=own_hosts
add address=0.0.0.0/8 comment="defconf: RFC6890" list=no_forward_ipv4
add address=169.254.0.0/16 comment="defconf: RFC6890" list=no_forward_ipv4
add address=224.0.0.0/4 comment="defconf: multicast" list=no_forward_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=no_forward_ipv4
add address=127.0.0.0/8 comment="defconf: RFC6890" list=bad_ipv4
add address=192.0.0.0/24 comment="defconf: RFC6890" list=bad_ipv4
add address=192.0.2.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=198.51.100.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=203.0.113.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=240.0.0.0/4 comment="defconf: RFC6890 reserved" list=bad_ipv4
add address=0.0.0.0/8 comment="defconf: RFC6890" list=not_global_ipv4
add address=10.0.0.0/8 comment="defconf: RFC6890" list=not_global_ipv4
add address=100.64.0.0/10 comment="defconf: RFC6890" list=not_global_ipv4
add address=169.254.0.0/16 comment="defconf: RFC6890" list=not_global_ipv4
add address=172.16.0.0/12 comment="defconf: RFC6890" list=not_global_ipv4
add address=192.0.0.0/29 comment="defconf: RFC6890" list=not_global_ipv4
add address=192.168.0.0/16 comment="defconf: RFC6890" list=not_global_ipv4
add address=198.18.0.0/15 comment="defconf: RFC6890 benchmark" list=not_global_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=not_global_ipv4
add address=224.0.0.0/4 comment="defconf: multicast" list=bad_src_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=bad_src_ipv4
add address=0.0.0.0/8 comment="defconf: RFC6890" list=bad_dst_ipv4
add address=224.0.0.0/4 comment="defconf: RFC6890" list=bad_dst_ipv4
add address=10.2.0.0/16 list=private_lans
add address=192.168.0.0/16 list=private_lans
add address=172.20.215.128/27 list=border_gates
add address=10.2.4.10 list=hairpin_dst
add address=10.2.4.20 list=hairpin_dst
add address=10.2.0.0/16 list=hairpin_src
add address=192.168.222.0/24 list=hairpin_src
add address=192.168.223.0/24 list=hairpin_src
/ip firewall filter
add action=accept chain=input comment="defconf: accept ICMP after RAW" protocol=icmp
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=accept chain=input comment="SSH access from everywhere" dst-port=1983 protocol=tcp
add action=accept chain=input comment=Wireguard dst-port=51820 protocol=udp
add action=accept chain=input dst-port=8291 protocol=tcp src-address-list=private_lans
add action=accept chain=input dst-port=3784,4784,3785 in-interface-list=peers protocol=udp src-address-list=own_hosts
add action=accept chain=input in-interface-list=peers protocol=ospf
add action=accept chain=input dst-port=179 in-interface-list=peers protocol=tcp src-address-list=own_hosts
add action=accept chain=input dst-port=53 protocol=udp src-address-list=own_hosts
add action=accept chain=input dst-port=53 protocol=tcp src-address-list=own_hosts
add action=drop chain=input comment="defconf: drop all not coming from trusted" in-interface-list=!trusted
add action=accept chain=forward comment="PortFW: For some reason, Fasttrack doesn't work for these." connection-mark=conn_portfw connection-state=established,related
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=jump chain=forward comment="Traffic rules for new connections" connection-state=new jump-target=traffic_rules
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid log=yes log-prefix=INVALID
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=drop chain=forward comment="defconf: drop bad forward IPs" src-address-list=no_forward_ipv4
add action=drop chain=forward comment="defconf: drop bad forward IPs" dst-address-list=no_forward_ipv4
add action=accept chain=forward comment="Check if counters >0. If not, DROP here" log=yes log-prefix="[ACPT]"
add action=accept chain=traffic_rules comment="Allow all ICMP echo" icmp-options=8:0-255 protocol=icmp
add action=accept chain=traffic_rules comment="Allow from ISPs only when DNAT" connection-nat-state=dstnat in-interface-list=ISP
add action=accept chain=traffic_rules comment="Allow internet for public 44" in-interface-list=DMZ out-interface-list=WANMYAS
add action=log chain=traffic_rules comment="Log drops (not wan)" in-interface-list=!WANMYAS log=yes log-prefix="[BLOCKED]" out-interface-list=!WANMYAS src-address=!192.168.222.150
add action=drop chain=traffic_rules
/ip firewall mangle
add action=mark-routing chain=output comment="wg-tunnels: bgate1 over FTTH" dst-address=*.*.*.* dst-port=51821 new-routing-mark=default_isp_only passthrough=no protocol=udp
add action=mark-routing chain=output comment="wg-tunnels: bgate2 over FTTH" dst-address=*.*.*.* dst-port=51821 new-routing-mark=default_isp_only passthrough=no protocol=udp
/ip firewall nat
add action=masquerade chain=srcnat comment="wg-tunnels: FTTH" routing-mark=default_isp_only
add action=masquerade chain=srcnat comment="Hairpin all PortFWs" dst-address-list=hairpin_dst src-address-list=hairpin_src
add action=masquerade chain=srcnat comment="defconf: masquerade WAN interface" out-interface=vlan2
/ip firewall raw
add action=accept chain=prerouting comment="defconf: enable for transparent firewall"
add action=accept chain=prerouting comment="defconf: accept DHCP discover" dst-address=255.255.255.255 dst-port=67 in-interface-list=LAN protocol=udp src-address=0.0.0.0 src-port=68
add action=drop chain=prerouting comment="defconf: drop bogon IP's" src-address-list=bad_ipv4
add action=drop chain=prerouting comment="defconf: drop bogon IP's" dst-address-list=bad_ipv4
add action=drop chain=prerouting comment="defconf: drop bogon IP's" src-address-list=bad_src_ipv4
add action=drop chain=prerouting comment="defconf: drop bogon IP's" dst-address-list=bad_dst_ipv4
add action=drop chain=prerouting comment="defconf: drop non global from WAN" in-interface-list=WAN src-address-list=not_global_ipv4
add action=drop chain=prerouting comment="defconf: drop forward to local lan from WAN" dst-address-list=private_lans in-interface-list=WAN
add action=drop chain=prerouting comment="defconf: drop local if not from default IP range" in-interface-list=LAN src-address=!10.2.79.0/24
add action=drop chain=prerouting in-interface-list=DEV src-address=!192.168.222.0/24
add action=drop chain=prerouting in-interface-list=DEVP src-address=!192.168.223.0/24
add action=drop chain=prerouting comment="defconf: drop bad UDP" port=0 protocol=udp
add action=jump chain=prerouting comment="defconf: jump to ICMP chain" jump-target=icmp4 protocol=icmp
add action=jump chain=prerouting comment="defconf: jump to TCP chain" jump-target=bad_tcp protocol=tcp
add action=accept chain=prerouting comment="defconf: accept everything else from LAN" in-interface-list=trusted
add action=accept chain=prerouting comment="defconf: accept everything else from WAN" in-interface-list=WAN
add action=drop chain=prerouting comment="defconf: drop the rest"
add action=drop chain=bad_tcp comment="defconf: TCP flag filter" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=fin,syn
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=fin,rst
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=fin,!ack
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=fin,urg
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=syn,rst
add action=drop chain=bad_tcp comment=defconf protocol=tcp tcp-flags=rst,urg
add action=drop chain=bad_tcp comment="defconf: TCP port 0 drop" port=0 protocol=tcp
add action=accept chain=icmp4 comment="defconf: echo reply" icmp-options=0:0 limit=5,10:packet protocol=icmp
add action=accept chain=icmp4 comment="defconf: net unreachable" icmp-options=3:0 protocol=icmp
add action=accept chain=icmp4 comment="defconf: host unreachable" icmp-options=3:1 protocol=icmp
add action=accept chain=icmp4 comment="defconf: protocol unreachable" icmp-options=3:2 protocol=icmp
add action=accept chain=icmp4 comment="defconf: port unreachable" icmp-options=3:3 protocol=icmp
add action=accept chain=icmp4 comment="defconf: fragmentation needed" icmp-options=3:4 protocol=icmp
add action=accept chain=icmp4 comment="defconf: echo" icmp-options=8:0 limit=5,10:packet protocol=icmp
add action=accept chain=icmp4 comment="defconf: time exceeded " icmp-options=11:0-255 protocol=icmp
add action=drop chain=icmp4 comment="defconf: drop other icmp" protocol=icmp
/ip route
add check-gateway=ping comment=default_isp disabled=no distance=1 dst-address=0.0.0.0/0 gateway=*.*.*.* routing-table=default_isp scope=30 suppress-hw-offload=no target-scope=10
/routing bgp connection
add as=64513 disabled=no local.address=172.20.215.132 .role=ibgp name=BorderGate1 remote.address=172.20.215.129/32 .as=64513 routing-table=default_myas
add as=64513 disabled=no local.address=172.20.215.132 .role=ibgp name=BorderGate2 remote.address=172.20.215.130/32 .as=64513 routing-table=default_myas
/routing ospf area range
add area=ospf-area-1 disabled=no prefix=172.20.215.128/27
add area=ospf-area-1 disabled=no prefix=192.0.2.0/24
/routing ospf interface-template
add area=ospf-area-1 auth=md5 auth-id=1 auth-key=**************** cost=100 dead-interval=10s disabled=no hello-interval=5s interfaces=wg-bg1-ftth,wg-bg2-ftth
add area=ospf-area-1 disabled=no interfaces=dum0 passive
add area=ospf-area-1 disabled=no interfaces=vlan44 passive
/routing rule
add action=lookup-only-in-table comment="Connections must never go through other links" disabled=no routing-mark=default_isp_only table=default_isp
add action=lookup comment="main table" disabled=no table=main
add action=lookup comment="default 44net" disabled=no src-address=192.0.2.0/24 table=default_myas
add action=lookup comment="default sonic" disabled=no table=default_isp
/tool sniffer
set filter-ip-address=192.0.2.210/32 filter-ip-protocol=udp filter-port=51820

[divB]

I am finding it difficult to understand what you are saying, but I will assume the mikrotik has 3 public ip addresses on it.
And the incoming wg packets all come into the mikrotik via either the 2.249 or 2.253 interfaces but directed at the .210 address.

Normally for this, I would use routing rules to send the return packets out the correct port based on source address.
But in this case the source address is always .210

I think in this case you can use a dst-nat rule that should (maybe) work.

Yes, this the correct setup but no, I don’t think that’s necessary (or should be necessary). It’s plain routing with public IPs as it was done before NAT even existed. The source address is indeed (and should always be) .210. It is a public IP, the whole /24 is announced and internal OSPF/iBGP make sure that this IP can always be reached (over possibly multiple redundant links).

I would really like to avoid NAT as much as possible

that is a correct behavior from the wireguard to do rpf check (anti spoofing) before sending you handshake reply - as any other session based connection.

That was my initial thought and indeed it would be the obvious explanation. However, the source and destination IPs are never rewritten. Wireguard server is always .210. What routing happens in between is totally irrelevant.

Now there is one weird possibility: Since it’s locally generated packet, that RouterOS’ wireguard does not answer with source .210 but .249 or .253 (since these are also public IPs of the router). But in this case, the packets would be discarded at the road warrier client. Note that RouterOS not even transmits and response packets.

try having lamp stack with ssl, or build ipsec using your scenario - and see what happens.

Works flawlessly

[divB]

Ok, major update: RouterOS did send the return packages out but I had the wrong filter in the packet sniffer (192.0.2.210/32 instead of 192.0.2.0/24). One definitely should take breaks in between.

Now the situation is clear: Even though I request the connection to 192.0.2.210, the return packet does not originate from 192.0.2.210 but from 192.0.2.177, the source IP of the wireguard tunnel. This packet is then discarded by the client because it doesn’t match server.

Source IP of locally generated packet

Ok, no problem, I can just use pref-src, so I thought. I added a route filter to set pref-src and it indeed gets set in the routing table:

/ip/route/print detail
[...]
   DAb   dst-address=0.0.0.0/0 routing-table=default_myasn pref-src=192.0.2.210 gateway=172.20.215.130 immediate-gw=192.0.2.176%wg-bg2-ftth distance=200 scope=40 target-scope=30 suppress-hw-offload=no 

   D b   dst-address=0.0.0.0/0 routing-table=default_myasn pref-src=192.0.2.210 gateway=172.20.215.129 immediate-gw=192.0.2.185%wg-bg1-ftth distance=200 scope=40 target-scope=30 suppress-hw-offload=no

Yet still the packets leave with source IP 192.0.2.177, instead of 192.0.2.210. Why does RouterOS ignore my pref-src??

Source NAT

I hate NAT, I really don’t want to use NAT but fine, since pref-src isn’t working, I’ll just do src-nat. So I thought.

/ip/firewall/nat<SAFE> print detail
[...]
12    chain=srcnat action=src-nat to-addresses=192.0.2.210 protocol=udp src-address=192.0.2.177 src-port=51820

The entry is matched (based on packet counter) but yet again, the packages still leave as 192.0.2.177 instead of 192.0.2.210. Again, RouterOS is ignoring my config. Why does RouterOS ignore my srcnat entry?

[anav]

Diagram please detailing the wans etc… It could be a well known wireguard routing issue but a diagram will help orient me to your network.

[wiseroute]

@divb,

“try having lamp stack with ssl, or build ipsec using your scenario - and see what happens.”
Works flawlessly

I don’t know how you made your scenario work for it - but if you did, probably the whole world won’t need for complex load balancers and proxies. just to keep persistent sessions.

Now the situation is clear: Even though I request the connection to 192.0.2.210, the return packet does not originate from 192.0.2.210 but from 192.0.2.177, the source IP of the wireguard tunnel. This packet is then discarded by the client because it doesn’t match server.

it’s obviously re written somewhere. either you have source nat on your interface or something else?

full routing or nat is just options - depends on the requirements. engineers created nat for reasons.

for your wireguard problem - why don’t you just make it to listen on the wan interface ip? just for simplicity because you hated nat?

[divB]

I continued to debug and the issue is, as I stated, that RouterOS uses the wrong source address which should be handled via pref-src. Since this is a new topic, I started a new thread: http://forum.mikrotik.com/t/routeros-blatantly-ignores-pref-src-can-this-really-be-a-bug/180360/1

@wiseroute:

it’s obviously re written somewhere. either you have source nat on your interface or something else?

No, I describe it in the post above. I assume it’s a bug (or at least very bad implementation) of wireguard that does not set source address properly when generating local packets. For local packets without explicit source address, if there are multiple IP addresses, the operating system needs to decide which one to use. And RouterOS decides to use the wrong one (and ignores my request to use the correct one)

for your wireguard problem - why don’t you just make it to listen on the wan interface ip? just for simplicity because you hated nat?

Oh gosh, I wish life (RouterOS) would be so hassle free.
This was my first obvious idea but if I don’t miss anything simple, that’s not possible: RouterOS does not allow wireguard to “bind” or “listen” on a specific interface. It just listens on every interface on the router! Crazy, but yeah, that’s sadly how it is. “Bugfixes” are to use firewall filters but this clearly doesn’t fix the issue I am seeing.

[wiseroute]

@divb,

I assume it’s a bug (or at least very bad implementation) of wireguard that does not set source address properly when generating local packets.

ok. let us find out…

you have 3 interfaces in the router. 2 wan and 1 loopback (wg ip). now… from those 3 interfaces - 210, 253, 177 which ip did the wg use? does it ring a bell? ospf router-id?

ok. now… can you differentiate these 3 pictures

|-------------------------|
wg/loopback        internet
  |-------------------------|
  
              |---------------|
  wg----router          internet
              |----------------|
              
      |-----wg-------------|
   router                 internet 
      |-----wg-------------|

i hope you get the idea about persistency.

[divB]

@divb,

I assume it’s a bug (or at least very bad implementation) of wireguard that does not set source address properly when generating local packets.

ok. let us find out…

you have 3 interfaces in the router. 2 wan and 1 loopback (wg ip). now… from those 3 interfaces - 210, 253, 177 which ip did the wg use? does it ring a bell? ospf router-id?



ok. now… can you differentiate these 3 pictures […]
i hope you get the idea about persistency.

Sorry, I am not yet able to follow. No bell rings yet. What does ospf router id have to do with it? Any locally generated packet can be distinguished in 3 different cases:

• The response packet for an existing TCP connection where the socket initially opened by a client connecting to it. In this case, per definition of a TCP connection, the source address must be identical to the address that was used to establish the connection. Doesn’t matter how many interfaces/routes
• Same as above but for UDP (this is the wireguard scenario).
• A new TCP connection is created (e.g. “telnet mx.google.com 25”)

Since UDP is stateless, the state of a connection is managed by the userspace application (in this case wireguard). The initial request came to address 192.0.2.210. When wireguard creates the responses packet, it knows already which “virtual connection” it belongs to and it’s its duty to set the source address properly. See this for the bigger picture: https://blog.cloudflare.com/everything-you-ever-wanted-to-know-about-udp-sockets-but-were-afraid-to-ask-part-1/#sourcing-packets-from-a-wildcard-socket

It seems RouterOS’s wireguard implementation misses this … it does not specify source address and leaves it up to the OS to decide. This is not a feature, there is no excuse for this being the “right thing”. It’s just a bug.

I don’t see how any of this has to do with multiple interfaces, WAN’s etc.


PS: The .210 is not a loopback interface. It’s vlan44 and 192.0.2.210 is just assigned the address on this network on the router. But there are others connected via Ethernet. But doesn’t matter for this discussion

[edepalos]

[In the line “add action=accept chain=input comment=Wireguard dst-port=51820 protocol=udp” are you sure you are not missing quote marks?

[divB]

Yes I think quotes are only needed if comment has spaces or special characters. Just a simple word still works without.