iOS Wireguard vs. RouterOS - No handshake

hoeser · July 18, 2022, 5:29pm

After an exhaustive beating of my face against the keyboard in numerous google searches, wiki’s, forum posts, etc, I am finally conceding to asking for help with this one.

Very basic road-warrior setup - can’t get a handshake to work.

RouterOS X86 v7.3.1 on a Cisco C220-M3 (works great in all other regards)
iPhone 13 Pro - Wireguard for iOS 1.0.15 (26)

Here’s the RouterOS Config

[admin@C220-Core] /interface/wireguard> print
Flags: X - disabled; R - running
 0  R name="wg_roadwarrior" mtu=1420 listen-port=51457 private-key="private key" public-key="public key"
[admin@C220-Core] /interface/wireguard> peers
[admin@C220-Core] /interface/wireguard/peers> print
Flags: X - DISABLED
Columns: INTERFACE, PUBLIC-KEY, ENDPOINT-ADDRESS, ENDPOINT-PORT, ALLOWED-ADDRESS, PERSISTENT-KEEPALIVE
#   INTERFACE     PUBLIC-KEY                                    ENDPOINT-ADDRESS  ENDPOINT-PORT  ALLOWED-ADDRESS    PERSISTENT-KEEPALIVE
;;; WireGuard Peer - Dave's iPhone
0   wg_roadwarrior  [Public key]                                0  192.168.205.20/32  
[admin@C220-Core] /interface/wireguard/peers> /ip/address
[admin@C220-Core] /ip/address> print
Flags: D - DYNAMIC
Columns: ADDRESS, NETWORK, INTERFACE
 #   ADDRESS            NETWORK         INTERFACE
xxx 
14   192.168.205.1/24   192.168.205.0   wg_roadwarrior
xxx
[admin@C220-Core] /ip/address>

Also - I’ve got these firewall rules:

 0    chain=input action=accept in-interface=wg_roadwarrior log=no log-prefix="" <-- this is at the top of the chain
1  chain=input action=accept protocol=udp in-interface=wan_interface dst-port=51457 log=no log-prefix=""

The iphone config is as follows:

Interface
Name: roadwarrior
Public key: (the public key specified in the peer on RouterOs for the phone)
Addresses: 192.168.205.20/32

Peer:

Public Key: (the public key of the routeros interface)
Endpoint: my_public_ip:51457
Allowed IPs: 192.168.205.0/24

On the RouterOS side when looking at the stats for the iPhone peer, I see traffic on the peer in both directions when attempting ping the phone or having the phone attempt to ping the router - but the handshake is not successful and fails with the generic 5 second retry on the handshake. I am getting the same issue when using a Macbook as well. I added a friends phone on another wireless carrier with the same result.

I’m really pulling my hair out because I followed the roadwarrior guide and its very simple… just not working. I did try rebooting the router, enabling/disabling the interface, the peers, re-creating all of it with new keys, etc. No dice. Any insight would be appreciated.

anav · July 18, 2022, 5:35pm

*** Would need to see the complete config aka IP routes and IP firewall rules for best review.

What you ONLY need for input chain on the SERVER side is
add chain=input action=accept dst-port=51457 protocol=udp comment=“allow wireguard handshake”

If you want to give your iphone access to the config…
add action=accept chain=input in-interface=wg_roadwarrior dst-port=winboxport protocol=tcp src-address=192.168.205.20/32 { allows admin to config local router remotely }

(then after connected, use IOS MT APP to configure the router - may need some additional router config changes) ***.

okay I see you have the idea of the above covered, so that is good!!
+++++++++++++++++++++++++++++++++++++++++
IOS device:
What do you have as dns server 192.168.205.1 ??

Also you have the peers correct partially.
If this is only to config the router, you are fine.
If you wish to access subnets you have to put in the subnet IP address or applicable single IPs being reached (such as servers).
If you wish to access internet then you need to enter 0.0.0.0/0 which ipso facto includes all of the above.

hoeser · July 18, 2022, 5:41pm

[admin@TRG-C220M3-Core] /ip/route> print
Flags: D - DYNAMIC; A - ACTIVE; c, s, y - COPY; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#      DST-ADDRESS        GATEWAY                    DISTANCE
2  As  0.0.0.0/0          wan_interface                    0
  DAc+ 10.0.0.0/22        eth1_lan                          0
  DAc  10.50.44.47/32     wan_interface                  0
 DAc  192.168.205.0/24   wg_roadwarrior                 0

Firewall is long and complex for the internet stuff - but the top two rules are ultra simple and I’ve disabled literally anything else that was not an accept rule in attempt to troubleshoot this.

anav · July 18, 2022, 5:49pm

In addition, its not clear if you have any forward chain rules to allow IOS client to any subnets, if that is the intention…
Would need that along with the allowed IPs on the IPHONE settings…

anav · July 18, 2022, 5:50pm

Yeah the DAC route is there so any return traffic from your IPhone should get back into the tunnel.

Will need the complete config however to see what may be preventing connectivity.
Are you sure you are getting a public IP on the MT router??

hoeser · July 18, 2022, 6:03pm

I’m certain I’m getting a public IP on the router, yes. Again I can see the RX and TX counters for the Wireguard peer ticking up when either side attempts to reach the other, but it never successfully handshakes. I even have another WG interface on that same public IP on a different port doing a lan bridge to another RouterOS site - which works fine.

I’ve also just audited this same config using that other site, with the same iphone, which works. I think this is the most alarming point because I’m starting to wonder if this is an X86 problem. the other box is a 3011.

anav · July 18, 2022, 6:06pm

Both using version 7.3.1 ??
So the more important question is can you access the x86 router from your iphone, nevermind not seeing the handshake, as data is passing back and forth…
In other words, do you have any proof that the handshake did work. Its only a one time hit on the log.

anav · July 18, 2022, 6:08pm

On your input chain rule drop the i_n-interface=WAN_interface_ part of the rule and see what happens?

Other than that will need to see the complete config (minus any public IPs of course).

hoeser · July 18, 2022, 6:19pm

Both of the tested routers are 7.3.1 - One is X86, the other is an RB3011.
I can confirm I have disabled input filtering entirely on the X86 router and I still get the same result. I have basically bypassed the entire firewall and it still doesn’t work.
The wireguard log on the iphone shows the “sending handshake initiation” and then the “handshake did not complete after 5 seconds” error - when this is happening I can see the RX/TX counter going up on the peer on the RouterOS box. That said, is there debug logging I can turn up for wireguard on routeros?

hoeser · July 18, 2022, 6:22pm

oh, and #4, yes I can confirm the the phone can’t hit the router and the router can’t hit the phone. ICMP timeouts both ways and I can’t hit the webserver. (X86 box) - all works same config on the 3011.

anav · July 18, 2022, 7:31pm

Other than reinstalling the RoS, I am at a loss to explain, other than theV86 is not getting a public IP.
As noted, would need to see complete config minus any public iPs.

A network diagram may help as well.

hoeser · July 18, 2022, 7:43pm

After a few more hours at this one I discovered the root cause. The good news is it’s not a bug. The bad news is that I’m not sure how to fix it.

The box has two gateways -
#1 is a Cable provider on the MAIN route table - this is my commercial connection with some static IPs, but a relatively poor upstream bandwidth.
#2 is a residential fiber to the home connection, very fast upstream. It is on the “fiber” route table in router OS.

Clients behind the router get mangled out the desired gateway through various rules. But the Wireguard listener is obviously on the router itself.

The firewall allows the wireguard incoming connections only on the fiber interface, but it appears that the responses outbound (for example, to the iphone) are going out the MAIN route table. Since wireguard is entirely untracked from a connection perspective, not sure this is fixable without a re-architecture. I can’t really mangle my way out of this one, I don’t think.

Znevna · July 18, 2022, 8:09pm

Never too late to mention dual wan with a broken config, I guess.
So the title has nothing to do with the actual problem…

anav · July 18, 2022, 8:14pm

Bad news: Tis why the OP should have provided a network diagram and full config from the getgo instead of wasting everyones time.

Good news: Now that we know the probable cause, post your complete config and if possible a network diagram. Yes this very much fixable, once you provide your dirty underwear…

hoeser · July 18, 2022, 8:41pm

A beating is deserved, that’s fair enough.

Here is a complete /export minus the obviously not required bits (ppp secrets, ipsec config for vpns, etc).

Please note: There are definitely a bunch of extra firewall rules in here right now due to mid-stream debugging this. Certainly some do absolutely nothing or are altogether just dumb, broken, or not required.

# jul/18/2022 12:29:28 by RouterOS 7.3.1
#

/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no name=eth1_lan
set [ find default-name=ether2 ] disable-running-check=no name=eth2_tagged
set [ find default-name=ether5 ] disable-running-check=no name=eth3_cogeco
set [ find default-name=ether6 ] disable-running-check=no name=eth4_gvs
set [ find default-name=ether3 ] disable-running-check=no name=eth5_bellfibe
set [ find default-name=ether4 ] disable-running-check=no name=eth6

/interface ovpn-server
add name=ovpn-in1-chops user=xxx

/interface wireguard
add listen-port=52522 mtu=1420 name=wg_embybuds
add listen-port=52520 mtu=1420 name=wg_thad
add listen-port=52521 mtu=1420 name=wg_trgmobile

/interface vlan
add interface=eth2_tagged name=vlan63_addfree vlan-id=63
add interface=eth2_tagged name=vlan66_dmz vlan-id=66
add interface=eth2_tagged name=vlan67_mgmt vlan-id=67
add interface=eth2_tagged name=vlan69_vpn vlan-id=69
add interface=eth2_tagged name=vlan70_emby vlan-id=70
add interface=eth2_tagged name=vlan99_heartbeat vlan-id=99

/disk
set usb1 disabled=no
set usb1-part1 disabled=no

/interface list
add name=WAN
add name=TRUSTED_LANS
add name=INET_FWD_ALLOWED
add name=LAN-FWD-ALLOW
add name=EMBY-ALLOWED
add name=WIREGUARD_TRUSTED


/interface pppoe-client
add disabled=no interface=eth5_bellfibe name=pppoe-bellfibe profile=bell-fibe user=xxx

/ip vrf
add disabled=yes interfaces=pppoe-bellfibe name=bellfibe

/queue simple
add max-limit=20M/150M name=gvs queue=ethernet-default/ethernet-default target=eth4_gvs total-queue=ethernet-default

/routing table
add disabled=no fib name=bellfibe

/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192

/interface l2tp-server server
set default-profile=default enabled=yes use-ipsec=required

/interface list member
add interface=eth3_cogeco list=WAN
add interface=eth1_lan list=TRUSTED_LANS
add interface=eth1_lan list=INET_FWD_ALLOWED
add interface=vlan63_addfree list=INET_FWD_ALLOWED
add interface=vlan66_dmz list=INET_FWD_ALLOWED
add interface=eth4_gvs list=INET_FWD_ALLOWED
add interface=vlan66_dmz list=LAN-FWD-ALLOW
add interface=vlan67_mgmt list=LAN-FWD-ALLOW
add interface=vlan63_addfree list=LAN-FWD-ALLOW
add interface=eth4_gvs list=LAN-FWD-ALLOW
add interface=vlan63_addfree list=TRUSTED_LANS
add interface=eth3_cogeco list=EMBY-ALLOWED
add interface=vlan63_addfree list=EMBY-ALLOWED
add disabled=yes interface=vlan67_mgmt list=INET_FWD_ALLOWED
add disabled=yes interface=ovpn-in1-chops list=LAN-FWD-ALLOW
add interface=pppoe-bellfibe list=WAN
add interface=eth1_lan list=LAN-FWD-ALLOW
add interface=wg_thad list=WIREGUARD_TRUSTED
add interface=wg_trgmobile list=WIREGUARD_TRUSTED
add interface=wg_trgmobile list=TRUSTED_LANS
add interface=wg_trgmobile list=INET_FWD_ALLOWED
add interface=vlan70_emby list=LAN-FWD-ALLOW


/interface wireguard peers
add allowed-address=192.168.203.2/32,192.168.1.0/24 comment="WireGuard Peer - Thad" endpoint-address=XXX \
    endpoint-port=13231 interface=wg_thad public-key="XXX"
add allowed-address=192.168.205.20/32 comment="WireGuard Peer - Dave's iPhone" interface=wg_trgmobile public-key=\
    "XXX"
add allowed-address=192.168.205.10/32 comment="WireGuard Peer - Dave's Macbook Air" interface=wg_trgmobile \
    public-key="XXX"
add allowed-address=192.168.205.30/32 comment="Thaddeus Test" interface=wg_trgmobile public-key=\
    "XXX"
add allowed-address=192.168.206.20/32 comment="Dave iPhone Emby" interface=wg_embybuds public-key=\
    "XXX"

/ip address
add address=10.0.2.1/22 interface=eth1_lan network=10.0.0.0
add address=PUBLIC_IP_RANGE.122/29 interface=eth3_cogeco network=PUBLIC_IP_RANGE.120
add address=PUBLIC_IP_RANGE.123/29 interface=eth3_cogeco network=PUBLIC_IP_RANGE.120
add address=PUBLIC_IP_RANGE.126/29 interface=eth3_cogeco network=PUBLIC_IP_RANGE.120
add address=PUBLIC_IP_RANGE.124/29 interface=eth3_cogeco network=PUBLIC_IP_RANGE.120
add address=PUBLIC_IP_RANGE.125/29 interface=eth3_cogeco network=PUBLIC_IP_RANGE.120
add address=192.168.66.1/24 interface=vlan66_dmz network=192.168.66.0
add address=192.168.67.1/24 interface=vlan67_mgmt network=192.168.67.0
add address=192.168.63.1/24 interface=vlan63_addfree network=192.168.63.0
add address=192.168.120.1/24 interface=eth4_gvs network=192.168.120.0
add address=192.168.69.1/24 interface=vlan69_vpn network=192.168.69.0
add address=192.168.66.11/24 interface=vlan66_dmz network=192.168.66.0
add address=10.0.0.1/22 interface=eth1_lan network=10.0.0.0
add address=192.168.203.1/30 interface=wg_thad network=192.168.203.0
add address=192.168.205.1/24 interface=wg_trgmobile network=192.168.205.0
add address=10.255.255.1/30 interface=vlan99_heartbeat network=10.255.255.0
add address=192.168.70.1/24 interface=vlan70_emby network=192.168.70.0
add address=192.168.206.1/24 interface=wg_embybuds network=192.168.206.0

/ip dns
set allow-remote-requests=yes servers=xxx,10.0.0.6

/ip firewall filter
add action=accept chain=input comment="ACCEPT ALL FROM WIREGUARD TEST" in-interface=wg_trgmobile
add action=accept chain=input comment="ALLOW WIREGUARD PORTS" dst-port=52520-52522 in-interface=pppoe-bellfibe \
    protocol=udp
add action=accept chain=input comment="ALLOW WIREGUARD PORTS" dst-address=PUBLIC_IP_RANGE.122 dst-port=52521 in-interface=\
    eth3_cogeco protocol=udp
add action=drop chain=input comment="DROP INVALID" connection-state=invalid disabled=yes
add action=accept chain=forward comment="ALLOW ESTABLISHED RELATED FOR ALL" connection-state=established,related
add action=accept chain=input comment="ACCEPT INPUT FROM TRUSTED_LANS" in-interface-list=TRUSTED_LANS
add action=accept chain=forward comment="ALLOW FORWARD - INTERNET ALLOWED ONLY" in-interface-list=INET_FWD_ALLOWED \
    out-interface-list=WAN
add action=accept chain=forward comment="ALLOW FORWARD - TRUSTED LAN FOWARDS" in-interface-list=TRUSTED_LANS \
    out-interface-list=LAN-FWD-ALLOW
add action=accept chain=forward comment="ALLOW FORWARD - TRUSTED LAN TO CHOPSVPN" dst-address=192.168.78.0/24 \
    in-interface-list=TRUSTED_LANS
add action=accept chain=input comment="ALLOW INPUT - WIREGUARD TRUSTED" in-interface-list=WIREGUARD_TRUSTED
add action=accept chain=output comment="ALLOW OUTPUT - WIREGUARD TRUSTED" out-interface-list=WIREGUARD_TRUSTED
add action=accept chain=forward comment="ALLOW FORWARD - THAD TO LAN" in-interface-list=WIREGUARD_TRUSTED \
    out-interface=eth1_lan
add action=accept chain=forward comment="ALLOW FORWARD - LAN TO THAD" in-interface=eth1_lan out-interface-list=\
    WIREGUARD_TRUSTED
add action=accept chain=forward comment="ALLOW FORWARD - EMBY USERS TO EMBY DMZ" in-interface=wg_embybuds \
    out-interface=vlan70_emby
add action=accept chain=forward comment="ALLOW EMBY DMZ BOX TO MOUNT NFS FROM NEUTRON" dst-address=10.0.0.13 \
    dst-port=111,649,2049,35010-35012 in-interface=vlan70_emby protocol=tcp src-address=192.168.70.10
add action=accept chain=forward comment="ALLOW EMBY DMZ BOX TO MOUNT NFS FROM NEUTRON" dst-address=10.0.0.13 \
    dst-port=111,649,2049,35010-35012 in-interface=vlan70_emby protocol=udp src-address=192.168.70.10
add action=accept chain=input comment="TRGBoyz VPN" dst-address=PUBLIC_IP_RANGE.122 in-interface=eth3_cogeco protocol=l2tp
add action=accept chain=input comment="TRGBoyz VPN" dst-address=PUBLIC_IP_RANGE.122 in-interface=eth3_cogeco protocol=\
    ipsec-esp
add action=accept chain=input comment="ACCEPT INPUT - VPNs - IPSEC OVPN ETC" dst-address=PUBLIC_IP_RANGE.122 dst-port=\
    500,4500,1701,52768 in-interface=eth3_cogeco protocol=udp
add action=accept chain=input comment="TRGBoyz & ChopsCallhome VPN" dst-address=PUBLIC_IP_RANGE.122 dst-port=52768 \
    in-interface=eth3_cogeco protocol=tcp
add action=accept chain=forward comment=CH2A-Test dst-address=192.168.77.200
add action=accept chain=forward comment=MobileMike dst-address=192.168.77.58
add action=accept chain=input comment="ACCEPT ICMP IF NOT INTERNET" in-interface-list=!WAN protocol=icmp
add action=accept chain=forward comment="ACCEPT FORWARD Gaming-Stuff UDP" dst-address=192.168.66.50 dst-port=\
    2456-2458,3979,7780-9780,25565,25575,26800-29015,32123-32125,42420,41234 out-interface=vlan66_dmz protocol=udp
add action=accept chain=forward comment="ACCEPT FORWARD Gaming-Stuff TCP" dst-address=192.168.66.50 dst-port=\
    2456-2458,3979,7780-9780,25565,25575,26800-29015,32123-32125,42420,41234 out-interface=vlan66_dmz protocol=tcp
add action=drop chain=forward comment="DROP MAIL IN CASE OF MALWARE" dst-port=25,587,2525 out-interface=eth3_cogeco \
    protocol=tcp
add action=accept chain=forward comment="ACCEPT EMBY TRAFFIC TO EMBY-SERVERS FROM INTERFACES EMBY-ALLOWED" \
    dst-address-list=EMBY-SERVERS dst-port=8096 in-interface-list=EMBY-ALLOWED out-interface=eth1_lan protocol=tcp
add action=accept chain=forward comment="REDACTED OpenVPN" dst-address=192.168.66.10 dst-port=62234 in-interface=\
    eth3_cogeco out-interface=vlan66_dmz protocol=udp
add action=accept chain=forward comment=REDACTEDDev dst-address=192.168.66.151 dst-port=22,80,443,3306 in-interface=\
    eth3_cogeco out-interface=vlan66_dmz protocol=tcp
add action=accept chain=forward comment="dmzweb forwards" dst-address=192.168.66.160 dst-port=80,443,8080 \
    in-interface=eth3_cogeco out-interface=vlan66_dmz protocol=tcp
add action=accept chain=forward comment="Allow ssh/http/httpd connections from DMZ to LAN git server" dst-address=\
    10.0.0.40 dst-port=22,80,443 in-interface=vlan66_dmz out-interface=eth1_lan protocol=tcp
add action=accept chain=forward comment="ACCEPT FORWARD Pi-Hole to AD-DNS" dst-address-list=AD-DNS dst-port=53 \
    protocol=udp src-address=192.168.66.2
add action=accept chain=forward comment="Allow forwards for wifi_vlan63 to DMZ for DNS" dst-port=53 in-interface=\
    vlan63_addfree out-interface=vlan66_dmz protocol=udp
add action=drop chain=forward comment="Drop DNS queries from Addfree Network to Internet" dst-port=53 in-interface=\
    vlan63_addfree out-interface=eth3_cogeco protocol=udp
add action=drop chain=forward comment="DROP FORWARD"
add action=accept chain=input comment="ALLOW Established/Related INPUT" connection-state=established,related
add action=accept chain=input comment="ACCEPT INPUT FOR DNS FROM NOT INTERNET" dst-port=53 in-interface-list=!WAN \
    protocol=udp
add action=drop chain=input comment="DROP INPUT"
add action=accept chain=output comment="ACCEPT Established/Related OUTPUT" connection-state=established,related
add action=accept chain=output comment="ACCEPT OUTPUT FOR UDP/53 ROUTERDNS" dst-port=53 protocol=udp
add action=drop chain=output comment="DROP OUTPUT" disabled=yes


/ip firewall mangle
add action=mark-packet chain=prerouting comment="Cogeco incoming packets" in-interface=eth3_cogeco new-packet-mark=\
    cogeco-packet passthrough=yes
add action=mark-packet chain=prerouting comment="Bell incoming packets" in-interface=pppoe-bellfibe new-packet-mark=\
    bell-packet passthrough=yes
add action=mark-connection chain=prerouting comment="Mark incoming connections from bell for tracking" in-interface=\
    pppoe-bellfibe new-connection-mark=bell
add action=mark-connection chain=prerouting comment="Mark incoming connections from Cogeco for tracking" \
    in-interface=eth3_cogeco new-connection-mark=cogeco
add action=mark-routing chain=prerouting comment="Single gateway Fibe clients by list" dst-address-list=!LOCAL \
    new-routing-mark=bellfibe src-address-list=BELL-FIBE-CLIENTS
add action=mark-connection chain=prerouting comment="Mark connections for Dual gateway with Nth 2/1" \
    connection-state=new dst-address-list=!LOCAL new-connection-mark=bell nth=2,1 src-address-list=\
    DUAL-GATEWAY-CLIENTS
add action=mark-routing chain=prerouting comment="Mark routing for connections for dual gateway" connection-mark=bell \
    dst-address-list=!LOCAL new-routing-mark=bellfibe src-address-list=DUAL-GATEWAY-CLIENTS
add action=mark-routing chain=output comment="Send wireguard out the fast interface" new-routing-mark=bellfibe \
    protocol=udp src-port=52520,52521
add action=mark-routing chain=prerouting disabled=yes dst-port=52520 in-interface-list=WAN new-routing-mark=bellfibe \
    protocol=udp
add action=mark-routing chain=output dst-port=13231 new-routing-mark=bellfibe protocol=udp


/ip firewall nat
add action=masquerade chain=srcnat comment="Hairpin NAT for rules not matching an IN" dst-address=10.0.0.0/22 \
    src-address=10.0.0.0/22
add action=masquerade chain=srcnat comment="Masq to Steve" dst-address=192.168.78.100 src-address=10.0.0.0/22
add action=masquerade chain=srcnat comment="CH2A Test" dst-address=192.168.77.200 src-address=10.0.0.70
add action=masquerade chain=srcnat comment=MobileMike dst-address=192.168.77.58 src-address=10.0.0.70
add action=masquerade chain=srcnat comment="TEST 10.0.0.16 - FIBE" dst-address-list=!LOCAL out-interface=\
    pppoe-bellfibe src-address-list=DUAL-GATEWAY-CLIENTS
add action=masquerade chain=srcnat comment="TEST 10.0.0.16 - COGECO" dst-address-list=!LOCAL out-interface=\
    eth3_cogeco src-address-list=DUAL-GATEWAY-CLIENTS
add action=masquerade chain=srcnat comment="SECONDARY INTERNET (BELL FIBE) MASQUERADE BY ROUTING MARK" \
    dst-address-list=!LOCAL out-interface=pppoe-bellfibe routing-mark=bellfibe src-address-list=BELL-FIBE-CLIENTS
add action=masquerade chain=srcnat comment="PRIMARY INTERNET (COGECO) MASQUERADE" dst-address-list=!LOCAL \
    out-interface=eth3_cogeco src-address-list=NAT_ALLOWED_SUBNETS
add action=dst-nat chain=dstnat comment="Win2k19 Games Server " dst-address=PUBLIC_IP_RANGE.122 dst-port=\
    2456-2458,3979,7780-8095,8097-9780,25565,26800-29015,32123-32125,42420,41234 protocol=udp to-addresses=\
    192.168.66.50
add action=dst-nat chain=dstnat comment="Win2k19 Games Server" dst-address=PUBLIC_IP_RANGE.122 dst-port=\
    2456-2458,3979,7780-8095,8097-9780,25565,26800-29015,32123-32125,42420,41234 protocol=tcp to-addresses=\
    192.168.66.50
add action=dst-nat chain=dstnat comment="Win2k19 Games - For TRGBoyz NAT" dst-address=192.168.77.1 dst-port=\
    25000-30000 protocol=tcp to-addresses=192.168.66.50
add action=dst-nat chain=dstnat comment="REDACTED OpenVPN" dst-address=PUBLIC_IP_RANGE.126 dst-port=62234 in-interface=\
    eth3_cogeco protocol=udp to-addresses=192.168.66.10
add action=dst-nat chain=dstnat comment=Web dst-address=PUBLIC_IP_RANGE.124 dst-port=80,443,8080,65123 in-interface=\
    eth3_cogeco protocol=tcp to-addresses=192.168.66.160
add action=dst-nat chain=dstnat comment="WFC WEb" dst-address=PUBLIC_IP_RANGE.123 dst-port=80 in-interface=eth3_cogeco \
    protocol=tcp to-addresses=192.168.66.160 to-ports=80
add action=dst-nat chain=dstnat comment="Emby for EMBY-USERS List" dst-address=PUBLIC_IP_RANGE.122 dst-port=8096 \
    in-interface=pppoe-bellfibe protocol=tcp src-address-list=EMBY-USERS to-addresses=10.0.0.17 to-ports=8096
add action=dst-nat chain=dstnat comment="REDACTEDDev - Web" dst-address=PUBLIC_IP_RANGE.126 dst-port=80,443 in-interface=\
    eth3_cogeco protocol=tcp to-addresses=192.168.66.151
add action=dst-nat chain=dstnat comment="REDACTEDDev - SSH - Aaron" dst-address=PUBLIC_IP_RANGE.126 dst-port=22422 \
    in-interface=eth3_cogeco protocol=tcp src-address=24.57.244.15 to-addresses=192.168.66.151 to-ports=22
add action=dst-nat chain=dstnat comment="REDACTEDDev- DB - List" dst-address=PUBLIC_IP_RANGE.126 dst-port=3306 \
    in-interface=eth3_cogeco protocol=tcp src-address-list=REDACTED-DB-ACCESS to-addresses=192.168.66.151
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=PUBLIC_IP_RANGE.121 pref-src=0.0.0.0 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-bellfibe pref-src=0.0.0.0 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=192.168.69.0/24 gateway=vlan69_vpn pref-src=0.0.0.0 routing-table=bellfibe \
    scope=30 suppress-hw-offload=no target-scope=10
add disabled=no dst-address=192.168.66.0/24 gateway=vlan66_dmz routing-table=bellfibe suppress-hw-offload=no
add disabled=no dst-address=192.168.1.0/24 gateway=192.168.203.2 routing-table=main suppress-hw-offload=no
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=pppoe-bellfibe pref-src=0.0.0.0 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10

/ip service
set telnet disabled=yes
set ftp disabled=yes

/system hardware
set allow-x86-64=yes

anav · July 18, 2022, 9:13pm

So you have to figure out what you want do do regarding your ISPs and LAN subnets.
How do you differentiate between this… some IPs, entire subnets etc…??

It would appear you have two subnets you wish to have through wireguard.
add interface=wg_thad list=WIREGUARD_TRUSTED
add interface=wg_trgmobile list=WIREGUARD_TRUSTED

However, no clue based on your config what these interfaces really are ??
Okay I see it now, there are two subnets identified.
add address=192.168.203.1/30 interface=wg_thad network=192.168.203.0
add address=192.168.205.1/24 interface=wg_trgmobile network=192.168.205.0

But they are not identified anywhere??? Ip address, etc… ???
Where do they come from???

Continuing, it appears one of them “wg_thad” is solely to be used with wireguard, but the other, wg_trgmobile, is confusing in that its part of the trusted lans and inet_fwd allowed.
I see a conflict on the latter in that you want that traffic to go out the tunnel but you seem to state it should go out local internet and be able to access LOCAL LAN subnets.

add interface=wg_thad list=WIREGUARD_TRUSTED
add interface=wg_trgmobile list=WIREGUARD_TRUSTED
add interface=wg_trgmobile list=TRUSTED_LANS
add interface=wg_trgmobile list=INET_FWD_ALLOWED
++++++++++++++++++++++++++++
Also, it appears you have duplicates for the bellfibe connection?

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=PUBLIC_IP_RANGE.121 pref-src=0.0.0.0 routing-table=main scope=30
suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-bellfibe pref-src=0.0.0.0 routing-table=main scope=30
suppress-hw-offload=no target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=pppoe-bellfibe pref-src=0.0.0.0 routing-table=main scope=30
suppress-hw-offload=no target-scope=10

hoeser · July 18, 2022, 9:48pm

It appears when I was messing with the config prior to the export I pushed one of those bell routes to main that should be on route table “bell”.

Gateway selection is done by some mangle rules . It’s in very home lab test state right now and I realize it’s a mess. The bell connection is new and the other has been there for years. I’ve been experimenting for a few days.

The Thad WireGuard is essentially a lan bridge to a fellow home lab. The trgmobile is the road warrior WireGuard which will eventually be my devices talking to home when I’m on the road.

I am recognizing now I will probably need routing rules by subnet.

hoeser · July 18, 2022, 10:17pm

Essentially my plan was

All traffic out the cable/cogeco interface except traffic marked for bell via a mangle rule based on a source address list. That mangle rule is listed and pushes the traffic onto the bellfibe route table.

This was obviously the least disruptive because it didn’t require much thought about subnetting off the lan by intended path to the internet. And this worked fairly well until the WireGuard issue at the router level.

I also experimented with nth connection balancing on some hosts (which also worked fine).

The problem is just getting the WireGuard into and out of the bell interface when it is not the least weighted default gateway on the router itself and is not (really) part of the main route table.

anav · July 19, 2022, 11:45am

Ensuring the wireguard warrior can access the router subnets does not require mangling.

Than_wg I do not understand about this lan bridge AT ALL.
There is no such LAN on your local router and if there is such a LAN on a different router somewhere, how are you connecting to this remote LAN?

Can you draw a diagram and maybe it will be clearer as well.

Thus far I only see the need for one subnet NOT AN ADDRESS LIST, to go out wireguard but that subnet does not exist on your router??

hoeser · July 19, 2022, 1:26pm

I never said the wireguard roadwarrior access required mangling.

I said I used the mangling to control the wan access for internal lan machines out specific gateways. The wireguard has nothing to do with the mark-routing mangling. The lan is 10.0.0.0/22 and some of those machines were going out the fiber (bell) interface based on mangle rules assinging the bellfibe routing mark, based on an address list BELL-FIBE-CLIENTS.

The wireguard lan bridge is simple… wg_thad has a /30 p2p subnet on it and at the other end is the 3011 peer. 192.168.1.0/24 is routed over that link. Thad’s LAN is 192.168.1.0. He has a static route on his side to my 10.0.0.0/22 and I have a static route to his 192.168.1.0/24.

What I understand now is that it is a very fundamental problem and is on the router itself. Looking at my entire config is superfluous to the actual core issue. The problem is caused by the router being configured with two default gateways on the main table - one is Cogeco with a Distance of 1. The other is bell with a distance of 2. I allowed (input) 52521 (wireguard udp) on the bell interface - this allows incoming wireguard traffic just fine on that interface, but wireguard on the router always preferred to send wireguard output out the cogeco link because the distance is shorter. For example, icmp ping over the wireguard link would come inbound on the Bell and the reply would actually go out the Cogeco interface.

I ended up going with the simplest fix and swapped the distances between the two so the router always prefers the Bell link. This works because I don’t have a need for the router to prefer the cogeco link at this time. I’m not saying there isn’t a better way of doing this… I even tried output mangling on the router side to force traffic from wireguard on the router out the bell interface - This actually sort of worked, but looking at the packet sniffs the router was sending out the traffic on the bell interface with a src address of the cogeco interface… and the replies actually would come back on the cogeco interface (insane/terrible but it worked).

What I ended up doing was swapping distance between 0.0.0.0/0 (cogeco) and 0.0.0.0/0 (bell) on the main table to ensure the router’s traffic prefers the Bell link. This solves the problem I was having with my wireguard traffic at the router level leaving on the incorrect (what I should say is “not desired”) link.

Sorry that I don’t have time to diagram this at scale because I’m ironically at work making diagrams. At any rate, the problem is mostly solved and I appreciate the sanity checks and the flogging.