Community discussions

MikroTik App
 
mikbrew
just joined
Topic Author
Posts: 12
Joined: Tue Oct 27, 2015 9:14 pm

Route Wireguard traffic through specific WAN interface

Thu Nov 03, 2022 4:29 pm

Well met, ye Masters of the Advanced Networking Dimension. We've travelled many moons from the lands of Africa seeking your sagely advice.

We have a multi-WAN setup and want to route all Wireguard traffic through a specific WAN interface, pppoe-vpn, which is assigned a static IP at ISP level. Here's the network diagram:
network-diagram.PNG

And this is the config:
# RouterOS 7.6
# model = RB750Gr3

/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment=defconf name=bridge

/interface pppoe-client
add disabled=no interface=ether1 name=pppoe-backup use-peer-dns=yes
add disabled=no interface=ether1 name=pppoe-vpn use-peer-dns=yes
add disabled=no interface=ether1 name=pppoe-wa use-peer-dns=yes

/interface wireguard
add listen-port=54321 mtu=1420 name=wireguard0

/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN

/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik

/ip hotspot profile
set [ find default=yes ] html-directory=hotspot

/ip pool
add name=dhcp ranges=192.168.10.101-192.168.10.254

/ip dhcp-server
add address-pool=dhcp interface=bridge lease-time=12h name=dhcp0

/port
set 0 name=serial0

/routing table
add disabled=no fib name=vpn

/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5

/ip neighbor discovery-settings
set discover-interface-list=LAN

/ip settings
set rp-filter=loose

/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=pppoe-vpn list=WAN

/interface wireguard peers
add allowed-address=192.168.9.3/32 comment=mac interface=wireguard0 \
    persistent-keepalive=25s

/ip address
add address=192.168.10.83/24 comment=defconf interface=bridge network=\
    192.168.10.0
add address=192.168.9.83 comment="wireguard interface local  ip" interface=\
    wireguard0 network=192.168.9.0

/ip cloud
set ddns-update-interval=15m

/ip dhcp-client
add comment=defconf disabled=yes interface=ether1

/ip dhcp-server lease
add address=192.168.10.93 lease-time=23h59m59s mac-address=yy:yy:yy:yy:yy:yy \
    server=dhcp0 use-src-mac=yes

/ip dhcp-server network
add address=192.168.10.0/24 comment=defconf dns-server=192.168.10.83 gateway=\
    192.168.10.83 netmask=24

/ip dns
set allow-remote-requests=yes

/ip dns static
add address=192.168.10.83 comment=defconf name=router.lan

/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="accept wireguard" dst-port=54321 \
    in-interface=pppoe-vpn protocol=udp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    disabled=yes ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    disabled=yes ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related disabled=yes hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN

/ip firewall mangle
add action=mark-routing chain=prerouting comment=\
    "mark wireguard connections - prerouting" dst-port=54321 \
    new-routing-mark=vpn passthrough=no protocol=udp
add action=mark-routing chain=output comment=\
    "mark wireguard connections - prerouting" new-routing-mark=vpn \
    passthrough=no protocol=udp src-port=54321

/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface=pppoe-vpn
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface=pppoe-wa

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-wa pref-src="" \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-vpn \
    routing-table=vpn scope=30 suppress-hw-offload=no target-scope=10

/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6

/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=\
    33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN

/system clock
set time-zone-name=Africa/Johannesburg

/tool mac-server
set allowed-interface-list=LAN

/tool mac-server mac-winbox
set allowed-interface-list=LAN

/tool sniffer
set filter-interface=all filter-ip-protocol=udp filter-port=54321

Disabling these 2 mangle rules...
/ip firewall mangle
add action=mark-routing chain=prerouting comment=\
    "mark wireguard connections - prerouting" dst-port=54321 \
    new-routing-mark=vpn passthrough=no protocol=udp
add action=mark-routing chain=output comment=\
    "mark wireguard connections - prerouting" new-routing-mark=vpn \
    passthrough=no protocol=udp src-port=54321

...and attempting to connect a peer through the static IP yields the following via the packet sniffer (actual listening port redacted but let's pretend it's 54321 as per above config):

sniffer-default.PNG

The packet is received through the pppoe-vpn interface but is responded to from the pppoe-wa interface - the default interface. As a result, no handshake takes place.

Enabling the 2 mangle rules yields the following result:
sniffer-3.PNG


The packets are being responded to on the correct interface but
1. There is a big delay in the response.
2. The response is not being sent through the same port it was requested from - the peer is connecting from port 24099 but the response is being sent to port 22505.

As a result, no handshake takes place.

Some info that may or may not be relevant:
1. The static IP of interface pppoe-vpn is assigned at ISP level. On the router itself, we get assigned a dynamic NAT'ed (we beg pardon if this isn't the correct term) IP (10.230.68.216 in this case) and we don't see the actual static IP anywhere on the router.
2. When using pppoe-vpn as the default gateway using the default routing table, the handshake takes place without issue.

Please advise, oh wise Ones - what are we doing wrong?
You do not have the required permissions to view the files attached to this post.
Last edited by mikbrew on Thu Nov 03, 2022 5:41 pm, edited 2 times in total.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19125
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Route Wireguard traffic through specific WAN interface

Thu Nov 03, 2022 4:44 pm

To be clear, is your wireguard instance on the router acting as a server for handshake or is it connecting to something else as a client?
What is the purpose of the wireguard. So far it seems the requirement is for local users to go out a specific WAN but to what for example.

Also you are missing an IP address for wireguard in the config.
 
mikbrew
just joined
Topic Author
Posts: 12
Joined: Tue Oct 27, 2015 9:14 pm

Re: Route Wireguard traffic through specific WAN interface

Thu Nov 03, 2022 4:53 pm

To be clear, is your wireguard instance on the router acting as a server for handshake or is it connecting to something else as a client?
The Wireguard instance on the router is indeed acting as a server for handshake (that's the intention, anyway).

What is the purpose of the wireguard. So far it seems the requirement is for local users to go out a specific WAN but to what for example.
The purpose is for peers to connect to the local network remotely, and access hosts within the local network, for remote work.

Also you are missing an IP address for wireguard in the config.
Noted and will add an IP address under
/ip addresses
for the Wireguard interface - the plan was to sort that out after establishing a reliable handshake.
 
mikbrew
just joined
Topic Author
Posts: 12
Joined: Tue Oct 27, 2015 9:14 pm

Re: Route Wireguard traffic through specific WAN interface

Thu Nov 03, 2022 5:20 pm

Realised an error in the mangle rules -
src-address-type=""
After removing this condition, the packet sniffer now looks like this:

sniffer-3.PNG
The packets are being responded to on the correct interface but
1. There is a big delay in the response.
2. The response is not being sent through the same port it was requested from - the peer is connecting from port 24099 but the response is being sent to port 22505.

As a result, still no handshake takes place.

Updating the OP with this info as well.
You do not have the required permissions to view the files attached to this post.
Last edited by mikbrew on Thu Nov 03, 2022 5:34 pm, edited 1 time in total.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19125
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Route Wireguard traffic through specific WAN interface

Thu Nov 03, 2022 5:27 pm

(1) Why isnt the primary WAN pppoe not part of the wan list??
In other words
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=pppoe-vpn list=WAN

add interface=pppoe-wa list=WAN

(2) Persistent keep alive for the incoming remote user .3/32 makes no sense. Should be set at the client.

(3) When you construct your IP address say 192.168.9.1/24 this will cause the router to auto generate a routing.
<dac> dst-address=192.168.3.0/24 gwy=wireguard0 table=main

This means that when the remote users access the LAN for traffic, the return traffic will be correctly routed through the tunnel that was established over the VPN WAN link.

(4) Where more work would be required is if:
a. you wanted remote users to go out your internet like out pppoe-wa for instance, OR
b. you wanted local user to go out wireguard (lets say if you had another router as a client and there were remote subnets local users needed to reach).


(5) Finally yes disable the mangle rules as their need is not clear yet but may be needed later or some variant of.

Now looking at your current route rules for pppoe, wireguard handshake is not working, maybe due to the that fact that you don't have the standard route for pppoe-vpn so if you have multiple wans you at least need the standard routing for each WAN, for standard traffic to the router etc.

FROM:
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-wa pref-src="" \
routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-vpn pref-src=\
"" routing-table=vpn scope=30 suppress-hw-offload=no target-scope=10
TO:
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-wa table=main
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=pppoe-vpn table=main

In this regard normal traffic will flow and the handshake should work. Note I added a distance of 2, to your vpn network.
This will ensure all users go out the pppoe-wa WAN connection for internet.
 
mikbrew
just joined
Topic Author
Posts: 12
Joined: Tue Oct 27, 2015 9:14 pm

Re: Route Wireguard traffic through specific WAN interface

Thu Nov 03, 2022 6:37 pm

anav's much appreciated Sagely advice...

OK, have added all pppoe interfaces to WAN list, removed persistent keep alive for peer (it was indeed also set on client), added IP for Wireguard interface (which added a dynamic routing to the main routing table), disabled the mangle rules, and updated the pppoe-vpn route.

Config now looks like:
# RouterOS 7.6
# model = RB750Gr3

/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment=defconf name=bridge

/interface pppoe-client
add disabled=no interface=ether1 name=pppoe-backup use-peer-dns=yes
add disabled=no interface=ether1 name=pppoe-vpn use-peer-dns=yes
add disabled=no interface=ether1 name=pppoe-wa use-peer-dns=yes

/interface wireguard
add listen-port=54321 mtu=1420 name=wireguard0

/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN

/routing table
add disabled=no fib name=vpn

/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5

/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=pppoe-vpn list=WAN
add interface=pppoe-wa list=WAN
add interface=pppoe-backup list=WAN

/interface wireguard peers
add allowed-address=192.168.9.3/32 comment=mac interface=wireguard0

/ip address
add address=192.168.10.83/24 comment=defconf interface=bridge network=\
    192.168.10.0
add address=192.168.9.83/24 comment="wireguard interface local  ip" \
    interface=wireguard0 network=192.168.9.0

/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="accept wireguard" dst-port=54321 \
    in-interface=pppoe-vpn protocol=udp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    disabled=yes ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    disabled=yes ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related disabled=yes hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN

/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    out-interface-list=WAN

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-wa pref-src="" \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=pppoe-vpn pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10

Packet sniffer shows packet coming in through pppoe-vpn but response still goes out through the default gateway, pppoe-wa:

sniffer-4.PNG
And no handshake.

Is this masquerade rule correct?
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    out-interface-list=WAN
You do not have the required permissions to view the files attached to this post.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19125
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Route Wireguard traffic through specific WAN interface  [SOLVED]

Thu Nov 03, 2022 7:06 pm

Yes masquerade rule is correct, it was me that erred.
Lets fix this for you.

By the way what is with .83 LOL, its when I graduated University, so its a nice number!

Please add this third route and routing rule........
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-wa pref-src="" \
routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=pppoe-vpn pref-src=\
"" routing-table=main scope=30 suppress-hw-offload=no target-scope=10

add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=pppoe-vpn table=vpn

/routing rule
add interface=src-address=IPaddress (of pppoe-vpn) action=lookup-only-in-table table=vpn

That should allow the handshake
Problem is that that IP is not static right ???
If not then you will need a script to keep it correct.
 
mikbrew
just joined
Topic Author
Posts: 12
Joined: Tue Oct 27, 2015 9:14 pm

Re: Route Wireguard traffic through specific WAN interface

Thu Nov 03, 2022 7:42 pm

Lets fix this for you.
Brilliant. Handshake now takes place and client peer can remotely access local network and hosts. Thank you Sagely One, much appreciated.

By the way what is with .83 LOL, its when I graduated University, so its a nice number!
We preferred 83 over the default 88 because 83 > 88.

Problem is that that IP is not static right ???
If not then you will need a script to keep it correct.
Indeed. Have seen script guides floating about - will consult one and implement.

Client peer's internet is also going out via pppoe-wa. This is what we want, but for the record just wondering what about the setup is allowing this? Since earlier you stated:
(4) Where more work would be required is if:
a. you wanted remote users to go out your internet like out pppoe-wa for instance, OR...
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19125
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Route Wireguard traffic through specific WAN interface

Thu Nov 03, 2022 8:35 pm

That is handled mostly by firewall rules.
Since yours are terrible and by that I mean loosey goosey, I dont think there will be an issue.
If you had better rule sets, then one would have to consider ensuring that specific traffic was permissible.

The MT OS default construct is to block a few known things but allow everything else all other traffic., known or unknown!
I always recommend moving to the construct of only allowing known things and blocking everything else, known or unknown.

For example.
add chain=forward action=accept in-interface=wg dst-address=local_subnet
add chain=forward action=accept in-interface=wg out-interface-list=WAN
(which can be done by adding wg to LAN interface list, and thus would follow this rule
add chain=forward action=accept in-interface-list=LAN out-interface-list=WAN



If one of your clients was another mT router with subnets, one would have to consider additional routes so the router knows were to send return traffic (coming from internet or local subnet devices).
 
mikbrew
just joined
Topic Author
Posts: 12
Joined: Tue Oct 27, 2015 9:14 pm

Re: Route Wireguard traffic through specific WAN interface

Thu Nov 03, 2022 9:20 pm

Sagely One performs a Yurchenko double pike like it ain't no thang...

Understood. Your ways make more sense than those we're accustomed to so we shall gladly adopt them. We appreciate your guidance.

Who is online

Users browsing this forum: nik247 and 24 guests