PBR to ensure traffic coming from WireGuard tunnel goes back to WireGuard tunnel rOS v7.6

guilhermegsl1 · December 31, 2022, 1:59pm

Hello guys. I’ve been trying to setup a routing policy using mangle rules to ensure my traffic coming from VPN interface is answered and goes back through the VPN.
I have RB750Gr3 (R1) with public static WAN IP and hAC ac² (R2) under another ISPs CGNAT. I’ve successfully setup wireguard tunnel between them, R1 LAN is 192.168.98.0/24 and R2’s 10.1.0.1/24. Allowed IPs in R1 peer is 10.1.0.0/24 and in R2 allowed IPs is set 0.0.0.0/0. I have the following mangle rules and routes:

/ip firewall mangle
add action=accept chain=prerouting connection-mark=no-mark connection-state=established,related
add action=mark-connection chain=prerouting in-interface=wireguard1 new-connection-mark=wireguard1-conn passthrough=no
add action=mark-routing chain=prerouting connection-mark=wireguard1-conn new-routing-mark=rtab-1 passthrough=no
add action=mark-connection chain=input in-interface=wireguard1 new-connection-mark=wireguard1-input-conn passthrough=no
add action=mark-routing chain=output connection-mark=wireguard1-input-conn log=yes new-routing-mark=rtab-1 passthrough=no

/routing table
add comment="wireguard1" disabled=no fib name=rtab-1

/ip route
add comment="wireguard1" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=wireguard1 pref-src="" routing-table=rtab-1 scope=30 suppress-hw-offload=no target-scope=10

Also have 0.0.0.0/0 with pppoe-out1 as gateway for my wan connection.

I have a DVR I want to access through my R1’s public ip address. I created a dst-nat rule on R1 for that:

/ip firewall nat
add action=dst-nat chain=dstnat comment="DVR R2" dst-port=52002 in-interface-list=WAN protocol=tcp to-addresses=10.1.0.250

The connection to DVR works perfectly when I’m on R1’s site, don’t even need nat rule. The problem is when I’m accessing from WAN IP. If I place the output mark-routing rule using the same connection mark as the prerouting one, I can’t access the DVR from R1’s public IP anymore and get type 3, code 4 ICMP on R2’s log. That’s why I added the second connection mark in input chain and output routing rule, without the output routing mark I loose access to R2’s from wireguard.
Also I couldn’t figure out why, when I enable passthrough in mark-connection rules, they stop working. From what I have been reading about, it should NOT work when passthrough is disable.

When I’m in R1’s site I use R2’s LAN IP (10.1.0.1) to access it. I don’t use IPs on the tunnel itself.

Here is a full export of the R2 config:

/interface bridge
add comment=Admin name=bridge1

/interface pppoe-client
add add-default-route=yes comment="Link 1" disabled=no interface=ether1 max-mru=1480 max-mtu=1480 name=pppoe-out1 use-peer-dns=yes user=mypppoeuser

/interface wireguard
add comment=S2S listen-port=13232 mtu=1420 name=wireguard1

/interface vlan
add comment=Guest interface=bridge1 name=vlan100 vlan-id=100

/interface ethernet switch port
set 1 default-vlan-id=1 vlan-mode=secure
set 2 default-vlan-id=1 vlan-mode=secure
set 3 default-vlan-id=1 vlan-mode=secure
set 4 default-vlan-id=1 vlan-mode=secure
set 5 default-vlan-id=1 vlan-mode=secure

/interface list
add name=WAN

/ip pool
add comment=Admin name=pool1 ranges=10.1.0.100-10.1.0.200
add comment=Guest name=pool2 ranges=192.168.100.100-192.168.100.200

/ip dhcp-server
add add-arp=yes address-pool=pool1 comment=Admin interface=bridge1 lease-time=12h name=server1
add add-arp=yes address-pool=pool2 comment="Guest VLAN100" interface=vlan100 name=server2

/queue simple
add comment="Limit Guest VLAN100" max-limit=15M/15M name=queue1 queue=pcq-upload-default/pcq-download-default target=vlan100

/routing table
add comment="wireguard1" disabled=no fib name=rtab-1

/interface bridge port
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
add bridge=bridge1 interface=ether4
add bridge=bridge1 interface=ether5
add bridge=bridge1 interface=wlan1

/interface ethernet switch vlan
add independent-learning=no ports=switch1-cpu,ether5 switch=switch1 vlan-id=100
add independent-learning=no ports=switch1-cpu,ether2,ether3,ether4,ether5 switch=switch1 vlan-id=1

/interface list member
add interface=pppoe-out1 list=WAN

/interface wireguard peers
add allowed-address=0.0.0.0/0 comment="ap wg1" endpoint-address=x.x.x.x endpoint-port=13232 interface=wireguard1 persistent-keepalive=30s public-key="***"

/ip address
add address=10.1.0.1/24 comment=Admin interface=bridge1 network=10.1.0.0
add address=192.168.100.1/24 interface=vlan100 network=192.168.100.0

/ip dhcp-server lease
add address=10.1.0.250 client-id=1:d8:77:8b:fe:17:56 comment=DVR mac-address=xxxxxxxxxxxx server=server1

/ip dhcp-server network
add address=10.1.0.0/24 comment=Admin dns-server=10.1.0.1 gateway=10.1.0.1 netmask=24 ntp-server=10.1.0.1
add address=192.168.100.0/24 comment=Guest dns-server=10.1.0.1 gateway=192.168.100.1 netmask=24 ntp-server=10.1.0.1

/ip dns
set allow-remote-requests=yes cache-size=6144KiB max-udp-packet-size=512 servers=8.8.8.8

/ip firewall filter
add action=fasttrack-connection chain=forward connection-mark=!fasttrack-off hw-offload=yes in-interface-list=WAN
add action=accept chain=input comment=DNS dst-port=53 in-interface-list=!WAN protocol=udp
add action=accept chain=input dst-port=53 in-interface-list=!WAN protocol=tcp
add action=accept chain=input comment=NTP dst-port=123 in-interface-list=!WAN protocol=udp
add action=accept chain=input comment="wireguard1" in-interface=wireguard1
add action=accept chain=forward in-interface=wireguard1
add action=drop chain=input comment="Isolate Guest VLAN100" in-interface=vlan100
add action=accept chain=input connection-state=established,related in-interface=vlan100
add action=drop chain=forward in-interface=vlan100 out-interface-list=!WAN
add action=accept chain=forward in-interface=vlan100
add action=accept chain=input comment="Input" connection-state=established,related in-interface-list=WAN
add action=drop chain=input in-interface-list=WAN
add action=accept chain=forward comment="LAN" connection-state=established,related in-interface-list=WAN
add action=drop chain=forward connection-nat-state=!dstnat in-interface-list=WAN

/ip firewall mangle
add action=accept chain=prerouting connection-mark=no-mark connection-state=established,related
add action=mark-connection chain=prerouting comment="wireguard1" in-interface=wireguard1 new-connection-mark=wireguard1-conn passthrough=no
add action=mark-routing chain=prerouting connection-mark=wireguard1-conn new-routing-mark=rtab-1 passthrough=no
add action=mark-connection chain=input in-interface=wireguard1 new-connection-mark=wireguard1-input-conn passthrough=no
add action=mark-routing chain=output connection-mark=wireguard1-input-conn log=yes new-routing-mark=rtab-1 passthrough=no

/ip firewall nat
add action=redirect chain=dstnat comment="Redirect DNS" dst-port=53 in-interface-list=!WAN protocol=udp
add action=redirect chain=dstnat dst-port=53 in-interface-list=!WAN protocol=tcp
add action=accept chain=srcnat comment="WireGuard S2S" out-interface=wireguard1
add action=masquerade chain=srcnat comment="Internet" out-interface-list=WAN

/ip route
add comment="wireguard1" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=wireguard1 pref-src="" routing-table=rtab-1 scope=30 suppress-hw-offload=no target-scope=10

I appreciate any help or suggentions
Thanks in advance.
Regards

anav · December 31, 2022, 6:31pm

Wrong approach…

Also vague requrirements. I made the assumption that AC2 users need to go out R1 for internet… If not then we can discuss later, but after you make changes.
You need to post complete config of both this snippet nonsense is not helpful.
/export file=anynameyouwish ( minus router serial # and and public WANIP information )

In any case, lets recap.,
The AC2 is the client for the connection and the idea is that users from the hapac2 need to reach the main router and the internet of the main router hence the 0.0.0.0/0 for allowed peers to the main router on the AC2. Later on its discovered that the incoming users to AC2 also point to 0.0.0.0/0 for peer settings as remote users from R1 and perhaps the internet will need to access a server on AC2.

To ensure LAN users on AC2 can reach the main router.

table - check
route - check
routing rule ???

/routing rule add src-address=10.1.0.0/24 action=lookup table=rtab-1
Note: If you want users never to be able to access local wan in case tunnel is down, then use action=lookup-only-in-table option.

Just to be on the safe side add this extra routing rule, order is important
/routing rule add dst-address=10.1.0.1 action=lookup table=main { this will allow the admin on the subnet to reach the local router AC2 for config purposes }
/routing rule add src-address=10.1.0.0/24 action=lookup table=rtab-1

Firewall rules NEED WORK.
Wrong approach, turn fastrack back on…
I also despise over use of ! symbol…
Also why is there a wireguard input chain rule as this is the CGNAT which has no accessible IP…
Why do your guests have full access to the router…??

Cleaned up for you
/firewall filter rules
{input chain}
add action=accept chain=input comment=“defconf: accept established,related,untracked” connection-state=established,related,untracked
add action=drop chain=input comment=“defconf: drop invalid” connection-state=invalid
add action=accept chain=input comment=“defconf: accept ICMP” protocol=icmp
add action=accept chain=input in-interface-list=authorized
add action=accept chain=input comment=“Allow LAN DNS queries-UDP” \ {and NTP *** services if required etc}
dst-port=53,123 in-interface-list=LAN protocol=udp
add action=accept chain=input comment=“Allow LAN DNS queries - TCP”
dst-port=53 in-interface-list=LAN protocol=tcp
add action=drop chain=input comment="drop all else
{forward chain}
add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related
add action=accept chain=forward comment=“defconf: accept established,related, untracked” connection-state=established,related,untracked
add action=drop chain=forward comment=“defconf: drop invalid” connection-state=invalid
add action=accept chain=forward comment=“allow internet traffic” in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment=“allow port forwarding” connection-nat-state=dstnat
add action=accept chain=forward src-address=10.1.0.0/24 out-interface=wireguard1
add action=accept chain=forward in-interface=wireguard1 dst-address=10.1.0.250/32 { to permit access to server }
add action=drop chain=forward comment=“drop all else”

No mangling required and removed.

Note: Authorized access list is those who will be accessing the router for config purpose.
Based on specific single IP addresses, could be
a. local admin IPs on 10.1.0.0/24 subnet
b. remote admin IPs coming from 192.168.98.0/24 subnet via wireguard
c. remote admin IPs (road warrior Wireguard to R1 and then to R2 via wireguard) for example let say you have a laptop for the road with wireguard address of 172.10.10.3/32 (see #7).

(5) YOu need to complete your interface list…
/interface list
add name=WAN
add name=LAN

/interface list member
add interface=pppoe-out1 list=WAN
add interface=bridge1 list=LAN
add interface=V100 list=LAN

(6) There is no need to masquerade wireguard traffic from MT to MT for this scenario at least on AC2
You only need the one rule.
add action=masquerade chain=srcnat comment=“Internet” out-interface-list=WAN

Not sure what you are trying to do with redirect of port 53, more explanation required on your intention so for now keeping it simple and removed.

(7) What is missing is the IP address for the Wireguard network… ???

Lets say we establish a proper wg working setup. Note I despise using same wirguard name on two different but connected devices, makes it confusing.
ON R1 the server for the connection
/ip address
add 172.10.10.1/24 interface=wireguardR1

ON AC2
add 172.10.10.2/24 interface=wireguardR2

(8) Routes for any traffic coming from R1, a route is already created via the IP address for pinging and config access purposes from R1, but what aobut subnet traffic.
Any traffic coming from R1 hits the local LAN and needs a return path.
/ip route
add dst-address=192.168.98.0/24 gwy=wireguardR2 table=main

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

As for R1…
I have the same issues, I dont see a need for mangling yet.

For allowed IPs on R1 for the peer settings of AC2
allowed ips=172.10.10.2/32,10.1.0.0/24

If you had a road warrior lets say a laptop
another peer entry would be
allowed-ips=172.10.10.3/32

You will need an an additional route for R2 traffic to get back in the tunnel or any local traffic/internet traffic heading to the server
/ip route
add dst-address=10.1.0.0/24 gwy=wireguard1 table=main.

I do not see a need for Table and Routing rule or anything fancy on R1 yet…!!!

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

So with that out of the way we can focus on the DVR issue.
Its not clear where this DVR is located but from your config entry the DVR is located on AC2.

You want local users on R1 and internet users via WAN on R1 to access DVR.
Hence like the firewall rules for AC2 you will need a rule such as.

You will need port forwarding rule in the forward chain.
add action=accept chain=forward comment=“allow port forwarding” connection-nat-state=dstnat

The format of the dstnat rule…
add action=srcnat chain=src-nat dst-address=staticpublicWANIP dst-port=xxxxxx protocol=yyy
to-address=10.1.0.250 to-ports required only if different ports (aka port translation required)

FOR SOURCNAT this is where it gets tricky…

Since the server is not local you dont need hairpin nat rule.
So the standard source nat rule is required.
However the dilemma are the external users coming in from the internet.
They will be pushed out the tunnel as their destination address matches that in the allowed IPs, —> 10.1.0.250
Due to allowed IPs at ac2 of 0.0.0.0/0 they will be accepted at that end.

The problem is that the AC2 will not know where to send the return traffic as its unknown ahead of time.
The solution is to source nat all traffic heading out the wireguard tunnel…
Thus here are your two sourcenat rules.

add action=src-nat chain=srcnat out-interface=ether1 to-address=publicIPofWAN
add action=masquerade chain=srcnat out-interface=wireguard1 src-address=!192.168.98.0/24

Since you have a fixed wanIP the standard rule is modified slightly.
The second rule ensures that all internet traffic entering the tunnel is given a source address of 172.10.10.1( if its not from the local subnet on R1) when it reaches AC2 and thus the return traffic will be returned to R1 which will un-sourcnat the traffic back…

In this way, you can easily distinguish what is R1 traffic at R2 if that is needed.

There is additional work required most likely for firewall rules, but as stated earlier clean up both configs and then post them for review.

guilhermegsl1 · December 31, 2022, 11:01pm

Thanks for answering anav. Right now can’t access the routers and make changes but I think I couldn’t explain the problem very well, also not a good english speaker/writer here.

First of all, I could ask for a public IP on R2 for my ISP, would cost a little bit more and not use wireguard from R1, but I’m trying to setup it like this to learn more about VPN and routing.

I have R1, with public wan IP, let’s say it’s 1.1.1.1. I have R2 (ac²) under cgnat and need to access devices connected on it’s bridge1 (10.1.0.0/24). One of these devices is my DVR (10.1.0.250:52002/tcp). I have set up site to site wireguard between them successfully, I can access R2 (10.1.0.1/24) from R1 (192.168.98.0/24). I was using a static route:

R1 (192.168.98.0/24)

add disabled=no distance=1 dst-address=10.1.0.0/24 gateway=wireguard1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10

R2 (10.1.0.0/24)

add disabled=no distance=1 dst-address=192.168.98.0/24 gateway=wireguard1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10

I can access both sites from the other including the router (192.168.98.1 and 10.1.0.1). The problem I am trying to solve is: I need to access this DVR from outside the R2’s LAN and have public IP on R1 and a VPN working. What I tried then:

My firewall on R1 doesn’t drop if traffic is dst-nated, so I created a rule on R1:

add action=dst-nat chain=dstnat comment="To DVR" dst-port=52002 in-interface-list=WAN protocol=tcp to-addresses=10.1.0.250

With this the dvr receives a connection when I connect to my public 1.1.1.1 IP on R1 using port 52002, but it arrives with source address being the address of whatever i’m connected to (mobile data, a public wifi or whatever not being 192.168.98.0/24 which is the route I have set in R2 to go through wireguard tunnel). Because of that R2 can’t route it through wireguard beacause the route is just 192.168.98.0/24. Having that in mind, I tried to setup a PBR (Could be called PBR?) based on connection mark. I don’t want to route internet traffic from R2 through R1 using the wireguard tunnel, but I want to route to any address that starts a connection from wireguard tunnel to 10.1.0.0/24 to be routed back to wireguard tunnel. I didn’t use a routing rule because I didn’t find a way to do this. I can use a routing rule to route traffic back to 192.168.98.0/24 ans scr-nat traffic in R1 to 192.168.98.1, but wouldn’t it be better if R2 just could answer from any ip through the tunnel? I know that it will require more cpu than a routing rule, but as the routers being all mine, just home use, this wouldn’t be a problem for me.

The mangling I set up is working, I can access 10.1.0.1 from 192.168.98.0/24, I can access 10.1.0.0/24 from 192.168.98.0/24, I can access 10.1.0.250:52002 from 192.168.98.0/24 or from any internet ip address using the R1 public IP and port. Connection arrives to 10.1.0.250 with src-address being the internet address and is routed through wireguard to R1, R1 does the src-nat to the internet back and all works, but it looks I’m using more rules than it should be necessary to do so. I achieved what I needed but I think by the wrong way.

Thanks again, will post the exports probably during the week when I can play a little more with the routers.
Regards