Community discussions

MikroTik App
 
dalami
Member Candidate
Member Candidate
Topic Author
Posts: 134
Joined: Mon Dec 12, 2011 9:18 am

Route internet through Wireguard

Thu Nov 24, 2022 9:34 pm

Not only is this not working - I don't understand *why* it is not working. There's obviously something basic here I don't get. Please use whatever size hammer is required to drive the point into my thick head.

I have an office network with a hEX that behaves quite nicely. This has a static IP connection to the internet. LAN is 192.168.0.0/24 and it serves Wireguard as 10.23.1.0/24.
I have a cloud server running Ubuntu Linux - also has a static IP. This serves Wireguard as 10.23.2.0/24. This also connects to the office as 10.23.1.10.

Previously my hEX (actually, another MT device behind the hEX but ignore that for now) was the Wireguard server for my remote access. I have moved that role to my cloud server - it has a much faster internet connection. I retain a WG connection between my cloud server and my office hEX.

For...reasons...I need to be able to reach the internet (well...really only 1 external IP but the principle is the same) from my cloud server based clients via my office IP. So the path is <wg peer> --> <cloud server> --> <office hEX> --> internet

Straight wg connections work just fine. From within my office network I can reach the internet in general and the target IP in particular perfectly. From my cloud server I can reach the internet directly and my office network perfectly, also cloud wireguard peers work fine accessing the office - yet every attempt I've made to perform the specific routing fails. It looks from a traceroute I hit the hEX on the wg IP and it just disappears. I tried doing some packet sniffing and I *think* the issue is srcnat is not being applied - I just don't understand why.

Looking at other posts it seems I *may* need to start learning policy routing for this - I've experimented a bit and so far no luck.

My excerpted export below. I deleted most of what I thought was irrelevant. I left some of my filtering rules in place just in case they're either relevant or worthy of comment.

On the subject of understanding why - looking at my srcnat rules you'll see three rules commented "Wireguard to others". The multiple rules are my just throwing things at this to try to get it to work but...I see traffic on all three rules. The first rule references the interface directly - how do the other IP based rules ever get processed?
# nov/24/2022 10:29:13 by RouterOS 7.5rc2
# software id = 6MPE-115F
#
# model = RB760iGS
/interface wireguard
add listen-port=<mywgport> mtu=1420 name=wg0
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/routing table
add disabled=yes fib name=wireguard
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1-Internet list=WAN
/interface wireguard peers
add allowed-address=10.23.1.10/32,10.23.2.0/24 \
    endpoint-address=<my.linux.wg> endpoint-port=<mywgport> interface=wg0 \
    persistent-keepalive=25s public-key="myverysuperspecialsecret"
/ip address
add address=<my.pub.ip>/30 comment="Public IP" interface=ether1-Internet \
    network=<my.pub.net>
add address=192.168.0.1/24 comment="Primary LAN" interface=bridge network=\
    192.168.0.0
add address=10.23.1.1/24 interface=wg0 network=10.23.1.0
/ip dns
set servers=192.168.0.2,1.1.1.1,8.8.8.8
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall filter
add action=drop chain=input comment="drop temporary IPSEC blacklist" \
    src-address-list=ipsec-temp-blacklist
add action=accept chain=input in-interface=wg0
add action=accept chain=input comment=\
    "accept established,related...and untracked" connection-state=\
    established,related,untracked
add action=accept chain=input comment=\
    "Allow IPSEC traffic (established via policy)" in-interface=\
    ether1-Internet ipsec-policy=in,ipsec
add action=accept chain=input comment="Accept Wireguard" dst-port=<mywgport> \
    in-interface=ether1-Internet protocol=udp
add action=accept chain=input comment="Accept new,untracked from Bridge" \
    connection-state=new,untracked in-interface=bridge
add action=accept chain=input comment="Allow self-connections - for The Dude" \
    dst-address-type=local src-address-type=local
add action=accept chain=input comment="Allow IPSEC connections" dst-port=\
    500,4500 in-interface=ether1-Internet protocol=udp
add action=accept chain=input comment="Allow IPSEC connections" in-interface=\
    ether1-Internet protocol=ipsec-esp
add action=accept chain=input comment=\
    "Allow IPSEC connections - seems unnecessary." in-interface=\
    ether1-Internet protocol=ipsec-esp
add action=accept chain=input comment="Accept SSH,Webcfg,Winbox from Bridge" \
    dst-port=22,80,8291 in-interface=bridge protocol=tcp
add action=accept chain=input comment="Accept SSH,Webcfg,Winbox from IPSec" \
    dst-port=22,80,8291 ipsec-policy=in,ipsec protocol=tcp
add action=accept chain=output comment="Section Break" disabled=yes
add action=drop chain=input comment="drop ssh brute forcers" \
    src-address-list=ssh_blacklist
add action=drop chain=input comment="dropping port scanners" \
    src-address-list="port scanners"
add action=drop chain=input comment="drop all from Internet via Fail2Ban" \
    in-interface=ether1-Internet src-address-list=fail2ban
add action=drop chain=input comment=\
    "dropping blacklist - Joshaven compiled Spamhaus, malc0de, dshield" \
    src-address-list=blacklist
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=jump chain=input comment="Jump to RFC SSH Chain" connection-state=\
    new dst-port=22 in-interface=ether1-Internet jump-target="RFC SSH Chain" \
    protocol=tcp
add action=jump chain=input comment="Jump to RFC Port Scanners UDP Chain" \
    in-interface=ether1-Internet jump-target="RFC Port Scanners UDP Chain" \
    protocol=udp src-address-list=!non-public
add action=jump chain=input comment="Jump to RFC Port Scanners TCP Chain" \
    in-interface=ether1-Internet jump-target="RFC Port Scanners TCP Chain" \
    protocol=tcp src-address-list=!non-public
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="Block DHCP to/from Internet" \
    in-interface=ether1-Internet port=67-68 protocol=udp
add action=accept chain=input comment=\
    "Accept all Bridge input (safety to prevent being locked out)" \
    in-interface=bridge
add action=drop chain=input comment="drop all from Internet" in-interface=\
    ether1-Internet
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    disabled=yes in-interface-list=!LAN
add action=drop chain=input comment=\
    "Final Rule - shouldn't be needed but JIC CYA..."
add action=accept chain=forward comment=\
    "accept in ipsec policy - must be before fasttrack" ipsec-policy=in,ipsec
add action=accept chain=forward comment=\
    "accept out ipsec policy - must be before fasttrack" ipsec-policy=\
    out,ipsec
add action=accept chain=forward comment="Forward ProtectedExternal" \
    src-address-list=ProtectedExternal
add action=accept chain=forward comment=\
    "avoid fasttrack for OpenDrive (to use Simple Queue for rate limiting)" \
    connection-state="" packet-mark=OpenDrive
add action=accept chain=forward comment="Forward from Wireguard" \
    in-interface=wg0
add action=accept chain=forward comment="Forward to Wireguard" out-interface=\
    wg0
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment="dropping port scanners" \
    src-address-list="port scanners"
add action=drop chain=forward comment="drop all from Internet via Fail2Ban" \
    in-interface=ether1-Internet src-address-list=fail2ban
add action=drop chain=forward comment=\
    "dropping blacklist - Joshaven compiled Spamhaus, malc0de, dshield" \
    src-address-list=blacklist
add action=fasttrack-connection chain=forward comment="fasttrack non-ipsec" \
    connection-mark=!ipsec connection-state=established,related disabled=yes \
    hw-offload=yes
add action=accept chain=forward comment=\
    "Accept established,related - and untracked" connection-state=\
    established,related,untracked
add action=jump chain=forward comment="Jump to RFC Port Scanners TCP Chain" \
    jump-target="RFC Port Scanners TCP Chain" protocol=tcp src-address-list=\
    !non-public
add action=jump chain=forward comment="Jump to Vulnerable TCP Chain" \
    jump-target=vulnerabilities-tcp protocol=tcp src-address-list=!non-public
add action=jump chain=forward comment="Jump to RFC Port Scanners UDP Chain" \
    jump-target="RFC Port Scanners TCP Chain" protocol=udp src-address-list=\
    !non-public
add action=jump chain=forward comment="Jump to Vulnerable UDP Chain" \
    jump-target=vulnerabilities-udp protocol=udp src-address-list=!non-public
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
add action=drop chain=forward comment="drop all from fake LAN on Internet" \
    connection-nat-state=!srcnat connection-state="" in-interface=\
    ether1-Internet src-address-list=non-public
add action=accept chain=forward comment="Forward LAN-to-LAN" \
    dst-address-list=non-public in-interface=bridge out-interface=bridge
add action=accept chain=forward comment="Forward LAN-to-WAN" \
    dst-address-list=!non-public in-interface=bridge out-interface=\
    ether1-Internet
add action=accept chain=forward comment="Forward from WAN if DSTNATed" \
    connection-nat-state=dstnat connection-state=new in-interface=\
    ether1-Internet
add action=accept chain=forward comment="Forward ICMP from LAN" in-interface=\
    bridge protocol=icmp
add action=accept chain=output comment="Section Break" disabled=yes
/ip firewall nat
add action=src-nat chain=srcnat comment="Wireguard to others" out-interface=\
    wg0 to-addresses=10.23.1.1
add action=src-nat chain=srcnat comment="Wireguard to others" dst-address=\
    192.168.0.0/24 src-address=10.23.1.0/24 to-addresses=192.168.0.1
add action=src-nat chain=srcnat comment="Wireguard to others" dst-address=\
    !10.23.1.0/24 src-address=10.23.1.0/24 to-addresses=<my.pub.ip>
add action=src-nat chain=srcnat comment="Wireguard to others - never see packets here" dst-address=\
    10.23.1.0/24 src-address=<my.pub.ip> to-addresses=10.23.1.1
add action=src-nat chain=srcnat comment="LAN to Wireguard" dst-address=\
    10.23.1.0/24 src-address=192.168.0.0/24 to-addresses=192.168.0.1
add action=src-nat chain=srcnat comment=LAN out-interface=bridge \
    to-addresses=192.168.0.1
add action=src-nat chain=srcnat comment=\
    "Should be last srcnat rule. non-IPSec to Internet NAT to public IP" \
    ipsec-policy=out,none out-interface=ether1-Internet to-addresses=\
    <my.pub.ip>
add action=src-nat chain=srcnat comment="In theory - either this rule or the a\
    bove rule aren't needed. Because we have an explicit \"srcnat-accept\" rul\
    e above for the IPSec networks no \"IPSec policy\" matching should be requ\
    ired. Simple srcnat to public IP for internet access." out-interface=\
    ether1-Internet to-addresses=<my.pub.ip>
add action=masquerade chain=srcnat comment=\
    "Should be no forking reason for this. But."
/ip route
add comment="Default internet" disabled=no dst-address=0.0.0.0/0 gateway=\
    <my.pub.gw>
add comment="Cloud Wireguard network" disabled=no distance=1 \
	dst-address=10.23.2.0/24 gateway=10.23.1.10 pref-src="" \
	routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/routing rule
add action=lookup-only-in-table disabled=yes dst-address=<the.special.external.address> \
    interface=wg0 table=wireguard
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Route internet through Wireguard

Thu Nov 24, 2022 10:40 pm

From your collection of srcnat rules, all you should need is just one, and its #7. It covers access from anywhere to internet, so including from remote 10.23.2.x peer.

You don't need policy routing for this, access from 10.23.2.x/24 to internet is same as access from LAN to internet, it doesn't matter that it's behind another tunnel.

Firewall filter isn't stopping it either, it's allowed by forward rule #5 ("Forward from Wireguard").

So it should work. Perhaps something you deleted wasn't as irrelevant as you thought?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Route internet through Wireguard

Thu Nov 24, 2022 10:55 pm

You need to sort out the IP address usage so it more coherent for your wireguard as I think this is your major issue.

Proposing since you have this in the correct format to keep the hex at
The MT client device has a wireguard address of 10.23.1.1/24 OKAY
now give the ubunut address=10.23.1.2 ( NOT ZERO and not different from hex)

Alternatively you could have the ubuntu with 10.23.2.1 and the hex with IP address of 10.23.2.2/24 !!! ( what I would do )

+++++++++++++++++++++++++++++++++++
On the ubuntu allowed IPs would have to be 0.0.0.0/0 as you have at least one user that wants internet access out the hex.
otherwise you would have 10.23.2.2/32 for the hex if you chose the alternative above and any subnets on the hex that were going to head to the ubuntu OR any hex subnets users on the ubuntu were going to visit. However as stated since you have one user on the ubuntu going out internet at the hex, the entry of 0.0.0.0/0 covers all addresses.
+++++++++++++++++++++++++++++++++++

What you need to do on the ubuntu is make sure that user is directed out the tunnel for internet, in terms being allowed by any firewall rules and routed/forced vice going out the local ubunti wani connection - cannot help you with that as its not an MT issue.

++++++++++++++++++++++++++++++++++++++++

On the mT device,
lets say the IP address is 10.23.2.1/24

Your allowed IPs should consist of 10.23.2.0/24, + any subnets coming from ubuntu, + any ubuntu subnets that local hex users need to visist.
IN your case it it seems it will just be 10.23.2.0/24
as that will permit pinging of ubuntu from hex but also allow any client attaching via wg on ubuntu be it 10.23.2.3 or 10.23.2.25 for example.

You may wish to consider on the ubunut to put the subnet or IP address of the admin on the hex device ( in allowed IPs ) as you may wish to be able to config the ubuntu via wireguard from the hex, (besides the current plan of remote client connection direct to the ubuntu. The good news is already included in 0.0.0.0/0

++++++++++++++++++++++++
FW rules.

(1) I see that anyone coming in via WG is allowed access to the router itself, you may wish to consider limiting by IP address in some instances but probably not here not required.

(2) I dont understand why you have this rule as client........... There is no handshake proposed in this direction. IF it is indeed different and you do need it then keep it.
add action=accept chain=input comment="Accept Wireguard" dst-port=<mywgport> \
in-interface=ether1-Internet protocol=udp


(3) After more review your firewalls are a bloated mess based on fear and watching way too many useless youtube videos, and could be simplified dramatically but that is another issue.

(4) Couldnt disagree more with your logic on this statement.
add action=drop chain=input comment=\
"Final Rule - shouldn't be needed but JIC CYA..."

In fact, I would get rid of most of the other rules because you have this excellent rule as your last rule in the input chain - KISS.

(5) I dont understand this rule for starters and second its misguided if you think its somehow okay or viable to do what I think you are attempting to do...... Also on the surface of it so hypocritical to be almost conspiracy level on the input chain and then attempt to sabotage yourself on the forward chain LOL.
add action=accept chain=forward comment="Forward ProtectedExternal" \
src-address-list=ProtectedExternal



(6) I have no clue what you are doing with wireguard in the NAT chain.
All you need basically is one default rule for sourcenat in general and in some cases a wireguard associated rule
add action=masquerade chain=srcnat out-interface-list=WAN

If you had hex users going out ubuntu, there is no need to sourcnat them because you simply put the subnet on the allowed IPs on the ubuntu and there are no real issues.
I suppose it might save you setting a route for the subnet on the ubuntu so return traffic would go back into the tunnel. Mostly sourcenat is used when sending users out to a third party vpn server.

However, you dont have traffic in that direction, its reverse..................
Therefore there is no need for any sourcenat wireguard entries........

(7) NOT REQUIRED>....
add comment="Cloud Wireguard network" disabled=no distance=1 \
dst-address=10.23.2.0/24 gateway=10.23.1.10 pref-src="" \
routing-table=main scope=30 suppress-hw-offload=no target-scope=10


The IP address of 10.23.2.1/24 gateway=wg0 on the hex means that the router automatically creates a route
<dac> dst-address=10.23.2.0/24 gateway=wg0 table= main


(8) what is the purpose of this rule.........???? ONce again you have not stated any requirements for local hex users so this is useless.
/routing rule
add action=lookup-only-in-table disabled=yes dst-address=<the.special.external.address> \
interface=wg0 table=wireguard
 
dalami
Member Candidate
Member Candidate
Topic Author
Posts: 134
Joined: Mon Dec 12, 2011 9:18 am

Re: Route internet through Wireguard

Fri Nov 25, 2022 7:13 am

From your collection of srcnat rules, all you should need is just one, and its #7. It covers access from anywhere to internet, so including from remote 10.23.2.x peer.

So it should work. Perhaps something you deleted wasn't as irrelevant as you thought?
First of all - thank you for your response. I agree with you that it *should* work - the fact that it doesn't is quite frustrating. I'm assuming it's not a bug and is a misconfiguration on my part but I can't see it.

As for the deleted stuff - I doubt it but I will be revising and re-uploading a new export. If after stripping down to bare metal I get this working I can always selectively add stuff back.
 
dalami
Member Candidate
Member Candidate
Topic Author
Posts: 134
Joined: Mon Dec 12, 2011 9:18 am

Re: Route internet through Wireguard

Fri Nov 25, 2022 8:14 am

You need to sort out the IP address usage so it more coherent for your wireguard as I think this is your major issue.

Proposing since you have this in the correct format to keep the hex at
The MT client device has a wireguard address of 10.23.1.1/24 OKAY
now give the ubunut address=10.23.1.2 ( NOT ZERO and not different from hex)

Alternatively you could have the ubuntu with 10.23.2.1 and the hex with IP address of 10.23.2.2/24 !!! ( what I would do )
I didn't give a full narrative as I thought an explicit export was what was desired on the forum. If I'm being invited to share my intent fully then...

My understanding is I cannot have two wireguard servers (actually, not just wireguard) serving the same network. This would be a basic violation of routing by having multiple gateways or routes for the same network. Therefore while my initial wireguard network was 10.23.1.0/24, hosted by my hEX, I've setup a second network of 10.23.2.0/24 hosted by my cloud server. My remotes have both servers defined as peers - though some clients don't support this properly but that's another story. I'm probably going to eventually phase out and eliminate the hEX based network as that location will be closed and make all this academic - but I need to support my particular external route for today's purposes. And I want to actually understand this so I can implement things correctly in the future.

I also have legacy IPSec and SSTP connections left over - they'll be ported to wireguard as soon as convenient. Which I've been saying for a while...never quite get around to it. At least I'm done with OpenVPN.

On the linux server I have two wireguard interfaces defined. wg0 is the 10.23.2.0/24 network that serves most of my remote peers - the hEX does not directly appear here. wg1 has the address 10.23.1.10 to provide a connection to the hEX. So if and when the location and network associated with the hEX goes away I don't have much housekeeping to cleanup on the linux server. So explicit IP's:

linux - 10.23.2.1 on wg0 and 10.23.1.10 on wg1
hEX - 10.23.1.1 on wg0

I do need to clearly state - my wireguard point-to-point and accessing networks behind the peers or server has been working fine. The only thing I'm having an issue with is accessing an external host from a wireguard remote through the wireguard server (hEX). So most of my mess is functional - though I'm not disputing it's a mess.
On the ubuntu allowed IPs would have to be 0.0.0.0/0 as you have at least one user that wants internet access out the hex.
Yes, the desired external IP is listed in the allowedips for the cloud server's peer connection to the hEX. Along with all my other various networks - some will continue, some will be eliminated in the future.
[Peer]
PublicKey = secret
AllowedIPs = 10.23.1.0/24,192.168.0.0/24,10.59.97.0/24,10.41.0.0/24,10.21.3.0/24,10.22.3.0/24,192.168.88.0/24,68.156.140.190
Endpoint = <hEX.external.IP>:<port>
What you need to do on the ubuntu is make sure that user is directed out the tunnel for internet, in terms being allowed by any firewall rules and routed/forced vice going out the local ubunti wani connection - cannot help you with that as its not an MT issue.
As far as I can tell I've got that correct. If wasn't clear in my OP I can see the traffic going from my linux server to the hEX - but I'm not seeing proper NAT in the ICMP communication when the requests come from wireguard. The ICMP definitely hits the router and tries to go out - it just makes requests to the internet from 10.23.1.10 - which obviously fails.
(1) I see that anyone coming in via WG is allowed access to the router itself, you may wish to consider limiting by IP address in some instances but probably not here not required.

(2) I dont understand why you have this rule as client........... There is no handshake proposed in this direction. IF it is indeed different and you do need it then keep it.
add action=accept chain=input comment="Accept Wireguard" dst-port=<mywgport> \
in-interface=ether1-Internet protocol=udp
Absolutely required because as stated above this is also a wireguard server itself.
(4) Couldnt disagree more with your logic on this statement.
add action=drop chain=input comment=\
"Final Rule - shouldn't be needed but JIC CYA..."

In fact, I would get rid of most of the other rules because you have this excellent rule as your last rule in the input chain - KISS.
I very well may kill the other rules. Reason for my comment - is my other rules do kill 99% of inbound garbage. And because I aggressively block attacking IPs I usually have 0 packets reaching this final rule.
(5) I dont understand this rule for starters and second its misguided if you think its somehow okay or viable to do what I think you are attempting to do...... Also on the surface of it so hypocritical to be almost conspiracy level on the input chain and then attempt to sabotage yourself on the forward chain LOL.
add action=accept chain=forward comment="Forward ProtectedExternal" \
src-address-list=ProtectedExternal
I can, and probably am, doing things wrong. I'm trying to be obsessively paranoid in blocking external attacks - but there are specific remotes I need to allow in (primarily my VoIP provider for the office phones). So they have no business talking directly to the router in the input - but do need to pass consistently through the forward.
(6) I have no clue what you are doing with wireguard in the NAT chain.
All you need basically is one default rule for sourcenat in general and in some cases a wireguard associated rule
add action=masquerade chain=srcnat out-interface-list=WAN
Desperately flinging things at the wall to see what sticks. And trying to understand how interface vs IP rules somehow match differently - even though it seems they shouldn't when certain addresses should only be present on certain interfaces. Also trying to explicitly declare rules where possible - for some reason I have it stuck in my head that masquerade is "bad" for static addresses.
If you had hex users going out ubuntu, there is no need to sourcnat them because you simply put the subnet on the allowed IPs on the ubuntu and there are no real issues.
I suppose it might save you setting a route for the subnet on the ubuntu so return traffic would go back into the tunnel. Mostly sourcenat is used when sending users out to a third party vpn server.
hEX needs to reach ubuntu - does not need to pass through ubuntu for internet.
 
dalami
Member Candidate
Member Candidate
Topic Author
Posts: 134
Joined: Mon Dec 12, 2011 9:18 am

Re: Route internet through Wireguard

Fri Nov 25, 2022 8:21 am

New firewall. First - all my src-nat has condensed to:
add action=src-nat chain=srcnat comment="Should be last srcnat rule. non-IPSec to Internet NAT to public IP" ipsec-policy=\
    out,none out-interface=ether1-Internet to-addresses=<my.external.ip>
add action=src-nat chain=srcnat comment="In theory - either this rule or the above rule aren't needed. Because we have an expli\
    cit \"srcnat-accept\" rule above for the IPSec networks no \"IPSec policy\" matching should be required. Simple srcnat to p\
    ublic IP for internet access." out-interface=ether1-Internet to-addresses=<my.external.ip>
add action=masquerade chain=srcnat comment="Should be no forking reason for this. But."
Watching the packet/byte counters, rules #1 and #3 have traffic. Rule #2 - the one I think isn't needed - has no traffic. I need to test some of my other peers to see if anything has broken but at the moment, at least with rule #3 masquerade catching strays, my general network traffic is online and there's no difference for my wireguard.

On the filter side, before I start deleting, a question: for this application, when traffic is coming is sourced from a remote wireguard peer (the linux server) and trying to reach the internet through the hEX via wireguard - is this a "input" or a "forward"? Or both?
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Route internet through Wireguard

Fri Nov 25, 2022 2:45 pm

First srcnat rule covers traffic to internet. Second is useless, because it would take subset of traffic already handled by first. Third affects everything else passing through router, so connections between local and VPN subnets, forwarded ports if you have any, etc. It shouldn't be needed. If you get rid of it and something breaks, it means that you have some missing routes or firewalls (not necessarily on this router) that don't work with real addresses.

As for filter, don't delete anything, rules are processed in order, so for testing just move the one allowing everything from WG up. If it's first, nothing further will touch those packets. Input = to router itself, forward = to somewhere else, VPN->internet is the latter.

Logging is your friend, do e.g.:
/ip firewall mangle
add chain=prerouting dst-address=<your special address> action=log
add chain=forward dst-address=<your special address> action=log
add chain=postrouting dst-address=<your special address> action=log
and you'll see how far it gets. If there's more traffic to it, you can add other conditions like single port. It can be even some that's not in use (protocol=tcp dst-port=12345), so you won't be able to connect to anything, but you will still see outgoing packets.

You can also mark selected connections, which will handle even changes that may be done by NAT, e.g.:
/ip firewall mangle
add chain=prerouting connection-mark=debug action=log log-prefix=debug
add chain=prerouting connection-state=new src-address=10.23.2.x dst-address=x.x.x.x protocol=tcp dst-port=80 action=mark-connection new-connection-mark=debug log=yes log-prefix=debug-NEW passthrough=yes
add chain=forward connection-mark=debug action=log log-prefix=debug
add chain=postrouting connection-mark=debug action=log log-prefix=debug
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Route internet through Wireguard

Fri Nov 25, 2022 4:23 pm

Do not understand this comment.
I do need to clearly state - my wireguard point-to-point and accessing networks behind the peers or server has been working fine. The only thing I'm having an issue with is accessing an external host from a wireguard remote through the wireguard server (hEX). So most of my mess is functional - though I'm not disputing it's a mess

Its problematic for me because its a jumbled statement that begs for more clarity.
Also, you said the ubuntu (cloud server) is now your server for WG connection?????????


Put the phrase in plain english or a diagram may be better. or give an example. that makes it clear.
It means nothing to me at the moment. Dont know who is originating traffic, where the traffic goes next etc...... what site is has to reach etc......
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Route internet through Wireguard

Fri Nov 25, 2022 5:07 pm

Two WG servers (hEX, Cloud), client connects to Cloud, wants to access x.x.x.x, but it needs to be routed via hEX.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Route internet through Wireguard

Fri Nov 25, 2022 9:13 pm

Well, yes and no maybe so, no clearer as he specifically states that he has moved the wg server aspect from hex to cloud, you are insinuating no he has not.
So will wait for the op to come clean. :-)

As a matter of fact I am done here as the op is in between and not firm on a final design.
Once he knows what is required a set of requirements are established i'm tired of chasing information.
 
dalami
Member Candidate
Member Candidate
Topic Author
Posts: 134
Joined: Mon Dec 12, 2011 9:18 am

Re: Route internet through Wireguard

Fri Nov 25, 2022 11:44 pm

First srcnat rule covers traffic to internet. Second is useless, because it would take subset of traffic already handled by first. Third affects everything else passing through router, so connections between local and VPN subnets, forwarded ports if you have any, etc. It shouldn't be needed. If you get rid of it and something breaks, it means that you have some missing routes or firewalls (not necessarily on this router) that don't work with real addresses.

As for filter, don't delete anything, rules are processed in order, so for testing just move the one allowing everything from WG up. If it's first, nothing further will touch those packets. Input = to router itself, forward = to somewhere else, VPN->internet is the latter.
Thank you again.
Logging is your friend, do e.g.:
So...I turned on the log. And I'm more confused. My results - first for ping from the bridged lan inside the hEX network:
Nov 25 13:20:50 Foxy-hEX prerouting: in:bridge out:(unknown 0), connection-state:new src-mac 00:25:90:4c:61:48, proto ICMP (type 8, code 0), 192.168.0.2-><sp.ec.ial.ip>, len 84
Nov 25 13:20:50 Foxy-hEX forward: in:bridge out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new src-mac 00:25:90:4c:61:48, proto ICMP (type 8, code 0), 192.168.0.2-><sp.ec.ial.ip>, len 84
Nov 25 13:20:50 Foxy-hEX postrouting: in:bridge out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new src-mac 00:25:90:4c:61:48, proto ICMP (type 8, code 0), 192.168.0.2-><sp.ec.ial.ip>, len 84
Nov 25 13:20:51 Foxy-hEX prerouting: in:bridge out:(unknown 0), connection-mark:ICMP connection-state:new,snat src-mac 00:25:90:4c:61:48, proto ICMP (type 8, code 0), 192.168.0.2-><sp.ec.ial.ip>, NAT (192.168.0.2-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
Nov 25 13:20:51 Foxy-hEX forward: in:bridge out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new,snat src-mac 00:25:90:4c:61:48, proto ICMP (type 8, code 0), 192.168.0.2-><sp.ec.ial.ip>, NAT (192.168.0.2-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
Nov 25 13:20:51 Foxy-hEX postrouting: in:bridge out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new,snat src-mac 00:25:90:4c:61:48, proto ICMP (type 8, code 0), 192.168.0.2-><sp.ec.ial.ip>, NAT (192.168.0.2-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
Nov 25 13:20:52 Foxy-hEX prerouting: in:bridge out:(unknown 0), connection-mark:ICMP connection-state:new,snat src-mac 00:25:90:4c:61:48, proto ICMP (type 8, code 0), 192.168.0.2-><sp.ec.ial.ip>, NAT (192.168.0.2-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
Nov 25 13:20:52 Foxy-hEX forward: in:bridge out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new,snat src-mac 00:25:90:4c:61:48, proto ICMP (type 8, code 0), 192.168.0.2-><sp.ec.ial.ip>, NAT (192.168.0.2-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
Nov 25 13:20:52 Foxy-hEX postrouting: in:bridge out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new,snat src-mac 00:25:90:4c:61:48, proto ICMP (type 8, code 0), 192.168.0.2-><sp.ec.ial.ip>, NAT (192.168.0.2-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
And then via the cloud wireguard connection:
Nov 25 13:22:57 Foxy-hEX prerouting: in:wg0 out:(unknown 0), connection-state:new proto ICMP (type 8, code 0), 10.23.1.10-><sp.ec.ial.ip>, len 84
Nov 25 13:22:57 Foxy-hEX forward: in:wg0 out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new proto ICMP (type 8, code 0), 10.23.1.10-><sp.ec.ial.ip>, len 84
Nov 25 13:22:57 Foxy-hEX postrouting: in:wg0 out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new proto ICMP (type 8, code 0), 10.23.1.10-><sp.ec.ial.ip>, len 84
Nov 25 13:22:58 Foxy-hEX prerouting: in:wg0 out:(unknown 0), connection-mark:ICMP connection-state:new,snat proto ICMP (type 8, code 0), 10.23.1.10-><sp.ec.ial.ip>, NAT (10.23.1.10-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
Nov 25 13:22:58 Foxy-hEX forward: in:wg0 out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new,snat proto ICMP (type 8, code 0), 10.23.1.10-><sp.ec.ial.ip>, NAT (10.23.1.10-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
Nov 25 13:22:58 Foxy-hEX postrouting: in:wg0 out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new,snat proto ICMP (type 8, code 0), 10.23.1.10-><sp.ec.ial.ip>, NAT (10.23.1.10-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
Nov 25 13:22:59 Foxy-hEX prerouting: in:wg0 out:(unknown 0), connection-mark:ICMP connection-state:new,snat proto ICMP (type 8, code 0), 10.23.1.10-><sp.ec.ial.ip>, NAT (10.23.1.10-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
Nov 25 13:22:59 Foxy-hEX forward: in:wg0 out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new,snat proto ICMP (type 8, code 0), 10.23.1.10-><sp.ec.ial.ip>, NAT (10.23.1.10-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
Nov 25 13:22:59 Foxy-hEX postrouting: in:wg0 out:ether1-Internet, packet-mark:ICMP connection-mark:ICMP connection-state:new,snat proto ICMP (type 8, code 0), 10.23.1.10-><sp.ec.ial.ip>, NAT (10.23.1.10-><my.pub.ip>)-><sp.ec.ial.ip>, len 84
The only things I see different, other than the obvious source IP address, is that the wireguard doesn't have a src-mac shown. Which I assume is because wireguard doesn't have a MAC to show. I explicitly see (I think) NAT transformations occurring. Which means...what? That for some reason (the core issue) the replies are not being forwarded to my wireguard peer?

I tried inserting a simple unconditional forward as my first filter rule = "add chain=forward action=accept". I even added them for input and output. No difference.
 
dalami
Member Candidate
Member Candidate
Topic Author
Posts: 134
Joined: Mon Dec 12, 2011 9:18 am

Re: Route internet through Wireguard

Fri Nov 25, 2022 11:55 pm

Maybe I found something...or not.

The target IP I'm trying to reach has some kind of firewall protections itself (it's a 3rd-party service out of my control) and only lets specific IP's in. Which is why I need to route my traffic through my current hEX IP. It also doesn't respond to ping - but when things are working traceroute (using mtr) works fine.

Just for fun, I tried adding Cloudflare (1.1.1.1) to the cloud->hEX wireguard allowedIPS. I can ping fine. So...I *think* that means something else is going on. The only other hint I have is when I tried mtr to 1.1.1.1 - from my 192.168.0.0/24 network I see the full path. From the wireguard cloud peer I only see the start and end - the links in between don't show responses.

Does that provide any useful insight? I can try logging some of that traffic but it gets big.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Route internet through Wireguard  [SOLVED]

Sat Nov 26, 2022 1:56 am

What's the thing you actually need to access on <sp.ec.ial.ip>? You're probably not doing this just to ping it. So test that real thing. If it's e.g. web server, then watch for TCP ports 80/443 (unless it uses some other port). Either using the second set of logging rules with connection mark, or add ports to first ones:
/ip firewall mangle
add chain=prerouting dst-address=<sp.ec.ial.ip> protocol=tcp dst-port=80,443 action=log log-prefix=request
add chain=forward dst-address=<sp.ec.ial.ip> protocol=tcp dst-port=80,443 action=log log-prefix=request
add chain=postrouting dst-address=<sp.ec.ial.ip> protocol=tcp dst-port=80,443 action=log log-prefix=request
add chain=prerouting src-address=<sp.ec.ial.ip> protocol=tcp src-port=80,443 action=log log-prefix=response
add chain=forward src-address=<sp.ec.ial.ip> protocol=tcp src-port=80,443 action=log log-prefix=response
add chain=postrouting src-address=<sp.ec.ial.ip> protocol=tcp src-port=80,443 action=log log-prefix=response
And you'll see both outgoing and incoming trafic (or lack of it).

Seemingly non-responding hops in mtr are because they have "random" source addresses, but your WG tunnel doesn't allow them. On Linux server, you need allowed address 0.0.0.0/0 (= anything) for hEX as peer.
 
dalami
Member Candidate
Member Candidate
Topic Author
Posts: 134
Joined: Mon Dec 12, 2011 9:18 am

Re: Route internet through Wireguard

Sat Nov 26, 2022 5:19 am

I would say this is embarrassing - except I'm used to displaying my stupidity. Especially on this forum.

The particular form my stupidity has taken now was revealed (at least partially) by your logging instructions. Since I saw absolutely no traffic for my actual use (VNC, which means TCP to port 5900) I was more confused than ever. Until I decided to try initiating a VNC connection directly from my linux server - and it worked!

So...first I have to thank you again for the assistance and instructions so far. I've now established that I'm not completely, just mostly, incompetent when it comes to setting up my MT rules as they're obviously working - they're even working as expected. The gateway can't route traffic that never reaches it.

Now I have to figure out why my Windows client is not reaching the hEX correctly for this particular traffic. Let's see if I can add some logging on my cloud server via nftables...yes I can. And try again...and absolutely no traffic from my VNC viewer. Yet ping does generate packets. Hmm...

Windows routing tables look right. Wireguard allowedIPS looks right. Windows firewall isn't blocking outbound. What's left?

Out of desperation, and sheer morbid curiosity, let's try a different VNC viewer...and it $#@!ing works! It works!

So now I have to figure out why TurboVNC either isn't respecting my routing table. Which is somewhat off-topic here.

Thank you again for this learning opportunity.

Who is online

Users browsing this forum: rplant and 37 guests