Internet via WireGuard

Fructose3075 · Sat Sep 03, 2022 4:48 pm

Hi

I'm trying to set up Mikrotik router as a WireGuard peer and then route requests to specific addresses via the WireGuard peer on a VPS in a datacenter. It seems to work, but very slow. Strange enough, if I'm running some sort of speedtest, i.e. fast.com, it gets up to 50-70Mbps, but general web browsing its extremely slow.
I've tried playing around with MTU with little success. Added change-mss in the mangle chain like some suggested on the internet, but it did not help either. I've connected with my phone to the same wg peer as the router to test the experience and everything is as fast as without wg.

What I'm trying to achieve:
General internet access
LAN(192.168.11.0)->ether1->target website
Access to VPNLISTed sites:
LAN(192.168.11.0)->ether1->wg0->VPS wg peer->target website
Access from VPNed addresses in LAN:
LAN(192.168.11.0)->ether1->wg0->target website

Clients reaching home network via wghome should be able to access 192.168.11.0/24 and access internet like they are part of the LAN.

I'm new to the forum, but it seems that the configuration is the first thing that gets asked, so here is the /export compact from the router

# sep/03/2022 16:03:00 by RouterOS 7.4.1

/interface bridge
add admin-mac=2C:C8:1B:58:D0:27 auto-mac=no comment=defconf name=bridge
add name=bridge-iot
/interface ethernet
set [ find default-name=ether1 ] comment="Internet access"
set [ find default-name=ether2 ] comment=LAN
/interface wireguard
add comment="gate.homedomain.com outgoing WireGuard interface" listen-port=29411 mtu=1420 name=wg0
add comment="incoming WireGuard interface" listen-port=55800 mtu=1420 name=wghome
/caps-man security
add authentication-types=wpa2-psk encryption=aes-ccm group-encryption=aes-ccm name=tikapp-sec
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
add authentication-types=wpa2-psk comment=defconf disable-pmkid=yes mode=dynamic-keys name=GATE supplicant-identity=MikroTik
add authentication-types=wpa2-psk mode=dynamic-keys name=IoT supplicant-identity=MikroTik
add authentication-types=wpa2-psk disable-pmkid=yes mode=dynamic-keys name=loopwpa2 supplicant-identity=MikroTik
/interface wireless
set [ find default-name=wlan1 ] disabled=no frequency=auto installation=outdoor mode=ap-bridge security-profile=loopwpa2 ssid=loop24
set [ find default-name=wlan2 ] band=5ghz-a/n/ac channel-width=20/40mhz-Ce disabled=no frequency=auto installation=indoor mode=ap-bridge security-profile=loopwpa2 ssid=loop
set [ find default-name=wlan3 ] band=5ghz-a/n/ac channel-width=20/40mhz-XX disabled=no hide-ssid=yes installation=indoor mode=bridge security-profile=GATE ssid=SYNC-58D02A \
    wireless-protocol=nv2
add default-forwarding=no disabled=no mac-address=2E:C8:1B:58:D0:28 master-interface=wlan1 name=wlan4 security-profile=IoT ssid=IoT24 wds-default-bridge=bridge-iot wps-mode=\
    disabled
/interface wireless nstreme
set wlan3 enable-nstreme=yes enable-polling=no
/ip pool
add comment="Default local clients pool" name=dhcp ranges=192.168.11.100-192.168.11.200
add comment="iot virtual wireless interface DHCP" name=iot-pool ranges=192.168.111.100-192.168.111.200
/ip dhcp-server
add address-pool=dhcp interface=bridge lease-time=1h name=defconf
add address-pool=iot-pool interface=bridge-iot lease-time=1h name=iot-dhcp
/queue type
add kind=pfifo name=pfifo250 pfifo-limit=250
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
add disabled=no name=default-v2
/routing ospf area
add disabled=yes instance=default-v2 name=backbone-v2
/routing table
add disabled=no fib name=wg-route
/caps-man manager interface
set [ find default=yes ] forbid=yes
add comment=defconf disabled=no interface=bridge
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether2
add bridge=bridge comment=defconf ingress-filtering=no interface=wlan1
add bridge=bridge comment=defconf ingress-filtering=no interface=wlan2
add bridge=bridge comment=defconf ingress-filtering=no interface=wlan3
add bridge=bridge-iot ingress-filtering=no interface=wlan4
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add comment="WireGuard is a way to get to the internet" interface=wg0 list=WAN
add comment="WireGuard gateway for external clients to connect home" interface=wghome list=LAN
/interface wireguard peers
add allowed-address=0.0.0.0/0,::/0 comment=gate.homedomain.com endpoint-address=88.99.77.55 endpoint-port=55670 interface=wg0 persistent-keepalive=25s public-key=\
    "publickey1"
add allowed-address=10.0.20.2/32 comment=MacBook interface=wghome public-key="publickey2"
add allowed-address=10.0.20.3/32 comment=iPhone interface=wghome public-key="publickey3"
/interface wireless cap
set caps-man-addresses=127.0.0.1 interfaces=wlan1,wlan2
/ip address
add address=192.168.11.1/24 comment=defconf interface=ether2 network=192.168.11.0
add address=192.168.111.1/24 interface=bridge-iot network=192.168.111.0
add address=10.8.0.5/24 comment="WireGuard peer address provided by VPS" interface=wg0 network=10.8.0.0
add address=10.0.20.1/24 comment="Wireguard interface for home calling" interface=wghome network=10.0.20.0
/ip dhcp-client
add comment=defconf interface=ether1 use-peer-dns=no
/ip dhcp-server lease
add address=192.168.11.167 client-id=1:cc:d2:81:80:9:2 mac-address=CC:D2:81:80:09:02 server=defconf
/ip dhcp-server network
add address=192.168.11.0/24 comment=defconf dns-server=192.168.11.1 domain=homedomain.com gateway=192.168.11.1 netmask=24
add address=192.168.111.0/24 comment="IoT DHCP Server" dns-server=1.1.1.1 gateway=192.168.111.1 netmask=24
/ip dns
set allow-remote-requests=yes max-concurrent-tcp-sessions=50 use-doh-server=https://one.one.one.one/dns-query verify-doh-cert=yes
/ip firewall address-list
add address=twitter.com list=VPNLIST
add address=192.168.11.0/24 list=allowed-ips
add address=10.0.20.0/24 list=allowed-ips
add address=192.168.11.167 comment=AlwaysViaWG list=VPNed
/ip firewall filter
add action=jump chain=forward comment="jump to kid-control rules" jump-target=kid-control
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=input comment="Drop wrong interface incoming" in-interface-list=!LAN src-address-list=allowed-ips
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=accept chain=input comment="Accept ICMP" protocol=icmp
add action=accept chain=input comment="WireGuard incoming interface wghome" dst-port=55800 protocol=udp
add action=accept chain=input comment="Allow router access from allowed-ips list" src-address-list=allowed-ips
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" disabled=yes dst-address=127.0.0.1
add action=drop chain=input comment="Drop all undefined above"
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward dst-address=192.168.11.0/24 src-address=192.168.111.0/24
add action=drop chain=forward connection-nat-state=!dstnat connection-state=new disabled=yes in-interface=wg0
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=change-mss chain=forward new-mss=1380 out-interface=wg0 protocol=tcp tcp-flags=syn tcp-mss=1381-65535
add action=mark-routing chain=prerouting connection-nat-state=!dstnat dst-address=!192.168.11.0/24 new-routing-mark=wg-route passthrough=no src-address-list=VPNed
add action=mark-routing chain=prerouting comment="Mark packets to be routed via WireGuard" connection-nat-state=!dstnat dst-address-list=VPNLIST new-routing-mark=wg-route \
    passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=dst-nat chain=dstnat comment="Vaultwarden internet access" disabled=yes dst-port=31281 in-interface=all-ethernet protocol=tcp to-addresses=192.168.11.11 to-ports=\
    31281
add action=dst-nat chain=dstnat comment="Re-route all DNS requests back to Mikrotik. Except for one device" dst-address=!192.168.11.1 dst-port=53 log-prefix=dns-route \
    protocol=udp src-address=!192.168.11.167 to-addresses=192.168.11.1 to-ports=53
/ip route
add disabled=no distance=1 dst-address="" gateway=*C routing-table=*400 scope=30 suppress-hw-offload=no target-scope=10
add comment="Route all marked via WireGuard interface" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=wg0 pref-src="" routing-table=wg-route suppress-hw-offload=no
/routing rule
add action=lookup-only-in-table routing-mark=wg-route table=wg-route

anav · Sat Sep 03, 2022 7:44 pm

Just to be clear, you are initiating the requests on the MIkrotik Router and then want to connect via wireguard to VPS server instance on the cloud.
Is the VPS an MT CHR or some other OS device?
In any case, need at least the wireguard and firewall rules and routing rules of the VPS if not MT, and the full config if its MT.
A network diagram never hurts.......

The requirement seems straightforward.
a. Lan users on mikrotik go out local router and local ISP for WAN connections.
b. Lan users on mikrotik for specific WANIPs? need to traverse the wireguard tunnel to go out the internet at the VPS.
c. LAn users DO NOT require access to any subnets that may be associated at the VPS device.
d. The admin may wish to access the mikrotik router via wireguard for config purposes after connecting to the VPS device ??
e. The admin may wish to access the LAN on the mikrotik router via wireguard after connecting to the VPS device ??

Looking further your explanation gets a bit weird, understand
LAN to WAN via ISP (LAN(192.168.11.0)->ether1->target website)
LAN to WAN via Wgo to vpn sites. (LAN(192.168.11.0)->ether1->wg0->VPS wg peer->target website)
Number 3 makes no sense, as there is no traffic ORIGINATING on a VPN site heading your way)........ At least from what I can gather.
(Access from VPNed addresses in LAN:
LAN(192.168.11.0)->ether1->wg0->target website)

In any case either your verbiage and path or wrong or both.... it would be vpnsite---> wg0---> LAN ?????

anav · Sat Sep 03, 2022 7:53 pm

(1) Why two bridges,,,,,,,,,
(2) OKAY now I have to ask WHY to Wireguard interfaces ???

You would be better served by one bridge and two vlans or more depending
bridge1
vlan20 = LAN
vlan40 = IOT
vlanxx= wifi users home
vlanyy =wifi users gues

etc.................

(3) Okay perhaps a little clearer here.......
add comment="WireGuard is a way to get to the internet" interface=wg0 list=WAN
add comment="WireGuard gateway for external clients to connect home" interface=wghome list=LAN

External users, coming from where????
Do you mean road warriors that need to access the home router LAN??
You only need one wireguard interface to accomplish the above.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Okay you want to keep it clean.
WIREGUARD to home directly for admin or road warriors
WIREGUARD TO VPS only for LAN users and only for specific sites........

The problem is you are allowing LAN users to go out ALL INTERNET through wg0 and not just specific sites.
SO which is it>> ALL internet or only some destinations???

anav · Sat Sep 03, 2022 8:24 pm

HOW MANY internet vpn sites are you talking about here as it directly impacts the design of the config...... for the LAN users to access thru wg0??

Okay, I think I understand better your setup and requirements and trying to keep things the same, the following suggestions...

(1) FROM: /interface wireguard peers
add allowed-address=0.0.0.0/0,::/0 comment=gate.homedomain.com endpoint-address=88.99.77.55 endpoint-port=55670 interface=wg0 persistent-keepalive=25s public-key=\
"publickey1"

(2) TO:
/interface wireguard peers
add allowed-address=wgIPofVPS,publicIP#1,publicIP#2, PublicIP#3...............................PublicIP#ZZ comment=gate.homedomain.com endpoint-address=88.99.77.55 endpoint-port=55670 interface=wg0 persistent-keepalive=25s public-key= "publickey1"

(3) Assuming the entire LAN needs access to those addresses, and they are clearly delineated on the wireguard peers, then you dont need to worry about the route interfering with the already existing Route through the Local MT router ISP connection, which is to 0.0.0.0/0

However you will need a router to tell the router for those addresses send the traffic through the wgo gateway
/ip route
add dst-address-list=VPNSITES gwy=wg0 table=main

however there is no such thing as a dst-address-list in MT config, so you can either create a route per VPN SITE or mangle traffic heading towards those addresses.

First thing is to create a table entry which will be used to select a new routing-mark entry in mangle rules.
/routing table add name=thruWG fib

Then we identify the traffic.
/ip firewall filter mangle
add chain=prerouting connection-mark=no-mark action=mark-connection new-connection mark=VPNs src-address=192.168.11.0/24 dst-address-list=VPNSITES passthrough=yes

where
/ip firewall address list
add IP=publicIP#1 list=VPNSITES
add IP=publicIP#2 list=VPNSITES
....................................................
add IP=publicIP#ZZ list=VPNSITES

So we mark any traffic coming from the LAN heading to those public IPs.
then we mark the routing path.

add chain=prerouting connection-mark=VPNs action=mark-routing new-routing-mark=thruWG passthrough=yes

Next we add a route rule so that the router knows how to handle such traffic.........
/routing rule add routing-mark=thruWG action=lookup-only-in-table table=wg0

Finally I believe we need a general route so that the router can actually apply the routing rule.
add dst-address=0.0.0.0 gwy=wg0 table=main distance=5

Thus all traffic will normally use the default route out the ISP and never look at the wg table.
However marked traffic will now be able to enter the wg0 tunnel.

Well I see this was a futile exercise, it appears you are trying to send out all requests for twitter out the wg0 tunnel.
They have hundreds of different addresses and not sure if there is a way to successfully do what you want............

In other words, the only way to do this may be to send all internet requests through wg0..............................

Fructose3075 · Sat Sep 03, 2022 10:44 pm

HOW MANY internet vpn sites are you talking about here as it directly impacts the design of the config...... for the LAN users to access thru wg0??

Hey @anav, thanks for looking into it. The list is quite limited (14) and it is not twitter per se. I'm managing the list by adding required IPs to the VPNLIST address list at the moment. I think it is easier to do it this way than trying to add those addresses to the WG rule.
I'm attaching a sketch of the traffic flow I am trying to achieve and a diagram. Hopefully it makes sense.
Side note - WireGuard connections work fine. I tried connecting to the VPS directly from my laptop using a WireGuard client and it works fine with acceptable 30-40 Mbps speed. The connection to the servers in VPNLIST from the LAN do work and traceroute shows that they go through wg0, but the web experience is bad. Not snappy at all. Speedtests though show about the same mbps as a direct connection.

The requirement seems straightforward.
a. Lan users on mikrotik go out local router and local ISP for WAN connections.
b. Lan users on mikrotik for specific WANIPs? need to traverse the wireguard tunnel to go out the internet at the VPS.
c. LAn users DO NOT require access to any subnets that may be associated at the VPS device.
d. The admin may wish to access the mikrotik router via wireguard for config purposes after connecting to the VPS device ??
e. The admin may wish to access the LAN on the mikrotik router via wireguard after connecting to the VPS device ??

a-c are correct
d. The admin (road warrior) may wish to access the mikrotik router via wireguard for config purposes
e. The admin (road warrior) may wish to access the LAN on the mikrotik router via wireguard

VPS is used
(1) by all LAN devices for access to a limited number of sites (list=VPNLIST)
(2) by two specific devices (list=VPNed) on the LAN that must access internet _only_ via VPS

I have a routing table for that
/routing table add disabled=no fib name=wg-route

with only one route set up in it
/ip route add comment="Route all marked via WireGuard interface" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=wg0 pref-src="" routing-table=wg-route suppress-hw-offload=no

a routing rule to look up in this table only for marked traffic
/routing rule add action=lookup-only-in-table routing-mark=wg-route table=wg-route

and two rules in prerouting chain to mark traffic
/ip firewall mangle
add action=mark-routing chain=prerouting connection-nat-state=!dstnat dst-address=!192.168.11.0/24 new-routing-mark=wg-route passthrough=no src-address-list=VPNed
add action=mark-routing chain=prerouting comment="Mark packets to be routed via WireGuard" connection-nat-state=!dstnat dst-address-list=VPNLIST new-routing-mark=wg-route \
passthrough=no

It seems very much like your suggested config. But HTTP(s) browsing is extremely slow

VPS is an Ubuntu server with firewall accepting the UDP port set for the WireGuard and the following wg0.conf

# Server
[Interface]
PrivateKey = privkey
Address = 10.8.0.1/24
ListenPort = #####
PreUp = 
PostUp =  iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE; iptables -A INPUT -p udp -m udp --dport ##### -j ACCEPT; iptables -A FORWARD -i wg0 -j ACCEPT; iptables -A FORWARD -o wg0 -j ACCEPT; 
PreDown = 
PostDown = 

# Client: mikrotik
[Peer]
PublicKey = pubkey1
PresharedKey = key1
AllowedIPs = 10.8.0.5/32

Fructose3075 · Sat Sep 03, 2022 11:15 pm

I've found it!

/ip firewall add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes

^^^This is the reason. Once disabled, everything works as expected. I now have to figure out how to make this rule more specific, so that the whole traffic marking ideas will keep working while the most of general purpose traffic will be fasttracked to take the load off of CPU...

anav · Sun Sep 04, 2022 1:20 am

On the VPS allowed IPs you only have the IP address of the mikrotk wireguard.
it should look like.
AllowedIPs = 10.8.0.5/32,192.168.11.0/24

anav · Sun Sep 04, 2022 1:25 am

It would be nice to remove the need for mangling.................
If you want to then simply make 10 or whatever routes

add dst-address=publicIP#1 gwy=wg0 table= main
add dst-address=publicP#2 gwy=wg0 table=main
.................................................................................
add dst-address=publicIP#14 gwy=wg0 table=main.

DONE PHUCK mangle.

Fructose3075 · Sun Sep 04, 2022 10:19 am

On the VPS allowed IPs you only have the IP address of the mikrotk wireguard.
it should look like.
AllowedIPs = 10.8.0.5/32,192.168.11.0/24

My understanding was that adding 192.168.11.0/24 to the allowed list will open up access to lan for other peers of the VPS. Am I right? If so, I do not expect it to work, there is no scenario, as road warriors connect to the router directly.

Fructose3075 · Sun Sep 04, 2022 10:26 am

It would be nice to remove the need for mangling.................
If you want to then simply make 10 or whatever routes

add dst-address=publicIP#1 gwy=wg0 table= main
add dst-address=publicP#2 gwy=wg0 table=main
.................................................................................
add dst-address=publicIP#14 gwy=wg0 table=main.

DONE PHUCK mangle.

This is interesting. So having a proper routing table from both sides will allow me get the fasttrack back and will allow to get rid of mangling. I will definitely try this! Thanks for your help!

anav · Sun Sep 04, 2022 2:20 pm

On the VPS allowed IPs you only have the IP address of the mikrotk wireguard.
it should look like.
AllowedIPs = 10.8.0.5/32,192.168.11.0/24
My understanding was that adding 192.168.11.0/24 to the allowed list will open up access to lan for other peers of the VPS. Am I right? If so, I do not expect it to work, there is no scenario, as road warriors connect to the router directly.

Your understanding is flawed, read the link here and then if you still dont understand we can discuss!
viewtopic.php?t=182340
Focus on: (1) GENERIC SETTINGS FOR WG DEVICES sub paragraph ---> Important Discussion of Allowed IPs

Glad you got it working.

Note: allowed IP determines traffic exiting or entering the tunnel ( routes provide a path for traffic ) and FIREWALL RULES are used to deny or allow traffic to reach interfaces.

Internet via WireGuard [SOLVED]

Internet via WireGuard

Re: Internet via WireGuard

Re: Internet via WireGuard

Re: Internet via WireGuard

Re: Internet via WireGuard

Re: Internet via WireGuard

Re: Internet via WireGuard

Re: Internet via WireGuard [SOLVED]

Re: Internet via WireGuard

Re: Internet via WireGuard

Re: Internet via WireGuard

Who is online