Wireguard routing and access

benny532017 · September 1, 2023, 1:06pm

I would be very glad of some advice if at all possible.

My main office has a CCR2004 as it’s main router and I am trying to make a configuration where I can do site-site VPNs with wireguard and reach the various networks from any of the networks.

This has basically been working fine with ‘road warriors’ which are multiple peers on a single wireguard interface, however I now would like to scale this further with a second wireguard interface because these different wireguard networks are for different things, different customers etc. so should make things easier to track and manage.

The problem I’m having is that I can’t reach the device on wireguard2 from devices that are remote to the office network but connected via wireguard1, but I can when I’m at the office on 192.168.93.0/24.

I’ve looked at the configs for a long time to try and make sure I’ve not missed anything or done anything stupid, I’ve also tried with all firewall rules disabled to try and rule that out but I am now banging my head on a wall - I suspect there’s something fundamental that I’m not understanding.

The device peering to wireguard2 is a Teltonika RUT241 with the following config:

root@Teltonika-RUT241:~# wg
interface: CTQ_HQ
  public key: kWQdRPOM9vuD0nCUeBhiPNPteXfG+vaXOhizO32gJRk=
  private key: (hidden)
  listening port: 51820

peer: teAy83xdR/QZMQInmoNXHhCtwfT+QLLLZpJzppR3o1g=
  endpoint: 193.237.82.109:13232
  allowed ips: 192.168.35.1/32, 192.168.93.0/24, 192.168.34.0/24
  latest handshake: 14 seconds ago
  transfer: 7.11 KiB received, 10.72 KiB sent
  persistent keepalive: every 25 seconds

Can someone offer me their thoughts? I feel like I’m missing something like a route, allowed IPs or something I can’t think of. Many thanks in advance!

Below is the config for the CCR:

# 2023-09-01 13:51:41 by RouterOS 7.10.2
# software id = RK30-EMVZ
#
# model = CCR2004-1G-12S+2XS
# serial number = .....
/interface bridge
add name=bridge1
add name=bridge2_guest
add name=bridge3_church
/interface ethernet
set [ find default-name=ether1 ] comment="DSL Modem"
set [ find default-name=sfp-sfpplus1 ] auto-negotiation=no speed=1Gbps
set [ find default-name=sfp-sfpplus2 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=sfp-sfpplus3 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=sfp-sfpplus4 ] auto-negotiation=no comment="Guest Network" speed=1Gbps
set [ find default-name=sfp-sfpplus5 ] auto-negotiation=no comment="Church Connection" speed=1Gbps
set [ find default-name=sfp-sfpplus6 ] auto-negotiation=no comment="Church network O&M" speed=1Gbps
set [ find default-name=sfp-sfpplus7 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=sfp-sfpplus8 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=sfp-sfpplus9 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=sfp-sfpplus10 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=sfp-sfpplus11 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=sfp-sfpplus12 ] auto-negotiation=no speed=1Gbps
set [ find default-name=sfp28-1 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=sfp28-2 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=pppoe-out1 use-peer-dns=yes user=##############
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1
add comment=VoltServer listen-port=13232 mtu=1420 name=wireguard2
/interface list
add name=WAN
add name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add comment=Guest name=dhcp_pool2 ranges=10.10.10.10-10.10.10.100
add comment="Main Office LAN" name=dhcp_pool3 ranges=192.168.93.100-192.168.93.250
add comment=Church name=churchPool ranges=10.10.11.10-10.10.11.254
/ip dhcp-server
add address-pool=dhcp_pool3 interface=bridge1 lease-time=1h name=dhcp1
add address-pool=dhcp_pool2 interface=bridge2_guest lease-time=1h name=dhcp2
add address-pool=churchPool interface=bridge3_church lease-time=1h name=dhcp3_Church
/port
set 0 name=serial0
set 1 name=serial1
/queue tree
add comment="Guest Network Limit" max-limit=14M name=Total parent=global
add max-limit=10M name=GuestDownload packet-mark=guest-Download-pkts parent=Total
add comment="Guest Network Limit" max-limit=30M name="Church Network Limit" parent=global
add max-limit=20M name=ChurchDownload packet-mark=church-Download-pkts parent="Church Network Limit"
add max-limit=4M name=GuestUpload packet-mark=guest-Upload-pkts parent=Total
add max-limit=10M name=ChurchUpload packet-mark=church-Upload-pkts parent="Church Network Limit"
/queue type
add kind=sfq name=sfq-default sfq-perturb=10
/queue simple
add max-limit=9500k/35M name=sfq-default queue=sfq-default/sfq-default target=192.168.93.0/24
/interface bridge port
add bridge=bridge1 interface=sfp-sfpplus1
add bridge=bridge1 interface=sfp-sfpplus2
add bridge=bridge1 interface=sfp-sfpplus3
add bridge=bridge2_guest comment="Guest Network" interface=sfp-sfpplus4
add bridge=bridge3_church comment="Church Network" interface=sfp-sfpplus5
add bridge=bridge3_church comment="Church netowrk O&M" interface=sfp-sfpplus6
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface detect-internet
set detect-interface-list=all
/interface list member
add interface=pppoe-out1 list=WAN
add interface=bridge1 list=LAN
add interface=wireguard1 list=LAN
add interface=wireguard2 list=LAN
/interface ovpn-server server
set auth=sha1,md5
/interface wireguard peers
add allowed-address=192.168.34.2/32 comment="###" interface=wireguard1 persistent-keepalive=10s public-key="###"
add allowed-address=192.168.34.3/32,192.168.169.0/24 comment="hAP AC_RW1" interface=wireguard1 persistent-keepalive=10s public-key="###"
add allowed-address=192.168.34.4/32,192.168.170.0/24 comment=mAP_2n_RW2 interface=wireguard1 persistent-keepalive=10s public-key="###"
add allowed-address=192.168.34.5/32,192.168.8.0/24 comment=mango1 interface=wireguard1 persistent-keepalive=25s public-key="###"
add allowed-address=192.168.34.6/32 comment="###" interface=wireguard1 public-key="###="
add allowed-address=192.168.35.2/32,10.9.8.0/24 comment="###" interface=wireguard2 persistent-keepalive=25s public-key="###"
add allowed-address=192.168.34.7/32 comment="enigmatic (###)" interface=wireguard1 persistent-keepalive=25s public-key="###"
/ip address
add address=10.10.10.1/24 comment="Guest Network" interface=bridge2_guest network=10.10.10.0
add address=192.168.93.1/24 interface=bridge1 network=192.168.93.0
add address=192.168.34.1/24 interface=wireguard1 network=192.168.34.0
add address=10.10.11.1/24 comment="Church Network" interface=bridge3_church network=10.10.11.0
add address=172.30.30.2/24 comment="###" interface=bridge1 network=172.30.30.0
add address=192.168.35.1/24 interface=wireguard2 network=192.168.35.0
add address=10.9.8.2/24 disabled=yes interface=bridge1 network=10.9.8.0
/ip dhcp-client
add interface=sfp-sfpplus12
add disabled=yes interface=ether1
/ip dhcp-server lease
.
.
.
/ip dhcp-server network
add address=10.10.10.0/24 gateway=10.10.10.1
add address=10.10.11.0/24 gateway=10.10.11.1
add address=192.168.93.0/24 dns-server=1.1.1.1,8.8.4.4 gateway=192.168.93.1 netmask=24
/ip firewall address-list
.
.
.
/ip firewall filter
add action=accept chain=forward dst-address=192.168.93.137 dst-port=22 in-interface=pppoe-out1 protocol=tcp
add action=accept chain=input dst-port=13231 protocol=udp
add action=accept chain=input comment=wireguard2 dst-port=13232 protocol=udp
add action=accept chain=input comment=wireguard2 dst-port=13232 protocol=tcp
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface=ether1
add action=drop chain=forward comment="Drop to bogon list" dst-address-list=Bogons
add action=drop chain=forward comment="Drop dodgy countries" src-address-list=CountryIPBlocks
add action=accept chain=forward comment="defconf: accept established,related" connection-state=established,related
add action=drop chain=forward src-address=211.37.174.46
add action=drop chain=forward connection-nat-state=!dstnat in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=forward comment=all-connections new-connection-mark=all-connection passthrough=yes
add action=mark-packet chain=forward comment=Guest-Download-pkts connection-mark=all-connection dst-address-list="Guest Network" new-packet-mark=guest-Download-pkts passthrough=no
add action=mark-packet chain=forward comment=guest-Upload-pkts connection-mark=all-connection new-packet-mark=guest-Upload-pkts passthrough=no src-address-list="Guest Network"
add action=mark-packet chain=forward comment=church-Upload-pkts connection-mark=all-connection new-packet-mark=church-Upload-pkts passthrough=no src-address-list="Church Network"
add action=mark-packet chain=forward comment=Church-Download-pkts connection-mark=all-connection dst-address-list="Church Network" new-packet-mark=church-Download-pkts passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
add action=dst-nat chain=dstnat comment="SSH Port Forward to Sadam" dst-address=193.237.82.109 dst-port=22 in-interface=pppoe-out1 protocol=tcp src-address=0.0.0.0/0 to-addresses=192.168.93.137 to-ports=22
add action=dst-nat chain=dstnat comment="FTP Port 21 Forward to Sadam" disabled=yes dst-address=193.237.82.109 dst-port=21 in-interface=pppoe-out1 protocol=tcp src-address=0.0.0.0/0 to-addresses=192.168.93.137 to-ports=21
add action=dst-nat chain=dstnat comment="TFTP Port 69 Forward to Sadam" disabled=yes dst-address=193.237.82.109 dst-port=69 in-interface=pppoe-out1 protocol=udp src-address=0.0.0.0/0 to-addresses=192.168.93.137 to-ports=69
add action=dst-nat chain=dstnat comment="FTP Port 20 Forward to Sadam" disabled=yes dst-address=193.237.82.109 dst-port=20 in-interface=pppoe-out1 protocol=tcp src-address=0.0.0.0/0 to-addresses=192.168.93.137 to-ports=20
add action=netmap chain=dstnat comment="ICMP Port Forward to Sadam" disabled=yes dst-address=193.237.82.109 in-interface=pppoe-out1 protocol=icmp src-address=0.0.0.0/0 to-addresses=192.168.93.137
add action=dst-nat chain=dstnat comment="Wireguard to sadam tcp" dst-address=193.237.82.109 dst-port=51820 in-interface=pppoe-out1 protocol=udp src-address=0.0.0.0/0 to-addresses=192.168.93.137 to-ports=51820
add action=dst-nat chain=dstnat comment=Mikrotik_Wireguard_TCP disabled=yes dst-address=198.185.159.144 dst-port=13231 protocol=tcp src-port=13231 to-addresses=192.168.93.1 to-ports=13231
add action=dst-nat chain=dstnat comment=Mikrotik_Wireguard_UDP disabled=yes dst-address=198.185.159.144 dst-port=13231 protocol=udp src-port=13231 to-addresses=192.168.93.1 to-ports=13231
add action=dst-nat chain=dstnat comment="get to sadam grafana from outside but don't think this works" disabled=yes dst-port=3000 protocol=tcp to-addresses=192.168.93.137 to-ports=3000
add action=dst-nat chain=dstnat comment="this is the rule that messes everything up" disabled=yes dst-port=80 protocol=tcp to-addresses=192.168.93.137 to-ports=3000
/ip route
add disabled=no distance=1 dst-address=192.168.169.0/24 gateway=wireguard1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add dst-address=192.168.88.0/24 gateway=wireguard1
add disabled=no distance=1 dst-address=192.168.170.0/24 gateway=wireguard1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no dst-address=192.168.8.0/24 gateway=wireguard1 routing-table=main suppress-hw-offload=no
add comment="voltserver The O2" disabled=no distance=1 dst-address=10.9.8.0/24 gateway=wireguard2 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no dst-address=192.168.34.7/32 gateway=wireguard1 routing-table=main suppress-hw-offload=no
add disabled=yes distance=1 dst-address=192.168.35.2/32 gateway=wireguard2 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address=192.168.93.0/24
set ssh address=192.168.93.0/24
set api disabled=yes
set winbox address=192.168.93.0/24,192.168.34.0/24
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/London
/system identity
set name=CCR2004
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=0.uk.pool.ntp.org
/system resource irq rps
set ether1 disabled=no
/tool bandwidth-server
set enabled=no
/tool graphing interface
add allow-address=192.168.93.0/24
/tool graphing queue
add allow-address=192.168.93.0/24
/tool graphing resource
add allow-address=192.168.93.0/24
/tool romon
set enabled=yes
/tool sniffer
set filter-interface=all filter-ip-address=192.168.34.1/32 filter-ip-protocol=icmp only-headers=yes

anav · September 1, 2023, 2:43pm

Provide a clear diagram of what you would like to achieve, show both wireguard networks and which subnets should be accessing which subnets and which mobile user should be accessing which routers/subnets etc…
Aka a plan that is understood.

benny532017 · September 1, 2023, 3:31pm

Thanks for the replay already anav.

Diagram attached

Behaviour I have so far is, from PC02 I can successfully ping:
192.168.93.1
192.168.34.1
192.168.35.1
10.9.8.1
10.9.8.11

But from PC01 I can ping:
192.168.93.1
192.168.34.1
and not:
192.168.35.1
10.9.8.1
10.9.8.11

ohhhhhhhhhhhhhhh hahahaha I don’t have 192.168.35.0/24 in the wireguard client on PC01 - I’ve just added this and now it works.

anav, I know you ask for a diagram and I drew one out before and I didn’t spot this - you asking made me draw a specific one I saw the problem - my apologies!!!

I’ll post this anyway as it might help someone in the future (if it shows up in a search)

Okay so if I may continue this thread, the plot thickens - now I have a situation where the web GUI of ‘Device A’ loads very slowly and sometimes not at all on PC02 but works perfectly on PC01. I spotted the MSS clamping section in your guide anav, but that’s pushing my knowledge - any ideas or guesses on what could be causing this behaviour?

Thanks

B

anav · September 1, 2023, 4:48pm

You still need more clarity at least for simple me!
I only see one MT device and assuming only one connection to the internet on the diagram through this device the CCS2004.

What is Teltonika?
Is it a router attached to the internet why the connection to a subnet on the MT. Is this a double nat local downstream router??

The PC with a wireguard Client WHY???
The CRS2004 should act as the wireguard host for all LOCAL devices ??

benny532017 · September 1, 2023, 6:30pm

Sorry I’ve tried to simplify it but have probably overdone it!

These devices are not at the same location:

The CCR2004 is at our office which has a fixed IP address so the other devices can initiate towards it. The LAN at the office is 192.168.93.0/24 and .1 is the CCR as shown.

The Teltonika box is a remote cellular modem/router combo so has LAN 10.9.8.0/24 - as its internet comes from a cellular modem it probably has carrier NAT.

The PC with a wireguard Client - this is so that PC (PC01 in the drawing) can connect back to the “Office network” from starbucks or wherever he happens to be in the world. I have shown it connected to my home network which is 192.168.9.0/24 I have a few other “road warrior” devices not shown in the diagram, a mixture of Mikrotik and non-Mikrotik which have wireguard interfaces with IP addresses in the 192.168.34.0/24 subnet and these all work perfectly as desired - with the wireguard providing connectivity back the office.

Hope that makes sense? Have I left anything ambiguous?

B

anav · September 1, 2023, 8:37pm

Yes, a clearer set of requirements.

I have no idea why you are showing 192.168.35.0/24 connected to telefonica?/
Does this mean you would like that subnet to reach the 10.9.8.0/24 subnet ???

IS your home router a mikrotik router, the one that PCO1 is behind and why is 192.16.34.0/24 connected to it???
Should I assume that you mean for that subnet to reach your home subnet of 192.168.9.0/24

In other words
a. identify all user(s)/device(s) and groups of users/devices including admin
b. identify what traffic they require.

Then we can plan and configure accordingly.

benny532017 · September 1, 2023, 10:08pm

Maybe it’s an odd way of depicting a logical network. 192.168.35.0/24 is the wireguard subnet, so the wireguard2 interface on the CCR2004 is 192.168.35.1 and the wireguard interface on the Teltonika is 192.168.35.2 - since the Teltonika is configured with the public IP address of the CCR as the endpoint in the peer config, that’s how it makes the wireguard tunnel. So in terms of reaching 10.9.8.0/24 - I would like anything on 192.168.35.0/24 to be able to reach it (which it can) and indeed anything on 192.168.93.0/24 (this also works).

My home router is currently a generic DSL router but I’m about to replace it with a Mikrotik one - 192.168.34.7 is the address of the wireguard interface on that PC, the address the home router gives the PC (for internet access) is 192.168.9.100. I don’t really care if 192.168.9.0/24 reaches anyof these other subnets (for the time being at least), but I want that PC to be able to reach 192.168.93.0/24 (via the wireguard interface), as it could be anywhere. By adding routes on the CCR and allowed IPs in the carious wireguard configs I seem to be able to get around, I just have this weird problem where http is really slow or broken when going 192.168.93.0/24 → 10.9.8.11 but works fine 192.168.34.0/24 → 10.9.8.11

anav · September 2, 2023, 2:46pm

I started going through the configs making recommendations until I got to the CCR peers.
I then saw an hapac and map2n and mango and something else NOT depicted on the diagram.
I am unable to help when the facts are not made available and then come out in dribs and drabs… when facts & truths are not forthcoming I move along.
Hopefully someone else with lower expectations will help. Especially since its straight forward.

benny532017 · September 2, 2023, 8:55pm

No problem anav, I understand and am already very thankful that you have helped me with my main problem - I really appreciate you taking the time. I should also say a big thank you for your other forum posts which I have found very useful, as well as your guide “Wireguard Success For The Beginner” - really valuable stuff!

For the sake of anyone who comes past this thread in the future, those other devices are other “road warriors” some are Mikrotik and some are not and they all work as intended - giving the user of that device access to the main office LAN 192.168.93.0/24

I’ll play around with some packet captures and see if I can troubleshoot this remaining issue, but for now I will call this problem solved.

anav · September 3, 2023, 1:11am

I have some ideas on how to adjust…
Why the different wireguard interfaces you need only one.
You have two options,
a. use firewall rules to control who goes where
b. you can use two subnets to describe one wireguard interface ( ideal for your situation ) as you seem to want to have two separate uses…

The key for success is how you setup the firewall at the RELAY router.