Weird Wireguard subnet problem

Hello

We are implementing a pilot with Wireguard on a small Mikrotik RB450 running RouterOS V7.6 as Tunnel Server for an internal network composed of 2 LANs: 192.168.0.0/24 (adm lan) and 172.16.0.0/20 (gpon lan). Wireguard is configured with it’s virtual interface “wireguard1” on subnet 192.168.32.1/24. My PC connects to the tunnel using IP 192.168.32.2/24 and I can ping any host on subnets 192.168.0.0/24, 192.168.1.0/24 and only the first 254 hosts on subnet 172.16.0.0/20. This happens also if I use the PING tool on RouterOS and use as source address the wireguard1 address on subnet 192.168.32.1/24.

RB450 IP routes:

  • DAd 0.0.0.0/0 192.168.1.254 1
  • DAc 172.16.0.0/20 ether5-gpon 0
  • 0 As 192.168.0.0/24 172.16.0.1 1
  • DAc 192.168.1.0/24 ether3-Oi 0
  • DAc 192.168.32.0/24 wireguard1 0

Ping command results:

  • ping src-address=192.168.32.1 address=192.168.1.254 - OK

  • ping src-address=192.168.32.1 address=192.168.0.100 - OK

  • ping src-address=192.168.32.1 address=172.16.0.50 - OK

  • ping src-address=192.168.32.1 address=172.16.1.10 - timeout!

Any clues why this is happening?
wireguard.png

Without seeing your config of RB450 or connected clients, no clue.

My first guess would be somewhere an allowed address with the wrong mask.

I did the tests without the wireguard client to enphasize the error does not seem related to the wireguard client configuration. It happens based on the ip packet source address. If i ping any node on subnet 172.16.0.0/20 from the RB450 terminal window they all work. Only when I specify the source address as being 192.168.32.1 attributed to the wireguard1 interface the problem occurs.

  • ping 172.16.8.1 - OK

  • ping 172.16.8.1 src-address=192.168.32.1 - TIMEOUT

Please find below both the wireguard client configuration and the RB450 export configuration.

Wireguard Windows 10 Client config

[Interface]
PrivateKey = XXXXXXXXXXXXXXXXXXXXXXXX
Address = 192.168.32.2/24
DNS = 192.168.0.100

[Peer]
PublicKey = XXXXXXXXXXXXXXXXXXXXXXXXX
AllowedIPs = 192.168.32.0/24, 192.168.0.0/24, 172.16.0.0/20
Endpoint = d4500d85ddc1.sn.mynetname.net:13231
PersistentKeepalive = 10

RB450 export configuration

oct/20/2022 16:50:59 by RouterOS 7.6

software id = JTLI-04BH

model = RB760iGS

serial number = D4500D85DDC1

/interface ethernet set [ find default-name=ether1 ] advertise=1000M-half,1000M-full comment=“adm LAN” disabled=yes loop-protect=on
/interface ethernet set [ find default-name=ether2 ] disabled=yes
/interface ethernet set [ find default-name=ether3 ] advertise=1000M-half,1000M-full comment=“wan - Oi Fiber” name=ether3-Oi
/interface ethernet set [ find default-name=ether4 ] disabled=yes
/interface ethernet set [ find default-name=ether5 ] advertise=1000M-half,1000M-full comment=“gpon LAN” name=ether5-gpon
/interface ethernet set [ find default-name=sfp1 ] disabled=yes
/interface wireguard add listen-port=13231 mtu=1420 name=wireguard1
/disk set sd1-part1 name=disk1
/interface list add comment=defconf name=WAN
/interface list add comment=defconf name=LAN
/interface list add name=“adm lan”
/interface list add name=“gpon lan”
/interface lte apn set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik
/port set 0 name=serial0
/routing bgp template set default disabled=no output.network=bgp-networks
/routing ospf instance add disabled=yes name=default-v2
/routing ospf area add disabled=yes instance=default-v2 name=backbone-v2
/snmp community set [ find default=yes ] name=apolom-ro
/system logging action set 0 memory-lines=2500
/system logging action set 1 disk-file-name=disk1/logs/log disk-lines-per-file=10000
/system logging action set 3 remote=172.16.0.50 src-address=172.16.15.254
/dude set data-directory=disk1/dude-data enabled=yes
/ip neighbor discovery-settings set discover-interface-list=none
/ip settings set rp-filter=strict
/ipv6 settings set max-neighbor-entries=8192
/interface detect-internet set internet-interface-list=WAN lan-interface-list=LAN wan-interface-list=WAN
/interface list member add comment=“gpon subnet” interface=ether5-gpon list=LAN
/interface list member add comment=“Oi Fibra” interface=ether3-Oi list=WAN
/interface list member add comment=“WG clients are LAN” interface=wireguard1 list=LAN
/interface wireguard peers add allowed-address=192.168.32.2/32 comment=“Lenovo PC” interface=wireguard1 public-key=“VpvXEZzZBCc3N0cXW/UcvO6v/nth0XIT/9wo4jGTTm8=”
/interface wireguard peers add allowed-address=192.168.32.3/32 interface=wireguard1 public-key=“RyzB8itPnzMiC3WEOEfsDUov2vS3Y6WPIfe1Gh9E3D4=”
/ip address add address=172.16.15.254/20 comment=“gpon subnet” interface=ether5-gpon network=172.16.0.0
/ip address add address=192.168.32.1/24 comment=“WireGuard subnet” interface=wireguard1 network=192.168.32.0
/ip cloud set ddns-enabled=yes
/ip dhcp-client add comment=“Oi Fibra” interface=ether3-Oi use-peer-dns=no use-peer-ntp=no
/ip dns set servers=192.168.0.100,192.168.0.101
/ip firewall address-list add address=192.168.0.100 comment=“Lista de servidores DNS da rede APOLOM” list=internal-DNS-Servers
/ip firewall address-list add address=192.168.0.101 comment=“Lista de servidores DNS da rede APOLOM” list=internal-DNS-Servers
/ip firewall address-list add address=192.168.0.128/28 comment=“Bloco de enderecos sem acesso \E0 Internet - rede adm - faixa de 192.168.0.128 a 192.168.0.143” list=Block-addresses
/ip firewall address-list add address=172.16.0.128/28 comment=“Bloco de enderecos sem acesso \E0 Internet - rede gpon - faixa de 172.16.0.128 a 172.16.0.143.” list=Block-addresses
/ip firewall filter add action=accept chain=input comment=“Allow Wireguard” dst-port=13231 protocol=udp
/ip firewall filter add action=drop chain=input comment=“Drop DNS requests from Internet - UDP” disabled=yes dst-port=53 in-interface-list=WAN protocol=udp
/ip firewall filter add action=drop chain=input comment=“Drop DNS requests from Internet - TCP” disabled=yes dst-port=53 in-interface-list=WAN protocol=tcp
/ip firewall filter add action=accept chain=input comment=“defconf: accept established,related,untracked” connection-state=established,related,untracked
/ip firewall filter add action=accept chain=input comment=“defconf: accept ICMP” protocol=icmp
/ip firewall filter add action=accept chain=input comment=“defconf: accept to local loopback (for CAPsMAN)” dst-address=127.0.0.1
/ip firewall filter add action=drop chain=input comment=“defconf: drop invalid” connection-state=invalid log-prefix=“INVALID - "
/ip firewall filter add action=drop chain=input comment=“defconf: drop all not coming from LAN” in-interface-list=!LAN log-prefix=“NOT LAN - "
/ip firewall filter add action=drop chain=forward comment=“Bloqueia acesso \E0 Internet para lista de enderecos bloqueados” dst-address-list=”” log-prefix=Blocked-IP-Address- out-interface-list=WAN src-address-list=Block-addresses
/ip firewall filter add action=accept chain=forward comment=“defconf: accept in ipsec policy” ipsec-policy=in,ipsec
/ip firewall filter add action=accept chain=forward comment=“defconf: accept out ipsec policy” ipsec-policy=out,ipsec
/ip firewall filter add action=accept chain=forward comment=“Allow DNS forwarder form internal DNS servers” disabled=yes dst-port=53 out-interface-list=WAN protocol=udp src-address-list=internal-DNS-servers
/ip firewall filter add action=accept chain=forward comment=“Allow DNS forwarder form internal DNS servers” disabled=yes dst-port=53 out-interface-list=WAN protocol=tcp src-address-list=internal-DNS-servers
/ip firewall filter add action=drop chain=forward comment=“Drop DNS traffic for everybody else” disabled=yes dst-port=53 out-interface-list=WAN protocol=udp
/ip firewall filter add action=drop chain=forward comment=“Drop DNS traffic for everybody else” disabled=yes dst-port=53 out-interface-list=WAN protocol=tcp
/ip firewall filter add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related hw-offload=yes
/ip firewall filter add action=accept chain=forward comment=“defconf: accept established,related, untracked” connection-state=established,related,untracked
/ip firewall filter add action=drop chain=forward comment=“defconf: drop invalid” connection-state=invalid
/ip firewall filter add action=drop chain=forward comment=“defconf: drop all from WAN not DSTNATed” connection-nat-state=!dstnat connection-state=new in-interface-list=WAN log-prefix=“WAN not DSTNATed - "
/ip firewall nat add action=masquerade chain=srcnat comment=“defconf: masquerade” ipsec-policy=out,none out-interface-list=WAN
/ip firewall nat add action=dst-nat chain=dstnat comment=“Rule for SoftEther VPN Server” disabled=yes dst-address-type=local dst-port=5555 log=yes protocol=tcp to-addresses=172.16.0.20 to-ports=5555
/ip firewall nat add action=dst-nat chain=dstnat comment=“Rule for L2TP access” disabled=yes dst-port=500 log=yes protocol=udp to-addresses=172.16.0.20
/ip firewall nat add action=dst-nat chain=dstnat comment=“Rule for L2TP access” disabled=yes dst-port=4500 log=yes protocol=udp to-addresses=172.16.0.20
/ip route add disabled=no dst-address=192.168.0.0/24 gateway=172.16.0.1 routing-table=main suppress-hw-offload=no
/ip service set telnet disabled=yes
/ip service set ftp disabled=yes
/ip service set www disabled=yes
/ip service set ssh address=172.16.0.0/20,192.168.0.0/24,192.168.32.0/24
/ip service set www-ssl address=172.16.0.0/20,192.168.0.0/24,192.168.32.0/24 certificate=Webfig disabled=no
/ip service set api disabled=yes
/ip service set winbox address=172.16.0.0/20,192.168.0.0/24,192.168.32.0/24,127.0.0.1/32
/ip service set api-ssl disabled=yes
/ipv6 dhcp-client add add-default-route=yes interface=ether3-Oi pool-name=apolom-ipv6 request=address use-peer-dns=no
/ipv6 firewall address-list add address=::/128 comment=“defconf: unspecified address” list=bad_ipv6
/ipv6 firewall address-list add address=::1/128 comment=“defconf: lo” list=bad_ipv6
/ipv6 firewall address-list add address=fec0::/10 comment=“defconf: site-local” list=bad_ipv6
/ipv6 firewall address-list add address=::ffff:0.0.0.0/96 comment=“defconf: ipv4-mapped” list=bad_ipv6
/ipv6 firewall address-list add address=::/96 comment=“defconf: ipv4 compat” list=bad_ipv6
/ipv6 firewall address-list add address=100::/64 comment=“defconf: discard only " list=bad_ipv6
/ipv6 firewall address-list add address=2001:db8::/32 comment=“defconf: documentation” list=bad_ipv6
/ipv6 firewall address-list add address=2001:10::/28 comment=“defconf: ORCHID” list=bad_ipv6
/ipv6 firewall address-list add address=3ffe::/16 comment=“defconf: 6bone” list=bad_ipv6
/ipv6 firewall filter add action=accept chain=input comment=“defconf: accept established,related,untracked” connection-state=established,related,untracked
/ipv6 firewall filter add action=drop chain=input comment=“defconf: drop invalid” connection-state=invalid
/ipv6 firewall filter add action=accept chain=input comment=“defconf: accept ICMPv6” protocol=icmpv6
/ipv6 firewall filter add action=accept chain=input comment=“defconf: accept UDP traceroute” port=33434-33534 protocol=udp
/ipv6 firewall filter add action=accept chain=input comment=“defconf: accept DHCPv6-Client prefix delegation.” dst-port=546 protocol=udp src-address=fe80::/10
/ipv6 firewall filter add action=accept chain=input comment=“defconf: accept IKE” dst-port=500,4500 protocol=udp
/ipv6 firewall filter add action=accept chain=input comment=“defconf: accept ipsec AH” protocol=ipsec-ah
/ipv6 firewall filter add action=accept chain=input comment=“defconf: accept ipsec ESP” protocol=ipsec-esp
/ipv6 firewall filter add action=accept chain=input comment=“defconf: accept all that matches ipsec policy” ipsec-policy=in,ipsec
/ipv6 firewall filter add action=drop chain=input comment=“defconf: drop everything else not coming from LAN” in-interface-list=!LAN
/ipv6 firewall filter add action=accept chain=forward comment=“defconf: accept established,related,untracked” connection-state=established,related,untracked
/ipv6 firewall filter add action=drop chain=forward comment=“defconf: drop invalid” connection-state=invalid
/ipv6 firewall filter add action=drop chain=forward comment=“defconf: drop packets with bad src ipv6” src-address-list=bad_ipv6
/ipv6 firewall filter add action=drop chain=forward comment=“defconf: drop packets with bad dst ipv6” dst-address-list=bad_ipv6
/ipv6 firewall filter add action=drop chain=forward comment=“defconf: rfc4890 drop hop-limit=1” hop-limit=equal:1 protocol=icmpv6
/ipv6 firewall filter add action=accept chain=forward comment=“defconf: accept ICMPv6” protocol=icmpv6
/ipv6 firewall filter add action=accept chain=forward comment=“defconf: accept HIP” protocol=139
/ipv6 firewall filter add action=accept chain=forward comment=“defconf: accept IKE” dst-port=500,4500 protocol=udp
/ipv6 firewall filter add action=accept chain=forward comment=“defconf: accept ipsec AH” protocol=ipsec-ah
/ipv6 firewall filter add action=accept chain=forward comment=“defconf: accept ipsec ESP” protocol=ipsec-esp
/ipv6 firewall filter add action=accept chain=forward comment=“defconf: accept all that matches ipsec policy” ipsec-policy=in,ipsec
/ipv6 firewall filter add action=drop chain=forward comment=“defconf: drop everything else not coming from LAN” in-interface-list=!LAN
/snmp set contact=“Andr\E9 Muzi” enabled=yes location=“Control Room” trap-generators=interfaces trap-interfaces=all trap-version=2
/system clock set time-zone-name=America/Sao_Paulo
/system identity set name=bastion
/system logging add action=remote topics=error
/system logging add action=remote topics=critical
/system logging add action=remote topics=warning
/system logging add action=disk topics=error
/system note set note=”\r
\nThis system belongs to APOLOM and can be used by authorized users only.\r
\nIts use is monitored.\r
\n\r
\n”
/system ntp client set enabled=yes
/system ntp server set broadcast=yes enabled=yes manycast=yes
/system ntp client servers add address=0.br.pool.ntp.org
/system ntp client servers add address=1.br.pool.ntp.org
/system ntp client servers add address=2.br.pool.ntp.org
/system ntp client servers add address=3.br.pool.ntp.org
/system scheduler add comment=“Auto Restart schedule” interval=2w name=Reboot on-event=“system reboot” policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=sep/30/2022 start-time=22:30:00
/tool bandwidth-server set enabled=no
/tool e-mail set address=mail.nextin.com.br from=someone@somewhere.com port=587 tls=starttls user=someone@somewhere.com
/tool graphing interface add interface=ether1
/tool graphing interface add interface=ether3-Oi
/tool graphing interface add interface=ether5-gpon
/tool graphing queue add
/tool graphing resource add
/tool mac-server set allowed-interface-list=none
/tool mac-server mac-winbox set allowed-interface-list=none
/tool mac-server ping set enabled=no
/tool sniffer set filter-interface=ether3-Oi
/tool traffic-monitor add comment=“Alarm for Internet Port - Download” disabled=yes interface=ether3-Oi name=tmon-download-log on-event=“:log info "WAN download reached 150Mbps"” threshold=150000000 traffic=received
/tool traffic-monitor add comment=“Alarm for Internet Port - Upload” disabled=yes interface=ether3-Oi name=tmon-upload-log on-event=“:log info "WAN upload reached 30Mbps"” threshold=30000000

Does everything in 172.16.0.0/20 subnet have route to 192.168.32.0/24?

Subnet 172.16.0.0/20 is directly attached to RB450, hence has a direct route to it. All IP nodes on the first “class-c” address range are reachable, but no other. And I cannot see any configuration where mask /24 was used by mistake. Even on the PC, when wireguard client is activated, we have a route to subnet 172.16.0.0/20. See below:

PC IPV4 Route Table with Wireguard active

IPv4 Route Table

Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.50.1 192.168.50.101 25
127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
127.0.0.1 255.255.255.255 On-link 127.0.0.1 331
127.255.255.255 255.255.255.255 On-link 127.0.0.1 331
172.16.0.0 255.255.240.0 On-link 192.168.32.2 5
172.16.15.255 255.255.255.255 On-link 192.168.32.2 261
192.168.0.0 255.255.255.0 On-link 192.168.32.2 5
192.168.0.255 255.255.255.255 On-link 192.168.32.2 261
192.168.1.0 255.255.255.0 On-link 192.168.32.2 5
192.168.1.255 255.255.255.255 On-link 192.168.32.2 261
192.168.32.0 255.255.255.0 On-link 192.168.32.2 5
192.168.32.2 255.255.255.255 On-link 192.168.32.2 261
192.168.32.255 255.255.255.255 On-link 192.168.32.2 261
192.168.50.0 255.255.255.0 On-link 192.168.50.101 281
192.168.50.101 255.255.255.255 On-link 192.168.50.101 281
192.168.50.255 255.255.255.255 On-link 192.168.50.101 281
224.0.0.0 240.0.0.0 On-link 127.0.0.1 331
224.0.0.0 240.0.0.0 On-link 192.168.50.101 281
255.255.255.255 255.255.255.255 On-link 127.0.0.1 331
255.255.255.255 255.255.255.255 On-link 192.168.50.101 281

The route that matters is to 192.168.32.0/24. If device in 172.16.0.0/20 subnet has RB450 as its default gateway, it’s covered. But if some other device used RB3011 as gateway, and if RB3011 didn’t have route to 192.168.32.0/24, such device wouldn’t be reachable from 192.168.32.x. But on second look, it can’t be exactly this, because if it works with 192.168.0.0/24 behind RB3011, then RB3011 must have route to 192.168.32.0/24. Anyway, you see where I’m going with this, it’s important to look not only at VPN->target direction, but also at target->VPN. It’s also possible that I’m misunderstanding something, the image and description seemed a bit confusing.

My objective with this scenario is to have a dedicated Wireguard VPN Server to allow external third party professionals to access ONLY subnet 172.16.0.0/20 via VPN for support purposes. No need whatsoever to access any other subnet. As subnet 172.16.0.0/20 is local to the RB450, I’m expecting that no route nor default gateway woiuld be needed .

Mauricio

Small note, your Win10 Wireguard-config is not fully correct ;

[Interface]
PrivateKey = XXXXXXXXXXXXXXXXXXXXXXXX
Address = 192.168.32.2/24
DNS = 192.168.0.100

You need to specify a /32 here !
Each WG-endpoint receives a /32 (well … not really “receives” off course but you get the picture…)

Hello jvanhambelgium

As far as the tutorials go, the /32 is only used on the “wireguard peer” definition at the RouterOS side, right? The W10 WG client config reflects my attempts to make it work as intended.
As far as I understood the Wireguard tutorials available, the “production W10 client” config file should be something like:

[Interface]
PrivateKey = XXX
Address = 192.168.32.2/29 (only 6 addresses needed in total - from .2 to .7)
DNS = 8.8.8.8

[Peer]
PublicKey = XXX
AllowedIPs = 172.16.0.0/20
Endpoint = 99999999999.sn.mynetname.net:13231
PersistentKeepalive = 30

Help me if I’m wrong. THANKS!

Mauricio

Don’t confuse address for interface with allowed address.

In theory you could have stumbled upon a bug or something, especially with these larger subnets.
Your story makes sense.
This 172.16.1.10 device that you are trying to ping, can you tell me something about it (Windows? Linux? custom appliance)? What is it ?
Does it have a gateway set ? To where ?
Can you do some tcpdump on this device to see IF something arrives in the first place?

On the RB450, did you enable all sorts of firewall-logging ? To see if something hits against certain (unexpected rules) or something?

Echo others

(1) Windows client should be 192.168.32**.2/32**

(2) TRY:
DNS=192.168.32.1 just to see if it makes a difference as I dont see why its not working…

(3) For troubleshooting purpose please change RP-filter to loose (very few people use strict, and certainly verbotten for dual WAN).

(4) For troubleshooting turn this to NONE< It is not an understood feature and it has caused issues in the past.
/interface detect-internet set internet-interface-list=WAN lan-interface-list=LAN wan-interface-list=WAN

(5) I am horrible and dont undertstand IP address nomenclature so this seems weird to me
/ip address add address=172.16.15.254/20 comment=“gpon subnet” interface=ether5-gpon network=172.16.0.0

What happens if you use 172.16.0.1/20 for IP address…

(6) I dont know what the heck you are trying to accomplish with DNS services but you have many rules and Im sure they are screwing with your config.
Someone expert on DNS should help as I think it needs attention.
The forward chain rules as is now are blocking DNS… but its a router service which is input chain… in any case I am confused..

(7) Can you explain this route…/ip route add disabled=no dst-address=192.168.0.0/24 gateway=172.16.0.1 routing-table=main suppress-hw-offload=no

Lets discuss the fact that there is no such subnet identified on your config.
I only see two DNS servers? 192.168.0.100/101 but nothing else indicating this lan subnet???

Not necessarily. Address with /24 mask (or generally non-/32 mask) is fine, it’s just useless without the rest of /24 in allowed addresses, because it will create route to /24, but it won’t be of any use if not allowed.

Edit: And for @anav, play with this nice tool: http://www.subnetmask.info/ (172.16.0.0, default, 4096 hosts, Calculate => /20; List Networks => 172.16.0.0-172.16.15.255 = same network)

Being nitpicky again, but please focus your attention on other areas which may be a problem, dont waste that very large MT brain of yours!!!

Ungrateful animal, I’m trying to enlighten you. :wink:

As for the problem, it just needs some debugging. If something consistently doesn’t work, it’s a dream. Just think about what should happen and in what order, and then go step by step and find where it fails. Keep the ping running and check if you see outgoing packets on right interface, if they arrive to target device (pick some where you can check it), if it sends any responses, where to exactly, if they arrive there, etc.

It’s very important, not for traffic to device, but for traffic from device. If source is RB450’s 172.16.15.254, then it’s direct communication. But if it’s 192.168.32.1, then device has to use its default gateway to reach it. If default gateway is RB450, there’s no problem. But if it’s RB3011, it’s different. One thing I see now, it would be asymmetric routing, which could be blocked by RB3011’s firewall.

Thanks for all the help so far.

I’ll try to simplify the scenario to the bare minimum, do the suggested troubleshooting and post the results here.

Mauricio

The missing part for me is as follows:

I dont see the 192.168.0.0 network you are saying is behind the RB450.

To be clear what is true?

a. the RB450 is a Wireguard server for the windows10 client,
B. the RB3011 is the wireguard server for the RB450 client
c. the RB3011 and RB450 are not connected via wg.

We are in the middle of a router upgrade. RB3011 will be replaced by a CCR1009 as main router. RB3011 will replace RB450 as Wireguard VPN Server. Roles that will be implemented:

  • CCR1009 - Main router for both internal LANs (192.168.0.0/24 and 172.16.0.0/20), with connections to both Cable ISP (primary) and Fiber ISP (fallback)
  • RB3011 - Wireguard VPN Server for a single internal LAN (172.16.0.0/20), with connection only to Fiber ISP

Cable ISP allows it’s cable modem to operate in bridge mode, hence CCR1009 will get a valid IPV4 public address
Fiber ISP does not allow it’s ONU to operate in bridge mode, hece ONU is a router with 4 ports and reserved IPV4 subnet address (192.168.1.0/24).

I’ll post the final diagram after the changes and testing.

Mauricio

I see no reason to use the RB3011, it can all be done on the CCR1009.