Using WireGuard for select IP addresses

RouterOS Beginner Basics
1

I have a working MikroTik environment with two WireGuard interfaces: one for users to access internal resources and one for the router to connect to a remote VPS with its own WireGuard server. The use case for the second interface is to route all traffic from the LAN bound for specific IP addresses, which are defined in an address list. The problem is no traffic past the handshake is being routed to the remote host.

This is the WireGuard setup in the MikroTik router:

/interface wireguard
add disabled=no listen-port=55667 mtu=1420 name=linode private-key="privateKey"
/interface wireguard peers
add allowed-address=0.0.0.0/0,::/0 disabled=no endpoint-address=addressOfVPS endpoint-port=51001 interface=linode persistent-keepalive=30s preshared-key="presharedKey" public-key="publicKey"

I have a routing table and route configured for it:

/ip route
add dst-address=0.0.0.0/0 gateway=linode routing-table=to_vpn
/ipv6 route
add dst-address=::/0 gateway=linode routing-table=to_vpn
/routing table
add fib name=to_vpn

I also have IP addresses assigned:

/ip address
add address=10.0.0.2/30 disabled=no interface=linode network=10.0.0.0
/ipv6 address
add address=fdc9:281f:4d7:9ee9::3/128 advertise=no auto-link-local=yes disabled=no eui-64=no interface=linode no-dad=no

Finally, I set up firewall rules to allow traffic and NAT:

/ip firewall filter
add action=accept chain=forward comment="For Linode VPN" in-interface-list=VLAN out-interface=linode
/ip firewall mangle
add action=mark-connection chain=prerouting connection-state=new dst-address-list=route_to_vpn in-interface-list=VLAN new-connection-mark=to_vpn passthrough=yes
add action=mark-routing chain=prerouting connection-mark=to_vpn in-interface-list=VLAN new-routing-mark=to_vpn passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat connection-mark=to_vpn disabled=no out-interface=linode
/ipv6 firewall filter
add action=accept chain=forward comment="For Linode VPN" in-interface-list=VLAN out-interface=linode
/ipv6 firewall nat
add action=masquerade chain=srcnat connection-mark=to_vpn out-interface=linode !to-address !to-ports

However, I can’t get any traffic to pass over this connection. I have an address list named route_to_vpn under /ipv6 firewall and /ip firewall, and each has a domain name that was auto-resolved to two IP addresses. Even using ping, I couldn’t get traffic to pass over this connection:

/ping target.invalid%linode count=4
  SEQ HOST                                 SIZE TTL TIME       STATUS                                                                                                                                          
    0 ipAddress                                                timeout                                                                                                                                         
    1 ipAddress                                                timeout                                                                                                                                         
    2 ipAddress                                                timeout                                                                                                                                         
    3 ipAddress                                                timeout                                                                                                                                         
    sent=4 received=0 packet-loss=100%

However, using /ping with the WAN interface works just fine. WireGuard on the VPS side shows the handshake is complete and that the keepalive messages are being received. However, I used netcat to listen on the VPN port and saw absolutely no traffic when I ran ping.

I’m certain I’ve overlooked something silly, and I would appreciate help finding it. If I need to post my full config, then please let me know. Also, if you know a simpler way to do this task, then I would appreciate any insight.

2

Can I assume that the LAN users going out the VPS for specific WANIP addresses, have a firewall rule so that only traffic to those addresses is permitted. Must be a long list and thus more practical to allow all addresses on the wireguard rule itself.

add chain=forward action=accept in-interface=VLAN dst-address-list=route_to_vpn out-interface=linode

So they are allowed to use wireguard, but we need to let the router know where to send the traffic.....
Hence the mangling, One big change is NOT to use the same name for both the connection mark and routing mark, bad things could happen.

/ip firewall mangle
add action=mark-connection chain=forward connection-mark=no-mark 
dst-address-list=route_to_vpn in-interface-list=VLAN new-connection-mark=vps_send passthrough=yes
add action=mark-routing chain=prerouting connection-mark=vps_send new-routing-mark=to_vpn passthrough=no
3

Yeah, said resource is behind a long list of rotating IP addresses. Hence, it’s easier to just resolve the domain name whenever the TTL expires and dynamically update the address list that way.

I was unaware that the connection and routing marks need to be different. I’ll fix that immediately.

4

I narrowed the cause of the problem to masquerading by RouterOS. Debug logging for WireGuard showed packets are being sent with the destination’s IP address as the source:

Jul 04 15:46:54 MULTI kernel: wireguard: wg0: Packet has unallowed src IP (destinationIPv6Address) from peer 8 (mikrotikRouter:port)
Jul 04 15:46:54 MULTI kernel: wireguard: wg0: Packet has unallowed src IP (destinationIPv4Address) from peer 8 (mikrotikRouter:port)

After disabling the masquerade rules for VPN traffic (cf. my original post), the sending device’s LAN IP address was used instead. All of these should be permitted by AllowedIPs on the client side, which is set to 0.0.0.0/0 and ::0/0, so there is clearly a separate issue on the WireGuard server side (which I will investigate separately).

5

After extensive testing and reconfiguration, this is where I am:

  • WireGuard debug logging on the WireGuard server complains about unallowed source IP addresses, but these are normal and can be ignored. This is why I thought MikroTik was somehow adding the wrong source address.

kernel: wireguard: wg0: Packet has unallowed src IP (2606:4700:10::6814:179b) from peer 1 (myRouter:myPort)
kernel: wireguard: wg0: Packet has unallowed src IP (104.20.23.155) from peer 1 (myRouter:myPort)

  • Packet captures show the target server is responding to the initial TCP handshake, but this response is never making it back to the client. It goes from the target, through the VPS (the WireGuard server), through the hEX S (the WireGuard client), and then it stops.

On this network, the web browser is behind a hAP ax³, a CRS112, and a hEX S. The hEX S is the only path to and from the Internet, and it looks like it’s not sending WireGuard response packets to the CRS112. At least, I assume this based on packet captures taken on the hEX S and the CRS112. The hEX S shows the packet that should be going back to the browser, but the CRS112 never sees it.

No other traffic appears to have this problem. The browser is on a VLAN, but tagging occurs at the CRS112 and not at the hEX S. Regardless, it’s part of an established connection.

EDIT: Oh! There are no firewall rules configured on any device other than the hEX S. It’s the only device performing routing in this LAN.

6

Are you saying you still need help or that its working??
If you still need help, I am happy to assist provided you include the latest configs of all devices and a network diagram explaining the relationships between them. (traffic flow, vlans, wireguard etc. )

7

Yes, I do need help. I apologize for the incomplete thought I left in my last comment. Attached are the configuration exports for the router (hEX S) and the core switch (CRS112) involved. This is what I expect to happen:

  • Packets bound for the VPS-hosted WireGuard server should hit the router on interface COMMON_VLAN (which physically comes in through ether2) and then exit via interface linode. This is working as expected.
  • Packets coming back from the VPS-hosted WireGuard server should come in via the interface linode and go to the core switch via interface COMMON_VLAN (which physically goes out via ether2). This is not working as expected.

The relevant network setup is AT&T Gateway > MikroTik router > MikroTik core switch > MikroTik hAP ax³ > Test client. Packets aren’t making it to the core switch, so I didn’t provide a config for the hAP ax³.

As you can see in the configs, there are VLANs involved. I am testing from VLAN ID 10, but since the connection is being tracked, I don’t expect VLAN tagging to be at fault.

I took packet captures on the router (watching interface linode and then ether2 and COMMON_VLAN) and on the core switch (watching interface spf10, which is connected to ether2 on the router). The return packets from linode are supposed to go to COMMON_VLAN and then on to the core switch, but I can’t see them hitting COMMON_VLAN on the router or any interface on the core switch. (While it is possible hardware offloading is making the packets invisible on the core switch, the same should not be true of the router.)

Note: I took the packet captures in the tx direction, so the packets should already have passed through the firewall. Also, I only tested IPv4 when taking the packet captures, but the same problem occurs with IPv6 over WireGuard.

There is another WireGuard interface (homevpn) you will see in the router config that are unrelated to this setup. That is for external clients to access internal resources, and it’s working as expected.

EDIT: Corrected the incoming and outgoing interfaces for client packets.

core_switch.rsc (5.9 KB)
router.rsc (20.2 KB)

8

Even if the wireguard is 'UNRELATED' one has to make sure there is not interference between them but correct usually not a problem.

Normally problems that you describe, fall into two main areas, peer settings ( and associated needed routes) and firewall rules. However your config is very complex and I think the main issue beside many small ones is the way you are moving traffic to vps. Your explanation has not made it clear and thus why I asked for a network diagram. Make it simple.................. Identify which vlans are to go out VPS.

it would seem that the challenge stems from using dst-address as the determining factor if traffic should go out VPS. You state these IP addresses are not static in that they change from time to time.
I am not a script monkey but suffice to say, your remit is to ensure the firewall address list of those external addresses is kept up to date somehow!!!!

Observations:

  1. Your ether5 settings do not make sense.
    a. you describe it in Bridge PORTS as an access port for for the base vlan (99)
    b. you describe it in Bridge VLANS as a trunk port (carrying vlans 10,20) and properly as an access port for vlan99.

Conclusion, you are trying to describe ether5 as a hybrid port.

If so then the two parts of Bridge settings have to match an all to say
From: add bridge=bridge comment=defconf frame-types=admit-only-untagged-and-priority-tagged interface=ether5 internal-path-cost=10 path-cost=10 pvid=99

TO: add bridge=bridge comment=defconf frame-types=admit-all interface=ether5 internal-path-cost=10 path-cost=10 pvid=99

  1. Once you go vlans, then the bridge has lesser role to play in all settings.
    REMOVE from /interface list members.

  2. What is the purpose of setting tcp syn cookies yes???

  3. Your wireguard peer settings make no sense to me in terms of the HOME VPN.
    I am assuming that is for external users, likely yourself as admin to be able to reach the router for config or device/data purposes????? This is not the vps wireguard. WhOLAy do you have endpoints listed and why are they all different. All the clients, by the way should have the same endpoint which is your listening port at 55776.
    The peer line should look like the following on your router, take peer 3 for example:
    add allowed-address=192.168.10.64/32 interface=homevpn name=peer3 public-key="----"

  4. While discussing endpoints, You correctly assigned the VPS listening port on the peer settings to 55667, but why did you also assign it to your own device for the same wireguard. It can be any random number and its confusing for me to duplicate something that should be set on the vps device and only noted in your router peer settings. Recommend changing it.

  5. Regarding the peer settings for linode. MISSING are a few items. see below:
    add allowed-address=0.0.0.0/0 comment="Linode passthrough" endpoint-address=linodeVPS endpoint-port=55667 interface=linode name=peer7 persistent-keep-alive=30s public-key="+++"

    1. I personally do not like the nomenclature you are using for interface lists, find it very confusing.
      a. you have five vlans SO how is it you have identified 7 vlan list members????
      b. Nothing wrong with outside list, or base list
      What you need is for the LAN list to describe the LAN
      Linode is not part of a VLAN interface or LAN interface but one could consider homevpn as part of the LAN interface..... as you will want to come in to the router on that vpn, and access either the LAN or also the router.

    /interface list
    add comment=defconf name=WAN
    add comment=defconf name=LAN
    add name=BASE

    /interface list member
    add comment=defconf interface=ether1 list=WAN
    +++++++++++++++++++++++++++++++++
    add interface=BASE_VLAN list=LAN
    add interface=COMMON_VLAN list=LAN
    add interface=OUTSIDE_VLAN list=LAN
    add interface=WORK_VLAN list=LAN
    add interface=GUEST_VLAN list=LAN
    add interface=homevpn list=LAN
    +++++++++++++++++++++++++++++
    add interface=BASE_VLAN list=BASE
    add interface=homevpn list=BASE

  6. No ideas the purpose of external IP addresses is for in your IP address list............. okay I see it now, discussed later. However, there is no firewall address list entry for the EXTERNAL ADDRESSES you want users to reach via VPS, so I will make up a name called ROTATING

  7. I prefer simplicity not overkill, especially to get the core functionality desired working. If you want to get fancy and add more, do so AFTER, the config is working for traffic flow desired vice focussing on blocking. I

    With that in mind, here is my take on your firewall filter rules, put into proper order as well and with consistent themes. For example so many blocking rules and yet allow all users access to router, so we will be surgical! Do not log the drop all rules, your logs will fill up and exhaust the memory.

    /ip firewall address-list
    add address= { vlan99 address of admin #1 } list=AUTHORIZED comment=admin wired PC on v99
    add address= { vlan99 address of admin #2 } list=AUTHORIZED comment=admin wifi laptop on v99
    add address= { vlan99 address of admin #3 } list=AUTHORIZED comment=admin wifi iphone on v99
    add address={ homevpn address of admin #1 } list=AUTHORIZED comment=admin laptop
    add address={ homevpn address of admin #2} list=AUTHORIZED comment=admin iphone

    note1: if only admin is on base vlan (v99) then one rule suffices for vlan99:
    add address=192.168.1.0/24 list=AUTHORIZED

    /ip firewall filter
    add action=accept chain=input connection-state=established,related,untracked
    add action=accept chain=input protocol=icmp
    add action=drop chain=input connection-state=invalid
    add action=accept chain=input comment="Allow WireGuard" dst-port=55776 protocol=udp
    add action=accept chain=input comment="Admin access" in-interface-list=BASE scr-address-list=AUTHORIZED
    add action=accept chain=input comment="Users to services" in-interface-list=LAN
    dst-port=53,123 protocol=udp
    add action=accept chain=input comment="Users to services" in-interface-list=LAN
    dst-port=53 protocol=tcp
    add action=drop chain=input comment="drop all else"
    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    add action=fasttrack-connection chain=forward connection-state=established,related
    connection-mark=no-mark
    add action=accept chain=forward connection-state=established,related,untracked
    add action=accept chain=forward comment="Internet" in-interface-list=LAN out-interface-list=WAN
    add action=accept chain=forward comment="WG to COMMON" in-interface=homevpn log=yes log-prefix=vpnforward out-interface=COMMON_VLAN
    add action=accept chain=forward comment="VPS Traffic Outbound" in-interface-list=LAN out-interface=linode comment="allows any user to reach VPS"
    add action=accept chain=forward comment="admin to LAN" in-interface-list=BASE src-address-list=AUTHORIZED out-interface-list=LAN
    add action=accept chain=forward comment="Allow cameras (UDP)" dst-port=554 in-interface=COMMON_VLAN out-interface=OUTSIDE_VLAN protocol=udp
    add action=accept chain=forward comment="Allow cameras (TCP)" dst-port=80,443,554 in-interface=BASE_VLAN out-interface=OUTSIDE_VLAN protocol=tcp
    add action=drop chain=forward comment="drop everything else"

    **GET RID OF RAW RULES for now
    **
    10. Now for the NAT rules and what you are doing here for vpn seems overly complex and confusing so will try to simplify.
    It would appear you have one group (isolated net) that should go only to external2 (confusing as you state GUEST and WORK users, which is it. you only identify one vlan, 192.168.5.1/24 as being part of this address list?????????????????????
    You have one user that should only go to external3 and also one user that should only go to external4

    What is not clear, is if those users ARE EXEMPT from going out VPS. What happens if the destination address of their traffic matches one of the external addresses you noted????

    It would seem that all "normal traffic" all other users that is, goes out external1.**
    **
    /ip firewall nat
    add action=src-nat chain=srcnat out-interface-list=WAN src-address-list=isolated_net to-addresses=ExternalIP2
    add action=src-nat chain=srcnat out-interface-list=WAN src-address=192.168.2.32 to-addresses=ExternalIP4
    add action=src-nat chain=srcnat out-interface-list=WAN src-address=192.168.2.34 to-addresses=ExternalIP3
    +++++++++++++++++++++++++++++++++++++++++++++++++++++
    add action=src-nat chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN to-addresses=externalIP1
    add action=masquerade chain=srcnat comment="vps traffic" out-interface=linode

  8. No idea what you are trying to accomplish with mangle rules but in the line of keeping it simple, get rid of what you have.
    We need to mangle traffic heading for specific addresses and send that traffic out VPS.

/ip firewall mangle
add action=mark-connection chain=prerouting in-interface-list=LAN connection-mark=no-mark
dst-address-list=ROTATING dst-address-type=!local new-connection-mark=outgoing-vps
passthrough=yes
add action=mark-routing chain=prerouting connection-mark=outgoing-vps
new-routing-mark=to_vpn  passthrough=no

/routing table
add fib name=to_vpn

/ip route
add address=0.0.0.0/0 gateway=linode routing-table=to_vpn
  1. For your winbox allowed addresses (currently base vlan), dont forget to add specfic IP addresses of the admin coming in wireguard.
  2. the plain mac-server has no encryption, so we set it to none.
    /tool mac-server
    set allowed-interface-list=none
    /tool mac-server mac-winbox
    set allowed-interface-list=BASE
This should get you much closer........ Also there are many unanswered questions and assumptions I had to make...............  get there eventually
9

Haha didnt even get to the switch yet. LOL

You are not using standard bridge vlan filtering so unable to comment.

10

Thank you for that comprehensive analysis. Some of the concerns you raised are the result of tinkering over time (e.g., ether5, which currently has nothing connected to it but was intended to be a physical access port for the management VLAN in case of emergencies). I’ll review and fix those this weekend.

After further packet captures, the problem does seem to be that incoming packets that pass through the linode interface are never going to the bridge. I looked at a packet capture of normal return traffic for the system I’m testing with, and incoming packets went from COMMON_VLAN > bridge > ether2, which then shuffled them off to the core switch (CRS112). The VLAN tag is added before or at the transition to bridge.

You’re spot on that this is a route or VLAN tagging problem, and that’s where I’ll focus my initial troubleshooting.

11

ANAV is always a great help to get fixed. He has helped me many times.