Wireguard VRF Issues

Hi All,

Apologies as I am new to Mikrotik. I am trying to solve a scenario where I am required to contain an entire segment of a network into a single VRF including the WireGuard interface/operations. I am not having any luck.

Here is the breakdown -

  • All interfaces associated with this segment of the network are in a single vrf
  • I have tried this without a VRF and it works fine
  • As soon as put all interfaces (wg0, vlan100, ether2) in the VRF it stops working and the Mikrotik starts sending ICMP Destination Unreachable (Port Unreachable) to the initiator

Other things -

  • I can route all the way through (no routing issues)
  • I have tried this on 7.20.4 and 7.21beta8 with the same result
  • Testing this using a hEX but the plan is to do this config on a CCR2116-12G-4S+

Here is a stripped down config of what I am using -

/interface bridge add name=bridge1 protocol-mode=none vlan-filtering=yes
/interface bridge port add bridge=bridge1 hw=no interface=ether2
/interface bridge vlan add bridge=bridge1 tagged=bridge1,ether2 vlan-ids=100
/interface vlan add interface=bridge1 name=vlan100 vlan-id=100

/interface wireguard add listen-port=13231 mtu=1420 name=wg0
/interface wireguard peers add allowed-address= client-allowed-address= interface=wg0 name=peer5 public-key=""

/ip vrf add interfaces=vlan100,wg0 name=wg_vrf

/ip address add address=100.100.100.1 interface=wg0 network=100.100.100.1
/ip address add address=172.30.3.2/30 interface=vlan100 network=172.30.3.0

/ip route add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=172.30.3.1@wg_vrf routing-table=wg_vrf scope=30 target-scope=10 vrf-interface=wg_vrf

/ip route add disabled=no distance=1 dst-address=100.100.100.0/24 gateway=wg0 routing-table=wg_vrf vrf-interface=wg_vrf

Any advice would be much appreciated.

Thank you

Rule #12 seemingly applies, JFYI:

Your incomplete configuration except hat a lot of nonsenses:

  • VLAN ID 1303 is tagged for bridge1 (and ether2) but there is no VLAN interface defined to make use of it.

  • You have a VLAN interface defined for VLAN ID 100, but that VLAN ID is associated with no ports.

  • You appear to use only IPv4 but the WG peers has only IPv6 address in allowed-address.

  • You added ether2 to the VRF instance, although ether2 is a slave port (member of the bridge).

  • What is this???

A very small portion of the internet? :wink:
https://www.calculator.net/ip-subnet-calculator.html?cclass=any&csubnet=24&cip=0.0.0.0&ctype=ipv4&x=Calculate

*) wireguard - added VRF option (CLI only) (additional fixes);

does anyone tested that?
i did try, no luck

Worked for me. I only did a quick test, so there may be things lurking in the shadows.

thanks lurker888.

am i doinf right?

No, the vrf property of the WireGuard interface is not made for what you intend to have in your configuration above. You should clear its current value.

You set that property if you want the outer UDP packets, that transport the encrypted tunnel, to be sent and received in a particular VRF.

It's not used to put the inside tunnel in a VRF, for this purpose you use the traditional way of including wg1 in the interfaces property of the VRF.

i think i understood what u are saying.

You set that property if you want the outer UDP packets, that transport the encrypted tunnel, to be sent and received in a particular VRF.

Are u willing to give me just basic e.g. so i can play around

Thank you for your reply. I changed the original output to make it less convoluted but ended up just making it worse. I’ve updated the original post.

Thanks for trying it out. I will try it on another device and see if I have the same issue.

one more @CGGXANNX

if i enable VRF (CLI only) , i'm losing the link, that why im playing with /rules

This is what im gettign:

@wildrodeo9142
More a question than an answer, but i can see in the snippet you posted these interfaces:
bridge1
ether2
vlan100
wg0
and a routing table (for the vrf):
wg_vrf

What is the vrf-interface wg_vrf ? (wg_vrf is a vrf and a routing table, not an interface from what I understand).

This describes an example of vrf for the tunnel (underlay, encrypted) case: Feature Request: Wireguard over VRF - #29 by lurker888, the post under it by divB gives a bit of example code.

For the vrf interface itself being in the VRF, an example can be found here: Feature Request: Wireguard over VRF - #35 by lurker888

Well when VRF in the code settles down to some stability, not sure I even want to explore this.
What would work for me is a zoom tutorial with lurker and CGG providing a walk through of all the vrf and wireguard possibilities in/out/and twisted around. Would even make small donation for the teaching session!!

1 Like

The link doesn't contain exactly what I'm looking for, but it does have a lot of useful information.

this is frim divB

/interface/wireguard/set wg-bg1-wwan vrf=wwan

/interface/wireguard/set wg-bg2-wwan vrf=wwan

makes lot of sense

Yes, @lurker888 above has given one example, where all the traffic is to be routed through WG, but the outer WG packets themself, of course need to use the regular WAN.

I have another example for one of the scenarios where this new ability to specify the VRF for WG is very useful, it is when you have multiple WAN, with one let's say WAN1 acting as the main gateway because it has high download bandwidth (1Gbps down, 100Mbps up), but is behind CGNAT with no reachable public IP addresses (and the upload is a bit low), and you have a 2nd WAN with symmetric 250Mbps up/down and public IP address. In this case, this 2nd WAN is ideal to be your WireGuard endpoint to your router (and to reach the LAN network behind it), because of the public IP address and higher upload capacity.

But because WAN1 is our default gateway (and maybe WAN2 is setup as failover) in the main routing table, any attempt to use WG solely on WAN2 would involve another routing table where WAN2 is used as the default gateway.

There were many threads and posts on this forum such as this one Wireguard in 2nd WAN - RouterOS / General - MikroTik community forum or this one WireGuard Multi-WAN Policy Routing - RouterOS / General - MikroTik community forum where people found out that convoluted workarounds (usually DSTNAT and/or SRCNAT together with mangle rules need to be involved) are required if we want WireGuard to answer using any routing table other than main. The added support for VRF with WG in 7.21 will greatly simplify this scenario.

In this example we have two WAN, WAN1 is DHCP client on ether1 and WAN2 is PPPoE client on ether2. Both have dynamic IP addresses, but the one from DHCP is a CGNAT address. The default route from WAN1 is added to main with distance 1, and the default route for WAN2 is added with distance 2.

/ip dhcp-client
add default-route-tables=main interface=ether1

/interface pppoe-client
add add-default-route=yes default-route-distance=2 interface=ether2 \
   name=pppoe-out1 user=... password=....

# configure interface list, input firewall, masquerade etc... if needed

If we now add a WG interface and peers:

/interface wireguard
add listen-port=21204 mtu=1420 name=wg1

# configure address, subnet, and peers
# /ip address add interface=wg1 ...
# /interface wireguard peers add interface=wg1 ...

If we now take our phone and use the WG app on the phone to try to connect to the endpoint specified with the IP address of pppoe-out1 (the address of WAN2), we'll see that the handshake never complete, the app shows Tx bytes being sent, but 0 Rx bytes are received. The reason for this has been explained very well by @lurker888 in the 2 threads I linked above. And you could also employ the workarounds from these threads if you want. But with the VRF support in 7.21 you can do this instead:

  • Create a VRF instance dialup and add pppoe-out1 to it:

    /ip vrf
    add interfaces=pppoe-out1 name=dialup
    

    We can now see the dynamically added FIB table dialup, and the default route with pppoe-out1 as gateway has been moved to this table, and is no longer in main.

  • Next we set the vrf property of the wg1 WireGuard instance to dialup. I don't know if it's intended by MikroTik or because it's beta, but in my tests I had to disable the wg1 interface first, then set the property, then re-enable wg1 for it to fully work, just setting the property live wasn't enough.

    /interface wireguard disable wg1
    /interface wireguard set wg1 vrf=dialup
    /interface wireguard enable wg1
    
  • There is no need for extra NAT and mangle rules!

Once this is done, you'll can try with the WG app again and see that connecting to the endpoint using the public IP address of WAN2 now works correctly, as intended.

Moving pppoe-out1 to another VRF however has another important effect. The main VRF now no longer has access to WAN2. In the main routing table there is no more route using pppoe-out1 as gateway, and WAN2 is currently not working as failover anymore.

To have WAN2 as failover again, first we need to "leak" a default route using pppoe-out1 to the main table, with higher distance (because it's failover):

/ip route
add distance=5 dst-address=0.0.0.0/0 gateway=pppoe-out1@dialup routing-table=main

However, if we now unplug the cable from ether1, or disable the DHCP client on ether1 (disable WAN1), so that WAN2 becomes the default route in the main VRF / routing table, we'll notice that while the router itself is still able to go to the internet normally, the clients in our LAN (for example a 192.168.88.6 client on bridge) no longer has internet access, everything times out.

The reason is because although the packets from those clients can successfully be sent out (through the pppoe-out1 gateway) to the internet, any response packets when they arrive to the router, are in the dialup VRF. And in this VRF there are no proper routes to the LAN subnets. There is only one single default route with 0.0.0.0/0 as destination and pppoe-out1 as gateway.

For these packets to be able to be forwarded to the clients, routes must be added to the dialup routing table. We need to "leak" routes for all LAN destinations that we want to be able to use pppoe-out1 as gateway to the internet, the routes would be copies of the ones in main, with @main specified, for example:

/ip route 
add dst-address=192.168.88.0/24 gateway=bridge@main routing-table=dialup 
add dst-address=10.1.20.0/24 gateway=vlan20@main routing-table=dialup 
add dst-address=172.18.4.0/22 gateway=192.168.88.15@main routing-table=dialup 

With that the client 192.168.88.6 would have internet connection again, because the return packets know the proper route to use (via bridge interface in VRF main).

CGGXANNX Thanks a lot