After spending hours over multiple evenings I'm just at my wits end once again and hoping someone knows what the heck is going on.
Background: A wireguard server on port 51820 should be accessible from multiple VRFs (at same address+port!). Sounds simple, right? Not with RouterOS. It's putting every conceivable stone in your way...
Before moving forward, three constraints:
I do have existing route leaks between the VRFs which are working and they should remain untouched!
The wireguard server should be accessible over the same IP address/port from multiple VRFs
The responses from the wireguard server should have a defined, predictable source address
Based on this topic I can easily make port 51820 accessible from multiple VRFs but frustratingly wireguard generates response packets with source address 0.0.0.0 which means the source address comes from the local IP (or pref-src) of the first route lookup. Too bad this is a connected route where I couldn't even set a pref-src if I wanted, but in any case, the source address of the response packet is not necessarily the same as the destination address of the request packet. So, this doesn't satisfy #3.
Add a route from vrf-main to vrf-as via the macvlan bridge (per constraint #1 above, this route must not interfere with existing routes for anything in 10.227.79.0/24):
Add a route for the return packet in vrf-as. Critically, Use pref-src to make sure that the return packets will have the correct source address (192.0.2.222):
Everything makes sense: Host 10.227.79.104 (from vlan3 in vrf-main) is establishing a connection, it enters via ether1/br-main/vlan3/vrrp3. Then SNAT happens, as expected, the source address is translated to 172.18.1.21 and packet forwarded to xvrf-main2. It goes through the br-xvrf and eventually arrives in vrf-as via xvrf-as ... exactly as expected! Wireguard generates a response, again exactly as expected, it uses source address 192.0.2.222, again exactly as expected (due to out pref-src in the route). Packet is sent to 172.18.1.21 via vrf-as.
But then? Silence. Nothing else!
At the same time, I am not sure if it's related and expected but there are two conntrack entries:
I have exhausted my wildest imaginations: I have tried using DNAT hacks, mangle/route-mark hacks, different ports, using raw with "notrack" targets to avoid one of the conntrack entries and I even tried SNAT-ing to a non-local IP with the same trick from here.
Why is RouterOS trying sooo hard to make nothing with VRFs work as expected?
That is a really interesting case. VRFs, WireGuard sockets, conntrack, and NAT together can create some very unexpected behavior on RouterOS.
One thing I would look at is separating the service endpoint from the VRF routing domain for example using a dedicated loopback/service address and controlled route leaking rather than trying to make the same WireGuard listener behave across multiple VRFs. Locally generated WireGuard replies can behave differently from normal forwarded traffic, which makes these scenarios tricky.
This type of problem is actually one of the reasons we built TunGuard. Not because WireGuard itself is difficult, but because once you add real-world requirements like multiple sites, NAT, CGNAT, provisioning, remote access, and migrations, the operational side becomes the hard part.
TunGuard takes a different approach: userspace WireGuard engine, automated peer management, MikroTik provisioning, and an optional SSH jump host for remote access.
First, like I already wrote in the other thread(s), the tunnel workaround with MACVLAN is only really needed when you need to communicate between one side being a router's service running in main, and the other in some other VRF(s). In your test setup, the communication is between two real VRFs (not main) so normally, the usual route leaking mechanism as documented by MikroTik can be use without the dummy bridge and the extra MACVLAN.
But wait! You knew all that because you were the one who started that thread!
Having said that, if you still want to use / abuse the MACVLAN "tunnel" solution then what follow is a "solution" to be integrated with that MACVLAN stuff.
First I will start from the easiest, that requires almost no modification to the configuration with MACVLAN, but with a big constraint. The constraint is that your WG client don't need to use that specific 192.0.2.222 WG endpoint address, and has the ability to specify the endpoint address as a private (non-public) IP address, in short, the constraint is that the WG clients are in the network managed by your router, and not coming from the outside internet for example.
I just did a test configuration and nothing much to be done for it to work. In fact, starting from the test configuration of this post: Accessing router-hosted services (DNS, HTTP, etc) from a VRF - #9 by CGGXANNX a single additional SRCNAT rule was required. And I only linked to that post for the definition of the VRFs, the MACVLANs, and the addresses. The NAT or mangle rules from that post is not required (they are needed for accessing services in main only).
From the screenshot you can see that the Windows VM which has 192.168.88.254 as address is in the VRF tertiary. The WireGuard is set up to listen in the secondary interface. You can see that there aren't even any VRF route leaking between the two VRFs required. No routes from secondary to tertiary and vice versa.
WG client on Windows is working, and Windows is able to ping the address of the router inside WG (10.5.25.1).
Now if we apply that, starting from your configuration:
This #3 route above is not needed at all.
Same with this #4 route. It can be removed.
As for this SRCNAT rule at #5:
It's the only additional thing that is needed (as I wrote above), however the SRCNAT rule should be like this instead:
This is the only additional thing needed (and it replaces your #5)!
Now is the very important part: For the devices in vrf-main (those connected to the interfaces vlan3, vrrp3, etc... When they specify the remote endpoint of the WG server, they should use 172.18.1.20:51820, not 192.0.2.222:51820! And of course, it's preferable that you turn on the "Responder" checkbox in the setting of the peers of the wg-mobile interface.
That was the big constraint. The devices use the internal address inside the MACVLAN "network" as endpoint address. If the router acts as default gateway for those clients, then this work fine.
The reason why this work is:
The incoming UDP packet that tries to establish the handshake has destination 172.18.1.20:51820, it arrive at the router in the xvrf-main VRF. In this VRF, the best route for destination 172.18.1.0/24 is the connected route specifying xvrf-main2 as outgoing interface.
The packet, before going out through this interface, is handled by SRCNAT (the masquerade rule above), so its source address is changed to 172.18.1.21, the address attached to xvrf-main2.
It goes into the dummy bridges and come out at the side of the xvrf-as interface, because this interface has 172.18.1.20 as assigned address. This interface is also in vrf-as, so it's accepted and handled by WireGuard.
WireGuard will create the response packet, if you read the other thread, you'll know that this packet at the beginning has the destination address 172.18.1.21, but the source address is not yet defined.
Luckily for us, in vrf-as the best route for destination 172.18.1.21 is the connected route with xvrf-as specified as outgoing gateway. There is no ambiguity. Also, if this route is used and the packet has no source address yet, then there is only one best address to be used as source, which is 172.18.1.20 attached to xvrf-as.
As a result, the response packet will have source 172.18.1.20:51820 and destination address 172.18.1.21 and it will be put into the dummy bridge.
The packet will then arrive at the interface xvrf-main2, because this interface as 172.18.1.21 assigned. Here connection tracking will be able to find the matching pair 172.18.1.20:51820 and 172.18.1.21:xxxx previously marked as SRCNAT-ed. It will be able to undo the NAT-ing and restore the original address and port of the WG client in place of 172.18.1.21:xxxx. The packet will be routed back to the client.
So this pretty straightforward. We don't even need any VRF route leaking, because the routing between VRF is through the tunnel.
The reason we needed the SRCNAT masquerade rule is because it helps selecting the correct route for the answer packet at the vrf-as side. If you don't like that SRCNAT rule, you can remove it, however, in the routing table of vrf-as you should assign routes for destination address ranges of the WG client (the addresses in xvrf-main) with gateway=172.18.1.21@vrf-as, so that the response packet can be sent through the MACVLAN tunnel.
In the next post I will outline the few modifications needed, so that 192.0.2.222:51820 can be used to specify the endpoint address in the WG client, instead of 172.18.1.20:51820.
Now, what if the clients can't use 172.18.1.20 as endpoint, and absolutely have to use 192.0.2.222 and we want to stick to the MACVLAN tunnel workaround?
A first thought would be to simply add a DSTNAT rule, that transforms the destination 192.0.2.222:51820 to 172.18.1.20:51820. So the packets will have both DSTNAT and SRCNAT applied. Having both is normally not a problem, and it works like that elsewhere too. However not in this configuration. Probably because both of the two ends of the MACVLAN tunnel share the same connection tracking table that apparently has no separation between VRFs, having a connection with both DSTNAT and SRCNAT coming to the other end of the table, where it should be tracked again as another connection, somehow confuse conntrack and you get 3 conntrack entries appearing at the sametime, with some permutation of the source/destnation/reply source/reply destination, where two will quickly timeout and the remaining entry is a wrong one. As a result conntrack is unable to reverse the NAT-ing, and it doesn't work.
Which means that we can unfortunately not use the trivial solution of DSTNAT. To be able to use 192.0.2.222 as endpoint destination address, the other end must answer for that destination, as well as use that address as source for the return packet. And from other thread, we new that using pref-src is not effective.
So, the solution is to assign 192.0.2.222 as the main address of xvrf-as. There is no problem with assigning the same 192.0.2.222 address to multiple interfaces in the same VRF, so the other interface that currently has 192.0.2.222 assigned needs no modification. What we have to make sure, however, is that the two connected routes generated by the two /ip address assignments do not have the same dst-address, otherwise we'll create an ECMP group and that is problematic.
And we want WG to really use 192.0.2.222 as source address, and not sometime that, sometime 172.18.1.20. Which means the 192.0.2.222 address must really replace 172.18.1.20 in the /ip address assignment for xvrf-as, not existing side-by-side.
But we also don't want to have to change the other places that currently specify 172.18.1.20 as gateway address of the static routes. Which means we should proceed in the following order:
First, we don't want to lose the current connected route with dst-address=172.18.1.0/24 gateway=xvrf-as@vrf-as, this route normally disappears when we remove the 172.18.1.20/24/ip address assignment. So we'll first add this static connected route:
We also don't want to have to modify the routes in other tables that currently use something like gateway=172.18.1.20@vrf-xxxxx, so we'll make sure the all the xvrf-xxxxx interfaces will reach xvrf-as for the destination 172.18.1.20. For that we'll add a static ARP entry for each of the other xvrf-xxxxx interfaces:
We can now add the 192.0.2.222 address to xvrf-as, but we'll add that as a /32 address, so that the connected route it produces does not interfere with the routes created by the assignment of the real original interface that also has 192.0.2.222:
Of course, now we have to modify the masquerade rule from the previous post, so that it now uses the condition dst-address=192.0.2.222 instead of dst-address=172.18.1.20:
That's all the modifications needed. Devices in vrf-main should be able to specify 192.0.2.222:51820 as endpoint for the WG peer setting.
Similar to what was said in the previous post. If you don't like to apply SRCNAT (the masquerade rule) and want to keep the source addresses (real addresses of the clients) then you can remove the masquerade rule if you add routes for all the needed destinations with gateway=172.18.1.21@vrf-as routing-table=vrf-as.
Once again, thank you so much for taking so much time and the thorough response.
Let me just first clarify the background: I would like the VPN (wireguard) server to be available from everywhere, no matter if internal or external networks, internet under 192.0.2.222:51820. To add a bit complication, the actual IP of the router is actually 192.0.2.210. However, there is/will be a backup router 192.0.2.211 which operates with VRRP on 192.0.2.222.
Indeed I studied the old thread again and this is how I got things working before I started with all of this VRF mess.
Having said that, it's not that I want to use/abuse the MACVLAN solution for this, but I was thinking it's the only way it can work.
I actually got something very similar to your first suggestion working already, even without SNAT but the issue is that it's just not the public IP then.
Unfortunately with your second solution, I am still not able to make it work. In fact, I run into the same issue from the original post In this thread: I see the response packet going out of vrf-as but nothing more:
I am aware that with this setup I have the ECMP route that you mentioned to avoid because 192.0.2.222 is also /32 on the VRRP interface. However, I removed it and it made no change. I don't think it's related to this.
So I think there must be something else wrong.
Now, if I think about it, I believe your second proposal is pretty much the same as the proposal in my initial post. The only difference is that your solution avoids the pref-src issue.
I am not sure which other thread you are referring to (this one?) but I think this doesnt apply here: As explained in that linked thread, there are two route lookups, an initial one (which sets pref-src) and a route adjustment step. The latter one is used for policy routing. In my original question I had the issue that the route was picked at the route adjustment stage (due to complex routing scenarios) but at this point the source address was already selected.
Now, #4 in original post of this thread does generate a route which is indeed consulted in the very initial routing step which is used to set the source address. You can see that it works, in original post wireguard is indeed answering from 192.0.2.222, as expected.
I tried removing that route and/or pref-src and it stopped working, so I'm pretty sure that's doing its job.
The issue that's left in both our cases is the same: Wireguard's response packet from 192.0.2.222 to 172.18.1.21 is stuck somewhere and does not proceed beyond vrf-as in both cases.
Have you tried out your proposal and did it work? If so, there must still be something wrong with my config.
If not, I am afraid hitting yet another RouterOS bug, possibly what you were saying here already:
To avoid unnecessary guesswork, I am sharing attached my full config for reference.
Yes, that should not be the problem with the setup above.
I've not had time yes to fully read your post and study your posted configuration, but about this question then yes, it was tested on the Frankenstein CHR setup made up for the MACVLAN tunnel adventure from previous threads. The setup has:
WAN (DHCP client) on ether1, which is in the wan VRF, it needs this for the router to be able to go to the internet to check for update, sync time, etc...
The is secondary VRF which has ether3 and the 172.20.80.2/24 subnet. This is where I also have the PC that manages the CHR.
The main LAN (with DHCP server) is in the tertiary VRF, has the 192.168.88.1/24 subnet on ether2.
Router services like DNS, WinBox, WebFig are in main, so it uses the trick from here to be able to reach them from other VRFs (like from LAN in tertiary, or from my PC 172.20.80.1 in secondary).
There is PPPoE server which put the clients in the tertiary VRF. These client needs to access the internet, so there are scripts that add the return route to them from wan. PPPoE clients are put in the 192.168.99.0/24 subnet.
Which means from my PC at 172.20.80.1 I can make WG connection to the router at 172.20.80.2 (on ether3) being in the secondary VRF.
This 172.20.80.2 plays the role of your 192.0.2.222 address, and is reachable from outside of the router, if plugged to the right port(s).
The interface wg1 itself as well as the subnet inside the tunnel (10.5.25.1/24), is in the main VRF, but this is not important for the test, you can move the wg1interface to other VRFs if you want. What important is the vrf=secondary parameter on the interface definition above, because that determines the VRF of the WireGuard UDP wrapper packets.
The vrf-link bridge is created with 4 MACVLAN interfaces, vrf-link-main, vrf-link-secondary, vrf-link-tertiary, vrf-link-wan. They originally have the addresses 172.18.0.20-23 assigned. However with the change made for WireGuard as described in the post above, 172.18.0.21 has been removed from vrf-link-secondary, replaced by the address that should be used for accessing WG, which is 172.20.80.2 (same as address on ether3). 172.18.0.21 now only exists as ARP response that map to the MAC address of vrf-link-secondary (see ARP table).
With the setup, a Win11 client which is in the LAN network (tertiary VRF) with address (obtained via DHCP) 192.168.88.254 can establish a WG connection to 172.20.80.2 (which is in secondary), and get a working tunnel. I can ping 10.5.25.1 from the Win11 machine with address inside tunnel 10.5.25.2. On the router, the peer show correct last handshake and current endpoint information.
Thank you so much for sharing! I put this on a test CHR and only changed minor things (disabled DHCP server, replaced wireguard keys, 192.168.88 --> 10.227.79, 10.5.25 --> 10.227.33 to match my ranges easily.
Indeed, this works! Weird. So I need to carefully dissect what the difference is to my other config where I am pretty sure I did exactly the same!
One thing that prevents me from doing a direct comparison is something super weird though: As you can see the only filters I added are udp plus the port. So I expect to see the packet to arrive at ether2 first. But you can see from the screenshot that only vrf-* devices are seen in the packet sniffer. How can this be?
By contrast, in my sniffer outputs from above, the packet first arrives in ether1, then br-main, then vlan3, then vrrp3. Sure, your config doesnt have a bridge, VLANs and VRRP but still the first entry must be ether2.
Another thing that I don't understand is that in my original packet capture, the source NAT is clearly visible after the packet is received from physical interface (num: 5).
However, in the screenshot above the source address is still the original client IP (10.227.79.43).
And the more I stare at it, the less it makes sense to me: Why even vrf-link-main shows up?? This is the link to main table, should have nothing to do with this. And vrf-link-wan?? And with all of this, how does the actual wireguard tunnel seem to work? Like I can ping the tunnel endpoint AND handshake AND data with the client looks good.
Ha ha ah, wow, thank you for the screenshot with the Packet Sniffer. That bothered me so I decided to removed all the unnecessary stuffs from the Frankenstein config, and reduced it to this, that "works":
Let's look at the export in details: Gone are the WAN stuffs, as well as the PPPoE stuffs, and all the DSTNAT and manging that make the router's services in main accessible. What left are:
On ether2 LAN 192.168.88.1/24 with DHCP server in tertiary VRF.
A secondary VRF where WireGuard wg1 is listening, as well as ether3 with main IP address 172.20.80.2/24 (my PC at 172.20.80.1 is connected to this).
Subnet inside tunnel is 10.5.25.1/24 and is in main. Remote peer has 10.5.25.2.
Win11 with WG client connected to ether2 getting address 192.168.88.254 via DHCP (is in tertiary VRF). WG client is listening on port 13131.
Win11 WG client connects to endpoint address 172.20.80.2 which is part of the secondary VRF on the CHR.
Now for the MACVLAN tunnel part:
Dummy bridge vrf-link with 3 MACVLAN vrf-main, vrf-secondary, vrf-tertiary, with address assignment:
With vrf-link-secondary getting a /32 address assignment being the same as the address on ether3, 172.20.80.2.
Because vrf-link-secondary only automatically gets a dynamic /32 route from the address assignment, we also manually add the route to 172.18.0.0/24 with the MACVLAN interface as gateway. The purpose is so that when we have the a return address 172.18.0.22 it's directed to the MACVLAN tunnel, toward the other MACVLAN on VRF tertiary:
If that route is present and enabled everything works, if that route is disabled, the handshake hangs and never completes!
Now let's look at the original config in the older post that also worked. Actually this route in main to reach 192.168.88.0/24 also existed there, just not explicit, but as combination of these two routes:
which directs the packets (because 192.168.88.x is satisfied by 0.0.0.0/0 ) to the gateway 172.18.0.23, which is the address on vrf-link-wan of VRF wan. The packets will be received on that interface, and in VRF wan we have this route:
that knows where to put the packets for destination 192.168.88.0/24.
So the response packets being sent to 192.168.88.254 were pushed through vrf-link-main and vrf-link-wan as a result, and you saw them your the sniffer screenshot.
In the new reduced configuration that detour route has been replaced by:
We have the blue expected connection entry, with correct NAT information (Reply Dst. Address 172.18.0.22 from masquerade). And we would think that when the packets go out of the tunnel on the vrf-link-secondary side, another conntrack entry would have been created between 172.18.0.22 and 172.20.80.2. But we don't see that.
Instead there is a conntrack entry as pointed by the orange arrow, that has source 172.20.80.2:51820 (ok, understandable if this is catching the return traffic when the WG side of the router response), but instead of the expected destination address 172.18.0.22 (because that was what WG see incoming, 172.18.0.22 is also listed in the Current Client Enpoint field), we have the original source address 192.168.88.254:13131 as destination!!!
And what is even more strange, is that this conntrack entry has a SRCNAT applied to the 172.20.80.2:51820 source:
So how did this connection that absolutely requires the route in main come out of existence??? I have no idea how to explain this at all. You've been correct all the time with this take:
Wow that's insane!! Thank you so much for going through this with me, there's no way I could have caught that.
I reproduced your new config and even better, my config from the original post in this thread starts to work once I add the following route to my main table:
In other words, assigning the /32 directly to the vrf-link interface as you suggested works. However, this in my original post still works and is in my opinion slightly more elegant:
Adding the static route in main is not desired because I made peace abandoning my main VRF due to all the issues (route leaking etc). Now it seems I again need to keep static return routes for all possible wireguard client networks in main
I do not fully grasp what's happening here but to me it seems it's not wireguard: First, as I posted above, the log shows that wireguard responds. Second, the return packet can be seen in the sniffer.
Instead, it seems the return packet has to go through "main" in order for SNAT/masquerde back translation to work. In other words, there seems to be some weird edge case where NAT back translation with a VRF fails.
This smells like yet another bug and I think I am going to file another one... adding to my big stash of (still open and unanswered) bug reports...
.
