TL;DR: After the public IP address of my RB5009 changes, Wireguard tunnels behind the router do not work anymore since they are still source NATed to the old public IP address.
I’m running a RB5009 with ROS 7.18.1 at home, connected to an ISP which offers a public IPv4 address via DHCP. One of my Proxmox VE hosts is connected directly to the router at ether8. This PVE host runs an OPNsense and CHR VM, both VMs are connected with their WAN interfaces to vmbr0, which is a bridge interface directly connected to the ethernet interface of the PVE host itself. Both VMs get a private IP address from the RB5009 through DHCP, which is their default gateway. Both VMs have a wireguard interface configured with a persistent keepalive of 10s, which connects to a VM at a Hetzner datacenter. While this setup works just fine, whenever I get a new public IP address from my ISP, the wireguard tunnels (and only those) do not work anymore.
Starting the packet sniffer at the WAN interface of the RB5009, it shows that connections for the wireguard tunnels are still source NATed to the old IP public address. As soon as I reset the corresponding connections in the tracking table or set the udp-stream-timeout lower than 10s (i.e. lower than the persistent keepalive of the wireguard tunnels), the tunnels come up again and everything works just fine. Since I’m using a masquerade NAT rule for outgoing connections at the WAN interface, a change of the public IP should reset the connection tracking entries, but this seems not to be the case here.
Thank you for the script, however it doesn’t really answer my question. I guess I should be more specific: Is this expected behaviour or a possible bug?
It works as designed. To elaborate a bit, there is a difference between NAT action=masquerade and action=src-nat. In case masquerade is used, the conntrack entries are purged automatically. For src-nat they are not. In this case ou can clear them using a script as suggested. This doesn’t affect connections behaving in a “more usual” manner, where some sort of keepalive and ephemeral ports are used, because when the source port is changed, a new (and now correct) conntrack entry is created.
If you go with the script approach, I would probably filter on the ip/protocol/port that is used specifically for your wg connections. Also, a variable $bound is passed to the lease script that informs you whether there is a new binding or a new IP address. Using this as condition in your script, you won’t purge the connections unless they have to be.
To elaborate a bit, there is a difference between NAT action=masquerade and action=src-nat. In case masquerade is used, the conntrack entries are purged automatically. For src-nat they are not.
Well, exactly that’s the reason I think it’s a bug, because there is no src-nat rule in place. I’m using masquerade only:
/ip/firewall/nat/print
Flags: X - disabled, I - invalid; D - dynamic
0 ;;; masquerade
chain=srcnat action=masquerade out-interface-list=wan log=no log-prefix=""
In may experience the masquerade part of conntrack works correctly and purges the appropriate entries.
I’m not trying to be dismissive, but I would assume that there is something going on other than (or besides) what you are describing. Maybe the wg tunnels are not actually initiated on the VMs? How exactly are you getting a new address - is it during renew? The list could go on, but I’m only guessing. Please provide more information.
I’m not trying to be dismissive, but I would assume that there is something going on other than (or besides) what you are describing. Maybe the wg tunnels are not actually initiated on the VMs? How exactly are you getting a new address - is it during renew? The list could go on, but I’m only guessing. Please provide more information.
Since there is no way to initiate the wg connection from the outside (there are no DNAT rules configured), the VMs initiate the connection. A new IP address is usually assigned during maintenance windows of my ISP, so the renew fails and the router gets a new lease with a new IP address from a different DHCP server.
If I can provide any more information or config, please let me know.
There you go. I left as much as possible in there, but removed some sections, namely WiFi (for CAPsMAN), DHCP static leases and firewall filter rules. The router itself has a wg interface as well, which connects to the exact same peer as the VMs. This tunnel works without issues after a new public IP address is assigned, which is another indication that the problem is related to the SNAT connection tracking table.
Looked it up amongst the kernel patches. For some time period the behavior was indeed to purge the connections if the address changed (pr added/removed). This was reverted, and it is not the behavior any more. The purging of the entries only happens on link down. (At least for the 5.6 that Mikrotik bases v7 on. There were some other helper-related fixes later on, such as in 5.14, but even if those were backported, they would not really help you.)
Through a quick google search I’ve found several pieces of software impacted by this, and the foolproof solution is to purge the entries on triggers by the dhpcd, so basically on-lease scripts.
Wow, looks like you dug really deep, thank you very much! I will open a support ticket to get an official confirmation and so it can be removed from the documentation.
Since you’re seeing something that seems to completely contradict the documentation, I actually prototyped the issue fully. (I drank too much coffee and have too much time on my hands.)
In this case of course I control the DHCP server.
When I force the client to renew (and of course it’s NAK’d, and instead gets a different address), the conntrack entry is not cleared, and its reply-dst is unchanged. Of course it can’t usefully forward traffic, however as long as it’s receiving packets from the side behind the NAT, it keeps renewing the timeout.
If I force the output interface down, the conntrack entry is purged. (Regardless of whether there is any IP change later when it’s brought up again.)
src-nat is executed after firewall filter … so dropping invalid traffic with firewall filter rules won’t work. Raw filters only have “prerouting” and “output” chains, but something in “postrouting” would be needed.