FreeNAS Box with a couple of jails running various jails (Static DHCP records 192.168.1.3-8)
What I would like to happen is to have the jail at 192.168.1.5 to route all internet via the NordVPN IKEv2 connection. All other devices connecting to the network should continue connecting to the internet directly.
I have followed the NordVPN guide to set up the VPN client and the connection seems to be established, but I am still not able to get the jail to route via the VPN using connection marks and a mangle rule.
Is this somehow achievable?
Do I need to run the hEX as a router (I had the FreeNAS box double-nat’ed and wanted to move away from this setup). Unfortunately I am unable to use the hEX as my only router, due to some magic that Telekom has where there is no username or password for the connection
Post the export of the current setup, there must be a minor mistake in it. See my automatic signature below regarding how to remove sensitive information without breaking consistence of the configuration.
The connection-mark and src-address-list on the /ip ipsec mode-config row are used to dynamically create an action=src-nat rule in /ip firewall nat once the IPsec session is established. The rule causes all connections whose initial packet matches these match conditions to get src-nated to the address obtained via the mode-config, so that it would match to the IPsec policy. So packets whose source address before NAT matches the src-address-list, and whose connection-mark matches the connection-mark value of the rule.
Your mangle rule assigning the connection-mark value, action=mark-connection chain=prerouting connection-mark=no-mark dst-address=192.168.133.5 new-connection-mark=via-NordVPN passthrough=yes, matches on dst-address rather than src-address. So this rule only marks the connections of that jail as late as when handling the first response packets of these connections, which is too late.
So change dst-address to src-address in the mangle rule, and remove src-address-list from the mode-config row. If you want the .5 jail to be able to talk to local devices, you have to add a match condition to the mangle rule which will prevent if from matching on connections to LAN addresses, such as dst-address=!192.168.0.0/16.
What does /ip firewall mangle print stats show? When you stop a traceroute for a minute and then start it again and print the stats again, can you see the packet count to increase for the mangle rule?
If not, when running traceroute to x.x.x.x in the jail, make the command line window on the Mikrotik as wide as your screen allows and run /tool sniffer quick ip-address=x.x.x.x to see whether the source address is indeed 192.168.133.5.
I can see a minute amount of traffic captured (18 packets), and only when the jail is restarted, but none of the subsequent traffic (pings, traceroutes, etc) seems to get captured.
While pinging google, the sniffer captured the following:
That’s normal, as the rule only matches on the first packet of each connection, because only the first one not only matches the src-address and dst-address but also has no connection-mark assigned yet.
This is the part that suggests what to look for, see below.
This clearly shows the source address is correct.
So the answer is here: /ip address
add address=192.168.133.2/24 interface=ether2 network=192.168.133.0
…
/ip route
add distance=1 gateway=192.168.133.1
In another words, the Mikrotik is a default gateway for the jail, but Mikrotik’s own default gateway is in the same subnet like the jail’s IP address. So when the Mikrotik forwards a packet from the jail, it finds out that it has to forward it out the same interface through which it has received it, so it sends an ICMP message “a better router than is available for this destination in the same subnet” to the jail, which accepts that message and starts sending subsequent packets towards that destination directly to the 192.168.133.1. You can see those packets in the capture because the Mikrotik acts as a bridge for these packets, but since it does not route them any more, it applies no IP processing at all on them.
So there are two ways to deal with this:
to split the networks, i.e. to use a dedicated subnet as the WAN one of the Mikrotik
to implement policy routing for the traffic from that jail, which will use another gateway for this traffic, preventing this from happening.
The latter one can also act as a prevention of traffic leakage through the normal path if the tunnel is down (sometimes called a “killswitch”).
To implement this, you would add a bridge interface with no member ports: /interface bridge add name=br-blackhole protocol-mode=none
Next, you would add a default route via that interface into a distinct routing table: /ip route add routing-mark=via-NordVPN gateway=br-blackhole
And last, you would add the following mangle rule after (below) the action=mark-connection one: connection-mark=via-NordVPN in-interface=bridge action=mark-routing routing-mark=via-NordVPN
One more thing that is wrong is that the Mikrotik’s own IP address (192.168.133.2) is attached to ether2 rather than to the acual IP interface, which is bridge. This wrong setup seems to be allowed to permit some migration scenarios, but it causes some weird effects. It is definitely not related to your issue, but nevertheless worth fixing.
I decided to go with the policy based approach you described, as I would really prefer to keep the devices on the same subnet.
I seem to have followed the instructions quite closely, but still not having any luck. The jail still has access to the internet (tested by manually disabling the ipsec tunnel) and traceroute still shows the first hop as
I also looked at the firewall stats, and this is still behaving exactly the same as it was before, except now the mark routing rule also showing traffic. Each time the jail connects to the network, there are 4 packets captured.
ip firewall mangle print stats
Flags: X - disabled, I - invalid, D - dynamic
# CHAIN ACTION BYTES PACKETS
0 prerouting mark-connection 1 230 8
1 prerouting mark-routing 1 230 8
Ok now that I have updated the Mikrotik settings and set it to act as a router in Quickset as well as updating the DHCP to issue leases with 192.168.133.2 as the gateway, there is definitely more activity on the mangle rules.
Unfortunately, I now have no access to the outside world from the 192.168.133.5 jail. The gateway is correctly set via DHCP to 192.168.133.2, and all other device on the network are working fine. Clearly the black hole is working, but I am missing some config to direct the traffic through the tunnel :shrug:
Once you change even a single bit in the configuration using any other tool than Quickset, never use Quickset again.
Sorry, my mistake, your configuration is indeed unusual The jail → internet direction is fine, the mistake is in the internet → jail direction. The action=mark-routing rule assigns the routing-mark also to the internet->jail packets, which is wrong. Normally, the way to avoid this is to set in-interface=bridge1, but that fails in your case because the in-interface of the internet->jail packets is also bridge1. So remove this match condition from that rule, and instead of it, use dst-address=!192.168.133.0/24 like in the action=mark-connection rule instead:
The jail → internet direction is fine, the mistake is in the internet → jail direction. The action=mark-routing rule assigns the routing-mark also to the internet->jail packets, which is wrong. Normally, the way to avoid this is to set in-interface=bridge1, but that fails in your case because the in-interface of the internet->jail packets is also bridge1. So remove this match condition from that rule, and instead of it, use dst-address=!192.168.133.0/24 like in the action=mark-connection rule instead: