I’m trying to set up a HAP-AX2 running ROS 7.20 for a VRF Lite setup. I need a VPN tunnel with some IPs routed down it for trusted users but don’t want to route via the tunnel for guest users - hence wanting two route tables.
I’ve created a ‘bridge-guest’, given it an IP and assigned one of the ethernet ports to it. Set up DHCP for both ‘bridge’ and ‘bridge-guest’ and everything works as expected (without the VRF isolation yet).
Create a VRF-Guest and assign ‘bridge-guest’ to it and I can ping the router as long as I don’t activate ‘VLAN Filtering’ on ‘bridge-guest’. I would like to understand why but it’s not a show stopper as I don’t need VLANs on either network.
Problem is with NAT. I see connections being NAT’d but nothing gets back through and I think I can see post-NAT traffic coming back out the WAN interface presumably following the default route. I’ve seen a lot of mention of mangle rules and tried setting up a pair to mark the connection and then route-mark the returning packets but I don’t see anything hitting the second rule and I’m not even sure if any of it is necessary of I'm barking up the wrong tree. Does NAT require help putting the returning traffic back on the correct VRF? If so, is this the right way to do it?
Edit: My 'trusted' LAN and external WAN port (currently connected within another working network for testing) are both in the main VRF and my guest network is in the VRF-Guest.
Edit2: Add the mangle rules that I tried back in to the config.
Corrected the mangle rules and I can now get pings to work fine. DNS lookups are also fine...but web traffic is just painfully slow (like slower than dial-up speed - got a download running and it's clocking in at 903 bytes per second!). I thought it might be an MTU issue but I'm getting a full 1500 MTU and can send df pings up to 1472 as expected. CPU usage is 1% or less. Passthrough on or off for the first mangle rule doesn't seem to make any difference either way.
You’ve actually figured out the main thing by yourself. Congrats!
Your current issue is that the fasttrack feature is basically incompatible with routing based on routing marks.
To verify that this is your problem, simply disable fasttrack support entirely by going into IP→Settings and disableing fastpath support there.
If this fixes your problem, the best fix is to re-enable fastpath, but exempt marked traffic from being fasttracked. This is done by modifying your action=fasttrack-connection rule in your firewall filter/forward chain and adding the match criterion connection-mark=no-mark to it. This way fasttrack will still apply to your normal (non-guest) traffic, but guest traffic will be exempt from it and will still be correctly routed.
The only thing to be aware of is that the exemption will only apply to newly created connections (or to put it another way: once a connection is fasttracked it stays that way.)
Fantastic, cheers Lurker that's absolutely nailed it. Not sure which setting under settings since "Allow Fast Path" is ticked, "IPv4 Fast Path Active" is not ticked (and isn't toggleable) and "IPv4 Fasttract Active" is ticked but toggleable.
Adding "connection-mark=no" to the fasttrack firewall rule brings the speed back up to normal - at least I can top out my internet connection.
When you say "Your current issue is that the fasttrack feature is basically incompatible with routing based on routing marks." it sort of implies there might be a better way of doing this without routing marks...or am I just reading too much into that?
My instructions about deactivating fastpath were accurate but terse. The two indicators that you can’t change are just that: read-only. The reason for them is that they give you feedback whether the feature is actually active. Disabling fastpath (the only one you can toggle) deactivates fasttrack as well. You can confirm this at any time, nothing will go awry.
As for the second part of your inquiry about there being another way. There is, however that isn’t what I was implying What you are doing is totally fine, especially because usually guest traffic is lower in volume (and therefore the efficiency of fasttrack is not really needed), and you often want to restrict its bandwidth usage with queues, which for fasttracked traffic are also unavailable (except for interface queues.) So I actually wouldn’t change things.
The other way is only possible if the subnets on your normal and guest networks don’t overlap (which they really have no reason to), in which case the whole mangling/marking thing can be replaced by a routing rule of the form: /routing rule add interface=wan dst-address=gu.est.sub.net/24 action=lookup-only-in-table table=guest-routing-tableThis is fully compatible with fasttrack.
Thanks Lurker. Quite happy with the setup as-is. Like you say, it's not going to push the bounds - I think it's going to end up on an 80/20 connection so not a problem. Interesting that I could achieve largely the same thing as VRF (at least for this requirement) by manually adding a routing table...which is pretty much what I started off trying to achieve. Would need a firewall rule or two to stop traffic between the two but quite possibly simpler. Cheers.
Yep. With 80/20 fasttrack doesn’t matter to you. If you’re looking at any feature that’s not compatible with it, simply turn it off.
Just an additional note: You should really repeat your second mangle rule (the mark-routing one) in the mangle/output chain, without the in-interface matcher. This is because you may want to provide additional services running on your router, and for those the packet flow is not through mangle/prerouting, but trough mangle/output. (That this works out ok for ping doesn’t mean that it will work similarly for other things - it’s just a side effect of kernel behavior.)
Your whole setup in fact doesn’t require any additional routing table or vrf. The isolation of the two networks can simply be handled by the firewall.
Cheers, I'll have a look at the mangle output rules and see if I can wrap my head around that. Some time when it's not very much a post-Friday requisite Beer O'Clock!
As for the possibility of firewall rules, yes and no. Isolation, totally but there is (at least potentially) a need to route to specific IPs normally over the internet for the guest but to route to the same IPs via a VPN for the non-guest. It's just about remote access to servers for maintenance (hosted without NAT) when they provide publicly accessible services. Could probably also be done with policy routing but I'm used to the cost of that making it unattractive and effectively separate hardware is a less error-prone way of doing it. You can imagine the hideous complexity of my home network with me working from home, my wife working from home, untrusted IoT devices that can access the internet, untrusted IoT that can't, home automation and a teenager. It is a delightful mix of VLANs, VEF, ZBF, Cisco, Mikrotik, mdns, IKE, IKEv2, Wireguard, WiFi, basis, RADIUS, dot1x and port security. Not that I make life complicated for myself!
I suspected that you also wanted other policy routing stuff as well, that’s why I didn’t start off with telling you to just use the firewall Obviously, if you need the functionality then you need it.
Thanks Lurker. I've managed to get back to this and had a look at the rules and man pages. Am I reading this correctly that it's similar to firewall rule chains forward vs input in that the output mangle will apply the route mark to packets generated by the router. For example, if I were to get the router to provide DNS to clients on the guest network, the reply packets from the router would need to be tagged with the correct VRF in order to be properly routed via the correct routing table rather than dumped out the default route on the 'main' routing table. That about right?
/ip firewall mangle
add action=mark-connection chain=prerouting comment="Mark outbound connections from VRF-Guest" connection-state=new in-interface=bridge-guest new-connection-mark=VRF-Guest
add action=mark-routing chain=prerouting comment="Add correct VRF tag to incoming packets that are part of a connection marked as VRF-Guest" connection-mark=VRF-Guest in-interface-list=WAN new-routing-mark=VRF-Guest
add action=mark-routing chain=output comment="Add correct VRF tag for packets generated by router that are part of a connection marked as VRF-Guest" connection-mark=VRF-Guest new-routing-mark=VRF-Guest
Sorry for the short answer, but that’s all. You have reviewed the packet flow diagrams and you have come to the correct conclusion. Another forum member has taken upon themself to provide a visual guide to the packet flow: look up @jaclaz and the Ultimate Mikrotik packet flow diagram.
What you are doing is totally fine, especially because usually guest traffic is lower in volume (and therefore the efficiency of fasttrack is not really needed), and you often want to restrict its bandwidth usage with queues, which for fasttracked traffic are also unavailable (except for interface queues.) So I actually wouldn’t change things.