We have a production setup that is using muliple VPS versions of x86 RouterOS. We have an issue that we’ve had for a while but never really done anything about it as the work around “works”!, I thought it might be worthwhile asking here if our understanding of how vrrp should work is correct.
I have replicated the production setup in a lab using a pair of 1009’s and get the same behaviour.
/interface bridge
add name=LANBridge
/interface ethernet
set [ find default-name=ether1 ] name=Morten-WAN
/interface vrrp
add interface=Morten-WAN interval=3s name=vrrp-102 vrid=102
/ip pool
add name=dhcp_pool1 ranges=192.168.55.2-192.168.55.254
/ip dhcp-server
add address-pool=dhcp_pool1 disabled=no interface=LANBridge name=dhcp1
/interface bridge port
add bridge=LANBridge interface=ether5
/ip address
add address=192.168.88.1/24 interface=Morten-WAN network=192.168.88.0
add address=192.168.55.1/24 interface=LANBridge network=192.168.55.0
add address=192.168.40.71/24 interface=Morten-WAN network=192.168.40.0
add address=192.168.40.73 interface=vrrp-102 network=192.168.40.73
add address=192.168.40.74/24 interface=vrrp-102 network=192.168.40.0
p dhcp-server network
add address=192.168.55.0/24 dns-server=8.8.8.8 gateway=192.168.55.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8
/ip firewall nat
add action=src-nat chain=srcnat out-interface=vrrp-102 to-addresses=\
192.168.40.73
/ip route
add distance=10 gateway=192.168.40.1%vrrp-102 pref-src=192.168.40.73
add distance=15 gateway=192.168.40.1%Morten-WANode]
So the key here is the ip addresses set on the vrrp interface
add address=192.168.40.73 interface=vrrp-102 network=192.168.40.73
This is the correct way in ROS to define an ipaddress for the vrrp (assuming you have an ip of the same subnet defined on the host interface of the vrrp interface)
You’ll see below that address in the code we have the additional line
add address=192.168.40.74/24 interface=vrrp-102 network=192.168.40.0
If we don’t add this address (with note the /24 (same as the host interface)) the the ip route statement which should be highest becomes unreachable
/ip route
add distance=10 gateway=192.168.40.1%vrrp-102 pref-src=192.168.40.73
add distance=15 gateway=192.168.40.1%Morten-WAN
So the question is - is this correct behaviour? On the production system I have to add an additional Ip address with the /26 (in that case) to make the routes work ok, and with the production boxes being on public ip’s i’d rather save them up.
Thanks
You’ll see below that address in the code we have the additional line
add address=192.168.40.74/24 interface=vrrp-102 network=192.168.40.0
>
> If we don't add this address (with note the /24 (same as the host interface)) the the ip route statement which should be highest becomes unreachable
I'm confused. Which IP route statement becomes unreachable?
> ```text
/ip firewall nat
add action=src-nat chain=srcnat out-interface=vrrp-102 to-addresses=\
192.168.40.73
Here you are saying any traffic that is going out the “vrrp-102” interface is src-nat’d to the 192.168.40.73 address. This may be the true point of trouble. Are we able to step back a little. We may want to talk why you are using VRRP on the public side and why you are src-nat’ing to a shared VRRP address instead of the public interface IP for each router and relying on routing to return the packet to the correct device.
Ok, some clarity;
add distance=10 gateway=192.168.40.1%vrrp-102 pref-src=192.168.40.73
This route becomes unreachable if i remove the vrrp address with the /24 on the end. (VRRP addresses should be /32)
(the 192.168.40.x range is used as a simulator for the public ip addresses)
Why wouldn’t I want the connections going out with the Vrrp address? Regardless of what side the vrrp address is of the router the requirements are the same, two routers accessible by a single ip address (one at time of course with a shared mac address). The production servers are there to accept incoming connections from the internet and as such they connect to the vrrp address, so the devices would expect a reply from the vrrp address.
To me that makes sense. The VRRP interface is a /32, here you add a route that says the gateway is 192.168.40.1 on the VRRP interface. An address that is not included in 192.168.40.73/32 that is assigned to that interface. So in order for the router to reach 192.168.40.1 a subnet mask that includes that IP and .73 is required. In your case you found that a /24 works. I’d assume a /25 would achieve the same result. Either way…
The route you are highlighting as no dst-address specified. The default value of this is 0.0.0.0/0. This means you are trying to from a routing perspective force the use of the VRRP address from what I can tell. This would take precedence when your source NAT doesn’t.
In general this all seems off and not at the fault of MikroTik and VRRP but more so design. So let’s talk more design. The more you can clue us in to what you are really trying to do the more we can help you. Sometimes the best fix is to bring a better design to the table than some trickery to try to get the device to do something it really shouldn’t.
You hinted at servers that may be behind these routers. My suspicion is you have a service, for simplicity let’s say it’s a web-site, that you want to be available behind each router. In MikroTik land adding src and dst nat rules isn’t enough for the router to reply to ARP requests for an IP. Some others can chime in but the simple answer is to add the IP address to the appropriate interface. From their the appropriate NAT rules will take effect and route accordingly. I’ve read about using black-hole routes and proxy-arp but that didn’t appear to work for me in the latest versions. Additionally I’m not a fan of proxy-arp and generally avoid it like the plague to avoid knock-on effects of turning it on. A middle of the road solution is to use anycast. Assign the IP as an anycast IP (/32) to both interfaces similar to VRRP. You will not get duplicate IP addresses because everyone thinks the IP is part of it’s own network and will co-exist happily. Both will reply to ARP requests and which-ever lands first will win. Now doesn’t that sound nice. Kind of. Some people may prefer a more dictated approach especially if their are state consistency concerns. Here you could leverage VRRP and assign as many IPs (I don’t know of a limitation) to that VRRP interface. This would allow you to use typical mechanics to control the state of those shared public IPs more effectively. You of course would need the NAT rules in place for all shared addresses.
Notice that none of these solutions require adding a static route. We use VRRP to claim the IP and build the layer 2 to layer 3 mapping in ARP. The router then uses 1:1 NAT, can be the all ports or specific, to move the traffic to the appropriate server behind the router. Each router would have the NAT rules configured.
Now the down-side with this configuration is that the src and dst nat rules will cause the server on the non-active side to work, if they are a 1:1 NAT without any specificity. If we add specificity appropriately we can have the 1:1 relationship only for the application traffic, our example being a web-site, and all other traffic would fall to a lower placed common masquerade to the interface IP for that side. This would probably the most ideal solution. It would allow the server to do most of it’s day to day business like updates or management while still serving the web-site appropriately when the VRRP state changes and shifts traffic that direction.
If your “application” or “usage” is different we can talk more about possible ways to solve your problem. Thankfully VRRP in MikroTik has some scripting control points like on-backup and on-master that we could leverage to shift a generic 1:1 NAT between devices when the state change happens by enabling or disabling the appropriate NAT rules on demand.
Here’s a quick and dirty drawing. I used separate networks behind the router but you could fairly easily do something similar on a shared segment behind the router (possibly layering VRRP there for the more traditional use of a single gateway IP).
