I am trying to route 2 LANs over 2 ISP connections. I want servers on LAN1 to route over a more expensive internet connection and users on LAN2 to route over a cheaper route. This mostly works but it breaks routing between the LANs.
To complicate matters the public server IPs are 1:1 NATs from the ISP.
How can I get this to work without breaking the routing between the LANs?
/ip address
# 1:1 NAT IPs from expensive ISP
add address=154.xxx.xxx.2/xx interface=Lo0 network=154.xxx.xxx.xxx
add address=154.xxx.xxx.3/xx interface=Lo0 network=154.xxx.xxx.xxx
add address=154.xxx.xxx.4/xx interface=Lo0 network=154.xxx.xxx.xxx
# actual IP address at expensive ISP
add address=197.xxx.xxx.2/xx interface=ether1 network=197.xxx.xxx.xxx
# IP address at cheap ISP
add address=172.160.0.2/16 interface=ether2 network=172.160.0.0
# lan IP addresses
add address=192.168.1.0/24 interface=ether3 network=192.168.1.0
add address=192.168.2.0/24 interface=ether4 network=192.168.2.0
/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=lan1 passthrough=yes src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=lan2 passthrough=yes src-address=192.168.2.0/24
/ip route
# route servers over expensive route
add distance=1 gateway=197.xxx.xxx.xxx routing-mark=lan1
# route users over cheap route
add distance=1 gateway=172.16.0.1 routing-mark=lan2
/ip firewall nat
# src NAT server IPs back to 1:1 IPs
add action=src-nat chain=srcnat protocol=tcp src-address=192.168.1.2 to-addresses=154.xxx.xxx.2
add action=src-nat chain=srcnat protocol=tcp src-address=192.168.1.3 to-addresses=154.xxx.xxx.3
add action=src-nat chain=srcnat protocol=tcp src-address=192.168.1.4 to-addresses=154.xxx.xxx.4
# Masquerade user traffic
add action=masquerage chain=srcnat out-interface=ether2 src-address=192.168.2.0/24
The problem I was having is that the srcnat for I needed for 1:1 at the ISP was also NATing traffic from LAN1 to LAN2. There are probably better solutions with packet marking or something, but adding an accept rule to the src-nat chain before srcnat happens seems to work.
So my NAT rules look like this now:
# src NAT server IPs back to 1:1 IPs for internet, but do not NAT server traffic to LAN
add action=accept chain=srcnat comment="Do not NAT test-web forward" dst-address=192.168.2.0/24 src-address=192.168.1.0/24
# LAN traffic accounted for, now do srcnats for 1:1
add action=src-nat chain=srcnat protocol=tcp src-address=192.168.1.2 to-addresses=154.xxx.xxx.2
add action=src-nat chain=srcnat protocol=tcp src-address=192.168.1.3 to-addresses=154.xxx.xxx.3
add action=src-nat chain=srcnat protocol=tcp src-address=192.168.1.4 to-addresses=154.xxx.xxx.4
Possibly better but untested is to exclude LAN2 in the src-nat rule:
# src NAT server IPs back to 1:1 IPs for internet, but do not NAT server traffic to LAN
add action=src-nat chain=srcnat dst-address=!192.168.2.0/24 protocol=tcp src-address=192.168.1.2 to-addresses=154.xxx.xxx.2
add action=src-nat chain=srcnat dst-address=!192.168.2.0/24 protocol=tcp src-address=192.168.1.3 to-addresses=154.xxx.xxx.3
add action=src-nat chain=srcnat dst-address=!192.168.2.0/24 protocol=tcp src-address=192.168.1.4 to-addresses=154.xxx.xxx.4
Finally got a chance to try this out and it doesn’t work. The second route shows blue in the GUI and some reading says this route will never be used. I will try again tomorrow with packet marks and will hopefully be able to force the 2nd route.
Got this working with by fixing my mangle rule. What I was missing is that traffic for 192.168.2.0 from 192.168.1.0 was also getting NAT according to my packet mark.