Two Public Gateways, One Natted LAN - Mangle/Route Issues

RouterOS General
1

I have an RB450 running 3.24 connected to two different ISPs. I would like to access the RB433 itself via either ISP and would like the packets come back out the same gateway that they went into.

(I am actually trying to do much more complicated things, but am having trouble with this particular piece and so have tried to whittle the problem down to its simplest form.)

I have a setup that appears to do the right thing: I can log into the router itself from the internet using either gateway. Using Packet Sniffer on Winbox, I can see the packets traverse the correct interface depending on which address I use.

The confusing thing is: if I either disconnect the cable to the first gateway, or disable its default route, then access to the router via the second gateway ceases to work. The Packet Sniffer reports the packets arriving at the router, but not going back out. If I am connecting via the second gateway, I am not sure why the default route pointing to the first gateway comes into play.

Here are my Mangle Rules and Routing. There are no NAT or Route Rules set up at present. One ISP (ether5) is the default, and connections coming from the second (ether1, connected to a cellular modem) one are marked and routed based on the in-interface. The remaining ports (ether2-4) are bridged.

 /ip ad pr
Flags: X - disabled, I - invalid, D - dynamic
 #   ADDRESS            NETWORK         BROADCAST       INTERFACE
 0   192.168.100.2/24   192.168.100.0   192.168.100.255 LocalBridge
 1   192.168.1.2/24     192.168.1.0     192.168.1.255   ether5
 2 D 70.192.134.134/24  70.192.134.0    70.192.134.255  ether1




 /ip route pr det
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 0 A S  dst-address=0.0.0.0/0 gateway=192.168.1.11 interface=ether5 gateway-state=reachable distance=1 scope=30 target-scope=10

 1 A S  dst-address=0.0.0.0/0 gateway=70.192.134.1 interface=ether1 gateway-state=reachable distance=1 scope=30 target-scope=10 routing-mark=Cell-route

 2 ADC  dst-address=70.192.134.0/24 pref-src=70.192.134.134 interface=ether1 distance=0 scope=10

 3 ADC  dst-address=192.168.1.0/24 pref-src=192.168.1.2 interface=ether5 distance=0 scope=10

 4 ADC  dst-address=192.168.100.0/24 pref-src=192.168.100.2 interface=LocalBridge distance=0 scope=10




/ip route rule pr
Flags: X - disabled, I - inactive
[admin@sts-dev] > /ip firewall mangle pr
Flags: X - disabled, I - invalid, D - dynamic
 0   ;;; Cell conn
     chain=prerouting action=mark-connection new-connection-mark=in-Cell-conn passthrough=yes in-interface=ether1

 1   chain=output action=mark-routing new-routing-mark=Cell-route passthrough=yes connection-mark=in-Cell-conn

So it appears with this setup that, if the link via my first ISP goes down, the link via my second ISP goes down, too.

I know I must be missing something here, so if anyone could point it missing link, I would appreciate it.

Thanks,

jw

2

I told myself I would post a working configuration if I ever got this figured out, so here it is.

I have a natted, bridged subnet that has two gateways to the internet. One gateway is cheap(simonsNet), the other is very expensive on a per-bit basis (cellNet). All outbound traffic should go over the cheap interface, with no failover. All inbound traffic should return over the same interface as it came in on.

Addresses

/ ip address
add address=192.168.100.2/24 network=192.168.100.0 broadcast=192.168.100.255 interface=signalBridge 
add address=192.168.1.105/24 network=192.168.1.0   broadcast=192.168.1.255   interface=simonsNet
add address=75.216.4.4/24    network=75.216.4.0    broadcast=75.216.4.255    interface=cellNet

The interfaces are a natted bridged subnet, and two gateways to the internet
Mangle

/ ip firewall mangle
add chain=prerouting action=mark-connection new-connection-mark=cellNet_conn in-interface=cellNet

Mark all connections originating on the cellNet interface

add chain=output     action=mark-routing new-routing-mark=to_cellNet connection-mark=cellNet_conn

Make a routing-mark on all packets originating on the router itself belonging to connections that orginated on the cellNet. These will need special routing to get them to the cellNet

add chain=prerouting action=mark-routing new-routing-mark=to_cellNet connection-mark=cellNet_conn in-interface=signalBridge

Make a routing-mark on all packets originating on the bridge interface that belong to connections originating on the cellNet gateway. These will also need special routing to get them to the cellNet
Routing

/ ip route
add dst-address=0.0.0.0/0 gateway=75.216.4.1   routing-mark=to_cellNet
add dst-address=0.0.0.0/0 gateway=192.168.1.11

The first handles all packets marked to go to the cellNet. The second handles everything else.

NATing

/ ip firewall nat 
add chain=srcnat out-interface=simonsNet action=masquerade

Only need to masquerade the simonsNet, since that is the default gateway used for outbound connections. Do not want users on the bridge to use the cellNet connection for outbound connections (too expensive)

/ip firewall nat add chain=dstnat dst-address=192.168.1.105  dst-port=443  protocol=tcp action=dst-nat to-addresses=192.168.100.1 to-ports=443 comment=sig1webISP1Dst
/ip firewall nat add chain=srcnat src-address=192.168.100.1  src-port=443  protocol=tcp action=src-nat to-addresses=192.168.1.105 to-ports=443 comment=sig1webISP1Src

Port forwarding for https on the simonsNet

/ip firewall nat add chain=dstnat dst-address=75.216.4.4     dst-port=443  protocol=tcp action=dst-nat to-addresses=192.168.100.1 to-ports=443 comment=sig1webISP2Dst
/ip firewall nat add chain=srcnat src-address=192.168.100.1  src-port=443  protocol=tcp action=src-nat to-addresses=75.216.4.4    to-ports=443 comment=sig1webISP2Src

Port forwarding for https on the cellNet


Hope this proves useful to someone.

Regards,

jw