PBR Output Chain Routing Adjustment not Selecting Alternative Interface of a Dual WAN Link within a Single ISP Subnet

RouterOS General
1

Hi,

I have a similar use-case to a number of scenarios already discussed on this forum - two wan links connected to a single subnet with the same ISP gateway, both of my interfaces are configured using the ISP’s DHCP. This part I’ve successfully configured using the VRRP hack.

I am then trying to setup stable forwarding of the connections - all packets belonging to a connection initiated via wan1 to always flow via wan1 etc. Even this part seems to work successfully.

What I can’t get working is setting up the similar thing but this time also for the output chain - so that the responses to traffic sent to the router coming via wan1 (ie ping) are also sent back via wan1. This I am unable to get right. It seems the routing adjustment that’s supposed to happen at the end of the output chain doesn’t change the routing decision to the alternative wan interface.


The relevant parts of my setup are as follows:

  • 192.168.0.0/16 is my LAN
  • 10.10.10.0/24 is the ISP WAN subnet (wan1=10.10.10.115 and wan2=10.10.10.116, GW: 10.10.10.1)
  • I want all of the LAN-originated (from-intra mark) traffic to leave via wan1
  • responses to wan1-inbound requests to be sent via wan1
  • responses to wan2-inbound requests to be sent via wan2
  • running on ros v7.1.1
/ip dhcp-client
add interface=wan1
add default-route-distance=2 interface=wan2

/routing table
add disabled=no fib name=wan1
add disabled=no fib name=wan2

/routing rule
add action=lookup disabled=no routing-mark=wan2 table=wan2
add action=lookup disabled=no routing-mark=wan1 table=wan1

/ip route
add dst-address=10.10.10.0/24 gateway=wan1 routing-table=wan1
add dst-address=10.10.10.0/24 gateway=wan2 routing-table=wan2

> print
  D d+ 0.0.0.0/0         10.10.10.1         2
  DAd+ 0.0.0.0/0         10.10.10.1         1
  DAc+ 10.10.10.0/24   wan2                 0
  DAc+ 10.10.10.0/24   wan1                 0
 ...
 0  As  10.10.10.0/24   wan1                 1
 1  As  10.10.10.0/24   wan2                 1
 
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=wan1 new-connection-mark=from-wan1 passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=wan2 new-connection-mark=from-wan2 passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address=!192.168.0.0/16 new-connection-mark=from-intra passthrough=no
add action=mark-routing chain=output connection-mark=from-wan1 new-routing-mark=wan1 passthrough=no
add action=mark-routing chain=output connection-mark=from-wan2 new-routing-mark=wan2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=from-intra dst-address=!192.168.0.0/16 new-routing-mark=wan1 passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat out-interface=wan1 src-address=192.168.0.0/16
add action=masquerade chain=srcnat out-interface=wan2 src-address=192.168.0.0/16

All the traffic originated from the LAN works as expected, however, traffic from the outside arriving to wan2 is being responded via wan1 as show in the following dump:

/tool/sniffer/packet> print detail 
 0 time=14.322 num=1 direction=rx src-mac=30:5A:3A:74:FE:AD dst-mac=00:00:5E:00:01:02 interface=wan2 
   src-address=<external.ip>:45928 dst-address=10.10.10.116:13231 protocol=ip ip-protocol=udp size=190 cpu=0 
   ip-packet-size=176 ip-header-size=20 dscp=0 identification=57553 fragment-offset=0 ttl=55 

 1 time=14.332 num=2 direction=tx src-mac=00:00:5E:00:01:01 dst-mac=30:5A:3A:74:FE:AD interface=wan1 
   src-address=10.10.10.115:13231 dst-address=<external.ip>:45928 protocol=ip ip-protocol=udp size=134 cpu=0 
   ip-packet-size=120 ip-header-size=20 dscp=34 identification=2163 fragment-offset=0 ttl=64

See the request arrives at wan2 (with its VRRP mac *:02) but the response leaves via wan1 (VRRP *:01).

I suspect the initial routing decision made before entering the output chain picks the default route from the @main table for which the wan1 interface has higher priority than wan2 and later when the output chain is supposed to do the routing adjustment based on the mangling, it tries to override with the gateway=wan2 rule/route but since that resolves to the same gateway IP, it doesn’t change the actual output interface to wan2.

What am I doing wrong?

2

Your /ip/route/export shows that only routes to 10.10.10.0/24 exist in the additional routing tables wan1 and wan2, whereas you need default routes there (dst-address=0.0.0.0/0). Could it be this simple?

Your current /routing/rule rows have no actual effect. If no eligible route is available in the required table (wan1 or wan2), RouterOS uses main, and since you want traffic from the LAN subnet to use WAN 1 and the route via WAN 1 is preferred in routing table main, your goal is fulfilled but not by means of policy routing. To prevent the fallback to main, you have to change the action in the routing rules to lookup-only-in-table. Once you do that, the traffic marked with wan1 or wan2 will stop passing through until you add the default routes to these tables.

Yet another issue, which is currently also hidden by the absence of default routes in wan1 and wan2 and the fact that the fallback to main is not prevented, is that your action=mark-routing connection-mark=from-intra rule matches on dst-address, which is not correct because in the prerouting chain, mangle stands before dstnat. So while passing mangle in prerouting, the packets that came in via WAN still have the WAN IP addresses as destination, so also the responses to requests from LAN get the routing-mark and never reach the LAN even though they get un-src-nated later. So change dst-address=!192.168.0.0/16 to src-address=192.168.0.0/16 to fix this.

And last, since you’ve set passthrough=no in the action=mark-connection dst-address=!192.168.0.0/16 rule, initial packets of connections coming from LAN will get the connection-mark but not the routing-mark. So it will work for as long as you want connections from LAN to use WAN 1, thanks to routing table main. But as soon as you decide to change that, you’ll start pulling your hair again.

3

You want these routes instead of yours:

/ip route
add dst-address=0.0.0.0/0 gateway=10.10.10.1%wan1 routing-table=wan1
add dst-address=0.0.0.0/0 gateway=10.10.10.1%wan2 routing-table=wan2
4

cheers guys, got it working thanks to your help!