Dual WAN policy routing with DHCP

RouterOS Beginner Basics
1

I’m trying to set up dual WAN connections without luck. The firewall filter and magle rules mark the connections and create routing marks. Packets marked (for example to 1.2.3.4) seem to get mark, but cannot figure out how to get the packets to use different WAN interface. With setup below, traffic in and out work but only using ether5-lte interface.

The dhcp-client inserts the routes to main table for both interfaces. How do I get it to create entries to separate routing tables? In the nexthops the table remains empty and outgoing traffic marked with to_adsl routing mark results “no route to host” (icmp message destination unreachable returned).
routes_1.png
routes_nexthops.png

[admin@gw254] > /ip route print           
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          ether5-lte                3
 1 A S  0.0.0.0/0                          ether1-gateway            5
 2 ADS  0.0.0.0/0                          51.150.189.94             3
 3  DS  0.0.0.0/0                          71.107.104.1              5
 4 ADC  10.4.1.0/24        10.4.1.254      ether5-lte                0
 5 ADC  10.3.0.0/16        10.3.1.254      ether2-master-l...        0
 6 ADC  51.150.189.88/29   51.150.189.93   ether5-lte                0
 7 ADC  71.107.104.0/23    71.107.104.172  ether1-gateway            0




[admin@gw254] > ip route print detail 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=ether5-lte gateway-status=ether5-lte reachable check-gateway=ping distance=3 scope=30 target-scope=30 routing-mark=to_lte 
 1 A S  dst-address=0.0.0.0/0 gateway=ether1-gateway gateway-status=ether1-gateway reachable check-gateway=ping distance=5 scope=30 target-scope=30 routing-mark=to_adsl 
 2 ADS  dst-address=0.0.0.0/0 gateway=51.150.189.94 gateway-status=51.150.189.94 reachable via  ether5-lte distance=3 scope=30 target-scope=10 vrf-interface=ether5-lte 
 3  DS  dst-address=0.0.0.0/0 gateway=71.107.104.1 gateway-status=71.107.104.1 reachable via  ether1-gateway distance=5 scope=30 target-scope=10 vrf-interface=ether1-gateway 
 4 ADC  dst-address=10.4.1.0/24 pref-src=10.4.1.254 gateway=ether5-lte gateway-status=ether5-lte reachable distance=0 scope=10 
 5 ADC  dst-address=10.3.0.0/16 pref-src=10.3.1.254 gateway=ether2-master-local gateway-status=ether2-master-local reachable distance=0 scope=10 
 6 ADC  dst-address=51.150.189.88/29 pref-src=51.150.189.93 gateway=ether5-lte gateway-status=ether5-lte reachable distance=0 scope=10 
 7 ADC  dst-address=71.107.104.0/23 pref-src=71.107.104.172 gateway=ether1-gateway gateway-status=ether1-gateway reachable distance=0 scope=10




[admin@gw254] > /export 
# dec/16/2016 16:59:22 by RouterOS 6.37.3
# software id = 5008-63CH
#

/ip firewall filter
add action=accept chain=input comment="default configuration" protocol=icmp
add action=accept chain=input comment="default configuration" connection-state=established,related
add action=drop chain=input comment="default configuration" in-interface=ether1-gateway
add action=drop chain=input comment="default configuration" in-interface=ether5-lte
add action=fasttrack-connection chain=forward comment="default configuration" connection-state=established,related disabled=yes
add action=accept chain=forward comment="default configuration" connection-state=established,related
add action=drop chain=forward comment="default configuration" connection-state=invalid
add action=drop chain=forward comment="default configuration" connection-nat-state=!dstnat connection-state=new in-interface=ether1-gateway
add action=drop chain=forward comment="default configuration" connection-nat-state=!dstnat connection-state=new in-interface=ether5-lte

/ip firewall mangle
add action=mark-connection chain=prerouting comment="force adsl-interface" dst-address=92.122.94.47 new-connection-mark=adsl_mark passthrough=yes
add action=mark-connection chain=input connection-mark=no-mark in-interface=ether5-lte new-connection-mark=lte_mark passthrough=yes
add action=mark-connection chain=input connection-mark=no-mark in-interface=ether1-gateway new-connection-mark=adsl_mark passthrough=yes
add action=mark-routing chain=prerouting connection-mark=lte_mark dst-address-type="" new-routing-mark=to_lte passthrough=yes
add action=mark-routing chain=prerouting connection-mark=adsl_mark dst-address-type="" new-routing-mark=to_adsl passthrough=yes
add action=mark-connection chain=output connection-mark=no-mark new-connection-mark=adsl_mark out-interface=ether1-gateway passthrough=yes
add action=mark-connection chain=output connection-mark=no-mark new-connection-mark=lte_mark out-interface=ether5-lte passthrough=yes

/ip firewall nat
add action=masquerade chain=srcnat comment="Hairpin NAT" dst-address=10.3.0.0/16 dst-address-list=!10.3.0.0/16 out-interface=ether2-master-local protocol=tcp src-address=10.3.0.0/16
add action=masquerade chain=srcnat comment="NAT masquerade for outgoing connections" out-interface=ether1-gateway
add action=masquerade chain=srcnat comment="NAT masquerade for outgoing connections" out-interface=ether5-lte

/ip route
add check-gateway=ping distance=3 gateway=ether5-lte routing-mark=to_lte target-scope=30
add check-gateway=ping distance=5 gateway=ether1-gateway routing-mark=to_adsl target-scope=30

/ip route rule
add routing-mark=to_adsl table=to_adsl
add routing-mark=to_lte table=to_lte
2

Use routing filters to set routing table for dhcp routes, see e.g. here.

3

Thank you Sob about help. The routing filter was the correct approach and setup is ALMOST working.

All traffic normally goes to lte-connection and MOST packets marked with to_adsl connection-mark to adsl-connection. With most I mean that for example ping to 4.2.2.2 (marked with magle-rules below) works, except with huge packet loss, something like 40%. Also incomining dstnat:tted connections work, but with packet loss (ssh-connection looks like working, but hangs occasionally).

Any idea where to look for to problem?

/ip route rule
add action=lookup-only-in-table comment=LAN dst-address=10.3.0.0/16 table=main
add comment="lookup from table to_lte with routing mark to_lte" routing-mark=to_lte table=to_lte
add comment="lookup from table to_adsl with routing mark to_adsl" routing-mark=to_adsl table=to_adsl
add table=to_lte
add table=to_adsl

/routing filter
add chain=dynamic-in distance=10 set-distance=1 set-route-comment="Set by route filter for to_lte" set-routing-mark=to_lte
add chain=dynamic-in distance=11 set-distance=1 set-route-comment="Set by route filter for to_adsl" set-routing-mark=to_adsl

With above, the routing table is like:

admin@gw254] /ip route rule> /ip route print detail 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 ADS  ;;; Set by route filter for to_lte
        dst-address=0.0.0.0/0 gateway=51.150.189.94 gateway-status=51.150.189.94 reachable via  ether5-lte distance=1 scope=30 target-scope=10 routing-mark=to_lte vrf-interface=ether5-lte 
 1 ADS  ;;; Set by route filter for to_adsl
        dst-address=0.0.0.0/0 gateway=71.107.104.1 gateway-status=71.107.104.1 reachable via  ether1-gateway distance=1 scope=30 target-scope=10 routing-mark=to_adsl vrf-interface=ether1-gateway 
 2 ADC  dst-address=10.4.1.0/24 pref-src=10.4.1.254 gateway=ether5-lte gateway-status=ether5-lte reachable distance=0 scope=10 
 3 ADC  dst-address=10.3.0.0/16 pref-src=10.3.1.254 gateway=ether2-master-local gateway-status=ether2-master-local reachable distance=0 scope=10 
 4 ADC  dst-address=51.150.189.88/29 pref-src=51.150.189.93 gateway=ether5-lte gateway-status=ether5-lte reachable distance=0 scope=10 
 5 ADC  dst-address=71.107.104.0/23 pref-src=71.107.104.172 gateway=ether1-gateway gateway-status=ether1-gateway reachable distance=0 scope=10




/interface ethernet
set [ find default-name=ether1 ] name=ether1-gateway rx-flow-control=auto tx-flow-control=auto
set [ find default-name=ether2 ] name=ether2-master-local poe-out=off rx-flow-control=auto tx-flow-control=auto
set [ find default-name=ether3 ] master-port=ether2-master-local name=ether3-slave-local poe-out=off
set [ find default-name=ether4 ] master-port=ether2-master-local name=ether4-slave-local poe-out=off
set [ find default-name=ether5 ] name=ether5-lte poe-out=off rx-flow-control=auto tx-flow-control=auto

/ip settings
set rp-filter=loose

/ip firewall filter
add action=accept chain=input comment="default configuration" protocol=icmp
add action=accept chain=input comment="default configuration" connection-state=established,related
add action=drop chain=input comment="default configuration" in-interface=ether1-gateway
add action=drop chain=input comment="default configuration" in-interface=ether5-lte
add action=fasttrack-connection chain=forward comment="default configuration" connection-state=established,related disabled=yes
add action=accept chain=forward comment="default configuration" connection-state=established,related
add action=drop chain=forward comment="default configuration" connection-state=invalid
add action=drop chain=forward comment="default configuration" connection-nat-state=!dstnat connection-state=new in-interface=ether1-gateway
add action=drop chain=forward comment="default configuration" connection-nat-state=!dstnat connection-state=new in-interface=ether5-lte

/ip firewall mangle
add action=mark-connection chain=prerouting comment="Set adsl_mark to https://wtfismyip.com/text " connection-state=new dst-address=167.114.174.158 new-connection-mark=adsl_mark passthrough=yes
add action=mark-connection chain=prerouting comment="Good ping target 4.2.2.2 " connection-state=new dst-address=4.2.2.2 new-connection-mark=adsl_mark passthrough=yes
add action=mark-connection chain=input connection-mark=no-mark in-interface=ether5-lte new-connection-mark=lte_mark passthrough=yes
add action=mark-connection chain=input connection-mark=no-mark in-interface=ether1-gateway new-connection-mark=adsl_mark passthrough=yes
add action=mark-routing chain=prerouting connection-mark=lte_mark dst-address-type="" new-routing-mark=to_lte passthrough=yes
add action=mark-routing chain=prerouting connection-mark=adsl_mark dst-address-type="" new-routing-mark=to_adsl passthrough=yes
add action=mark-connection chain=forward connection-state=new in-interface=ether1-gateway new-connection-mark=adsl_mark
add action=mark-connection chain=forward connection-state=new in-interface=ether5-lte new-connection-mark=lte_mark
add action=mark-routing chain=output connection-mark=adsl_mark new-routing-mark=to_adsl out-interface=ether1-gateway passthrough=no
add action=mark-routing chain=output connection-mark=lte_mark new-routing-mark=to_lte out-interface=ether5-lte passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat comment="Hairpin NAT" dst-address=10.3.0.0/16 dst-address-list=!10.3.0.0/16 out-interface=ether2-master-local protocol=tcp src-address=10.3.0.0/16
add action=masquerade chain=srcnat comment="NAT masquerade for outgoing connections" out-interface=ether1-gateway
add action=masquerade chain=srcnat comment="NAT masquerade for outgoing connections" out-interface=ether5-lte
add action=dst-nat chain=dstnat comment=DSTNAT-22 dst-address-type=local dst-port=22 protocol=tcp to-addresses=10.3.1.1 to-ports=22
4

I’m looking at it, but I don’t see anything obviously wrong. Try to find what happens. Run ping on 4.2.2.2 and see where those packets go (using Tools->Torch or log rule in postrouting). You can also try to replace connection-state=new in connection marking rules with connection-mark=no-mark.