Failover and Selective Load-Balancing Issue

jcbhltz909 · May 20, 2024, 6:50pm

Hi everyone,

I have dual WAN DHCP interfaces (WAN0 and WAN1) and I am trying to set up failover with selective load-balancing. However, I’m encountering an issue where, after some time, traffic starts being routed through WAN1 even though WAN0 is still active. Disabling and re-enabling WAN0 does not resolve the issue, and traffic continues to route through WAN1.

Here’s what I aim to achieve:

10.10.110.5 should use WAN0 for port 80.
10.10.110.5 should use WAN1 for port 81.
All other traffic should be routed through WAN0. If WAN0 fails, traffic should be routed through WAN1.
CloudDNS should use the active WAN interface for updates.

While I know this could be done with simple routing, I want the flexibility to add marks in the future, and ideally, I’d like to set it up this way.

Below is my current configuration

Any guidance or suggestions would be greatly appreciated!

Thank you!

/ip dhcp-client
add add-default-route=no interface=WAN0 script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"WAN0\"] gateway=\$\"gateway-address\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"WAN0_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"WAN1_conn\"]\r\
    \n" use-peer-dns=no use-peer-ntp=no
add add-default-route=no disabled=yes interface=WAN1 script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"WAN1\"] gateway=\$\"gateway-address\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"WAN0_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"WAN1_conn\"]" use-peer-dns=no use-peer-ntp=no
/ip firewall mangle
add action=accept chain=prerouting comment="local-access" dst-address-list=local in-interface-list=LAN
add action=mark-routing chain=output dst-address-list=mikrotik-cloud new-routing-mark=to_WAN0 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=established,related in-interface=WAN0 new-connection-mark=\
    WAN0_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=established,related in-interface=WAN1 new-connection-mark=WAN1_conn \
    passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!local dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=WAN0_conn passthrough=yes
add action=mark-connection chain=prerouting dst-port=80 new-connection-mark=WAN0_conn passthrough=yes protocol=tcp src-address=10.10.110.5
add action=mark-connection chain=prerouting dst-port=81 new-connection-mark=WAN1_conn passthrough=yes protocol=tcp src-address=10.10.110.5
add action=mark-routing chain=prerouting connection-mark=WAN0_conn in-interface-list=LAN new-routing-mark=to_WAN0 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1_conn in-interface-list=LAN new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN0_conn dst-address-list=!local new-routing-mark=to_WAN0 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_conn dst-address-list=!local new-routing-mark=to_WAN1 passthrough=yes
/ip route
add comment=WAN0 distance=1 dst-address=9.9.9.9/32 gateway=132.68.132.1 scope=10 target-scope=10
add comment=WAN1 distance=1 dst-address=8.26.56.26/32 gateway=168.91.21.129 scope=10 target-scope=10
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=8.26.56.26 scope=10 target-scope=11
add comment=WAN0 dst-address=64.6.64.6/32 gateway=112.68.132.1 scope=10
add comment=WAN0 dst-address=208.67.220.220/32 gateway=112.68.132.1 scope=10
add comment=WAN1 dst-address=208.67.222.222/32 gateway=138.91.21.129 scope=10
add comment=WAN1 dst-address=64.6.65.6/32 gateway=168.91.21.129 scope=10
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=64.6.64.6 routing-table=to_WAN0 scope=10 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=64.6.65.6 routing-table=to_WAN0 scope=10 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=208.67.222.222 routing-table=to_WAN1 scope=10 target-scope=11
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=208.67.220.220 pref-src="" routing-table=to_WAN1 scope=10 \
    suppress-hw-offload=no target-scope=11

sindy · May 25, 2024, 6:39pm

There are some typos in your setup, like 132.68.132.1 instead of 112.68.132.1 at one place, but I suspect it’s a consequence of obfuscation and that the actual addresses of the gateways are different. Other than that, there is nothing clearly wrong in the setup except the scope=10 on the default routes, I just find it not optimal in terms that each of your 6 default routes has its own distinct “physical” route, so unreachability of 64.6.64.6 alone is sufficient for packets marked with to_WAN0 to start flowing through WAN1.

I would use just four “physical” routes in total, two per WAN, and then double each of the default routes in order that none of the effective routes depended on reachability of a particular “canary” address:

add comment=WAN0 dst-address=9.9.9.9/32 gateway=112.68.132.1 scope=10
add comment=WAN0 dst-address=64.6.64.6/32 gateway=112.68.132.1 scope=10

add comment=WAN1 dst-address=8.26.56.26/32 gateway=168.91.21.129 scope=10
add comment=WAN1 dst-address=208.67.222.222/32 gateway=168.91.21.129 scope=10

add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=9.9.9.9 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=64.6.64.6 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=8.26.56.26 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=208.67.222.222 target-scope=11

add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=9.9.9.9 routing-table=to_WAN0 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=64.6.64.6 routing-table=to_WAN0 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=8.26.56.26 routing-table=to_WAN0 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=208.67.222.222 routing-table=to_WAN0 target-scope=11

add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=8.26.56.26 routing-table=to_WAN1 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=208.67.222.222 routing-table=to_WAN1 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=9.9.9.9 routing-table=to_WAN1 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=64.6.64.6 routing-table=to_WAN1 target-scope=11

If that does not help, it will require some debugging while it actually happens.

anav · May 25, 2024, 8:16pm

I would want to see the complete config to make any assessments because the rules are often integrated to some extent ( affect each other ).

You stated that WAN0 is primary amd WAN1 is secondary and thus…
However, I dont like your setup for recursive. In fact it almost looks like mixing up nested and flat recursive but you are not using any faux addresses as they are all real and pingable.
Distances are wrong!! You have as noted by Sindy some gross errors in the ISP gateway IPs, I see one listed as =112.68.132.1 and one as 138.91.21.129

In fact, both of us interpreted the correct IP address as being the other LOL. So you know there is an issue!!

If you want recursive flat with your setup using THREE different DNS checks…

Recursive Flat Routes Main Table:
{ Farthest Hop }
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=64.6.64.6 scope=10 target-scope=11
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=208.67.220.220 scope=10 target-scope=11
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=8.26.56.26 scope=10 target-scope=11
add check-gateway=ping distance=5 dst-address=0.0.0.0/0 gateway=64.6.65.6 scope=10 target-scope=11
add check-gateway=ping distance=6 dst-address=0.0.0.0/0 gateway=208.67.222.222 scope=10 target-scope=11
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
{ Closer Hop }
add comment=WAN0 distance=1 dst-address=9.9.9.9/32 gateway=132.68.132.1 scope=10 target-scope=10 comment=WAN0
add comment=WAN0 distance=2 dst-address=64.6.64.6/32 gateway=132.68.132.1 scope=10 target-scope=10 comment=WAN0
add comment=WAN0 distance=3 dst-address=208.67.220.220/32 gateway=132.68.132.1 scope=10 target-scope=10 comment=WAN0
add comment=WAN1 distance=4 dst-address=8.26.56.26/32 gateway=168.91.21.129 scope=10 target-scope=10 comment=WAN1
add comment=WAN1 distance=5 dst-address=64.6.65.6/32 gateway=168.91.21.129 scope=10 target-scope=10 comment=WAN1
add comment=WAN1 distance=6 dst-address=208.67.222.222/32 gateway=168.91.21.129 scope=10 target-scope=10 comment=WAN1

DONE! Main table routes.

Now for Special Table Routes.
add distance=1 dst-address=0.0.0.0/0 gateway=132.68.132.1 routing-table=to_WAN0 comment=WAN0
add distance=1 dst-address=0.0.0.0/0 gateway=168.91.21.129 routing-table=to_WAN1 comment=WAN1

Done with all routes !

+++++++++++++++++++++++++++++++
Based on the above, not convinced mangles dont need looking at LOL.

anav · May 25, 2024, 8:30pm

If you wanted nested Recursive… it would be like so and just using two DNS checks. First we use a bogus or faux address,

add distance=1 dst-address=0.0.0.0/0 gateway=10.100.100.10 routing-table=main scope=10 target-scope=14
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add check-gateway=ping distance=2 dst-address=10.100.100.10/32 gateway=9.9.9.9 routing-table=main scope=10 target-scope=13
add check-gateway=ping distance=3 dst-address=10.100.100.10/32 gateway=208.67.220.220 routing-table=main scope=10 target-scope=13
add check-gateway=ping distance=4 dst-address=10.100.100.10/32 gateway=8.26.56.26 routing-table=main scope=10 target-scope=13
add check-gateway=ping distance=5 dst-address=10.100.100.10/32 gateway=208.67.222.222 routing-table=main scope=10 target-scope=13
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add distance=2 dst-address=9.9.9.9/32 gateway=132.68.132.1 routing-table=main scope=10 target-scope=12 comment=WAN0
add distance=3 dst-address=208.67.220.220/32 gateway=132.68.132.1 routing-table=main scope=10 target-scope=12 comment=WAN0
add distance=4 dst-address=8.26.56.26/32 gateway=168.91.21.129 routing-table=main scope=10 target-scope=12 comment=WAN1
add distance=5 dst-address=208.67.222.222/32 gateway=168.91.21.129 routing-table=main scope=10 target-scope=12 comment=WAN1

anav · May 25, 2024, 8:40pm

Now all user will use WAN0, and WAN1 is the defacto secondary failover option.
Now to differentiate a single user by port Mangling is certainly the option that comes to mind.

What you have is a holy mess of mangling that makes zeros sense to the requirements you stated.

/ip mangle
add chain=forward action=mark-connection connection-mark=no-mark in-interface-list=LAN src-address=10.10.110.5 dst-port=81 protocol=tcp
new-connection-mark=captureWAN1 passthrough=yes
add chain=forward action=mark-connection connection-mark=no-mark in-interface-list=LAN src-address=10.10.110.5 dst-port=81 protocol=udp
new-connection-mark=captureWAN1 passthrough=yes { only if udp is required }
add chain=prerouting action=mark-route connection-mark=captureWAN1 new-routing-mark=to_WAN1 passthrough=no

DONE!
Any traffic from that IP going to port 81 should be forced out WAN1.
We dont need to do anything for port 80 as WAN0 is the primary and will go out that path anyway.

By the way,
if WAN0 fails, traffic for both ports will go out WAN1
If WAN1 fails, traffic for both ports will go out WAN0

anav · May 25, 2024, 8:48pm

Finally, I also believe your scripts are wrong or should I say the application of the combination of scripts and IP DHCP settings.

In other words you should select
a. default route=YES
b. distance=255

For both WANs.