I've successfully implemented the advanced routing failover using:

https://wiki.mikrotik.com/wiki/Advanced ... _Scripting

When the cable modem fails the connection switches over immediately to an LTE backup and that works well.

I've noticed, however, that when the cable modem resumes that during the connection cycle that it initially goes through the approach has trouble for around 50 seconds or so before successfully switching back.

I believe the route cause is that a route that has check-gateway = ping can become active even if the gateway IP address is not pingable. This happens in particular if the interface over which the gateway IP is routed goes down before the gateway becomes unreachable.

So the sequence that happens is:

1. Cable modem is fine... all gateway check IPs are pingable. (I use 3 instead of 2 for extra redundancy)
2. Cable modem goes down, interface goes down. The routes with the check-gateway = ping become inactive immediately because they cannot be routed to.
3. Cable modem starts coming back up (but doesn't fully negotiate the link with the ISP). The interface becomes active again.
4. THE PROBLEM: The routes with the check-gateway = ping become active again. This is true even though the gateway cannot be pinged. The default route switches back to the cable modem.
5. 20 seconds or so later, (two consecutive 10 second ping checks fail). The routes become inactive and we switch back to LTE.
6. This bounces back and forth a couple times before the cable modem link is fully up.

In step 4... if a route with a check-gateway was only activated if that gateway was pingable (seems sensible?) this issue wouldn't occur.

Does the above make sense and if so is that a feature we can get in the queue? An alternative is that the router should be trying to ping the gateway IPs even when the route isn't active so it knows their status at all times.

A

Post your routing configuration, it is not easy to understand without looking at it.
I suspect you have defined interface as gateway instead of IP address.

Here's my current routing table:

Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 A S 0.0.0.0/0 PureVPN-PPTP 5
1 A S ;;; LTE
0.0.0.0/0 192.168.50.1 1
2 A S ;;; VirtualCable
0.0.0.0/0 10.1.1.1 2
3 S ;;; VirtualLTE
0.0.0.0/0 10.2.2.2 3
4 DS 0.0.0.0/0 PureVPN-PPTP 5
5 A S ;;; LTE
1.0.0.1/32 192.168.50.1 1
6 SB 1.0.0.1/32 20
7 A S ;;; CABLE
1.1.1.1/32 67.253.120.1 1
8 SB 1.1.1.1/32 20
9 A S ;;; LTE
4.2.2.2/32 192.168.50.1 1
10 SB 4.2.2.2/32 20
11 A S 10.1.1.1/32 208.67.222.222 1
12 S 10.1.1.1/32 1.1.1.1 1
13 S 10.1.1.1/32 216.239.35.0 1
14 A S 10.2.2.2/32 1.0.0.1 1
15 S 10.2.2.2/32 4.2.2.2 1
16 S 10.2.2.2/32 208.67.220.220 1
17 A S ;;; LTE PureVPN-PPTP
45.74.35.0/24 192.168.50.1 1
18 SB ;;; PureVPN-PPTP Blackhole
45.74.35.0/24 20
19 ADS 45.74.35.1/32 192.168.50.1 0
20 ADC 45.74.35.7/32 45.74.35.8 PureVPN-PPTP 0
21 ADC 67.253.120.0/21 67.253.126.143 ether1-gateway 0
22 ADC 192.168.50.0/24 192.168.50.10 ether5-gateway 0
23 S 192.168.50.0/24 ether5-gateway 1
24 ADC 192.168.88.0/24 192.168.88.1 bridge1 0
25 A S ;;; Cable Modem
207.32.194.24/32 67.253.120.1 1
26 SB ;;; Cable Modem blackhole
207.32.194.24/32 20
27 A S ;;; LTE
208.67.220.220/32 192.168.50.1 1
28 SB 208.67.220.220/32 20
29 A S ;;; CABLE
208.67.222.222/32 67.253.120.1 1
30 SB 208.67.222.222/32 20
31 A S ;;; CABLE
216.239.35.0/32 67.253.120.1 1
32 SB 216.239.35.0/32 20


And the corresponding config export. The VPN is strictly for an inbound connection and can be ignored (as can the LAPTOP routing mark stuff).

add distance=5 gateway=PureVPN-PPTP routing-mark=PPTP
add check-gateway=ping comment=LTE distance=1 gateway=192.168.50.1 routing-mark=LAPTOP
add comment=VirtualCable distance=2 gateway=10.1.1.1
add comment=VirtualLTE distance=3 gateway=10.2.2.2
add comment=LTE distance=1 dst-address=1.0.0.1/32 gateway=192.168.50.1 scope=10
add distance=20 dst-address=1.0.0.1/32 type=blackhole
add comment=CABLE distance=1 dst-address=1.1.1.1/32 gateway=67.253.120.1 scope=10
add distance=20 dst-address=1.1.1.1/32 type=blackhole
add comment=LTE distance=1 dst-address=4.2.2.2/32 gateway=192.168.50.1 scope=10
add distance=20 dst-address=4.2.2.2/32 type=blackhole
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=208.67.222.222 scope=10
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=1.1.1.1 scope=10
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=216.239.35.0 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=1.0.0.1 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=4.2.2.2 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=208.67.220.220 scope=10
add comment="LTE PureVPN-PPTP" distance=1 dst-address=45.74.35.0/24 gateway=192.168.50.1
add comment="PureVPN-PPTP Blackhole" distance=20 dst-address=45.74.35.0/24 type=blackhole
add distance=1 dst-address=192.168.50.0/24 gateway=ether5-gateway
add comment="Cable Modem" distance=1 dst-address=207.32.194.24/32 gateway=67.253.120.1
add comment="Cable Modem blackhole" distance=20 dst-address=207.32.194.24/32 type=blackhole
add comment=LTE distance=1 dst-address=208.67.220.220/32 gateway=192.168.50.1 scope=10
add distance=20 dst-address=208.67.220.220/32 type=blackhole
add comment=CABLE distance=1 dst-address=208.67.222.222/32 gateway=67.253.120.1 scope=10
add distance=20 dst-address=208.67.222.222/32 type=blackhole
add comment=CABLE distance=1 dst-address=216.239.35.0/32 gateway=67.253.120.1 scope=10
add distance=20 dst-address=216.239.35.0/32 type=blackhole

On your first part of post, where you have the printed route table, there are 4 routes with destination 0.0.0.0/0 which are the routes to the internet.
I don't see the rules for these on the second part where you have posted the export of the route table.
From what I can see on the first part, you have interface specified as gateway, and that configuration will not work with interface as gateway, it does need IP addresses.
There are also a lot of blackhole routes. What is the purpose of those?

Please, try to post the export of what is at our interest, there are a lot of routing rules for my eyes (getting old I know )

Here... let me break things up to be more readable. Sorry about that.

First... These routes force traffic to specific destinations over specific routes. Either via the CABLE connection with gateway 67.253.120.1 or via the LTE gateway at 192.168.50.1. The blackholes are there because if the specific interface is down it will revert back to using the backup route via the 0.0.0.0 routes and we don't want to do that. I have 6 of these destinations (3 for each route). This is a bit overkill vs the recommended two in the wiki.

add comment=LTE distance=1 dst-address=1.0.0.1/32 gateway=192.168.50.1 scope=10
add distance=20 dst-address=1.0.0.1/32 type=blackhole
add comment=CABLE distance=1 dst-address=1.1.1.1/32 gateway=67.253.120.1 scope=10
add distance=20 dst-address=1.1.1.1/32 type=blackhole
add comment=LTE distance=1 dst-address=4.2.2.2/32 gateway=192.168.50.1 scope=10
add distance=20 dst-address=4.2.2.2/32 type=blackhole
add comment=LTE distance=1 dst-address=208.67.220.220/32 gateway=192.168.50.1 scope=10
add distance=20 dst-address=208.67.220.220/32 type=blackhole
add comment=CABLE distance=1 dst-address=208.67.222.222/32 gateway=67.253.120.1 scope=10
add distance=20 dst-address=208.67.222.222/32 type=blackhole
add comment=CABLE distance=1 dst-address=216.239.35.0/32 gateway=67.253.120.1 scope=10
add distance=20 dst-address=216.239.35.0/32 type=blackhole

Next are the "virtual" hops described in the wiki. One for Cable (preferred) and one for LTE (backup). These are the 0.0.0.0 you were asking about.

add comment=VirtualCable distance=2 gateway=10.1.1.1
add comment=VirtualLTE distance=3 gateway=10.2.2.2

Next we add the gateways (via recursive routing) for the virtual hops with the check-gateways.

add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=208.67.222.222 scope=10
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=1.1.1.1 scope=10
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=216.239.35.0 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=1.0.0.1 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=4.2.2.2 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=208.67.220.220 scope=10

You can ignore the rest of these as they serve other purposes. Hope that clears things up. Request remains... I don't think a route should become active that has check-gateway=ping if the corresponding gateway cannot be pinged.

add distance=5 gateway=PureVPN-PPTP routing-mark=PPTP
add check-gateway=ping comment=LTE distance=1 gateway=192.168.50.1 routing-mark=LAPTOP
add comment="LTE PureVPN-PPTP" distance=1 dst-address=45.74.35.0/24 gateway=192.168.50.1
add comment="PureVPN-PPTP Blackhole" distance=20 dst-address=45.74.35.0/24 type=blackhole
add distance=1 dst-address=192.168.50.0/24 gateway=ether5-gateway
add comment="Cable Modem" distance=1 dst-address=207.32.194.24/32 gateway=67.253.120.1
add comment="Cable Modem blackhole" distance=20 dst-address=207.32.194.24/32 type=blackhole

This is how I see it: So, maybe the problem is with the interface going down. I don't have a lab to test it, but I suspect it is about the interface going down. Normally the route should remain Static Active. What makes a route not active is the interface going down, and when it is not active it is not checking the gateway. When interface comes up, causes the route to become active and starts pinging. During this phase it can become reachable.
Can you verify this?

Anyway, I would start the implementation with one Host to check (I've done make successful backup routes with one Host per link) and if that is okay pass to the double recursive solution.

Yes that is exactly what happens. Interface goes down and the routes become inactive. When the interface comes back up they become active again even though the gateway IPs cannot be pinged. It then takes 20 seconds before they become inactive again (two 10 second interval pings have to fail).

My "ask" is that a route with check-gateway=ping should NOT go from inactive -> active unless the ping successfully passes even if the reason the route became inactive was due to the interface going down.

BTW - everything "works" it's just that during the recovery phase I lose internet access for about a minute while it switches over. This is because it needs to timeout the check-gateway pings. If RouterOS did the gateway check ping I'm requesting here I believe the cutover would be almost seamless.

It is a seamless process, I have done many with desired result and no issues.
That is why I recommend to start with a basic setup with only one host per link to check. Also, why should the interface of the modem go down? The modem might lose connection but the ethernet link should remain up. I suspect that ethernet link coming up triggers the route to become active before it can check for a ping.

Every so often the cable modem reboots (I think due to maintenance by the cable provider). Hence why the interface goes down is out of my control.

Either way is my request unreasonable? i.e. a route with check-ping should only become active if the gateway IP is ping-able?

Adam

Either way is my request unreasonable? i.e. a route with check-ping should only become active if the gateway IP is ping-able?

No, not unreasonable at all . It should become active only if ping is successful.
My suspicion though (since I haven't tested such a thing), is about the ethernet link. As you know, a route does have an IP as gateway and interface attached to it. When either of them goes down makes the route inactive. Maybe the interface link coming up makes the route active before it starts pinging the gateway, I have to find time to test it if this is true or not.