I have set up RR on many devices, I have one where it is not failing over despite target device being unreachable.
so
pilot route is 1.1.1.1/32 via 192.168.10.1 scope 10 check gateway ping
recursive route is 0.0.0.0/0 via 1.1.1.1 target scope 30 check gateway ping
But the recursive route is not going to unreachable status even though the 1.1.1.1 is unreachable. When I ping 1.1.1.1 i get ICMP Net Unreachable from the upstream router, is the fact that I get an ICMP response causing the route to stay active?
If I change the target gateway for 1.1.1.1 to .2 (so it is invalid) then the recursive route goes to USI status as I would expect.
(we have an ISP outage which has highlighted the issue so testing is limited at present)
This target-scope=30 is what causing the observed behavior. When you specify a value that large, the other fallbackdst-address=0.0.0.0/0 route can also be used to reach 1.1.1.1 (because its scope is 30, which satisfies the "less than or equal 30" condition dictated by target-scope=30) . So 1.1.1.1 is pingable even when the gateway 192.168.10.1 is down.
That's why when you configure recursive routes, you should keep target-scope as small as strictly required. Refer to the guide linked by @jaclaz above for some sane value. If you apply the guide, the dst-address=1.1.1.1/32 gateway=192.168.10.1 route will have scope=11, and the route with dst-address=0.0.0.0/0 gateway=1.1.1.1 route will have the strict minimum possible target-scope=11 too.
That way that route cannot accidentally use the other fallback route, which has dst-address=0.0.0.0/0 scope=30 to lookup 1.1.1.1. Because 30 is larger than the condition target-scope=11.