Recursive routing not failing over - possible icmp issue

I have set up RR on many devices, I have one where it is not failing over despite target device being unreachable.

so

pilot route is 1.1.1.1/32 via 192.168.10.1 scope 10 check gateway ping

recursive route is 0.0.0.0/0 via 1.1.1.1 target scope 30 check gateway ping

But the recursive route is not going to unreachable status even though the 1.1.1.1 is unreachable. When I ping 1.1.1.1 i get ICMP Net Unreachable from the upstream router, is the fact that I get an ICMP response causing the route to stay active?

If I change the target gateway for 1.1.1.1 to .2 (so it is invalid) then the recursive route goes to USI status as I would expect.

(we have an ISP outage which has highlighted the issue so testing is limited at present)

Check scopes and target-scopes, just in case:
Simple recursive failover for bears of little brain

Also post config vice giving sparse information.

This target-scope=30 is what causing the observed behavior. When you specify a value that large, the other fallback dst-address=0.0.0.0/0 route can also be used to reach 1.1.1.1 (because its scope is 30, which satisfies the "less than or equal 30" condition dictated by target-scope=30) . So 1.1.1.1 is pingable even when the gateway 192.168.10.1 is down.

That's why when you configure recursive routes, you should keep target-scope as small as strictly required. Refer to the guide linked by @jaclaz above for some sane value. If you apply the guide, the dst-address=1.1.1.1/32 gateway=192.168.10.1 route will have scope=11, and the route with dst-address=0.0.0.0/0 gateway=1.1.1.1 route will have the strict minimum possible target-scope=11 too.

That way that route cannot accidentally use the other fallback route, which has dst-address=0.0.0.0/0 scope=30 to lookup 1.1.1.1. Because 30 is larger than the condition target-scope=11.

Initially the target scop WAS set to 10 (which doesn't work on newer versions as scope has to be less than targetscope), so it was set to 11.

In any case I redid the route setup and it all started working as expected so I am putting it down as a glitch.

This is a setup that has been known to work..