Simpler Failover for two Gateways I found working

The newish Netwatch options have not been reported as being used (successfully or not) by anyone AFAICR, everyone seems to be using the old or new ICMP probe.

Anyway some netwatch related discussions are here:
Netwatch ICMP settings
Netwatch tool

The one I personally find the more promising (but that it won't work everywhere, it depends on the connection you have) is the one checking a known hop on the traceroute:

accept-icmp-time-exceeded=yes can be used together with a manually set low ttl value to monitor Internet connectivity, without relying on a specific endpoint.

For example, you can monitor a public IP address, but that address can filter your ICMP request, or just become unreachable itself, if the Netwatch probe is using this address to monitor Internet connectivity this would cause a false alarm.

To make sure you can reach the Internet, it's generally enough to make sure you can reach a device a few routing hops away. Low time to live value will expire in transit to the specified host you want to monitor - each router passing the ICMP packet will subtract "1" from TTL value, upon TTL reaching 0, ICMP "time exceeded" packet will be generated, and sent back to the Netwatch probe. If all other fail thresholds are not broken, this response will be considered a success.

BUT (still personally) I never had 8.8.8.8 fail because reply was refused from the server, of course it may happen, but it must be a very rare case.

Hi,

I've read the documentation of "accept-icmp-time-exceeded=yes" but I'm not sure I understand it well.

What would happen if you enable "accept-icmp-time-exceeded=yes", set a low TTL (like 3 or 4 for example), and then ping google's 8.8.8.8 address in an hypothetical/post-apocalyptic future where Google decides it won't respond to pings because they are saturated?

Would the netwatch script still consider it a "success"?

If it goes through 3 or 4 hops while attempting to reach 8.8.8.8, yes, because TTL will become 0.

As an example, let's see a traceroute from my PC:

C:\>tracert 8.8.8.8

Rilevazione instradamento verso dns.google [8.8.8.8]
su un massimo di 30 punti di passaggio:

 1    <1 ms    <1 ms    <1 ms  [REDACTED] <- Local router/gateway
 2     7 ms     7 ms     6 ms  [REDACTED] <- ISP
 3     8 ms     7 ms     7 ms  [REDACTED] <- PRIVATE-ADDRESS-BBLK-RFC1918-IANA-RESERVED
 4    13 ms    12 ms    12 ms  172.19.184.70 <- PRIVATE-ADDRESS-BBLK-RFC1918-IANA-RESERVED
 5    12 ms    12 ms    12 ms  172.19.177.62 <- PRIVATE-ADDRESS-BBLK-RFC1918-IANA-RESERVED
 6    12 ms    12 ms    12 ms  195.22.196.170 <- still in Italy
 7    12 ms    11 ms    11 ms  142.250.168.148 <- this is already google
 8    13 ms    13 ms    13 ms  192.178.99.215 <- google
 9    13 ms    12 ms    12 ms  108.170.232.181 <- google
10    12 ms    12 ms    12 ms  dns.google [8.8.8.8] <- google

Rilevazione completata.

In this case a TTL set to:
ttl=1 would mean "nothing"
2<=ttl<=6 would mean "ISP is working, cannot say more"
6<ttl<=10 would mean "ISP is working and I can reach across the pond"

So, probably in this case the "sweet spot" would be ttl=7 or 8.

I am assuming that something like the following is roughly accurate:

As I see it the issues (if any) may arise if - for whatever reasons - the number of hops increases or decreases by more than 2.

Thanks a lot for the info!

However, I still have many doubts about this method. What would happen to our "sweet spot" when some hops are just ignoring our packets?

Take a look at this tracert from a Vodafone FTTH router in Spain:

tracert 8.8.8.8

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

1    <1 ms    <1 ms    <1 ms  192.168.40.1
2     *        *        *     Request timed out.
3     *        *        *     Request timed out.
4     8 ms     8 ms     8 ms  212.166.147.222
5     8 ms     9 ms     8 ms  108.170.252.253
6     8 ms     8 ms     9 ms  74.125.37.87
7     9 ms     9 ms     8 ms  dns.google [8.8.8.8]

Trace complete.

What happens with those disrespectful routers that are insultingly ignoring our ping requests? Do they affect the TTL of the packet or not? What would happen if the magic TTL number we painstakingly chose after many deliberations is exactly the hop that doesn't want to respond?

Actually the chance that routers do not send back an informational reply like "time exceeded" is much higher than that they do not reply to a ping. The informational replies are normally restricted to "once every X packets" and it is not know how long the measurement interval is. I would not use that, use only ping and DNS queries.

AFAICU (but nobody seems to want to actually test) a hop is a hop, and TTL will become TTL-1 after having crossed it.

The reply "Request timed out." should substantially mean "I won't tell you who I am, but I will nonetheless forward your packet to the next hop towards destination." what actually happens is that the hop is asked by traceroute "Who are you?" before and besides the actual tracing of the packet and if the hop doesn't respond at all, traceroute prints "Request timed out", but this has nothing to do with the packet TTL being decreased or not, if there is a hop after that one, then the TTL is decreased, otherwise it isn't.

So, in your example, you start with a TTL of 5, you will reach hop #5, i.e. 108.170.252.253 (which is already google).

And BTW I wouldn't call those routers "disrespectful", but rather "tight-lipped" :wink:.

Mmm, with all due respect, I believe that pe1chl is right in this case!

Take a look at how Microsoft implemented the TRACERT tool in Windows:

https://web.archive.org/web/20150314205734/https://support.microsoft.com/en-us/kb/162326

The TRACERT diagnostic utility determines the route taken to a destination by sending Internet Control Message Protocol (ICMP) echo packets with varying IP Time-To-Live (TTL) values to the destination. Each router along the path is required to decrement the TTL on a packet by at least 1 before forwarding it, so the TTL is effectively a hop count. When the TTL on a packet reaches 0, the router should send an ICMP Time Exceeded message back to the source computer.

TRACERT determines the route by sending the first echo packet with a TTL of 1 and incrementing the TTL by 1 on each subsequent transmission until the target responds or the maximum TTL is reached. The route is determined by examining the ICMP Time Exceeded messages sent back by intermediate routers. Note that some routers silently drop packets with expired TTLs and are invisible to TRACERT.

TRACERT prints out an ordered list of the routers in the path that returned the ICMP Time Exceeded message. If the -d switch is used (telling TRACERT not to perform a DNS lookup on each IP address), the IP address of the near- side interface of the routers is reported.

Those "tight-lipped" routers completely destroy my hope of using TTL in the netwatch script! :frowning:

I cannot see anything in your quote saying something different from what I posted.
A hop in traceroute decreases TTL.
A router "invisibile" to traceroute Is not a hop.

The issue I see Is the possible variation of the number of hops because the ISP (or the country network) may choose longer or shorter paths.

I think I'm not explaining myself clearly. I have just made a test here in a house that has FTTH supplied by Vodafone Spain.

I've added the following netwatch script:

/tool netwatch add accept-icmp-time-exceeded=yes disabled=no down-script=":log info "Damn, we are DOWN"" host=8.8.8.8 http-codes="" interval=30s name=prueba test-script="" timeout=5s ttl=3 type=icmp up-script=":log info "Hurrah, we are UP""

And the results are sadly just what pe1chl predicted :frowning:

Total failure :frowning:

However, if I change the TTL to 5, then the script happily shows: "Hurrah, we are UP" in the logs!

Yep.

But with TTL=5 you did not actually reach 8.8.8.8, you should have stopped at 108.170.252.253.

What happens with TTL=4?

And what happens with TTL=6?

I've testead all the TTLs from 1 to 7, here are the results:

TTL=1 hurrah, we are UP
TTL=2 damn, we are DOWN
TTL=3 damn, we are DOWN
TTL=4 damn, we are DOWN
TTL=5 hurrah, we are UP
TTL=6 hurrah, we are UP
TTL=7 hurrah, we are UP

And here's a screenshot of the traceroute to 8.8.8.8 from winbox

The use of addresses like 8.8.8.8 or 1.1.1.1 to test for "reachability of the internet" may seem like a good idea because these are large companies and services that will not be terminated without notice, but actually it is a poor choice because these are "anycast" addresses that are hosted in many places in the world and thus are quite close to your ISP. At my previous ISP they were even hosted in the ISP datacenter itself, and would be reachable even when there was an interruption at the internet exchange...

What is "wrong" (IMHO) is the DOWN with 4.

Also the downs with lower TTL, but these would make little sense anyway.

In your case, the "right" values are 5 and 6, but if you choose 5 and something changes in your ISP "path towards google" and another (silent) hop is inserted, you might have false downs.

Choosing 6 should be more reliable, still ... :grimacing: