After disabling those routes, still having issues with existing/new connections. I can make WAN2 fail in simulation, still working on WAN1 simulation. Here is some data:
What kind of issues in particular? Some connections do not get through although you assume they should, some connections do get through although you assume they should not, or some connections establish via a WAN that is simulated to be down?
Both print outputs show that everything works as expected:
when both WANs are simulated to be OK, routes 6 and 9 are active because their recursive gateways are reachable, and routes 7 and 8 are not used (not marked as Active) simply because they have higher distance values than 6 and 9 ones.
when WAN 2 is simulated to be down, routes 6 and 8 are active whereas 7 and 9 are marked as Inactive because their recursive gateway (74.6.143.25) is unreachable. So even though route 8 has higher distance than route 9, it becomes Active.
So what is the issue when WAN 2 is simulated to be down, i.e. what behaves different than you expect?
As for simulation of WAN 1 failure, it is enough to add the following firewall rule:
This will prevent the check-gateway pings from reaching the recursive gateway, having the same effect as if it becomes unreachable due to some issue further away in the network.
So what is the issue when WAN 2 is simulated to be down, i.e. what behaves different than you expect?
Existing clients connected have no path to internet. They are unable to re-establish existing connections in most cases. Seems like the router is still trying to send traffic to the down WAN.
Connections that are established at the moment WAN2 goes down cannot continue if there is NAT, because the connection tracking keeps sending the packets belonging to these connections out with the source address of the WAN2 interface even though they are routed via WAN1. So either the routers on the path to the destination drop them, or the server sends the response to this private address and thus they are routed to that private address within the context of the server’s network, so they cannot reach your router neither via WAN 1 nor via WAN 2. The clients have to initiate new connections whose first packet will be sent via WAN 1 and NATed accordingly. E.g. in case of ping, you have to wait 10 seconds so that the tracked connection is forgotten, and then try again.
You can use /tool/sniffer and /ip/firewall/connection/print detail where src-address~“sour.ce.i.p:port” dst-address~“de.st.i.p:port” to visualise this.
Thanks @sindy. So from your perspective, my setup is working as expected based on the information I was able to provide. I’ll I’m going to keep testing scenarios. Truly appreciate all the help this group has provided.
If you can confirm that clients who use routing table to-ISP2 establish their connections via WAN 2 while the internet is reachable through WAN 2, and establish new connections via WAN 1 while it is not, then yes, it works as expected. Including the fact that existing connections via WAN 2 do not automatically continue via WAN 1 after internet becomes unreachable via WAN 2.
10 seconds are ICMP default timeout. For TCP, there are different timeouts depending on the current state of the session (24 h if everything has been ACKed, just 5 min if I remember well if there are unACKed data, etc.). I was referring specifically to ICMP as it is both simple to test and the timeouts are short.
So the current solution is having problems with clients that have active sessions (which we know won’t work) AND trying to establish new connections. If I use the drop script OR another method (disconnect WAN, but gateway remains IP remains up), traffic is still being sent to both WAN’s, even though it looks down in the routing table.
I’m going to re-do this setup using ROSv6. Wonder if something is not ‘baked’ right in ROSv7 just yet. Will report my findings back soon.
ok @sindy so you 're referring to the check-gateway ping timeout…
Yes indeed, for the unacked and retransmitted TCP packets the default timeout is 5 minutes…
But there are as well 10 second timeouts for some TCP wait timeouts etc. and i thought you were referring to them.
Not even that, it’s a coincidence (or maybe not?) that the lifetime of a pinhole (tracked connection) in firewall is 10 s by default for pings, and that the check-gateway pings are also being sent 10 s apart.
I really only suggested a connection type (ping) that is the fastest one to die off from the connection tracking module of the firewall, allowing you to initiate a new connection rather than reusing the existing one very soon after imitating the failure of the uplink - you need to wait at max 10 seconds for the failure to be detected, then stop the outgoing ping for at least 10 seconds so that the pinhole could be dropped, and then ping again to see whether it succeeds via the working uplink or not (or, as @gutowscr471 says to be the case in ROS 7.1, whether it continues to be sent via the uplink that should be down).
I still don’t understand that behaviour, as when the route via WAN 2 in table to-ISP2 becomes inactive because the check-gateway ping fails, there is still the route via WAN 1 in that table, so there is no reason why packets bearing a routing-mark to-ISP2 should fall back to routing table main and use the route via WAN 2 from that table. So maybe the routing-mark is actually not assigned properly, but I’m tired asking people over and over again to post the complete configurations rather than just those parts they deem related to their issue.