Dear experts,
I read many interesting posts about recursive routes, first of all thanks to anyone who contributed investing his time to share experiences and help many who couldn’t learn networking.
@anav’s question was “why would you want to use ECMP” while yours was (apparently) “can I use ECMP or do I have to use distinct distance values”?
In ROS 6, you had to specify multiple gateways for the same destination as a list in a single route to get ECMP; in ROS 7, it is enough to configure multiple routes with same destination and distance to get ECMP.
In the specific case of using recursive next-hop search as a way to monitor the transparency of the uplink, it does not really matter whether you set the same or distinct distance to the routes that use the same actual gateway found via distinct canary addresses.
So yes, you can save yourself the burden of specifying a non-default value of distance.
Thanks @sindy, a side question was, if using ECMP, in this case is more CPU consuming and if there is any difference in case (using this example) 1.1.1.1 is not available.
I did some basic tests changing distance=2 for 9.9.9.9 and increasing to 3 for 8.8.8.8 but didn’t find any difference in WAN1 availability blocking 1.1.1.1.
There is no difference in CPU load as the canary addresses are not checked for every packet. The reason for using multiple canary addresses to check the transparency of the same uplink is to avoid false negatives, i.e. declaring the uplink dead although actually only the single canary address is dead. So the fact that the router continues using the uplink if you make 1.1.1.1 unreachable means it works as designed.
Well, using recursive eliminates the situation where connectivity to the ISP is fine but the connection from the ISP to the WWW is not working.
Having a second recursive, eliminates the issue where the connectivity to the ISP is fine, access to the WWW is fine, but the DNS site itself is down.
So, a small nuance, the second recursive, addresses the first DNS site being down BUT ALSO is still viably ensuring that the ISP can talk to the WWW.
( think of the case where the DNS site is down, but also the ISP is not connecting to the WWW )
After further thought this has nothing to do with ECMP, its simply a lazy way of doing recursive.
Since the first two recursives are going via the same gateway it matters little which route is used.
Since if both are available both can be randomly chosen by the router as the active route and will work fine..
Writing in non-native language has it limits.
Curiosity was to understand if, when setting different distance for the two “canary” addresses is enough and when first service is down, second route with increased distance is efficient to provide traffic continuity, or load-balancing the two addresses using ECMP could provide a better experience without sacrificing CPU usage (I still have some doubts on this, due to routing engine should distribute traffic via both active routes).
ECMP should not add significant CPU load over failover.
And it’s the scope/target-scope matter for recursive routes. The distance on the default route (i.e. dst-address=0.0.0.0/0) makes it failover (different distance) or ECMP (same distance for 0.0.0.0/0 routes) - but distance does not effect how recursive routes work. e.g. the first route lookup for a Internet address (0.0.0.0/0) get some canary address, and the 2nd route lookup of that canary is a “more specific route” (i.e. a /32) so that since since there is no conflict, distance= is not used since an exact/unique route match (dst-address=8.8.8.8/32) is found.
It’s when multiple routes match, that distance= is used.
Given that route print detail shows things like gateway-status=10.100.8.8 recursive via my.wan.gw.ip bridge-wan, I would expect the recursive routes to use the actual gateway straight away during the actual routing process, are you sure that the complete recursion path is followed for every single packet being routed?
I speaking more logically with distance=, whether it happens per packet IDK. I just don’t think there is any significant CPU impact based on using ECMP vs failover.
why do you keep on insisting on using the word ECMP. Its called random, or chaotic, there is no methodology for the router to choose route if distances are equal.
Are you telling me the router says Oh I picked ether1 last time, so I am going to be fair and pick ether2 next time???
I mean, if so, then it is the distance of the routes with dst-address=0.0.0.0/0 that plays a role, and the fact that the /32 routes to the canary addresses/virtual gateways have unique dst-addresses and thus their distance is irrelevant is, well… irrelevant
I think the same, but given that I don’t understand what makes ECMP via different WANs still work now as routing cache is gone (someone here has claimed that to be the case and attributed that to connection tracking to cache the gateway, but I never had enough determination to test that), I do not dare to speculate as for CPU cycles spent per packet with and without ECMP using the same WAN.
I do not dare to speculate as for CPU cycles spent per packet with and without ECMP using the same WAN.
Agreed. JustIn the pantheon of worries, ECMP’s CPU usage would not be the top of that list.
To be honest, it’s the using DNS servers as the canary address be my bigger worry. Unless the servers are tightly controlled, if the WAN assigned to 8.8.8.8 goes down ANY client that uses 8.8.8.8 will need either a backup DNS configured, or just fail if the DNS used as canary is on a dead WAN. Even if you assign DNS via DHCP, a client application could always use an alternative DNS - i.e. modern browsers can have their own mind about DNS servers.
No. While it may not be truly equal (and neither is PCC), but ECMP is not random/“chaotic”, it also uses a hash to decide to which route to choose if distance= is same (and same prefix length too AFAIK). The default ECMP hashing is just src+dst address, so it’s not “as random” as PCC if “both-address-and-ports” is used with PCC.
But the newest 7.16rc actually expands ECMP hashing algorithm. So essentially ECMP can do similar “both-addresses-and-ports” as PCC now. See the docs on /ip/settings to control how ECMP:
ipv4-multipath-hash-policy > (l3 | l4 | l3-inner; Default: l3)
IPv4 Hash policy used for ECMP routing in /ip/settings menu
l3 – layer-3 hashing of src IP, dst IP
l3-inner – layer-3 hashing or inner layer-3 hashing if available
l4 – layer-4 hashing of src IP, dst IP, IP protocol, src port, dst port
multipath-hash-policy > (l3 | l4 | l3-inner; Default: l3)
IPv6 Hash policy used for ECMP routing in /ipv6/settings menu
l3 – layer-3 hashing of src IP, dst IP, flow label, IP protocol
l3-inner – layer-3 hashing or inner layer-3 hashing if available
l4 – layer-4 hashing of src IP, dst IP, IP protocol, src port, dst port
Because the word acronym ECMP stands for Equal Cost Multi-Path, which is a name of the mechanism. The fact that someone decides to misuse it to “distribute” the traffic among multiple aliases of the same actual path is another can of worms - same like with PCC, you may legitimately want to send e.g. 60 % of connections one way and the remaining 40 % the other, so you use three routes with one actual gateway and two with another. So using all 5 for the same physical path is just an edge case of the same, nothing illegal.
Where does one invoke this ECMP.
By waving the magic wand??
Okay… ECMP (Equal cost multi-path) routes have multiple gateways (next-hop) values. All reachable next-hops are copied to FIB and are used to forward packets.
In reality we don’t have multiple gateways, its simply the same gateway on the closer hop.
I will agree that the further hop does have different gateway 1.1.1.1 vs 8.8.8.8 for example.
ECMP if does see same destination for different gateway in main, table auto assigns a + symbol, and uses some hash algorithm
Thus, and the crux of my question, there is no entry, config checkbox etc. for specifically ECMP.
I was going by the IP routing statement in documentation that stated… The candidate route with the lowest distance becomes an active route. If there is more than one candidate route with the same distance, the selection of the active route is arbitrary.
However, I could not find where it states explicitly the hashing is used to choose the active route in the case of ECMP. If you can point me too that… I am more apt to believe the fowl bird.
So interesting, nice to see @Amm0 stepping in. @Amm0, based in your comment from 7.16beta topic, are you using ECMP via different WANs without mangle rules?
I probably understood 40% from the more technical posts, is correct to assume that using ECMP (same distance for 0.0.0.0/0 routes), it doesn’t matter and is not useful when using the same WAN due to the fact that, in case one DNS service used as canary address is down, connections are dropped but having a different distance for 0.0.0.0/0 routes will fail-over to the higher one in case the distance=1 is unavailable?
Indeed, how can I have forgotten about this method of pseudo-random distribution that does not require any memory.