ECMP doesn't work for Load balancing

RouterOS General
21

For anyone later that follows this thread;;;;;;;

In this case since there is only one subnet and the prefix rule is basically doing the equivalent to:
/routiing rule
add action=lookup-only-in-table src-address=192.168.88.0/24 dst-address=192.168.88.0/24 table=main

The value in using the min-prefix=0 is that it allows any local subnet to subnet traffic to occur ( from or to source address )

++++++++++
Without any variation of the above rules, such traffic originating from source or responding from source goes out the next “force traffic to somewhere” routing rule.
So the variation is required prior to the force rule.

22

Sorry @anav, but your wording is so advanced that it is confusing even for me, although I know what you actually intended to say.

So let me try myself in a simpler language with more details: In most cases, we need to prevent the traffic towards local destinations from using other routing table than main. Until recently, the only way to do that using routing rules alone was to put (one or several) rules with dst-address matching the local subnets with action=lookup table=main before (above) the rule that matched on src-address and/or interface and action=lookup-only-in-table table=something-else-than-main. So traffic towards local destinations never reached the rule telling it to use another routing table.

The value of the min-prefix attribute is that it allows to use just a single rule for the same purpose, which is universal in terms that you do not need to specify the dst-address to match to manually; instead, it uses the dst-address of the existing routes in the routing table, taking their prefix length (a.k.a. subnet mask) into account.

As an example, let routing table main contain one default route provided by a DHCP client and three routes to “connected subnets”:
dst-address, gateway, distance
0.0.0.0/0, 10.0.0.1, 1
192.168.88.0/24, bridge1, 0
192.168.44.0/24, bridge2, 0
10.11.0.0/22,bridge3, 0

To make sure that traffic towards any of these three local subnets will be routed using table main, we need three rules:
dst-address=192.168.88.0/24 action=lookup table=main
dst-address=192.168.44.0/24 action=lookup table=main
dst-address=10.11.0.0/22 action=lookup table=main

If it is OK to use main for all private destinations, this can be reduced to just two rules:
dst-address=192.168.0.0/16 action=lookup table=main
dst-address=10.0.0.0/8 action=lookup table=main

But with min-prefix, a single rule is sufficient:
min-prefix=0 action=lookup table=main
It will look into table main, find the best matching route for the packet being processed, and if the prefix length (subnet mask) of the dst-address of that route is higher than the min-prefix value, it will tell the routing to use that routing table for the packet; otherwise, it will hand the packet over to the next routing rule as if it did not match it.

23

After ECMP becomes working, I am considering how to configure failover. Here is my config(bold part is the main section):

/ip route
add disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out8 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=120.x.x.x routing-table=cmip_route scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out2 routing-table=ct2 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-table=ct1 scope=30 suppress-hw-offload=no target-scope=10
add gateway=pppoe-out1 routing-table=ecmp
add gateway=pppoe-out2 routing-table=ecmp
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out3 routing-table=ecmp scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out4 routing-table=ecmp scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=119.29.29.29/32 gateway=pppoe-out8 routing-table=main scope=10 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=119.28.28.28/32 gateway=pppoe-out1 routing-table=main scope=10 suppress-hw-offload=no target-scope=10
add check-gateway=ping distance=1 gateway=119.29.29.29 target-scope=11
add check-gateway=ping distance=2 gateway=119.28.28.28 target-scope=11

And you mentioned what the addition of recursive checking of connectivity on ecmp does? Why do all people have to plus one to target-scope when using recursive routing?

Finally I have a requirement that I don’t know if I can implement, currently my configuration failover will transfer all traffic to pppoe-out1 but I would like to transfer it directly to ECMP’s load balancing instead of taking over all traffic through pppoe-out1 alone, after pppoe-out8 fails.


Another ROS issue I don’t understand is why the two things /routing/rules and /ip/route were separated after the 7.x release.

24

Slow down, man :smiley: Leaving aside that the topic title has become totally misleading already long ago, and that you have changed the objective all of a sudden, your bold rules are either incomplete or incorrect.

The whole idea of using the recursive next-hop search to monitor route transparency is to prevent the router from sending traffic via faulty routes. So if the goal is to use the route via pppoe-out8 as long as it works, and if that one has a problem, to evenly distribute the traffic among all the functional pppoe-out1 to pppoe-out4 routes using ECMP, you must monitor all those routes individually. So the overall setup would be

dst-address=ca.na.ry.1/32 gateway=pppoe-out1
dst-address=ca.na.ry.2/32 gateway=pppoe-out2
dst-address=ca.na.ry.3/32 gateway=pppoe-out3
dst-address=ca.na.ry.4/32 gateway=pppoe-out4
dst-address=ca.na.ry.8/32 gateway=pppoe-out8
dst-address=0.0.0.0/0 gateway=ca.na.ry.8 distance=1 check-gateway=ping target-scope=11
dst-address=0.0.0.0/0 gateway=ca.na.ry.1 distance=2 check-gateway=ping target-scope=11
dst-address=0.0.0.0/0 gateway=ca.na.ry.2 distance=2 check-gateway=ping target-scope=11
dst-address=0.0.0.0/0 gateway=ca.na.ry.3 distance=2 check-gateway=ping target-scope=11
dst-address=0.0.0.0/0 gateway=ca.na.ry.4 distance=2 check-gateway=ping target-scope=11

ca.na.ry.X must be a unique address for each WAN.

25

Also its nice to tells us different requirements, but if you dont post the most update config, we have no idea where things are at.
Thus everytime you make changes post the new config
/export file=anynameyouwish ( minus router serial number, any public WANIP information, keys etc..)

Also state all the requirements (the whole plan) as opposed to bits and pieces.

26

@sindy @anav

First of all, thank you very much for your help.
I am in the process of refining my configuration, when I will post a new topic and describe my configuration and my corresponding needs.
I hope I can get your guidance to make my configuration more robust.

27

@Amm0,

Also, I’m pretty sure that statement in the documentation is a simplification the author has used to avoid the need to explain that this requirement (for some route to exist in main) is only related to own outgoing traffic of the router, which indeed has to be routed using table main first, in order to ever get to mangle and eventually hit a rule there that tells it to run through routing again, using some other routing table this time.

28

I haven’t looked at this recently…so pretty sure you’re right. Now I do want to say I got invalid routes in some past V7 RouterOS versions with the rule was violated…

My usual setup is failover in main (distance=1…x, with some combo of 0-2x “ethernet” WAN + 1-2x LTE links), using routing table per WAN plus an “ecmp” route table as here. So it doesn’t come up in practice, since all WANs are in main routing table. Also LTE is an interface route, that might affect the logic too, IDK. I’ll run a test since you have me curious on exactly what works here.

29

Interesting, ECMP sure is much less complex in terms of sharing the load in case one of the WANs is not available.
In load balancing, it can turn into a huge nightmare of extra mangles and routes.

I am not convinced your recursive is correct I would do it this way.
dst-address=0.0.0.0/0 gateway=ca.na.ry.8 distance=1 check-gateway=ping scope=10 target-scope=12 comment=WAN08
dst-address=0.0.0.0/0 gateway=ca.na.ry.1 distance=2 check-gateway=ping scope=10 target-scope=12
dst-address=0.0.0.0/0 gateway=ca.na.ry.2 distance=2 check-gateway=ping scope=10 target-scope=12
dst-address=0.0.0.0/0 gateway=ca.na.ry.3 distance=2 check-gateway=ping scope=10 target-scope=12
dst-address=0.0.0.0/0 gateway=ca.na.ry.4 distance=2 check-gateway=ping scope=10 target-scope=12
++++++++++++++++++++++++++++++++
dst-address=ca.na.ry.8/32 gateway=pppoe-out8 distance=1 scope=10 target-scope=11
dst-address=ca.na.ry.1/32 gateway=pppoe-out1 distance=2 scope=10 target-scope=11
dst-address=ca.na.ry.2/32 gateway=pppoe-out2 distance=2 scope=10 target-scope=11
dst-address=ca.na.ry.3/32 gateway=pppoe-out3 distance=2 scope=10 target-scope=11
dst-address=ca.na.ry.4/32 gateway=pppoe-out4 distance=2 scope=10 target-scope=11

I do appreciate the detail in ca.na.ry.x though. :slight_smile: