ECMP ROUTE RoS V7 buG

hello sir,

I need use route ecmp in my router RoS V7

I have 2 ISP upstream

isp A = 100Mbps gw 192.168.19.1

isp B = 300Mbps gw 192.168.20.1

I have done add route

And why isn't the load balancing system working? 25% to ISP A and 75% to ISP B?

In reality, it's 1:1.

I was hoping for 25% to ISP A and 75% to ISPB.

Previously, I implemented it in RoS v6 and it worked normally.

However, in RoS v7, it doesn't work like in V6.

Please, is there any additional configuration to enable the load balance ratio calculation?

You won’t be able to do a 25%/75% on BGP if the dst-ip is the same, but that being said, ecmp is troublesome on BGP and not properly implemented - it is a known issue that’s been screamed and shouted at for many, many years.

sorry, we not use BGP sir,

I think it's because with your four copies of the same route having the same next-hops, when the reachable next-hops are copied to the FIB they will only create one next-hop entry. Which means in the end when the "best route" for destination 0.0.0.0/0 is installed into the FIB, it's one ECMP route with only two next-hop entries. You'll only get a 1:1 distribution.

If you have control of the gateway having the 192.168.20.1 address, you could, for example, add multiple IP addresses to it's interface facing you router, instead of just 192.168.20.1, add 192.168.20.2, 192.168.20.3, 192.168.20.4 (the interface has 4 IP addresses). Then on your MikroTik router, add the 4 default routes having gateways 192.168.20.1, 192.168.20.2, 192.168.20.3, 192.168.20.4.

Read this: IP Routing - RouterOS - MikroTik Documentation especially the Multipath (ECMP) routes, Nexthop Lookup, Forwarding Information Base sections.

from official site mikrotik make it like i made it, no problem with me being the same https://mikrotik.co.id/artikel/492/

That's not the official MikroTik site. That's some company named Citraweb from Indonesia

I think this is the last solution.
I'll try to find out how to create an IP alias on my modem, or maybe add a router near the modem.:grimacing:

Here is a test that you can do without the need to have devices making connections and mesuring bandwidth:

  1. Add some bogus ECMP routes:

    /ip route
    # with gateway 192.168.88.10
    add dst-address=10.7.8.0/22 gateway=192.168.88.10
    add dst-address=10.7.8.0/22 gateway=192.168.88.10
    add dst-address=10.7.8.0/22 gateway=192.168.88.10
    add dst-address=10.7.8.0/22 gateway=192.168.88.10
    add dst-address=10.7.8.0/22 gateway=192.168.88.10
    add dst-address=10.7.8.0/22 gateway=192.168.88.10
    # with gateway 192.168.88.11
    add dst-address=10.7.8.0/22 gateway=192.168.88.11
    add dst-address=10.7.8.0/22 gateway=192.168.88.11
    

    That's assuming you have a connected route with destination 192.168.88.0/24 (your bridge for example).

  2. The /ip route check command can be used to find out which next-hop will be chosen for a destination, so we'll write a script that just counts how many times 192.168.88.10 and 192.168.88.11 are chosen for the over 1000 destination addresses in the 10.7.8.0/22 range (10.7.8.1->.254, 10.7.9.1->.254, 10.7.10.1->.254, 10.7.11.1->.254):

    {
        :local bucket [:toarray ""];
    
        for cntr1 from=8 to 11 do={
            for cntr2 from=1 to=254 do={
                :local nhop [:tostr ([/ip route check dst-ip="10.7.$cntr1.$cntr2" once as-value]->"nexthop")];
                :set ($bucket->"$nhop") (($bucket->"$nhop") + 1);
            }
        }
    
        :put $bucket;
    }
    
  3. Result output for the 1016 destination addresses:

    192.168.88.10=499;192.168.88.11=517
    

    As you can see, it's nearly a 1:1 ratio. Because the next-hops are distinctly added for the ECMP route, and not 6 × 192.168.88.10 and 2 × 192.168.88.11 .

And for "science" here is the test, but with 8 distinct gateway addresses:

/ip route
add dst-address=10.7.8.0/22 gateway=192.168.88.11
add dst-address=10.7.8.0/22 gateway=192.168.88.12
add dst-address=10.7.8.0/22 gateway=192.168.88.13
add dst-address=10.7.8.0/22 gateway=192.168.88.14
add dst-address=10.7.8.0/22 gateway=192.168.88.15
add dst-address=10.7.8.0/22 gateway=192.168.88.16
add dst-address=10.7.8.0/22 gateway=192.168.88.17
add dst-address=10.7.8.0/22 gateway=192.168.88.18

Output from the script above:

192.168.88.11=108;192.168.88.12=129;192.168.88.13=129;192.168.88.14=133;192.168.88.15=138;192.168.88.16=133;192.168.88.17=112;192.168.88.18=134

With 8 possible gateways, the perfect distribution is 1016 ÷ 8 = 127 per next hop. And the result is pretty close (it depends on the hashing algorithm)

You might not need the support from the ISP modem or an extra router. Instead, you can try this:

  • On the MikroTik router, go to the IP -> ARP table, look for the entry for 192.168.20.1.
  • Make 3 copies of that entry, and save them but for IP addresses 192.168.20.2, 192.168.20.3, 192.168.20.4 (or other addresses if you have conflicts with existing devices).
  • Update the 4 ECMP 0.0.0.0/0 routes to use gateways 192.168.20.1 , 192.168.20.2 , 192.168.20.3 , 192.168.20.4.

Those addresses don't need to exist on the ISP modem. The MikroTik router only needs to know the MAC address to set as destination for the ethernet frame that it sends to the gateway. The destination IP address field of the packet has the external (on internet) IP addresses anyway, and not 192.168.20.2, or 192.168.20.3, etc...

1 Like

You might want to confirm with /routing/route/print as that shows the FIB to confirm the MAC trick works. I'm not 100% that work...

There is another thread that has more detail on the lack of asymmetric splits (25%/75%): ECMP recursive routes

It does work, here is some test setup I just did:

Router is CHR with minimal config. ether1 currently has DHCP client disabled and does not participate in the test. ether3 is WAN with 172.20.80.1 is the real gateway. Standard DHCP server is setup on ether2 with 192.168.88.1/24. Windows VM is connected to this and get IP address and has internet access (can ping forum.mikrotik.com).

As you can see, the host 172.20.80.11 doesn't exist (nothing answers ping).

Next the the ARP-copies trick is performed. ARP entries for 172.20.80.11-.14 are added. The default route with gateway=172.20.80.1 is disabled, and the 4 ECMP routes to the bogus gateway addresses are added instead.

In this screenshot you can see that the Windows VM has full internet connectivity, although it can't ping 172.20.80.11 (non existant host). The 4 routes are present in the routing table as ECMP routes.

The test with the script from the above post shows "equal" distribution among the 4 next-hops:

Next I enable DHCP client on ether1 and add a masquerade rule. This add another really different default route to the router (gateway 10.1.12.1 on ether1):

Ping test shows that depending on the destination, sometime you get TTL=55 (the ether3 route), sometime TTL=56 (the ether1 route).

And the test from the script above shows the distribution of the 1016 destinations across the 5 possible next-hops.

1 Like

Fair enough, quite the trick. I just wasn't if you were being theoretical :wink: