Community discussions

MikroTik App
 
powiadamiacz
just joined
Topic Author
Posts: 8
Joined: Mon Feb 03, 2020 6:48 pm

ECMP LoadBalancing

Fri May 22, 2020 3:49 pm

Hello everyone ..
i have configured two microtic routers with multipath (ECMP)

everythink wors fine BUT :)

scenario1
currently i have 3 routes to the same network. when i make the test
i make iperf from client1 -> server 1 on port 5901 -> router take routes path no1
i make iperf from client1 -> server 1 on port 5902 -> router take route path no2
i mak iperf from client1 -> server1 on port 5903 -> router take route path no3

scenario2
if i change setination IP
i make iperf from client1 -> server1 on port 5901 -> router take route path no1
i make iperf from cleitn1 ->server1:1(alias interface- other IP) on port 5902 -> router take route path no2
i make iperf from cleitn1 ->server1:2(alias interface - other IP ) on port 5903 -> router take route path no3

all tests was prepared at the same time.
conclusion is that ECMP works (in my case) per-src-dst-address combination load balancing.
now question. is there any way to change some settings that Mikrotic can take different route in traffic on the same IPS (src & dst) but other ports ? (in my case scenario no 1)
 
tberg
newbie
Posts: 41
Joined: Sun Sep 22, 2013 6:34 pm

Re: ECMP LoadBalancing

Fri May 22, 2020 5:36 pm

Not sure if there are other ways to do this, but one is to use firewall mangle rules and PCC (per connection classifier). Here is a link to a MUM presentation on the topic. https://mum.mikrotik.com/presentations/ ... 038928.pdf
 
sindy
Forum Guru
Forum Guru
Posts: 5378
Joined: Mon Dec 04, 2017 9:19 pm

Re: ECMP LoadBalancing

Fri May 22, 2020 5:44 pm

Yes, but not using ECMP. ECMP does not look at port numbers, period.

So you need routing marks and mangle rules assigning them if you want to distribute the traffic with same src and dst addresses across multiple routes.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
powiadamiacz
just joined
Topic Author
Posts: 8
Joined: Mon Feb 03, 2020 6:48 pm

Re: ECMP LoadBalancing

Fri May 22, 2020 9:43 pm

Yes, but not using ECMP. ECMP does not look at port numbers, period.

So you need routing marks and mangle rules assigning them if you want to distribute the traffic with same src and dst addresses across multiple routes.
Can You tell me how to do that ?
from example page there is only this
"/ ip firewall mangle
add chain=input in-interface=wlan1 action=mark-connection new-connection-mark=wlan1_conn
add chain=input in-interface=wlan2 action=mark-connection new-connection-mark=wlan2_conn
add chain=output connection-mark=wlan1_conn action=mark-routing new-routing-mark=to_wlan1
add chain=output connection-mark=wlan2_conn action=mark-routing new-routing-mark=to_wlan2
"
but this is reposible for symetric answear (packet incoming from wlan1 must return using the same interface)
 
sindy
Forum Guru
Forum Guru
Posts: 5378
Joined: Mon Dec 04, 2017 9:19 pm

Re: ECMP LoadBalancing

Fri May 22, 2020 11:51 pm

Can You tell me how to do that ?
Have a look at this recent post. If it is not 100% clear from there, ask further questions here.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
powiadamiacz
just joined
Topic Author
Posts: 8
Joined: Mon Feb 03, 2020 6:48 pm

Re: ECMP LoadBalancing

Sat May 23, 2020 1:08 am

Can You tell me how to do that ?
Have a look at this recent post. If it is not 100% clear from there, ask further questions here.
currently i have config as follow (there is better but still the traffic are not evenly divided. (as attachment sent 3 screens - 1) iperf to 5901 port 3 threads, 2)iperf to 5902 port 3 threads, 3) ipers to 5903 port 3 threads) - look at mark connection (not symetric)
Screenshot from 2020-05-22 23-57-08.png
Screenshot from 2020-05-22 23-57-59.png
Screenshot from 2020-05-22 23-58-40.png


/interface gre
add allow-fast-path=no keepalive=2s,2 local-address=172.31.13.2 name=I_WAN1 \
remote-address=172.31.16.2
add keepalive=2s,2 local-address=172.31.12.2 name=I_WAN2 remote-address=\
172.31.26.2
add keepalive=2s,2 local-address=172.31.11.2 name=I_WAN3 remote-address=\
172.31.36.2

/ip address
add address=172.31.50.1/30 interface=I_WAN1 network=172.31.50.0
add address=172.31.50.9/30 interface=I_WAN2 network=172.31.50.8
add address=172.31.50.13/30 interface=I_WAN3 network=172.31.50.12

/ip firewall address-list
add address=172.31.0.0/24 list=LAN

/ip firewall mangle
add action=mark-connection chain=prerouting comment="PCC rules" \
connection-mark=no-mark dst-address-list=!LAN dst-address-type=!local \
new-connection-mark=LAN-to-WAN1 passthrough=yes \
per-connection-classifier=both-addresses-and-ports:3/0 src-address-list=\
LAN
add action=mark-connection chain=prerouting connection-mark=no-mark \
dst-address-list=!LAN dst-address-type=!local new-connection-mark=\
LAN-to-WAN2 passthrough=yes per-connection-classifier=\
both-addresses-and-ports:3/1 src-address-list=LAN
add action=mark-connection chain=prerouting connection-mark=no-mark \
dst-address-list=!LAN dst-address-type=!local new-connection-mark=\
LAN-to-WAN3 passthrough=yes per-connection-classifier=\
both-addresses-and-ports:3/2 src-address-list=LAN
add action=mark-routing chain=prerouting comment=\
"Mark routing for upload packets from marked connections" \
connection-mark=LAN-to-WAN1 dst-address-list=!LAN new-routing-mark=WAN1 \
passthrough=no src-address-list=LAN
add action=mark-routing chain=prerouting connection-mark=LAN-to-WAN2 \
dst-address-list=!LAN new-routing-mark=WAN2 passthrough=no \
src-address-list=LAN
add action=mark-routing chain=prerouting connection-mark=LAN-to-WAN3 \
dst-address-list=!LAN new-routing-mark=WAN3 passthrough=no \
src-address-list=LAN

/ip route
add distance=3 dst-address=172.31.19.0/24 gateway=I_WAN1 routing-mark=WAN1
add distance=2 dst-address=172.31.19.0/24 gateway=I_WAN2 routing-mark=WAN2
add distance=1 dst-address=172.31.19.0/24 gateway=I_WAN3 routing-mark=WAN3
add check-gateway=ping distance=1 dst-address=172.31.19.0/24 gateway=\
I_WAN3,I_WAN2,I_WAN1
You do not have the required permissions to view the files attached to this post.
 
msatter
Forum Guru
Forum Guru
Posts: 1710
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: ECMP LoadBalancing

Sat May 23, 2020 2:21 am

If you want an exact distribution then the use of PCC is not the best. Also think about when you spread traffic, don't define the last line and just mark what is left over. You create then also a catch-all in case one of you earlier lines are not working.

I have spent an posting on this and have a look at NTH and that will spread evenly every connection.

viewtopic.php?f=2&t=159234&p=782225&hilit=nth#p782225

Use 3:1 then 2:1 and 1:1 and 1:1. 1:1 Is then the one you can omit and it collects all what is left and don't forget to set passthrough to NO because then the same connection will be evaluated again and get an other mark.

You are marking connections so you only have to do that once when it is a new connection status. If you need connections and routing then first mark the connections and then the routing based on the connection marking. As soon as you process a connection in any way the new connection status is removed and you loose the trigger to only mark new connections.
One RB4011 (cooled) and a RB760iGS (hEX S) in series. The 4011 Does PPPoE/IKEv2.
The cooler: viewtopic.php?f=3&t=138613&start=300#p799879
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.14
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 987
Joined: Tue Oct 11, 2005 4:53 pm

Re: ECMP LoadBalancing

Sat May 23, 2020 3:21 am

One way to evenly distribute the traffic among all interfaces with using pure ECMP (no routing marks, etc), is by disabling route cache in IP > Settings.

This will make each packet to go through a different interface in round-robin fashion (I think), instead of per src/dst ip.
So even a single connection can utilize all interfaces.

Of course disabling route cache may have other side-effects, even break features altogether, depending on the rest of your configuration on both sides.
 
sindy
Forum Guru
Forum Guru
Posts: 5378
Joined: Mon Dec 04, 2017 9:19 pm

Re: ECMP LoadBalancing

Sat May 23, 2020 8:55 am

@powiadamiacz, a statistics from just a few samples is no statistics at all. You'd have to create thousands of connections with different source an destination addresses and ports to see the actual evenness of their distribution across the rules/routes. The key here is how the 32-bit hash is generated from the total of 96 bits of the two IP addresses and ports.

What @msatter suggests is a more deterministic approach which distributes the connections
evenly among the paths - 1/3 of the total to the first, 1/2 of the remaining 2/3, which is another 1/3, to the second, and the rest, which is yet another 1/3, to the third. But that still says nothing about traffic volume distribution, as the total amount of data transported by any given connection is unpredictable.

@Cha0s, what you suggest would likely work, but sending individual packets of the same TCP session via different physical paths is what 9 of 10 dentists advice against as that may cause packets to arrive to destination in shuffled order. So a good exercise but a possible headache if deployed in production.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
powiadamiacz
just joined
Topic Author
Posts: 8
Joined: Mon Feb 03, 2020 6:48 pm

Re: ECMP LoadBalancing

Sat May 23, 2020 9:40 am

Aaa - one more

Regarding the settings below - the trafic not working when i remove the last line - ( i thought that this line is not nessesery ) - can you explain me ? :/

ip route
add distance=3 dst-address=172.31.19.0/24 gateway=I_WAN1 routing-mark=WAN1
add distance=2 dst-address=172.31.19.0/24 gateway=I_WAN2 routing-mark=WAN2
add distance=1 dst-address=172.31.19.0/24 gateway=I_WAN3 routing-mark=WAN3
add check-gateway=ping distance=1 dst-address=172.31.19.0/24 gateway=\
I_WAN3,I_WAN2,I_WAN1
 
msatter
Forum Guru
Forum Guru
Posts: 1710
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: ECMP LoadBalancing

Sat May 23, 2020 10:59 am

I am not a routing specialist but with distance and specific WAN you create a problem.

The connection with the shortest distance gets priority to transport traffic but then only for WAN3 traffic leaving WAN2+3 to be catched by the last routing rule.

Why not only use the routing-mark and set the distance to one for all three.
One RB4011 (cooled) and a RB760iGS (hEX S) in series. The 4011 Does PPPoE/IKEv2.
The cooler: viewtopic.php?f=3&t=138613&start=300#p799879
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.14
 
sindy
Forum Guru
Forum Guru
Posts: 5378
Joined: Mon Dec 04, 2017 9:19 pm

Re: ECMP LoadBalancing

Sat May 23, 2020 11:01 am

Does it stop working for iperf running outside the router or you've only tried pinging from the router itself? Packets sent by the router itself do not pass through the prerouting path but through the output one.

So if there is no other route to 172.31.19.0/24 (a default one would be sufficient) in routing table main (the one chosen if no routing-mark is assigned), it would explain this behaviour for locally originated packets only.

But even if you add connection-marking and routing-marking to chain output of mangle, it still won't work without a working route in table main because the handling of locally originated packets is a bit complex. A locally originated packet is first routed and only then mangled, and if mangling eventually results in assignment of a routing-mark, it gets routed one more time. So if the first routing step using table main finds no route, the further steps do not happen.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
sindy
Forum Guru
Forum Guru
Posts: 5378
Joined: Mon Dec 04, 2017 9:19 pm

Re: ECMP LoadBalancing

Sat May 23, 2020 11:08 am

I am not a routing specialist but with distance and specific WAN you create a problem.

The connection with the shortest distance gets priority to transport traffic but then only for WAN3 traffic leaving WAN2+3 to be catched by the last routing rule.
@msatter, this is a misleading statement. The distance value is only used to choose among routes with identical routing-mark and identical dst-address.

So if you have dst-address=0.0.0.0/0 distance=1 and dst-address=1.2.3.0/24 distance 10, the latter one will be used to deliver a packet to 1.2.3.4. Both routes match, but the most narrowly matching routes are always chosen.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
msatter
Forum Guru
Forum Guru
Posts: 1710
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: ECMP LoadBalancing

Sat May 23, 2020 11:39 am

@sindy Only the routing-mark differs and the destination address is in all four lines the same. Which is not present if run directly from the router as you wrote.
One RB4011 (cooled) and a RB760iGS (hEX S) in series. The 4011 Does PPPoE/IKEv2.
The cooler: viewtopic.php?f=3&t=138613&start=300#p799879
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.14
 
sindy
Forum Guru
Forum Guru
Posts: 5378
Joined: Mon Dec 04, 2017 9:19 pm

Re: ECMP LoadBalancing

Sat May 23, 2020 11:43 am

@sindy Only the routing-mark differs and the destination address is in all four lines the same.
Differing routing-mark means different routing table, so it always overrides distance as I wrote above.

If no route with the required routing-mark matches, routing table main is used instead (unless you explicitly prohibit such a fallback using /ip route rule action=lookup-only-in-table table=therouting-mark-value), but that's another can of worms.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 987
Joined: Tue Oct 11, 2005 4:53 pm

Re: ECMP LoadBalancing

Sat May 23, 2020 3:57 pm

@Cha0s, what you suggest would likely work, but sending individual packets of the same TCP session via different physical paths is what 9 of 10 dentists advice against as that may cause packets to arrive to destination in shuffled order. So a good exercise but a possible headache if deployed in production.
This method is not for every scenario. Obviously.
And obviously you don't use this on anything other than fixed/wired network with predictable performance (ie no congestion that can cause out of order packets etc).

And it can work. I've used this method for years in production where I needed to distribute traffic among multiple interfaces but NOT touch (in any way shape or form) the dreaded Connection Tracking Table.

As I said, this method is not for every scenario.

Who is online

Users browsing this forum: anon31337, Google [Bot], matxanh and 88 guests