Community discussions

 
User avatar
normis
MikroTik Support
MikroTik Support
Topic Author
Posts: 24268
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

ECMP

Tue Apr 07, 2009 9:54 am

We added some additional information and renamed one of the Wiki topic about ECMP:
http://wiki.mikrotik.com/wiki/ECMP_load ... masquerade

Some users complained about connection interruptions every 10-30min. After looking deeper into problem we were able to determine that Linux Kernel initiates a routing table flush every 10min. So after that connections are assigned to gateways once again and may or may not be on the same gateway.

If you have fully routed network (clients address can be routed via all available gateway), change of the gateway will have no ill effect, but in case you use masquerade, change of the gateway will result in change of the packet's source address and connection will be dropped.

This flush was introduced by Linux Kernel developers to eliminate possibility of DoS attacks on your routers.

So ... currently it's not really possible to make a fully working ECMP setup with Masquerade due to a Kernel design issue. As an alternative you can use this: http://wiki.mikrotik.com/wiki/NTH_load_ ... masquerade
No answer to your question? How to write posts
 
wispnz
just joined
Posts: 9
Joined: Sun Mar 15, 2009 7:38 am

Re: ECMP

Tue Apr 07, 2009 2:41 pm

Thank you for the update!

Will be testing it soon.

One question:

How will we modify this for a 3 or more WAN link scenario?

The reason I ask is that most of my sites have 3 or more Internet links per MT box.

Kind Regards,
Arno
 
nicopretorius
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Mon Nov 15, 2004 9:49 am

Re: ECMP

Fri Apr 10, 2009 11:14 am

Hi Normis, thank for the information re ECMP. I have changed my ECMP config to NTH load balancing with masquerade as per the Wiki. Everything seems to work fine except that the load balancing is very skewed towards the "odd" connection.

Below find the config that I have implemented on ROS3.22:
/ip firewall mangle
add action=mark-connection chain=prerouting comment="" disabled=no in-interface=local new-connection-mark=odd passthrough=yes src-address-list=odd
add action=mark-routing chain=prerouting comment="" disabled=no in-interface=local new-routing-mark=odd passthrough=no src-address-list=odd
add action=mark-connection chain=prerouting comment="" disabled=no in-interface=local new-connection-mark=even passthrough=yes src-address-list=even
add action=mark-routing chain=prerouting comment="" disabled=no in-interface=local new-routing-mark=even passthrough=no src-address-list=even
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=odd nth=2,1 passthrough=yes
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=odd nth=2,1 passthrough=yes src-address-list=!even
add action=add-src-to-address-list address-list=odd address-list-timeout=1d chain=prerouting comment="" connection-mark=odd disabled=no in-interface=local
add action=mark-routing chain=prerouting comment="" connection-mark=odd disabled=no in-interface=local new-routing-mark=odd passthrough=no
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=even nth=2,2 passthrough=yes
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=even nth=2,2 passthrough=yes src-address-list=!odd
add action=add-src-to-address-list address-list=even address-list-timeout=1d chain=prerouting comment="" connection-mark=even disabled=no in-interface=local
add action=mark-routing chain=prerouting comment="" connection-mark=even disabled=no in-interface=local new-routing-mark=even passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL1
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL2

/ip route
add check-gateway=arp comment="Route odd ToADSL1" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL1 routing-mark=odd
add check-gateway=arp comment="Route even ToADSL2" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL2 routing-mark=even
add check-gateway=arp comment="Route router traffic ToADSL2" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL2
If I investigate my firewall connections, address-lists and interface loads I see the following:

admin@MikroTik] /ip firewall address-list> print
Flags: X - disabled, D - dynamic
# LIST ADDRESS
0 D odd 192.168.254.2
1 D odd 192.168.0.101
2 D odd 192.168.0.62
3 D even 192.168.0.34
4 D odd 192.168.0.66
5 D odd 192.168.0.32
6 D even 192.168.254.1
7 D odd 192.168.0.15
8 D odd 192.168.0.31
9 D odd 192.168.0.10
10 D even 192.168.0.5
11 D odd 192.168.0.40
12 D odd 192.168.0.63
13 D odd 192.168.0.250

Most of the IP's are assigned to the "odd address list" i.e only 3 out of 14 to even. As a result the load on my odd interface is also much higher than the even interface. Am I missing something or what can be the reason for the "skew allocation"?
 
knects
just joined
Posts: 24
Joined: Tue Dec 02, 2008 1:49 pm

Re: ECMP

Fri Apr 10, 2009 9:25 pm

Nice to know that I was not smoking something funny back in Feb. when I reported this.


http://forum.mikrotik.com/viewtopic.php?f=2&t=29412

I will give this setup a try this weekend and see if it does better. Thanks for finding out the root of the problem.
 
nicopretorius
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Mon Nov 15, 2004 9:49 am

Re: ECMP

Sat Apr 11, 2009 10:38 am

I have changed my config to the one listed below in order to "resolve my problem" where my "odd address list" contained about 3-4 times more addresses than my "even address list"
/ip firewall mangle
add action=mark-connection chain=prerouting comment="" disabled=no in-interface=local new-connection-mark=even passthrough=yes src-address-list=even
add action=mark-routing chain=prerouting comment="" disabled=no in-interface=local new-routing-mark=even passthrough=no src-address-list=even
add action=mark-connection chain=prerouting comment="" disabled=no in-interface=local new-connection-mark=odd passthrough=yes src-address-list=odd
add action=mark-routing chain=prerouting comment="" disabled=no in-interface=local new-routing-mark=odd passthrough=no src-address-list=odd
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=odd nth=4,1 passthrough=yes
add action=add-src-to-address-list address-list=odd address-list-timeout=1d chain=prerouting comment="" connection-mark=odd disabled=no in-interface=local
add action=mark-routing chain=prerouting comment="" connection-mark=odd disabled=no in-interface=local new-routing-mark=odd passthrough=no
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=even nth=4,2 passthrough=yes
add action=add-src-to-address-list address-list=even address-list-timeout=1d chain=prerouting comment="" connection-mark=even disabled=no in-interface=local
add action=mark-routing chain=prerouting comment="" connection-mark=even disabled=no in-interface=local new-routing-mark=even passthrough=no
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=even nth=4,3 passthrough=yes
add action=add-src-to-address-list address-list=even address-list-timeout=1d chain=prerouting comment="" connection-mark=even disabled=no in-interface=local
add action=mark-routing chain=prerouting comment="" connection-mark=even disabled=no in-interface=local new-routing-mark=even passthrough=no
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=even nth=4,4 passthrough=yes
add action=add-src-to-address-list address-list=even address-list-timeout=1d chain=prerouting comment="" connection-mark=even disabled=no in-interface=local
add action=mark-routing chain=prerouting comment="" connection-mark=even disabled=no in-interface=local new-routing-mark=even passthrough=no


/ip firewall nat
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL1
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL2


/ip route
add comment="Router traffic To ADSL1" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL1
add check-gateway=arp comment="Route odd ToADSL1" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL1 routing-mark=odd
add check-gateway=arp comment="Route even ToADSL2" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL2 routing-mark=main
With the above config I do not get any duplication of addresses in the different address lists and I had a split of 8/7 (i.e. out of a total of 15 IP's) addresses respectively in my odd/even address lists.
 
knects
just joined
Posts: 24
Joined: Tue Dec 02, 2008 1:49 pm

Re: ECMP

Sat Apr 11, 2009 3:03 pm

Yes, this seems to be doing much better. I successfully downloaded an entire ISO without dropping. Obviously the flushing of the routing table was causing my issues. This solution is Ok, but the ECMP was better in that opening multiple connections from one host gave more "bandwidth" to that host since some of those connections would go over both links. I have not fully completed my setup but it does split the connections based on IP. I used the above posters mangle rules but alas I have less hosts at this location so could not really see if I was getting odd or even marks distributed better. Once I have this locked down I will try one of my other test sites and report back.


Will there be any changes made by Mikrotik to enable us to use the ECMP solution with masqueraded connections again? I really liked the connections getting balanced. Thanks for tearing it to this more deeply.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1369
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: ECMP

Sat Apr 11, 2009 3:21 pm

Does it affect the TCP performance when the routing table is flushed? Do we have packet drops or packet delays because of it?

To our Latvian friends: I see you have added my "Connections to the router itself" config that we developed with mcgaiver, to the ECMP WiKi but the editing is disabled so I was not able to add my other part of the config that works around the flushing problem so connections stay on their proper gateways even after flushed....

Oh and by the way, could we use Private Messages on this forum again please?
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
knects
just joined
Posts: 24
Joined: Tue Dec 02, 2008 1:49 pm

Re: ECMP

Sat Apr 11, 2009 6:09 pm

If you wanted to post it I would be happy to test it. My only problem with ECMP was the routing table flushing. If it was posted in your other thread, I missed it :(.
 
alphahawk
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Fri Mar 28, 2008 6:40 pm

Re: ECMP

Sun Apr 12, 2009 8:37 am

Is there a way to simulate flushing the routes?
 
iraq_soul
just joined
Posts: 17
Joined: Thu Oct 26, 2006 1:24 pm
Location: iraq

Re: ECMP

Sun Apr 12, 2009 12:04 pm

hi,
i have rb1000 and i have 2 wan and 1 local ip .... i have tried these setting but all the connection was odd or even there is no balancing and only one wan working ...
i dont want to balance between them because one wan is 4Mb and the other is 1Mb
is there any setting to do this ???
thanks.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1369
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: ECMP

Sun Apr 12, 2009 12:11 pm

I think the best way to go is to use ECMP which can balance between different links (4M and 1M) distributing across all links perfectly, and implement the workaround of the routing table flushing that I am using since months already. In the setups of all my remote clients (that contact me via givememorebandwidth AT gmail DOT com with WinBox login info) it works perfectly, so it is tested.


Setting up a proper ECMP route gives you controlled way of balance, you can balance 4/5 of all connections to one gateway and 1/5 to another. It also provides good fail-over - it can detect whether the gateway IP replies to icmp or arp requests, it can work with gateway interfaces instead of gateway IP addresses, making it perfect in the case with one ISP/same gateway/multiple connections. Another good thing about it is that when one user downloads with more TCP connections, they are distributed across the gateways so he can have ALL the bandwidth for himself when he needs it.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
iraq_soul
just joined
Posts: 17
Joined: Thu Oct 26, 2006 1:24 pm
Location: iraq

Re: ECMP

Sun Apr 12, 2009 12:40 pm

Thank you NetworkPro for your assistance and thank you for your fast reply
 
knects
just joined
Posts: 24
Joined: Tue Dec 02, 2008 1:49 pm

Re: ECMP

Sun Apr 12, 2009 5:20 pm

I actually am using gateway names with the nth config too. This configuration has been solid. I will be rolling this to a few test sites and then going back to the ECMP configuration. I will figure out the way to work around the flushes and post it for everyone.
 
nicopretorius
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Mon Nov 15, 2004 9:49 am

Re: ECMP

Sun Apr 12, 2009 6:26 pm

I have come across another nth load balancing example that was posted by Janisk:

http://wiki.mikrotik.com/wiki/NTH_load_ ... pproach%29

I have also implemented his approach. The distribution between the IP addresses with this example is better than the http://wiki.mikrotik.com/wiki/NTH_load_ ... masquerade example.

I agree that nth load balancing is not as granular as ECMP. However, it does not have any of the issues that requires one to implement policy routing to make ECMP work properly. Furthermore once you have forced certain connections via a particular interface with policy routing your load balacing is anyway skewed and then you are still uncertain whether you have catered for all "exceptions". Also as far as I'm concerned multiple PPTP connections will not work properly with ECMP because of the issues with policy routing PPTP.
/ip firewall mangle
add action=add-src-to-address-list address-list=ADSL1 address-list-timeout=0s chain="mark new unseen" comment="" disabled=no nth=2,1
add action=add-src-to-address-list address-list=ADSL2 address-list-timeout=0s chain="mark new unseen" comment="" disabled=no nth=2,2
add action=add-src-to-address-list address-list=seen address-list-timeout=0s chain="mark new unseen" comment="" disabled=no
add action=jump chain="mark new unseen" comment="" disabled=no jump-target="mark connection"
add action=mark-connection chain="mark connection" comment="" disabled=no new-connection-mark=ADSL1_conn passthrough=yes src-address-list=ADSL1
add action=mark-connection chain="mark connection" comment="" disabled=no new-connection-mark=ADSL2_conn passthrough=yes src-address-list=ADSL2
add action=mark-routing chain="mark connection" comment="" connection-mark=ADSL1_conn disabled=no new-routing-mark=ADSL1 passthrough=no
add action=mark-routing chain="mark connection" comment="" connection-mark=ADSL2_conn disabled=no new-routing-mark=ADSL2 passthrough=no
add action=mark-routing chain=prerouting comment="" connection-mark=ADSL1_conn disabled=no new-routing-mark=ADSL1 passthrough=no src-address-list=ADSL1
add action=mark-routing chain=prerouting comment="" connection-mark=ADSL2_conn disabled=no new-routing-mark=ADSL2 passthrough=no src-address-list=ADSL2
add action=jump chain=prerouting comment="" connection-state=new disabled=no in-interface=local jump-target="mark connection"
add action=jump chain=prerouting comment="" connection-state=new disabled=no in-interface=local jump-target="mark new unseen"


/ip firewall nat
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL1
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL2


/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL1
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL1 routing-mark=ADSL1
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL2 routing-mark=ADSL2
Also please note that the order of mangle rules in any of the nth load blancing examples are important. It will not work properly if you change the order of some the rules.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1369
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: ECMP

Sun Apr 12, 2009 7:58 pm

By the way, PPTP works with ECMP + route table flush workarounds, but L2TP does not. I mean from the router itself to a external router somewhere in the wild. Tested. I wonder how come L2TP does not work, what is so special about it that is different from PPTP for example? L2TP gets hit by the flush, the tunnel drops each 10 minutes +/- 2 or 3 or ?. I guess the connection-state=new does not catch it.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
alphahawk
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Fri Mar 28, 2008 6:40 pm

Re: ECMP

Mon Apr 13, 2009 8:14 am

By the way, PPTP works with ECMP + route table flush workarounds, but L2TP does not. I mean from the router itself to a external router somewhere in the wild. Tested. I wonder how come L2TP does not work, what is so special about it that is different from PPTP for example? L2TP gets hit by the flush, the tunnel drops each 10 minutes +/- 2 or 3 or ?. I guess the connection-state=new does not catch it.
Are you useing l2tp/ipsec? if so then yes it would be affected by the flush since it uses higher level protocol than the routing protocols. Thus when the routes are flushed everything in the upper layers have to reconnect. Since the ipsec is considered lvl4/5 (depending on who you talk to) it will have to reconnect on a route flush since that happens on lvl3.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1369
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: ECMP

Mon Apr 13, 2009 10:31 am

The mangle rules that we use (route table flush workarounds as I call em) fix the TCP connections (no reconnects, no loss in performance (not 100% analyzed for perf)) so they should fix everything. But L2TP somehow we miss. Maybe it is missed by connection-state=new as I said. Not sure.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Mon Apr 13, 2009 11:22 am

can we see that rules?..
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
changeip
Forum Guru
Forum Guru
Posts: 3803
Joined: Fri May 28, 2004 5:22 pm

Re: ECMP

Mon Apr 13, 2009 6:53 pm

The mangle rules that we use (route table flush workarounds as I call em) fix the TCP connections (no reconnects, no loss in performance (not 100% analyzed for perf)) so they should fix everything. But L2TP somehow we miss. Maybe it is missed by connection-state=new as I said. Not sure.
l2tp is UDP, and therefore has a very short conntrack entry.
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
alphahawk
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Fri Mar 28, 2008 6:40 pm

Re: ECMP

Thu Apr 16, 2009 10:18 pm

Normis,

Any update as to if and when mikrotik will be releasing a fix or work around other than using the nth for ecmp?
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Fri Apr 17, 2009 12:22 am

changing Kernel? will they do it at all?..
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1369
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: ECMP

Fri Apr 17, 2009 11:55 am

Well they could patch the kernel to get rid of the route table flushing or make it an option, so we can turn it off when we use masq/local ip addresses for customers.

By the way, me and a friend recently needed to route certain connections over interfaces in the same broadcast domain with the same gateway. Something like ECMP with the same gateway, but without PPPoE or PPTP tunnels to simplify things. MikroTik RouterOS currently can not handle this scenario. This is very sad.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
namwons
just joined
Posts: 8
Joined: Thu Aug 14, 2008 10:43 pm

Re: ECMP

Fri Apr 17, 2009 2:47 pm

I have been trying to get ECMP to work with this config on a RB150. Would your config work on that NetworkPro?
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Fri Apr 17, 2009 5:55 pm

MikroTik RouterOS currently can not handle this scenario. This is very sad.
ECMP - cannot. use http://wiki.mikrotik.com/wiki/NTH_load_ ... masquerade
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1369
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: ECMP

Fri Apr 17, 2009 7:07 pm

Well they could patch the kernel to get rid of the route table flushing or make it an option, so we can turn it off when we use masq/local ip addresses for customers.

By the way, me and a friend recently needed to route certain connections over interfaces in the same broadcast domain with the same gateway. Something like ECMP with the same gateway, but without PPPoE or PPTP tunnels to simplify things. MikroTik RouterOS currently can not handle this scenario. This is very sad.
Nth Can not as well ;)
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Fri Apr 17, 2009 8:47 pm

I don't understand your config. please, in more details =)

interface routing didn't help?
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1369
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: ECMP

Fri Apr 17, 2009 9:31 pm

I believe this to be the problem, this is the "route" that is at fault...:
Image

The gateway IP address is the same on ether1 ether2 ether3 ether4. When I try to set up the route without an IP address for gateway, it does not work at all. When I input the gateway IP, RouterOS always sends through ether1, despite "prefsrc" field. When I add the IP and the interface - the route works for a while, until table flush occurs.. even with the rules that I use, since they rely on route mark and the route on the picture routes marked traffic.

When routing is through PPPoE, PPTP etc, there is no such problem, when used with my flushing workaround rules, all is flawless.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
GuJack20
Trainer
Trainer
Posts: 322
Joined: Sat Jun 12, 2004 9:44 pm
Location: Tirana
Contact:

Re: ECMP

Sun Apr 19, 2009 8:29 pm

Continuing the ECMP discussion..

I the past days i have had complains about the ECMP problem (mostly with disconnection of MSN messenger) by 2-3 of our clients. The strange thing is that one other client that is using the ECMP with 6 external lines (mathematically speaking he should have disconnections more often) doesn't report disconnections. I checked his router and he has version 3.10, the others instead have newer ones. I downgraded to 3.10 one of the other routers and something surely changed. I don't see disconnections anymore (at least not as often as before). I'll have to test this in the upcoming days but for sure the downgrade has improved things.
Can anyone confirm this?? Maybe Normis can tell us when the Kernel writers have introduced the "Flushing" instruction, so we can use older versions of RouterOS for the clients needing ECMP.

The other problem is that other Open Source Linux Based Routers doesn't report this problem with the ECMP. I'm going to test one in the upcoming days, but it would be great if anyone has tested successfully ECMP with something else.
--Do you remember that guy who gave up? Neither does anybody else!
 
MrIC
Frequent Visitor
Frequent Visitor
Posts: 66
Joined: Tue Jan 13, 2009 11:34 pm

Re: ECMP

Mon Apr 20, 2009 12:47 am

@ GuJack20
can u pls upload MTv3.10 to test it with u ?
i can't found it
 
GuJack20
Trainer
Trainer
Posts: 322
Joined: Sat Jun 12, 2004 9:44 pm
Location: Tirana
Contact:

Re: ECMP

Mon Apr 20, 2009 1:42 am

just google

all_packages-x86-3.10.zip
or all_packages-mipsbe-3.10.zip if you have a routerboard.

The ECMP problem still exist but there is a slight improvement.
--Do you remember that guy who gave up? Neither does anybody else!
 
MrIC
Frequent Visitor
Frequent Visitor
Posts: 66
Joined: Tue Jan 13, 2009 11:34 pm

Re: ECMP

Mon Apr 20, 2009 2:48 am

just google

all_packages-x86-3.10.zip
or all_packages-mipsbe-3.10.zip if you have a routerboard.

The ECMP problem still exist but there is a slight improvement.
ok thanks
i will test 3.17 i think that it will doing great , i will back with result :)
 
MrIC
Frequent Visitor
Frequent Visitor
Posts: 66
Joined: Tue Jan 13, 2009 11:34 pm

Re: ECMP

Mon Apr 20, 2009 5:56 pm

3.23 arrived
http://www.mikrotik.com/download.html
so ECMP bug fixed or not ?
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1369
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: ECMP

Mon Apr 20, 2009 7:07 pm

...
By the way, me and a friend recently needed to route certain connections over interfaces in the same broadcast domain with the same gateway. Something like ECMP with the same gateway, but without PPPoE or PPTP tunnels to simplify things. MikroTik RouterOS currently can not handle this scenario. This is very sad.
This problem is attacked with routing-test package. Will test in matter of hours and post results + send supout to support e-mail. edit: no success.
Last edited by NetworkPro on Mon Apr 27, 2009 9:37 am, edited 1 time in total.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
GuJack20
Trainer
Trainer
Posts: 322
Joined: Sat Jun 12, 2004 9:44 pm
Location: Tirana
Contact:

Re: ECMP

Mon Apr 20, 2009 9:19 pm

NetworkPro. Can't you explain here the workaround of the ECMP problem? Or by email. I surely will do as discussed..
--Do you remember that guy who gave up? Neither does anybody else!
 
MrIC
Frequent Visitor
Frequent Visitor
Posts: 66
Joined: Tue Jan 13, 2009 11:34 pm

Re: ECMP

Tue Apr 21, 2009 4:45 am

same problem in 3.23 :(
messenger dissconnect every 30 min
i tried to routing mark msn & yahoo messenger but the same prob too

this is my rules
/ip firewall mangle
add action=mark-routing chain=prerouting comment="Y! routing mark" disabled=\
no dst-port=5050 new-routing-mark=Yahoo passthrough=no protocol=tcp
add action=mark-routing chain=prerouting comment="MSN routing mark" disabled=\
no dst-port=1863 new-routing-mark=MSN passthrough=no protocol=tcp

/ip route
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
WAN-1 routing-mark=Yahoo scope=30 target-scope=10
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
WAN-1 routing-mark=MSN scope=30 target-scope=10

is this rules right ? or is there any way to selove this prob ?
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1064
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: ECMP

Tue Apr 21, 2009 1:43 pm

I do not think this is related to the "flush of connections by kernel", at least in some cases.
For example, when you visit a page which would ask your identification, or would ask
for a confiramtion by entering a code, it will keep on recycling you every time you click
on login, bouncing back at the login or confirmation page. In this case we are aproximatly
returning to this page every 30 seconds, so it's obvious it has nothing to do with the flush of
connections. I tested this with http pages or https, same behaviour.
Take for example http://cqcounter.com/whois/ you will see what I mean

Now as gujack mentioned, donwgrading to version 3.10 this doesn't hapen any more.
So I think this is a problem realted to ecmp itself.
-Toni-
Don't crash the ambulance, whatever you do
 
alphahawk
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Fri Mar 28, 2008 6:40 pm

Re: ECMP

Fri Apr 24, 2009 12:21 am

Any update on patching the kernel or a fix for the ecmp normis?
 
changeip
Forum Guru
Forum Guru
Posts: 3803
Joined: Fri May 28, 2004 5:22 pm

Re: ECMP

Fri Apr 24, 2009 12:49 am

In this case we are aproximatly
returning to this page every 30 seconds, so it's obvious it has nothing to do with the flush of
connections.
A single web page load will sometimes use 20-30 connections... it's not just a single connection for each page. You are probably loading the captcha image on one connection and hitting their form on another.
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Fri Apr 24, 2009 1:01 am

A single web page load will sometimes use 20-30 connections... it's not just a single connection for each page.
didn't you forget about keep-alive connections?.. all modern browsers use them

C:\Documents and Settings\Chups>nslookup
> cqcounter.com
Name: cqcounter.com
Addresses: 75.160.72.234, 63.231.252.74
they use load balancing via DNS. maybe just browser sends request to different IP, so ECMP uses another gateway?
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
GuJack20
Trainer
Trainer
Posts: 322
Joined: Sat Jun 12, 2004 9:44 pm
Location: Tirana
Contact:

Re: ECMP

Fri Apr 24, 2009 5:24 pm

Chupaka... have you tried anything else than RouterOS for ECMP?? Any clue for a stable open source system?
--Do you remember that guy who gave up? Neither does anybody else!
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Sat Apr 25, 2009 12:47 pm

no, sorry, I'm RouterOS fan =)
and I used only 'Nth' (it was 'Random' really =) ), no ECMP
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1064
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: ECMP

Sat Apr 25, 2009 2:32 pm

Let's have an organized idea about how ECMP works.
So, when I attempt to connect to MSN Server for example, my computer initiates
a connection with MSN destination. It should look like this:

src-address=192.168.1.10:23000 dst-address=90.90.90.90:1863, right?

once this connection is established, RouterOS would keep it alive according to
the specified value in the conntrack for established TCP connection.
Now, we are told that the kernel flushes these established connections, which
would lead my computer to initiate another connection, sth. like this:

src-address=192.168.1.10:23001 dst-address=90.90.90.90:1863, right?

From the perspective of RouterOS this should be considered a "related connection".
Can't we make this related connection to go through the same gateway? Does conntrack
keeps track of gateway used? It looks not. Can we make a mangle to keeps track
of gateway used?
Do you think this is an approach to the solution, or is it completely wrong :)?
-Toni-
Don't crash the ambulance, whatever you do
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Sat Apr 25, 2009 2:57 pm

'make a mangle to keeps track of gateway used' means 'use Nth example from Wiki' =)

kernel flushes connections table, not closes all connections. your connection just can now use another gateway, so if you use src-nat, it brokes
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1064
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: ECMP

Sat Apr 25, 2009 5:24 pm

The nth will not keep gateway per connection but per IP.
It is not what I meant, I am hopping for any solution that will
keep gateway for connection.
Anyway it doesn't look it can be done.

The problem I have with nth is that first you can experience different
internet perfomance with two different adjacent computers, next,
most important I think, is you need to configure some extra scripts
for failover.
-Toni-
Don't crash the ambulance, whatever you do
 
MrIC
Frequent Visitor
Frequent Visitor
Posts: 66
Joined: Tue Jan 13, 2009 11:34 pm

Re: ECMP

Sun Apr 26, 2009 2:59 am

no, sorry, I'm RouterOS fan =)
and I used only 'Nth' (it was 'Random' really =) ), no ECMP
so ur using nth
i tried to use it but it didn't works fine .
all connections used only 1 wan .
so is there any changes u did to make NTH works fine ?
and if there is changed can u share it with us pls ?

another thing what is the best way ?
http://wiki.mikrotik.com/wiki/NTH_load_ ... masquerade
or http://wiki.mikrotik.com/wiki/NTH_load_ ... pproach%29
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Sun Apr 26, 2009 3:53 pm

The problem I have with nth is that first you can experience different
internet perfomance with two different adjacent computers, next,
most important I think, is you need to configure some extra scripts
for failover.
well, if you have two computers - yes, you need ECMP. if you have two hundred computers, Nth is quite enough. I love you, the law of large numbers =)

failover is just adding a route with gateway of another WAN and bigger distance
so ur using nth
i tried to use it but it didn't works fine .
all connections used only 1 wan .
so is there any changes u did to make NTH works fine ?
and if there is changed can u share it with us pls ?
actually, it was not Nth, it was my own setup developed from scratch. it worked fine while we use it =) and as far as I can see it was using the same principles as Nth in Wiki
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
MrIC
Frequent Visitor
Frequent Visitor
Posts: 66
Joined: Tue Jan 13, 2009 11:34 pm

Re: ECMP

Tue Apr 28, 2009 11:59 am

so ur using nth
i tried to use it but it didn't works fine .
all connections used only 1 wan .
so is there any changes u did to make NTH works fine ?
and if there is changed can u share it with us pls ?
actually, it was not Nth, it was my own setup developed from scratch. it worked fine while we use it =) and as far as I can see it was using the same principles as Nth in Wiki
now i'm using nth
but there is prob ..
proxy doesn't work
Image

so web proxy doesn't works with NTH or there is something wrong in my rules ?
 
User avatar
macgaiver
Forum Guru
Forum Guru
Posts: 1721
Joined: Wed May 18, 2005 5:57 pm
Location: Sol III, Sol system, Sector 001, Alpha Quadrant

Re: ECMP

Tue Apr 28, 2009 1:01 pm

With great knowledge comes great responsibility, because of ability to recognize id... incompetent people much faster.
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Tue Apr 28, 2009 6:11 pm

as I can see by 'Sent To Client', proxy is working. but is not caching =) did you enabled web cache at all?
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
MrIC
Frequent Visitor
Frequent Visitor
Posts: 66
Joined: Tue Jan 13, 2009 11:34 pm

Re: ECMP

Tue Apr 28, 2009 6:14 pm

as I can see by 'Sent To Client', proxy is working. but is not caching =) did you enabled web cache at all?
yes
settings :
enabled: yes
src-address: 0.0.0.0
port: 8080
parent-proxy: 0.0.0.0
parent-proxy-port: 0
cache-administrator: "master"
max-cache-size: none
cache-on-disk: yes
max-client-connections: 1000
max-server-connections: 1000
max-fresh-time: 3d
serialize-connections: no
always-from-cache: yes
cache-hit-dscp: 4
cache-drive: primary-master
Image

firewall :
chain=dstnat action=redirect to-ports=8080 protocol=tcp dst-port=80
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Tue Apr 28, 2009 7:14 pm

change max-cache-size: none no max-cache-size: unlimited

Added: but you cannot use Nth to load-balance transparent proxy. use ECMP for Web-Proxy or wait for PCC...
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
MrIC
Frequent Visitor
Frequent Visitor
Posts: 66
Joined: Tue Jan 13, 2009 11:34 pm

Re: ECMP

Tue Apr 28, 2009 10:11 pm

change max-cache-size: none no max-cache-size: unlimited

Added: but you cannot use Nth to load-balance transparent proxy. use ECMP for Web-Proxy or wait for PCC...
i think it works now
and will test it if there is prob i will disable proxy and wait for PCC
thanks
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Wed Apr 29, 2009 12:15 am

if there is prob i will disable proxy and wait for PCC
well, now all your proxy traffic should go through one gateway. but for proxy you may use ECMP - it's not long-time connections like instant messengers, etc.
just add two gateways for your main default route, and leave one default gateway to every routing table in your Nth config
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1369
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: ECMP

Wed Apr 29, 2009 12:35 am

@MrIC, The problem is probably that the proxy does not have a proper storage for the cache. It needs a drive that is partitioned that is initialized for it, has the proper folder structure. Maybe you could somehow try to rebuild the cache or make it use a secondary drive or microSD card... And dont forget to upgrade RouterOS version.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
Bibo
just joined
Posts: 10
Joined: Wed Oct 27, 2010 1:23 pm

Re: ECMP

Thu Oct 28, 2010 3:49 pm

Are there any fix or news about ECMP problem?
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Fri Oct 29, 2010 2:03 pm

what problem?..
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
Bibo
just joined
Posts: 10
Joined: Wed Oct 27, 2010 1:23 pm

Re: ECMP

Fri Oct 29, 2010 2:25 pm

routing table flush every 10min

also I had some problems with NAT:

I configured ECMP load balancing with masquerade, but when I'm switching from route 2 to route 3, I can't connect to RDC through WAN1, but WAN2 works. When I'm switching back to route 2 WAN1 work's without problems.
 2 A S  ;;; 1 GW
        0.0.0.0/0                          xxx.xxx.82.1        1       
 3 X S  ;;; 2 GWs
        0.0.0.0/0                          xxx.xxx.82.1        1       
                                           xxx.xxx.169.1      
                                           xxx.xxx.169.1      
 0   ;;; masquarade WAN1
     chain=srcnat action=masquerade out-interface=WAN1 

 1   ;;; masquarade WAN2
     chain=srcnat action=masquerade out-interface=WAN2 

 3   ;;; RDC to WAN1
     chain=dstnat action=dst-nat to-addresses=192.168.0.11 to-ports=3389 
     protocol=tcp in-interface=WAN1 dst-port=8000 

 4   ;;; RDC to WAN2
     chain=dstnat action=dst-nat to-addresses=192.168.0.11 to-ports=3389 
     protocol=tcp in-interface=WAN2 dst-port=8000
I'm planing to try PCC but i'm still curious :)
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Fri Oct 29, 2010 2:39 pm

routing table flush every 10min
it won't be fixed by MikroTik, it's in Kernel

about RDC: you need to track, which interface the connection came from and then route it back via the same connection. PCC example in Wiki do that
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
alphahawk
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Fri Mar 28, 2008 6:40 pm

Re: ECMP

Sat Oct 30, 2010 12:45 am

Are there any fix or news about ECMP problem?
There are quite a few people who would like to see the ECMP connection table flush reversed. Don't expect it to happen any time soon. Right now PCC is best option using only mikrotik.
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Sat Oct 30, 2010 2:11 am

flush is security/usability measure, I don't believe it will be removed
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
juaco
just joined
Posts: 15
Joined: Tue Nov 09, 2010 12:15 am

Re: ECMP

Wed Nov 10, 2010 12:37 am

Maybe some mangling can help with the ecmp/flush issue. Use add-src-to-address-list when a client is first routed through a gateway, and later, after the flush you can restore the route by matching from src-in-address-list and using route marks.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: ECMP

Wed Nov 10, 2010 10:32 am

that is why PPC was introduced, it does that for you in a manner you configure it, so you do not have to do tedious task of shuffling and mangling endless user connections by yourself. Just set up PPC and it will make short work of it, and connections will be recreated to same gateway they where before the flash. So you got security measure and working configuration that can be easily configured, IMHO, win-win situation.
 
juaco
just joined
Posts: 15
Joined: Tue Nov 09, 2010 12:15 am

Re: ECMP

Wed Nov 10, 2010 7:09 pm

janisk: what you say is right, i haven't used PCC though it does look really good. Anyway, does it work along with ECMP or replaces it completely? I am confused as to the situations where it's best to use ECMP or NTH or Random or PCC for loadbalancing/failover.

For now what i see is:

ECMP the "kernel routing/rpdb" way of loadbalancing. Automatic failover, has the flush issue.
PCC the "netfilter way", only mikrotik +3.24 and more customizable than ECMP alone on how you will sort connections, don't know if it handles its failover automatically as ECMP or has to be scripted.
NTH/Random: netfilter way, can do almost as everything as PCC with more work on designing the mangles, and otherwise can be used to support ECMP. Failover is scripted.

Am i missing something?
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Wed Nov 10, 2010 8:19 pm

PCC with 'both-addresses' classifier is like ECMP
PCC with 'both-addresses-and-ports' classifier is like Nth (well, varies)
PCC with 'src-address' classifier is the best way =)

about failover: to use ECMP-like failover on PCC, you need just add several routes (main and backups) with different distances for each routing table. also, look at http://wiki.mikrotik.com/wiki/Advanced_ ... _Scripting
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
acim
Member
Member
Posts: 424
Joined: Mon Sep 12, 2005 12:26 am
Location: Serbia
Contact:

Re: ECMP

Wed Apr 18, 2012 12:06 am

What about netmap to public pool (BGP prefix) and two assymetric uplinks with BGP advertising this prefix on both links? I suppose I can use ECMP in this case, am I right?

It's no problem to use PCC either, but because of assymetric uplinks (ratio 15:4) it looks much cleaner to me to use ECMP with 19 gateways (15+4) than to have 19 PCC rules and divide PCC identifier by 19.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1369
Joined: Mon Jan 05, 2009 6:23 pm
Location: Worldwide
Contact:

Re: ECMP

Wed Apr 18, 2012 3:47 am

Heads-up - If at each "table flush" your upload packets switch path this could negatively impact protocol performance.
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP

Wed Apr 18, 2012 1:28 pm

actually, ~5 rules will be enough: 4 for one uplink, and 1 for all the rest (w/o pcc)

but indeed, it would be nice to have a possibility to set a range of reminders (like in ports or addresses)

p.s. I don't think 1:4 will hurt you compared to 4:15 ;)
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.

Who is online

Users browsing this forum: No registered users and 123 guests