Page 1 of 1

BGP ECMP (multipathing)

Posted: Thu Jan 30, 2014 9:14 pm
by Crami
Hi There

I have setup where the Customer router (Router Boards) is connected to a MPLS ELAN (VPLS) cloud provided by a third party. We, the ISP are connected redundantly to that cloud with two Cisco Routers. I now set up two BGP
sessions from the Customer Routers to the ISP Routers for redundancy and announce the Prefix the Customer gets assigned and send a default route to them from the Cisco routers.

I now like to do load balancing over these two default gateways. I know that MikroTik has the standpoint that BGP can not do ECMP by design, but all other vendors I know can do it (Cisco, Juniper, Brocade, Fortinet etc.)

As the two BGP sessions do not end on the same router I can not use the loopback method mentioned in the BGP ECMP manual here: http://wiki.mikrotik.com/wiki/Manual:BG ... interfaces
But they reference a Cisco page to explain that other vendors do it the same way and that this is only the right way... fortunately on this same page Cisco explains how BGP multipath works! http://www.cisco.com/en/US/tech/tk365/t ... 45bf.shtml (Load Sharing When Dual-Homed to One Internet Service Provider (ISP) Through a Single Local Router)

Will MikroTik reconsider their standpoint about multipath BGP or do I have to switch back to Cisco for the customer routers?

Regards

Matthias

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 11:35 am
by mrz
It is a hack and we also can do it.
http://wiki.mikrotik.com/wiki/Manual:BG ... interfaces

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 1:05 pm
by Crami
Please explain how this is different to what I already wrote. Loopbacks are not possible. There are two (2) BGP sessions to two (2) different routers. So the destination can not be the same.

Or am I wrong ?

Regards

Matthias

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 1:35 pm
by mrz
Read that cisco article and mikrotik article, they both uses the same principle and loopback address.

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 2:05 pm
by Crami
Have you red the section: Load Sharing When Dual-Homed to One Internet Service Provider (ISP) Through a Single Local Router


This is what I like to accomplish. The other examples are clear and is common practice, I agree...

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 2:12 pm
by mrz
This is exactly the same setup as Load Sharing with the Loopback Address as a BGP Neighbor except that remote end has two routers.
Setup on mikrotik router would be the same, both remote peers establishes BGP session to loopback address.

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 2:18 pm
by Crami
Remote peers are not the problem. I want two default routes on the Mikrotik to the remote peers. That does not change anything if I make a session to remote peer loopbacks.

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 2:25 pm
by mrz
So to make it clear you have two BGP R1 and R2 running iBGP between then. They bothhave peering to ISPs router (R3).
And you want to have ECMP routes on R1 and R2?

shell code

R1------------\
 |             \
ibgp           ISP R3
 |             /
R2------------/

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 2:34 pm
by Crami
No, the other way around:
    +-------------- ISP_R1 -+
    |   EBGP                |
 R1-+                       | IBGP
    |   EBGP                |
    +-------------- ISP_R2 -+
R1 is Mikrotik
ISP_R1 and R2 are Cisco

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 2:46 pm
by mrz
Then the same setup as described in the manual
R1 has loopback address and both ISP routers should establish connection to loopback. Exact same config is also in cisco article.

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 2:59 pm
by Crami
Yes, exactly ... But with the little command "maximum-paths 2" in the bgp config!

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 3:26 pm
by mrz
It just limits ecmp gateways to two. Mikrotik does not limit amount of gateways.

Re: BGP ECMP (multipathing)

Posted: Fri Jan 31, 2014 3:45 pm
by Crami
So then the missing A on route 1 does not mean anything ?
> /ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  0.0.0.0/0                          212.25.9.1               20
 1  Db  0.0.0.0/0                          212.25.9.2               20
If I make traceroutes from a public ip behind the mikrotik (no nat) to several locations, it always takes the path over route 0.

Re: BGP ECMP (multipathing)

Posted: Fri Feb 14, 2014 10:35 am
by Crami
And now there is silence ...
Are you now convinced that there should be a "maximum-path" feature like in cisco ios to accomplish this? Or am I just to boring to be eligible for a rely.

Re: BGP ECMP (multipathing)

Posted: Mon Mar 03, 2014 11:42 pm
by Chupaka
maybe, post at least your BGP config first?..

Re: BGP ECMP (multipathing)

Posted: Tue Mar 04, 2014 10:28 am
by Crami
Shure, here ist the relevan config of the mikrotik router.
# feb/02/1970 15:21:57 by RouterOS 6.7
# software id = XHEM-GX3Y
#
/interface vlan
add comment="CES" interface=sfp1 l2mtu=4070 mtu=4000 name=sfp1-ces vlan-id=3214
/ip neighbor discovery
set sfp1-ces comment="CES"
/interface wireless security-profiles
/routing bgp instance
set default as=64512 router-id=z.y.x.w
add address=z.y.x.w/26 interface=sfp1-ces network=z.y.x.0
/routing bgp network
add network=a.b.c.d/27 synchronize=no
/routing bgp peer
add name=peerA remote-address=z.y.x.1 remote-as=8758 ttl=default update-source=sfp1-ces
add name=peerB remote-address=z.y.x.2 remote-as=8758 ttl=default update-source=sfp1-ces
peerA and peerB are Cisco Routers, which only announce a default route.
I'm happy for any inputs

Re: BGP ECMP (multipathing)

Posted: Tue Mar 04, 2014 1:55 pm
by mrz
Ok, sorry for that. Yes you are right your intended setup is currently not possible on MikroTik routers. BGP will install only one gateway.

Re: BGP ECMP (multipathing)

Posted: Wed Mar 05, 2014 10:10 am
by Crami
Thanks for your confirmation. At least I know now that I'm not doing something wrong. Are there plans to support this? Would be a very nice and appreciated feature.

Re: BGP ECMP (multipathing)

Posted: Wed Oct 22, 2014 7:04 am
by ttanner
I am really disappointed to see that this is not supported. We have 4 CCR1036-8G-2S+EMs in production that I need to do this to coming up.

What can we do to get this feature implemented? We have been migrating away from Cisco to Mikrotik, but missing features like this are really hurting us.

Re: BGP ECMP (multipathing)

Posted: Sat Nov 01, 2014 12:03 pm
by ada
Please add suport for many (2-3) ways for gateway for BGP

Re: BGP ECMP (multipathing)

Posted: Sat Nov 08, 2014 12:53 am
by engineertote
very interested

MT guys ,, can you please update the post if you are going to add this feature or not ?

Re: BGP ECMP (multipathing)

Posted: Thu Nov 26, 2015 1:44 pm
by omago
Any updates on this topic ?
Is BGP ECMP routing on Mikrotik's roadmap ?

Regards,

Jon

Re: BGP ECMP (multipathing)

Posted: Mon Nov 30, 2015 5:41 pm
by IPANetEngineer
Would like to know about this as well - Maybe this is a v7 feature?

Re: BGP ECMP (multipathing)

Posted: Tue Dec 01, 2015 8:42 pm
by Bergante
It would be really useful.

BGP multipathing can be used, for example, to turn a router into a load balancer for a cluster of servers (web, DNS, etc) and it's a technique used in many large installations.

This article describes it very well.
http://bits.shutterstock.com/2014/05/22 ... -software/

This depends on a multipath BGP implementation that can do a flow or hash based load balancing. In a configuration like this several servers would announce the same IP address via BGP and the routers would distribute connections among them. Flow based balancing (or some kind of IP address hashing) is needed so that all the packets belonging to a TCP connection are sent to the same servers.

This technique is very cheap to implement (unless you have to use expensive routers for it) and in my opinion it's much more reliable and scalable than the typical load balancer "magic boxes".

So, do it, please!

Re: BGP ECMP (multipathing)

Posted: Tue Oct 03, 2017 1:31 am
by C3H5N3O9
Any updates on this topic ?
Is BGP ECMP routing on Mikrotik's roadmap ?

Regards,

Re: BGP ECMP (multipathing)

Posted: Thu Feb 01, 2018 6:41 pm
by chubbs596
Any updates on this topic ?
Is BGP ECMP routing on Mikrotik's roadmap ?

Re: BGP ECMP (multipathing)

Posted: Thu Feb 01, 2018 6:47 pm
by pe1chl
I would guess that it is all in the RouterOS v7 super duper routing code!

Re: BGP ECMP (multipathing)

Posted: Wed Sep 05, 2018 7:35 pm
by Crami
More than four (4) years since my initial post, and still there is no proper ECMP support ...

Re: BGP ECMP (multipathing)

Posted: Thu Sep 06, 2018 3:15 am
by schadom
More than four (4) years since my initial post, and still there is no proper ECMP support ...

Welcome to Mikrotik... I guess they should remove the "Router" from "RouterOS", as it seems there had been no progress on the routing engine for years now. Instead hundreds of fixes for the wifi and access point stuff are appearing in every new ROS release. Mikrotik nowadays is a great access point and WISP supplier, but not a company I'd buy another edge/core router or switch from. Seems their focus and market orientation has changed after releasing the CCR-series. Sad story, as there would have been potential.

Re: BGP ECMP (multipathing)

Posted: Sat Sep 08, 2018 9:49 pm
by IPANetEngineer
I've been wanting to see this as well, but i'd rather have recursive routing in IPv6 for BGP fixed first. :-)

Re: BGP ECMP (multipathing)

Posted: Tue Nov 13, 2018 11:16 am
by artferrr
I've been wanting to see this as well, but i'd rather have recursive routing in IPv6 for BGP fixed first. :-)

maybe in ROS7 years after 10 I’m very impressed with the support of mikrotik for 2019, it is planned to fully transfer the kernel to devices from other manufacturers.

Re: BGP ECMP (multipathing)

Posted: Thu May 02, 2019 2:04 pm
by heribertos
I solved this for me with Routing Filters under Set Next-Hop-in with multiple Gateway-Addresses. This works for me even for BGP.

Please observe, that in CISCO the max-path n does 2 things: (1) Installation of best path and (n-1) looser pathes (2) Forming an ECMP Route from the multiple routes to the same destination prefix. The first part is not needed in MikroTik, because MikroTik installs always all BGP Routes in the Routing Table, of couse not neccessarily activ. Only the second part is required and could be done manually, by routing filters or by scripts.

Second thing to observe is, you will get never ever on any router 2 routes active pointing to the same destination prefix.

Re: BGP ECMP (multipathing)

Posted: Thu May 02, 2019 4:25 pm
by pe1chl
Second thing to observe is, you will get never ever on any router 2 routes active pointing to the same destination prefix.
But that is the idea of ECMP.... have 2 or more routes for the same prefix pointing to a different path, and the router has some mechanism to use them alternately for outgoing traffic.

Re: BGP ECMP (multipathing)

Posted: Thu May 02, 2019 5:17 pm
by heribertos
Sorry, for my confusing wording. I think we mean the same.

ECMP Route is for me 1 route with multiple gateways, it appears as 1 destination prefix in the table. Of course, that are 2 or more routes , but considered as 1 with respect to administrative distance.

With 2 active routes I ment 2 seperate entries in the table, and each one is marked as active, as the OP in his last post was asking, why 1 default route route is active and the second is not..
My comment is, that the second will never get active.

No issue. We mean the same.

Regards

Re: BGP ECMP (multipathing)

Posted: Tue Jul 02, 2019 11:18 pm
by RcRaCk2k
Came to that point today and i am disappointed that mikrotik developers are not able to give us some information if that feature is on a todo list or not.

Re: BGP ECMP (multipathing)

Posted: Wed Jul 03, 2019 4:00 am
by troybowman
Please add my +1 vote for ECMP, too.

Re: BGP ECMP (multipathing)

Posted: Fri Aug 16, 2019 10:01 pm
by Railander
as a workaround, ECMP works perfectly with OSPF instead of BGP, assuming your setup allows for it.
alternatively, the tip provided by heribertos using routing filters forcing the "Next-Hop-in" to your desired multipath gateways might work, though you'll lose the fully dynamic aspect of BGP by essentially enforcing static ECMP routes.

with that said, i'd really like to have ECMP supported in MPLS LDP. it's really weird that it works fine with OSPF but when you run LDP on top of it it just picks whatever gateway is listed first on the route, this is terrible for load-balancing VPLS.

Re: BGP ECMP (multipathing)

Posted: Mon Jun 22, 2020 9:36 pm
by ploquets
I solved this for me with Routing Filters under Set Next-Hop-in with multiple Gateway-Addresses. This works for me even for BGP.


Thanks for this, I could workaround this issue and now working!

Re: BGP ECMP (multipathing)

Posted: Thu Jul 02, 2020 9:50 pm
by RcRaCk2k
How have i to setup "Set In Nexthop" on Actions-Tab as @heribertos suggested?
What value have to be in there?

Re: BGP ECMP (multipathing)

Posted: Fri Sep 04, 2020 8:42 am
by peterh
with that said, i'd really like to have ECMP supported in MPLS LDP. it's really weird that it works fine with OSPF but when you run LDP on top of it it just picks whatever gateway is listed first on the route, this is terrible for load-balancing VPLS.
It is even worse. RouterOS "BGP" (I am using quotes because it is not RFC 4271 compliant) picks not just the first gateway for equal cost routes, but even for UNEQUAL cost routes, since it does not care whatsoever for IGP costs to reach the BGP next hop gateway. This not only violates RFC 4271 but essentially also means that RouterOS routes packets in an BGP or MPLS cloud essentially in a random fashion. Yes, packets reach the target, but to go from New York to Boston, RouterOS first sends you first to Los Angeles, then Seattle, detour through Chicago, and then to Boston, since its "BGP" has no understanding of the IGP topology in an AS.

With any other BGP implementation on the market (even free Open Source solutions), you get true recursive lookups of the BGP next hops within the IGP routing table including their costs, as it is described in RFC 4271. And yes, with most you even get ECMP.

Re: BGP ECMP (multipathing)

Posted: Fri Sep 04, 2020 9:00 am
by Crami
You are absolutely right. That's why I do not use mikrotik for larger scale BGP deployments. They should use something prooven as routing suite, like frr or bird underneath.

Re: BGP ECMP (multipathing)

Posted: Fri Sep 04, 2020 4:40 pm
by pe1chl
It is even worse. RouterOS "BGP" (I am using quotes because it is not RFC 4271 compliant) picks not just the first gateway for equal cost routes, but even for UNEQUAL cost routes, since it does not care whatsoever for IGP costs to reach the BGP next hop gateway. This not only violates RFC 4271 but essentially also means that RouterOS routes packets in an BGP or MPLS cloud essentially in a random fashion. Yes, packets reach the target, but to go from New York to Boston, RouterOS first sends you first to Los Angeles, then Seattle, detour through Chicago, and then to Boston, since its "BGP" has no understanding of the IGP topology in an AS.
That became clear to me pretty quickly when I started using RouterOS BGP, and I quickly decided to use a different AS number for every site.
(of course such decisions are only possible in a closed network not BGP-connected to internet)

Re: BGP ECMP (multipathing)

Posted: Fri Sep 04, 2020 4:44 pm
by peterh
The sad thing about this is that since the last 10 years, we see a creeping trend for "BGP everywhere". It is no longer just a WAN protocol for nation-wide ISPs and occasional Internet Exchange peerings, it first creeped into LANs and campus networks through MPLS and now we see whole data centre networks being constructed purely on BGP (i.e. RFC7938). That way you can scrap the underlying OSPF/ISIS completely and run on one protocol - BGP everywhere - from your spine to your leaf switches to your Internet peerings. MikroTik has catapulted itself out of that race with its current implementation. BGP ECMP is an absolute must in the year 2020 (actually it already was in 2010).

Re: BGP ECMP (multipathing)

Posted: Fri Sep 04, 2020 5:38 pm
by pe1chl
Similar to what that RFC proposes, I quickly switched to EBGP-only to avoid a lot of strange routing issues that I encountered at first when I had used the same AS for a couple of sites in the same region so there was a mix of IBGP/EBGP.
I now use IBGP only between routers at the same site that are directly linked.
Of course still no ECMP but at least no unpredictable routing.