BGP route reflectors and cluster-id

BigTrumpet · Sun Jan 21, 2018 9:06 pm

Hi all,
probably I misunderstand how to use cluster-id...

I have a mesh core network configured with OSPF and loopback ip's.
I added two BGP route reflectors RR1 and RR2, just to distribute dynamic routes (e.g. PPPoE clients for every node).
I configured, for the network nodes, two BGP peers with these two RR, and set route-reflect=yes.
Being that I want to have two Route Reflectors for redundancy, I set the same "cluster-id" to both Routers: 1.1.1.1.

BGP reflection seems working OK, but if I look into the routing table of any client router, I see the routes installed as twice: one is active and another is not.
I thought that being both Route Reflectors within a cluster (same cluster-id) they should have ignored routes coming from each other RR. Is'n it so?

This is RR1 config:

ROUTE REFLECTOR RR1

[admin@R1] > /routing bgp instance pr detail
Flags: * - default, X - disabled
 1    ;;; IBGP ROUTE REFLECTOR 1
      name="RR1" as=64600 router-id=1.1.1.1 redistribute-connected=no redistribute-static=no
      redistribute-rip=no redistribute-ospf=no redistribute-other-bgp=no out-filter="" cluster-id=1.1.1.1
      client-to-client-reflection=yes ignore-as-path-len=no routing-table=""

for the second RR:

ROUTE REFLECTOR RR2

[admin@R7] > /routing bgp instance pr detail
Flags: * - default, X - disabled
 1    ;;; IBGP ROUTE REFLECTOR 2
      name="RR2" as=64600 router-id=1.1.1.7 redistribute-connected=no redistribute-static=no
      redistribute-rip=no redistribute-ospf=no redistribute-other-bgp=no out-filter="" cluster-id=1.1.1.1
      client-to-client-reflection=yes ignore-as-path-len=no routing-table=""

And this is the configuration for every reflector client:

REFLECTOR CLIENT

[admin@R2] > /routing bgp instance pr detail
Flags: * - default, X - disabled
 1    name="IBGP" as=64602 router-id=1.1.1.2 redistribute-connected=yes redistribute-static=no
      redistribute-rip=no redistribute-ospf=no redistribute-other-bgp=no out-filter=""
      client-to-client-reflection=no ignore-as-path-len=no routing-table=""

[admin@R2] > /routing bgp peer pr detail
Flags: X - disabled, E - established
 0 E name="IBGP RR1" instance=IBGP remote-address=1.1.1.1 remote-as=64600 tcp-md5-key=""
     nexthop-choice=default multihop=yes route-reflect=yes hold-time=3m ttl=default in-filter=""
     out-filter="" address-families=ip update-source=1.1.1.2 default-originate=never remove-private-as=no
     as-override=no passive=no use-bfd=no

 1 E name="IBGP RR2" instance=IBGP remote-address=1.1.1.7 remote-as=64600 tcp-md5-key=""
     nexthop-choice=default multihop=yes route-reflect=yes hold-time=3m ttl=default in-filter=""
     out-filter="" address-families=ip update-source=1.1.1.2 default-originate=never remove-private-as=no
     as-override=no passive=no use-bfd=no


[admin@R2] > /ip route pr detail where bgp
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 0 ADb  dst-address=2.2.2.1/32 gateway=1.1.1.4 gateway-status=1.1.1.4 recursive via 10.99.0.5 eth2
        distance=20 scope=40 target-scope=30 bgp-as-path="64600,64604" bgp-origin=incomplete
        received-from=IBGP RR1

 1  Db  dst-address=2.2.2.1/32 gateway=1.1.1.4 gateway-status=1.1.1.4 recursive via 10.99.0.5 eth2
        distance=20 scope=40 target-scope=30 bgp-as-path="64600,64604" bgp-origin=incomplete
        received-from=IBGP RR2

 2 ADb  dst-address=2.2.2.2/32 gateway=1.1.1.4 gateway-status=1.1.1.4 recursive via 10.99.0.5 eth2
        distance=20 scope=40 target-scope=30 bgp-as-path="64600,64604" bgp-origin=incomplete
        received-from=IBGP RR1

 3  Db  dst-address=2.2.2.2/32 gateway=1.1.1.4 gateway-status=1.1.1.4 recursive via 10.99.0.5 eth2
        distance=20 scope=40 target-scope=30 bgp-as-path="64600,64604" bgp-origin=incomplete
        received-from=IBGP RR2

As you can see, for every /32 route, I actually get two copies, one from RR1 and another (inactive) from RR2.
What am I missing?

Massimo

StubArea51 · Wed Jan 24, 2018 3:56 pm

Your route reflectors should also be peered with each other (but not reflected) is that the way you have it setup?

Even if you don't set the cluster-id or set a different cluster id, you're still going to see one route active and one inactive because BGP in MIkroTik doesn't support ECMP. There is a way to do it within iBGP if OSPF is the IGP that advertises loopbacks for BGP and you are learning a prefix that has two or more paths learned by OSPF to reach the same prefix.

BigTrumpet · Fri Jan 26, 2018 6:35 pm

IPANetEngineer, thank you for your post.

Your route reflectors should also be peered with each other (but not reflected) is that the way you have it setup?

Yes, it is.

Even if you don't set the cluster-id or set a different cluster id, you're still going to see one route active and one inactive because BGP in MIkroTik doesn't support ECMP. There is a way to do it within iBGP if OSPF is the IGP that advertises loopbacks for BGP and you are learning a prefix that has two or more paths learned by OSPF to reach the same prefix.

So, it is normal to see active + inactive routes whitin the same RR cluster?
What's the point to use same cluster-id if I see no difference?

This is my topology. R1 and R7 are the route reflectors.
OSPF is the IGP that advertises loopbacks.

[admin@R2] > /ip route pr where ospf
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADo  1.1.1.1/32                         10.99.0.1               110
 1 ADo  1.1.1.3/32                         10.99.0.5               110
 2 ADo  1.1.1.4/32                         10.99.0.5               110
 3 ADo  1.1.1.5/32                         10.99.0.5               110
                                           10.99.0.1
 4 ADo  1.1.1.6/32                         10.99.0.1               110
 5 ADo  1.1.1.7/32                         10.99.0.5               110
 6 ADo  10.99.0.8/30                       10.99.0.5               110
 7 ADo  10.99.0.12/30                      10.99.0.5               110
 8 ADo  10.99.0.16/30                      10.99.0.1               110
 9 ADo  10.99.0.20/30                      10.99.0.1               110
10 ADo  10.99.0.24/30                      10.99.0.5               110

Thanks again.
Massimo

shaoranrch · Fri Jan 26, 2018 10:16 pm

Same cluster-id filtering only applies for updates between route-reflectors, if you didn't do it like this the reflectors would each have a copy of the others reflector's table, and this wouldn't change the fact that the clients would have 2 copies each (in this particular case) of the routes.

Most of the time route-reflectors have a huge load, they're afterall providing "the best" routes to their clients so you'd want to do whatever it's necessary to minimize their load.
Also the usage of different cluster-id does make sense when you know not all the clients will be able to connect to all the RR (either by design or constraints), this way you'd ensure all the RR-Clients get the routes as intended.

Here's a good talk about route-reflectors design pros and cons:

http://packetpushers.net/bgp-rr-design-part-1/

BigTrumpet · Sat Jan 27, 2018 9:34 pm

Same cluster-id filtering only applies for updates between route-reflectors, if you didn't do it like this the reflectors would each have a copy of the others reflector's table, and this wouldn't change the fact that the clients would have 2 copies each (in this particular case) of the routes.

Ok Rafael, but I do have a peer between the two Route Reflectors RR1 and RR2.
They have same cluster-id but different AS.
Is this the reason why they keep two different copies of routes?

Look at one client:

 0 ADb  dst-address=2.2.2.1/32 gateway=1.1.1.4 gateway-status=1.1.1.4 recursive via 10.99.0.5 eth2
        distance=20 scope=40 target-scope=30 bgp-as-path="64600,64604" bgp-origin=incomplete
        received-from=IBGP RR1

 1  Db  dst-address=2.2.2.1/32 gateway=1.1.1.4 gateway-status=1.1.1.4 recursive via 10.99.0.5 eth2
        distance=20 scope=40 target-scope=30 bgp-as-path="64600,64604" bgp-origin=incomplete
        received-from=IBGP RR2

As you see, route 0 (active) is received from RR1, while route 1 (inactive) is received from RR2.

Massimo

shaoranrch · Sat Jan 27, 2018 11:56 pm

Same cluster-id filtering only applies for updates between route-reflectors, if you didn't do it like this the reflectors would each have a copy of the others reflector's table, and this wouldn't change the fact that the clients would have 2 copies each (in this particular case) of the routes.
Ok Rafael, but I do have a peer between the two Route Reflectors RR1 and RR2.
They have same cluster-id but different AS.
Is this the reason why they keep two different copies of routes?

Look at one client:
Code: Select all
 0 ADb  dst-address=2.2.2.1/32 gateway=1.1.1.4 gateway-status=1.1.1.4 recursive via 10.99.0.5 eth2
        distance=20 scope=40 target-scope=30 bgp-as-path="64600,64604" bgp-origin=incomplete
        received-from=IBGP RR1

 1  Db  dst-address=2.2.2.1/32 gateway=1.1.1.4 gateway-status=1.1.1.4 recursive via 10.99.0.5 eth2
        distance=20 scope=40 target-scope=30 bgp-as-path="64600,64604" bgp-origin=incomplete
        received-from=IBGP RR2
As you see, route 0 (active) is received from RR1, while route 1 (inactive) is received from RR2.

Massimo

Ok the "client" (I quote this because this isn't a route-reflector client, in order for it to be a client must be in the same AS) is on a different AS than the reflectors, so this is an eBGP adj, lets assume this:

Client AS: 1000
RR-AS: 64600

Both routes are exactly equal and more than probably your filters are set such as they don't broadcast back those prefixes (don't use AS1000 as transit). This is normal.

The following applies to anything related to BGP not just RR:

Let's get this scenario:

We have 4 routers on four different AS, peering like this:

Router A (AS100) - Peers with Router B (AS200)
Router A (AS100) - Peers with Router C (AS300)
Routers C and B are connected to Router D (AS400) and this router publishes to C and B the subnet 172.16.0.0/24

B and C know about 172.16.0.0/24 with an AS_PATH = 400 i
B and C broadcast to A 172.16.0.0/24, so A will get two routes to 172.16.0.0/24

Route 1
NEXT_HOP: Router B
AS_PATH: 200 400 i

Route 2
NEXT_HOP: Router C
AS_PATH: 300 400 i

In terms of BGP best path algorithm, both routes are identical, yet Router A will only elect 1, however you'll see that router A will still have 2 copies of the same route, why? the reason is, both routers B and C keep announcing it to A, in order for the non-winning route to vanish from Router A, the router that made A know about it, must withdraw it from the updates it's sending to A.

Lets see what would happen (without adding filters here).

Router A runs the best path algorithm and chooses B's route as its active route (could have used C instead, it doesn't matter in this case)
It'll mark it as active and announce it back to all its neighbors except the one from whom it learnt it (Router B, is basically a split-horizon rule)
Router C will receive the update from A (with AS_PATH = 100 200 400 i), and will run the best path algorithm too, but it won't mark A's route as active because he's got a better one (lowest AS path)
Thus Router C will not mark his route as inactive, and it'll keep announcing it back to all its peers except the one it learnt it from (Router D)
A will keep the route from C on its memory because C keeps telling it about that route

I hope this make things clear.

If we see your example probably this, or the use of filters is making it keep both routes in memory.

Let's change a little bit the example:

Router A keeps the same
Router B is now the one that originates 172.16.0.0/24
Router C connects to Router D
Router D connects to Router E (AS500)
Router E connects to Router B

So now Router's C route to 172.16.0.0/24 (learnt via D) has an AS_PATH = 400 500 200 i
Router A has two routes:
Route 1
NEXT_HOP: Router B
AS_PATH: 200 i

Route 2
NEXT_HOP: Router C
AS_PATH: 300 400 500 200 i

We'll have two different scenarios here:
Scenario #1 Router A is not doing any filtering at all towards C
Then A will send the route it chooses as best to C and C will have a new route with an AS_PATH = 100 200 i, C will elect this one as the best one and since it was learnt via A, it will not publish it back, hence it'll not publish any routing information from 172.16.0.0/24 to A.
This scenario ends with A only having a single route to 172.16.0.0/24

Scenario #2 Router A filters everything sent to C, such as A is never used as a transit network
In this case A will never announce the 172.16.0.0/24 route back to C, due to this C won't ever know about a better route via A, so it'll keep the announce to A of its route to 172.16.0.0/24
This scenario ends with A having the 2 routes in memory but just 1 active

BigTrumpet · Sun Jan 28, 2018 8:02 pm

Thanks Rafael for your clarification.

Actually I got confused and designed my network using eBGP with private ASs.
Anyway, tried to change it using same AS for every node (iBGP).
This way, in order to have working Route reflectors, I had to set property route-reflect=yes on every client peer (not needed with previous eBGP configuration).
Again, every RR client has two routes (active+inactive), while RR routers only see one copy of the routes.

Massimo

BGP route reflectors and cluster-id

BGP route reflectors and cluster-id

Re: BGP route reflectors and cluster-id

Re: BGP route reflectors and cluster-id

Re: BGP route reflectors and cluster-id

Re: BGP route reflectors and cluster-id

Re: BGP route reflectors and cluster-id

Re: BGP route reflectors and cluster-id

Who is online