BGP multihoming - strange routing issue

sickology · Thu Mar 15, 2018 2:15 pm

Hello,

we have 2 mikrotik routers, each connected to its own external peer, and both are connected with iBGP.

Here is some testing:
RT1:

 /ip route> print where dst-address in 8.8.8.0/24
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  8.8.8.0/24                         212.162.29.61            20
 
/tool traceroute 8.8.8.8
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 4.69.201.158                       0%    4  37.7ms    37.7    37.7    37.8       0
 2 4.69.201.158                       0%    4  37.7ms    37.7    37.7    37.7       0
 3 72.14.243.192                      0%    4  37.7ms    37.8    37.7    37.8       0
 4 209.85.248.115                     0%    3  38.3ms    38.2    38.2    38.3       0
 5 8.8.8.8                            0%    3  37.8ms    37.8    37.8    37.9       0

RT2:

/ip route> print where dst-address in 8.8.8.0/24
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  8.8.8.0/24                         172.16.1.246            200
 1  Db  8.8.8.0/24                         149.6.30.105             20
 
 /tool traceroute 8.8.8.8
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 172.16.1.246                       0%    9   0.2ms     0.2     0.2     0.3       0
 2 4.69.201.158                       0%    9  37.8ms    37.8    37.7    37.8       0
 3 4.69.201.158                       0%    9  37.8ms    37.9    37.8    38.2     0.1
 4 72.14.243.192                      0%    9  38.6ms    38.2    37.8    39.4     0.5
 5 209.85.248.115                     0%    9  38.4ms    38.3    38.3    38.4       0
 6 8.8.8.8                            0%    9    38ms    37.9    37.9      38       0

Can anyone explain why is RT2 going through RT1 when tracing 8.8.8.8 when RT2 has the BGP route to this network with lesser administrative distance?

Another example, even more strange:
RT1:

 print where dst-address in 185.3.192.0/22
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  185.3.192.0/22                     185.1.87.116             20
 1  Db  185.3.192.0/22                     185.1.87.116             20
 2  Db  185.3.192.0/22                     212.162.29.61            20
 
/tool traceroute 185.3.192.34
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 185.3.192.34                       0%   17   0.3ms     0.4     0.3     0.5     0.1

RT2:

print where dst-address in 185.3.192.0/22
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  185.3.192.0/22                     172.16.1.246            200
 1  Db  185.3.192.0/22                     149.6.30.105             20
 
/tool traceroute 185.3.192.34
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 172.16.1.246                       0%    2   0.2ms     0.3     0.2     0.3     0.1
 2                                  100%    2 timeout
 3                                  100%    2 timeout
 4                                  100%    2 timeout
 5                                  100%    2 timeout
 6                                  100%    1 timeout

So, this is even stranger, the RT2 has BGP route to 185.3.192.0/22 with distance 20, and it is still sending packet to RT1 (distance 200), and than the packet drops somewhere?

If You need any other info please let me know.

Regards!

Vooray · Thu Mar 15, 2018 8:23 pm

Does RT2 have route to 149.6.30.105?

sickology · Fri Mar 16, 2018 9:38 am

HI,

yes it does:

ip route print detail where "149.6.30.105" in dst-address
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 0 A S  dst-address=0.0.0.0/0 gateway=149.6.30.105 gateway-status=149.6.30.105 reachable via  ether4-cogent distance=201 scope=30 target-scope=10

EDIT:
sorry, i didn't paste the right route, it is default one, and 149.6.30.105 is directly connected peer, so no route is necessary here.

EDIT2:
I think I got it... all the ibgp routes from RT1 were coming with local-pref of 100, while ebgp route didn't have that set at all (i guess it was default of 100), but also ebgp routes on RT2 were coming in with bgp-med of 28031 and I guess this was causing RT1 route to be active one. I've tested this with setting ebgp on RT2 with local pref of 200, and all traffic was sent to ebgp peer directly.

Vooray · Fri Mar 16, 2018 11:59 am

It looks like you had a problem with link to your 149.6.30.105 gateway. If you look at your first post - eBGP route 8.8.8.0/24 is not active. And AD is always > some protocol selection mechanics.

 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  8.8.8.0/24                         172.16.1.246            200
 1  Db  8.8.8.0/24                         149.6.30.105             20

And in your second post route to connected network with gateway is pointing to 0.0.0.0 only, but should point to directly connected network as well.
Can you run

ip route print detail where "149.6.30.105" in dst-address

once again?

sickology · Fri Mar 16, 2018 2:07 pm

Hm... it would be strange, because i could ping the peer gateway all the time from RT2.

If i understood correctly, AD is not that important for BGP if the route has local-pref set higher, or if local-pref is the same for 2 routes, than bgp-med is considered, and route with lower med is preferred (which was actually the problem in this case, i belive).

markmcn · Sat Mar 17, 2018 10:10 am

Easiest way to think about Administrative distance is it's used to determine best path between routing protocol EG
You learn 8.8.8.8/32 from RIP (AD120) OSPF (AD110) and BGP(20)
The challenge is which one do I believe is best? The answer is the one with the lowest AD.
So if you look at it this way the routing protocols don't look at AD directly as it's used for the router to determine who is the preferred protocol.
Are the routers R1 & R2 IBGP peered? If so this changes a few things

I'd suggest you take a look at the
Cisco BGP Best Path Selection Algorithm
It really should be your go do doc when trying to work out why one path is selected over another. I'd suggest you look though it step by step and you'll
probably find the reason for this.

sickology · Mon Mar 19, 2018 9:18 am

Yes, as I said in the first post, they are iBGP connected.

Thank you for the link, it is good source of information.

Regards!

Mon Mar 19, 2018 2:16 pm

Here you can find how best BGP route is selected in ROS:
https://wiki.mikrotik.com/wiki/Manual:B ... _Algorithm

BGP multihoming - strange routing issue

BGP multihoming - strange routing issue

Re: BGP multihoming - strange routing issue

Re: BGP multihoming - strange routing issue

Re: BGP multihoming - strange routing issue

Re: BGP multihoming - strange routing issue

Re: BGP multihoming - strange routing issue

Re: BGP multihoming - strange routing issue

Re: BGP multihoming - strange routing issue

Who is online