BGP Routes / Gateways Inactive

dsovereen · Mon Jan 03, 2022 10:07 pm

I am in the process of migrating some routers from Cisco to Mikrotik. I am having trouble with a specific use case: down-stream BGP customers, which we don't have many of.

[admin@rtr-1] > ip route print detail where dst-address in 64.7.165.0/24
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 ADo  dst-address=64.7.165.0/24 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 1  Db  dst-address=64.7.165.0/24 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=200 scope=40 target-scope=30 bgp-local-pref=100 bgp-med=0 bgp-origin=igp 
        received-from=rtr-2.sfldmi-2 

 2 ADo  dst-address=64.7.165.0/29 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 3 ADo  dst-address=64.7.165.8/30 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 4 ADo  dst-address=64.7.165.12/30 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 5 ADo  dst-address=64.7.165.16/28 gateway=64.7.163.116,64.7.163.117 gateway-status=64.7.163.116 reachable via  bridge_switch.100,64.7.163.117 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 
        ospf-metric=30 ospf-type=external-type-1 

 6 ADo  dst-address=64.7.165.17/32 gateway=64.7.163.117 gateway-status=64.7.163.117 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 7 ADo  dst-address=64.7.165.32/27 gateway=64.7.163.116,64.7.163.117 gateway-status=64.7.163.116 reachable via  bridge_switch.100,64.7.163.117 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 
        ospf-metric=30 ospf-type=external-type-1 

 8 ADo  dst-address=64.7.165.33/32 gateway=64.7.163.117 gateway-status=64.7.163.117 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 9  Db  dst-address=64.7.165.64/26 gateway=64.7.165.246 gateway-status=64.7.165.246 inactive distance=200 scope=40 target-scope=30 bgp-as-path="64514" bgp-local-pref=200 bgp-med=0 bgp-origin=igp 
        received-from=rtr-2.sfldmi-2 

10  Db  dst-address=64.7.165.64/26 gateway=64.7.160.230 gateway-status=64.7.160.230 inactive distance=200 scope=40 target-scope=30 bgp-as-path="64514,64514" bgp-local-pref=200 bgp-origin=igp 
        received-from=rtr-2.mdldmi-2 

11 ADo  dst-address=64.7.165.244/30 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 
admin@rtr-1] > ip route print detail where dst-address=64.7.160.228/30  
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 ADo  dst-address=64.7.160.228/30 gateway=64.7.163.118 gateway-status=64.7.163.118 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1

Not all of the above routes are important to the problem, but I wanted to leave as much detail as possible in case there are any helpful pieces in there. The problem is with routes 9 and 10. They are not active. They were when this router was a Cisco. I've read about scope and target scope and I believe these should be fine at their default values.

Route 9 has a gateway of 64.7.165.246 and a target scope of 30. Route 11 (64.7.165.244/30) resolves that gateway and has a scope of 20 (less than or equal to 30).

Route 10 has a similar issue.

What you see above is unaltered. This customer is using a Private ASN and announces these IP addresses to us in two different data centers. The /26 does not get advertised to the global Internet; it is part of a larger announcement, but within our network, the customer can control which data center the traffic goes to. Under normal operation, the traffic goes to the data center with one AS hop and it automatically fails over to the other data center where they advertise with two AS hops should the primary go offline for any reason.

I am new to scope and target scope (not used in Ciscos) but if I am understanding the Wiki and forum posts correctly, they should not need to be adjusted in this case. This router is running 6.48.6. I'm hoping someone more expert and experienced than me can help me see what my new-to-BGP-on-Mikrotik eyes are missing.

Thanks!

Dave

StubArea51 · Tue Jan 04, 2022 5:51 pm

It looks like this is an eBGP multi-hop peering with a private ASN...can you post the output of the peering config these routes are learned over?

dsovereen · Tue Jan 04, 2022 6:38 pm

The router that I posted output from above learns these routes from iBGP neighbors. It has a full iBGP peer mesh config and has no eBGP peers:

/routing bgp peer
add in-filter=bgp-ibgp-in name=rtr-1.twrvwi-4 out-filter=bgp-ibgp-out remote-address=64.7.163.120 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-2.mdldmi-1 out-filter=bgp-ibgp-out remote-address=64.7.163.119 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-1.mdldmi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.115 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-2.mdldmi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.118 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-1.sfldmi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.117 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-2.sfldmi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.116 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-3.sfldmi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.113 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-2.shbywi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.122 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-1.shbywi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.123 remote-as=13549 ttl=default

Here is output from the Mikrotik router handling the backup (2 AS Hop) eBGP peering session (not multi hop):

/routing bgp peer
add in-filter=bgp-ibgp-in name=rtr-1.mdldmi-1 out-filter=bgp-ibgp-out remote-address=64.7.163.114 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-2.mdldmi-1 out-filter=bgp-ibgp-out remote-address=64.7.163.119 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-1.mdldmi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.115 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-2.sfldmi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.116 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-1.sfldmi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.117 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-3.sfldmi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.113 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-2.shbywi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.122 remote-as=13549 ttl=default
add in-filter=bgp-ibgp-in name=rtr-1.shbywi-2 out-filter=bgp-ibgp-out remote-address=64.7.163.123 remote-as=13549 ttl=default
add default-originate=always in-filter=bgp-64.7.160.230-in name=64.7.160.230 out-filter=bgp-64.7.160.230-out remote-address=64.7.160.230 remote-as=64514 ttl=default
add in-filter=bgp-ibgp-in name=rtr-1.twrvwi-4 out-filter=bgp-ibgp-out remote-address=64.7.163.120 remote-as=13549 ttl=default

64.7.160.230 is the eBGP peer in question, using the private ASN. This Mikrotik router shows the route it learned via eBGP as active, even though it really should not be. The other route, with only one AS Hop should be active and this should be a non-Active backup route with 2 AS Hops:

[admin@rtr-2> ip route print detail where dst-address in 64.7.165.0/24
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 ADo  dst-address=64.7.165.0/24 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 1  Db  dst-address=64.7.165.0/24 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=200 scope=40 target-scope=30 bgp-local-pref=100 bgp-med=0 bgp-origin=igp 
        received-from=rtr-2.sfldmi-2 

 2 ADo  dst-address=64.7.165.0/29 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 3 ADo  dst-address=64.7.165.8/30 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 4 ADo  dst-address=64.7.165.12/30 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 5 ADo  dst-address=64.7.165.16/28 gateway=64.7.163.117,64.7.163.116 gateway-status=64.7.163.117 reachable via  bridge_switch.100,64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 
        ospf-metric=30 ospf-type=external-type-1 

 6 ADo  dst-address=64.7.165.17/32 gateway=64.7.163.117 gateway-status=64.7.163.117 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 7 ADo  dst-address=64.7.165.32/27 gateway=64.7.163.117,64.7.163.116 gateway-status=64.7.163.117 reachable via  bridge_switch.100,64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 
        ospf-metric=30 ospf-type=external-type-1 

 8 ADo  dst-address=64.7.165.33/32 gateway=64.7.163.117 gateway-status=64.7.163.117 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1 

 9 ADb  dst-address=64.7.165.64/26 gateway=64.7.160.230 gateway-status=64.7.160.230 reachable via  bridge_switch.56 distance=200 scope=40 target-scope=10 bgp-as-path="64514,64514" bgp-local-pref=200 bgp-origin=igp 
        received-from=64.7.160.230 

10  Db  dst-address=64.7.165.64/26 gateway=64.7.165.246 gateway-status=64.7.165.246 inactive distance=200 scope=40 target-scope=30 bgp-as-path="64514" bgp-local-pref=200 bgp-med=0 bgp-origin=igp 
        received-from=rtr-2.sfldmi-2 

11 ADo  dst-address=64.7.165.244/30 gateway=64.7.163.116 gateway-status=64.7.163.116 reachable via  bridge_switch.100 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1

Route 9, the route learned via eBGP (not multi hop) on this router is identified as Active presumably because route 10, which has fewer AS Hops, is showing the gateway as inactive and therefore cannot be Active. Based on my new learning and understanding of scopes and target scopes, the Mikrotik ought to be able to look up the route to this gateway (Route 11 in this output) and mark it Active.

Just so you have it, in case they might shed light on the problem, here are the route filters being used:

/routing filter
add action=accept address-family=ip bgp-as-path-length=0-100 chain=bgp-64.7.160.230-in comment=\
    "accept ip prefix 64.7.165.64/26 prefixes /26-32 with AS Path lengths 0-100, set Distance 200, set Localpref 200 (Customer)" prefix=64.7.165.64/26 prefix-length=26-32 set-bgp-local-pref=200 set-distance=200
add action=discard chain=bgp-64.7.160.230-in
add action=accept address-family=ip chain=bgp-64.7.160.230-out prefix=0.0.0.0/0
add action=discard chain=bgp-64.7.160.230-out

The other router, handling the eBGP session at the other data center where the advertisement has only one AS Hop and should be primary, is still a Cisco. Here is it's relevant config:

router bgp 13549
 neighbor 64.7.165.246 remote-as 64514
 neighbor 64.7.165.246 description CUST-23314 IPPORT-456
 neighbor 64.7.165.246 version 4
 neighbor 64.7.165.246 timers 10 30
 address-family ipv4
  neighbor 64.7.165.246 activate
  neighbor 64.7.165.246 default-originate
  neighbor 64.7.165.246 soft-reconfiguration inbound
  neighbor 64.7.165.246 prefix-list 64.7.165.246-in in
  neighbor 64.7.165.246 prefix-list 64.7.165.246-out out
  neighbor 64.7.165.246 route-map 64.7.165.246-in in

ip prefix-list 64.7.165.246-in seq 5 permit 64.7.165.64/26
ip prefix-list 64.7.165.246-in seq 10 deny 0.0.0.0/0 le 32
ip prefix-list 64.7.165.246-out seq 5 permit 0.0.0.0/0

route-map 64.7.165.246-in permit 10
 set local-preference 200
!
route-map 64.7.165.246-in permit 20

Thanks for looking at this with me. I am at a complete loss in understanding why this is happening.

Dave

dsovereen · Thu Jan 06, 2022 12:56 am

The cliffs notes version of the issue is that RouterOS is not resolving eBGP next-hops/gateways that have a target-scope of 30 with OSPF routes to them that have a scope of 20.

Is there no one else using Mikrotik with downstream eBGP customers using OSPF internally to resolve how to reach those eBGP gateways?

Is this a bug?

Dave

dsovereen · Thu Jan 06, 2022 5:35 pm

Problem finally found. It's was right in front of my eyes all along.

The bgp-ibgp-in routing filter, applied to iBGP peers (the contents of which did not make it into either of my posts, sorry) had a prefix range of 1-24 and needed to have a prefix range of 1-32 to allow for customer network announcements smaller than /24. It follows with a reject rule, which keeps the route in the routing table but not active so that one can change routing filters and have existing routes reprocessed against them without resetting the BGP session.

/routing filter
add action=accept address-family=ip bgp-communities=13549:666 chain=bgp-ibgp-in set-bgp-weight=65535 set-type=blackhole
add action=accept address-family=ip bgp-communities=13549:667 chain=bgp-ibgp-in set-bgp-weight=65535 set-type=blackhole
add action=accept address-family=ip chain=bgp-ibgp-in prefix=0.0.0.0/0 prefix-length=1-24
add action=reject address-family=ip chain=bgp-ibgp-in
add action=discard chain=bgp-ibgp-in

Glad to have finally figured this out. Thanks to everyone who looked a this with me.

Dave

BGP Routes / Gateways Inactive

BGP Routes / Gateways Inactive

Re: BGP Routes / Gateways Inactive

Re: BGP Routes / Gateways Inactive

Re: BGP Routes / Gateways Inactive

Re: BGP Routes / Gateways Inactive

Who is online