Mostly this works as expected. Recently the CPU load on our routers skyrocketed and the BGP announcement traffic to our route reflector heavily increased for no reason. Further investigation revealed a route announcement loop. Our router DCHAM-RTR01 startet playing announcement ping pong with DCHAM-RTR02. We simplyfied our network diagram down to the relevant hosts. We analyzed bgp announcements via wireshark and found that the follwing announcements were alternating in a less than second rhythm:
Code: Select all
+2a0b:1300:81dc::/48 via 2001:7f8::20ad:0:1 med 225 localpref 200 as-path 8365 35548 209915
+2a0b:1306:1dc::/48 via 2001:7f8::20ad:0:1 med 225 localpref 200 as-path 8365 35548 209915
+2a0b:1300:81dc::/48 via 2001:7f8:3d::1b1b:0:1 med 100 localpref 200 as-path 6939 2914 209915
+2a0b:1306:1dc::/48 via 2001:7f8:3d::1b1b:0:1 med 100 localpref 200 as-path 6939 2914 209915
+2a0b:1300:81dc::/48 via 2001:7f8::20ad:0:1 med 225 localpref 200 as-path 8365 35548 209915
+2a0b:1306:1dc::/48 via 2001:7f8::20ad:0:1 med 225 localpref 200 as-path 8365 35548 209915
+2a0b:1300:81dc::/48 via 2001:7f8:3d::1b1b:0:1 med 100 localpref 200 as-path 6939 2914 209915
+2a0b:1306:1dc::/48 via 2001:7f8:3d::1b1b:0:1 med 100 localpref 200 as-path 6939 2914 209915
We then used /routing/route/print detail where dst-address=... several times to catch the moment of routing table change on DCHAM-RTR01. There we found the reason:
Code: Select all
[admin@ICHAM-RTR01] > /routing/route/print detail where dst-address=2a0b:1300:81dc::/48
Flags: X - disabled, F - filtered, U - unreachable, A - active; c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - modem, a - ldp-address, l - ldp-mapping, y - copy; H - hw-offloaded; + - ecmp, B - blackhole
b afi=ip6 contribution=candidate dst-address=2a0b:1300:81dc::/48 routing-table=main gateway=2001:7f8:1::a500:8365:1 immediate-gw=fe80::4a8f:5aff:fe00:2d36%bonding1 distance=200 scope=40 target-scope=30
belongs-to="BGP IP6 routes from 2a01:55e0::b000"
bgp.peer-cache-id=*B000003 .as-path="8365,35548,209915" .communities=35548:16,64800:42005,64800:41005,64800:40002,64800:49999 .originator-id=194.39.187.3 .local-pref=200 .med=220 .atomic-aggregate=yes .origin=igp
debug.fwp-ptr=0x20355D20
b afi=ip6 contribution=candidate dst-address=2a0b:1300:81dc::/48 routing-table=main gateway=2001:7f8:1::a500:8365:1 immediate-gw=fe80::4a8f:5aff:fe00:2d36%bonding1 distance=200 scope=40 target-scope=30
belongs-to="BGP IP6 routes from 2a01:55e0::b001"
bgp.peer-cache-id=*B000006 .as-path="8365,35548,209915" .communities=35548:16,64800:42005,64800:41005,64800:40002,64800:49999 .originator-id=194.39.187.3 .local-pref=200 .med=220 .atomic-aggregate=yes .origin=igp
debug.fwp-ptr=0x20355D20
Ab afi=ip6 contribution=active dst-address=2a0b:1300:81dc::/48 routing-table=main gateway=2001:7f8:3d::1b1b:0:1 immediate-gw=2001:7f8:3d::1b1b:0:1%vlan-de-cix-ham distance=20 scope=40 target-scope=10
belongs-to="BGP IP6 routes from 2001:7f8:3d::1b1b:0:1"
bgp.peer-cache-id=*B00025A .as-path="6939,2914,209915" .communities=64800:42002,64800:41001,64800:40001,64800:49999 .local-pref=200 .med=100 .atomic-aggregate=no .origin=igp
debug.fwp-ptr=0x2035F840
b afi=ip6 contribution=candidate dst-address=2a0b:1300:81dc::/48 routing-table=main gateway=2001:7f8::20ad:0:1 immediate-gw=2001:7f8::20ad:0:1%vlan-de-cix-fra distance=20 scope=40 target-scope=10
belongs-to="BGP IP6 routes from 2001:7f8::20ad:0:1"
bgp.peer-cache-id=*B000172 .as-path="8365,35548,209915" .communities=35548:16,64800:42001,64800:41002,64800:40001,64800:49999 .local-pref=200 .med=225 .atomic-aggregate=no .origin=igp
debug.fwp-ptr=0x203605A0
[admin@ICHAM-RTR01] > /routing/route/print detail where dst-address=2a0b:1300:81dc::/48
Flags: X - disabled, F - filtered, U - unreachable, A - active; c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - modem, a - ldp-address, l - ldp-mapping, y - copy; H - hw-offloaded; + - ecmp, B - blackhole
b afi=ip6 contribution=candidate dst-address=2a0b:1300:81dc::/48 routing-table=main gateway=2001:7f8:3d::1b1b:0:1 immediate-gw=2001:7f8:3d::1b1b:0:1%vlan-de-cix-ham distance=20 scope=40 target-scope=10
belongs-to="BGP IP6 routes from 2001:7f8:3d::1b1b:0:1"
bgp.peer-cache-id=*B00025A .as-path="6939,2914,209915" .communities=64800:42002,64800:41001,64800:40001,64800:49999 .local-pref=200 .med=100 .atomic-aggregate=no .origin=igp
debug.fwp-ptr=0x2035F840
Ab afi=ip6 contribution=active dst-address=2a0b:1300:81dc::/48 routing-table=main gateway=2001:7f8::20ad:0:1 immediate-gw=2001:7f8::20ad:0:1%vlan-de-cix-fra distance=20 scope=40 target-scope=10 belongs-to="BGP IP6 routes from 2001:7f8::20ad:0:1"
bgp.peer-cache-id=*B000172 .as-path="8365,35548,209915" .communities=35548:16,64800:42001,64800:41002,64800:40001,64800:49999 .local-pref=200 .med=225 .atomic-aggregate=no .origin=igp
debug.fwp-ptr=0x203605A0
- DCHAM-RTR01 has 4 routes. Two via AMS-IX at DCHAM-RTR02 (immideate-gwxxx%bonding1) with med 220. One via DE-CIX-HAM with MED 100 (the correct active one). And one via DE-CIX-FRA with MED 225.
- DCHAM-RTR01 is publishing it’s active route to the route reflectors which are pushing them to DCHAM-RTR02.
- DCHAM-RTR02 is receiving the route with MED 100 via DCHAM-RTR01 and marks it as active because MED 100 is better than the MED 220 route via AMS-IX.
- DCHAM-RTR02 withdraws it’s announcent via AMS-IX with MED 220 from the route reflectors which are in turn withdrawing the two routes also from DCHAM-RTR01
- DCHAM-RTR01 has two routes left. One with MED 100 via DE-CIX-HAM. And one with MED 225 via DE-CIX-FRA. And now the error happens: DCHAM-RTR01 is marking the route with the higher MED (225) as active and pushes this route to the route reflectors replacing the MED 100 route.
- The MED 225 route is being pushed to DCHAM-RTR02 replacing the MED 100 route.
- DCHAM-RTR02 correctly sees that MED 200 route via AMS-IX is better than MED 225 route via DE-CIX-FRA and activates route via AMS-IX, pushing the route to the route reflectors.
- DCHAM-RTR01 get’s the MED 220 route via AMS-IX and now correctly activates the MED 100 route again, jumping back to step 1. The loop repeats.
It seems that for some weird reason in a special case the MED comparison is made wrong. It doesn’t make any sense why this is not happening all the time. And only when certain kind of routes are present. But this is what actually happens. We also ruled out any error being made by the route reflectors. Those are working as intended.
Routers are both on 7.3.1.
There is an open support case: SUP-84730