BGP bug - subtle but problematic issue with communities

RouterOS Forwarding Protocols
1

I’ve found a glitch in ROS BGP that leads to a mismatch in the state of two BGP peers, and posting this for community awareness.

Test Bed:
Router OS version 6.34 on CHR platform running on Virtualbox in GNS3.

The Glitch:
In some cases, a route can change states such that it is no longer accepted by a route filter.
When this happens, the originating router removes the prefix from its list of advertised routes, but fails to actually send a route withdrawal update message. Afterwards, no soft-updates (refresh) can remove the route from the peer’s routing table. The route should eventually expire when the TTL is reached, but only a complete stop/start of the peering session, or re-asserting the route and subsequently blocking it with a different filtering rule will clear the bugged route from the remote peer’s table otherwise.

How to replicate this behavior:
Not all accept->discard transitions are affected. I’m not sure what the underlying difference is, but this scenario causes it.

I configured a system which uses route filters to apply community strings to locally-originated routes and then filter outbound advertisements based on this community. There are 2 filters in play: global-out and peer-out. R1 (advertising router) will match locally-originated routes and append community 1:1 [global-out], and then filter routes to peer R2 such that only routes with community 1:1 will be sent [peer-out].

Configuration of R1:

/ip route
add distance=254 dst-address=192.168.1.0/24 type=blackhole
add distance=254 dst-address=192.168.2.0/24 type=blackhole

/routing filter
add action=accept bgp-communities=1:1 chain=peer-out
add action=discard chain=peer-out
add action=accept append-bgp-communities=1:1 chain=global-out locally-originated-bgp=yes

/routing bgp instance
set default out-filter=bgp-global-out as=65530

/routing bgp network
add network=192.168.1.0/24 synchronize=yes
add network=192.168.2.0/24 synchronize=yes

/routing bgp peer
add name=R2 out-filter=peer-out remote-address=10.1.2.2 remote-as=65520 ttl=default

The configuration of R2 is trivial - it accepts all routes from R1 over an eBGP session.

If you create a third network in /routing bgp network e.g. 192.168.3.0/24, with synchronize=no, then the global-out filter will not match this prefix as a “locally-originated” route, so 192.168.3.0/24 will not get the 1:1 community, thus the peer-out filter chain will block it from being advertised to R2 (it was testing this behavior that led me to find this bug).

However, if you modify an existing prefix (192.168.2.0/24) by changing synchronize yes->no, then the router will internally remove the 1:1 community from that prefix, and will then block it in the advertisements list - but the router fails to actually send the BGP Update message withdrawing the route from R2.

[admin@R1] /routing bgp advertisements> print
PEER     PREFIX               NEXTHOP          AS-PATH                                         ORIGIN     LOCAL-PREF
R2       192.168.1.0/24       10.1.2.1                                                         igp      

[admin@R2] /ip route> print where bgp
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  192.168.1.0/24                     10.1.2.1                 20
 1 ADb  192.168.2.0/24                     10.1.2.1                 20

A Wireshark capture shows that no withdraw update message was sent by R1.
If R2 or R1 perform a refresh, then R1 sends only the prefix 192.168.1.0/24 - R2 simply leaves the 192.168.2.0/24 prefix in its table because it did not receive any new information about that prefix.

I know this sounds like a fringe case, but it’s bad that there is a scenario where a filter begins to block a route, and the router thinks it has withdrawn the route, but in actuality it did not.

There is another issue with the route filters - changing the rule order by dragging a rule to a new position in the chain does not trigger the filter to re-evaluate the routes. Modifying an existing rule in the chain (even simply changing the comment) will cause the filter to re-evaluate the routes properly.

One last thing - if using a single filter (no global filter on the instance itself) then the behavior changes so that the BGP matcher will match locally-originated=yes, even if the network is set to synchronize=no, and therefore the bug is not applicable.

2

Does the bug persist if the filter action is reject instead of discard?

3

I hadn’t tried that, but yes it does.

The thing that originally clued me in was that I was using several different community strings and messing around, and I noticed that there was one route in R2 that had a community that I’d totally abandoned. I checked every single rule’s bgp-set-community action, and nowhere had that community. No static routes had that community. In fact, that route shouldn’t even be showing up at all!

Hmmmmm…
Hey, R1 says it’s not even advertising this at all!
George-Takei-oh-my.jpg
My next action was to spin up a Cisco router as another bgp client to see whether it also kept the “Sticky” route - and it did, so I knew the problem was in the advertising router… which is when I fired up Wireshark and discovered that there was never a “withdraw route” update.

I mean, it’s not like the Mikrotik devs need to abandon what they’re doing and work around the clock to fix this for this exact scenario- but a glitch like this could be pretty hard to troubleshoot in a production environment with no access to the remote peers, especially if there’s some other combination of conditions that lead to the same error where the router just forgets to tell everybody “hey I don’t have this route anymore”

4

Have you emailed support?

5

Nice work and as pukkita said definitely send support the details of your findings :slight_smile:

6

Done. I’ll update this thread with their response.

7

interesting post, with great painstaking detail. I can also say that I have seen such a problem with our mix of x86 MT Routers (variety of 6.x versions, doing bgp, ibgp etc.). Sorry for being vague…

Couple of weeks back, I had to chase down a problem where we are seeing a ‘ghost route’ on our edge routers, the source of that distribution was one of our peering routers injecting that route into our other bgp routers. The best I could tell was that this route prefix was being advertised on the peering fabric and was removed at some point in time. I could not locate this prefix on the peering router, however one of our other edge router was seeing this route coming from our peering edge router.. The only way to clear it out was to bounce the session.

Since this was affecting live traffic, I was not able to do a deep analysis…but it sounds very much like the issue you found.

8

I too think I have seen this issue on RB1100 and CCR platforms.

Ours was similar, withdrawn routes not disappearing. I thought it was due to feeds with large numbers of route changes but could never pinpoint it.

9

My lab was only using three prefixes, so it’s definitely not due to a large number of routes - but I can see how that could’ve been a possibility. I wonder what the underlying condition is - I’m sure my setup isn’t the only way to produce this behavior, but in my mind, it’s definitely got something to do with the process of re-evaluating the prefix filter.

I’m not entirely sure whether it’s a bug or a feature that unsynchronized locally-originated routes aren’t considered as “locally-originated” by a filter chain called from the BGP instance, but ARE considered locally-originated by the filter chain called from a peer’s configured out-filter. (it seems like a bug to me)

10

I am subscribing to this thread since it may be related to an age old issue with MikroTik’s BGP not properly sending withdraw update messages.
http://forum.mikrotik.com/t/inject-route-into-bgp-routing-table-problem/6929/1
http://forum.mikrotik.com/t/routeros-2-9-24-is-out/6777/28
http://forum.mikrotik.com/t/bgp-routing-problem-your-opinion/6459/1

I really hope there’s a solution to this. It has been a whole decade(!) and we still have the same problem described in the thread above.

11

I never heard anything back from Mikrotik except for the automatic “we received your email” reply.

It’s little chinks in the armor like this that keep ROS out of the big leagues of core routing IMO. I feel confident enough in my own knowledge of BGP and routing and ROS that I would be able to deploy a routing core with ROS, but “little” things like this can really baffle someone who’s just getting started in the complex world of BGP; someone who doesn’t realize that what they’re seeing isn’t normal.

12

Keep after them…MikroTik will eventually get it in the pipeline for a fix. MikroTik has patched just about all the bugs I’ve brought to them - sometimes quickly and sometimes not, but I think that’s the trade off for getting such an inexpensive router. There are only so many resources for product development/testing, code development, technical support, etc. However, I make sure to keep e-mailing them and it helps to keep visibility on the issue.

Also, the guys at MikroTik do read the forum a lot…keeping this thread active is the best way to get visibility to the MikroTik developers.

Post the ticket # in the thread when you get a chance

13

Do not worry, we will reply to the ticket when test will be complete.

Those links form 2006 is irrelevant in this case, because back then completely different implementation (quagga) was used.

14

The routing-test package was supposed to be a replacement for quagga.

The problem there is about your BGP Implementation, not Quagga. Prior to routing-test (quagga) we didn’t have any problem with prefixes not being withdrawn properly.

15

These old topics are irrelevant, according to topics problem was seen on v 2.8 (quagga), v2.9 and v2.9 routing test.
As I mentioned already, we will test problem discussed in THIS topic and let you know if it can be fixed in ROS v6.

16

MRZ can you confirm this is in fact a bug?

17

I can confirm there are two BUGs, 1. instance filter is not matching properly originated networks; 2. withdraw is not sent after route is discarded.

18

Hi Maris,

Any ETA on a fix for these bugs ?

19

I’ll let you know when there will be any news.

20

I did receive an update this morning from Mikrotik on this issue - it’s pretty much the same as what’s been said in this thread - being that if this bug is fixable in ROSv6 it will be done there.

Thanks, Mikrotik, for looking into this issue.

I didn’t focus on it much in this post, but there’s also the behavior that re-ordering the filter rules does not trigger them to be re-evaluated.
(so moving a rule that accepts a prefix to come before a rule that drops the prefix should cause this prefix to be advertised, but it does not unless I do something as simple as setting and removing a comment on the rule)