Total BGP Community noob

RouterOS Forwarding Protocols
1

Hi all,

Total noob when it comes to BGP.

I have a basic BGP setup with 2 transit providers and connected to 2 IX with multiple direct peering with those IX members. I also have downstream customers that I am sending them a default route and announcing there AS and Prefix using filter etc.

Here is my issue from one of the IX

To your transit and peers, you need to only announce your routes and your customer routes. You can’t just use a prefix-list to filter the routes because you may learn your customer routes from your transit or peers if their connection to you ever goes down. You should tag all of your customer routes and only allow routes with that customer BGP community to be announced to your transit and peers.

You should be tagging your customer routes with a specific BGP community and filter on that community in combination with your prefix-list in order to only send customer routes to your transit provider. Otherwise, you are effectively providing transit for your peers.

I think I understand what he means, but I have no clue how to setup communities. I have been reading alot but still not there yet. Issue starting to get serious because the IX cut our routes off and downstream customers starting to complain.

Any help here would be nice. Maybe a couple examples as im a damn nooobb…! Everything under one BGP instance.

Thx in advance.

noob bgp problem
2

Make a bgp-customer-in filter chain for each customer.
In each customer’s chain, allow each prefix they may advertise, and include a BGP action to set the community (more on this later).

On each BGP speaking router, you need a few standard BGP-out filters.
BGP-out-full
BGP-out-local-and-peers
BGP-out-local-only
BGP-out-none

Full and none should be obvious.
Customer-only should check for the presence of your customer community tag and allow such routes only. Same for local-and-peers

For my AS, I set up the following communities:
ASN:100 = transit routes
ASN:200 = peer routes
ASN:300 = customer routes
ASN:400 = my local routes

So a customer-facing router would tag all learned prefixes with 65500:300 (if my ASN were 65500)
A peer-facing router should tag learned prefixes with :200
Transit facing routers tag :100
When you originate your own prefixes, be sure to tag those with 400

To transits, your out filter should only allow :300 and :400
To private peerings, only allow :300 and :400
To customers who want local-only, send 300-400 (and maybe 200 if the customer wants to benefit from your private peerings)
To customers who want full routes, send it all
To peers, send 300-400

3

In each customer’s chain, allow each prefix they may advertise, and include a BGP action to set the community (more on this later).

What type of action do I set for them in the customers chain in filters?

My primary core router is transit, peer and customer facing.

4

I get the general idea, but would it be possible to get a couple CLI examples? There are so many options and tabs im clueless.

What would it look like setting up out filters for my transit peers using the above community string examples?

What would it look like setting up out filters for my downstream customers looking to benefit from my private peering sessions and obviously going out to transit upstream providers?

If I could just get an example or two, it should put me on the right track in terms of what and where to add things.

Again, main core router is a big 32 core intel device running a mix of 80 sfp and sfp+ nics so it is transit facing and customer facing device.


Regards,

5

Here’s an example input filter for CustomerX who has two CIDR blocks 10.100.128.0/22 and 10.200.64.0/21
This example allows the customer to send their CIDR or any smaller subsets of their CIDR no more specific than /24
All accepted prefixes are tagged with community 65500:300 (following my example scheme where :300 = customer prefixes)
65500 would of course be replaced with your actual ASN.

/routing filter
add action=passthrough chain=CustomerX-in prefix=0.0.0.0/0 prefix-length=0-32 set-bgp-communities=65500:300 comment="tag all prefixes with customer community"
add action=accept chain=CustomerX-in prefix=10.100.128.0/22 prefix-length=22-24 comment="accept customer-owned prefix and subnets"
add action=accept chain=CustomerX-in prefix=10.200.64.0/21 prefix-length=21-24 comment="accept customer-owned prefix and subnets" 
add action=discard chain=CustomerX-in prefix=0.0.0.0/0 prefix-length=0-32 comment="default discard all other prefixes"

Here are the “standardized” output filters:
(EDIT: Your transit providers would be configured with the “bgp-out-local” example below, but I would recommend making a custom copy of that chain with the transit provider’s name explicitly because you may want to send communities to a particular peer, and if the transit provider is using a standardized chain, you couldn’t do that. Peers should get specific custom chains as well, but they should basically mirror the below chains… in fact, you could probably make the custom chains all do the peer-specific things, and then jump into the standard ones for the filtering part)

add action=passthrough set-bgp-communities="" chain=bgp-out-local-nocustomer
add action=accept bgp-communities=65500:400 chain=bgp-out-local-nocustomer
add action=reject chain=bgp-out-local-nocustomer

add action=passthrough set-bgp-communities="" chain=bgp-out-local
add action=accept bgp-communities=65500:300 chain=bgp-out-local
add action=accept bgp-communities=65500:400 chain=bgp-out-local
add action=reject chain=bgp-out-local

add action=passthrough set-bgp-communities="" chain=bgp-out-local+peers
add action=accept bgp-communities=65500:200 chain=bgp-out-local+peers
add action=accept bgp-communities=65500:300 chain=bgp-out-local+peers
add action=accept bgp-communities=65500:400 chain=bgp-out-local+peers
add action=reject chain=bgp-out-local+peers

add action=passthrough set-bgp-communities="" chain=bgp-out-full
add action=accept prefix-length=0-24 chain=bgp-out-full

add action=passthrough set-bgp-communities="" chain=bgp-out-local-nocustomer
add action=accept bgp-communities=65500:400 chain=bgp-out-local-nocustomer
add action=reject chain=bgp-out-local-nocustomer

Note that these are pretty basic, but should give you the gist of how to do it.
I think that if you specify multiple communities on a single rule that it requires all of them to be present - you could try experimenting in the lab :wink:

Also note that these example output chains all wipe the community list clear. Of course if you need to send communities to peers for various reasons (many carriers offer community support for traffic engineering purposes) then you’d want to set those in a similar fashion as your input chains.

Community append is also useful because it leaves the existing community string intact and just adds things - think of them like stickers that mean whatever you want them to mean “fragile” “hazardous” “bulk” “expedited” etc - you could have as few or as many as needed. Some providers even keep your communities intact so that you can send communities through them to other networks out there…

Important:
Note that your output chains all trust that your community tagging is correct. There are no safety-catches for things like bogon routes, routes that you shouldn’t be advertising, etc. If it’s tagged, it ships. You could probably make an auxiliary chain that lists all of your prefixes that you should be allowed to route, and make your “local”, “local+peers” and “local-nocustomer” chains match it also, so that way, you only have one list of prefixes to maintain in each router, instead of having to edit multiple chains in multiple routers every time you add a prefix to your network… but then again, if you’re scrupulus at the gate (meaning that you always tag properly and always filter properly at the border / when you originate routes) then communities should be enough to properly filter your outgoing advertisements, while being scalable and requiring a minimum of configuration modifications to your network whenever new prefixes come along / old prefixes go away.

6

You know - I just looked into one item for this whole design - I don’t see how to set communities on routes that you originate yourself.
In Cisco land, you use a route-map on your network statement, but there’s no such thing here.

Perhaps the instance’s bgp-out chain is the way to do it - I’ll have to lab this up later…
Meanwhile, you could substitute rules which match your own CIDR blocks for output filter chains where community :400 was used in my examples.

7

OK I think im getting somewhere.

So the CLI examples you gave actually tag the corresponding prefixes.

What about the filter used for transit providers and peers?

This probably doesnt make sense, but this is what I got from your first post

chain=BGP out to Transit bgp-communities=“” invert-match=no action=accept
set-bgp-prepend-path=“” set-bgp-communities=XXXXX:100,22748:300,22748:400

chain=BGP out to Peers bgp-communities=“” invert-match=no action=accept
set-bgp-prepend-path=“” set-bgp-communities=22748:300,22748:400

Now how wrong is that? loll

Following your lastest example of tagging a customers prefix

;;; Accept customer-owned prefix and subnets XXX
chain=Customer IN prefix=xxx.xxx.87.0/24 prefix-length=22-24
invert-match=no action=accept set-bgp-prepend-path=“”
set-bgp-communities=“”

8

this should just be prefix-length=24 (and in fact, you could just leave this criteria out entirely if you only want to accept the prefix’s exact length)

Basically, if the customer has a /22, then it’s 22-24, if the customer has a 21, then it’s 21-24…

However, allowing a prefix-length of 22 from a prefix defined as a /24 won’t allow anything but a /24 anyway so no harm done, but it’s something to tidy up.

FYI: what this does is allow the customer to break their prefix into smaller advertisements if they want to. If I have x.x.32.0/22 as a prefix, then I could choose to send it to the world as x.x.32.0/23 and x.x.34.0/23 if your filters allowed prefix-length 22-23, but if I tried to send x.x.33.0/24 and your filter specified prefix-length 22-23, then your filter would block this announcement from me.

Also, make sure there’s a default “discard” action at the end of your customer-filter-IN chain.

9

Yeah, my post was kind of long, but basically, that filter should substantially be the same as the bgp-out-local example.

add action=passthrough set-bgp-communities="" chain=bgp-out-local
add action=accept bgp-communities=65500:300 chain=bgp-out-local
add action=accept bgp-communities=65500:400 chain=bgp-out-local
add action=reject chain=bgp-out-local

As I mentioned in my previous little “addendum” post, I don’t immediately see a way to make ROS put communities on routes the router originates itself (with the networks list) but I’m going to try monkeying around in GNS3 to see if I can make it happen…

… if not, then you’ll need to replace the 65500:400 rule with a rule that matches your own prefixes explicitly. I suppose those don’t change terribly often, so it wouldn’t be a huge scalability / manageability burden to just put your IP prefixes.

10

Well, it’s super-easy to originate communities if you’re using a blackhole route to originate your own prefixes into BGP.

On the blackhole route, go to the attributes tab, and just type in 65500:400 as the community. :slight_smile:

Originating it as a connected route (and I’m assuming OSPF) won’t let me do that to the dynamic route, so I’m going to see if an out-filter at the instance level takes care of it or not…

11

Don’t ever redistribute connect routes with BGP… Use synchronize:

/ip route add dst-address=192.0.2.0/24 bgp-communities=123:456 type=blackhole
/routing bgp network add network=192.0.2.0/24 synchronize=yes
12

Definitely don’t use redistribute. However, if the actual interface has the same address/mask as the network interface, then the connected route will be the one in the routing table that synchronize catches… or if you’re synchronizing based on reachability in an IGP like OSPF, then you can’t just slap a community on those routes.

The OSPF case is relevant - for instance if you want a border router to withdraw its prefixes when it is isolated from the internal AS topology, then you don’t want to use black hole routes to originate prefixes. If you use nailed-up blackhole routes in the border router, then you’re advertising a black hole to the world if this router gets isolated from the local AS, but other border routers are still available into the network.

As for how to tag a community onto locally-originated prefixes regardless of what protocol they’re synchronized to:

I’ve found that you can make a very simple BGP-out global filter rule that will automatically tag the router’s own originated routes with the correct community (e.g. 65500:400):

/routing filter add action=accept bgp-communities=65500:400 chain=tag-originated locally-originated-bgp=yes
/routing bgp instance set default out-filter=tag-originated

Doing this, any networks that synchronize on OSPF/Connected/RIP(yuck) routes will be tagged when the router attempts to export them to its peers. Furthermore, per-peer out-filter checks run after the global out-filter, so any per-peer output filter that depends on the presence of the 65500:400 community will find that the :400 community is in place as it should be. Just note that the community won’t be visible on the route in the routing table itself in the originating router.

13

All hell breaks loose with this config. I also have to disable the reject chains or else nothing gets advertised anywhere.

Instance
set default as=xxx out-filter=Tag-originated

filters
0 chain=Tag-originated bgp-communities=xxx:400 locally-originated-bgp=yes invert-match=no action=accept set-bgp-prepend-path=“”

1 ;;; BGP out to Transit Rule
chain=BGP out to Transit bgp-communities=“” invert-match=no action=passthrough set-bgp-prepend-path=“” set-bgp-communities=“”

2 ;;; BGP out to Transit - Customers
chain=BGP out to Transit bgp-communities=xxx:300 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

3 ;;; BGP out to Transit - Local
chain=BGP out to Transit bgp-communities=xxx:400 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

4 X ;;; BGP out to Transit Reject
chain=BGP out to Transit bgp-communities=“” invert-match=no action=reject set-bgp-prepend-path=“”

5 ;;; BGP out to Peers Rule
chain=BGP out to Peers bgp-communities=“” invert-match=no action=passthrough set-bgp-prepend-path=“” set-bgp-communities=“”

6 ;;; BGP out to Peers - Customers
chain=BGP out to Peers bgp-communities=xxx:300 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

7 ;;; BGP out to Peers - Local
chain=BGP out to Peers bgp-communities=xxx:400 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

8 ;;; BGP out to Peers - Peers
chain=BGP out to Peers bgp-communities=xxx:200 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

9 X ;;; BGP out to Peers - Reject
chain=BGP out to Peers invert-match=no action=reject set-bgp-prepend-path=“”

my prefixes

10 chain=BGP out to Transit prefix=xxx.0/24 bgp-communities=“” invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

11 chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

12 chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path=“”

13 chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path=“”

14 chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path=“”

15 chain=BGP out to Transit prefix=xxx.0/24 bgp-communities=“” invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

16 chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path=“”

17 chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

18 chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

customer prefix

19 chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

20 chain=isp1-in invert-match=no action=accept set-bgp-prepend-path=“”

customer prefix

21 ;;; Accept customer-owned prefix and subnets xxx
chain=Customer IN prefix=xxx0/24 prefix-length=0 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

22 ;;; Accept customer-owned prefix and subnets xx
chain=Customer IN prefix=xxx.0/24 prefix-length=0 invert-match=no action=accept set-bgp-prepend-path=“” set-bgp-communities=“”

23 ;;; Tag all prefixes with customer community
chain=Customer IN prefix=0.0.0.0/0 prefix-length=0-32 invert-match=no action=passthrough set-bgp-prepend-path=“” set-bgp-communities=xxx:300

Thats all I got

14

Ok - you got a few things wrong below. I think a good thing for you to do is to set up a dummy peer that doesn’t advertise, but accepts all routes - set it up as a bogus ASN like 65530 and try setting it up with your various chains to see what you get and what you don’t get, etc.

One comment - you probably don’t want to send :200 community-tagged routes to your peers because that means you’re giving transit to that peer for those routes. Peers only want to see you and downstream only.

Time to roll up my sleeves and comment on your rules.

Instance
set default as=xxx out-filter=Tag-originated

filters
 0   chain=Tag-originated bgp-communities=xxx:400 locally-originated-bgp=yes invert-match=no action=accept set-bgp-prepend-path=""

So long as the bgp instance uses Tag-originated as its out-filter, then this part is good to go. You should see these communities on your testbed.

 1   ;;; BGP out to Transit Rule
     chain=BGP out to Transit bgp-communities="" invert-match=no action=passthrough set-bgp-prepend-path="" set-bgp-communities="" 

 2   ;;; BGP out to Transit - Customers
     chain=BGP out to Transit bgp-communities=xxx:300 invert-match=no action=accept set-bgp-prepend-path="" set-bgp-communities="" 

 3   ;;; BGP out to Transit - Local
     chain=BGP out to Transit bgp-communities=xxx:400 invert-match=no action=accept set-bgp-prepend-path="" set-bgp-communities="" 

 4 X ;;; BGP out to Transit Reject
     chain=BGP out to Transit bgp-communities="" invert-match=no action=reject set-bgp-prepend-path=""

If your routes aren’t being announced with rule 4 active, then this means that your rules to set communities aren’t working right…
In the routes view, you can add the BGP Communities column to the display, if that helps… Do that and make sure your tagging rules are working when you receive routes.

FYI - I find it easier to read /routing filters export - but I know that’s harder to filter sometimes (no where x=y that I’m aware of)

skipping the out-to-peers rules, as the same comments apply there…

my prefixes

10   chain=BGP out to Transit prefix=xxx.0/24 bgp-communities="" invert-match=no action=accept set-bgp-prepend-path="" set-bgp-communities="" 
11   chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path="" set-bgp-communities="" 
12   chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path="" 
13   chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path="" 
14   chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path="" 
15   chain=BGP out to Transit prefix=xxx.0/24 bgp-communities="" invert-match=no action=accept set-bgp-prepend-path="" set-bgp-communities="" 
16   chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path="" 
17   chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path="" set-bgp-communities="" 
18   chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path="" set-bgp-communities="" 
19   chain=BGP out to Transit prefix=xxx.0/24 invert-match=no action=accept set-bgp-prepend-path="" set-bgp-communities=""

Okay - a few things here. First, make sure that there’s no criteria of bgp-communities=“” on your rules unless you really mean “this route must have no communities” because that’s what that means…
Also, you don’t need the set-bgp-communities=“” action on all of the accept rules so long as the first rule in the chain is passthrough and does this action. By the time the accept rules hit, the passthrough rule has already worked its magic.
Same holds for the prepend, but perhaps that’s just showing in the output of your “print where blahblah” commands…

Here’s where things got wonky:

customer prefix 

21   ;;; Accept customer-owned prefix and subnets xxx
     chain=Customer IN prefix=xxx0/24 [color=#FF0000]prefix-length=0[/color] invert-match=no action=accept set-bgp-prepend-path="" set-bgp-communities="" 

22   ;;; Accept customer-owned prefix and subnets xx
     chain=Customer IN prefix=xxx.0/24 [color=#FF0000]prefix-length=0[/color] invert-match=no action=accept set-bgp-prepend-path="" set-bgp-communities="" 

23   ;;; Tag all prefixes with customer community
     chain=Customer IN prefix=0.0.0.0/0 prefix-length=0-32 invert-match=no action=passthrough set-bgp-prepend-path="" set-bgp-communities=xxx:300

Rule 23 should be first in this chain

Then for each customer-owned prefix, prefix-length shouldn’t be zero because that would always fail. prefix-length should be a range of x-24 where x is the mask of the customer’s CIDR block (e.g. /22 → prefix-length=22-24)
Finally, after all of the “accept” rules for each customer prefix (no need to put bgp actions on each one, those are there just to enumerate exactly what IP ranges you will accept from the customer. The passthrough at the beginning does all of the “do this for every route” actions) you need a discard all rule.
chain=Customer IN action=discard

Maybe a little playing around in a lab environment would be helpful for you before you pull the trigger again.

15

Hi,

A question about this, when you set the “default-chain” attribute, is this chain used even if you set a different chain (a custom one) per peer?

It’s my understanding that this chain is only used whenever there’s no other chain explicitly set for the peer.

16

It uses both - at least according to my results when I did a quickie GNS3 lab to test what I was recommending here.
The global bgp-out filter on the instance itself will run first, and then any peer-out filter will run after that, so you could make, for instance, a global safety net that blocks all bogon prefixes or something like that, so that you can never accidentally allow them to a transit peer.
It appears that anything accepted by the global must also then be accepted by the per-peer, which makes sense to me.

In my configurations for this thread, the global-out simply tags all locally-originated prefixes (locally = inside this very same router, and not local-as) with the :400 community so that the per-peer filters will be able to recognize them as “my prefixes” and allow them.

17

Cool, nice to know this can be done too. Thanks for the reply

18

All seems better, but now we are seeing that we are announcing some prefixes that are not ours. It shows in the communities columns that 0:0 then example 174:xx then a couple others.. They are not all cogent after 0:0. We do not peer with or have Cogent in our mix. Where are these coming from and how do I block them. My IN filter is accept everything. Its beeing used in transit IN and Peers IN.

Regards,

Edit - If I disable one particular Upstream provider, They go away… There a couple here and there that are coming from somewhere else, but nothing like before. What am I missing.. :frowning: so close…

The other ones are coming from another IX Route server where peering with…

19

If you’re sending a route to a neighbor that shouldn’t be sent there, then one of two things is happening: either the route isn’t being properly tagged when you receive it or else the output filter to that neighbor is not properly using your tags.

Look at the communities on the route in your routing table to see if it has the right community (e.g xxxx:300) - if not then that’s why- fix the input filter on the peering where your network learns the route. If it IS tagged properly, then fix the output filter on the neighbor who’s incorrectly receiving the route.