Newbie BGP issues

RouterOS Forwarding Protocols
1

Guys & Gals,

New to BGP on a Mikrotik however having issues when using a single instance with multiple peers and route advertisement and traffic going out 1 path and back another through different parts of the country however when I set each peer as their own instance all works as expected.

BGP configuration:

/routing bgp instance
add as=588XX client-to-client-reflection=no name=588XX \
    redistribute-connected=yes redistribute-ospf=yes redistribute-static=yes \
    router-id=0.0.0.1
/routing bgp peer
add instance=588XX name=NSW-IX-1 out-filter=ix-nsw-out \
    remote-address=X.100.52.1 remote-as=76XX ttl=default
add instance=588XX  name=NSW-IX-2 out-filter=ix-nsw-out \
    remote-address=X.100.52.254 remote-as=76XX ttl=default update-source=\
    "vlan3098 - IX NSW"
add instance=588XX name=QLD-IX-1 nexthop-choice=force-self out-filter=\
    ix-qld-out-1 remote-address=X.100.76.1 remote-as=76XX ttl=default
add instance=588XX name=QLD-IX-2 nexthop-choice=force-self out-filter=\
    ix-qld-out-1 remote-address=X.100.76.254 remote-as=76XX ttl=default
add disabled=yes instance=588XX name=B1-Transit out-filter=ip-b1 \
    remote-address=X.122.114.234 remote-as=381XX ttl=default

/routing filter
add action=accept chain=ix-nsw-out prefix=X.17.250.0/23
add action=accept chain=ix-nsw-out prefix=X.17.252.0/23
add action=accept chain=ix-nsw-out prefix=X.241.148.0/23
add action=accept chain=ix-nsw-out prefix=X.27.248.0/24
add action=reject chain=ix-nsw-out
add action=accept chain=ix-qld-out-1 prefix=X.17.250.0/23
add action=accept chain=ix-qld-out-1 prefix=X.17.252.0/23
add action=accept chain=ix-qld-out-1 prefix=X.241.148.0/23
add action=accept chain=ix-qld-out-1 prefix=X.27.248.0/24
add action=reject chain=ix-qld-out-1
add action=accept chain=ip-b1 prefix=X.X.250.0/23
add action=accept chain=ip-b1 prefix=X.X.252.0/23
add action=accept chain=ip-b1 prefix=X.X.148.0/23
add action=accept chain=ip-b1 prefix=X.X.248.0/24
add action=accept chain=ip-b1 prefix=X.X.118.0/23
add action=reject chain=ip-b1
2

Okay - first of all, don’t do this:

It works but it’s wrong - like a teenager learning to drive and using their right foot on the gas and their left foot on the brake.
There’s almost never a good reason to redistribute routes into public BGP - For one thing, BGP considers redistributed routes as inferior to properly-originated routes when making routing decisions.

Follow Best Practices:

Properly originate your prefixes using network statements:

/routing bgp network
add network=X.17.250.0/23
add network=X.17.252.0/23
add network=X.241.148.0/23
add network=X.27.248.0/24

What these mean is that whenever an exactly-matching destination is active in the IP routing table, it should be advertised in BGP.
If you don’t check the “synchronize” checkbox (or specify synchronize=no on the command line), then these networks will be unconditionally advertised in BGP. This may or may not be good behavior for your network. Typical design is that you should require synchronization because if the router loses its connection to one of those networks, it should not be telling the Internet that it is a way to get to that destination anyway… (this comes into play more in larger scale networks, but if you start good habits early, then you’ll have an easier time when things grow)

If using synchronize=yes (which is the default behavior in most BGP implementations) then make sure that exactly these prefixes do appear in your routing table somewhere.
If you have a route for x.27.248.128/26 in your routing table, that will NOT activate the x.27.248.0/24 prefix - you must have a route for exactly x.27.248.0/24
You can create the master prefixes as blackhole routes if you want them to ALWAYS be announced (or just disable synchronization).
Blackhole routes also have the advantage of dumping traffic destined for un-assigned IPs within your blocks - for example if a certain unused range of IPs is getting scanned, then the scan packets just get discarded by your router without being forwarded along to something further in your network - or worse: forwards them to the default GW, causing ping-pong packets on your Internet links.

Use BGP Communities to control your prefixes better:

Next item, and most likely the source of your problems - you should consider defining some BGP communites to use on your routes and filter based on those.
I’m not sure whether all 4 of these prefixes are local to your network, or if you have customers announcing one or more of them to you using eBGP. If the latter, then it is crucial that you implement communities. If your direct connection to the customer goes down, but the customer has other active BGP routes to the Internet, then your router will learn the customer’s prefix from eBGP with one of your upstream providers. When you filter only by IP, it doesn’t matter where you learned the prefix - it only matters what prefix it is, so you will announce your customer’s prefix to other peers, even though all you’re going to do is hand it to one of your Internet connections and be done with it.

For any customer prefixes, you should make an IN-filter rule on their neighbor which sets/adds some community - e.g. 588XX:100
For any IX or Internet peering sessions, require that the customer prefix must also have community 588XX:100 in order to be permitted.

3

Thanks for the great reply!

So I reinstated the networks as your suggested as best practice. I’ve also overnight been looking at the communities and BGP prepend however neither seemed to make a difference.

At this stage I am only trying to configure the IX directly which we have 1 IX provider with two peers in different states of the country QLD & NSW.

When I visit the trusty speedtest.net and select a speedtest server in QLD the download comes directly however the upload goes via NSW to QLD. The IX provider publishes these on their website:

So I invested the time to try prepend thrice to the NSW peer however no change was noticed:

I also tried the reverse, 588XX:3

As you will see traffic is still back arsewoods:

As it stands my BGP configuration is:

/routing bgp instance
set default as=65000 disabled=yes
add as=588XX client-to-client-reflection=no name=588XX router-id=0.0.0.1

/routing bgp network
add network=XXX.17.250.0/23
add network=XXX.17.252.0/23
add network=XXX.241.148.0/23
add network=XXX.XXX.248.0/24
add network=XXX.122.118.0/23

/routing bgp peer
add in-filter=ix-nsw-in instance=588XX name=NSW-IX-1 out-filter=ix-nsw-out \
    remote-address=XXX.100.52.1 remote-as=7606 ttl=default
add in-filter=ix-nsw-in instance=588XX name=NSW-IX-2 out-filter=ix-nsw-out \
    remote-address=XXX.100.52.254 remote-as=7606 ttl=default update-source=\
    "vlan3098 - IX NSW"
add instance=588XX name=QLD-IX-1 nexthop-choice=force-self out-filter=\
    ix-qld-out-1 remote-address=XXX.100.76.1 remote-as=7606 ttl=default
add instance=588XX name=QLD-IX-2 nexthop-choice=force-self out-filter=\
    ix-qld-out-1 remote-address=XXX.100.76.254 remote-as=7606 ttl=default
add disabled=yes instance=588XX name=APEXn-B1-Transit out-filter=ip-b1 \
    remote-address=XXX.122.114.234 remote-as=38195 ttl=default

/routing filter
add action=accept chain=ix-nsw-in
add action=accept chain=ix-nsw-out prefix=XXX.17.250.0/23 \
    set-bgp-communities=588XX:3 set-bgp-prepend=3 set-distance=50
add action=accept chain=ix-nsw-out prefix=XXX.17.252.0/23 \
    set-bgp-communities=588XX:3 set-bgp-prepend=3
add action=accept chain=ix-nsw-out prefix=XXX.241.148.0/23 \
    set-bgp-communities=588XX:3 set-bgp-prepend=3
add action=accept chain=ix-nsw-out prefix=XXX.XXX.248.0/24 \
    set-bgp-communities=588XX:3 set-bgp-prepend=3
add action=reject chain=ix-nsw-out
add action=accept chain=ix-qld-out-1 prefix=XXX.17.250.0/23
add action=accept chain=ix-qld-out-1 prefix=XXX.17.252.0/23
add action=accept chain=ix-qld-out-1 prefix=XXX.241.148.0/23
add action=accept chain=ix-qld-out-1 prefix=XXX.XXX.248.0/24
add action=reject chain=ix-qld-out-1
add action=accept chain=ip-b1 prefix=XXX.17.250.0/23
add action=accept chain=ip-b1 prefix=XXX.17.252.0/23
add action=accept chain=ip-b1 prefix=XXX.241.148.0/23
add action=accept chain=ip-b1 prefix=XXX.XXX.248.0/24
add action=accept chain=ip-b1 prefix=XXX.122.118.0/23
add action=reject chain=ip-b1
4

The communities you’ve posted are how you modify the IX’s policy regarding routes towards you (your download)… and more broadly, they affect how the world sees your network via that peer.

The upload choice is to be made by your router’s view of the routing table. Are you receiving full routes, partial routes, or default-only?
If default-only, then there’s no surprise that one is always preferred over another - you basically only have one route to use: 0.0.0.0/0 and it then boils down to which default GW you’re choosing as the primary, and which is the backup. In that case, you really can’t make any specific policy for your outbound routing any more granular than just “primary/backup”

If you receive partial or full routes, then you’re going to have to do some detective work with your routing table - you’ll need to determine what the QLD speedtest server’s IP is, and then find out what prefixes you’re receiving for that destination, and determine why your router is choosing the NSW peer to reach this.

Obviously, a more specific route (longer prefix length) will win over a more generic route, but if there are two or more identical prefixes being learned from multiple peers, then the BGP Best Path Selection Algorithm determines which one will be used and activated in the IP routing table. (this is a Cisco document, but it’s pretty much the same thing for Mikrotik, which even supports the Weight metric). Note that it’s this algorithm which is the reason you want all of your peerings to be done in just one BGP instance. You will lose all of this granularity if you use multiple instances.

If you determine that you’re receiving the same route from both peers, and that the best path selection results in a sub-optimal choice, then you’ll have to implement some policy in your input filters which bumps or lowers the routes’ desirability based on some tangible criteria (e.g. prefer NSW for certain destination ASNs) Perhaps the IX sends communities to you on their announcements, and maybe they supply communities that denote the region they’re originated from. Say they send you a community that means “from NSW” - you could then add a filter rule which matches this community and sets the local_pref to 110 on the NSW peers.

Note that I mentioned local_pref - this is the most common way to dictate your entire ASN’s preference as to ways to reach some destination or other. Local Pref will beat any other metric except weight (which is not even included in BGP messages - it’s just a thing internal to any one router, and it was originally a Cisco-proprietary thing).