I am replacing a former OpenWRT-based P2P link with two RB912 devices.
So far the wireless part works like a charm, good signal and performance.
But I run into trouble with my BGP configuration.
Both sites form an AS (site A AS 64600, site B AS 64601).
Each site has a backbone core consisting of several routers. OSPF and iBGP is spoken within the core. Two routers in the core act as iBGP route reflectors.
ether1 of the MikroTik device is attached to the core. OSPF is working properly to reach all internal networks. I was also able to successfully setup the iBGP peering to the two iBGP route reflectors.
Between both MikroTik devices I established an eBGP peering using wlan1.
To make it easier to understand, I visualized the relevant part of the network in the attached picture. I will now focus on AS 64601 (though the same applies to AS 64600 too).
So far the peerings work. The RB also learns all routes from iBGP. But those routes are not propageted via my eBGP peering to the other RB. At the moment I do not have any filters in use on the RB.
Here is my BGP configuration:
[admin@BaseBox5-CPE] > /routing bgp export
mar/08/2014 22:04:24 by RouterOS 6.10
/routing bgp instance
set default as=64601 router-id=172.24.8.2
/routing bgp peer
add name=basebox5-ap nexthop-choice=force-self remote-address=172.24.3.33 remote-as=
64600 tcp-md5-key=XXX ttl=1 update-source=wlan1
add name=fortigate nexthop-choice=force-self remote-address=172.24.8.4 remote-as=64601
ttl=1 update-source=ether1
add name=cisco nexthop-choice=force-self remote-address=172.24.8.3 remote-as=64601 ttl=
1 update-source=ether1The BGP status shows, that prefixes are learned from my iBGP peers:
[admin@BaseBox5-CPE] > /routing bgp peer print status
Flags: X - disabled, E - established
0 E name="basebox5-ap" instance=default remote-address=172.24.3.33 remote-as=64600
tcp-md5-key="XXX" nexthop-choice=force-self multihop=no
route-reflect=no hold-time=3m ttl=1 in-filter="" out-filter="" address-families=ip
update-source=wlan1 default-originate=never remove-private-as=no as-override=no
passive=no use-bfd=no remote-id=172.24.4.30 local-address=172.24.3.34
uptime=22m36s prefix-count=1 updates-sent=0 updates-received=1 withdrawn-sent=0
withdrawn-received=0 remote-hold-time=1m30s used-hold-time=1m30s
used-keepalive-time=30s refresh-capability=yes as4-capability=yes
state=established
2 E name="cisco" instance=default remote-address=172.24.8.3 remote-as=64601
tcp-md5-key="" nexthop-choice=force-self multihop=no route-reflect=no hold-time=3m
ttl=1 in-filter="" out-filter="" address-families=ip update-source=ether1
default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no
remote-id=172.24.8.3 local-address=172.24.8.2 uptime=18m35s prefix-count=17
updates-sent=0 updates-received=17 withdrawn-sent=0 withdrawn-received=0
remote-hold-time=30s used-hold-time=30s used-keepalive-time=10s
refresh-capability=yes as4-capability=yes state=establishedAnd now the most interesting part (and what confuses me), there is no single route advertised by the RB. I would expect the RB to announce the routes learned from the iBGP peering to its eBGP peer.
[admin@BaseBox5-CPE] > /routing bgp advertisements print
PEER PREFIX NEXTHOP AS-PATH ORIGIN LOCAL-PREFAlso to confirm this, I do not receive any prefixes on the other RB.
[admin@BaseBox5-AP] > /routing bgp peer print status where name="BaseBox5-CPE"
Flags: X - disabled, E - established
5 E name="BaseBox5-CPE" instance=default remote-address=172.24.3.34 remote-as=64601
tcp-md5-key="XXX" nexthop-choice=force-self multihop=no
route-reflect=no hold-time=1m30s ttl=1 in-filter="" out-filter=""
address-families=ip update-source=wlan1 default-originate=if-installed
remove-private-as=no as-override=no passive=yes use-bfd=no remote-id=172.24.8.2
local-address=172.24.3.33 uptime=47m26s prefix-count=0 updates-sent=1
updates-received=0 withdrawn-sent=0 withdrawn-received=0 remote-hold-time=3m
used-hold-time=1m30s used-keepalive-time=30s refresh-capability=yes
as4-capability=yes state=establishedIs there anything special I need to configure in order to propagate/pass BGP routes to other eBGP peers? With my former OpenWRT device (which used Quagga as routing daemon) this worked as expected.
redistribute-other-bgp did not help (as it only controls the redistribution between multiple bgp instances and I only use a single instance).
The other redistribution commands are not what I want. I actually do not want any new routes to be announced by the RB. I only want the RB to pass already learned routes to other peers.
Announcing a new network by using the “network” command basically works (in a sense that the route will be advertised to all neighbors); but as stated above, this is not my intention. In the shown core network there are even more routing devices like VPN concentrators for instance. They all propagate certain summarized routes into BGP (which the RB learns through iBGP). I do not want to manually add all these networks on the RB again - that’s why I am going to use a dynamic routing protocol. Furthermore the network command makes the defined network originate at the RB, which is wrong for routes having their origin somewhere else (e.g. VPNs coming from other AS).
So I am missing the possibility to simply pass (and not to originate) BGP routes; which in my opinion is a very basic and vital feature. I most likely guess that I am missing some kind of configuration parameter which I haven’t figured out yet. Still hoping somebody might already have configured such a scenario with success and can tell me what’s wrong with my configuration. On the other hand I cannot exlude for sure whether this might be a bug in the current RouterOS version I am using (v6.10).
From my understanding (and I ran into this problem myself) you either need to set up your instance as a route-reflector, or create different instances for eBGP/iBGP (which is what I ended up doing) in a similar circumstance when I had a peering connection to Router A and a L2 to Router B from A and wanted to share the routes. I thought instance was for each AS you’d be running as, but as i’ve found you kind of need multiple instances to get by.
I have a eBGP instance as I wouldn’t be sharing eBGP routes with other eBGP neighbors, and an iBGP instance, actually currently an iBGP instance per site I connect to. I plan to move these into one instance as route reflectors though.
Interesting hint, I did not have had the idea to create two separate instances for iBGP and eBGP (though this should not be necessary).
So on the RB CPE I created a second instance called eBGP and moved my eBGP peer into this new instance. I activated redistribute-other-bgp for that new instance. According to your theory this should help redistributing the iBGP routes into eBGP. Unfortunately after the peering re-established, nothing changed. Only a route I manually added with the network statement was advertised to the other RB. No single iBGP learned route was redistributed.
I also did a second try by using a single instance and activating route reflection towards my eBGP peer. Same unchanged behaviour here. I guess this does not work since as far as I know route reflection only works within the same AS (which of course would make sense). On a border router towards another AS simply all known BGP routes should be advertised on that eBGP link (as long as they are not filtered out of course).
Interestingly routes learned via eBGP are being properly advertised towards the iBGP peers (without any fancy configuration). So when routes from eBGP are announced to iBGP (as you would expect), why does this fail vice versa?
Are both peers in both iBGP/eBGP instance set to redistribute other BGP? I have 2 way redistribution this way.
Are you running any filters blocking the routes?
Usually iBGP can be many small routes lower than /24’s etc, when talking to eBGP you usually want cleaner set routes and nothing less than a /24 ideally. Maybe /25, so forwarding iBGP straight to your eBGP peer, they may also be filtering you on their inbound.
I have configured the two instances like this:
[admin@BaseBox5-CPE] > /routing bgp instance export verbose
mar/10/2014 00:48:33 by RouterOS 6.10
/routing bgp instance
set default as=64601 client-to-client-reflection=yes !cluster-id
!confederation disabled=no ignore-as-path-len=no name=default out-filter=
"" redistribute-connected=no redistribute-ospf=no redistribute-other-bgp=
yes redistribute-rip=no redistribute-static=no router-id=172.24.8.2
routing-table=""
add as=64601 client-to-client-reflection=yes !cluster-id !confederation
disabled=no ignore-as-path-len=no name=eBGP out-filter=""
redistribute-connected=no redistribute-ospf=no redistribute-other-bgp=yes
redistribute-rip=no redistribute-static=no router-id=172.24.3.34
routing-table=""If I understood you correctly, this is what you suggested. I.e. two instances, one (default) for iBGP and another one (eBGP) for eBGP. Both have redistribute-other-bgp enabled.
And this is my peer configuration:
[admin@BaseBox5-CPE] > /routing bgp peer export verbose
According to my BGP status I receive several prefexis from the two iBGP peers.
[admin@BaseBox5-CPE] > /routing bgp peer print status detail
Flags: X - disabled, E - established
0 E name="basebox5-ap" instance=eBGP remote-address=172.24.3.33
remote-as=64600 tcp-md5-key="XXX"
nexthop-choice=force-self multihop=no route-reflect=yes hold-time=3m
ttl=1 in-filter="" out-filter="" address-families=ip update-source=wlan1
default-originate=never remove-private-as=no as-override=no passive=no
use-bfd=no remote-id=172.24.4.30 local-address=172.24.3.34 uptime=7m5s
prefix-count=2 updates-sent=2 updates-received=2 withdrawn-sent=1
withdrawn-received=2 remote-hold-time=1m30s used-hold-time=1m30s
used-keepalive-time=30s refresh-capability=yes as4-capability=yes
state=established
2 E name="cisco" instance=default remote-address=172.24.8.3 remote-as=64601
tcp-md5-key="" nexthop-choice=force-self multihop=no route-reflect=no
hold-time=3m ttl=1 in-filter="" out-filter="" address-families=ip
update-source=ether1 default-originate=never remove-private-as=no
as-override=no passive=no use-bfd=no remote-id=172.24.8.3
local-address=172.24.8.2 uptime=8m6s prefix-count=14 updates-sent=3
updates-received=15 withdrawn-sent=1 withdrawn-received=1
remote-hold-time=30s used-hold-time=30s used-keepalive-time=10s
refresh-capability=yes as4-capability=yes state=establishedBut I am still missing the advertisement of these routes towards my eBGP peer. I only see the single network I manually added by using the network statement.
[admin@BaseBox5-CPE] > /routing bgp advertisements print
PEER PREFIX NEXTHOP AS-PATH ORIGIN LOCAL-PREF
baseb... 172.24.8.0/23 172.24.3.34 igp
forti... 172.24.0.0/21 172.24.8.2 64600 igp 100
forti... 172.24.8.0/23 172.24.8.2 igp 100
cisco 172.24.0.0/21 172.24.8.2 64600 igp 100
cisco 172.24.8.0/23 172.24.8.2 igp 100Also as you can see from my configuration I do not use any filtering at all. Indeed it looks like something would filter out the routes; but there is nothing.
Of course I would apply some outgoing filters to my eBGP peering later when I got this working to supress smaller prefixes. There are also already summarized prefixes coming in from iBGP which entered the AS for example at the VPN concentrator and are reflected by the two iBGP route reflectors to all other BGP routers within the AS. The prefixes contain other customer networks connected via VPN which I want to pass along the 802.11n bridge to the other RB on the other AS. There might be also a connection to those VPN systems at the other AS which gives the desired redundancy as soon as the prefixes will be propagated through the 802.11n link between the two RB (like it did before with openwrt/quagga).
Are you also running ROS 6.10? Or can you spot any relevant difference between my posted configuration and yours?
Okay, this explains why the redistribution does not work.
But if I change my eBGP instance to AS 64600, then I would not have an eBGP peering between the two RB anymore. Instead this would result in the eBGP transition between the two AS to take place on my RB CPE (between the two instances). This is not really a feasable design and breakes almost all known BGP designs.
To summarize after all your experiences, the suggested solutions with route reflection and/or creating separate instances are all just workarounds. Normally a route propagation from iBGP to eBGP should work “out of the box” like all other BGP implementarion I know behave (Cisco, Juniper, Fortinet, Quagga). Curiously the way vice versa from eBGP into iBGP actually works with ROS.
In the end this seen behaviour makes it very hard (if not impossible) to integrate ROS based systems in an existing BGP infrastructure.
For me this sounds pretty much like a bug as I cannot see a real reason why iBGP to eBGP propagation should not work.
After gathering all these information I will try to raise a support ticket directly at MikroTik - and hope wether they can replicate and hopefully fix this.
This isn’t right. I’m running 2 instances on the same AS and redistributing BGP both ways between the 2 instances, no problems.
Here is my ROS (6.10) export:
/routing bgp instance
set default as=123456 redistribute-connected=yes redistribute-other-bgp=yes redistribute-static=yes router-id=1.1.1.2
add as=123456 name=to-other-core redistribute-connected=yes redistribute-other-bgp=yes redistribute-static=yes router-id=1.2.1.2
/routing bgp peer
add in-filter=DENY-DEFAULT1 multihop=yes name=other-site remote-address=1.1.1.1 remote-as=123456 ttl=default
add in-filter=DENY-DEFAULT instance=to-other-core name=other-core remote-address=1.2.1.1 remote-as=123456 ttl=defaultHave changed some things for my own privacy - but all 3 routers are in the same AS, the mikrotik almost acting as a bridge between them. I am redistributing routes in and out both ways.
We have two Edge routers, doing eBGP to different External Peers, and iBGP and OSPF running between the two Edge routers.
A Couple of hints / tips etc.
In our case we want to have a full table on both iBGP peers, as such we have route-reflection setting turned on. This is in two places.. one is “Client-to-Client” reflection setting under the default bgp instance setting, and the other is “Route-Reflector” under the BGP Peer Instance setting.
We pass routes to eBGP, via filters. Not just our routes but also routes for downstream BGP customers.
Meanwhile I could figure out why RouterOS does not properly propagate the BGP routes in my scenario. It is actually some “special” behaviour of the BGP implementation in RouterOS.
Normally you would expect all routes learned via BGP to be propagated to all other BGP peers (assuming there are no filters, loops, etc.). In other words, whatever you have in your BGP database/table should be forwarded.
Well, the BGP stack in RouterOS does one additional check while forwarding BGP routes: For each route which should be forwarded via BGP it checks whether that route is actually an active route in the regular routing table.
In my scenario I use OSPF with better/lower administrative distances to control the traffic flow within each AS while I use BGP to propagate the routes between the AS’. This means regarding the actual routing decision the OSPF route override the BGP route in the ROS routing table (fdb); which is intended. But since now the OSPF route is the active one in the routing table (and not the BGP route) this means that the BGP route will not be propagated to any other BGP peer either (because the check whether the BGP route is active fails).
Of course it depends on your actual network design, but in my case this makes running multiple routing protocols like BGP and OSPF simultaneously on a RouterOS device nearly impossible. From other vendors (Cisco, Fortinet, Juniper, Extreme, and software daemons like Quagga) I am used to have multiple routing protocols on certain devices. And having OSPF to control the intra-AS traffic while having BGP for route propagation outside the AS as nothing special for me either.
Finally I found that Juniper JunOS has a similiar feature in their BGP implemenation. They also check whether a BGP route is actually an active route in the routing table while propagating it to other BGP peers (while Cisco and Fortinet do not have such checks as far as I know). But - and that’s the good thing - JunOS has a configurable option called “advertise-inactive” where you can (de)activate this additional check.
Finally I ended up discussing this with MikroTik support some months ago. Here is what I wrote to summarize this issue:
MikroTik support told me that it is not possible to change the behaviour of the current v6 BGP stack.
But they are coding a new BGP implementation for v7 where they “could add this feature easily”.
So we can at least hope that the BGP implementation in v7 will be much better and solve some problems and limitations we currently have.
How is the check you speak of not just the loop prevention that iBGP does? I.e. iBGP only advertises locally originating routes thus needing a full mesh or route-reflector.
Good question - which most likely only the guy how coded the ROS BGP implementation can answer.
I am absolutely with you that there shouldn’t be any “implicit fancy automatic filters”. Nobody knows them (I am not aware that this feature which filters out non-active routes was documented somewhere), they are sometimes hard to understand, there is always some risk of breaking something, and it’s even worse when you are unable to disable them.
My guess is that they tried to implement some additional checks on-top to make sure all traffic is actually forwarded the same way as it is propagated by the routing protocol. In this scenario when you have iBGP, eBGP and OSPF co-existent, traffic for a BGP route may actually be forwarded by using a OSPF route (when OSPF overrides BGP in the fib). But that’s fine - and in my case exactly what I want.
My scenario is that I have OSPF and iBGP wihtin my AS and eBGP to other AS’. iBGP is used to carry the BGP routes from one end of my AS to the other end
← eBGP - [R1] ← iBGP → [R2] - eBGP →
Within my AS (i.e. between R1 and R2) I also use OSPF to intentionally influence the actual routing process within the AS (i.e. between R1 and R2). In my case I want to force the traffic flowing through a dedicated firewall if possible. To achieve this the firewall redistributes the BGP routes into OSPF with the firewall being the next-hop. So R1 and R2 still have their BGP adjacencies to exchange the BGP routes but when the firewall is online they “divert” the traffic through the firewall because the OSPF distance is better so the OSPF routes through the firewall are the active route used in the routing tables. This is quite nice because it lets me force traffic flowing through a firewall while still having the regular path as a fallback. But then R2 does not propagate the BGP routes from R1 anymore; because they are filtered out by this implicit “filter inactive routes” feature.
If I had the choice, I would love to see the same advertise-inactive option like JunOS has. Which means the currently existing “filter out inactive routes” code in ROS should be surrounded by an if-clause so you can skip those checks if you want.
Basically the ROS BGP stack is not that bad, but there are some quirks which can result in some real show-stoppers.
I hope that this information might also help other people if they are facing similiar “strange behaviour” of their ROS BGP. Just keep in mind that routes are only propagated to other eBGP peers when they are actually present and active in the regular routing table. And look forward to the new BGP stack in ROS v7.
