This question seems to come up a lot in the MikroTIk forums and I'm curious to see what different types of CCR BGP deployments are out there as well as the experience of the end user. Some users seem to have no issues with multiple public feeds and some aren't stable on only one feed.

So....

1) What kind CCR (s) are you using in your network
2) How many BGP feeds / Routes are you taking in
3) Are you stable or do you experience issues - if so, what kind
4) Software version used

We hear so many good/bad stories on CCR BGP, it might be helpful to have all the info in one thread.

Good afternoon

very strange request from you.
don't you have any CCR with BGP in production ?
aren't you "The Largest Carrier-Grade MikroTik Integrator in North America" ?

best regards
Thierry

This question seems to come up a lot in the MikroTIk forums and I'm curious to see what different types of CCR BGP deployments are out there as well as the experience of the end user. Some users seem to have no issues with multiple public feeds and some aren't stable on only one feed.

So....

1) What kind CCR (s) are you using in your network
2) How many BGP feeds / Routes are you taking in
3) Are you stable or do you experience issues - if so, what kind
4) Software version used

We hear so many good/bad stories on CCR BGP, it might be helpful to have all the info in one thread.

Great post.

We will be switching back from our cisco 7200 to the CCR on Sat morning for BGP.

1) CCR1036
2) 5 Feeds in total (local, international, peeing, a private peer, and Hurricane)
3) Was stable for 5 months on - then instability with random reboots (Tried a different CCR, new config, old config, ROS V6.10, Swapped SFP's, Ran BGP only on one interface)
4) V6.7 and V6.10

We are dedicating a CCR for ONLY BGP now on two SFP's with three vlans for int/local/peering. No queues, no advanced firewall handling etc and watchdog turned off as we suspect its a CPU issue and watchdog rebooting it. Another CCR will be used for the queues/firewall and IGP. We will try V6.13 on our BGP router

Will provide feedback on Monday if it is stable.

Good afternoon

very strange request from you.
don't you have any CCR with BGP in production ?
aren't you "The Largest Carrier-Grade MikroTik Integrator in North America" ?

best regards
Thierry

Touche!...you are correct, we ARE the largest Carrier-Grade Integrator in North America and deploy BGP/MPLS solutions to many large wireline/wisp carriers. We work with a lot of CCRs in BGP peering and have seen various issues. We tend to limit full table peerings on CCRs to no more than 2 based on the limitation of a multi-threaded BGP process. Since we are an integrator, most of our customers use Cisco and Juniper still for public BGP peering and are reluctant to migrate. Large carriers don't like being beta testers in general - stability and conservative design is the name of the game.

The intent of the thread is to try and centralize information and experiences on peering with full global BGP tables on CCR and see if there are common issues across software versions and/or types of peering routers.

1) What kind CCR (s) are you using in your network

CCR1036-12G-4S. Would really like to see a 12-port SFP version.

2) How many BGP feeds / Routes are you taking in

One transit provider on the CCR. Other transit providers are on other platforms, but the plan is to move to CCRs, one per provider. Taking full tables.
If things continue to go well, we'll also move edge aggregation to CCR. Core switching will be on other platforms.

3) Are you stable or do you experience issues - if so, what kind

Stable since deployment about 2 weeks ago. Most issues are just learning the platform, there are some oddities. Diagnostics are probably the biggest problem: 'show ip route' would be nice.

4) Software version used

6.12

1) What kind CCR (s) are you using in your network

CCR1036-12G-4S. Would really like to see a 12-port SFP version.

There is the CCR1016-12S-1S+, only half the cores but unless you are doing lots of data it should still allow it.

Or there is a CCR1072-1G-8S+ on the horizon I believe (Q3 2014), which is 72 cores and 8 (or more) SFP+ ports

2) How many BGP feeds / Routes are you taking in

One transit provider on the CCR. Other transit providers are on other platforms, but the plan is to move to CCRs, one per provider. Taking full tables.
If things continue to go well, we'll also move edge aggregation to CCR. Core switching will be on other platforms.

Why only 1 CCR per provider? (curious)

3) Are you stable or do you experience issues - if so, what kind

Stable since deployment about 2 weeks ago. Most issues are just learning the platform, there are some oddities. Diagnostics are probably the biggest problem: 'show ip route' would be nice.

There is this, but large tables can take a bit to get the answer, and it will show ever matching route, not just the preferred/taken route:

ros code

[admin@core1.bne1] > /ip route print where 8.8.8.8 in dst-address
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          <hidden>               1
 1 ADb  8.8.8.0/24                         <hidden>            200

We had problems around the point of 4 full feeds but I don't thing we ran the problem to ground. We were hitting various other CCR limitations at the time so it wasn't the most important item for resolution.

At this point I don't think it is possible to fully "integrate" Mikrotik equipment as part of a "carrier grade" solution. You can certainly interface Mikrotik equipment with carrier grade systems but I certainly would not describe any Mikrotik hardware as "carrier grade". That doesn't mean it isn't useful, good value or the appropriate answer for some applications.

There is the CCR1016-12S-1S+, only half the cores but unless you are doing lots of data it should still allow it.

For some reason I thought that was all 10 gig. Thanks for the tip!

Why only 1 CCR per provider? (curious)

Redundancy. One failure takes out only one provider. Low cost makes this very practical.

There is this, but large tables can take a bit to get the answer, and it will show ever matching route, not just the preferred/taken route:

ros code

[admin@core1.bne1] > /ip route print where 8.8.8.8 in dst-address
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          <hidden>               1
 1 ADb  8.8.8.0/24                         <hidden>            200

I really like that it shows the less-specifics. It gives functionality similar to 'show ip [route|bgp] <address> longer-prefixes' in cisco-land. It takes a good 60-90 seconds to do the lookup on a router with a full table though, making troubleshooting time consuming, which is incredibly frustrating in an outage situation.

Good afternoon

very strange request from you.
don't you have any CCR with BGP in production ?
aren't you "The Largest Carrier-Grade MikroTik Integrator in North America" ?

best regards
Thierry

Touche!...you are correct, we ARE the largest Carrier-Grade Integrator in North America and deploy BGP/MPLS solutions to many large wireline/wisp carriers. We work with a lot of CCRs in BGP peering and have seen various issues. We tend to limit full table peerings on CCRs to no more than 2 based on the limitation of a multi-threaded BGP process. Since we are an integrator, most of our customers use Cisco and Juniper still for public BGP peering and are reluctant to migrate. Large carriers don't like being beta testers in general - stability and conservative design is the name of the game.

The intent of the thread is to try and centralize information and experiences on peering with full global BGP tables on CCR and see if there are common issues across software versions and/or types of peering routers.

So with your experience you may know that

1) bgp needs a lot of cpu
2) bgp needs some memory
3) V6.x is not multi thread for BGP

but to help you

CCR1036-4G can handle with no problem 2 full table IPV4 + IPV6
if you need a lot of peers use x86 until v7 and 72 tilera cores

for compatibility as you seem not to be able to do testings it works fine with cisco, juniper, quagga, bird, huawey

our network is spread over europe with 9 bgp routers ( x86 ) with about 500 bgp sessions including peering and upstreams
it takes about 1 minute to propagate a full IPV4 table over all the network
you will not be able to do this even with SUP720 3BXL or MX80 ( we have tested in real production )

for public peering if you are connected on a peering point like AMSIX, DECIX, TOPIX, LUCIX ... you will see a lot of Mikrotik routers just with "ip neighbour" but they are not speaking about it

our customers are using CCR for connecting their upstreams and they have no issue but the secret is also about setup

regards
Thierry

1) We have 2 CCR1016-12G
2) We have BGP with 2 upstream with 2 session per upstream so, in total, 4 BGP session per router
3) We are stable, no issue at the moment with BGP
4) v6.12

TLCWEB Srl/Baleno Wireless the smaller WISP in south of Italy

We will be switching back from our cisco 7200 to the CCR on Sat morning for BGP.

1) CCR1036
2) 5 Feeds in total (local, international, peeing, a private peer, and Hurricane)
3) Was stable for 5 months on - then instability with random reboots (Tried a different CCR, new config, old config, ROS V6.10, Swapped SFP's, Ran BGP only on one interface)
4) V6.7 and V6.10

We are dedicating a CCR for ONLY BGP now on two SFP's with three vlans for int/local/peering. No queues, no advanced firewall handling etc and watchdog turned off as we suspect its a CPU issue and watchdog rebooting it. Another CCR will be used for the queues/firewall and IGP. We will try V6.13 on our BGP router

Will provide feedback on Monday if it is stable.

So far, very stable now for 54 hours on V6.13

we have been using 2x CCR1036-12G-4S since early 6.x without much issue. the catch, daily reboot on the router in off-peak hours.

first unit connected with 3 BGP sessions (2 full route, 1 local route ~80k), total throughput >300mbps

second unit, 2 BGP sessions (1 full route, 1 local route ~10k), total throughput >300mbps

so far so good

2 CCR both – CCR1036-12G-4S
2 feeds each with 2 sessions each feed, dualstack

1 carrier on fibre other on ethernet on each router

2 ipv4 on each with 244k routes and 488K the other
2 ipv6 on each with 15K and 12K on the other

software 6.12

No problems ...

Coming in to say CCR1036-8G-2S+ has been in production for nearly a month. Only taking default route still, but has 30K~ in iBGP/OSPF/IX routes without problem. Probably only about 12k are active routes though (lots of double ups over the IX's)

Peaking at about 600mbit / 100kpps aggregate bandwidth through a 802.3ad LACP trunk.

Have some basic ICMP stuff in the forward/input chain

Have some basic blocking in the input chain for Telnet/SSH

Have about 10 simple queues, with 2 mangle rules per queue (to match input/output traffic) on a VLAN interface that does 100-140mbit traffic at any point.

Mostly, I barely see the CPU above 10% at peak, so very stable and happy with performance so far.

Running ROS6.13

Spinning 2nd one up this week for stateful balance and failover. Will also start receiving full table's from 2 providers soon.

Ours have been running for 27 days with ROS 6.15 today and one full feed and a handfull of peers at an IX.

Works great so far!

Since we split our routing, i.e ospf/firewall rules/queues etc on one router and BGP on another CCR it has been running great.

system resource print
uptime: 8w3d9h14m22s
version: 6.13

Been running our CCR1036-12g-4s since they were available as a preproduction product with full bgp coming from two upstream providers totaling 730k routes passing 400mbps of traffic with no problems what so ever. This router only runs BGP on it though, no ospf or anything else. Currently running v6.9

Been running our CCR1036-12g-4s since they were available

How long ago was that?

full bgp coming from two upstream providers totaling 730k routes passing 400mbps of traffic with no problems what so ever

.

Good to hear. May I ask if you have done any reboots? Or table rebuilds? Interested in knowing how long it took to build the routing table.

1) What kind CCR (s) are you using in your network
two CCR-1036-12G

2) How many BGP feeds / Routes are you taking in
above 500k. One of the routers has 4 BGP peering sessions, the other has 5. At the moment we're peering with 2 ISPs, in the next month another two ISP peering sessions are planned.

3) Are you stable or do you experience issues - if so, what kind
twice, the issue was that all of the configured bridge interfaces were somehow wiped out from the running config. But I do not attribute this to BGP as it happen once long before BGP was deployed. In general - I haven't had any issues with it.

4) Software version used
6.17

Glad to see this thread has been useful...i've seen it linked in multiple other forum posts.

@Mikrotik - what is the status on balancing BGP across multiple cores?

Glad to see this thread has been useful...i've seen it linked in multiple other forum posts.

@Mikrotik - what is the status on balancing BGP across multiple cores?

That's something I would like to see too! This is something that bothers me, because it turns out that as far as routing is concerned - there's such a waste in capacity! So many cores stay idle while one of them is utilized to its limits.
I think I read somewhere that core load balancing is planned for version 7.0 but I'm not sure...

@Mikrotik - what is the status on balancing BGP across multiple cores?

As above poster said, I think this is being held off for v7.

There is a thread on here somewhere in relation to v7 where normis says they're alpha test builds are loading 15 million routes in 15 minutes, a million routes a minute is pretty darn good. You'd have to imagine this efficiency is being achieved with multi-core.

Hopefully filtering/adding/removing routes from a big route-table will be vastly improved too. Still running 6.13 on my main BGP routers but hell they are slow to add routes and filter looking for things.

My current setup:

3 x CCR1036-12G-4S running v6.18:

- London: 2 x full BGP transit, iBGP to Frankfurt, Madrid and London
- Frankfurt: 1x full BGP transit, iBGP to London
- Madrid: 1x full BGP transit, iBGP to London

1 x 1100AHx2 v5.26 (yes, it's old, but it's stable!):

- Dubin: Approx 20 peering sessions (INEX exchange), iBGP to London

The setup has been stable other than an issue where BGP 'disappears' in Frankfurt. At present I suspect a hardware issue - there is a thread about this issue that I have just written and is awaiting approval. (ETA: http://forum.mikrotik.com/viewtopic.php?f=14&t=88912)

My only criticism of BGP on RouterOS is the speed. It takes a good 5 minutes at present to do a ~500k route converge on a 1036. Fingers crossed that v7 will speed this up, then I will have no reason not to deploy more CCR's!

Chris

i found a minor bug in bgp filters. I dont know whether it was display or it was really the config. Router ccr 1036 8g version 6.15
BGP with 3 peers
2 peers to isp - isp1 and isp2
1 peer to client - c1
yesternight I added c1 to peers and added his subnet on filters accept. Strangely the IPs which were being advertised to isp1 and isp2 earlier, one of the subnet got the as path for c1. No as-path was altered in any of the filters. I removed the subnet of c1, the as path of that subnet got ok and other subnet got the as path. Then i disabled the advertising of that subnet and enable after few minutes the as path was empty now. If anyone wants the graphical representation I'll post it later on.

Which are use stable RouterOS version with CCR1072 ?
Thanks