This question seems to come up a lot in the MikroTIk forums and I’m curious to see what different types of CCR BGP deployments are out there as well as the experience of the end user. Some users seem to have no issues with multiple public feeds and some aren’t stable on only one feed.
So…
What kind CCR (s) are you using in your network
How many BGP feeds / Routes are you taking in
Are you stable or do you experience issues - if so, what kind
Software version used
We hear so many good/bad stories on CCR BGP, it might be helpful to have all the info in one thread.
very strange request from you.
don’t you have any CCR with BGP in production ?
aren’t you “The Largest Carrier-Grade MikroTik Integrator in North America” ?
We will be switching back from our cisco 7200 to the CCR on Sat morning for BGP.
CCR1036
5 Feeds in total (local, international, peeing, a private peer, and Hurricane)
Was stable for 5 months on - then instability with random reboots (Tried a different CCR, new config, old config, ROS V6.10, Swapped SFP’s, Ran BGP only on one interface)
V6.7 and V6.10
We are dedicating a CCR for ONLY BGP now on two SFP’s with three vlans for int/local/peering. No queues, no advanced firewall handling etc and watchdog turned off as we suspect its a CPU issue and watchdog rebooting it. Another CCR will be used for the queues/firewall and IGP. We will try V6.13 on our BGP router
Touche!..you are correct, we ARE the largest Carrier-Grade Integrator in North America and deploy BGP/MPLS solutions to many large wireline/wisp carriers. We work with a lot of CCRs in BGP peering and have seen various issues. We tend to limit full table peerings on CCRs to no more than 2 based on the limitation of a multi-threaded BGP process. Since we are an integrator, most of our customers use Cisco and Juniper still for public BGP peering and are reluctant to migrate. Large carriers don’t like being beta testers in general - stability and conservative design is the name of the game.
The intent of the thread is to try and centralize information and experiences on peering with full global BGP tables on CCR and see if there are common issues across software versions and/or types of peering routers.
CCR1036-12G-4S. Would really like to see a 12-port SFP version.
How many BGP feeds / Routes are you taking in
One transit provider on the CCR. Other transit providers are on other platforms, but the plan is to move to CCRs, one per provider. Taking full tables.
If things continue to go well, we’ll also move edge aggregation to CCR. Core switching will be on other platforms.
Are you stable or do you experience issues - if so, what kind
Stable since deployment about 2 weeks ago. Most issues are just learning the platform, there are some oddities. Diagnostics are probably the biggest problem: ‘show ip route’ would be nice.
There is the CCR1016-12S-1S+, only half the cores but unless you are doing lots of data it should still allow it.
Or there is a CCR1072-1G-8S+ on the horizon I believe (Q3 2014), which is 72 cores and 8 (or more) SFP+ ports
Why only 1 CCR per provider? (curious)
There is this, but large tables can take a bit to get the answer, and it will show ever matching route, not just the preferred/taken route:
[admin@core1.bne1] > /ip route print where 8.8.8.8 in dst-address
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
We had problems around the point of 4 full feeds but I don’t thing we ran the problem to ground. We were hitting various other CCR limitations at the time so it wasn’t the most important item for resolution.
At this point I don’t think it is possible to fully “integrate” Mikrotik equipment as part of a “carrier grade” solution. You can certainly interface Mikrotik equipment with carrier grade systems but I certainly would not describe any Mikrotik hardware as “carrier grade”. That doesn’t mean it isn’t useful, good value or the appropriate answer for some applications.
For some reason I thought that was all 10 gig. Thanks for the tip!
Why only 1 CCR per provider? (curious)
Redundancy. One failure takes out only one provider. Low cost makes this very practical.
There is this, but large tables can take a bit to get the answer, and it will show ever matching route, not just the preferred/taken route:
[> admin@core1.bne1> ] > /ip route print where 8.8.8.8 in dst-address
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 A S 0.0.0.0/0 1
1 ADb 8.8.8.0/24 200
I really like that it shows the less-specifics. It gives functionality similar to ‘show ip [route|bgp] longer-prefixes’ in cisco-land. It takes a good 60-90 seconds to do the lookup on a router with a full table though, making troubleshooting time consuming, which is incredibly frustrating in an outage situation.
CCR1036-4G can handle with no problem 2 full table IPV4 + IPV6
if you need a lot of peers use x86 until v7 and 72 tilera cores
for compatibility as you seem not to be able to do testings it works fine with cisco, juniper, quagga, bird, huawey
our network is spread over europe with 9 bgp routers ( x86 ) with about 500 bgp sessions including peering and upstreams
it takes about 1 minute to propagate a full IPV4 table over all the network
you will not be able to do this even with SUP720 3BXL or MX80 ( we have tested in real production )
for public peering if you are connected on a peering point like AMSIX, DECIX, TOPIX, LUCIX … you will see a lot of Mikrotik routers just with “ip neighbour” but they are not speaking about it
our customers are using CCR for connecting their upstreams and they have no issue but the secret is also about setup
Coming in to say CCR1036-8G-2S+ has been in production for nearly a month. Only taking default route still, but has 30K~ in iBGP/OSPF/IX routes without problem. Probably only about 12k are active routes though (lots of double ups over the IX’s)
Peaking at about 600mbit / 100kpps aggregate bandwidth through a 802.3ad LACP trunk.
Have some basic ICMP stuff in the forward/input chain
Have some basic blocking in the input chain for Telnet/SSH
Have about 10 simple queues, with 2 mangle rules per queue (to match input/output traffic) on a VLAN interface that does 100-140mbit traffic at any point.
Mostly, I barely see the CPU above 10% at peak, so very stable and happy with performance so far.
Running ROS6.13
Spinning 2nd one up this week for stateful balance and failover. Will also start receiving full table’s from 2 providers soon.
Been running our CCR1036-12g-4s since they were available as a preproduction product with full bgp coming from two upstream providers totaling 730k routes passing 400mbps of traffic with no problems what so ever. This router only runs BGP on it though, no ospf or anything else. Currently running v6.9
What kind CCR (s) are you using in your network
two CCR-1036-12G
How many BGP feeds / Routes are you taking in
above 500k. One of the routers has 4 BGP peering sessions, the other has 5. At the moment we’re peering with 2 ISPs, in the next month another two ISP peering sessions are planned.
Are you stable or do you experience issues - if so, what kind
twice, the issue was that all of the configured bridge interfaces were somehow wiped out from the running config. But I do not attribute this to BGP as it happen once long before BGP was deployed. In general - I haven’t had any issues with it.
Touche!..you are correct, we ARE the largest Carrier-Grade Integrator in North America and deploy BGP/MPLS solutions to many large wireline/wisp carriers. We work with a lot of CCRs in BGP peering and have seen various issues. We tend to limit full table peerings on CCRs to no more than 2 based on the limitation of a multi-threaded BGP process. Since we are an integrator, most of our customers use Cisco and Juniper still for public BGP peering and are reluctant to migrate. Large carriers don’t like being beta testers in general - stability and conservative design is the name of the game.