Community discussions

 
User avatar
IPANetEngineer
Trainer
Trainer
Topic Author
Posts: 963
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

CCR and Internet BGP - survey of user experiences

Tue May 20, 2014 9:33 pm

This question seems to come up a lot in the MikroTIk forums and I'm curious to see what different types of CCR BGP deployments are out there as well as the experience of the end user. Some users seem to have no issues with multiple public feeds and some aren't stable on only one feed.

So....

1) What kind CCR (s) are you using in your network
2) How many BGP feeds / Routes are you taking in
3) Are you stable or do you experience issues - if so, what kind
4) Software version used

We hear so many good/bad stories on CCR BGP, it might be helpful to have all the info in one thread.
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
wulfgard
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Wed Oct 17, 2012 1:06 pm
Location: France
Contact:

Re: CCR and Internet BGP - survey of user experiences

Thu May 22, 2014 3:19 pm

Good afternoon

very strange request from you.
don't you have any CCR with BGP in production ?
aren't you "The Largest Carrier-Grade MikroTik Integrator in North America" ?

best regards
Thierry
System and Network Engineer
Mikrotik Trainer - MTCNA MTCRE
Official French Mikrotik Distributor
 
paulsa
newbie
Posts: 34
Joined: Tue Oct 29, 2013 10:24 am

Re: CCR and Internet BGP - survey of user experiences

Thu May 22, 2014 4:55 pm

This question seems to come up a lot in the MikroTIk forums and I'm curious to see what different types of CCR BGP deployments are out there as well as the experience of the end user. Some users seem to have no issues with multiple public feeds and some aren't stable on only one feed.

So....

1) What kind CCR (s) are you using in your network
2) How many BGP feeds / Routes are you taking in
3) Are you stable or do you experience issues - if so, what kind
4) Software version used

We hear so many good/bad stories on CCR BGP, it might be helpful to have all the info in one thread.
Great post.

We will be switching back from our cisco 7200 to the CCR on Sat morning for BGP.

1) CCR1036
2) 5 Feeds in total (local, international, peeing, a private peer, and Hurricane)
3) Was stable for 5 months on - then instability with random reboots (Tried a different CCR, new config, old config, ROS V6.10, Swapped SFP's, Ran BGP only on one interface)
4) V6.7 and V6.10

We are dedicating a CCR for ONLY BGP now on two SFP's with three vlans for int/local/peering. No queues, no advanced firewall handling etc and watchdog turned off as we suspect its a CPU issue and watchdog rebooting it. Another CCR will be used for the queues/firewall and IGP. We will try V6.13 on our BGP router

Will provide feedback on Monday if it is stable.
 
User avatar
IPANetEngineer
Trainer
Trainer
Topic Author
Posts: 963
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: CCR and Internet BGP - survey of user experiences

Thu May 22, 2014 5:49 pm

Good afternoon

very strange request from you.
don't you have any CCR with BGP in production ?
aren't you "The Largest Carrier-Grade MikroTik Integrator in North America" ?

best regards
Thierry
:) Touche!...you are correct, we ARE the largest Carrier-Grade Integrator in North America and deploy BGP/MPLS solutions to many large wireline/wisp carriers. We work with a lot of CCRs in BGP peering and have seen various issues. We tend to limit full table peerings on CCRs to no more than 2 based on the limitation of a multi-threaded BGP process. Since we are an integrator, most of our customers use Cisco and Juniper still for public BGP peering and are reluctant to migrate. Large carriers don't like being beta testers in general - stability and conservative design is the name of the game.

The intent of the thread is to try and centralize information and experiences on peering with full global BGP tables on CCR and see if there are common issues across software versions and/or types of peering routers.
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
yogensha
just joined
Posts: 7
Joined: Thu May 22, 2014 8:10 pm

Re: CCR and Internet BGP - survey of user experiences

Thu May 22, 2014 8:27 pm

1) What kind CCR (s) are you using in your network
CCR1036-12G-4S. Would really like to see a 12-port SFP version.
2) How many BGP feeds / Routes are you taking in
One transit provider on the CCR. Other transit providers are on other platforms, but the plan is to move to CCRs, one per provider. Taking full tables.
If things continue to go well, we'll also move edge aggregation to CCR. Core switching will be on other platforms.
3) Are you stable or do you experience issues - if so, what kind
Stable since deployment about 2 weeks ago. Most issues are just learning the platform, there are some oddities. Diagnostics are probably the biggest problem: 'show ip route' would be nice.
4) Software version used
6.12
 
joegoldman
Member
Member
Posts: 386
Joined: Mon May 27, 2013 2:05 am

Re: CCR and Internet BGP - survey of user experiences

Fri May 23, 2014 9:55 am

1) What kind CCR (s) are you using in your network
CCR1036-12G-4S. Would really like to see a 12-port SFP version.
There is the CCR1016-12S-1S+, only half the cores but unless you are doing lots of data it should still allow it.

Or there is a CCR1072-1G-8S+ on the horizon I believe (Q3 2014), which is 72 cores and 8 (or more) SFP+ ports
2) How many BGP feeds / Routes are you taking in
One transit provider on the CCR. Other transit providers are on other platforms, but the plan is to move to CCRs, one per provider. Taking full tables.
If things continue to go well, we'll also move edge aggregation to CCR. Core switching will be on other platforms.
Why only 1 CCR per provider? (curious)
3) Are you stable or do you experience issues - if so, what kind
Stable since deployment about 2 weeks ago. Most issues are just learning the platform, there are some oddities. Diagnostics are probably the biggest problem: 'show ip route' would be nice.
There is this, but large tables can take a bit to get the answer, and it will show ever matching route, not just the preferred/taken route:

ros code

[admin@core1.bne1] > /ip route print where 8.8.8.8 in dst-address
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          <hidden>               1
 1 ADb  8.8.8.0/24                         <hidden>            200
 
NetNotGross
just joined
Posts: 12
Joined: Wed May 21, 2014 7:15 pm

Re: CCR and Internet BGP - survey of user experiences

Fri May 23, 2014 3:44 pm

We had problems around the point of 4 full feeds but I don't thing we ran the problem to ground. We were hitting various other CCR limitations at the time so it wasn't the most important item for resolution.

At this point I don't think it is possible to fully "integrate" Mikrotik equipment as part of a "carrier grade" solution. You can certainly interface Mikrotik equipment with carrier grade systems but I certainly would not describe any Mikrotik hardware as "carrier grade". That doesn't mean it isn't useful, good value or the appropriate answer for some applications.
Last edited by NetNotGross on Sun May 25, 2014 4:41 pm, edited 1 time in total.
 
yogensha
just joined
Posts: 7
Joined: Thu May 22, 2014 8:10 pm

Re: CCR and Internet BGP - survey of user experiences

Fri May 23, 2014 4:57 pm

There is the CCR1016-12S-1S+, only half the cores but unless you are doing lots of data it should still allow it.
For some reason I thought that was all 10 gig. Thanks for the tip!
Why only 1 CCR per provider? (curious)
Redundancy. One failure takes out only one provider. Low cost makes this very practical.
There is this, but large tables can take a bit to get the answer, and it will show ever matching route, not just the preferred/taken route:

ros code

[admin@core1.bne1] > /ip route print where 8.8.8.8 in dst-address
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          <hidden>               1
 1 ADb  8.8.8.0/24                         <hidden>            200
I really like that it shows the less-specifics. It gives functionality similar to 'show ip [route|bgp] <address> longer-prefixes' in cisco-land. It takes a good 60-90 seconds to do the lookup on a router with a full table though, making troubleshooting time consuming, which is incredibly frustrating in an outage situation.
 
User avatar
wulfgard
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Wed Oct 17, 2012 1:06 pm
Location: France
Contact:

Re: CCR and Internet BGP - survey of user experiences

Sat May 24, 2014 1:34 am

Good afternoon

very strange request from you.
don't you have any CCR with BGP in production ?
aren't you "The Largest Carrier-Grade MikroTik Integrator in North America" ?

best regards
Thierry
:) Touche!...you are correct, we ARE the largest Carrier-Grade Integrator in North America and deploy BGP/MPLS solutions to many large wireline/wisp carriers. We work with a lot of CCRs in BGP peering and have seen various issues. We tend to limit full table peerings on CCRs to no more than 2 based on the limitation of a multi-threaded BGP process. Since we are an integrator, most of our customers use Cisco and Juniper still for public BGP peering and are reluctant to migrate. Large carriers don't like being beta testers in general - stability and conservative design is the name of the game.

The intent of the thread is to try and centralize information and experiences on peering with full global BGP tables on CCR and see if there are common issues across software versions and/or types of peering routers.
So with your experience you may know that

1) bgp needs a lot of cpu
2) bgp needs some memory
3) V6.x is not multi thread for BGP

but to help you :)

CCR1036-4G can handle with no problem 2 full table IPV4 + IPV6
if you need a lot of peers use x86 until v7 and 72 tilera cores

for compatibility as you seem not to be able to do testings it works fine with cisco, juniper, quagga, bird, huawey

our network is spread over europe with 9 bgp routers ( x86 ) with about 500 bgp sessions including peering and upstreams
it takes about 1 minute to propagate a full IPV4 table over all the network
you will not be able to do this even with SUP720 3BXL or MX80 ( we have tested in real production )

for public peering if you are connected on a peering point like AMSIX, DECIX, TOPIX, LUCIX ... you will see a lot of Mikrotik routers just with "ip neighbour" but they are not speaking about it

our customers are using CCR for connecting their upstreams and they have no issue but the secret is also about setup :)

regards
Thierry
System and Network Engineer
Mikrotik Trainer - MTCNA MTCRE
Official French Mikrotik Distributor
 
Luth
newbie
Posts: 42
Joined: Sun Jun 05, 2011 4:04 pm
Location: Italy
Contact:

Re: CCR and Internet BGP - survey of user experiences

Sun May 25, 2014 11:06 am

1) We have 2 CCR1016-12G
2) We have BGP with 2 upstream with 2 session per upstream so, in total, 4 BGP session per router
3) We are stable, no issue at the moment with BGP
4) v6.12

TLCWEB Srl/Baleno Wireless the smaller WISP in south of Italy :-)
Luciano Talarico
TLCWEB Srl - Italy
Visit Mikrotik topics on our Linux by Myself website at linuxbms.com
 
paulsa
newbie
Posts: 34
Joined: Tue Oct 29, 2013 10:24 am

Re: CCR and Internet BGP - survey of user experiences

Mon May 26, 2014 3:33 pm

We will be switching back from our cisco 7200 to the CCR on Sat morning for BGP.

1) CCR1036
2) 5 Feeds in total (local, international, peeing, a private peer, and Hurricane)
3) Was stable for 5 months on - then instability with random reboots (Tried a different CCR, new config, old config, ROS V6.10, Swapped SFP's, Ran BGP only on one interface)
4) V6.7 and V6.10

We are dedicating a CCR for ONLY BGP now on two SFP's with three vlans for int/local/peering. No queues, no advanced firewall handling etc and watchdog turned off as we suspect its a CPU issue and watchdog rebooting it. Another CCR will be used for the queues/firewall and IGP. We will try V6.13 on our BGP router

Will provide feedback on Monday if it is stable.
So far, very stable now for 54 hours on V6.13
 
User avatar
sguox
Trainer
Trainer
Posts: 72
Joined: Fri Mar 09, 2012 6:23 pm

Re: CCR and Internet BGP - survey of user experiences

Thu May 29, 2014 6:53 am

we have been using 2x CCR1036-12G-4S since early 6.x without much issue. the catch, daily reboot on the router in off-peak hours.

first unit connected with 3 BGP sessions (2 full route, 1 local route ~80k), total throughput >300mbps

second unit, 2 BGP sessions (1 full route, 1 local route ~10k), total throughput >300mbps

so far so good
 
NodeMax
newbie
Posts: 38
Joined: Sun Sep 22, 2013 11:39 am

Re: CCR and Internet BGP - survey of user experiences

Thu May 29, 2014 6:26 pm

2 CCR both – CCR1036-12G-4S
2 feeds each with 2 sessions each feed, dualstack

1 carrier on fibre other on ethernet on each router

2 ipv4 on each with 244k routes and 488K the other
2 ipv6 on each with 15K and 12K on the other

software 6.12

No problems ... :D
 
joegoldman
Member
Member
Posts: 386
Joined: Mon May 27, 2013 2:05 am

Re: CCR and Internet BGP - survey of user experiences

Sun Jun 22, 2014 5:34 am

Coming in to say CCR1036-8G-2S+ has been in production for nearly a month. Only taking default route still, but has 30K~ in iBGP/OSPF/IX routes without problem. Probably only about 12k are active routes though (lots of double ups over the IX's)

Peaking at about 600mbit / 100kpps aggregate bandwidth through a 802.3ad LACP trunk.

Have some basic ICMP stuff in the forward/input chain

Have some basic blocking in the input chain for Telnet/SSH

Have about 10 simple queues, with 2 mangle rules per queue (to match input/output traffic) on a VLAN interface that does 100-140mbit traffic at any point.

Mostly, I barely see the CPU above 10% at peak, so very stable and happy with performance so far.

Running ROS6.13

Spinning 2nd one up this week for stateful balance and failover. Will also start receiving full table's from 2 providers soon.
 
selric
just joined
Posts: 6
Joined: Tue Jul 23, 2013 3:05 pm

Re: CCR and Internet BGP - survey of user experiences

Sun Jul 20, 2014 10:17 pm

Ours have been running for 27 days with ROS 6.15 today and one full feed and a handfull of peers at an IX.

Works great so far!
 
paulsa
newbie
Posts: 34
Joined: Tue Oct 29, 2013 10:24 am

Re: CCR and Internet BGP - survey of user experiences

Tue Jul 22, 2014 7:18 pm

Since we split our routing, i.e ospf/firewall rules/queues etc on one router and BGP on another CCR it has been running great.

system resource print
uptime: 8w3d9h14m22s
version: 6.13
 
rcwmoab
just joined
Posts: 16
Joined: Tue Jun 22, 2010 4:03 am
Location: Utah

Re: CCR and Internet BGP - survey of user experiences

Sun Jul 27, 2014 12:35 am

Been running our CCR1036-12g-4s since they were available as a preproduction product with full bgp coming from two upstream providers totaling 730k routes passing 400mbps of traffic with no problems what so ever. This router only runs BGP on it though, no ospf or anything else. Currently running v6.9
 
randvegeta
just joined
Posts: 8
Joined: Tue Jul 22, 2014 6:24 am

Re: CCR and Internet BGP - survey of user experiences

Mon Jul 28, 2014 10:15 am

Been running our CCR1036-12g-4s since they were available
How long ago was that?
full bgp coming from two upstream providers totaling 730k routes passing 400mbps of traffic with no problems what so ever
.

Good to hear. May I ask if you have done any reboots? Or table rebuilds? Interested in knowing how long it took to build the routing table.
 
lz1dsb
Member Candidate
Member Candidate
Posts: 222
Joined: Wed Aug 07, 2013 11:48 am

Re: CCR and Internet BGP - survey of user experiences

Thu Aug 21, 2014 5:17 pm

1) What kind CCR (s) are you using in your network
two CCR-1036-12G

2) How many BGP feeds / Routes are you taking in
above 500k. One of the routers has 4 BGP peering sessions, the other has 5. At the moment we're peering with 2 ISPs, in the next month another two ISP peering sessions are planned.

3) Are you stable or do you experience issues - if so, what kind
twice, the issue was that all of the configured bridge interfaces were somehow wiped out from the running config. But I do not attribute this to BGP as it happen once long before BGP was deployed. In general - I haven't had any issues with it.

4) Software version used
6.17
 
User avatar
IPANetEngineer
Trainer
Trainer
Topic Author
Posts: 963
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: CCR and Internet BGP - survey of user experiences

Tue Aug 26, 2014 5:11 pm

Glad to see this thread has been useful...i've seen it linked in multiple other forum posts.

@Mikrotik - what is the status on balancing BGP across multiple cores?
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
lz1dsb
Member Candidate
Member Candidate
Posts: 222
Joined: Wed Aug 07, 2013 11:48 am

Re: CCR and Internet BGP - survey of user experiences

Tue Aug 26, 2014 5:26 pm

Glad to see this thread has been useful...i've seen it linked in multiple other forum posts.

@Mikrotik - what is the status on balancing BGP across multiple cores?
That's something I would like to see too! This is something that bothers me, because it turns out that as far as routing is concerned - there's such a waste in capacity! So many cores stay idle while one of them is utilized to its limits.
I think I read somewhere that core load balancing is planned for version 7.0 but I'm not sure...
 
joegoldman
Member
Member
Posts: 386
Joined: Mon May 27, 2013 2:05 am

Re: CCR and Internet BGP - survey of user experiences

Wed Aug 27, 2014 2:37 am

@Mikrotik - what is the status on balancing BGP across multiple cores?
As above poster said, I think this is being held off for v7.

There is a thread on here somewhere in relation to v7 where normis says they're alpha test builds are loading 15 million routes in 15 minutes, a million routes a minute is pretty darn good. You'd have to imagine this efficiency is being achieved with multi-core.

Hopefully filtering/adding/removing routes from a big route-table will be vastly improved too. Still running 6.13 on my main BGP routers but hell they are slow to add routes and filter looking for things.
 
User avatar
bigcw
Frequent Visitor
Frequent Visitor
Posts: 95
Joined: Mon Sep 08, 2014 2:38 pm

Re: CCR and Internet BGP - survey of user experiences

Mon Sep 08, 2014 3:13 pm

My current setup:

3 x CCR1036-12G-4S running v6.18:

- London: 2 x full BGP transit, iBGP to Frankfurt, Madrid and London
- Frankfurt: 1x full BGP transit, iBGP to London
- Madrid: 1x full BGP transit, iBGP to London

1 x 1100AHx2 v5.26 (yes, it's old, but it's stable!):

- Dubin: Approx 20 peering sessions (INEX exchange), iBGP to London

The setup has been stable other than an issue where BGP 'disappears' in Frankfurt. At present I suspect a hardware issue - there is a thread about this issue that I have just written and is awaiting approval. (ETA: http://forum.mikrotik.com/viewtopic.php?f=14&t=88912)

My only criticism of BGP on RouterOS is the speed. It takes a good 5 minutes at present to do a ~500k route converge on a 1036. Fingers crossed that v7 will speed this up, then I will have no reason not to deploy more CCR's!

Chris
Ecom International Network - Operators of AS61337 with POPs in Europe and North America - www.ecomltd.co.uk
Colocker Data Centre - The data centre with a difference! - www.colocker.com
 
hashbang
Member Candidate
Member Candidate
Posts: 148
Joined: Sat Jul 26, 2014 6:38 pm

Re: CCR and Internet BGP - survey of user experiences

Sat Sep 13, 2014 11:11 am

i found a minor bug in bgp filters. I dont know whether it was display or it was really the config. Router ccr 1036 8g version 6.15
BGP with 3 peers
2 peers to isp - isp1 and isp2
1 peer to client - c1
yesternight I added c1 to peers and added his subnet on filters accept. Strangely the IPs which were being advertised to isp1 and isp2 earlier, one of the subnet got the as path for c1. No as-path was altered in any of the filters. I removed the subnet of c1, the as path of that subnet got ok and other subnet got the as path. Then i disabled the advertising of that subnet and enable after few minutes the as path was empty now. If anyone wants the graphical representation I'll post it later on.
 
rootemin
just joined
Posts: 4
Joined: Thu Oct 01, 2015 6:12 pm

Re: CCR and Internet BGP - survey of user experiences

Wed Feb 24, 2016 12:30 pm

Which are use stable RouterOS version with CCR1072 ?
Thanks

Who is online

Users browsing this forum: No registered users and 2 guests