What Mikrotik product is the fastest for BGP with 10-gig load with 2 BGP feeds ?

I thought I would ask:
- 2 BGP sessions with to Internet feeds
- 1 LAN
- All interfaces 10-gig
- Average sustained LAN traffic (to/from Internet) 2-to-4-Gig soon to be 4-to-8-Gig

My business is growing ....

North Idaho Tom Jones

You need a CCR1072 or CCR1036.
Same BGP speed since all load is on one core but 1072 seems better for your requirements.

Did not found how to pm here. Wonna honest answer? Go mx80 for your business.

Your CHR expertise and knowledge would be best put to use for this. It will be far faster than a "real" MikroTik off the shelf product.

Your CHR expertise and knowledge would be best put to use for this. It will be far faster than a "real" MikroTik off the shelf product.

thanks for your info/post

One of the things I am also weighing when considering a ROS based high-throughput system is the following:
- With Mikrotik hardware , I could use a CCR1072 or CCR1036 product ( 72 CPU cores or 36 CPU cores)
- With a CHR running under the free VmWare ESXi (basic free license), I have a limitation of 8 CPUs I can allocate to a virtual machine.
(It is possible to purchace a VmWare ESXi license (for $90k or so) that will allow more CPUs to a virtual machine)
(it is also possible to dump VMware and switch/rebuild over to a different Linux (Ubuntu) hyper-visor system that also allows more cores be allocated to a virtual machine)
- Per core , a Xeon can easily out perform any processor currently in any Mikrotik product.
- BGP at this time , only uses a single core (out of many processor cores available)
- Additional cores on a BGP system can and are still used by other processes such as routes and firewall rules and such

Although I am leaning toward CHR on a VmWare ESXi system with 8 CPU cores , I am just not sure how well it stacks up to a physical Mikrotik product with 36 or 72 cores , when the system is under full load running BGP and all of the other normal firewall/routes configurations at the same time.

At this time, I am running BGP on a VmWare ESXi 8-core system and also running another BGP on a Mikrotik CCR1036. The 36-core CCR1036 always shows one core at normally 100 percent (BGP I assume). The 8-core CHR sometimes hits 100 percent on one core (BGP) but normally averages around 35 percent (constantly changing from 2 percent to 100 percent). In my testing, the CHR appears to be always faster for everything. However , bandwidth demands are growing and I want to keep on top of what works best for the $

North Idaho Tom Jones

VMWare essentials is about £400 (sorry, I know you’re not gbp but I don’t know conversion rate). Very cheap way to get the performance you want.

1 core for bgp in Xeon will be about 3 times faster than a CCR and then your other cores will also be faster.

Better to have faster cores currently until multi core becomes a thing.

I've done some pretty extensive benchmarks with CHR and ESXi seems to be the most well optimized right now. KVM as ProxMox was able to get ~5mpps with a Xeon-d 1518 on Intel sfp+ interfaces. ESXi was able to push closer to 10mpps. CHR on ESXi seemed finicky though. If I remember right IRQ affinity in CHR was kind of screwy and would sometimes take several reboots to work correctly.

Did not found how to pm here. Wonna honest answer? Go mx80 for your business.

second that. if you could talk your upstreams into limiting your bgp-view (ie only local routes and no full feed) you might still have fun with CCRs.

Did not found how to pm here. Wonna honest answer? Go mx80 for your business.

second that. if you could talk your upstreams into limiting your bgp-view (ie only local routes and no full feed) you might still have fun with CCRs.

Would you mind elaborating a bit more in detail, why your suggestion towards an MX80 instead of a CCR? Our core currently runs on an CCR1036 with BGPv4 IPv4/v6 full-tables from two upstreams with no problems. We also did some testing with CHRs on x86 (VMWare ESXi), Intel 10Gbs NICs, three upstreams each with IPv4/v6 full-feeds and around 60 active peerings, which also worked out quite well. Indeed our network is very small compared to others, so no expierence with larger environments.

Some cons of CCRs I could think of are (ordered by priority):
- single-threaded BGP only
- missing BGPv4 SNMP MIBs
- slow BGP convergence time (seems MUCH faster on CHR/x86)
- route table search comparably slow (seems MUCH faster on CHR/x86)
- some stuff painful to configure on ROS (advanced bgp communities, routing filters, etc.)
- different support/hotfix/SLA model compared to other vendors on the market

Thanks

Good question is what kind of data you'll get from upstreams:

- 2 BGP sessions with to Internet feeds

If you're not an ISP, you can get only a few routes and even on low-end ROS devices it'll be ok as for CPU. So you should better care for throughput numbers and I'd go for testing both CCRs that are 10G capable (yes, 1036 of 1072 are the options for you). I think for its price you can find a reseller who'll be happy to provide you with the test units.

But if not ROS, why don't you consider even VyOS https://vyos.io/ in a VM?

Hi,, I think that the best one to do this is a CHR hosted by VMWare or HyperV, please take a look of this video / presentation from MUM at Berlin:

https://www.youtube.com/watch?time_cont ... cgdGA1W_0o

Did not found how to pm here. Wonna honest answer? Go mx80 for your business.

second that. if you could talk your upstreams into limiting your bgp-view (ie only local routes and no full feed) you might still have fun with CCRs.

Would you mind elaborating a bit more in detail, why your suggestion towards an MX80 instead of a CCR? Our core currently runs on an CCR1036 with BGPv4 IPv4/v6 full-tables from two upstreams with no problems. We also did some testing with CHRs on x86 (VMWare ESXi), Intel 10Gbs NICs, three upstreams each with IPv4/v6 full-feeds and around 60 active peerings, which also worked out quite well. Indeed our network is very small compared to others, so no expierence with larger environments.

Some cons of CCRs I could think of are (ordered by priority):
- single-threaded BGP only
- missing BGPv4 SNMP MIBs
- slow BGP convergence time (seems MUCH faster on CHR/x86)
- route table search comparably slow (seems MUCH faster on CHR/x86)
- some stuff painful to configure on ROS (advanced bgp communities, routing filters, etc.)
- different support/hotfix/SLA model compared to other vendors on the market

Thanks

my opinion is simply based on our own testing, experience and then going the JunOS/IOS-Path on several decisions after we tried MT but couldn't work around the caveats.

the biggest issue in our test process were the sometimes poor BGP designs ROS brings with it and we all here in this forum have heard the running gag about ROS7 and how it fill fix things

the issue with importing loopback IPs via OSPF (IGP) specifically for IPv6 is a deal breaker for any route-reflector-setup also between full-mesh-bgp-routers this issue prevails and while you can work around this with static routes it's totally besides the point of setting up a redundant core which then breaks intentionally because of static routes. (you can find lots of discussons here on BGP issues and specifically ipv6)
on strong VM-systems BGP convergence is quite fast, but then again CCR hardware does not converge fast - in fact it is breathtakingly slow to import a full table leave alone multiple full tables;
also in day to day management tasks the winbox would be nice but we get back to a sometimes quite cumbersome CLI for simply looking up a route;
working with VRFs is a head-banger and VPNV6 is simply not there; communities like gshut are not well known; designwise nonstop-forwarding can not be done; HA setups besides vrrp - nope; protecting the controlplane (aka cpu) has to be done in the firewall and disables IPv4 fast path; RP filter is either on for all interfaces or none which is a quite impossible situation for a border-router; management can not be done in a mgmt-vrf but can only be done on the main-table (another design issue); admin user's password data is obviously not one-way-encrypted leading to the latest security-issue (a total show stopper); BFD is simply broken and even MT advises not to use it - a real issue if you have some transport-hardware between you and the transit-carriers edge-router/peer-routers or simply between your own core routers.

most of these issues are much more softened (or simply non existend) with big-iron-hardware+software and for our network transporting also multi-gigabit-peers and transits it was simply not worth the hassle to implement MTs at the core and transit-edge; we are happily using CCRs on several points in our network transporting also heavy traffic-rates, we simply need a complete and robust ipv4+ipv6 feature-set and a well tested and developed BGP feature-set and timely bug-fixes if an issue occurs in our core or at our transit edge; at the CPE end we quite often use MTs as they do a good job there even routing 2 Gbps on the ARM plattform without a hassle including firewall-features (QoS etc);

in the end I know about several ISPs running their core on MTs and CCRs and it works (most of the time), but usually those are not in the 10Gbps range and do not care much about MPLS networking or similar advanced feature-sets (often ipv6 is also a non-issue as it is not used).

regards
hk