Community discussions

 
Wolfraider
newbie
Topic Author
Posts: 49
Joined: Wed Jul 15, 2015 8:06 pm

CHR Hardware

Mon Mar 18, 2019 10:38 pm

We are working on new replacement routers for our core network. We will have 2 independent 10Gb peers and full tables from each. What would be a good low cost server to run CHR on? I will admit that I have not been in the server field for several years and I am almost lost on all the new options. We are going to get 2 servers running ESXi and then run a CHR on each server/dedicate all resources to the CHR. We would like to keep costs down to $1000/each. Current load is 8.4Gbps peak each night with expected growth of 12Gbps by summers end spread across both routers.

I came up with the following but I am unsure if they would handle the load.
Dual Intel DP L5640 2.26GHz processors
32Gb DDR3 1333 Ram
2 - 500Gb SSD mirrored
Intel dual port 10Gb nic
$912
 
User avatar
Steveocee
Forum Guru
Forum Guru
Posts: 1092
Joined: Tue Jul 21, 2015 10:09 pm
Location: UK
Contact:

Re: CHR Hardware

Tue Mar 19, 2019 12:43 pm

Will you be licensing your ESXi installations? If not you can only use 8 vCPU's per machine so you'd have a lot of redundant cores. Saying that it's better to run WITHOUT HT for CHR so only 4 over.
Steve "Steveocee" Carter
PC Gamer, Airsofter, MikroTik Nerd
My Website - My MikroTik Tutorials
 
User avatar
TomjNorthIdaho
Forum Veteran
Forum Veteran
Posts: 981
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: CHR Hardware

Tue Mar 19, 2019 6:51 pm

For two 10-Gig BGP CHR routers , this is what I do (and it works good).

- One physical VMware ESXi box (Free version of VMware ESXi)
- Two Xeon CPUs - with a minimum of 10 cores per Xeon CPU.
- Lots of Xeon CPU cache helps
- Disable Hyper-threading
-- On 1st CHR , configure eight CPUs (CPUs 2-through-10) ((( dedicated configuration to make this CHR run only on the 1st Xeon CPU )))
-- On 2nd CHR , configure eight CPUs (CPUs 12-through-20) ((( dedicated configuration to make this CHR run only on the 2nd Xeon CPU )))
((( Note ; Each CHR will run on a different physical Xeon processor -and- each CHR will have access to the entire built-in Xeon CPU cache )))
((( Note ; The 2nd CHR which runs on the 2nd Xeon CPU might have slightly faster throughput )))
((( Note ; On your CHRs , use VMXNET-3 network interfaces only )))

You should have a second cold-spare physical VMware ESXi server ready to use if you have a hardware problem with the first physical box.
Your 2nd physical box could also be running a 3rd CHR to handle any OSPF traffic and/or routing to your customers.
Your 2nd physical box could also be running a 4th CHR to handle any customer bandwidth limiter configurations you may have to limit customer up/down bandwidths.

I always suggest a stand-by physical Hyper-Visor server (VMware ESXi) with some CHRs configured , licensed and ready to go) to minimize any possible down-time in the event you have some serious hardware failures. The spare stand-by physical box and spare stand-by CHRs when not needed will make a great LAB system for testing stuff.

CHR P-Unlimited licenses are cost effective (about $250 per license). The only real expense is the physical servers ( In my case SuperMicro servers ) and some 10-gig switches.

This is how I do my core networks (including BGP systems).

North Idaho Tom Jones
 
Wolfraider
newbie
Topic Author
Posts: 49
Joined: Wed Jul 15, 2015 8:06 pm

Re: CHR Hardware

Wed Mar 20, 2019 7:55 pm

We will be assigning a couple of our ESX licenses for this. We like the idea of running 2 physical servers instead of 1 just in case of hardware failure. We are currently running a couple CCR1072's but we have not brought up our second 10Gb trunk up yet or received full routes from either carrier.
 
User avatar
TomjNorthIdaho
Forum Veteran
Forum Veteran
Posts: 981
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: CHR Hardware

Wed Mar 20, 2019 8:26 pm

We will be assigning a couple of our ESX licenses for this. We like the idea of running 2 physical servers instead of 1 just in case of hardware failure. We are currently running a couple CCR1072's but we have not brought up our second 10Gb trunk up yet or received full routes from either carrier.
With good physical servers hosting your CHRs , I think you will see much better ROS functionality with CHR verses a physical Mikrotik for your BGP, routing, firewall and bandwidth management functions.

When you bring up your CHR , please post your full BGP table load times of the CHR and the physical Mikrotik. I would like to know how much faster the CHR is.

North Idaho Tom Jones
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1020
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: CHR Hardware

Thu Mar 21, 2019 3:27 pm

Hyper-V is hands down the best hypervisor for using a CHR as a BGP edge router. Mostly this is because MikroTIk spent a lot of time building the Hyper-V drivers for the CHR and they used off the shelf drivers for KVM/ESXi

The single biggest impact is to get a CPU with a higher clock speed and fewer cores vs. lots of cores at a lower clock speed.

Here is a presentation I did last year in Berlin on the topic and since then I've had a number of customers deploy on Hyper-V with a great deal of success and extremely rapid convergence times.

https://mum.mikrotik.com/presentations/ ... 562405.pdf
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
Wolfraider
newbie
Topic Author
Posts: 49
Joined: Wed Jul 15, 2015 8:06 pm

Re: CHR Hardware

Mon Mar 25, 2019 9:08 pm

We are not running eBGP as of yet but I will post numbers as soon as we get this implemented.

Your getting better performance with Hyper-V? I have heard others talk about better performance with ESXi. We could go either way. I'll post back numbers.
 
cmurrayis
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Fri May 15, 2009 4:31 am

Re: CHR Hardware

Fri Jun 07, 2019 7:00 am

Hyper-V is hands down the best hypervisor for using a CHR as a BGP edge router. Mostly this is because MikroTIk spent a lot of time building the Hyper-V drivers for the CHR and they used off the shelf drivers for KVM/ESXi

The single biggest impact is to get a CPU with a higher clock speed and fewer cores vs. lots of cores at a lower clock speed.

Here is a presentation I did last year in Berlin on the topic and since then I've had a number of customers deploy on Hyper-V with a great deal of success and extremely rapid convergence times.

https://mum.mikrotik.com/presentations/ ... 562405.pdf
@IPANetEngineer

We're wanting to do some testing with CHR running as a VM however we're needing more than 10gbps of throughput which generally isn't an issue however we're hitting 80% on our CCR1072's with around 800,000PPS

We're looking to use the Dell vep4600 as the hardware platform with the additional 8 x 10gbps modules. Given these are the Network Accelerated Xeon models we hoped for amazing results.

Do you have any recommendations?
 
User avatar
TomjNorthIdaho
Forum Veteran
Forum Veteran
Posts: 981
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: CHR Hardware

Fri Jun 07, 2019 6:27 pm

Hyper-V is hands down the best hypervisor for using a CHR as a BGP edge router. Mostly this is because MikroTIk spent a lot of time building the Hyper-V drivers for the CHR and they used off the shelf drivers for KVM/ESXi

The single biggest impact is to get a CPU with a higher clock speed and fewer cores vs. lots of cores at a lower clock speed.

Here is a presentation I did last year in Berlin on the topic and since then I've had a number of customers deploy on Hyper-V with a great deal of success and extremely rapid convergence times.

https://mum.mikrotik.com/presentations/ ... 562405.pdf
@IPANetEngineer

We're wanting to do some testing with CHR running as a VM however we're needing more than 10gbps of throughput which generally isn't an issue however we're hitting 80% on our CCR1072's with around 800,000PPS

We're looking to use the Dell vep4600 as the hardware platform with the additional 8 x 10gbps modules. Given these are the Network Accelerated Xeon models we hoped for amazing results.

Do you have any recommendations?
Here is what I would suggest to consider:
CPU (qty 2):
Intel Xeon E5 Family
Intel CM8066002024000 Xeon E5-2698 v4 - 2.2 GHz - 20-Core - 40 Threads - 50 MB Cache - LGA2011 Socket - OEM
# of Cores: 20-Core
Series: Intel Xeon E5 Family
L2 Cache: 20 x 256KB
L3 Cache: 50MB

With 50 MB of cache , you have a high probability of having more CPU cache hits. CPU Cache hits will help keep you CHR running at CPU clock speed instead of ram speed.
Also - do NOT use hyper-threading


North Idaho Tom Jones
 
angriukas
just joined
Posts: 10
Joined: Fri Nov 22, 2013 9:20 am
Location: Lithuania
Contact:

Re: CHR Hardware

Wed Jul 03, 2019 3:11 pm

Try to use Proxmox Virtualization Platform (PVE).
Just successfully tested CHR on PVE with KVM hypervisor.
CHR VM (virtual machine) supports following intefaces:
disk - SATA or Virtio (to boot CHR).
network - all types, but only Virtio and vmxnet (VMWare) supports 10G.
Better to use everywhere Virtio type of interfaces, because of better integration with PVE.
HA (High Availability) cluster can be built with min. 3 physical servers.
HA means - same file system on all of physical servers (same VM image - on all nodes). If Node1 will fail - VM automatically will be started on next node.
In case of HA, for best performance use host CPU or use lower CPU type of your nodes. I think default CPU type kvm64 will be also good enough.
We use PVE for years in our office, not for CHR, but for other tasks. If you like I can consult regarding PVE.
It's perfect opensource platform and is free of charge (with no subscription).

Who is online

Users browsing this forum: No registered users and 1 guest