Community discussions

MikroTik App
 
User avatar
Wolfraider
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 56
Joined: Wed Jul 15, 2015 8:06 pm

CHR Hardware

Mon Mar 18, 2019 10:38 pm

We are working on new replacement routers for our core network. We will have 2 independent 10Gb peers and full tables from each. What would be a good low cost server to run CHR on? I will admit that I have not been in the server field for several years and I am almost lost on all the new options. We are going to get 2 servers running ESXi and then run a CHR on each server/dedicate all resources to the CHR. We would like to keep costs down to $1000/each. Current load is 8.4Gbps peak each night with expected growth of 12Gbps by summers end spread across both routers.

I came up with the following but I am unsure if they would handle the load.
Dual Intel DP L5640 2.26GHz processors
32Gb DDR3 1333 Ram
2 - 500Gb SSD mirrored
Intel dual port 10Gb nic
$912
 
User avatar
Steveocee
Forum Guru
Forum Guru
Posts: 1130
Joined: Tue Jul 21, 2015 10:09 pm
Location: UK
Contact:

Re: CHR Hardware

Tue Mar 19, 2019 12:43 pm

Will you be licensing your ESXi installations? If not you can only use 8 vCPU's per machine so you'd have a lot of redundant cores. Saying that it's better to run WITHOUT HT for CHR so only 4 over.
Steve "Steveocee" Carter
PC Gamer, Airsofter, MikroTik Nerd
My Website - My MikroTik Tutorials
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1164
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: CHR Hardware

Tue Mar 19, 2019 6:51 pm

For two 10-Gig BGP CHR routers , this is what I do (and it works good).

- One physical VMware ESXi box (Free version of VMware ESXi)
- Two Xeon CPUs - with a minimum of 10 cores per Xeon CPU.
- Lots of Xeon CPU cache helps
- Disable Hyper-threading
-- On 1st CHR , configure eight CPUs (CPUs 2-through-10) ((( dedicated configuration to make this CHR run only on the 1st Xeon CPU )))
-- On 2nd CHR , configure eight CPUs (CPUs 12-through-20) ((( dedicated configuration to make this CHR run only on the 2nd Xeon CPU )))
((( Note ; Each CHR will run on a different physical Xeon processor -and- each CHR will have access to the entire built-in Xeon CPU cache )))
((( Note ; The 2nd CHR which runs on the 2nd Xeon CPU might have slightly faster throughput )))
((( Note ; On your CHRs , use VMXNET-3 network interfaces only )))

You should have a second cold-spare physical VMware ESXi server ready to use if you have a hardware problem with the first physical box.
Your 2nd physical box could also be running a 3rd CHR to handle any OSPF traffic and/or routing to your customers.
Your 2nd physical box could also be running a 4th CHR to handle any customer bandwidth limiter configurations you may have to limit customer up/down bandwidths.

I always suggest a stand-by physical Hyper-Visor server (VMware ESXi) with some CHRs configured , licensed and ready to go) to minimize any possible down-time in the event you have some serious hardware failures. The spare stand-by physical box and spare stand-by CHRs when not needed will make a great LAB system for testing stuff.

CHR P-Unlimited licenses are cost effective (about $250 per license). The only real expense is the physical servers ( In my case SuperMicro servers ) and some 10-gig switches.

This is how I do my core networks (including BGP systems).

North Idaho Tom Jones
 
User avatar
Wolfraider
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 56
Joined: Wed Jul 15, 2015 8:06 pm

Re: CHR Hardware

Wed Mar 20, 2019 7:55 pm

We will be assigning a couple of our ESX licenses for this. We like the idea of running 2 physical servers instead of 1 just in case of hardware failure. We are currently running a couple CCR1072's but we have not brought up our second 10Gb trunk up yet or received full routes from either carrier.
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1164
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: CHR Hardware

Wed Mar 20, 2019 8:26 pm

We will be assigning a couple of our ESX licenses for this. We like the idea of running 2 physical servers instead of 1 just in case of hardware failure. We are currently running a couple CCR1072's but we have not brought up our second 10Gb trunk up yet or received full routes from either carrier.
With good physical servers hosting your CHRs , I think you will see much better ROS functionality with CHR verses a physical Mikrotik for your BGP, routing, firewall and bandwidth management functions.

When you bring up your CHR , please post your full BGP table load times of the CHR and the physical Mikrotik. I would like to know how much faster the CHR is.

North Idaho Tom Jones
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1360
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: CHR Hardware

Thu Mar 21, 2019 3:27 pm

Hyper-V is hands down the best hypervisor for using a CHR as a BGP edge router. Mostly this is because MikroTIk spent a lot of time building the Hyper-V drivers for the CHR and they used off the shelf drivers for KVM/ESXi

The single biggest impact is to get a CPU with a higher clock speed and fewer cores vs. lots of cores at a lower clock speed.

Here is a presentation I did last year in Berlin on the topic and since then I've had a number of customers deploy on Hyper-V with a great deal of success and extremely rapid convergence times.

https://mum.mikrotik.com/presentations/ ... 562405.pdf
Global - MikroTik Support & Consulting - English | Español | Serbian | Danish +1 855-645-7684
https://iparchitechs.com/ecosystem/mikr ... consulting mikrotiksupport@iparchitechs.com
 
User avatar
Wolfraider
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 56
Joined: Wed Jul 15, 2015 8:06 pm

Re: CHR Hardware

Mon Mar 25, 2019 9:08 pm

We are not running eBGP as of yet but I will post numbers as soon as we get this implemented.

Your getting better performance with Hyper-V? I have heard others talk about better performance with ESXi. We could go either way. I'll post back numbers.
 
cmurrayis
Member Candidate
Member Candidate
Posts: 106
Joined: Fri May 15, 2009 4:31 am

Re: CHR Hardware

Fri Jun 07, 2019 7:00 am

Hyper-V is hands down the best hypervisor for using a CHR as a BGP edge router. Mostly this is because MikroTIk spent a lot of time building the Hyper-V drivers for the CHR and they used off the shelf drivers for KVM/ESXi

The single biggest impact is to get a CPU with a higher clock speed and fewer cores vs. lots of cores at a lower clock speed.

Here is a presentation I did last year in Berlin on the topic and since then I've had a number of customers deploy on Hyper-V with a great deal of success and extremely rapid convergence times.

https://mum.mikrotik.com/presentations/ ... 562405.pdf
@IPANetEngineer

We're wanting to do some testing with CHR running as a VM however we're needing more than 10gbps of throughput which generally isn't an issue however we're hitting 80% on our CCR1072's with around 800,000PPS

We're looking to use the Dell vep4600 as the hardware platform with the additional 8 x 10gbps modules. Given these are the Network Accelerated Xeon models we hoped for amazing results.

Do you have any recommendations?
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1164
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: CHR Hardware

Fri Jun 07, 2019 6:27 pm

Hyper-V is hands down the best hypervisor for using a CHR as a BGP edge router. Mostly this is because MikroTIk spent a lot of time building the Hyper-V drivers for the CHR and they used off the shelf drivers for KVM/ESXi

The single biggest impact is to get a CPU with a higher clock speed and fewer cores vs. lots of cores at a lower clock speed.

Here is a presentation I did last year in Berlin on the topic and since then I've had a number of customers deploy on Hyper-V with a great deal of success and extremely rapid convergence times.

https://mum.mikrotik.com/presentations/ ... 562405.pdf
@IPANetEngineer

We're wanting to do some testing with CHR running as a VM however we're needing more than 10gbps of throughput which generally isn't an issue however we're hitting 80% on our CCR1072's with around 800,000PPS

We're looking to use the Dell vep4600 as the hardware platform with the additional 8 x 10gbps modules. Given these are the Network Accelerated Xeon models we hoped for amazing results.

Do you have any recommendations?
Here is what I would suggest to consider:
CPU (qty 2):
Intel Xeon E5 Family
Intel CM8066002024000 Xeon E5-2698 v4 - 2.2 GHz - 20-Core - 40 Threads - 50 MB Cache - LGA2011 Socket - OEM
# of Cores: 20-Core
Series: Intel Xeon E5 Family
L2 Cache: 20 x 256KB
L3 Cache: 50MB

With 50 MB of cache , you have a high probability of having more CPU cache hits. CPU Cache hits will help keep you CHR running at CPU clock speed instead of ram speed.
Also - do NOT use hyper-threading


North Idaho Tom Jones
 
angriukas
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Fri Nov 22, 2013 9:20 am
Contact:

Re: CHR Hardware

Wed Jul 03, 2019 3:11 pm

Try to use Proxmox Virtualization Platform (PVE).
Just successfully tested CHR on PVE with KVM hypervisor.
CHR VM (virtual machine) supports following intefaces:
disk - SATA or Virtio (to boot CHR).
network - all types, but only Virtio and vmxnet (VMWare) supports 10G.
Better to use everywhere Virtio type of interfaces, because of better integration with PVE.
HA (High Availability) cluster can be built with min. 3 physical servers.
HA means - same file system on all of physical servers (same VM image - on all nodes). If Node1 will fail - VM automatically will be started on next node.
In case of HA, for best performance use host CPU or use lower CPU type of your nodes. I think default CPU type kvm64 will be also good enough.
We use PVE for years in our office, not for CHR, but for other tasks. If you like I can consult regarding PVE.
It's perfect opensource platform and is free of charge (with no subscription).
 
wpeople
Member
Member
Posts: 366
Joined: Sat May 26, 2007 6:36 pm

Re: CHR Hardware

Tue Sep 24, 2019 5:27 pm

for my experience (in LAB), Proxmox is much faster for Mikrotik virtualisation then ESXi.
Not tested with BGP, just routing and bandwidth test.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 1773
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CHR Hardware

Tue Sep 24, 2019 5:55 pm

for my experience (in LAB), Proxmox is much faster for Mikrotik virtualisation then ESXi.
Not tested with BGP, just routing and bandwidth test.

i had better results with ESXi than proxmox beyond 1gbps of live customer traffic

i have not tested hyper-V

looks like some optimal configuration define final performance
Last edited by chechito on Tue Sep 24, 2019 6:27 pm, edited 1 time in total.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 1773
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CHR Hardware

Tue Sep 24, 2019 6:25 pm

i propose a simplest and cheaper option to implement a CHR, maybe can be suitable ofr some implementations:

CPU: intel core i5 8400 (6 cores without HT) 2.8ghz base frequency
RAM: 8gb or ddr4 ram memory (split in 2 modules to take advantage of dual channel memory bandwidth)
Motherboard: z390 chipset motherboard
Storage: 120GB SSD
NIC: Dual SFP+ pci express NIC
Power supply: 80+ Gold, Active PFC, ATX power supply, 550watt
Standard or rack mount ATX enclosure
Line Interactive 2.5kva UPS

Aprox Cost: 800-900 USD

one option to improve performance is swapping for core i5 9600k CPU, then you easily get 30% more clock rate, with 150USD added cost
one option to improve performance is swapping for core i7 9700K CPU, then you easily get 30% more clock rate, plus to more cores (total 8 cores) with 300USD added cost

Then you can obtain 8 core 3.6ghz base clock CHR for about 1200USD

Off course is a very good idea implement a second CHR for high availability
 
User avatar
Maggiore81
Member
Member
Posts: 378
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CHR Hardware

Mon Nov 11, 2019 4:39 pm



We're wanting to do some testing with CHR running as a VM however we're needing more than 10gbps of throughput which generally isn't an issue however we're hitting 80% on our CCR1072's with around 800,000PPS

We're looking to use the Dell vep4600 as the hardware platform with the additional 8 x 10gbps modules. Given these are the Network Accelerated Xeon models we hoped for amazing results.

Do you have any recommendations?

Are you using IP FASTTRACK to lower the cpu load?
we use fasttrack and have 3gbps with about 10% cpu.
Dott. Elia Spadoni
---
Network Administrator
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
DUB
just joined
Posts: 3
Joined: Sat Mar 01, 2014 3:48 pm

Re: CHR Hardware

Fri Dec 20, 2019 5:36 am

Hyper-V is hands down the best hypervisor for using a CHR as a BGP edge router. Mostly this is because MikroTIk spent a lot of time building the Hyper-V drivers for the CHR and they used off the shelf drivers for KVM/ESXi

The single biggest impact is to get a CPU with a higher clock speed and fewer cores vs. lots of cores at a lower clock speed.

Here is a presentation I did last year in Berlin on the topic and since then I've had a number of customers deploy on Hyper-V with a great deal of success and extremely rapid convergence times.

https://mum.mikrotik.com/presentations/ ... 562405.pdf
@IPANetEngineer

We're wanting to do some testing with CHR running as a VM however we're needing more than 10gbps of throughput which generally isn't an issue however we're hitting 80% on our CCR1072's with around 800,000PPS

We're looking to use the Dell vep4600 as the hardware platform with the additional 8 x 10gbps modules. Given these are the Network Accelerated Xeon models we hoped for amazing results.

Do you have any recommendations?

Hi,
I have been looking at that product as well.

Very interested in:
"First to market with networking optimized Intel® Xeon® D-2100 x86-based processor
Accelerates packet processing with Intel® Data Plane Development Kit (DPDK)
Accelerates security encryption with Intel® QuickAssist Technology (QAT)"

It does sound like it should work pretty well, I am a little concerned about the slow clock speed on the D-2100 though...

Have you ever gotten any info if it works well?
I placed some technical questions with our Dell rep today, waiting on answers.....

Thank you,
DUB
 
okinawajoe
just joined
Posts: 1
Joined: Wed Sep 30, 2020 9:39 am

Re: CHR Hardware

Wed Sep 30, 2020 9:59 am

For two 10-Gig BGP CHR routers , this is what I do (and it works good).

- One physical VMware ESXi box (Free version of VMware ESXi)
- Two Xeon CPUs - with a minimum of 10 cores per Xeon CPU.
- Lots of Xeon CPU cache helps
- Disable Hyper-threading
-- On 1st CHR , configure eight CPUs (CPUs 2-through-10) ((( dedicated configuration to make this CHR run only on the 1st Xeon CPU )))
-- On 2nd CHR , configure eight CPUs (CPUs 12-through-20) ((( dedicated configuration to make this CHR run only on the 2nd Xeon CPU )))
((( Note ; Each CHR will run on a different physical Xeon processor -and- each CHR will have access to the entire built-in Xeon CPU cache )))
((( Note ; The 2nd CHR which runs on the 2nd Xeon CPU might have slightly faster throughput )))
((( Note ; On your CHRs , use VMXNET-3 network interfaces only )))

You should have a second cold-spare physical VMware ESXi server ready to use if you have a hardware problem with the first physical box.
Your 2nd physical box could also be running a 3rd CHR to handle any OSPF traffic and/or routing to your customers.
Your 2nd physical box could also be running a 4th CHR to handle any customer bandwidth limiter configurations you may have to limit customer up/down bandwidths.

I always suggest a stand-by physical Hyper-Visor server (VMware ESXi) with some CHRs configured , licensed and ready to go) to minimize any possible down-time in the event you have some serious hardware failures. The spare stand-by physical box and spare stand-by CHRs when not needed will make a great LAB system for testing stuff.

CHR P-Unlimited licenses are cost effective (about $250 per license). The only real expense is the physical servers ( In my case SuperMicro servers ) and some 10-gig switches.

This is how I do my core networks (including BGP systems).

North Idaho Tom Jones
Tom - I really appreciate all of your insight and interaction as it relates to running BGP on CHR nodes. We are a small MSO serving a rather demanding customer base, and currently peak between 30 - 36Gbps at peak hour, and run flat out at our entire 40Gbps during any Call of Duty updates. We presently have a single default route to our primary transit provider, though we are moving towards multiple transit providers and a need for capacity to run two BGP full tables.

Given these throughput requirements, can you share your thoughts on how you might approach this? After reading this thread, I was thinking perhaps two physical ESXi or Hyper-V hosts running in parallel, each servicing 20Gbps of traffic.

Would you present the CHRs as individual peers to the upstream transit providers, or apply some type of load balancing? When evaluating other carrier class routers capable of running dual full view tables, the exorbitant costs led me to search for alternatives, and that is how I came across your thread.
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1164
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: CHR Hardware

Fri Oct 02, 2020 12:33 am

For two 10-Gig BGP CHR routers , this is what I do (and it works good).

- One physical VMware ESXi box (Free version of VMware ESXi)
- Two Xeon CPUs - with a minimum of 10 cores per Xeon CPU.
- Lots of Xeon CPU cache helps
- Disable Hyper-threading
-- On 1st CHR , configure eight CPUs (CPUs 2-through-10) ((( dedicated configuration to make this CHR run only on the 1st Xeon CPU )))
-- On 2nd CHR , configure eight CPUs (CPUs 12-through-20) ((( dedicated configuration to make this CHR run only on the 2nd Xeon CPU )))
((( Note ; Each CHR will run on a different physical Xeon processor -and- each CHR will have access to the entire built-in Xeon CPU cache )))
((( Note ; The 2nd CHR which runs on the 2nd Xeon CPU might have slightly faster throughput )))
((( Note ; On your CHRs , use VMXNET-3 network interfaces only )))

You should have a second cold-spare physical VMware ESXi server ready to use if you have a hardware problem with the first physical box.
Your 2nd physical box could also be running a 3rd CHR to handle any OSPF traffic and/or routing to your customers.
Your 2nd physical box could also be running a 4th CHR to handle any customer bandwidth limiter configurations you may have to limit customer up/down bandwidths.

I always suggest a stand-by physical Hyper-Visor server (VMware ESXi) with some CHRs configured , licensed and ready to go) to minimize any possible down-time in the event you have some serious hardware failures. The spare stand-by physical box and spare stand-by CHRs when not needed will make a great LAB system for testing stuff.

CHR P-Unlimited licenses are cost effective (about $250 per license). The only real expense is the physical servers ( In my case SuperMicro servers ) and some 10-gig switches.

This is how I do my core networks (including BGP systems).

North Idaho Tom Jones
Tom - I really appreciate all of your insight and interaction as it relates to running BGP on CHR nodes. We are a small MSO serving a rather demanding customer base, and currently peak between 30 - 36Gbps at peak hour, and run flat out at our entire 40Gbps during any Call of Duty updates. We presently have a single default route to our primary transit provider, though we are moving towards multiple transit providers and a need for capacity to run two BGP full tables.

Given these throughput requirements, can you share your thoughts on how you might approach this? After reading this thread, I was thinking perhaps two physical ESXi or Hyper-V hosts running in parallel, each servicing 20Gbps of traffic.

Would you present the CHRs as individual peers to the upstream transit providers, or apply some type of load balancing? When evaluating other carrier class routers capable of running dual full view tables, the exorbitant costs led me to search for alternatives, and that is how I came across your thread.
Wow - You've got some good high-end questions. There is no single correct answer.
However , If it were me ... , I think I would consider something like this:
- One primary high-end VmWare ESXi server ( and possibly a second warm-spare server )
Two or Four Xeon CPUs ( lots of cores & lots of CPU cache
128 Gig or 256 Gig of RAM memory
One or Two high-throughput network card ( 100 Gig would be best )
- High throughput layer 2-switch ( 100 Gig ports would be best )

I would try this in a single VmWare ESXi server:
- Two CHR BGP-Peering routers
* Each BGP peering router has it's own 10-100-Gig interface - So you are burning up two ports for WANs to your BGP routers on the WAN interfaces
- A third CHR or PfSense router which is doing OSPF to my two BGP peering routers
* the two BGP routers would do OSPF to your OSPF router and no traffic would be going through an external switch * faster I/0 using using on the network interfaces *
- A fourth ( or more ) distribution routers routers on the same VmWare ESXi server ( these routers talk to your OSPF router - not the BGP servers )
* The WANs on your distribution routes don't go through an external switch * faster network I/O
** At his point , everything ( excluding your distribution router LANs ) is running inside one signe VmWare ESXi server * Faster network I/O network throughput because you are not going through external networks/switches
*** You only have four virtual machines on your VmWare ESXi server ( CHR#1-BGP & CHR#2-BGP & CHR-OSPF0router & CHR-Distribution-router )
Note - anything talking to your distribution router LANs should be going through their own switches *** BGP WANs and distribution LANs are not on the same switch *** keep it simple and keep it fast

Note : In theory , if you are on a fast enough VmWare ESXi server , virtual switches should outrun physical switches ( because a virtual switch is software instead of physical - and the only limitation on a virtual switch throughput is how fast is the CPU.


my thoughts
North Idaho Tom Jones
 
User avatar
Maggiore81
Member
Member
Posts: 378
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CHR Hardware

Sun Nov 08, 2020 12:05 pm

[/quote]

Wow - You've got some good high-end questions. There is no single correct answer.
However , If it were me ... , I think I would consider something like this:
- One primary high-end VmWare ESXi server ( and possibly a second warm-spare server )
Two or Four Xeon CPUs ( lots of cores & lots of CPU cache
128 Gig or 256 Gig of RAM memory
One or Two high-throughput network card ( 100 Gig would be best )
- High throughput layer 2-switch ( 100 Gig ports would be best )
my thoughts
North Idaho Tom Jones
[/quote]

What is the purpose of 128G of memory since the box needs to do BGP and routing?
It can happily run with 16G... or am I overseeing something?
Dott. Elia Spadoni
---
Network Administrator
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1164
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: CHR Hardware

Mon Nov 09, 2020 10:30 pm

Re: What is the purpose of 128G of memory since the box needs to do BGP and routing?

My answer: Build it and they will come

If you go skimpy on the RAM/CPU/Network-IO , it may come back later to bite you.
Let's say that later , you wanted to change a CHR router to a PfSense. or you wanted to add a PfSense router , you got RAM memory to do so.
Let's say you need a new firewall ( example PfSense ) , and you wanted to also install some additional packages such as :
bandwidthd
pfBlockerNG-devel
snort
squidguard
zabbix-agent
** and some additional on-line lists ( spammer black-lists, block china, block hacker IP addresses ) and have these lists auto update.

Who is online

Users browsing this forum: No registered users and 2 guests