I would really love to see a proper x86-64 release for bare metal installs.

There are a number of benchmarks out there between the various hyper-visors and one thing is quite clear. The hyper-visor IS in the way of maximum performance.

For example, VMWARE is the absolute slowest for BGP. Hyper-V is nearly an order of magnitude faster
For general routing, VMWARE has the highest throughput by a bit, Hyper-V the slowest.
KVM (usually via proxmox) sits right in the middle.

If you look at Linux BGP installs and compare virtualized vs bare metal, you can see a similar performance gap as routes are transported between userspace and kernel in high volumes.

We lack a high performance Mikrotik device for BGP. CCRs are routing monsters, but the general purpose computing performance is actually pretty poor due a bit to architecture but a lot to low Mhz.

Re: I would really love to see a proper x86-64 release for bare metal installs.
An ISO install for CHR ROS x86-64 does have some advantages over a hyper-visor installation:
- With an ISO install on bare metal, the ROS system gains full use of all CPU cache and all cores.
--- (The more CPU cache in the Xeon, the more likely your BGP is running at the faster CPU cache speed instead of external memory buss speed

Re: VMWARE is the absolute slowest for BGP
mabey not - CHR on a new modern Xeon (with lots of CPU cache) running my BGP (8-CPUs assigned via VmWare ESXi
- core running BGP (load avg 28 to 40 with very few (quick peaks to 100 %
- overall CPU load averages 15% (all CPUs combined)

In my opinion, fast hardware is the most important , then second is what ever hypervisor you prefer.

Info - fyi - my CHR (running BGP) under VmWare is able to btest to 127.0.0.1 and consistantly acheive 20 to 22 gig throughput (and this is now an older box).


North Idaho Tom Jones

Using a MUM presentation on the subject: "Using Mikrotik CHR as a BGP edge router"
https://mum.mikrotik.com/presentations/ ... 562405.pdf

loading the LINX table took 4 minutes and 46 seconds on VMWare and an i7-7700k. OUCH. Now add load, they pushed 5G through it and convergence went to 11 Minutes! Totally unacceptable.

proxmox was 1:34 and 9:03 which is a bit better, but still a big problem under load.

Hyper-v did it in 12 seconds. Under load, 41 seconds. THAT's usable.

Loading routes is a simple operation done many many times and latency from userspace to kernel is responsible for almost all BGP performance. Hyper-V's hypervisor architecture happens to have the lowest latency for this type of operation.

Hyper-v numbers look a lot like Linux+Quagga numbers but I can't find a Linux+quagga on such powerful hardware... So there is still room for improvement.

And guess what, the *old* x86 routeros is still king for BGP. CHR can't touch it except on Hyper-V, and CHR still slower despite probably being run on faster/more modern CPUs. There are plenty of gotcha's with x86 version, specifically the memory limit. We are fast approaching a full gig per peer (IPv4+IPv6).

Basically, it doesn't make a lot of sense to run a Windows computer just to run a CHR on just to get decent BGP speeds.

Further, if you actually want to have full routes at your towers because you are multi-homed you really don't have a great option.

It's really time for an x86-64 bit release *OR* just make an x86-64 bare metal *routerboard* , they did it before with the rb230, granted those were not the fastest things... You can get a LOT of Ram and Ghz in a small package in x86-64 these days.

Using a MUM presentation on the subject: "Using Mikrotik CHR as a BGP edge router"
https://mum.mikrotik.com/presentations/ ... 562405.pdf

loading the LINX table took 4 minutes and 46 seconds on VMWare and an i7-7700k. OUCH. Now add load, they pushed 5G through it and convergence went to 11 Minutes! Totally unacceptable.

proxmox was 1:34 and 9:03 which is a bit better, but still a big problem under load.

Hyper-v did it in 12 seconds. Under load, 41 seconds. THAT's usable.

Loading routes is a simple operation done many many times and latency from userspace to kernel is responsible for almost all BGP performance. Hyper-V's hypervisor architecture happens to have the lowest latency for this type of operation.

Hyper-v numbers look a lot like Linux+Quagga numbers but I can't find a Linux+quagga on such powerful hardware... So there is still room for improvement.

And guess what, the *old* x86 routeros is still king for BGP. CHR can't touch it except on Hyper-V, and CHR still slower despite probably being run on faster/more modern CPUs. There are plenty of gotcha's with x86 version, specifically the memory limit. We are fast approaching a full gig per peer (IPv4+IPv6).

Basically, it doesn't make a lot of sense to run a Windows computer just to run a CHR on just to get decent BGP speeds.

Further, if you actually want to have full routes at your towers because you are multi-homed you really don't have a great option.

It's really time for an x86-64 bit release *OR* just make an x86-64 bare metal *routerboard* , they did it before with the rb230, granted those were not the fastest things... You can get a LOT of Ram and Ghz in a small package in x86-64 these days.

I agree that bare metal sounds like the fastest possible solution - due to the fact that there would be no hypervisor overhead - which could allow all interfaces being directly managed by the bare metal ROS install.
However - the underlying issue to bare metal and CHR is the fact that there is no ISO for a bare metal install of a CHR.
So - A thought I have that might work to acheive a CHR on bare metal is:
** Only a thought & this depends if the CHR can directly talk to the network devices ***
Using a hyper visor (in my case VmWare ESXi would be to attempt something like this procedure ...
- 1st , get a CHR running on the hyper visor system
- 2nd , add a raw/dedicated disk to the hypervisor and attempt a raw copy (or some sort of a DD) from the hyper visor CHR disk system to the new raw/dedicated IDE hard disk.
- 3rd , after the copy , re-install the IDE hard disk into a bare-metal computer and attempt to boot the IDE hard disk.
---or--- another possible method ---
- a) on the hypervisor system, install the CHR system to a dedicated raw (semi-stand-alone) IDE hard disk.
- b) on the hypervisor , make the virtual hardware match the bare-metal hardware
- c) then remove the IDE hdd and install in in the bare metal and try booting the IDE hdd on the bare metal box.

*** I am pretty sure the HDD and HDD controller card might need to be IDE because I dont think there is any SAS,SATA,SCSI drivers in CHR
*** I suspect that a native Intel E1000 and/or E1000e network card should work because the drivers are already in the CHR
*** I have no idea about 10-gig network cards - but possibly if they can use ethernet drivers already in the CHR

Anybody have any thoughts ?