> 10 gigabit

RouterOS Virtualization
1

Have any of you been able to get more than 10 gigabits through an interface in CHR? We have a bonded 10G in vSphere that we’re presenting to a CHR and we start getting packet loss at about 8.5 gigabit/s. The CPU is in the 70% area, so we should be able to go higher.

I spun up another CHR to test packet loss through the troubled vSwitch. When the primary CHR is encountering about 5% packet loss, there is 0 packet loss on the test CHR.
So it’s not a physical issue.
It’s not likely a vSwitch issue.
It seems to be a virtual port issue or a Mikrotik\CHR problem.

2

Your license is probably restricting you.

While with a P10 license will restrict you to 10Gb per interface this probably includes your LAG interface too.

3

1st - check your license level.
2nd - If using VmWare ESXi, then use vmxnet3 interfaces

4

We have an unlimited license. I was using vmxnet3, I moved a couple of the adapters to use PCI pass-through and that helped. We’re still encountering issues in the router. The device that were using must not be capable of generating enough CPU power to be useful.

5

For me , do this test and let me know what you get:
btest to 127.0.0.1 ( udp send )
If this is not well over 10-Gig , then it’s your hardware - or a configuration issue.

6

I got 750 - 850 gigs.

7

What type of physical CPUs are you using ( Xeon ? )
What is the VMware ESXi processor type ?

  • Try disabling hyper-threading
  • How many other virtual hosts ( virtual machines ) and how many CPUs are total assigned to all virtual hosts ?
  • How many CPUs are assigned to your CHR ?
  • Is your VMware ESXi the latest newest version ( at least 6.7.0 latest patch release ?
  • Do you have any other CHRs on other boxes to test to ?
  • How are you measuring speed ? ( from what to what ? )
8

It’s a “4 CPUs x Intel(R) Core™ i7-7700K CPU @ 4.20GHz” in a Maxxwave Vengeance.

Hyperthreading is disabled.
All CPUs assigned to this CHR.
We created a second CHR when troubleshooting the problem, simply to be a ping destination on the same vSwitch.
It’s still at 6.5. We’re not using vCenter, so we’re waiting for a good maintenance window to upgrade it.
No other CHRs on local boxes.
The speed I’m measuring is live traffic going through the router, from various CDN cache boxes behind it to a variety of networks upstream.

9

FYI - Here is my CHR ( running on a slow physical box ) :
btest-300Gig.png

10

I suppose any btest to 127.0.0.1 faster than 100 Gig may be fast enough for 10-Gig interfaces or some 40-Gig interfaces.

In your virtual machine BIOS setup , disable all un-necessary devices (cd-rom, serial ports, parallel-printer ports).
In your physical box, verify you are not re-using interrupts. Shared interrupts create a slight delay where the OS has to figure out which device made the interrupt.

11

Here are some updated CPU load and throughput charts, with all of our latest changes.
graph (21).png
graph (22).png
graph (23).png
graph (24).png
graph (25).png
graph (27).png
graph (26).png
graph (28).png
graph (31).png
graph (30).png
graph (29).png
graph (33).png
graph (32).png

12

I did remove as much virtual hardware as I could.

I’ll have them check the host BIOS for things to shut off.

13

Try to play with interface queue.
What the difference in test CHR and production on load /tool profiler?

14

The only thing the test CHR is for being a ping destination. It has one interface, one IP address, etc

The queues are set to multi-queue-ethernet-default for the virtual NICs. I forgot to change the settings on the ones I did PCI passthrough on. Does the RouterOS driver for Intel 10G NICs support multiple queues?