Page 1 of 1

> 10 gigabit

Posted: Wed Jun 10, 2020 3:36 am
by Hammy
Have any of you been able to get more than 10 gigabits through an interface in CHR? We have a bonded 10G in vSphere that we're presenting to a CHR and we start getting packet loss at about 8.5 gigabit/s. The CPU is in the 70% area, so we should be able to go higher.

I spun up another CHR to test packet loss through the troubled vSwitch. When the primary CHR is encountering about 5% packet loss, there is 0 packet loss on the test CHR.
So it's not a physical issue.
It's not likely a vSwitch issue.
It seems to be a virtual port issue or a Mikrotik\CHR problem.

Re: > 10 gigabit

Posted: Wed Jun 10, 2020 5:27 pm
by elbob2002
Your license is probably restricting you.

While with a P10 license will restrict you to 10Gb per interface this probably includes your LAG interface too.

Re: > 10 gigabit

Posted: Tue Jun 23, 2020 2:55 am
by TomjNorthIdaho
1st - check your license level.
2nd - If using VmWare ESXi, then use vmxnet3 interfaces

Re: > 10 gigabit

Posted: Tue Jun 23, 2020 2:59 am
by Hammy
We have an unlimited license. I was using vmxnet3, I moved a couple of the adapters to use PCI pass-through and that helped. We're still encountering issues in the router. The device that were using must not be capable of generating enough CPU power to be useful.

Re: > 10 gigabit

Posted: Tue Jun 23, 2020 10:06 pm
by TomjNorthIdaho
We have an unlimited license. I was using vmxnet3, I moved a couple of the adapters to use PCI pass-through and that helped. We're still encountering issues in the router. The device that were using must not be capable of generating enough CPU power to be useful.
For me , do this test and let me know what you get:
btest to 127.0.0.1 ( udp send )
If this is not well over 10-Gig , then it's your hardware - or a configuration issue.

Re: > 10 gigabit

Posted: Tue Jun 23, 2020 10:10 pm
by Hammy
We have an unlimited license. I was using vmxnet3, I moved a couple of the adapters to use PCI pass-through and that helped. We're still encountering issues in the router. The device that were using must not be capable of generating enough CPU power to be useful.
For me , do this test and let me know what you get:
btest to 127.0.0.1 ( udp send )
If this is not well over 10-Gig , then it's your hardware - or a configuration issue.
I got 750 - 850 gigs.

Re: > 10 gigabit

Posted: Tue Jun 23, 2020 10:15 pm
by TomjNorthIdaho
What type of physical CPUs are you using ( Xeon ? )
What is the VMware ESXi processor type ?
- Try disabling hyper-threading
- How many other virtual hosts ( virtual machines ) and how many CPUs are total assigned to all virtual hosts ?
- How many CPUs are assigned to your CHR ?
- Is your VMware ESXi the latest newest version ( at least 6.7.0 latest patch release ?
- Do you have any other CHRs on other boxes to test to ?
- How are you measuring speed ? ( from what to what ? )

Re: > 10 gigabit

Posted: Tue Jun 23, 2020 10:21 pm
by Hammy
It's a "4 CPUs x Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz" in a Maxxwave Vengeance.

Hyperthreading is disabled.
All CPUs assigned to this CHR.
We created a second CHR when troubleshooting the problem, simply to be a ping destination on the same vSwitch.
It's still at 6.5. We're not using vCenter, so we're waiting for a good maintenance window to upgrade it.
No other CHRs on local boxes.
The speed I'm measuring is live traffic going through the router, from various CDN cache boxes behind it to a variety of networks upstream.

Re: > 10 gigabit

Posted: Tue Jun 23, 2020 10:28 pm
by TomjNorthIdaho
FYI - Here is my CHR ( running on a slow physical box ) :

Re: > 10 gigabit

Posted: Tue Jun 23, 2020 10:39 pm
by TomjNorthIdaho
I suppose any btest to 127.0.0.1 faster than 100 Gig may be fast enough for 10-Gig interfaces or some 40-Gig interfaces.

In your virtual machine BIOS setup , disable all un-necessary devices (cd-rom, serial ports, parallel-printer ports).
In your physical box, verify you are not re-using interrupts. Shared interrupts create a slight delay where the OS has to figure out which device made the interrupt.

Re: > 10 gigabit

Posted: Tue Jun 23, 2020 10:39 pm
by Hammy
Here are some updated CPU load and throughput charts, with all of our latest changes.

Re: > 10 gigabit

Posted: Wed Jun 24, 2020 12:31 am
by Hammy
I suppose any btest to 127.0.0.1 faster than 100 Gig may be fast enough for 10-Gig interfaces or some 40-Gig interfaces.

In your virtual machine BIOS setup , disable all un-necessary devices (cd-rom, serial ports, parallel-printer ports).
In your physical box, verify you are not re-using interrupts. Shared interrupts create a slight delay where the OS has to figure out which device made the interrupt.
I did remove as much virtual hardware as I could.

I'll have them check the host BIOS for things to shut off.

Re: > 10 gigabit

Posted: Wed Jun 24, 2020 7:51 am
by kujo
Try to play with interface queue.
What the difference in test CHR and production on load /tool profiler?

Re: > 10 gigabit

Posted: Wed Jun 24, 2020 3:42 pm
by Hammy
Try to play with interface queue.
What the difference in test CHR and production on load /tool profiler?
The only thing the test CHR is for being a ping destination. It has one interface, one IP address, etc

The queues are set to multi-queue-ethernet-default for the virtual NICs. I forgot to change the settings on the ones I did PCI passthrough on. Does the RouterOS driver for Intel 10G NICs support multiple queues?