Community discussions

MikroTik App
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Topic Author
Posts: 761
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

> 10 gigabit

Wed Jun 10, 2020 3:36 am

Have any of you been able to get more than 10 gigabits through an interface in CHR? We have a bonded 10G in vSphere that we're presenting to a CHR and we start getting packet loss at about 8.5 gigabit/s. The CPU is in the 70% area, so we should be able to go higher.

I spun up another CHR to test packet loss through the troubled vSwitch. When the primary CHR is encountering about 5% packet loss, there is 0 packet loss on the test CHR.
So it's not a physical issue.
It's not likely a vSwitch issue.
It seems to be a virtual port issue or a Mikrotik\CHR problem.
-----
Mike Hammett

The Brothers WISP
 
elbob2002
Frequent Visitor
Frequent Visitor
Posts: 70
Joined: Tue May 15, 2018 8:15 pm
Location: Ireland

Re: > 10 gigabit

Wed Jun 10, 2020 5:27 pm

Your license is probably restricting you.

While with a P10 license will restrict you to 10Gb per interface this probably includes your LAG interface too.
RB3011UiAS, RB750GR3, CRS328-24P-4S+, CRS125-24G-1S, CRS125-24G-1S-2HnD, 8 x CHR
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1104
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: > 10 gigabit

Tue Jun 23, 2020 2:55 am

1st - check your license level.
2nd - If using VmWare ESXi, then use vmxnet3 interfaces
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Topic Author
Posts: 761
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: > 10 gigabit

Tue Jun 23, 2020 2:59 am

We have an unlimited license. I was using vmxnet3, I moved a couple of the adapters to use PCI pass-through and that helped. We're still encountering issues in the router. The device that were using must not be capable of generating enough CPU power to be useful.
-----
Mike Hammett

The Brothers WISP
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1104
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: > 10 gigabit

Tue Jun 23, 2020 10:06 pm

We have an unlimited license. I was using vmxnet3, I moved a couple of the adapters to use PCI pass-through and that helped. We're still encountering issues in the router. The device that were using must not be capable of generating enough CPU power to be useful.
For me , do this test and let me know what you get:
btest to 127.0.0.1 ( udp send )
If this is not well over 10-Gig , then it's your hardware - or a configuration issue.
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Topic Author
Posts: 761
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: > 10 gigabit

Tue Jun 23, 2020 10:10 pm

We have an unlimited license. I was using vmxnet3, I moved a couple of the adapters to use PCI pass-through and that helped. We're still encountering issues in the router. The device that were using must not be capable of generating enough CPU power to be useful.
For me , do this test and let me know what you get:
btest to 127.0.0.1 ( udp send )
If this is not well over 10-Gig , then it's your hardware - or a configuration issue.
I got 750 - 850 gigs.
-----
Mike Hammett

The Brothers WISP
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1104
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: > 10 gigabit

Tue Jun 23, 2020 10:15 pm

What type of physical CPUs are you using ( Xeon ? )
What is the VMware ESXi processor type ?
- Try disabling hyper-threading
- How many other virtual hosts ( virtual machines ) and how many CPUs are total assigned to all virtual hosts ?
- How many CPUs are assigned to your CHR ?
- Is your VMware ESXi the latest newest version ( at least 6.7.0 latest patch release ?
- Do you have any other CHRs on other boxes to test to ?
- How are you measuring speed ? ( from what to what ? )
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Topic Author
Posts: 761
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: > 10 gigabit

Tue Jun 23, 2020 10:21 pm

It's a "4 CPUs x Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz" in a Maxxwave Vengeance.

Hyperthreading is disabled.
All CPUs assigned to this CHR.
We created a second CHR when troubleshooting the problem, simply to be a ping destination on the same vSwitch.
It's still at 6.5. We're not using vCenter, so we're waiting for a good maintenance window to upgrade it.
No other CHRs on local boxes.
The speed I'm measuring is live traffic going through the router, from various CDN cache boxes behind it to a variety of networks upstream.
-----
Mike Hammett

The Brothers WISP
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1104
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: > 10 gigabit

Tue Jun 23, 2020 10:28 pm

FYI - Here is my CHR ( running on a slow physical box ) :
You do not have the required permissions to view the files attached to this post.
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1104
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: > 10 gigabit

Tue Jun 23, 2020 10:39 pm

I suppose any btest to 127.0.0.1 faster than 100 Gig may be fast enough for 10-Gig interfaces or some 40-Gig interfaces.

In your virtual machine BIOS setup , disable all un-necessary devices (cd-rom, serial ports, parallel-printer ports).
In your physical box, verify you are not re-using interrupts. Shared interrupts create a slight delay where the OS has to figure out which device made the interrupt.
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Topic Author
Posts: 761
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: > 10 gigabit

Tue Jun 23, 2020 10:39 pm

Here are some updated CPU load and throughput charts, with all of our latest changes.
You do not have the required permissions to view the files attached to this post.
-----
Mike Hammett

The Brothers WISP
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Topic Author
Posts: 761
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: > 10 gigabit

Wed Jun 24, 2020 12:31 am

I suppose any btest to 127.0.0.1 faster than 100 Gig may be fast enough for 10-Gig interfaces or some 40-Gig interfaces.

In your virtual machine BIOS setup , disable all un-necessary devices (cd-rom, serial ports, parallel-printer ports).
In your physical box, verify you are not re-using interrupts. Shared interrupts create a slight delay where the OS has to figure out which device made the interrupt.
I did remove as much virtual hardware as I could.

I'll have them check the host BIOS for things to shut off.
-----
Mike Hammett

The Brothers WISP
 
kujo
Member Candidate
Member Candidate
Posts: 164
Joined: Sat Jun 18, 2016 10:17 am
Location: Ukraine

Re: > 10 gigabit

Wed Jun 24, 2020 7:51 am

Try to play with interface queue.
What the difference in test CHR and production on load /tool profiler?
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Topic Author
Posts: 761
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: > 10 gigabit

Wed Jun 24, 2020 3:42 pm

Try to play with interface queue.
What the difference in test CHR and production on load /tool profiler?
The only thing the test CHR is for being a ping destination. It has one interface, one IP address, etc

The queues are set to multi-queue-ethernet-default for the virtual NICs. I forgot to change the settings on the ones I did PCI passthrough on. Does the RouterOS driver for Intel 10G NICs support multiple queues?
-----
Mike Hammett

The Brothers WISP

Who is online

Users browsing this forum: No registered users and 8 guests