Community discussions

MikroTik App
 
User avatar
ScottReed
Member Candidate
Member Candidate
Topic Author
Posts: 114
Joined: Thu Sep 24, 2009 9:47 pm
Location: Montana / Western Massachusetts

CHR tx-queue-drops-per-second

Thu Apr 02, 2020 9:42 pm

CHR 6.45.8
RouterMaxx Vengeance on ESXi 6.7U3 / i7-7700 / 32GB

Using as an edge router connected to IX and ISP. 90+ BGP peering sessions. Partial routes from ISP.

Any sort of load on any of the interfaces results in upwards of ~3000 tx-queue-drops-per-second.

For example:

/interface monitor-traffic interface=sfp-sfpplus7
                         name:  sfp-sfpplus7
        rx-packets-per-second:       129 961
           rx-bits-per-second:     290.7Mbps
     fp-rx-packets-per-second:             0
        fp-rx-bits-per-second:          0bps
          rx-drops-per-second:             0
         rx-errors-per-second:             0
        tx-packets-per-second:       218 756
           tx-bits-per-second:       2.1Gbps
     fp-tx-packets-per-second:             0
        fp-tx-bits-per-second:          0bps
          tx-drops-per-second:             0
    tx-queue-drops-per-second:         2 836
         tx-errors-per-second:             0

This is happening on both the front expansion MaxxWave 4-Port 10G module and the read expansion HotLava St. Helens 4-Port 10G card. Using FS modules.

CHR CPU ~40% @ 5Gbps
ESXi CPU Pkg ~80% @ 5Gbps

I'm scheduling a time to disable hyper-threading in the BIOS to hopefully obtain some more performance, but I'm doubtful this will be the fix.

Outside of NorthIdahoTom's great CHR performance tips, is there anything else anyone can think of doing?
 
User avatar
ScottReed
Member Candidate
Member Candidate
Topic Author
Posts: 114
Joined: Thu Sep 24, 2009 9:47 pm
Location: Montana / Western Massachusetts

Re: CHR tx-queue-drops-per-second

Fri Apr 03, 2020 3:28 pm

We were unable to get Hyper Threading disabled on Aptio BIOS v2.18.1263

The Advanced/CPU Configuration menu shows all grey options that cannot be modified. Surprisingly, Hyper Threading wasn't listed which is strange because the i7-7700 certainly supports it and it shows "enabled" in ESXi.
 
faisali
Member Candidate
Member Candidate
Posts: 180
Joined: Fri Oct 08, 2010 5:11 am

Re: CHR tx-queue-drops-per-second

Fri May 08, 2020 6:00 am

Just a suggestion:-
try to change change the queue type to multi-queue-ethernet
and also look into how to enable that for the nic via the hypervisor / Vmware.
 
PortalNET
Member Candidate
Member Candidate
Posts: 132
Joined: Sun Apr 02, 2017 7:24 pm

Re: CHR tx-queue-drops-per-second

Thu Dec 07, 2023 11:43 pm

Just a suggestion:-
try to change change the queue type to multi-queue-ethernet
and also look into how to enable that for the nic via the hypervisor / Vmware.
Hi guys i am having the same issue on RoS V7.12 x64bits running with 2 intel X520 da2 cards..

anyone find a fix for that?

on interface queue i have set only-hardware-queue.. can i change it to multi-queue-ethernet?


EDIT1: After changing in Queues-list / Interface Queues to multi-queue-ethernet-default.. running over 10mins now and tx queue drops 0 and RX Erros 0 also..
 
User avatar
ahmdzaki18
just joined
Posts: 12
Joined: Fri Oct 06, 2023 7:52 pm
Location: Jakarta, Indonesia
Contact:

Re: CHR tx-queue-drops-per-second

Sun Dec 17, 2023 11:28 am

Try to using cpu affinity for dedicated CHR cpu and SR-IOV to bypass esxi kernel for using dedicated CHR NIC.

Virtualization always be a nightmare for networking if not using ideal configuration.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1177
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: CHR tx-queue-drops-per-second

Mon Dec 18, 2023 12:20 pm

”Try to using cpu affinity for dedicated CHR cpu and SR-IOV to bypass esxi kernel for using dedicated CHR NIC.”

Yeah, that should be pinned somewhere as best practice.
 
wpeople
Member
Member
Posts: 380
Joined: Sat May 26, 2007 6:36 pm

Re: CHR tx-queue-drops-per-second

Sun Apr 14, 2024 9:39 pm

if you want to run your router in VM, its highly recommended to use PCI-passthru for passing the whole NIC to the VM.

SR-IOV is great - unless you want to have quite a big traffic - since it can do only 1 or 2 rxtx-queue per port, while the X520DA-2 NIC can have like 64 or 128 queues per port.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1177
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: CHR tx-queue-drops-per-second

Mon Apr 15, 2024 3:30 pm

Not necessarily. It ultimately depends on how well the driver is developed specifically for each solution. With a single NIC used solely by one guest OS, the difference is probably not even measurable with modern drivers. The major difference is that a NIC using PCI passthrough (VMware DirectPath) becomes locked to a single guest, whereas with SR-IOV it can be shared by multiple guests. NIC RX/TX queue sizes are directly related to the hw and device driver's ability.

There are plenty of articles online that describe all of this in detail.
 
User avatar
inteq
Member
Member
Posts: 416
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: CHR tx-queue-drops-per-second

Wed May 01, 2024 5:01 pm

Just a suggestion:-
try to change change the queue type to multi-queue-ethernet
and also look into how to enable that for the nic via the hypervisor / Vmware.
Hi guys i am having the same issue on RoS V7.12 x64bits running with 2 intel X520 da2 cards..

anyone find a fix for that?

on interface queue i have set only-hardware-queue.. can i change it to multi-queue-ethernet?


EDIT1: After changing in Queues-list / Interface Queues to multi-queue-ethernet-default.. running over 10mins now and tx queue drops 0 and RX Erros 0 also..
Thank you.
Worked for me inside a CHR.
Was wondering why the counter increased even with very low bandwidth used.
Read the wiki...a bit smarter now, but still dumb in this regard.
drops.png
You do not have the required permissions to view the files attached to this post.

Who is online

Users browsing this forum: intelvtd and 2 guests