CHR tx-queue-drops-per-second

RouterOS Virtualization
1

CHR 6.45.8
RouterMaxx Vengeance on ESXi 6.7U3 / i7-7700 / 32GB

Using as an edge router connected to IX and ISP. 90+ BGP peering sessions. Partial routes from ISP.

Any sort of load on any of the interfaces results in upwards of ~3000 tx-queue-drops-per-second.

For example:


/interface monitor-traffic interface=sfp-sfpplus7
                         name:  sfp-sfpplus7
        rx-packets-per-second:       129 961
           rx-bits-per-second:     290.7Mbps
     fp-rx-packets-per-second:             0
        fp-rx-bits-per-second:          0bps
          rx-drops-per-second:             0
         rx-errors-per-second:             0
        tx-packets-per-second:       218 756
           tx-bits-per-second:       2.1Gbps
     fp-tx-packets-per-second:             0
        fp-tx-bits-per-second:          0bps
          tx-drops-per-second:             0
    tx-queue-drops-per-second:         2 836
         tx-errors-per-second:             0

This is happening on both the front expansion MaxxWave 4-Port 10G module and the read expansion HotLava St. Helens 4-Port 10G card. Using FS modules.

CHR CPU ~40% @ 5Gbps
ESXi CPU Pkg ~80% @ 5Gbps

I’m scheduling a time to disable hyper-threading in the BIOS to hopefully obtain some more performance, but I’m doubtful this will be the fix.

Outside of NorthIdahoTom’s great CHR performance tips, is there anything else anyone can think of doing?

2

We were unable to get Hyper Threading disabled on Aptio BIOS v2.18.1263

The Advanced/CPU Configuration menu shows all grey options that cannot be modified. Surprisingly, Hyper Threading wasn’t listed which is strange because the i7-7700 certainly supports it and it shows “enabled” in ESXi.

3

Just a suggestion:-
try to change change the queue type to multi-queue-ethernet
and also look into how to enable that for the nic via the hypervisor / Vmware.

4

Hi guys i am having the same issue on RoS V7.12 x64bits running with 2 intel X520 da2 cards..

anyone find a fix for that?

on interface queue i have set only-hardware-queue.. can i change it to multi-queue-ethernet?


EDIT1: After changing in Queues-list / Interface Queues to multi-queue-ethernet-default.. running over 10mins now and tx queue drops 0 and RX Erros 0 also..

5

Try to using cpu affinity for dedicated CHR cpu and SR-IOV to bypass esxi kernel for using dedicated CHR NIC.

Virtualization always be a nightmare for networking if not using ideal configuration.

6

Yeah, that should be pinned somewhere as best practice.

7

if you want to run your router in VM, its highly recommended to use PCI-passthru for passing the whole NIC to the VM.

SR-IOV is great - unless you want to have quite a big traffic - since it can do only 1 or 2 rxtx-queue per port, while the X520DA-2 NIC can have like 64 or 128 queues per port.

8

Not necessarily. It ultimately depends on how well the driver is developed specifically for each solution. With a single NIC used solely by one guest OS, the difference is probably not even measurable with modern drivers. The major difference is that a NIC using PCI passthrough (VMware DirectPath) becomes locked to a single guest, whereas with SR-IOV it can be shared by multiple guests. NIC RX/TX queue sizes are directly related to the hw and device driver’s ability.

There are plenty of articles online that describe all of this in detail.

9

Thank you.
Worked for me inside a CHR.
Was wondering why the counter increased even with very low bandwidth used.
Read the wiki…a bit smarter now, but still dumb in this regard.
drops.png