Interface packet Drops on an octacore xeon with intel nics

gustkiller · Wed Oct 13, 2010 4:37 pm

Hi,

I cant understand why we're getting rx drops. its an HP ML 350 Server with only intel pcixpress gigabit ethernet nics. low traffic and we're getting rx drops

i tried to enable RPS on the interfaces that are getting drops, fixed one cpu to each card that having drops and still getting drops

any idea how to fix it?

gustkiller · Fri Oct 15, 2010 4:19 pm

just to keep you informed , i found out that the problem is caused by OSPF BUG in rc01 - rc02 prerelease. When i disabled ospf no more RX drops!

exe · Sun Feb 13, 2011 1:21 am

Hi,

Can you post PCI tab with IRQs?

What is the model of motherboard?

gustkiller · Sun Feb 13, 2011 1:38 am

it is a HP Proliant ML 350

# DEVICE VENDOR NAME IRQ
0 01:04.6 Hewlett-Packard Company Proliant iLO2 virtual UART (... 11
1 01:04.4 Hewlett-Packard Company Proliant iLO2 virtual USB co... 11
2 01:04.2 Compaq Computer Corporation Integrated Lights Out Proce... 11
3 01:04.0 Compaq Computer Corporation Integrated Lights Out Contro... 11
4 01:03.0 ATI Technologies Inc ES1000 (rev: 2) 3
5 03:00.0 Broadcom Corporation NetXtreme II BCM5708 Gigabit... 7
6 02:00.0 Broadcom EPB PCI-Express to PCI-X Bri... 0
7 14:01.1 Intel Corporation 82546GB Gigabit Ethernet Con... 7
8 14:01.0 Intel Corporation 82546GB Gigabit Ethernet Con... 4
9 13:08.0 Hewlett-Packard Company Smart Array E200i (SAS Contr... 4
10 13:04.0 Broadcom BCM5785 [HT1000] PCI/PCI-X B... 0
11 12:00.0 Broadcom EPB PCI-Express to PCI-X Bri... 0
12 0f:00.1 Intel Corporation 82571EB Gigabit Ethernet Con... 4
13 0f:00.0 Intel Corporation 82571EB Gigabit Ethernet Con... 10
14 09:00.0 Intel Corporation 82572EI Gigabit Ethernet Con... 5
15 06:00.0 Intel Corporation 82572EI Gigabit Ethernet Con... 7
16 05:01.0 Intel Corporation 6311ESB/6321ESB PCI Express ... 255
17 05:00.0 Intel Corporation 6311ESB/6321ESB PCI Express ... 255
18 04:00.3 Intel Corporation 6311ESB/6321ESB PCI Express ... 0
19 04:00.0 Intel Corporation 6311ESB/6321ESB PCI Express ... 0
20 00:1f.2 Intel Corporation 631xESB/632xESB/3100 Chipset... 5
21 00:1f.0 Intel Corporation 631xESB/632xESB/3100 Chipset... 0
22 00:1e.0 Intel Corporation 82801 PCI Bridge (rev: 217) 0
23 00:1d.7 Intel Corporation 631xESB/632xESB/3100 Chipset... 7
24 00:1d.3 Intel Corporation 631xESB/632xESB/3100 Chipset... 4
25 00:1d.2 Intel Corporation 631xESB/632xESB/3100 Chipset... 10
26 00:1d.1 Intel Corporation 631xESB/632xESB/3100 Chipset... 5
27 00:1d.0 Intel Corporation 631xESB/632xESB/3100 Chipset... 7
28 00:1c.0 Intel Corporation 631xESB/632xESB/3100 Chipset... 255
29 00:16.0 Intel Corporation 5000 Series Chipset FBD Regi... 0
30 00:15.0 Intel Corporation 5000 Series Chipset FBD Regi... 0
31 00:13.0 Intel Corporation 5000 Series Chipset Reserved... 0
32 00:11.0 Intel Corporation 5000 Series Chipset Reserved... 0
33 00:10.2 Intel Corporation 5000 Series Chipset FSB Regi... 0
34 00:10.1 Intel Corporation 5000 Series Chipset FSB Regi... 0
35 00:10.0 Intel Corporation 5000 Series Chipset FSB Regi... 0
36 00:05.0 Intel Corporation 5000 Series Chipset PCI Expr... 0
37 00:04.0 Intel Corporation 5000 Series Chipset PCI Expr... 0
38 00:03.0 Intel Corporation 5000 Series Chipset PCI Expr... 0
39 00:02.0 Intel Corporation 5000 Series Chipset PCI Expr... 0
40 00:00.0 Intel Corporation 5000Z Chipset Memory Control... 0

[admin@mk_brd01] /system resource pci>

exe · Sun Feb 13, 2011 1:52 am

Thank you, you are fast.

Maybe it is good idea to try to fix this 7 and 4 irq conflicts to avoid rx drops and other problems ?

Bishop · Sun Feb 13, 2011 8:06 pm

Thank you, you are fast.

Maybe it is good idea to try to fix this 7 and 4 irq conflicts to avoid rx drops and other problems ?

I have a similar problem http://forum.mikrotik.com/viewtopic.php?f=1&t=49153

Mon Feb 14, 2011 8:45 am

I'm sorry, but your "RX-Drop-Phobia" is just funny at best.

As far as i know there are 2 possible reasons for "RX Drops":

1) Driver drops packet cause it is convinced that packet is unnecessary/unusable (usually management frames like "pause" frames, multicast frames, frames that arrived 2nd time, frames with wrong header info etc) - basically driver is doing good thing by keeping out the bad stuff from your software

2) Drivers rx buffer is full, it can't receive more frames, cause there are not enough resources to process already received frames and empty the buffer.

Usually 2nd option are indicated by 6 7 8 digit numbers in RX drops over 24h

Chupaka · Mon Feb 14, 2011 5:32 pm

Thanks for the info. Maybe you can post 'TX Drop' reasons? on VLAN interface, I have about 100 dropped TX packets per second at ~130 Mbps TX rate. Ethernet drop/error counters are at zero

UPD: seems like problem was solved by increasing 'ethernet-default' queue length from 50 to 200 packets %)

UPD2: from 200 to 400

alexspils · Mon Feb 14, 2011 6:42 pm

I have this problem too. x86, intel 82572EI. increazing ethernet default to 400 do not help

nikhil · Sat Jul 23, 2011 2:01 am

anybody have a solution . Using 82576 too many 200000 packets in and out . Dual quad core (8cores) cant handle it.

exe · Sat Jul 23, 2011 9:44 am

You need to make irqs and cores balance and then you will not have this problem in v5

If you have more lanes (pci-e) per one ethernet that is the better and you can setup it so packet drops will never appear.

nikhil · Sat Jul 23, 2011 12:32 pm

You need to make irqs and cores balance and then you will not have this problem in v5

If you have more lanes (pci-e) per one ethernet that is the better and you can setup it so packet drops will never appear.

I have an 8 core system
4 x 82576 gige Ports - shows 7rx and 7tx each ethernet auto (like eth2-tx-rx-0 there are 7 of these , similarly there are 14 for each 82576 eth )
2x onboard broadcom 5708 (this is pci-x bridge but im not using this anymore)

How do I do it in v5 . Have v5.4 right now installed. I want it to scale to 500k or more pps how can this be done ?
Where do I go what do I do to put in more lanes
Would getting an opteron 12 + 12 core = 24 core system help me process more even though they have a lower clock speed?

exe · Sat Jul 23, 2011 2:16 pm

You should have 8 total.

Go in resources>irq and put different core for every lane (eth2-tx-rx-0 ... eth2-tx-rx-1 .. eth2-tx-rx-2 etc). Put auto for some and for some put fixed core like for eth2-tx-rx-XX (XX is number, 0-3 fixed and 4-7 put auto).

Do that for every ethernet and you should be fine.

nikhil · Sun Jul 24, 2011 10:57 pm

Done it let me watch tomm. no significant change so far. Right now the heavy packet senders are on a separate router but strangely on that I have 4 cores and it shows 8 streams per ethernet (same 82576 Intel) . It shows 4 streams of rx (eth0-rx-0 to 3) and 4 streams of tx ( eth0-tx-0 to 3) which means I have 4 ports (eth0,1,2,3) = 32 streams for all 4 eth

While on the 8 core system i have (eth0-txrx-0 to 7) 8 streams . Still total works out to 32 streams (4 eth *

. But the naming of ethX-txrx-X vs ethX-rx-X + ethX-tx-X why is there a difference ?

Does changing the no. of packets (pfifo) in ethernet-default (in queuing) also affect anything? It was 100 changed to 100000

nikhil · Mon Jul 25, 2011 1:44 pm

Hasnt made much of a difference my cpu loads are still the same.

NetworkPro · Fri Nov 11, 2011 4:24 am

Did you turn off RPS and maybe try the multi-queue ?

martini · Sun Nov 13, 2011 11:34 pm

Maybe you try to install linux ?? gentoo or ubuntu... On 8 core Xeon on mikrotik (2 Gb/s traffic, 600000 pps) without firewall and queue 15-20% cpu load. On same system and same traffic but with linux cpu load ~ 1%

With firewall (8000 rules) and queue (queue tree 8000 rules) on mikrotik cpu load - 50-70% and TX/RX drops on interface.
With queue (12000 rules) on linux cpu load ~ 2-4% and no TX/RX drops.

Question - maybe mikrotik need to change something ??

NetworkPro · Mon Nov 14, 2011 12:08 am

Martini, that kind of traffic could not be utilising the CPU only 1% ? Maybe Linux need to change something.

martini · Mon Nov 14, 2011 12:58 am

1500 users going through this router, why this traffic cant utilising CPU for 1% ??

@shaper172:~# uptime
15:15:24 up 14 days, 14:16, 1 user, load average: 0.00, 0.01, 0.05

----------------------------------

bwm-ng v0.6 (probing every 0.500s), press 'h' for help
input: /proc/net/dev type: rate
| iface Rx Tx Total
==============================================================================
eth1: 247.39 Mb/s 923.58 Mb/s 1.14 Gb/s
eth2: 923.44 Mb/s 246.43 Mb/s 1.14 Gb/s
------------------------------------------------------------------------------
total: 1.14 Gb/s 1.14 Gb/s 2.29 Gb/s

-----------------------------

top - 16:09:59 up 14 days, 15:11, 1 user, load average: 0.04, 0.03, 0.05
Tasks: 84 total, 1 running, 83 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 99.6%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4123928k total, 404336k used, 3719592k free, 90460k buffers
Swap: 4190204k total, 0k used, 4190204k free, 230644k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 3184 1800 1260 S 0 0.0 0:01.10 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd

------------------

eth1 Link encap:Ethernet HWaddr 00:1b:21:92:01:d0
inet addr:10.10.7.7 Bcast:10.10.7.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe92:1d0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:46726066747 errors:0 dropped:0 overruns:0 frame:0
TX packets:47601321122 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:31720165396132 (31.7 TB) TX bytes:50510319588460 (50.5 TB)
Memory:d8800000-d8820000

eth2 Link encap:Ethernet HWaddr 00:1b:21:92:01:d1
inet addr:10.10.8.8 Bcast:10.10.8.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe92:1d1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:47601307667 errors:0 dropped:0 overruns:0 frame:0
TX packets:46722252334 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:50527044557868 (50.5 TB) TX bytes:31642723711897 (31.6 TB)
Memory:d8820000-d8840000

Any questions ???

NetworkPro · Mon Nov 14, 2011 2:07 am

When you tested the same with RouterOS, did you turn off RPS ? If not - please can you test again?

And make a screenshot of Tools->Profile. As well as a supout to eventually send to support.

Under Linux, can we see how many CPU are the ethernet controllers using, for interrupt requests etc ?

P.S. Please can you show the exact linux version you are using, exact Kernel and exact drivers with exact NIC models. Thank you.

martini · Mon Nov 14, 2011 1:56 pm

)) yes , i turn off RPS.
NetworkPro - i know about IRQ, RPS, RFS, MSI-X ... Some years ago i create very intresting topic about MSI-X and IRQ support on intel Nics in mikrotik)), after that discussion mikrotik team included new intel driver in ROS.

Distributor ID: Ubuntu
Description: Ubuntu 11.10
Release: 11.10
Codename: oneiric

-------------------

Linux 3.0.0-12-generic-pae #20-Ubuntu SMP

-------------------

cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
0: 117 1152 271 382 862 378 329 919 IO-APIC-edge timer
1: 0 0 0 0 1 0 1 0 IO-APIC-edge i8042
7: 0 0 0 0 0 0 0 0 IO-APIC-edge parport0
8: 0 0 1 0 0 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
14: 7 6 5 7 4 6 4 6 IO-APIC-edge ata_piix
15: 0 0 0 0 0 0 0 0 IO-APIC-edge ata_piix
16: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb5
17: 414408 1098 5676 1100 1100 1100 1104 1100 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb2
18: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
19: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
65: 47826404 4 4 3 3 3 2 4 PCI-MSI-edge eth0
66: 0 0 2 0 1 0 0 0 PCI-MSI-edge eth1
67: 2525800625 1 2 1 126106 2 110324 2 PCI-MSI-edge eth1-TxRx-0
68: 2513065077 2 7 2 4382 1 46784 4 PCI-MSI-edge eth1-TxRx-1
69: 140349 2 2498340200 6 2 3 251 3 PCI-MSI-edge eth1-TxRx-2
70: 132417 2 3 3 31 11 2485760026 4 PCI-MSI-edge eth1-TxRx-3
71: 141761 11 2511046380 4 309 3 2 61 PCI-MSI-edge eth1-TxRx-4
72: 259489 124 419 127 2480696332 123 123 121 PCI-MSI-edge eth1-TxRx-5
73: 2500197820 4 0 2 3 5 110192 8 PCI-MSI-edge eth1-TxRx-6
74: 131975 3 2 41 6 2 2480229584 8 PCI-MSI-edge eth1-TxRx-7
75: 0 0 6 2 2 2 0 6 PCI-MSI-edge eth2
76: 2540912381 3 0 12 206326 0 4679 11 PCI-MSI-edge eth2-TxRx-0
77: 0 2 3 2519446249 166 2 5 5 PCI-MSI-edge eth2-TxRx-1
78: 0 2 6 3 4 2507669239 5 1 PCI-MSI-edge eth2-TxRx-2
79: 0 3 5 1 3 2494796162 74 4 PCI-MSI-edge eth2-TxRx-3
80: 0 2518890968 3 4 2 5 4 2 PCI-MSI-edge eth2-TxRx-4
81: 0 6 3 1 5 6 2 2488018578 PCI-MSI-edge eth2-TxRx-5
82: 2509797338 5 1 7 5 3 132741 6 PCI-MSI-edge eth2-TxRx-6
83: 0 2 3 2488619734 3 2 4 6 PCI-MSI-edge eth2-TxRx-7
84: 1 1 1 0 1 0 0 1 PCI-MSI-edge ioat-msi
85: 1 0 0 1 0 0 0 0 PCI-MSI-edge radeon
NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 7225524 1691936 7499082 2003085 6842025 1535041 1903862 3782494 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 0 0 0 0 IRQ work interrupts
RES: 251012 8405 91379 83820 41566 32190 14002 15273 Rescheduling interrupts
CAL: 989 2740 367 559 99551 4830 356 455 Function call interrupts
TLB: 774 284 2405 2921 595 217 2200 3056 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 4369 4369 4369 4369 4369 4369 4369 4369 Machine check polls

----------------------

Cannot create screenshot of Ros and supout, because after mikrotik i install Linux on that Router and it serve many users.
I can say that in tool/Profile i see only ethernet load about 5-10%, but in resourses/cpu i see 100% interrupt usage (every 2 TX/RX vector from ethernet was on its own core)

changeip · Mon Nov 14, 2011 6:14 pm

martini, keep up the good work and troubleshooting on this issue. I see the same problems even on dual core boxes - lots of RX drops even though the hardware isnt loaded. Im wondering if its just poor kernel parameter tuning in ROS or something.

martini · Mon Nov 14, 2011 6:27 pm

martini, keep up the good work and troubleshooting on this issue. I see the same problems even on dual core boxes - lots of RX drops even though the hardware isnt loaded. Im wondering if its just poor kernel parameter tuning in ROS or something.

on some routers (X86 with mikrotik) i change IRQ mapping (try to play whith it), this decrease RX drops or stop count RX drop.

NetworkPro · Tue Dec 25, 2012 11:33 pm

This stuff here is important in x86 context. Well done martini and macgaiver.

Fresh new questions kind of on the topic (also sent to support and waiting optimistic answer):

- What Kernel Options / tweaks of this sort (https://help.ubuntu.com/community/BootO ... el_Options ACPI, APIC, IRQ Balance) does RouterOS have in it? Are they pre-tweaked for each platform / hardware combination? Or if auto-chosen on boot-up - how are they chosen?

- Will RouterOS make use of the most hardware capabilities available in the Chipset+CPU even if the x86 BIOS does does not enable some/has problematic implementation?

- What clock source does RouterOS use even when the BIOS does not support enabling the HPET functionality of the x86 platform?

- Does the clock source affect how much pps the router will pass?

- Does x86 platform somehow limit the performance of HTB and PCQ in RouterOS - what are the relation between clock source/megahertz/memory bandwidth and how much pps in sub-queues will HTB and PCQ pass - without delay and drop? How many sub-queues per CPU cycle are processed at the same time?

- What parallelism takes place, in order to fully utilize the available x86 speed in terms of CPU<->Memory/CPU<->BUS bandwidth so that pps performance would be competitive and there would not be out-of order packets?

Some answers are found in Linux Kernel source code but buried too deep in there

Waiting for optimistic answers. If the answers are not optimistic then I am happy I pointed where improvements can be made.

Thank you.

Wed Jan 02, 2013 2:19 pm

About RX drops with some additional testing i got to conclusion that

Rx Drops shows all the packets that Router can't handle:

1) if you disable ipv6 package all IPv6 packets directed to the router (not bridged trough) will be captured by Rx drops
2) if you enable sniffer - there will be no Rx drops at all, cause all packets now will be handled by sniffer
3) if you receive a packet (not bridged trough) with VLAN tag that you do not know anything about those packets also will be captured by Rx drops

I haven't tested multicast, but i imagine picture is the same.

So bottom line - there are no need to worry about Rx Drops - it is just another way to see how much useless traffic comes to your Router

Chupaka · Sun Jan 06, 2013 12:33 am

3) if you receive a packet (not bridged trough) with VLAN tag that you do not know anything about those packets also will be captured by Rx drops

it would be nice to classify those drops, like in big switches

'VLAN Drop', 'CRC Drop', etc... MikroTik? =)

osvaldotcf · Sat Mar 09, 2013 3:33 am

I have the same, or not, problem with RB1200 RouterOS v6rc11.

On hotspot all clients with 0 tx rate.

omidkosari · Thu Dec 07, 2017 2:53 pm

About RX drops with some additional testing i got to conclusion that

Rx Drops shows all the packets that Router can't handle:

1) if you disable ipv6 package all IPv6 packets directed to the router (not bridged trough) will be captured by Rx drops
2) if you enable sniffer - there will be no Rx drops at all, cause all packets now will be handled by sniffer
3) if you receive a packet (not bridged trough) with VLAN tag that you do not know anything about those packets also will be captured by Rx drops

I haven't tested multicast, but i imagine picture is the same.

So bottom line - there are no need to worry about Rx Drops - it is just another way to see how much useless traffic comes to your Router

What about Tx Drops ?

Fri Dec 08, 2017 8:08 am

What about Tx Drops ?

OMG that was 5 years old post

TX drop is much more rare, imagine you have 1Gbps and 100Mbps interfaces on your router and you are trying to receive 200Mbps on Gbps interface and send it out on 100Mbps interface, 1Gpbs will have no problem receiving 200Mbps, and at that point in time it is unknown where this traffic will be directed. As outgoing interface can handle only 100Mbps, rest will be dropped as TX drops.

This is much more common to Wireless interface that can have transfer rate jumping.

omidkosari · Fri Dec 08, 2017 12:38 pm

What about Tx Drops ?
OMG that was 5 years old post

Drops are zombies . They can re-alive after 100 years . 5 years is not so much

TX drop is much more rare, imagine you have 1Gbps and 100Mbps interfaces on your router and you are trying to receive 200Mbps on Gbps interface and send it out on 100Mbps interface, 1Gpbs will have no problem receiving 200Mbps, and at that point in time it is unknown where this traffic will be directed. As outgoing interface can handle only 100Mbps, rest will be dropped as TX drops.

This is much more common to Wireless interface that can have transfer rate jumping.

I think in my situation that is not the exact reason.
Actually i see Tx Drops on Vlan interfaces only (you can see in attached image that the ether1 does not have any drops). The vlans may be QinQ . But even those with simple one vlan tag has Tx Drops also .

Capture1.JPG

Capture2.JPG

I shared related info and hope you help me to solve it .

/interface ethernet
set [ find default-name=ether1 ] mtu=1512
set [ find default-name=ether2 ] mtu=1512
set [ find default-name=ether3 ] mtu=1512
set [ find default-name=ether4 ] mtu=1512


/interface vlan
add interface=vlan1705 name=vlan24 vlan-id=24
add interface=ether1 name=vlan100 vlan-id=100
add interface=ether1 name=vlan102 vlan-id=102
add interface=ether1 name=vlan103 vlan-id=103
add interface=ether1 name=vlan104 vlan-id=104
add interface=ether1 name=vlan105 vlan-id=105
add interface=ether1 name=vlan106 vlan-id=106
add interface=ether1 name=vlan108 vlan-id=108
add interface=ether1 name=vlan109 vlan-id=109
add interface=ether1 name=vlan119 vlan-id=119
add interface=vlan1705-4 name=vlan300 vlan-id=300
add interface=vlan3924 name=vlan1705 vlan-id=1705
add interface=vlan3924-3 name=vlan1705-3 vlan-id=1705
add interface=vlan3924-4 name=vlan1705-4 vlan-id=1705
add interface=ether1 name=vlan3518 vlan-id=3518
add interface=ether1 name=vlan3924 vlan-id=3924
add interface=ether3 name=vlan3924-3 vlan-id=3924
add interface=ether4 name=vlan3924-4 vlan-id=3924

/queue interface pr where interface~"^ether" 
 # INTERFACE                                 QUEUE                                    ACTIVE-QUEUE                                
 0 ether1                                    only-hardware-queue                      only-hardware-queue                         
 1 ether2                                    only-hardware-queue                      only-hardware-queue                         
 2 ether3                                    only-hardware-queue                      only-hardware-queue                         
 3 ether4                                    only-hardware-queue                      only-hardware-queue              
   
/driver pr
 # DRIVER                                                                                                                         
 0 USB Keyboard/Mouse                                                                                                             
 1 Broadcom NX2 Ethernet

/sys resource pci pr
 # DEVICE   VENDOR                                                 NAME                                                        IRQ
 0 3f:06.3  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
 1 3f:06.2  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
 2 3f:06.1  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
 3 3f:06.0  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
 4 3f:05.3  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
 5 3f:05.2  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
 6 3f:05.1  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
 7 3f:05.0  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
 8 3f:04.3  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
 9 3f:04.2  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
10 3f:04.1  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
11 3f:04.0  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
12 3f:03.4  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Te...          0
13 3f:03.2  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller RA...          0
14 3f:03.1  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ta...          0
15 3f:03.0  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller (r...          0
16 3f:02.5  Intel Corporation                                      Xeon 5500/Core i7 QPI Physical 1 (rev: 5)                     0
17 3f:02.4  Intel Corporation                                      Xeon 5500/Core i7 QPI Link 1 (rev: 5)                         0
18 3f:02.1  Intel Corporation                                      Xeon 5500/Core i7 QPI Physical 0 (rev: 5)                     0
19 3f:02.0  Intel Corporation                                      Xeon 5500/Core i7 QPI Link 0 (rev: 5)                         0
20 3f:00.1  Intel Corporation                                      Xeon 5500/Core i7 QuickPath Architecture System A...          0
21 3f:00.0  Intel Corporation                                      Xeon 5500/Core i7 QuickPath Architecture Generic ...          0
22 3e:06.3  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
23 3e:06.2  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
24 3e:06.1  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
25 3e:06.0  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
26 3e:05.3  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
27 3e:05.2  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
28 3e:05.1  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
29 3e:05.0  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
30 3e:04.3  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
31 3e:04.2  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
32 3e:04.1  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
33 3e:04.0  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ch...          0
34 3e:03.4  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Te...          0
35 3e:03.2  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller RA...          0
36 3e:03.1  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller Ta...          0
37 3e:03.0  Intel Corporation                                      Xeon 5500/Core i7 Integrated Memory Controller (r...          0
38 3e:02.5  Intel Corporation                                      Xeon 5500/Core i7 QPI Physical 1 (rev: 5)                     0
39 3e:02.4  Intel Corporation                                      Xeon 5500/Core i7 QPI Link 1 (rev: 5)                         0
40 3e:02.1  Intel Corporation                                      Xeon 5500/Core i7 QPI Physical 0 (rev: 5)                     0
41 3e:02.0  Intel Corporation                                      Xeon 5500/Core i7 QPI Link 0 (rev: 5)                         0
42 3e:00.1  Intel Corporation                                      Xeon 5500/Core i7 QuickPath Architecture System A...          0
43 3e:00.0  Intel Corporation                                      Xeon 5500/Core i7 QuickPath Architecture Generic ...          0
44 04:00.0  Hewlett-Packard Company                                Smart Array G6 controllers (rev: 1)                          11
45 03:00.1  Broadcom Corporation                                   NetXtreme II BCM5709 Gigabit Ethernet (rev: 32)              10
46 03:00.0  Broadcom Corporation                                   NetXtreme II BCM5709 Gigabit Ethernet (rev: 32)              11
47 02:00.1  Broadcom Corporation                                   NetXtreme II BCM5709 Gigabit Ethernet (rev: 32)               7
48 02:00.0  Broadcom Corporation                                   NetXtreme II BCM5709 Gigabit Ethernet (rev: 32)              11
49 01:04.6  Hewlett-Packard Company                                Proliant iLO2 virtual UART (rev: 0)                          10
50 01:04.4  Hewlett-Packard Company                                Proliant iLO2 virtual USB controller (rev: 0)                 5
51 01:04.2  Compaq Computer Corporation                            Integrated Lights Out  Processor (rev: 3)                     5
52 01:04.0  Compaq Computer Corporation                            Integrated Lights Out Controller (rev: 3)                    10
53 01:03.0  ATI Technologies Inc                                   ES1000 (rev: 2)                                               4
54 00:1f.2  Intel Corporation                                      82801JI (ICH10 Family) 4 port SATA IDE Controller...          7
55 00:1f.0  Intel Corporation                                      82801JIB (ICH10) LPC Interface Controller (rev: 0)            0
56 00:1e.0  Intel Corporation                                      82801 PCI Bridge (rev: 144)                                   0
57 00:1d.7  Intel Corporation                                      82801JI (ICH10 Family) USB2 EHCI Controller #1 (r...          3
58 00:1d.3  Intel Corporation                                      82801JI (ICH10 Family) USB UHCI Controller #6 (re...          4
59 00:1d.2  Intel Corporation                                      82801JI (ICH10 Family) USB UHCI Controller #3 (re...          5
60 00:1d.1  Intel Corporation                                      82801JI (ICH10 Family) USB UHCI Controller #2 (re...          4
61 00:1d.0  Intel Corporation                                      82801JI (ICH10 Family) USB UHCI Controller #1 (re...          3
62 00:1c.2  Intel Corporation                                      82801JI (ICH10 Family) PCI Express Root Port 3 (r...        255
63 00:1c.0  Intel Corporation                                      82801JI (ICH10 Family) PCI Express Root Port 1 (r...        255
64 00:14.2  Intel Corporation                                      5520/5500/X58 I/O Hub Control Status and RAS Regi...          0
65 00:14.1  Intel Corporation                                      5520/5500/X58 I/O Hub GPIO and Scratch Pad Regist...          0
66 00:14.0  Intel Corporation                                      5520/5500/X58 I/O Hub System Management Registers...          0
67 00:0e.4  Intel Corporation                                      unknown device (rev: 19)                                      0
68 00:0e.3  Intel Corporation                                      unknown device (rev: 19)                                      0
69 00:0e.2  Intel Corporation                                      unknown device (rev: 19)                                      0
70 00:0e.1  Intel Corporation                                      unknown device (rev: 19)                                      0
71 00:0e.0  Intel Corporation                                      unknown device (rev: 19)                                      0
72 00:0d.6  Intel Corporation                                      unknown device (rev: 19)                                      0
73 00:0d.5  Intel Corporation                                      5520/5500 Physical Layer Port 1 (rev: 19)                     0
74 00:0d.4  Intel Corporation                                      5520/5500/X58 Physical Layer Port 0 (rev: 19)                 0
75 00:0d.3  Intel Corporation                                      unknown device (rev: 19)                                      0
76 00:0d.2  Intel Corporation                                      unknown device (rev: 19)                                      0
77 00:0d.1  Intel Corporation                                      unknown device (rev: 19)                                      0
78 00:0d.0  Intel Corporation                                      unknown device (rev: 19)                                      0
79 00:0a.0  Intel Corporation                                      5520/5500/X58 I/O Hub PCI Express Root Port 10 (r...          0
80 00:09.0  Intel Corporation                                      5520/5500/X58 I/O Hub PCI Express Root Port 9 (re...          0
81 00:08.0  Intel Corporation                                      5520/5500/X58 I/O Hub PCI Express Root Port 8 (re...          0
82 00:07.0  Intel Corporation                                      5520/5500/X58 I/O Hub PCI Express Root Port 7 (re...          0
83 00:06.0  Intel Corporation                                      5520/X58 I/O Hub PCI Express Root Port 6 (rev: 19)            0
84 00:05.0  Intel Corporation                                      5520/X58 I/O Hub PCI Express Root Port 5 (rev: 19)            0
85 00:04.0  Intel Corporation                                      5520/X58 I/O Hub PCI Express Root Port 4 (rev: 19)            0
86 00:03.0  Intel Corporation                                      5520/5500/X58 I/O Hub PCI Express Root Port 3 (re...          0
87 00:02.0  Intel Corporation                                      5520/5500/X58 I/O Hub PCI Express Root Port 2 (re...          0
88 00:01.0  Intel Corporation                                      5520/5500/X58 I/O Hub PCI Express Root Port 1 (re...          0
89 00:00.0  Intel Corporation                                      5520 I/O Hub to ESI Port (rev: 19)                            0



/sys resource irq pr
Flags: ro - read-only 
 #    IRQ USERS                                                                                       CPU ACTIVE-CPU         COUNT
 0      1 i8042                                                                                      auto          0             3
 1      9 acpi                                                                                       auto          1             0
 2     12 i8042                                                                                      auto          2             3
 3     14 ide0                                                                                       auto          3             0
 4     15 ide1                                                                                       auto          4             0
 5     17 ata_piix                                                                                   auto          5             7
 6     20 usb2                                                                                       auto          6             0
          usb1                                                                                
 7     22 usb6                                                                                       auto          7            36
          usb4                                                                                
 8     23 usb5                                                                                       auto          8             0
          usb3                                                                                
 9     66 hpsa0                                                                                      auto          9        94 698
10     70 eth0-0                                                                                     auto         10 4 000 526 560
11     71 eth0-1                                                                                     auto         11 1 032 486 196
12     72 eth0-2                                                                                     auto         12    90 196 898
13     73 eth0-3                                                                                     auto         13   517 809 070
14     74 eth0-4                                                                                     auto         14 1 960 955 982
15     75 eth0-5                                                                                     auto         15    91 867 428
16     76 eth0-6                                                                                     auto          0 3 076 872 916
17     77 eth0-7                                                                                     auto          1   318 379 144
18     78 eth1-0                                                                                     auto          2   976 625 561
19     79 eth1-1                                                                                     auto          3    15 096 680
20     80 eth1-2                                                                                     auto          4     1 097 316
21     81 eth1-3                                                                                     auto          5    29 147 707
22     82 eth1-4                                                                                     auto          6     3 136 236
23     83 eth1-5                                                                                     auto          7       844 382
24     84 eth1-6                                                                                     auto          8   468 421 761
25     85 eth1-7                                                                                     auto          9    39 497 220
26     86 eth2-0                                                                                     auto         10   611 631 342
27     87 eth2-1                                                                                     auto         11    13 727 240
28     88 eth2-2                                                                                     auto         12       935 089
29     89 eth2-3                                                                                     auto         13     1 421 833
30     90 eth2-4                                                                                     auto         14   315 378 643
31     91 eth2-5                                                                                     auto         15       817 139
32     92 eth2-6                                                                                     auto          0       634 491
33     93 eth2-7                                                                                     auto          1    32 778 547
34     94 eth3-0                                                                                     auto          2 2 279 155 393
35     95 eth3-1                                                                                     auto          3    12 097 538
36     96 eth3-2                                                                                     auto          4     1 299 703
37     97 eth3-3                                                                                     auto          5     1 372 216
38     98 eth3-4                                                                                     auto          6 2 292 517 611
39     99 eth3-5                                                                                     auto          7       833 238
40    100 eth3-6                                                                                     auto          8       610 256
41    101 eth3-7                                                                                     auto          9    28 320 528

NetworkPro · Sun Dec 10, 2017 11:55 am

Hi

- What is the queue of the ether1 interface ? If you increase it, do you have less TX Drops ?

Change from only-hardware-queue to some Queue type multi-queue-ethernet-default and increase from 50 to 1000 only for test ? (could add jitter for those packets that will not be dropped anymore which is not a good thing for most protocols)

- Does the CPU handle the PPS over the ether1 interface correctly, do you see the load distributed over cores if you check with Tools-> Profile?

- What version is the RouterOS ? MTUs are suspect.

- Replace the server with newer hardware platform at least for test, could indicate problem with how Intel/HP do things in this machine.

- Replace the NIC. Intel Driver is a suspect.

omidkosari · Tue Dec 12, 2017 3:44 pm

Change from only-hardware-queue to some Queue type multi-queue-ethernet-default

Thanks ( couldn't find karma+ )
It seems solved . I will inform you if happens again .

PortalNET · Mon Dec 11, 2023 1:09 am

Hi

- What is the queue of the ether1 interface ? If you increase it, do you have less TX Drops ?

Change from only-hardware-queue to some Queue type multi-queue-ethernet-default and increase from 50 to 1000 only for test ? (could add jitter for those packets that will not be dropped anymore which is not a good thing for most protocols)

- Does the CPU handle the PPS over the ether1 interface correctly, do you see the load distributed over cores if you check with Tools-> Profile?

- What version is the RouterOS ? MTUs are suspect.

- Replace the server with newer hardware platform at least for test, could indicate problem with how Intel/HP do things in this machine.

- Replace the NIC. Intel Driver is a suspect.

Hi guys sorry to bump on old thread..

We have the same issue running a Dell R620 (2 cpus E5-2697v2 + 128GB RAM DDR3) with 3 Intel X520DA2 one is dell OEM 2SFP+ WITH 2RJ45 + 2 slots pci-e 3.0 with Intel X520 da2 and 1 slot with Mellanox MCX355ECBT 40gbps dual..

RouterOS v7.12 stable with L6 license.

We decided to give it a test on a old R620 sell server.. because it had enough pci slots so we could make a BRAS server pppoe and would be able to load the nearly 1.6k pppoe sessiones we have split in 3 ccr1036 models into one box only.. and indeed the CPU usage ate rates over 3.5Gbps download speeds.. are only 2% to 3% on the R620 server..
but after few hours sometimes couple of days the server crashes and need to reboot, because the intel ports stop responding.. we have analyzed and at first we had RX-DRPOS counter increasing.. then we have changed the interface queues from hardware-only to multi-queue-ethernet-default... and for a few hours the RX-drops stopped.. but after we started with RX-ERRORS.. increaseing slowly and when they reach around 25k or more drops...ports stops responding...

so far have not managed to fix the issue.. had to go back to the CCRs and split the BRAS servers again..

Anyone has ideas?.. the Mellanox cards do not get errors at all.. we have setup a lab with 2 R620.. both with Mellanox cards MCX355ECBT with firmware upgraded and switched to ethernet mode.. we could get troughput from one server to the other running r0S v7 of 64gbps wich is the max supporte by the pci-e 3.0 8x pci cards and slots.. with no errors.. we only get the error with the intel x520 models attached below the pics of the devices.

X520 issue.png

Dell X520.png

Interface packet Drops on an octacore xeon with intel nics

Interface packet Drops on an octacore xeon with intel nics

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel ni

Re: Interface packet Drops on an octacore xeon with intel nics

Re: Interface packet Drops on an octacore xeon with intel nics

Re: Interface packet Drops on an octacore xeon with intel nics

Who is online