I’m seeing EXTREMELY poor iperf3 results when doing testing between two linux servers that have Mellanox ConnectX-3 dual port NICs, each with one port connected to my CRS354. So poor that I know I might either have a hardware issue on one/both of my ports or something is wrong with my switch config that I just can’t find. I have also attached my entire CRS354 config. It’s pretty basic at this moment as I just got it and wanted to get all of this straightened out before I start doing more advanced/permanent config.
NOTE I have tested the below with three different sets/brands of QSFP+ DAC cables.
Below are iperf3 results (both single and 2 parallel connections) between two Linux boxes each of which are connected to the CRS354 with DAC QSFP+ cables. As you can see the results are absolutely abysmal.
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 4.18 MBytes 35.1 Mbits/sec 153 9.90 KBytes
[ 5] 1.00-2.00 sec 4.59 MBytes 38.5 Mbits/sec 153 1.41 KBytes
[ 5] 2.00-3.00 sec 10.5 MBytes 87.9 Mbits/sec 322 2.83 KBytes
[ 5] 3.00-4.00 sec 4.23 MBytes 35.4 Mbits/sec 143 1.41 KBytes
[ 5] 4.00-5.00 sec 6.68 MBytes 56.0 Mbits/sec 227 1.41 KBytes
[ 5] 5.00-6.00 sec 5.22 MBytes 43.8 Mbits/sec 187 1.41 KBytes
[ 5] 6.00-7.00 sec 255 KBytes 2.09 Mbits/sec 14 2.83 KBytes
[ 5] 7.00-8.00 sec 5.29 MBytes 44.4 Mbits/sec 217 1.41 KBytes
[ 5] 8.00-9.00 sec 6.46 MBytes 54.2 Mbits/sec 203 1.41 KBytes
[ 5] 9.00-10.00 sec 4.36 MBytes 36.6 Mbits/sec 147 1.41 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 51.7 MBytes 43.4 Mbits/sec 1766 sender
[ 5] 0.00-10.00 sec 51.5 MBytes 43.2 Mbits/sec receiver
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 6.57 MBytes 55.1 Mbits/sec 229 2.83 KBytes
[ 7] 0.00-1.00 sec 10.0 MBytes 84.1 Mbits/sec 319 1.41 KBytes
[SUM] 0.00-1.00 sec 16.6 MBytes 139 Mbits/sec 548
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 4.10 MBytes 34.4 Mbits/sec 143 1.41 KBytes
[ 7] 1.00-2.00 sec 4.16 MBytes 34.9 Mbits/sec 134 9.90 KBytes
[SUM] 1.00-2.00 sec 8.26 MBytes 69.3 Mbits/sec 277
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 4.60 MBytes 38.6 Mbits/sec 140 2.83 KBytes
[ 7] 2.00-3.00 sec 2.92 MBytes 24.5 Mbits/sec 114 2.83 KBytes
[SUM] 2.00-3.00 sec 7.52 MBytes 63.1 Mbits/sec 254
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 764 KBytes 6.26 Mbits/sec 30 2.83 KBytes
[ 7] 3.00-4.00 sec 3.60 MBytes 30.2 Mbits/sec 120 5.66 KBytes
[SUM] 3.00-4.00 sec 4.35 MBytes 36.5 Mbits/sec 150
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 7.21 MBytes 60.5 Mbits/sec 256 17.0 KBytes
[ 7] 4.00-5.00 sec 7.77 MBytes 65.2 Mbits/sec 241 18.4 KBytes
[SUM] 4.00-5.00 sec 15.0 MBytes 126 Mbits/sec 497
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 11.4 MBytes 95.9 Mbits/sec 381 5.66 KBytes
[ 7] 5.00-6.00 sec 2.42 MBytes 20.3 Mbits/sec 79 1.41 KBytes
[SUM] 5.00-6.00 sec 13.9 MBytes 116 Mbits/sec 460
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 2.36 MBytes 19.8 Mbits/sec 95 1.41 KBytes
[ 7] 6.00-7.00 sec 3.48 MBytes 29.2 Mbits/sec 103 7.07 KBytes
[SUM] 6.00-7.00 sec 5.84 MBytes 49.0 Mbits/sec 198
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 2.11 MBytes 17.7 Mbits/sec 60 1.41 KBytes
[ 7] 7.00-8.00 sec 1.30 MBytes 10.9 Mbits/sec 42 9.90 KBytes
[SUM] 7.00-8.00 sec 3.42 MBytes 28.7 Mbits/sec 102
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 5.72 MBytes 48.0 Mbits/sec 193 2.83 KBytes
[ 7] 8.00-9.00 sec 255 KBytes 2.09 Mbits/sec 6 5.66 KBytes
[SUM] 8.00-9.00 sec 5.97 MBytes 50.1 Mbits/sec 199
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 6.03 MBytes 50.6 Mbits/sec 218 2.83 KBytes
[ 7] 9.00-10.00 sec 3.91 MBytes 32.8 Mbits/sec 132 1.41 KBytes
[SUM] 9.00-10.00 sec 9.94 MBytes 83.4 Mbits/sec 350
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 50.9 MBytes 42.7 Mbits/sec 1745 sender
[ 5] 0.00-10.00 sec 50.6 MBytes 42.4 Mbits/sec receiver
[ 7] 0.00-10.00 sec 39.8 MBytes 33.4 Mbits/sec 1290 sender
[ 7] 0.00-10.00 sec 39.6 MBytes 33.2 Mbits/sec receiver
[SUM] 0.00-10.00 sec 90.7 MBytes 76.1 Mbits/sec 3035 sender
[SUM] 0.00-10.00 sec 90.2 MBytes 75.6 Mbits/sec receiver
Now below, you will see two different iperf3 results. The first is a direct connection between NIC1 on Server1 and NIC1 on Server 2. The second is a direction connection between NIC2 on Server1 and NIC2 on Server2. I thought it important to test direct connections with both cables that I’m connecting to the CRS354 to eliminate them as the issue.
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.74 GBytes 32.1 Gbits/sec 0 1.46 MBytes
[ 5] 1.00-2.00 sec 3.72 GBytes 32.0 Gbits/sec 0 1.55 MBytes
[ 5] 2.00-3.00 sec 3.73 GBytes 32.1 Gbits/sec 0 1.55 MBytes
[ 5] 3.00-4.00 sec 3.73 GBytes 32.0 Gbits/sec 190 1.30 MBytes
[ 5] 4.00-5.00 sec 3.72 GBytes 32.0 Gbits/sec 0 1.38 MBytes
[ 5] 5.00-6.00 sec 3.73 GBytes 32.0 Gbits/sec 0 1.38 MBytes
[ 5] 6.00-7.00 sec 3.72 GBytes 32.0 Gbits/sec 0 1.55 MBytes
[ 5] 7.00-8.00 sec 3.73 GBytes 32.1 Gbits/sec 0 1.55 MBytes
[ 5] 8.00-9.00 sec 3.49 GBytes 30.0 Gbits/sec 0 1.75 MBytes
[ 5] 9.00-10.00 sec 3.10 GBytes 26.6 Gbits/sec 0 1.75 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 36.4 GBytes 31.3 Gbits/sec 190 sender
[ 5] 0.00-10.00 sec 36.4 GBytes 31.3 Gbits/sec receiver
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.25 GBytes 19.3 Gbits/sec 0 1.31 MBytes
[ 7] 0.00-1.00 sec 2.12 GBytes 18.2 Gbits/sec 0 1.36 MBytes
[SUM] 0.00-1.00 sec 4.37 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.31 MBytes
[ 7] 1.00-2.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.36 MBytes
[SUM] 1.00-2.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.38 MBytes
[ 7] 2.00-3.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 2.00-3.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.45 MBytes
[ 7] 3.00-4.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 3.00-4.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.45 MBytes
[ 7] 4.00-5.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 4.00-5.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.45 MBytes
[ 7] 5.00-6.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 5.00-6.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.45 MBytes
[ 7] 6.00-7.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 6.00-7.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.45 MBytes
[ 7] 7.00-8.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 7.00-8.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 2.18 GBytes 18.8 Gbits/sec 0 1.45 MBytes
[ 7] 8.00-9.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 8.00-9.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.45 MBytes
[ 7] 9.00-10.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.54 MBytes
[SUM] 9.00-10.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 21.9 GBytes 18.8 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 21.9 GBytes 18.8 Gbits/sec receiver
[ 7] 0.00-10.00 sec 21.8 GBytes 18.7 Gbits/sec 0 sender
[ 7] 0.00-10.00 sec 21.7 GBytes 18.7 Gbits/sec receiver
[SUM] 0.00-10.00 sec 43.6 GBytes 37.5 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 43.6 GBytes 37.5 Gbits/sec receiver
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.78 GBytes 32.5 Gbits/sec 0 1.56 MBytes
[ 5] 1.00-2.00 sec 3.77 GBytes 32.4 Gbits/sec 0 1.56 MBytes
[ 5] 2.00-3.00 sec 3.63 GBytes 31.2 Gbits/sec 0 1.56 MBytes
[ 5] 3.00-4.00 sec 3.40 GBytes 29.2 Gbits/sec 0 1.56 MBytes
[ 5] 4.00-5.00 sec 3.37 GBytes 28.9 Gbits/sec 0 1.56 MBytes
[ 5] 5.00-6.00 sec 3.41 GBytes 29.3 Gbits/sec 0 1.56 MBytes
[ 5] 6.00-7.00 sec 3.44 GBytes 29.6 Gbits/sec 0 1.56 MBytes
[ 5] 7.00-8.00 sec 3.43 GBytes 29.5 Gbits/sec 0 1.56 MBytes
[ 5] 8.00-9.00 sec 3.43 GBytes 29.4 Gbits/sec 0 1.56 MBytes
[ 5] 9.00-10.00 sec 3.41 GBytes 29.3 Gbits/sec 0 1.56 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 35.1 GBytes 30.1 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 35.1 GBytes 30.1 Gbits/sec receiver
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.17 GBytes 18.6 Gbits/sec 0 1.33 MBytes
[ 7] 0.00-1.00 sec 2.17 GBytes 18.6 Gbits/sec 0 1.38 MBytes
[SUM] 0.00-1.00 sec 4.34 GBytes 37.3 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.33 MBytes
[ 7] 1.00-2.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.38 MBytes
[SUM] 1.00-2.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.33 MBytes
[ 7] 2.00-3.00 sec 2.18 GBytes 18.8 Gbits/sec 0 1.38 MBytes
[SUM] 2.00-3.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.33 MBytes
[ 7] 3.00-4.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.38 MBytes
[SUM] 3.00-4.00 sec 4.37 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.33 MBytes
[ 7] 4.00-5.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.38 MBytes
[SUM] 4.00-5.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.33 MBytes
[ 7] 5.00-6.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.38 MBytes
[SUM] 5.00-6.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.48 MBytes
[ 7] 6.00-7.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 6.00-7.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.48 MBytes
[ 7] 7.00-8.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 7.00-8.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.48 MBytes
[ 7] 8.00-9.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 8.00-9.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.48 MBytes
[ 7] 9.00-10.00 sec 2.18 GBytes 18.7 Gbits/sec 0 1.46 MBytes
[SUM] 9.00-10.00 sec 4.36 GBytes 37.5 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 21.8 GBytes 18.7 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 21.8 GBytes 18.7 Gbits/sec receiver
[ 7] 0.00-10.00 sec 21.8 GBytes 18.7 Gbits/sec 0 sender
[ 7] 0.00-10.00 sec 21.8 GBytes 18.7 Gbits/sec receiver
[SUM] 0.00-10.00 sec 43.6 GBytes 37.5 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 43.6 GBytes 37.5 Gbits/sec receiver
I have also confirmed HW = yes is on for all the qsfpplus ports as you can see below:
# INTERFACE BRIDGE HW PVID PRIORITY PATH-COST INTERNAL-PATH-COST HORIZON
53 H qsfpplus1-1 bridge yes 110 0x80 10 10 none
54 I H qsfpplus1-2 bridge yes 110 0x80 10 10 none
55 I H qsfpplus1-3 bridge yes 110 0x80 10 10 none
56 I H qsfpplus1-4 bridge yes 110 0x80 10 10 none
57 H qsfpplus2-1 bridge yes 110 0x80 10 10 none
58 I H qsfpplus2-2 bridge yes 110 0x80 10 10 none
59 I H qsfpplus2-3 bridge yes 110 0x80 10 10 none
60 I H qsfpplus2-4 bridge yes 110 0x80 10 10 none
CRS354.txt (4.66 KB)