Hello, im trying to get 10Gbe network to work in my lab, but i have some problems. !0Gb connection drops to a 10-100Mb/s speeds when tested with iperf3.
My setups looks like this:
- 3x dell R620 with NC523 NICs, connected with fiber [ FTLX8571D3BCL-GM SFP+ modules] - Proxmox Cluster
- Dell R510 with Dell Broadcom 10GbE NIC 57711 NIC, connected with fiber [ FTLX8571D3BCL-GM SFP+ module] - Freenas box
Im using 10Gb network for iscsi/nfs, only those four servers are connected to network. Initially im getting stable 9Gb/s on all nodes as in this log:
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.240, port 48040
[ 5] local 192.168.10.5 port 5201 connected to 192.168.10.240 port 48042
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.05 GBytes 9.02 Gbits/sec
[ 5] 1.00-2.00 sec 1.06 GBytes 9.14 Gbits/sec
[ 5] 2.00-3.00 sec 1.07 GBytes 9.15 Gbits/sec
[ 5] 3.00-4.00 sec 1.06 GBytes 9.09 Gbits/sec
[ 5] 4.00-5.00 sec 1.06 GBytes 9.14 Gbits/sec
But after 10-20 minutes of uptime im getting 10-100Mb/s, but only from some nodes, for example node 1 to freenas is low, but node 2 to freenas is fast, and also node 1 to node 2 is fast. Sometimes connection is slow in one direction.
[ 5] 0.00-10.00 sec 10.6 GBytes 9.13 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.242, port 37644
[ 5] local 192.168.10.5 port 5201 connected to 192.168.10.242 port 37646
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 559 KBytes 4.55 Mbits/sec
[ 5] 1.01-2.00 sec 4.30 MBytes 36.1 Mbits/sec
[ 5] 2.00-3.00 sec 14.9 MBytes 126 Mbits/sec
[ 5] 3.00-4.00 sec 173 MBytes 1.45 Gbits/sec
Ifdown ifup retores speed to 9-10Gb/s but id dosent matter if i do it on proxmox node or on freenas box. I was suspecting many thing [changed NIC to intel, mellanox, broadcom. Added additional cooling], but today ive tested network with other switch [Dell PCM8024 in M1000e blade chassis]. And after two hours everything was stable with constant network traffic. Ive switched back to Mikrotik switch, and after 20 minutes network become unstable and speed started to drop to Mb/s ranges.
I would be glad if someone could push me in right direction, else i will have to find other switch for my lab.