Excellent suggestions.
The loop is blocked by RSTP and the NICs appear to be capable of driving a 10 Gbe connection. For the test, I ran iperf3 across the directly attached path between the NAS and Proxmox servers. The test data shows the NAS to Proxmox link handling nearly full wire speed.
Attached are the results of several iperf3 tests. There are numerous retransmits and the congestion window is vanishingly small. When not testing, the network is nearly idle with little to no traffic.
From a Wireshark capture I see several [TCP window specified by the receiver is now completely full] and numerous [Expert Info (Note/Sequence): This frame is a (suspected) retransmission] messages.
Note: I updated the drawing to clarify the details. All test results are from the iperf server output.
Any suggestions on how to track down the cause of the retransmissions?
Thanks again.
NAS (10Gbe) <--> POPOS WORKSTATION (5Gbe)
10.10.10.2 10.10.10.128
-----------------------------------------------------------
Server listening on 5201 (test #20)
-----------------------------------------------------------
Accepted connection from 10.10.10.128, port 34844
[ 5] local 10.10.10.2 port 5201 connected to 10.10.10.128 port 34852
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 290 MBytes 2.43 Gbits/sec
[ 5] 1.00-2.00 sec 345 MBytes 2.90 Gbits/sec
[ 5] 2.00-3.00 sec 209 MBytes 1.75 Gbits/sec
[ 5] 3.00-4.00 sec 334 MBytes 2.80 Gbits/sec
[ 5] 4.00-5.00 sec 346 MBytes 2.91 Gbits/sec
[ 5] 5.00-6.00 sec 346 MBytes 2.90 Gbits/sec
[ 5] 6.00-7.00 sec 347 MBytes 2.91 Gbits/sec
[ 5] 7.00-8.00 sec 346 MBytes 2.90 Gbits/sec
[ 5] 8.00-9.00 sec 346 MBytes 2.91 Gbits/sec
[ 5] 9.00-10.00 sec 345 MBytes 2.90 Gbits/sec
[ 5] 10.00-10.00 sec 1.61 MBytes 2.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 3.18 GBytes 2.73 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201 (test #21)
-----------------------------------------------------------
Accepted connection from 10.10.10.128, port 43704
[ 5] local 10.10.10.2 port 5201 connected to 10.10.10.128 port 43706
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.01 sec 177 MBytes 1.47 Gbits/sec 91 533 KBytes
[ 5] 1.01-2.00 sec 243 MBytes 2.07 Gbits/sec 689 253 KBytes
[ 5] 2.00-3.01 sec 238 MBytes 1.99 Gbits/sec 1056 288 KBytes
[ 5] 3.01-4.00 sec 239 MBytes 2.02 Gbits/sec 367 255 KBytes
[ 5] 4.00-5.00 sec 197 MBytes 1.65 Gbits/sec 363 864 KBytes
[ 5] 5.00-6.01 sec 183 MBytes 1.52 Gbits/sec 785 881 KBytes
[ 5] 6.01-7.00 sec 174 MBytes 1.48 Gbits/sec 975 106 KBytes
[ 5] 7.00-8.01 sec 228 MBytes 1.90 Gbits/sec 583 483 KBytes
[ 5] 8.01-9.00 sec 287 MBytes 2.42 Gbits/sec 589 486 KBytes
[ 5] 9.00-10.00 sec 321 MBytes 2.69 Gbits/sec 696 508 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.24 GBytes 1.92 Gbits/sec 6194 sender
NAS (10Gbe) <--> DEBIAN WORKSTATION (10Gbe)
10.10.10.2 10.10.10.240
-----------------------------------------------------------
Server listening on 5201 (test #22)
-----------------------------------------------------------
Accepted connection from 10.10.10.240, port 49492
[ 5] local 10.10.10.2 port 5201 connected to 10.10.10.240 port 49500
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 303 MBytes 2.54 Gbits/sec
[ 5] 1.00-2.00 sec 392 MBytes 3.29 Gbits/sec
[ 5] 2.00-3.00 sec 400 MBytes 3.36 Gbits/sec
[ 5] 3.00-4.00 sec 396 MBytes 3.32 Gbits/sec
[ 5] 4.00-5.00 sec 396 MBytes 3.32 Gbits/sec
[ 5] 5.00-6.00 sec 407 MBytes 3.42 Gbits/sec
[ 5] 6.00-7.00 sec 413 MBytes 3.47 Gbits/sec
[ 5] 7.00-8.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 8.00-9.00 sec 438 MBytes 3.67 Gbits/sec
[ 5] 9.00-10.00 sec 460 MBytes 3.86 Gbits/sec
[ 5] 10.00-10.00 sec 679 KBytes 3.90 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 3.94 GBytes 3.38 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201 (test #23)
-----------------------------------------------------------
Accepted connection from 10.10.10.240, port 47520
[ 5] local 10.10.10.2 port 5201 connected to 10.10.10.240 port 47536
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 223 MBytes 1.87 Gbits/sec 78 184 KBytes
[ 5] 1.00-2.00 sec 223 MBytes 1.87 Gbits/sec 91 65.5 KBytes
[ 5] 2.00-3.00 sec 252 MBytes 2.11 Gbits/sec 63 128 KBytes
[ 5] 3.00-4.00 sec 225 MBytes 1.89 Gbits/sec 66 191 KBytes
[ 5] 4.00-5.00 sec 213 MBytes 1.78 Gbits/sec 75 182 KBytes
[ 5] 5.00-6.01 sec 196 MBytes 1.63 Gbits/sec 54 248 KBytes
[ 5] 6.01-7.00 sec 197 MBytes 1.66 Gbits/sec 75 181 KBytes
[ 5] 7.00-8.01 sec 226 MBytes 1.89 Gbits/sec 84 152 KBytes
[ 5] 8.01-9.01 sec 201 MBytes 1.68 Gbits/sec 98 221 KBytes
[ 5] 9.01-10.00 sec 200 MBytes 1.69 Gbits/sec 81 231 KBytes
[ 5] 10.00-10.00 sec 467 KBytes 1.77 Gbits/sec 0 234 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.11 GBytes 1.81 Gbits/sec 765 sender
NAS (10Gbe) <--> MACOS (1Gbe)
10.10.10.2 10.10.10.101
-----------------------------------------------------------
Server listening on 5201 (test #24)
-----------------------------------------------------------
Accepted connection from 10.10.10.101, port 50871
[ 5] local 10.10.10.2 port 5201 connected to 10.10.10.101 port 50872
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 103 MBytes 865 Mbits/sec
[ 5] 1.00-2.00 sec 106 MBytes 889 Mbits/sec
[ 5] 2.00-3.00 sec 108 MBytes 904 Mbits/sec
[ 5] 3.00-4.00 sec 103 MBytes 864 Mbits/sec
[ 5] 4.00-5.00 sec 108 MBytes 903 Mbits/sec
[ 5] 5.00-6.00 sec 108 MBytes 905 Mbits/sec
[ 5] 6.00-7.00 sec 108 MBytes 904 Mbits/sec
[ 5] 7.00-8.00 sec 108 MBytes 905 Mbits/sec
[ 5] 8.00-9.00 sec 106 MBytes 889 Mbits/sec
[ 5] 9.00-10.00 sec 104 MBytes 876 Mbits/sec
[ 5] 10.00-10.00 sec 119 KBytes 738 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 1.04 GBytes 890 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201 (test #25)
-----------------------------------------------------------
Accepted connection from 10.10.10.101, port 50876
[ 5] local 10.10.10.2 port 5201 connected to 10.10.10.101 port 50877
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 109 MBytes 916 Mbits/sec 90 65.1 KBytes
[ 5] 1.00-2.00 sec 107 MBytes 896 Mbits/sec 54 159 KBytes
[ 5] 2.00-3.00 sec 109 MBytes 911 Mbits/sec 55 109 KBytes
[ 5] 3.00-4.00 sec 109 MBytes 916 Mbits/sec 75 132 KBytes
[ 5] 4.00-5.00 sec 108 MBytes 904 Mbits/sec 71 118 KBytes
[ 5] 5.00-6.00 sec 109 MBytes 912 Mbits/sec 67 145 KBytes
[ 5] 6.00-7.00 sec 109 MBytes 913 Mbits/sec 72 160 KBytes
[ 5] 7.00-8.00 sec 82.5 MBytes 692 Mbits/sec 48 115 KBytes
[ 5] 8.00-9.00 sec 110 MBytes 919 Mbits/sec 59 146 KBytes
[ 5] 9.00-10.00 sec 82.6 MBytes 693 Mbits/sec 47 92.0 KBytes
[ 5] 10.00-10.00 sec 80.6 KBytes 1.03 Gbits/sec 0 93.4 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.01 GBytes 867 Mbits/sec 638 sender
NAS (10Gbe) <--> Windows 11 (WSL2) (10Gbe)
10.10.10.2 10.10.10.232
-----------------------------------------------------------
Server listening on 5201 (test #26)
-----------------------------------------------------------
Accepted connection from 10.10.10.232, port 50232
[ 5] local 10.10.10.2 port 5201 connected to 10.10.10.232 port 50234
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 276 MBytes 2.32 Gbits/sec
[ 5] 1.00-2.00 sec 397 MBytes 3.33 Gbits/sec
[ 5] 2.00-3.00 sec 397 MBytes 3.33 Gbits/sec
[ 5] 3.00-4.00 sec 458 MBytes 3.84 Gbits/sec
[ 5] 4.00-5.00 sec 376 MBytes 3.15 Gbits/sec
[ 5] 5.00-6.00 sec 397 MBytes 3.33 Gbits/sec
[ 5] 6.00-7.00 sec 412 MBytes 3.46 Gbits/sec
[ 5] 7.00-8.00 sec 377 MBytes 3.16 Gbits/sec
[ 5] 8.00-9.00 sec 386 MBytes 3.24 Gbits/sec
[ 5] 9.00-10.00 sec 391 MBytes 3.28 Gbits/sec
[ 5] 10.00-10.00 sec 915 KBytes 3.10 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 3.78 GBytes 3.25 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201 (test #27)
-----------------------------------------------------------
Accepted connection from 10.10.10.232, port 50236
[ 5] local 10.10.10.2 port 5201 connected to 10.10.10.232 port 50238
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.01 sec 190 MBytes 1.59 Gbits/sec 568 506 KBytes
[ 5] 1.01-2.00 sec 277 MBytes 2.34 Gbits/sec 227 555 KBytes
[ 5] 2.00-3.00 sec 234 MBytes 1.96 Gbits/sec 213 639 KBytes
[ 5] 3.00-4.00 sec 204 MBytes 1.71 Gbits/sec 307 426 KBytes
[ 5] 4.00-5.00 sec 265 MBytes 2.22 Gbits/sec 220 528 KBytes
[ 5] 5.00-6.00 sec 274 MBytes 2.30 Gbits/sec 271 595 KBytes
[ 5] 6.00-7.00 sec 274 MBytes 2.30 Gbits/sec 366 221 KBytes
[ 5] 7.00-8.00 sec 272 MBytes 2.28 Gbits/sec 0 917 KBytes
[ 5] 8.00-9.00 sec 229 MBytes 1.92 Gbits/sec 590 363 KBytes
[ 5] 9.00-10.00 sec 177 MBytes 1.49 Gbits/sec 617 402 KBytes
[ 5] 10.00-10.00 sec 1.08 MBytes 2.21 Gbits/sec 0 406 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.34 GBytes 2.01 Gbits/sec 3379 sender
NAS (10Gbe) <--> Proxmox (10Gbe) DIRECT VIA DAC
10.5.0.2 10.5.0.7
-----------------------------------------------------------
Server listening on 5201 (test #30)
-----------------------------------------------------------
Accepted connection from 10.5.0.7, port 46402
[ 5] local 10.5.0.2 port 5201 connected to 10.5.0.7 port 46406
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 703 MBytes 5.89 Gbits/sec
[ 5] 1.00-2.00 sec 675 MBytes 5.66 Gbits/sec
[ 5] 2.00-3.00 sec 1.00 GBytes 8.61 Gbits/sec
[ 5] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 4.00-5.00 sec 1.09 GBytes 9.38 Gbits/sec
[ 5] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 6.00-7.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 7.00-8.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 9.00-10.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 10.00-10.00 sec 640 KBytes 8.53 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 10.0 GBytes 8.60 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201 (test #31)
-----------------------------------------------------------
Accepted connection from 10.5.0.7, port 39562
[ 5] local 10.5.0.2 port 5201 connected to 10.5.0.7 port 39566
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 955 MBytes 8.01 Gbits/sec 53 707 KBytes
[ 5] 1.00-2.00 sec 921 MBytes 7.73 Gbits/sec 87 582 KBytes
[ 5] 2.00-3.00 sec 782 MBytes 6.56 Gbits/sec 29 737 KBytes
[ 5] 3.00-4.00 sec 852 MBytes 7.14 Gbits/sec 45 475 KBytes
[ 5] 4.00-5.00 sec 759 MBytes 6.37 Gbits/sec 36 787 KBytes
[ 5] 5.00-6.00 sec 797 MBytes 6.69 Gbits/sec 35 307 KBytes
[ 5] 6.00-7.00 sec 960 MBytes 8.05 Gbits/sec 31 492 KBytes
[ 5] 7.00-8.00 sec 915 MBytes 7.68 Gbits/sec 23 192 KBytes
[ 5] 8.00-9.00 sec 697 MBytes 5.85 Gbits/sec 27 1.41 KBytes
[ 5] 9.00-10.00 sec 721 MBytes 6.05 Gbits/sec 16 1.21 MBytes
[ 5] 10.00-10.00 sec 206 KBytes 7.11 Gbits/sec 0 1.21 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 8.16 GBytes 7.01 Gbits/sec 382 sender
You do not have the required permissions to view the files attached to this post.