Community discussions

MikroTik App
 
FurthestCastle
just joined
Topic Author
Posts: 4
Joined: Sun Oct 23, 2022 2:10 am

Asymmetric and slow throughput

Sun Oct 23, 2022 2:58 am

Hello all,

I am new here and have searched the forum for similar issues but have not found a resolution yet. Apologies if the answer is hiding in plain sight.

The environment consists of three Mikrotik CRS switches. A network diagram is attached. The CRS309 is my top-of-rack switch for my two servers and pfsense router. The other two CRS switches are for access.

VLANs are routed through the pfsense machine in a router-on-a-stick configuration. No router functions are used in the CRS switches.

The issue I am having is throughput. Using iperf3, traffic between the 10 Gbe hosts never tests greater than 4 Gbps (using the -P 4 option). Usually it is 2.5 Gbps. Additionally, traffic bottlenecks asymmetrically. For example, between my 10 Gbe hosts I will see ~2.5 Gbps in one direction and ~1.8 Gbps in the other. Or, if testing between a 1Gbe host and a 10 Gbe server I will see 940 Mbps up and <800 Mbps down.

For clarity, the throughput issue is on the same VLAN (10) and not routed traffic. The routed traffic is minimal to non-exisitent. Only one bridge exists in the configuration and hardware offload appears to be working. Jumbo frames are NOT being used.

Obviously, there are many variables. But I am not certain my CRS configurations are correct.

Any guidance is GREATLY appreciated!

Thanks in advance,
-Todd
You do not have the required permissions to view the files attached to this post.
 
User avatar
smyers119
Member Candidate
Member Candidate
Posts: 232
Joined: Sat Feb 27, 2021 8:16 pm
Location: USA

Re: Asymmetric and slow throughput

Mon Oct 24, 2022 2:16 am

first thing i would do is confirm whatever device I am testing from is actually able to create 10Gbps of packets to begin with. Maybe test on a loopbak? or more preferably isolate 2 hosts on 1 switch and test between them

Next I would check my STP settings, you have a lot of duplicate pathways between switches.
 
FurthestCastle
just joined
Topic Author
Posts: 4
Joined: Sun Oct 23, 2022 2:10 am

Re: Asymmetric and slow throughput

Tue Oct 25, 2022 1:52 am

Excellent suggestions.

The loop is blocked by RSTP and the NICs appear to be capable of driving a 10 Gbe connection. For the test, I ran iperf3 across the directly attached path between the NAS and Proxmox servers. The test data shows the NAS to Proxmox link handling nearly full wire speed.

Attached are the results of several iperf3 tests. There are numerous retransmits and the congestion window is vanishingly small. When not testing, the network is nearly idle with little to no traffic.

From a Wireshark capture I see several [TCP window specified by the receiver is now completely full] and numerous [Expert Info (Note/Sequence): This frame is a (suspected) retransmission] messages.

Note: I updated the drawing to clarify the details. All test results are from the iperf server output.

Any suggestions on how to track down the cause of the retransmissions?

Thanks again.


NAS (10Gbe) <--> POPOS WORKSTATION (5Gbe)
10.10.10.2		10.10.10.128
-----------------------------------------------------------
Server listening on 5201 (test #20)
-----------------------------------------------------------
Accepted connection from 10.10.10.128, port 34844
[  5] local 10.10.10.2 port 5201 connected to 10.10.10.128 port 34852
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   290 MBytes  2.43 Gbits/sec                  
[  5]   1.00-2.00   sec   345 MBytes  2.90 Gbits/sec                  
[  5]   2.00-3.00   sec   209 MBytes  1.75 Gbits/sec                  
[  5]   3.00-4.00   sec   334 MBytes  2.80 Gbits/sec                  
[  5]   4.00-5.00   sec   346 MBytes  2.91 Gbits/sec                  
[  5]   5.00-6.00   sec   346 MBytes  2.90 Gbits/sec                  
[  5]   6.00-7.00   sec   347 MBytes  2.91 Gbits/sec                  
[  5]   7.00-8.00   sec   346 MBytes  2.90 Gbits/sec                  
[  5]   8.00-9.00   sec   346 MBytes  2.91 Gbits/sec                  
[  5]   9.00-10.00  sec   345 MBytes  2.90 Gbits/sec                  
[  5]  10.00-10.00  sec  1.61 MBytes  2.89 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.18 GBytes  2.73 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #21)
-----------------------------------------------------------
Accepted connection from 10.10.10.128, port 43704
[  5] local 10.10.10.2 port 5201 connected to 10.10.10.128 port 43706
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec   177 MBytes  1.47 Gbits/sec   91    533 KBytes       
[  5]   1.01-2.00   sec   243 MBytes  2.07 Gbits/sec  689    253 KBytes       
[  5]   2.00-3.01   sec   238 MBytes  1.99 Gbits/sec  1056    288 KBytes       
[  5]   3.01-4.00   sec   239 MBytes  2.02 Gbits/sec  367    255 KBytes       
[  5]   4.00-5.00   sec   197 MBytes  1.65 Gbits/sec  363    864 KBytes       
[  5]   5.00-6.01   sec   183 MBytes  1.52 Gbits/sec  785    881 KBytes       
[  5]   6.01-7.00   sec   174 MBytes  1.48 Gbits/sec  975    106 KBytes       
[  5]   7.00-8.01   sec   228 MBytes  1.90 Gbits/sec  583    483 KBytes       
[  5]   8.01-9.00   sec   287 MBytes  2.42 Gbits/sec  589    486 KBytes       
[  5]   9.00-10.00  sec   321 MBytes  2.69 Gbits/sec  696    508 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.24 GBytes  1.92 Gbits/sec  6194             sender




NAS (10Gbe) <--> DEBIAN WORKSTATION (10Gbe)
10.10.10.2		10.10.10.240
-----------------------------------------------------------
Server listening on 5201 (test #22)
-----------------------------------------------------------
Accepted connection from 10.10.10.240, port 49492
[  5] local 10.10.10.2 port 5201 connected to 10.10.10.240 port 49500
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   303 MBytes  2.54 Gbits/sec                  
[  5]   1.00-2.00   sec   392 MBytes  3.29 Gbits/sec                  
[  5]   2.00-3.00   sec   400 MBytes  3.36 Gbits/sec                  
[  5]   3.00-4.00   sec   396 MBytes  3.32 Gbits/sec                  
[  5]   4.00-5.00   sec   396 MBytes  3.32 Gbits/sec                  
[  5]   5.00-6.00   sec   407 MBytes  3.42 Gbits/sec                  
[  5]   6.00-7.00   sec   413 MBytes  3.47 Gbits/sec                  
[  5]   7.00-8.00   sec   429 MBytes  3.60 Gbits/sec                  
[  5]   8.00-9.00   sec   438 MBytes  3.67 Gbits/sec                  
[  5]   9.00-10.00  sec   460 MBytes  3.86 Gbits/sec                  
[  5]  10.00-10.00  sec   679 KBytes  3.90 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.94 GBytes  3.38 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #23)
-----------------------------------------------------------
Accepted connection from 10.10.10.240, port 47520
[  5] local 10.10.10.2 port 5201 connected to 10.10.10.240 port 47536
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   223 MBytes  1.87 Gbits/sec   78    184 KBytes       
[  5]   1.00-2.00   sec   223 MBytes  1.87 Gbits/sec   91   65.5 KBytes       
[  5]   2.00-3.00   sec   252 MBytes  2.11 Gbits/sec   63    128 KBytes       
[  5]   3.00-4.00   sec   225 MBytes  1.89 Gbits/sec   66    191 KBytes       
[  5]   4.00-5.00   sec   213 MBytes  1.78 Gbits/sec   75    182 KBytes       
[  5]   5.00-6.01   sec   196 MBytes  1.63 Gbits/sec   54    248 KBytes       
[  5]   6.01-7.00   sec   197 MBytes  1.66 Gbits/sec   75    181 KBytes       
[  5]   7.00-8.01   sec   226 MBytes  1.89 Gbits/sec   84    152 KBytes       
[  5]   8.01-9.01   sec   201 MBytes  1.68 Gbits/sec   98    221 KBytes       
[  5]   9.01-10.00  sec   200 MBytes  1.69 Gbits/sec   81    231 KBytes       
[  5]  10.00-10.00  sec   467 KBytes  1.77 Gbits/sec    0    234 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.11 GBytes  1.81 Gbits/sec  765             sender



NAS (10Gbe) <--> MACOS (1Gbe)
10.10.10.2		10.10.10.101

-----------------------------------------------------------
Server listening on 5201 (test #24)
-----------------------------------------------------------
Accepted connection from 10.10.10.101, port 50871
[  5] local 10.10.10.2 port 5201 connected to 10.10.10.101 port 50872
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   103 MBytes   865 Mbits/sec                  
[  5]   1.00-2.00   sec   106 MBytes   889 Mbits/sec                  
[  5]   2.00-3.00   sec   108 MBytes   904 Mbits/sec                  
[  5]   3.00-4.00   sec   103 MBytes   864 Mbits/sec                  
[  5]   4.00-5.00   sec   108 MBytes   903 Mbits/sec                  
[  5]   5.00-6.00   sec   108 MBytes   905 Mbits/sec                  
[  5]   6.00-7.00   sec   108 MBytes   904 Mbits/sec                  
[  5]   7.00-8.00   sec   108 MBytes   905 Mbits/sec                  
[  5]   8.00-9.00   sec   106 MBytes   889 Mbits/sec                  
[  5]   9.00-10.00  sec   104 MBytes   876 Mbits/sec                  
[  5]  10.00-10.00  sec   119 KBytes   738 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.04 GBytes   890 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #25)
-----------------------------------------------------------
Accepted connection from 10.10.10.101, port 50876
[  5] local 10.10.10.2 port 5201 connected to 10.10.10.101 port 50877
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   109 MBytes   916 Mbits/sec   90   65.1 KBytes       
[  5]   1.00-2.00   sec   107 MBytes   896 Mbits/sec   54    159 KBytes       
[  5]   2.00-3.00   sec   109 MBytes   911 Mbits/sec   55    109 KBytes       
[  5]   3.00-4.00   sec   109 MBytes   916 Mbits/sec   75    132 KBytes       
[  5]   4.00-5.00   sec   108 MBytes   904 Mbits/sec   71    118 KBytes       
[  5]   5.00-6.00   sec   109 MBytes   912 Mbits/sec   67    145 KBytes       
[  5]   6.00-7.00   sec   109 MBytes   913 Mbits/sec   72    160 KBytes       
[  5]   7.00-8.00   sec  82.5 MBytes   692 Mbits/sec   48    115 KBytes       
[  5]   8.00-9.00   sec   110 MBytes   919 Mbits/sec   59    146 KBytes       
[  5]   9.00-10.00  sec  82.6 MBytes   693 Mbits/sec   47   92.0 KBytes       
[  5]  10.00-10.00  sec  80.6 KBytes  1.03 Gbits/sec    0   93.4 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.01 GBytes   867 Mbits/sec  638             sender


NAS (10Gbe) <--> Windows 11 (WSL2) (10Gbe)
10.10.10.2		10.10.10.232

-----------------------------------------------------------
Server listening on 5201 (test #26)
-----------------------------------------------------------
Accepted connection from 10.10.10.232, port 50232
[  5] local 10.10.10.2 port 5201 connected to 10.10.10.232 port 50234
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   276 MBytes  2.32 Gbits/sec                  
[  5]   1.00-2.00   sec   397 MBytes  3.33 Gbits/sec                  
[  5]   2.00-3.00   sec   397 MBytes  3.33 Gbits/sec                  
[  5]   3.00-4.00   sec   458 MBytes  3.84 Gbits/sec                  
[  5]   4.00-5.00   sec   376 MBytes  3.15 Gbits/sec                  
[  5]   5.00-6.00   sec   397 MBytes  3.33 Gbits/sec                  
[  5]   6.00-7.00   sec   412 MBytes  3.46 Gbits/sec                  
[  5]   7.00-8.00   sec   377 MBytes  3.16 Gbits/sec                  
[  5]   8.00-9.00   sec   386 MBytes  3.24 Gbits/sec                  
[  5]   9.00-10.00  sec   391 MBytes  3.28 Gbits/sec                  
[  5]  10.00-10.00  sec   915 KBytes  3.10 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.78 GBytes  3.25 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #27)
-----------------------------------------------------------
Accepted connection from 10.10.10.232, port 50236
[  5] local 10.10.10.2 port 5201 connected to 10.10.10.232 port 50238
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec   190 MBytes  1.59 Gbits/sec  568    506 KBytes       
[  5]   1.01-2.00   sec   277 MBytes  2.34 Gbits/sec  227    555 KBytes       
[  5]   2.00-3.00   sec   234 MBytes  1.96 Gbits/sec  213    639 KBytes       
[  5]   3.00-4.00   sec   204 MBytes  1.71 Gbits/sec  307    426 KBytes       
[  5]   4.00-5.00   sec   265 MBytes  2.22 Gbits/sec  220    528 KBytes       
[  5]   5.00-6.00   sec   274 MBytes  2.30 Gbits/sec  271    595 KBytes       
[  5]   6.00-7.00   sec   274 MBytes  2.30 Gbits/sec  366    221 KBytes       
[  5]   7.00-8.00   sec   272 MBytes  2.28 Gbits/sec    0    917 KBytes       
[  5]   8.00-9.00   sec   229 MBytes  1.92 Gbits/sec  590    363 KBytes       
[  5]   9.00-10.00  sec   177 MBytes  1.49 Gbits/sec  617    402 KBytes       
[  5]  10.00-10.00  sec  1.08 MBytes  2.21 Gbits/sec    0    406 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.34 GBytes  2.01 Gbits/sec  3379             sender



NAS (10Gbe) <--> Proxmox (10Gbe) DIRECT VIA DAC
10.5.0.2		10.5.0.7

-----------------------------------------------------------
Server listening on 5201 (test #30)
-----------------------------------------------------------
Accepted connection from 10.5.0.7, port 46402
[  5] local 10.5.0.2 port 5201 connected to 10.5.0.7 port 46406
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   703 MBytes  5.89 Gbits/sec                  
[  5]   1.00-2.00   sec   675 MBytes  5.66 Gbits/sec                  
[  5]   2.00-3.00   sec  1.00 GBytes  8.61 Gbits/sec                  
[  5]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec                  
[  5]   5.00-6.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  5]   6.00-7.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  5]   7.00-8.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  5]   8.00-9.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  5]   9.00-10.00  sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]  10.00-10.00  sec   640 KBytes  8.53 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.0 GBytes  8.60 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #31)
-----------------------------------------------------------
Accepted connection from 10.5.0.7, port 39562
[  5] local 10.5.0.2 port 5201 connected to 10.5.0.7 port 39566
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   955 MBytes  8.01 Gbits/sec   53    707 KBytes       
[  5]   1.00-2.00   sec   921 MBytes  7.73 Gbits/sec   87    582 KBytes       
[  5]   2.00-3.00   sec   782 MBytes  6.56 Gbits/sec   29    737 KBytes       
[  5]   3.00-4.00   sec   852 MBytes  7.14 Gbits/sec   45    475 KBytes       
[  5]   4.00-5.00   sec   759 MBytes  6.37 Gbits/sec   36    787 KBytes       
[  5]   5.00-6.00   sec   797 MBytes  6.69 Gbits/sec   35    307 KBytes       
[  5]   6.00-7.00   sec   960 MBytes  8.05 Gbits/sec   31    492 KBytes       
[  5]   7.00-8.00   sec   915 MBytes  7.68 Gbits/sec   23    192 KBytes       
[  5]   8.00-9.00   sec   697 MBytes  5.85 Gbits/sec   27   1.41 KBytes       
[  5]   9.00-10.00  sec   721 MBytes  6.05 Gbits/sec   16   1.21 MBytes       
[  5]  10.00-10.00  sec   206 KBytes  7.11 Gbits/sec    0   1.21 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  8.16 GBytes  7.01 Gbits/sec  382             sender
You do not have the required permissions to view the files attached to this post.
 
FurthestCastle
just joined
Topic Author
Posts: 4
Joined: Sun Oct 23, 2022 2:10 am

Re: Asymmetric and slow throughput

Wed Nov 09, 2022 1:57 am

Solved, mostly.

The bottleneck in my setup was bridged interfaces on the hosts. The Truenas machine created a bridge for the jails. After replacing the 10G interface with an unused 1G interface, network bandwidth was >9 Gbps!

Who is online

Users browsing this forum: normis and 95 guests