Community discussions

MikroTik App
 
d3k
just joined
Topic Author
Posts: 2
Joined: Sat Sep 11, 2021 2:08 am

Slow TCP throughput on CRS309 with VLAN trunking

Sat Sep 11, 2021 3:16 am

Hello All,

I'm using a CRS309 as a switch to provide L2 connectivity for a home lab. I have two of these devices and am not using any of the L3 features. I basically just want VLAN trunking to a few ports facing ESXi hosts and access VLANs on others facing some physical systems.

Problem: Slow TCP throughput on local LAN when frames are passed through the MikroTik CRS309 into a trunked ESXi host.

Environment: Currently, I have only one CRS309 switch in the lab, one ESXi host, two windows physical systems, and two VMs running in the ESXi host: a palo alto firewall and a windows vm. All physical machines are connected with 10gbit nics and known-good cables/transceivers. The ESXi vswitch configuration is all defaults with normal VLAN tagging on the port groups.

If I run iperf between two physical systems on access ports I get good performance with TCP and UDP:
[client02] <--> (sfpp2)[crs309](sfpp3) <--> [server03]
TCP performance:
C:\iperf-3.1.3-win64>iperf3 -c 10.1.3.3 -t 0
Connecting to host 10.1.3.3, port 5201
[ 4] local 10.1.3.133 port 52819 connected to 10.1.3.3 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 338 MBytes 2.83 Gbits/sec
[ 4] 1.00-2.00 sec 334 MBytes 2.80 Gbits/sec
[ 4] 2.00-3.00 sec 339 MBytes 2.85 Gbits/sec
[ 4] 3.00-3.72 sec 244 MBytes 2.83 Gbits/sec

------------------------------------------

However, if the iperf server is a VM inside of my ESXi host the TCP performance is abysmal, but UDP performance is great:
[client02] <--> (sfpp2)[crs309](sfpp5) <--> (vmnic6)[esxi05 - vswitch0](vlan 3 port group)<-->[virtual windows server]
TCP performance:
C:\iperf-3.1.3-win64>iperf3 -c 10.1.3.23 -t 0
Connecting to host 10.1.3.23, port 5201
[ 4] local 10.1.3.133 port 50293 connected to 10.1.3.23 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1.00 MBytes 8.37 Mbits/sec
[ 4] 1.00-2.00 sec 1.50 MBytes 12.6 Mbits/sec
[ 4] 2.00-3.00 sec 1.50 MBytes 12.6 Mbits/sec

UDP performance:
C:\iperf-3.1.3-win64>iperf3 -c 10.1.3.23 -t 0 -u -b 0
Connecting to host 10.1.3.23, port 5201
[ 4] local 10.1.3.133 port 52311 connected to 10.1.3.23 port 5201
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 315 MBytes 2.64 Gbits/sec 40320
[ 4] 1.00-2.00 sec 312 MBytes 2.62 Gbits/sec 39930
[ 4] 2.00-3.00 sec 316 MBytes 2.65 Gbits/sec 40420
[ 4] 3.00-4.00 sec 314 MBytes 2.63 Gbits/sec 40160


If I take the mikrotik out of the picture and connect my desktop to a 10gbit port on the ESXi host and configure the port groups appropriately, TCP throughput is unhindered.
[client02] <--> (vmnic7)[esxi05 - vswitch3](private port group)<-->[virtual windows server]
Note: I don't have a screen cap from this test, but I've ran it several times: it's fine. Similarly, speed tests between two VMs are fine, as is traffic routed to the internet via a 1gbit provider link on another port on the ESXi host.

I feel I must be missing something in the config. I've tried many variations of flipping features and flags on and off
Here is the current configuration:
[admin@sw02] > export hide-sensitive
# sep/11/2021 06:52:51 by RouterOS 6.49beta54
# software id = D8LP-5LB6
#
# model = CRS309-1G-8S+
/interface bridge
add admin-mac=2C:C8:1B:A6:9E:9E auto-mac=no comment=defconf ingress-filtering=yes name=bridge vlan-filtering=\
    yes
/interface ethernet
set [ find default-name=sfp-sfpplus2 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full,2500M-full,5000M-full,10000M-full
/interface vlan
add interface=bridge name=vlan2 vlan-id=2
add interface=bridge name=vlan3 vlan-id=3
add interface=bridge name=vlan4 vlan-id=4
add interface=bridge name=vlan5 vlan-id=5
add interface=bridge name=vlan6 vlan-id=6
add interface=bridge name=vlan7 vlan-id=7
add interface=bridge name=vlan8 vlan-id=8
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf hw=no interface=ether1
add bridge=bridge comment="client02 on vlan1" interface=sfp-sfpplus1
add bridge=bridge comment="client02 on vlan3" interface=sfp-sfpplus2 pvid=3
add bridge=bridge comment="server03 on vlan3" interface=sfp-sfpplus3 pvid=3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=esxi05 interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7 pvid=7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
/interface bridge vlan
add bridge=bridge comment=ad tagged=sfp-sfpplus5,sfp-sfpplus8,sfp-sfpplus1 vlan-ids=2
add bridge=bridge comment=backup tagged=sfp-sfpplus5,sfp-sfpplus8,sfp-sfpplus1 untagged=sfp-sfpplus3 vlan-ids=3
add bridge=bridge comment=vcenter tagged=sfp-sfpplus5,sfp-sfpplus8,sfp-sfpplus1 vlan-ids=4
add bridge=bridge comment=hypervisor tagged=sfp-sfpplus5,sfp-sfpplus1,sfp-sfpplus8 vlan-ids=5
add bridge=bridge comment=sql tagged=sfp-sfpplus1,sfp-sfpplus5,sfp-sfpplus8 vlan-ids=6
add bridge=bridge comment=storage tagged=sfp-sfpplus1,sfp-sfpplus5,sfp-sfpplus8 untagged=sfp-sfpplus7 vlan-ids=\
    7
add bridge=bridge comment="desktop alternate" tagged=sfp-sfpplus1,sfp-sfpplus5 untagged=sfp-sfpplus2 vlan-ids=8
/interface list member
add interface=ether1 list=LAN
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
add interface=sfp-sfpplus5 list=LAN
add interface=sfp-sfpplus6 list=LAN
add interface=sfp-sfpplus7 list=LAN
add interface=sfp-sfpplus8 list=LAN
/ip address
add address=10.1.1.12/24 comment=defconf interface=sfp-sfpplus1 network=10.1.1.0
/ip dns
set servers=10.1.2.3,1.1.1.1
/ip route
add distance=1 gateway=10.1.1.1
/system clock
set time-zone-name=Asia/Bangkok
/system identity
set name=sw02
/system package update
set channel=testing
/system routerboard settings
set boot-os=router-os
[admin@sw02] >


Note:
I have tried testing/dev/stable firmware for fun and have found the issue on all three branches. I have VLANs tagged on sfpplus8 and sfpplus1 because I had intended these to go to other switches, but those are disconnected now.

Any help would be much appreciated.
 
d3k
just joined
Topic Author
Posts: 2
Joined: Sat Sep 11, 2021 2:08 am

Re: Slow TCP throughput on CRS309 with VLAN trunking

Sat Sep 11, 2021 9:43 am

Addendum, I decided to try another mikrotik switch running SwOS, a CSS610-8G-2S-IN. In this switch I've trunked the appropriate VLANs, connected my PC to a 1gbit port, and the ESXi host to one of the two 10gbit ports.

Here are the results with the CSS610 switch:
C:\iperf-3.1.3-win64>iperf3 -c 10.1.2.3 -t 0
Connecting to host 10.1.2.3, port 5201
[ 4] local 10.1.1.111 port 51430 connected to 10.1.2.3 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 111 MBytes 933 Mbits/sec
[ 4] 1.00-2.00 sec 112 MBytes 942 Mbits/sec
[ 4] 2.00-3.00 sec 112 MBytes 942 Mbits/sec
[ 4] 3.00-4.00 sec 112 MBytes 942 Mbits/sec
[ 4] 4.00-5.00 sec 112 MBytes 942 Mbits/sec
[ 4] 5.00-5.87 sec 97.6 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-5.87 sec 658 MBytes 940 Mbits/sec sender
[ 4] 0.00-5.87 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

--> Edit: Plugged my PC into one of the 10gbit ports and tested:
C:\iperf-3.1.3-win64>iperf3 -c 10.1.2.3 -t 0
Connecting to host 10.1.2.3, port 5201
[ 4] local 10.1.1.112 port 57248 connected to 10.1.2.3 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 335 MBytes 2.81 Gbits/sec
[ 4] 1.00-2.00 sec 313 MBytes 2.62 Gbits/sec
[ 4] 2.00-3.00 sec 314 MBytes 2.64 Gbits/sec
[ 4] 3.00-4.00 sec 316 MBytes 2.65 Gbits/sec
[ 4] 4.00-5.00 sec 322 MBytes 2.70 Gbits/sec
[ 4] 5.00-6.00 sec 322 MBytes 2.70 Gbits/sec
...

----------
So... definitely something with the CRS309 device or the config.
 
rcjokibbe
just joined
Posts: 1
Joined: Sat Jan 01, 2022 2:51 pm

Re: Slow TCP throughput on CRS309 with VLAN trunking

Tue Jan 04, 2022 9:46 pm

I might have similar symptoms when using a Mikrotik HAP AC2 router. The Iperf server is a VM on ESXi and is connected to HAP AC2 via trunk port. Client is wired.

Iperf when using HAP AC2 as router with VLANs:
Connecting to host lx-server, port 5201
[ 6] local 192.168.24.75 port 50468 connected to 192.168.22.4 port 5201
[ ID] Interval Transfer Bandwidth
[ 6] 0.00-1.00 sec 34.9 MBytes 293 Mbits/sec
[ 6] 1.00-2.00 sec 46.3 MBytes 388 Mbits/sec
[ 6] 2.00-3.00 sec 46.1 MBytes 387 Mbits/sec
[ 6] 3.00-4.00 sec 45.7 MBytes 383 Mbits/sec
[ 6] 4.00-5.00 sec 46.1 MBytes 386 Mbits/sec
[ 6] 5.00-6.00 sec 46.1 MBytes 386 Mbits/sec
[ 6] 6.00-7.00 sec 45.6 MBytes 382 Mbits/sec
[ 6] 7.00-8.00 sec 46.1 MBytes 386 Mbits/sec
[ 6] 8.00-9.00 sec 46.2 MBytes 387 Mbits/sec
[ 6] 9.00-10.00 sec 45.7 MBytes 383 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -

Iperf when using Untangle VM as router with VLANs:
Connecting to host lx-server, port 5201
[ 6] local 192.168.24.75 port 51083 connected to 192.168.22.4 port 5201
[ ID] Interval Transfer Bandwidth
[ 6] 0.00-1.00 sec 112 MBytes 941 Mbits/sec
[ 6] 1.00-2.00 sec 110 MBytes 923 Mbits/sec
[ 6] 2.00-3.00 sec 111 MBytes 931 Mbits/sec
[ 6] 3.00-4.00 sec 111 MBytes 929 Mbits/sec
[ 6] 4.00-5.00 sec 110 MBytes 925 Mbits/sec
[ 6] 5.00-6.00 sec 104 MBytes 876 Mbits/sec
[ 6] 6.00-7.00 sec 111 MBytes 928 Mbits/sec
[ 6] 7.00-8.00 sec 111 MBytes 928 Mbits/sec
[ 6] 8.00-9.00 sec 111 MBytes 928 Mbits/sec
[ 6] 9.00-10.00 sec 110 MBytes 921 Mbits/sec

Who is online

Users browsing this forum: coffee1978, DanMos79, EsaqzpHot, icemending and 93 guests