I'm using a CRS309 as a switch to provide L2 connectivity for a home lab. I have two of these devices and am not using any of the L3 features. I basically just want VLAN trunking to a few ports facing ESXi hosts and access VLANs on others facing some physical systems.
Problem: Slow TCP throughput on local LAN when frames are passed through the MikroTik CRS309 into a trunked ESXi host.
Environment: Currently, I have only one CRS309 switch in the lab, one ESXi host, two windows physical systems, and two VMs running in the ESXi host: a palo alto firewall and a windows vm. All physical machines are connected with 10gbit nics and known-good cables/transceivers. The ESXi vswitch configuration is all defaults with normal VLAN tagging on the port groups.
If I run iperf between two physical systems on access ports I get good performance with TCP and UDP:
[client02] <--> (sfpp2)[crs309](sfpp3) <--> [server03]
TCP performance:
C:\iperf-3.1.3-win64>iperf3 -c 10.1.3.3 -t 0
Connecting to host 10.1.3.3, port 5201
[ 4] local 10.1.3.133 port 52819 connected to 10.1.3.3 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 338 MBytes 2.83 Gbits/sec
[ 4] 1.00-2.00 sec 334 MBytes 2.80 Gbits/sec
[ 4] 2.00-3.00 sec 339 MBytes 2.85 Gbits/sec
[ 4] 3.00-3.72 sec 244 MBytes 2.83 Gbits/sec
------------------------------------------
However, if the iperf server is a VM inside of my ESXi host the TCP performance is abysmal, but UDP performance is great:
[client02] <--> (sfpp2)[crs309](sfpp5) <--> (vmnic6)[esxi05 - vswitch0](vlan 3 port group)<-->[virtual windows server]
TCP performance:
C:\iperf-3.1.3-win64>iperf3 -c 10.1.3.23 -t 0
Connecting to host 10.1.3.23, port 5201
[ 4] local 10.1.3.133 port 50293 connected to 10.1.3.23 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1.00 MBytes 8.37 Mbits/sec
[ 4] 1.00-2.00 sec 1.50 MBytes 12.6 Mbits/sec
[ 4] 2.00-3.00 sec 1.50 MBytes 12.6 Mbits/sec
UDP performance:
C:\iperf-3.1.3-win64>iperf3 -c 10.1.3.23 -t 0 -u -b 0
Connecting to host 10.1.3.23, port 5201
[ 4] local 10.1.3.133 port 52311 connected to 10.1.3.23 port 5201
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 315 MBytes 2.64 Gbits/sec 40320
[ 4] 1.00-2.00 sec 312 MBytes 2.62 Gbits/sec 39930
[ 4] 2.00-3.00 sec 316 MBytes 2.65 Gbits/sec 40420
[ 4] 3.00-4.00 sec 314 MBytes 2.63 Gbits/sec 40160
If I take the mikrotik out of the picture and connect my desktop to a 10gbit port on the ESXi host and configure the port groups appropriately, TCP throughput is unhindered.
[client02] <--> (vmnic7)[esxi05 - vswitch3](private port group)<-->[virtual windows server]
Note: I don't have a screen cap from this test, but I've ran it several times: it's fine. Similarly, speed tests between two VMs are fine, as is traffic routed to the internet via a 1gbit provider link on another port on the ESXi host.
I feel I must be missing something in the config. I've tried many variations of flipping features and flags on and off
Here is the current configuration:
Code: Select all
[admin@sw02] > export hide-sensitive
# sep/11/2021 06:52:51 by RouterOS 6.49beta54
# software id = D8LP-5LB6
#
# model = CRS309-1G-8S+
/interface bridge
add admin-mac=2C:C8:1B:A6:9E:9E auto-mac=no comment=defconf ingress-filtering=yes name=bridge vlan-filtering=\
yes
/interface ethernet
set [ find default-name=sfp-sfpplus2 ] advertise=\
10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full,2500M-full,5000M-full,10000M-full
/interface vlan
add interface=bridge name=vlan2 vlan-id=2
add interface=bridge name=vlan3 vlan-id=3
add interface=bridge name=vlan4 vlan-id=4
add interface=bridge name=vlan5 vlan-id=5
add interface=bridge name=vlan6 vlan-id=6
add interface=bridge name=vlan7 vlan-id=7
add interface=bridge name=vlan8 vlan-id=8
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf hw=no interface=ether1
add bridge=bridge comment="client02 on vlan1" interface=sfp-sfpplus1
add bridge=bridge comment="client02 on vlan3" interface=sfp-sfpplus2 pvid=3
add bridge=bridge comment="server03 on vlan3" interface=sfp-sfpplus3 pvid=3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=esxi05 interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7 pvid=7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
/interface bridge vlan
add bridge=bridge comment=ad tagged=sfp-sfpplus5,sfp-sfpplus8,sfp-sfpplus1 vlan-ids=2
add bridge=bridge comment=backup tagged=sfp-sfpplus5,sfp-sfpplus8,sfp-sfpplus1 untagged=sfp-sfpplus3 vlan-ids=3
add bridge=bridge comment=vcenter tagged=sfp-sfpplus5,sfp-sfpplus8,sfp-sfpplus1 vlan-ids=4
add bridge=bridge comment=hypervisor tagged=sfp-sfpplus5,sfp-sfpplus1,sfp-sfpplus8 vlan-ids=5
add bridge=bridge comment=sql tagged=sfp-sfpplus1,sfp-sfpplus5,sfp-sfpplus8 vlan-ids=6
add bridge=bridge comment=storage tagged=sfp-sfpplus1,sfp-sfpplus5,sfp-sfpplus8 untagged=sfp-sfpplus7 vlan-ids=\
7
add bridge=bridge comment="desktop alternate" tagged=sfp-sfpplus1,sfp-sfpplus5 untagged=sfp-sfpplus2 vlan-ids=8
/interface list member
add interface=ether1 list=LAN
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
add interface=sfp-sfpplus5 list=LAN
add interface=sfp-sfpplus6 list=LAN
add interface=sfp-sfpplus7 list=LAN
add interface=sfp-sfpplus8 list=LAN
/ip address
add address=10.1.1.12/24 comment=defconf interface=sfp-sfpplus1 network=10.1.1.0
/ip dns
set servers=10.1.2.3,1.1.1.1
/ip route
add distance=1 gateway=10.1.1.1
/system clock
set time-zone-name=Asia/Bangkok
/system identity
set name=sw02
/system package update
set channel=testing
/system routerboard settings
set boot-os=router-os
[admin@sw02] >
Note:
I have tried testing/dev/stable firmware for fun and have found the issue on all three branches. I have VLANs tagged on sfpplus8 and sfpplus1 because I had intended these to go to other switches, but those are disconnected now.
Any help would be much appreciated.