crs309 poor 10gb performance

Hello,

I have bought a CRS309-1G-8S+IN for upgrade my lan network to 10Gb.
I have connected my wmware server with to dac sfp+ 10Gb cable , my qnap nas with an S+RJ10 module and my computer in 2.5Gb with also an S+RJ10 module.

i have also an aruba 1930 connected with dac cable aruba for the the equipements which don’t need 10G.

when my vmware server is connected on the aruba, i have average 250MB in wrting and 600MB in reading

When i connect the vmware server on the crs309, i have 250MB in writing but only 15MB in reading.

With my computer connected in 2.5GB , i have 250MB in writing/reading.

i’m a noob with routerOS and i don’t undersand why i have so poor reading performance with my vmware server.

Best regards

Hi,


Most likely, the traffic is processed by the CPU instead of the hardware, and that is the reason for slow networking. Post your interface and ip config, please, so we can analyze it.

/interface export
/ip export

Hello,

i have try with switchOS, not better.
now i can’t restart on RoutrOs, i click on system → boot router os but always restart on switchOs…

After a reset welcom back RouterOS!
so here the interface export

/interface bridge
add admin-mac=78:9A:18:XX:XX:XX auto-mac=no comment=defconf name=bridge \
    vlan-filtering=yes
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
/interface bridge vlan
add bridge=bridge tagged=sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3 vlan-ids=10
/interface list member
add interface=ether1 list=WAN
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
add interface=sfp-sfpplus5 list=LAN
add interface=sfp-sfpplus6 list=LAN
add interface=sfp-sfpplus7 list=LAN
add interface=sfp-sfpplus8 list=LAN

ip - export :

/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip address
add address=10.28.201.238/24 comment=defconf interface=sfp-sfpplus1 network=10.28.201.0
/ip dns
set servers=10.28.201.155

Your configuration looks pretty straightforward, i.e. no routing, just switching, which is good. What speeds are the ports reporting? Do they show up as 10G? Are all the devices in the same LAN subnet?

hello,

yes all the devices are in the same subnet.
In the doubt i will check again this evening but it seem the ports show up as 10G.
As you say, my configuration is straightforward, i just want 10G switching nothing more!

I keep you informed later.
Thanks

hello,

i have checked, the concerned ports show up as 10G when connected with dac cables.

I’m no expert with MT devices but shouldn’t the IP be assigned to the bridge and not the physical interface in that configuration?

Hello,
i assign the ip on the bridge but no change.

I just noticed that when i do a iperf between my computer (connected in 2.5G on the crs 309) and the crs309, i have 2.5Gbit in upload but only 250Mbit in download.
if i do an internet speed test, i have also 250Mbit upload and 600Mbit in upload (600Mbit is the max of my internet contract).
Finally the problem is not only with my vmware server…
That drives me crazy

What is the processor consumption while performing a speedtest?

I test with btest and the cpu consumption is to 50% and the hardware offloading is enabled.
for an internet speed test no more 2% consumption

Did you try to enable flow control (both Tx and Rx) on all involved ports?

The thing that bothers a switch the most is speed change - from faster to slower port. In this case switch has to buffer data and we all know that buffer bloat is bad. So when that tiny buffer fills up, switch has two choices: 1) drop next frame or 2) signal transmitter that it has to pause sending … the later is done by deploying flow control. The former is default behaviour, but hurts TCP throughput imensely … because in this case transmitter has to re-transmit and with relatively high-latency (milliseconds on 10Gbps networks is high latency) this kills throughput.

I see you have VLAN filtering turned on, maybe VLANs are not configured properly. Would you be able to share a full config export and removing any sensitive info (public IPs, passphrases, usernames, serial numbers, etc)

While performing a bandwidth test take a look under Tools → Profile. Click Start for CPU all and see what’s hogging up resources.

Post #11 above (by @jericho63) IMO shows that throughput problems are not due to traffic hitting CPU (it’d be much higher than 2% at 2.5Gbps) but some other reasons, internal to how switch chip handles the traffic.
High CPU load while running btest is usual but has nothing with normal traffic handling (which is port-to-port, not port-to-CPU as in case of btest).

Does it work correctly if you use 10Gbit on both connections? (or do you only have asymmetric connections).

Sometimes flow control helps but you might need to go to the switch port settings and set an ingress/egress rate limit to see if you can get closer to the actual hardware line rate.

One way to identify this is if an upload from 2.5 GB connection is line speed (to a 10GB connection) but download is under 1GB then this is most likely the issue.

@mkx
i have try to enable flow control following your reply.
For the test (compturer to crs with btest) i have enabled it only for the port where is connected my computer but not better, i have always a througput in download arround 250-300Mbits with btest and an througput at 2.5Gbits in upload, internet speedtest 195Mbits in download / 595Mbits un upload.
If i try a test with my nas (with nas performance tester 1.7https://www.808.dk/?code-csharp-nas-performance) , connected on the crs309 as my computer, i have 211MBytes in wrtting and 269MBytes in reading.

@darthgizm0
i have disbled valn filtering since a while but no change.
during a test with btest in receive mode, profile show arround 20% on network and 20% on btest.

@biomesh
I have vmware server and a qnap nas connected in 10G and my computer in 2.5G.
But i have also througput issues with my virtuals machines to internet or to the NAS.
I will try an upload from my computer to one vm and see the througput.

The best way to test is using iperf on both ends. As long as the machines on both ends are powerful enough to generate/receive 10Gbit worth of traffic, you won’t hit any other bottlenecks like disk IO etc.

i know i will have a bottleneck in reason of the disk i/o limit on the vmware server because they are not ssd but for the moment i have a througput below it.. and it is very frustrating.
i will perform new tests this evening.

here the result of iperf client side (computer 2.5G as server / server vmware client):

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   237 MBytes  1.98 Gbits/sec
[  4]   1.00-2.00   sec   160 MBytes  1.35 Gbits/sec
[  4]   2.00-3.00   sec   188 MBytes  1.57 Gbits/sec
[  4]   3.00-4.00   sec   203 MBytes  1.70 Gbits/sec
[  4]   4.00-5.00   sec   187 MBytes  1.57 Gbits/sec
[  4]   5.00-6.00   sec   192 MBytes  1.61 Gbits/sec
[  4]   6.00-7.00   sec   192 MBytes  1.61 Gbits/sec
[  4]   7.00-8.00   sec   192 MBytes  1.61 Gbits/sec
[  4]   8.00-9.00   sec   195 MBytes  1.63 Gbits/sec
[  4]   9.00-10.00  sec   191 MBytes  1.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.89 GBytes  1.62 Gbits/sec                  sender
[  4]   0.00-10.00  sec  1.89 GBytes  1.62 Gbits/sec                  receiver

and here client side where server vmware is server and computer in 2.5G as client:

[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.01   sec  8.16 MBytes  67.9 Mbits/sec
[  5]   1.01-2.00   sec  48.3 MBytes   408 Mbits/sec
[  5]   2.00-3.01   sec  20.4 MBytes   169 Mbits/sec
[  5]   3.01-4.00   sec  7.29 MBytes  61.4 Mbits/sec
[  5]   4.00-5.00   sec  7.75 MBytes  65.0 Mbits/sec
[  5]   5.00-6.01   sec  8.72 MBytes  72.9 Mbits/sec
[  5]   6.01-7.00   sec  9.38 MBytes  79.1 Mbits/sec
[  5]   7.00-8.00   sec  8.27 MBytes  69.2 Mbits/sec
[  5]   8.00-9.00   sec  10.1 MBytes  85.3 Mbits/sec
[  5]   9.00-10.01  sec  10.7 MBytes  89.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.01  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.01  sec   139 MBytes   117 Mbits/sec                  receiver