I got my workstation (with Mellanox Connectx-2) connected per DAC to the CSS326. WIthin SWOS it tells me “10G”, but if I try to copy something to the servers (1Gbit) I only get 30-180mbit traffic speed. If I connect the workstation with a 1Gbit cable instead, I get around 980 mbit/s. If I connect the Workstation directly to one of the servers per DAC, then I get the full 10gbit link. What’s the problem?
Edit: I made some tests:
It doesn’t matter which Server/PC is using the Mellanox as sending-device (I tested both cards): it will only be between 20-400mbit/s, depending on the OS (linux is faster than windows in sending). Receiving is about 980mbit/s (full Gbit; I only have one DAC right now).
I found someone in a different forum who has the same problem with 2* Mellanox Connectx-2 and the CSS326 with DAC.
I’m also suffering some problems with CSS326 and SwOS 2.5. Downgrade OS to solve problem? That sounds wired. I was expecting SwOS are rock solid like RouterOS, but seems I’m totally wrong.
Seems Mikrotik official not care much about SwOS. Is CSS326 ready for production usage? What OS version should I use?
Yes, I’m not the only one with SFP-> 1Gbit transfer problems. Downgrading solved the problem. 2.3 and 2.4 are running fine. So far it is a very good switch!
Same problem here as well.
While TCP traffic between 1G ↔ 10G is reasonably Ok - UDP traffic has huge losses and out-of-order issues.
Streaming in LAN (NVIDIA GameStream) is unusable with SwOS 2.5 and 2.6.
We are using two CSS326-24G-2S+ with both firmware 2.7 and we can’t get above 170mbps any ideas?
Traffic path:
Server → switch → CSS326-24G-2S+ → SFP+ → SPF+ → CSS326-24G-2S+ → Switch - > Server
We’ve recently purchased two CSS326-24G-2S+ switches.
They have a 10gbps connection between the two switches / sites. (two datacenter sites)
The CSS326-24G-2S+ switches are connected by 1gbps coper to our switch infrastructure in either site.
When we generate traffic from one site through the other site so:
Server → switch → CSS326-24G-2S+ → SFP+ → SPF+ → CSS326-24G-2S+ → Switch - > Server
We get stuck at 170mbps.
This is very strange.
What do we need to do to be able to get around 1gbps of traffic through from 1 site to the second site?
Yes, just noticed the same bad SFP+ → 1GB performance bottleneck… SMB performance from Server 2016 → Windows 10 even had as low as 600 KB/s from the 10GB-Link.
It’s also on current 2.7 firmware. Downgraded to 2.3, speed returned to normal.
I sometimes see the same thing, but it’s intermittent for me. Sometimes the speeds are just fine.
MikroTik, let me know if you need log files etc. to help your investigations.
Hello, since v2.7 we have had difficulties in repeating your reported performance issues on CSS326.
Do all of you experience performance problems only with SMB transfers on CSS326 with SwOS 2.7?
Does any other bandwidth testing method also give the same bad results?
For us it’s usually SMB, we have had to return to 2.3.
We thought 2.7 had fixed the issues of 2.5 and 2.6, but 2.7 drops to back crawling without any rhyme or reason, so we had to go back to 2.3.
I can confirm the performance problem since 2.5. We read half the speed from a 10GB port than we can write. E.g. reading is about 50MByte/s, writing is > 100MByte/s.
This is independend on the protocol. Appears on SMB, FTP, SFTP.
According to Mikrotik support the buffer handling changed since 2.3/2.4.
SwitchOS 2.3 is not the solution as it brings the switch into an endless reboot loop if 10G DAC are connected while boot.
Unfortunately the 317/326 switches are not applicable for us because of the performance reduction.
Could the performance issues be caused by bursty traffic and too small buffers in the switch? So the traffic is below 1 Gb/s on average, but comes in short 10 Gb/s bursts that overflow a too small buffer resulting in packet loss? The issue is not theoretical, I’ve seen it already a few times in cases like traffic enters a gigabit switch port and leaves through a 100 Mb port (or a 300 Mb microwave link). Software queues in RouterOS are usually large enough, but hardware (such as a switch chip) is limited, and higher speeds need larger buffers (but not too large - bufferbloat is bad too). Rule of thumb for the best throughput of a TCP flow is the RTT multiplied by the data rate. So the issue gets worse when traffic comes over the Internet and/or over wireless networks (often much longer RTT compared to a wired LAN). What is the hardware buffer size in CRS/CSS 3xx series switches? CPU in a CRS is too weak to handle much traffic, so larger queues possible in RouterOS don’t help. This is important, the switch buffer size should be known and also be tunable. Good example of (more expensive) hardware that got it right is NEC iPasolink microwave links - default queue of 64 KB is too small for a 300 Mb/s link, but can be increased to 1024 KB which works much better when moving traffic at such speeds from the Internet to a few hundreds of customers on my network (mostly wireless).
Since we upgraded the connection between the CRS317 switches to 10G, any of the customers connected to any of the CRS317-1G-16S+RM switches would have 10-12-13 Mbps download speed and 500 Mbps-900 Mbps upload speed (depending of the speedtest server). I’ve noticed that in a speedtest, the further (geographically) the test server, the more that the speed on download degrades. When the speedtest server is closer (200 Km away for example), the download speed is 100 Mbps vs 850 Mbps normally, and further more, at 2000 km away, is 12 Mbps vs 750 Mbps. If we make a speedtest in the US, the speed would degrade even more, would be for example 5 Mbps instead of 500 Mbps. I don’t understand how could this happen and how the distance could determine the speed of a connection through such a switch. I attached some examples.
Stupid question: where can i find the 2.3 swos firmware? On the Mikrotik website there is only an archive for routeros versions, not for the swos. And google finds only mikrotik’s website which has no archive for older swos
Can confirm anything after SwOS 2.4 does not work well at all when receiving data on a 10G port and sending it out on a 1G port.
There is significant packet re-ordering, using default settings, both with and without flow control enabled link partners. I don’t expect all that much at this price point but delivering packets in order is a basic requirement for a switch.
on 1G port: iperf3 -s
on 10G port: iperf3 -c <IP of 1G client> -i 1
If you do a packet capture the packets are all jumbled up (very visible in wireshark)
Note: the actual performance impact of packet re-ordering varies between operating systems, configurations and RTT of the TCP connection. And some benchmarks may not “see it” if they use a ton of paralell TCP connections.
