QSFP+ 40Gbps crs326-24s+2q+rm

RouterOS General
1

Hi all,

I have recently bought the crs326-24s+2q+rm and connected a Freenas and ESXi host with QSFP+ 40Gbps cables. (Have 2x Mellanox and 2x Mikrotik cables). In the servers i use the Mellanox Connectx-3 with latest 2020 firmware. And yes, they are in ETH modus.

With SwOS i can utilize the 40Gbps connection. The problem i have is with RouterOS. I only get 100Mbps traffic speed, maybe for a fraction of a second 200Mbps. No where near the 40Gbps. What am i missing?

RouterOS v6.48.6 (long-term) - copy action between 2 datastores on the freenas server and dd test within linux vm
Autonegotiation = off and fixed on 40Gb. (=on gives same result)
Flow control off
Interfaces are added to bridge
Everything else we left default
Tried mikrotik and mellanox cables

Direct connection freenas ↔ esxi utilizes the 40Gbps
Freenas ↔ SwOS ↔ Esxi utlizes the 40Gbps
Freenas ↔ RouterOS ↔ ESXi 100-200Mbps

I want to use RouterOS, because with SwOS there is the issue that when i reboot a server, the link does not come up. disabling/reenable auto negotiation or a reboot, fixes the connection. This is not a stable config and after a year it still hasn’t been fixed. It should be fixed in RouterOS and i haven’t seen this issue with RouterOS.
And final limitation of SwOS, there is no option to configure jumbo frames. And with QSFP and SFP+ ports, you definately want jumbo frames. (2,5x faster)

So i want to use RouterOS and program the switch cpu only. I hope somebody has an answer to this problem.

Attachments:
SwOS - 2GB ISO file copy between 2 datastores - 9GB
RouterOS - same ISO copy to same datastores - 130Mbps
4BC117DC-CCEF-4868-AFD3-AB9AEDECA975.jpeg
5954A4BB-F4DD-4430-8E77-916466E45BB3.jpeg

2

Tried stable release 6.49.2 again, but this time it seems to work ok. Performance is as expected.

SOLVED!

3

Thanks You for sharing The solution

4

Well that was a short Eureka. After booting everything up again, i experienced the same issue with RouterOS 6.49.2. Speed is around 100Mbps.
Switching to SwOS gives me full speed. I’m lost at what is causing this. Enabling/disabling autonegotiation does not work and i haven’t changed a thing since yesterday after the upgrade.

If someone has a tip to push me in the right direction, please do.

5

Make sure you only have one bridge defined and it has the H designated to show it’s hardware based, if you define more than one bridge the second one will be using the CPU of the switch which is not capable of much traffic.