CRS812 DDQ v7.23.2 and RDMA limited on 25G rate

RouterOS General
1

Hello,

I am trying to configure RDMA on a MikroTik 812. While I can establish the connection without issues, I cannot track down why the throughput is hard-capped at 25 Gbps.
Setup details:

  • Host 1 (ESXi 8.0): Mellanox 100GbE NIC (QSFP28) connected to port qsfp56-dd-1-1 via a DAC breakout cable (2x 100GbE QSFP28).
  • Host 2 (TrueNAS Enterprise): Mellanox 100GbE NIC (QSFP28) connected to port qsfp56-dd-2-1 via a standard 100GbE QSFP28 DAC cable.

The issue and troubleshooting steps:

  • Baseline configuration (attached below): RDMA throughput is strictly capped at 25 Gbps.

  • Disabling IP DSCP Mapping: Traffic defaults to the primary queue. Read speeds jump to the full 100 Gbps, but writes predictably drop to ~40 MB/s with latency spiking out of control due to packet drops.

  • Changing the Lossless Traffic Class: If I reassign the Lossless Traffic Class from 3

    to TC 2 or TC 7, even the primary queue gets throttled to 25 Gbps.

It behaves as if RouterOS/the switch ASIC is applying the QoS shaping/policing to a single 25G lane instead of the aggregate 4x25G (100GbE) link.
Has anyone encountered this behavior or found a way to resolve it?

My config:

26082026.rsc (6.5 KB)

2

After upgrading firmware to 7.24.1 and software to 7.24.1 is much better - on the same configuration.:slightly_smiling_face: :upside_down_face:

I lost three days on this :confused:

I suspect the root cause was a firmware/software incompatibility—the firmware was older than the running software.

3

image

It is strongly recommended to upgrade the bootloader after upgrading RouterOS. To upgrade the bootloader, execute command "/system routerboard upgrade" in CLI, followed by a reboot. Alternatively, navigate to the GUI System → RouterBOARD menu and click the "Upgrade" button, then reboot the device.