QSFP+ 40 Gbe DAC only works if I force it to 10 Gbps

[zmarty]

Hardware

• Switch: MikroTik CRS326-24S+2Q+RM
• QSFP+ 40Gbe DAC cable: FS 3m (10ft) Generic Compatible 40G QSFP+ Passive Direct Attach Copper Cable
• Network card: Mellanox ConnectX-3 MCX353A-FCBT
• Connection is (Switch) <— DAC —> (Network card, PC with Ubuntu 20.04)

Software

• RouterOS for MikroTik CRS326-24S+2Q+RM: v6.47.2
• Network card firmware (latest available): 2.42.5000

Symptoms

• DAC cable gets recognized, but there is no link

What I tried in RouterOS to fix the problem

• Turned off auto negotiation
• Set the port to 40 Gbe manually on both ends (disabled auto negotiation and set the speed manually)
• Set the port to 10 Gbe manually on both ends (disabled auto negotiation and set the speed manually)

It works but only if I set it to 10 Gbe. In the Interface list I see only the first channel out of 4 is used when I transfer files, so I only hit max 10 Gbps.

Any ideas?

[zmarty]

Seems to be a compatibility problem with the FS.com DAC cable, not with DAC cables in general.

I bought a Wiitek cable and that one got recognized immediately as 40 Gbps and it works fine.

I sent this information to FS.com too, but my guess is Mikrotik needs to look at this.

Since I have a cable that works, I am willing to ship the FS.com DAC cable with issues to Mikrotik for testing. Let me know if you need it.

[iamspartacus]

Seems to be a compatibility problem with the FS.com DAC cable, not with DAC cables in general.

I bought a Wiitek cable and that one got recognized immediately as 40 Gbps and it works fine.

I sent this information to FS.com too, but my guess is Mikrotik needs to look at this.

Since I have a cable that works, I am willing to ship the FS.com DAC cable with issues to Mikrotik for testing. Let me know if you need it.

Hey zmarty I’m wondering if you did any speed/performance testing between two devices connected to your switch and those Mellanox cards. I have the same NICs and I’m getting terrible performance results connecting two Linux servers through my CRS354 QSFP+ ports. The cables I’ve tried (3 sets) all connect fine and the link shows as 40Gb (with auto negotiation off and speed set manually to 40Gb) but the performance is less than 1Gbps. I’m wondering, if you’ve seen good performance, if you’d be willing to DM me and share some insight into your config or look at mine and see what I might be missing. Thanks!