Hello,
I hope you are well.
I’ve recently upgraded my homelab server and installed a Mellanox ConnectX-4 Lx MCX4111A-ACAT (SFP28 capable) to future-proof.
As my Mikrotik CRS309-1G-8S+IN is only 10G, I’ve used an SFP+ 10GbE transceiver on both ends (cannot use fibre as the cabling is not fibre yet).
The card was initially on 14.24.1000 firmware (https://network.nvidia.com/support/firmware/connectx4lxen/) from 2 December 2018. There were problems bringing the link up. As far as I understand this is a known issue - I needed to switch auto negotiation off, force 10G full-duplex on the server side without touching anything on the Mikrotik side. After that the link worked flawlessly - I stress-tested it for hours at peak 10G traffic.
Then I tried to update firmware (the card is actually still supported - the newest firmware 14.32.1908 is from 18 September 2025) - and the card refuses to work at all. mstlink says the link is up and there are no errors shown anywhere, dmesg shows Link ACTIVE and no errors as well. But no traffic flows at all.
Firmwares past 14.25.1020 from 2 May 2019 have the following known issue in Release notes (e. g. https://docs.nvidia.com/networking/display/connectx4lxfirmwarev14321908/known+issues):
Description: In Ethernet mode, at 10/40GbE speeds, only NO-FEC in Force mode is supported. Other user configurations are overridden.
Workaround: N/A
Keywords: Ethernet, 10GbE, 40GbE, RS-FEC
Discovered in Version: 14.25.1020
I read the known issue as the FEC is mode is force-disabled for me and I do not need to do anything. Despite that, I tried forcing FEC off using different methods (using mstconfig --dev ... set PHY_FEC_OVERRIDE_P1= with different values; using ethtool --set-fec) and on both sides (/interface ethernet set [ find default-name=sfp-sfpplusX ] auto-negotiation=no fec-mode=off speed=10G-baseT [as far as I understand fec-mode=off is default anyway and non-applicable to 10G-baseT anyway]) to no avail.
By bisecting which version is the last that works, it turns out the only firmware version the card still works on is 14.25.1020 - one past the initial 14.24.1000 and allegedly the version where the known issue should already manifest itself. But it does not.
First thought is that something is wrong with the transceiver (especially considering mine are non-branded). But I have a few of them from different batches (e.g. some of them are 1G/2.5G/5G/10G and some are 2.5G/5G/10G), I tried switching them around - and all of them work flawlessly on firmware versions 14.24.1000 and 14.25.1020 and do not work on any version past - so it’s unlikely the transceivers are literally broken. And even on firmwares past 14.25.1020 the link looks up per mstlink.
Could it be that the newer firmwares started being that much more picky about non-branded transceivers?
I’m new to server-grade network cards (previously I had a Supermicro server with an Intel 10GbE card but I never did anything with it - it just worked), so maybe I’m missing anything obvious to a (network) administrator who deals with these cards daily?
Am I misinterpreting the known issue?
Or is this Mellanox card known not to work in 10GbE mode past 14.25.1020?
If you have any thoughts what else I might debug, please, let me know.