CRS804-4DDQ Inconsistent Cage Behavior

Hardware MikroTik hardware general
1

Before I part with this CRS804-4DDQ, does anyone have a practical fix for one cage (out of 4) failing only with DGX Spark over passive DAC?

I’m considering selling my MikroTik CRS804-4DDQ because I could not get all four QSFP56-DD cages to work at 200GbE with DGX Spark/GB10 systems over passive DAC. 3 worked perfectly, one didn't... Before listing it, I would appreciate any specific guidance from people who have used this switch with NVIDIA ConnectX-7 endpoints.

I tested four GIGABYTE AI TOP ATOM / DGX Spark GB10 systems, plus an NVIDIA DGX Spark Founders Edition. All endpoints use ConnectX-7 networking.

Using the same endpoint and same passive DAC, physical cages 1, 3, and 4 establish a stable 200GbE link. Physical cage 2 does not establish a 200GbE link to any tested Spark/GB10 endpoint, but it does link when forced to 100GbE.

I tested both:

  • Amphenol NJAAKK-N911 passive DAC
  • NADDOD Q56-200G-CU1 passive DAC

The result was consistent across multiple endpoints (5, different brands) and (5 different cables--2 cable models): cages 1, 3, and 4 link at 200G; cage 2 does not at 200G but does at forced 100G.

I upgraded the switch from RouterOS 7.19.6 to RouterOS 7.24.1 and also upgraded RouterBOARD firmware to 7.24.1. The tested 200G configuration was:

auto-negotiation=no
speed=200G-baseCR4
fec-mode=fec91

As a control test, I connected physical cage 2 directly to physical cage 3 using the Amphenol DAC. Both interfaces were removed from bridge forwarding to avoid a Layer-2 loop. With both sides set to 200G-baseCR4 and fec91, they established link-ok at 200GbE, full duplex, FEC91.

This makes me think cage 2 is electrically functional, but has a lower signal margin or a passive-DAC interoperability issue specifically with these ConnectX-7/DGX Spark endpoints. I ruled out bridging, IP addressing, jumbo frames, and Linux interface state because the failure occurs before physical link establishment.

MikroTik support’s conclusion was that this is a passive-DAC compatibility issue and recommended active DAC, AOC, or optical modules/fiber. They advised that an RMA or replacement would not necessarily change the behavior because cage 2 links at 200G to another CRS804 cage.

Before I decide whether to sell the switch or buy different media, I’d appreciate any experience with:

  • CRS804-4DDQ and DGX Spark / GB10 / ConnectX-7 endpoints
  • Passive DAC models and lengths that work across all four cages
  • Any relevant RouterOS, ConnectX firmware, speed, or FEC settings
  • Whether active DAC/AOC/optics were actually necessary in a similar setup
2

Quick update: I’m still working through this with MikroTik support. They targeted a fix for 7.25beta5, which I’ve now installed for both RouterOS and switch-marvell, but cage 2 still fails with auto-negotiation and forced 200G/FEC91.

I’ve sent the comparison outputs and a fresh support file to MikroTik. I’ll update this thread when we have more guidance or a confirmed fix.

3

Same issue here — in my case it is cage 1 (interface qsfp56-dd-1-1); cages 2 and 3 link fine at 200G. I ran an isolation test that I hope helps narrow this down for you.

Environment

  • CRS804-4DDQ, RouterOS 7.24.5 stable, RouterBOARD/switch-marvell firmware 7.24.5
  • Endpoint: GB10 / DGX Spark with ConnectX-7 (QSFP-DD cage exposes two PCI functions / two MACs on one physical 200G cable)
  • Cables: NADDOD Q56-200G-CU1 (passive, 1 m) — same model as yours
  • Failing-port config: auto-negotiation=no, speed=200G-baseCR4, fec-mode=fec91 — byte-identical to the working cages

Symptom No link at 200G with the ConnectX-7 endpoint; forcing 100G links fine.

Everything I tried on cage 1 (all no-link at 200G)

  • auto-negotiation=no + 200G-baseCR4 + fec91 (baseline)
  • auto-negotiation=yes (both sides AN enabled)
  • fec-mode=fec91 / fecrs / fec74 / fec-off
  • Interface disable/enable cycle (forced re-train)
  • Host side: autoneg on/off, FEC auto / RS (the working cage negotiates Auto → RS)

Module telemetry is normal throughout: sfp-module-present=yes, NADDOD Q56-200G-CU1, 1 m, PAM4, eeprom-checksum=good, power class 1 — identical to what the working cages report.

The key isolation result I moved the same cable, same host port, same config from cage 1 to an unused cage (4). It established link-ok, 200G, FEC91 within seconds. Moving it back to cage 1: no-link again. Repeatable.

Conclusion: the cable, the endpoint's PCI function, the module EEPROM, and the RouterOS config are all proven good. The only variable is the cage itself.

Additional observation — the failing cage DOES train at 200G, it just cannot stay up Before the isolation test, the switch log showed the failing port link up at 200G twice, with two different port configs:

  • config A (defaults: AN=yes, fec-mode=auto): link up 200G, held 43 seconds, then link down
  • config B (forced: AN=no, 200G-baseCR4, fec91): link up 200G, held 2 min 36 s, then link down — permanently

Two different hold times = the teardown is not a fixed timer; it is error-driven. The link trains (so the PHY is alive) but runs at a BER the FEC cannot sustain for more than a couple of minutes, then the PHY drops it and cannot retrain. This is the classic signature of marginal receiver signal margin, not a dead port. Also observed: while the port keeps retrying 200G, the switch's phy-temperature climbs and the fans go to overdrive (visible in /system health), and the management plane (SSH) becomes intermittently unresponsive for seconds — consistent with a sustained link-training retry storm.

Why this supports your signal-margin theory Your control test showed the failing cage trains 200G against another CRS804 cage — meaning that cage's PHY/SerDes can physically do 200G. Combined with my result (same endpoint+cable works on another cage), this looks like a receiver/equalization interoperability limit between that specific cage and the ConnectX-7 transmitter over passive DAC — i.e. marginal signal integrity, not a dead port.

Questions for support

  1. Is there any per-port receiver equalization / link-training adaptation setting exposed (or planned) for the QSFP56-DD cages? A way to tune the failing cage's RX or adjust TX emphasis to recover margin would solve this without new media.
  2. If not, can you confirm whether "links at 100G but not 200G with a compliant passive DAC and a compliant 200G endpoint" qualifies as a hardware defect / RMA criterion for this model, or is it documented as expected per-cage variance?
  3. What is the current status of the fix targeted for 7.25 (mentioned in the OP)? I'm happy to test 7.25rc1 or any beta/RC you point me to — I have a reproducible 30-second test (move the cable to another cage) and can report back with monitor output.

One extra observation While the failing cage stays populated and keeps retrying 200G, the switch's management plane (SSH) becomes intermittently unresponsive for seconds at a time — possibly a link-training retry storm from the failing cage. Worth watching if you get similar reports.

Happy to provide full /export, monitor output for failing vs working cages, and endpoint ethtool output on request.

4

Hi,

I have a similar (?) issue with sfp56 cage 4 on a CRS812-DDQ.

I have many phy and crc errors which are corrected by FEC. I use four mikrotik 3m sfp28 DAC XS+DA0003 along with connectx-4 lx nics.
All sfp56 ports are errors free but the cage 4. CRS812DD, connectx-4 lx, XS+DA0003, disabled FEC: CRC e link phy error

To rule out fec failure in correcting errors you can try mlxlink in nvidia mst package. This tool can show port errors:

sudo mlxlink -d /dev/mst/mt4117_pciconf0 -c
[sudo: authenticate] Password:

Operational Info
----------------
State                              : Active
Physical state                     : LinkUp
Speed                              : 25GbE
Width                              : 1x
FEC                                : Firecode FEC
Loopback Mode                      : No Loopback
Auto Negotiation                   : ON
Tx Precoding Status                : unknown / no active link
Rx Precoding Status                : unknown / no active link

Port Info
---------
Label Port                         : 1
Local Port                         : 1

Supported Info
--------------
Enabled Link Speed                 : 0x38007013 (25G,10G,1G)
Supported Cable Speed              : 0x08001001 (25G,10G,1G)

Troubleshooting Info
--------------------
Status Opcode                      : 0
Group Opcode                       : N/A
Recommendation                     : No issue was observed

Tool Information
----------------
Firmware Version                   : 14.32.1912
MFT Version                        : 4.37.0-154

Physical Counters and BER Info
------------------------------
Time Since Last Clear [Min]        : 3.3
Effective Physical Errors          : 0
Raw Physical Errors Per Lane       : 0
Effective Physical BER             : 15E-255
Raw Physical BER                   : 15E-255
Link Down Counter                  : 0
Link Error Recovery Counter        : 0

According to Effective Physicall Errors, Raw Physical Errors Per Lane,Effective Physical BER and Raw Physical BER you can read uncorrected and corrected errors.

https://network.nvidia.com/products/adapter-software/firmware-tools/

https://www.mellanox.com/downloads/MFT/mft-4.37.0-154-x86_64-deb.tgz

I tried auto negotitions on and off (25GbaseCR), FEC auto and FC both on switch ports and nic ports.

As suggested by @drotz I am waiting for 7.25 stable release.
I have not opened a ticket yet.

Best Regards