I got a CRS518-16XS-2XQ a couple days ago and I've been able to use sfp28 and sfp+ DAC cables without any problems. I just got a Chelsio T62100-CR and I am unable to connect it to the qsfp28 ports with a DAC reliably.
If I connect the two qsfp28 ports together on the switch, in a loop, then it works at 100gb. Same, if I connect the two ports on the NIC, in a loop, it connects. Green lights all around.
But if I try to go from the NIC to the switch with either of the 2 DAC cables that I have... no dice. Ever so often I'll get some blinking lights but nothing solid. Around the same time I get an event warning logged for "CH_WARN_RATELIMIT: Port 0 link down, reason: Link Down". I'm running Windows server 2022.
I've tried forcing the link speed to 100gb and that doesn't help. I'm using this cable:
100G QSFP28 DAC Cable - 100GBASE-CR4 QSFP28 to QSFP28 Passive Direct Attach Copper Twinax Cable for Mellanox Ethernet MCP1600-C002, 2-Meter(6.5ft)
I know my DAC says mellanox, and I'm using a chelsio card, but using the DAC cables in a loop works. My other sfp28 and sfp+ DAC are also coded as Mellanox and they work perfect on this switch. I was able to get 1 pair of connections working at 100gb once randomly. But then I rebooted the CRS518 and now I'm stuck with them not working again. I feel like I'm missing something simple here. Any advice?