Community discussions

MikroTik App
 
mwastart
just joined
Topic Author
Posts: 3
Joined: Thu Sep 01, 2022 10:43 pm

CRS518-16XS-2XQ to Chelsio T62100-CR via qsfp28 DAC?

Thu Sep 01, 2022 10:48 pm

I got a CRS518-16XS-2XQ a couple days ago and I've been able to use sfp28 and sfp+ DAC cables without any problems. I just got a Chelsio T62100-CR and I am unable to connect it to the qsfp28 ports with a DAC reliably.

If I connect the two qsfp28 ports together on the switch, in a loop, then it works at 100gb. Same, if I connect the two ports on the NIC, in a loop, it connects. Green lights all around.

But if I try to go from the NIC to the switch with either of the 2 DAC cables that I have... no dice. Ever so often I'll get some blinking lights but nothing solid. Around the same time I get an event warning logged for "CH_WARN_RATELIMIT: Port 0 link down, reason: Link Down". I'm running Windows server 2022.

I've tried forcing the link speed to 100gb and that doesn't help. I'm using this cable:
100G QSFP28 DAC Cable - 100GBASE-CR4 QSFP28 to QSFP28 Passive Direct Attach Copper Twinax Cable for Mellanox Ethernet MCP1600-C002, 2-Meter(6.5ft)

I know my DAC says mellanox, and I'm using a chelsio card, but using the DAC cables in a loop works. My other sfp28 and sfp+ DAC are also coded as Mellanox and they work perfect on this switch. I was able to get 1 pair of connections working at 100gb once randomly. But then I rebooted the CRS518 and now I'm stuck with them not working again. I feel like I'm missing something simple here. Any advice?
 
mwastart
just joined
Topic Author
Posts: 3
Joined: Thu Sep 01, 2022 10:43 pm

Re: CRS518-16XS-2XQ to Chelsio T62100-CR via qsfp28 DAC?

Sat Sep 03, 2022 9:25 pm

I just installed RouterOS version 7.6beta4 and it looks like my problem is solved. Both QSFP28 ports are coming up reliably as 100gb. Thanks Mikrotik!
 
mwastart
just joined
Topic Author
Posts: 3
Joined: Thu Sep 01, 2022 10:43 pm

Re: CRS518-16XS-2XQ to Chelsio T62100-CR via qsfp28 DAC?

Sat Sep 03, 2022 9:57 pm

Nevermind... I posted too soon. The connections work more reliably... but the connections aren't perfect yet. Previously, the switch ports would get stuck in a weird loop where it will show Vendor information for the module for a couple seconds and then it will show no Vendor information for a couple seconds and flip-flop back and forth without getting connected.

With Router OS v7.6beta4, it will now fully establish the connection at 100gbps but there is still something not right. In the space of just a couple minutes, port qsfp28-1-1 reported 4 link down events and qsfp28-2-1 reported 1 link down event. Router OS 7.5 would never count those link down events. Also, I tested hot swapping the QSFP28 DAC cables and they never come back up. I have to reboot the switch to get the connections to work again.

I did get port qsfp28-1-1 to work once for several days on Router OS v7.2.3. I feel like Router OS v7.6beta4 is a big step in the right direction... but there is still something wonky with getting the connections to establish correctly over QSFP28. Are there any reports or something I can run that would help Mikrotik to debug this issue? I'd like to help more if I can. Thanks!

Who is online

Users browsing this forum: Majestic-12 [Bot] and 68 guests