Community discussions

MikroTik App
 
ChristianRiesen
just joined
Topic Author
Posts: 8
Joined: Thu Jul 28, 2022 12:22 pm

Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Thu Jul 28, 2022 12:48 pm

Short version: Can't get an SFP28 DAC to work between sfp28-2 and an Ubiquiti switch, just no link lights up at any time and after trying many things I'm looking for ways I could further debug this to find what's wrong.

Long version: I have a CCR2004-1G-12S+2XS that has 25 gigs coming in on sfp28-1 and I would like to connect it on sfp28-2 to my network. On the Switch side, I have a USW-Pro-Aggregation from Ubiquiti. The switch then is connected to a server by a SFP28 DAC. The server has an intel card and the cable is a "LENOVO DCG 3m Passive 25G SFP28 DAC Cable" with a part number of 7Z57A03558 according to my online shop of choice. I also tried a cable form Ubiquiti, namely the UACC-DAC-SFP28-3M one.
Now the Lenovo cable works between switch and server. Unplugging it from the server and plugging it into the router does not work. I tried another Lenovo (probably, though not 100% certain that it's the same) DAC and that one didn't work either. Then I tried the Ubiquiti cable and the same state, no blinking lights, or at least static light up to see a link was established. I did try all 4 ports on the switch with the server, they all seem to operate just fine. Then I tried the ports with all three cables, and the router, and no go.
I do have a 1G sfp module in the router that connects to one on the switch at the moment, merely to have some kind of connection going and of course I removed that on both sides just to be sure, before any testing to not create a loop. There is nothing else plugged into the router, just line in, and then out only over the 1g module, or only the second sfp28 port when testing.

During testing I tried setting values on both sides by disabling auto negotation, setting the ports speed manually, enabling, disabling flow control, etc. Nothing worked.

I have not seen any errors or warnings or anything else pop up anywhere on both sides. My conclusion is I'm missing something somewhere, as in I'm not looking in the right place or not running the right commands to get more debugging output. Searching for this problem has sent me to tons of places where I can buy DAC cables or the Mikrotik router, so far none where I would be able to read up on how to debug this better.

Any pointers to where I might find more help are greatly appreciated
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Thu Jul 28, 2022 1:10 pm

1) I do not understand if you try or not between CCR and Server, if work...

2) You contact Ubiquiti forum?
 
ChristianRiesen
just joined
Topic Author
Posts: 8
Joined: Thu Jul 28, 2022 12:22 pm

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Thu Jul 28, 2022 1:25 pm

1) I do not understand if you try or not between CCR and Server, if work...

2) You contact Ubiquiti forum?
1) I have not tried that, but that combination would also not yield me anything that is useful. I will give it a shot though.

2) No, since the Switch seems to be able to talk to the Server just fine, I'm suspecting something is maybe not configured properly on the Mikrotik side (by me) but I'm unable to find any further information regarding that.
 
ChristianRiesen
just joined
Topic Author
Posts: 8
Joined: Thu Jul 28, 2022 12:22 pm

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Thu Jul 28, 2022 2:20 pm

Just to be sure tested the server <-> CR2004 connection as well, and that doesn't seem to work either. I haven't tried much more, but as it stands, that looks pretty bleak for me.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Thu Jul 28, 2022 2:29 pm

[...] that combination would also not yield me anything that is useful [...]
it doesn't matter if it's useful or not, obviously it's a test to understand where the problem is...
and the fact that you haven't thought about it and I have, means that you don't have to take things for granted,
maybe now another forum user directly writes a solution that neither of us had thought of....

The question in point 2) was a consequence of not knowing point 1)

3) Have you tried to swap sfp28-2 and sfp28-1 ???
4) To connect the server or the switch on sfp28-1 ?
5) To connect the "25 gigs coming in on sfp28-1" directly to the server or switch?
6) To connect the "25 gigs coming in on sfp28-1" on sfp28-2 ?
 
ChristianRiesen
just joined
Topic Author
Posts: 8
Joined: Thu Jul 28, 2022 12:22 pm

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Thu Jul 28, 2022 7:25 pm

[...] that combination would also not yield me anything that is useful [...]
it doesn't matter if it's useful or not, obviously it's a test to understand where the problem is...
and the fact that you haven't thought about it and I have, means that you don't have to take things for granted,
maybe now another forum user directly writes a solution that neither of us had thought of....

The question in point 2) was a consequence of not knowing point 1)

3) Have you tried to swap sfp28-2 and sfp28-1 ???
4) To connect the server or the switch on sfp28-1 ?
5) To connect the "25 gigs coming in on sfp28-1" directly to the server or switch?
6) To connect the "25 gigs coming in on sfp28-1" on sfp28-2 ?
Thank you very much for your help, I very much appreciate it. Exactly for those kinds of things I came here, to see what did I miss or because I was banging my head against the problem, not try yet that would yield some obvious answers.

I moved the 25 gig incoming module from sfp28-1 to sfp28-2 and it worked sort of. It showed an uptime of 2 seconds on the log then didn't recover anymore, even leaving it for about 5 minutes, as that sometimes can take a bit from experience (it's the ISP uplink). Switching it back also took about 20 seconds for the link to reestablish but then it worked again flawlessly on sfp28-1. Not sure what is happening there, could be another config that I might need to change as my ISP is rather vigilant and kills connections on their side quickly if they run rogue.

I have not tried switching everything around though including all the configurations. I would like to try to avoid it unless absolutely neccessary for debugging this.

The 25 gig module from the isp (that lives in sfp28-1 usually) picks up a link on both the switch and the server.

Attaching the DAC from server or from the switch to sfp28-1 doesn't give me a link either though.

I do get a connection to the outside world through the 1 gig link I make from the router to the switch with the sfp module I had laying about, so all the parts with routing etc seem to work just fine. It's really coming down to sfp28-2 not wanting to cooperate. Is there any command or log I could look at closer, an option to enable to see more output on what's going on when I try to plug the DAC in?

What I do see, on the interface, on both sides, is that it recognizes that there is a DAC, but no connection is made. See below for how that looks like in the two interfaces.
Image
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Thu Jul 28, 2022 9:09 pm

I do not have that hardware to do some experiments,
try to write one e-mail to support@mikrotik.com

:(
 
tdw
Forum Guru
Forum Guru
Posts: 1841
Joined: Sat May 05, 2018 11:55 am

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Fri Jul 29, 2022 4:52 pm

During testing I tried setting values on both sides by disabling auto negotation, setting the ports speed manually, enabling, disabling flow control, etc. Nothing worked.
Was that just speed autonegotiation? SFP28 introduced FEC to detect and correct link errors, it needs to be the same at both ends.
 
ChristianRiesen
just joined
Topic Author
Posts: 8
Joined: Thu Jul 28, 2022 12:22 pm

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Fri Jul 29, 2022 6:30 pm

During testing I tried setting values on both sides by disabling auto negotation, setting the ports speed manually, enabling, disabling flow control, etc. Nothing worked.
Was that just speed autonegotiation? SFP28 introduced FEC to detect and correct link errors, it needs to be the same at both ends.
Not certain. On Mikrotik side it shows speed and "full duplex" as a checkmark if I disable it, on the Uqibuiti side I can choose 25G FDX or auto-negotiation, see image below.

Image

Do you mean forward error correction? I don't know how that would apply here, but then I'm unsure what to do with this issue.
 
tdw
Forum Guru
Forum Guru
Posts: 1841
Joined: Sat May 05, 2018 11:55 am

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Fri Jul 29, 2022 7:31 pm

Do you mean forward error correction?
Yes

There are comments in the Ubiquiti forums suggesting it is always enabled on their devices, e.g. https://community.ui.com/questions/Feat ... 9986d4b714

There appears to be nothing in the Mikrotik help pages or wiki, but there is mention in the changelog:
What's new in 6.49 (2021-Oct-06 11:55):
*) sfp28 - changed FEC auto mode to disabled;
*) winbox - added "fec-mode" parameter under "Interface/Ethernet" menu;
and
What's new in 6.47 (2020-Jun-02 07:38):
*) sfp28 - added support for FEC modes (fec74 and fec91), with fec91 mode already enabled by default;

Edit to add:
Vendors appear to use different terminology for the two modes:
Reed-Solomon / RS-FEC / IEEE Clause 91 and Clause 108 - a ‘stronger’ FEC and is most prevalent.
Firecode / FC-FEC / BASE-R / IEEE Clause 74 - a ‘weaker’ FEC but introduces less latency on the link when used in comparison to RS-FEC.

and just as auto speed negotiation between vendors doesn't always work, neither does auto FEC mode negotiation.

Some DAC and optics will require FEC to support their full distance. From a Cisco datasheet for their 25G copper DAC 1m, 1.5m, and 2m cables do not require FEC on the host ports; 2.5m and 3m cables require BASE-R FEC (also known as FC-FEC) on the host ports; 4m and 5m cables require RS-FEC on the host ports.
 
ChristianRiesen
just joined
Topic Author
Posts: 8
Joined: Thu Jul 28, 2022 12:22 pm

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Sat Jul 30, 2022 10:36 am

Do you mean forward error correction?
Yes

There are comments in the Ubiquiti forums suggesting it is always enabled on their devices, e.g. https://community.ui.com/questions/Feat ... 9986d4b714

There appears to be nothing in the Mikrotik help pages or wiki, but there is mention in the changelog:
What's new in 6.49 (2021-Oct-06 11:55):
*) sfp28 - changed FEC auto mode to disabled;
*) winbox - added "fec-mode" parameter under "Interface/Ethernet" menu;
and
What's new in 6.47 (2020-Jun-02 07:38):
*) sfp28 - added support for FEC modes (fec74 and fec91), with fec91 mode already enabled by default;

Edit to add:
Vendors appear to use different terminology for the two modes:
Reed-Solomon / RS-FEC / IEEE Clause 91 and Clause 108 - a ‘stronger’ FEC and is most prevalent.
Firecode / FC-FEC / BASE-R / IEEE Clause 74 - a ‘weaker’ FEC but introduces less latency on the link when used in comparison to RS-FEC.

and just as auto speed negotiation between vendors doesn't always work, neither does auto FEC mode negotiation.

Some DAC and optics will require FEC to support their full distance. From a Cisco datasheet for their 25G copper DAC 1m, 1.5m, and 2m cables do not require FEC on the host ports; 2.5m and 3m cables require BASE-R FEC (also known as FC-FEC) on the host ports; 4m and 5m cables require RS-FEC on the host ports.
I wish I could get a 2 meter cable, or 1.5 as that's more what I need here, but I can only get 1 and 3 meters it seems.

I tried digging into the switch side on this topic and it's a desert concerning that. But this is now a new parameter I can look into on the router and see if that changes anything, thank you.
 
ChristianRiesen
just joined
Topic Author
Posts: 8
Joined: Thu Jul 28, 2022 12:22 pm

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Sat Jul 30, 2022 1:35 pm

So far I have not been able to change anything regarding FEC on any side. I can't find it anywhere in the Mikrotik side and I can find a command to enter on Ubiquiti side, but not where to enter it properly. It says something about "(UBNT) (Interface 0/29)#no fec" which I understand is the prompt up to # and behind it the command, but that seems to be running within some UI shell that I have not yet found how to start from SSHing into the switch.

I dropped a message to both MikroTik and Ubiquiti support now in hopes to figure this out at one point.
 
ChristianRiesen
just joined
Topic Author
Posts: 8
Joined: Thu Jul 28, 2022 12:22 pm

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Wed Aug 03, 2022 3:01 pm

Support at MikroTik helped me along and now it works. Here my steps for posterity and in case someone else runs into the same issue.
1. Update (packages) of the router to 7.4. If you can't see them, choose "upgrade" channel, that should give you the option. Packages 4.x wont do
2. After the update is done and rebooted, upgrade firmware by running /system routerboard upgrade on the console. Reboot again after that!
3. Set the port on the switch under port profile overrride in the link speed field to 25G FDX. This will lock all SFP28 ports to be 25G only, the interface will also inform you. LLDP MED and Spanning Tree Protocol enabled, the rest disabled (default state on my machine).
4. For the sfp28-2 port on the router, set the following. Rate Select: high. FEC Mode: fec74 (important, no other setting seems to give a link). Auto negotiation: unchecked (port will report failure to autonegotiate otherwise). Tx and Rx Flow Control, both: Off. Speed: 25Gbps. Full Duplex: checked.

This finally gave me a link and so far it has been stable without the slightest hiccup in the past 2 hours.

Thanks a ton to the support for leading me through this, to get to a working solution in the end!
 
User avatar
doridian
newbie
Posts: 26
Joined: Mon Jan 20, 2014 3:08 pm

Re: Debugging SFP28 DAC connection between CCR2004 and Ubiquiti switch

Sat Jan 07, 2023 8:30 am

Support at MikroTik helped me along and now it works. Here my steps for posterity and in case someone else runs into the same issue.
1. Update (packages) of the router to 7.4. If you can't see them, choose "upgrade" channel, that should give you the option. Packages 4.x wont do
2. After the update is done and rebooted, upgrade firmware by running /system routerboard upgrade on the console. Reboot again after that!
3. Set the port on the switch under port profile overrride in the link speed field to 25G FDX. This will lock all SFP28 ports to be 25G only, the interface will also inform you. LLDP MED and Spanning Tree Protocol enabled, the rest disabled (default state on my machine).
4. For the sfp28-2 port on the router, set the following. Rate Select: high. FEC Mode: fec74 (important, no other setting seems to give a link). Auto negotiation: unchecked (port will report failure to autonegotiate otherwise). Tx and Rx Flow Control, both: Off. Speed: 25Gbps. Full Duplex: checked.

This finally gave me a link and so far it has been stable without the slightest hiccup in the past 2 hours.

Thanks a ton to the support for leading me through this, to get to a working solution in the end!
Thank you so much for posting your steps. I have been fighting with the exact same device combination all evening and fec74 fixed it immediately! :D

Who is online

Users browsing this forum: Matta and 20 guests