QSFP28 breakout question around autonegotiation and force setting port speeds when broken out

Hardware
1

I’ve been configuring a CRS520-4XS-16XQ-RM, with some new servers. I’ve got each sub interface of the QSFP28-1 port set to auto negotiate. This is working, all servers connected stay connected, but I’ve noticed when the server connected to QSFP28-1-1 reboots, it’ll then take down the link for QSFP28-1-2, QSFP28-1–3 and QSFP28-1 -4, when it renegotiates I believe. Read on the help pages, it’s best to hard set the port speed to 25gbit on each sub interface. I was wondering is that the only fix, or can I also remove any advertised port speeds above 25gbit, and get the same effect, or would that not be recommended?

My thinking is I can then preset this across most of the switch sub interfaces, and I don’t have to worry if I’m connecting up a 1,10,25 gbit server, the auto negotiation will take care of it correctly.

Has anyone tried and run this setup for extended period of time, or shall I just hard set each sub interface as needed?

Many Thanks

2

How often do you change the devices connected to that QSFP28 Port?
(and with devices running the connection at a different speed)

Settings a fixed speed seems to me safer, and not that much inconvenient, unless you continuously replace the devices.

3

Yeah fixed speeds is fine really, these servers won’t change for years, but more of a set once and not worry, but easy enough to add the steps to documentation.

4

Might be best then to file a support ticket with your request.
This is a user forum.
MT staff comes by here from time to time but not everyone nor everywhere.

A support ticket will make sure at least 1 person from MT staff has to look at your request.

5

I’ve now set each port to 25G-baseCR, and the issue is sadly still happening.

/interface ethernet
set [ find default-name=qsfp28-1-1 ] auto-negotiation=no fec-mode=fec91 speed=25G-baseCR
set [ find default-name=qsfp28-1-2 ] auto-negotiation=no fec-mode=fec91
set [ find default-name=qsfp28-1-3 ] auto-negotiation=no fec-mode=fec91 speed=25G-baseCR
set [ find default-name=qsfp28-1-4 ] auto-negotiation=no fec-mode=fec91
set [ find default-name=qsfp28-2-1 ] auto-negotiation=no fec-mode=fec91 speed=25G-baseCR
set [ find default-name=qsfp28-2-2 ] auto-negotiation=no fec-mode=fec91
set [ find default-name=qsfp28-2-3 ] auto-negotiation=no fec-mode=fec91 speed=25G-baseCR
set [ find default-name=qsfp28-2-4 ] auto-negotiation=no fec-mode=fec91

Have also logged a ticket SUP-209238 for this issue

The docs say this is the correct configuration, but I’m guessing it’s a bug or I’ve missed something.

Note the -2 and -4 interfaces say their 25G-baseCR in Winbox, I think the 2nd and 4th port just inherit whatever the other sets due to how the splitting up of the port works.

6

Little Update, I’ve updated all 3 servers with the latest HP SSP, so it’s got the latest firmware for the Intel E810, the problem still persists.

Edit: SUP-209238 has been replied back to and MikroTik have managed to replicate the problem

7

Hi, we are experiencing the same issue. We are attempting to use a 100 Gbit to 4×25 Gbit breakout, but the hosts are not being split correctly.

8

Also using a E810?

9

We are using the Intel XXV710-DA2 NIC

10

Guessing you’e set the ports to 25G-CR? also I had to set the fec-mode on both sides, and the Intel auto and the MikroTik Auto, were different defaults

/interface ethernet
set [ find default-name=qsfp28-1-1 ] auto-negotiation=no fec-mode=fec91 speed=25G-baseCR
set [ find default-name=qsfp28-1-2 ] auto-negotiation=no fec-mode=fec91
set [ find default-name=qsfp28-1-3 ] auto-negotiation=no fec-mode=fec91 speed=25G-baseCR
set [ find default-name=qsfp28-1-4 ] auto-negotiation=no fec-mode=fec91

I’m still having my issue, support thinks the issue is with the e810 card, will be interesting if you get the same issue with the 710. Although not sure if you have two or more systems with a a 710

11

At the moment we have FEC on auto mode.
Also at the nic.
Every host connected to same QSFP is flapping.

/interface ethernet set [ find ]  qsfp28-4-1  fec-mode=auto l2mtu=9570  speed=25G-baseCR
/interface ethernet set [ find ]  qsfp28-4-2  fec-mode=auto l2mtu=9570  speed=25G-baseCR
/interface ethernet set [ find ]  qsfp28-4-3  fec-mode=auto l2mtu=9570  speed=25G-baseCR
/interface ethernet set [ find ]  qsfp28-4-4  fec-mode=auto l2mtu=9570  speed=25G-baseCR
12

Set the fec to fec91 on both sides, and see if that fixes it. I couldn’t pass any traffic with my setup until those matched.

OH also make sure you NVM is up to date

13

Could you let me know which breakout cable you have?

14

I have 4 of these https://mikrotik.com/product/xq_bc0003_xs_

15

XXV710-DA2 does not support FEC91 :smiley:

Proxmox always uses: CL108 RS-FEC
i40e 0000:0b:00.0 ens2f0: NIC Link is Up, 25 Gbps Full Duplex, Requested FEC: CL108 RS-FEC, Negotiated FEC: CL108 RS-FEC, Autoneg: False, Flow Control: RX/TX

16

Ah then you’ll be using fec74

ice 0000:12:00.0 nic1_1: NIC Link is up 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg Advertised: On, Autoneg Negotiated: False, Flow Control: None

oddly mine doesn’t always report all NICs coming up in the dmesg, yet they are up.. wtf