I have connected 2 computer, each one with a dual sfp28 port connectx-4 lx, connected to port 1 e 2, computer A, and port 3 e 4, computer B.
Port 1 and 3 on PCs are for standard traffic. Port 2 and 4 are configured on PCs and switch for RoCEV2.When I test RoCEV2 connection between computer A and B I found several CRC and link phy errors only on port 4.
Errors appear also if monitoring nic port connected to switch port 4 "without" traffic.
Errors are read using mellanox utility mlxlink
I switched cable and computer to switch port, but errors on port 4 reappear.
I have FEC disabled but I read on IEEE 802.3by the FEC is mandatory for long dac cables, and that 3meters DAC cables (as XS+DA0003) is reported as borderline.
CRS812DD configuration is done accordind to Quality of Service | RouterOS Manual and mellanox nics according to that.
Did I make a mistake by not setting the FEC?
Someone else with a (+/-) similar issues or configuration?
I have swaped nics, switch ports, cables and computers (!!) and the result is always switch port 4 with some errors.
Enabling FEC all errors are corrected and reported as corrected.
I tried to set switch port 4 and nic port to 10Gbps (10GBaseCR on switch) and there are not any errors with the same 3m XS+DA0003 dac (note this dac is sfp/sfp+/sfp28 compatible).
@drotz Thank you for advice, I definitely give ROS 7.25 a try (I would prefer the stable version, but if it isn't released "soon," I'll switch to the RC) but seems an hw issue (sfp28 cage?)
Again thank you
You can see if the module you inserted into the sfp28 cage works if you get your phone, open the camera on the phone and point the camera on your phone at the sfp28 TX part, you should see a dot (laser), if there's a laser, means it powers up.
DO NOT LOOK AT IT WITH YOUR OWN EYES (Even tho SM is the more dangerous one and more capable of doing damage to retinas and MM tends to be a bit less dangerous, regardless don't do it.)
So if the module powers on by itself, the only other thing that can be is that the spf cage doesn't recognise the module, which is extremely rare with Mikrotik imo, or you got a "bad quality" module that doesn't have a DWDM chip inside for readouts that Mikrotik can read info about the module from.
It's highly unlikely to be a HW issue unless something was under very high temperature for a long time and cooked itself (bad ventilation, rj45 modules..).
An update: disabling "auto negotiation" both on CSR812 port and connectx-4 lx port and setting 25G baseCR, seems solve the errors issue (no errors during 4.30 hours monitoring interval). @drotz Unfortunately I dont have optical modules, only mikrotik XS-DA0003 "combo" dacs (sfp/sfp+/sfp28 compatible) and cisco sfp+ dacs.
For a 3 m 25G passive DAC I would first try it with FEC enabled, preferably RS-FEC/FEC91.
IEEE does allow 25G over 3 m copper without FEC, but only if the cable/channel meets the stricter no-FEC requirements. Since you already confirmed that enabling FEC corrects the errors, that suggests the link is simply marginal without FEC rather than necessarily indicating a faulty CRS812 port.
I’d only investigate the port further if you still get uncorrectable FEC errors, CRC errors, link drops, or if that port shows a dramatically higher corrected-error rate than the others with the same cable/NIC setup.
Can't talk CRS812DD but i'm using two set CRS520-4XS-16XQ (ros 7.21.5 + 7.21.4) with XS-DA0003 cables. They are connecting CX4 interfaces and no problem observed. Auto neg. enabled and fec auto (at status page FEC says off) gives 25Gbps per port. Also using auto neg disabled, port speed 10 Gbps manually set with intel x520, no problem there as well.
Edit: Sorry cable was DQ+BC0003-DS+ not XS+DA0003 so whole writings inappropriate... I'll try to find XS+DA0003 try and report later.
