Community discussions

MikroTik App
 
Frostbyte
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 92
Joined: Mon Dec 25, 2017 1:42 am

CRS328-24P-4S+ & Intel X520-DA2 Port Flapping

Fri Apr 07, 2023 6:08 pm

Greetings,

I have a CRS328-24P-4S+ which I am using for a couple of years now; so far I've been satisfied with said device, but recently I'm facing a really annoying behavior.
I've got an ESXi 6.7U3 machine with an Intel X520-DA2 card installed onto it and I'm trying to achieve 10G link(s) with my CRS328 device.

I am using MikroTik S+85DLC03D SFP+ modules on the CRS328 side and Official Intel E10GSFPSR SFP+ modules on the X520-DA2 side.

Initially, my CRS328 device was left on RouterOS 6.49.7 (the reason being that I wasn't sure which, if any, console commands have changed on RouterOS 7.*).
The ESXi machine recognized the Intel X520-DA2 card from the very first time I installed it, utilizing the ixgben driver 1.8.7.

Under these versions, the ESXi showed "Link Down" and the CRS328 would not switch the corresponding SFP+ interface(s) into running mode (also had the "Rx Loss" flag ticked).
After a little bit of research, I saw that many people were mentioning that the Intel X520-DA2 cards do not support Auto Negotiation; therefore I configured fixed 10G speeds, where applicable, on both the ESXi and the CRS328 sides. However, the behavior remained exactly the same.

At that time, I had the idea to eject and re-insert the SFP+ modules on the X520-DA2.
Then, I noticed that the "Rx Loss" flag on the corresponding SFP+ interface(s) went away on the CRS328 and the interfaces started to flap (remaining down for about 5 seconds, then switched to running for half a second, then down again for another 5 seconds and so forth).
On the ESXi side, even for that split second of runtime, I was still getting "Link Down".

Not wanting to give up just yet, I decided to update the ESXi ixgben driver (as I saw that this was a resolution for some people on Reddit).
According to VMware's Compatibility Guide, the latest available ixgben driver for ESXi 6.7U3 was 1.14.2.0, thus I procured it and performed the due procedures.
After said ixgben driver update, the "Rx Loss" flag on the corresponding SFP+ interface(s) went away permanently on the CRS328 side, but the interfaces still retained their aforementioned flapping behavior exactly as before.
This time around, however, on the ESXi side I would see "10000 Mbps, full duplex" for that split second of runtime.

Next up I wondered; what if this is a CRS328-specific issue (having stumbled across a few Reddit and forum posts, which mentioned switching the device into SwOS mode did the trick, plus a MikroTik forum post reporting severe flapping behavior between CRS328 and CRS317 devices).
Within the mountains of information which I perused, I saw that for some people updating to RouterOS 7.* resolved the issue, since it was reportedly fixed(?) somewhere between RouterOS 7.3 and 7.4.

Before executing any update, however, I decided to do a little test with my RB4011iGS+5HacQ2HnD device (also on RouterOS 6.49.7) that I also had in hand.
I thought that it wouldn't hurt, plus it would provide me with more information about this conundrum that I found myself into.
Et voila! The RB4011 device achieved a perfectly stable 10G link with the X520-DA2, no fuss, no muss, everything as expected on the ESXi side as well.

So, I decided to update my CRS328 device into the latest available Stable RouterOS release, which was 7.8.
After performing the RouterBOARD and device firmware update on the CRS328 device to 7.8, the only thing that changed was the flapping behavior becoming faster (would stay down for a second, then switch to running for another second, then down again for another second and so forth).

I would also like to mention that in every step during the troubleshooting, a fixed 1G speed configuration between the CRS328 and the ESXi would also work flawlessly, with no evidence of flapping whatsoever.

Ideally, I would've liked to open a ticket to MikroTik support about this directly, however their Jira doesn't allow me to login (always stating that the CAPTCHA I entered was invalid).
If anyone from MikroTik support happens to come across this thread, please also advise on how to resolve this as well, so I can provide the due supout.rif files as well.

Thank you and apologies for the lengthy analysis.

PS: I also have a Synology DS1819+ with an Intel X710 card and Official Intel E10GSFPSR SFP+ modules that works flawlessly (on 802.3ad 2x10G) for more than two years with the same CRS328 device and two more MikroTik S+85DLC03D SFP+ modules.
 
User avatar
mhenriques
newbie
Posts: 47
Joined: Sat Mar 23, 2019 8:45 pm
Location: BRAZIL
Contact:

Re: CRS328-24P-4S+ & Intel X520-DA2 Port Flapping

Thu Jul 20, 2023 8:25 pm

Did you ever get this 10Gbit connection working on CRS328-24P switch?

Who is online

Users browsing this forum: Bing [Bot] and 25 guests