Hi everyone!
Recently I bought an Intel X520-DA2 to put on my x86 server that is running RouterOs v7.8, because I was reaching the limit of 1GB on my ethernet interfaces.
Changed it up and I found some strange things going on:
I boot up RouterOs and the two interfaces were displaying, so far so good, then I notice that the L2MTU value of both 10gb interfaces were 0, I don't know if makes any difference but I found it strange because my other ethernet port has recognized the L2MTU value of 9200.
Moving on, I put traffic on those 2 10GB ports, it worked, but at some time, both ports stops responding, I have to disable and enable to make it work back.
I thought it may be the power supply from the computer that wasn't handling the power of the NIC, I changed it, and the problem still there.
Anyone have an idea of how I can fix this?
Re: Problems with X520-DA2
My org uses X520-DA2 in some x86 routers, and other than a couple of quirks that I haven't gotten around to posting about yet, they have been working surprisingly well for us.
The concept of L2MTU is a RouterOS-specific thing, and has to be explicitly supported by the underlying driver for the interface. RouterOS includes many drivers for third-party network interfaces, but MikroTik does not necessarily go to the trouble of extending/modifying every driver they include from Linux for every third-party interface to support all custom RouterOS "extensions". In the case of interfaces such as the X520 whose drivers have no concept of "L2MTU", you should set normal MTU to be at minimum whatever you would need your minimum L2MTU to be.
I should mention that we are currently running X520 on RouterOS 6.49, bare-metal (not CHR). We have not experienced the lock-up/stopped responding issue you describe. So perhaps there is a RouterOS 7 specific problem there. Or if you are using CHR, perhaps there is an issue with your underlying VM hypervisor, and/or if you are exposing them to the RouterOS CHR guest with SR-IOV, perhaps there are certain bugs or issues related to that as well (either on the hardware, hypervisor, or RouterOS side). Since we run ours without virtualization, it's possible that we have simply avoided all of those issues (and with us being on ROS 6 still, which doesn't support SR-IOV anyway, we wouldn't have seen it if that is potentially a variable in the equation).
The concept of L2MTU is a RouterOS-specific thing, and has to be explicitly supported by the underlying driver for the interface. RouterOS includes many drivers for third-party network interfaces, but MikroTik does not necessarily go to the trouble of extending/modifying every driver they include from Linux for every third-party interface to support all custom RouterOS "extensions". In the case of interfaces such as the X520 whose drivers have no concept of "L2MTU", you should set normal MTU to be at minimum whatever you would need your minimum L2MTU to be.
I should mention that we are currently running X520 on RouterOS 6.49, bare-metal (not CHR). We have not experienced the lock-up/stopped responding issue you describe. So perhaps there is a RouterOS 7 specific problem there. Or if you are using CHR, perhaps there is an issue with your underlying VM hypervisor, and/or if you are exposing them to the RouterOS CHR guest with SR-IOV, perhaps there are certain bugs or issues related to that as well (either on the hardware, hypervisor, or RouterOS side). Since we run ours without virtualization, it's possible that we have simply avoided all of those issues (and with us being on ROS 6 still, which doesn't support SR-IOV anyway, we wouldn't have seen it if that is potentially a variable in the equation).
Re: Problems with X520-DA2
I actually experienced this on a Dell R730 with logical processors/hyper threading enabled. I am running 7.15.2. When I disabled logical processors in BIOS it became stable. I do still get some RX-ERRORS that I am confused about, it happens on all ports regardless of what SFP is plugged in it seems. Performance still seems pretty good so I wonder if these errors are just certain packets the driver doesn't like.
I'm assuming this is all driver related. I wish I knew more about what the BEST NICs are to run on this platform and if Mikrotik could provide any clarity on the RX ERRORS, there aren't a ton, but they add up to the thousands in a few days even on lightly loaded interfaces.
I'm assuming this is all driver related. I wish I knew more about what the BEST NICs are to run on this platform and if Mikrotik could provide any clarity on the RX ERRORS, there aren't a ton, but they add up to the thousands in a few days even on lightly loaded interfaces.