I am testing out two 25gbit ConnectX-4 NICs (MCX4121A-ACAT - latest fw from nvidia’s site) and I am experiencing a problem where if I insert one NIC it works fine, if I try to add a second, the old NIC stops working. Removing the new NIC that was added doesn’t fix the original NIC either. I tried int reset, disabling and re-enabling, rebooting etc. The only thing that fixes it is re-installing RouterOS. What am I missing here? Did the first NIC get blacklisted/disabled with no way in the CLI to reset it or is there a special way to truly reset this NIC or at least wipe all NIC/interface config so a complete reinstall is not needed.
I do suspect the issue may be related to not enough PCI lanes on the server or something like that, but RouterOS should not be blocking out the NIC with no way to use it again until a full reinstall, that’s what I would like to solve to make trouble shooting easier.