I have searched through a few posts on this forum, and do not find a lot of topics on it so I’m not sure how many are using this.
I am newer to SRIOV, but I have some older hardware where I am trying to squeeze out as much performance as possible from the CHR. I have a connect X3 and virtualization the CHR with proxmox.
Under settings > Resources > PCI, I see the connect X3 VF detected, but it does not show up under /interfaces/ethernet.
Is it just simply that CHR does not have the drivers to do PCI pass through/SRIOV for?
SR-IOV is built into the device driver for the network card in your virtual host operating system so you won’t actually see it in CHR. But you might notice that the CPU load and software interrupt rate are significantly reduced at full network throughput when SR-IOV is enabled. If the NIC doesn’t show up, it probably means that the virtual host (Proxmox) isn’t configured correctly.
Ok thanks. I will dig through it more then. When I do the same method with an ubuntu VM for example, I do see the NIC present in the VM. But it passes it along as the Mellanox X3 VF, which was what lead to me wondering if that needs to have drivers within CHR to do so. I will do more research, but it sounds like within the CHR if SRIOV is working, it should show up as a generic network card for a lack of a better term?
The CHR is totaly unaware if the virtual driver uses SR-IOV or not. It just uses the NIC that is provided by the virtual guest where the CHR is located. So if you can’t see the NIC in the virtual guest configuration it won’t either appear in the CHR.
Virtual driver and sriov are two mutually exclusive methods. If the chr is unaware of sriov, then it is NOT using it. Sriov is a method for presenting the physical card to the guest OS. The guest should then use it’s own driver for whatever the physical card is (Intel, mellanox, emulex in my case).
I am successfully running sriov under kvm (Ubuntu) with several guests (also Ubuntu) and all of them see the VF of my emulex card as a physical card, loading their own be2net driver. It is working as it is supposed to be: partitioning the card and splitting it’s resources (rx/tx queues, vlan filter, multicast mac’s, etc.) equally amongst guests. I have successfully managed to run vlan tagging within the guest and even running multicast and ipv6 over the tagged vlan within the guest.
But I was unable to make chr (and x86) to use sriov or pci passthrough (which is quite similar). As it was already mentioned, the routeros sees the pci adapter, but does not load it’s hardware driver. But no hardware driver means no sriov, so I assume that chr can not be used in this environment not because it does not support it (the kernel is fine, as the VF is visible as a pci adapter), but because it is intentionally limited to virtio drivers only, or at least I didn’t find a way to load the native driver for the card, both with sriov or with pci passthrough.
Thought of dropping here and leave a little word of advice.
I’ve been in a head scratcher for the last 2 days in which Proxmox backed CHR router was behaving mysteriously. After much debugging I just realized by default and VF out of a Mellanox card using the upstream linux kernel driver mlx4 will drop by default any L2 traffic not directly addressed to its defined MAC address or multicast, broadcast.
This works fine if the VF port is used as a direct port in which all traffic is directly addressed to the router but it will create many problems if the VF is attached to a bridge and you need such bridge to drop traffic containing other mac addresses.
The fix for this would be setting the VF as trusted and enabling promiscuous mode to it but the mainline kernel driver has no provisions for that and the Mellanox provided driver (MLNX_OFED) is just too old to work any recent proxmox versions.
You need to switch mellanox nic on host to switchdev so you don’t need promisc on to see traffic. Worked fine for me after that was done.(Non chr use case)
In general there are VF kernel modules for specific nics. I.e. x710 Intel uses i40e kernel module, VF functions use iavf module within vms. I’m not sure of how to verify what kernel modules are available short of looking at one of the gpl source code dumps someone got from mikrotik and dumped to GitHub, or pinging support.i have x710e working with sr-iov on chr myself.
I have a connect-x 5 myself but haven’t tried sr-iov specifically with chr, only Intel so far. But have sr-iov on connect-x working in other guest OS’s
