RouterOS x86, no support for Chelsio T540 VF?

RouterOS General
1

hello,

i have RouterOS x86 running in a virtual machine (bhyve on FreeBSD 15.0). the virtual machine has four Ethernet PCI network devices:

 2 device="00:05.0" name="Virtio network device (rev: 0)" vendor="Red Hat, Inc." category="Ethernet controller" vendor-id="0x1af4" device-id="0x1000" irq=10 memory=0xC000E000-0xC000FFFF io=0x2080-0x20BF 

 3 device="00:05.1" name="Virtio network device (rev: 0)" vendor="Red Hat, Inc." category="Ethernet controller" vendor-id="0x1af4" device-id="0x1000" irq=10 memory=0xC000C000-0xC000DFFF io=0x2040-0x207F 

 4 device="00:05.2" name="Virtio network device (rev: 0)" vendor="Red Hat, Inc." category="Ethernet controller" vendor-id="0x1af4" device-id="0x1000" irq=11 memory=0xC000A000-0xC000BFFF io=0x2000-0x20DF 

 5 device="00:06.0" name="T540-CR Unified Wire Ethernet Controller [VF] (rev: 0)" vendor="Chelsio Communications Inc" category="Ethernet controller" vendor-id="0x1425" device-id="0x5803" irq=0 
   memory=0xC0012000-0xC0012FFF,0xC0000000-0xC01FFFFF,0xC0008000-0xC0009FFF

however, while the “Virtio” devices are detected correctly, the Chelsio VF is not:

[admin@lab1.i.lfns.org.uk] /system> /interface/print
Flags: R - RUNNING
Columns: NAME, TYPE, ACTUAL-MTU, MAC-ADDRESS
#   NAME    TYPE      ACTUAL-MTU  MAC-ADDRESS      
0 R ether1  ether           1500  06:00:00:00:05:08
1 R ether2  ether           1500  06:00:00:00:05:09
2 R ether3  ether           1500  06:00:00:00:05:0C
3 R lo      loopback       65536  00:00:00:00:00:00
[admin@lab1.i.lfns.org.uk] /system>

i assume this is because the kernel is missing a driver for this card. is there some kind of HCL for RouterOS x86 that lists supported drivers? or is it worth asking MikroTik support to add the appropriate driver?

unfortunately i can’t use CHR as it doesn’t support bhyve (because MikroTik apparently wrongly believe it’s paravirtualisation).

2

A quick google search I’ve found there’s a way to boot on bhyve guide
Can’t try myself, but there’s a bunch of guides how to boot ROS7, maybe this could solve your problem

3

thanks, i did see that but it seems it involves a lot of low-level work on the CHR image and i’m worried it’s going to break when updating. so i might try it, but it would help to know if CHR includes the driver for this card, then :slight_smile:

4

Amm0 has a dedicated GitHub repository with the updated scripts/mods, AND pre-made corrected images, see here:
http://forum.mikrotik.com/t/router-os-7-on-uefi/156661/1


https://github.com/tikoci/fat-chr/releases

You can try one of the pre-made images.
There are no issues when updating, once an image works it means that the partitioning and filesystems are fine and the update process does not change anything related to partitioning or volume formatting/filesystems, it only writes new files.

5

thanks jaclaz, this works a lot better than the x86 image (if only for improved licensing :slight_smile:

unfortunately though, it still does not seem to include the driver for the T540 VF:

[admin@lab1.i.lfns.org.uk] > /system/resource/print
                   uptime: 4m7s
                  version: 7.16.1 (stable)
               build-time: 2024-10-10 14:03:32
         factory-software: 7.1
              free-memory: 291.1MiB
             total-memory: 512.0MiB
                      cpu: AMD
                cpu-count: 1
            cpu-frequency: 3699MHz
                 cpu-load: 0%
           free-hdd-space: 71.2MiB
          total-hdd-space: 89.2MiB
  write-sect-since-reboot: 880
         write-sect-total: 880
        architecture-name: x86_64
               board-name: CHR FreeBSD BHYVE
                 platform: MikroTik
[admin@lab1.i.lfns.org.uk] > /system/resource/pci/print
Columns: DEVICE, VENDOR, NAME, IRQ
#  DEVICE   VENDOR                         NAME                                                    IRQ
0  00:00.0  Network Appliance Corporation  unknown device (rev: 0)                                   0
1  00:04.0  unknown                        unknown (rev: 0)                                          0
2  00:05.0  Red Hat, Inc.                  Virtio network device (rev: 0)                           10
3  00:05.1  Red Hat, Inc.                  Virtio network device (rev: 0)                           10
4  00:05.2  Red Hat, Inc.                  Virtio network device (rev: 0)                           11
5  00:06.0  Chelsio Communications Inc     T540-CR Unified Wire Ethernet Controller [VF] (rev: 0)    0
6  00:1f.0  Intel Corporation              82371SB PIIX3 ISA [Natoma/Triton II] (rev: 0)             0
[admin@lab1.i.lfns.org.uk] > /interface/print
Flags: R - RUNNING
Columns: NAME, TYPE, ACTUAL-MTU, MAC-ADDRESS
#   NAME    TYPE      ACTUAL-MTU  MAC-ADDRESS      
0 R ether1  ether           1500  06:00:00:00:05:08
1 R ether2  ether           1500  06:00:00:00:05:09
2 R ether3  ether           1500  06:00:00:00:05:0C
3 R lo      loopback       65536  00:00:00:00:00:00
[admin@lab1.i.lfns.org.uk] >

i didn’t see anything in all_packages that seems relevant either.

6

Is there a real reason to use passthrough instead of virtio driver or maybe try to use SR-IOV?

7

on FreeBSD host, at least, using virtio in the guest with bridge(4) on the host prevents the host from using LRO/TSO which reduces TCP performance.

or maybe try to use SR-IOV?

i am using SR-IOV - that’s what creates the PCI device i’m passing through. this is why it shows up as a ‘VF’ device.

8

in the mean time i filed feature request SUP-169557 to add the driver, but if anyone has any other suggestions, i’m all ears.

9

When using CHR, the NIC drivers must be compatible with and managed by the virtual machine host operating system (ie FreeBSD).

So, MikroTik support won’t really help here - you’ll need to check out the FreeBSD Forums instead.

10

sorry, but this is not correct in the situation i have described here.

when passing a PCI device through to a virtual machine, it’s up to the virtual machine kernel (e.g., the CHR kernel) to support the device, not the host system.

edit: unless you mean that CHR does not support PCI passthrough at all? that would be understandable (although i would find it surprising) but it’s still up to MikroTik to fix it.

11

No, virtual machines support only a limited set of virtual drivers required for the virtual guest to function properly. The host OS drivers are responsible for managing this support.

You need to configure it properly like: https://forums.freebsd.org/threads/sr-iov-chelsio-error-in-guest.70653/. If you run into more issues, try posting a question on the FreeBSD forum.

12

before i write a longer response, can i please ask, do you know what SR-IOV is?

13

Yep, it’s also up to the host OS drivers to enable SR-IOV support. The virtual machine can check if SR-IOV is available, but if you try to turn it on without driver support, you’ll just get an error or warning. The host OS still needs proper support for everything required by the virtual machine, so if something’s not working, that’s where you should start troubleshooting.

And just to make it crystal clear, you can’t manage any host drivers from within CHR; everything needs to be done on the host OS.

14

correct. host OS has SR-IOV enabled, this is working in all virtual machines on this host except for RouterOS VM.

The virtual machine can check if SR-IOV is available, but if you try to turn it on without driver support, you’ll just get an error or warning.

right. please read my original post, which is about whether RouterOS x86 / CHR has driver for Chelsio T540-CR VF device. i do not get error or warning, but device does not appear in /interface/print.

this is nothing to do with FreeBSD - question is whether RouterOS has driver for this device.

15

This is where you might be misunderstanding things. It’s all about the virtual machine support at the host OS level, not the guest OS.

RouterOS x86 and CHR are completely different. RouterOS x86 is for bare-metal installations, while CHR is designed for virtual machines that require proper support for virtual drivers. If the virtual drivers in the virtual host and its host OS aren’t set up correctly, you won’t be able to use them in CHR.

16

no, it’s not. the entire point of PCI passthrough is that the PCI device is directly accessible in the virtual machine, and the VM’s kernel is responsible for identifying the device and attaching the appropriate device driver to it.

the only thing that needs to be supported at the hypervisor level is to detect the PCI card and attach it to the VM – which, as i said, is not a problem on FreeBSD at all.

the problem here is that RouterOS does not provide the required device driver for the PCI device.

i’m afraid you simply do not understand what VM PCI passthrough means.

RouterOS x86 and CHR are completely different. blah blah bha…

and neither of them have the driver for Chelsio T540-CR Virtual Function device. which is what my entire post is about.

17

let me provide an example from a non-RouterOS Linux system, which is a VM with a T540 VF PCI passthrough device.

we can see the device in lspci:

1? 3!media ~% lspci -v 
00:00.0 Host bridge: Network Appliance Corporation Device 1275
        Flags: bus master, fast devsel, latency 0
        Capabilities: <access denied>

00:04.0 Non-Volatile memory controller: Device fb5d:0a0a (prog-if 02 [NVM Express])
        Flags: bus master, fast devsel, latency 0
        Memory at 800004000 (64-bit, non-prefetchable) [size=16K]
        Memory at c000e000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: <access denied>
        Kernel driver in use: nvme

00:05.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem
        Subsystem: Red Hat, Inc. Device 0009
        Flags: bus master, fast devsel, latency 64, IRQ 18
        I/O ports at 2000 [size=512]
        Memory at c000c000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: <access denied>
        Kernel driver in use: virtio-pci

00:06.0 Non-Volatile memory controller: Device fb5d:0a0a (prog-if 02 [NVM Express])
        Flags: bus master, fast devsel, latency 0
        Memory at 800000000 (64-bit, non-prefetchable) [size=16K]
        Memory at c000a000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: <access denied>
        Kernel driver in use: nvme

00:07.0 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
        Subsystem: Chelsio Communications Inc Device 0000
        Flags: bus master, fast devsel, latency 0
        Memory at c0010000 (32-bit, non-prefetchable) [size=4K]
        Memory at c0000000 (32-bit, non-prefetchable) [size=32K]
        Memory at c0008000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: <access denied>
        Kernel driver in use: cxgb4vf

00:1f.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
        Flags: bus master, fast devsel, latency 0

4!media ~%

‘00:07.0 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]’ - this is our Virtual Function device, exposed from the host via PCI passthrough.

and you can see lspci says “Kernel driver in use: cxgb4vf” - meaning the Linux kernel in the VM has attached the “cxgb4vf” driver to this device.

and in “dmesg” we can see the Linux kernel has detected this as a network device:

[    1.021179] cxgb4vf 0000:00:07.0: Using assigned MAC ACL: 06:00:00:00:05:a2
[    1.024707] cxgb4vf 0000:00:07.0: eth0: Chelsio VF NIC PCIe MSI-X
[    7.057836] cxgb4vf 0000:00:07.0: eth0: passive DA port module inserted
[    7.057847] cxgb4vf 0000:00:07.0 eth0: link up, 10Gbps, full-duplex, RX/TX PAUSE

and indeed this network device shows up in Linux:

# ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
    link/ether 06:00:00:00:05:a2 brd ff:ff:ff:ff:ff:ff

the problem i’m reporting in this thread is that, apparently, the RouterOS kernel is missing “cxgb4vf” driver, so it cannot detect the PCI device as a network interface.

18

after poking around “routeros-7.16.1.npk”, the problem is apparently that ROS ships the driver for “cxgb4” (the PF device) but not “cxgb4vf” (the VF device). hopefully, this means it’s fairly simply for MT to enable the VF driver – but we’ll see what support says.

19

CHR runs fine on FreeBSD as long as you set up the VM with a compatible VF for SR-IOV or configure the standard drivers properly. That said, SR-IOV won’t give you any extra performance unless you’re sharing the NIC with multiple VMs so the easiest way to get started with bhyve with is to bridge a tap interface to virtio-net.

Just a heads-up: as explained previously you can’t manage host drivers directly from within CHR, everything has to be set up on the host OS. PCI passthrough isn’t helpful here because CHR lacks built-in support for the Chelsio T540, so you’re limited to the SR-IOV-based VFs CHR supports (unless you can get MikroTik to add special SR-IOV drivers for the T540, which isn’t likely).

20

no, it doesn’t, because the CHR image is not compatible with bhyve – you need to use the third-party image that jaclaz posted earlier. the stock CHR image won’t boot on bhyve and is explicitly documented in the CHR documentation as not supporting bhyve. (i believe MT should modify it to support bhyve, but as things stand, it does not.)

as long as you set up the VM with a compatible VF

literally the entire reason i posted this thread was to ask if the Chelsio T540 VF is, or could be a compatible VF – you’re the one who tried to derail it into “the host OS needs to support the network card and the guest OS doesn’t matter”.

Just a heads-up: as explained previously you can’t manage host drivers directly from within CHR, everything has to be set up on the host OS.

this has nothing to do with what i’m talk about in this thread.

PCI passthrough isn’t helpful here because CHR lacks built-in support for the Chelsio T540

yes, thank you, finally you accept the point i’ve been trying to make since you first posted here. the entire point of this thread is to ask about support the T540 VF driver in RouterOS. nothing at all to do with the host OS.

edit: however, it seems like RouterOS does support the Chelsio T540, it just doesn’t support the VF driver – which should be trivial for MT to add.