Huzzah! I got it to work last night. Well, mostly. My kernel’s missing a bunch of what PVE (er, PXVirt) needs to actually run VM’s, and I don’t have any Ethernet interfaces yet (used USB for this one), but progress is progress!
I have an NVMe to SATA adapter inside the CCR2116 with one SSD hooked up for the root filesystem, and it boots via TFTP from another 2116.
Truly impressive!
Hi Sirbryan?
Did you finally manage to add all the required stuff for running VMs?
I’m still sorting out networking issues. Since I don’t have native drivers for the Alpine CPU’s SFP+ ports and for the Marvell ASIC, I’m using a USB ethernet dongle. There also appears to be something messed up in the bridge configuration, so it can’t attach VMs to any bridges I create.
If I can get a kernel compiled with all the support needed for both Proxmox and the router’s own hardware, it will be more practical.
I am able to get a VM to boot on it though, just without networking, so that part’s nice. I have TrueNAS working with a couple of SDD’s passed through (using an NVME to SATA adapter on the 2116’s motherboard).
For those following along:
After two weeks of working with Claude on porting an Alpine V2 Ethernet driver I found elsewhere ( GitHub - bcyangkmluohmars/linux-alpine-v2: Linux 6.12 LTS kernel port for Annapurna Labs Alpine V2 SoC (Ubiquiti UNVR, UDM-Pro, QNAP) · GitHub ), I now have a kernel driver that works with the management 1G port (ether13 on the CCR2116 and mgmt1 on the RDS2216). It also starts up the four in-CPU 10G ports that connect to the onboard Presterra chip, but since that’s not initializing properly (at the moment), that’s a 40G bridge to nowhere.
On the CCR2116 I was originally using the hard drive connected to an NVMe → SATA adapter, with a Realtek USB Ethernet adapter. I then moved the drive to USB, and plugged in an Intel X710 dual SFP+ card to an NVMe→PCIe adapter. Since the NVMe slot is x4 PCIe Gen 3, the X710 card easily handles 10Gbps, and should be able to do more without a problem (x4 * 8 = ~32Gbps). (That’s the backup plan in case I can’t get the Prestera started.)
Since we’re loading the kernel via TFTP, and using a USB→SATA adapter, the NAND is left untouched. Switching to an NFS root would mean a near-touchless experience in setting up a cluster of 2116’s as VM hosts. Of course, with only 1Gbps (for now), you’re not going to want to load up a bunch of data-heavy VM’s. An external card would be required for that (and that part is easy enough).
I’m running a 5.10 kernel, mainly because it’s the first version with native support for the Prestera chip, and it’s closest to the MikroTik 5.6.3 kernel patches released in Spring of 2022. (I've tried 6.1 and 6.12, but they don’t initialize the serial port properly, so I can’t debug what’s going on at boot.)
Here’s a screenshot with the 2116 running the ARM64 port of Proxmox (PXVirt) with btop in the shell.
(EDIT: The RDS2216 looks the same, just with 32GB of RAM instead of 16.)
By the way, VM’s are working. I was having issues with the bridging because MikroTik’s original patches create lots of optimizations on the network stack and break standard Linux bridging.
Now, with the kernel driver also working on ether13, we can bridge straight to that, or to a third-party card.
I don’t have PCIe passthrough working yet; still digging into what IOMMU support exists for the SOC.
You're going to give this router a new lease of life. Can the RAM be upgraded by swapping the memory modules?
It’s all soldered to the motherboard. About all you can do with these is put something in the NVMe slot. With NVMe to PCIe adapters you can put anything you want, be it storage or networking or possibly even a video card (oooo, haven’t tried that one yet…..).
I envision most users would have a handful of smaller VM’s, or 2-3 busier VM’s, each with 4-8GB of RAM.
The fact is the only other 16-core ARM64 box is the Honeycomb LX2, which is more expensive before getting a case, storage, RAM, etc. The next level up for ARM64 is Ampere (including future Ampere-driven CCR’s).
The Intel competition for the CCR2116 is mini-PC’s with laptop processors, like the MS-01 or MS-A1, each with ~16-20 threads, also at a higher price (by the time you add RAM).
RDS2216 is the least expensive NVMe server I’ve been able to find. That’s my next target.
Woot! I got things working on the RDS2216 too. It sees all the NVME drives, but with no IOMMU support, PCIE-passthrough isn't working yet.
One setback: getting the Prestera switch chip to work requires using Marvell's NDA-protected CPSS, or designing a kernel that's compatible with MikroTik's kernel modules.
The mainline kernel support for the Prestera chips requires a specific board design and special firmware that the kernel loads into the chip at boot up. On the 2x16's, the chip gets its firmware some other way (ROM?) and doesn't respond to the Prestera mainline kernel module.
To get 10-40Gbps out of these things requires a PCIe-connected card, either via the NVME port on the CCR2116, or the SFF-8644 connector on the RDS2216. Unfortunately, my SFF8644 cable and adapter don't seem to be working, at least not with the Chelsio 40G card I tried. A different NVME->Oculink->PCIe adapter I have on the 2116 works fine with an Intel X710 card on the 2116, so there's hope.
I always enjoy reading about your progress. Thank you for keeping us updated.
