 CHR using Apple Virtualization & QEMU via macOS UTM

.


See post #65 below for UTM CHR installation instructions for the
Most streamline way to install RouterOS CHR on Intel-based any macOS UTM
https://forum.mikrotik.com/viewtopic.php?t=204805#p1131200

https://forum.mikrotik.com/viewtopic.php?t=204805#p1125233
chr-utm-screenshot.png
I didn’t intend on this thread to have 60+ posts… Which now includes multiple GitHub projects…
CHR re-partitioning: https://github.com/tikoci/fat-chr
UTM packaging to ZIP/URL - “mikropkl” https://github.com/tikoci/mikropkl
Old Apple Virtualization Only UTM packaging to ZIP/URL https://github.com/tikoci/chr-utm

At this point the rest of the content in this post below is now historic:
I’d been experiment using Apple Virtualization Framework (https://developer.apple.com/documentation/virtualization) using the option in MacOS-version of UTM (https://mac.getutm.app).

I normally using VMWare Fusion/ESXi. But UTM (in Apple, NOT QEMU, mode) seems to work okay in my limited testing for a few Linux images. For a lark, I wanted to try CHR using UTM+AppleVMF. Issue is Apple’s Virtualization Framework ONLY support UEFI, which CHR – for some unknown reason includes EFI boot files, but the diskpart stuff isn’t right.

As it turns out, I somehow got it work. I use CHR on Mac for testing configscripts, so “working” is about all I need. I tried to write up what I did, but post if you try and doesn’t work. I do know that CHR does not show the logon prompt on the emulated video display (although the serial port is mirrored as the screen shown below) but serial port and network work fine. /tool/speed-test in a UTM+Apple VM matched a similar bridged network using VMWare Fusion. Since UTM+Apple (e.g. no QEMU) uses less CPU and boot WAY quicker than VMWare Fusion, so far pretty good.

Even more historic information, potentially could be removed:
See #15 post below for more streamlined process to using CHR with Apple virtualization, on Intel-based Macs:
https://forum.mikrotik.com/viewtopic.php?t=204805#p1059466

The following is the “manual” way now…



This post had the clue on UEFI issues with CHR: http://forum.mikrotik.com/t/router-os-7-on-uefi/156661/5
While I did NOT use the script…the post was 100% correct:
For some reason, Mikrotik does actually includes the right bits for UEFI support…but UEFI requires a FAT16 partition — NOT the ext2 that’s the boot partition in the CHR .IMG file — so CHR does not work unless some Legacy BIOS is used (which Apple, and other VM platforms, do not offer).

The Mikrotik’s help for CHR on Vultr has the big clue to how to avoid needing a script (which will not work on Mac) — boot to the SystemRescure image. See https://help.mikrotik.com/docs/display/ROS/CHR+Vultr+installation

So the Vultr instruction ALMOST works for UTM + Apple Virtualization: e.g. booting UTM with Apple Virtualization enabled to the “SystemRescueCD”… But those need to be COMBINE with the @kriszos to convert the IMG file’s partition from ext2 to fat16. Most of the write up below is the process involved in that…

Inside UTM, select:

  • hit “+” to add a new VM
  • select “Linux” as the type
  • check the “Use Apple Virtualization” box
  • pick the SystemRescueCD as the “Boot ISO” (after downloading the ISO: https://www.system-rescue.org)
  • pick cores/memory as desired
  • pick a disk size - RouterOS does not need a big disk… I used 1GB but can be smaller/bigger as desired
  • skip shared directory (hit continue)
  • now, pick a name for the VM, I used “AppleCHR” and IMPORTANTLY check “Open VM Settings”
  • dialog with VM setting will appear:
  • under Network, you may to change to use “Bridge” mode (or add more network interface… or less likely, use shared if you really like multiple NATs)
  • under Devices section, use “+ New…” to add a “Serial” port (below item network) – default is bring up serial port a new window which is what you want
  • reviews other setting, but serial above is about only CRITICAL thing to add
  • hit “Save”
  • in UTM main window hit Play icon to start the new VM


    I’m guessing Vultr is more forgiving than Apple about UEFI. The solution to this is same as @kriszos post above allude: convert the ext2 partition to fat16. And this can be done after the “dd” in SystemRescueCD. The specific steps I used:
  • follow other steps from https://help.mikrotik.com/docs/display/ROS/CHR+Vultr+installation
  • at same SystemRescueCD’s terminal…
  • “mountall” after CHR has been extracted/copyed to the /dev/vda disk
  • “mkdir /tmp/vda1” to create folder to store RouterOS EFI files in ext2 boot partition
  • “cp -r /dev/vda1/* /tmp/vda1” to copy the EFI files
  • “umount /dev/vda1” and “umount /dev/vda2” to un-mount the boot and main partitions
  • “startx” - to launch X11 (probably some CLI to “parted” work too, but GUI is helpful with disks IMO)
  • Hit the icon for “gparted” in the task bar (or run “gparted” from a Terminal in X11 desktop)
  • Use the drop-down in upper left to select the /dev/vda disk. You should see two partitions: one ext2 and one ext4.
  • Right click on the first one, marked “boot” on right, and select format.
  • Pick “fat16” as the format type. This will “cue” the operation.
  • Next commit the, use menus or hit the green checkbox in toolbar. Assuming it says success in status window & still marked “boot”…closed gparted.
  • Bring up a terminal window from menu/taskbar, and again type “mountall”
  • At same terminal, use “cp -r /tmp/vda1 /dev/vda1” to copy the original files back to the re-formatted 1st partition.
  • Finally “poweroff”
  • CHR should be “installed” and bootable at this point, except you need to remove the SystemRescueCD from UTM – since it will boot to RouterOS at this point.

Note: I only tested on Intel Mac – but I believe UTM with Apple Virtualization framework will work on M1/M2/M3 (ARM) Mac via Rosetta. If someone tries, that LMK if it works.

Did you experienced any FS corruption running on UTM with Apple Virtualization Framework? Related to this still open issue for UTM → https://github.com/utmapp/UTM/issues/4840
I tried with Debian 12.5 (arm64, without Rosetta) which installation is using kernel 6.1 and FS gets corrupted eventually on M1 since some kernel patches for Silicon (M) chips support are made in 6.2, but not sure if these kernel patches are directly related to FS corruption and M chip support or is UTM bug with Apple Virtualization Framework in general (following github issue comments maybe it is only UTM bug).

@Ammo, thanks for very interesting info! Personally I love Parallels Desktop but for various reasons we are exploring alternative solutions. UTM/VMF might be an option when it becomes stable enough. Will definitely look into it further..

For me VFM and VirtioFS seems to be stable using with Docker on M1, have some databases running with it in containers (even some with x86_64 arch using Rosetta), so I guess UTM needs to improve stability and it will be great free VM alternative on Macs.

Well, I’m surprised CHR even worked – why I shared. Not really not sure why there isn’t a pre-made CHR image that works with UEFI since X86 does. I’m sure there is probably a more optimized path to a working .IMG file than my write up since I just wrote down what I did than figure out the best way to do it.

When you use Apple Virtualization in UTM, I’m not sure UTM does very much other than call the needed Apple API. Since the framework is new, my bet is Apple. For example, the serial console works with Apple VMs, but not the display. I know CHR shows the login prompt on VMWare, but using UTM+Apple Virtualization is video display is blank, but winbox etc via network does so not an issue for me. BUT… without the serial port enabled… it look like it doesn’t work…

I just used Ubuntu using Apple Virtualization – that does seem to work, at least for a week. But I don’t really stress any of my VMs. I’m trying to ween myself off VMWare overall (Fusion is more useful for me since I use ESXi and the VMs work same on both).

The one problem UTM solve is VMWare Fusion’s boot speed ( http://forum.mikrotik.com/t/mikrotik-routeros-boot-speed-is-very-slow-vmware/166801/3 and others) – it seems to be a recurring problem. UTM+Apple, boots in few seconds.

My only complaint so far is Apple does not support USB passthrough, which is useful to get an LTE modem into CHR.

I tried this tonight WITHOUT using UTM, only Apple. I used some swift from an Apple sample project that used VZEFIBootLoader() & another sample with the serial console window. And changed the disk image to use same converted CHR disk image (e.g. 1st/boot just changed from EXT2 to FAT16 type, as described in first post) as disk. And CHR still works. Since everyone like icons on Mac included a “teaser”:

The real trick was @kriszos finding here about FAT16 being required by UEFI specs – which is NOT how Mikrotik package the CHR image (e.g. they use EXT2):
http://forum.mikrotik.com/t/router-os-7-on-uefi/156661/5

To confirm,

This window is not the VGA graphics from the CHR VM, but a SwiftUI window connecting to the serial port of the CHR?

Correct, it’s a serial console in window using https://github.com/migueldeicaza/TermKit to deal with ANSI. Using Swift without UTM, the VGA display is blank – same as UTM.

It’s possible Apple is using the GPU variant of VirtIO display (QEMU’s virtio-gpu-pci) – which does not have VGA emulation & likely CHR wants VGA… Apple lets you set size not a specific driver (https://developer.apple.com/documentation/virtualization/vzvirtiographicsscanoutconfiguration).

Did a try on mac arm and it doesn’t work for me.

  1. SystemRescueCD can’t boot and I’ve used Kali Live (Apple silicon ARM64)
  2. After performing all the steps VM can’t boot with the same error as SystemRescueCD - “EFI Boot Options returned. Exiting VM”
    Something was already discussed on GitHub but it was about arm on arm - ARM64 Linux not booting with Apple Virtualization #5517.
    Screenshot 2024-02-23 at 00.16.03.png
    Emulation works fine, but slightly slow.

Probably AMPERE (coming soon) may solve that?

Yup. Apparently, the specific are Rosetta will help with any X86’s running inside Apple Virtualized machine – the actual Linux disto needs to also be same arm. So “Ampere” support is needed. Actually looked it up: https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta
There was a footnote:

Rosetta doesn’t support the bootstrapping or installation of Intel Linux distributions on Mac computers with Apple silicon using the Virtualization framework. Intel Linux distributions can run using the Virtualization framework on Intel-based Mac computers without the need for this translation capability.

I am glad that my work help you resolve your problem. Hopefully MikroTik will provide us with uefi image packaged by them without this ugly workarounds. More and more modern hypervisors stop providing BIOS options at all my own research come from need to run CHR on LXD that didn’t supported BIOS at the time.

Totally @kriszos - it was worth a search for UEFI in the forum… but I would have never got to the partition scheme in the RAW image was the reason EFI didn’t work & given up before trying gdisk :wink:.

I filed a feature request for UEFI support as SUP-144667 after my post here. I’m not sure they got the issue is generically UEFI support in CHR:

We checked our suggestion list, and you are the only person who asked about Apple Hypervisor.
We noted it, but it is not a big chance that it will be implemented soon.

But tried.

Should more of us file support/feature requests for generic UEFI support in CHR?

SUP-145276

I took @kriszos’s “gdisk+bash” script, with small modification to fetch RouterOS in loop, and put it into a GitHub repo “fat-chr” to build FAT UEFI images automatically via GitHub’s Action CI.

Since 7.14 stable came out today, I tried out my GH Action script with @kriszos’s gdisk+bash script & have 7.14 stable IMG that work without SystemRestoreCD – I just trigger a build in GitHub and download the .IMG. And 7.14 CHR is running without UTM’s QEMU, just Apple’s.

IMG files built by GitHub and work with Apple (and likely other/newer hypervisors that require proper UEFI support) are here:
https://github.com/tikoci/fat-chr/releases
I would not use these for production - but to spin up a VM for testing, might be helpful.

This greatly simplify the process of adding a CHR to Intel-based UTM using it’s “Apple mode”.
In UTM, you need to check the “Use Apple” in wizard and still pick an ISO and create some disk… then in the setting BEFORE starting:

  • create new VM in UTM “+” > Virtualize > Linux
  • check the “Use Apple Virtualization” & pick some file for ISO boot image – going to remove later so doesn’t matter what (could be text file whatever)
  • click “next” to everything else in UTM setup wizard…and at end check “Open VM Setting”… then in those settings:
  • remove the disk the wizard added
  • remove the ISO image
  • remove the display
  • add a serial port
  • under disk, use “Import” and pick .RAW image from GitHub
  • give it a name if you want (default will be “Linux #”)
  • then… start it with Play icon… a console window will appear with RouterOS login [via serial —but you won’t know – a VGA CLI look the same, so serial saves some bits/complexity if you think about]

Note: The image only fix the disk part of UEFI support. Something else is needed for VGA display. I think Apple’s VirtIO driver does not have VGA mode, which is why Display will show nothing since CHR is liking setting VGA mode. Bash nor GitHub can fix that… something in CHR have to change to deal with the newer VirtIO VGA-less display drivers.

Great work @Amm0
chr_utm_i.png
Didn’t try yet over Rosetta on M, but it could be issues with FS corruption as I mentioned in my previous post (or is just native ARM issue). Too bad there is no USB passthrough support.

Rosetta on ARM Mx only helps with X86 AFTER a ARM-based Linux is loaded. The boot process does not go through Rosetta, only user applications. So this won’t work for non-Intel Macs. :frowning:. Apple docs say:

Rosetta doesn’t support the bootstrapping or installation of Intel Linux distributions on Mac computers with Apple silicon using the Virtualization framework. Intel Linux distributions can run using the Virtualization framework on Intel-based Mac computers without the need for this translation capability.

So still QEMU on M1/M2/… but “UEFI boot” for CHR should be selectable in other hypervisors using these images I think.

But Mikrotik does have “AMPERE” on their download page (grayed out). To me, that’s Mikrotik messed-up marketing for ARM64 (“aarch64”) as ISO coming soon - but dunno. But gives hope for M1+ Macs.

I used VMWare Fusion for a long while (since it come out) but CPU use is ~half with UTM+Apple on my main Intel Mac for a CHR VM. And various problems with VMWare freezing when booting CHR also gone – it boots in few seconds.

Ah yes, I forgot about Rosetta is not supporting Linux bootstrapping. Well we will need to wait CHR ROS for aarch64 then…

I’m excited to see what people come up with now that the Ampere ISO is out. I tried messing around with it but I can’t get things to boot.