 CHR using Apple Virtualization & QEMU via macOS UTM

Amm0 · Tue Feb 20, 2024 5:25 am

.

See post #65 below for UTM CHR installation instructions for the
Most streamline way to install RouterOS CHR on Intel-based any macOS UTM
viewtopic.php?t=204805#p1131200

viewtopic.php?t=204805#p1125233

chr-utm-screenshot.png

I didn't intend on this thread to have 60+ posts... Which now includes multiple GitHub projects...
CHR re-partitioning: https://github.com/tikoci/fat-chr
UTM packaging to ZIP/URL - "mikropkl" https://github.com/tikoci/mikropkl
Old Apple Virtualization Only UTM packaging to ZIP/URL https://github.com/tikoci/chr-utm

At this point the rest of the content in this post below is now historic:

I'd been experiment using Apple Virtualization Framework (https://developer.apple.com/documentati ... ualization) using the option in MacOS-version of UTM (https://mac.getutm.app).

I normally using VMWare Fusion/ESXi. But UTM (in Apple, NOT QEMU, mode) seems to work okay in my limited testing for a few Linux images. For a lark, I wanted to try CHR using UTM+AppleVMF. Issue is Apple's Virtualization Framework ONLY support UEFI, which CHR – for some unknown reason includes EFI boot files, but the diskpart stuff isn't right.

As it turns out, I somehow got it work. I use CHR on Mac for testing configscripts, so "working" is about all I need. I tried to write up what I did, but post if you try and doesn't work. I do know that CHR does not show the logon prompt on the emulated video display (although the serial port is mirrored as the screen shown below) but serial port and network work fine. /tool/speed-test in a UTM+Apple VM matched a similar bridged network using VMWare Fusion. Since UTM+Apple (e.g. no QEMU) uses less CPU and boot WAY quicker than VMWare Fusion, so far pretty good.

Even more historic information, potentially could be removed:

See #15 post below for more streamlined process to using CHR with Apple virtualization, on Intel-based Macs:
viewtopic.php?t=204805#p1059466

The following is the "manual" way now....

This post had the clue on UEFI issues with CHR: viewtopic.php?p=1025068&hilit=UEFI#p933799
While I did NOT use the script...the post was 100% correct:
For some reason, Mikrotik does actually includes the right bits for UEFI support...but UEFI requires a FAT16 partition — NOT the ext2 that's the boot partition in the CHR .IMG file — so CHR does not work unless some Legacy BIOS is used (which Apple, and other VM platforms, do not offer).

The Mikrotik's help for CHR on Vultr has the big clue to how to avoid needing a script (which will not work on Mac) — boot to the SystemRescure image. See https://help.mikrotik.com/docs/display/ ... stallation

So the Vultr instruction ALMOST works for UTM + Apple Virtualization: e.g. booting UTM with Apple Virtualization enabled to the "SystemRescueCD"... But those need to be COMBINE with the @kriszos to convert the IMG file's partition from ext2 to fat16. Most of the write up below is the process involved in that...

Inside UTM, select:
- hit "+" to add a new VM
- select "Linux" as the type
- check the "Use Apple Virtualization" box
- pick the SystemRescueCD as the "Boot ISO" (after downloading the ISO: https://www.system-rescue.org)
- pick cores/memory as desired
- pick a disk size - RouterOS does not need a big disk... I used 1GB but can be smaller/bigger as desired
- skip shared directory (hit continue)
- now, pick a name for the VM, I used "AppleCHR" and IMPORTANTLY check "Open VM Settings"
- dialog with VM setting will appear:
- under Network, you may to change to use "Bridge" mode (or add more network interface... or less likely, use shared if you really like multiple NATs)
- under Devices section, use "+ New..." to add a "Serial" port (below item network) – default is bring up serial port a new window which is what you want
- reviews other setting, but serial above is about only CRITICAL thing to add
- hit "Save"
- in UTM main window hit Play icon to start the new VM

I'm guessing Vultr is more forgiving than Apple about UEFI. The solution to this is same as @kriszos post above allude: convert the ext2 partition to fat16. And this can be done after the "dd" in SystemRescueCD. The specific steps I used:
- follow other steps from https://help.mikrotik.com/docs/display/ ... stallation
- at same SystemRescueCD's terminal...
- "mountall" after CHR has been extracted/copyed to the /dev/vda disk
- "mkdir /tmp/vda1" to create folder to store RouterOS EFI files in ext2 boot partition
- "cp -r /dev/vda1/* /tmp/vda1" to copy the EFI files
- "umount /dev/vda1" and "umount /dev/vda2" to un-mount the boot and main partitions
- "startx" - to launch X11 (probably some CLI to "parted" work too, but GUI is helpful with disks IMO)
- Hit the icon for "gparted" in the task bar (or run "gparted" from a Terminal in X11 desktop)
- Use the drop-down in upper left to select the /dev/vda disk. You should see two partitions: one ext2 and one ext4.
- Right click on the first one, marked "boot" on right, and select format.
- Pick "fat16" as the format type. This will "cue" the operation.
- Next commit the, use menus or hit the green checkbox in toolbar. Assuming it says success in status window & still marked "boot"...closed gparted.
- Bring up a terminal window from menu/taskbar, and again type "mountall"
- At same terminal, use "cp -r /tmp/vda1 /dev/vda1" to copy the original files back to the re-formatted 1st partition.
- Finally "poweroff"
- CHR should be "installed" and bootable at this point, except you need to remove the SystemRescueCD from UTM – since it will boot to RouterOS at this point.

Note: I only tested on Intel Mac – but I believe UTM with Apple Virtualization framework will work on M1/M2/M3 (ARM) Mac via Rosetta. If someone tries, that LMK if it works.

optio · Tue Feb 20, 2024 5:34 pm

Did you experienced any FS corruption running on UTM with Apple Virtualization Framework? Related to this still open issue for UTM -> https://github.com/utmapp/UTM/issues/4840
I tried with Debian 12.5 (arm64, without Rosetta) which installation is using kernel 6.1 and FS gets corrupted eventually on M1 since some kernel patches for Silicon (M) chips support are made in 6.2, but not sure if these kernel patches are directly related to FS corruption and M chip support or is UTM bug with Apple Virtualization Framework in general (following github issue comments maybe it is only UTM bug).

Larsa · Tue Feb 20, 2024 5:54 pm

@Ammo, thanks for very interesting info! Personally I love Parallels Desktop but for various reasons we are exploring alternative solutions. UTM/VMF might be an option when it becomes stable enough. Will definitely look into it further..

optio · Tue Feb 20, 2024 6:03 pm

For me VFM and VirtioFS seems to be stable using with Docker on M1, have some databases running with it in containers (even some with x86_64 arch using Rosetta), so I guess UTM needs to improve stability and it will be great free VM alternative on Macs.

Amm0 · Tue Feb 20, 2024 9:18 pm

Well, I'm surprised CHR even worked – why I shared. Not really not sure why there isn't a pre-made CHR image that works with UEFI since X86 does. I'm sure there is probably a more optimized path to a working .IMG file than my write up since I just wrote down what I did than figure out the best way to do it.

When you use Apple Virtualization in UTM, I'm not sure UTM does very much other than call the needed Apple API. Since the framework is new, my bet is Apple. For example, the serial console works with Apple VMs, but not the display. I know CHR shows the login prompt on VMWare, but using UTM+Apple Virtualization is video display is blank, but winbox etc via network does so not an issue for me. BUT... without the serial port enabled... it look like it doesn't work...

I just used Ubuntu using Apple Virtualization – that does seem to work, at least for a week. But I don't really stress any of my VMs. I'm trying to ween myself off VMWare overall (Fusion is more useful for me since I use ESXi and the VMs work same on both).

The one problem UTM solve is VMWare Fusion's boot speed ( viewtopic.php?p=1048889&hilit=fusion#p1048889 and others) – it seems to be a recurring problem. UTM+Apple, boots in few seconds.

My only complaint so far is Apple does not support USB passthrough, which is useful to get an LTE modem into CHR.

Amm0 · Wed Feb 21, 2024 10:39 am

I tried this tonight WITHOUT using UTM, only Apple. I used some swift from an Apple sample project that used VZEFIBootLoader() & another sample with the serial console window. And changed the disk image to use same converted CHR disk image (e.g. 1st/boot just changed from EXT2 to FAT16 type, as described in first post) as disk. And CHR still works. Since everyone like icons on Mac included a "teaser":

The real trick was @kriszos finding here about FAT16 being required by UEFI specs – which is NOT how Mikrotik package the CHR image (e.g. they use EXT2):
viewtopic.php?p=1025068&hilit=UEFI#p933799

sirbryan · Thu Feb 22, 2024 12:45 am

I tried this tonight WITHOUT using UTM, only Apple. I used some swift from an Apple sample project that used VZEFIBootLoader() & another sample with the serial console window.

To confirm,

This window is not the VGA graphics from the CHR VM, but a SwiftUI window connecting to the serial port of the CHR?

Amm0 · Thu Feb 22, 2024 5:50 am

I tried this tonight WITHOUT using UTM, only Apple. I used some swift from an Apple sample project that used VZEFIBootLoader() & another sample with the serial console window.
To confirm,

This window is not the VGA graphics from the CHR VM, but a SwiftUI window connecting to the serial port of the CHR?

Correct, it's a serial console in window using https://github.com/migueldeicaza/TermKit to deal with ANSI. Using Swift without UTM, the VGA display is blank – same as UTM.

It's possible Apple is using the GPU variant of VirtIO display (QEMU's virtio-gpu-pci) – which does not have VGA emulation & likely CHR wants VGA... Apple lets you set size not a specific driver (https://developer.apple.com/documentati ... figuration).

stmx38 · Fri Feb 23, 2024 12:34 am

Did a try on mac arm and it doesn't work for me.
1. SystemRescueCD can't boot and I've used Kali Live (Apple silicon ARM64)
2. After performing all the steps VM can't boot with the same error as SystemRescueCD - "EFI Boot Options returned. Exiting VM"
Something was already discussed on GitHub but it was about arm on arm - ARM64 Linux not booting with Apple Virtualization #5517.

Screenshot 2024-02-23 at 00.16.03.png

Emulation works fine, but slightly slow.

Probably AMPERE (coming soon) may solve that?

Amm0 · Fri Feb 23, 2024 12:39 am

Did a try on mac arm and it doesn't work for me.
[...]
Probably AMPERE (coming soon) may solve that?

Yup. Apparently, the specific are Rosetta will help with any X86's running inside Apple Virtualized machine – the actual Linux disto needs to also be same arm. So "Ampere" support is needed. Actually looked it up: https://developer.apple.com/documentati ... th_rosetta
There was a footnote:

Rosetta doesn’t support the bootstrapping or installation of Intel Linux distributions on Mac computers with Apple silicon using the Virtualization framework. Intel Linux distributions can run using the Virtualization framework on Intel-based Mac computers without the need for this translation capability.

kriszos · Wed Feb 28, 2024 11:08 am

I am glad that my work help you resolve your problem. Hopefully MikroTik will provide us with uefi image packaged by them without this ugly workarounds. More and more modern hypervisors stop providing BIOS options at all my own research come from need to run CHR on LXD that didn't supported BIOS at the time.

Amm0 · Wed Feb 28, 2024 6:12 pm

I am glad that my work help you resolve your problem.

Totally @kriszos - it was worth a search for UEFI in the forum.... but I would have never got to the partition scheme in the RAW image was the reason EFI didn't work & given up before trying gdisk

.

I filed a feature request for UEFI support as SUP-144667 after my post here. I'm not sure they got the issue is generically UEFI support in CHR:

We checked our suggestion list, and you are the only person who asked about Apple Hypervisor.
We noted it, but it is not a big chance that it will be implemented soon.

But tried.

sirbryan · Wed Feb 28, 2024 7:26 pm

Should more of us file support/feature requests for generic UEFI support in CHR?

kriszos · Wed Feb 28, 2024 8:58 pm

Should more of us file support/feature requests for generic UEFI support in CHR?

SUP-145276

Amm0 · Thu Feb 29, 2024 7:04 pm

I took @kriszos's "gdisk+bash" script, with small modification to fetch RouterOS in loop, and put it into a GitHub repo "fat-chr" to build FAT UEFI images automatically via GitHub's Action CI.

Since 7.14 stable came out today, I tried out my GH Action script with @kriszos's gdisk+bash script & have 7.14 stable IMG that work without SystemRestoreCD – I just trigger a build in GitHub and download the .IMG. And 7.14 CHR is running without UTM's QEMU, just Apple's.

IMG files built by GitHub and work with Apple (and likely other/newer hypervisors that require proper UEFI support) are here:
https://github.com/tikoci/fat-chr/releases
I would not use these for production - but to spin up a VM for testing, might be helpful.

This greatly simplify the process of adding a CHR to Intel-based UTM using it's "Apple mode".
In UTM, you need to check the "Use Apple" in wizard and still pick an ISO and create some disk... then in the setting BEFORE starting:
- create new VM in UTM "+" > Virtualize > Linux
- check the "Use Apple Virtualization" & pick some file for ISO boot image – going to remove later so doesn't matter what (could be text file whatever)
- click "next" to everything else in UTM setup wizard...and at end check "Open VM Setting"... then in those settings:
- remove the disk the wizard added
- remove the ISO image
- remove the display
- add a serial port
- under disk, use "Import" and pick .RAW image from GitHub
- give it a name if you want (default will be "Linux #")
- then... start it with Play icon... a console window will appear with RouterOS login [via serial —but you won't know – a VGA CLI look the same, so serial saves some bits/complexity if you think about]

Note: The image only fix the disk part of UEFI support. Something else is needed for VGA display. I think Apple's VirtIO driver does not have VGA mode, which is why Display will show nothing since CHR is liking setting VGA mode. Bash nor GitHub can fix that... something in CHR have to change to deal with the newer VirtIO VGA-less display drivers.

optio · Thu Feb 29, 2024 8:34 pm

Great work @Amm0

chr_utm_i.png

Didn't try yet over Rosetta on M, but it could be issues with FS corruption as I mentioned in my previous post (or is just native ARM issue). Too bad there is no USB passthrough support.

Amm0 · Thu Feb 29, 2024 8:54 pm

Rosetta on ARM Mx only helps with X86 AFTER a ARM-based Linux is loaded. The boot process does not go through Rosetta, only user applications. So this won't work for non-Intel Macs.

. Apple docs say:

Rosetta doesn’t support the bootstrapping or installation of Intel Linux distributions on Mac computers with Apple silicon using the Virtualization framework. Intel Linux distributions can run using the Virtualization framework on Intel-based Mac computers without the need for this translation capability.

So still QEMU on M1/M2/... but "UEFI boot" for CHR should be selectable in other hypervisors using these images I think.

Amm0 · Thu Feb 29, 2024 9:11 pm

But Mikrotik does have "AMPERE" on their download page (grayed out). To me, that's Mikrotik messed-up marketing for ARM64 ("aarch64") as ISO coming soon - but dunno. But gives hope for M1+ Macs.

I used VMWare Fusion for a long while (since it come out) but CPU use is ~half with UTM+Apple on my main Intel Mac for a CHR VM. And various problems with VMWare freezing when booting CHR also gone – it boots in few seconds.

optio · Thu Feb 29, 2024 10:03 pm

Ah yes, I forgot about Rosetta is not supporting Linux bootstrapping. Well we will need to wait CHR ROS for aarch64 then...

sirbryan · Tue Mar 05, 2024 4:16 am

I'm excited to see what people come up with now that the Ampere ISO is out. I tried messing around with it but I can't get things to boot.

ayufanpl · Tue Mar 05, 2024 10:17 am

I was able to boot Ampere/ARM64 ISO on Qemu with KVM acceleration: https://github.com/ayufan-research/mikrotik-qemu-arm64. Works just fine. Tested on Raspberry PI 5 and Rock 5B.

Amm0 · Tue Mar 05, 2024 5:44 pm

I'm excited to see what people come up with now that the Ampere ISO is out. I tried messing around with it but I can't get things to boot.

I tried on Equinox Metal, but couldn't get it work. I do not think it's Mikrotik fault... Metal uses iPXE with netboot.xyz for custom OSes. I use VMWare on X86... so I gave up quick since I don't know iPXE stuff (although similar problem as here – what's the kernel & initfs to use for RouterOS ). But kinda did want to see it boot on something before spelunking on ARM Mac. But no success on Metal.

I don't have an ARM Mac handy, but next time I can check I'll see what happens. At the end of the day, Apple Virtualization needs some EFI executable file and a ramdisk with RouterOS file system (e.g. routeros.npk uncompressed AFAIK) — just how it find/pick them may be tricky align.

Amm0 · Tue Mar 05, 2024 6:00 pm

I was able to boot Ampere/ARM64 ISO on Qemu with KVM acceleration: https://github.com/ayufan-research/mikrotik-qemu-arm64. Works just fine. Tested on Raspberry PI 5 and Rock 5B.

Thanks for sharing your scripts. Some working examples always help.

FWIW... this is a likely a good call:

 ... qemu-system-aarch64 ...   
   -vga none -nographic -monitor none 
  -serial chardev:term0 -chardev stdio,id=term0

RouterOS X86 seems to have problems with VirtIO displays (e.g. on Apple, some Hyper-V, etc) — doubt that be fixed in ARM64.

Still... might be worth a check to see if virtual display actually works with "Ampere"+QEMU

sirbryan · Tue Mar 05, 2024 6:13 pm

I'm excited to see what people come up with now that the Ampere ISO is out. I tried messing around with it but I can't get things to boot.
I tried on Equinox Metal, but couldn't get it work. I do not think it's Mikrotik fault... Metal uses iPXE with netboot.xyz for custom OSes. I use VMWare on X86... so I gave up quick since I don't know iPXE stuff (although similar problem as here – what's the kernel & initfs to use for RouterOS ). But kinda did want to see it boot on something before spelunking on ARM Mac. But no success on Metal.

I don't have an ARM Mac handy, but next time I can check I'll see what happens. At the end of the day, Apple Virtualization needs some EFI executable file and a ramdisk with RouterOS file system (e.g. routeros.npk uncompressed AFAIK) — just how it find/pick them may be tricky align.

I have a Studio and MacBook Pro. You've given me a couple ideas to try. There is an efiboot.img on the ISO. It also has all the packages on the ISO. Kernel boots on Apple and on QEMU but doesn't know where init is.

Amm0 · Tue Mar 05, 2024 7:59 pm

I have a Studio and MacBook Pro. You've given me a couple ideas to try. There is an efiboot.img on the ISO. It also has all the packages on the ISO. Kernel boots on Apple and on QEMU but doesn't know where init is.

I'm not expect here, but re-packaging I can do. I added ZIP file output to my GitHub "X86 UEFI CHR builder", while it doesn't have symlinks and stuff. If you extract the ZIP file, you can see how the IMG get's structured without mounting the disk on Mac (since it's cannot read the 2nd partition):
https://github.com/tikoci/fat-chr/relea ... 25-testing

The CHR ARM64 ISO has a very different structure... My wild ass guess is somehow though the sysconfig stuff it mounts routeros-7.15beta4-arm64.npk as the ramdisk. While it ends in .npk, it's still a normal LZMA compressed file (e.g. 7-Zip) that EFI boot loader tries to find.

But I'm really not the expert on UEFI and Linux boot details, especially on ARM64... If someone figures out the magic sauce, I can automate it.

ayufanpl · Tue Mar 05, 2024 8:45 pm

I have a Studio and MacBook Pro. You've given me a couple ideas to try. There is an efiboot.img on the ISO. It also has all the packages on the ISO. Kernel boots on Apple and on QEMU but doesn't know where init is.

This is likely problem of UEFI or HVF accelerator implemented in QEMU on MacOS.

In general if you disable `Use Hypervisor` kernel will boot. The UTM requires to use `NVME` disk for storage, and I was not able to configure CDROM. I cannot force UTM to use `virtio-blk-pci` for CDROM device. Once you install (somehow) the ARM64 MikroTik will boot in non-hypervisor mode via UTM.

sysconfig for x86 is to boot via legacy BIOS.

ayufanpl · Tue Mar 05, 2024 8:56 pm

OK. You can boot/install via UTM:

1. Create new VM => Virtualize => Linux.
2. For Virtualization Engine: Keep `Use Apple Virtualization` unmarked. For Boot Image: Select arm64.iso. Then, continue.
3. Set min 512MB of memory, and at least one CPU core, at least 1GB of disk size. Then, continue a few times.
4. Edit then VM Settings.
5. In QEMU, unselect Use Hypervisor.
6. Go to Display, and click Delete. Then go to Sound and click Delete.
7. Under Devices, add Serial.
8. Go to VirtIO Drive, change Interface to NVMe.
9. Go to USB Drive, and Delete. This remove the iso image.
10. Under Drives, click New. Select VirtIO, click Import, find mikrotik-arm64.iso.
11. You should be able to start and install. After you install, you can remove the ISO drive.

Of course Hypervisor will not work, so the performance will be slow.

sirbryan · Tue Mar 05, 2024 9:26 pm

Sweet. I got it installed and booted.

Now to figure out how to boot it under Apple's Virtualization Framework to get closer to the CPU.

stmx38 · Tue Mar 05, 2024 9:59 pm

Works for me as well, but I almost forgot that ROS is payed software - we need CHR for ARM

With CHR using x86 emulation we can play for free and boot time is similar to that ARM.

Amm0 · Wed Mar 06, 2024 2:09 am

OK. You can boot/install via UTM:

Just for fun, I tried the ARM64 ISO, emulated in UTM on Intel Mac — WORKED.

Instructions almost just work on Intel UTM with QEMU emulation.... Clearly the NVMe disk seem required (tried IDE or VirtIO – neither worked). But for Intel UTM, the ISO image seems to need to be first converted manually to a .qcow2 with VirtIO for the 1st partition to boot before importing it.

qemu-img convert -O qcow2 mikrotik-7.15beta4-arm64.iso mikrotik-7.15beta4-arm64.qcow2

Amm0 · Wed Mar 06, 2024 5:31 am

Now for Apple Virtualization on ARM MacOS... My best guess is the new RouterOS "ARM64 ISO" is missing a needed "virtio_blk.ko" (at least doesn't appear in ISO or any .NPK)
And since VirtIO is only* disk option on Apple Virtualization, kinda problematic (*if there isn't virtio_blk, there isn't NBD in RouterOS, while NBD is supported by Apple's APIs, it's not in UTM with Apple VM set). Explain why QEMU seem requires NVMe emulation to boot in UTM...

I think the "ARM virtio disk problem" is same problem as video one: just another missing different virtio driver...

patrikg · Wed Mar 06, 2024 10:05 am

Sorry

Secure warning !!

Be careful if you download images other than from the official Mikrotik one.

optio · Wed Mar 06, 2024 10:37 am

warning secured...

ayufanpl · Wed Mar 06, 2024 10:54 am

Just for fun, I tried the ARM64 ISO, emulated in UTM on Intel Mac — WORKED.

Instructions almost just work on Intel UTM with QEMU emulation.... Clearly the NVMe disk seem required (tried IDE or VirtIO – neither worked). But for Intel UTM, the ISO image seems to need to be first converted manually to a .qcow2 with VirtIO for the 1st partition to boot before importing it.
Code: Select all
qemu-img convert -O qcow2 mikrotik-7.15beta4-arm64.iso mikrotik-7.15beta4-arm64.qcow2

The step 10. does this conversion (or rather attachment of RAW).

The NVMe device is required as MikroTik requires some Serial Number to be configured to get Hardware ID. The VirtIO does not allow that.

I'm personally running ARM64 variant on my Rock 5B now via KVM virtualization, and it achieves maximum throughput!

Amm0 · Wed Mar 06, 2024 3:32 pm

Code: Select all
qemu-img convert -O qcow2 mikrotik-7.15beta4-arm64.iso mikrotik-7.15beta4-arm64.qcow2
The step 10. does this conversion (or rather attachment of RAW).

Yeah that may be a bug in UTM (or gremlins)... it seem to try to convert it, and then reports in dialog it cannot find mikrotik-7.15beta4-arm64.qcow. Just something I noticed. (Keep in mind, I tested on Intel Mac using emulation, so may not happen on ARM Mac)

The NVMe device is required as MikroTik requires some Serial Number to be configured to get Hardware ID. The VirtIO does not allow that.

Ah yes, the license scheme is different. Which also makes sense why there isn't a virtio blk driver for it either.

I'm personally running ARM64 variant on my Rock 5B now via KVM virtualization, and it achieves maximum throughput!

Nice! There is hope.

sirbryan · Wed Mar 06, 2024 5:38 pm

I'm personally running ARM64 variant on my Rock 5B now via KVM virtualization, and it achieves maximum throughput!

I'll have to try it via KVM on my SolidRun Honeycomb LX2 that's collecting dust. It doesn't support UEFI boot and their UEFI shim is out of date. Otherwise I'd have spent some more time on booting it natively.

ayufanpl · Wed Mar 06, 2024 5:40 pm

I'm personally running ARM64 variant on my Rock 5B now via KVM virtualization, and it achieves maximum throughput!
I'll have to try it via KVM on my SolidRun Honeycomb LX2 that's collecting dust. It doesn't support UEFI boot and their UEFI shim is out of date. Otherwise I'd have spent some more time on booting it natively.

Via KVM it works just fine. No objections there, and especially a better idea for something that will rather be unsupported by MikroTik.

Amm0 · Wed Mar 06, 2024 6:22 pm

I personally think it better to always use a hypervisor. In years of using Mikrotik, never once tried metal X86 before here.

Mikrotik need to just build a ARM64 CHR version of "AMPERE", since it's not just Apple where this comes up.

And the "security warning" in this thread on UEFI image... easily fixed by Mikrotik changing their build to add one more CHR image with FAT as boot. I put GitHub on that task, and since CI runners are free in public project, easy to share. FWIW, the entire build process is observable/auditable, but it's a fair warning. And I'd rather not be involved package building.

optio · Wed Mar 06, 2024 6:53 pm

And it is also possible to compare checksums from original files is someone is concerned...

sirbryan · Thu Mar 07, 2024 5:52 am

I made a tiny bit of headway trying to boot the installer on my Honeycomb LX2.

Using a UEFI image from SolidRun, the ISO boots via USB. It looks for an NVMe drive (SATA are ignored), and then burps because it can't find the CD-ROM.

Based on what we saw with QEMU, I wonder if it expects the CD-ROM to be on the PCIe bus or something. Booting from a SATA drive doesn't make a difference. Going to try two NVMe drives just for fun.

sirbryan · Thu Mar 07, 2024 7:02 am

Huzzah! Just like with QEMU, two NVMe drives works. But it doesn't see any of the NICs or USB interfaces. That would probably explain why the install kernel doesn't see the USB drive after it takes over from UEFI. In System/Resources/PCI it lists just the NVMe drives.

I wonder what cards in Ampere systems they're targeting with this. The onboard NXP ports aren't available (yet).

Amm0 · Thu Mar 07, 2024 7:13 am

But it doesn't see any of the NICs or USB interfaces.

Isn't there some extra-nic.npk you can install – maybe already did – just a thought.

That would probably explain why the install kernel doesn't see the USB drive after it takes over from UEFI.

It does replicate QEMU where the CD-ROM had to be transformed into a disk image. IDK about modern bootloaders, but AFAIK EFI will mount the CD-ROM for an OS install to START. But when the "real" OS boot, it depends on its own driver for CD-ROM. But a physical disk (or emulated one) be same ID/etc in both EFI and RouterOS kernel.

ayufanpl · Thu Mar 07, 2024 10:21 am

The installer does iterate all devices and try to mount the is9660. It does not care about whether this is in a physical CDROM, USB drive, or NVMe drive. So, the drive with a CDROM image needs to be discoverable. Anyway, don't expect USB drivers to work here.

Amm0 · Thu Mar 07, 2024 7:37 pm

All reasons why I'm pro-virtualization. Good news is seems the world is coalescing around VirtIO since Microsoft and even Apple support.... since dealing with device drivers has been a PITA my entire life.

FWIW, I did see @normis report WRT to "AArch64" on the 7.15beta thread:

CHR images are coming in next betas

This be good news for Mac ARM users. I suspect if it work under Apple Virtualization on ARM Mac... it work any other KVM hypervisor using AArch64. e.g. Apple is kinda "worse case" since only supports VirtIO. KVM long offered VirtIO, but most Linux hypervisors do have other supporting functions too... but with Apple it's 100% VirtIO drivers (and clearly a limited subset of them too).

ayufanpl · Thu Mar 07, 2024 9:09 pm

This be good news for Mac ARM users. I suspect if it work under Apple Virtualization on ARM Mac... it work any other KVM hypervisor using AArch64. e.g. Apple is kinda "worse case" since only supports VirtIO. KVM long offered VirtIO, but most Linux hypervisors do have other supporting functions too... but with Apple it's 100% VirtIO drivers (and clearly a limited subset of them too).

Mac does not offer KVM. It does offer HVF and Virtualization framework. Qemu can support anything HVF.

Amm0 · Thu Mar 07, 2024 10:07 pm

This be good news for Mac ARM users. I suspect if it work under Apple Virtualization on ARM Mac... it work any other KVM hypervisor using AArch64. e.g. Apple is kinda "worse case" since only supports VirtIO. KVM long offered VirtIO, but most Linux hypervisors do have other supporting functions too... but with Apple it's 100% VirtIO drivers (and clearly a limited subset of them too).
Mac does not offer KVM. It does offer HVF and Virtualization framework. Qemu can support anything HVF.

True. I just found it telling that even Apple adopted KVM's VirtIO in "HVF", not their own drivers/VMXNET3/something ... Even to the extent that "HVF" is likely a "even more pure" VirtIO-based hypervisor, than even KVM

More a thought that HVF kinda useful for general VM testing.... if OS has poor VirtIO support, you'll get nothing from Apple.

sirbryan · Fri Mar 08, 2024 6:10 am

After messing around with it as bare metal on the Honeycomb (extra-nics was installed), I booted back into Ubuntu and got it working through QEMU/KVM. With 2 or 8 cores, I can get it to receive 1.5Gbps and transmit 700Mbps, so I'm guessing there's a bit of optimization yet to be had. iperf3 on the host can get 8Gbps to/from my Mac Studio using the stock NXP drivers.

Looking forward to ARM64 CHR and hopefully some more optimization.

ayufanpl · Fri Mar 08, 2024 3:09 pm

True. I just found it telling that even Apple adopted KVM's VirtIO in "HVF", not their own drivers/VMXNET3/something ... Even to the extent that "HVF" is likely a "even more pure" VirtIO-based hypervisor, than even KVM

More a thought that HVF kinda useful for general VM testing.... if OS has poor VirtIO support, you'll get nothing from Apple.

This is incorrect:

1. The HVF.framework is similar to KVM: it provides a function to run a second-level isolated hypervisor / virtual machine. It is very bare bones and requires to handle everything. The HVF is used by QEMU to to provide isolation boundary. QEMU for connected devices, can emulate any underlying device type as long as it is programmed, this is why you can attach non-virtio emulated devices to the VM. The HVF.framework was introduced as a way to provide user-space accessible virtualization, similar to how the `/dev/kvm` is exposed on Linux.
2. On MacOS there's also Virtualization.framework. It is Apple's own QEMU-like emulator. The Virtualization.framework does only expose VirtIO-like devices. Underneath the Virtualization.framework uses HVF.framework.

The UTM can use:

1. QEMU that uses HVF, and does emulate various type of devices - all that are supported by the QEMU.
2. QEMU that uses TCG, basically emulate everything, including CPU.
3. Apple Virtualization via Virtualization.framework, very limted to only a VirtIO-like devices.

Amm0 · Fri Mar 08, 2024 5:58 pm

Thanks @ayufanpl. Used VMWare for decades and for a time built iOS apps. Most server/apps/OS supported VMWare, while KVM was hit/miss. Seems the reverse is happening, which is good since VMWare is now pretty expensive. While I know UNIX/Linux well enough, all the KVM/QEMU stuff I'm trying to learn. Seems complicated onion.

But I now get what you mean when you said "HVF" – that QEMU support for Apple's Virtualization.framework. That's nifty – I thought Virtualization.framework was only in UTM actually. But I though you made-up an weird acronym for "Apple's Virtualization.framework" (which has not acronym AFAIK).

UTM with Apple (outside CHR image problem) "just working" got me thinking... So I did try the Virtualization.framework with Apple's Swift sample to directly to boot CHR, which worked: viewtopic.php?p=1061786#p1057843
I kinda liked the generic concept of have an "/Application" that put some nice SwiftUI (perhaps NOT a terminal) around some Linux thing more generically. At some point, I'll put the "MacOS CHR" code/app on GitHub (perhaps, "AHR"

) when I have time. Hopefully Mikrotik will have a ARM64 CHR image by then too.

ayufanpl · Fri Mar 08, 2024 8:26 pm

Thanks @ayufanpl. Used VMWare for decades and for a time built iOS apps. Most server/apps/OS supported VMWare, while KVM was hit/miss. Seems the reverse is happening, which is good since VMWare is now pretty expensive. While I know UNIX/Linux well enough, all the KVM/QEMU stuff I'm trying to learn. Seems complicated onion.

...

I kinda liked the generic concept of have an "/Application" that put some nice SwiftUI (perhaps NOT a terminal) around some Linux thing more generically. At some point, I'll put the "MacOS CHR" code/app on GitHub (perhaps, "AHR" ) when I have time. Hopefully Mikrotik will have a ARM64 CHR image by then too.

If you are coming from VMWare please checkout the Proxmox VE. It is brilliant. Build on top of Debian, QEMU and KVM. And Open Source. The company behind it offers paid support.

Yes, there are number of apps build on top of `Virtualization.framework` to provide GUI. Since, all heavy lifting is done by Apple to provide emulated hardware support so building GUI is significantly easier task. The UTM provides a GUI for both: Virtualization.framework and QEMU.

Amm0 · Fri Mar 08, 2024 9:03 pm

If you are coming from VMWare please checkout the Proxmox VE. It is brilliant. Build on top of Debian, QEMU and KVM. And Open Source. The company behind it offers paid support.

LOL. Yeah I recently just installed Proxmox VE on an old server to test it... It better from a UI POV for sure. And the disk management is just more logical too. Plus seem QEMU has come a long way since I start with VMWare in the 90s. CHR was an addition on ESXi a few years, but it super handy for dealing with all the networking stuff vSwitch can't & avoid need the even more price vSphere stuff. e.g. RouterOS VRRP on two VMWare ESXI does similar HA things to vSphere & easier, and $100/server is a bargain. On Proxmox VE, there is all Linux newer data plane stuff. Haven't gotten there yet with Proxmox VE.

It was the Fusion to UTM part of conversion that got me here. It was nice to be able test a VMX-based VM on Mac Fusion, and just copy files to ESXi (or vise versa). But if want to do same with Proxmox, I need KVM/QEMU on Mac...

One note: @TomjNorthIdaho (who runs an open btest server from his ISP) is in same boat re VMWare. @ayufanpl, if you had any Proxmox advice, might want to share it here: viewtopic.php?t=203953&hilit=vmware

ayufanpl · Fri Mar 08, 2024 10:16 pm

It was the Fusion to UTM part of conversion that got me here. It was nice to be able test a VMX-based VM on Mac Fusion, and just copy files to ESXi (or vise versa). But if want to do same with Proxmox, I need KVM/QEMU on Mac...

One note: @TomjNorthIdaho (who runs an open btest server from his ISP) is in same boat re VMWare. @ayufanpl, if you had any Proxmox advice, might want to share it here: viewtopic.php?t=203953&hilit=vmware

Not really. The disk images are transferable between Proxmox VE and UTM, since both are based on QEMU, both can boot UEFI, both use VirtIO for most of emulated devices. They are super similar on how they configure QEMU and interact with it. The problem with UTM is that it can run amd64 or arm64 natively depending on MacOS arch. The Proxmox VE at least now is only amd64. So, you can transfer UTM <-> PVE only when running on Intel Macs. There are some GitHub repos trying to bring PVE to arm64 with mixed degree of success.

Kartone · Thu Jun 06, 2024 11:51 am

For those who, like me, struggled to have ROS running on Apple Silicon. I'm sharing the config that, for me, solved the issue:
I'm running UTM Version 4.5.3 (99) on Apple M2 Max, Sonoma 14.5.

Hope it helps.

optio · Thu Jun 06, 2024 8:03 pm

That is system emulation with QEMU, you can run any supported architecture like that on any system where you can run QEMU, this topic is related to CHR virtualization using Apple Virtualization Framework.
With Apple VF CHR arm64 raw image won't boot, but x86_64 it does after changing FS type on ESP in image, in arm64 image ESP seeps to be ok, but it won't boot, didn't have time to investigate more.

pcDidakt · Mon Jul 08, 2024 7:55 pm

Hi Kartone and others

Thanks for that! I am new on ARM and I do know there are different types of ARM Cores etc. Do we know if the AMPERE and Apple Silicon are fully compatible on CPU Level? If so, aren't there optimizations in QEMU so that most of the code runs natively? I mean that we need to emulate the IO is not that strange.

Actually I have never used CHRx86 that much on Bare metal so I know if they supported modules with drivers for Network Cards or it was up to the user to buy NE2000, Intel exxx compatible etc?

Keep up the good work and I would LOVE to see a Youtube snippet about how it looks and runs!

/Johan Löfgren
Sweden

Amm0 · Mon Feb 10, 2025 5:43 am

Given the Super Bowl in US... Amm0's container&script superstore is now offering ...

Easiest way to install a RouterOS* — via URL:

utm://downloadVM?url=https://github.com/tikoci/chr-utm/releases/download/v7.17.2/RouterOS.utm.zip

*assuming you have an Intel Mac with UTM installed — To install UTM for Mac, see https://mac.getutm.com .

On Intel macOS, just cut-and-paste URL below into Safari — including the "utm://downloadVM?" which tell UTM to start the process:

utm://downloadVM?url=https://github.com/tikoci/chr-utm/releases/download/v7.17.2/RouterOS.utm.zip

After using URL, UTM will prompt you download it, and will automatically add CHR to the virtual machine assuming you accept the download.

UTM prompt after utm url.png

You can also download the image directly from the "Release" section on the GitHub project that, essentially, ZIP you a directory with a .plist file and .raw image (and icon). If you expand the ZIP, UTM should see it as virtual machine then (as it will have the ".utm" ending when unzip). So same image packaged for UTM (as a ZIP) is under "Releases" from the GitHub tikoci/chr-utm repo:
https://github.com/tikoci/chr-utm/releases

In either case, you should be able to just start it once it installed in UTM — no configuration of serial ports or disks needed.
To start, click the ">" Start icon next to "RouterOS", and a console will appear

CHR started after install using serial console.png

The username is of course "admin", and CHR still uses no password.

Note: RouterOS will use a UTM "shared" (NATed) network by default. With macOS defaults (i.e. macOS firewall enabled), no port should be available beyond you local Mac. You can feel free to adjust any networking as desired — including none if just want to test scripts or commands. But you can add multiple network adapters, or "bridge" CHR to a real interface if desired (i.e. to avoid NAT once it's configured with passwords and a RouterOS firewall). For example, you can add a USB ethernet dongle to Mac, add that interface as "bridge" in RouterOS UTM image "Network" settings. Anyway, there are network setting to adjust in UTM to present more network (say for testing) or use multiple "shared" ones.

Above uses Apple's "App URLs", which get send to UTM app and is same scheme as "UTM Gallery" uses to install images. The image inside ZIP iimage based on the tikoci/fat-chr "no-gdisk" method, and the image is set to use Apple Virtualization which is super fast at bring up CHR.

Finally to use multiple CHR, you'll need to rename any existing ones first named "RouterOS". So if you want to add TWO CHRs, you change "RouterOS" to "RouterOS 1" and then use same URL/download to add another CHR (and perhaps rename that one to "RouterOS 2"). Basically, it won't automatically install a CHR if same already exists.

Amm0 · Sat Feb 22, 2025 3:31 am

I updated the readme on my Intel macOS CHR "automated" images for UTM. I've included a bit more information on setup and usage:
https://github.com/tikoci/chr-utm

Also, If anyone has a .plist file and some details on image used for ARM64 UTM that known working... I can package that up similar to Intel on same GitHub project – I'm just don't have a ARM Mac always on hand to test, so avoid publishing anything for Apple Silicon macOS's UTM.

optio · Sat Feb 22, 2025 11:40 am

I never managed to boot ARM64 ROS image using AVF, only with QEMU in UTM, same UTM QEMU setup

config.plist.zip

works on Intel and Silicon Mac with unmodified CHR image. I would also like to know if anybody succeed to boot ARM64 ROS with AVF on Silicon.

Amm0 · Mon Feb 24, 2025 7:09 pm

Thanks @optio. I still have not added to my project to build QEMU variants.

But did just update the Apple Intel macOS-based "UTM CHR" to 7.18:

utm://downloadVM?url=https://github.com/tikoci/chr-utm/releases/latest/download/RouterOS.utm.zip

With an update README on usage:
https://github.com/tikoci/chr-utm?tab=r ... hr-for-utm

Since one use for CHR on Apple is testing... While not fully baked, there is an `expect` script that will automatically accept license and reset password. And links to UTM's AppleScript which can control UTM (and osascript allow AppleScript to be run from bash FWIW) that can tweak machine setting, start/stop etc.

optio · Mon Feb 24, 2025 8:32 pm

Nice work.
FWIW, ARM64 CHR can run in QEMU with image provided by MT without modifications if no additional space on it is needed.

Amm0 · Mon Feb 24, 2025 8:52 pm

Nice work.
FWIW, ARM64 CHR can run in QEMU with image provided by MT without modifications if no additional space on it is needed.

My plan is to build three version:
UTM with Apple (Intel)
UTM with QEMU (Intel)
UTM with QEMU (ARM64)

I also want to figure out how to update the UUIDs in the .plist per build too. I'll hopefully get it this week.

But I've gone from waiting 5 minutes for VMWare Fusion to just start... to going from reading the "new release thread" to have CHR running in a 2-3 minutes for entire process. I'm some AppleScript away to UTM from automating some testing.

If anyone had input on a good default mem/cpu's to use, I'm open. I left UTMs defaults, but perhaps CHR's min specs.

optio · Mon Feb 24, 2025 9:02 pm

I also want to figure out how to update the UUIDs in the .plist per build too. I'll hopefully get it this week.

You can try to automate with PlistBuddy (/usr/libexec/PlistBuddy) in script. PlistBuddy is macOS provided CLI tool for editing .plists

Amm0 · Mon Feb 24, 2025 10:39 pm

I also want to figure out how to update the UUIDs in the .plist per build too. I'll hopefully get it this week.
You can try to automate with PlistBuddy (/usr/libexec/PlistBuddy) in script. PlistBuddy is macOS provided CLI tool for editing .plists

Oh plenty of tools on macOS for plist. Issue is the UTM ZIP files are built at GitHub Action "in cloud" on Ubuntu... And thus no PlistBuddy (or even `defaults`) on Linux...

I'm sure some `sed` could work.... but the nitty-gritty details are I wanted to try Apple's new config language, pkl-lang.org since that does read a plist on Linux, and supports "templating" so the GitHub build can provide the variables to `pkl` to generate a new .plist as part of build. For automation, it's actually more useful to set serial to a "Psuedo-TTY", and stuff like cpu/disk. And pkl script should work both at GitHub and a local macOS to "clone" as UTM VM. But I actually only got as far there wasn't some thing similar to plistbuddy for linux... And... there are really no pkl-lang examples, other than: https://pkl-lang.org/main/current/langu ... plate.html.

optio · Mon Feb 24, 2025 11:11 pm

Ah, forgot it's under Linux, maybe with plistutil for conversion + some XML edit tool like xmlstarlet? plistutil comes with libplist-utils package. plistutil is not even needed if .plist is not in binary format or it can be used to convert to JSON format and such can be modified with jq tool (instead using xmlstarlet) and then convert back to .plist XML format.

Amm0 · Wed Mar 05, 2025 7:36 pm

Nice work.
FWIW, ARM64 CHR can run in QEMU with image provided by MT without modifications if no additional space on it is needed.
My plan is to build three version:
UTM with Apple (Intel)
UTM with QEMU (Intel)
UTM with QEMU (ARM64)

I finally got around to streamlining this. It's actually in a new GitHub — mikropkl.

Since it has a better framework, and I plan to retire the chr-utm version at some point - since mikropkl has same Apple, plus now QEMU images. Basically I can create ready-to-use UTM virtual machine based on a "manifest" file written in `pkl` (https://www.pkl-lang.org) that deal with UTM's .plist and other packaging details. So creating any new variants of QEMU or Apple CHR is now pretty straightforward - it just a new half dozen line file in ./Manifests. The Makefile and build will do the rest.

For example, I now have "ROSE" images that include 3 small 100Mb disks that can be used to test ROSE features, without having to manually add disks to config. In the new scheme, QEMU images are always based on unmodified CHR images from Mikrotik (while Apple images still require processing to convert ext2 to fat inside the .img file). And all VM packages come standard with TWO serial ports, one going to UTM window console, and another wired to some pseudo-tty device (i.e. a "file" like /dev/stty00X). But still Apple Virtualization support is limited to Intel.

mikropkl-packages-in-utm.png

FWIW, even the icon style (solid or line) and color are customizable in the manifest file, and selected for now based on gaudiness - in solidarity with the WinBox4 pro-color&contrast movement.

I have tested all these images on Intel macOS with UTM, but not Apple Silicon based macOS. If anyone want to try the aarch64 QEMU image on ARM-based Mac, and report back that be good info to know if it just worked (although it's based on @optio's example above) - but it possible some tweak may be need for ARM macOS, dunno.

The download CHR images for UTM are here: https://github.com/tikoci/mikropkl/rele ... chr-7.18.1

With the following images currently being built:

CHR.Apple.x86_64.utm.zip
CHR.QEMU.aarch64.utm.zip
CHR.QEMU.x86_64.utm.zip
ROSE.QEMU.aarch64.utm.zip
ROSE.QEMU.x86_64.utm.zip

optio · Wed Mar 05, 2025 9:18 pm

Great work. I will try them on Silicon Mac when I find some time and report back.

Amm0 · Thu Mar 06, 2025 10:05 pm

And the UTM packager now deals with versions, so there is a 7.19beta4 release, with QEMU using "real CHR" (and the Apple Virtualization version still needs custom FAT boot image):
https://github.com/tikoci/mikropkl/rele ... -7.19beta4

Also, renamed the generated package/ZIP names to include the version. While long, you can rename it once import or alias'ed in UTM.

I'll probably split the new UTM package into a new topic at some point. But to use the new "mikropkl" UTM packages of CHR for any macOS, it's basically:

Install UTM (via "brew install --cask utm" or https://mac.getutm.app) if you don't have it.
In Terminal or in Safari's URL, use the correct UTM URL from https://github.com/tikoci/mikropkl/releases - for example for Apple Silicon with 7.19beta4, with 3 spare disks, can be launched from Terminal using:
Code: Select all
```
open 'utm://downloadVM?url=https://github.com/tikoci/mikropkl/releases/download/chr-7.19beta4/rose.chr.aarch64.qemu.7.19beta4.utm.zip'
```
(note the 'single quotes' around the URL)
UTM will open and prompt to download and add the new CHR
If accepted, hit the "Start" icon on "rose.chr.aarch64.qemu.7.19beta4" shown on left in UTM
A terminal window will appear connected to CHR. Login is "admin" without password, speed in free version is 1Mb/s.
After login, "/ip/address/print" will show you the "shared" network IP, and you can use that in Winbox or Webfig from the local computer only.

All ".qemu." version use the same CHR .img from Mikrotik. The ".apple." version uses one that get repackaged by GitHub to convert ext2 to FAT (as well described in 60+ posts above).

The ready-to-use QEMU images for UTM on macOS - that use "real" image – are what's new here.

monk · Sat Mar 08, 2025 2:41 pm

Hi!

I tried to run those machines from https://github.com/tikoci/mikropkl/releases (chr.aarch64.qemu.7.19beta4.utm and rose.chr.aarch64.qemu.7.19beta4.utm - used the utm:// links for both) on my Mac Mini with Apple M2 Pro processor and on anything I try I get only such error.

utm chr error.png

What am I missing here to run it successfully.

Amm0 · Sat Mar 08, 2025 9:50 pm

I tried to run those machines from https://github.com/tikoci/mikropkl/releases (chr.aarch64.qemu.7.19beta4.utm and rose.chr.aarch64.qemu.7.19beta4.utm - used the utm:// links for both) on my Mac Mini with Apple M2 Pro processor and on anything I try I get only such error.

Thanks for testing! I had not tried it on Apple Silicon, I'm a bit of a dinosaur on that front and still use Intel Mac. So the aarch64 images are based on @optio's working configuration provided a few post above.

In diff'ing @optio's config from the GitHub generated config, I suspect the issue is the generated `config.plist` needs to have both setting below set to "false" - while GitHub/pkl are using "true":

		<key>Hypervisor</key>
		<false/>

		<key>RNGDevice</key>
		<false/>

Also, the CPU type is wrong, it's using "default" which may not be "cortex-a710" which is what @optio uses. So this also need to change:

		<key>CPU</key>
		<string>cortex-a710</string>

I'll re-generate an image with those settings for 7.19beta4 today, to exactly match @optio's configuration which is known to work.

You should be able to edit these "manually":
1. In UTM, right click on the CHR machine, select "Show in Finder".
2. Close UTM app to allow modifying the configuration without potential conflict.
3. In Finder, right click on same machine, select "Show Package Contents"
4. Select "config.plist" and right-click and open in an editor. If there are no default editors, select "Other", then find "TextEdit".
5. Change the above RNGDevice and Hypervisor from <true/> to <false/>
5. Change the above CPU from "default" to "cortex-a710"
6. Save file and close editor.
7. Restart UTM and try machine again.

optio · Sat Mar 08, 2025 10:31 pm

@Amm0, yes issue is with enabled QEMU Hypervisor for aarch64 CHR, it should be disabled, I had same issue as @monk when I was setting this up before, for X86_64 doesn't have affect since it's different architecture. Regarding RNG device, it is also working when it is enabled, maybe it will approve performance when enabled for aarch64 CHR. Also aarch64 works on cortex-a710, cortex-a72 (Default) and cortex-a76 CPU series.

machines1.png

machines2.png

Also QEMU Hypervisor can be disabled over UTM GUI for VM setup:

hypervisor.png

When rebooting aarch64 CHR noticed this error, it's working anyway, could be CHR ROS image issue:

reboot-aarch64.png

Amm0 · Sat Mar 08, 2025 11:19 pm

@Amm0, yes issue is with enabled QEMU Hypervisor for aarch64 CHR, it should be disabled, I had same issue as @monk when I was setting this up before, for X86_64 doesn't have affect since it's different architecture. Regarding RNG device, it is also working when it is enabled, maybe it will approve performance when enabled for aarch64 CHR. Also aarch64 works on cortex-a710, cortex-a72 (Default) and cortex-a76 CPU series.

I re-built the 7.19beta4 and 7.18.1 images to exactly match @optio's plist, so cortex-a710 and no RNG for now. I do think the RNG shouldn't be a problem. But I recall the "no hypervisor" part in thread, but missed that in diff's.

Images with above changes:
https://github.com/tikoci/mikropkl/rele ... -7.19beta4

Or directly to ROSE + QEMU + 7.19beta4 for Apple Silicon (aarch64) from Terminal app:

open 'utm://downloadVM?url=https://github.com/tikoci/mikropkl/releases/download/chr-7.19beta4/rose.chr.aarch64.qemu.7.19beta4.utm.zip'

When rebooting aarch64 CHR noticed this error, it's working anyway, could be CHR ROS image issue:
reboot-aarch64.png

Yeah I don't know how the CHR software ID is generated, nor if it's correlated with UTM "machine identifier" UUID in config. Something could go wrong there, and "cloud lic" part is some clue I think. I have tested with all images running on Intel. If running code locally, with "make utm-install; make utm-start" which add all built images and starts them with AppleScript from Makefile)

EDITS: I changed the URL to releases to include a "chr-" & rebuilt 7.18.1 (stable) and 7.19beta4 (testing) to use revised "no hypervisor" and other settings for Apple Silicon

monk · Sun Mar 09, 2025 5:53 pm

@Amm0, yes issue is with enabled QEMU Hypervisor for aarch64 CHR, it should be disabled, I had same issue as @monk when I was setting this up before, for X86_64 doesn't have affect since it's different architecture. Regarding RNG device, it is also working when it is enabled, maybe it will approve performance when enabled for aarch64 CHR. Also aarch64 works on cortex-a710, cortex-a72 (Default) and cortex-a76 CPU series.
I re-built the 7.19beta4 and 7.18.1 images to exactly match @optio's plist, so cortex-a710 and no RNG for now. I do think the RNG shouldn't be a problem. But I recall the "no hypervisor" part in thread, but missed that in diff's.

Images with above changes:
https://github.com/tikoci/mikropkl/rele ... -7.19beta4

Or directly to ROSE + QEMU + 7.19beta4 for Apple Silicon (aarch64) from Terminal app:
Code: Select all
open 'utm://downloadVM?url=https://github.com/tikoci/mikropkl/releases/download/chr-7.19beta4/rose.chr.aarch64.qemu.7.19beta4.utm.zip'
When rebooting aarch64 CHR noticed this error, it's working anyway, could be CHR ROS image issue:
reboot-aarch64.png
Yeah I don't know how the CHR software ID is generated, nor if it's correlated with UTM "machine identifier" UUID in config. Something could go wrong there, and "cloud lic" part is some clue I think. I have tested with all images running on Intel. If running code locally, with "make utm-install; make utm-start" which add all built images and starts them with AppleScript from Makefile)

EDITS: I changed the URL to releases to include a "chr-" & rebuilt 7.18.1 (stable) and 7.19beta4 (testing) to use revised "no hypervisor" and other settings for Apple Silicon

This new image works like magic! Thank you so much!

Amm0 · Sun Mar 09, 2025 10:04 pm

Or directly to ROSE + QEMU + 7.19beta4 for Apple Silicon (aarch64) from Terminal app:
Code: Select all
open 'utm://downloadVM?url=https://github.com/tikoci/mikropkl/releases/download/chr-7.19beta4/rose.chr.aarch64.qemu.7.19beta4.utm.zip'
This new image works like magic! Thank you so much!

Good to hear. Yeah it pretty cool to see them just open up, without going through dozen steps.

CHR 7.19beta4 x 4 (Tiled Terminals).png

I could write a bit more notes on using them, but happy to hear it works since I can't easily test aarch64 macOS.

sirbryan · Mon Mar 10, 2025 6:05 pm

Nice work Amm0. I opened the URL on my Mac Studio and it came right up. I'm able to browse into and it works well. Just need to license it for more bandwidth.

Also testing CHR on an Ampere box (Q80-30) and loving all this ARM64 stuff.

Amm0 · Mon Mar 10, 2025 8:07 pm

FWIW, I updated the README.md with a lot more information. I'll summarize some key things below.

While all generally works fine with Wi-Fi adapter... it is not the quickest path. This is NOT a CHR thing, as it effects all virtualization on Mac. So wiring up a Ethernet dongle is recommend if you're looking to see more native-like performance from the network interface. Or... if anyone wants to learn about /queue, bring up 2 x CHR images, both "bridged" to a Wi-Fi adapter, and run a /tool/bandwidth-test...you will quickly see variability. TL;DR is if you are using the Wi-Fi interface, using fq_codel queue in UTM CHR does clean up the performance somewhat.

In particular on further automation options, which I left unexplored. But the same utm:// URL scheme has a few more simple commands for automation. So you can use some

open 'utm://start?name=chr.aarch64.qemu.7.9beta4'

to start a VM. UTM's docs are good here in describing some simple usages on macOS with Automator and Shortcuts:
https://docs.getutm.app/advanced/remote-control/
So you can in a more few steps, you can be starting/stopping CHR from Siri if desired. And the same Automator and Shortcuts can be used with the REST API to go beyond starting/stopping in those tools.

Also, it now trivial to add new UTM CHR VM configurations for download, so if anyone wants a more "customized CHR"... feel free to post the suggested configuration. I added the ROSE images to both test the "customizability", but also it handy to not have to modify UTM setting after download.

I opened the URL on my Mac Studio and it came right up. I'm able to browse into and it works well. Just need to license it for more bandwidth.

The current images all default to "help.mikrotik.com CHR recommandation" memory of 1GB and 2 CPU cores. On a Mac Studio, you might be able to more generous.

On licensing, I added a section in README.md about it. But they do offer a 60-day "trial" mode. So if you have/create an account at www.mikrotik.com/client, then to enable a 10Gb/s license on any CHR, the command is just:

/system/license/renew account=$myaccount password=$mypassword level=p10

and enabling it does not need a credit card or anything. @normis has pointed out the only restriction is upgrade is not allowed.

Since I think a primary use case be testing... you would not want to upgrade & likely use a new version before 60-day even. Now for long-term use, yes, you should buy a license. But I cannot see Mikrotik objecting to MORE people testing the "beta" or "rc", or new features like "ROSE" - the later @normis is also asking folk to "test" – but how was no described. So the "ROSE" CHR image published, at least for macOS, allow someone to try the ROSE features safely. But I do not speak for Mikrotik, and not a lawyer on licensing rules.

But.. essentially, it's one command and your password to move it from "free" to "trial" mode.

While the underlying GitHub re-packaging is as transparent as possible, It is worth repeating a previous commentary:

Sorry
Secure warning !!

Be careful if you download images other than from the official Mikrotik one.

And to clarify this part for the "mikropkl" images:

For the *chr.*.qemu* VM downloads, all use unmodified CHR .img files download as part of a GitHub Action. Anyone curious can see the build logs here: https://github.com/tikoci/mikropkl/actions.

For the *chr.*.apple* VM downlaods, these use images that are originally download from Mikrotik.com, but have a script changes partitioning table (as discussed in dozen of post above). Specifically, the *.apple.* images on mikropkl's GitHub, use CHR images build in another project, tikoci/fat-chr, with those build logs here: https://github.com/tikoci/fat-chr/actions

At the end of the day, a lot of trust is placed in Microsoft / GitHub, since I cannot make that more transparent than it already is... Flip side is GitHub does have various security scanning tools to look for possible supply chain attacks that are enabled in tikoci projects. And if the supply chain or other attack was within Mikrotik's image (e.g. in their build process), nothing to be done about the risk either, nor any transparency on that process. It's a tricky world on these things.

And as alternative, it is totally possible to build everything locally. The README.md describes how to this in Build locally on macOS section.

Please comment if anyone has idea to make the process more secure.

Amm0 · Sat Mar 15, 2025 8:50 pm

For anyone messing around with Apple Virtualization and ARM-based Macs, I saw this on YouTube today:
https://www.youtube.com/watch?v=r3wpXhL3iA0
while about Slackware linux, it was interesting to see the kernel options he used aarch64 / "Apple Silicon" with Apple Virtualization. One thing I noticed was using console=hvc0 as a kernel option for Apple Virtualization on aarch64. And there was another article about Arch and Apple Virtualization that also suggest using "console=hvc0" too.

At some point I'll break down and get a new ARM Mac, since I still think it's possible to get RouterOS to run on Apple VZ using right ARM64 image or options. So mainly cataloging the clues.