 CHR using Apple Virtualization & 🐧 QEMU for Linux - Ready-to-use!

[Amm0]

Interesting. I do get the /system/check-installation on ARM64 emulated on Intel Mac... I tried both UTM "import" (using utm:// link) and "alias" (downloading the ZIP/UTM file, which runs in place). I'd thought it maybe just the import. Both QEMU ARM64 images do fail for me too.

I think in mikropkl, I've always used the "real" images. Only the fat-chr project did I repackage ARM64/aarch64 in a few build. I downloaded the QEMU ARM64 CHR from GitHub, and MikroTik version...

diff show them identical between what's download and MikroTik's image. Strange...

But... when you add UTM CHR "by hand", UTM converts the chr*.img to chr*.qcow2 when creating a new machine in its UI. mikropkl actually take the more conservative approach to leaving it a "RAW" .img — since that's what MikroTik offers. That less space efficient but perhaps .qcow does more....

One suspect is disk geometry may get calculated differently, someplace, and that's what's trigger the error. Perhaps UTM/qemu, or limitation of "RAW" images (which may not easy to "know" the geometry, where as qcow2 may encode that. e.g. QEMU has the "right info" to return for qcow2 but raw – IDK, but that's my thought.

If image is identical, something is going wrong elsewhere... Perhaps size is not the only reason UTM always converts raw to qcow (or. maybe ASIF in some cases)... If any calculation is "off-by-one", that likely "trip" the /system/check-installation.

I could try the repacking, but I liked that I could say "it's the 'real' image". But have the code from chr-fat to the conversion, but need to dig around more next I have a Silicon Mac handy. It could easily be something else not aligned, like perhaps the efvars.fd (where EFI data is stored), or disk GUID need to match something (GUID in .plist are auto-generated on GitHub when built).

Both Intel QEMU and Apple, pass the /system/check-installation. But QEMU with ARM64 fails the check-installation on Intel (using emulation and Silicon Mac (using virtualization).

The workaround is remove the disk and add the "official" one, that will convert it to QCOW and fixes the error. Now you'd need to save any data for this approach...if these was not a test system and/or save configuration/backup

But ironically the one that **is** repacked for EFI, Apple Virtualization, it says "installation is ok":

[admin@MikroTik] > /system/check-installation 
  status: installation is ok
[admin@MikroTik] > /system/resource/print
                   uptime: 2h55m2s                                             
                  version: 7.22 (stable)                                       
               build-time: 2026-03-09 08:38:02                                 
         factory-software: 7.1                                                 
              free-memory: 674.7MiB                                            
             total-memory: 1024.0MiB                                           
                      cpu: Intel(R)                                                 
        architecture-name: x86_64                                              
               board-name: CHR Apple Inc. Apple Virtualization Generic Platform
                 platform: MikroTik          

And the QEMU on Intel, do same. Just ARM64, and just with using .raw images, do you get the 'check-installation. UTM imports and converts to qcow2, which work to avoid the check-instillation` error.

[optio]

Not sure that just conversion to QCOW format will be enough. Just tried with converted image, same issues.

Converted with qemu-img convert -f raw chr-7.22-arm64.img chr-7.22-arm64.qcow2and then chr-7.22-arm64.qcow2 image is added as NVMe drive (not as raw).

[Amm0]

TL;DR: "bad image" is repo'able with QEMU "everywhere", not just UTM

See report-arm64-chr-check-installation-failures

Repo "bad image" on Apple Silicon

Although I did not doubt and see in ARM64-emulation on Intel. I did try on "real" Apple Silicon, and got the same error with /system/check-installation with ARM64 CHR & QEMU (with UTM), either manually downloading from MikroTik or using the mikropkl packages. No major trick I could find with just UTM UI and "Stock" Packages either.

Tried using libvirt on GitHub Actions...

To get the bottom of the ARM64 /system/installation issue, I added experimental "libvirt support" to mikropkl. So future packages will include a valid libvirt XML to launch CHR with same matching params as UTM QEMU, but uses Linux KVM/QEMU instead. (extra files are ignored by UTM, and tiny). My goal was not libvirt support...I did that to test CHR on a Linux aarch64 in GitHub Actions to if we get the /system/check-installation issue there. GitHub uses Ubuntu ARM64, idea see what works there, no Apple-ism.... But this did not "fix" the /system/check-installation "bad image" problem on ARM64. This largely tried the option we use in UTM, but did see if other QEMU options or packaging might fix.

Stuck CoPilot on using an Intel Mac with ARM64 images using brew's qemu-system-aarch64

After many iteration of trying and testing QEMU setting with CHR ARM64 images, and deep dive into the structure of the package. The whole story is catalog in GitHub - tikoci/mikropkl: `pkl` packaged UTM virtual machines, including RouterOS CHR · GitHub project, mainly in the Lab/**/NOTES.md files, with some in AGENTS.md/CLAUDE.md (that might have more details of learnings). The TL;DR: MikroTik has fix something to avoid the error in ARM64 CHR with QEMU OR QEMU needs to support emulate

Research and Findings

ARM64 investigation gory details. More CHR learned were captured in other CLAUDE/AGENTS too.

The report linked at top summarized it well here:

The deeper reason is architectural: we extracted and disassembled the checker binary from both x86 and ARM64 CHR images. The x86 checker always succeeds — after scanning for hardware files, it unconditionally runs a fallback program (/bin/milo) and returns success. The ARM checker has no such fallback — when hardware descriptor files are missing, it returns failure, which RouterOS reports as "damaged system package: bad image." This is a design difference in MikroTik's firmware, not a QEMU configuration problem.

We explored whether different QEMU settings could resolve this:

• Using a different CPU model (we tried four variants including ones that match real Armada7040 hardware): no effect — the check doesn't care about CPU model, it cares about hardware descriptors
• Disabling ACPI to force QEMU to expose better hardware info: RouterOS then can't find the disk at all, because it relies on ACPI to discover the virtual storage controller
• Using MMIO transport (virtio-blk-device) with ACPI disabled: RouterOS kernel doesn't have virtio-mmio drivers — stalls at boot
• Injecting SMBIOS data to mimic MikroTik hardware: changes board-name display but doesn't affect the check
• Patching the device tree with Marvell hardware identifiers: kernel ignores the DTB when ACPI is present
• qcow2 instead of raw disk format: no difference — disk format is irrelevant
• Different UEFI firmware: no effect — the firmware starts fine either way, the problem is downstream in RouterOS itself

Every path leads to either "boot works, check fails" or "disk not found, nothing works." There's no configuration of QEMU's standard virtual machine type that satisfies RouterOS's hardware expectations for ARM64.

"We" in above. AI wrote up NOTES and tests scripts ... I asked for a "report" (long prompt) and reviewed and light edited the report-arm64-chr-check-installation-failures - the NOTES.md were all CoPilot+Sonnet's doing (read them all but it more itself it keep track of what done/tried... in case I come back... But CoPilot knows how slice-and-dice CHR now, see tooling below.

macOS tools for FAT and disk images

I did not know these, CoPilot did:

# List files on FAT filesystem without mounting
mdir -i /tmp/efi.fat ::
mdir -i /tmp/efi.fat ::/EFI/BOOT/

# Extract kernel from FAT partition
mcopy -i /tmp/efi.fat ::/EFI/BOOT/BOOTAA64.EFI /tmp/kernel

# Identify kernel type
file /tmp/kernel

# Mount CHR image partition on macOS (read-only)
hdiutil attach -nomount chr-7.22.img
# then: diskutil list to find partition device, mount manually

# Check QEMU firmware file sizes (pflash must match)
ls -la /usr/local/share/qemu/edk2-*
ls -la /opt/homebrew/share/qemu/edk2-*     # Apple Silicon

(brew install mtools get the FAT mcopy/mdir commands ... learn something new everyday. )

[Amm0]

CHR with QEMU Tests using GitHub CI

mikropkl which builds the UTM images now has a GitHub Action "workflow" that tests via QEMU path. I turned this support into a "standard feature" of the UTM images. QEMU test use options, largely, same as UTM image uses (and QEMU setting can, for the most part, be added to UTM, if needed). Both do use same package, since the "UTM Packages" are just ZIP files to Linux.

This means there are finally some "sanity tests" tests. I do test the images before any get marked at "Released" in GitHub. I published 7.22 and 7.23beta3 with new QEMU support. See mikropkl README.md for details

QEMU on macOS "Silicon" (aarch64)

This is on GitHub using QEMU directly (via brew install qemu) but using a "real" Mac (as GitHub Action Runner).

Note One critical problem is Mac-on-GitHub do not support Hypervisor.framework, it been disabled. This is need for UTM or QEMU, or Apple Virtualization. So test below are using QEMU tcg emulation. But all images do boot and work.

The qemu.sh script is same as in "UTM bundles" - but does detect the hypervisor support, so the "real" Mac test, the included qemu.sh did correctly "downgrade".

From Test: QEMU Direct Boot · tikoci/mikropkl@f0bdd3a · GitHub

Example test (1 of 5)

It actually a long to show all test. But here is what one looks like. All images purposed are changed, showing just apple.aarch64 (for 7.23beta2) here:

Boot chr.aarch64.apple.7.23beta2
Config: arch=aarch64 mem=1024M cpus=2 disks=1
Runner: aarch64, KVM=0 → expected accel=tcg
Port: 9180, Timeout: 60s
Starting chr.aarch64.apple.7.23beta2 (port 9180, accel=tcg,tb-size=256)...
QEMU PID=15445 — log: /tmp/qemu-chr.aarch64.apple.7.23beta2.log
Serial: socat - UNIX-CONNECT:/tmp/qemu-chr.aarch64.apple.7.23beta2-serial.sock
Monitor: socat - UNIX-CONNECT:/tmp/qemu-chr.aarch64.apple.7.23beta2-monitor.sock
Serial capture PID=15449 → /tmp/qemu-chr.aarch64.apple.7.23beta2-serial.log
✓ Binary: qemu-system-aarch64
✓ Accelerator: tcg
✓ UEFI pflash present
✓ Port forwarding: host 9180 → guest 80
Attempt 1/12: not ready (qemu_pid=15445 state=R< cpu= 99.8%) — waiting 5s
Attempt 2/12: not ready (qemu_pid=15445 state=S< cpu= 20.1%) — waiting 5s
Attempt 3/12: not ready (qemu_pid=15445 state=S< cpu= 12.0%) — waiting 5s
✓ RouterOS HTTP up after 30s
✓ GET /system/resource
✓ GET /interface
✓ GET /ip/address
✓ GET /ip/route
✓ GET /ipv6/address
✓ GET /ipv6/route
/system/resource:
{
"architecture-name": "arm64",
"board-name": "CHR QEMU QEMU Virtual Machine",
"build-time": "2026-03-13 09:52:01",
"cpu": "ARM64",
"cpu-count": "2",
"cpu-load": "7",
"free-hdd-space": "72286208",
"free-memory": "819560448",
"platform": "MikroTik",
"total-hdd-space": "86183936",
"total-memory": "1073741824",
"uptime": "24s",
"version": "7.23beta2 (development)",
"write-sect-since-reboot": "1048",
⊘ Skipping check-installation on aarch64 (see CLAUDE.md)
2026/03/20 03:25:29 socat[15449] W exiting on signal 15
✓ chr.aarch64.apple.7.23beta2: ALL CHECKS PASSED
Boot chr.aarch64.qemu.7.23beta2
Boot chr.x86_64.apple.7.23beta2
Boot chr.x86_64.qemu.7.23beta2
Boot rose.chr.aarch64.qemu.7.23beta2
Boot rose.chr.x86_64.qemu.7.23beta2

MacOS results for all images

Note Test for X86 on ARM are skipped, the emulation is too slow. As noted, tsg mode is used, which from previous tests is actually extremely slow at X86 virtualization on ARM (it's a known thing actually, not CHR's fault here & esoteric case mainly as an "extreme test") - so the X86 on ARM is always skipped. X86 test always run all builds, including ARM64 ones (see further below).

This is the summary from "mac-15":

══════════════════════════════════════════════════════════
Boot timing — aarch64/Darwin runner (3 passed, 0 failed, 3 skipped)
══════════════════════════════════════════════════════════
✓ chr.aarch64.apple.7.23beta2: 30s (tcg)
✓ chr.aarch64.qemu.7.23beta2: 30s (tcg)
⊘ chr.x86_64.apple.7.23beta2: skipped (x86_64 on aarch64)
⊘ chr.x86_64.qemu.7.23beta2: skipped (x86_64 on aarch64)
✓ rose.chr.aarch64.qemu.7.23beta2: 30s (tcg)
⊘ rose.chr.x86_64.qemu.7.23beta2: skipped (x86_64 on aarch64)
══════════════════════════════════════════════════════════

"UTM Package" on ARM64/aarch64 with Ubuntu

From GitHub Action: Test: QEMU Direct Boot · tikoci/mikropkl@f0bdd3a · GitHub

Summary

══════════════════════════════════════════════════════════
Boot timing — aarch64/Linux runner (3 passed, 0 failed, 3 skipped)
══════════════════════════════════════════════════════════
✓ chr.aarch64.apple.7.22: 29s (tcg)
✓ chr.aarch64.qemu.7.22: 29s (tcg)
⊘ chr.x86_64.apple.7.22: skipped (x86_64 on aarch64)
⊘ chr.x86_64.qemu.7.22: skipped (x86_64 on aarch64)
✓ rose.chr.aarch64.qemu.7.22: 22s (tcg)
⊘ rose.chr.x86_64.qemu.7.22: skipped (x86_64 on aarch64)
══════════════════════════════════════════════════════════

"UTM Package" on Intel/x86_64 with Ubuntu

From GitHub Action: Test: QEMU Direct Boot · tikoci/mikropkl@f0bdd3a · GitHub

Summary

══════════════════════════════════════════════════════════
Boot timing — x86_64/Linux runner (6 passed, 0 failed, 0 skipped)
══════════════════════════════════════════════════════════
✓ chr.aarch64.apple.7.23beta2: 29s (tcg)
✓ chr.aarch64.qemu.7.23beta2: 30s (tcg)
✓ chr.x86_64.apple.7.23beta2: 13s (kvm)
✓ chr.x86_64.qemu.7.23beta2: 13s (kvm)
✓ rose.chr.aarch64.qemu.7.23beta2: 29s (tcg)
✓ rose.chr.x86_64.qemu.7.23beta2: 13s (kvm)
══════════════════════════════════════════════════════════

7.22 GitHub Test (Overview Page)

The same test, except skipping Mac (e.g. emulated QEMU is not particular valid since most "actually real" Mac's support the Hypervisor.framework. From Test: QEMU Direct Boot · tikoci/mikropkl@f0bdd3a · GitHub

QEMU Support Doc

For deeper overview of how the qemu.sh used but test works on downloaded imaged, see mikropkl/Files/QEMU.md at main · tikoci/mikropkl · GitHub

Stay tuned for more updates

[Amm0]

Please check out my new site around much of the CHR image/repackaging/etc work here.

CHR Images — tikoci.github.io

Please see updated "top post" above for more details, with a summary of the related work in GitHub tikoci/*.

For our "long time readers" of the thread, here is one more piece of the puzzle of CHR - guest tools...

Lab Report: Guest Tool Support in CHR 7.22 and 7.23beta

As part of my "remodeling" of mikropkl (which make the UTM package, and now qemu.sh/qemu.cfg included too), I had the clunkers (here, CoPilot using Claude Sonnet or Opus) run some "experiments" on CHR. One I did not mention here checking out the "Guest Tool" of RouterOS as viewed from QEMU. "Guest tools" let the Hypervisor (or with QEMU, scripts linked to the hypervisor) do things like a proper shutdown, get stats, and, apparently, even issue commands.

Nothing in packages, qemu.sh, or else mikropkl uses QGA yet. But it be one way to install packages more automatically, or get stats. I looked this since most of current mikropkl project is focused on building "UTM" packages (which are ZIP files) without repackage if not needed. But like to "expand" the QEMU stuff and support local use better than download images directory (e.g. use sh or bun script locally to "build" a CHR)...and the "guest tool" likely can automate package install, and collect stats from RouterOS POV (which can be checked against qemu's elaborate set details on its performance available via some socket while a machine is running.

TL;DR: two findings about "QGA":

• X86 works better than documented using VirtIO-based Guest Tools from QEMU.
• ARM64 has no support at all

AI wrote a report here - based on real testing using the python - on the test/results of trying CHR against the full set of guest tool operations to figure out which one worked, and which ones did not. See:

Lab/qemu-guest-agent/NOTES.md

[Amm0]

Update on Apple Silicon support

I should also note the underlying Apple Silicon issue is more clear these days. I never updated this thread. AFAIK, RouterOS is using arm32 binaries in the ARM64 CHR. But the issue is macOS under Apple Silicon (aarch64/ARM64) does not support 32-bit ARM32 binaries.

So RouterOS 32-bit binary's are why UTM nor QEMU cannot actually use "native" Hypervisor.framework on Apple Silicon (which underpins the Virtualization.framework discussed here, _and -accel hvf in QEMU terms). On macOS on Intel, it is really is just the FAT32 and EFI that's needed to get access to non-emulated (QEMU -accel tcg). But on macOS on Apple Silicon (M1/M2/M3/M4/...), it the ARM32 init and few other ones that prevent RouterOS CHR from booting on Mx Mac's — so the binary types CANNOT be fixed by re-packaging and only fix be from MikroTik to build a "true"/pure ARM64 version of CHR ARM64.

QuickCHR test results from last 60 days...

Now testing using emulation seems to be a15-30% penalty. More exact data from quickchr is below ( GitHub - tikoci/quickchr: `quickchr` - Fastest path to MikroTik RouterOS CHR instance for testing and scripts · GitHub is robust CLI around the basic QEMU support in the "CHR Images" download above).

quickchr has an elaborate CI system that does extensive cross-platform testing of bunx @tikoci/quickchr ..., so the various boot times are collected. So the effect of QEMU -accel is tracked ("accel" which is what controls if an emulated CPU is used of if "native" KVM/HVF is used).

platform	accel	RouterOS	boots	dropped	median	min	max
linux-arm64	tcg	7.24rc3	61	0	33.9s	33.8s	34s
linux-arm64	tcg	7.24rc2	108	0	33.9s	33.8s	34s
linux-arm64	tcg	7.24rc1	51	0	38.5s	33.8s	38.6s
linux-arm64	tcg	7.23.3	261	0	33.9s	28.9s	34.4s
linux-arm64	tcg	7.23.2	1095	0	38.5s	29s	45.5s
linux-arm64	tcg	7.23.1	306	0	38.5s	29s	45.2s
linux-arm64	tcg	7.21.5	218	0	36.4s	29s	40.5s
linux-arm64	tcg	7.21.4	76	0	36.5s	29s	40.7s
linux-arm64	tcg	7.20.8	98	0	33.6s	29.6s	34.1s
linux-arm64	tcg	7.20.7	71	0	33.6s	33.5s	33.6s
linux-x86	kvm	7.24rc3	61	0	23.8s	23.6s	23.9s
linux-x86	kvm	7.24rc2	162	0	23.8s	23.6s	24.2s
linux-x86	kvm	7.24rc1	62	0	25.8s	23.7s	34.2s
linux-x86	kvm	7.24beta3	26	1	27.8s	23.7s	34.4s
linux-x86	kvm	7.23.3	252	0	23.8s	23.5s	25.6s
linux-x86	kvm	7.23.2	1050	7	25.8s	23.6s	34.5s
linux-x86	kvm	7.23.1	300	2	25.8s	23.6s	34.4s
linux-x86	kvm	7.21.5	221	0	25.8s	23.6s	32s
linux-x86	kvm	7.21.4	62	0	25.8s	23.7s	32.1s
linux-x86	kvm	7.20.8	105	0	23.5s	23.5s	23.5s
linux-x86	kvm	7.20.7	78	0	23.5s	23.5s	23.6s
macos-arm64	tcg	7.24rc3	61	0	29.6s	24.4s	35s
macos-arm64	tcg	7.24rc2	135	0	29.6s	24.6s	35.1s
macos-arm64	tcg	7.24rc1	35	0	29.5s	24.1s	29.9s
macos-arm64	tcg	7.23.3	56	0	29.6s	24.4s	35.4s
macos-arm64	tcg	7.23.2	357	0	29.6s	24s	40s
macos-arm64	tcg	7.23.1	67	0	29.5s	24.1s	34.7s
macos-arm64	tcg	7.21.5	196	0	29.6s	24.2s	34.9s
macos-arm64	tcg	7.21.4	35	0	29.6s	24.5s	34.9s
macos-arm64	tcg	7.20.8	60	0	29.3s	23.8s	34.5s
macos-arm64	tcg	7.20.7	33	0	29.4s	24.3s	34.8s
macos-x86	hvf	7.24rc3	7	0	38.9s	36.3s	45.7s
macos-x86	hvf	7.24rc1	8	0	35.9s	31.3s	38.3s
macos-x86	hvf	7.23.3	25	0	33.9s	30.7s	39s
macos-x86	hvf	7.23.2	37	0	35.3s	30s	40.9s
macos-x86	hvf	7.23.1	17	0	33.9s	30.6s	40.1s
macos-x86	hvf	7.21.5	7	0	34.5s	30s	34.8s
macos-x86	hvf	7.21.4	8	0	34.5s	31.4s	35.8s
macos-x86	hvf	7.20.8	5	0	33.5s	30.3s	34.9s
macos-x86	hvf	7.20.7	4	0	32.1s	29.8s	39.4s
macos-x86	tcg	7.23.3	84	0	34.1s	33.8s	44.6s
macos-x86	tcg	7.20.8	3	0	34s	33.8s	38.8s
macos-x86	tcg	7.20.7	3	0	34.1s	33.7s	38.9s
windows-x86	tcg	7.24rc3	60	0	34s	33.9s	46.6s
windows-x86	tcg	7.24rc2	133	0	34s	33.9s	45.8s
windows-x86	tcg	7.24rc1	64	0	34s	33.9s	44.1s
windows-x86	tcg	7.23.3	29	0	34s	33.9s	39s
windows-x86	tcg	7.23.2	359	0	34s	28.9s	46.2s
windows-x86	tcg	7.23.1	14	0	34s	28.9s	34.3s
windows-x86	tcg	7.21.5	222	0	34s	28.8s	45.6s
windows-x86	tcg	7.21.4	37	0	34s	33.8s	40s
windows-x86	tcg	7.20.8	37	0	33.8s	28.7s	38.8s
windows-x86	tcg	7.20.7	35	0	33.8s	28.7s	34s

I should note that there have been ZERO bugs found in ANY RouterOS version from there tests, which do more than just boot but also license/device-mode/snapshots/guest-tools/users/licensing/etc using REST/console/QGA. But the above times are strictly boot to "REST API works". Only issues found in above tests is where outside of "core" RouterOS. e.g. when emulation is required, like Apple Silicon, tests failed.. The only MikroTik-related things are occasional transitory download issues, which has now been hardened against various DNS-related failures and/or trying manual DNS lock, explict IPv4 IPv6 as fallback to handle various MikroTik infastructure related things in quickchr.