CHR ARM64 images are broken?

RouterOS Virtualization
1

General (x64) image partition 2 like below:

.
├── bin
│   ├── bash
│   └── milo
├── boot
├── dev
│   ├── bootdev
│   └── bootpart
├── lost+found
├── nova
│   └── etc
│       └── serial
├── rw
│   ├── autorun.scr
│   └── REBOOT
├── SHOW_LICENSE
├── UPGRADED
└── var
    └── pdb
        └── system
            └── image

10 directories, 10 files

Images of ARM64 (7.15beta9 testing) like below:
source: https://cdn.mikrotik.com/routeros/7.15beta9/chr-7.15beta9-arm64.img.zip

├── rw
└── var
    └── pdb
        └── system

4 directories, 0 files

When booting with the above image, cannot find some files.

[    0.740142] process '/init' started with executable stack
[    0.749886]  sda: sda1 sda2
[    0.754014] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
opendir: No such file or directory
ERROR: no system package found!
[    1.758207] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[    1.759258] SMP: stopping secondary CPUs
[    1.759869] Kernel Offset: disabled
[    1.760614] CPU features: 0x0000a,68002238
[    1.761060] Memory Limit: none
[    1.761340] Rebooting in 5 seconds..

test procedure (log)

root@debian-4gb-fsn1-2:~# curl -skL --retry 3 --connect-timeout 3 -o chr.img.zip https://download.mikrotik.com/routeros/7.15beta9/chr-7.15beta9-arm64.img.zip
unzip chr.*.zip
rm -rf chr.*.zip
Archive:  chr.img.zip
  inflating: chr-7.15beta9-arm64.img
root@debian-4gb-fsn1-2:~# # Load nbd module
sudo -E modprobe nbd
root@debian-4gb-fsn1-2:~# qemu-img convert -f raw -O qcow2 chr-*.img chr.qcow2
root@debian-4gb-fsn1-2:~# sudo -E qemu-nbd -c /dev/nbd0 chr.qcow2
root@debian-4gb-fsn1-2:~# sudo -E rm -rf /tmp/chr*
sudo -E mkdir /tmp/chr-test/
root@debian-4gb-fsn1-2:~# sudo -E mount /dev/nbd0p1 /tmp/chr-test/

root@debian-4gb-fsn1-2:~# ls /tmp/chr-test/ -l
total 1
drwxr-xr-x 3 root root 512 Mar 27 20:41 EFI
root@debian-4gb-fsn1-2:~# sudo -E mkdir /tmp/chr-p2/
root@debian-4gb-fsn1-2:~# sudo -E mount /dev/nbd0p2 /tmp/chr-p2/
root@debian-4gb-fsn1-2:~# tree /tmp/chr-p2
-bash: tree: command not found
root@debian-4gb-fsn1-2:~# apt install tree
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  tree
0 upgraded, 1 newly installed, 0 to remove and 14 not upgraded.
Need to get 50.9 kB of archives.
After this operation, 169 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main arm64 tree arm64 2.1.0-1 [50.9 kB]
Fetched 50.9 kB in 0s (1,621 kB/s)
Selecting previously unselected package tree.
(Reading database ... 34031 files and directories currently installed.)
Preparing to unpack .../tree_2.1.0-1_arm64.deb ...
Unpacking tree (2.1.0-1) ...
Setting up tree (2.1.0-1) ...
Processing triggers for man-db (2.11.2-2) ...
root@debian-4gb-fsn1-2:~# tree /tmp/chr-p2
/tmp/chr-p2
├── rw
└── var
    └── pdb
        └── system

5 directories, 0 files
root@debian-4gb-fsn1-2:~#
2

Wrong comparison method from the start.
Especially because you haven’t even explained well what you’re comparing and where you downloaded the files from,
what procedure did you use, etc., etc., etc.

You’ve definitely made some mistakes or the file you’re checking is a corrupt or incomplete download.

To me the official download appears to be all correct and without problems.

3

They work fine on ARM64 AMPERE machine type on Hetzner. Where are you trying to boot them?

4

I want to use them on Oracle Cloud (AMPERE/ARM64) instance.
But I have also tried to use it on Hetzner (CAX11 - ARM64/Ampere plan), and got the same result. I will try it again.

5

# procedure on CX11 (Intel x86) → working well

apt update && apt install -y unzip
curl -L https://cdn.mikrotik.com/routeros/7.15beta9/chr-7.15beta9.img.zip > mikrotik-chr.zip
funzip mikrotik-chr.zip > mikrotik-chr.img
dd if=./mikrotik-chr.img of=/dev/sda bs=1M
sync && reboot

# procedure on CAX11 (ARM64/AMPERE) → still failed

apt update && apt install -y unzip
curl -L https://cdn.mikrotik.com/routeros/7.15beta9/chr-7.15beta9-arm64.img.zip > mikrotik-chr.zip
funzip mikrotik-chr.zip > mikrotik-chr.img
dd if=./mikrotik-chr.img of=/dev/sda bs=1M
sync && reboot

Anyone knows what is the root cause of this? Thank you!

6

Can you provide some errors error code, debugging info ??
Is the dev the same in x86 and ARM ?? /dev/sda
And funzip ???

7

The devs are the same (/dev/sda) as below:

(With Hetzner CX11 Intel/x86 plan)

root@debian-2gb-fsn1-1:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            939M     0  939M   0% /dev
tmpfs           192M  628K  192M   1% /run
/dev/sda1        19G  1.7G   17G  10% /
tmpfs           960M     0  960M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda15      241M  138K  241M   1% /boot/efi
tmpfs           192M     0  192M   0% /run/user/0
/dev/nbd0p1      32M  3.4M   29M  11% /tmp/chr-p1
/dev/nbd0p2      90M   18M   72M  21% /tmp/chr-p2
root@debian-2gb-fsn1-1:~#

(With Hetzner CAX11 ARM64 plan)

root@debian-4gb-fsn1-2:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           383M  812K  382M   1% /run
/dev/sda1        38G  1.7G   35G   5% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda15      244M  152K  244M   1% /boot/efi
tmpfs           383M     0  383M   0% /run/user/0
root@debian-4gb-fsn1-2:~#

Full test procedure on x86:

root@debian-2gb-fsn1-1:~# curl -skL --retry 3 --connect-timeout 3 -o chr.img.zip https://download.mikrotik.com/routeros/7.15beta9/chr-7.15beta9.img.zip
root@debian-2gb-fsn1-1:~# unzip chr.*.zip
Archive:  chr.img.zip
  inflating: chr-7.15beta9.img
root@debian-2gb-fsn1-1:~# rm -rf chr.*.zip
root@debian-2gb-fsn1-1:~# sudo -E modprobe nbd
root@debian-2gb-fsn1-1:~# qemu-img convert -f raw -O qcow2 chr-*.img chr.qcow2
root@debian-2gb-fsn1-1:~# sudo -E qemu-nbd -c /dev/nbd0 chr.qcow2
root@debian-2gb-fsn1-1:~# sudo -E rm -rf /tmp/chr*
root@debian-2gb-fsn1-1:~# sudo -E mkdir /tmp/chr-p1/
root@debian-2gb-fsn1-1:~# sudo -E mkdir /tmp/chr-p2/
root@debian-2gb-fsn1-1:~# sudo -E mount /dev/nbd0p1 /tmp/chr-p1/
root@debian-2gb-fsn1-1:~# sudo -E mount /dev/nbd0p2 /tmp/chr-p2/
root@debian-2gb-fsn1-1:~# tree /tmp/chr-p1
/tmp/chr-p1
├── EFI
│   └── BOOT
│       └── BOOTX64.EFI
├── lost+found
└── map

4 directories, 2 files
root@debian-2gb-fsn1-1:~# tree /tmp/chr-p2
/tmp/chr-p2
├── bin
│   ├── bash
│   └── milo
├── boot
├── dev
│   ├── bootdev
│   └── bootpart
├── lost+found
├── nova
│   └── etc
│       └── serial
├── rw
│   ├── autorun.scr
│   └── REBOOT
├── SHOW_LICENSE
├── UPGRADED
└── var
    └── pdb
        └── system
            └── image

11 directories, 10 files
root@debian-2gb-fsn1-1:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            939M     0  939M   0% /dev
tmpfs           192M  628K  192M   1% /run
/dev/sda1        19G  1.7G   17G  10% /
tmpfs           960M     0  960M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda15      241M  138K  241M   1% /boot/efi
tmpfs           192M     0  192M   0% /run/user/0
/dev/nbd0p1      32M  3.4M   29M  11% /tmp/chr-p1
/dev/nbd0p2      90M   18M   72M  21% /tmp/chr-p2
root@debian-2gb-fsn1-1:~#
8

# procedure on CAX11 (ARM64/AMPERE) → still failed

apt update && apt install -y unzip
curl -L https://cdn.mikrotik.com/routeros/7.15beta9/chr-7.15beta9-arm64.img.zip > mikrotik-chr.zip
funzip mikrotik-chr.zip > mikrotik-chr.img
dd if=./mikrotik-chr.img of=/dev/sda bs=1M
sync && reboot

Do you know where I did it wrong?
I have updated the post to give more info about this.
Thank you!

9

Procedure to analyze the filesystem (due to the kernel panic info)

# Initialize environment
sudo -E apt update
sudo -E apt -y install curl qemu-utils rsync unzip zip dosfstools gdisk tree

# Download image for ARM64
curl -skL --retry 3 --connect-timeout 3 -o chr.img.zip https://download.mikrotik.com/routeros/7.15beta9/chr-7.15beta9.img.zip
unzip chr.*.zip
rm -rf chr.*.zip

# Load nbd module
sudo -E modprobe nbd

# Convert to qcow2
qemu-img convert -f raw -O qcow2 chr-*.img chr.qcow2

# Connect ndb
sudo -E qemu-nbd -c /dev/nbd0 chr.qcow2

# Create tmp dir
sudo -E rm -rf /tmp/chr*
sudo -E mkdir /tmp/chr-p1/
sudo -E mkdir /tmp/chr-p2/

# Mount
sudo -E mount /dev/nbd0p1 /tmp/chr-p1/
sudo -E mount /dev/nbd0p2 /tmp/chr-p2/

# List all files on partition-1
tree /tmp/chr-p1

# List all files on partition-2
tree /tmp/chr-p2

Normally the partition-2 seems have more files as below (for x86 image):

root@debian-2gb-fsn1-1:~# tree /tmp/chr-p2
/tmp/chr-p2
├── bin
│   ├── bash
│   └── milo
├── boot
├── dev
│   ├── bootdev
│   └── bootpart
├── lost+found
├── nova
│   └── etc
│       └── serial
├── rw
│   ├── autorun.scr
│   └── REBOOT
├── SHOW_LICENSE
├── UPGRADED
└── var
    └── pdb
        └── system
            └── image

11 directories, 10 files
root@debian-2gb-fsn1-1:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            939M     0  939M   0% /dev
tmpfs           192M  628K  192M   1% /run
/dev/sda1        19G  1.7G   17G  10% /
tmpfs           960M     0  960M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda15      241M  138K  241M   1% /boot/efi
tmpfs           192M     0  192M   0% /run/user/0
/dev/nbd0p1      32M  3.4M   29M  11% /tmp/chr-p1
/dev/nbd0p2      90M   18M   72M  21% /tmp/chr-p2
root@debian-2gb-fsn1-1:~#

But with the ARM64 image, it seems not complete.

root@debian-4gb-fsn1-2:~# tree /tmp/chr-p2
/tmp/chr-p2
├── rw
└── var
    └── pdb
        └── system

5 directories, 0 files
root@debian-4gb-fsn1-2:~#
10

Could you kindly give me the complete procedure of installation of it?

Because I use the same way to install it on ARM64 as below, and it cannot boot up correctly (which x86 is working fine).
https://community.hetzner.com/tutorials/mikrotik-chr-basic-setup

11

https://wiki.mikrotik.com/wiki/Manual:CHR_Hetzner

32

curl -L https://cdn.mikrotik.com/routeros/7.15beta9/chr-7.15beta9.img.zip | funzip | dd of=/dev/sda bs=1M

64

curl -L https://cdn.mikrotik.com/routeros/7.15beta9/chr-7.15beta9-arm64.img.zip | funzip | dd of=/dev/sda bs=1M

Inside the image file are present two partitions, and if you use dd for overwrite the filesystem from the same filesystem…
do not save the file on dev/sda but on another or on memory…

12

Yes, one is for x86. Another one is for ARM64 (which cannot correctly boot with the same procedure).
My question is why ARM64 image (chr-7.15beta9-arm64.img.zip) seems broken and likely it’s related to the boot up issue.

13

If it’s a problem here, why the same procedure is working with x86 images. I think they should have the same problem?
Thank you for replying.

14

Unquestionably
0.RouterOS Boot
map
0.RouterOS Boot\EFI
(empty)
0.RouterOS Boot\EFI\BOOT
BOOTX64.EFI
0.RouterOS Boot\lost+found
(empty)
1.RouterOS
SHOW_LICENSE
UPGRADED
1.RouterOS[SYS]
Journal
1.RouterOS\bin
bash
milo
1.RouterOS\boot
(empty)
1.RouterOS\dev
bootdev
bootpart
1.RouterOS\lost+found
(empty)
1.RouterOS\nova
(empty)
1.RouterOS\nova\etc
serial
1.RouterOS\rw
REBOOT
autorun.scr
1.RouterOS\var
(empty)
1.RouterOS\var\pdb
(empty)
1.RouterOS\var\pdb\system
image

0.RouterOS Boot
(missing map)
0.RouterOS Boot\EFI
(empty)
0.RouterOS Boot\EFI\BOOT
BOOTAA64.EFI (instead of BOOTX64.EFI)
missing: 0.RouterOS Boot\lost+found
1.RouterOS
missing: SHOW_LICENSE
missing: UPGRADED
var_pdb_system_image (instead “image” on var/pdb/system)
1.RouterOS[SYS]
Journal
missing: 1.RouterOS\bin
missing: bash
missing: milo
1.RouterOS\boot
(empty)
1.RouterOS\dev
bootdev
bootpart
1.RouterOS\lost+found
(empty)
missing: 1.RouterOS\nova
missing: 1.RouterOS\nova\etc
missing: serial
1.RouterOS\rw
missing: REBOOT
missing: autorun.scr
1.RouterOS\var
(empty)
1.RouterOS\var\pdb
(empty)
1.RouterOS\var\pdb\system
missing: image

15

Yes, this is my question here. Does these missing files cause the problem here?
Or ARM64 image just uses the different behavior or layout for the file system (which I don’t know, so I put a question mark)

16

Also missing “bootloader” at the start of the image…
FA 31 C0 8E D0 BC 00 7C 89 E6 50 07 50 1F FB FC BF 00 06 B9 00 01 F2 A5 EA 1D 06 00 00 BE BE 07 B3 04 80 3C 80 74 23 80 3C 00 75 09 83 C6 10 FE CB
75 EF CD 18 BE 9B 06 AC 3C 00 74 0B 56 BB 07 00 B4 0E CD 10 5E EB F0 EB FE 8B 14 8B 4C 02 89 F5 BF 05 00 BB 00 7C B8 01 02 57 CD 13 5F 73 0C 31 C0
CD 13 4F 75 ED BE 7C 06 EB CC BF FE 7D 81 3D 55 AA 75 C0 89 EE EA 00 7C 00 00 45 72 72 6F 72 20 6C 6F 61 64 69 6E 67 20 6F 70 65 72 61 74 69 6E 67
20 73 79 73 74 65 6D 00 4D 69 73 73 69 6E 67 20 6F 70 65 72 61 74 69 6E 67 20 73 79 73 74 65 6D …
64 bit is simply empty… all bytes set to 00

17

Sounds like something wrong with their image packing script for ARM64.

18

Sorry for confusion, it seems the build server has an issue, internal builds worked, but not public release versions. Please wait for next betas to test this.

In short, publicly downloadable ARM64 CHR is not working right now.

19

Thank you for replying. I will try it again with the next release.

20

The files in partition-2 seem complete. But still have problem here:

UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table
      FS0: Alias(s):HD0a1b:;BLK1:
          PciRoot(0x0)/Pci(0x5,0x7)/Pci(0x0,0x0)/Scsi(0x0,0x1)/HD(1,GPT,82A03785-E9A5-48A2-BAD3-F186DB97256A,0x800,0x10800)
     BLK3: Alias(s):
          VenHw(93E34C7E-B50E-11DF-9223-2443DFD72085,00)
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x5,0x7)/Pci(0x0,0x0)/Scsi(0x0,0x1)
     BLK2: Alias(s):
          PciRoot(0x0)/Pci(0x5,0x7)/Pci(0x0,0x0)/Scsi(0x0,0x1)/HD(2,GPT,FA1BF794-8C0D-4CBE-B632-63824D860B6A,0x11000,0x2E000)


Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell> ize: cluster chain corrupt
FATDirSize: cluster chain corrupt
FATDirSize: cluster chain corrupt

Files in partitions:

/tmp/chr-p1
└── EFI
    └── BOOT
        └── BOOTAA64.EFI

3 directories, 1 file
/tmp/chr-p2
├── boot
├── lost+found
├── rw
└── var
    └── pdb
        └── system
            └── image

7 directories, 1 file