Kernel doesn't boot on RPI 4 when root is defined as root=ZFS=zroot/ROOT/dataset

[IronDusts]

Using the official Centos 7.9 image provided here: Here

How to reproduce:

• Build ZFS from source according to these instructions.
• Build RPM packages, install them, build DKMS, load module, create pool, create a root dataset, set mointpoint, rsync the enture root volume on it, chroot and create initramfs, check with lsinitrd if everything zfs related is included... full migration to a ZFS root.
• Change the cmdline from root=/dev/mmcblk0p3 to root=ZFS=zroot/ROOT/centos and reboot.

Expected result:
System boots normally or if there's an issue with the initrd go to initramfs shell.

Actual result:
System goes into kernel panic.

• If stated 'rootwait' then the system waits indefinitely
• Even if try to add rd.shell it still goes to kernel panic without going to shell and giving troubleshoot option

Investigation steps and things I have checked:

• When change back the cmdline from root=ZFS=zroot/ROOT/centos to root=/dev/mmcblk0p3 then the system boots normally, and the pool gets imported with no issues.
• When booting into init=/bin/bash with root=ZFS=zroot/ROOT/centos the system still won't boot and go into kernel panic even if rd.shell is enabled
• When booting into init=/bin/bash with root=/dev/mmcblk0p3 the zfs module is not loaded. Even if I do modprobe zfs the zfs commands still do not work with module not loaded error. After manually mounting /proc and /sys then modprobing zfs everything works normal. The pool gets imported but I cannot continue booting with zfs root because the kernel cmdline already has /dev/mmcblk0p3 stated as root

[TheOneValen]

I have the same Problem. I am using openzfs 2.0.4 and compiled it myself for debian, arm64. I configured my system to boot into dropbear (ssh) and I can import the zpool there. I have no clue as to why the system does not boot in dmesg :(

[nabijaczleweli]

Are the modules in the initrd? Is the userspace tooling? What's the state and failure mode after you ssh in? Do you have some (virtualisable) way to reproduce this?

[TheOneValen]

The modules are in the initrd, as well as the userspace tooling. I can import the pool with alternate root and chroot into it. I have no idea how I could virtualize this situation though.

[nabijaczleweli]

Hm. What if you boot with "verbose debug" (and without "quiet")? You should see all errors and logs in /run/initramfs/initramfs.debug by the time you log in.

[IronDusts]

Hello,

I don't think the issue is about what was included or not included in the initramfs.

I cannot even access the initramfs shell.

Let me put it very simple.
with kernel parameter root=/dev/some/block/device - I get initramfs loaded and then a shell to troubleshoot whatever I need. That that's true even if I don't have the drivers for needed the rootfs or even if the rootfs path is wrong then I still can operate within the initram shell.

Same kernel, same initramfs image but kernel root parameter is:
root=ZFS/pool/some/device

Don't even get initramfs shell. It's not about if the kernel module is there or not, I cannot even load the initramfs at all, got kernel panic right away.

I believe that for some reason for raspberry pi the kernel can't handle ZFS root path and can't even start loading the initramfs.

[nabijaczleweli]

These are probably unrelated, yes

[TheOneValen]

Hm. What if you boot with "verbose debug" (and without "quiet")? You should see all errors and logs in /run/initramfs/initramfs.debug by the time you log in.

Finally I see something, thank you. There is an error that the pool cannot be imported.

Message: cannot import 'callisto': no such pool or dataset
	Destroy and re-create the pool from
	a backup source.

But manually importing it with the tools in the initramfs is fine o_O?

[nabijaczleweli]

Outdated zpool.cache, most likely! If you chroot into your real root, remove /etc/zfs/zpool.cache, and regenerate the initrd, you should be g2g next boot. If you want to keep using the cache (which you don't have to, some say it's faster on import, but it's also a hassle, YMMV), then you should be able to do zpool cachefile=none yourpool, then zpool cachefile=/etc/zfs/zpool.cache yourpool (or zpool cachefile="" yourpool) after you boot into your real system.

[TheOneValen]

OK, that sounds fixable. Could you point me to some documentation on how I can refresh that cache?

[nabijaczleweli]

Sorry, sent too early. See updated comment above.

[TheOneValen]

Thank you! It was a zpool.cache problem. I have a mirror and I just replaced the last drive of the original configuration. The initrd was not updated. After chroot and updating the cache file (it was not up to date) and regenerating the initrd, the system boots again. Just some curiosity:

• removing the zpool.cache file -> it will not boot
• updating the kernel to 5.10 (from 5.9) -> will not boot, module & zpool.cache up to date, manual import works

[nabijaczleweli]

Hm, what errors do you get in either case? The first point is particularly odd, since a nonexistent zpool.cache is very explicitly supported (and I boot many systems off cache-less pools).

[TheOneValen]

Do I have to set cachefile=none before booting an initrd without a cache file? I'll try to reproduce the state and get the log from initramfs.debug.

[nabijaczleweli]

I don't think so? But I don't know conclusively, sorry.

(When testing, please also take a look at whether /etc/zfs/zpool.cache (a) doesn't exist (b) exists but is empty or (c) is nonempty. I have a minor suspicion about a subtle bug in handling of (b).)