Fedora - ZFS/2.0.0 : zfs modules not loaded

[gogi983]

System information

Type	Version/Name
Distribution Name	Fedora
Distribution Version	33
Linux Kernel	5.8 --> 5.9
Architecture	amd64
ZFS Version	2.0.0
SPL Version	2.0.0

Describe the problem you're observing

Using Fedora 33 actually with ZFS on Root :

NAME FSTYPE
sda
├─sda1 bios_boot_part
├─sda2 vfat
├─sda3 zfs_member 5000

The problem appeared since zfs/0.8.5 version release. After a kernel-upgrade, Fedora keeps the last three kernels available on the system.
Despite the fact that dkms builds correctly the modules for the new kernel, and all the files and/or links are correctly included in the initramfs images for all the installed kernels, one can not boot anymore with previous ones already installed.
Example :

$ dkms status
zfs, 0.8.5, 5.8.18-300.fc33.x86_64, x86_64: installed
zfs, 0.8.5, 5.9.10-200.fc33.x86_64, x86_64: installed

$ ls -l /var/lib/dkms/zfs/
укупно 2
drwxr-xr-x. 4 root root 5 25. нов. у 12:53 0.8.5
lrwxrwxrwx. 1 root root 35 23. нов. у 22:31 kernel-5.8.18-300.fc33.x86_64-x86_64 -> 0.8.5/5.8.18-300.fc33.x86_64/x86_64
lrwxrwxrwx. 1 root root 35 25. нов. у 12:53 kernel-5.9.10-200.fc33.x86_64-x86_64 -> 0.8.5/5.9.10-200.fc33.x86_64/x86_64

$ ls -l /var/lib/dkms/zfs/0.8.5/
укупно 2
drwxr-xr-x. 3 root root 3 23. нов. у 22:31 5.8.18-300.fc33.x86_64
drwxr-xr-x. 3 root root 3 25. нов. у 12:53 5.9.10-200.fc33.x86_64
lrwxrwxrwx. 1 root root 18 23. нов. у 22:28 source -> /usr/src/zfs-0.8.5

$ sudo lsinitrd /boot/initramfs-5.9.10-200.fc33.x86_64.img | grep zfs
[судо] лозинка за корисника gogi:
zfs
-rw-r--r-- 1 root root 35 Oct 23 17:08 etc/modprobe.d/zfs.conf
-rw-r--r-- 1 root root 324 Oct 23 17:30 etc/udev/rules.d/90-zfs.rules
-rwxr-xr-x 1 root root 32752 Oct 23 17:30 usr/lib64/libzfs_core.so.1.0.0
lrwxrwxrwx 1 root root 32 Oct 23 17:30 usr/lib64/libzfs_core.so.1 -> ../../lib64/libzfs_core.so.1.0.0
-rwxr-xr-x 1 root root 505696 Oct 23 17:30 usr/lib64/libzfs.so.2.0.0
lrwxrwxrwx 1 root root 27 Oct 23 17:30 usr/lib64/libzfs.so.2 -> ../../lib64/libzfs.so.2.0.0
-rwxr-xr-x 1 root root 269 Oct 23 17:30 usr/lib/dracut/hooks/cleanup/99-zfs-needshutdown.sh
-rwxr-xr-x 1 root root 2081 Oct 23 17:30 usr/lib/dracut/hooks/cmdline/95-parse-zfs.sh
-rwxr-xr-x 1 root root 2553 Oct 23 17:30 usr/lib/dracut/hooks/mount/98-mount-zfs.sh
-rwxr-xr-x 1 root root 2619 Oct 23 17:30 usr/lib/dracut/hooks/pre-mount/90-zfs-load-key.sh
-rwxr-xr-x 1 root root 484 Oct 23 17:30 usr/lib/dracut/hooks/shutdown/20-export-zfs.sh
-rwxr-xr-x 1 root root 4860 Oct 23 17:30 usr/lib/dracut-zfs-lib.sh
-rw-r--r-- 1 root root 778788 Oct 23 17:30 usr/lib/modules/5.9.10-200.fc33.x86_64/extra/zfs.ko.xz
-rw-r--r-- 1 root root 152 Oct 23 17:30 usr/lib/modules-load.d/zfs.conf
-rwxr-xr-x 1 root root 2015 Oct 23 17:30 usr/lib/systemd/system-generators/dracut-zfs-generator
lrwxrwxrwx 1 root root 20 Oct 23 17:30 usr/lib/systemd/system/initrd.target.wants/zfs-import.target -> ../zfs-import.target
-rw-r--r-- 1 root root 365 Oct 23 17:30 usr/lib/systemd/system/zfs-env-bootfs.service
-rw-r--r-- 1 root root 498 Oct 23 17:30 usr/lib/systemd/system/zfs-import-cache.service
-rw-r--r-- 1 root root 465 Oct 23 17:30 usr/lib/systemd/system/zfs-import-scan.service
-rw-r--r-- 1 root root 101 Oct 23 17:30 usr/lib/systemd/system/zfs-import.target
drwxr-xr-x 2 root root 0 Oct 23 17:30 usr/lib/systemd/system/zfs-import.target.wants
lrwxrwxrwx 1 root root 25 Oct 23 17:30 usr/lib/systemd/system/zfs-import.target.wants/zfs-env-bootfs.service -> ../zfs-env-bootfs.service
lrwxrwxrwx 1 root root 27 Oct 23 17:30 usr/lib/systemd/system/zfs-import.target.wants/zfs-import-cache.service -> ../zfs-import-cache.service
lrwxrwxrwx 1 root root 26 Oct 23 17:30 usr/lib/systemd/system/zfs-import.target.wants/zfs-import-scan.service -> ../zfs-import-scan.service
-rwxr-xr-x 1 root root 28368 Oct 23 17:30 usr/sbin/mount.zfs
-rwxr-xr-x 1 root root 138304 Oct 23 17:30 usr/sbin/zfs

$ sudo lsinitrd /boot/initramfs-5.8.18-300.fc33.x86_64.img | grep zfs
zfs
-rw-r--r-- 1 root root 35 Oct 23 17:08 etc/modprobe.d/zfs.conf
-rw-r--r-- 1 root root 324 Oct 23 17:30 etc/udev/rules.d/90-zfs.rules
-rwxr-xr-x 1 root root 32752 Oct 23 17:30 usr/lib64/libzfs_core.so.1.0.0
lrwxrwxrwx 1 root root 32 Oct 23 17:30 usr/lib64/libzfs_core.so.1 -> ../../lib64/libzfs_core.so.1.0.0
-rwxr-xr-x 1 root root 505696 Oct 23 17:30 usr/lib64/libzfs.so.2.0.0
lrwxrwxrwx 1 root root 27 Oct 23 17:30 usr/lib64/libzfs.so.2 -> ../../lib64/libzfs.so.2.0.0
-rwxr-xr-x 1 root root 269 Oct 23 17:30 usr/lib/dracut/hooks/cleanup/99-zfs-needshutdown.sh
-rwxr-xr-x 1 root root 2081 Oct 23 17:30 usr/lib/dracut/hooks/cmdline/95-parse-zfs.sh
-rwxr-xr-x 1 root root 2553 Oct 23 17:30 usr/lib/dracut/hooks/mount/98-mount-zfs.sh
-rwxr-xr-x 1 root root 2619 Oct 23 17:30 usr/lib/dracut/hooks/pre-mount/90-zfs-load-key.sh
-rwxr-xr-x 1 root root 484 Oct 23 17:30 usr/lib/dracut/hooks/shutdown/20-export-zfs.sh
-rwxr-xr-x 1 root root 4860 Oct 23 17:30 usr/lib/dracut-zfs-lib.sh
lrwxrwxrwx 1 root root 65 Oct 23 17:30 usr/lib/modules/5.8.18-300.fc33.x86_64/weak-updates/zfs.ko.xz -> ../../../../../lib/modules/5.9.10-200.fc33.x86_64/extra/zfs.ko.xz
-rw-r--r-- 1 root root 778788 Oct 23 17:30 usr/lib/modules/5.9.10-200.fc33.x86_64/extra/zfs.ko.xz
-rw-r--r-- 1 root root 152 Oct 23 17:30 usr/lib/modules-load.d/zfs.conf
-rwxr-xr-x 1 root root 2015 Oct 23 17:30 usr/lib/systemd/system-generators/dracut-zfs-generator
lrwxrwxrwx 1 root root 20 Oct 23 17:30 usr/lib/systemd/system/initrd.target.wants/zfs-import.target -> ../zfs-import.target
-rw-r--r-- 1 root root 365 Oct 23 17:30 usr/lib/systemd/system/zfs-env-bootfs.service
-rw-r--r-- 1 root root 498 Oct 23 17:30 usr/lib/systemd/system/zfs-import-cache.service
-rw-r--r-- 1 root root 465 Oct 23 17:30 usr/lib/systemd/system/zfs-import-scan.service
-rw-r--r-- 1 root root 101 Oct 23 17:30 usr/lib/systemd/system/zfs-import.target
drwxr-xr-x 2 root root 0 Oct 23 17:30 usr/lib/systemd/system/zfs-import.target.wants
lrwxrwxrwx 1 root root 25 Oct 23 17:30 usr/lib/systemd/system/zfs-import.target.wants/zfs-env-bootfs.service -> ../zfs-env-bootfs.service
lrwxrwxrwx 1 root root 27 Oct 23 17:30 usr/lib/systemd/system/zfs-import.target.wants/zfs-import-cache.service -> ../zfs-import-cache.service
lrwxrwxrwx 1 root root 26 Oct 23 17:30 usr/lib/systemd/system/zfs-import.target.wants/zfs-import-scan.service -> ../zfs-import-scan.service
-rwxr-xr-x 1 root root 28368 Oct 23 17:30 usr/sbin/mount.zfs
-rwxr-xr-x 1 root root 138304 Oct 23 17:30 usr/sbin/zfs

At boot with one of the previous kernels, system starts booting correctly, loading "dracut-premount" hook and then flooding with the following :

dracut-premount : the ZFS modules are not loaded
dracut-premount : Try running ´/sbin/modprobe/zfs´ as root to load them

This is obviously not the behaviour that we want, all the more that at this stage of boot process we don't have any root access through a console.
The trick is to boot with the last kernel (i.e here 5.9.10-*) or to uninstall all zfs modules installed and reinstall only one module for a given kernel installed on the system, and then boot on that kernel...
So if I have some issue with a given kernel and I want to boot on a previous one, I have to first remove all zfs dkms modules installed, then rebuild and install the modules for the kernel I want to boot on, and then boot on it, but of course I will be the only one usable.

Please feel free to ask for more informations if needed.
Thanks.

[gregory-lee-bartholomew]

$ sudo lsinitrd /boot/initramfs-5.9.10-200.fc33.x86_64.img | grep zfs
...
lrwxrwxrwx 1 root root 65 Oct 23 17:30 usr/lib/modules/5.8.18-300.fc33.x86_64/weak-updates/zfs.ko.xz -> ../../../../../lib/modules/5.9.10-200.fc33.x86_64/extra/zfs.ko.xz

It looks like you might be running into the same weak modules problem I hit recently. If this is the same problem, the work around I suggested here: #11128 might work.

[gogi983]

$ sudo lsinitrd /boot/initramfs-5.9.10-200.fc33.x86_64.img | grep zfs
...
lrwxrwxrwx 1 root root 65 Oct 23 17:30 usr/lib/modules/5.8.18-300.fc33.x86_64/weak-updates/zfs.ko.xz -> ../../../../../lib/modules/5.9.10-200.fc33.x86_64/extra/zfs.ko.xz

It looks like you might be running into the same weak modules problem I hit recently. If this is the same problem, the work around I suggested here: #11128 might work.

That's right, in fact we're running in the same issue, and I tried both the workarounds you suggested in your thread, but none of them helped in my case :

• add "rd.driver.pre=zfs"
• disable building of "WEAK_MODULES"

The issue persists here for me with Fedora 33 and fresh upgrade to kernel 5.9.13 x86_64, so I had to do the hard method to go back and boot on kernel 5.8.18.

EDIT : I forgot to precise that I'm now running not anymore with ZFS/0.8.5 but ZFS/2.0.0 and the issue still persists. So I'm going to update the title of this thread too.

[gregory-lee-bartholomew]

I forgot to precise that I'm now running not anymore with ZFS/0.8.5 but ZFS/2.0.0 and the issue still persists. So I'm going to update the title of this thread too.

gogi: Since you have a current system that is exhibiting this problem (we think), you might be able to help us find a fix.

Would you mind trying the following steps in order and checking the results?:

1. Verify that your non-working initramfs contains a weak module by running the following command and verifying that the kernel versions shown on the command line and in the output do not match (substitute the initramfs/kernel version as necessary).

$ sudo lsinitrd /boot/initramfs-5.9.10-200.fc33.x86_64.img | grep zfs.ko

2. Delete (or rename) the bad initramfs and remove the zfs module for only that initramfs version with the follow (again substitute the version number as necessary).

$ sudo rm /boot/initramfs-5.9.10-200.fc33.x86_64.img
$ sudo dkms remove -m zfs -v 2.0.0 -k 5.9.10-200.fc33.x86_64

3. Rebuild the zfs module and regenerate the initramfs.

$ sudo dkms install -m zfs -v 2.0.0 -k 5.9.10-200.fc33.x86_64
$ sudo dracut -f /boot/initramfs-5.9.10-200.fc33.x86_64.img 5.9.10-200.fc33.x86_64

4. If your system is still exhibiting this weak module problem, it should still show non-matching kernel versions when you rerun the command from step 1.

$ sudo lsinitrd /boot/initramfs-5.9.10-200.fc33.x86_64.img | grep zfs.ko

5. Assuming that you've verified the erroneous behavior, repeat step 2, but before running the commands from step 3, run the following command to disable weak modules.

$ sudo sh -c 'echo "NO_WEAK_MODULES=\"yes\"" >> /usr/src/zfs-2.0.0/dkms.conf'

6. After adding NO_WEAK_MODULES="yes" to /usr/src/zfs-2.0.0/dkms.conf, rerun the commands from steps 3 and 4 and report back here if adding NO_WEAK_MODULES="yes" to /usr/src/zfs-2.0.0/dkms.conf fixed the problem. If this fixes the problem, then I or someone else can submit a pull request to get that line added to zfs's dkms.conf by default.

Thanks!

[gogi983]

@gregory-lee-bartholomew

I tested everything you said (eventhough I already did the same steps yesterday before answering you and it didn't work), and actually today it is working.
But there was one step missing there in your steps : if kernel modules for more than one kernel were previously built without "NO_WEAK_MODULES=yes", then one must first delete manually the corresponding folders :

$sudo rm -rf /lib/modules/(kernel-ID)/weak-updates

for each kernel installed. Otherwise, the procedure doesn't work. By the way we don't need to quote "yes".

I even tried to do it the manner it should be done when a kernel upgrade occurs, I mean I tested the procedure by removing kernel 5.9.13, initramfs image and dkms modules for kernel 5.8.18, reinstalling dkms modules and rebuilding initramfs image for
kernel 5.8.18, and reinstalling kernel 5.9.13 through a "dnf upgrade".

Conclusion is the procedure once again worked fine. So you touched the point, congratulations. As far as I'm concerned it's a go for a pull request.

I would like to add that I tried two variants for "NO_WEAK_MODULES=yes" :

1. I followed your suggestion and put it in /usr/src/zfs-2.0.0/dkms.conf
2. I created /etc/dkms/zfs.conf and put option in it.

Both of them work fine, but in my opinion, option 2 may be more relevant as this is the path where config files should be placed. At last zfs will probably change version in future, so path /usr/src/zfs-x.x.x/ will follow these changes, but this config option has to remain, no matter what version we'll upgrade to in future.

In all cases, thank you for helping in contribution to solve this issue.

[gregory-lee-bartholomew]

if kernel modules for more than one kernel were previously built without "NO_WEAK_MODULES=yes", then one must first delete manually the corresponding folders

I wondered if that might be the case. Thanks for pointing out that detail. I guess the fix will still work since I think a new zfs version will trigger the removal and rebuild of all the zfs modules across all the currently installed kernels?

in my opinion, option 2 may be more relevant as this is the path where config files should be placed.

Yes, user-specified config files should be placed under /etc. Package-provided config files, however, are supposed to be placed under /usr. The idea is that if both exist, the /etc (user-specified) one should win/override the one provided in the package.

Again, thanks for your help!