zfs-dkms-2.1.13 missing dkms.conf file

[stuartthebruce]

System information

Type	Version/Name
Distribution Name	Rocky Linux
Distribution Version	8.8
Kernel Version	4.18.0-477.27.1
Architecture	x86_64
OpenZFS Version	2.1.13

Describe the problem you're observing

After updating from zfs-dkms-2.1.12 to zfs-dkms-2.1.13 now seeing,

[root@repo1 ~]# dkms status
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.12/source/dkms.conf does not exist.

However, ZFS appears to be working just fine.

Describe how to reproduce the problem

Update zfs-dkms rpm package to 2.1.13.

Include any warning/errors/backtraces from the system logs

[root@repo1 ~]# cat /etc/redhat-release 
Rocky Linux release 8.8 (Green Obsidian)

[root@repo1 ~]# uname -a
Linux repo1 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Wed Sep 20 15:55:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[root@repo1 ~]# rpm -q zfs-dkms
zfs-dkms-2.1.13-1.el8.noarch

[root@repo1 ~]# cat /sys/module/zfs/version 
2.1.13-1

[root@repo1 ~]# dkms status
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.12/source/dkms.conf does not exist.

For comparison and identical system (HW and SW) still running 2.1.12 reports,

[root@repo2 ~]# cat /etc/redhat-release 
Rocky Linux release 8.8 (Green Obsidian)

[root@repo2 ~]# uname -a
Linux repo2 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Wed Sep 20 15:55:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[root@repo2 ~]# rpm -q zfs-dkms
zfs-dkms-2.1.12-1.el8.noarch

[root@repo2 ~]# cat /sys/module/zfs/version 
2.1.12-1

[root@repo2 ~]# dkms status
zfs/2.1.12, 4.18.0-477.15.1.el8_8.x86_64, x86_64: installed
zfs/2.1.12, 4.18.0-477.21.1.el8_8.x86_64, x86_64: installed
zfs/2.1.12, 4.18.0-477.27.1.el8_8.x86_64, x86_64: installed

[danie-dejager]

This is happening to me too. But my ZFS is not working fine. The mount points goes away until I can fix the dkms modules. Fixing it I remove all dkms versions and then build and install 2.1.13. That usually works for me but I've had scenarios where I can't get the host up within 1 hour. How can I manually fix this if the config is not found in an efficient manner?

[AllKind]

This is likely due to the issue, that #13182 tried to resolve.
Read the "Motivation and Context" section.

As a workaround maybe this helps:
#15253 (comment)

If the dkms modules are not getting build:
#15415

Edit: this only applies to .deb packages... overlooked the .rpm thing. Might be a similar workaround for that.
Maybe just this 'dangling symlink' needs to be removed.
Sorry, I should take a rest...

[eoli3n]

I get this with 2.1.11 on latest arch linux.

root@work ~# mkinitcpio -P
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
==> Using configuration file: '/etc/mkinitcpio.conf'
  -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
==> Starting build: '6.1.58-1-lts'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'xhci_pci'
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [zfs]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-lts.img'
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
==> Using default configuration file: '/etc/mkinitcpio.conf'
  -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux.img --microcode /boot/intel-ucode.img
==> Starting build: '6.5.7-arch1-1'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'xhci_pci'
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [zfs]
==> ERROR: module not found: 'zavl'
==> ERROR: module not found: 'znvpair'
==> ERROR: module not found: 'zunicode'
==> ERROR: module not found: 'zcommon'
==> ERROR: module not found: 'zfs'
==> ERROR: module not found: 'spl'
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux.img'
==> WARNING: errors were encountered during the build. The image may not be complete.
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
==> Using default configuration file: '/etc/mkinitcpio.conf'
  -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux-fallback.img -S autodetect --microcode /boot/intel-ucode.img
==> Starting build: '6.5.7-arch1-1'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'qed'
==> WARNING: Possibly missing firmware for module: 'aic94xx'
==> WARNING: Possibly missing firmware for module: 'bfa'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
==> WARNING: Possibly missing firmware for module: 'wd719x'
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'xhci_pci'
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [zfs]
==> ERROR: module not found: 'zavl'
==> ERROR: module not found: 'znvpair'
==> ERROR: module not found: 'zunicode'
==> ERROR: module not found: 'zcommon'
==> ERROR: module not found: 'zfs'
==> ERROR: module not found: 'spl'
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
==> WARNING: errors were encountered during the build. The image may not be complete.


root@work ~# dkms status
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.11/source/dkms.conf does not exist.

edit:

It seems unrelated because 2.2.0 is installed.

root@work ~# pacman -Qs zfs
local/zfs-dkms 2.2.0-1 (archzfs-dkms)
    Kernel modules for the Zettabyte File System.

[AllKind]

I was able to reproduce the error (on Fedora 38):

root@work ~# dkms status
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.11/source/dkms.conf does not exist.

The dangling sysmlink is /var/lib/dkms/zfs/<VERSION>/source which points to /usr/src/zfs-<VERSION>.
Deleting the symlink, or the entire /var/lib/dkms/zfs/<VERSION> version directory resolves the broken dkms status.

[eoli3n]

root@work ~# rm -Rf /var/lib/dkms/zfs/2.1.11/
root@work ~# dkms status
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.12/source/dkms.conf does not exist.

As I can't build latest zfs, maybe I'll stop here, until latest zfs is fixed :)

[AllKind]

@eoli3n

File: /var/lib/dkms/zfs/2.1.12/source/dkms.conf does not exist.

You have the same thing with version 2.1.12!
Just run rm -rf /var/lib/dkms/zfs/2.1.12

[eoli3n]

root@work ~# rm -rf /var/lib/dkms/zfs/2.1.12
root@work ~# dkms status
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.13/source/dkms.conf does not exist.
root@work ~# rm -rf /var/lib/dkms/zfs/2.1.13
root@work ~# dkms status
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.9/source/dkms.conf does not exist.
root@work ~# rm -rf /var/lib/dkms/zfs/2.1.9
root@work ~# dkms status
zfs/2.2.0, 6.1.55-1-lts, x86_64: installed (original_module exists)
zfs/2.2.0, 6.1.58-1-lts, x86_64: installed

Long trip :) Thanks

[stuartthebruce]

I can confirm that manually removing the dangling symbolic link /var/lib/dkms/zfs/2.1.12/source -> /usr/src/zfs-2.1.12 makes the original error go away,

[root@repo1 ~]# rpm -qa | grep zfs
zfs-dkms-2.1.13-1.el8.noarch
libzfs5-2.1.13-1.el8.x86_64
zfs-2.1.13-1.el8.x86_64

[root@repo1 ~]# dkms status
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.1.12/source/dkms.conf does not exist.

[root@repo1 ~]# ls -l /var/lib/dkms/zfs/2.1.*
/var/lib/dkms/zfs/2.1.12:
total 0
drwxr-xr-x 3 root root 20 Jul  4 10:32 4.18.0-477.13.1.el8_8.x86_64
lrwxrwxrwx 1 root root 19 Jul  4 10:30 source -> /usr/src/zfs-2.1.12

/var/lib/dkms/zfs/2.1.13:
total 0
drwxr-xr-x 3 root root 20 Sep 30 09:26 4.18.0-477.27.1.el8_8.x86_64
lrwxrwxrwx 1 root root 19 Sep 30 09:24 source -> /usr/src/zfs-2.1.13

[root@repo1 ~]# ls -l /usr/src/zfs-2.1.12
ls: cannot access '/usr/src/zfs-2.1.12': No such file or directory

[root@repo1 ~]# rm /var/lib/dkms/zfs/2.1.12/source 
rm: remove symbolic link '/var/lib/dkms/zfs/2.1.12/source'? y

[root@repo1 ~]# dkms status
zfs/2.1.13, 4.18.0-477.27.1.el8_8.x86_64, x86_64: installed

[evelikov]

I can confirm that manually removing the dangling symbolic link /var/lib/dkms/zfs/2.1.12/source -> /usr/src/zfs-2.1.12 makes the original error go away,

Did you manage to figure out what broke that symlink in the first place?

[stuartthebruce]

I can confirm that manually removing the dangling symbolic link /var/lib/dkms/zfs/2.1.12/source -> /usr/src/zfs-2.1.12 makes the original error go away,

Did you manage to figure out what broke that symlink in the first place?

No. But, I have now also ended up with the same broken symlink on some systems after upgrading from 2.1.12 to 2.2.0 rather than 2.1.13.

[AllKind]

@stuartthebruce
#15415 (comment)

[AllKind]

@behlendorf
As #15415 has been merged, this issue should be resolved (as long as it goes into the next 2.1 and 2.2 releases).