[0.8.5 too] was 0.8.4 dkms won't compile unless I remove 0.8.3 folder from /var/lib/dkms/zfs/

[Germano0]

System information

Type	Version/Name
Distribution Name	CentOS
Distribution Version	8.2
Linux Kernel	4.18.0-193.6.3.el8_2.x86_64
Architecture	x86_64
ZFS Version	0.8.4-1
SPL Version	0.8.4-1

Describe the problem you're observing

I updated to CentOS 8.2 and ZFS 0.8.4, so after the reboot I had to reload ZFS module (due #10462) so I ran

# modprobe zfs
modprobe: FATAL: Module zfs not found in directory /lib/modules/4.18.0-193.6.3.el8_2.x86_64

to fix this error I tried

[root@machine user]# dkms autoinstall
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/0.8.3/source/dkms.conf does not exist.

Since it is a kind of error I have seen on other packages like Wireguard, I knew that if after an package update, dkms autoinstall failed, I have to remove the older folder in /var/lib/dkms, so I did it:

[root@machine user]# ls /var/lib/dkms/zfs/
0.8.3  0.8.4  kernel-4.18.0-147.5.1.el8_1.x86_64-x86_64  kernel-4.18.0-147.8.1.el8_1.x86_64-x86_64
[root@machine user]# rm -rf /var/lib/dkms/zfs/0.8.3/
[root@machine user]# dkms autoinstall

and then the machine started to compile the new module

[Hermanio]

Ran into the exact same issue, removing /var/lib/dkms/zfs/0.8.3/ did the trick.

[AllKind]

I remember having that problem with other dkms modules.
I needed to do the steps manually.
first: dkms build ..., then: dkms install ...
might be a dkms problem?

[Germano0]

it's happening also with 0.8.5 not compiling unless I remove 0.8.4 folder

# dkms autoinstall
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/0.8.4/source/dkms.conf does not exist.

# rm -rf /var/lib/dkms/zfs/0.8.4
# dkms autoinstall
everything compiled successfully

[jeremyvisser]

This still happens without fail every single ZFS and kernel upgrade on Fedora 33.

Yesterday I did a regular upgrade, which broke ZFS:

• From:
  • kernel-core-5.10.19-200.fc33.x86_64
  • zfs-dkms-2.0.3-1.fc33.x86_64
• To:
  • kernel-core-5.10.21-200.fc33.x86_64
  • zfs-dkms-2.0.4-1.fc33.x86_64

On the following boot, dkms.service output the following logs:

Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.0.3/source/dkms.conf does not exist.

Looking at /var/lib/dkms/zfs there's a whole lot of stale broken symlinks left behind, as well as the old source dir for 2.0.3:

% ls -l /var/lib/dkms/zfs
total 112
drwxr-xr-x. 1 root root 150 Mar  3 08:53 2.0.3
drwxr-xr-x. 1 root root  58 Mar 14 00:13 2.0.4
lrwxrwxrwx. 1 root root  36 Feb 19 18:07 kernel-5.10.10-200.fc33.x86_64-x86_64 -> 2.0.3/5.10.10-200.fc33.x86_64/x86_64
lrwxrwxrwx. 1 root root  36 Feb 19 20:41 kernel-5.10.15-200.fc33.x86_64-x86_64 -> 2.0.3/5.10.15-200.fc33.x86_64/x86_64
lrwxrwxrwx. 1 root root  36 Mar 14 00:13 kernel-5.10.19-200.fc33.x86_64-x86_64 -> 2.0.4/5.10.19-200.fc33.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Jan 23 07:56 kernel-5.10.8-200.fc33.x86_64-x86_64 -> 2.0.1/5.10.8-200.fc33.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Jan 26 10:04 kernel-5.10.9-201.fc33.x86_64-x86_64 -> 2.0.1/5.10.9-201.fc33.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Sep 19  2019 kernel-5.2.14-200.fc30.x86_64-x86_64 -> 0.8.1/5.2.14-200.fc30.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Sep 20  2019 kernel-5.2.15-200.fc30.x86_64-x86_64 -> 0.8.1/5.2.15-200.fc30.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Oct  3  2019 kernel-5.2.16-200.fc30.x86_64-x86_64 -> 0.8.2/5.2.16-200.fc30.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Oct  3  2019 kernel-5.2.17-200.fc30.x86_64-x86_64 -> 0.8.2/5.2.17-200.fc30.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Oct  9  2019 kernel-5.2.18-200.fc30.x86_64-x86_64 -> 0.8.2/5.2.18-200.fc30.x86_64/x86_64
lrwxrwxrwx. 1 root root  34 Aug 18  2019 kernel-5.2.8-200.fc30.x86_64-x86_64 -> 0.8.1/5.2.8-200.fc30.x86_64/x86_64
lrwxrwxrwx. 1 root root  34 Sep  4  2019 kernel-5.2.9-200.fc30.x86_64-x86_64 -> 0.8.1/5.2.9-200.fc30.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Dec  4  2019 kernel-5.3.13-300.fc31.x86_64-x86_64 -> 0.8.2/5.3.13-300.fc31.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Dec 12  2019 kernel-5.3.14-300.fc31.x86_64-x86_64 -> 0.8.2/5.3.14-300.fc31.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Dec 13  2019 kernel-5.3.15-300.fc31.x86_64-x86_64 -> 0.8.2/5.3.15-300.fc31.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Dec 22  2019 kernel-5.3.16-300.fc31.x86_64-x86_64 -> 0.8.2/5.3.16-300.fc31.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Jan 25  2020 kernel-5.4.10-200.fc31.x86_64-x86_64 -> 0.8.3/5.4.10-200.fc31.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Feb  2  2020 kernel-5.4.14-200.fc31.x86_64-x86_64 -> 0.8.3/5.4.14-200.fc31.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Feb  2  2020 kernel-5.4.15-200.fc31.x86_64-x86_64 -> 0.8.3/5.4.15-200.fc31.x86_64/x86_64
lrwxrwxrwx. 1 root root  34 Jan 10  2020 kernel-5.4.7-200.fc31.x86_64-x86_64 -> 0.8.2/5.4.7-200.fc31.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 May 13  2020 kernel-5.5.18-200.fc31.x86_64-x86_64 -> 0.8.4/5.5.18-200.fc31.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Sep 24 09:24 kernel-5.8.10-200.fc32.x86_64-x86_64 -> 0.8.4/5.8.10-200.fc32.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Sep 28 16:53 kernel-5.8.11-200.fc32.x86_64-x86_64 -> 0.8.4/5.8.11-200.fc32.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Oct  8 00:04 kernel-5.8.12-200.fc32.x86_64-x86_64 -> 0.8.5/5.8.12-200.fc32.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Nov  5 00:17 kernel-5.8.17-300.fc33.x86_64-x86_64 -> 0.8.5/5.8.17-300.fc33.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Nov  8 11:04 kernel-5.8.18-300.fc33.x86_64-x86_64 -> 0.8.5/5.8.18-300.fc33.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Dec 16 09:43 kernel-5.9.13-200.fc33.x86_64-x86_64 -> 2.0.0/5.9.13-200.fc33.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Dec 25 00:40 kernel-5.9.15-200.fc33.x86_64-x86_64 -> 2.0.0/5.9.15-200.fc33.x86_64/x86_64

The problem is the existence of the 2.0.3 directory (/var/lib/dkms/zfs/2.0.3), which was evidently not correctly cleaned up.

I can manually work around the problem by running:

$ rm -r /var/lib/dkms/zfs/2.0.3

But obviously it's not a reasonable expectation for users to be doing this.

[SpComb]

See #9891 -> 4d6043f

Is this fixed in zfs 2.0.x packages? The fix is at package uninstall time, so it only takes effect for upgrades from 2.0.x.

[jeremyvisser]

This happened to me again last night on Fedora 35.

The sequence of events that led up to it were:

• Starting state (as at 2021-12-03):
  • zfs-dkms-2.1.1-1.fc35
  • kernel-core-5.15.6-200.fc35
• ZFS upgrade (on 2021-12-16, which worked fine):
  • Upgrade: zfs-dkms 2.1.1-1.fc35 → 2.1.2-1.fc35
• Kernel upgrade (on 2021-12-23, this is where things broke):
  • Install: kernel-core-5.15.10-200.fc35
  • dkms.service fails with:

    Error! Could not locate dkms.conf file.
    File: /var/lib/dkms/zfs/2.1.1/source/dkms.conf does not exist.
    

  • Noticed that /var/lib/dkms/zfs/2.1.1 still exists (probably left behind by prior upgrade)

Even though the problem only triggered on the kernel upgrade, I think the root cause happened during the zfs-dkms upgrade, where it failed to clean up the old /var/lib/dkms/zfs/$old_version directory.

The issue is still reliably reproducible, so it definitely has not been fixed.

[denizzzka]

Still happens with /var/lib/dkms/zfs/2.0.1/ during moving to a new kernel and zfs 2.1.2

[jeremyvisser]

Having a bit of a poke around, it seems the zfs-dkms RPM is all kinds of broken:

• Determining the version doesn’t make any sense:

  zfs/rpm/generic/zfs-dkms.spec.in

  Lines 89 to 90 in b3ab290

  	NEWEST_VER=$(dkms status zfs | tr -d , | sort -r -V | awk '/installed/{print $2; exit}')
  	if [ "$NEWEST_VER" != "%{version}" ] ; then

  If you look at the output, you’ll see:

  % sudo dkms status zfs 2>&-
  zfs/2.1.2, 5.14.10-300.fc35.x86_64, x86_64: installed
  % sudo dkms status zfs 2>&- | sed 's/,//g' | sort -r -V | awk '/installed/{print $2; exit}'
  5.14.10-300.fc35.x86_64
  

  This means that the above line 90 will be:

  if [ "5.14.10-300.fc35.x86_64" != "2.1.2" ] ; then
  

  Which makes no sense. This may have been broken by dell/dkms@f83b758.

• This zfs_config.h does not exist:

  zfs/rpm/generic/zfs-dkms.spec.in

  Lines 104 to 107 in b3ab290

  	CONFIG_H="/var/lib/dkms/%{module}/%{version}/*/*/%{module}_config.h"
  	SPEC_META_ALIAS="@PACKAGE@-@VERSION@-@RELEASE@"
  	DKMS_META_ALIAS=`cat $CONFIG_H 2>/dev/null |
  	awk -F'"' '/META_ALIAS\s+"/ { print $2; exit 0 }'`

  Or as shipped in zfs-dkms-2.1.2:

  CONFIG_H="/var/lib/dkms/zfs/2.1.2/*/*/zfs_config.h"
  SPEC_META_ALIAS="zfs-2.1.2-1"
  DKMS_META_ALIAS=`cat $CONFIG_H 2>/dev/null |
      awk -F'"' '/META_ALIAS\s+"/ { print $2; exit 0 }'`
  

  This file does not exist, but a similar one does exist although it doesn’t contain what you’re looking for:

  % ls /var/lib/dkms/zfs/2.1.2/*/*/zfs_config.h
  no matches found: /var/lib/dkms/zfs/2.1.2/*/*/zfs_config.h
  % find -L /var/lib/dkms/zfs -name '*zfs_config.h*'
  /var/lib/dkms/zfs/2.1.2/source/zfs_config.h.in
  % rpm -ql zfs-dkms | grep 'zfs_config.h'          
  /usr/src/zfs-2.1.2/zfs_config.h.in
  

  Just to be explicit, $DKMS_META_ALIAS is empty:

  % DKMS_META_ALIAS=`cat $CONFIG_H 2>/dev/null |
      awk -F'"' '/META_ALIAS\s+"/ { print $2; exit 0 }'`
  % echo $DKMS_META_ALIAS 
  

  I don’t know why zfs_config.h isn’t being created properly.

• As a result, this will never work:

  zfs/rpm/generic/zfs-dkms.spec.in

  Lines 108 to 112 in b3ab290

  	if [ "$SPEC_META_ALIAS" = "$DKMS_META_ALIAS" ]; then
  	echo -e
  	echo -e "Uninstall of %{module} module ($SPEC_META_ALIAS) beginning:"
  	dkms remove -m %{module} -v %{version} --all %{!?not_rpm:--rpm_safe_upgrade}
  	fi

  Which would effectively be doing:

  SPEC_META_ALIAS="zfs-2.1.2-1"
  DKMS_META_ALIAS=""  # effectively empty string as per above
  if [ "zfs-2.1.2-1" = "" ]; then
      dkms remove -m zfs -v 2.1.2 --all --rpm_safe_upgrade    # will never execute
  fi
  

  This at least proves that it’s impossible for it to work correctly, and backs up the above claims that things are not being cleaned up properly.

So yeah, very broken.

[jeremyvisser]

After looking into the history of this issue, which has surfaced multiple times (#8160, dell/dkms#25, #6902, etc.), I am convinced the complexity of the existing scripts are not justified with the simplicity of the problem. I think a more reliable solution is actually quite minimal:

%preun
dkms remove -m %{module} -v %{version} --all

%posttrans
/usr/lib/dkms/common.postinst %{module} %{version}

Notes:

• The above snippet replaces the existing %post and %preun sections.
• See ordering information at https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/. As you can see, %post is before %preun, but %posttrans happens last, which is more desirable here.
• This avoids parsing anything (less prone to future breakage)
• Even though existing logic tries hard to avoid uninstalling on upgrade (e.g. fc28→fc29) this is honestly only an inefficiency and the cost of risking breaking things is not worth the efficiency gains. As such, unconditionally reinstall.
• The /usr/lib/dkms/common.postinst always reinstalls anyway (https://github.com/dell/dkms/blob/1194db850c4d2391daf69cfb1a30ed17517a2bf9/dkms_common.postinst#L133-L137) so we might as well make this reliable