ZoL 0.7.13 + Linux Kernel 5.0.4: GPL-incompatible module zfs.ko uses GPL-only symbol 'preempt_schedule_notrace'

[grahamc]

System information

Type	Version/Name
Distribution Name	NixOS
Distribution Version	19.03 (beta)
Linux Kernel	5.0.4
Architecture	aarch64-linux
ZFS Version	0.7.13
SPL Version	0.7.13

Describe the problem you're observing

Kernel module build fails in MODPOST:

  LD [M]  /build/source/module/zfs/zfs.o
  Building modules, stage 2.
  MODPOST 7 modules
FATAL: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'preempt_schedule_notrace'

I'm also seeing the following warnings:

In file included from /build/source/module/zfs/vdev_raidz_math_aarch64_neon.c:30:0:
/build/source/module/zfs/vdev_raidz_math_aarch64_neon_common.h:126:0: warning: "ASM_BUG" redefined
 #define ASM_BUG() ASSERT(0)

                 from /build/source/module/zfs/vdev_raidz_math_aarch64_neon.c:26:
/nix/store/avplvw4j265f6brz3g368sqrlzkyabas-linux-5.0.4-dev/lib/modules/5.0.4/source/arch/arm64/include/asm/asm-bug.h:52:0: note: this is the location of the previous definition
 #define ASM_BUG() ASM_BUG_FLAGS(0)

In file included from /build/source/module/zfs/vdev_raidz_math_aarch64_neonx2.c:29:0:
/build/source/module/zfs/vdev_raidz_math_aarch64_neon_common.h:126:0: warning: "ASM_BUG" redefined
 #define ASM_BUG() ASSERT(0)

                 from /build/source/module/zfs/vdev_raidz_math_aarch64_neonx2.c:29:
/nix/store/avplvw4j265f6brz3g368sqrlzkyabas-linux-5.0.4-dev/lib/modules/5.0.4/source/arch/arm64/include/asm/asm-bug.h:52:0: note: this is the location of the previous definition
 #define ASM_BUG() ASM_BUG_FLAGS(0)

Describe how to reproduce the problem

Attempt to compile the module. In this particular case, if you have a machine with Nix running aarch64:

nix-build https://github.com/nixos/nixpkgs/archive/23fd1394dc6bef03a2ff010278a24403a97f0647.tar.gz --system aarch64-linux -A linuxPackages_5_0.zfs

Include any warning/errors/backtraces from the system logs

[grahamc]

The following have the same problem:

• 0.7.13 / 5.0.3 nix-build https://github.com/nixos/nixpkgs/archive/bf3c19558244f038dce7993274da241e72e03da1.tar.gz --system aarch64-linux -A linuxPackages_5_0.zfs
• 0.7.13 / 5.0.2 (nix-build https://github.com/nixos/nixpkgs/archive/e6b124b3b57eb327f777349086a8a67325f6c949.tar.gz --system aarch64-linux -A linuxPackages_5_0.zfs)
• 0.7.13 / 5.0.1 (nix-build https://github.com/nixos/nixpkgs/archive/2aa471997fd485a0678acb4b5d470da2f8eebbcb.tar.gz --system aarch64-linux -A linuxPackages_5_0.zfs)

[nilsmeyer]

As far as I can tell this has always been EXPORT_SYMBOL_GPL, did you set CONFIG_PREEMPT?

[grahamc]

Hmm, yeah ... Not sure what is going on there. I do have CONFIG_PREEMPT set, attached is the full config.

config.txt

[grahamc]

The last time NixOS was able to build ZFS on aarch64 for the latest stable was 4.19.12, I'll start bisecting.

[grahamc]

I bisected 5.0 to 4.19 and torvalds/linux@0e9da3f is the first offending commit

Based on git merge-base b12a912 00203ba (the two parents of 0e9da3f), I've tried bisecting 40e020c129cfc991e8ab4736d2665351ffd1468d to 00203ba40d40d7f33857416adfb18adaf0e40123 but both ends built just fine.

My colleague is rebasing torvalds/linux@0e9da3f on to torvalds/linux@b12a912 and I'll be bisect that next.

[grahamc]

OK I finished that rebase, and it narrowed down to this commit: torvalds/linux@7fcf2b0 -- calling it a day for now, and will look more tomorrow.

[grahamc]

Attached are build logs and the results of the original attempted build (nix-build https://github.com/nixos/nixpkgs/archive/23fd1394dc6bef03a2ff010278a24403a97f0647.tar.gz --system aarch64-linux -A linuxPackages_5_0.zfs)

• nix-build-zfs-kernel-0.7.13-5.0.4.drv-0.log
• nix-build-zfs-kernel-0.7.13-5.0.4.drv-0.tar.gz.

objdump (attached) shows https://github.com/zfsonlinux/zfs/blob/a8c2b7ebc64d23ffe15e2b168aa0bea7a4897133/module/zfs/vdev_disk.c#L702-L779 is calling preempt_schedule_notrace

[behlendorf]

@grahamc thanks for isolating the offending commit. It looks like when they switched over to using percpu_ref's in blkg_tryget() they inadvertently converted it to the GPL-only symbol. This looks to be specific to the way preemption is handled on aarch64.

This code is pulled in though the CONFIG_BLK_CGROUP option. So if you can live without this functionality you could try disabling it until we find a good way to handle this.

[grahamc]

Ah, excellent! Unfortunately, I can't turn that off so that workaround doesn't apply for me. However, I'm able to use 4.19 for the the next 4-5 months.

Thank you for doing this research and getting to the workaround!

[vcunat]

So, in NixOS we give up on ZFS in latest-kernel stuff for aarch64? (example)

[grahamc]

Something like that, but let's move that discussion to our own issue tracker NixOS/nixpkgs#58959 :)