Unable to compile on ARM64 with clang

[derzahla]

System information

Type	Lenovo c630 w/Snapdragon sdm850 cortex.a75-cortex.a55
Distribution Name	Arch
Distribution Version	Current w/ clang 13.0.1
Kernel Version	5.17.0
Architecture	ARM64
OpenZFS Version	2.1.4 and latest git

Describe the problem you're observing

While building under GCC works fine, I am unable to with clang+lto or -lto

Describe how to reproduce the problem

Compile linux kernel with ZFS +clang or compile ZFS as a dkms module for a kernel compiled with clang

Include any warning/errors/backtraces from the system logs

LTO [M] drivers/bluetooth/btrsi.lto.o

LTO [M] drivers/target/target_core_user.lto.o
GEN .version
CHK include/generated/compile.h
GEN .tmp_initcalls.lds
GEN .tmp_symversions.lds
LTO vmlinux.o
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2150099659
ld.lld: error: couldn't allocate output register for constraint '{v7}' at line 2150100117
ld.lld: error: couldn't allocate input reg for constraint '{v1}' at line 2150100832
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2150101251
ld.lld: error: couldn't allocate output register for constraint '{v7}' at line 2150101709
ld.lld: error: couldn't allocate input reg for constraint '{v1}' at line 2150102484
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157696494
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157700699
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157704887
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157709206
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157714137
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157718730
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157724053
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157728241
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157732948
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157738385
ld.lld: error: couldn't allocate output register for constraint '{v16}' at line 2157781218
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157782350
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157786517
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157789877
ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
make: *** [Makefile:1156: vmlinux] Error 1
==> ERROR: A failure occurred in build().
Aborting...

More detail is obtained when not using LTO, here is output without LTO and using with "make v=1" :

CC fs/proc/inode.o
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:151:2: error: couldn't allocate output register for constraint '{v0}'
NEON_INIT_LOOP();
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:75:6: note: expanded from macro 'NEON_INIT_LOOP'
asm("eor %[ZERO].16b,%[ZERO].16b,%[ZERO].16b\n"
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:154:3: error: couldn't allocate output register for constraint '{v7}'
NEON_MAIN_LOOP(NEON_DONT_REVERSE);
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:93:6: note: expanded from macro 'NEON_MAIN_LOOP'
asm("ld1 { %[SRC].4s }, %[IP]\n"
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:157:2: error: couldn't allocate input reg for constraint '{v1}'
NEON_FINI_LOOP();
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:112:6: note: expanded from macro 'NEON_FINI_LOOP'
asm("st1 { %[ACC0].4s },%[DST0]\n"
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:190:2: error: couldn't allocate output register for constraint '{v0}'
NEON_INIT_LOOP();
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:75:6: note: expanded from macro 'NEON_INIT_LOOP'
asm("eor %[ZERO].16b,%[ZERO].16b,%[ZERO].16b\n"
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:193:3: error: couldn't allocate output register for constraint '{v7}'
NEON_MAIN_LOOP(NEON_DO_REVERSE);
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:93:6: note: expanded from macro 'NEON_MAIN_LOOP'
asm("ld1 { %[SRC].4s }, %[IP]\n"
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:196:2: error: couldn't allocate input reg for constraint '{v1}'
NEON_FINI_LOOP();
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:112:6: note: expanded from macro 'NEON_FINI_LOOP'
asm("st1 { %[ACC0].4s },%[DST0]\n"
^
6 errors generated.
make[3]: *** [scripts/Makefile.build:288: fs/zfs/zcommon/zfs_fletcher_aarch64_neon.o] Error 1
make[2]: *** [scripts/Makefile.build:550: fs/zfs/zcommon] Error 2
make[1]: *** [scripts/Makefile.build:550: fs/zfs] Error 2
make[1]: *** Waiting for unfinished jobs....

[rincebrain]

The problem is that, while GCC is perfectly happy with explicit register allocations when it's been told not to do certain kinds of instructions with compile flags, clang is not, and passing -mgeneral-regs-only (as Linux does generally on AArch64, among others) makes the magic smoke come out.

CFLAGS_REMOVE_zfs_fletcher_aarch64_neon.o += -mgeneral-regs-only in the Makefile will make that stop, for instance, but I think technically it could choose to emit FPU instructions outside of the kfpu_begin/end bounded regions if you do that without further changes, even though I can't imagine a reason it would decide it was beneficial here.

[derzahla]

Thanks for the quick response. That is what was suggested on IRC. I did a 'make clean' and placed the line (direct copy and paste) "CFLAGS_REMOVE_zfs_fletcher_aarch64_neon.o += -mgeneral-regs-only" below the 4th line of the fs/zfs/Makefile and recompiled. It didnt seem to make any difference

[rincebrain]

It's true, I did suggest it there too.

That said, for builtin, it's possible it wants a path like fs/zfs/... - I don't know how the way it calculates paths interacts when you do it as builtin.

As far as as a module, it should work, removing the flag did work for me, and adding that line in the Makefile did make the flag, and that error set go away (though it promptly produced the same class of problem with the NEON raidz instructions, but the same dance over there worked too).

[derzahla]

Thanks! It compiles as a module with after taking care of the two raidz neon instructions. Any change adding those two would also allow it to compile builtin?

[derzahla]

Update: So compiling completes but the system does not boot off its ZFS root. Either when compiled as a module or built in

[derzahla]

Coming back to this, I'm not sure where to add the 3 lines anymore, seems the code structure has changed? I tried adding:

CFLAGS_REMOVE_zfs_fletcher_aarch64_neon.o += -mgeneral-regs-only
CFLAGS_REMOVE_vdev_raidz_math_aarch64_neon.o += -mgeneral-regs-only
CFLAGS_REMOVE_vdev_raidz_math_aarch64_neonx2.o += -mgeneral-regs-only

to the three Makefile.in's that exist in master:

./include/Makefile.in
./module/Makefile.in
./Makefile.in

But build errors out the same as without the fix. Can you tell me what Im doing wrong, note: I am building as a module this time, not built in. output from dkms build:

 CC [M]  /var/lib/dkms/zfs/git/build/module/os/linux/zfs/zvol_os.o
  CC [M]  /var/lib/dkms/zfs/git/build/module/zcommon/zfs_fletcher_aarch64_neon.o
  CC [M]  /var/lib/dkms/zfs/git/build/module/zfs/vdev_raidz_math_aarch64_neon.o
  CC [M]  /var/lib/dkms/zfs/git/build/module/zfs/vdev_raidz_math_aarch64_neonx2.o
  LD [M]  /var/lib/dkms/zfs/git/build/module/spl.o
/var/lib/dkms/zfs/git/build/module/os/linux/zfs/zvol_os.c:1114:3: warning: ignoring return value of function declared with 'warn_unused_result' attribute [-Wunused-result]
                add_disk(zv->zv_zso->zvo_disk);
                ^~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
1 warning generated.
/var/lib/dkms/zfs/git/build/module/zfs/vdev_raidz_math_aarch64_neonx2.c:215:24: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
#pragma GCC diagnostic pop
                       ^
1 warning generated.
  LD [M]  /var/lib/dkms/zfs/git/build/module/zfs.o
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157731689
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157735894
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157740082
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157744401
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157749332
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157753925
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157759248
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157763436
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157768143
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157773580
ld.lld: error: couldn't allocate output register for constraint '{v16}' at line 2157812348
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157813480
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157817647
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157821007
ld.lld: error: couldn't allocate input reg for constraint '{v4}' at line 2157829425
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157833708
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157837068
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157840959
ld.lld: error: couldn't allocate input reg for constraint '{v4}' at line 2157845316
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157849552
ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
make[4]: *** [scripts/Makefile.build:418: /var/lib/dkms/zfs/git/build/module/zfs.o] Error 1
make[3]: *** [Makefile:1844: /var/lib/dkms/zfs/git/build/module] Error 2
make[3]: Leaving directory '/usr/lib/modules/5.18.0-asahi-20220608-1-ARCH/build'

[rincebrain]

Haven't tested this, but you probably want lines like CFLAGS_REMOVE_foo.o and the corresponding CFLAGS_foo.o in module/Kbuild.in for Linux kernel land (though note that they actually need to be in Kbuild, in the end, so don't forget to run configure to regenerate Kbuild after editing Kbuild.in).

[ardbiesheuvel]

Just a drive by comment: if permitted, the compiler may use NEON registers for spilling or for inline memsets and memcpys, e.g., to initialize stack variables. If this occurs outside of kernel_neon_begin()/end(), the kernel code will be clobbering user space data, because (unlike on 32-bit ARM), we don't disable access to the FP/SIMD registers when entering the kernel, and while we preserve them eagerly (as the task may be scheduled out and migrate to another CPU), the restore is only done lazily, but only when/after kernel_neon_begin()/end() have been used.