unknown pseudo-op

[Dmole]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	18.04.6
Kernel Version	5.10.183
Architecture	aarch64
OpenZFS Version	master 70ea484

Describe the problem you're observing

build after copy-builtin from master on aarch64

Describe how to reproduce the problem

build after copy-builtin from master on aarch64

Include any warning/errors/backtraces from the system logs

fs/zfs/icp/asm-aarch64/blake3/b3_aarch64_sse2.S:54: Error: unknown pseudo-op: `.cfi_negate_ra_state'      

fs/zfs/icp/asm-aarch64/blake3/b3_aarch64_sse2.S:558: Error: unknown pseudo-op: `.cfi_negate_ra_state'     

fs/zfs/icp/asm-aarch64/blake3/b3_aarch64_sse2.S:611: Error: unknown pseudo-op: `.cfi_negate_ra_state'     

make[2]: *** [fs/zfs/icp/asm-aarch64/blake3/b3_aarch64_sse2.o] Error 1                                    
make[1]: *** [fs/zfs] Error 2
make: *** [fs] Error 2                               
make: *** Waiting for unfinished jobs....            

[FormulaAgile]

we have the same problem here

[ziggythehamster]

I can replicate this on EL8 with ZFS 2.2.2, but not with ZFS 2.1.14 and not on EL9. (and obviously only on aarch64 since this is an aarch64 specific optimization)

EL8 ships:

• GCC 8
• glibc 2.28
• binutils 2.30

EL9 ships:

• GCC 11
• glibc 2.34
• binutils 2.35

It is most likely one of those variables causing the issue. A newer toolchain can be installed on EL8, but that feels brittle to me.

[ziggythehamster]

This was introduced by #14728 and thus all releases of ZFS 2.2 will have this issue.

[Dmole]

I encountered the issue with

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	18.04.6
GCC Version	7.4

But not with

Type	Version/Name
Distribution Name	Debian
Distribution Version	12
Kernel Version	5.10.205
Architecture	aarch64
OpenZFS Version	master cae502c
GCC Version	12.2

So maybe documentation like

https://openzfs.github.io/openzfs-docs/Developer%20Resources/Building%20ZFS.html

should just be updated to list minimum build requirement versions.
Because anywhere this issue is going to happen is going to be EOL soon anyway.

[ziggythehamster]

Because anywhere this issue is going to happen is going to be EOL soon anyway.

RHEL8 becomes EOL in 2029, so this either needs to be made conditional (I think this is just an assertion?) or officially the newer toolchain has to be made the minimum requirement.

I'm currently starting with GCC 9 to see what the earliest gcc-toolset-* version is that works.

[Dmole]

Right; I forgot it was only CentOS 8 that EOL-ed in 2021.

Maybe @mcmilk and @behlendorf can comment.

[ziggythehamster]

GCC 9 doesn't work, but GCC 10 does work. To avoid having exclusively one version of the package using GCC10 that could break in a weird way, I am unconditionally using GCC10 on EL8 in my build script and both x86_64 and aarch64 work fine on both 2.1 and 2.2 (whereas 2.2 on aarch64 GCC8 doesn't work)

The only issue I see with this is that I will need to patch the zfs-dkms spec to depend on the GCC 10 toolchain.

If you want to try this experiment at home:

dnf install scl-utils gcc-toolset-10
# extract and whatever
scl run gcc-toolset-10 -- ./configure # or whatever args you want
scl run gcc-toolset-10 -- make rpm-utils rpm-dkms # or whatever targets you want

[ziggythehamster]

I ended up having to use gcc-toolset-11 because gcc-toolset-10 uses the wrong (system) version of annobin if you have both the system and toolset version of annobin installed. gcc-toolset-11 doesn't have this problem. (https://bugzilla.redhat.com/show_bug.cgi?id=1960299)

Also, patching things so that it uses the right toolchain was a nightmare. KERNEL_CC will still run as from $PATH even if you were to set CCAS and AS, so I ended up having to also set PATH and LD_LIBRARY_PATH.

[mcmilk]

dnf install scl-utils gcc-toolset-12
scl run gcc-toolset-12 -- ./configure
scl run gcc-toolset-12 -- make rpm-utils rpm-dkms

Yes, it will generate the correct packages, but installation will fail, because users of those packages will likely not have gcc-toolset-12 installed.... and even when, we need to tell the packages, that they rely on gcc-12 :/

@behlendorf - I really think, that we should extend the linux spl layer with some kfpu_begin/kfpu_end wrappers - so we can just use them. The spl module has the correct license and other functions like __alloc_percpu or free_percpu are used there already. What do you think, can I create a PR for this?

For the .cfi_negate_ra_state - I will try to find a fix for that ...

[ziggythehamster]

Yeah, in order for the built RPMs to install properly (and further for DKMS packages to work), I have to modify the spec files to depend on the right toolset version and also append CC/CCAS/LD/LD_LIBRARY_PATH/NM/PATH (where the path variables are calculated by /opt/rh/*/enable) to configure and make.

This isn’t ideal. :)

I think the cfi_negate_ra_state issue could be resolved by regenerating the .S with an older toolset, and having the knowledge of what the proper machine code is supposed to be. I tried this and got a completely different result. I tried with gcc11 and clang13 and neither of those gave me remotely similar machine code to what’s committed either.

[mcmilk]

The source for generating the assembler files is located here:
https://github.com/mcmilk/BLAKE3-tests/blob/master/contrib/simde.sh

[ziggythehamster]

Yeah, that’s what I tried. I get the expected differences (the function name is different, there’s not a compiler annotation), but also the assembly is very different. I don’t know arm assembly (and for that matter, x86 SIMD is somewhat dark magic to me even though I vaguely remember x86-16 assembly), so I can’t ascertain whether the output is correct or not.

[mcmilk]

I have created two new files, they are located here: b3_aarch64_sse2.S and b3_aarch64_sse41.S

They work on my local almalinux-8 with gcc8 ...