openzfs-2.0.0-rc7 zpool import hangs

[zviratko]

System information

Type	Version/Name
Distribution Name	Gentoo Linux
Distribution Version	~amd64
Linux Kernel	5.9.11-gentoo
Architecture	amd64
ZFS Version	2.0.0-rc7
SPL Version	2.0.0-rc7

Describe the problem you're observing

After updating from openzfs-2.0.0-rc6 to openzfs-2.0.0-rc7 I am observing a hang on boot.
Running debug shell in initramfs (I am running root on ZFS, initramfs is genkernel/linuxrc) I see that "zpool import" takes 1m16s (consistently) to return with correct ata. Not sure how long it took on rc6, but it surely didn't take more than a second or so to mount the pool.

EDIT: This was a red herring caused by missing zpool.cache. The actual import simply never completes.

My root pool is on a 2 drive NVMe mirror.
There is also a boot pool on the same NVMe pair.

There is also a data pool on 6 SATA drives with 2 SSD cache drives for L2ARC and SLOG.

Drives and pools are probably detected correctly, it just takes a long time.

I waited for ~15 minutes and the system didn't boot, I'll try waiting longer when I can, maybe something can be dug up from a running system.

I don't think the problem is in the userland utilities: 2.0.0-rc6 kernel modules + 2.0.0-rc7 userland work just fine.

Include any warning/errors/backtraces from the system logs

I see nothing in dmesg indicating some sort of I/O error or hang, no errors anywhere. Let me know how I can help debug this. Unfortunately the initramfs environment is limited so I can't look at stack or something like that (at least not easily).

[behlendorf]

If possible could you verify the delay isn't present in 2.0.0-rc6. I understand the initramfs environment is limited but perhaps you could still determine if it appears to be IO or cpu bound?

[zviratko]

2.0.0-rc6 works just fine, boot is swift including importing all pools (rpool in initramfs and the ata pool after that).
2.0.0-rc6 kmod + 2.0.0-rc7 userland also works fine

[zviratko]

To elaborate more - I haven't tried running plain "zpool import" with 2.0.0-rc6, so it's possible it's a red herring and the ramdisk runs something else (I think it actually does something like "mount -t zfs $noideawhat")

I don't think it's anything-bound - "time zpool import" reported 1m16s real, rest was pretty much zero. Drive LEDs indicate nothing.

[behlendorf]

I don't see anything in the release notes which would be a likely cause. The most plausible kernel side changes from rc6 to rc7 which might have caused this would be a3a4b8d or ccfa35c, could you try reverting each of those changes to see if it helps?

[zviratko]

Thank you. I will try and report back (tomorrow).

I do have some parameters set not default:
options zfs zfs_per_txg_dirty_frees_percent=0 l2arc_noprefetch=0 l2arc_write_max=33554432 l2arc_write_boost=33554432 zfs_vdev_scrub_max_active=4 zfs_vdev_scrub_min_active=0 zfs_scrub_min_time_ms=100

And this should be the command that hangs in initramfs (at least for ~15 minutes, possibly forever)
/sbin/zpool import -N -c /etc/zfs/zpoo.cache -f rpool

[zviratko]

I ran the zpool import in background so I could gather something

1. cpu is doing nothing
2. drives are doing nothing
3. zpool import is in D state and photo of stack is attached :-)
   

[zviratko]

Success!

Reverting a3a4b8d did the trick.
Pool imported instantly and the server booted just fine.

Let me know if I can help debug it further.

[behlendorf]

Thanks that definitely narrows it down. @zviratko were you able to reproduce the issue with the default module tunings? Were there any background activities (scrub resilver, initialize, etc)which might have been in progress when trying to import? Or anything else non-standard about the pool? Do both pools experience this long delay or only one? @amotin might have some suggestions for additional things to check.

[zviratko]

I tried increasing zfs_vdev_scrub_min_active(=2), zfs_vdev_initializing_min_active(=2), zfs_vdev_initializing_max_active(=3) (when are those two used?) with no effect. I also tried with and without the cache file.

There was nothing going on in the pools, and the root pool is a simple one.

The only non-standard things is that rpool doesn't have all features enabled, those are missing:

rpool
redaction_bookmarks
redacted_datasets
bookmark_written
log_spacemap
livelist
device_rebuild
zstd_compress

And the only relevant tuning is "libata.fua=1" on cmdline (not sure if that's the default nowadays).
I'm running a 5.9.11 kernel, maybe nobody tested on that yet?

I will retest with defaults and also the data pool when I can, but I'm pretty sure that patch just sets the number of requests to zero somewhere... :)

[zviratko]

Oh yeah, also bleeding edge versions everywhere, like
gcc (Gentoo 10.2.0-r3 p4) 10.2.0

[JordanP-Dev]

If it helps to narrow it down, I am also running Gentoo with kernel 5.9.11 and zfs 2.0.0-rc7 and my pool loads fine. My pool does have all features enabled, though.