Deadlock when writing to ZFS 2.0.1 filesystem from Docker container

[Do-while-Bacon]

System information

Type	Version/Name
Distribution Name	CentOS
Distribution Version	7.9.2009 (Core)
Linux Kernel	3.10.0-1160.11.1
Architecture	x86_64
ZFS Version	2.0.1 (kmod)
SPL Version	2.0.1

Describe the problem you're observing

After updating my NAS from ZFS 0.8.5 to 2.0.1 and performing a zpool upgrade, I started encountering an issue where my docker containers (running local on the NAS) that mounted ZFS volumes would deadlock at 100% CPU as soon as they tried to perform any heavy write operations to the pool. I observed this behavior with both the linuxserver.io Sabnzbd and qBittorrent containers. The containers would appear to function normally until I tried to download a Linux ISO, then, the download would get stuck, container would lock at 100% CPU, and nothing would work to kill or stop the container until I rebooted.

I was able to work around this issue by downgrading ZFS packages to 2.0.0. Everything is working correctly again.

Describe how to reproduce the problem

Create a RAID-Z2 pool using OpenZFS 2.0.1 on CentOS 7.9 (my pool has a both an L2ARC and a SLOG device)
Install Docker CE 20.10 (problem occurs with 19.03 too)
Launch a linuxserver.io Sabnzbd container, passing a ZFS volume to /config and /downloads
Attempt to download a NZB
Download will begin and then immediately deadlock

Include any warning/errors/backtraces from the system logs

There was no relevant log output from the docker application or in syslog, however, I did strace the process while it was locked at 100% CPU and it was repeating this system call over and over:

strace: Process 20733 attached
select(0, NULL, NULL, NULL, {tv_sec=2, tv_usec=562986}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=3, tv_usec=0}) = 0 (Timeout)

[sempervictus]

@Do-while-Bacon: when you say "zfs volumes" do you mean ZVOLs or volumes as files in the ZFS filesystem?

[Do-while-Bacon]

@sempervictus Sorry for the confusion. I just meant passing through directories on a ZFS filesystem to a docker container.

[marshalleq]

This may be the same issue I, and some others have been experiencing since upgrading to ZFS 2.0.1 on unraid. We've even gone to the extent of separately building a kernel with both 2.0.0 and 2.0.1 separately and swapping them out. The issue is repeatable in that on 2.0.1 it always occurs and on 2.0.0 it always doesn't occur. This prevents us from using 2.0.1.

I can post logs if need be - the unraid thread starts here but the basics are that the docker tab stops being responsive and in one case docker is exhibiting 100% CPU behaviour.

[aerusso]

@marshalleq This may be asking a lot, but would you be willing to bisect the responsible commit?

[marshalleq]

Hey, I would be willing to do most things, but in this case I'm not actually sure what you're asking. :)

[aerusso]

Ah! This is performing a binary search in the commit history. You've established that 2.0.0 is "good" and 2.0.1 is "bad", and then you compile halfway through the commits---and test again. Then you can figure out if the culprit commit was before or after the midpoint you tested. This will take about ~6 re-compilation/re-tests (because 2**6 is roughly the number of commits between 2.0.0 and 2.0.1). Here's a tutorial (for wine, but you can just skip the wine-specific stuff---you already know how to compile zfs).

git bisect chooses the commit for you, and you just mark whether it was good or bad at each step.

[marshalleq]

Oh I see. OK, I'll see if I can do that, no problem. I may have to wait til the weekend as I'm away for work, but one of the other Unraid guys may also be able to help. Thanks.

[Joly0]

I am having the same problems with unraid, can only tell its happening when running some docker containers (like amp-dockerized or jdownloaded) and others not. Problem is not appearing in 2.0.0 and started with 2.0.1. Cpu usage is pinned to 100% making the server basically unusable and forcing one to hard-reboot it to use it again.

[Joly0]

@aerusso I tested a bit and build some custom kernels based on a few commits and i figured out, the issue started with commit 1c2358c
This specific commit fixed my problem with lancache docker container giving out errors due to wrong syscalls and at the same time made my server basically unusable when using other docker containers like amp-dockerized and jdownloader pinning my cpu to 100% and makes it unable to do anything, stop the containers, stop the storage-array or even restart.

I rly hope this helps to identify this problem and help to fix it.

[behlendorf]

PR #11484 contains the proposed fix which applies cleanly to OpenZFS 2.0.1. Any help verifying the fix would be welcome so we can get it applied for 2.0.2.

[Joly0]

Alright, going to build a custom kernel based on that commit, this might take about half an hour - 1 hour, will give a result asap