zed doesn't reap script zombies

[nabijaczleweli]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	sid as of 2021-03-19T16:07:26+01:00
Linux Kernel	5.10.0-4-amd64
Architecture	amd64
ZFS Version	2.0.3-1

Describe the problem you're observing

Consider this htop snippet

    PID USER      PRI  NI   RES   SHR S CPU% MEM%   TIME+  Command
      1 root       20   0  9676  6040 S  0.0  0.2  0:10.32 systemd --system --deserialize 21
   1941 root       20   0  3452  2704 S  0.0  0.1  0:02.19 └─ zed -F
   2138 root       20   0  3452  2704 S  0.0  0.1  0:00.06    ├─ zed -F
   2139 root       20   0  3452  2704 S  0.0  0.1  0:00.00    ├─ zed -F
  10602 root       20   0     0     0 Z  0.0  0.0  0:00.40    └─ history_event-z

And this log:

nab@d0:~/src$ journalctl -u zfs-zed -b
-- Journal begins at Thu 2020-10-01 15:24:48 CEST, ends at Fri 2021-03-19 16:08:42 CET. --
Mar 12 15:14:12 d0 systemd[1]: Started ZFS Event Daemon (zed).
Mar 12 15:14:13 d0 zed[1941]: ZFS Event Daemon 2.0.3-1 (PID 1941)
Mar 12 15:14:13 d0 zed[1941]: Processing events since eid=0
Mar 12 15:14:36 d0 zed[1941]: Missed 1 events
Mar 12 15:14:48 d0 zed[1941]: Missed 1 events
Mar 12 15:14:49 d0 zed[1941]: Missed 1 events
Mar 12 15:15:02 d0 zed[1941]: Killing hung "history_event-zfs-list-cacher.sh" pid=10602
Mar 12 15:15:02 d0 zed[1941]: Missed 1 events
Mar 12 15:15:09 d0 zed[1941]: Missed 1 events

Describe how to reproduce the problem

Enable history_event-zfs-list-cacher.sh I guess? Dunno.

[aerusso]

How long does it take for this command to run?

zfs list -H -o name,mountpoint,canmount,atime,relatime,devices,exec,readonly,setuid,nbmand,encroot,keylocation,org.openzfs.systemd:requires,org.openzfs.systemd:requires-mounts-for,org.openzfs.systemd:before,org.openzfs.systemd:after,org.openzfs.systemd:wanted-by,org.openzfs.systemd:required-by,org.openzfs.systemd:nofail,org.openzfs.systemd:ignore

(on the pool in question)

This command is being run every time there's a set, create, "finish receiving", import, destroy, or rename event. Ideally, we wouldn't have to drag up the metadata for every dataset in the pool every time that happens, but we never got around to addressing that.

I suspect that command takes a long time. Is this problem new since 0.8.6, or are you upgrading from a much earlier version?

[nabijaczleweli]

That was also my cursory diagnosis; the command you posted takes between 4 and 8 seconds with real=sys (user negligibly close to 0) on a good day (compare #11491) – it doesn't appear to happen every time or straight from cold boot, but if there's a lot of activity (or the clock jumps maybe? which, this being a VM on my work laptop, happens relatively often) it definitely does.

This pool started off at 0.8.6 and it also happened then, I've kept upgrading roughly as new versions hit Debian, and I've always ended up with a zombie script after a few days.