Strange behaviour of canmount=off

[SoongNoonien]

System information

Type	Version/Name
Distribution Name	Gentoo
Distribution Version	Gentoo Base System release 2.6
Linux Kernel	5.1.5-gentoo
Architecture	amd64
ZFS Version	0.8.0-r0-gentoo
SPL Version	0.8.0-r0-gentoo

Describe the problem you're observing

I'm using the canmount=off property to make children of different pools appear in the same directory as described in the man page. Unfortunately, this seems to be kind of broken since 0.8.0. I've isolated the problem to this list of commands:

/tmp # truncate -s 1G test0.img
/tmp # truncate -s 1G test1.img
/tmp # zpool create test0 /tmp/test0.img
/tmp # zpool create test1 /tmp/test1.img
/tmp # zfs create test0/a
/tmp # zfs create test1/b
/tmp # zfs set mountpoint=none test0
/tmp # zfs set mountpoint=/a test0/a
/tmp # zfs set canmount=off test1
/tmp # zfs set mountpoint=/a test1
/tmp # zfs list -o name,mounted,canmount,mountpoint
NAME     MOUNTED  CANMOUNT  MOUNTPOINT
test0         no        on  none
test0/a      yes        on  /a
test1         no       off  /a
test1/b      yes        on  /a/b
/tmp # zfs unmount -a
/tmp # zfs mount -a
/tmp # zfs unmount -a
/tmp # zfs mount -a
cannot mount '/a': directory is not empty
/tmp #

Apparently zfs mount -a isn't able to resolve the mountpoints anymore. I found two different workarounds for this issue.
One can simply avoid using canmount to inherit mountpoints by setting the mountpoints manually. For this example zfs set mountpoint=none test1 and zfs set mountpoint=/a/b test1/b.
The stranger solution is swapping the names of the pools. Or simply choose any new name for test0 to make it appear after test1 in zfs list. (e.g. test2) So the name order of the pools seems to have an impact on the disentanglement of mountpoints.
I was able to reproduce the issue on Gentoo and Arch but not on Openindiana.

[johnnyjacq16]

I am not sure but it seems that systemd.mount could solve this issue, by setting the zfs mountpoint of the dataset to legacy, placing it into the fstab file, then tell systemd which mountpoint you want to be mounted first and after creating a dependency between the mountpoints.
example below in the fstab file:
<pool/dataset> /var/log zfs x-systemd.after=var.mount
The following would tell systemd to mount /var/log mountpoint after /var mount has been mounted.
Also a service could be used
<pool/data> /home zfs x-systemd.before=zfs-mount.service
this will mount /home before the zfs-mount.service starts.
The status of mountpoints can be check
systemctl status -- -.mount
shows the status of the root mountpoint.
For more information see https://jlk.fjfi.cvut.cz/arch/manpages/man/systemd.mount.5
read the section default dependencies and fstab which explains x-systemd.after= ,x-systemd.before=.

[aerusso]

Take a look at zfs-mount-generator, it produces the systemd.mount units (instead of requiring you to manually put things in your /etc/fstab). And systemd should be able to resolve these "tangled" mounts in a sane way. It should be able to get these things to mount correctly at boot up.

@SoongNoonien Can you see if that resolves your use case?

[SoongNoonien]

Thanks for your efforts but I do not use systemd.
@aerusso Therefore I can't test if it resolves the issue in my exact use case.
As I said in my first comment I already found two workarounds. I implemented the first one to able to boot properly again. But the actual problem is the resolution of zfs mount -a . A manual disentanglement with zfs mount <pool/dataset> works just fine, so I'am sure both of your suggestions will also work.
Perhaps the parallel mounting could be the problem #8092. It seems to be new to 0.8.0 and I haven't found it in 0.7.13. But I'm really not sure about that, because this seems to be active in Illumos as well and I wasn't able to reproduce the issue on Openindiana.
What do you think?

[SoongNoonien]

I took a closer look on this. Now I'm pretty sure, it is the parallel mounting. I looked in the commit history and found the last commit before "OpenZFS 8115 - parallel zfs mount" (a10d50f) which simply is "Tag 0.8.0-rc2" (af2e841).
So I've started my Arch Linux virtual machine and cloned the current master branch of ZFS. I rolled back to "Tag 0.8.0-rc2" (git reset --hard af2e8411dacbc694b1aaf9074e68a9d12270e74c) and built and installed it. Afterwards I wasn't able to reproduce the issue anymore.
Then I rolled back the whole VM and started again. But this time I rolled back the git clone to "OpenZFS 8115 - parallel zfs mount" (git reset --hard a10d50f999511d304f910852c7825c70c9c9e303). Afterwards I was able to reproduce the issue again. I think this shows that "OpenZFS 8115 - parallel zfs mount" (a10d50f) is responsible for this issue.
Because of the SIMD problems of the new kernels I had to install an old kernel to be able to compile these old versions of ZFS.

Version overview:

System on which the issue is not reproduceable:

Type	Version/Name
Distribution Name	Arch Linux
Linux Kernel	4.20.1-arch1-1-ARCH
Architecture	amd64
ZFS Version	0.8.0-rc2
SPL Version	0.8.0-rc2

System on which the issue is reproduceable:

Type	Version/Name
Distribution Name	Arch Linux
Linux Kernel	4.20.1-arch1-1-ARCH
Architecture	amd64
ZFS Version	0.8.0-rc2_1_ga10d50f99
SPL Version	0.8.0-rc2_1_ga10d50f99

I hope this helps.

[kusumi]

The problem here seems to be that zfs umount -a tries to unmount in reverse-strcmp order, while pthreads on parallel mounts can race each other. This will likely happen with or without using canmount if the same race condition happens.

Below is basically same as what op has pasted here, except that it creates regfiles to make this clearer.

# cat ./test0.sh
zpool create -f p1 <device>
zpool create -f p2 <device>

zfs create p1/f1
zfs create p2/f2

zfs set mountpoint=none p1
zfs set mountpoint=/mntpt p1/f1
touch /mntpt/file_for_p1

zfs set canmount=off p2
zfs set mountpoint=/mntpt p2
touch /mntpt/f2/file_for_p2

zfs list -o name,mounted,canmount,mountpoint
zfs unmount -a
zfs mount -a

tree /mntpt

zfs list -o name,mounted,canmount,mountpoint
zfs unmount -a
zfs mount -a

As shown below, p2/f2 mount has been shadowed by p1/f1 mount, hence zfs unmount -a trying to unmount /mntpt/f2 first will fail with "not mounted".

+ zpool create -f p1 sdb
+ zpool create -f p2 sdc
+ zfs create p1/f1
+ zfs create p2/f2
+ zfs set mountpoint=none p1
+ zfs set mountpoint=/mntpt p1/f1
+ touch /mntpt/file_for_p1
+ zfs set canmount=off p2
+ zfs set mountpoint=/mntpt p2
+ touch /mntpt/f2/file_for_p2
+ zfs list -o name,mounted,canmount,mountpoint
NAME   MOUNTED  CANMOUNT  MOUNTPOINT
p1          no        on  none
p1/f1      yes        on  /mntpt
p2          no       off  /mntpt
p2/f2      yes        on  /mntpt/f2
+ zfs unmount -a
+ zfs mount -a
+ tree /mntpt
/mntpt
|-- f2
`-- file_for_p1

1 directory, 1 file
+ zfs list -o name,mounted,canmount,mountpoint
NAME   MOUNTED  CANMOUNT  MOUNTPOINT
p1          no        on  none
p1/f1      yes        on  /mntpt
p2          no       off  /mntpt
p2/f2      yes        on  /mntpt/f2
+ zfs unmount -a
umount: /mntpt/f2: not mounted.
cannot unmount 'p2/f2': umount failed
+ zfs mount -a
cannot mount '/mntpt': directory is not empty

Having a small delay in zfs_mount_task(), will make zfs mount -a mount in the order zfs unmount -a expects. This heuristic defeats concept of parallel mount for mountpoints that overlap (one contains another), but will likely catch most cases.
@behlendorf do you have any thought on this ?

# git diff
diff --git a/lib/libzfs/libzfs_mount.c b/lib/libzfs/libzfs_mount.c
index 649c232aa..c74aa77cf 100644
--- a/lib/libzfs/libzfs_mount.c
+++ b/lib/libzfs/libzfs_mount.c
@@ -1450,6 +1450,7 @@ zfs_mount_task(void *arg)

                if (!libzfs_path_contains(mountpoint, child))
                        break; /* not a descendant, return */
+               usleep(10 * 1000);
                zfs_dispatch_mount(mp->mnt_hdl, handles, num_handles, i,
                    mp->mnt_func, mp->mnt_data, mp->mnt_tp);
        }

With above change, now both regfiles are visible. The last zfs mount -a failure is caused by /mntpt not being empty for p1/f1 mount (i.e. at this point /mntpt/f2 exists). It has nothing to do with the race.

+ zpool create -f p1 sdb
+ zpool create -f p2 sdc
+ zfs create p1/f1
+ zfs create p2/f2
+ zfs set mountpoint=none p1
+ zfs set mountpoint=/mntpt p1/f1
+ touch /mntpt/file_for_p1
+ zfs set canmount=off p2
+ zfs set mountpoint=/mntpt p2
+ touch /mntpt/f2/file_for_p2
+ zfs list -o name,mounted,canmount,mountpoint
NAME   MOUNTED  CANMOUNT  MOUNTPOINT
p1          no        on  none
p1/f1      yes        on  /mntpt
p2          no       off  /mntpt
p2/f2      yes        on  /mntpt/f2
+ zfs unmount -a
+ zfs mount -a
+ tree /mntpt
/mntpt
|-- f2
|   `-- file_for_p2
`-- file_for_p1

1 directory, 2 files
+ zfs list -o name,mounted,canmount,mountpoint
NAME   MOUNTED  CANMOUNT  MOUNTPOINT
p1          no        on  none
p1/f1      yes        on  /mntpt
p2          no       off  /mntpt
p2/f2      yes        on  /mntpt/f2
+ zfs unmount -a
+ zfs mount -a
cannot mount '/mntpt': directory is not empty

[behlendorf]

@kusumi do you happen to know if the reason this wasn't reproducible on illumos is simply down to the timing? Adding a delay might make this less likely but it would be nice to have a properly robust solution.

[kusumi]

@behlendorf
Not sure about illumos (I don't use illumos), but I can/will check how this works on FreeBSD 12. FreeBSD also has this parallel mount feature.

and yes, random delay is definitely not what we want to do to fix it. I did it to showcase the problem is in timing of pthreads which eventually call /bin/mount.

[kusumi]

@behlendorf
The same script worked fine on FreeBSD. With some printf debugging, it seems this really is a matter of timing. I also don't see any significant difference in parallel mount logic in FreeBSD.

Basically, a dataset whose mountpoint is descendant of another mountpoint gets mounted at a cost of extra pthread_create(3) or pthread_cond_signal(3) (depends on state of pthread pool), so mount(2) usually gets invoked in that order.

(If that's the case the (only?) consistent fix might be to serialize mounts for those that overlap. This shouldn't be difficult as there is an example of it in zfs_foreach_mountpoint() when serial_mount is true.)

Edited:
Looks like we can fix this (i.e. make it behave similarly to FreeBSD) without forcing serial mount with kusumi@3fa419b .

The main reason parallel mount behaves differently on ZoL is that
pthreads fork/execs another process for /bin/mount, whereas illumos
and FreeBSD calls mount(2) and nmount(2) respectively.

By taking a global lock around /bin/mount (and also /bin/umount)
call, ZoL will behave similarly to other implementations.

[kusumi]

There is an env var called ZFS_SERIAL_MOUNT to force non-parallel mount.
Define this variable to enable.

This is undocumented (only appears in source code), so may not be stable i/f in the future.

[cyphar]

I think I've seen the underlying issue on openSUSE a couple of times -- sometimes on-boot zfs mount -a ends up mounting a child dataset before the parent. I'll give ZFS_SERIAL_MOUNT a shot then.