systemd-mount-generator breaks with default mountpoints due to `mount` canonicalizing the source argument when calling `mount.zfs`

[chenxiaolong]

System information

Type	Version/Name
Distribution Name	Fedora
Distribution Version	33
Linux Kernel	5.9.16-200.fc33.x86_64
Architecture	x86_64
ZFS Version	2.0.1-1
SPL Version	2.0.1-1
util-linux Version	2.36-3.fc33.x86_64

Describe the problem you're observing

With systemd .mount units, the mount command gets run in CWD /, which causes mount to turn the source argument into an absolute path before calling mount.zfs. This is problematic with the default mountpoint scheme where the dataset name is the mountpoint. When mount is called with pool/dataset as the source, mount.zfs gets called with /pool/dataset.

This can be worked around when mounting manually by:

• Passing --no-canonicalize to mount
• Running mount from anywhere else besides /

Unfortunately, I don't think systemd .mount units support either of those options. Would it be feasible to try and detect this somehow in mount.zfs or maybe there's a better workaround available?

Describe how to reproduce the problem

1. Enable systemd-mount-generator:

touch /etc/zfs/zfs-list.cache/pool

2. Create a test dataset and unmount it:

zfs create pool/test
zfs unmount pool/test

3. Reload systemd and try to mount the volume:

systemctl daemon-reload
systemctl start pool-test.mount

4. The mount fails with:

filesystem '/pool/test' cannot be mounted, unable to open the dataset

---

The mount canonicalization behavior can also be reproduced by:

$ zfs create pool/test
$ zfs unmount pool/test
$ cd /
$ mount pool/test /pool/test -t zfs -o defaults,atime,relatime,dev,exec,rw,suid,nomand,zfsutil
filesystem '/pool/test' cannot be mounted, unable to open the dataset
$ cd /tmp
$ mount pool/test /pool/test -t zfs -o defaults,atime,relatime,dev,exec,rw,suid,nomand,zfsutil
# ^^ Works outside of /
$ umount /pool/test
$ mount pool/test /pool/test -t zfs -o defaults,atime,relatime,dev,exec,rw,suid,nomand,zfsutil --no-canonicalize
# ^^ Also works with --no-canonicalize

[chenxiaolong]

Looks like #6437 was the fix for the same issue in zfs mount -a. Unfortunately, that --no-canonicalize trick won't work in mount units.

[sammko]

Same problem on archlinux. Do you know what has changed to cause this? The same setup worked just fine before.

[sammko]

For now, I patched my systemd to pass the --no-canonicalize option to mount.

diff --git a/src/core/mount.c b/src/core/mount.c
index 8e83de0ba8..ba62be96c9 100644
--- a/src/core/mount.c
+++ b/src/core/mount.c
@@ -1036,6 +1036,8 @@ static void mount_enter_mounting(Mount *m) {
                         r = exec_command_append(m->control_command, "-t", p->fstype, NULL);
                 if (r >= 0 && !isempty(opts))
                         r = exec_command_append(m->control_command, "-o", opts, NULL);
+                if (r >= 0)
+                        r = exec_command_append(m->control_command, "--no-canonicalize", NULL);
         } else
                 r = -ENOENT;
         if (r < 0)
@@ -1096,6 +1098,8 @@ static void mount_enter_remounting(Mount *m) {
                         r = exec_command_append(m->control_command, "-w", NULL);
                 if (r >= 0 && p->fstype)
                         r = exec_command_append(m->control_command, "-t", p->fstype, NULL);
+                if (r >= 0)
+                        r = exec_command_append(m->control_command, "--no-canonicalize", NULL);
         } else
                 r = -ENOENT;
         if (r < 0)

I'm not sure if it should be done unconditionally, but systemd.mount(5) says that What= should be an absolute path, so I hope nothing relies on it doing the canonicalization, relative to /, which makes it completely pointless anyway.

I feel the proper solution is to add an option to systemd.mount(5), but I think the origin of the problem should be discovered first.

[aerusso]

This is happening because, in util-linux's cannonicalize.c:canonicalize_path_restricted, the realpath function determines that the string pool/test is a bonafide path---and prepends / to it. This will not happen if there is no file or directory matching the name of the dataset (i.e., /pool/test). (That's not a nice workaround, but it is a workaround.)

I don't think we at ZoL can do anything about this---I think we will need to ask systemd to either set --no-canonicalize by default, or at least provide an option to disable canonicalization (or for util-linux to reverse this decision). I do not understand why util-linux made this decision---many other filesystems do not have the "device" as a genuine filesystem object.

[aerusso]

We could actually fix this on the ZoL side---by accepting paths that start with /. We could shield that behavior behind another mount option, say "undocanonicalization", and pass that as an option to zfs in the zfs-mount-generator. 🤦

[chenxiaolong]

I've submitted a feature request for systemd here systemd/systemd#18188 asking if they'd be open to either passing --no-canonicalize unconditionally or making it an optional parameter in mount units.

At the same time, I'd love to see @aerusso's idea of having a undocanonicalization zfs mount option also. That would help for current/older distros where a potential systemd patch likely wouldn't be backported.

[chenxiaolong]

This has been fixed by util-linux/util-linux@372ce5b, so I'll close this issue. I'm happy with patching my local util-linux until its next release gets picked up by distros.