Unability to remove vdev with different ashift values is not documented

[erictapen]

System information

Type	Version/Name
Distribution Name	NixOS
Distribution Version	20.09
Linux Kernel	5.4.83
Architecture	x86_64
ZFS Version	2.0.0-rc3
SPL Version	2.0.0-rc3

Describe the problem you're observing

When having vdevs with different ashift values in one pool, it is impossible to zpool remove one of these, even if there is sufficient disk space. This brought me into a hairy situation recently, when I added all the drives I own to the pool and was then surprised to find out that I wasn't able to remove any of them, as I somehow had one vdev with a different ashift value than the others. I experienced this on 0.8.5, but also verified it on 2.0.0-rc3.

What I'd expect: This should at least be documented in zpool-remove(8) and zpool-add(8). Also it would be nice to get an error when mixing different ashift's which would need to be confirmed with -f, but maybe it is a legitimate use case to mix ashift in a pool?

Describe how to reproduce the problem

$ zfs version
zfs-2.0.0-rc3
zfs-kmod-2.0.0-rc3

$ dd if=/dev/zero bs=1M count=5000 of=vdev1
...

$ dd if=/dev/zero bs=1M count=5000 of=vdev2
...

$ sudo zpool create -o ashift=9 ztest /path/to/vdev1

$ sudo zpool add -o ashift=10 ztest /path/to/vdev2

$ sudo zpool remove ztest /path/to/vdev2
cannot remove /path/to/vdev2: invalid config; all top-level vdevs must have the same sector size and not be raidz.

[ahrens]

I like those ideas, thanks!

[erictapen]

Also I noticed that if there is not sufficient space left in the pool to remove the vdev, zfs will complain only about that fact. Maybe it'd make sense to fail early due to mixed ashifts, so users don't waste time freeing space in a pool, where they cannot remove vdevs anyway?

[erictapen]

Thank you so much for documenting this, @smammy @behlendorf !