[2.2] spa: make read/write queues configurable

[robn]

Backporting #15675 to 2.2.

Note that master has removed ZTI_BATCH and added ZTI_SYNC. This PR matches 2.2, that is, it accepts batch instead of sync. Comments and manpages have been updated accordingly.

Motivation and Context

We are finding that as customers get larger and faster machines (hundreds of cores, large NVMe-backed pools) they keep hitting relatively low performance ceilings. Our profiling work almost always finds that they're running into bottlenecks on the SPA IO taskqs. Unfortunately there's often little we can advise at that point, because there's very few ways to change behaviour without patching.

Description

This commit adds two load-time parameters zio_taskq_read and zio_taskq_write that can configure the READ and WRITE IO taskqs directly.

This achieves two goals: it gives operators (and those that support them) a way to tune things without requiring a custom build of OpenZFS, which is often not possible, and it lets us easily try different config variations in a variety of environments to inform the development of better defaults for these kind of systems.

Because tuning the IO taskqs really requires a fairly deep understanding of how IO in ZFS works, and generally isn't needed without a pretty serious workload and an ability to identify bottlenecks, only minimal documentation is provided. Its expected that anyone using this is going to have the source code there as well.

Sponsored-by: Klara, Inc.
Sponsored-by: Wasabi Technology, Inc.

How Has This Been Tested?

This PR compiled and sanity checked only. Assuming it works based on #15675 and CI results.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

@robn this change also needs to be fixed up to resolve the CentOS 7 compile warning.

https://build.openzfs.org/builders/CentOS%207%20x86_64%20%28TEST%29/builds/1559

[robn]

Ahh crap, this is from before the FreeBSD fixes too. Wow, how'd I stuff that up. #15696 is similarly busted.

Will push update in an hour or two.

[robn]

Pushed.