Optimize allocation throttling.

[amotin]

Remove mc_lock use from metaslab_class_throttle_*(). The math there
is based on refcounts and so atomic, so the only race possible there
is between zfs_refcount_count() and zfs_refcount_add(). But in most
cases metaslab_class_throttle_reserve() is called with the allocator
lock held, which covers the race. In cases where the lock is not
held, GANG_ALLOCATION() or METASLAB_MUST_RESERVE are set, and so we
do not use zfs_refcount_count(). And even if we assume some other
non-existing scenario, the worst that may happen from this race is
few more I/Os get to allocation earlier, that is not a problem.

Move locks and data of different allocators into different cache
lines to avoid false sharing. Group spa_alloc_* arrays together
into single array of aligned struct spa_alloc spa_allocs. Align
struct metaslab_class_allocator.

How Has This Been Tested?

On 80-thread FreeBSD system doing ~220K 16KB ZVOLs writes profiler shows reduction of lock contention on allocator locks from 7.3% to 5.8%. In case of file writes, where parallel dnode sync more actively uses multiple allocators, the difference must be more dramatic, since mc_lock is global and not per-allocator.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[amotin]

Can somebody comment on this? It is open for two weeks and it is not a rocket science.

[IsaacVaughn]

Is there any further improvement if you switch zfs_refcount_add to zfs_refcount_add_many and zfs_refcount_remove to zfs_refcount_remove_many? There is a comment claiming that refs must be added individually to be removed individually, but this seems contradictory to the implementation of zfs_refcount as an atomic uint64. Perhaps the slots were previously implemented in a different manner?

Also, perhaps add a comment somewhere documenting the possible racy behavior if the functions are called with no allocation lock and neither GANG_ALLOCATION or METASLAB_MUST_RESERVE set. Although the race does appear harmless, it seems a potential pitfall for future developers who may be surprised at the resulting behavior.

[amotin]

Is there any further improvement if you switch zfs_refcount_add to zfs_refcount_add_many and zfs_refcount_remove to zfs_refcount_remove_many? There is a comment claiming that refs must be added individually to be removed individually, but this seems contradictory to the implementation of zfs_refcount as an atomic uint64.

It would be good, but present implementation is required for debug builds, in which the refcount code can be made to match allocations with frees to detect any mismatches, and _many() would not match. But it affects only blocks with multiple copies, so the difference should not be very dramatic.