Improve zfs send performance by bypassing the ARC

[ahrens]

Motivation and Context

When doing a zfs send on a dataset with small recordsize (e.g. 8K),
performance is dominated by the per-block overheads. This is especially
true with zfs send --compressed, which further reduces the amount of
data sent, for the same number of blocks. Several threads are involved,
but the limiting factor is the send_prefetch thread, which is 100% on
CPU.

The main job of the send_prefetch thread is to issue zio's for the
data that will be needed by the main thread. It does this by calling
arc_read(ARC_FLAG_PREFETCH). This has an immediate cost of creating
an arc_hdr, which takes around 14% of one CPU. It also induces later
costs by other threads:

• Since the data was only prefetched, dmu_send()->dmu_dump_write() will
  need to call arc_read() again to get the data. This will have to look
  up the arc_hdr in the hash table and copy the data from the scatter ABD
  in the arc_hdr to a linear ABD in arc_buf. This takes 27% of one CPU.
• dmu_dump_write() needs to arc_buf_destroy() This takes 11% of one CPU.
• arc_adjust() will need to evict this arc_hdr, taking about 50% of one
  CPU.

Description

All of these costs can be avoided by bypassing the ARC if the data is
not already cached. This commit changes zfs send to check for the
data in the ARC, and if it is not found then we directly call
zio_read(), reading the data into a linear ABD which is used by
dmu_dump_write() directly.

In addition to improving the performance of zfs send, this change
makes zfs send not pollute the ARC cache. In most cases the data will
not be reused, so this allows us to keep caching useful data in the MRU
(hit-once) part of the ARC.

How Has This Been Tested?

The performance improvement is best expressed in terms of how many
blocks can be processed by zfs send in one second. This change
increases the metric by 50%, from ~100,000 to ~150,000. When the amount
of data per block is small (e.g. 2KB), there is a corresponding
reduction in the elapsed time of zfs send >/dev/null (from 86 minutes
to 58 minutes in this test case).

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

I've resubmitted the builds which failed due to the unrelated failure of the alloc_class_013_pos test. The Ubuntu 18.04 failure also appears to be unrelated. It may have been introduced by recent commit cccbed9, it's the first time I've seen this particular failure in the CI.

[ahrens]

re-pushed the same changes to try and get the tests to run to completion.

[behlendorf]

Only expected failures in the updated push. This is ready to merge.