OpenZFS 6569 - large file delete can starve out write ops

[alek-p]

This is a port of the delete throttle work from OpenZFS, see: openzfs/openzfs#214

The core issue I've found is that there is no throttle for how many deletes get assigned to one TXG.
As a results when deleting large file(s) we end up filling consecutive TXGs with deletes/frees,
then write throttling other (more important) write ops.

There is an easy test case for this problem. Try deleting several large files (at least 1/2 TB) while
you do write ops on the same pool.
What we've seen is performance of these write ops (let's call it sideload I/O) would drop to zero.

More specifically the problem is that dmu_free_long_range_impl() can/will fill up all of the
dirty data in the pool "instantly", before many of the sideload ops can get in.
So sideload performance will be impacted until all the files are freed.
The solution we have tested at Nexenta (with positive results) creates a relatively simple throttle for how many "free" ops we let into one TXG.

[mention-bot]

@alek-p, thanks for your PR! By analyzing the history of the files in this pull request, we identified @ahrens, @behlendorf and @DKOI to be potential reviewers.

[alek-p]

I wonder if this patch should wait for lazy unmount to be integrated. Reason being that w/o lazy unmount the unmount will fail w/ EBUSY if there are files being freed since we will hold open the referecnes to these files. The delete throttle patch makes this window of how long we're freeing these files longer hence the EBUSY unmount window is longer.
I'll can look into what it takes to implement lazy unmount, unless someone has already done the research?

[behlendorf]

@alek-p I think I can save you a little time. Lazy unmounts are something the Linux kernel should handle entirely for us in the VFS. Providing the --lazy flag to umount(8) will detach the namespace but defer the destruction of the super block until the last reference is dropped. I believe the only thing missing is extending the zfs umount command to accept a -l flag and pass it all the way through to umount(8).

[behlendorf]

The change LGTM but I'd like to get a fresh run through the automated testing before merging this. @alek-p can you rebase this branch on the current master and force update the branch on GitHub. That will trigger another test run.

[behlendorf]

@alek-p I've rebased this branch against master and resolved the trivial conflict.

[dinatale2]

Missing the Ported-by: line in the commit message as well.

[dinatale2]

Style warning from the buildbot on this line. Space after cast.

[behlendorf]

This patch diverges slightly from what was merged to OpenZFS in late December. Let's port this again using the finalized version of the patch.

[gmelikov]

@behlendorf i'll port it again.

[behlendorf]

Superseded by #5706.