Performance degradation in NVMe L2ARC configured ZFS upgrading from 2.1.4 to 2.1.6

[voraharshit]

System information

Type	Version/Name
Distribution Name	Amazon Linux 2
Distribution Version	Amazon Linux 2 AMI 2.0.20230119.1 x86_64
Kernel Version	5.10.165-143.735.amzn2.x86_64
Architecture	x86_64
OpenZFS Version	2.1.6

Describe the problem you're observing

Upgrading from ZFS 2.1.4 to ZFS 2.1.6, I'm seeing increased read latency and reduced bandwidth, IOPS.

I bisected it to the following commit: 39e038cdc61a9af2549fa3ed1f42c16dbcbe4ca6
I have the l2arc_exclude_special tunable set to 0 as is the default. Reverting this commit seems to fix the issue. The issue is only affecting performance for a NVMe L2ARC configured ZFS. ZFS without L2ARC is unaffected.

[allanjude]

Do you have a special vdev in your pool?

[voraharshit]

No, I do not have a special vdev.

[voraharshit]

Please let me know if I can provide any additional details or if there was any advice on how I should investigate this further

[KungFuJesus]

This isn't surprising at all looking at the diff. The test for "is cacheable" is considerably more expensive. Particularly replacing this macro with the function that's there now:
DMU_OS_IS_L2CACHEABLE

The dnode check had some function calls buried in there that might have amounted to similar complexity as I do see at least one function call in there for "is metadata". Still, it's not too hard to see why it'd be slower.

[behlendorf]

@voraharshit thanks for reporting this. If possible could you provide some detail on how significantly this impacted read latency and IOPs. @gamanakis could you look in how we can best optimize this.

[gamanakis]

I should be able to come up with a PR.

[gamanakis]

@voraharshit would you be able to apply #14563 and see if it helps?

[voraharshit]

Thank you @behlendorf, @gamanakis for looking into this! I'll try the PR and get back with some performance details.

[voraharshit]

#14563 is working perfectly, and I'm seeing performance numbers similar to before commit 39e038cdc61a9af2549fa3ed1f42c16dbcbe4ca6

[gamanakis]

Thank you @voraharshit!