l2arc_noprefetch documentation is unclear - what does this do?

[recklessnl]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	10
Linux Kernel	5.4.14
Architecture	x64
ZFS Version	0.8.4
SPL Version	0.8.4-pve1

Describe the problem you're observing

I'm trying to tune L2ARC for maximum performance but I'm having trouble understand how exactly it operates in ZFS. The device that I want to use as an L2ARC cache vdev is an Intel P4608 enterprise SSD which is an x8 PCIe3 SSD device. It features 2 seperate pools of 3.2TB, each with x4 lanes, and I want to stripe both of these together for the combined speed and IOPS (would use x8 PCIe lanes for this). You can view additional stats about this drive here to give you an idea of the performance it is capable of.
RAM on this system is 512GB in total. I want to stripe this device as a cache vdev, so total L2ARC would be ~6.4TB.

Random reads and smaller size reads will surely be much faster compared to the pool of disks, but I'm confused about sequential reads and the docs do not explain properly. I do want sequential cached reads to be pulled from these drives as well for now because I can't see the harddrive pool outperforming this SSD when striped. I'm planning to use zpool add poolname cache ssd1 ssd2 as command, this will stripe the SSDs together instead of creating a JBOD pool, right?

Additionally, I'm seeing information that you need to set the l2arc_noprefetch tunable to 0 in order to properly allow seqential reads but is this how it actually works? Does it not do sequential reads unless you set that to 0 (default is 1) or am I not understanding it correctly?

https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/ZFS%20on%20Linux%20Module%20Parameters.html#l2arc-noprefetch

I'm also wondering what other performance tuning I should do in order to get the most out of L2ARC with modern hardware and hope you guys can give me some pointers.

[shodanshok]

When issuing sequential reads, much data is prefetched - loaded before the actual demand load. These prefetched buffers are stored in the MRU list and tagged with a "prefetched" flag. If these buffers are referenced by a demand load, the "prefetched" flag is cleared (and if multiple references happen, they are stored in the MFU list).

By default, any buffer flagged with the "prefetched" tag is not eligible for L2ARC, so that one-shot sequential reads does not pollute the L2ARC. Multiple sequential reads will gradually flag more buffer as eligible for L2ARC (clearing the "prefetched" flag), and the same will do random reads for these prefetched buffers.

The rationale is that HDDs are quite fast for sequential reads, and so it is better to use the available L2ARC for random reads. This is especially true for large pools (ie: 12+ vdevs), where the combined sequential read speed can easily be >1GB/s. However, if having a large and fast L2ARC as your Intel drive, you can set l2arc_noprefetch=0 to ignore the "prefetched" flag and to always cache any buffer which is to be evicted by ARC.

I agree that the docs are not very clear on this point.

[richardelling]

Since I wrote the module parameters doc, I'm a bit biased, but always looking to improve.

In the modules parameters, the range says:
0=write prefetched but unused buffers to cache devices, 1=do not write prefetched but unused buffers to cache devices

There is not a relationship between "sequential" and "prefetch" access patterns. They are orthogonal ideas. Sequential most often refers to contiguous LBA regions on a HDD (there is no equivalent for SSD). Prefetch is done on an object's ordered blocks. It is rare that prefetches are also sequential.

With this knowledge, how would we write this document better?

[shodanshok]

@richardelling "sequential", in my reply above, refers to contiguous reads as done by the user application - ie: the same a cp or dd does. I was not speaking about physically sequential LBAs. Is this view incorrect?

[richardelling]

@shodanshok Yes, I think your interpretation of sequential as in the object (a file is an object in a dataset) is correct. This is where the ZFS prefetcher works: traversing the list of blocks in an object.

[recklessnl]

Thank you for the replies, and thank you for clearing up some of the confusion @shodanshok.
I think with 2.0 coming up and introducing persistent L2ARC, plus the SSD revolution still developing now with PCI 4.0, there will be more interest in L2ARC in the coming years so it's a good time to update some of the documentation.

I'm not as versed in this as you guys are so I can give the perspective from an intermediate user. I found the documentation surrounding this to be confusing enough to create an issue here and in my opinion I think it would help if we added an example for this in the docs too, to explain it more in layman's terms on what this setting is actually doing.

L2ARC was first designed in a time where very fast NVMe SSDs didn't exist. The default recommendation for L2ARC these days should be a fast NVMe SSD, preferably more striped like I'm planning to do with the Intel P4608. However, this default setting would prevent the SSD from properly delivering sequential reads of bigger files, that I do want cached on there. Some users might not want that but I feel most will want to with newer technology.

The main thing I've learned is that this setting should be disabled for large sized, NVMe based flash storage cache devices - because these will likely always outperform an even very high amount of striped rust disks. However, if you don't want it to do larger sequential reads, then this setting should be set to 1. I might not be fully understanding it right but I feel the documentation should at least give an example in layman's terms like I tried to do here. It would help with the understanding of how the L2ARC works in general too.

[richardelling]

@recklessnl I think you missed an important point. If the data is touched, then it would not be tagged as "prefetch" and therefore would be eligible for L2ARC.

Also, there are many other considerations for L2ARC that are much more important than prefetching tunables. However, that is beyond scope for an issue.

Lastly, L2ARC exists because the cost of RAM >> cost of SSD. That cost difference is reduced over time, so you are always better off investing in RAM if the cost difference isn't large. Back in the time when L2ARC was first being developed, systems had 2-4GB of RAM and SSDs were 32GB. Today, many systems can easily hold 1.5TB of RAM. So for L2ARC to be cost effective, your working set size needs to be > 1.5TB, which is not common for cache-friendly workloads.

Finally, the ARC tunables are per-node, not per-pool, so there can be no decisions on tunables based on the configuration of a single pool.

[recklessnl]

Also, there are many other considerations for L2ARC that are much more important than prefetching tunables.

I'm very curious if you could give me some pointers in this, and regardless of this specific issue, it's good information to have. Would you share some of the more important tuneables? I would appreciate it (and I also asked about it in the OP, so it is somewhat relevant).

As far as the prefetching goes, thanks for clearing it up more, this makes more sense now.

[adamdmoss]

l2arc_noprefetch=1 is absolutely disastrous for l2arc reads too, so I think there's another angle... or a terrible bug.

[richardelling]

@adamdmoss l2arc_noprefetch has no real affect on non-prefetch reads. What behaviour do you see that indicates otherwise?

[adamdmoss]

@adamdmoss l2arc_noprefetch has no real affect on non-prefetch reads. What behaviour do you see that indicates otherwise?

# echo 0 > /sys/module/zfs/parameters/l2arc_noprefetch                    
# drop;precache /zfs4/250b/SteamLibrary/steamapps/common/BorderlandsPreSequel
0:09.59 elapsed (sys:2.74 user:0.07)
# echo 1 > /sys/module/zfs/parameters/l2arc_noprefetch
# drop;precache /zfs4/250b/SteamLibrary/steamapps/common/BorderlandsPreSequel
0:44.18 elapsed (sys:3.15 user:0.12)
# drop;precache /zfs4/250b/SteamLibrary/steamapps/common/BorderlandsPreSequel
0:44.01 elapsed (sys:3.14 user:0.08)
# echo 0 > /sys/module/zfs/parameters/l2arc_noprefetch
# drop;precache /zfs4/250b/SteamLibrary/steamapps/common/BorderlandsPreSequel
0:09.47 elapsed (sys:2.67 user:0.07)
# drop;precache /zfs4/250b/SteamLibrary/steamapps/common/BorderlandsPreSequel
0:09.66 elapsed (sys:2.69 user:0.08)
# echo 1 > /sys/module/zfs/parameters/l2arc_noprefetch             
# drop;precache /zfs4/250b/SteamLibrary/steamapps/common/BorderlandsPreSequel
0:43.13 elapsed (sys:3.14 user:0.07)

... and so on. (drop = echo 3 > /proc/sys/vm/drop_caches, precache = read data [in this case 8GB already warm in l2arc] at max rate)

It's trivially reproducible here.

[richardelling]

That seems like a contrived experiment. What about real life where folks don't go around dropping caches?

[adamdmoss]

The dropped cache is so the l2arc gets hit rather than the arc, that being the whole point of the test...?

[richardelling]

The question at hand is whether prefetched but unused data should be sent to L2ARC. Obviously, during prefetch, there is only speculation that the data will be used, but no real confidence. There is some period of time between the data being prefetched and its eviction from ARC. That time is based on many variables, such as the size of the MRU and the churn rate. If the MRU size is small and the churn rate is high, then caching prefetched data makes sense. However, when you're in that mode, life isn't very pleasant and there are better cures for the problem (better than kicking the can down to L2ARC).

One method to observe how well the prefetcher is working is to monitor prefetch hit rate in arcstats and monitor the zfetchstats. If the prefetch hit rate is high and MRU size is low, then it is probably a good idea to enable prefetched data caching in L2ARC.