blake3 vs. fletcher4

[mabod]

Hello,

I am trying to understand the benefits of blake3 vs. fletcher4 checksums. But I dont get it.
What is a good reason to move to blake3? Performance wise I believe fletcher4 is still faster. Is that right?

[IvanVolosyuk]

The fletcher4 checksum should be the fastest by far. I believe the only reason to switch to a different checksum algorithm is if you use encryption or dedup. I might be wrong though. ;)

…

On Thu, Oct 19, 2023 at 6:55 PM mabod ***@***.***> wrote: Hello, I am trying to understand the benefits of blake3 vs. fletcher4 checksums. But I dont get it. What is a good reason to move to blake3? Performance wise I believe fletcher4 is still faster. Is that right? — Reply to this email directly, view it on GitHub <#15421>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HN2N47VBWW7CRJI4JTYADMG3AVCNFSM6AAAAAA6GXUYRWVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZVG42TENJSGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[robn]

Blake3 is a cryptographic hash. Unless you need cryptographically-strong checksums (either you’re using dedup, or you have a particular application that requires it), then the default fletcher4 is still faster.

[dodexahedron]

And, for what it's worth, on CPUs from the past decade, SHA512 tends to be quite a bit faster than SHA256 while not costing you anything except in the edge case that those extra 32 bytes would have made the difference between a record getting inlined in the dnode or not.

And blake3 is significantly faster than either of those (often multiple times faster), on x86 and 64-bit ARM variants, with the gains being more apparent with increasing size of input to it (so, bigger records benefit even more).

Regardless, hashing throughput is extremely unlikely to be a bottleneck or even more than a rounding error in the performance of ZFS on any system. Any amount of compression of any algorithm will cost a lot more than what the hash algorithm costs, in terms of CPU load, yet that'd also be an unwise thing to disable in most cases.

[IvanVolosyuk]

The last part is interesting. How can I actually measure the compression / hashing throughput in ZFS? I did a bit on benchmarking using command line tool for blake3, lz4 and fletcher4 (ripped AVX512f version from ZFS). I've got some numbers like

fletcher 30GB/s
lz4 5GB/s (compress)
blake3 3GB/s

Obviously, I didn't measured the same code we have in ZFS, but it gave me rough idea about the performance numbers.

There is also consideration of compressed ARC. Every time I want to access compressed data in ARC it will be decompressed, but hashing happens only once when data is read from disk.

[LittleNewton]

@IvanVolosyuk
Could you please share your CPU information with me? I have recompiled OpenSSL with the flag -march=native, and the result showed that SHA-256 is much faster than SHA-512. My host is a Hyper-V Debian virtual machine based on an Intel Core 12700K CPU.

[IvanVolosyuk]

I have AMD 7950x3d, but how is related? I didn't check sha{256,512} peformance, just hashing and compression.

[LunarLambda]

Worth noting that the 12700K disables AVX512 depending on chip serial and microcode version, and AMD Zen 4 architecture only used a 256-bit datapath for AVX512. So benchmark results could vary strongly here.

In any case it would be good to take note of which extensions are being used for which hash in this discussion.