feature@fast_dedup does not become 'active' despite dedup being actively used

[adamdmoss]

System information

Type	Version/Name
Distribution Name	ubuntu
Distribution Version	24.04.1
Kernel Version	6.8
Architecture	x64
OpenZFS Version	git master @ 46c4f2c

Describe the problem you're observing

I upgraded two of my existing pools to the openzfs-2.3 compat profile, which included enabling feature@fast_dedup. I then enabled dedup=on on several of each of these pool's datasets and generally used the pools for a few days including copying some redundant/deduppable data around.

zfs5  dedupratio                     1.06x                          -
zfs5  dedup_table_size               299M                           -
zfs5  dedup_table_quota              auto                           default
zfs5  feature@fast_dedup             active                         local
zfs4-on-8tbgold  dedupratio                     1.22x                          -
zfs4-on-8tbgold  dedup_table_size               1.70G                          -
zfs4-on-8tbgold  dedup_table_quota              auto                           default
zfs4-on-8tbgold  feature@fast_dedup             enabled                        local

On both pools my dedupratio mutated and grew (as did dedup_table_size!), but only the zfs5 pool reports feature@fast_dedup=active; on the other pool it's merely 'enabled'.

I conclude that one of three things is happening:

1. fast dedup is indeed being used on zfs4-on-8tbgold but feature@fast_dedup is not correctly flipping over to 'active'.
2. For some reason zfs4-on-8tbgold is correctly doing dedup but it's exclusively using the old 'slow dedup' path / dedup tables / format.
3. As documented in zpool-features(7) the flag has reverted to 'enabled' because the dedup table is empty; with my recent activity and dedupratio I'm doubtful that this is the case but I'd be happy to try to verify this.

(1) would be a bug, (2) isn't a bug per se but is mysterious and inscrutable AFAICT. I suspect that (2) is what's actually happening because (for whatever reason) I haven't prompted creation of a new dedup table on zfs4-on-8tbgold; the mechanisms for doing so are somewhat vague in zpool-features(7) but I'll have a little fiddle around.

Include any warning/errors/backtraces from the system logs

No obvious errors reported.

[robn]

Not clear to me: did you have dedup on these pools before the upgrade? feature@fast_dedup only goes active the first time a new dedup table is created; existing ones are not converted.

Could you post output of zdb -D <pool>?

[adamdmoss]

Thanks for the response.

Not clear to me: did you have dedup on these pools before the upgrade?

Yes - zfs4-on-8tbgold had some dedup before, zfs5 (I believe) did not.

feature@fast_dedup only goes active the first time a new dedup table is created; existing ones are not converted.

I saw that documented, but it was vague on how a new dedup table is created. I tried changing the data checksum type, the dedup verification checksum type, and of course adding (and removing) dedup=on to new datasets FWIW.

Could you post output of zdb -D <pool>?

% sudo zdb -D zfs4-on-8tbgold 
[sudo] password for adam: 
DDT-sha256-zap-duplicate: 284025 entries, size 648044544 on disk, 209244160 in core
DDT-sha256-zap-unique: 1940810 entries, size 1365381120 on disk, 440872960 in core

dedup = 1.24, compress = 1.63, copies = 1.00, dedup * compress / copies = 2.02

% sudo zdb -D zfs5           
DDT-sha256-zap-duplicate: 51609 entries, size 97672704 on disk, 134283264 in core
DDT-sha256-zap-unique: 716468 entries, size 245345280 on disk, 173244416 in core
DDT-log-sha256-0: 1058 log entries

dedup = 1.06, compress = 1.41, copies = 1.00, dedup * compress / copies = 1.49

[robn]

So I'm 99% sure that its all fine. zfs4-on-8tbgold is using the old format, so the feature is enabled but not active. zfs5 dedup was enabled after, so it's active, and the zdb output shows the DDT-log-* object.

(Not 100% sure because an empty log object on zfs4-on-8tbgold wouldn't be shown, but that would be a very difficult thing to hit on any kind of active pool, and would also mean the feature flag was set wrong. I'm happy to play the numbers on this one!)

So I guess some output and/or doc updates might be in order! At least, zdb should probably show the actual structure & flags on each table, not just ape zpool -D. For you though, what questions were you trying to answer, and where would you have expected the answer to be?

[pondering documentation a little] The zpool-features(7) entry for fast_dedup is technically correct, but really only describes when the feature flag changes, which is kind of an implementation detail.

What we don't really seem to have is something a bit higher-level that explains the concepts, like a single conceptual "table" potentially spanning multiple objects, being shared across the pool but activated per-dataset, and being created when the first block is written and destroyed with the last one is freed.

So yeah, what would you have liked to see?

[adamdmoss]

So yeah, what would you have liked to see?

Good question, thanks for asking! :)

I think I'd either want reassurance that my pool/system isn't suffering by being locked into the old table format, or more concrete information on how to activate the new table format. i.e. I'm okay with the old tables becoming read-only/shrink-only and new tables just being incrementally used for dedups of fresh data; I don't expect a wholesale conversion (though of course I'd be delighted ;) ).

Failing any of the above, I think I'd want it explicitly documented that existing pools with any existing slow-dedup don't actually get fast-dedup ever (if that's actually true). Or whether (as I suspect) slow-dedup itself is a bit faster since fast-dedup landed but won't get the full benefit without a fast-dedup table. (I don't know if that's true either, or if there are numbers to back that up.)

[robn]

I've just finished throwing this together: 077e182. Just a small zdb patch to show a lot better what's really on disk:

$ zdb -D tank
DDT-blake3: version=1 [FDT]; flags=0x03 [FLAT LOG]; rootobj=65
DDT-blake3-zap-duplicate: dspace=0xa3400; mspace=0x88000; entries=1888
DDT-blake3-zap-unique: dspace=0x121800; mspace=0x108000; entries=2968
DDT-log-blake3-0: flags=0x01 [FLUSHING]; obj=66; len=0x0; txg=649; entries=0
DDT-log-blake3-1: flags=0x00; obj=67; len=0x160000; txg=657; entries=360

$ zdb -D tank
DDT-blake3: version=0 [LEGACY]; flags=0x00; rootobj=1
DDT-blake3-zap-duplicate: dspace=0x6e800; mspace=0x48000; entries=1824
DDT-blake3-zap-unique: dspace=0xba400; mspace=0x88000; entries=2752

This isn't really the full answer, but at least it's something we can probably ship immediately that makes it possible to figure out where things are at without the guesswork we had to do.

If you feel like it, give it a whirl. I'll make a PR out of it later today.

I think I'd either want reassurance that my pool/system isn't suffering by being locked into the old table format, or more concrete information on how to activate the new table format. i.e. I'm okay with the old tables becoming read-only/shrink-only and new tables just being incrementally used for dedups of fresh data; I don't expect a wholesale conversion (though of course I'd be delighted ;) ).

Heh, of course! Unfortunately there's no way right now to turn a traditional table into a new one. So your only real option is to somehow drain the existing table and refill it, which means really means a send into a new dedup-capable pool.

(Alas, it just wasn't in scope for the project and though we thought about trying to do it as a side-effect of the main work, it turns out there's some weird edge cases that need a proper solve. Still, it could be done, and we did leave space for both partial and full conversion; for example, see above where my FDT flags are FLAT LOG - it's possible to add a log to a traditional table, but there's no way to ask for that right now).

So yeah, I think documentation is likely to be the whole thing for now. I'm not sure where we really put that right now; zfsconcepts(7) probably needs an updated description of dedup in general, but I'm not really sure where to put upgrade or migration notes. zpool-upgrade(8) maybe, but maybe better would be a whole new page (reminds me a bit of Perl's perldelta(1)).

For you, I think the answer is that existing dedup tables will perform the way they always have. They might be a little quicker due to incidental changes (eg, ZAP objects in general can now reclaim some of their internal space, and we're now sorting table updates to be a bit more efficient), but I wouldn't expect it to be big or consistent enough to document it as an expected performance increase.

And yes, we should say that!

[IvanVolosyuk]

I guess the answer is no - LEGACY will not be gradually upgraded to FDT unless legacy dedup table is cleared first. At least for now. Might be useful to document that, to avoid having higher expectation for users doing the upgrade.

[adamdmoss]

@robn Hey thanks, that patched output is more usable! (I'm not quite so keen on dspace and mspace being printed as hex now but I can live. :) )

But also, hold the front page!

% zpool get all|grep dedup|grep feature
zfs4-on-8tbgold    feature@fast_dedup             active                         local
zfs5               feature@fast_dedup             active                         local

% sudo zdb -D zfs4-on-8tbgold
DDT-sha256: version=0 [LEGACY]; flags=0x00; rootobj=1
DDT-sha256-zap-duplicate: dspace=0x26a06000; mspace=0xc78d000; entries=284362
DDT-sha256-zap-unique: dspace=0x51648000; mspace=0x1a47f000; entries=1947726
DDT-blake3: version=1 [FDT]; flags=0x03 [FLAT LOG]; rootobj=3393
DDT-blake3-zap-duplicate: dspace=0x96c000; mspace=0x580000; entries=20312
DDT-blake3-zap-unique: dspace=0x62eb000; mspace=0x3f40000; entries=215688
DDT-log-blake3-0: flags=0x03 [FLUSHING CHECKPOINT]; obj=3394; len=0xa60000; txg=14872423; entries=69633
    checkpoint: 0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000
DDT-log-blake3-1: flags=0x00; obj=3397; len=0x280000; txg=14872457; entries=18069

So a legacy DDT and a new FDT DDT can live side-by-side, I just had to use dedup=blake3 because only the old the sha256 table was locked into the old DDT. (I'd already tried dedup=blake3 but I guess in my previous attempt I didn't keep any new data around for the table to be visible before I gave up in disappointment.)

Better still, after shuffling some data around:

DDT-sha256: version=0 [LEGACY]; flags=0x00; rootobj=1
DDT-sha256-zap-duplicate: dspace=0x4ddc000; mspace=0x14e1000; entries=868
DDT-sha256-zap-unique: dspace=0x51687000; mspace=0x1a491000; entries=341192
DDT-blake3: version=1 [FDT]; flags=0x03 [FLAT LOG]; rootobj=3393
DDT-blake3-zap-duplicate: dspace=0x742e000; mspace=0x4008000; entries=274586
DDT-blake3-zap-unique: dspace=0x2e8e7000; mspace=0x1e2b0000; entries=1631980
DDT-log-blake3-0: flags=0x00; obj=3394; len=0x0; txg=14874686; entries=0
DDT-log-blake3-1: flags=0x01 [FLUSHING]; obj=3397; len=0x0; txg=14874678; entries=0

So the old tables even shrink, after a fashion, as old data gets destroyed or manually rewritten. At least, the 'duplicate' table gets shrunk; the 'unique' table seems to not shrink despite having far fewer entries now, or I haven't crossed some magical emptiness% threshold.

This means that the behavior I was kinda expecting (new data uses the FDT, old data slowly retires-out of the legacy DDT, in effect there's an incremental movement away from the old tables and towards the new tables) does actually happen, it just requires an explicit dedup checksum type change (hey, blake3 is awesome anyway) and if one can afford the time then re-copying some datasets around.

[robn]

Ahh yes, it's sort of an unclear low-level detail: the top "table" concept (where the legacy/FDT option happens) is based on checksum. So it worked - new checksum means new blocks, which created a new table. Of course to clear everything, ultimately you're going to have to rewrite every single block.

The ZAPs not shrinking is almost certainly because you haven't removed enough entries to clear an entire DDT ZAP data block (~190 entries). It can only reclaim when the entire block is zeroed, and since you have so many more entries there and they're randomly distributed, its gonna take a lot more work to empty it out.

Anyway, this is interesting! I hadn't ever thought about a checksum change as a migration option. I will now! Not sure if its generally useful, but not a bad tool to have in the kit.

(also you're now the world's foremost expert on FDT operations, sorry 😆)

[adamdmoss]

Thus => this defect isn't a defect, just some expectations slightly out of alignment and some magik incantations to close the gap.

I'll close this issue but feel free to reopen if it's helpful to morph it into a doc defect etc etc.
Thanks!