[ENH]: perform collection hard deletes from garbage collector

[codetheweb]

Description of changes

This updates the garbage collector to transition soft deleted collections to hard deleted when eligible.

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

I updated the proptest with a new transition for deleting collections.

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

n/a

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[codetheweb]

• [ENH]: wire GC v2 to new cleanup modes & call FinishDatabaseDeletion from garbage collector #4671
• [ENH]: soft delete databases, add FinishDatabaseDeletion gRPC method to hard delete databases #4627
• [ENH]: perform collection hard deletes from garbage collector #4605 👈 (View in Graphite)
• [ENH]: sysdb changes to support moving collection hard deletes to garbage collector #4607
• [ENH]: add validation during garbage collection for empty file paths #4586
• [ENH]: garbage collector v2 orchestrator (supports forking) #4468
• [ENH]: allow deleting v0 from version file & correctly propagate errors on DeleteCollectionVersion #4579
• [ENH]: add operator for garbage collection to list all files used by collection version #4466
• [ENH]: add operator to compute versions to garbage collect from version graph #4464
• [ENH]: add orchestrator to construct version graph for garbage collection #4463
• [ENH]: add batch get version file paths method to Sysdb #4432 : 1 other dependent PR (#4447 )
• [ENH]: SysDb should return lineage, version file paths and root collection ID on collections #4557
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codetheweb]

changes in this file provide a little more debugging info + a new defensive check

[propel-code-bot]

Enable Automatic Hard Deletion of Collections via Garbage Collector

This PR implements major architectural changes to the garbage collection system in Chroma, allowing the orchestrator to perform hard deletion of collections that were previously soft deleted and have become eligible for permanent removal. The change propagates through several crates, introducing new status query APIs, collection dependency handling, data structure extensions, and test fixture modifications, with particular focus on guaranteeing correctness for forked/dependent collections and their deletion lifecycles.

Key Changes:
• The garbage collector orchestrator (garbage_collector_orchestrator_v2.rs) now checks for soft-deleted collections and, when eligible (i.e., no dependent/forked live children), invokes hard deletion via sysdb.
• Added collection dependency graph construction in the garbage collector and use of topological order for deletion, ensuring children are deleted before parents.
• New sysdb APIs and gRPC methods (batch_get_collection_soft_delete_status, finish_collection_deletion) to support both querying and performing hard deletes; interfaces extended in sysdb.rs and wire/proto contracts updated.
• ComputeVersionsToDeleteOperator now receives the set of soft-deleted collections, marking all their versions for deletion.
• Proptest/test harnesses and the in-memory reference model were updated for the new collection lifecycle, with additional transitions and invariant checks (e.g., verifying hard deletes truly remove metadata from sysdb).
• Dependency improvements in collection graph construction, test helpers, and structure typing (e.g., separation of soft and hard delete states; explicit collection deletion sentinel values).

Affected Areas:
• garbage_collector_orchestrator_v2.rs
• sysdb.rs (and test_sysdb.rs)
• types.rs (types and API contracts)
• construct_version_graph_orchestrator.rs (defensive checks, graph construction)
• operators/compute_versions_to_delete_from_graph.rs
• proptest_helpers/garbage_collector_reference.rs/garbage_collector_under_test.rs

Potential Impact:

Functionality: Transitions deletion from manual/user-initiated steps to a fully orchestrated, topologically correct hard delete after all dependencies are met, affecting retention semantics, lifecycle guarantees, and overall data integrity for collections and their descendants.

Performance: Minimal direct perf impact; some additional sysdb queries and graph traversals may add overhead for large fork trees, but these are generally asynchronous/batch operations.

Security: Correct erasure of hard-deleted collections prevents data retention leaks; security posture enhanced if orchestrated properly.

Scalability: Imposes a dependency graph and topological sort for deletes, but approach is inherently scalable as all edges are explicit. Soft delete batch queries scale with the number of collections eligible for GC.

Review Focus:
• Correctness of dependency/resolution logic, particularly the topological sort and condition checks for hard delete eligibility.
• Backward compatibility of new sysdb API surfaces, including error handling and return data for new calls.
• Invariants/defensive checks in both garbage collection and graph construction.
• Test coverage and assertions for new collection lifecycle states.
• Impact on distributed/parallel GC scenarios where collection lifecycles may race.

Testing Needed

• Run full proptest suite to validate all new transition states and invariants, especially hard-deletion propagation through collection forks and dependencies.
• Run integration tests to ensure sysdb contains no metadata for hard-deleted collections and that no files or segments remain.
• Validate that cycles and edge cases in collection dependency graphs (e.g., forks with mixed live/dead descendants) result in the correct set of deleted collections.
• Verify soft deletion and eligibility semantics remain correct in both test and mock sysdb implementations.

Code Quality Assessment

rust/sysdb/src/sysdb.rs: API extended cleanly; method naming is consistent. Implementation for 'finish_collection_deletion' and batch queries follows Rust async conventions.

rust/garbage_collector/src/operators/compute_versions_to_delete_from_graph.rs: Logic extended to support soft-deleted collections; retains clear separation of concerns and remains testable.

rust/garbage_collector/tests/proptest_helpers/garbage_collector_reference.rs: Substantial state and transition refactor, strong invariants, and testability improved.

rust/garbage_collector/src/construct_version_graph_orchestrator.rs: Structural improvements with defensive checks; code cleanliness maintained.

others: Internal type wiring and glue code follow Rust conventions. Test and ref code are verbose but structured.

rust/garbage_collector/src/garbage_collector_orchestrator_v2.rs: Substantial new logic, clear structure, good use of error handling and tracing. Functions remain within idiomatic bounds. Watch for error propagation and partial state updates.

Best Practices

Testing:
• Rich property-based/proptest testing with genuine model/reference validation.
• Defensive runtime invariant checks for graph integrity and lifecycle correctness.

Modularization:
• Separation of sysdb, orchestrator, operator, and reference/test code.
• Public APIs and test APIs clearly separated.

Error Handling:
• Consistent error propagation via enums and custom error types.
• Defensive error-to-internal mapping in orchestration.

Documentation:
• Method comment headers and code docstrings present; inline developers notes added on key invariants.

Potential Issues

• If a fork dependency graph is large and deeply nested, the toposort and eligibility checks could be computationally heavier than before.
• Errors or partial failures in sysdb during batch soft delete status queries or hard delete calls may leave collections in inconsistent states unless fully transactional or carefully handled.
• Backward compatibility: Services expecting previous manual delete step must now access sysdb via new APIs for deletion completeness.
• Potential edge cases if a collection's child is restored or recreated between soft-delete and hard-delete steps.

This summary was automatically generated by @propel-code-bot

[Sicheng-Pan]

if we iterate in reverse topological order and soft delete the collections, it should be guaranteed that all children are soft deleted in each step right?

[codetheweb]

no, because the graph is comprised of both alive and soft deleted collections
we could break out of the loop on the first alive collection we see (which I think is what you're saying?) but then it's possible to miss collections eligible for hard deletion