[BUG]: don't try reading HNSW files as sparse indices when performing garbage collection on SPANN collections #4401
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… garbage collection on SPANN collections
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
Sicheng-Pan
reviewed
Apr 29, 2025
| @@ -73,7 +74,7 @@ impl ComputeUnusedFilesOperator { | |||
| for segment_compaction_info in older_segment_info.segment_compaction_info.iter() { | |||
| for (file_type, file_paths) in &segment_compaction_info.file_paths { | |||
| // For hnsw_index files, just add it without comparing with newer version. | |||
| if file_type == "hnsw_index" { | |||
| if file_type == "hnsw_index" || file_type == HNSW_PATH { | |||
There was a problem hiding this comment.
nit: Should we create a enum and use pattern matching here instead? That could be more robust in the future if we introduce more files in the future
There was a problem hiding this comment.
I spent a while investigating. This is not straightforward to do as we would need to push the enum into the proto and/or add a wrapper type for CollectionSegmentInfo.
Alternatively, adding just a SegmentFileType enum and then manually constructing it here works, but slightly decreases code quality everywhere else (we now have an enum but it's almost always consumed as SegmentFileType::_.as_str().to_string()) and is really only a stopgap solution.
Ok to file a ticket to add a file type enum to the proto and merge for now?
There was a problem hiding this comment.
Created a ticket and merging.
https://github.com/chroma-core/hosted-chroma/issues/2508
Sicheng-Pan
approved these changes
Apr 29, 2025
warpbuild-benchmark-bot bot
added a commit
to WarpBuilds/chroma
that referenced
this pull request
Apr 29, 2025
… garbage collection on SPANN collections (chroma-core#4401)
chroma-droid
pushed a commit
that referenced
this pull request
Apr 29, 2025
… garbage collection on SPANN collections (#4401) ## Description of changes Collections that use a HNSW index have a segment file path called `hnsw_index`. However, collections with a SPANN index have a segment file path called `hnsw_path`. Currently garbage collection logic treats `hnsw_index` as HNSW files but was treating `hnsw_path` as sparse indices because we never match on that file type name. This PR fixes the logic so that both `hnsw_index` and `hnsw_path` file types are treated as HNSW files. ## Test plan _How are these changes tested?_ Added test fails on main. ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_ n/a
eculver
pushed a commit
that referenced
this pull request
Apr 30, 2025
This PR cherry-picks the commit 12a633c onto release/2025-04-25. If there are unresolved conflicts, please resolve them manually. Co-authored-by: Max Isom <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
Collections that use a HNSW index have a segment file path called
hnsw_index. However, collections with a SPANN index have a segment file path calledhnsw_path. Currently garbage collection logic treatshnsw_indexas HNSW files but was treatinghnsw_pathas sparse indices because we never match on that file type name.This PR fixes the logic so that both
hnsw_indexandhnsw_pathfile types are treated as HNSW files.Test plan
How are these changes tested?
Added test fails on main.
Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?
n/a