[ENH]: Add spann metrics

[sanketkedia]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Adds metrics for various operations during compaction
• New functionality
  • ...

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

None

[sanketkedia]

• [ENH]: Add spann metrics #4492 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[github-actions]

Please tag your PR title with one of: [ENH | BUG | DOC | TST | BLD | PERF | TYP | CLN | CHORE]. See https://docs.trychroma.com/contributing#contributing-code-and-ideas

[propel-code-bot]

Instrumentation of SPANN Operations with Detailed Metrics

This PR adds comprehensive metrics instrumentation to SPANN index operations, particularly during compaction processes. It implements counters and histograms for various operations like reassignments, splits, head creation/deletion, and measures latency for key operations. The PR moves stopwatch functionality to a shared utility and ensures metrics are properly tracked throughout the compaction lifecycle.

Key Changes:
• Added SpannMetrics struct with counters and histograms for all SPANN operations
• Added WriteStats struct to track atomic counters during index operations
• Instrumented code paths to record metrics at key points in compaction workflow
• Added detailed latency tracking for garbage collection, commit, and flush operations
• Refactored Stopwatch utility to be reusable across crates

Affected Areas:
• rust/index/src/spann/types.rs
• rust/tracing/src/util.rs
• rust/cache/src/foyer.rs
• rust/segment/src/distributed_spann.rs
• rust/blockstore/src/arrow/flusher.rs
• rust/blockstore/src/types/flusher.rs

Potential Impact:

Functionality: No changes to core functionality, but enhances observability of SPANN operations

Performance: Minimal overhead from metrics collection, with performance data now available for analysis

Security: No security implications

Scalability: Helps diagnose scalability issues by providing visibility into compaction bottlenecks

Review Focus:
• Implementation of Stopwatch utility and its usage patterns
• Placement of metrics collection points in code paths
• Appropriate use of atomic operations for counter increments
• Naming and organization of metrics for discoverability

Testing Needed

• Verify metrics are properly emitted during compaction operations
• Check that latency measurements accurately reflect operation durations
• Ensure metrics don't add significant overhead to SPANN operations

Code Quality Assessment

rust/tracing/src/util.rs: Good encapsulation of the Stopwatch functionality

rust/blockstore/src/types/flusher.rs: Potential issue with unimplemented!() in num_entries() method for MemoryBlockfileFlusher

rust/index/src/spann/types.rs: File has become significantly larger and could benefit from refactoring into multiple files

Best Practices

Metrics:
• Each operation has a dedicated counter
• Latency measurements use histograms
• Consistent attribute tagging with collection_id

Rust:
• Using Arc<AtomicU32/64> for thread-safe counters
• Appropriate use of shared utility code
• Proper implementation of Drop trait for automatic metric recording

Potential Issues

• The num_entries() method uses unimplemented!() for MemoryBlockfileFlusher which could cause runtime panics
• Frequent string conversions for collection_id may add unnecessary overhead
• Tracking multiple sub-categories of reassigns may create too many metrics

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

[CriticalError]

The num_blocks method for BlockfileFlusher::MemoryBlockfileFlusher uses unimplemented!(). If a SpannIndexFlusher (from chroma-index) is created with MemoryBlockfileFlusher instances, which is possible if SPANN uses in-memory blockfiles, calling flush() on SpannIndexFlusher will invoke num_blocks() on these memory flushers (e.g., in SpannIndexFlusher::flush when calculating num_pl_blocks_flushed). This will lead to a panic.

Consider returning a value that is semantically correct for in-memory flushers. If "flushed blocks" primarily refers to units written to persistent storage, 0 might be appropriate. If it represents a conceptual block, 1 could be an alternative. Returning 0 seems safer to prevent panics and align with the idea of blocks flushed to disk.

Suggested change

	BlockfileFlusher::MemoryBlockfileFlusher(_) => unimplemented!(),
	BlockfileFlusher::MemoryBlockfileFlusher(_) => 0,

⚡ Committable suggestion

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.