[BUG] Retry chroma-load upserts when rate limited

[jasonvigil]

Description of changes

When the upsert operations are rate-limited, the "block" of 100 records in the operation is never inserted. Therefore, the cardinality of the test data set can never grow beyond the point of the failed (rate limited) upsert.

Also, when starting a new workload, we need to reset the cardinality and heap for the data set, so that it starts again from cardinality of 0.

Test plan

Tested locally in tilt. Verified that the cardinality of the test data set continues to grow, even after hitting rate limit errors on previous upsert attempts. Also verified that the cardinality is reset to 0 when initializing a new workload.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Fix Rate-Limited Upserts in Chroma-Load with Retry Mechanism

This PR addresses a bug where upsert operations that are rate-limited would prevent the test data set from growing beyond the point of failure. The solution implements a retry mechanism with exponential backoff for rate-limited upsert operations, and ensures cardinality is reset when starting a new workload.

Key Changes:
• Add retry mechanism with exponential backoff for rate-limited upsert operations
• Reset cardinality and cardinality heap when initializing a new workload
• Add error handling for rate-limited requests (HTTP 429)

Affected Areas:
• rust/load/src/data_sets.rs
• rust/load/Cargo.toml
• Cargo.lock

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

[BestPractice]

Using format!("{err:?}").contains("429") to check for rate limiting is fragile and depends on the specific error format. It would be more robust to check for specific error types or error codes. If the ChromaClient has a specific error type for rate limiting, consider matching on that instead.

[propel-code-bot]

[PerformanceOptimization]

The clone() operations inside the loop can lead to unnecessary memory allocations on each retry. Consider moving the creation of entries outside the loop and only clone when necessary for retries.

[propel-code-bot]

[BestPractice]

Consider using rand to add jitter to your backoff delay to prevent thundering herd problems when multiple clients retry at the same time:

Suggested change

	// Exponential backoff with max delay
	// Exponential backoff with jitter and max delay
	let jitter = rand::random::<f32>() * delay.as_millis() as f32 * 0.1;
	let jittered_delay = delay.checked_add(std::time::Duration::from_millis(jitter as u64)).unwrap_or(delay);
	delay = std::cmp::min(delay * 2, max_delay);

⚡ Committable suggestion

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.