Skip to content

[ENH] Implement log forking #4326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Apr 22, 2025
Merged

[ENH] Implement log forking #4326

merged 4 commits into from
Apr 22, 2025

Conversation

Sicheng-Pan
Copy link
Contributor

@Sicheng-Pan Sicheng-Pan commented Apr 18, 2025 •
edited
Loading

Description of changes

Summarize the changes made by this PR.

  • Improvements & Bug fixes
    • N/A
  • New functionality
    • Implement log forking. Currently it is only implemented for legacy log service, by naively cloning the log offsets and records.

Test plan

How are these changes tested?

  • Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?

@github-actions GitHub Actions
Copy link

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

This was referenced Apr 18, 2025
Merged
Merged
Merged
Merged
@Sicheng-Pan Graphite App
Copy link
Contributor Author

Sicheng-Pan commented Apr 18, 2025 •
edited
Loading

This stack of pull requests is managed by Graphite. Learn more about stacking.

@Sicheng-Pan Sicheng-Pan marked this pull request as ready for review April 18, 2025 23:26
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-18-_enh_implement_log_forking branch from 9840862 to 5c80da6 Compare April 21, 2025 17:25
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-17-_enh_wire_up_collection_forking_for_python branch from a41a043 to a6aadbd Compare April 21, 2025 17:25
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-18-_enh_implement_log_forking branch from 5c80da6 to d47ad9a Compare April 21, 2025 17:53
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-17-_enh_wire_up_collection_forking_for_python branch from a6aadbd to ffdcc6c Compare April 21, 2025 17:53
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-18-_enh_implement_log_forking branch from d47ad9a to 67eba39 Compare April 21, 2025 18:24
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-17-_enh_wire_up_collection_forking_for_python branch from ffdcc6c to c9b045b Compare April 21, 2025 18:24
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-18-_enh_implement_log_forking branch 2 times, most recently from b9599fd to c17ac56 Compare April 22, 2025 20:21
sanketkedia
sanketkedia reviewed Apr 22, 2025
View reviewed changes
go/pkg/log/repository/log.go
}
}()

sourceBounds, err := queriesWithTx.GetBoundsForCollection(ctx, sourceCollectionID)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

handle the case when no row exists in collection table. This can happen right after you create a collection before adding any data

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be handled by the query

sanketkedia
sanketkedia reviewed Apr 22, 2025
View reviewed changes
go/pkg/log/store/db/queries.sql.go
SELECT
COALESCE(record_compaction_offset_position, 0) AS record_compaction_offset_position,
COALESCE(record_enumeration_offset_position, 0) AS record_enumeration_offset_position
SELECT record_compaction_offset_position, record_enumeration_offset_position
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change should not be there. Will revert

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a change introduced from main

@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-17-_enh_wire_up_collection_forking_for_python branch from 1b64a55 to b9a2835 Compare April 22, 2025 21:44
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-18-_enh_implement_log_forking branch from c17ac56 to 3a995a4 Compare April 22, 2025 21:44
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-17-_enh_wire_up_collection_forking_for_python branch from b9a2835 to 71ce8e3 Compare April 22, 2025 21:52
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-18-_enh_implement_log_forking branch from 3a995a4 to 42437c0 Compare April 22, 2025 21:53
@Sicheng-Pan Graphite App
Copy link
Contributor Author

Sicheng-Pan commented Apr 22, 2025 •
edited
Loading

Merge activity

  • Apr 22, 7:04 PM EDT: A user started a stack merge that includes this pull request via Graphite.
  • Apr 22, 7:10 PM EDT: Graphite rebased this pull request as part of a merge.
  • Apr 22, 7:10 PM EDT: A user merged this pull request with Graphite.

@Sicheng-Pan Sicheng-Pan changed the base branch from sicheng/04-17-_enh_wire_up_collection_forking_for_python to graphite-base/4326 April 22, 2025 23:07
@Sicheng-Pan Sicheng-Pan changed the base branch from graphite-base/4326 to main April 22, 2025 23:08
@Sicheng-Pan Sicheng-Pan force-pushed the sicheng/04-18-_enh_implement_log_forking branch from 42437c0 to 0b3979c Compare April 22, 2025 23:09
@Sicheng-Pan Sicheng-Pan merged commit 1fe8a68 into main Apr 22, 2025
4 checks passed
warpbuild-benchmark-bot bot added a commit to WarpBuilds/chroma that referenced this pull request Apr 24, 2025
@warpbuild-benchmark-bot
[ENH] Implement log forking (chroma-core#4326)
d8018e4
warpbuild-benchmark-bot bot added a commit to WarpBuilds/chroma that referenced this pull request Apr 24, 2025
@warpbuild-benchmark-bot
[ENH] Implement log forking (chroma-core#4326)
14afe23
@Sicheng-Pan Sicheng-Pan deleted the sicheng/04-18-_enh_implement_log_forking branch May 28, 2025 23:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sanketkedia sanketkedia sanketkedia approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Sicheng-Pan @sanketkedia