[ENH] Bootstrap a wal3 log from existing content.

[rescrv]

Description of changes

This adds a "bootstrap" call to wal3 that copies the data semi-atomically. It is reasoned to be
safe to do concurrent with all other log operations, but may leave a FragmentSeqNo(1) lying around
in the event that a log is initialized at the same time it's bootstrapping. This should never
happen in our system so thankfully we can just leave the hole.

See #4558 for the reasoning.

Test plan

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

N/A

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Add Bootstrap Functionality to wal3 Log from Existing Content

This PR introduces a new bootstrap method to the wal3 log writer, enabling initialization of a new log from an existing set of records. The function provides a semi-atomic process to copy data into a new log, with checks to avoid bootstrapping over already-initialized or in-use logs, and is designed to be idempotent and safe for recovery if partially completed. Two integration tests are added to validate both populated and empty bootstrap scenarios.

Key Changes:
• Implemented LogWriter::bootstrap async method for initializing a log from supplied messages/offset.
• Updated fragment upload logic to avoid creating empty fragments.
• Adjusted internal error handling around atomic file creation and manifest updates during bootstrap.
• Added two integration tests: one for bootstrapping a log with records and one for empty bootstraps.
• Minor update: Derived Default for Limits struct in reader module.

Affected Areas:
• rust/wal3/src/writer.rs
• rust/wal3/src/reader.rs
• rust/wal3/tests/test_k8s_integration_83_bootstrap.rs
• rust/wal3/tests/test_k8s_integration_84_bootstrap_empty.rs

Potential Impact:

Functionality: Enables initializing Rust-based logs from arbitrary existing data, facilitating system migrations and recovery scenarios.

Performance: No significant runtime path changes; only affects manual/admin/bootstrap operations.

Security: No sensitive information exposed, but as with all initialization routines, care must be taken to avoid race conditions.

Scalability: No impact on log scalability or concurrency in normal operation; bootstrap is a one-time admin operation.

Review Focus:
• Correctness and atomicity of the bootstrap process.
• Error handling and rollback on partial failure.
• Test coverage for edge cases, such as empty log or concurrent initialization.

Testing Needed

• Run the new integration tests to verify both non-empty and empty bootstrapping.
• Exercise concurrent initialization (if feasible) to observe contention/error handling.
• Manually test idempotency by re-running bootstrap under failure scenarios.

Code Quality Assessment

rust/wal3/src/writer.rs: Well-reasoned, clear inline documentation, explains invariants and known tradeoffs.

rust/wal3/src/reader.rs: Minor improvement with Default derivation; otherwise unchanged.

rust/wal3/tests/test_k8s_integration_83_bootstrap.rs: Good scenario and data validation.

rust/wal3/tests/test_k8s_integration_84_bootstrap_empty.rs: Test coverage for empty bootstraps.

Best Practices

Modularization:
• New API surface (bootstrap) is clearly separated

Atomicity:
• Partial failure recovery considered, idempotency noted

Testing:
• Relevant integration tests for new code

Documentation:
• Clear doc-comments and inline reasoning

Error Handling:
• Explicit error paths, safe to re-run on failure

Potential Issues

• Very rare edge case where a 'hole' (FragmentSeqNo(1)) is left if initialization and bootstrap race, but this is predicated not to occur in expected workflows.
• If bootstrap is used outside migration/one-shot admin scenarios, it could subvert concurrency guarantees.
• Any changes to upload_parquet or manifest error handling in the future should re-examine bootstrap robustness.

This summary was automatically generated by @propel-code-bot

[codetheweb]

might be good to have a basic test for this