poll new blocks in the committer when in live mode

[iuwqyir]

TL;DR

Added support for real-time block committing in addition to backfill mode.

What changed?

• Added a new getBlockToCommitUntil function that determines the end block for committing based on work mode
• In real-time mode, the committer now checks the latest block from RPC and uses that as the upper bound if it's less than the calculated end block
• Created a fetchBlockDataToCommit function that handles different data retrieval strategies based on work mode:
  • In backfill mode: retrieves data from staging storage
  • In real-time mode: uses a BoundlessPoller to fetch data directly without saving to staging first
• Updated the getBlockNumbersToCommit function to accept a context parameter
• Refactored getSequentialBlockDataToCommit to use the new functions

How to test?

1. Run the application in real-time mode and verify it correctly commits blocks up to the latest block from RPC
2. Run the application in backfill mode and verify it commits blocks in batches of blocksPerCommit
3. Verify that when switching between modes, the committer correctly adjusts its behavior

Why make this change?

This change enables the committer to operate in two distinct modes:

1. Backfill mode: for historical data processing where blocks are first staged then committed
2. Real-time mode: for processing current blocks directly from the RPC without staging

This dual-mode approach improves efficiency by allowing real-time processing to skip the staging step when immediate data is needed, while maintaining the ability to process historical data in batches.

Summary by CodeRabbit

• New Features
  • Improved block data fetching logic to dynamically adjust commit ranges based on current mode and blockchain state.
• Bug Fixes
  • Updated gap handling to skip unnecessary operations in live mode, reducing redundant processing.
• Tests
  • Enhanced test coverage to reflect new context-based method signatures and work mode handling.

[coderabbitai]

Caution

Review failed

The pull request is closed.

"""

Walkthrough

The pull request refactors the Committer logic to propagate context, modularizes block data fetching based on work mode, and dynamically determines the commit range using RPC state. It introduces new helper methods, updates method signatures, and adjusts gap handling to skip in live mode. Associated tests are updated for context usage.

Changes

File(s)	Change Summary
internal/orchestrator/committer.go	Refactored getBlockNumbersToCommit to accept context; added getBlockToCommitUntil and fetchBlockDataToCommit; updated logic for commit range and data fetching; modified gap handling for live mode.
internal/orchestrator/committer_test.go	Updated tests to set workMode explicitly and pass context to getBlockNumbersToCommit and related calls.

Sequence Diagram(s)

sequenceDiagram
    participant Caller
    participant Committer
    participant Storage
    participant RPC
    participant Poller

    Caller->>Committer: getSequentialBlockDataToCommit(ctx)
    Committer->>Committer: getBlockNumbersToCommit(ctx)
    Committer->>Committer: getBlockToCommitUntil(ctx, latestCommittedBlockNumber)
    alt Backfill mode
        Committer->>Storage: GetStagingBlockData(blockNumbers)
        alt Data missing
            Committer->>Committer: handleMissingStagingData()
        end
    else Live mode
        Committer->>Poller: PollWithoutSaving(ctx, blockNumbers)
    end
    Committer-->>Caller: BlockData or nil

Loading

Possibly related PRs

• setup workmode determination and update #224: Adds workMode support and dynamic mode switching to the Committer, which is directly built upon by the current PR.
• add possibility to poll without saving #225: Refactors polling logic and introduces PollWithoutSaving, which is now used in the new live mode block data fetch logic.

Suggested reviewers

• nischitpra
• catalyst17
• AmineAfia
  """

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between cc624f5 and abb607d.

📒 Files selected for processing (2)

• internal/orchestrator/committer.go (4 hunks)
• internal/orchestrator/committer_test.go (15 hunks)

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[iuwqyir]

• publish metrics only when actually publishing #227
• poll new blocks in the committer when in live mode #226 👈 (View in Graphite)
• add possibility to poll without saving #225
• setup workmode determination and update #224
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[coderabbitai]

Actionable comments posted: 1

🔭 Outside diff range comments (1)

internal/orchestrator/committer.go (1)

131-143: ⚠️ Potential issue

Guard against endBlock preceding startBlock

When the node is in live mode and the RPC’s latest block equals (or is less than) the last-committed block, endBlock can be ≤ startBlock.
The current code will compute a negative blockCount, panicking when creating the slice.

+ if endBlock.Cmp(startBlock) < 0 {
+     return []*big.Int{}, nil
+ }

Insert this short-circuit right before blockCount is calculated.

🧹 Nitpick comments (2)

internal/orchestrator/committer_test.go (1)

28-29: Avoid reaching into unexported state from tests

Every test tweaks committer.workMode directly. That couples the test-suite to internal implementation details and will break as soon as the field becomes private/removed.
Prefer exercising the public surface instead, e.g.

committer := NewCommitter(mockRPC, mockStorage, WithCommitterWorkModeChan(nil))
setWorkMode(committer, WorkModeBackfill) // helper that sends on the channel

or expose a small setter used only in tests.
This keeps behaviour contracts intact while allowing the struct layout to evolve.

Also applies to: 44-45, 68-69, 95-96, 122-123, 146-147, 173-174, 200-201, 225-226, 251-252, 286-287, 325-326, 351-352, 475-476, 522-523

internal/orchestrator/committer.go (1)

146-162: Consider caching latest RPC block in live mode

getBlockToCommitUntil hits the RPC every commit cycle.
On high-frequency triggers this becomes needless load. A lightweight cache with TTL (e.g. 1-2 s) would cut traffic while staying fresh enough for live committing.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 918459c and cc624f5.

📒 Files selected for processing (2)

• internal/orchestrator/committer.go (4 hunks)
• internal/orchestrator/committer_test.go (15 hunks)

🧰 Additional context used

🧬 Code Graph Analysis (1)

internal/orchestrator/committer_test.go (1)

internal/orchestrator/work_mode_monitor.go (1)

• WorkModeBackfill (21-21)

🔇 Additional comments (1)

internal/orchestrator/committer.go (1)

291-294: Double-check metric semantics when skipping gap handling in live mode

The function increments GapCounter / MissedBlockNumbers even when live mode intentionally ignores gaps.
If gaps are expected in live streaming this will inflate the metrics and trigger alerts.

Verify the desired behaviour; otherwise move the increment after the live-mode early-return.