Skip to content

fix(consensus): record where timed-out executions stalled - #1811

Open
danieljrc888 wants to merge 2 commits into
v0.123-devfrom
claude/genlayer-validator-timeout-7a640c
Open

danieljrc888 wants to merge 2 commits into
v0.123-devfrom
claude/genlayer-validator-timeout-7a640c

Conversation

@danieljrc888

@danieljrc888 danieljrc888 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Leader and validator exec-timeout receipts now keep the partial stats of the cancelled run, plus a new execution_stats.timeout_progress:
    • phase: before_genvm, awaiting_manager, awaiting_executor_connect, awaiting_vm, awaiting_manager_result, host_call:<method>, retry_backoff or after_genvm
    • node_step: last timing step the node reached
    • retries, last_host_call, host_idle_ms, host_call_counts
  • Each slot expiry also logs a warning with the transaction hash, address and progress.

Why

Exec-timeout receipts are synthetic and carry execution_stats: null. The outer slot budget fires and cancels the run, which discards the GenVM context, so a hang leaves no record of where it stalled.

We hit this investigating a Studionet transaction on 2026-10-29 (0xf6812b66779d93f7acd4dea5abe04d1bc351a5278752223c2d18bcc1cb9e5a28), where 3 of 5 validators hung for exactly 600s and the round ended in CONSENSUS_VALIDATOR_EXEC_TIMEOUT with no majority. Receipts could not tell a stalled LLM call from a stuck storage read, so the stalled component is still unproven.

This is diagnostics only: it does not change timeouts, votes, quorum or replacement behavior, and it does not fix the hang.

Testing done

  • New tests in tests/unit/test_genvm_timeout_progress.py (phase transitions, retry backoff, a storage read stuck in flight, a node whose run never returns)
  • New tests in tests/unit/consensus/test_validator_exec_timeout.py (leader and validator receipts keep the stats, the snapshot is taken before cancelling, a failing snapshot still yields a normal timeout)
  • Tests drive the real run_genvm_host retry loop and the real host_loop. Mutating the wiring (dropped ctx, no bind_context, no backoff marks, no mark_run_finished, leader snapshot after cancel) fails the suite
  • Unit suite on a clean tree: 1795 passed, 7 skipped (excluding test_rpc_endpoint_manager.py, as CLAUDE.md does). Black and ruff pass on the changed files.

Not tested: a real GenVM or router hang, or the receipt actually reaching Postgres and the explorer. The field uses the existing receipt serialization, so no migration is needed. To check it end to end, set CONSENSUS_VALIDATOR_EXEC_TIMEOUT_SECONDS very low on a local Studio.

Decisions made

  • backend/node/genvm/origin/ is untouched. It mirrors genvm-manager and a re-vendor would overwrite local edits, so host calls are tracked by a decorator on Studio's Host handlers and by genvm.base.Context.
  • The snapshot is taken before exec_task.cancel(), so it describes the stall and not the cleanup.
  • Snapshot failures are swallowed and logged, because diagnostics must not turn a timeout into a consensus failure.
  • awaiting_manager_result means every connection the VM opened has finished, nested executors included, so a stall in wait_terminal or artifact fetch does not read as a VM hang. It is derived from stats the host loop already reports.
  • A retry resets per-attempt stats and host-call tracking, so a timeout on a later attempt does not mix in the dead attempt. The errors that explain the retry stay.
  • Only the handlers that can block are tracked: storage_read, resolve_call_contract_executor and get_balance_gen_wei.

Known limits

  • host_idle_ms is the time since a tracked host call started or ended, so untracked calls do not reset it and it also grows during an in-flight call.
  • awaiting_vm does not separate LLM or web calls from contract computation, because those calls do not pass through the host. Pinning a stall on the router or a provider still needs the run's genvm_log or router logs.
  • genvm_id and boot_id are not included, since they live inside origin/run_genvm.
  • Validators cancelled by quorum (CONSENSUS_VALIDATOR_QUORUM_REACHED) still have no stats, so hangs cut short by quorum stay hidden.
  • This ships on the v0.123 line. Production on v0.121 does not get it until it moves to v0.123.

Checks

  • I have tested this code
  • I have reviewed my own PR
  • I have created an issue for this PR
  • I have set a descriptive PR title compliant with conventional commits

Reviewing tips

Start with _partial_execution_stats in backend/consensus/base.py, then Context.progress in backend/node/genvm/base.py.

Leader and validator exec-timeout receipts are synthetic and carried
execution_stats: null. The outer slot budget cancels the run and discards
the GenVM context, so a hang leaves no trace of the phase it stalled in.

Snapshot the stats and progress of the live run just before cancelling it
and store them on the timeout receipt, under execution_stats.timeout_progress,
plus a warning log. Host calls are tracked in Studio's Host and Context rather
than in the vendored origin/ mirror, which a re-vendor would overwrite.
A failing snapshot leaves execution_stats null and never affects the timeout.

Validated with the unit suite on a clean tree: 1789 passed, 7 skipped.
@danieljrc888 danieljrc888 self-assigned this Nov 5, 2026
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f9658869-1dd1-4ba4-ad56-595f9f4029a4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-06T16:31:16.158800Z caef6c1 New commits
🔒 Security Review ✅ Completed 2026-11-05T23:02:13.384957Z c6d7ba9 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…ress

The first version could mislead on the question it exists to answer.

- awaiting_vm stayed set after the VM closed its connection, so a manager stall in wait_terminal or artifact fetch read like a VM or LLM hang. The new awaiting_manager_result phase covers that window.
- A retry only dropped two stats, so a timeout on a later attempt mixed host timings and call counts from the dead attempt. Per-attempt stats and host-call tracking now reset; the errors that explain the retry stay.
- mark_run_finished now runs on every exit from the GenVM run, including errors.
- Node.timing_callback is a method, so a Node no longer needs the cycle collector to release its deep-copied snapshot.

The VM counts as gone once every connection it opened has finished, nested executors included, derived from the stats the host loop already reports so origin stays untouched.

Tests now drive the real run_genvm_host retry loop and the real host_loop. Mutating the wiring (dropped ctx, no bind_context, no backoff marks, no mark_run_finished, leader snapshot after cancel) previously left the suite green; each mutant now fails.

tests/unit: 1795 passed, 7 skipped on a clean copy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant