Skip to content

fix(consensus): record where timed-out executions stalled (v0.121) - #1812

Open
danieljrc888 wants to merge 3 commits into
v0.121from
claude/genlayer-validator-timeout-v0121
Open

danieljrc888 wants to merge 3 commits into
v0.121from
claude/genlayer-validator-timeout-v0121

Conversation

@danieljrc888

@danieljrc888 danieljrc888 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What

Backport of #1811 to v0.121, the line production runs.

  • Leader and validator exec-timeout receipts now keep the partial stats of the cancelled run, plus a new execution_stats.timeout_progress:

    • phase: before_genvm, awaiting_manager, awaiting_executor_connect, awaiting_vm, awaiting_manager_result, host_call:<method>, retry_backoff or after_genvm
    • node_step: last timing step the node reached
    • retries, last_host_call, host_idle_ms, host_call_counts
  • Each slot expiry also logs a warning with the transaction hash, address and progress.

  • Separate commit: explorer/package.json now declares viem, which contractSchema.ts imports. A fresh install nests it under genlayer-js, so the explorer image no longer builds. This is unrelated to the diagnostics and would also have blocked any v0.121 release, since the explorer has no lockfile.

Why

The hangs were seen on production, not on studio-next. On 2026-10-29, transaction 0xf6812b66779d93f7acd4dea5abe04d1bc351a5278752223c2d18bcc1cb9e5a28 had 3 of 5 validators hang for exactly 600s, in a window with 9 such rounds across five contracts. The transaction is found on studio.genlayer.com and not on studio-next.genlayer.com.

Exec-timeout receipts carry execution_stats: null, so these hangs leave no record of where they stalled. #1811 fixes that on v0.123-dev, but production would not get it until v0.123.0. By then GenVM's 300s client timeout should cut these calls short, so the current behavior would never be measured.

This is diagnostics only: it does not change timeouts, votes, quorum or replacement behavior, and it does not fix the hang.

Testing done

  • New tests in tests/unit/test_genvm_timeout_progress.py and tests/unit/consensus/test_validator_exec_timeout.py, matching fix(consensus): record where timed-out executions stalled #1811 adapted to v0.121. They drive the real run_genvm_host retry loop and the real host_loop, and mutating the wiring fails the suite
  • _FakeHost in test_run_genvm_host_state_copy.py gains a no-op bind_context
  • Unit suite on a clean tree: 880 passed, 7 skipped, against 866 and 7 without this change (excluding test_rpc_endpoint_manager.py, as CLAUDE.md does)
  • Black passes on the changed files. v0.121 has no ruff.toml; with ruff's default config there are 8 findings, all on lines this PR does not touch.

Not tested: a real hang on GenVM v0.2.16, or the receipt actually reaching Postgres and the explorer. The field uses the existing receipt serialization, so no migration is needed.

Decisions made

  • Differences from fix(consensus): record where timed-out executions stalled #1811: Host gains a small bind_context, since v0.121 does not pass it the context. Only storage_read and get_balance are tracked, as resolve_call_contract_executor does not exist here. Backoff is marked at v0.121's retry sleep.
  • awaiting_manager_result starts at NOTIFY_FINISHED, so a stall in the manager after the VM returned does not read as a VM hang.
  • A retry resets per-attempt stats and host-call tracking, so a timeout on a later attempt does not mix in the dead attempt. The errors that explain the retry stay.
  • Timeout receipts leave out delete_genvm_body and delete_genvm_request_error, because raw manager errors can name internal hosts and receipts reach public RPC. The failure flags stay.
  • backend/node/genvm/origin/ is untouched, as in fix(consensus): record where timed-out executions stalled #1811.
  • The snapshot is taken before exec_task.cancel(), so it describes the stall and not the cleanup.
  • Snapshot failures are swallowed and logged, because diagnostics must not turn a timeout into a consensus failure.

Risks and rollout

  • storage_read and get_balance now pass through a decorator on every transaction. The cost is a few dict operations next to a thread hop, but it is on the hot path.
  • Merging does not deploy this. It needs a stable release (likely v0.121.26) and promotion, which also moves Docker Hub :latest.
  • Rollback is reverting this commit. Receipts already written keep the extra field, and existing readers tolerate it.

Known limits

  • host_idle_ms is the time since a tracked host call started or ended, so untracked calls do not reset it and it also grows during an in-flight call.
  • Normal receipts of a run whose last DELETE failed can still carry delete_genvm_request_error and delete_genvm_body, as they did before this PR.
  • awaiting_vm does not separate LLM or web calls from contract computation, because those calls do not pass through the host. Pinning a stall on the router or a provider still needs router logs.
  • Validators cancelled by quorum (CONSENSUS_VALIDATOR_QUORUM_REACHED) still have no stats, so hangs cut short by quorum stay hidden.

Checks

  • I have tested this code
  • I have reviewed my own PR
  • I have created an issue for this PR
  • I have set a descriptive PR title compliant with conventional commits

Reviewing tips

The logic matches #1811, so the v0.121-specific parts are Host.bind_context and the retry sleep in run_genvm_host, both in backend/node/genvm/base.py.

Backport of #1811 to the line production runs. Exec-timeout receipts are
synthetic and carried execution_stats: null, because the outer slot budget
cancels the run and discards the GenVM context. On GenVM v0.2.16 the LLM client
has no request timeout, so a stalled call only ends at that budget and leaves
no trace.

Snapshot the stats and progress of the live run just before cancelling it and
store them under execution_stats.timeout_progress, plus a warning log. Host
calls are tracked in Studio's Host and Context rather than in the vendored
origin/ mirror. A failing snapshot leaves execution_stats null and never
affects the timeout.

Differences from #1811: Host gains bind_context, only storage_read and
get_balance are tracked, and backoff is marked at v0.121's retry sleep.

Validated with the unit suite on a clean tree: 874 passed, 7 skipped.
@danieljrc888 danieljrc888 self-assigned this Nov 6, 2026
@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8e903b76-9348-4602-9082-aa9b84b18a8a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-06T16:31:09.891836Z 04a4fea New commits
🔒 Security Review ✅ Completed 2026-11-06T04:17:20.504590Z afea5fe PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

contractSchema.ts imports a type from viem, which only came in through
genlayer-js. The explorer has no lockfile, and a fresh install now nests viem
under genlayer-js, so next build fails with "Cannot find module 'viem'" and
the image build with it. Reproduced from v0.121's package.json without this
PR's other changes; with viem declared, next build passes.
…ress

The first version could mislead on the question it exists to answer.

- awaiting_vm stayed set after the VM closed its connection, so a manager stall in wait_terminal or artifact fetch read like a VM or LLM hang. The new awaiting_manager_result phase covers that window.
- A retry only dropped two stats, so a timeout on a later attempt mixed host timings and call counts from the dead attempt. Per-attempt stats and host-call tracking now reset; the errors that explain the retry stay.
- mark_run_finished now runs on every exit from the GenVM run, including errors.
- Node.timing_callback is a method, so a Node no longer needs the cycle collector to release its deep-copied snapshot.
- Timeout receipts leave out delete_genvm_body and delete_genvm_request_error: raw manager errors can name internal hosts, and receipts reach public RPC. The failure flags stay.

Tests now drive the real run_genvm_host retry loop and the real host_loop. Mutating the wiring (dropped ctx, no bind_context, no backoff marks, no mark_run_finished, leader snapshot after cancel) previously left the suite green; each mutant now fails.

tests/unit: 880 passed, 7 skipped on a clean copy.
@sonarqubecloud

sonarqubecloud Bot commented Nov 6, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant