Repository navigation
fix(consensus): record where timed-out executions stalled - #1811
danieljrc888 wants to merge 2 commits into
Conversation
Leader and validator exec-timeout receipts are synthetic and carried execution_stats: null. The outer slot budget cancels the run and discards the GenVM context, so a hang leaves no trace of the phase it stalled in. Snapshot the stats and progress of the live run just before cancelling it and store them on the timeout receipt, under execution_stats.timeout_progress, plus a warning log. Host calls are tracked in Studio's Host and Context rather than in the vendored origin/ mirror, which a re-vendor would overwrite. A failing snapshot leaves execution_stats null and never affects the timeout. Validated with the unit suite on a clean tree: 1789 passed, 7 skipped.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…ress The first version could mislead on the question it exists to answer. - awaiting_vm stayed set after the VM closed its connection, so a manager stall in wait_terminal or artifact fetch read like a VM or LLM hang. The new awaiting_manager_result phase covers that window. - A retry only dropped two stats, so a timeout on a later attempt mixed host timings and call counts from the dead attempt. Per-attempt stats and host-call tracking now reset; the errors that explain the retry stay. - mark_run_finished now runs on every exit from the GenVM run, including errors. - Node.timing_callback is a method, so a Node no longer needs the cycle collector to release its deep-copied snapshot. The VM counts as gone once every connection it opened has finished, nested executors included, derived from the stats the host loop already reports so origin stays untouched. Tests now drive the real run_genvm_host retry loop and the real host_loop. Mutating the wiring (dropped ctx, no bind_context, no backoff marks, no mark_run_finished, leader snapshot after cancel) previously left the suite green; each mutant now fails. tests/unit: 1795 passed, 7 skipped on a clean copy.
What
execution_stats.timeout_progress:phase:before_genvm,awaiting_manager,awaiting_executor_connect,awaiting_vm,awaiting_manager_result,host_call:<method>,retry_backofforafter_genvmnode_step: last timing step the node reachedretries,last_host_call,host_idle_ms,host_call_countsWhy
Exec-timeout receipts are synthetic and carry
execution_stats: null. The outer slot budget fires and cancels the run, which discards the GenVM context, so a hang leaves no record of where it stalled.We hit this investigating a Studionet transaction on 2026-10-29 (
0xf6812b66779d93f7acd4dea5abe04d1bc351a5278752223c2d18bcc1cb9e5a28), where 3 of 5 validators hung for exactly 600s and the round ended inCONSENSUS_VALIDATOR_EXEC_TIMEOUTwith no majority. Receipts could not tell a stalled LLM call from a stuck storage read, so the stalled component is still unproven.This is diagnostics only: it does not change timeouts, votes, quorum or replacement behavior, and it does not fix the hang.
Testing done
tests/unit/test_genvm_timeout_progress.py(phase transitions, retry backoff, a storage read stuck in flight, a node whose run never returns)tests/unit/consensus/test_validator_exec_timeout.py(leader and validator receipts keep the stats, the snapshot is taken before cancelling, a failing snapshot still yields a normal timeout)run_genvm_hostretry loop and the realhost_loop. Mutating the wiring (droppedctx, nobind_context, no backoff marks, nomark_run_finished, leader snapshot after cancel) fails the suitetest_rpc_endpoint_manager.py, asCLAUDE.mddoes). Black and ruff pass on the changed files.Not tested: a real GenVM or router hang, or the receipt actually reaching Postgres and the explorer. The field uses the existing receipt serialization, so no migration is needed. To check it end to end, set
CONSENSUS_VALIDATOR_EXEC_TIMEOUT_SECONDSvery low on a local Studio.Decisions made
backend/node/genvm/origin/is untouched. It mirrors genvm-manager and a re-vendor would overwrite local edits, so host calls are tracked by a decorator on Studio'sHosthandlers and bygenvm.base.Context.exec_task.cancel(), so it describes the stall and not the cleanup.awaiting_manager_resultmeans every connection the VM opened has finished, nested executors included, so a stall inwait_terminalor artifact fetch does not read as a VM hang. It is derived from stats the host loop already reports.storage_read,resolve_call_contract_executorandget_balance_gen_wei.Known limits
host_idle_msis the time since a tracked host call started or ended, so untracked calls do not reset it and it also grows during an in-flight call.awaiting_vmdoes not separate LLM or web calls from contract computation, because those calls do not pass through the host. Pinning a stall on the router or a provider still needs the run'sgenvm_logor router logs.genvm_idandboot_idare not included, since they live insideorigin/run_genvm.CONSENSUS_VALIDATOR_QUORUM_REACHED) still have no stats, so hangs cut short by quorum stay hidden.Checks
Reviewing tips
Start with
_partial_execution_statsinbackend/consensus/base.py, thenContext.progressinbackend/node/genvm/base.py.