Repository navigation
fix(consensus): record where timed-out executions stalled (v0.121) - #1812
danieljrc888 wants to merge 3 commits into
Conversation
Backport of #1811 to the line production runs. Exec-timeout receipts are synthetic and carried execution_stats: null, because the outer slot budget cancels the run and discards the GenVM context. On GenVM v0.2.16 the LLM client has no request timeout, so a stalled call only ends at that budget and leaves no trace. Snapshot the stats and progress of the live run just before cancelling it and store them under execution_stats.timeout_progress, plus a warning log. Host calls are tracked in Studio's Host and Context rather than in the vendored origin/ mirror. A failing snapshot leaves execution_stats null and never affects the timeout. Differences from #1811: Host gains bind_context, only storage_read and get_balance are tracked, and backoff is marked at v0.121's retry sleep. Validated with the unit suite on a clean tree: 874 passed, 7 skipped.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
contractSchema.ts imports a type from viem, which only came in through genlayer-js. The explorer has no lockfile, and a fresh install now nests viem under genlayer-js, so next build fails with "Cannot find module 'viem'" and the image build with it. Reproduced from v0.121's package.json without this PR's other changes; with viem declared, next build passes.
…ress The first version could mislead on the question it exists to answer. - awaiting_vm stayed set after the VM closed its connection, so a manager stall in wait_terminal or artifact fetch read like a VM or LLM hang. The new awaiting_manager_result phase covers that window. - A retry only dropped two stats, so a timeout on a later attempt mixed host timings and call counts from the dead attempt. Per-attempt stats and host-call tracking now reset; the errors that explain the retry stay. - mark_run_finished now runs on every exit from the GenVM run, including errors. - Node.timing_callback is a method, so a Node no longer needs the cycle collector to release its deep-copied snapshot. - Timeout receipts leave out delete_genvm_body and delete_genvm_request_error: raw manager errors can name internal hosts, and receipts reach public RPC. The failure flags stay. Tests now drive the real run_genvm_host retry loop and the real host_loop. Mutating the wiring (dropped ctx, no bind_context, no backoff marks, no mark_run_finished, leader snapshot after cancel) previously left the suite green; each mutant now fails. tests/unit: 880 passed, 7 skipped on a clean copy.
|



What
Backport of #1811 to
v0.121, the line production runs.Leader and validator exec-timeout receipts now keep the partial stats of the cancelled run, plus a new
execution_stats.timeout_progress:phase:before_genvm,awaiting_manager,awaiting_executor_connect,awaiting_vm,awaiting_manager_result,host_call:<method>,retry_backofforafter_genvmnode_step: last timing step the node reachedretries,last_host_call,host_idle_ms,host_call_countsEach slot expiry also logs a warning with the transaction hash, address and progress.
Separate commit:
explorer/package.jsonnow declaresviem, whichcontractSchema.tsimports. A fresh install nests it undergenlayer-js, so the explorer image no longer builds. This is unrelated to the diagnostics and would also have blocked anyv0.121release, since the explorer has no lockfile.Why
The hangs were seen on production, not on studio-next. On 2026-10-29, transaction
0xf6812b66779d93f7acd4dea5abe04d1bc351a5278752223c2d18bcc1cb9e5a28had 3 of 5 validators hang for exactly 600s, in a window with 9 such rounds across five contracts. The transaction is found onstudio.genlayer.comand not onstudio-next.genlayer.com.Exec-timeout receipts carry
execution_stats: null, so these hangs leave no record of where they stalled. #1811 fixes that onv0.123-dev, but production would not get it untilv0.123.0. By then GenVM's 300s client timeout should cut these calls short, so the current behavior would never be measured.This is diagnostics only: it does not change timeouts, votes, quorum or replacement behavior, and it does not fix the hang.
Testing done
tests/unit/test_genvm_timeout_progress.pyandtests/unit/consensus/test_validator_exec_timeout.py, matching fix(consensus): record where timed-out executions stalled #1811 adapted tov0.121. They drive the realrun_genvm_hostretry loop and the realhost_loop, and mutating the wiring fails the suite_FakeHostintest_run_genvm_host_state_copy.pygains a no-opbind_contexttest_rpc_endpoint_manager.py, asCLAUDE.mddoes)v0.121has noruff.toml; with ruff's default config there are 8 findings, all on lines this PR does not touch.Not tested: a real hang on GenVM v0.2.16, or the receipt actually reaching Postgres and the explorer. The field uses the existing receipt serialization, so no migration is needed.
Decisions made
Hostgains a smallbind_context, sincev0.121does not pass it the context. Onlystorage_readandget_balanceare tracked, asresolve_call_contract_executordoes not exist here. Backoff is marked atv0.121's retry sleep.awaiting_manager_resultstarts atNOTIFY_FINISHED, so a stall in the manager after the VM returned does not read as a VM hang.delete_genvm_bodyanddelete_genvm_request_error, because raw manager errors can name internal hosts and receipts reach public RPC. The failure flags stay.backend/node/genvm/origin/is untouched, as in fix(consensus): record where timed-out executions stalled #1811.exec_task.cancel(), so it describes the stall and not the cleanup.Risks and rollout
storage_readandget_balancenow pass through a decorator on every transaction. The cost is a few dict operations next to a thread hop, but it is on the hot path.v0.121.26) and promotion, which also moves Docker Hub:latest.Known limits
host_idle_msis the time since a tracked host call started or ended, so untracked calls do not reset it and it also grows during an in-flight call.DELETEfailed can still carrydelete_genvm_request_erroranddelete_genvm_body, as they did before this PR.awaiting_vmdoes not separate LLM or web calls from contract computation, because those calls do not pass through the host. Pinning a stall on the router or a provider still needs router logs.CONSENSUS_VALIDATOR_QUORUM_REACHED) still have no stats, so hangs cut short by quorum stay hidden.Checks
Reviewing tips
The logic matches #1811, so the
v0.121-specific parts areHost.bind_contextand the retry sleep inrun_genvm_host, both inbackend/node/genvm/base.py.