Skip to content

fix(recovery): late verify timeouts fail the candidate, not the run - #36

Merged
giveen merged 1 commit into
mainfrom
fix/late-verify-timeout-recovery
Oct 2, 2026
Merged

giveen merged 1 commit into
mainfrom
fix/late-verify-timeout-recovery

Conversation

@giveen

@giveen giveen commented Oct 2, 2026

Copy link
Copy Markdown
Owner

From the live campaign: hyper_connection died as failed ... ctest timed out after 60s — no RunFinished in its journal, 0 LLM calls recorded — after doing real search work (rich correctness-failure detail and all).

Fix 1: demote late verify errors to gate outcomes

finalize's winner re-verify and both shape-gate verifies used ?, so one hung test unwound the whole target (cursor advanced, journal unclosed, checkpoint kept):

  • finalize: a re-verify error now fails Gate 1–2 instead — normal flow continues to a fallback with closed journal, cleared checkpoint, and a specific cause (winner re-verify timed out: … vs generic rebuild failure).
  • shape gate: a verify error restores the patched test file and Skips the gate ("could not determine" ≠ "candidate wrong"). Only an actual reported failure can fail it.
  • New exec::is_timeout helper (all tool timeouts bail with "… timed out …").

Fix 2: recover usage from dead runs

The campaign Err arm now replays the failed run's journal and delta-accounts (calls, tokens) exactly like account() — so budgets stay honest and a resumed-then-failed target is never double-counted. Previously the usage vanished (observed: real spend recorded as 0).

Recovery semantics (unchanged, verified)

  • Journal always closes (RunFinished), checkpoints clear on the fallback path, cursor advances, state saves after every target — resume behaves exactly as before.
  • failed still means "genuine run error" (discover/baseline/etc.); timeouts now surface as fallback with cause instead.
  • Delta accounting mirrors account(), so a failed target that is later manually rerun can't double-count.

Validation

  • cargo build --release clean; cargo test 121 passed (2 new: is_timeout_matches_tool_timeouts, journal_usage_sums_journaled_calls); pytest 21 passed.
  • Safe to land mid-campaign: only affects future targets/runs (the running campaign keeps its old image).

A hyper_connection target died as "failed ... ctest timed out after 60s"
with no RunFinished in its journal and 0 LLM calls recorded, after doing
real search work. Two gaps:

- finalize's winner re-verify and the shape gate's verifies used `?`, so a
  hung test unwound the whole target (cursor advanced, journal unclosed,
  checkpoint kept). Now a verify error in finalize demotes to gate failure
  (fallback with a closed journal, specific "re-verify timed out" cause);
  shape-gate verify errors restore the test file and Skip the gate (only an
  actual reported failure can fail it).
- campaign's Err arm never accounted usage: recover (calls, tokens) from the
  dead run's journal (delta-accounted, so resumes never double-count) and
  keep budgets honest.

Adds exec::is_timeout plus unit tests for it and journal_usage.
@giveen
giveen merged commit fb4efad into main Oct 2, 2026
2 checks passed
@giveen
giveen deleted the fix/late-verify-timeout-recovery branch October 2, 2026 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant