Repository navigation
Verification loop: agents must prove their work before a thread goes green #296
Description
The strongest cross-vertical finding of round 5. Sources: TestSprite (verifier-as-MCP, agent-callable), XBOW (validator fleet must produce a working proof before a finding is reported; zero false positives across 1,060 findings), Cleric (Verify stage against the live environment), Dagger (declared output contracts: the run isn't done until the required artifact exists), Anthropic's harness post (agents mark features done without end-to-end testing unless forced).
Give threads a verification stage distinct from review: the agent (or a fresh-context verifier thread, retired after each mission to avoid bias) must run the tests, build the app, click through the flow in the embedded browser (#155), and attach the evidence as a run artifact (#244). Failure hands the fixer a structured bundle: failing step, screenshots/logs, root-cause hypothesis, pinned to a checkpoint. Thread goals (#238) declare required outputs up front; 'done' requires the proof, not the claim. Verified-by-execution is a much stronger trust signal than LLM-judge approval (#256) and should feed the PR autopilot (#255).
Activity
currentbits commentedon Aug 17, 2026
User-pain research (round 6) makes this the #1 priority corroboration: 'agent says tests passed, it did not run them' is called the single most-reported failure of autonomous agents in 2026. Key detail from postmortems: the transcript is structurally unreliable as an audit trail — verification must check ground truth (re-run tests, diff the filesystem, query git state), never the agent's self-report. Also the correlated-reviewer failure is getting public attention (AI reviewing AI trained on the same patterns); that is why the verifier must be execution-based, and review cross-model (#259).
currentbits commentedon Aug 17, 2026
Implemented on
coder/verification-loop-agents-must-prove-thei-3f9cdf.Shape. A thread carries a
verifyCommand(shell string). When set, a run that would landdoneinstead runs that command in the thread's worktree (project path when it has none), after the turn checkpoint so the evidence pins to a sha that exists. Exit 0 → the thread goes green and theVerifyResultis stored as evidence. Non-zero → it does not go green; the agent is handed a structured bundle (command, exit code or timeout, checkpoint sha, attempt N of M, log tail, "state a root-cause hypothesis, fix the cause, re-run it yourself") as a new turn, up toMAX_FIX_ATTEMPTS(2), after which the thread landsfailedwith alastErrornaming the command.Deliberately no new
ThreadStatus. A verifying thread staysworking— the ask is a gate, not a fourth colour, and a new status would have meant touching every switch in the sidebar, settle and waiting logic.electron/verify.js— engine: spawn under/bin/sh -c, detached, 10-min timeout withkillTree, capped log tail,buildFixPrompt.electron/runner.js— the gate inafterSuccessfulTurn, the single choke point every provider (generic/claude/codex/kimi/opencode/workflow) lands through. Fix turns are delivered the same way orchestrator wake-ups are.electron/store.js/services.js/ipc.js— persistence,threads:setVerifyCommand,threads:runVerify(manual "Verify now").Tests: core 38/38, renderer 795/795, memory 122/122, electron 918/919 — the one failure is the pre-existing
web-integrationreconnect test, unchanged from main.Not covered here, since both depend on unlanded work: embedded-browser click-through (#155) and screenshots/videos as run artifacts (#244). The evidence today is the command's log tail on the thread. When #244 lands an artifact store,
VerifyResultis where the artifact ids attach. Thread goals declaring required outputs up front (#238) would replace the single command with a contract; the gate's shape does not change.Merge pull request #331 from currentbits/coder/verification-loop-agen…
Merge origin/main into the observability branch (#280)
currentbits commentedon Aug 17, 2026
Research note (GitHub agentic autofix): the cheapest strong gate — after an agent claims a fix, re-run the ORIGINAL failing check (test, linter, scanner) and only surface the thread for review when the check actually closes. Validate-by-rerun against the original failure, not a generic test pass.
currentbits commentedon Aug 17, 2026
Research note (review pedagogy round): the single highest-leverage heuristic from GitHub's 2026 playbook — 'a test that fails on pre-change behavior'. Mechanical for an orchestrator: run the agent's new test against the PRE-CHANGE worktree and show green/red. A test that passes on both sides proves nothing.
currentbits commentedon Aug 18, 2026
From the Codex doc-mine: Stop/SubagentStop continuation hooks — a hook returning decision:block turns its reason into a new user prompt, which is an EXTERNALIZED verification loop ('run the tests again') the orchestrator doesn't have to own. SubagentStop also receives last_assistant_message for cheap review gates. A clean way to make verification userland-configurable.
currentbits commentedon Aug 18, 2026
Round 20: Prime Agent (open-sourced 5 Aug, docs) expresses this issue as a spawn flag:
Two rules worth stealing even if we never ship their adapter (#464):
Solenta should be able to attach a gate command to any thread, any provider, and keep the thread in an explicit verifying/blocked state until the gate passes or the turn budget is hit. Do not special-case Prime.
currentbits commentedon Aug 18, 2026
Round 24: danluu's "The benchmarkpocalypse" (2026-08-18, front page of HN) is required reading for this issue. He ran a SOTA agent in a loop for a month optimizing a regex engine against the rebar suite. Findings that bear directly on how Solenta verifies agent work:
None of this argues against the verification loop; it argues the loop must be adversarial by design.
currentbits commentedon Aug 18, 2026
Verified done (with deliberate design deviations).
electron/verify.jsgate runs the thread'sverifyCommandafter each successful turn; thread only settles done on proof (runner.js:1090-1106). Failure builds a structured fix prompt (command, exit cause, checkpoint sha, capped log, root-cause-hypothesis request) with up to 2 fix attempts before failing. Evidence persisted asthread.verifyand shown in VerifyCard. Deviations: main process runs the command directly instead of a fresh-context verifier thread (stronger — no agent self-report), bundle is a prompt not a JSON blob, no screenshots/click-through beyond what the verify command itself does, and the gate is opt-in per thread. Tests: verify-gate/verify/verify-services/verifyCard.