Skip to content

ISS-14158: Define when to capture and guard screenshot runs - #435

Merged
shafty023 merged 3 commits into
mainfrom
vibe/iss-14158-capture-budget
Nov 16, 2026
Merged

shafty023 merged 3 commits into
mainfrom
vibe/iss-14158-capture-budget

Conversation

@shafty023

@shafty023 shafty023 commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • ISS-14158: vibe and fleet-orchestrator now say when to capture screenshots and when one visual QA pass is enough. A new capture guard refuses repeated main baselines and oversized capture sweeps, and every ETA is computed from the remaining queue. See plugins/vibe/skills/vibe/references/quality-loop-visual-qa.md:83 and plugins/vibe/skills/vibe/scripts/capture-guard.mjs:71.

  • AC-1, when to capture: references/quality-loop-visual-qa.md gains the block when-to-capture with the approved criterion 1 text. By default, one visual-qa reviewer reads the diff plus the stories. Captures happen only for a shared component's pixels, for a criterion or ruling that asks for visual proof, or for a finding the reviewer cannot settle. quality-loop.md and the fleet QA gate point at that block. The verify worker captures only in those cases, or when the session has a reference image for the screen. After fixes, handoff reruns visual QA as one pass scoped to the changes since the last reviewed capture.

  • AC-2, one main baseline: the fleet QA gate and the verify worker compare against the ticket's one main baseline, never a new one because main moved, and a difference outside the ticket's diff is main's. The guard refuses any second main baseline for the ticket.

  • AC-3 and AC-4, the capture guard. New skills/vibe/scripts/capture-guard.mjs, with subcommands begin, end, status and facts. begin refuses (exit 79):

    • a main commit already captured for the ticket;
    • a second main baseline;
    • more than 240 shots (pages times states);
    • more than about 20 minutes at the higher of 5 s per shot and the slowest rate the ticket has measured.

    It only refuses; nothing requires a capture. end records ok or failed once, with the measured seconds. A running entry holds its place while its owner lives, or for its own estimate (never under 20 minutes). The ledger is the fleet program's captures directory when this clone is a writer worktree on exactly one program's board, otherwise the clone's own ledger. The fleet ledger and the clone's ledger mirror each other.

  • Overrides, fixed shape: a refusal is lifted only by a Ruling line in the program rulings, written as - Ruling YYYY-MM-DD (<source>): "<the operator's exact words>" <TICKET> capture-guard override: <kind>.

    • The decision text after the source parenthesis must begin with the line's only double-quoted span (no curly quotes), and that span must equal the spec's quote.
    • The ticket (exact slug) and the marker must come after that span.
    • baseline lifts only the two baseline refusals; budget lifts only the shot and minute refusals.
    • Lines are split exactly as the shared Ruling matcher splits them. references/state.md tells recorders this shape.
  • Fleet slot: fleet.sh ... slot heavy <owner> <dir> --capture <spec.json> <command...>.

    • It runs the guard with the program's rulings.md before the load check and before any slot is taken.
    • It times the command inside the slot, so the recorded seconds exclude the queue wait.
    • A TERM, INT or HUP stops the command and frees the slot, locally and on a remote host, and the entry records the real exit.
    • A slot without --capture is unchanged.
  • AC-5, ETAs: the fleet reporting rule and the Remaining work report template require every ETA for queued work to be computed from the remaining queue count, and restated the moment a step adds more than 30 minutes. See plugins/vibe/skills/fleet-orchestrator/SKILL.md:212 and plugins/vibe/skills/fleet-orchestrator/references/templates.md:126.

  • AC-6: each guard rule has a test that fails on base 8a858bb or under its mutation (see Test plan). See plugins/vibe/skills/vibe/scripts/capture-guard.test.mjs:80 and plugins/vibe/skills/fleet-orchestrator/scripts/fleet.test.mjs:663.

  • Versions: vibe 1.1.91. vibe-core is unchanged. Instruction words: vibe is about 496 words longer than in 1.1.86 (origin/main 8a858bb), as the CHANGELOG states.

Affected plugins and files

  • vibe:
    • New skills/vibe/scripts/capture-guard.mjs and its test.
    • skills/fleet-orchestrator/scripts/fleet.sh and fleet.test.mjs.
    • skills/fleet-orchestrator/SKILL.md, references/fleet-qa-gate.md, references/state.md and references/templates.md.
    • skills/vibe/references/quality-loop.md and quality-loop-visual-qa.md.
    • skills/handoff/SKILL.md and references/handoff-ui.md.
    • agents/vibe-verify-worker.md.
    • New tools/python/test_vibe_iss_14158_capture_contract.py.
    • The prompt-split and 1079 contracts, extended or re-pointed.
    • Release pins in test_pr_takeover_skill_contract.py and test_vibe_ws8_load_contract.py.
    • Both plugin.json files and the README version line.
  • tools/vibe-bench/src/reply-checker.test.ts: version pin.
  • CHANGELOG.md: vibe 1.1.91.

Rulings applied

  • Criterion 1 text, approved by the operator as written. It is placed verbatim in one canonical block, and nothing in it makes captures a requirement.
  • Q-14158-1: one canonical copy of criterion 1 in the vibe visual QA reference. The fleet QA gate gets a pointer line and no copy.
  • Q-14158-2: keep both limits, 240 shots and about 20 minutes, whichever hits first, with a 5 s per shot floor. The six refusal lines and their shared ending were approved as written.
  • Q-14158-3: offset the added text by collapsing duplicated sentences into their canonical owners, and state the true word count rather than force it to zero.
  • Q-14158-4: an override must be about overriding the guard. Later superseded by Q-14158-5.
  • Q-14158-5: stop parsing English. An override is an explicit capture-guard override: baseline or budget marker, and each kind lifts only its own refusals.
  • Q-14158-6: the fixed override shape described above. The approved state.md sentence documents it.
  • Q-14158-7: rerun the two refused pre-PR lanes once, in their CI shape, on a heavy slot under the load gate.
  • Q-14158-8: no further node-tests runs. The timing failures outside this diff are environmental and are covered by the gate override below, citing ISS-14184.
  • Also approved:
    • the wording that tells callers to run end after a capture, and the verify worker's capture rule including the reference image case;
    • the state.md sentence saying a marker is added only when the operator's words ask for that override;
    • accepting the direct-caller --rulings limit below.

Accepted limits

  • begin --facts trusts the facts it is given. Remote fleet captures read them on the host that holds the worktree.
  • A direct, non-fleet caller can name any file through --rulings. The guard cannot verify that a local file is the program rulings or the ticket body. fleet.sh always passes the program rulings.
  • The override fails closed in the fixed shape. Operator words that themselves contain double quotes never match, and neither does any other line shape.
  • A remote fleet capture is recorded only in the program ledger. The orchestrator's status --ledger <orchDir>/captures does not list direct captures, although the guard refuses across both ledgers for local fleet captures.

Overlap

Follow-up outside this repository

  • symphony-alpha: pnpm vibe shots and the proof scripts that call it should run the capture guard's begin before capturing and end after, so the guard refuses at the capture command itself.

Known main failures

  • CI job "Vibe Eval T0" (npm test in tools/vibe-bench): tools/vibe-bench/src/phase3.test.ts fails on an expired quarantine deadline. It fails the same way on main 8a858bb; this diff does not touch that file. The recorded lane identity of that job:

    {"job":{"name":"Vibe Eval T0","runs-on":"ubuntu-latest","steps":[{"uses":"actions/checkout@v4"},{"uses":"actions/setup-node@v4","with":{"node-version":"24"}},{"uses":"astral-sh/setup-uv@v6","with":{"python-version":"3.13"}},{"run":"uv sync --frozen --group dev"},{"name":"Install deterministic benchmark tooling","run":"npm ci","working-directory":"tools/vibe-bench"},{"name":"Type-check benchmark contracts","run":"npm run typecheck","working-directory":"tools/vibe-bench"},{"name":"Validate local plugin structure","run":"npm run validate:plugins","working-directory":"tools/vibe-bench"},{"name":"Run deterministic benchmark contracts","run":"npm test","working-directory":"tools/vibe-bench"},{"name":"Run deterministic Codex mirror contracts","run":"uv run pytest plugins/vibe/evals/codex/test_run.py"}],"timeout-minutes":10},"matrix":{},"version":1,"workflow":{"concurrency":{"cancel-in-progress":true,"group":"ci-${{ github.event.pull_request.number || github.ref }}"},"name":"CI","on":{"pull_request":null,"push":{"branches":["main"]}}}}

Test plan

Final candidate tree: tested-head treeSha256 4d21d5084cd477d7f7ac87483cc2ecea09d7d2369d5a7767a4087fe603d1f66b.

  • Targeted tests:
    • On the final tree, on a second machine's heavy slot: node --test plugins/vibe/skills/vibe/scripts/capture-guard.test.mjs plugins/vibe/skills/fleet-orchestrator/scripts/fleet.test.mjs passed 64/64.
    • On the preceding candidate trees of this branch, where only the guard and its tests changed after: node --test plugins/vibe/skills/fleet-orchestrator/scripts/generic.test.mjs plugins/vibe/skills/fleet-orchestrator/scripts/slot-lock.test.mjs plugins/vibe/skills/vibe/scripts/skill-split.test.mjs passed.
    • uv run pytest -q on these contracts passed (69 passed): plugins/vibe/tools/python/test_vibe_iss_14158_capture_contract.py, test_vibe_prompt_split_contract.py, test_vibe_1074_lessons_contract.py, test_vibe_1078_lessons_contract.py, test_vibe_1079_lessons_contract.py, test_vibe_110_single_use_version_check_contract.py, test_vibe_connection_restore_contract.py, test_vibe_stranded_writer_recovery_contract.py and test_vibe_no_person_names_contract.py.
  • Second commit (tests only): fleet.test.mjs gains two cases, a missing or unreadable --capture spec is a usage error before any load check, slot or command, and --capture with --containerized-e2e-build is accepted in either order. node --test plugins/vibe/skills/fleet-orchestrator/scripts/fleet.test.mjs passed 44/44 on a heavy slot; moving the spec checks after the load check fails the first case, and accepting the flags in one order only fails the second.
  • Mutation proofs. Each was applied, tested and restored inside one heavy-slot run, with a byte-for-byte compare after:
    • Every new test fails on base 8a858bb.
    • Every guard rule fails its test when reverted and passes when restored. The rules covered:
      • the main baseline rules;
      • the shot and minute budgets and the slowest measured rate;
      • the running-entry time bound;
      • end refusing finished entries;
      • recording seconds outside the queue wait;
      • signal handling, local and remote, including the stop before a remote slot starts;
      • recording the real remote exit;
      • the guard running before the load check;
      • the default and mirrored ledgers, and the canonical fleet home;
      • the override markers and kind scoping;
      • the fixed shape: the span first, the whole-span match, a single span, no curly quotes, the decision after the source parenthesis, and the shared line split.
    • On the final tree (second machine): splitting on \n only fails the line-split test, and restoring it passes.
  • node plugins/vibe/skills/fleet-orchestrator/scripts/instruction-words.mjs --base 8a858bb4 --changelog CHANGELOG.md --version 1.1.91: net +496, check passes.
  • Pre-PR lane runs on tree 4d21d508:
    • plugin-version-bump, plugin-manifest-versions, lint (uv run ruff check .), typecheck (uv run pyright) and pytest (uv run pytest plugins/) all passed.
    • vibe-eval-t0 fails only on tools/vibe-bench/src/phase3.test.ts, test "local driver snapshots one committed tree and writes one immutable dual-runtime result" (an expired quarantine deadline). The failure is identical on main 8a858bb (see Known main failures).
    • tools-ts[vibe-workers]: 848/848 on the rerun, after timeouts under load.
    • node-tests: no failure in this diff's files across three full runs. The only failures were timing tests outside the diff, under load:
      • plugins/vibe-core/lib/vibe/scripts/e2e-lock.test.mjs, tests "one SIGINT allows the repository teardown budget to finish without forced escalation" and "a second signal explicitly escalates a stuck command group", in the second machine's run;
      • plugins/vibe/skills/codex-fleet-monitor/scripts/recover.test.mjs, test "a second supervisor on the host exits while the first holds the lock", cancelled at 20 s in a local rerun, and passed in 2.3 s on the second machine.
    • node-tests was clean on tree b4a490dc earlier.
    • The other tools-ts rows were not run, under the targeted-tests ruling.

Vibe gate override: ISS-14184: node-tests failed only on timing tests outside this diff under load (the e2e-lock.test.mjs signal tests and recover.test.mjs's second-supervisor test), each passing on another full run or alone; this diff's files passed in every run

Adds the capture guard (refuses repeated main baselines and captures over 240 shots or about 20 minutes, with a fixed-shape ruling override), the fleet.sh --capture slot, the when-to-capture block for visual QA, one-pass visual QA after fixes, and queue-based ETAs. vibe 1.1.91.
Adds two fleet.test.mjs cases: a missing or unreadable capture spec is a usage error before any load check, slot or command, and the capture and containerized e2e build flags are accepted in either order.
@shafty023
shafty023 merged commit 716a568 into main Nov 16, 2026
11 of 12 checks passed
@shafty023
shafty023 deleted the vibe/iss-14158-capture-budget branch November 16, 2026 17:47
shafty023 added a commit that referenced this pull request Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant