Repository navigation
ISS-14162: Fix the quarantined vibe eval cases - #437
Conversation
Move every ClosedLoop and symphony-alpha specific line out of vibe and vibe-core into the new optional vibe-closedloop-provider plugin, which ships text and data only. With the provider bound, sessions keep today's behavior; without it, vibe and fleet work from the repository itself: the person's description, plans in the private git directory, the AGENTS.md test command, the lockfile install, and a requested PR or named recipient at the end. - provider binding, provider text assembly and data access in vibe-core - guards read their rules from the bound provider and stay silent outside an activated chat - main sync gates its provider-only machinery on the provider's recipe - the recipes opt-in file is validated and withholds recipes with one reason - the provider is offered for install once, never re-enabled or reinstalled - generic scan gate, generic walk fixture, eval arms, and generic twins for the provider-bound tests - vibe 1.1.81, vibe-core 1.0.5, vibe-closedloop-provider 1.0.0
- writer ledgers parse capabilities by shape; grants are checked against the session binding at register and launch - the Claude launcher resolves provider text for its roles - writers inherit the lead's binding through a Provider line, enforced by the Codex launcher, the Claude launcher and the Agent guard - the recipes opt-in file is validated and judged in the session repository - adopting a session keeps its binding; moved provider releases re-resolve - every installed provider's plan folder stays private - generic fleet recovery ids and duties, the worktree exclude rule, the guardrails formatter from the verify recipe - dangling reference paths fixed, provider text free of ticket ids and home paths, the inventory snapshot and mapping checked in pytest - the eval provider arm installs provider-resolved text - bundles build byte-identically from any working directory
Bind the tracker in the tracker-duty tests, run spawned scripts outside the real checkout so the cwd repository leg cannot pick it, derive the provider line in fleet tests from the fixture binding, fix the shadowed prototype deploymentHost check and rebuild dist, list provider.mjs as a vibe-core wrapper, and record the changelog word count.
Pass the lead's binding to remote fleet writers, read the recipes opt-in from the main checkout for older sessions and for a named checkout, know UI files by extension without an areas recipe, never write a legacy provider record generic, keep old hook scope records active, restore the exact ticket pattern for Prior removal, keep clerkOrgId for older vibe, report a main sync with no check as needing review, require the sole workflow for an undeclared lane source, keep generic plans in the common git dir, move the repository E2E text into provider blocks, and stop offering an install over an unreadable provider.
…c-vibe-v2 # Conflicts: # CHANGELOG.md # plugins/vibe/.claude-plugin/plugin.json # plugins/vibe/.codex-plugin/plugin.json
…ider - provider.mjs assemble writes a vibe file merged for the session's binding, and assemble --blocks gives every bound block of a loaded SKILL.md in one call; the reading rules use them - each generic region and its provider block share their identical lines once, so assembled text is unchanged and the provider is smaller - only an unbound capability falls back to the generic text: a bound provider that is gone, a missing block or an unreadable record stops with its reason and fix (exit 4) in the CLI, the launcher, every script and the hooks, and a guard refuses the action it guards - session records are written whole and renamed into place - the 1.1.82 word count is measured at this tree
ISS-14039 (#427) took vibe 1.1.82 and vibe-core 1.0.5. Keep both sides of the claude-worker launcher (the broken-provider mapping and the review model), rebuild the bundles, drop the ticket slugs the merge brought into runtime files, and remeasure the word count against the new main.
ISS-14028 and ISS-14024 (#428) took vibe 1.1.84 and vibe-core 1.0.7. The unit-status guard keeps the bound provider's rule and gains the chat's own program board; its tests keep both cases. The fleet-resume header covers both changes, the bundles are rebuilt, the ticket slugs the merge brought into runtime comments are dropped, the launcher mark written into installed launchers stays as an allowlisted compatibility line, and the word count is remeasured against the new main.
…e check A bound session now behaves like main. The start is one version check when the provider skill's description names its calls, and --closedloop/--graph are accepted again as aliases of --connection; otherwise the read-only provider probe runs first, and a check with no states records unprobed instead of binding generic. Each capability binds on its own connection, so a missing graph keeps the ClosedLoop text and restores the graph. The check writes the bound text once per repository and binding, and readers open files from that folder instead of assembling each one. check always exits 0 and prints the setup block in every state. Also: precedent stays live-only, a brief without a Provider: line inherits the lead's binding, the preflight finds the checkout as main did when bound, vibe-bench binds through the real check, and the eval runner gains a provider-installed arm.
…-generic - tools/vibe-closedloop-build assembles plugins/vibe plus the provider into the committed plugins/vibe-closedloop (the `vibe` entry), with cold deploy and environment sections in their own files, recipes inline, and a CI drift check. plugins/vibe is published as `vibe-generic`. - Retire runtime provider text: the rule paragraph (two generic sentences stay, dropped by the build), provider-text.md, the bind-time folder, model-run assemble/block/path/tree, the install offer, provider-install, provider-match, the skill-listed flag and the provider skill. Discovery reads only the provider bundled in vibe. - Each install finds its own plugin name and marketplace; with both installed the ClosedLoop vibe runs the hooks and binds, and the check reports both-installed. - Eval arms install the built folder (with the repository opt-in file in the case folder) and plugins/vibe as `vibe`; no harness assembly. - Resolve run.py's private homes and every main guard to real paths.
…the word-count entry - provider.mjs knownProviderContracts lists the provider bundled in every vibe install or checkout, enabled or not; local-plans uses it so a plan written while bound stays private when the session is not, whichever vibe the lookup order picks. Binding and lookup orders are unchanged. - The vibe CHANGELOG entry names the full Markdown scope of its word count again, with the generated folder's exclusion stated after it.
… 1.1.97 - The build places the provider's whole-file references (ticket-template, closedloop-graph, ticket-units) at main's paths in plugins/vibe-closedloop and rewrites provider: names to main's relative forms; provider/ holds only provider.json, whose placed map providerPath resolves through. A build test requires every Markdown file reference in the built tree to resolve. - vibe 1.1.97, vibe-core 1.0.13 (VIBE_CORE_MIN raised to it); #432's new cloud code follows the branch's generic rules.
The built ClosedLoop vibe named no connection in its Plugin version command, so sessions invented --connection names and lost the setup check eval. A provider block now carries main's text, naming --closedloop and --graph; generic vibe keeps its --connection form.
Every case in the vibe eval quarantine now passes its unchanged grader, or is retired by a ruling, so eval-quarantine.json is empty. - vibe: skill, handoff, quality-loop and change-worker text fixes for the quarantined Claude and Codex cases. - vibe-bench: the Claude stager uses the installed skill (writer-role cases also read the change worker), marks later turns, stages every case on the no-selection path, and installs Pillow with pinned hashes. The private-artifact rule flags only identity roots. - The pm-prd-children prompt schema can express the checks it grades. The exec precedent case is retired. - ISS-14325: the handoff diff rules build one line index per file and find identifier uses in one pass, so the inventory finishes on large diffs. - ISS-14136: the duplicate-helper rule treats a built mirror under a .vibe-build.json marker as the same definition as its source. - vibe 1.1.100, vibe-closedloop-provider 1.0.3; plugins/vibe-closedloop rebuilt.
Targeted pre-PR evidencePublished HEAD:
Matching post-merge lanes: plugin-version-bump: |
Summary
Ticket: ISS-14162.
.vibe-build.jsonmarker (tools/vibe-closedloop-build/src/build.ts:44), and the rule checks for it (plugins/vibe/skills/handoff/scripts/review-gap-rules.mjs:95).Formerly quarantined cases
"Before" is the baseline arm on main before this work: 3 runs per case on Codex, 2 on Claude. "After" is the latest measured arm on this code: 5 runs per case unless noted. The last column is the Codex cloud pass on this PR (first pass, then the two reruns), when the case differed between the arms.
session-questions.mjs askresult, so asking the person, the right reply, failed the check. Ruled "Fix the case": the prompt records{"ask":true}(plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/prompt.md:23), and asks-about-cancel replaces reply-is-none (plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/graders/asks-about-cancel.md:1).pnpm-lock.yamland said "pre-check", which the graders rejected. The graders were widened as ruled (plugins/vibe/evals/pm-owned-lockfile-recovery/graders/keeps-intended-lockfile.md:5), and the vibe SKILL.md install-drift rule was made precise.NEEDS_CHANGE(plugins/vibe/agents/vibe-environment-worker.md:120).Also in this PR
checkGuardedConstants. It split the file's whole prefix for every line lookup, and it searched the whole file again for every definition..vibe-build.jsonwithbuiltFromat the built plugin's root (tools/vibe-closedloop-build/src/build.ts:196).builtFrom, as one definition. It does so only when that source exists (plugins/vibe/skills/handoff/scripts/review-gap-rules.mjs:100).Rulings
vibe.baseRefper process. The built-mirror marker (option 1), and its same-diff hardening as a follow-up.Deliberate test changes
loadEvalQuarantineandevalQuarantineStatustake an optional file argument (default the committed file) and accept an empty entries list.How it was checked
Known main failures
tools-ts[vibe-workers]) times out on this machine under load on main too. On main 7d5bacf, in a fresh worktree, 5 of 863 tests timed out (main-sync, main-sync-main-comparison, scope-intake). On this branch's final head, 2 of 863 timed out in main-sync. This PR does not touch tools/vibe-workers or its built dist. CI runs the lane on this PR.The lane as the evidence names it
Disclosures
plugin-version.mjs adoptbound it on resume. It started before ISS-13446: Run vibe and fleet in any repository #426, on a vibe that kept its plans in the worktree's ignored.closedloop-ai/vibe-plans/folder; bound vibe reads them from the git directory, so the session-plan script finds none for this session. The private plan-folder check was therefore run with the session-plan script of that earlier vibe (1.1.94), and it passed.vibe.baseRefin each process's environment and never in a git config file.Follow-ups
.vibe-build.jsononly when it is present at base or produced by the drift check. Today it trusts a marker added in the same diff; this repository's drift check ties the committed marker to the build.Stack
The base is main. #426 has merged. This merges after #433.