Repository navigation
feat(vibe): triage manifest and class pipeline skills (ISS-13332, ISS-13335) - #362
shingorani888 wants to merge 2 commits into
Conversation
- vibe-triage.mjs classifies a request into a class and writes a manifest (reason, evidence, policy steps, review scope) - environment (defer) and desktop (skip) recommendations are logged in skipped[] with reason and evidence; text alone recommends nothing - recommendations take effect only with VIBE_TRIAGE_APPLY_RECOMMENDATIONS=1; off by default, so the session flow is unchanged (ISS-13333, ISS-13334 await the owner) - four class pipeline skills cite the shared rules in pipeline-core.md - bump vibe to 1.1.15 (ISS-13332, ISS-13335) Testing: vibe-triage.test.mjs, test_vibe_triage_contract.py, pytest plugins/vibe Risks: None identified; default behavior unchanged Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Main concern is vision fit. Everything here only does something behind VIBE_TRIAGE_APPLY_RECOMMENDATIONS, and the two decisions it serves (ISS-13333, ISS-13334) are still open. So today we ship a classifier, four skills, a rules file and two contract suites that change no behavior, and every session pays for the extra startup step and the skill listing. The classifier is also a third, disagreeing copy of the area lists in pr-facts.mjs and handoff-inventory.mjs, and its keyword lists come from a seven-ticket sample. I'd rather hold this until the owner rules, then land the smallest version (script + one reference, one shared area list) that the ruling actually needs.
Also, the description says the release smoke is required before merge. That changed on 2026-10-28 (ISS-13321, now on main's CONTRIBUTING.md): it's optional, so don't block on it.
| const PROTOTYPE_PATH = /^apps\/prototypes\//; | ||
| const METRIC_PATH = /(^|\/)(metrics?|analytics|cost|reconcil\w*|aggregat\w*)(\/|[-.])/i; | ||
| const UI_PATH = /\.(tsx|css|scss)$|^packages\/(app|design-system)\/|^apps\/(app|web|storybook)\/|^apps\/desktop\/src\/renderer\//; | ||
| const BACKEND_PATH = /^apps\/(api|mcp|relay|realtime|doc-sync)\/|^packages\/(database|api|github|ai|loops-api|cost)\/|^apps\/desktop\/src\/(main|server)\/|\.sql$|schema\.prisma$/; |
There was a problem hiding this comment.
This is a third copy of the area lists we already have, and they disagree: pr-facts.mjs UI / DESKTOP_HOST (Desktop = renderer + packages/app) and handoff-inventory.mjs BACKEND_PREFIXES (no packages/github|ai|loops-api|cost), versus DESKTOP_PATH = /^apps\/desktop\/|^packages\/app\// here. vibe targets symphony-alpha for now, so symphony-alpha paths are fine, but there should be one shared list the handoff scripts and this script all read. Shouldn't the Desktop relevance check just reuse DESKTOP_HOST?
| const METRIC_WORDS = /\b(metrics?|reconcil\w*|cost|spend|tokens used|kpi|aggregate|rollup|totals?|percentages?)\b/i; | ||
| const TRIVIAL_WORDS = /\b(copy|wording|typo|label|colou?r|spacing|padding|margin|font|tooltip)\b/i; | ||
| const UI_WORDS = /\b(page|screen|table|columns?|chips?|buttons?|menus?|dialog|drawer|board|lanes?|headers?|toolbar|checkbox|icons?|sort control|sidebar|modal|layout|bars?)\b/i; | ||
| const BACKEND_WORDS = /\b(api|endpoint|route|migration|schema|database|webhook|audit|ledger|write paths?|backend|server-only|atomic upsert)\b/i; |
There was a problem hiding this comment.
These word lists are tuned on the 7 tickets in the PR description (board, lanes, sort control, ledger, audit, write paths, atomic upsert, tokens used). Seven tickets is a small sample to set routing rules from. What evidence do we have that it classifies the next 50 tickets correctly, for example an eval over a larger set of past tickets, and how does a misclass show up to the person?
| [CLASSES.Backend]: { scope: "consolidated", gates: ["workflow-code-review"] }, | ||
| [CLASSES.DocsConfig]: { scope: "consolidated-light", gates: [] }, | ||
| [CLASSES.Mockup]: { scope: "design-pass", gates: [] }, | ||
| [CLASSES.DataMetric]: { scope: "consolidated-adversarial", gates: ["logical-qa"] }, |
There was a problem hiding this comment.
Nothing reads steps.review.scope or steps.extraGates anywhere in the plugin: I grepped for consolidated, design-pass, as-today and the gate names, and the only hits are the triage files. The class skills say these are "recorded in the manifest for the consolidated reviewer", but there is no consolidated reviewer. What is this data for today?
| const byText = classifyByText(text); | ||
| const sources = byPaths ? ["paths"] : ["text"]; | ||
| let chosen = byPaths ?? byText; | ||
| if (byPaths && byText.cls === CLASSES.DataMetric && chosen.cls !== CLASSES.DataMetric) { |
There was a problem hiding this comment.
This lets request text override known paths, which contradicts triage.md ("Touched paths decide once they are known"). METRIC_WORDS is broad enough that it fires on normal UI asks: --paths docs/guide.md with "Update the runbook for the cost dashboard" comes back data-metric, env run, Desktop run, and the text-only "Rename the Total column header on the Projects board" is data-metric with the logical-qa gate. Is that intentional? If so, the reference should say text can override paths; if not, drop the override.
| ## Specific to this class | ||
|
|
||
| - Review scope: `consolidated-adversarial`. | ||
| - Environment: recommended `run`; Desktop: recommended `run` when a Desktop-relevant surface is in the evidence, otherwise recorded `unknown` and run. Neither is ever left out, so the triage switch changes nothing for this class (PC-1). |
There was a problem hiding this comment.
This isn't what the script does. recommend() returns skip for Desktop on a ui-feature whose paths have no apps/desktop / packages/app file, and vibe-triage.test.mjs line 66 asserts exactly that (WEB_UI_PATHS -> ACTIONS.Skip). With the switch on, a web-only multi-file UI change leaves Desktop out, so "Neither is ever left out, so the triage switch changes nothing for this class" is false. One of the two needs to change, and the contract test should pin whichever is true.
|
|
||
| Once, in "Starting new" after the requirements brief exists: | ||
|
|
||
| `node scripts/vibe-triage.mjs classify --request "<the request>" [--ticket-file <file>] [--paths <likely files from the brief>] --out <file>` |
There was a problem hiding this comment.
Where do <likely files from the brief> come from? The requirements worker's brief returns a route and a FEATURE_MAP id, not a file list. So either the orchestrator passes no paths, and the manifest never recommends anything even with the switch on, or it guesses files, and those guesses are then treated as "touched paths". That's the same text-derived evidence the text-only guard exists to reject, just one step removed. And since the class is never re-classified (line 16), with the switch on a guess that misses packages/app skips Desktop for the whole session. The replay table's oracle column uses the merged PR's real file list, so it doesn't measure this path.
| 7. Classify the work and stand up the environment (section 3). Run | ||
| `scripts/vibe-triage.mjs classify` and keep its manifest | ||
| (`references/triage.md`); for the class it names, other than `unknown` | ||
| or `mockup`, also load that class's pipeline skill. Section 3 runs now unless |
There was a problem hiding this comment.
With the switch off (the only shipped state), this step changes no behavior, but every session still pays for it: one more script call, a manifest file, and loading a class skill whose content is "the handoff runs every review it runs today". The four new top-level skills also add their names and descriptions to the skill listing every Codex and Claude Code session loads, in every repo the plugin is installed in, not just vibe sessions. Since ISS-13333 / ISS-13334 are still open, can we hold the step and the class skills until there's a ruling, or at least make them references under skills/vibe/references/ that only load when effective is something other than run?
| @@ -0,0 +1,18 @@ | |||
| --- | |||
| name: vibe-pipeline-backend | |||
| description: Class pipeline for a vibe session whose triage manifest says backend; internal, loaded by the vibe skill. | |||
There was a problem hiding this comment.
These four skills are each a few lines of "cites PC-1 to PC-5, restates none" plus a review-scope string nothing reads. They don't need to be skills: a skill is listed and discoverable in every session, and description: ...internal, loaded by the vibe skill doesn't stop that. One table in triage.md (it already has the classes table) would carry the same content with no listing cost and no pipeline-core.md / contract-test scaffolding to keep the five rules "in exactly one file".
| "name": "vibe", | ||
| "description": "Vibe sessions, handoff, seed refresh, and ClosedLoop ticket workflows", | ||
| "version": "1.1.13", | ||
| "version": "1.1.15", |
| text = section(TRIAGE.read_text(), "## The switch") | ||
| assert "`VIBE_TRIAGE_APPLY_RECOMMENDATIONS` is off by default" in text | ||
| assert "every step's `effective` is `run`" in text | ||
| assert "ISS-13333, ISS-13334" in text |
There was a problem hiding this comment.
Nit: this pins ticket ids as test contract. When ISS-13333 / ISS-13334 get ruled, someone edits the reference and this test breaks for a reason that has nothing to do with behavior. The applyRecommendations / effective assertions in the node test already cover what matters.
…n switch Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Purpose
PRD-823, workstream WS1: ISS-13332 (T1 classifier and manifest schema) and ISS-13335 (T5 class pipeline skills plus routing). T3 (ISS-13333, defer the environment) and T4 (ISS-13334, skip Desktop) are applied automatically when the evidence shows the step is not needed; there is no opt-in switch (decision: defer and skip are done automatically, not recommended). T2 (mid-run upgrade rules) is out of scope.
What changed (plugin
vibe)skills/vibe/scripts/vibe-triage.mjs(+.test.mjs):classifyandcheck. Classes: trivial-ui, ui-feature, backend, docs-config, data-metric, mockup, unknown. Unknown stays unknown, never guessed. The manifest records class, reason, evidence, the policy steps and the review scope. Onlyenvironment(defer) anddesktop(skip) carry adecision;effectivefollows it automatically. Each is logged inskipped[]with reason and evidence, andcheckfails when a log entry, reason or evidence is missing, or wheneffectivedisagrees with the decision. A class resting on request text alone leaves nothing out.skills/vibe/references/pipeline-core.md: shared rules PC-1 to PC-5, each in exactly one file. Four class skills (vibe-pipeline-trivial-ui,-ui-feature,-backend,-docs-config) cite every rule and restate none.tools/python/test_vibe_triage_contract.pypins both.skills/vibe/references/triage.mdand three small edits toskills/vibe/SKILL.md(Starting new step 7, the Desktop bullet, References) that read the manifest'seffective.pnpm test:lanesand the repo's e2e path filter, never from the class. Review scope and extra gates (workflow-code-review,logical-qa) are only recorded in the manifest; handoff still runs every review it runs today.Version and merge order
Bumped vibe 1.1.13 -> 1.1.15 (
.claude-pluginand.codex-plugin), built on origin/main b4b6ad8 (vibe 1.1.13). The sibling WS0 branch uses 1.1.14. Expect CHANGELOG.md,plugins/vibe/README.md(version line) and both plugin.json files to conflict with whichever merges second; resolve by keeping both CHANGELOG entries (newest first) and the higher version.Classifier check on 7 replay tickets (symphony-alpha merged PRs)
Real class taken from the merged PR diff. Request text only = the ticket brief; oracle paths = the merged PR's own file list (leaks ground truth, so an upper bound).
Defer or skip decided where the real change needed the step: 0 of 3 (all on oracle paths); text-only defers or skips nothing by design. This is in-sample: the heuristics were tuned after reading these briefs. Out of sample, a title-only run over the last 72 ISS commits in symphony-alpha found that letting text alone decide skips mis-skipped 4 of 17 decisions (2 of 9 commits); that is why text alone now leaves nothing out. No class upgrade (T2) exists in this release, so a wrong class cannot be corrected mid-run; a path-based defer or skip is therefore the only automatic one.
Tests run
node --test plugins/vibe/skills/vibe/scripts/vibe-triage.test.mjs: 12 pass (re-run after removing the switch)pytest plugins/vibe/tools/python/test_vibe_triage_contract.py: 8 pass after removing the switch (fullpytest plugins/vibewas 493 pass before that change and has not been re-run)tools/plugin-manifestsnpm test: 12 pass;ruff checkon the new Python test: passnode --testover all plugin suites: not completed (machine load average ~380 from other jobs; one unrelated timeout incl-sweep/.../app-server-worker-recovery.black-box.test.mjs, a file this PR does not touch)node tools/vibe-release-smoke/smoke.mjs(needs a Mac with the ChatGPT app signed in). Required before merge per CONTRIBUTING.md; its pass still needs to be added here./update-documentationwas not available in this session; please regenerate if you prefer.🤖 Generated with Claude Code