Skip to content

feat(vibe): WS0 measurement tooling, replay and session telemetry (ISS-13326 to ISS-13331) - #361

Draft
shingorani888 wants to merge 2 commits into
mainfrom
feat/vibe-fleet-opt-v2-ws0-measurement
Draft

shingorani888 wants to merge 2 commits into
mainfrom
feat/vibe-fleet-opt-v2-ws0-measurement

Conversation

@shingorani888

Copy link
Copy Markdown
Contributor

Purpose

WS0 (Measurement) of PRD-823: ISS-13326 (M1 corpus selector), ISS-13327 (M2 runner), ISS-13328 (M3 scorer), ISS-13329 (M4 session telemetry), ISS-13330 (M5 uncertainty and rework scoring), ISS-13331 (M6 baseline report). Plugin vibe only. Built on vibe 1.1.13 (origin/main b4b6ad8); bumps to 1.1.14 (the sibling WS1 branch uses 1.1.15).

What changed

  • plugins/vibe/evals/replay/ (new; not an eval case, no prompt.md): corpus-select.mjs, git-enrich.mjs, criteria.mjs, classes.mjs (M1); runner.mjs, workspace.mjs, claude-arm.mjs, live-deps.mjs (M2); scorer.mjs, graders.mjs (M3); uncertainty.mjs (M5); baseline-report.mjs (M6); cli.mjs; replay.test.mjs.
  • M4: skills/vibe/scripts/session-telemetry.mjs, transcript-tokens.mjs (+tests); new PostToolUse hook hooks/post-tool-use.mjs + telemetry-launch.mjs (+test, hooks.json); e2e-lock.mjs records slot wait; vibe-sessions.mjs telemetry command and telemetry in show.
  • tools/python/test_vibe_measurement_contract.py; hooks/pre-tool-use.test.mjs now pins the PostToolUse registration; manifests, README version line, CHANGELOG entry.

What each measure can and cannot do

  • Unknown is null, never 0 or a guess. Session record: touch rewrites the record by spreading it, so telemetry survives; touch itself takes no telemetry fields, hence the separate telemetry command. The environment-result schema rejects unknown fields, so environment minutes are not captured automatically: they are recorded only when a caller passes --environment-minutes (workflow run duration is still not read from anywhere).
  • Launches: PostToolUse on Agent inside a vibe session worktree. Slot wait: time from the waiter entry to acquisition in e2e-lock.mjs. Tokens: Claude transcripts only. Codex tokens/launches are not read and the Codex hook behavior for subagent tool names is unverified.
  • Replay wall time uses the stream result.duration_ms. The runner is writer-only (vibe change worker); a full-session replay with ticket and environment stubs is not built.
  • M3 functional score: the human change's test files copied onto the agent tree and run once per file with the product repo's own command (caller supplies testCommand); result is the per-file exit code. UI work without tests is scored by criteria graders only (noisier).
  • M5: flag-dependent numbers (flagged count, precision, unflagged wrong) are null until a run records flagged assumptions (S1). Matching is by label, not wording. Grades and code-stickiness survival are taken as given, not computed here.
  • M6: the report builder is shipped; the baseline table itself has not been produced. That needs live runs (history-free clones of symphony-alpha, N>=3 per ticket per arm, Claude spend) that were not run in this PR.

Test plan

  • node --test over all 114 plugins/**/*.test.mjs files: 907/911 pass. Failures: app-server-worker-recovery.black-box (fails the same way on pristine origin/main, EPIPE); project-resume-cross-package.black-box and e2e-lock "run serializes two commands" failed only under the full-suite load and pass alone (this branch and main); the PostToolUse registration test, fixed in the second commit.
  • New tests run: replay.test.mjs 22/22, session-telemetry, transcript-tokens, post-tool-use, e2e-lock 14/14, vibe-sessions all pass.
  • pytest plugins/vibe/tools/python: 468 passed. tools/plugin-manifests vitest 12/12 and tsc clean.
  • ruff check on the new Python file passes (repo-wide ruff 0.17 reports pre-existing findings; pyright not run).
  • NOT run: node tools/vibe-release-smoke/smoke.mjs (needs a Mac signed in to ChatGPT; AGENTS.md requires it before merge). Needs to be run on the final commit by a maintainer.
  • CHANGELOG entry was written by hand in the file's format because /update-documentation is not available in this session; please regenerate if preferred.

🤖 Generated with Claude Code

shingorani888 and others added 2 commits October 28, 2026 17:44
Replay corpus selector, runner, scorer, uncertainty and rework scoring and
baseline report under plugins/vibe/evals/replay (M1 to M3, M5, M6). Session
telemetry in the vibe session record (M4): worker launches from a PostToolUse
hook, heavy-slot wait from e2e-lock.mjs, transcript tokens, wall time and
environment minutes through vibe-sessions.mjs telemetry. Unmeasured values are
null. Bumps vibe to 1.1.14.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@shafty023 shafty023 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replay tooling and session telemetry reviewed against the vibe direction. Main concerns: the replay arm can read the oracle and earlier runs, its class rules are another copy of the area lists we already keep, and the telemetry hook rewrites the whole session record with no coordination, which can drop fields like the environment URLs. Telemetry stays local (nothing leaves the machine), but most of it is never recorded because nothing calls the telemetry command.

prepareClone({ sourceRepo, ticket, workspace: clone, oracleDir: oracle });
prepared.set(ticket.slug, clone);
}
return { workspace: copyForRun({ prepared: clone, dest: path.join(workRoot, ticket.slug, `${arm.name}-${n}`) }) };

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every run's workspace sits next to <workRoot>/<slug>/oracle and next to the earlier runs. The arm gets unrestricted Bash (claude-arm.mjs), so cat ../oracle/files/<test> hands it the human tests. And since the plan runs the arms in turn per ticket, candidate-1 can read baseline-1's finished tree and baseline-1-out/stream.jsonl. contaminationMentions only counts the slug, #<pr> and the sha, so neither leak shows up in the score. Can the oracle and the run outputs live somewhere the arm can't reach, for example a separate root per run with the oracle kept outside workRoot?

const DOCS_CONFIG = /(^|\/)(docs|\.claude|\.agents|\.github)\/|\.(md|mdx|ya?ml|json|toml)$|(^|\/)(AGENTS|CLAUDE)\.md$/;
const MOCKUP = /^apps\/prototypes\//;
const UI = /^(packages\/app|packages\/design-system|apps\/app|apps\/web|apps\/storybook|apps\/desktop\/src\/renderer)\/|\.(tsx|css|scss)$/;
const BACKEND = /^(apps\/api|apps\/mcp|apps\/relay|apps\/realtime|packages\/database|packages\/api|apps\/desktop\/src\/(main|server))\//;

@shafty023 shafty023 Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are symphony-alpha paths (apps/api, packages/design-system, apps/desktop/src/renderer, apps/prototypes). That part is fine for now: vibe targets symphony-alpha today. The concern is that this is another copy of the area lists we already keep in pr-facts.mjs and handoff-inventory.mjs. Can the replay classes read one shared list instead, so the three can't drift?

const record = readSessionRecord(worktree);
if (!record) return null;
const telemetry = change(record.telemetry);
writeSessionRecord(worktree, { ...record, telemetry });

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This re-reads and rewrites the whole session record, now from a hook on every Agent call and from e2e-lock inside a worker's Bash, with no coordination with the orchestrator's own vibe-sessions.mjs writes. What happens is: the hook reads the record, a background environment worker records environment-result, then the hook writes back its stale copy plus telemetry and the environment URLs are gone. writeSessionRecord also truncates before writing, so a reader in that window hits a JSON parse error, and two Agent calls finishing together drop a launch. Can telemetry go in its own file next to vibe-session.json so a counter never touches the record vibe depends on?

}

/** Records the measured cost the options carry in the session record's telemetry; options not given leave their measures as they were. */
function recordTelemetry() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who calls this? Nothing in the skills or agents runs vibe-sessions.mjs telemetry, so in a real session tokens, wallTimeMs and environmentMinutes stay null forever. Only launches (Claude Code only) and slot wait ever get recorded. If the plan is to wire it in later, it should go in a script handoff already runs, not as another SKILL.md step every session pays for.

process.stdout.write(`${JSON.stringify({ status: "busy", holder: attempt.holder })}\n`);
return BUSY_EXIT;
}
recordSlotWait(owner);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only a run that got the slot records its wait. A run that waited the whole window and exited BUSY records nothing, so slotWaitMs reads lowest exactly when contention is worst. Shouldn't the busy path record its wait too, counted apart from acquisitions?

import { parseTranscript, tokenTotals } from "../../skills/vibe/scripts/transcript-tokens.mjs";

export const WRITER_AGENT = "vibe:vibe-change-worker";
export const DEFAULT_BUDGET_USD = 15;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do $15 and 45 minutes come from? cli.mjs run has no flag for either, so at 3 per class (18 tickets) x 2 arms x 3 runs the default ceiling is about $1,600 per replay, set by a constant nobody picked on purpose. Can the cap be the caller's, with run refusing to start without one?

"--allowedTools", "Bash", "Read", "Write", "Edit", "Glob", "Grep", "Skill", "Agent",
"--disallowedTools", "Bash(gh:*)", "Bash(git push:*)",
"--max-budget-usd", String(budgetUsd),
...(arm.pluginDir ? ["--plugin-dir", arm.pluginDir] : []),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The baseline arm leaves the plugin dir out, so it runs whatever vibe the machine has installed, plus every other installed plugin and hook (this PR's PostToolUse hook included). Two machines, or one machine after an update, give different baselines. Can the baseline pin a plugin dir too and record the version it ran?

Comment thread CHANGELOG.md

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Entries are listed newest-first; each plugin section is treated as released when merged to `main`.

### vibe v1.1.14

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main is at vibe 1.1.16 now, so 1.1.14 is already behind. AGENTS.md also says not to hand-edit CHANGELOG.md or README.md; rebase and run /update-documentation for the entry and the version line. The release smoke isn't a merge requirement anymore (ISS-13321), so that note can come out of the test plan.

@@ -0,0 +1,38 @@
// Task classes the replay corpus is stratified by (plan section 4 class table).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: "plan section 4 class table", "WS0 M4", "WS2 S1/S4" and "T1" (here and in the other new files) point at a plan that isn't in this repo. Say what the thing is or drop the reference.

assert not hook_guard_routed(PLUGIN_ROOT, "PostToolUse", "telemetry-launch", tool="Bash")
guard = read(PLUGIN_ROOT / "hooks" / "telemetry-launch.mjs")
assert "inVibeSession" in guard
assert "return null" in guard

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: these check that strings exist in the source ("return null" in guard, "MIN_RUNS = 3" in runner). The JS tests already pin the behavior, and these stay green for any file that happens to contain the text. I'd drop this file.

@shafty023

Copy link
Copy Markdown
Collaborator

We're planning a start-to-PR benchmark for vibe (ISS-13401), and parts of this PR fit it. Can you cut this PR down to the reusable pieces and drop the rest?

Keep:

  • transcript-tokens.mjs, fixed: it sums the placeholder output_tokens on Claude assistant lines, so output totals are wrong. Read output from result.modelUsage (or OTEL api_request events) instead.
  • the results shape and the report builder.

Drop:

  • the PostToolUse telemetry hook (on by default, no off switch, never records tokens or wall time in a real session);
  • the whole-file session-record rewrites (they race with e2e-lock and can wipe fields);
  • the isolated change-worker replay arm and the fixed $15 / 45 minute constants.

The benchmark runs locally on a developer machine against a seeded environment, on both Claude Code and Codex, and its results go in a required section of the PR body. The driver will come from that ticket, so this PR can stay small. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants