Repository navigation
feat(vibe): WS0 measurement tooling, replay and session telemetry (ISS-13326 to ISS-13331) - #361
shingorani888 wants to merge 2 commits into
Conversation
Replay corpus selector, runner, scorer, uncertainty and rework scoring and baseline report under plugins/vibe/evals/replay (M1 to M3, M5, M6). Session telemetry in the vibe session record (M4): worker launches from a PostToolUse hook, heavy-slot wait from e2e-lock.mjs, transcript tokens, wall time and environment minutes through vibe-sessions.mjs telemetry. Unmeasured values are null. Bumps vibe to 1.1.14. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Replay tooling and session telemetry reviewed against the vibe direction. Main concerns: the replay arm can read the oracle and earlier runs, its class rules are another copy of the area lists we already keep, and the telemetry hook rewrites the whole session record with no coordination, which can drop fields like the environment URLs. Telemetry stays local (nothing leaves the machine), but most of it is never recorded because nothing calls the telemetry command.
| prepareClone({ sourceRepo, ticket, workspace: clone, oracleDir: oracle }); | ||
| prepared.set(ticket.slug, clone); | ||
| } | ||
| return { workspace: copyForRun({ prepared: clone, dest: path.join(workRoot, ticket.slug, `${arm.name}-${n}`) }) }; |
There was a problem hiding this comment.
Every run's workspace sits next to <workRoot>/<slug>/oracle and next to the earlier runs. The arm gets unrestricted Bash (claude-arm.mjs), so cat ../oracle/files/<test> hands it the human tests. And since the plan runs the arms in turn per ticket, candidate-1 can read baseline-1's finished tree and baseline-1-out/stream.jsonl. contaminationMentions only counts the slug, #<pr> and the sha, so neither leak shows up in the score. Can the oracle and the run outputs live somewhere the arm can't reach, for example a separate root per run with the oracle kept outside workRoot?
| const DOCS_CONFIG = /(^|\/)(docs|\.claude|\.agents|\.github)\/|\.(md|mdx|ya?ml|json|toml)$|(^|\/)(AGENTS|CLAUDE)\.md$/; | ||
| const MOCKUP = /^apps\/prototypes\//; | ||
| const UI = /^(packages\/app|packages\/design-system|apps\/app|apps\/web|apps\/storybook|apps\/desktop\/src\/renderer)\/|\.(tsx|css|scss)$/; | ||
| const BACKEND = /^(apps\/api|apps\/mcp|apps\/relay|apps\/realtime|packages\/database|packages\/api|apps\/desktop\/src\/(main|server))\//; |
There was a problem hiding this comment.
These are symphony-alpha paths (apps/api, packages/design-system, apps/desktop/src/renderer, apps/prototypes). That part is fine for now: vibe targets symphony-alpha today. The concern is that this is another copy of the area lists we already keep in pr-facts.mjs and handoff-inventory.mjs. Can the replay classes read one shared list instead, so the three can't drift?
| const record = readSessionRecord(worktree); | ||
| if (!record) return null; | ||
| const telemetry = change(record.telemetry); | ||
| writeSessionRecord(worktree, { ...record, telemetry }); |
There was a problem hiding this comment.
This re-reads and rewrites the whole session record, now from a hook on every Agent call and from e2e-lock inside a worker's Bash, with no coordination with the orchestrator's own vibe-sessions.mjs writes. What happens is: the hook reads the record, a background environment worker records environment-result, then the hook writes back its stale copy plus telemetry and the environment URLs are gone. writeSessionRecord also truncates before writing, so a reader in that window hits a JSON parse error, and two Agent calls finishing together drop a launch. Can telemetry go in its own file next to vibe-session.json so a counter never touches the record vibe depends on?
| } | ||
|
|
||
| /** Records the measured cost the options carry in the session record's telemetry; options not given leave their measures as they were. */ | ||
| function recordTelemetry() { |
There was a problem hiding this comment.
Who calls this? Nothing in the skills or agents runs vibe-sessions.mjs telemetry, so in a real session tokens, wallTimeMs and environmentMinutes stay null forever. Only launches (Claude Code only) and slot wait ever get recorded. If the plan is to wire it in later, it should go in a script handoff already runs, not as another SKILL.md step every session pays for.
| process.stdout.write(`${JSON.stringify({ status: "busy", holder: attempt.holder })}\n`); | ||
| return BUSY_EXIT; | ||
| } | ||
| recordSlotWait(owner); |
There was a problem hiding this comment.
Only a run that got the slot records its wait. A run that waited the whole window and exited BUSY records nothing, so slotWaitMs reads lowest exactly when contention is worst. Shouldn't the busy path record its wait too, counted apart from acquisitions?
| import { parseTranscript, tokenTotals } from "../../skills/vibe/scripts/transcript-tokens.mjs"; | ||
|
|
||
| export const WRITER_AGENT = "vibe:vibe-change-worker"; | ||
| export const DEFAULT_BUDGET_USD = 15; |
There was a problem hiding this comment.
Where do $15 and 45 minutes come from? cli.mjs run has no flag for either, so at 3 per class (18 tickets) x 2 arms x 3 runs the default ceiling is about $1,600 per replay, set by a constant nobody picked on purpose. Can the cap be the caller's, with run refusing to start without one?
| "--allowedTools", "Bash", "Read", "Write", "Edit", "Glob", "Grep", "Skill", "Agent", | ||
| "--disallowedTools", "Bash(gh:*)", "Bash(git push:*)", | ||
| "--max-budget-usd", String(budgetUsd), | ||
| ...(arm.pluginDir ? ["--plugin-dir", arm.pluginDir] : []), |
There was a problem hiding this comment.
The baseline arm leaves the plugin dir out, so it runs whatever vibe the machine has installed, plus every other installed plugin and hook (this PR's PostToolUse hook included). Two machines, or one machine after an update, give different baselines. Can the baseline pin a plugin dir too and record the version it ran?
|
|
||
| The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Entries are listed newest-first; each plugin section is treated as released when merged to `main`. | ||
|
|
||
| ### vibe v1.1.14 |
There was a problem hiding this comment.
main is at vibe 1.1.16 now, so 1.1.14 is already behind. AGENTS.md also says not to hand-edit CHANGELOG.md or README.md; rebase and run /update-documentation for the entry and the version line. The release smoke isn't a merge requirement anymore (ISS-13321), so that note can come out of the test plan.
| @@ -0,0 +1,38 @@ | |||
| // Task classes the replay corpus is stratified by (plan section 4 class table). | |||
There was a problem hiding this comment.
Nit: "plan section 4 class table", "WS0 M4", "WS2 S1/S4" and "T1" (here and in the other new files) point at a plan that isn't in this repo. Say what the thing is or drop the reference.
| assert not hook_guard_routed(PLUGIN_ROOT, "PostToolUse", "telemetry-launch", tool="Bash") | ||
| guard = read(PLUGIN_ROOT / "hooks" / "telemetry-launch.mjs") | ||
| assert "inVibeSession" in guard | ||
| assert "return null" in guard |
There was a problem hiding this comment.
Nit: these check that strings exist in the source ("return null" in guard, "MIN_RUNS = 3" in runner). The JS tests already pin the behavior, and these stay green for any file that happens to contain the text. I'd drop this file.
|
We're planning a start-to-PR benchmark for vibe (ISS-13401), and parts of this PR fit it. Can you cut this PR down to the reusable pieces and drop the rest? Keep:
Drop:
The benchmark runs locally on a developer machine against a seeded environment, on both Claude Code and Codex, and its results go in a required section of the PR body. The driver will come from that ticket, so this PR can stay small. Thanks. |
Purpose
WS0 (Measurement) of PRD-823: ISS-13326 (M1 corpus selector), ISS-13327 (M2 runner), ISS-13328 (M3 scorer), ISS-13329 (M4 session telemetry), ISS-13330 (M5 uncertainty and rework scoring), ISS-13331 (M6 baseline report). Plugin
vibeonly. Built on vibe 1.1.13 (origin/main b4b6ad8); bumps to 1.1.14 (the sibling WS1 branch uses 1.1.15).What changed
plugins/vibe/evals/replay/(new; not an eval case, noprompt.md):corpus-select.mjs,git-enrich.mjs,criteria.mjs,classes.mjs(M1);runner.mjs,workspace.mjs,claude-arm.mjs,live-deps.mjs(M2);scorer.mjs,graders.mjs(M3);uncertainty.mjs(M5);baseline-report.mjs(M6);cli.mjs;replay.test.mjs.skills/vibe/scripts/session-telemetry.mjs,transcript-tokens.mjs(+tests); new PostToolUse hookhooks/post-tool-use.mjs+telemetry-launch.mjs(+test,hooks.json);e2e-lock.mjsrecords slot wait;vibe-sessions.mjs telemetrycommand andtelemetryinshow.tools/python/test_vibe_measurement_contract.py;hooks/pre-tool-use.test.mjsnow pins the PostToolUse registration; manifests, README version line, CHANGELOG entry.What each measure can and cannot do
touchrewrites the record by spreading it, sotelemetrysurvives;touchitself takes no telemetry fields, hence the separatetelemetrycommand. The environment-result schema rejects unknown fields, so environment minutes are not captured automatically: they are recorded only when a caller passes--environment-minutes(workflow run duration is still not read from anywhere).Agentinside a vibe session worktree. Slot wait: time from the waiter entry to acquisition ine2e-lock.mjs. Tokens: Claude transcripts only. Codex tokens/launches are not read and the Codex hook behavior for subagent tool names is unverified.result.duration_ms. The runner is writer-only (vibe change worker); a full-session replay with ticket and environment stubs is not built.testCommand); result is the per-file exit code. UI work without tests is scored by criteria graders only (noisier).Test plan
node --testover all 114plugins/**/*.test.mjsfiles: 907/911 pass. Failures:app-server-worker-recovery.black-box(fails the same way on pristine origin/main, EPIPE);project-resume-cross-package.black-boxande2e-lock"run serializes two commands" failed only under the full-suite load and pass alone (this branch and main); the PostToolUse registration test, fixed in the second commit.pytest plugins/vibe/tools/python: 468 passed.tools/plugin-manifestsvitest 12/12 and tsc clean.ruff checkon the new Python file passes (repo-wide ruff 0.17 reports pre-existing findings; pyright not run).node tools/vibe-release-smoke/smoke.mjs(needs a Mac signed in to ChatGPT; AGENTS.md requires it before merge). Needs to be run on the final commit by a maintainer./update-documentationis not available in this session; please regenerate if preferred.🤖 Generated with Claude Code