Skip to content

feat(evals): agent-DX eval harness + wave-0 docs patches - #755

Closed
sam-goodwin wants to merge 2 commits into
mainfrom
claude/nice-kirch-b6a02c
Closed

sam-goodwin wants to merge 2 commits into
mainfrom
claude/nice-kirch-b6a02c

Conversation

@sam-goodwin

Copy link
Copy Markdown
Member

Agent-DX eval harness: measures how well coding agents (harness × model) build, test, and deploy real apps with alchemy guided only by our docs — and closes the loop by turning transcript triage into docs patches that get A/B re-measured against a deployed docs preview.

evals/

  • Runner: (task × harness × model × trial) cells, doppler-injected secrets (alchemy-v2/dev), isolated workspaces outside the repo (no CLAUDE.md inheritance), per-run CLAUDE_CONFIG_DIR, stage e<runId> with Alchemy.localState(), journaled lifecycle in results/*.jsonl.
  • Adapters for claude-code (exercised), codex, opencode, pi (pending auth; disabled in evals.config.ts).
  • Verification pyramid: tsc → grader re-deploy + stack outputs → health → hidden functional → agent-authored-test re-run → destroy-clean, banked partial credit, headline binary e2ePass.
  • Two task modes:
tasks/pastebin      "spec"  — pinned API contract, deterministic verify/checks.ts, oracle answer/
tasks/pastebin-web  "user"  — 4-sentence user-voice prompt; graded by a black-box
                              verifier agent probing the DEPLOYED app against hidden verify/intent.md
  • Docs are a runner input: --docs-url (deploy website/ to a stage, point a wave at the preview) — the docs-iteration axis.

Wave-0 results (Claude Code × Fable 5, C1 docs-only)

task docs e2e turns $ min
pastebin prod ✅ 69 6.47 12.8
pastebin patched preview ✅ 49 4.34 7.1
pastebin-web prod ✅ 126 14.75 22.0
pastebin-web patched preview ✅ 133 16.27 23.4

pastebin −29% turns against patched docs; pastebin-web flat because its sinks are the round-2 TanStack patches (in this PR, not in that frozen preview). Patches move only the buckets they target.

Docs patches (attributed from transcript triage)

  • outputs.mdx — ## as section (.as<T>() narrowing), cheat-sheet row; baseline's feat: remove most HKTs and simplify types #1 sink (~11 turns)
  • workers.mdx — example returns worker.url.as<string>(); new "Handle the request" section
  • testing-a-stack.mdx — POST-JSON example, 4xx-is-a-value note, multi-file fileParallelism note
  • test-harness.mdx — what destroy(Stack) does, stage mismatch = 0 succeeded, NO_DESTROY convention
  • tanstack-start.mdx — minimal app skeleton (router/routeTree.gen/createServerFn), headless serverFn verification guidance, 1042 first-request note

Design + wave report: processes/Evals/ (local); plan summary in evals/README.md.

🤖 Generated with Claude Code

Runner (harness x model x task matrix, doppler-injected creds, isolated
workspaces, gated verification pyramid, journaled lifecycle), four harness
adapters (claude-code exercised; codex/opencode/pi pending auth), two tasks
(spec-mode pastebin with oracle + hidden checks; user-voice pastebin-web
graded by a black-box verifier agent against hidden intent), and eight docs
patches attributed from wave-0 transcript triage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alchemy-version-bot

alchemy-version-bot Bot commented Jul 7, 2026 •

Copy link
Copy Markdown
Contributor

Install the packages built from this commit:

alchemy

bun add alchemy@https://pkg.ing/alchemy/662ea8a

@alchemy.run/better-auth

bun add @alchemy.run/better-auth@https://pkg.ing/@alchemy.run/better-auth/662ea8a

@alchemy.run/pr-package

bun add @alchemy.run/pr-package@https://pkg.ing/@alchemy.run/pr-package/662ea8a

Remove seeded smoke.test.ts and stub alchemy.run.ts from task templates —
a seeded exemplar contaminates the measurement (agents copy whatever example
is present, inflating the testing-docs score; the stub leaks the Stack shape
getting-started should teach). Templates now ship only installed deps,
tsconfig, and env plumbing; env constraints (localState, NO_DESTROY teardown
gate) move into prompt environment notes. L5a grades all agent-authored
tests. Oracle re-validated 75/75 after the change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sam-goodwin sam-goodwin closed this Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant