Repository navigation
feat(evals): agent-DX eval harness + wave-0 docs patches - #755
Closed
sam-goodwin wants to merge 2 commits into
Closed
sam-goodwin wants to merge 2 commits into
sam-goodwin wants to merge 2 commits into
Conversation
Runner (harness x model x task matrix, doppler-injected creds, isolated workspaces, gated verification pyramid, journaled lifecycle), four harness adapters (claude-code exercised; codex/opencode/pi pending auth), two tasks (spec-mode pastebin with oracle + hidden checks; user-voice pastebin-web graded by a black-box verifier agent against hidden intent), and eight docs patches attributed from wave-0 transcript triage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Install the packages built from this commit: alchemy bun add alchemy@https://pkg.ing/alchemy/662ea8a@alchemy.run/better-auth bun add @alchemy.run/better-auth@https://pkg.ing/@alchemy.run/better-auth/662ea8a@alchemy.run/pr-package bun add @alchemy.run/pr-package@https://pkg.ing/@alchemy.run/pr-package/662ea8a |
Remove seeded smoke.test.ts and stub alchemy.run.ts from task templates — a seeded exemplar contaminates the measurement (agents copy whatever example is present, inflating the testing-docs score; the stub leaks the Stack shape getting-started should teach). Templates now ship only installed deps, tsconfig, and env plumbing; env constraints (localState, NO_DESTROY teardown gate) move into prompt environment notes. L5a grades all agent-authored tests. Oracle re-validated 75/75 after the change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agent-DX eval harness: measures how well coding agents (harness × model) build, test, and deploy real apps with alchemy guided only by our docs — and closes the loop by turning transcript triage into docs patches that get A/B re-measured against a deployed docs preview.
evals/alchemy-v2/dev), isolated workspaces outside the repo (no CLAUDE.md inheritance), per-runCLAUDE_CONFIG_DIR, stagee<runId>withAlchemy.localState(), journaled lifecycle inresults/*.jsonl.evals.config.ts).e2ePass.--docs-url(deploywebsite/to a stage, point a wave at the preview) — the docs-iteration axis.Wave-0 results (Claude Code × Fable 5, C1 docs-only)
pastebin −29% turns against patched docs; pastebin-web flat because its sinks are the round-2 TanStack patches (in this PR, not in that frozen preview). Patches move only the buckets they target.
Docs patches (attributed from transcript triage)
outputs.mdx—## assection (.as<T>()narrowing), cheat-sheet row; baseline's feat: remove most HKTs and simplify types #1 sink (~11 turns)workers.mdx— example returnsworker.url.as<string>(); new "Handle the request" sectiontesting-a-stack.mdx— POST-JSON example, 4xx-is-a-value note, multi-filefileParallelismnotetest-harness.mdx— whatdestroy(Stack)does, stage mismatch =0 succeeded,NO_DESTROYconventiontanstack-start.mdx— minimal app skeleton (router/routeTree.gen/createServerFn), headless serverFn verification guidance, 1042 first-request noteDesign + wave report:
processes/Evals/(local); plan summary inevals/README.md.🤖 Generated with Claude Code