test(http-recorder): match cassettes prompt-agnostically for transport tests - #98
Merged
Merged
Conversation
…t tests The three `session.llm native recorded` scenarios are the only failures on `local/amicode`, so the fork's `test` job is red and every PR into it inherits a red check — including #93, whose own suite is 3366 pass / 3 fail with zero failures of its own. Cause is ours. AMICODE-PATCHES.md ADDENDUM 2 consolidated the prompt so `system.ts provider()` returns `[providerBase(model), PROMPT_COMMUNICATING]`, appending `communicating.txt` for every model family. The cassettes still record the pre-patch prompt, so all three break on a diff of prose. Those tests assert TRANSPORT mechanics — that a tool loop is driven to a final text answer. The prompt is incidental, so matching on it couples them to text they do not test. Patching the recorded prompt instead would work until the next prompt edit, and that patch already has three addenda. `RecorderOptions.match` is an existing seam (types.ts:85, threaded through `http()` to `recordingLayer`) that nothing used. Adds `promptAgnosticMatcher` alongside `defaultMatcher` and passes it from this one test. The prompt arrives in three shapes, all covered: instructions OpenAI Responses system[] Anthropic Messages input[].role=="system" the OpenCode proxy System-message content is replaced with a placeholder rather than deleted, so "a system message is present, with content" stays part of the match. Verified narrow, in test/matching.test.ts: it ignores prompt prose in all three shapes, and still rejects a different user message, model, tool set, URL, or a missing-vs-present system message. 49 pass in http-recorder; the three recorded scenarios go 3 fail -> 3 pass; typecheck green across 23 packages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three
session.llm native recordedscenarios are the only failures onlocal/amicode— the base branch'stestjob reports5 pass / 2 failand those three are the entire cause. So the fork's base is red, every PR into it inherits a red check, and nothing in the fork can be validated. #93 is the current casualty: its own suite is 3366 pass / 3 fail, with zero failures in its own files.Cause is ours
AMICODE-PATCHES.mdADDENDUM 2 consolidated the prompt sosystem.ts provider()returns[providerBase(model), PROMPT_COMMUNICATING], appendingcommunicating.txtfor every model family. The cassettes still record the pre-patch prompt, so all three break — on a diff of prose.Why not just re-record or patch the fixtures
Re-recording needs live credentials for all three providers (OpenAI API key and an OAuth JSON blob, Zen console token + org id, Anthropic key) and makes real API calls.
Patching the recorded prompt text would work until the next prompt edit — and that patch already carries three addenda. It would be the fourth fix to the same recurring break.
What this does instead
Those scenarios assert transport mechanics: that a tool loop is driven to a final text answer. The prompt is incidental, so matching on it couples them to text they don't test.
RecorderOptions.matchis an existing seam (types.ts:85, threaded throughhttp()intorecordingLayer) that nothing used. This addspromptAgnosticMatcheralongsidedefaultMatcherand passes it from this one test.The prompt arrives in three different shapes, all covered:
instructionssystem[]input[].role == "system"System-message content is replaced with a placeholder rather than deleted, so "a system message is present, with content" remains part of the match.
Verified narrow
The risk with a laxer matcher is that it stops catching real differences, so
test/matching.test.tspins both directions. It ignores prompt prose in all three shapes, and still rejects:Results:
http-recorder49 pass / 0 fail; the three recorded scenarios go 3 fail → 3 pass;typecheckgreen across 23 packages.Scope
Test infrastructure only — no
src/file outsidehttp-recorderimports this package, so nothing shipped changes. Deliberately does not touchrequestDiff, so failure messages are unchanged.