Skip to content

test(http-recorder): match cassettes prompt-agnostically for transport tests - #98

Merged
jack-champagne merged 1 commit into
local/amicodefrom
fix/prompt-agnostic-fixture-match
Jul 29, 2026
Merged

test(http-recorder): match cassettes prompt-agnostically for transport tests#98
jack-champagne merged 1 commit into
local/amicodefrom
fix/prompt-agnostic-fixture-match

Conversation

@jack-champagne

Copy link
Copy Markdown
Member

The three session.llm native recorded scenarios are the only failures on local/amicode — the base branch's test job reports 5 pass / 2 fail and those three are the entire cause. So the fork's base is red, every PR into it inherits a red check, and nothing in the fork can be validated. #93 is the current casualty: its own suite is 3366 pass / 3 fail, with zero failures in its own files.

Cause is ours

AMICODE-PATCHES.md ADDENDUM 2 consolidated the prompt so system.ts provider() returns [providerBase(model), PROMPT_COMMUNICATING], appending communicating.txt for every model family. The cassettes still record the pre-patch prompt, so all three break — on a diff of prose.

Why not just re-record or patch the fixtures

Re-recording needs live credentials for all three providers (OpenAI API key and an OAuth JSON blob, Zen console token + org id, Anthropic key) and makes real API calls.

Patching the recorded prompt text would work until the next prompt edit — and that patch already carries three addenda. It would be the fourth fix to the same recurring break.

What this does instead

Those scenarios assert transport mechanics: that a tool loop is driven to a final text answer. The prompt is incidental, so matching on it couples them to text they don't test.

RecorderOptions.match is an existing seam (types.ts:85, threaded through http() into recordingLayer) that nothing used. This adds promptAgnosticMatcher alongside defaultMatcher and passes it from this one test.

The prompt arrives in three different shapes, all covered:

shape provider
instructions OpenAI Responses
system[] Anthropic Messages
input[].role == "system" the OpenCode proxy

System-message content is replaced with a placeholder rather than deleted, so "a system message is present, with content" remains part of the match.

Verified narrow

The risk with a laxer matcher is that it stops catching real differences, so test/matching.test.ts pins both directions. It ignores prompt prose in all three shapes, and still rejects:

  • a different user message
  • a different model
  • a different tool set
  • a different URL
  • a missing-vs-present system message

Results: http-recorder 49 pass / 0 fail; the three recorded scenarios go 3 fail → 3 pass; typecheck green across 23 packages.

Scope

Test infrastructure only — no src/ file outside http-recorder imports this package, so nothing shipped changes. Deliberately does not touch requestDiff, so failure messages are unchanged.

…t tests

The three `session.llm native recorded` scenarios are the only failures on
`local/amicode`, so the fork's `test` job is red and every PR into it inherits a
red check — including #93, whose own suite is 3366 pass / 3 fail with zero
failures of its own.

Cause is ours. AMICODE-PATCHES.md ADDENDUM 2 consolidated the prompt so
`system.ts provider()` returns `[providerBase(model), PROMPT_COMMUNICATING]`,
appending `communicating.txt` for every model family. The cassettes still record
the pre-patch prompt, so all three break on a diff of prose.

Those tests assert TRANSPORT mechanics — that a tool loop is driven to a final
text answer. The prompt is incidental, so matching on it couples them to text
they do not test. Patching the recorded prompt instead would work until the next
prompt edit, and that patch already has three addenda.

`RecorderOptions.match` is an existing seam (types.ts:85, threaded through
`http()` to `recordingLayer`) that nothing used. Adds `promptAgnosticMatcher`
alongside `defaultMatcher` and passes it from this one test.

The prompt arrives in three shapes, all covered:
  instructions            OpenAI Responses
  system[]                Anthropic Messages
  input[].role=="system"  the OpenCode proxy

System-message content is replaced with a placeholder rather than deleted, so
"a system message is present, with content" stays part of the match.

Verified narrow, in test/matching.test.ts: it ignores prompt prose in all three
shapes, and still rejects a different user message, model, tool set, URL, or a
missing-vs-present system message. 49 pass in http-recorder; the three recorded
scenarios go 3 fail -> 3 pass; typecheck green across 23 packages.
@jack-champagne
jack-champagne merged commit 27b8ed1 into local/amicode Jul 29, 2026
1 of 4 checks passed
@jack-champagne
jack-champagne deleted the fix/prompt-agnostic-fixture-match branch July 29, 2026 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant