Repository navigation
test(harness): Count quoted sample lines by phrasing - #541
Merged
Merged
Conversation
omesser
force-pushed
the
test/244-sample-line-quotes
branch
from
September 8, 2026 21:37
0c7b928 to
22f02ba
Compare
The #244 seam: a quote counter over the loaded personality's sample lines, and three prompt phrasings behind AI_BUDDY_BENCH_FRAMING. Test code only. Not yet run against a model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1800 wakes of gemma-4-e2b-it-4bit: 4 quoted sample lines in 118 prose replies, and the break rate wanders 0-20% between runs of one framing. No wording changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1200 wakes of gpt-oss-20b-MXFP4-Q8, two runs per framing. No sample line quoted; the framings still cannot be separated. Its breaks are the thinking trace spending LOCAL_MAX_TOKENS, so they say nothing about wording — noted on the constant too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
omesser
force-pushed
the
test/244-sample-line-quotes
branch
from
September 10, 2026 05:45
22f02ba to
cc4ddce
Compare
Owner
Author
|
The conclusion about needing to handle reasoning vs "final" response and adjusting max tokens is critical! This needs to be handled specifically |
This was referenced Sep 10, 2026
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The short version: a suspicion about our prompt turned out to be wrong, and nothing needs changing. This PR adds the measurement that shows it, so nobody has to wonder again. It has now been run on two models, which agree.
The background, for anyone not steeped in this
Every Character ships a
personality.txt. It ends with eight to ten examplesentences showing how that character talks — the cat's file has
"What is that one? Show me."and so on.When the buddy decides what to do next, it asks a model for a reply in a fixed
shape: a Behavior name on the first line (
prowl,nap), and optionally aspoken line under it. If the reply has no recognisable Behavior name, the app
cannot act on it — it falls back to just speaking the text. We call that a
contract break.
The suspicion (#244): those example sentences sit near the top of the
prompt, and they are all bare lines with no Behavior name. Maybe a small model
copies their shape — or copies one outright — and answers with a naked
sentence instead of a Behavior. The evidence was thin and the issue said so:
two replies out of forty, on one model.
What this measures
Two numbers, over many decisions:
sentences word-for-word — the thing the suspicion is actually about.
And it tries three wordings of the prompt to see if any reduces the breaks:
todayframedafterThe result on the first model
1,800 decisions against a local
gemma-4-e2b-it-4bit— 200 per run, threeruns per wording, about 26 minutes. Each run is a fresh process, so no run
inherits another's habits.
todayframedafterRead the rows, not the columns. The same wording, run three times, swings
from 0% to 20.5%. That swing is bigger than any difference between the three
wordings, so none of them can be called better — the numbers are noise, not
signal.
afterlooks lowest, buttodayalso scored zero once. No wordingchanged.
And the original suspicion doesn't hold. Copying an example sentence
happened 4 times in 1,800 decisions — 0.2%. Three of the four were the
shortest line in the file (
"You may continue.") returned on its own.What actually breaks the format is the model inventing a Behavior name that
doesn't exist —
Bat,grin,grooms,Chirps at movement. That last one ittook from the personality's ordinary prose, not from the example lines. So the
fix for contract breaks is a reply schema that requires a behavior field
(#144); the example sentences are not the cause.
A second model, and what it changes
One model is a thin base for a decision, so the same protocol was run against a
local
gpt-oss-20b-MXFP4-Q8. 1,200 decisions — 200 per run, tworuns per wording rather than three, because this model is roughly twenty times
slower per decision and the three wordings took 8.4 hours of model time as
it was. That is the one place this run is smaller than the first; everything
else matches.
todayframedafterOn the suspicion, this is as clean as it gets: zero copied example sentences
in 1,200 decisions. Only 8 of those 1,200 replies were prose at all. Across
both models that is 4 copies in 3,000 decisions. The example sentences are not
teaching models to answer with a bare line.
On the wordings, the same verdict as before. Reading the rows again: the
widest gap between two runs of one wording is 9 points (
framed, 47.5% and38.5%), and the spread between the three wordings' averages is 8 points. The
noise still covers the difference, so still no wording can be called better.
But this model breaks the format for a different reason, and that matters
more than the table.
gpt-oss-20bis a reasoning model: it thinks inprose before it answers, and that thinking is billed against the same token
budget as the reply. The app allows a local model 512 tokens. On about 40% of
decisions this model spends all 512 thinking and returns nothing at all —
an empty reply, which the harness records as a break. Checked directly against
the server, 6 of 20 requests came back with
finish_reason=lengthand emptycontent, and the median request used 479 of its 512 tokens.
So its breaks are not a choice about wording — they are a budget running out.
Rewording the personality cannot fix that, which is why the three wordings land
so close together here. The 512-token cap is worth its own look; it is not
this PR's to change, and the measurement is now recorded on the constant.
Where that leaves the conclusion
It stands, and rests on two models instead of one — but only for the part
the second model could actually speak to:
models, 3,000 decisions, 4 copies. This is the finding that got stronger.
gemmathe run-to-run noise swamps it, ongpt-ossthe breaks are a tokenbudget no wording touches.
gpt-ossusually ran out of budget before it could invent anything, so it neither
confirms nor contradicts that half. Richer AI interactions: what the model API and the Harness can carry #144's schema is still the argued fix,
on the same evidence as before.
Two models is still two models, and both are local quantised builds.
What changed in the code
Test code, plus one comment on a shipped constant. No shipped behavior moved,
and no personality file was touched.
example sentences. It matches only whole sentences, after flattening case,
punctuation and spacing, so a paraphrase does not count — the number is a
floor, never inflated.
AI_BUDDY_BENCH_FRAMING.so the next reader finds the answer instead of re-running it.
LOCAL_MAX_TOKENSrecording that 512 is not enough forgpt-oss-20b, since that constant's comment already discusses exactly thisfailure and now has a number for it.
How to verify
Six unit tests cover the matching logic with fixed strings and need no model.
One runs against the real
characters/cat/personality.txt, so a brokenextractor fails loudly instead of quietly reporting zero.
To reproduce a run against a live server:
AI_BUDDY_DIRECTOR_BASE_URL=http://localhost:8000 \ AI_BUDDY_DIRECTOR_MODEL=gpt-oss-20b-MXFP4-Q8 \ AI_BUDDY_BENCH_WAKES=200 AI_BUDDY_BENCH_FRAMING=after \ cargo test -p ai-buddy measure_the_reply_contract -- --ignored --nocaptureWhat this cannot tell you
At roughly 20 seconds a decision, the third round would have cost another
four hours; the two rounds it did get agree with each other closely on
today(35.5% and 35.0%) and less closely onframed(47.5% and 38.5%).run took 43 minutes early and 96 minutes late. That affects the wall-clock
numbers above, not the break rates, but it is why the run was cut at two
rounds.
gpt-oss-20bcould not stand in for them: it ran out of tokens before it gotthat far.
gemma"copies" was a sample line used inside a longersentence of the model's own. The counter calls that a copy; the prompt
explicitly allows a signature phrase to recur, so it is arguably the model
doing as it was told.
#244 is left open — the acceptance criteria look met, but closing it is
@omesser's call.
— Claude Code agent (Coder), on @omesser's behalf.
🤖 Generated with Claude Code