Skip to content

test(harness): Count quoted sample lines by phrasing - #541

Merged
omesser merged 3 commits into
mainfrom
test/244-sample-line-quotes
Sep 10, 2026
Merged

omesser merged 3 commits into
mainfrom
test/244-sample-line-quotes

Conversation

@omesser

@omesser omesser commented Sep 8, 2026 •

Copy link
Copy Markdown
Owner

The short version: a suspicion about our prompt turned out to be wrong, and nothing needs changing. This PR adds the measurement that shows it, so nobody has to wonder again. It has now been run on two models, which agree.

The background, for anyone not steeped in this

Every Character ships a personality.txt. It ends with eight to ten example
sentences showing how that character talks — the cat's file has
"What is that one? Show me." and so on.

When the buddy decides what to do next, it asks a model for a reply in a fixed
shape: a Behavior name on the first line (prowl, nap), and optionally a
spoken line under it. If the reply has no recognisable Behavior name, the app
cannot act on it — it falls back to just speaking the text. We call that a
contract break.

The suspicion (#244): those example sentences sit near the top of the
prompt, and they are all bare lines with no Behavior name. Maybe a small model
copies their shape — or copies one outright — and answers with a naked
sentence instead of a Behavior. The evidence was thin and the issue said so:
two replies out of forty, on one model.

What this measures

Two numbers, over many decisions:

  1. How often the reply breaks the format (no usable Behavior name).
  2. How many of those breaks are the model handing back one of the example
    sentences word-for-word
    — the thing the suspicion is actually about.

And it tries three wordings of the prompt to see if any reduces the breaks:

Wording What it does
today the prompt as it ships
framed marks the examples as how the character speaks, not a format to copy
after moves the personality below the formatting instruction

The result on the first model

1,800 decisions against a local gemma-4-e2b-it-4bit — 200 per run, three
runs per wording, about 26 minutes. Each run is a fresh process, so no run
inherits another's habits.

Wording Replies that broke the format, per run Of those, copied examples
today 13.5%, 11.5%, 0% 1, 0, 0
framed 9.5%, 0%, 20.5% 0, 0, 0
after 0.5%, 0%, 3.5% 0, 0, 3

Read the rows, not the columns. The same wording, run three times, swings
from 0% to 20.5%. That swing is bigger than any difference between the three
wordings, so none of them can be called better — the numbers are noise, not
signal. after looks lowest, but today also scored zero once. No wording
changed.

And the original suspicion doesn't hold. Copying an example sentence
happened 4 times in 1,800 decisions — 0.2%. Three of the four were the
shortest line in the file ("You may continue.") returned on its own.

What actually breaks the format is the model inventing a Behavior name that
doesn't exist — Bat, grin, grooms, Chirps at movement. That last one it
took from the personality's ordinary prose, not from the example lines. So the
fix for contract breaks is a reply schema that requires a behavior field
(#144); the example sentences are not the cause.

A second model, and what it changes

One model is a thin base for a decision, so the same protocol was run against a
local gpt-oss-20b-MXFP4-Q8. 1,200 decisions — 200 per run, two
runs per wording rather than three, because this model is roughly twenty times
slower per decision and the three wordings took 8.4 hours of model time as
it was. That is the one place this run is smaller than the first; everything
else matches.

Wording Replies that broke the format, per run Of those, copied examples
today 35.5%, 35.0% 0, 0
framed 47.5%, 38.5% 0, 0
after 44.0%, 40.0% 0, 0

On the suspicion, this is as clean as it gets: zero copied example sentences
in 1,200 decisions.
Only 8 of those 1,200 replies were prose at all. Across
both models that is 4 copies in 3,000 decisions. The example sentences are not
teaching models to answer with a bare line.

On the wordings, the same verdict as before. Reading the rows again: the
widest gap between two runs of one wording is 9 points (framed, 47.5% and
38.5%), and the spread between the three wordings' averages is 8 points. The
noise still covers the difference, so still no wording can be called better.

But this model breaks the format for a different reason, and that matters
more than the table.
gpt-oss-20b is a reasoning model: it thinks in
prose before it answers, and that thinking is billed against the same token
budget as the reply. The app allows a local model 512 tokens. On about 40% of
decisions this model spends all 512 thinking and returns nothing at all —
an empty reply, which the harness records as a break. Checked directly against
the server, 6 of 20 requests came back with finish_reason=length and empty
content, and the median request used 479 of its 512 tokens.

So its breaks are not a choice about wording — they are a budget running out.
Rewording the personality cannot fix that, which is why the three wordings land
so close together here. The 512-token cap is worth its own look; it is not
this PR's to change
, and the measurement is now recorded on the constant.

Where that leaves the conclusion

It stands, and rests on two models instead of one — but only for the part
the second model could actually speak to:

  • Sample lines are not the cause of contract breaks. Now measured on two
    models, 3,000 decisions, 4 copies. This is the finding that got stronger.
  • No wording is better. Holds on both models, for different reasons: on
    gemma the run-to-run noise swamps it, on gpt-oss the breaks are a token
    budget no wording touches.
  • Invented Behavior names are still a one-model observation. gpt-oss
    usually ran out of budget before it could invent anything, so it neither
    confirms nor contradicts that half. Richer AI interactions: what the model API and the Harness can carry #144's schema is still the argued fix,
    on the same evidence as before.

Two models is still two models, and both are local quantised builds.

What changed in the code

Test code, plus one comment on a shipped constant. No shipped behavior moved,
and no personality file was touched.

  • A counter that spots when a reply hands back one of the loaded personality's
    example sentences. It matches only whole sentences, after flattening case,
    punctuation and spacing, so a paraphrase does not count — the number is a
    floor, never inflated.
  • The three prompt wordings above, selectable with AI_BUDDY_BENCH_FRAMING.
  • The conclusion from both models written into the benchmark's own doc comment,
    so the next reader finds the answer instead of re-running it.
  • A note on LOCAL_MAX_TOKENS recording that 512 is not enough for
    gpt-oss-20b, since that constant's comment already discusses exactly this
    failure and now has a number for it.

How to verify

cargo test                                  # 365 + 463 + 23 + 10 pass, 4 ignored
cargo clippy --all-targets -- -D warnings   # clean
cargo fmt --all -- --check                  # clean

Six unit tests cover the matching logic with fixed strings and need no model.
One runs against the real characters/cat/personality.txt, so a broken
extractor fails loudly instead of quietly reporting zero.

To reproduce a run against a live server:

AI_BUDDY_DIRECTOR_BASE_URL=http://localhost:8000 \
AI_BUDDY_DIRECTOR_MODEL=gpt-oss-20b-MXFP4-Q8 \
AI_BUDDY_BENCH_WAKES=200 AI_BUDDY_BENCH_FRAMING=after \
cargo test -p ai-buddy measure_the_reply_contract -- --ignored --nocapture

What this cannot tell you

  • Two models, and the second got two runs per wording rather than three.
    At roughly 20 seconds a decision, the third round would have cost another
    four hours; the two rounds it did get agree with each other closely on
    today (35.5% and 35.0%) and less closely on framed (47.5% and 38.5%).
  • The second model's throughput halved over the session — the same 200-wake
    run took 43 minutes early and 96 minutes late. That affects the wall-clock
    numbers above, not the break rates, but it is why the run was cut at two
    rounds.
  • Nothing here was measured on the other models test: Measure the reply-contract rate on local models, and what it exposed #230 saw inventing names, and
    gpt-oss-20b could not stand in for them: it ran out of tokens before it got
    that far.
  • One of the four gemma "copies" was a sample line used inside a longer
    sentence of the model's own. The counter calls that a copy; the prompt
    explicitly allows a signature phrase to recur, so it is arguably the model
    doing as it was told.

#244 is left open — the acceptance criteria look met, but closing it is
@omesser's call.

— Claude Code agent (Coder), on @omesser's behalf.

🤖 Generated with Claude Code

@omesser
omesser force-pushed the test/244-sample-line-quotes branch from 0c7b928 to 22f02ba Compare September 8, 2026 21:37
omesser and others added 3 commits September 9, 2026 22:39
The #244 seam: a quote counter over the loaded personality's sample
lines, and three prompt phrasings behind AI_BUDDY_BENCH_FRAMING.
Test code only. Not yet run against a model.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1800 wakes of gemma-4-e2b-it-4bit: 4 quoted sample lines in 118 prose
replies, and the break rate wanders 0-20% between runs of one framing.
No wording changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1200 wakes of gpt-oss-20b-MXFP4-Q8, two runs per framing. No sample
line quoted; the framings still cannot be separated. Its breaks are the
thinking trace spending LOCAL_MAX_TOKENS, so they say nothing about
wording — noted on the constant too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@omesser
omesser force-pushed the test/244-sample-line-quotes branch from 22f02ba to cc4ddce Compare September 10, 2026 05:45
@omesser omesser added documentation Improvements or additions to documentation research Defining or executing a research into an aspect: tech, direction, decision local models Regarding Local model inference specifically prompting LLM prompting facilities, shape and contents labels Sep 10, 2026
@omesser

omesser commented Sep 10, 2026

Copy link
Copy Markdown
Owner Author

The conclusion about needing to handle reasoning vs "final" response and adjusting max tokens is critical! This needs to be handled specifically

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation local models Regarding Local model inference specifically prompting LLM prompting facilities, shape and contents research Defining or executing a research into an aspect: tech, direction, decision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant