Repository navigation
test: Measure the reply-contract rate on local models, and what it exposed - #230
Merged
Merged
Conversation
This was referenced Sep 2, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
omesser
force-pushed
the
feat-175-contract-rate
branch
from
September 2, 2026 12:50
0a32f59 to
c79a29c
Compare
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 2, 2026
This was referenced Sep 15, 2026
6 of 12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #175.
measure_the_reply_contract_failure_rateis an#[ignore]d test insrc-tauri/src/model.rsthat runs 40 varied wakes against a live local server and counts what came back. It classifies withModelDirector::wakeitself rather than a copy of the parser, so the number cannot drift from what the app actually does, and it loads the shippedcharacters/catpackage so the prompt is the one production sends.AI_BUDDY_BENCH_WAKESsets the sample size.Rebased onto main after #233. Every row below is a post-#233 run.
What it measured
gemma4:latest, 8B Q4_K_Mllama3.2:1b, 1Bgemma-4-e2b-it-4bit#175 expected size to drive the rate. Instruction tuning drives it harder: a 4-bit
e2bbuild plays a Behavior on nine wakes in ten, while a 1B swings between one and eight.The 1B is a range because it is unstable
Four runs of the same model on the same code gave 12%, 30%, 65% and 78% Behaviors played. That is not sampling noise around a mean — the whole run shares one session, so once a small model starts inventing names it keeps doing it and the run locks into a groove. Its invented names are visible in the replies:
snub,scratching, andgroomingwhere the manifest declaresgroom. The 8B repeated 95% twice.Recording a single figure for the 1B would have been the more quotable and less true answer, so the table gives the range and the README says to read it as "unreliable" rather than as a percentage.
Some of the remaining breaks are ours
Two of the four prose replies from the
e2bbuild were the verbatim line"What is that one? Show me."— a sample line incharacters/cat/personality.txt. A small model reads sample lines as replies to imitate whole, and a sample line carries no Behavior name, so quoting one breaks the contract by construction. That is worth weighing when writing them (#156), and it means the residual rate is not purely the model's.#233 held up, on all three servers
case-only— a reply that kept the contract and was refused on capitalisation — was 0% in every run, against 95%, 75% and 100% before the fix. Behaviors played rose from 0%, 2% and 0% to 95%, 12–78% and 90%.The comparison stayed exact-modulo-case rather than fuzzy, which the runs also show:
Greeter(8B),grooming(1B) and the quoted sample lines still became speech, because no Character declares them.oMLX's own break rate read 0% before and 10% after. That is the same single-session variance the 1B exposed, not a regression: the fix can only turn a refusal into an acceptance, never manufacture prose.
Verify
cargo test --all-targets(514) andnode --test tests/(26) green on the rebase.AI_BUDDY_DIRECTOR_BASE_URL=http://localhost:11434 AI_BUDDY_DIRECTOR_MODEL=gemma4 cargo test -p ai-buddy measure_the_reply_contract -- --ignored --nocapture#[ignore]keeps it out of ordinary runs; nothing in CI reaches a model server.🤖 Generated with Claude Code