Skip to content

test: Measure the reply-contract rate on local models, and what it exposed - #230

Merged
omesser merged 5 commits into
mainfrom
feat-175-contract-rate
Sep 2, 2026
Merged

omesser merged 5 commits into
mainfrom
feat-175-contract-rate

Conversation

@omesser

@omesser omesser commented Sep 2, 2026 •

Copy link
Copy Markdown
Owner

Closes #175. measure_the_reply_contract_failure_rate is an #[ignore]d test in src-tauri/src/model.rs that runs 40 varied wakes against a live local server and counts what came back. It classifies with ModelDirector::wake itself rather than a copy of the parser, so the number cannot drift from what the app actually does, and it loads the shipped characters/cat package so the prompt is the one production sends. AI_BUDDY_BENCH_WAKES sets the sample size.

Rebased onto main after #233. Every row below is a post-#233 run.

What it measured

Server Model Behavior played Prose instead Failed
Ollama gemma4:latest, 8B Q4_K_M 95% 2% 2%
Ollama llama3.2:1b, 1B 12–78% 22–88% 0%
oMLX gemma-4-e2b-it-4bit 90% 10% 0%

#175 expected size to drive the rate. Instruction tuning drives it harder: a 4-bit e2b build plays a Behavior on nine wakes in ten, while a 1B swings between one and eight.

The 1B is a range because it is unstable

Four runs of the same model on the same code gave 12%, 30%, 65% and 78% Behaviors played. That is not sampling noise around a mean — the whole run shares one session, so once a small model starts inventing names it keeps doing it and the run locks into a groove. Its invented names are visible in the replies: snub, scratching, and grooming where the manifest declares groom. The 8B repeated 95% twice.

Recording a single figure for the 1B would have been the more quotable and less true answer, so the table gives the range and the README says to read it as "unreliable" rather than as a percentage.

Some of the remaining breaks are ours

Two of the four prose replies from the e2b build were the verbatim line "What is that one? Show me." — a sample line in characters/cat/personality.txt. A small model reads sample lines as replies to imitate whole, and a sample line carries no Behavior name, so quoting one breaks the contract by construction. That is worth weighing when writing them (#156), and it means the residual rate is not purely the model's.

#233 held up, on all three servers

case-only — a reply that kept the contract and was refused on capitalisation — was 0% in every run, against 95%, 75% and 100% before the fix. Behaviors played rose from 0%, 2% and 0% to 95%, 12–78% and 90%.

The comparison stayed exact-modulo-case rather than fuzzy, which the runs also show: Greeter (8B), grooming (1B) and the quoted sample lines still became speech, because no Character declares them.

oMLX's own break rate read 0% before and 10% after. That is the same single-session variance the 1B exposed, not a regression: the fix can only turn a refusal into an acceptance, never manufacture prose.

Verify

  • cargo test --all-targets (514) and node --test tests/ (26) green on the rebase.
  • Re-take a row: AI_BUDDY_DIRECTOR_BASE_URL=http://localhost:11434 AI_BUDDY_DIRECTOR_MODEL=gemma4 cargo test -p ai-buddy measure_the_reply_contract -- --ignored --nocapture
  • The #[ignore] keeps it out of ordinary runs; nothing in CI reaches a model server.

🤖 Generated with Claude Code

omesser and others added 4 commits September 2, 2026 15:41
)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@omesser
omesser force-pushed the feat-175-contract-rate branch from 0a32f59 to c79a29c Compare September 2, 2026 12:50
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Measure the reply-contract failure rate on a local model, as the before number for #144

1 participant