Repository navigation
Description
Related: #156 (the personality convention that introduced sample lines), #175/#230 (the benchmark that spotted this), #144 (structured replies, which would moot it).
What happens
The Character Prompt puts {personality} first, and every personality now ends with eight to ten complete verbatim utterances ("It has been heard to say: …"). Only afterwards does the prompt say:
Reply with the behavior name on the first line. An optional spoken line may follow on the next line.
Those teach two different shapes. A sample line carries no Behavior name, so a model that imitates the examples rather than the instruction produces a contract break by construction — the reply is refused, as_speech catches it, and the buddy talks instead of acting.
Seen in #230's run against oMLX gemma-4-e2b-it-4bit: two of its four prose replies were the verbatim line "What is that one? Show me." from characters/cat/personality.txt.
How much to believe
Thin: one model, two instances out of forty wakes. The other two models measured broke the contract differently — by inventing names (Sulks, delight, snub, grooming), not by quoting lines. So this is a well-motivated mechanism with weak support, not an established pattern. Confirming or dismissing it is most of the work here.
What to do
Measure before changing anything. measure_the_reply_contract_failure_rate (the #[ignore]d test in src-tauri/src/model.rs) already counts contract breaks over 40 wakes; AI_BUDDY_BENCH_WAKES raises the sample.
- Count quotes, not just breaks. Extend the harness to report how many prose replies are verbatim (or near-verbatim) sample lines from the loaded personality. That turns the hypothesis into a number and is useful whatever the answer.
- Compare prompt phrasings on the same model and sample size — at least: today's wording; one that marks the quoted lines as how the character speaks, not how to format a reply; and one that moves the personality after the format instruction. Small models lock into a groove within a session (test: Measure the reply-contract rate on local models, and what it exposed #230), so a single 40-wake run per variant is not enough to separate 5% from 10% — use a larger sample, or several runs, and say which.
- Keep whichever wording wins, or record that the difference was noise.
Do not change the personality convention itself on suspicion. The sample lines are the reason the voices read as well as they do (#156); the fix, if one is needed, belongs in how the prompt frames them.
Acceptance
- The harness reports a sample-line-quote count alongside the existing buckets.
- At least two prompt phrasings measured on one model, with the sample size stated and the instability caveat honoured.
- A recorded decision: the wording changed and why, or the effect was not distinguishable from noise.
- If Richer AI interactions: what the model API and the Harness can carry #144 lands first, close this instead — a schema requiring a behavior field cannot be satisfied by a bare line.
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsDone
Activity
test(harness): Record the #244 measurement in the benchmark
test(harness): Count quoted sample lines by phrasing
test(harness): Record the #244 measurement in the benchmark
test(harness): Count quoted sample lines by phrasing
test(harness): Record the #244 measurement in the benchmark
test(harness): Record the #244 measurement on a second model
omesser commentedon Sep 26, 2026
#541 already ran the measurement this issue asks for. The prompt stays as shipped.
Three phrasings (
today,framed,after), one fresh process per run, the shipped cat personality. The harness that counted quoted sample lines is on main in that pull request.gemma-4-e2b-it-4bit, 200 wakes × 3 runs × 3 phrasings (1,800):todayframedafterThe same phrasing swung from 0% to 20.5%. That swing covers every gap between phrasings. Quoted sample lines: 4 of 1,800.
gpt-oss-20b-MXFP4-Q8, 200 × 2 runs × 3 phrasings (1,200). Two runs, because a wake took about 20 seconds and the six runs took 8.4 hours.todayframedafterZero quotes in 1,200. The widest gap between two runs of one phrasing is 9 points; the spread between the three averages is 8. About 40% of this model's breaks were an empty reply after the 512-token budget of that run was spent thinking.
Across both models, 4 quoted sample lines in 3,000 wakes. Sample lines are not what produces a bare reply, and no phrasing is distinguishable from the run-to-run noise. Invented Behavior names stay a one-model observation on
gemma; #144's schema is still the argued fix for those. #763 later removed the verdict from the test's doc comment, so this comment is the record.— Grok Build agent, on @omesser's behalf.