You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Measure the reply-contract failure rate on a local model, as the before number for #144 #175
Related: #167 (local model serving, which landed the plumbing), #144 (structured replies — this is the before number for its first item).
Why
#167's last acceptance line asked for a measured reply-contract failure rate against a local model, and it is the one line that PR did not do: it is a measurement exercise rather than a code change, so it was left out rather than half-done.
The number matters because both failure modes are silent. character_prompt asks for a Behavior name on line one and an optional spoken line on line two. parse_proposal rejects prose; as_speech turns anything unparsable into dialogue; a hard failure becomes Wake::Failed and StaticDirector takes the turn. Nothing in the app says which of those happened, so "the buddy feels dumber on my local model" is currently unfalsifiable.
Constrained decoding is the fix, and #144 item 1 decides its shape. It should be argued with a number rather than an intuition.
Scope
Run a scripted set of wakes against a local model and count the outcomes:
accepted by parse_proposal as a declared Behavior,
fell through as_speech into speech,
ended as Wake::Failed.
Vary the wake so the prompts are not identical — the reactive happened values (Poke, Throw, Summon, Perch) plus ambient are enough. Sample size only has to be big enough to tell 5% from 50%; a few dozen wakes is plenty.
Do it for the model #167's README recommends (gemma4 on Ollama), and ideally one smaller model, since the hypothesis is that size drives the failure rate.
Write the result into the README's local-serving section next to the existing warning, so the claim there stops being qualitative.
Not in scope
Fixing the rate. That is #144: whatever structured-reply substrate it picks has to be expressible as response_format on chat-completions, because that is the only knob Ollama, llama.cpp, LM Studio and vLLM share.
Acceptance
The three counts, for at least one local model, recorded in the README with the model and the hardware.
#167 closed via #173. This is the leftover: measure parse_proposal / as_speech / Wake::Failed on the README's recommended local model (gemma4 on Ollama) and write the counts next to the existing warning.
Activity
omesser commentedon Sep 2, 2026
#167 closed via #173. This is the leftover: measure
parse_proposal/as_speech/Wake::Failedon the README's recommended local model (gemma4on Ollama) and write the counts next to the existing warning.Measure the reply-contract failure rate against a live local model (#175