Skip to content

Measure the reply-contract failure rate on a local model, as the before number for #144 #175

Closed

Description

@omesser

Related: #167 (local model serving, which landed the plumbing), #144 (structured replies — this is the before number for its first item).

Why

#167's last acceptance line asked for a measured reply-contract failure rate against a local model, and it is the one line that PR did not do: it is a measurement exercise rather than a code change, so it was left out rather than half-done.

The number matters because both failure modes are silent. character_prompt asks for a Behavior name on line one and an optional spoken line on line two. parse_proposal rejects prose; as_speech turns anything unparsable into dialogue; a hard failure becomes Wake::Failed and StaticDirector takes the turn. Nothing in the app says which of those happened, so "the buddy feels dumber on my local model" is currently unfalsifiable.

Constrained decoding is the fix, and #144 item 1 decides its shape. It should be argued with a number rather than an intuition.

Scope

Run a scripted set of wakes against a local model and count the outcomes:

  • accepted by parse_proposal as a declared Behavior,
  • fell through as_speech into speech,
  • ended as Wake::Failed.

Vary the wake so the prompts are not identical — the reactive happened values (Poke, Throw, Summon, Perch) plus ambient are enough. Sample size only has to be big enough to tell 5% from 50%; a few dozen wakes is plenty.

Do it for the model #167's README recommends (gemma4 on Ollama), and ideally one smaller model, since the hypothesis is that size drives the failure rate.

Write the result into the README's local-serving section next to the existing warning, so the claim there stops being qualitative.

Not in scope

Fixing the rate. That is #144: whatever structured-reply substrate it picks has to be expressible as response_format on chat-completions, because that is the only knob Ollama, llama.cpp, LM Studio and vLLM share.

Acceptance

Activity

  1. omesser commented on Sep 2, 2026

    @omesser
    OwnerAuthor

    #167 closed via #173. This is the leftover: measure parse_proposal / as_speech / Wake::Failed on the README's recommended local model (gemma4 on Ollama) and write the counts next to the existing warning.

  2. added this to the V1 milestone on Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions