Repository navigation
Description
Related: #144 (structured replies — the substrate this depends on; it decides the shape, not this issue), #11 (the HTTP stand-in), #16 (Harness attachment), #18 (settings binding), ADR-0004, ADR-0008, DESIGN decision 5.
Why
The buddy wakes on a pace all day. AI_BUDDY_DIRECTOR_WAKE_SECS starts at 120 seconds and backs off from there (doubling by default, capped at two hours), and every Poke, Throw, Grab, Perch, and Summon is a reactive wake on top of that. An always-on desk toy billing a hosted API for a two-line Behavior pick is the wrong shape: it puts a meter on idling. It also sends the frontmost application name and the wall clock off the machine on every wake — that is what follow_up in crates/core/src/director.rs puts in the prompt. Local serving is the privacy-and-cost-native fit: the wake budget stops mattering, and Sensing context never leaves the desk.
README already documents Ollama, and it works. That is "it happens to work", not a story. The key is a dummy string the stand-in demands and Ollama ignores; nothing checks the server is up or the model is pulled; and the 20-second timeout and 80-token cap in src-tauri/src/model.rs were sized against hosted models.
The second reason is the reply contract. character_prompt asks for a Behavior name on line 1 and an optional spoken line on line 2, then "Propose nothing else." parse_proposal rejects prose; as_speech turns anything unparseable into dialogue; a hard failure becomes Wake::Failed and StaticDirector takes the turn. Both failure modes are silent. A 3–8B local model breaks that contract far more often than gpt-4o-mini does.
Scope
The servers
All five speak OpenAI chat-completions at /v1/chat/completions, so completions_url reaches every one of them unchanged. This is mostly documentation, defaults, and a smoke test — not a new transport.
| Server | Default base URL | Auth | Schema-constrained output |
|---|---|---|---|
| Ollama | http://localhost:11434 |
none — the documented SDK examples pass api_key='ollama', "required but ignored" |
native format; OpenAI-layer response_format still to verify |
llama.cpp server |
http://localhost:8080 |
--api-key optional, default none (env LLAMA_API_KEY) |
yes — response_format as json_object+schema or json_schema, plus native grammar / json_schema |
| LM Studio | http://localhost:1234 |
no key in the documented examples (authentication is a separate feature) | yes — OpenAI-shaped response_format: {type: json_schema, …} |
| vLLM | http://localhost:8000 |
--api-key / VLLM_API_KEY, optional; their docs warn it does not protect every endpoint |
yes — response_format: json_schema |
Apple MLX (mlx_lm.server) |
http://localhost:8080 |
none documented | not documented — the request-field list has no response_format |
Two notes worth carrying into the README. llama.cpp and vLLM also expose /v1/responses (llama.cpp converts it into a chat-completions request internally), so an explicit full URL keeps working either way. MLX's own docs say the server is not recommended for production and implements only basic security checks.
What needs building
- Do not require a fake key for a local base URL.
endpoint()returnsNonewhensecret_key()isNone, and that is the only reasonAI_BUDDY_DIRECTOR_API_KEY=ollamaexists in the README. The transport is already fine without one:headers()setsAuthorizationonly when the key is non-empty. Let a loopback or private-network base URL configure with no key, and keep the key required for every other host so a missing cloud key still means Static. - A preflight capability check. Before the first wake, ask the configured origin whether it is up and whether the model is loaded.
GET /v1/modelsis common to all five; llama.cpp adds/props(n_ctx,total_slots,chat_template) and a public/health; Ollama has/api/tags. Server down, or model not pulled: one clear log line naming the origin and the reason, and stay on Static. It must not become a blocking startup step — ADR-0004 keeps the model out of the frame loop, and the Completer already runs on a worker thread the loop onlytry_recvs. - Timeouts and caps sized for local hardware.
TIMEOUTis a fixed 20 seconds. A cold local model can spend longer than that loading weights, so the first wake fails and the buddy is quietly Static for the rest of the session. Make the timeout configurable, and consider a longer budget for the first call than for later ones. - Reasoning-effort knobs do not generalize.
reasoning: {effort: "low"}is sent only on the/v1/responsespath. Local reasoning models (Qwen3, gpt-oss) emit thinking on chat-completions, andmax_tokens: 80truncates it — the visible content comes back empty or cut off, which parses as garbage and falls to Static. llama.cpp acceptsreasoning_effort: "none"on chat-completions, vLLM has per-model reasoning parsers, Ollama hasthink. Decide whether to send a per-server "do not think" hint or to raise the cap when thinking is detected. - Model-size guidance and a recommended default. Name a model this repo has actually run: size, quantization, and the machine it was tested on, so
AI_BUDDY_DIRECTOR_MODEL=llama3.2stops being an unexplained example. Document the naming differences too — Ollama wants a tag (llama3.2:3b), llama.cpp reports the gguf path unless started with--alias, MLX wants a Hugging Face repo id.
Small models and the reply contract
Constrained decoding is the mitigation, and four of the five servers support it. This issue must not choose the schema: #144 item 1 decides the structured-reply substrate. What belongs here is the record that local serving is the strongest argument for that work, and the constraint that whatever #144 lands has to be expressible as response_format on chat-completions, because that is the only knob these servers share.
Until then, measure it. Run a scripted set of wakes against a local 3–8B model and count how many replies parse_proposal accepts, how many fall through as_speech into speech, and how many end as Wake::Failed. That number is the argument.
Not in scope
- Bundling or downloading a model with the app. The user brings their own server.
- A second inference path beside the one session. ADR-0008 and DESIGN decision 5 both reject it; this is the same Completer with a different base URL.
- Anything that changes the Harness story (Harness attachment and the Action Log #16). When a Harness attaches it is the Director, and a local server is a stand-in like any other base URL.
- Native, non-OpenAI APIs — Ollama's
/api/chat, llama.cpp's/completion. Reach for them only if a capability probe genuinely cannot be done otherwise.
Acceptance
- README documents Ollama plus at least one of llama.cpp / LM Studio / vLLM / MLX with the exact env for each, and the buddy runs from those instructions with
AI_BUDDY_DIRECTOR_API_KEYunset. No cloud key anywhere in the flow. - A local base URL whose server is stopped, or whose model is not pulled, produces one log line naming the origin and the reason, and the sprite keeps moving on Static. No freeze, no stall in the frame loop, no silent nothing.
scripts/probe-model.sh(--probe-model) covers a local origin: reachable or not, the model list from/v1/models, and whether the configured model is in it. It already runs the same env and the same HTTP client as the overlay, so this is added coverage rather than a new tool.- Every documented server is listed with the model the repo actually tested and the hardware it was tested on.
- The reply-contract failure rate for the recommended local model is measured and written down, so Richer AI interactions: what the model API and the Harness can carry #144 has a before number.
Activity
omesser commentedon Sep 2, 2026
#173 landed on main (
c5dc9f5)..local) configures with no key; a setAI_BUDDY_DIRECTOR_API_KEYis still sent (oMLX and others that take--api-key)./v1/modelsoff the frame loop, logs one line if the origin is down or the model is missing, and stays on Static.gemma4:lateston Apple silicon), oMLX, llama.cpp, LM Studio, vLLM, andmlx_lm.server, with copy-paste env for Ollama (no key) and oMLX (with key).scripts/probe-model.shcovers the local origin.Closing this. The last acceptance line (measure the reply-contract failure rate for #144) was split to #175 rather than half-done here.