Skip to content

First-class local model serving: run the Director against Ollama, llama.cpp, LM Studio, vLLM, or MLX with no cloud key #167

Closed

Description

@omesser

Related: #144 (structured replies — the substrate this depends on; it decides the shape, not this issue), #11 (the HTTP stand-in), #16 (Harness attachment), #18 (settings binding), ADR-0004, ADR-0008, DESIGN decision 5.

Why

The buddy wakes on a pace all day. AI_BUDDY_DIRECTOR_WAKE_SECS starts at 120 seconds and backs off from there (doubling by default, capped at two hours), and every Poke, Throw, Grab, Perch, and Summon is a reactive wake on top of that. An always-on desk toy billing a hosted API for a two-line Behavior pick is the wrong shape: it puts a meter on idling. It also sends the frontmost application name and the wall clock off the machine on every wake — that is what follow_up in crates/core/src/director.rs puts in the prompt. Local serving is the privacy-and-cost-native fit: the wake budget stops mattering, and Sensing context never leaves the desk.

README already documents Ollama, and it works. That is "it happens to work", not a story. The key is a dummy string the stand-in demands and Ollama ignores; nothing checks the server is up or the model is pulled; and the 20-second timeout and 80-token cap in src-tauri/src/model.rs were sized against hosted models.

The second reason is the reply contract. character_prompt asks for a Behavior name on line 1 and an optional spoken line on line 2, then "Propose nothing else." parse_proposal rejects prose; as_speech turns anything unparseable into dialogue; a hard failure becomes Wake::Failed and StaticDirector takes the turn. Both failure modes are silent. A 3–8B local model breaks that contract far more often than gpt-4o-mini does.

Scope

The servers

All five speak OpenAI chat-completions at /v1/chat/completions, so completions_url reaches every one of them unchanged. This is mostly documentation, defaults, and a smoke test — not a new transport.

Server Default base URL Auth Schema-constrained output
Ollama http://localhost:11434 none — the documented SDK examples pass api_key='ollama', "required but ignored" native format; OpenAI-layer response_format still to verify
llama.cpp server http://localhost:8080 --api-key optional, default none (env LLAMA_API_KEY) yes — response_format as json_object+schema or json_schema, plus native grammar / json_schema
LM Studio http://localhost:1234 no key in the documented examples (authentication is a separate feature) yes — OpenAI-shaped response_format: {type: json_schema, …}
vLLM http://localhost:8000 --api-key / VLLM_API_KEY, optional; their docs warn it does not protect every endpoint yes — response_format: json_schema
Apple MLX (mlx_lm.server) http://localhost:8080 none documented not documented — the request-field list has no response_format

Two notes worth carrying into the README. llama.cpp and vLLM also expose /v1/responses (llama.cpp converts it into a chat-completions request internally), so an explicit full URL keeps working either way. MLX's own docs say the server is not recommended for production and implements only basic security checks.

What needs building

  • Do not require a fake key for a local base URL. endpoint() returns None when secret_key() is None, and that is the only reason AI_BUDDY_DIRECTOR_API_KEY=ollama exists in the README. The transport is already fine without one: headers() sets Authorization only when the key is non-empty. Let a loopback or private-network base URL configure with no key, and keep the key required for every other host so a missing cloud key still means Static.
  • A preflight capability check. Before the first wake, ask the configured origin whether it is up and whether the model is loaded. GET /v1/models is common to all five; llama.cpp adds /props (n_ctx, total_slots, chat_template) and a public /health; Ollama has /api/tags. Server down, or model not pulled: one clear log line naming the origin and the reason, and stay on Static. It must not become a blocking startup step — ADR-0004 keeps the model out of the frame loop, and the Completer already runs on a worker thread the loop only try_recvs.
  • Timeouts and caps sized for local hardware. TIMEOUT is a fixed 20 seconds. A cold local model can spend longer than that loading weights, so the first wake fails and the buddy is quietly Static for the rest of the session. Make the timeout configurable, and consider a longer budget for the first call than for later ones.
  • Reasoning-effort knobs do not generalize. reasoning: {effort: "low"} is sent only on the /v1/responses path. Local reasoning models (Qwen3, gpt-oss) emit thinking on chat-completions, and max_tokens: 80 truncates it — the visible content comes back empty or cut off, which parses as garbage and falls to Static. llama.cpp accepts reasoning_effort: "none" on chat-completions, vLLM has per-model reasoning parsers, Ollama has think. Decide whether to send a per-server "do not think" hint or to raise the cap when thinking is detected.
  • Model-size guidance and a recommended default. Name a model this repo has actually run: size, quantization, and the machine it was tested on, so AI_BUDDY_DIRECTOR_MODEL=llama3.2 stops being an unexplained example. Document the naming differences too — Ollama wants a tag (llama3.2:3b), llama.cpp reports the gguf path unless started with --alias, MLX wants a Hugging Face repo id.

Small models and the reply contract

Constrained decoding is the mitigation, and four of the five servers support it. This issue must not choose the schema: #144 item 1 decides the structured-reply substrate. What belongs here is the record that local serving is the strongest argument for that work, and the constraint that whatever #144 lands has to be expressible as response_format on chat-completions, because that is the only knob these servers share.

Until then, measure it. Run a scripted set of wakes against a local 3–8B model and count how many replies parse_proposal accepts, how many fall through as_speech into speech, and how many end as Wake::Failed. That number is the argument.

Not in scope

  • Bundling or downloading a model with the app. The user brings their own server.
  • A second inference path beside the one session. ADR-0008 and DESIGN decision 5 both reject it; this is the same Completer with a different base URL.
  • Anything that changes the Harness story (Harness attachment and the Action Log #16). When a Harness attaches it is the Director, and a local server is a stand-in like any other base URL.
  • Native, non-OpenAI APIs — Ollama's /api/chat, llama.cpp's /completion. Reach for them only if a capability probe genuinely cannot be done otherwise.

Acceptance

  • README documents Ollama plus at least one of llama.cpp / LM Studio / vLLM / MLX with the exact env for each, and the buddy runs from those instructions with AI_BUDDY_DIRECTOR_API_KEY unset. No cloud key anywhere in the flow.
  • A local base URL whose server is stopped, or whose model is not pulled, produces one log line naming the origin and the reason, and the sprite keeps moving on Static. No freeze, no stall in the frame loop, no silent nothing.
  • scripts/probe-model.sh (--probe-model) covers a local origin: reachable or not, the model list from /v1/models, and whether the configured model is in it. It already runs the same env and the same HTTP client as the overlay, so this is added coverage rather than a new tool.
  • Every documented server is listed with the model the repo actually tested and the hardware it was tested on.
  • The reply-contract failure rate for the recommended local model is measured and written down, so Richer AI interactions: what the model API and the Harness can carry #144 has a before number.

Activity

  1. omesser commented on Sep 2, 2026

    @omesser
    OwnerAuthor

    #173 landed on main (c5dc9f5).

    • A local base URL (loopback / RFC1918 / unique-local IPv6 / .local) configures with no key; a set AI_BUDDY_DIRECTOR_API_KEY is still sent (oMLX and others that take --api-key).
    • Startup preflight GETs /v1/models off the frame loop, logs one line if the origin is down or the model is missing, and stays on Static.
    • Local defaults: 120s timeout, 512 token cap.
    • README lists Ollama (tested: gemma4:latest on Apple silicon), oMLX, llama.cpp, LM Studio, vLLM, and mlx_lm.server, with copy-paste env for Ollama (no key) and oMLX (with key). scripts/probe-model.sh covers the local origin.

    Closing this. The last acceptance line (measure the reply-contract failure rate for #144) was split to #175 rather than half-done here.

  2. added this to the V1 milestone on Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions