Skip to content

feat: Run the Director against a local model server with no key - #173

Merged
omesser merged 3 commits into
mainfrom
feat-local-model-serving
Sep 2, 2026
Merged

omesser merged 3 commits into
mainfrom
feat-local-model-serving

Conversation

@omesser

@omesser omesser commented Sep 1, 2026 •

Copy link
Copy Markdown
Owner

Implements #167. Running the Director against Ollama needed AI_BUDDY_DIRECTOR_API_KEY=ollama — a string the stand-in demanded and the server ignored — and nothing said whether that server was up or had the model. Now a local base URL needs no key, and the app says once, at startup, when the server cannot answer.

What

  • No key for a local server. endpoint() and config() both gated on secret_key(), so both had to change (the issue named only the first). A loopback, RFC1918, or .local base URL configures with an empty key, which headers() already omits rather than sending as a made-up Bearer. Every other host still requires one: a missing cloud key must not silently become an unauthenticated request.
  • A startup preflight that only reports. spawn_preflight asks /v1/models on a worker thread and prints one line when the answer is no. It gates nothing — a failed wake already falls back to Static per turn (ADR-0004 keeps the model off the frame loop, and this stays off it too). The line exists so a buddy that went quiet is not a mystery. preflight_verdict is pure, so the decision is tested without a server, and a server that lists nothing recognisable gets the benefit of the doubt.
  • Local-sized timeout and cap. 20s/80 tokens were chosen for hosted calls. Local defaults are 120s (a cold model loads weights on the first call) and 512 tokens, both overridable with AI_BUDDY_DIRECTOR_TIMEOUT_SECS and AI_BUDDY_DIRECTOR_MAX_TOKENS.
  • README gains "A local model server": why local, the five servers with base URLs and model-naming, which one was actually tested, the probe command, the log lines, and what small models do to the reply contract.

One judgement call worth flagging: the issue asked whether to send a "do not think" hint or raise the cap for local reasoning models. This raises the cap. reasoning_effort is not a field all five servers accept, and a strict one rejects the whole request over an unknown field, so the portable half is the one that ships.

Verify

Ollama was running with gemma4:latest, so this was checked against a real server rather than reasoned about:

$ AI_BUDDY_DIRECTOR_BASE_URL=http://localhost:11434 \
  AI_BUDDY_DIRECTOR_MODEL=gemma4 scripts/probe-model.sh
  key    none (local server)
  200 {"object":"list","data":[{"id":"gemma4:latest",...}]}
  model gemma4 is served
POST http://localhost:11434/v1/chat/completions
  ok pong

No AI_BUDDY_DIRECTOR_API_KEY anywhere in that flow. gemma4 matching gemma4:latest is deliberate — Ollama reports a tag the user does not type.

Both failure paths, also against the real server:

director: http://localhost:11439 unreachable: ... Connection refused; staying on StaticDirector until it answers
director: http://localhost:11434 model "llama3.2" is not served; it has gemma4:latest

The first line is from the running app, which kept going.

cargo test --all-targets 390 green, node --test tests/ 23 green, hooks pass. Six new tests cover the local/remote split, the preflight verdicts, and the local defaults.

Not covered: the issue's last acceptance line asks for a measured reply-contract failure rate for a local model, as a before number for #144. That is a measurement exercise rather than a code change, so it is tracked as #175 instead of half-done here.

🤖 Generated with Claude Code

omesser and others added 2 commits September 2, 2026 00:28
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cursor
cursor Bot force-pushed the feat-local-model-serving branch 2 times, most recently from 0773a4c to efc80fe Compare September 2, 2026 00:37
Comment thread README.md Resolved
Comment thread src-tauri/src/model.rs Resolved
Addresses inline comments on PR #173.

README:
- Fix env table: API key is optional for local, not forbidden
- Add oMLX to server table (http://localhost:8000, requires key)
- Add Auth column showing which servers require/support keys
- Add example commands for Ollama (no key) and oMLX (with key)
- Add probe examples for both servers
- Stop claiming local servers need no key as a blanket statement

model.rs:
- Rewrite is_local docstring: local means key is optional, not that
  servers ship with no auth
- Add test verifying present key is used even when base is local

probe-model.sh:
- Fix comment claiming local needs no key

Facts verified:
- oMLX requires API key by default (--api-key or OMLX_API_KEY)
- Ollama still defaults to no auth
- llama.cpp, LM Studio, vLLM support optional --api-key
- mlx_lm.server has no native auth (reverse proxy required)

Co-authored-by: Oded Messer <odedmesser@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants