Repository navigation
feat: Run the Director against a local model server with no key - #173
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cursor
Bot
force-pushed
the
feat-local-model-serving
branch
2 times, most recently
from
September 2, 2026 00:37
0773a4c to
efc80fe
Compare
omesser
commented
Sep 2, 2026
Addresses inline comments on PR #173. README: - Fix env table: API key is optional for local, not forbidden - Add oMLX to server table (http://localhost:8000, requires key) - Add Auth column showing which servers require/support keys - Add example commands for Ollama (no key) and oMLX (with key) - Add probe examples for both servers - Stop claiming local servers need no key as a blanket statement model.rs: - Rewrite is_local docstring: local means key is optional, not that servers ship with no auth - Add test verifying present key is used even when base is local probe-model.sh: - Fix comment claiming local needs no key Facts verified: - oMLX requires API key by default (--api-key or OMLX_API_KEY) - Ollama still defaults to no auth - llama.cpp, LM Studio, vLLM support optional --api-key - mlx_lm.server has no native auth (reverse proxy required) Co-authored-by: Oded Messer <odedmesser@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #167. Running the Director against Ollama needed
AI_BUDDY_DIRECTOR_API_KEY=ollama— a string the stand-in demanded and the server ignored — and nothing said whether that server was up or had the model. Now a local base URL needs no key, and the app says once, at startup, when the server cannot answer.What
endpoint()andconfig()both gated onsecret_key(), so both had to change (the issue named only the first). A loopback, RFC1918, or.localbase URL configures with an empty key, whichheaders()already omits rather than sending as a made-up Bearer. Every other host still requires one: a missing cloud key must not silently become an unauthenticated request.spawn_preflightasks/v1/modelson a worker thread and prints one line when the answer is no. It gates nothing — a failed wake already falls back to Static per turn (ADR-0004 keeps the model off the frame loop, and this stays off it too). The line exists so a buddy that went quiet is not a mystery.preflight_verdictis pure, so the decision is tested without a server, and a server that lists nothing recognisable gets the benefit of the doubt.AI_BUDDY_DIRECTOR_TIMEOUT_SECSandAI_BUDDY_DIRECTOR_MAX_TOKENS.One judgement call worth flagging: the issue asked whether to send a "do not think" hint or raise the cap for local reasoning models. This raises the cap.
reasoning_effortis not a field all five servers accept, and a strict one rejects the whole request over an unknown field, so the portable half is the one that ships.Verify
Ollama was running with
gemma4:latest, so this was checked against a real server rather than reasoned about:No
AI_BUDDY_DIRECTOR_API_KEYanywhere in that flow.gemma4matchinggemma4:latestis deliberate — Ollama reports a tag the user does not type.Both failure paths, also against the real server:
The first line is from the running app, which kept going.
cargo test --all-targets390 green,node --test tests/23 green, hooks pass. Six new tests cover the local/remote split, the preflight verdicts, and the local defaults.Not covered: the issue's last acceptance line asks for a measured reply-contract failure rate for a local model, as a before number for #144. That is a measurement exercise rather than a code change, so it is tracked as #175 instead of half-done here.
🤖 Generated with Claude Code