model: serve through geistd (GEISTSHELL_GEISTD) — resident session, pinned prefix, same decoder - #155
Merged
Merged
Conversation
…inned prefix, same decoder Agent runs are short processes and loading the GGUF in each one was the expensive part. With GEISTSHELL_GEISTD set (Unix socket path or host:port, GEISTD_TOKEN off loopback) the GEIST adapter talks to a geistd daemon (geist-serve >= 0.2 + pin) instead of loading libgeist: the session and its KV cache live there and outlive the process, the constant prefix is pinned there, and a run pays only for the tokens it adds. The decode paths are untouched: every geist_session_* call now goes through an eng_* layer with two implementations. Remote: prefill appends (a shadow of the daemon's history turns geistd's whole-context prefill into an append), reset keeps the pinned prefix, peek returns the whole logit vector, token_to_str is a local table fetched once (the decoder scans the vocabulary by surface form) and yields nullptr for EOS/end-of-turn so the existing `piece == nullptr` test still ends a turn. Errors from the daemon are logged and mapped to geist statuses. Wired at the three GEIST config sites (agent, run, eval) and in chat (which then skips the file resolve). deps/geistd/geistd_client.h is the vendored single-header client; jsmn from deps/jsmn. Verified with SmolLM2 behind geistd: free decode, the constrained decoder producing (recommend (kind finish) (reason ...)), two agent runs, and the real eval path with the same outcome as in-process (256 tokens, rejected). test_model_adapter gains a geistd path (GEISTSHELL_TEST_GEISTD), and test/smoke_geistd.sh drives it plus smoke_eval_real through a daemon. Also: append_bytes no longer claims [static n] for n == 0 (UBSan), and map_geist_status covers GEIST_E_STALE_CALIBRATION from the pinned engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…re the first system header for getaddrinfo Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…by the client implementation) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agent runs are short processes; loading the GGUF in each was the expensive part. With
GEISTSHELL_GEISTD(Unix socket path orhost:port;GEISTD_TOKENoff loopback) the GEIST adapter talks to a geistd daemon instead of loading libgeist: session + KV cache live there and outlive the process, the constant prefix is pinned there, a run pays only for the tokens it adds.geist_session_*call goes through aneng_*layer with an in-process and a remote implementation (append-style prefill via a history shadow, pin-aware reset, full logit vector, vocabulary pieces fetched once, EOS/end-of-turn →nullptras before).deps/geistd/geistd_client.hvendored.(recommend (kind finish) (reason …)), two agent runs, andsmoke_eval_real.shwith the same outcome as in-process (256 tokens, rejected).test_model_adaptergains a geistd path (GEISTSHELL_TEST_GEISTD);test/smoke_geistd.shdrives it.make test: 61 passed.append_bytesno longer claims[static n]for n == 0 (UBSan);map_geist_statuscoversGEIST_E_STALE_CALIBRATION.🤖 Generated with Claude Code