Repository navigation
serve: report cached prompt tokens, llama.cpp timings and GET /v1/status - #107
Merged
Merged
Conversation
…/status The conversation cache's reuse is kept per request (GET /metrics shows it), but the APIs did not report it, so clients showed a cached prompt as 0%: - OpenAI: usage.prompt_tokens_details.cached_tokens, streamed and not. - Anthropic: the final usage splits input_tokens from cache_read_input_tokens (message_start can only know the whole prompt). - llama.cpp's `timings` block on the last chunk and on a whole reply (prompt_n, cache_n, prompt/predicted ms and per second), from the engine's own clock: what llama.cpp's clients already show as speed. - GET /v1/status (with the API key when one is set): the model and its window, images, the APIs, whether a request is running, the last request's timings and the GPU and RAM from the telemetry, for a front-end that polls its server rather than guessing. An engine without a clock (MockEngine) reports no timings and 0 cached. Tests: serve/test_server.py UsageAndStatus.
architectds
added a commit
to architectds/collabosm
that referenced
this pull request
Sep 29, 2026
…load Measured on the A100-40G on 2026-09-29. Prefill on real text (code and prose) was 240-410 t/s: the GPU sat idle 72% of each prompt while Strata read its n-gram table, 16 random 4 KB O_DIRECT reads a token, from Colab's PersistentDisk (~2,500 reads/s whatever the queue depth). /content is overlayfs on a loop device that caches its backing file (dio=0). A buffered read caches the table twice and evicts itself, but one O_DIRECT pass leaves it in RAM once. strata_warm does that pass after every engine start (serve.sh), about 90 s in the background. - Prefill is now 905 / 1,202 / 1,301 / 1,376 t/s at 4K / 18K / 28K / 59K. - Decode went from 42 to 61 t/s (66 after a 34K prompt, 72 on a next turn). - STRATA_PREFILL_TIMING puts the rest in Strata's GPU kernels: at 33K, QSA attention 36%, GDN 17%, the MoE 27% (waiting on PCIe 3%). - 16K prompt chunks were measured slower below 30K, so the recipe keeps Strata's own flags. STRATA_ENGINE_ARGS can still change them per recipe. The recipe now runs the fork's r2 engine, whose image encoder read a test picture right. It pins fd1e7d9, where Strata's server reports cached prompt tokens, llama.cpp timings and GET /v1/status (sent upstream as Niko1221/Strata#107). The speed rail, /props image upload and the idle clock now work as they do for ExLlamaV3. strata_weights fetches the GGUF and the image encoder at the model's pinned revision with Xet, with setup.py's .done marks: 84.5 GB in 413 s, where setup.py's one stream per file took most of a 22-minute start. READY now takes 19 min 46 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
|
Thanks @architectds - merged into the 0.1.23 branch, after #122 (both added GET routes; kept |
This was referenced Sep 29, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The conversation cache reuses prompt prefixes, and
/metricsalready records the reuse per request. But the chat APIs don't report it, so clients show a cached prompt as 0% cached. Front-ends that read speed from llama.cpp'stimingsshow nothing.This PR adds the missing fields and changes nothing else in the replies:
OpenAI (
/v1/chat/completions):usage.prompt_tokens_details.cached_tokensis the part of the prompt the conversation cache held. It's on the last streamed chunk and on a whole reply.Anthropic (
/v1/messages): the finalusagesplitsinput_tokensfromcache_read_input_tokens, the way Anthropic's API counts them. That's the stream'smessage_deltaand a whole reply.message_startstill carries the whole prompt, the only count known before the engine reads it.llama.cpp's
timingsgo on the last chunk and on a whole reply:prompt_n,cache_n,prompt_ms,prompt_per_second,predicted_n,predicted_ms,predicted_per_secondand the per-token ms. They come from the engine's own clock, the same numbers/metricsshows.GET /v1/statusis a small read-only summary for front-ends that poll their server rather than guess. It's behind the API key when one is set. It gives:The Monitor's
/metricsis unchanged.An engine without a clock (the mock engine) reports no
timingsand 0 cached tokens.Tested:
python -m unittest serve.test_server serve.test_mcppasses. The newUsageAndStatusclass covers both APIs, streaming,/v1/statusand the no-clock case.cached_tokens: 2784, withtimingscache_n: 2784, prompt_n: 20;/v1/statusfor its speed panel and for image upload.