Repository navigation
Conversation
…iends The engine already reports per-request prompt/decode timings, reuse and draft counts on its DONE line, but they stayed in server.py. Emit them in llama-server's dialect (prompt_n, prompt_ms, prompt_per_second, predicted_n, predicted_ms, predicted_per_second, plus optional cache_n and draft_n/draft_n_accepted) as a top-level timings object on the final streamed chunk, the non-streaming response, and the Anthropic message_delta - everywhere the usage object already goes. Proxies that chart per-request prefill/decode speed read exactly this shape; llama-swap's activity log, for one, shows unknown rates without it and prefill/decode/cache/draft columns with it. Verified end to end through llama-swap v257: a 10.6k-token request charts PP 467 t/s, decode 19.7 t/s, draft acceptance 32/59, and multi-turn requests show their reused tokens in the cache column.
Owner
Owner
Niko1221
added a commit
that referenced
this pull request
Sep 29, 2026
…ccepted), from PR #83 Co-Authored-By: mikicvi <mikicvi@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
MistVVK
pushed a commit
to MistVVK/XeStrata
that referenced
this pull request
Oct 3, 2026
…ccepted), from PR Niko1221#83 Co-Authored-By: mikicvi <mikicvi@users.noreply.github.com>
Hugua700
pushed a commit
to Hugua700/Strata
that referenced
this pull request
Oct 6, 2026
…ccepted), from PR Niko1221#83 Co-Authored-By: mikicvi <mikicvi@users.noreply.github.com>
anon761
pushed a commit
to anon761/Strata-upstream
that referenced
this pull request
Oct 6, 2026
…ccepted), from PR Niko1221#83 Co-Authored-By: mikicvi <mikicvi@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The engine already reports per-request prompt/decode timings, conversation-cache reuse and draft counts on its
DONEline, but those numbers stayed insideserver.py. This emits them everywhere theusageobject already goes, in llama-server's dialect — a top-leveltimingsobject on the final streamed chunk, the non-streaming response, and the Anthropicmessage_delta:(
cache_n/draft_*only when the engine reported them.)Why: proxies and monitors that chart per-request prefill/decode speed read exactly this shape — llama-swap's activity log (the llama-server, vLLM and TabbyAPI dialects it accepts), and anything else that already understands llama.cpp responses. Without it, Strata requests show token counts but
unknownrates. Nothing else changes: fields are additive, and clients that don't knowtimingsignore it.Verified end to end through llama-swap v257: streaming and non-streaming
/v1/chat/completions, and/v1/messages, all produce activity rows with prefill/decode rates, cached-token counts on reused prompts, and draft acceptance — e.g. a 10.6k-token request charts PP 467 t/s, decode 19.7 t/s, drafts 32 of 59 accepted.