Skip to content

serve: llama-server-style timings on responses (prefill/decode rates for llama-swap & friends) - #83

Closed
mikicvi wants to merge 2 commits into
Niko1221:mainfrom
mikicvi:llama-timings
Closed

mikicvi wants to merge 2 commits into
Niko1221:mainfrom
mikicvi:llama-timings

Conversation

@mikicvi

@mikicvi mikicvi commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

The engine already reports per-request prompt/decode timings, conversation-cache reuse and draft counts on its DONE line, but those numbers stayed inside server.py. This emits them everywhere the usage object already goes, in llama-server's dialect — a top-level timings object on the final streamed chunk, the non-streaming response, and the Anthropic message_delta:

"timings": {"prompt_n": 10597, "prompt_ms": 22683.0, "prompt_per_second": 467.3,
            "predicted_n": 64, "predicted_ms": 3248.0, "predicted_per_second": 19.7,
            "cache_n": 0, "draft_n": 59, "draft_n_accepted": 32}

(cache_n/draft_* only when the engine reported them.)

Why: proxies and monitors that chart per-request prefill/decode speed read exactly this shape — llama-swap's activity log (the llama-server, vLLM and TabbyAPI dialects it accepts), and anything else that already understands llama.cpp responses. Without it, Strata requests show token counts but unknown rates. Nothing else changes: fields are additive, and clients that don't know timings ignore it.

Verified end to end through llama-swap v257: streaming and non-streaming /v1/chat/completions, and /v1/messages, all produce activity rows with prefill/decode rates, cached-token counts on reused prompts, and draft acceptance — e.g. a 10.6k-token request charts PP 467 t/s, decode 19.7 t/s, drafts 32 of 59 accepted.

mikicvi and others added 2 commits September 28, 2026 19:37
…iends

The engine already reports per-request prompt/decode timings, reuse and
draft counts on its DONE line, but they stayed in server.py. Emit them in
llama-server's dialect (prompt_n, prompt_ms, prompt_per_second,
predicted_n, predicted_ms, predicted_per_second, plus optional cache_n
and draft_n/draft_n_accepted) as a top-level timings object on the final
streamed chunk, the non-streaming response, and the Anthropic
message_delta - everywhere the usage object already goes.

Proxies that chart per-request prefill/decode speed read exactly this
shape; llama-swap's activity log, for one, shows unknown rates without
it and prefill/decode/cache/draft columns with it. Verified end to end
through llama-swap v257: a 10.6k-token request charts PP 467 t/s,
decode 19.7 t/s, draft acceptance 32/59, and multi-turn requests show
their reused tokens in the cache column.
@Niko1221

Copy link
Copy Markdown
Owner

Thanks @mikicvi - #107 covers the same timings block (plus cached tokens and /v1/status), so I merged that one and added your draft fields (draft_n, draft_n_accepted) to it, credited in the commit. I'll close this one when 0.1.23 is out.

@Niko1221

Copy link
Copy Markdown
Owner

Closing: #107 (merged for 0.1.23) carries the same timings block, with your draft_n / draft_n_accepted fields added and credited. Thank you @mikicvi!

@Niko1221 Niko1221 closed this Sep 29, 2026
Niko1221 added a commit that referenced this pull request Sep 29, 2026
…ccepted), from PR #83

Co-Authored-By: mikicvi <mikicvi@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
MistVVK pushed a commit to MistVVK/XeStrata that referenced this pull request Oct 3, 2026
…ccepted), from PR Niko1221#83

Co-Authored-By: mikicvi <mikicvi@users.noreply.github.com>
Hugua700 pushed a commit to Hugua700/Strata that referenced this pull request Oct 6, 2026
…ccepted), from PR Niko1221#83

Co-Authored-By: mikicvi <mikicvi@users.noreply.github.com>
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
…ccepted), from PR Niko1221#83

Co-Authored-By: mikicvi <mikicvi@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants