Skip to content

/metrics: expose the speculative draft counts (offered / accepted) for llm-visuals #457

Closed

Description

@blaakstone

Hi, I run Strata 0.1.34 with Qwen3.8-Flash-Next IQ3_S at 262K on my RTX 5090 behind llama-swap (119-135 tok/s decode, ~5,000 tok/s prefill, 0.8 s to resume a 71K conversation), thank you for it.

I monitor it live with llm-visuals, a terminal dashboard that polls Strata's /metrics every second (its Strata adapter is on the branch feature/strata). Throughput, context, layers and requests work; the speculative decoding panel stays empty: "Strata does not report draft acceptance".

The server already has the MTP draft counts: serve/server.py reads them from the engine's DONE line (self.last.update(drafts_accepted=..., drafts_offered=...)), returns them in the response timings as draft_n / draft_n_accepted (#83), and the server log prints them ("drafts accepted 601 of 1181").

They are missing from GET /metrics: not in live, not in the requests history, not in totals (checked on 0.1.34). From the response timings I only get the figure once a request has finished; what I would like to see in llm-visuals is the acceptance while the model generates.

Proposal (additive, no change to existing fields):

  • in live, while generating: "drafts_offered" and "drafts_accepted" so far for the current request (if the engine can report them before DONE);
  • in each requests entry: the same two counts for the finished request (null when the engine did not report them, as draft_n is omitted today);
  • in totals: running sums of both.

Activity

  1. Niko1221 commented on Oct 2, 2026

    @Niko1221
    Owner

    Added in 0.1.35: each /metrics request entry has drafts_offered and drafts_accepted (null when the engine didn't report them), and totals keeps their running sums. Live counts during a request need an engine protocol change, so they're for later. Thanks for the idea!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions