Repository navigation
server : add --spec-draft-max-slots to stop drafting under concurrency - #151
Merged
Merged
Conversation
Speculative decoding wins for a single stream but loses once several slots generate together: the batch of all slots already keeps the GPU busy, and every draft adds a verify row per slot. With --spec-draft-max-slots N the server starts no new drafts while more than N slots are generating; a draft already in flight is still used. Verification is unchanged, so outputs are unchanged. Default 0 keeps the current behaviour. gfx1151 (Ryzen AI Max+ 395), Qwen3.8-Flash-Next ROCmFP4-FAST with its MTP head (n-max 3), -np 8 -kvu, aggregate t/s at 1/2/4/8 concurrent streams: MTP always 56.5 / 57.9 / 64.8 / 69.9 max-slots 2 56.5 / 58.8 / 66.5 / 76.9 no MTP 38.7 / 54.0 / 67.5 / 78.3 Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Speculative decoding helps one stream but costs throughput once several slots generate together. The batch of all
slots already keeps the GPU busy, and each draft adds verify rows for every slot. On Qwen3.8-Flash-Next with its MTP
head, MTP at 8 concurrent streams runs about 11% below plain decode (69.9 vs 78.3 t/s aggregate).
This adds
--spec-draft-max-slots N(envLLAMA_ARG_SPEC_DRAFT_MAX_SLOTS). While more than N slots are generating,the server starts no new drafts. A draft already in flight is still verified and used. The verification path is
untouched, so accepted tokens are chosen the same way. The default 0 keeps the current behaviour (no limit).
Changes are server-side only (
common/arg.cpp,common/common.h,tools/server/server-context.cpp, 20 lines).No kernel, graph or model code is touched.
Additional information
Prior art / attribution: vLLM has the same mechanism as
speculative_disable_by_batch_sizein itsspeculative decoding config (skip drafting once the running batch exceeds a size). This PR adds the equivalent
for llama-server; the threshold here was measured on this model and hardware.
Single-stream speculative tuning is unaffected. This only adds a concurrency cutoff.
Written by Claude Opus 5.5 (Claude Code) as part of a Qwen3.8-Flash-Next decode-throughput investigation. Benchmarks were run on a Ryzen AI Max+ 395.