Skip to content

Skip minimum speculative decoding probability for MTP and DFlash2 drafts with probabilistic draft sampling - #31294

Open
praneshgo wants to merge 1 commit into
ggml-org:masterfrom
praneshgo:pgonegandla/spec-p-min-rejection
Open

Skip minimum speculative decoding probability for MTP and DFlash2 drafts with probabilistic draft sampling#31294
praneshgo wants to merge 1 commit into
ggml-org:masterfrom
praneshgo:pgonegandla/spec-p-min-rejection

Conversation

@praneshgo

@praneshgo praneshgo commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview

Follow-up to discussion from #30663 (comment).

--spec-draft-p-min stops a draft once the draft's top probability drops below p_min. That fits greedy drafts, where the target keeps a drafted token only if it picks the same one. With --spec-draft-sampling probabilistic the draft is sampled and verified by rejection sampling: a sampled token is accepted with probability min(1, p/q), so the acceptance at a position is Σ min(p, q), which the draft's top probability does not tell. A flat but well-matched draft gets cut although the target would accept most of its tokens, and the cut fires more as the temperature rises.

This PR skips the p_min cut for sampled MTP and DFlash2 drafts. Greedy drafting and temp-0 requests are unchanged. The output distribution is the same either way, since the cut does not depend on the drawn token; only throughput changes.

The help text now says p_min is not applied to sampled draft-mtp and draft-dflash drafts.

Performance

Master and this PR were both run with --spec-draft-sampling probabilistic and --spec-draft-p-min 0.75. Same method as #27694: 11 temperatures, 12 prompts x 256 tokens, 3 seeds, arms interleaved. Temp 0 gave identical text in every config.

Δ is this PR vs master at each temperature, over the 12 prompts x 3 seeds:

  • tokens/target pass: generated tokens / target forward passes = generated / (generated - accepted draft tokens). Each target pass emits one token plus the draft tokens it accepts.
  • t/s: generation throughput reported by the server (predicted_per_second, prompt processing excluded), mean per request.

Tokens/target pass shows the acceptance effect alone; t/s also includes the cost of the extra drafted tokens and the larger verify batch.

Models: Qwen3.8-27B Q4_K_M with its DFlash2 draft and the ggml-org MTP head (Q8_0); Muse-Glimmer-30B Q4_K_M with z-lab's DFlash2 draft.

Summary

Δ t/s vs master, mean of per-temp values, temps 0.1-1.0 / 0.5-1.0:

Config n_max RTX 5090 DGX Spark
Qwen3.8-27B + DFlash2 3 +32.5% / +43.3% +12.0% / +16.9%
Muse-Glimmer-30B + DFlash2 3 +20.2% / +26.5% +12.7% / +17.7%
Qwen3.8-27B + MTP 3 +28.5% / +36.7% +9.7% / +14.0%
Qwen3.8-27B + MTP 8 +19.7% / +22.3% -

RTX 5090

RTX 5090, Qwen3.8-27B + DFlash2, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% -0.8%
0.1 +1.2% +6.8%
0.2 +3.4% +14.6%
0.3 +4.3% +20.0%
0.4 +7.9% +24.1%
0.5 +10.8% +30.6%
0.6 +12.4% +33.8%
0.7 +18.0% +47.4%
0.8 +20.1% +45.6%
0.9 +24.4% +51.8%
1.0 +27.3% +50.6%
mean 0.1-1.0 +13.0% +32.5%
mean 0.5-1.0 +18.8% +43.3%

RTX 5090, Muse-Glimmer-30B + DFlash2, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% +0.9%
0.1 +1.4% +4.0%
0.2 +7.9% +14.3%
0.3 +1.1% +9.5%
0.4 +8.6% +15.3%
0.5 +9.3% +22.9%
0.6 +11.4% +24.2%
0.7 +15.3% +26.7%
0.8 +18.6% +25.1%
0.9 +20.5% +25.8%
1.0 +26.8% +34.0%
mean 0.1-1.0 +12.1% +20.2%
mean 0.5-1.0 +17.0% +26.5%

RTX 5090, Qwen3.8-27B + MTP, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% -0.2%
0.1 +1.6% +7.7%
0.2 +3.0% +12.8%
0.3 +7.8% +20.5%
0.4 +8.1% +23.7%
0.5 +12.8% +29.6%
0.6 +13.4% +30.9%
0.7 +16.6% +34.6%
0.8 +21.8% +39.6%
0.9 +26.1% +43.7%
1.0 +31.6% +42.0%
mean 0.1-1.0 +14.3% +28.5%
mean 0.5-1.0 +20.4% +36.7%

RTX 5090, Qwen3.8-27B + MTP, n_max 8

temp Δ tokens/target pass Δ t/s
0.0 0.0% -0.0%
0.1 +2.5% +11.1%
0.2 +3.6% +14.0%
0.3 +8.2% +18.1%
0.4 +10.3% +19.8%
0.5 +15.8% +20.0%
0.6 +21.0% +23.5%
0.7 +21.5% +20.4%
0.8 +28.3% +23.4%
0.9 +33.9% +24.4%
1.0 +34.7% +22.0%
mean 0.1-1.0 +18.0% +19.7%
mean 0.5-1.0 +25.9% +22.3%

On the RTX 5090 part of the gain comes from CUDA graphs: with the cut the verify batch size changes from step to step and the graph keeps being re-captured. With GGML_CUDA_DISABLE_GRAPHS=1 (Qwen3.8 + DFlash2, temp 1.0) the gain is +20.9% t/s for +27.5% tokens per target pass. On DGX Spark t/s follows tokens per pass.

DGX Spark

DGX Spark, Qwen3.8-27B + DFlash2, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% +1.6%
0.1 +1.5% -0.1%
0.2 +2.9% +4.5%
0.3 +6.0% +7.2%
0.4 +6.3% +7.1%
0.5 +9.5% +10.3%
0.6 +11.7% +11.8%
0.7 +15.7% +15.3%
0.8 +19.5% +19.3%
0.9 +21.1% +19.6%
1.0 +26.9% +24.9%
mean 0.1-1.0 +12.1% +12.0%
mean 0.5-1.0 +17.4% +16.9%

DGX Spark, Muse-Glimmer-30B + DFlash2, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% -0.2%
0.1 +0.9% +2.8%
0.2 +0.3% +0.7%
0.3 +4.2% +5.8%
0.4 +8.5% +11.1%
0.5 +8.0% +10.8%
0.6 +11.1% +13.0%
0.7 +13.1% +14.0%
0.8 +14.4% +15.4%
0.9 +30.7% +30.7%
1.0 +21.9% +22.5%
mean 0.1-1.0 +11.3% +12.7%
mean 0.5-1.0 +16.5% +17.7%

DGX Spark, Qwen3.8-27B + MTP, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% -0.0%
0.1 +1.7% +1.5%
0.2 +1.6% +1.5%
0.3 +5.9% +4.8%
0.4 +8.2% +6.0%
0.5 +13.1% +9.9%
0.6 +13.7% +9.0%
0.7 +17.7% +12.6%
0.8 +21.1% +15.4%
0.9 +24.0% +15.9%
1.0 +31.2% +21.0%
mean 0.1-1.0 +13.8% +9.7%
mean 0.5-1.0 +20.1% +14.0%

SPEED-Bench

Qwen3.8-27B + DFlash2, n_max 3, temp 1.0, OSL 512, concurrency 1, master and this PR both with --spec-draft-sampling probabilistic --spec-draft-p-min 0.75. coding = --category coding, qualitative = --category all --limit 15. Dataset rows that only hold the placeholder text "FULL BENCHMARK DATA SHOULD BE FETCHED FROM THE SOURCE USING SPECDEC_BENCH" are excluded, leaving 70 coding and 82 qualitative prompts. t/s = generated tokens / decode time over all requests.

RTX 5090

benchmark Δ tokens/target pass Δ t/s
coding +48.0% +68.3%
qualitative +42.6% +60.8%

DGX Spark

benchmark Δ tokens/target pass Δ t/s
coding +47.5% +46.7%
qualitative +42.1% +41.1%

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: AI was used to partially assist while making the code edits and to understand the contexts of the code base.

Pranesh Gonegandla
spec : skip p_min for sampled MTP and DFlash2 drafts
091b9fd 
Under rejection sampling the acceptance at a position is 1 - TV(p, q),
which the draft's top probability does not tell, so the p_min cutoff
dropped draft tokens the target would have accepted. Skip it when
MTP or DFlash2 samples the draft. draft-simple keeps it: each extra
draft token costs a full draft model pass.
@github-actions github-actions Bot added documentation Improvements or additions to documentation server labels Oct 28, 2026
@praneshgo

praneshgo commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Adding the performance section as a separate comment for better visibility.

Performance

Master and this PR were both run with --spec-draft-sampling probabilistic and --spec-draft-p-min 0.75. Same method as #27694: 11 temperatures, 12 prompts x 256 tokens, 3 seeds, arms interleaved. Temp 0 gave identical text in every config.

Δ is this PR vs master at each temperature, over the 12 prompts x 3 seeds:

  • tokens/target pass: generated tokens / target forward passes = generated / (generated - accepted draft tokens). Each target pass emits one token plus the draft tokens it accepts.
  • t/s: generation throughput reported by the server (predicted_per_second, prompt processing excluded), mean per request.

Tokens/target pass shows the acceptance effect alone; t/s also includes the cost of the extra drafted tokens and the larger verify batch.

Models: Qwen3.8-27B Q4_K_M with its DFlash2 draft and the ggml-org MTP head (Q8_0); Muse-Glimmer-30B Q4_K_M with z-lab's DFlash2 draft.

Summary

Δ t/s vs master, mean of per-temp values, temps 0.1-1.0 / 0.5-1.0:

Config n_max RTX 5090 DGX Spark
Qwen3.8-27B + DFlash2 3 +32.5% / +43.3% +12.0% / +16.9%
Muse-Glimmer-30B + DFlash2 3 +20.2% / +26.5% +12.7% / +17.7%
Qwen3.8-27B + MTP 3 +28.5% / +36.7% +9.7% / +14.0%
Qwen3.8-27B + MTP 8 +19.7% / +22.3% -

RTX 5090

RTX 5090, Qwen3.8-27B + DFlash2, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% -0.8%
0.1 +1.2% +6.8%
0.2 +3.4% +14.6%
0.3 +4.3% +20.0%
0.4 +7.9% +24.1%
0.5 +10.8% +30.6%
0.6 +12.4% +33.8%
0.7 +18.0% +47.4%
0.8 +20.1% +45.6%
0.9 +24.4% +51.8%
1.0 +27.3% +50.6%
mean 0.1-1.0 +13.0% +32.5%
mean 0.5-1.0 +18.8% +43.3%

RTX 5090, Muse-Glimmer-30B + DFlash2, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% +0.9%
0.1 +1.4% +4.0%
0.2 +7.9% +14.3%
0.3 +1.1% +9.5%
0.4 +8.6% +15.3%
0.5 +9.3% +22.9%
0.6 +11.4% +24.2%
0.7 +15.3% +26.7%
0.8 +18.6% +25.1%
0.9 +20.5% +25.8%
1.0 +26.8% +34.0%
mean 0.1-1.0 +12.1% +20.2%
mean 0.5-1.0 +17.0% +26.5%

RTX 5090, Qwen3.8-27B + MTP, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% -0.2%
0.1 +1.6% +7.7%
0.2 +3.0% +12.8%
0.3 +7.8% +20.5%
0.4 +8.1% +23.7%
0.5 +12.8% +29.6%
0.6 +13.4% +30.9%
0.7 +16.6% +34.6%
0.8 +21.8% +39.6%
0.9 +26.1% +43.7%
1.0 +31.6% +42.0%
mean 0.1-1.0 +14.3% +28.5%
mean 0.5-1.0 +20.4% +36.7%

RTX 5090, Qwen3.8-27B + MTP, n_max 8

temp Δ tokens/target pass Δ t/s
0.0 0.0% -0.0%
0.1 +2.5% +11.1%
0.2 +3.6% +14.0%
0.3 +8.2% +18.1%
0.4 +10.3% +19.8%
0.5 +15.8% +20.0%
0.6 +21.0% +23.5%
0.7 +21.5% +20.4%
0.8 +28.3% +23.4%
0.9 +33.9% +24.4%
1.0 +34.7% +22.0%
mean 0.1-1.0 +18.0% +19.7%
mean 0.5-1.0 +25.9% +22.3%

On the RTX 5090 part of the gain comes from CUDA graphs: with the cut the verify batch size changes from step to step and the graph keeps being re-captured. With GGML_CUDA_DISABLE_GRAPHS=1 (Qwen3.8 + DFlash2, temp 1.0) the gain is +20.9% t/s for +27.5% tokens per target pass. On DGX Spark t/s follows tokens per pass.

DGX Spark

DGX Spark, Qwen3.8-27B + DFlash2, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% +1.6%
0.1 +1.5% -0.1%
0.2 +2.9% +4.5%
0.3 +6.0% +7.2%
0.4 +6.3% +7.1%
0.5 +9.5% +10.3%
0.6 +11.7% +11.8%
0.7 +15.7% +15.3%
0.8 +19.5% +19.3%
0.9 +21.1% +19.6%
1.0 +26.9% +24.9%
mean 0.1-1.0 +12.1% +12.0%
mean 0.5-1.0 +17.4% +16.9%

DGX Spark, Muse-Glimmer-30B + DFlash2, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% -0.2%
0.1 +0.9% +2.8%
0.2 +0.3% +0.7%
0.3 +4.2% +5.8%
0.4 +8.5% +11.1%
0.5 +8.0% +10.8%
0.6 +11.1% +13.0%
0.7 +13.1% +14.0%
0.8 +14.4% +15.4%
0.9 +30.7% +30.7%
1.0 +21.9% +22.5%
mean 0.1-1.0 +11.3% +12.7%
mean 0.5-1.0 +16.5% +17.7%

DGX Spark, Qwen3.8-27B + MTP, n_max 3

temp Δ tokens/target pass Δ t/s
0.0 0.0% -0.0%
0.1 +1.7% +1.5%
0.2 +1.6% +1.5%
0.3 +5.9% +4.8%
0.4 +8.2% +6.0%
0.5 +13.1% +9.9%
0.6 +13.7% +9.0%
0.7 +17.7% +12.6%
0.8 +21.1% +15.4%
0.9 +24.0% +15.9%
1.0 +31.2% +21.0%
mean 0.1-1.0 +13.8% +9.7%
mean 0.5-1.0 +20.1% +14.0%

SPEED-Bench

Qwen3.8-27B + DFlash2, n_max 3, temp 1.0, OSL 512, concurrency 1, master and this PR both with --spec-draft-sampling probabilistic --spec-draft-p-min 0.75. coding = --category coding, qualitative = --category all --limit 15. Dataset rows that only hold the placeholder text "FULL BENCHMARK DATA SHOULD BE FETCHED FROM THE SOURCE USING SPECDEC_BENCH" are excluded, leaving 70 coding and 82 qualitative prompts. t/s = generated tokens / decode time over all requests.

RTX 5090

benchmark Δ tokens/target pass Δ t/s
coding +48.0% +68.3%
qualitative +42.6% +60.8%

DGX Spark

benchmark Δ tokens/target pass Δ t/s
coding +47.5% +46.7%
qualitative +42.1% +41.1%

@praneshgo praneshgo mentioned this pull request Oct 28, 2026
Merged
@praneshgo
praneshgo marked this pull request as ready for review October 28, 2026 06:50
@praneshgo
praneshgo requested review from a team and ngxson as code owners October 28, 2026 06:50
@praneshgo

praneshgo commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

@ruixiang63, @gaugarg-nv, @ORippler can you please review this PR? Thanks.

@praneshgo praneshgo changed the title Skip minimum speculative decoding probability for sampled MTP and DFlash2 drafts Skip minimum speculative decoding probability for MTP and DFlash2 drafts with probabilistic draft sampling Oct 28, 2026
@ruixiang63 ruixiang63 self-assigned this Oct 28, 2026
sourenaraya added a commit to sourenaraya/llama.cpp that referenced this pull request Oct 28, 2026
@sourenaraya
INFERENCE.md - 8642541 base (+28), ggml-org#31294 in, ggml-org#30792/g…
266879d 
…gml-org#30804 out

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ngxson ngxson Awaiting requested review from ngxson ngxson is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

@ruixiang63 ruixiang63

Labels

documentation Improvements or additions to documentation server

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@praneshgo @ruixiang63