Extend probabilistic sampling for DFlash2 drafts by praneshgo · Pull Request #30663 · ggml-org/llama.cpp

praneshgo · 2026-10-16T09:44:31Z

Overview

This PR is an extension of #27694. This PR adds probabilistic drafter for DFlash2 technique and reuses the same verify-accept-reject logic added in #27694 which is currently on upstream.

Adding some performance data taken over multiple HW running models from different families below:

Performance

RTX 5090

Qwen3.8-27B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	-0.20%
0.1	-0.43%	+0.97%
0.2	-0.44%	+0.44%
0.3	+0.72%	+0.04%
0.4	+2.76%	+2.04%
0.5	+0.13%	+0.34%
0.6	+3.10%	+4.46%
0.7	+2.16%	+2.07%
0.8	+2.08%	+1.75%
0.9	+3.82%	+3.72%
1.0	+4.33%	+4.20%

Muse-Glimmer-30B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	-0.34%
0.1	+1.54%	+1.76%
0.2	+2.82%	+4.12%
0.3	+2.44%	+3.20%
0.4	+3.16%	+3.78%
0.5	+3.79%	+1.40%
0.6	+0.91%	-0.61%
0.7	+0.82%	-3.08%
0.8	+3.28%	+3.68%
0.9	-0.16%	-1.13%
1.0	+11.86%	+12.33%

DGX Spark

Qwen3.8-27B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	+0.01%
0.1	+0.74%	+1.16%
0.2	-1.00%	-0.70%
0.3	-0.89%	-0.89%
0.4	+1.58%	+0.77%
0.5	+2.61%	+2.51%
0.6	+1.76%	+2.35%
0.7	+3.38%	+3.02%
0.8	+2.07%	+2.15%
0.9	+3.64%	+3.41%
1.0	+4.63%	+4.82%

Muse-Glimmer-30B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	+0.07%
0.1	+2.09%	+2.43%
0.2	+1.29%	+1.63%
0.3	+4.44%	+3.99%
0.4	-1.34%	+2.15%
0.5	+4.19%	-0.46%
0.6	+1.03%	+0.96%
0.7	+0.44%	+0.55%
0.8	+3.53%	+4.10%
0.9	+5.72%	+6.26%
1.0	+3.12%	+3.56%

RTX Pro 6000

Qwen3.8-27B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	-2.26%
0.1	-0.05%	-0.01%
0.2	-0.80%	-0.48%
0.3	+1.23%	+1.51%
0.4	+1.57%	+1.51%
0.5	-0.25%	-0.36%
0.6	+3.14%	+2.48%
0.7	+2.56%	+2.67%
0.8	+3.46%	+3.23%
0.9	+4.07%	+4.12%
1.0	+3.89%	+4.52%

Muse-Glimmer-30B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	-0.66%
0.1	+3.37%	+3.39%
0.2	+2.49%	+2.36%
0.3	+3.10%	+2.15%
0.4	+5.39%	+1.30%
0.5	-1.76%	+1.92%
0.6	+3.10%	+3.03%
0.7	+1.59%	+0.74%
0.8	+4.04%	+3.52%
0.9	+1.56%	+2.09%
1.0	+7.05%	+7.09%

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: AI was used to partially assist while making the code edits and to understand the contexts of the code base.

praneshgo · 2026-10-16T10:24:31Z

Adding the performance section as a separate comment for better visibility

Performance

RTX 5090

Qwen3.8-27B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	-0.20%
0.1	-0.43%	+0.97%
0.2	-0.44%	+0.44%
0.3	+0.72%	+0.04%
0.4	+2.76%	+2.04%
0.5	+0.13%	+0.34%
0.6	+3.10%	+4.46%
0.7	+2.16%	+2.07%
0.8	+2.08%	+1.75%
0.9	+3.82%	+3.72%
1.0	+4.33%	+4.20%

Muse-Glimmer-30B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	-0.34%
0.1	+1.54%	+1.76%
0.2	+2.82%	+4.12%
0.3	+2.44%	+3.20%
0.4	+3.16%	+3.78%
0.5	+3.79%	+1.40%
0.6	+0.91%	-0.61%
0.7	+0.82%	-3.08%
0.8	+3.28%	+3.68%
0.9	-0.16%	-1.13%
1.0	+11.86%	+12.33%

DGX Spark

Qwen3.8-27B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	+0.01%
0.1	+0.74%	+1.16%
0.2	-1.00%	-0.70%
0.3	-0.89%	-0.89%
0.4	+1.58%	+0.77%
0.5	+2.61%	+2.51%
0.6	+1.76%	+2.35%
0.7	+3.38%	+3.02%
0.8	+2.07%	+2.15%
0.9	+3.64%	+3.41%
1.0	+4.63%	+4.82%

Muse-Glimmer-30B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	+0.07%
0.1	+2.09%	+2.43%
0.2	+1.29%	+1.63%
0.3	+4.44%	+3.99%
0.4	-1.34%	+2.15%
0.5	+4.19%	-0.46%
0.6	+1.03%	+0.96%
0.7	+0.44%	+0.55%
0.8	+3.53%	+4.10%
0.9	+5.72%	+6.26%
1.0	+3.12%	+3.56%

RTX Pro 6000

Qwen3.8-27B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	-2.26%
0.1	-0.05%	-0.01%
0.2	-0.80%	-0.48%
0.3	+1.23%	+1.51%
0.4	+1.57%	+1.51%
0.5	-0.25%	-0.36%
0.6	+3.14%	+2.48%
0.7	+2.56%	+2.67%
0.8	+3.46%	+3.23%
0.9	+4.07%	+4.12%
1.0	+3.89%	+4.52%

Muse-Glimmer-30B

temp	Δ accepted len	Δ t/s
0.0	+0.00%	-0.66%
0.1	+3.37%	+3.39%
0.2	+2.49%	+2.36%
0.3	+3.10%	+2.15%
0.4	+5.39%	+1.30%
0.5	-1.76%	+1.92%
0.6	+3.10%	+3.03%
0.7	+1.59%	+0.74%
0.8	+4.04%	+3.52%
0.9	+1.56%	+2.09%
1.0	+7.05%	+7.09%

praneshgo · 2026-10-16T10:24:55Z

@gaugarg-nv, @ruixiang63 can you please take a look at this PR?

gaugarg-nv · 2026-10-17T12:41:25Z

The change looks good to me.

ggerganov · 2026-10-17T13:12:15Z

I just wonder whether we should first merge the sampling tests that @danbev is working on: #30213. Would give some extra confidence if things are correct.

danbev · 2026-10-18T15:20:10Z

I'll revisit #30213 tomorrow, other things came in the way last week.

ruixiang63 · 2026-10-20T16:40:28Z

Do we have DFlash2 model for Muse-Glimmer-30B? @praneshgo

praneshgo · 2026-10-20T16:55:46Z

Do we have DFlash2 model for Muse-Glimmer-30B? @praneshgo

Yes, there is a DFlash2 model for Muse-Glimmer-30B. I found the drafter model at https://huggingface.co/z-lab/Muse-Glimmer-30B-DFlash2-GGUF and the corresponding target model is at https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF

ruixiang63 · 2026-10-20T17:30:04Z

                        for (int32_t k = 0; k < top_k; ++k) {
-                            sum += std::exp(scores[k] - scores[predecessor]);
+                            probs[k] = std::exp((scores[k] - scores[predecessor]) / temp);
+                            sum += probs[k];
                        }
                        if (1.0f / sum < params.p_min) {
                            break;
                        }
+                        for (int32_t k = 0; k < top_k; ++k) {
+                            probs[k] /= sum;
+                        }


I am thinking about params.p_min here: it still truncates on the draft’s max probability, which was a good proxy for greedy acceptance. Under rejection sampling the expected acceptance is Σ_x min(p(x), q(x)) = 1 − TV(p, q), which doesn’t depend on how peaked q is (draft token probability), so positions cut off by p_min could well have been accepted by the target. Ideally the cutoff would be an estimate of 1 − TV(p, q), but we don’t have p (target verified token probability) at draft time. Should we disable p_min when use rejection sampling?

Good point, I agree. Under rejection sampling the acceptance at a position is 1 − TV(p, q), and the draft's top probability doesn't say much about that, so p_min can cut tokens the target would have accepted.

For this PR:

p_min is 0 by default, so it only kicks in when it is set. All the numbers in the PR were measured with it not set.

draft-simple and draft-mtp already apply p_min the same way in probabilistic mode (top probability of the draft distribution at the target's temperature), so DFlash2 follows the existing behavior.

Instead of changing it for DFlash2 alone here, I think p_min under rejection sampling should be handled for all three drafters together. draft-simple and draft-mtp run one draft forward pass per token, so stopping early also saves draft compute. Dropping the cutoff trades that saving for more tokens the target may accept. Finding out which side wins needs measuring, which I can take up in a follow-up PR.

Please add a follow-up for this.

PR #31294 addresses the above discussion.

anskumar01 · 2026-10-22T14:17:57Z

It is approved. Can it be merged now?

praneshgo · 2026-10-22T14:42:13Z

@gaugarg-nv can you please help merge this PR if it looks okay? Thank you.

* spec : add probabilistic sampling for DFlash2 drafts * spec : apply review suggestions --------- Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>

spec : add probabilistic sampling for DFlash2 drafts

14626b5

gaugarg-nv marked this pull request as ready for review October 17, 2026 12:41

gaugarg-nv requested a review from a team as a code owner October 17, 2026 12:41

ggerganov self-assigned this Oct 17, 2026

ggerganov approved these changes Oct 20, 2026

View reviewed changes

ggerganov reviewed Oct 20, 2026

View reviewed changes

Comment thread common/speculative.h Outdated

danbev approved these changes Oct 20, 2026

View reviewed changes

Comment thread common/speculative.cpp Outdated

ruixiang63 reviewed Oct 20, 2026

View reviewed changes

ruixiang63 approved these changes Oct 20, 2026

View reviewed changes

gaugarg-nv approved these changes Oct 21, 2026

View reviewed changes

spec : apply review suggestions

6d90ee5

gaugarg-nv approved these changes Oct 23, 2026

View reviewed changes

gaugarg-nv merged commit 1ab09d6 into ggml-org:master Oct 23, 2026
16 of 19 checks passed

leehack mentioned this pull request Oct 23, 2026

Qualify post-v0.6.0 upstream correctness fixes and consumer compatibility leehack/llamadart-native#112

Open

6 tasks

praneshgo mentioned this pull request Oct 28, 2026

Skip minimum speculative decoding probability for MTP and DFlash2 drafts with probabilistic draft sampling #31294

Open

ggerganov mentioned this pull request Oct 28, 2026

llama.cpp : bump version to 0.8.0 #31309

Merged

Conversation

praneshgo commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

RTX 5090

DGX Spark

RTX Pro 6000

Requirements

Uh oh!

praneshgo commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

RTX 5090

DGX Spark

RTX Pro 6000

Uh oh!

praneshgo commented Oct 16, 2026

Uh oh!

gaugarg-nv commented Oct 17, 2026

Uh oh!

ggerganov commented Oct 17, 2026

Uh oh!

danbev commented Oct 18, 2026

Uh oh!

Uh oh!

Uh oh!

ruixiang63 commented Oct 20, 2026

Uh oh!

praneshgo commented Oct 20, 2026

Uh oh!

ruixiang63 Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

praneshgo Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

ruixiang63 Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

praneshgo Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

anskumar01 commented Oct 22, 2026

Uh oh!

praneshgo commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

praneshgo commented Oct 16, 2026 •

edited

Loading

praneshgo commented Oct 16, 2026 •

edited

Loading