Repository navigation
spec: DFlash2 drafters (port of llama.cpp#27816) + HIP top-k for large rows - #2
Merged
Merged
Conversation
… for large rows Ports upstream DFlash2 support (b10f9ca, "spec : add DFlash2 support (local convolution + candidate selector)") onto this fork's DFlash code: - dflash.cpp: DFlash2 hparams (block/conv/selector keys), per-layer local convolutions around attention and FFN, the target's logit scale/softcap on the draft logits, and the selector lattice (top-k candidates + pairwise transition scores per block position) written to the pre-norm output (this fork's name for upstream's h_nextn) - speculative.cpp: the greedy lattice walk (p_min on the softmax of the winning score, n_min); block tokens request no logits for DFlash2 - llama-context: DFlash2 graph node budget - ggml-cuda: upstream's HIP radix top-k, so TOP_K over a 248k vocab stays on the GPU instead of falling back to the CPU (test-backend-ops TOP_K 445/445 on gfx1151, rows up to 524288) - common: keep bounded recurrent-state rollback (n_rs_seq) for draft-dflash, as upstream does; without it every partial accept restores a checkpoint and replays the accepted tokens M-RoPE handling from the upstream commit is not needed here: the Qwen3.8 drafter has one 64-wide section, identical to NEOX with its positions. Qwen3.8-27B (Hadamard-rotated Q4_0 + W4A4 prompt path) + Qwen3.8-27B-DFlash2 q8_0 on gfx1151, -ctxcp 0, --spec-draft-p-min 0, 256 tokens greedy: mean acceptance length 6.71 code / 4.11 prose, matching upstream (6.71 / 4.25). Greedy output identical to no-draft on code and short prompts; prose flips one near-tie word at char 470 (batched verification rounding). Co-authored-by: Zihan Zhang <tiancaizhangdaxian@sjtu.edu.cn> Co-authored-by: Jian Chen <jianchen0311@gmail.com> Co-authored-by: Xuan-Son Nguyen <son@huggingface.co> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports upstream DFlash2 (ggml-org/llama.cpp#27816) onto this fork's DFlash code, so a single ROCm server can use both the Hadamard W4A4 prompt path (#1) and DFlash2 drafting.
What changes
src/models/dflash.cpp):common/speculative.cpp): a greedy pass over the lattice, withp_minandn_min.TOP_Kover the 248k vocab fell back to the CPU, and code decode ran at 11 tok/s instead of 28.common/common.cpp): draft-dflash keeps bounded recurrent-state rollback (n_rs_seq), as upstream does. Without it, every partial accept restores a checkpoint and replays the accepted tokens.Checks
test-backend-ops -o TOP_K -b ROCm0: 445/445, rows up to 524288. CONCAT and FILL also pass.Acceptance: Qwen3.8-27B + Qwen3.8-27B-DFlash2 q8_0, greedy, 256 tokens.
Greedy text vs no drafter: code and short prompts are identical. The prose prompt flips one near-tie word at character 470; that is batched-verification rounding.
Speed on gfx1151, with the rotated H32 file and the W4A4 prompt path. The server runs with
-ctxcp 0 --spec-draft-n-max 16 --spec-draft-p-min 0. Decode is in tok/s, best of 3; the box is shared, so treat these as indicative.Notes
--spec-draft-p-minis 0.75, which cuts DFlash2 drafts short (5.4 instead of 6.7 tokens per step). Pass--spec-draft-p-min 0.🤖 Generated with Claude Code