Repository navigation
llama : build the k-pool sparse selection independently of the kq_mask shape - #31243
Merged
Merged
Conversation
…k shape qwen4exp (QSA) and glm5-next pick cache cells by index and no longer read the kq_mask values, but they still shaped the selection like the dense kq_mask. Build the selection as [n_kv, n_tokens] over the cache cells instead, and split it into the streams of the K view right before the attention. This lets the dense mask change its layout (e.g. per-sequence row lists) without touching the sparse layers.
ggerganov
approved these changes
Oct 27, 2026
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
sanmai
approved these changes
Oct 27, 2026
sanmai
left a comment
sanmai
left a comment
Contributor
There was a problem hiding this comment.
Smoke-tested with QFN and GLM 5.3.
ServeurpersoCom
approved these changes
Oct 27, 2026
ServeurpersoCom
left a comment
ServeurpersoCom
left a comment
Contributor
There was a problem hiding this comment.
LGTM once @sanmai's nits are in, the build_fused_top_k_mask cleanup can also go in a follow-up since it's outside the diff.
sanmai
approved these changes
Oct 27, 2026
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
…k shape (ggml-org#31243) * llama : build the k-pool sparse selection independently of the kq_mask shape qwen4exp (QSA) and glm5-next pick cache cells by index and no longer read the kq_mask values, but they still shaped the selection like the dense kq_mask. Build the selection as [n_kv, n_tokens] over the cache cells instead, and split it into the streams of the K view right before the attention. This lets the dense mask change its layout (e.g. per-sequence row lists) without touching the sparse layers. * improve comment Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * llama : drop the unused build_fused_top_k_mask args, review nits --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
PapyKahan
added a commit
to PapyKahan/llama.cpp
that referenced
this pull request
Oct 28, 2026
Upstream's version is kept wherever it does what the branch did; the branch's stays where upstream has none: - MoE cache: the constructor takes upstream's no_alloc flag beside the CPU share, buffers come from llama_memory_bufs::alloc_buf; phases and host mode as before. - build_moe_ffn: the branch's experts lambda (cache slots, host mode, shared batches) with upstream's DeepSeek-4.1 and Laguna changes. - qwen4exp: the selection is built over the cache cells as upstream now does (ggml-org#30965, ggml-org#31243), the branch keeps its query chunks (ggml-org#30815's earlier version); upstream's hc_mix outputs and merged tensors. - CUDA MMQ: the tile map of active MoE tiles also launches upstream's fused gate/up/GLU kernel (ggml-org#31188, ggml-org#29948); the fused paths zero the rows of negative ids as ggml_cuda_mul_mat_id does. - CPU MUL_MAT_ID: upstream's -1 skip (ggml-org#26631) and the branch's two-pass grouping agree, negative ids are skipped. - tests: the QSA chunk tests move with upstream's archs tests; a MUL_MAT_ID result marked by ggml_mul_mat_id_set_negative_ids runs on CUDA again, only unmarked ones are excluded ([TAG_MUL_MAT_ID_SKIP]). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P3vd8iuiuxiev7wMPyJmSf
iki007
pushed a commit
to iki007/llama.cpp
that referenced
this pull request
Oct 29, 2026
…d375e7) The -X theirs replay of 188 commits onto master 0ac1a7f dropped upstream's side where both changed a hunk. This commit sets the 4 files that differ to the one-time merge resolution (7d9d375e7), so the tip's tree equals it: ggml-sycl/dequantize.hpp (upstream's vectorized reorder stores, ggml-org#30976: the 32-thread q5_K get_rows kernel that getrows.cpp now launches, q4_K vec stores; our IQ3 reorder kernels keep their own layout), ggml-sycl/fattn.cpp (upstream's DPAS kernel for a q4_0 / q8_0 cache, ggml-org#31125, gets its own enum value and is tried after our XMX kernel), src/models/qwen4exp.cpp (ggml-org#31243 type_mask in our mask chain order), tests/test-backend-ops.cpp. Assisted-by: Claude Opus 5.5
yzhou51
pushed a commit
to yzhou51/llama.cpp
that referenced
this pull request
Oct 30, 2026
…k shape (ggml-org#31243) * llama : build the k-pool sparse selection independently of the kq_mask shape qwen4exp (QSA) and glm5-next pick cache cells by index and no longer read the kq_mask values, but they still shaped the selection like the dense kq_mask. Build the selection as [n_kv, n_tokens] over the cache cells instead, and split it into the streams of the K view right before the attention. This lets the dense mask change its layout (e.g. per-sequence row lists) without touching the sparse layers. * improve comment Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * llama : drop the unused build_fused_top_k_mask args, review nits --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
wispoffates
added a commit
to wispoffates/llama.cpp
that referenced
this pull request
Oct 31, 2026
…ack on overlap (placement-independent results) The fused router rounds differently from the unfused softmax / argsort / get_rows / norm chain. For batches of more than TOPK_MOE_ROWS_PER_BLOCK rows the fusion was refused whenever an output overlapped a live input, so the path -- and the bits -- depended on where the allocator placed the router logits. Upstream ggml-org#31243 shifted the Flash-Next layout: layers 15-26 flipped to fused in most ubatches (same-text KLD 0.0007 vs v5.33) and the first request after load differed from every later one (the compute buffer is laid out again after it). Now the kernel writes to pool memory and the results are copied into place: always fused. GGML_CUDA_TOPK_MOE_STAGE=0 restores the fallback. Assisted-by: Claude Opus 5.5 (Claude Code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
qwen4exp (QSA) and glm5-next pick cache cells by index through the k-pool indexer, and since #31009 neither reads the KQ mask values. The selection is still shaped like the dense KQ mask though (
build_fused_top_k_masktakes the row layout from it, and both models assert that the selection has the mask's shape), so the sparse layers stay tied to the layout of the dense mask.This PR builds the selection as
[n_kv, n_tokens]over the cache cells and splits it into the streams of the K view right before the attention, so the dense mask can change its layout without touching these models. This is needed for #29510, where the unified cache attends through per-sequence K/V row lists and the KQ mask follows those lists. There qwen4exp currently needs an opt-out, which also misses its MTP graph.Additional information
Tested on RTX PRO 6000 and GB10:
test-llama-archs -a "qwen4exp|glm5-next"passes (CUDA, CPU, Meta)test-llama-archswithindexer.top_koverridden to 8 so that the top-k actually selects: PPL identical to master for unified/split cache, FA on/off, CUDA/CPU (16 configs)llama-parallelgreedy outputs identical, MTP server outputs identical. At 32k the PPL is within run-to-run noise (top-k ties)Requirements