Repository navigation
CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows - #29948
Merged
ggerganov merged 14 commits intoOct 25, 2026
Merged
Conversation
github-actions
Bot
added
testing
Contributor
Author
|
Adding @am17an and @JohannesGaessler for review |
Contributor
Author
gentle reminder for the review |
Contributor
|
Sorry, I hadn't prioritized looking at this PR for review because it is still marked as a draft. |
anujj
marked this pull request as ready for review
October 8, 2026 12:05
Comment on lines
+1880
to
+1887
| if (up->op != GGML_OP_MUL_MAT || !ggml_cuda_mmq_fusion_supported(x_up->type) || y->type != GGML_TYPE_F32 || | ||
| ggml_get_op_params_i32(up, 1) != GGML_HINT_NONE || ggml_get_op_params_i32(gate, 1) != GGML_HINT_NONE) { | ||
| return false; | ||
| } | ||
|
|
||
| if (!ggml_cuda_should_fuse_mul_mat(up, gate, glu)) { | ||
| return false; | ||
| } |
Contributor
There was a problem hiding this comment.
I'm not sure this is logically correct. You can fuse matrices with the Hadamard hind, but whether you should is a different question. So the correct place to have this check would be in ggml_cuda_should_fuse_mul_mat_q.
Contributor
Author
There was a problem hiding this comment.
Moved the hint check into ggml_cuda_should_fuse_mul_mat_q. If I understand currently it does not trigger here: the hint is only set on the KV-cache rotation matmuls (F32 src0, not followed by a GLU), while the fused path only takes quantized gate/up weights.
Comment on lines
+1106
to
+1107
| const int jt = has_fusion ? blockIdx.x : blockIdx.y; | ||
| const int it = has_fusion ? blockIdx.y : blockIdx.x; |
Contributor
There was a problem hiding this comment.
Why are you swapping blockIdx.x and blockIdx.y?
Contributor
Author
There was a problem hiding this comment.
The fused kernel cannot use stream-k, so it runs the xy tiling with the column tiles innermost, the same order as the stream-k path (jt = tmp % ntx). With the row tiles innermost the concurrently running blocks each read a different weight tile, so the gate+up weights of a layer (100 MB for Qwen3.8-27B Q4_K vs 24 MB of L2 on spark) are streamed from VRAM once per column tile; with the column tiles innermost they share one weight tile and only y (~12 MB at 2048 tokens) has to stay in L2. Fused gate/up time per layer on spark, Qwen3.8-27B pp2048, row tiles innermost vs this PR: Q4_K 14.5 vs 10.1 ms, Q8_0 20.7 vs 9.1 ms, IQ4_XS 10.7 vs 8.7 ms.
Contributor
There was a problem hiding this comment.
Sorry, I'm not sure I follow. The point I'm trying to get at is that there is a pre-existing logic for how to use blockIdx.x and blockIdx.y in terms of calculating pointer offsets without stream-k. This adds additional complexity and I'm not sure there is a technical benefit to it. The reason for the indexing in the stream-k path is that that seemed to work the best for the variable usage of experts in MoE models. I don't think it matters for tiling. Or if it does, please change it for the non-fused tiling case as well.
Comment on lines
+558
to
+600
| // Fused gate/up/GLU write-back: accumulators l and l + 2 hold the up and gate values of the same output row and column. | ||
| template<ggml_type type, int J, bool fallback> | ||
| static __device__ __forceinline__ void ggml_cuda_mmq_write_back_mma_glu( | ||
| const float * __restrict__ sum, const ggml_glu_op glu_op, const float glu_limit, | ||
| const int * __restrict__ ids_dst, float * __restrict__ dst, const int stride, const int i_max, const int j_max) { | ||
| #ifdef TURING_MMA_AVAILABLE | ||
| typedef tile<16, 8, int> tile_C; | ||
| static_assert(tile_C::I == 2*GGML_CUDA_MMQ_FUSION_ROWS, "each C tile must hold one chunk of up rows and one of gate rows"); | ||
|
|
||
| constexpr int rows_per_warp = ggml_cuda_mmq_get_rows_per_warp(type, J, fallback); | ||
| constexpr int ntx = rows_per_warp/tile_C::I; // Number of x minitiles per warp. | ||
|
|
||
| const int i0 = (threadIdx.y / ntx) * (ntx*tile_C::I); | ||
|
|
||
| #pragma unroll | ||
| for (int j0 = 0; j0 < J; j0 += ntx*tile_C::J) { | ||
| #pragma unroll | ||
| for (int n = 0; n < ntx; ++n) { | ||
| #pragma unroll | ||
| for (int l = 0; l < tile_C::ne/2; ++l) { | ||
| const int j = j0 + (threadIdx.y % ntx) * tile_C::J + tile_C::get_j(l); | ||
|
|
||
| if (j > j_max) { | ||
| continue; | ||
| } | ||
|
|
||
| const int i = (i0 + n*tile_C::I)/2 + tile_C::get_i(l); | ||
|
|
||
| if (fallback && i > i_max) { | ||
| continue; | ||
| } | ||
|
|
||
| const int sum_index = (j0/tile_C::J + n)*tile_C::ne + l; | ||
| dst[ids_dst[j]*stride + i] = ggml_cuda_op_glu_single(glu_op, sum[sum_index + 2], sum[sum_index], glu_limit); | ||
| } | ||
| } | ||
| } | ||
| #else | ||
| GGML_UNUSED_VARS(sum, glu_op, glu_limit, ids_dst, dst, stride, i_max, j_max); | ||
| NO_DEVICE_CODE; | ||
| #endif // TURING_MMA_AVAILABLE | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
Avoid adding a whole new function and extend the template for the existing one instead.
Comment on lines
+1880
to
+1883
| if (up->op != GGML_OP_MUL_MAT || !ggml_cuda_mmq_fusion_supported(x_up->type) || y->type != GGML_TYPE_F32 || | ||
| ggml_get_op_params_i32(up, 1) != GGML_HINT_NONE || ggml_get_op_params_i32(gate, 1) != GGML_HINT_NONE) { | ||
| return false; | ||
| } |
Contributor
There was a problem hiding this comment.
I think this is missing a check that up and gate have the same type.
Contributor
Author
There was a problem hiding this comment.
That is covered by ggml_cuda_should_fuse_mul_mat, called right below: it returns false unless up->src[0] and gate->src[0] have the same type, shape and strides, and ggml_cuda_mul_mat_q asserts the same. Happy to make it explicit here if you prefer.
| @@ -1565,14 +1670,26 @@ void mul_mat_q_switch_J(ggml_backend_cuda_context & ctx, const mmq_args & args, | |||
| } | |||
| } | |||
|
|
|||
| template <ggml_type type, bool fallback, ggml_prec prec_src1> | |||
| static void mul_mat_q_switch_fusion(ggml_backend_cuda_context & ctx, const mmq_args & args, cudaStream_t stream) { | |||
| if constexpr (ggml_cuda_mmq_fusion_supported(type)) { | |||
Contributor
There was a problem hiding this comment.
Suggested change
| if constexpr (ggml_cuda_mmq_fusion_supported(type)) { | |
| if constexpr (!fallback && ggml_cuda_mmq_fusion_supported(type)) { |
In order to avoid compiling template specializations that make no sense to use. Add a check to disable fusion for cases with fallback == false and then you can also simplify the code a bit by using the assumption that nrows_x is a multiple of 128. Relevant PR: #29953 .
anujj
force-pushed
the
cuda-mmq-fused-gate-up-interleaved
branch
from
October 9, 2026 12:19
0071207 to
b6e0f03
Compare
4 tasks done
Contributor
Author
|
Addressed all the comments |
Contributor
|
I ran this on Ada (RTX 4090, clocks locked to 2235 MHz, Qwen3.8-27B Q4_K, llama-bench -ngl 99 -r 5):
It helps on Ada too, though by less than on GB10 / RTX 5090. I also timed a single FFN block at the 27B shape (Q5_K, 5120 -> 17408 -> 5120, 512 tokens, median of 30 runs): 1441 us on master, 1414 us with this PR, and 1401 us with the down-projection GLU fusion in #30489. The fused launch is roughly at parity with the stock gate/up pair it replaces (~884 us vs ~880 us), so on Ada the gain comes mostly from removing the GLU kernel rather than from MMA-side savings; the down-side approach measures slightly faster per FFN. One interaction worth knowing about: With #30489 also in the tree, the two split coverage per FFN: this PR anchors the GLU first, so on the FFNs it accepts the down-side fusion does not run. On Ada the down-side fusion alone measures slightly faster (3030 vs 3023 t/s at pp512); both together 3040. Gating the fusion to cases where the plain dispatch would not pick stream-k might be worth checking on Ada. |
Contributor
Author
|
Thanks for the Ada numbers and the per-FFN breakdown. Scope. This PR is deliberately limited to the dense path and five types to keep it reviewable. The MUL_MAT_ID extension is already implemented locally on top of it (Qwen3.6-35B-A3B Q4_K_M, pp16K vs master: +10.2% on GB10, +6.1% on RTX 5090, perplexity bit-identical) and I plan to push it, and the remaining MMQ types, once the dense part is settled. Fused matmul vs the two stock matmuls. It depends on the type and the GPU, though. Per FFN layer, pp2048, 27B shape (5120 -> 17408), median time of the two stock MMQ launches vs the fused one, and the whole gate/up block:
Gating. Already in the PR since b6e0f03: #30489. Unsloth dynamic quants are a concrete case: some layers have different gate and up types, which this PR cannot interleave, so #30489 should take those layers while this PR takes the rest. I am measuring master / this PR / #30489 / both on Qwen3.8-27B UD-Q4_K_M on spark and will post the numbers here. |
Contributor
Author
|
Measured on spark with #30489 cherry-picked onto this PR. Qwen3.8-27B unsloth UD quants, pp16K, tok/s vs master on the same base:
UD-Q4_K_M has 30 of 64 FFN layers with different gate and up types. |
Contributor
Author
@JohannesGaessler : pinging here to move this up in your review stack |
Comment on lines
+1924
to
+1930
| // The fused kernel quantizes src1 to Q8_1, only fuse if the src1 precision of up and gate allows it. | ||
| for (const ggml_tensor * mm : { up, gate }) { | ||
| const ggml_prec prec_src1 = (ggml_prec) ggml_get_op_params_i32(mm, 3); | ||
| if (prec_src1 != GGML_PREC_UNDEFINED && prec_src1 != GGML_PREC_Q8 && prec_src1 != GGML_PREC_Q4) { | ||
| return false; | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
What cases does this intend to cover? It seems to me that if the precision is set to a value other than these then MMQ should not be used regardless of fusion.
Contributor
Author
There was a problem hiding this comment.
You're right that it isn't fusion-specific: the unfused path would hit the assert in ggml_cuda_mmq_get_prec_src1 for those values as well. Removed.
|
|
||
| template <ggml_type type, int J, bool fallback> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q8_0( | ||
| const char * __restrict__ x, int * __restrict__ x_tile, const int kbx0, const int i_max, const int stride) { | ||
| template <ggml_type type, int J, bool fallback, bool has_fusion = false> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q8_0( |
Contributor
There was a problem hiding this comment.
Suggested change
| template <ggml_type type, int J, bool fallback, bool has_fusion = false> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q8_0( | |
| template <ggml_type type, int J, bool fallback, bool has_fusion> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q8_0( |
Please require an explicit specification to avoid accidental misuse.
|
|
||
| template <ggml_type type, int J, bool fallback> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q4_K( | ||
| const char * __restrict__ x, int * __restrict__ x_tile, const int kbx0, const int i_max, const int stride) { | ||
| template <ggml_type type, int J, bool fallback, bool has_fusion = false> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q4_K( |
Contributor
There was a problem hiding this comment.
Suggested change
| template <ggml_type type, int J, bool fallback, bool has_fusion = false> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q4_K( | |
| template <ggml_type type, int J, bool fallback, bool has_fusion> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q4_K( |
|
|
||
| template <ggml_type type, int J, bool fallback> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q5_K( | ||
| const char * __restrict__ x, int * __restrict__ x_tile, const int kbx0, const int i_max, const int stride) { | ||
| template <ggml_type type, int J, bool fallback, bool has_fusion = false> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q5_K( |
Contributor
There was a problem hiding this comment.
Suggested change
| template <ggml_type type, int J, bool fallback, bool has_fusion = false> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q5_K( | |
| template <ggml_type type, int J, bool fallback, bool has_fusion> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q5_K( |
|
|
||
| template <ggml_type type, int J, bool fallback> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q6_K( | ||
| const char * __restrict__ x, int * __restrict__ x_tile, const int kbx0, const int i_max, const int stride) { | ||
| template <ggml_type type, int J, bool fallback, bool has_fusion = false> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q6_K( |
Contributor
There was a problem hiding this comment.
Suggested change
| template <ggml_type type, int J, bool fallback, bool has_fusion = false> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q6_K( | |
| template <ggml_type type, int J, bool fallback, bool has_fusion> static __device__ __forceinline__ void ggml_cuda_mmq_load_tiles_q6_K( |
| // The mul_mat_q kernel implements "stream-k" work partitioning as described in https://arxiv.org/abs/2301.03598 | ||
|
|
||
| template <ggml_type type, int J, bool fallback, ggml_prec prec_src1> | ||
| template <ggml_type type, int J, bool fallback, ggml_prec prec_src1, bool has_fusion = false> |
Contributor
There was a problem hiding this comment.
Suggested change
| template <ggml_type type, int J, bool fallback, ggml_prec prec_src1, bool has_fusion = false> | |
| template <ggml_type type, int J, bool fallback, ggml_prec prec_src1, bool has_fusion> |
Comment on lines
+1106
to
+1107
| const int jt = has_fusion ? blockIdx.x : blockIdx.y; | ||
| const int it = has_fusion ? blockIdx.y : blockIdx.x; |
Contributor
There was a problem hiding this comment.
Sorry, I'm not sure I follow. The point I'm trying to get at is that there is a pre-existing logic for how to use blockIdx.x and blockIdx.y in terms of calculating pointer offsets without stream-k. This adds additional complexity and I'm not sure there is a technical benefit to it. The reason for the indexing in the stream-k path is that that seemed to work the best for the variable usage of experts in MoE models. I don't think it matters for tiling. Or if it does, please change it for the non-fused tiling case as well.
| } | ||
|
|
||
| template <ggml_type type, int J, bool fallback, ggml_prec prec_src1> | ||
| template <ggml_type type, int J, bool fallback, ggml_prec prec_src1, bool has_fusion = false> |
Contributor
There was a problem hiding this comment.
Suggested change
| template <ggml_type type, int J, bool fallback, ggml_prec prec_src1, bool has_fusion = false> | |
| template <ggml_type type, int J, bool fallback, ggml_prec prec_src1, bool has_fusion> |
Comment on lines
+1482
to
+1485
| CUDA_SET_SHARED_MEMORY_LIMIT((mul_mat_q<type, J, false, prec_src1, has_fusion>), nbytes_shared); | ||
| if constexpr (!has_fusion) { | ||
| CUDA_SET_SHARED_MEMORY_LIMIT((mul_mat_q<type, J, true, prec_src1>), nbytes_shared); | ||
| } |
Contributor
There was a problem hiding this comment.
Suggested change
| CUDA_SET_SHARED_MEMORY_LIMIT((mul_mat_q<type, J, false, prec_src1, has_fusion>), nbytes_shared); | |
| if constexpr (!has_fusion) { | |
| CUDA_SET_SHARED_MEMORY_LIMIT((mul_mat_q<type, J, true, prec_src1>), nbytes_shared); | |
| } | |
| CUDA_SET_SHARED_MEMORY_LIMIT((mul_mat_q<type, J, /*fallback =*/ false, prec_src1, has_fusion>), nbytes_shared); | |
| if constexpr (!has_fusion) { | |
| CUDA_SET_SHARED_MEMORY_LIMIT((mul_mat_q<type, J, /*fallback =*/ true, prec_src1>), nbytes_shared); | |
| } |
To make it a bit more clear what the purpose is.
| } | ||
|
|
||
| template <ggml_type type, bool fallback, ggml_prec prec_src1> | ||
| template <ggml_type type, bool fallback, ggml_prec prec_src1, bool has_fusion = false> |
Contributor
There was a problem hiding this comment.
Suggested change
| template <ggml_type type, bool fallback, ggml_prec prec_src1, bool has_fusion = false> | |
| template <ggml_type type, bool fallback, ggml_prec prec_src1, bool has_fusion> |
anantshri
added a commit
to anantshri/llama.cpp
that referenced
this pull request
Oct 19, 2026
…1019 (CUDA MMQ GLU fusion)
Contributor
Author
|
#29948 (comment) |
For prompt processing, run the gate and up matmuls of a dense FFN as one mul_mat_q launch and apply the GLU in the write-back. - The loaders take the gate weights as a second pointer and fill the weight tile with up and gate rows interleaved in chunks of 8, so each thread holds an up value and the matching gate value (rows r and r + 8 of an MMA tile). - The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[] per thread. The fused write-back computes the GLU from registers and writes half the rows. - The fused path uses the regular tiling (no stream-k) with the column tiles innermost so that neighbouring blocks share weight tiles. - Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer.
Contributor
|
I forgot: I'll still want to check the performance diff myself before merge. |
Contributor
PerformanceTable generated with
Changing the layout of the CUDA grid seems to make a small difference in some cases, on average it seems to be faster. Thank you for spotting it. At least for LLaMA 3 8b it seems to be necessary to have a large |
JohannesGaessler
added
the
merge ready
6 tasks
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
…p rows (ggml-org#29948) * cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows For prompt processing, run the gate and up matmuls of a dense FFN as one mul_mat_q launch and apply the GLU in the write-back. - The loaders take the gate weights as a second pointer and fill the weight tile with up and gate rows interleaved in chunks of 8, so each thread holds an up value and the matching gate value (rows r and r + 8 of an MMA tile). - The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[] per thread. The fused write-back computes the GLU from registers and writes half the rows. - The fused path uses the regular tiling (no stream-k) with the column tiles innermost so that neighbouring blocks share weight tiles. - Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer. * cuda: assert the fused MMQ gate argument and trim redundant comments * cuda: move the fused MMQ hint check to should_fuse and add TODOs * cuda: pass prec_src1 to the fused MMQ write-back * cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back * cuda: require has_fusion explicitly, drop the fusion-only src1 precision check * cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile * cuda: inline the MMQ tiling-efficiency check * tests: one fused gate/up MMQ case per type and GLU op * cuda: always launch the fused MMQ kernel with one block per tile * tests: shorten the fused gate/up MMQ test comment * cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs * cuda: run the fused MMQ kernel on the plain tiling path * cuda: put the column tiles innermost in the MMQ plain tiling path
dc740
added a commit
to dc740/llama.cpp
that referenced
this pull request
Oct 28, 2026
…d gate/up rows (ggml-org#29948)" This reverts commit c49edf0. The fused path rewrote the shared MMQ plain tiling so column tiles run innermost. gfx906 gains nothing from the fusion (MMA-only types) but pays the slower tiling on every large-batch matmul: GLM-5.3 pp dropped ~25-30% (proven by revert A/B: 10.47 -> 15.01 t/s). Assisted-by: Devin
1 task done
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).
PapyKahan
added a commit
to PapyKahan/llama.cpp
that referenced
this pull request
Oct 28, 2026
Upstream's version is kept wherever it does what the branch did; the branch's stays where upstream has none: - MoE cache: the constructor takes upstream's no_alloc flag beside the CPU share, buffers come from llama_memory_bufs::alloc_buf; phases and host mode as before. - build_moe_ffn: the branch's experts lambda (cache slots, host mode, shared batches) with upstream's DeepSeek-4.1 and Laguna changes. - qwen4exp: the selection is built over the cache cells as upstream now does (ggml-org#30965, ggml-org#31243), the branch keeps its query chunks (ggml-org#30815's earlier version); upstream's hc_mix outputs and merged tensors. - CUDA MMQ: the tile map of active MoE tiles also launches upstream's fused gate/up/GLU kernel (ggml-org#31188, ggml-org#29948); the fused paths zero the rows of negative ids as ggml_cuda_mul_mat_id does. - CPU MUL_MAT_ID: upstream's -1 skip (ggml-org#26631) and the branch's two-pass grouping agree, negative ids are skipped. - tests: the QSA chunk tests move with upstream's archs tests; a MUL_MAT_ID result marked by ggml_mul_mat_id_set_negative_ids runs on CUDA again, only unmarked ones are excluded ([TAG_MUL_MAT_ID_SKIP]). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P3vd8iuiuxiev7wMPyJmSf
6 tasks
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
…oE + old-15 PR57/63) Merged from the planned 08 fused-prefill and 09 fused-moe: the two are not hunk-separable (the fusions and the mmvq dispatch share ggml_cuda.cu try_fuse, mmvq.cu, norm.cu, common.cuh and the ggml_cuda_mm_fusion_args plumbing), and the old-08-only fusion set was not correct (fusion-on greedy text differed from fusion-off). Contents: - old-08 fusions: gdn-conv, ple-conv, norm-gated, topk-moe, the norm/unary/ getrows/argsort kernels, the CPU references and the try_fuse arms. - old-13: fused MoE gate+up+GLU MMQ, the mmvq short-K item-split/ksplit, the routed-compact MMQ (mmq.cuh/mmq-config), MWR, the mmq gate instances. - old-15 PR ggml-org#57/ggml-org#63: the mmvq multi-row/row-loop decode (GGML_MMVQ_RDNA4_ DECODE_GRID, nrows_loop, the lf_nval halving reduction), ssm_gate_beta/ ssm_conv_l2_gatebeta, the norm rms_norm_q8_1 refactor and quantize_mmq_q8_1_glu. The MoE-cache-aware arms (moe_cache_get_cold/redirect, the B1 stand-down) and the qwen4exp/HC arms are deliberately NOT here -- they belong to the slab-moe and qwen4exp blocks, and their absence leaves the resident-only path (bit-identical). Gates: build all OK; fusion on == off (4B greedy text); test-logits-width-probe width_purity PASS; MUL_MAT 2444/2444. Amended (r132): the Q8_1 quantize cache is keyed by the ROOT tensor's identity but filled from the (possibly offset) view's pointer, so an offset view stored its rows under the root identity and a later op with the same root and canonical row layout could hit them. Every key walk that reaches the cache now stops at the first pointer change (this block: the four mmvq sites and the unary mul_q8_1 site; block 10 adds its own GLU site), so offset-free views keep the shared root entry and an offset view gets its own arena entry. A new q8_1_src_range_ok() validates the source read range against the root buffer's allocation before every quantize_row_q8_1_cuda / fused gated-q8_1 launch, so a stale source pointer (a compute buffer reallocated under a still-matching cache entry) aborts loudly with the layout logged instead of faulting with HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION or silently reading reused memory. Reported in PR ggml-org#218 (@slavap), whose V620 hit the fault on decode. Not reachable on the maintainer's model set (0 offset-view keys in 14336-60416 key walks per config across 7 configs, probed on gfx1201), so this is a correctness fix for configurations we do not run: 12/12 same-seed configs byte-identical, MTP acceptance identical (1859/3416, 1420/1710, 2171/2483, 372/375 on the four axes), prefill-logit KLD 0.000000. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r161): the routed fused gate arm is back. Upstream ggml-org#29948's `GGML_ASSERT(gate && !ids && ...)` in ggml_cuda_mul_mat_q is the *dense* arm's precondition, but the ids path below dispatches this block's `has_gate` kernel (ggml_cuda_mul_mat_q_switch_type_gate) over the routed weight tables, which is exactly what the MoE gate/up+GLU prefill fusion calls with `ids != nullptr`, so the assert aborted every config that reached it (issue ggml-org#241: an lfm2moe server prefill; reproduced on gfx1151 and on gfx1201 with GGML_PAIR_OFF=1). `ids` is no longer part of the assert -- bias/scale stay dense-only because the routed arm sets none, and a routed call can never reach upstream's dense interleaved kernel. Gate: fused == unfused on the reporting MoE (byte-identical greedy text; prefill-logit KLD -0.000000 / same-top-p 100.000 %).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the gate/up + GLU fusion, which today only covers small batches (MMVQ/MMVF), to prompt-processing batch
sizes on the MMQ path.
Alternative to #28702 implementing @am17an's suggestion there: keep one accumulator and apply the GLU in the write-back instead of carrying a second accumulator tile for the gate as discussed here: #28702 (comment)
For prompt processing, the gate and up
MUL_MATs of a dense FFN and the followingGLUrun as a singlemul_mat_qlaunch. The small-batch MMVQ/MMVF fusion, other backends and the weight layout are unchanged.Implimentation
randr + 8of the MMA tile). Chunks of 8 rather than 16 also cover the one-tile-per-warp configs.sum[]are unchanged; the write-back applies the GLU from registers and writes 64 output rows per tile.Please note: I have a plan to validate and enable it for the other GGML also.
Performance
DGX Spark (GB10),
llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5, tok/s:RTX 5090,
llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5, tok/s:Correctness
test-backend-ops: 19 newMUL_MAT_VEC_FUSIONcases hit the MMQ path (129 rows with bounds checks at one and two MMA tiles per warp, 256 rows without, GEGLU / SWIGLU_OAI / SWIGLU_CLAMP, batched). Full suite passes on GB10; the MUL_MAT suites also pass on an RTX 5090.GGML_CUDA_DISABLE_FUSION=1for all five types, also with-ot ffn_gate=CPU.Requirements