Repository navigation
CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3 (recreated) - #28552
Merged
Merged
Conversation
…24546) * adjust ncols_picker for routed MoE in mul_mat_q_case function * Adding CDNA, RDNA2 and RDNA4 * fix: update mmq_use_routed_moe_ncols_picker to include NVIDIA + Volta support * feat: enhance mmq configuration for various architectures with moe_ncols_min_cc support * refactor: replace moe_ncols_min_cc with use_typical_moe_ncols in mmq configuration files * HIP: mmq: enable typical moe ncols on RDNA4 --------- Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
Assisted-by: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011SYPfRhKoUpU3gMsGxq6go
JohannesGaessler
left a comment
JohannesGaessler
left a comment
Contributor
There was a problem hiding this comment.
Let me quickly check the RDNA4 performance as well but LGTM based on static code analysis.
| // On RDNA3 and RDNA4 it is faster to pick the tile size against this value instead of ne12. | ||
| int64_t ncols_opt = ne12; | ||
| if (GGML_CUDA_CC_IS_RDNA3_0(cc) || GGML_CUDA_CC_IS_RDNA4(cc)) { | ||
| ncols_opt = (ne12*n_expert_used + ne02 - 1) / ne02; |
Contributor
There was a problem hiding this comment.
I'm not sure the rounding actually makes a difference here but since this is just based on empirical evidence anyways I think it doesn't matter.
github-actions
Bot
added
ggml
Contributor
Performance
|
JohannesGaessler
approved these changes
Sep 8, 2026
IMbackK
approved these changes
Sep 9, 2026
Patt92
pushed a commit
to Patt92/llama.cpp
that referenced
this pull request
Sep 9, 2026
Also retires the hand-tuned MUL_MAT_ID J thresholds. Upstream d4abd57 (ggml-org#28552) re-landed routed-MoE tile sizing in the shape its maintainer asked for -- an ncols_opt field on mmq_args, decided host-side -- and gated it to RDNA3.0 and RDNA4. mmq.cu extends that gate to RDNA3.5 and drops the local heuristic and its GGML_CUDA_MMQ_ID_J override with it.
zbrad
pushed a commit
to zbrad/llama.cpp
that referenced
this pull request
Sep 10, 2026
…gml-org#28552) Recreated from ggml-org#24546 --------- Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz> * CUDA: pick MMQ tile size against ncols_opt set on the host side Assisted-by: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011SYPfRhKoUpU3gMsGxq6go --------- Co-authored-by: ravel7524 <58877666+ravel7524@users.noreply.github.com> Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
1 task done
liquidspikes
added a commit
to liquidspikes/llama.cpp
that referenced
this pull request
Sep 16, 2026
justinappler
added a commit
to justinappler/llama.cpp-strix-halo
that referenced
this pull request
Sep 26, 2026
…NA3.5 Upstream ggml-org#28552/ggml-org#28935 land the routed-MoE ncols picker on RDNA3.5, which Finding ggml-org#10 measured flat against our static J=48 cap, so the cap goes. Upstream ggml-org#28102 routes WMMA to MMA_F16 at D=256 and bypasses our tile config at prefill; guard it off on RDNA3.5 pending the G/N A/B. Assisted-by: Claude Opus 5.5
roverdrubber
added a commit
to roverdrubber/llama.cpp
that referenced
this pull request
Oct 15, 2026
…backend changes A quantized weight that takes the dequantize and hipBLAS path was dequantized to F16, so any activation past the F16 range came back as inf. On RDNA3 the 27B crosses into that path at 129 rows, which is why a long prompt returned NaN logits while a short one did not. Quantized types now dequantize to F32 on AMD; NVIDIA keeps F16 where its hardware is fast at it. The regression test added later in this series (activations in [-4000, 4000], both sides of the 128-row cutover) returned inf for Q2_K, Q6_K and Q8_0 before this change. The same commit carries the backend half of the upstream sync: the MMQ and vec-dot changes (ggml-org#28552, ggml-org#26705), fast_bf16_hardware_available and the BF16 fallback (ggml-org#28846), the GGML_CUDA_FA_QUANTS build option, and ggml-org#26308 and ggml-org#28897. The integrated-GPU host-buffer path stays disabled on HIP as well, because it corrupts output there (upstream ggml-org#26209, reverted in ggml-org#28604). MUL_MAT and MUL_MAT_ID pass on gfx1100 (1304/1304 and 892/892 at the tip of this series). Assisted-by: Claude Opus 5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Recreated #24546
Additional information
Added some remarks from @JohannesGaessler on top of the original.
Requirements