Skip to content

CUDA: Fuse GDN alpha/beta projections - #29187

Draft
ynankani wants to merge 1 commit into
ggml-org:masterfrom
ynankani:ynankani/gdn_alpha_beta_fuse_decode
Draft

ynankani wants to merge 1 commit into
ggml-org:masterfrom
ynankani:ynankani/gdn_alpha_beta_fuse_decode

Conversation

@ynankani

@ynankani ynankani commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Fuse GDN alpha/beta projections and MMVQ small-batch epilogues. This PR fuses below two paths

  1. Fuse the GDN Alpha/Beta producer chain: The kernel uses one block per head, token, and projection, distributes the dot-product reduction across threads. Currently supporting projection weights in BF16, F16, F32, or Q8_0(Q8_0 is dequantized during dot product).
gate = softplus(dot(W_alpha, x) + dt) * ssm_a
beta = sigmoid(dot(W_beta, x))

We fuse the below producer chain in one CUDA launch:

alpha: matrix multiplication -> reshape -> add dt -> softplus -> multiply ssm_a -> reshape
beta:  matrix multiplication -> reshape -> sigmoid

Additional information

Performance RTX-6000 PRO-BLACKWELL
GPU Model Microbatch Test t/s master t/s PR Speedup
RTX PRO 6000 Blackwell qwen3.8 27B BF16 1 pp256 28.84 28.83 1
RTX PRO 6000 Blackwell qwen3.8 27B BF16 2 pp256 50.42 54.27 1.08
RTX PRO 6000 Blackwell qwen3.8 27B BF16 3 pp256 74.65 80.26 1.08
RTX PRO 6000 Blackwell qwen3.8 27B BF16 4 pp256 99.62 106.97 1.07
RTX PRO 6000 Blackwell qwen3.8 27B BF16 5 pp256 122.04 130.55 1.07
RTX PRO 6000 Blackwell qwen3.8 27B BF16 6 pp256 146.13 156.47 1.07
RTX PRO 6000 Blackwell qwen3.8 27B BF16 7 pp256 168.69 180.56 1.07
RTX PRO 6000 Blackwell qwen3.8 27B BF16 8 pp256 193.91 207.42 1.07
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 1 pp256 67.17 66.7 0.99
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 2 pp256 109.96 123.84 1.13
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 3 pp256 156.18 174.22 1.12
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 4 pp256 198.58 219.97 1.11
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 5 pp256 233.04 254.81 1.09
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 6 pp256 265.05 289.14 1.09
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 7 pp256 283.98 305.94 1.08
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 8 pp256 307.1 331.34 1.08
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 1 pp256 62.84 63.13 1
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 2 pp256 103.46 117.27 1.13
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 3 pp256 147.75 166.15 1.12
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 4 pp256 187.64 210.23 1.12
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 5 pp256 221.46 245.1 1.11
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 6 pp256 252.27 279.08 1.11
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 7 pp256 271.57 296.5 1.09
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 8 pp256 297.18 321.46 1.08
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 1 pp256 71.46 71.87 1.01
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 2 pp256 131.16 132.88 1.01
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 3 pp256 181.09 184.21 1.02
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 4 pp256 223.01 223.88 1
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 5 pp256 248.27 254.37 1.02
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 6 pp256 281.8 296.84 1.05
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 7 pp256 345.51 353.65 1.02
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 8 pp256 385.81 393.84 1.02
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 1 pp256 261.9 267.04 1.02
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 2 pp256 434.56 445.85 1.03
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 3 pp256 593.72 619.35 1.04
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 4 pp256 613.54 710.99 1.16
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 5 pp256 727.69 816.13 1.12
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 6 pp256 843.74 923.62 1.09
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 7 pp256 934.71 1008.7 1.08
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 8 pp256 1005.5 1095.82 1.09

It is showing up to 13% perf benefits on RTX-6000-PRO-BLACKWELL, will collect scores on other hardware also

Requirements

@github-actions github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Sep 20, 2026
@ynankani
ynankani marked this pull request as ready for review September 21, 2026 12:04
@ynankani
ynankani requested review from a team and ggerganov as code owners September 21, 2026 12:04
@am17an

am17an commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

You should break this PR into 2 separate pieces. One for each change

@ynankani

Copy link
Copy Markdown
Contributor Author

You should break this PR into 2 separate pieces. One for each change

Sure, will split the changes

@ynankani
ynankani marked this pull request as draft September 21, 2026 15:33
@ynankani
ynankani force-pushed the ynankani/gdn_alpha_beta_fuse_decode branch from cee25e2 to 6e6c343 Compare September 23, 2026 05:45
@ynankani ynankani changed the title CUDA: Fuse GDN alpha/beta projections and MMVQ small-batch epilogues CUDA: Fuse GDN alpha/beta projections Sep 23, 2026
@ynankani
ynankani marked this pull request as ready for review September 23, 2026 05:57
@am17an

am17an commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

why are the results so good with nvfp4 and noisy with q4_k_m?

@am17an am17an self-assigned this Sep 23, 2026
@am17an

am17an commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

I actually couldn't reproduce these results on a RTX Pro 6000. I tried -pp 256 -ub 1,2,3,4,5,6,7,8

@ynankani

Copy link
Copy Markdown
Contributor Author

why are the results so good with nvfp4 and noisy with q4_k_m?

Q4_K_M checkpoint had GDN projections quantized to Q8_0, so we dequantize Q8_) in that path which affects perf

@ynankani

Copy link
Copy Markdown
Contributor Author

I actually couldn't reproduce these results on a RTX Pro 6000. I tried -pp 256 -ub 1,2,3,4,5,6,7,8

I collected perf numbers with speedbench on coding dataset. I will collect perf numbers with llama-bench with small ub.

@am17an

am17an commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Q4_K_M checkpoint had GDN projections quantized to Q8_0, so we dequantize Q8_) in that path which affects perf

Compared to the NVFP4 model which has f32?

@ynankani

Copy link
Copy Markdown
Contributor Author

Q4_K_M checkpoint had GDN projections quantized to Q8_0, so we dequantize Q8_) in that path which affects perf

Compared to the NVFP4 model which has f32?

Sorry, missed to mention, for NVFP4 checkpoint from both unsloth and radixark are in bf16 prec while for Q4_K_M checkpoint from unsloth it is in Q8_0.

@am17an

am17an commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

I think what is happening is that bf16 path is falling to cublas for these thin matrices i.e. MMVF is only getting enabled for bs=1. What this points to is that we need better kernels for bf16 for n=2..8, the bf16 cublas one is too slow, forcing MMVF there seems to help

baykalokandemir added a commit to baykalokandemir/llama.cpp-bells-tortured that referenced this pull request Sep 28, 2026
…lat in server A/B; not adopted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018co7fcJoEQXMxLezg7HpMU
@ynankani

Copy link
Copy Markdown
Contributor Author

I think what is happening is that bf16 path is falling to cublas for these thin matrices i.e. MMVF is only getting enabled for bs=1. What this points to is that we need better kernels for bf16 for n=2..8, the bf16 cublas one is too slow, forcing MMVF there seems to help

Sorry for the delay was out on holiday for some time.
Nice catch and thanks for the patch you suggested. It makes sense to use mmvf even for batch size 2 to 8 instead of slow cublass path.
I am getting below perf sweep after applying the patch, can you please confirm if you are also getting similar results?

Performance RTX-6000 PRO-BLACKWELL Collected with -p 256 -n 0 -ub 1,2,3,4,5,6,7,8 -r 5
GPU Model Microbatch Test t/s master t/s PR Speedup
RTX PRO 6000 Blackwell qwen3.8 27B BF16 1 pp256 28.84 28.83 1
RTX PRO 6000 Blackwell qwen3.8 27B BF16 2 pp256 50.42 54.27 1.08
RTX PRO 6000 Blackwell qwen3.8 27B BF16 3 pp256 74.65 80.26 1.08
RTX PRO 6000 Blackwell qwen3.8 27B BF16 4 pp256 99.62 106.97 1.07
RTX PRO 6000 Blackwell qwen3.8 27B BF16 5 pp256 122.04 130.55 1.07
RTX PRO 6000 Blackwell qwen3.8 27B BF16 6 pp256 146.13 156.47 1.07
RTX PRO 6000 Blackwell qwen3.8 27B BF16 7 pp256 168.69 180.56 1.07
RTX PRO 6000 Blackwell qwen3.8 27B BF16 8 pp256 193.91 207.42 1.07
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 1 pp256 67.17 66.7 0.99
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 2 pp256 109.96 123.84 1.13
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 3 pp256 156.18 174.22 1.12
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 4 pp256 198.58 219.97 1.11
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 5 pp256 233.04 254.81 1.09
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 6 pp256 265.05 289.14 1.09
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 7 pp256 283.98 305.94 1.08
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 8 pp256 307.1 331.34 1.08
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 1 pp256 62.84 63.13 1
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 2 pp256 103.46 117.27 1.13
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 3 pp256 147.75 166.15 1.12
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 4 pp256 187.64 210.23 1.12
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 5 pp256 221.46 245.1 1.11
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 6 pp256 252.27 279.08 1.11
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 7 pp256 271.57 296.5 1.09
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 8 pp256 297.18 321.46 1.08
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 1 pp256 71.46 71.87 1.01
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 2 pp256 131.16 132.88 1.01
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 3 pp256 181.09 184.21 1.02
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 4 pp256 223.01 223.88 1
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 5 pp256 248.27 254.37 1.02
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 6 pp256 281.8 296.84 1.05
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 7 pp256 345.51 353.65 1.02
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 8 pp256 385.81 393.84 1.02
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 1 pp256 261.9 267.04 1.02
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 2 pp256 434.56 445.85 1.03
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 3 pp256 593.72 619.35 1.04
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 4 pp256 613.54 710.99 1.16
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 5 pp256 727.69 816.13 1.12
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 6 pp256 843.74 923.62 1.09
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 7 pp256 934.71 1008.7 1.08
RTX PRO 6000 Blackwell qwen3.6 35B-A3B UD-Q4_K_M 8 pp256 1005.5 1095.82 1.09

@am17an

am17an commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Yes, but you should make the bf16 change in separate PR, we can merge that and then independently check this PR again.

@IMbackK

IMbackK commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Seams slightly beneficial for HIP gpus too, no surprise.

GPU Model Microbatch size Test t/s master t/s ynankani/gdn_alpha_beta_fuse_decode Speedup
MI100 qwen35 27B Q5_K_M 1 pp128 26.23 27.13 1.03
MI100 qwen35 27B Q5_K_M 2 pp128 44.28 45.02 1.02
MI100 qwen35 27B Q5_K_M 4 pp128 67.09 67.97 1.01
MI100 qwen35 27B Q5_K_M 8 pp128 116.42 124.27 1.07
MI100 qwen35 27B Q5_K_M 128 pp128 431.32 432.86 1.00
GPU Model Microbatch size Test t/s master t/s ynankani/gdn_alpha_beta_fuse_decode Speedup
RX 7900 XTX qwen35 27B Q5_K_M 1 pp128 32.94 33.32 1.01
RX 7900 XTX qwen35 27B Q5_K_M 2 pp128 58.71 61.12 1.04
RX 7900 XTX qwen35 27B Q5_K_M 4 pp128 91.37 92.78 1.02
RX 7900 XTX qwen35 27B Q5_K_M 8 pp128 115.16 116.52 1.01
RX 7900 XTX qwen35 27B Q5_K_M 128 pp128 642.59 642.27 1.00
GPU Model Microbatch size Test t/s master t/s ynankani/gdn_alpha_beta_fuse_decode Speedup
AI PRO R9700 qwen35 27B Q5_K_M 1 pp128 27.10 27.66 1.02
AI PRO R9700 qwen35 27B Q5_K_M 2 pp128 48.25 49.72 1.03
AI PRO R9700 qwen35 27B Q5_K_M 4 pp128 80.58 82.29 1.02
AI PRO R9700 qwen35 27B Q5_K_M 8 pp128 98.65 99.18 1.01
AI PRO R9700 qwen35 27B Q5_K_M 128 pp128 702.75 702.02 1.00


const int cc = ggml_cuda_info().devices[cuda_ctx->device].cc;
if (!use_cuda_graph || cuda_ctx->curr_stream_no != 0 ||
!GGML_CUDA_CC_IS_NVIDIA(cc) || cc < GGML_CUDA_CC_TURING) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
!GGML_CUDA_CC_IS_NVIDIA(cc) || cc < GGML_CUDA_CC_TURING) {
GGML_CUDA_CC_IS_MTHREADS(cc) || (GGML_CUDA_CC_IS_NVIDIA(cc) && cc < GGML_CUDA_CC_TURING)) {

@ynankani

ynankani commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Marking this as draft till this #29633 is merged and will reopen once #29633 is merged and have recollected perf sweep numbers for this PR on the updated master.

Signed-off-by: ynankani <ynankani@nvidia.com>
@ynankani
ynankani force-pushed the ynankani/gdn_alpha_beta_fuse_decode branch from 3d3d159 to e75c82b Compare October 5, 2026 14:22
@ynankani

ynankani commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Performance improvements dropped to 2-4% after #29633. Not sure if this much complexity is ok to be introduced for small perf improvements.

Performance RTX-6000 PRO-BLACKWELL
GPU Model Microbatch Test t/s master t/s PR Speedup
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 1 pp256 65.17 67.41 1.03
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 2 pp256 122.07 125.78 1.03
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 3 pp256 171.08 176.25 1.03
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 4 pp256 214.91 222.47 1.04
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 5 pp256 250.7 257.61 1.03
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 6 pp256 283.35 287.42 1.01
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 7 pp256 300.09 308.37 1.03
RTX PRO 6000 Blackwell qwen3.8 27B RadixArk NVFP4 8 pp256 324.66 331.63 1.02
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 1 pp256 62.13 62.88 1.01
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 2 pp256 116.4 118.16 1.02
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 3 pp256 163.42 165.6 1.01
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 4 pp256 206.35 210.35 1.02
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 5 pp256 240.61 243.76 1.01
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 6 pp256 273.42 278.22 1.02
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 7 pp256 291.84 295.94 1.01
RTX PRO 6000 Blackwell qwen3.8 27B Unsloth NVFP4 8 pp256 315.21 319.39 1.01
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 1 pp256 71.02 71.81 1.01
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 2 pp256 131.05 133.43 1.02
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 3 pp256 180.46 183.47 1.02
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 4 pp256 222.06 225.68 1.02
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 5 pp256 246.38 254.18 1.03
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 6 pp256 290.17 297.86 1.03
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 7 pp256 344.03 354.61 1.02
RTX PRO 6000 Blackwell qwen3.8 27B UD-Q4_K_M 8 pp256 385.02 394.01 1.02

@am17an

am17an commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Let's hold off then. I would think something like automatic fusion using nvrtc is a better long term solution like the inductor framework. Perhaps worth a PoC.

@ynankani

ynankani commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Let's hold off then. I would think something like automatic fusion using nvrtc is a better long term solution like the inductor framework. Perhaps worth a PoC.

Agreed that a generic solution would be better long term. Few questions regarding this:

  1. What should be the scope of auto-fuse(pointwise/broadcast chains only, or the full ladder up through reductions and matmul epilogues)?
  2. Do you feel nvrtc is essential or just a possible code generation path? I mean an AOT compiled generic interpreter over the op table can be a viable solution? that might avoid an additional runtime dependency and cache warm-up.
  3. Should we get rid of ggml_cuda_try_fuse() altogether or should both paths coexist?

@am17an

am17an commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

All are open for discussion. From what I experimented with, the virtual table doesn't yield much benefit vs nvrtc. I tried fusing elementwise point wise and row wise ops.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants