opencl: improve fa, allow dk512 for gemma-4, improve dk64 by lhez · Pull Request #30266 · ggml-org/llama.cpp

lhez · 2026-10-10T05:00:47Z

Overview

This PR extracts part of #26430 and enables dk = 512 with gqa = 4 and gqa = 8 for FA so gemma-4 does not fall back to CPU when FA is enabled. It also improves dk = 64 with gqa = 8 for gpt-oss-20b.

Additional information

Details

gpt-oss-20b

master

model	size	params	backend	ngl	fa	lm	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	OpenCL	99	1	none	tg256 @ d16384	16.59 ± 0.08

PR

model	size	params	backend	ngl	fa	lm	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	OpenCL	99	1	none	tg256 @ d16384	29.01 ± 0.08

gemma-4-E4B

master

Fallback to CPU

PR

model	size	params	backend	ngl	fa	lm	test	t/s
gemma4 E4B Q4_0	4.49 GiB	7.52 B	OpenCL	99	1	none	tg256 @ d16384	20.58 ± 0.39

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes, asked Codex to review and extract/refactor from opencl: extend, speed up and widen the MQ/cluster decode flash-attention (head sizes 64-512, GQA 2-16) #26430. I reviewed and checked Codex's results.

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

lhez · 2026-10-10T18:19:30Z

@max-krasnyansky please review and ack when you get a chance

CISC

I would have pushed the nits myself if I could. :)

CISC · 2026-10-11T11:46:03Z

+    std::map<std::tuple<int, int, int>, cl_kernel> f32_f16_mq_decode;
+    std::map<std::tuple<int, int, int>, size_t>    f32_f16_mq_decode_wg;
+    std::map<std::tuple<int, int, int>, int>       f32_f16_mq_decode_hs;
+    std::set<std::tuple<int, int, int>>           f32_f16_mq_decode_attempted;


Suggested change

std::set<std::tuple<int, int, int>> f32_f16_mq_decode_attempted;

std::set<std::tuple<int, int, int>> f32_f16_mq_decode_attempted;

CISC · 2026-10-11T11:50:28Z

+    cl_int err;
+    const std::string & compile_opts = backend_ctx->kernel_compile_opts;
+    const std::string opencl_c_std = "CL" + std::to_string(backend_ctx->opencl_c_version.major) +
+                                   "." + std::to_string(backend_ctx->opencl_c_version.minor);


Suggested change

"." + std::to_string(backend_ctx->opencl_c_version.minor);

"." + std::to_string(backend_ctx->opencl_c_version.minor);

This one just because it wasn't an even 4.

max-krasnyansky · 2026-10-11T18:50:22Z

I would have pushed the nits myself if I could. :)

Yeah, I need to figure out how to allow updates from ggml-maintainers in our fork.

ggml-opencl keeps this branch's implementation where upstream added its own version of the same feature: cooperative-K dp4a GEMM (ggml-org#30741, = cok), the act_vload4 dp4a GEMMs (ggml-org#30753, = alds4), the narrow l4_lm / q1_0 nsg1 variants (ggml-org#30850), the FA rework (ggml-org#30266), the expert-sort buf+offset (ggml-org#30418, kept the aligned-copy path) and the PREC_F32 MUL_MAT_ID decline (ggml-org#30365, covered by the src1 precision floor). The gemm_coopk_* kernels are not carried. Taken from upstream: IM2COL_3D (ggml-org#30895), get_rows i32 and cpy i32<->f32 (ggml-org#30756), CLAMP on non-contiguous views (ggml-org#30186), the A6X fixes (ggml-org#30176: a623 detection, constant-fold workaround, no cpy pack kernel), rms_norm full subgroup (ggml-org#30310), the rope build fix (ggml-org#30893), the mxfp4 flat GEMV row bound, the extra q5_0/q5_1/q5_K GEMV tail clamps (ggml-org#30421) and query_valid after the softcap in the q4_0/q8_0 FA prefill (ggml-org#30546). The q4_0 q1_vec FA decode now applies the softcap on the dp4a path too. Speculative decoding keeps this branch's llama_batch path: LiliCorr drafters (ggml-org#30506) are refused at load. The server keeps the tree verify and gives the logprobs of ggml-org#27196 the batch row each accepted token was sampled from. The two GDN n_tokens > K tests are dropped with upstream's relaxed contract.

…0266) * opencl: enable Gemma-4 E4B GPU decode Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> * opencl: extend Gemma-4 GPU decode to E2B Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> * opencl: optimize DK64 GQA8 decode Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> * opencl: optimize DK128 GQA4 decode Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> --------- Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

lhez and others added 4 commits October 9, 2026 21:02

opencl: enable Gemma-4 E4B GPU decode

4e7810d

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

opencl: extend Gemma-4 GPU decode to E2B

3110731

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

opencl: optimize DK64 GQA8 decode

cdf8763

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

opencl: optimize DK128 GQA4 decode

0861277

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

github-actions Bot added ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend labels Oct 10, 2026

lhez marked this pull request as ready for review October 10, 2026 18:19

lhez requested a review from a team as a code owner October 10, 2026 18:19

lhez added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Oct 10, 2026

max-krasnyansky approved these changes Oct 10, 2026

View reviewed changes

CISC approved these changes Oct 11, 2026

View reviewed changes

max-krasnyansky merged commit f2cc728 into ggml-org:master Oct 11, 2026
20 checks passed

This was referenced Oct 13, 2026

opencl: multiple optimizations of decode and prefill flash attention on Adreno #30182

Draft

opencl: decode gemma-4 global attention through the FA decomposition at depth #30247

Draft

ggerganov mentioned this pull request Oct 15, 2026

ggml : bump version to 0.27.0 ggml-org/ggml#1659

Merged

BrewTestBot mentioned this pull request Oct 15, 2026

ggml 0.27.0 Homebrew/homebrew-core#318733

Merged

1 task done

Navigation Menu

Repository navigation

opencl: improve fa, allow dk512 for gemma-4, improve dk64 - #30266

opencl: improve fa, allow dk512 for gemma-4, improve dk64#30266
max-krasnyansky merged 4 commits into
ggml-org:masterfrom
qualcomm:lh/opencl-gemma4-decode-minimal

lhez commented Oct 10, 2026

Uh oh!

lhez commented Oct 10, 2026

Uh oh!

CISC left a comment

Uh oh!

CISC Oct 11, 2026

Uh oh!

CISC Oct 11, 2026

Uh oh!

Uh oh!

max-krasnyansky commented Oct 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	std::set<std::tuple<int, int, int>> f32_f16_mq_decode_attempted;
	std::set<std::tuple<int, int, int>> f32_f16_mq_decode_attempted;

	"." + std::to_string(backend_ctx->opencl_c_version.minor);
	"." + std::to_string(backend_ctx->opencl_c_version.minor);

Conversation

lhez commented Oct 10, 2026

Overview

Additional information

gpt-oss-20b

gemma-4-E4B

Requirements

Uh oh!

lhez commented Oct 10, 2026

Uh oh!

CISC left a comment

Choose a reason for hiding this comment

Uh oh!

CISC Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

CISC Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

max-krasnyansky commented Oct 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants