vulkan : copy strided f16 KV once on AMD so reads spread across memory channels by Nathanw1014 · Pull Request #27703 · ggml-org/llama.cpp

Nathanw1014 · 2026-08-25T13:37:39Z

Overview

f16 KV layout does not engage all memory channels. This PR engages all channels by copying each layers f16 KV once into a scratch so that the KV cache reads are sequential. Reuse of scratch introduced in PR #25494

Additional information

Gating set to AMD where effect was measured, RTX 3070 did not benefit, Intel and any other vendor have not been tested.

Consecutive KV rows of one head are read n_kv_heads × head_dim × 2 bytes apart. Typical model KV geometry spacing is an even multiple of the 256B channel-interleave block that Mesa documents, causing only a subset of memory channels to engage.

KV geometry (heads × head_dim)	example models	row spacing	÷ 256 B	channels engaged (of 16)
2 × 128	small GQA-heavy models	512 B	2	8
4 × 128	Qwen3-Coder-30B (benchmarked)	1024 B	4	4
2 × 256	Qwen3.6-35B class (benchmarked)	1024 B	4	4
4 × 256	Qwen3.8-27B (benchmarked)	2048 B	8	2
8 × 128	Llama-3.1-8B class	2048 B	8	2
32 × 128	MHA-era 7B models	8192 B	32	1

The tradeoff is scratch size, 1 layer of KV

KV geometry	example models	KV layers	scratch/token	@32k	@128k	% of f16 KV cache
2 × 128	small GQA-heavy (typ. 32L)	32	1 KiB	32 MiB	128 MiB	~3.1%
4 × 128	Qwen3-Coder-30B (measured)	48	2 KiB	64 MiB	256 MiB	2.1%
2 × 256	Qwen3.6-35B (hybrid, 10/40 attn)	10	2 KiB	64 MiB	256 MiB	10%
4 × 256	Qwen3.8-27B (hybrid, 16/65 attn)	16	4 KiB	128 MiB	512 MiB	6.25%
8 × 128	Llama-3.1-8B class	32	4 KiB	128 MiB	512 MiB	3.1%
32 × 128	MHA-era 7B models	32	16 KiB	512 MiB	2 GiB	3.1%

TG is measured unaffected, stridding tax increases at depth. Worst case model (hybrid, 10/40 attention layers) regress ~1% in prefill @ 32k, back within run variance @ 64k.
All runs: llama-bench -m <model> -fa 1 -p 512 -n 0 -d 0,16384,32768,65536 -r 1 on Radeon 8060S (RADV STRIX_HALO), f16 KV. Stock = master 3af988f.

Qwen3-Coder-30B-A3B UD-Q6_K_XL - stock:

model	size	params	backend	ngl	fa	test	t/s
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512	1147.43 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d16384	376.48 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d32768	203.10 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d65536	72.42 ± 0.00

Qwen3-Coder-30B-A3B UD-Q6_K_XL - patched:

model	size	params	backend	ngl	fa	test	t/s
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512	1145.28 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d16384	506.22 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d32768	317.94 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d65536	186.54 ± 0.00

Qwen3.8-27B Q8_0 - stock:

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512	339.37 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d16384	240.11 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d32768	133.87 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d65536	64.18 ± 0.00

Qwen3.8-27B Q8_0 - patched:

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512	338.04 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d16384	270.60 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d32768	227.04 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d65536	172.65 ± 0.00

Qwen3.6-35B-A3B UD-Q4_K_XL - stock:

model	size	params	backend	ngl	fa	test	t/s
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512	1114.82 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d16384	814.97 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d32768	663.33 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d65536	482.44 ± 0.00

Qwen3.6-35B-A3B UD-Q4_K_XL - patched:

model	size	params	backend	ngl	fa	test	t/s
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512	1113.00 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d16384	812.31 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d32768	656.72 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d65536	478.51 ± 0.00

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes : Claude (Opus 4.8 / Opus 5 / Fable 5)
Assisted with benchmarking, analysis, and review; design + implementation directed by me

jeffbolznv · 2026-08-25T13:48:32Z

This seems fine. Might need broader testing to make sure it doesn't regress other AMD devices. Do you know what Qwen3.6-35B-A3B UD-Q4_K_XL didn't benefit?

0cc4m

Looks good overall and I can confirm the performance improvement. Great work.

0cc4m · 2026-08-26T12:29:50Z

+layout(local_size_x = 256, local_size_y = 1, local_size_z = 1) in;
+
+layout (binding = 0) readonly buffer A {f16vec4 data_a[];};
+layout (binding = 1) writeonly buffer D {f16vec4 data_b[];};


Since you're just copying memory you can probably just use regular ivec4. You don't need float16 explicit arithmetic types here.

0cc4m · 2026-08-26T12:34:33Z

        }
+        // Strided-copy counterpart for f16 KV (same scratch layout, no dequant).
+        if (tname == "f16") {
+            string_to_spv("dequant_f16_transpose", "dequant_f16_transpose.comp", {});


I don't think dequant is right, it's just a transpose copy shader. It's not immediately f16-specific either, e.g. bf16 kv could be handled by the same shader. It could also be written in a generic way to handle copies of any (reasonable) width.

I had also thought about commenting that this could just use the normal copy shader, no new shader needed. But this dedicated shader might perform a bit better? I'm not sure it will, it may be worth doing the experiment of just using the normal copy shader.

Thanks, the existing copy shader had the same performance as a dedicated shader.

hschougaard · 2026-08-31T06:22:59Z

Does this effort cover bf16 as well?

0cc4m · 2026-09-03T11:07:17Z

No, but there probably isn't a reason why it couldn't.

0cc4m · 2026-09-04T08:22:18Z

@Nathanw1014 Please rebase to fix the test conflict and check bf16.

0cc4m · 2026-09-11T12:57:46Z

@Nathanw1014

Nathanw1014 · 2026-09-21T14:40:17Z

I have extended the scratch path to bf16 too, scaler does benefit but slightly: depth at 64k (PP 470.1 vs 477) Llama-3.2-1B.

bf16 measured on coopmat1 using RADV_EXPERIMENTAL=bfloat16 had a better result, depth at 64k (PP 471 vs 753) Llama-3.2-1B.

0cc4m

I can see perf improvements on GCN and RDNA3.5 (Strix Halo), but a regression on RDNA4. Any theories about that? Maybe we have to disable the feature for it.

@Nathanw1014

… channels Cherry-pick of ggml-org/llama.cpp PR ggml-org#27703 (ggml-org#27703) Author: Nathan Wilson (@Nathanw1014) Reason for our setup: AMD-specific: decode perf at large context by spreading KV reads across memory channels

Assisted-by: Claude (Opus 4.8 / Opus 5)

The f16 clause needs only a strided copy, not a dequant, so drop the dedicated shader and dispatch the existing copy pipeline instead. The copy is described in source memory order so the strided access lands on the writes, which stream, instead of the reads, which would concentrate on a few memory channels. Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5)

Assisted-by: Claude Opus 5

jakehardee-wq · 2026-10-14T18:14:26Z

Tested on a different Strix Halo variant, in case broader AMD coverage helps: Ryzen AI Max+ PRO 495 (Radeon 8065S, gfx1151, RADV STRIX_HALO), 192GB, Mesa RADV 26.0.8, kernel 7.0.0-38, Ubuntu 26.04.1.

Model: MiniMax M2.7 UD IQ4_XS (230B A10B MoE), f16 KV. llama-bench -fa 1 -p 512 -n 0 -d 0,16384,32768,65536 -r 1

test	stock (`23b0202`)	this PR (`3fdbb71`)	speedup
pp512	309.22	309.69	1.00x
pp512 @ d16384	69.58	190.75	2.74x
pp512 @ d32768	14.80	132.33	8.94x
pp512 @ d65536	5.59	88.44	15.82x

No change at depth 0, and the gain grows with depth, reaching about 16x at 64k. For reference, with a q8_0 KV cache at d32768 and ubatch 512: stock 132.8, this PR 133.5, so q8_0 is unchanged and f16 now matches it.

I originally hit this as a severe f16 slowdown at depth, plus amdgpu "device lost" from long submissions, reported in #30364. This PR resolves it on this hardware.

vlforit · 2026-10-15T12:41:28Z

PR vulkan : copy strided f16 KV once on AMD so reads spread across memory channels #27703 <- 733f57a

llama-bench -fa 1 -m /Qwen3.8-27B-Q4_K_M.gguf -b 512 -ub 512 -d 0,16384,32768,65536 -r 1 -n 0
No such file or directory
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: dot2 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | n_batch |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | --: | --------------: | -------------------: |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    27.32 B | Vulkan     |  -1 |     512 |   1 |           pp512 |        986.88 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    27.32 B | Vulkan     |  -1 |     512 |   1 |  pp512 @ d16384 |        782.12 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    27.32 B | Vulkan     |  -1 |     512 |   1 |  pp512 @ d32768 |        645.05 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    27.32 B | Vulkan     |  -1 |     512 |   1 |  pp512 @ d65536 |        477.17 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    27.32 B | Vulkan     |  -1 |     512 |   1 |  pp512 @ d98304 |        375.16 ± 0.00 |

PR cpu: add F16 SWIGLU_OAI reference implementation and validation #29200 <- 5a638ae <- one commit before

llama-bench -fa 1 -m /Qwen3.8-27B-Q4_K_M.gguf -b 512 -ub 512 -d 0,16384,32768,65536 -r 1 -n 0
No such file or directory
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: dot2 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | n_batch |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | --: | --------------: | -------------------: |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    27.32 B | Vulkan     |  -1 |     512 |   1 |           pp512 |        989.79 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    27.32 B | Vulkan     |  -1 |     512 |   1 |  pp512 @ d16384 |        799.90 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    27.32 B | Vulkan     |  -1 |     512 |   1 |  pp512 @ d32768 |        675.64 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    27.32 B | Vulkan     |  -1 |     512 |   1 |  pp512 @ d65536 |        398.51 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    27.32 B | Vulkan     |  -1 |     512 |   1 |  pp512 @ d98304 |        297.28 ± 0.00 |

Test	PR #29200	PR #27703	27703/29200
pp512	989.79	986.88	0.9971
pp512 @ d16384	799.90	782.12	0.9778
pp512 @ d32768	675.64	645.05	0.9547
pp512 @ d65536	398.51	477.17	1.1974
pp512 @ d98304	297.28	375.16	1.2619

…y channels (ggml-org#27703) * vulkan : copy strided f16 KV once so reads spread across memory channels Assisted-by: Claude (Opus 4.8 / Opus 5) * tests : cover the f16 KV copy to scratch Assisted-by: Claude (Opus 4.8 / Opus 5) * vulkan : gate the f16 KV scratch copy to AMD Assisted-by: Claude (Opus 4.8 / Opus 5) * vulkan : reuse the generic copy shader for the f16 KV scratch fill The f16 clause needs only a strided copy, not a dequant, so drop the dedicated shader and dispatch the existing copy pipeline instead. The copy is described in source memory order so the strided access lands on the writes, which stream, instead of the reads, which would concentrate on a few memory channels. Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5) * vulkan : extend the strided KV scratch copy to bf16 Assisted-by: Claude Opus 5 * vulkan : clean up comments Assisted-by: Claude Opus 5 * consolidate copy shaders * disable on RDNA4 --------- Co-authored-by: Ruben Ortlam <rortlam@redhat.com>

Nathanw1014 requested review from a team and ggerganov as code owners August 25, 2026 13:37

github-actions Bot added testing Everything test related Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Aug 25, 2026

0cc4m approved these changes Aug 26, 2026

View reviewed changes

voidsurfer mentioned this pull request Sep 6, 2026

Strix Halo Vulkan stack: FA KV-quant, mmid, dense GEMM, delta-net prefill, DSv4 sparse attention halo-box/strix-llama.cpp#17

Merged

0cc4m mentioned this pull request Sep 21, 2026

vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 #27952

Merged

Nathanw1014 force-pushed the vulkan-fa-f16-kv-contig-v2 branch from 4d9e62f to 1781f45 Compare September 21, 2026 13:48

0cc4m reviewed Sep 22, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp Resolved

noamsto mentioned this pull request Sep 27, 2026

Investigate Qwen3.8-27B decode speed on gfx1151 (~22 tok/s, ~9 min per grading call) noamsto/nix-amd-ai#152

Open

6 tasks

0cc4m mentioned this pull request Oct 14, 2026

Eval bug: Vulkan: MiniMax-M2 prompt processing collapses at long context with ubatch above 128 on Strix Halo (gfx1151); long submits trigger amdgpu "device lost" #30364

Closed

Nathanw1014 and others added 7 commits October 14, 2026 09:00

vulkan : copy strided f16 KV once so reads spread across memory channels

24c1920

Assisted-by: Claude (Opus 4.8 / Opus 5)

tests : cover the f16 KV copy to scratch

5a718c4

Assisted-by: Claude (Opus 4.8 / Opus 5)

vulkan : gate the f16 KV scratch copy to AMD

f4a0888

Assisted-by: Claude (Opus 4.8 / Opus 5)

vulkan : extend the strided KV scratch copy to bf16

18c8eae

Assisted-by: Claude Opus 5

vulkan : clean up comments

a5141ac

Assisted-by: Claude Opus 5

consolidate copy shaders

60f13f7

0cc4m force-pushed the vulkan-fa-f16-kv-contig-v2 branch from 1781f45 to 60f13f7 Compare October 14, 2026 08:27

disable on RDNA4

3fdbb71

0cc4m approved these changes Oct 14, 2026

View reviewed changes

0cc4m added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Oct 15, 2026

ggerganov merged commit 733f57a into ggml-org:master Oct 15, 2026
27 of 28 checks passed

vlforit mentioned this pull request Oct 15, 2026

Misc. bug: llama-server Denial Of Service exchaust VRAM #30577

Open

ggerganov mentioned this pull request Oct 15, 2026

ggml : bump version to 0.27.0 ggml-org/ggml#1659

Merged

BrewTestBot mentioned this pull request Oct 15, 2026

ggml 0.27.0 Homebrew/homebrew-core#318733

Merged

1 task done

tczank mentioned this pull request Oct 16, 2026

[Regression]: draft-mtp decode ~10-13% slower on master vs a02c7f58c (RADV, RDNA3) — verify-step slowdown + cache-reuse-dependent acceptance instability #30638

Open

ruc1j mentioned this pull request Oct 23, 2026

密な GEMV とバッファの配置をメモリチャネルの交互配置に照らして監査する ruc1j/XeStrata-890M#13

Closed

Conversation

Nathanw1014 commented Aug 25, 2026

Overview

Additional information

Requirements

Uh oh!

jeffbolznv commented Aug 25, 2026

Uh oh!

0cc4m left a comment

Choose a reason for hiding this comment

Uh oh!

0cc4m Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

0cc4m Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

jeffbolznv Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Nathanw1014 Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

hschougaard commented Aug 31, 2026

Uh oh!

0cc4m commented Sep 3, 2026

Uh oh!

0cc4m commented Sep 4, 2026

Uh oh!

0cc4m commented Sep 11, 2026

Uh oh!

Nathanw1014 commented Sep 21, 2026

Uh oh!

0cc4m left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jakehardee-wq commented Oct 14, 2026

Uh oh!

Uh oh!

vlforit commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

vlforit commented Oct 15, 2026 •

edited

Loading