Repository navigation
vulkan: chunk batched mat-vec at the column counts that are slow on RDNA3.5 - #1
Merged
Merged
Conversation
…DNA3.5
On Strix Halo (RADV, Mesa 26.1) the mat-vec shader variants for 3, 5 and 6
columns -- and 4 for q4_K -- run several times slower per call than 1, 2 and
4. GGML_VK_PERF_LOGGER on Qwen3.8-Flash-Next: the q6_K LM head 2.24 ms at
n=1/2/4 but 12.8 at 3 and 18.5 at 5; q8_0 10240x2560 122 us at 1, 734 at 3,
1177 at 5; q4_K 12288x2560 85 us at 1, 180 at 4. Speculative verification and
multi-slot decode live exactly there.
Such batches are dispatched as chunks of column counts measured to scale
({4,2,1} for q8_0 and q6_K, pairs for the other quants), at the cost of
re-reading the weights once per chunk; 9-16 columns are kept on chunked
mat-vec instead of the mat-mat kernels, which are worse still at those widths
(a 3.3 MB q8_0 weight at n=21: 756 us against 17.5 per mat-vec column); and
q8_0 at two columns stays off the q8_1 integer-dot path, where it is 19x
slower than the f32 path. GGML_VK_MMV_NO_SPLIT=1 disables all of it.
Measured on the same tree: four bare slots decoding concurrently 53.5 t/s
aggregate (14.4 each) against 47.8 on ROCm; every shape in the census fixed;
single-stream within noise.
(cherry picked from commit d737d6e)
Collaborator
Author
|
I have not tried it but i'm assuming so because i don't think this change is model specific even if it was benchmarked with one. Happy to test it further before merging |
Collaborator
Author
|
this seems to bring no difference to 27B performance but affects Flash-Next version so worth merging |
gaetan-puleo
pushed a commit
to rcfa/strix-llama.cpp
that referenced
this pull request
Oct 16, 2026
…DNA3.5 (halo-box#1) (cherry picked from commit ca94157)
gaetan-puleo
pushed a commit
to rcfa/strix-llama.cpp
that referenced
this pull request
Oct 16, 2026
…sured The column chunking (halo-box#1) was measured on Qwen3.8-Flash-Next, where the hot shapes are q8_0 and q6_K. The K-quants and the q4_0 family were put on a 2-column chunk without a measurement, and on a dense Qwen3.8-27B UD-Q4_K_XL that re-reads 16 GiB of weights once per chunk for nothing: the single dispatch scales fine at 3, 5, 6 and 8 columns there. llama-bench pp tok/s, Vulkan (RADV, Mesa 26.1.7), -b 8: cols master NO_SPLIT ggml-org d08c787 2 23.5 23.0 23.0 3 23.0 32.9 33.1 5 28.5 43.1 46.7 8 28.8 47.4 59.9 Restrict chunking, and the 9..16 column chunk path, to q8_0 and q6_K. Every other type takes upstream's dispatch. GGML_VK_MMV_NO_SPLIT=1 still disables the rest. MTP decode on the same model (n_max 4, 3 runs): 9.6/9.8/9.9 tok/s on master -> 14.8/15.1/15.2 with this change (15.6/15.9/15.7 with NO_SPLIT, the q6_K tensors of this model are still chunked). Fixes halo-box#25 together with the n_rs_seq change. Assisted-by: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01LTnaB4ucGc5vkzctYns4Nq (cherry picked from commit d22fa65)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Strix Halo (RADV, Mesa 26.1) the mat-vec shader variants for 3, 5 and 6 columns — and 4 for q4_K — run several times slower per call than 1, 2 and 4.
GGML_VK_PERF_LOGGERon Qwen3.8-Flash-Next: the q6_K LM head 2.24 ms at n=1/2/4 but 12.8 at 3 and 18.5 at 5; q8_0 10240x2560 122 us at 1, 734 at 3, 1177 at 5; q4_K 12288x2560 85 us at 1, 180 at 4. Speculative verification and multi-slot decode live exactly there.Such batches are dispatched as chunks of column counts measured to scale, at the cost of re-reading the weights once per chunk. 9–16 columns stay on chunked mat-vec rather than the mat-mat kernels, which are worse still at those widths. q8_0 at two columns stays off the q8_1 integer-dot path, where it is 19x slower than f32.
GGML_VK_MMV_NO_SPLIT=1disables all of it.Measured: four bare slots decoding concurrently 53.5 t/s aggregate (14.4 each) against 47.8 on ROCm; every shape in the census fixed; single-stream within noise.
Scope: gated on
VK_VENDOR_ID_AMDand env-disableable — behaviour is unchanged on every other vendor. It is a driver/hardware heuristic tuned for RDNA3.5, which is why it is here rather than inhalo-box/llama.cpp.Build-verified with
-DGGML_VULKAN=ONonfedora:44(vulkan-headers, glslc, glslang, spirv-tools — same toolchain asmimiron/llamacpp-vulkan's build stage).