Repository navigation
Eval bug: Vulkan: count_experts.comp zero_skipped_rows() loops over every id in every thread #31163
Description
In count_experts.comp, the zero_skipped_rows() loop (from #26631) runs over every expert id in every thread instead of each thread checking the ids specific to itself. Swapping the two loops speeds up prompt processing 5%-35% depending on the model . Generation isn't affected significantly since it's only one token.
Here's a fix that Opus 5.5 wrote and I tested:
diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp b/ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp
index a46a0619..37767aec 100644
--- a/ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp
+++ b/ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp
@@ -56,7 +56,7 @@ void zero_skipped_rows() {
const uint num_elements = p.ne00 * p.ne01;
const uint tid = gl_LocalInvocationID.x;
- for (uint idx = 0; idx < num_elements; idx++) {
+ for (uint idx = tid; idx < num_elements; idx += BLOCK_SIZE) {
const uint i01 = fastdiv(idx, p.ne00mp, p.ne00L);
const uint i00 = idx - i01 * p.ne00;
@@ -65,7 +65,7 @@ void zero_skipped_rows() {
}
const uint d_off = i00 * p.stride_dst + i01 * p.batch_stride_dst;
- for (uint i = tid; i < p.ne0_dst; i += BLOCK_SIZE) {
+ for (uint i = 0; i < p.ne0_dst; i++) {
data_dst[d_off + i] = 0.0f;
}
}| model | experts (used/total) | pp2048 master | pp2048 patched | change | tg128 master | tg128 patched |
|---|---|---|---|---|---|---|
| granite-3.1-3b-a800m Q4_K_M | 8/40 | 6399.06 | 8625.55 | +35% | 316.36 | 316.48 |
| DeepSeek-V2-Lite-Chat 16B Q4_K_M | 6/64 | 3976.99 | 4369.85 | +10% | 188.27 | 187.15 |
| gpt-oss-20b MXFP4 | 4/32 | 3297.97 | 3511.33 | +6% | 161.61 | 161.59 |
| Qwen3.6-35B-A3B Q4_K_XL (-ncmoe 24) GPU w/ CPU | 8/256 | 754.48 | 791.08 | +5% | 28.14 | 27.77 |
| Qwen3-1.7B Q8_0 (dense, control) | - | 8330.22 | 8283.40 | -1% | 221.37 | 220.96 |
RX 7800 XT, RADV (Mesa 25.2.8), Linux 6.8, measured on master 2280f0f
llama-bench -m granite-3.1-3b-a800m-instruct-Q4_K_M.gguf -m gpt-oss-20b-MXFP4.gguf -ngl 99 -fa 1 -lm none -ub 2048 -p 512,2048 -n 128 -r 3
llama-bench -m DeepSeek-V2-Lite-Chat-Q4_K_M.gguf -ngl 99 -fa 1 -lm none -ub 2048 -p 512,2048 -n 128 -r 3
llama-bench -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -ngl 99 -ncmoe 24 -fa 1 -ub 2048 -p 512,2048 -n 64 -r 3
llama-bench -m Qwen3-1.7B-Q8_0.gguf -ngl 99 -fa 1 -lm none -ub 2048 -p 512,2048 -n 128 -r 5
test-backend-ops -b Vulkan0: all 21303 tests pass with the patch (including MUL_MAT_ID 1109/1109 and its 170 skip cases).
Activity
[-]Vulkan: count_experts.comp zero_skipped_rows() loops over every id in every thread[/-][+]Eval bug: Vulkan: count_experts.comp zero_skipped_rows() loops over every id in every thread[/+]SG-Amadeus commentedon Oct 27, 2026
I've opened a draft PR with a potential fix on #31266. @andrewd207 Could you test it on your setup ？
andrewd207 commentedon Oct 27, 2026
@SG-Amadeus I tested your PR and it has the same speedup effect, maybe a tiny bit faster even. Sadly not quite as dramatic as your results :) Wow + 76%.
It's a linear problem. Each block repeats the loops over the experts so Vulkan had a progressively worse problem the more blocks a prompt needed.
Vulkan is faster than HIP on my system :)
Tested on RX 7800 XT, RADV (Mesa 25.2.8 and Mesa 26.2.5), Linux 6.8, base 2280f0f. test-backend-ops -o MUL_MAT_ID: 1109/1109 pass.
pp2048, -ngl 99 -fa 1 -ub 2048, two interleaved rounds:RX 7800 XT, RADV (Mesa 25.2.8), Linux 6.8, master 2280f0f, pp2048
RX 7800 XT, RADV (Mesa 26.2.5), Linux 6.8, master 2280f0f, pp2048
HIP: ROCm 7.0.3, gfx1101, same commit.
llama-bench -ngl 99 -fa 1 -ub 2048 -p 512,2048 -n 128 -r 3, mean of 2 interleaved rounds.SG-Amadeus commentedon Oct 28, 2026
Thanks for testing! The large speedup mainly recovers the regression from #26631. Your fix gives essentially the same normal PP performance. I'm concerned that one thread has to zero the whole row. That might be slow when there are many skipped rows or each row is large.