Feature: Added LIGHTNING_INDEXER support for Deepseek V4 ops on Vulkan Backend by shenron0101 · Pull Request #27453 · ggml-org/llama.cpp

shenron0101 · 2026-08-20T18:13:44Z

Overview

Updated PR from the previous closed one.

LIGHTNING_INDEXER: Added support for F32, F16, BF16, and several quantized formats.

The Lightning Indexer has only been implemented in the CPU backend, CUDA backend but not in the Vulkan backend. This is an implementation of the Indexer for DeepSeek V4 on the Vulkan backend.

Assisted-by: OpenCode

Currently the Hyper-connection Operations for the Vulkan backend are being implemented by @kh0pper. The one missing operation in the PR was the lightning indexer.

##26585 — "vulkan: tiled transpose for 0<->2 permuted CONT"

The original PR which was closed:
##26548 — "vulkan: add DeepSeek-V4 hyper-connection fused ops (DSV4_HC_COMB/PRE/POST)".

Additional information

Implements GGML_OP_LIGHTNING_INDEXER for Vulkan. Previously CPU, CUDA, SYCL, Metal only.
K types: f32, f16, bf16, q8_0, q5_1, q5_0, q4_1, q4_0, iq4_nl.
Extracts the FA_TYPE_* values and helpers from flash_attn_base.glsl into a new shared fa_types.glsl, so the indexer reuses FA's dequant path rather than redeclaring it.
Two SPIR-V variants selected on subgroup_arithmetic && subgroup_require_full_support, same pattern as ssm_scan.

Numbers (RTX 3090, test-backend-ops perf, K=f16)

Shape	µs	GFLOPS
nh=32 kv=256	10.81	195.6
nh=64 kv=256	19.50	216.8
nh=32 kv=4096	81.55	414.7
nh=64 kv=4096	159.16	425.0

Correctness evidence

test-backend-ops: 180/180 LIGHTNING_INDEXER cases, 16931/16931 full Vulkan suite.
GGML_VULKAN_CHECK_RESULTS against CPU on live DeepSeek-V4-Flash shapes: avg_err 3.42e-10.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: GPT-5.6 was used in the development.

…ane dot-product reduction moved from a shared-memory tree to subgroupAdd.

ggml-gh-bot · 2026-08-20T18:18:44Z

Hi @shenron0101, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

cdanis · 2026-08-21T00:33:01Z

Tried this out on a Strix Halo (AMD gfx1151). Used model -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-IQ2_XXS

180/180 passed, no problem on ./bin/test-backend-ops -b Vulkan0 -o LIGHTNING_INDEXER

No apparent change in performance or perplexity as compared to current master.

kh0pper · 2026-08-21T03:07:30Z

Possible explanation for the null result @cdanis measured on gfx1151: most of what indexer fusion can remove on this hardware already went away when the 0<->2 permuted CONT transpose landed (#26585, in master since Aug 19).

Same box (gfx1151 / RADV), DeepSeek-V4-Flash IQ3_XXS, 9040-token prompt (indexer n_kv 2304, -ub 512), before vs after that transpose fix:

	before	after
indexer chain	45.13% of prefill	5.17%
indexer glue (what fusion removes)	44.64%	4.15%
decode glue	0.66%	0.63%

In absolute terms the glue went 67.5 s -> 3.6 s of an 87.7 s prefill. That caps indexer fusion at roughly +4.3% prefill / +0.5% decode at this context length, and the fused kernel also has to absorb the indexer matmul (1.01% of prefill), which is a tuned MUL_MAT f16 today — so the realistic band includes outcomes that come out slightly negative.

Two caveats that cut the other way: this is n=1 at a single context length, and the indexer glue is O(n_kv) while the HC ops are O(1), so the indexer's share grows with depth. If this change is going to show up anywhere it is at depth — d24576 or a 32k prompt — rather than at the default. Worth measuring there before treating the null as final.

Minor: the HC ops referenced in the description are #26578 (reopened today), not #26548, which was superseded.

cdanis · 2026-08-21T08:37:19Z

I wasn't too surprised with the null result given the lack of fused HC ops. I'll try again combined with the re-opened PR, thanks :)

0cc4m · 2026-08-21T06:26:44Z

-
 layout (binding = 1) readonly buffer K_PACKED_BF16 { u16vec4 data[]; } k_packed_bf16;
+
+#ifndef FA_K_ONLY


I don't think this is needed, unused bindings make no difference and you'd just need something like #define FaTypeV 0 to avoid the compile issue.

jeffbolznv

The ggml-vulkan changes look mostly good to me, I just had a couple minor comments. I didn't do a detailed review of the test changes. I'm surprised such large changes were needed, but I didn't look closely.

jeffbolznv · 2026-08-21T15:47:04Z

+    barrier();
+
+    float score = 0.0;
+    for (uint h = 0; h < n_heads; ++h) {


Looks like each head is independent, so there's probably a lot of opportunity to optimize. But this version is clear and fine for the initial implementation.

Tested a version with optimizations that we can add if needed.

Shape Baseline Optimized Speedup

nh=32 kv=256 464.0 GFLOPS 486.2 GFLOPS 1.05x

nh=64 kv=256 469.0 GFLOPS 497.3 GFLOPS 1.06x

nh=32 kv=4096 468.9 GFLOPS 559.4 GFLOPS 1.19x

nh=64 kv=4096 473.8 GFLOPS 576.2 GFLOPS 1.22x

nh=32 kv=65536 461.3 GFLOPS 563.1 GFLOPS 1.22x

nh=64 kv=65536 467.3 GFLOPS 568.9 GFLOPS 1.22x

Shall I leave this as a seperate PR?

Yes, leave for a future change

This reverts commit fdcbdd9.

0cc4m · 2026-08-26T11:40:43Z

@ggml-org/ggml-cuda CUDA is failing these new tests. Not sure about the other CI issues.

am17an · 2026-08-26T12:08:16Z

cc @fairydreaming

jeffbolznv

I haven't reviewed the details of the test changes, but the ggml-vulkan code LGTM.

fairydreaming · 2026-08-26T17:25:18Z

The test changes look like the friendly AI agent noticed that in real life K tensor is non-contiguous and decided to test non-contiguous inputs in general, so it made the necessary changes for this and some additional friendly refactors all over the place. While the intention is admirable, the execution is a bit confusing, so personally I'd purge it with fire, maybe except the part where GGML_TYPE_IQ4_NL is added to the list of tested types.

jeffbolznv · 2026-08-26T17:31:15Z

IMO it would be better to fix the supports_op for the other backends.

am17an · 2026-08-26T17:44:36Z

test-backend-ops changes look weird. It's just a normal op, no need to add these tests. We've been running this with the model which uses it for quite some time.

fairydreaming · 2026-08-26T18:55:09Z

IMO it would be better to fix the supports_op for the other backends.

@jeffbolznv OK, I'm going to investigate the problem for CUDA now, let's talk about what to change in tests later.

fairydreaming · 2026-08-26T20:09:36Z

From what I see in logs, the failing cases all follow the pattern where there is NaN present both in CPU and in tested backend, examples:

CUDA: [NONE] NaN at index 264 (CUDA0=-nan CPU=-nan) [NONE] NaN at index 40 (CUDA0=-nan CPU=-nan) [NONE] NaN at index 599 (CUDA0=nan CPU=nan)
Vulkan: [NONE] NaN at index 77 (Vulkan0=-nan CPU=-nan) - yeah it doesn't pass the new test cases as well
Graviton CPU: [NONE] NaN at index 3 (CPU=nan CPU=nan)

Also they are NONE tensors (sentinels?), so likely the problem is that this new test case doesn't initialize sentinels and sometimes their elements are randomly NaN. I see that test_generic_op used in new test cases only initializes source tensors and doesn't do anything to sentinels.

My opinion is unchanged - these new cases shall be removed, we should focus on testing realistic inputs used by models (non-contiguous KQ mask doesn't seem be to such case, AFAIK q and weights are contiguous too). If we want to add tests with k as a view we should follow the test_flash_attn_ext example where it's controlled by bool kv_view parameter and tensor initialization is very simple.

0cc4m · 2026-08-26T21:11:08Z

Makes sense. @shenron0101 please refactor the tests. What was the reason for the changes?

shenron0101 · 2026-08-27T08:20:35Z

Yup, it's as @fairydreaming mentioned. The Agent picked up that K may be a view/non-contiguous in a real model.

I have reverted the test cases and I'll leave the GGML_TYPE_IQ4_NL for a later change if needed.

…n Backend (ggml-org#27453) * vulkan: add LIGHTNING_INDEXER op * vulkan: updated lightning_indexer.comp and ggml-vulkan.cpp with 128-lane dot-product reduction moved from a shared-memory tree to subgroupAdd. * vulkan: cleanup; Skip bounds checks * vulkan: cleanup FA_K_ONLY * Revert "vulkan: cleanup FA_K_ONLY" This reverts commit fdcbdd9. * vulkan: restore interleaved K/V buffer ordering * vulkan: Remove FA_K_ONLY * vulkan: Revert flash_attn_dequant * vulkan: Revert tests in backend-ops.cpp

shenron0101 added 2 commits August 21, 2026 01:48

vulkan: add LIGHTNING_INDEXER op

9283284

vulkan: updated lightning_indexer.comp and ggml-vulkan.cpp with 128-l…

1065050

…ane dot-product reduction moved from a shared-memory tree to subgroupAdd.

shenron0101 requested review from a team and ggerganov as code owners August 20, 2026 18:13

github-actions Bot added testing Everything test related Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Aug 20, 2026

0cc4m reviewed Aug 21, 2026

View reviewed changes

jeffbolznv reviewed Aug 21, 2026

View reviewed changes

shenron0101 added 2 commits August 22, 2026 22:19

vulkan: cleanup; Skip bounds checks

00c191d

vulkan: cleanup FA_K_ONLY

fdcbdd9

jeffbolznv reviewed Aug 22, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/vulkan-shaders/flash_attn_dequant.glsl Resolved

Revert "vulkan: cleanup FA_K_ONLY"

82c0ba2

This reverts commit fdcbdd9.

shenron0101 requested a review from jeffbolznv August 22, 2026 16:24

shenron0101 added 2 commits August 23, 2026 02:24

vulkan: restore interleaved K/V buffer ordering

b3d7873

vulkan: Remove FA_K_ONLY

00899e3

0cc4m reviewed Aug 24, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/vulkan-shaders/flash_attn_dequant.glsl Resolved

vulkan: Revert flash_attn_dequant

89e7f7e

0cc4m approved these changes Aug 26, 2026

View reviewed changes

jeffbolznv approved these changes Aug 26, 2026

View reviewed changes

vulkan: Revert tests in backend-ops.cpp

8c7bc9a

0cc4m approved these changes Aug 27, 2026

View reviewed changes

0cc4m merged commit cb30059 into ggml-org:master Aug 27, 2026
23 of 29 checks passed

ggerganov mentioned this pull request Sep 4, 2026

ggml : bump version to 0.23.0 ggml-org/ggml#1618

Merged

BrewTestBot mentioned this pull request Sep 4, 2026

ggml 0.23.0 Homebrew/homebrew-core#302393

Merged

1 task done


		layout (binding = 1) readonly buffer K_PACKED_BF16 { u16vec4 data[]; } k_packed_bf16;

		#ifndef FA_K_ONLY

Shape	Baseline	Optimized	Speedup
nh=32 kv=256	464.0 GFLOPS	486.2 GFLOPS	1.05x
nh=64 kv=256	469.0 GFLOPS	497.3 GFLOPS	1.06x
nh=32 kv=4096	468.9 GFLOPS	559.4 GFLOPS	1.19x
nh=64 kv=4096	473.8 GFLOPS	576.2 GFLOPS	1.22x
nh=32 kv=65536	461.3 GFLOPS	563.1 GFLOPS	1.22x
nh=64 kv=65536	467.3 GFLOPS	568.9 GFLOPS	1.22x

Conversation

shenron0101 commented Aug 20, 2026

Overview

Additional information

Requirements

Uh oh!

ggml-gh-bot Bot commented Aug 20, 2026

Uh oh!

cdanis commented Aug 21, 2026

Uh oh!

kh0pper commented Aug 21, 2026

Uh oh!

cdanis commented Aug 21, 2026

Uh oh!

0cc4m Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

jeffbolznv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jeffbolznv Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

shenron0101 Aug 22, 2026

Choose a reason for hiding this comment

Uh oh!

shenron0101 Aug 22, 2026

Choose a reason for hiding this comment

Uh oh!

jeffbolznv Aug 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

0cc4m commented Aug 26, 2026

Uh oh!

am17an commented Aug 26, 2026

Uh oh!

jeffbolznv left a comment

Choose a reason for hiding this comment

Uh oh!

fairydreaming commented Aug 26, 2026

Uh oh!

jeffbolznv commented Aug 26, 2026

Uh oh!

am17an commented Aug 26, 2026

Uh oh!

fairydreaming commented Aug 26, 2026

Uh oh!

fairydreaming commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0cc4m commented Aug 26, 2026

Uh oh!

shenron0101 commented Aug 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

fairydreaming commented Aug 26, 2026 •

edited

Loading