Skip to content

Vulkan: Mamba-1 selective scan on the GPU #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bong-water-water-bong merged 1 commit into from
Sep 26, 2026
Merged

Vulkan: Mamba-1 selective scan on the GPU #116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/vulkan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ The commit the engine pinned before is kept as the tag `vulkan-upstream-<sha12>`
| [ggml-org#28243](https://github.com/ggml-org/llama.cpp/pull/28243) (Daniel Han, Ryan Monsurate) | Qwen3.8-Flash-Next's NextN/MTP draft head (`--spec-type draft-mtp`), and the fix for `-md` loading the target model instead of the draft file | `62484fba` on v0.5.0 |
| [ggml-org#21412](https://github.com/ggml-org/llama.cpp/pull/21412) (Echo Labs) | Zyphra's Zamba2 (Mamba-2 layers plus shared transformer blocks, per-layer LoRA adapters merged at conversion) and its converter | `5ce79a56` |
| ours: `vulkan: SSM_SCAN for d_state 64` | Vulkan's Mamba-2 scan had pipelines only for a 128/256 state, so Zamba2-2.7B/7B (state 64) ran the scan on the CPU in every layer; two test-backend-ops cases with their shapes | `a32f0d6f` |
| ours: `vulkan: SSM_SCAN for Mamba-1` | Vulkan's scan only handled Mamba-2, so every Mamba-1 layer (Mamba, Falcon-Mamba, Zamba v1, BlackMamba) ran on the CPU; a per-channel kernel with the state in registers (d_state 16), three test-backend-ops cases | `adb2af5d` |

Each upstream PR is reviewed line by line before it is pinned. #28243: it touches only the qwen4exp model, its
converter, and two lines of the shared speculative decoding (the `-md` path fix, and
Expand Down Expand Up @@ -93,6 +94,8 @@ v0.5.0 + #28243 + #21412 + the d_state-64 scan (`a32f0d6f`); Zamba2 converted wi
| 2.7B-instruct | 1541 (was 614) | 45.2 (was 4.3: scan on the CPU) | correct |
| 7B-Instruct (2 shared blocks, RoPE) | 523 (was 29) | 17.1 (was 2.1) | correct, 16.7 tok/s |

Mamba-1 scan (`adb2af5d`), `state-spaces/mamba-370m-hf` F16, `llama-bench -p 256 -n 64`: pp256 228 → **6736** tok/s, tg64 25.8 → **173.5** tok/s; greedy completions identical to the CPU-scan build; test-backend-ops SSM_SCAN 17/17.

Large models need `--ctx-size`: without it llama-server allocates the KV cache
for the model's full trained context (262,144 tokens for Qwen3.8), which does not
fit next to Flash-Next's 104 GiB of weights.
Expand Down
2 changes: 1 addition & 1 deletion registry/architectures.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.",
"sources": {
"llama.cpp (vulkan)": "a32f0d6f35fe215fe4c4e267800d0555dcc9a4b4",
"llama.cpp (vulkan)": "adb2af5d59940741884a5fddaf0188aad30d14d5",
"llama.cpp (hrx)": "5556bf2cc513ab67cb8b179620d40dd97cb6427a",
"zinc": "3a35e76d64ebb91e2d82e16ddda20ee865ce1d45"
},
Expand Down
2 changes: 1 addition & 1 deletion third_party/llama.cpp-vulkan
Open in desktop
Submodule llama.cpp-vulkan updated 5 files
+1 −0 ggml/src/ggml-vulkan/ggml-vulkan-types.h
+18 −7 ggml/src/ggml-vulkan/ggml-vulkan.cpp
+93 −0 ggml/src/ggml-vulkan/vulkan-shaders/ssm_scan_m1.comp
+1 −0 ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp
+2 −0 tests/test-backend-ops.cpp
Loading