Repository navigation
Vulkan: Mamba-1 selective scan on the GPU - #116
Merged
Merged
Conversation
…anning on the CPU) third_party/llama.cpp-vulkan -> 1bit/vulkan-upstream adb2af5d (a32f0d6f + one commit of ours). Vulkan's SSM_SCAN handled only Mamba-2, so every Mamba-1 layer (Mamba, Falcon-Mamba, and the Zyphra Zamba v1 / BlackMamba still to come) ran its scan on the CPU. The new ssm_scan_m1 shader runs one invocation per channel with the 16-long state in registers, matching the CPU reference including groups. state-spaces/mamba-370m-hf F16 on Strix Halo: pp256 228 -> 6736 tok/s, tg64 25.8 -> 173.5 tok/s; greedy completions identical to the CPU-scan build; test-backend-ops SSM_SCAN 17/17 (three Mamba-1 cases). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Docs7 for 1bit-monster/engine
Commit |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
pushed a commit
that referenced
this pull request
Sep 26, 2026
…e tokenizer.json vocabs third_party/llama.cpp-vulkan -> 1bit/vulkan-upstream ee6e4e0d (adb2af5d + two commits of ours): - model: Zamba v1 (Zyphra/Zamba-7B-v1). Mamba-1 layers split into two heads (per-head x_proj/dt_proj; B/C enter the scan as groups, on the Mamba-1 Vulkan scan from #116) plus one shared transformer block on concat(h, embeddings), no RoPE, stored once. Teacher-forced top-1 vs transformers FP32: 96/96; greedy continuations identical; pp256 543, tg64 14.9 tok/s, all compute on Vulkan0. - gguf-py: LlamaHfVocab gave every token score -1000, so llama.cpp's SPM tokenizer picked merges arbitrarily for any model with only a SentencePiece-style BPE tokenizer.json (Zamba, Zamba2, ...): Zamba-7B-v1 tokenized 318/1500 wikitext lines like HF. Scores are now -rank of the producing merge: 1500/1500. Upstream bug; GGUFs converted before need reconverting. registry: ZambaForCausalLM -> vulkan; registry_check PASS for Zamba-7B-v1 and the reconverted Zamba2-1.2B-instruct. docs/vulkan.md lists both commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
added a commit
that referenced
this pull request
Sep 26, 2026
…e tokenizer.json vocabs (#117) * Zyphra Zamba v1 on Vulkan, and a converter fix for SentencePiece-style tokenizer.json vocabs third_party/llama.cpp-vulkan -> 1bit/vulkan-upstream ee6e4e0d (adb2af5d + two commits of ours): - model: Zamba v1 (Zyphra/Zamba-7B-v1). Mamba-1 layers split into two heads (per-head x_proj/dt_proj; B/C enter the scan as groups, on the Mamba-1 Vulkan scan from #116) plus one shared transformer block on concat(h, embeddings), no RoPE, stored once. Teacher-forced top-1 vs transformers FP32: 96/96; greedy continuations identical; pp256 543, tg64 14.9 tok/s, all compute on Vulkan0. - gguf-py: LlamaHfVocab gave every token score -1000, so llama.cpp's SPM tokenizer picked merges arbitrarily for any model with only a SentencePiece-style BPE tokenizer.json (Zamba, Zamba2, ...): Zamba-7B-v1 tokenized 318/1500 wikitext lines like HF. Scores are now -rank of the producing merge: 1500/1500. Upstream bug; GGUFs converted before need reconverting. registry: ZambaForCausalLM -> vulkan; registry_check PASS for Zamba-7B-v1 and the reconverted Zamba2-1.2B-instruct. docs/vulkan.md lists both commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs/vulkan: the Zamba v1 numbers ran every scan on the Mamba-1 Vulkan kernel Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 20, 2026
…ixes the main build) (#477) * Pin llama.cpp 8792923f: the SET_ROWS add-on hook the private kernels need (census G1) gpu-kernels be68256d (#476) compiles addons/hip-hrx/dispatch-hip-set-rows-q8.cpp against HipSetRowsArgs, which only the fork branch 1bit/hrx-set-rows-q8 had: with the private kernels the release build failed at dispatch-hip-set-rows-q8.cpp:39. Fork #116 squash-merged that branch onto 1bit/hrx-vulkan-patched (8792923f8, the same tree as the head proven here). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * registry: regenerated at llama.cpp 8792923f Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 20, 2026
* Pin llama.cpp 8792923f: the SET_ROWS add-on hook the private kernels need (census G1) gpu-kernels be68256d (#476) compiles addons/hip-hrx/dispatch-hip-set-rows-q8.cpp against HipSetRowsArgs, which only the fork branch 1bit/hrx-set-rows-q8 had: with the private kernels the release build failed at dispatch-hip-set-rows-q8.cpp:39. Fork #116 squash-merged that branch onto 1bit/hrx-vulkan-patched (8792923f8, the same tree as the head proven here). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * registry: regenerated at llama.cpp 8792923f Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hw-gate: build the private HIP kernels, as the release does The gate built llama.cpp without the gpu-kernels add-on, so it passed engine main 1834ceb, whose pins do not compile together with the private kernels (dispatch-hip-set-rows-q8.cpp:39, HipSetRowsArgs). The build phase now fetches the add-on at the pinned commit (scripts/fetch-private.sh, the runner's gh login) and passes GGML_HRX_HIP_ADDON_DIR like cmake/hrx.cmake; it reconfigures when the gpu-kernels pin moves, fails when libggml-hrx.so lacks ggml_hrx_hip_addon_register, and prints only the compiler's error lines (the job log is public). A box without read access stops with exit 2; HW_GATE_PRIVATE=0 gates the public part. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hw-gate: re-record the goldens and KLD bases with the private HIP kernels The gate now builds llama.cpp with the gpu-kernels add-on (be68256d), as the release does, so the kat goldens and the KLD base logits are recorded from that build (llama.cpp 8792923f, hrx-system 82804197). Recorded with scripts/hw-gate.sh --record --phases kat,kld,needle on strixhalo (performance mode), from the binaries of hw-gate run 39498508232. Why the old goldens no longer fit (ZAYA1-8B Q4_K_M, wiki.test 8 x 512, performance mode): KLD vs the CPU (--device none): public build 0.0381 (p99 0.431, same top 92.45%) add-on build 0.0357 (p99 0.384, same top 92.60%) KLD vs the F16 model on the CPU: public 0.5798, add-on 0.5816 (+-0.019), CPU Q4_K_M 0.5986 add-on vs the old (public) base: 0.0069; hip.mul_mat_id_prefill alone 0.0056, the dense prefill tile (ragged lm_head) alone 0.0003 The kat failure was prefix-reuse token 0, a near-tie: CPU: green -0.942, The -0.964 public: The -0.931, green -1.068 (old golden) add-on: green -0.860, The -1.060 (new golden) The add-on now picks the same token the CPU picks. Kernel checks: hrx_prefill_mmid q4_K/q6_K on the ZAYA expert tensors (T 512 and 135, top-1) match the f16-activation reference to 7e-7; the ragged lm_head (q6_K 2048 x 262147, T 1..512) has the same NMSE vs the CPU in its last 3 columns as in the rest (~5e-5), 2 repeats each; test-backend-ops MUL_MAT_ID 132/132, 3 runs. Base sha256 changes: zaya1-8b, qwen3.8-27b, qwen3-4b-iq4xs. Qwen3-0.6B and Qwen3-Coder-30B-A3B record the same bytes as before. Only one kat text changed (zaya1-8b prefix-reuse). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Vulkan's
SSM_SCANhandled only Mamba-2 (a scalar decay per head), so every Mamba-1 layer ran its scan on the CPU: Mamba, Falcon-Mamba, and the Zyphra models still to come (Zamba v1's two-head Mamba-1, BlackMamba).third_party/llama.cpp-vulkan→1bit/vulkan-upstreamadb2af5d(the current pina32f0d6f+ one commit of ours):ssm_scan_m1.comp, one invocation per channel with the d_state-16 state in registers, matching the CPU reference's Mamba-1 branch including groups; three new test-backend-ops cases (Zamba v1's n_group=2 decode shape, an odd token count across 3 sequences).Measured on Strix Halo,
state-spaces/mamba-370m-hfF16:Greedy completions on three prompts are identical between the two builds. test-backend-ops
SSM_SCAN17/17 on Vulkan0. Registry regenerated (only the recorded pin changes).🤖 Generated with Claude Code