Repository navigation
feat: CUDA port of TurboQuant3 KV cache — 3.47x compression, 98.5% of F16 decode speed on RTX 5090 - #3
Closed
signalnine wants to merge 15 commits into
Conversation
Ports the Metal turbo3 implementation to NVIDIA CUDA. End-to-end working
on RTX 5090 with Qwen3.5 35B A3B Q4_K_M: 3.47x KV compression at 32K
context, ~4x max context extension (256K → 1M tokens on 32GB VRAM).
New files:
- ggml-cuda/turbo-quant.cuh — block_turbo3_0 layout, WHT sign arrays,
3-bit centroid LUT, dequant helpers,
quantize kernel (set_rows path)
- ggml-cuda/turbo-wht.cu/.cuh — GGML_OP_TURBO_WHT CUDA kernel; 128-thread
blocks, in-place butterfly in shared memory,
forward + inverse WHT via compile-time template
- ggml-cuda/template-instances/fattn-vec-instance-turbo3_0-turbo3_0.cu
— VEC flash-attention instance for D=64/128/256
Modified files:
- dequantize.cuh — dequantize_turbo3_0 (produces float2 pairs)
- convert.cu — all 5 to-fp16/fp32 dispatchers
- fattn-common.cuh — vec_dot_fattn_vec_KQ_turbo3_0, dequantize_V_turbo3_0,
dispatcher extensions
- fattn-vec.cuh — turbo3 treated as unquantized (f16-style nthreads_KQ)
- fattn.cu — route turbo3 exclusively to VEC kernel; add dispatch macro
- set-rows.cu — k_set_rows_turbo3 kernel: per-128-elem group quantization
with WHT rotation and norm correction
- ggml-cuda.cu — supports_op + compute dispatch for TURBO_WHT + SET_ROWS
- llama-kv-cache.cpp — +2 tensor overhead for rotation matrices
Benchmark (RTX 5090, Qwen3.5 35B A3B Q4_K_M, FA on):
KV memory @32k: 702 MiB (f16) → 202 MiB (turbo3) = 3.47x compression
Max context: ~256K (f16) → ~1M (turbo3) = ~4x extension
Decode @short: 233 t/s (q8_0) → 190 t/s (turbo3) = 0.82x
Prefill @32k: 6335 t/s (q8_0) → 1215 t/s (turbo3) = 0.19x
Note: prefill speed degrades significantly vs Metal (Metal: 0.99x q8_0 at all
contexts; CUDA: 0.19x at 32K). Root cause: turbo3 currently uses the VEC
flash-attention kernel; q8_0 uses the more efficient TILE/MMA kernels at long
context. TILE/MMA support for turbo3 is the next milestone.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove the 5-line early-return that forced turbo3 onto the VEC flash attention kernel. The VEC kernel is still used for decode (Q->ne[1]==1) via the existing dispatch logic, but prefill now goes through the Turing MMA kernel (RTX 5090 is SM 12.0 >> 7.5). launch_fattn already pre-dequantizes K/V to FP16 when need_f16_K/V are set (which TILE/MMA always pass as true). Our ggml_get_to_fp16_cuda and ggml_get_to_fp16_nc_cuda dispatchers for TURBO3_0 — added in the original CUDA port commit — provide that conversion automatically. Stride recalculation (nb11 = nb11*bs*sizeof(half)/ts) also works out correctly for turbo3 (bs=32, ts=14): nb11*32*2/14 = ne[0]*sizeof(half). ✓ Before (VEC only): After (MMA for prefill): 2K prefill: 5032 t/s (0.73× q8_0) 6734 t/s (0.98× q8_0) 8K prefill: 3110 t/s (0.46× q8_0) 6613 t/s (0.98× q8_0) 32K prefill: 1215 t/s (0.19× q8_0) 6168 t/s (0.97× q8_0) Matches Metal M5 Max result (0.99× q8_0 flat across all context sizes). Decode unchanged (VEC, ~0.64-0.82× q8_0 depending on context depth). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e (71→94 t/s, 98.5% of F16)
k_set_rows_turbo3 was the decode bottleneck: 1 thread/group serial kernel
gave 3.1% GPU utilisation (36.5 µs × 80 calls/token = ~21% of decode budget).
Replace with a fully parallel kernel — 1 block per 128-element group,
128 threads per block (one thread per element):
• Shared-memory WHT butterfly (7 stages, no atomics)
• Warp-reduce L2 norm + inter-warp accumulate via smem
• qs packed with __shfl_sync (4-wide gather), signs with __ballot_sync
• Reconstruction norm same pattern; one write per sub-block (warp lane 0)
Also tighten flash-attention dequant paths (fattn-common.cuh):
• vec_dot_fattn_vec_KQ_turbo3_0: elem0/elem1 always share the same
turbo3 block — load qs/signs once instead of twice per pair
• dequantize_V_turbo3_0: ne==4 fast path — load one qs byte and one
signs byte for all 4 elements; unrolled float2 / half2 output pairs
Benchmark (Qwen3.5 35B, RTX 5090, tg128):
Before: 71.86 t/s (0.75× q8_0)
After: 94.04 t/s (0.985× q8_0, within measurement noise of parity)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
seanrasch
pushed a commit
to seanrasch/llama-cpp-turboquant
that referenced
this pull request
Mar 27, 2026
Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed TheTom#3 (TURBO_D). TheTom#1 and TheTom#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TheTom
added a commit
that referenced
this pull request
Mar 27, 2026
Complete experiment log: #1 4-mag LUT: 15.1 at 8K (BEST, +38%) #2 Batched extract: 13.7 (+25%) #3 Inline FA block: 13.5 (I-cache pressure) #4 Deferred norm: 12.9 (loses ILP) #5 2-pair half2: 12.0 (ternary overhead) #6 Select chain: 11.9 (branches kill) #7 Bit-arithmetic: 11.6 (ALU too heavy) #8 FMA branchless: 11.4 (ALU still too heavy) #9 Named-reg ternary: 10.3 (branches worst) #10 Main (8-LUT): 10.95 (baseline) #11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai
signalnine
marked this pull request as draft
March 27, 2026 05:50
Author
|
Looks like I've got some quality issues with larger context windows, converting to draft while I work those out. |
The outer loop in vec_dot_fattn_vec_KQ_turbo3_0 stepped k_KQ_0 by `nthreads` (8), but the Q register file is loaded in blocks of `nthreads*cpy_ne` (32) elements per thread — the same pattern used by the f16/bf16 VEC kernels. This caused thread t>0 to pair K element (16s + 2t) against Q element (64*(s/4) + 8t + 2*(s%4)), a complete index mismatch. Every generated token had garbage attention scores. Fix: match the f16 kernel pattern — step by nthreads*cpy_ne, add an inner k_KQ_1 loop over cpy_ne pairs, and index Q_v as Q_v[k_KQ_0/nthreads + k_KQ_1]. Also clean up stale "PPL 23.5 vs 6.19" TODO comments in llama-graph.cpp that documented the symptom of this bug. Tested on RTX 5090, Qwen3.5-35B-A3B-Q4_K_M: - PPL (wikitext-2): 6.2023 → 6.2996 (+1.57%, within 5% target) - NIAH: 11/11 at 4K–256K (matches q8_0; was 0/11 before fix) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Author
Bug fix: VEC flash-attention Q/K stride mismatch (commit 4c91451)Root cause
This caused thread This kernel is used for all generation steps (n_tokens ≤ 2), so every generated token had garbage attention scores. Prefill (MMA kernel path) was unaffected. Fix (3 lines changed)// Before — wrong stride, Q/K indices misaligned for thread t > 0:
for (int k_KQ_0 = 0; k_KQ_0 < D/2; k_KQ_0 += nthreads) {
const int k_KQ = k_KQ_0 + (threadIdx.x % nthreads);
...
const float2 qv = ((const float2 *) Q_v)[k_KQ_0/nthreads];
// After — matches f16 kernel pattern:
constexpr int cpy_nb = ggml_cuda_get_max_cpy_bytes();
constexpr int cpy_ne = cpy_nb / 4;
for (int k_KQ_0 = 0; k_KQ_0 < D/2; k_KQ_0 += nthreads*cpy_ne) {
for (int k_KQ_1 = 0; k_KQ_1 < cpy_ne; ++k_KQ_1) {
const int k_KQ = k_KQ_0 + (threadIdx.x % nthreads)*cpy_ne + k_KQ_1;
...
const float2 qv = ((const float2 *) Q_v)[k_KQ_0/nthreads + k_KQ_1];Test results (RTX 5090, Qwen3.5-35B-A3B-Q4_K_M)Perplexity (wikitext-2, 512 ctx):
NIAH single-needle (11 depth points, q8_0 vs turbo3):
Aggregate score is identical (turbo3 = q8_0). The few single-cell differences are in opposite directions and consistent with model-level retrieval variance, not KV compression degradation. Speed (llama-bench):
Prefill is at parity. Generation is ~5% slower due to the centroid lookup overhead — this model is compute-bound during decoding (Q4_K_M MoE weights dominate bandwidth), so the 3.47x smaller KV cache doesn't help here. On a weight-fp16 or large-batch workload the bandwidth savings would show. |
|
Thats interesting. I will give it a try. I gave the fork from Madreag/turbo3-cuda a run tonight and it had also issues at large context sizes. q8 Madreag/turbo3-cuda turboquant signalnine/llama-cpp-turboquant (this PR) TG Performance is better on large kv-caches, but still not on par with q8 quants |
chrisqianz
referenced
this pull request
in chrisqianz/llama-cpp-turboquant-cuda
Mar 28, 2026
…n#29 Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed spiritbuun#3 (TURBO_D). spiritbuun#1 and spiritbuun#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
nalditopr
pushed a commit
to nalditopr/llama-cpp-turboquant
that referenced
this pull request
Mar 28, 2026
Optimizations from RotorQuant PR analysis: - TheTom#2: Flat no-WHT dequant kernel (256 threads, 4 elems/thread, no shmem, no syncthreads) — 32x fewer kernel launches than per-block version - TheTom#3: Shared memory centroid LUT in flat dequant kernel - Fused V*attn CUDA kernel (ready for future custom op integration) - Fixed ggml_turbo_wht assert: ne[0] % 32 (was 128) for WHT32 blocks Attempted direct MMVQ for V (no dequant) — incorrect because WHT rotation on attention weights corrupts the V dot product. WHT orthogonality only holds when both sides use matching block structure. Reverted to working WHT linearity approach (cast + post-matmul inv WHT). Final benchmarks on Qwen3.5-35B-A3B (RTX 5090): Prefill Decode vs f16 f16 baseline: 6860 187 100% turbo3 FA: 6832 188 100% ← parity! turbo3 MMVQ: ~60 139 74% Compression: 4.6x KV cache Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
@signalnine , would you be so kind and take a look at #13 ? |
TheTom#13) Models with n_embd_head_k not divisible by QK_TURBO3_GROUP (128) — e.g. GLM-4.7 Flash / DeepSeek2 MLA with head_dim=576 — previously hard-crashed with GGML_ASSERT(ne00 % QK_TURBO3_GROUP == 0) in set-rows.cu. Fix: - supports_op for SET_ROWS: return false when turbo3 and ne00 % 128 != 0, so llama.cpp falls back to an unquantised KV path instead of asserting - get_best_fattn_kernel: return BEST_FATTN_KERNEL_NONE for turbo3 when K head dim is not 128-aligned (VEC kernel only instantiated for D∈{64,128,256}) Affected models can now load with -ctk turbo3 -ctv turbo3; the non-aligned heads silently use f16 KV while 128-aligned heads continue to use turbo3. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…heTom#13) Models with n_embd_head_k not divisible by 128 (e.g. GLM-4.7 Flash, DeepSeek2 MLA with head_dim=192/576) previously crashed with GGML_ASSERT in set-rows.cu or segfaulted in CPU flash attention. Changes: - llama-kv-cache.cpp: detect incompatible head_dim at KV cache init, log a warning, and auto-fall back to q8_0 instead of crashing - ggml-cuda.cu supports_op: return false for turbo3 SET_ROWS when ne00 % 128 != 0, and for TURBO_WHT when ne[0] % 128 != 0 - fattn.cu: return BEST_FATTN_KERNEL_NONE for turbo3 at non-128 D - ggml-cpu.c: add turbo3 entry to type_traits_cpu (vec_dot + from_float) so CPU flash attention can handle turbo3 K/V for models where CUDA FA returns NONE (e.g. D=192 which has no CUDA FA kernel for any type) - set-rows.cu: add tail kernel for non-128 remainder (future use) - turbo-wht.cu: head-dim-aware processing with tail pass-through - ops.cpp: CPU WHT head-dim-aware with tail identity copy Tested: DeepSeek-Coder-V2 (head_dim=192) now loads with -ctk turbo3 and auto-falls back to q8_0 with clear warning. Qwen3.5 (head_dim=128) continues to use turbo3 at full speed (223 t/s). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Models with head_dim divisible by 64 but not 128 (e.g. DeepSeek2 MLA
head_dim=192, GLM-4.7 Flash head_dim=576) now use 64-element WHT
groups instead of crashing or falling back to q8_0.
Key changes:
**64-element WHT groups:**
- turbo-quant.cuh: 64-element sign arrays + FWHT-64 + rotate_64
- set-rows.cu: templated on GROUP_SIZE {128,64}, reads group_size
from SET_ROWS op_params (set by llama-kv-cache based on head_dim)
- turbo-wht.cu: templated on group_size, head-dim-aware dispatch
- ggml.h/ggml.c: ggml_turbo_wht() now takes explicit group_size
parameter (0=auto) to handle MLA where output dim differs from
K head dim
- ops.cpp: CPU WHT parameterized on group_size
**MLA fix — missing Q rotation in K-only build_attn:**
- build_attn(inp_attn_k, ...) had no turbo Q pre-rotation, while
SET_ROWS was applying WHT to K. Result: <unrotated_Q, rotated_K>
= garbage. Now all three build_attn variants apply Q rotation.
- Inverse WHT moved inside build_attn_mha (before v_mla projection)
and also added to the non-FA attention path.
**Guards + fallback:**
- supports_op, fattn.cu, llama-graph.cpp: relaxed from %128 to %64
- llama-kv-cache.cpp: falls back to q8_0 only when head_dim%64!=0
- CPU type_traits_cpu turbo3 entry for CPU FA vec_dot
Tested: GLM-4.7 Flash (head_dim=576) 208 t/s, DeepSeek-V2 (192) 143 t/s,
Qwen3.5 (128) 223 t/s — all producing correct output with turbo3.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…e versa) Users can now mix turbo3 and q8_0 independently for K and V caches: -ctk turbo3 -ctv q8_0 # compress K more, keep V at higher quality -ctk q8_0 -ctv turbo3 # opposite tradeoff Changes: - New VEC FA template instances for turbo3/q8_0 cross-type pairs (fattn-vec-instance-turbo3_0-q8_0.cu, fattn-vec-instance-q8_0-turbo3_0.cu) - fattn-vec.cuh: extern declarations for mixed instances - fattn.cu: dispatch + allow turbo3/q8_0 mix through the K!=V type guard - llama-graph.cpp: inverse WHT guard now checks V type (not K type) — when K=turbo3 but V=q8_0, V values are NOT WHT-rotated so inverse WHT must not fire. For MLA, V is a view of K so v->type correctly reflects K's turbo type. Tested: - 16/16 coherence tests pass (4 models × 4 KV combos) - NIAH at 4K+32K: all combos within baseline variance (10-11/11) q8_0/q8_0: 10/11, q8_0/turbo3: 10/11, turbo3/turbo3: 10/11, turbo3/q8_0: 9/11 (1 extra miss, normal variance) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
quantize_row_turbo3_0_ref was a stub that only stored the L2 norm and zeroed out qs/signs — any CPU-only path (-ngl 0) silently produced garbage output. Now implements full PolarQuant: L2 norm → normalize → forward WHT rotation (signs1 → butterfly → signs2) → 3-bit centroid quantize → pack qs/signs → corrected norm (grp_norm / recon_norm). WHT group size (128 or 64) is communicated from the CPU SET_ROWS handler via a global variable, read from the op_params that llama-kv-cache.cpp sets based on head_dim. This handles models with different K and V head dims (e.g. DeepSeek2 K=192→64, V=128→128). Tested CPU-only (-ngl 0): - Mixtral 8x7B (head_dim=128): correct output at 4.5 t/s - DeepSeek-Coder-V2 (head_dim=192): correct output at 18.2 t/s - GPU paths unchanged (Qwen 203 t/s, DeepSeek 164 t/s, GLM 202 t/s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds turbo2: 2-bit PolarQuant with WHT rotation, 2.5 bits/value, 6.4x compression vs f16. Same architecture as turbo3 but with a 4-centroid Lloyd-Max codebook instead of 8. Block format: block_turbo2_0 = 10 bytes per 32 values - norm (fp16, 2B): corrected L2 norm (grp_norm / recon_norm) - qs (8B): 2-bit centroid indices, 4 per byte Usage: -ctk turbo2 -ctv turbo2 Full stack: - ggml.h/ggml-common.h: GGML_TYPE_TURBO2_0 enum + block_turbo2_0 struct - ggml-turbo-quant.c: CPU quantize (with WHT) + dequantize - turbo-quant.cuh: CUDA centroids, midpoints, nearest-centroid, dequant - set-rows.cu: k_set_rows_turbo2 kernel (GROUP_SIZE-templated, parallel WHT) - dequantize.cuh + convert.cu: turbo2 to f16/f32 conversion - fattn-common.cuh: vec_dot_KQ_turbo2 + dequantize_V_turbo2 - fattn-vec.cuh + fattn.cu: VEC kernel dispatch + template instances - Mixed types: turbo2/q8_0 cross-type FA instances - common/arg.cpp: CLI --cache-type-k turbo2 - llama-graph.cpp + llama-kv-cache.cpp: graph + cache integration NIAH (Qwen3.5 35B, RTX 5090, 4K-512K): 4K-16K: 11/11 | 32K: 9/11 | 64K: 10/11 | 128K-512K: 11/11 Coherence: 31/31 across 4 models x all KV combos (GPU + CPU) Speed: 228 t/s decode on Qwen3.5 (comparable to turbo3 223 t/s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The mixed KV types commit changed the inverse WHT guard from checking k->type to v->type. For MLA, V is a view of K's first 512 elements (v->ne[0]=512, 512%128=0 → group=128). But K was quantized with 64-group WHT (k->ne[0]=576, 576%128≠0 → group=64). The mismatch produced garbage output on GLM-4.7 Flash. Fix: derive group_size from K when K is turbo (since K determines the rotation used during quantization), from V only when K is not turbo. Tested: 16/16 coherence (4 models × 4 KV combos), including GLM-4.7. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
InnerQ equalizes K channel variances before WHT rotation to reduce quantization error on models with anisotropic K distributions. Enabled via TURBO_INNERQ=N env var (N = calibration token count). Pipeline: calibrate per-channel K² stats → compute scales → apply scale to K before WHT (SET_ROWS) → apply 1/scale to Q and V output via ggml_turbo_wht op (scale passed as src[1]). Math: <Q/s, s*K> = <Q, K> preserves dot products. Key design: - d_innerq_active starts at 0 (CUDA zero-init) — kernel never multiplies by uninitialized scales - Auto-disables for 64-group models (group_size < 128) where channel mapping is not 1:1 across WHT groups - Auto-disables when max channel ratio < 1.2 (already balanced) - Cross-TU coordination via turbo-innerq.cu/cuh for scale_inv tensor updates between SET_ROWS and graph-side WHT - Scale_inv tensor stored in KV cache, initialized to identity Also: turbo2 (2-bit) now requires head_dim divisible by 128. At 64-group WHT, the 4-centroid codebook doesn't have enough resolution — DeepSeek (head_dim=192) produced garbage. turbo2 auto-falls back to q8_0 for non-128-aligned heads with a warning. Tested: 18/18 coherence (4 models × 4+ KV combos), InnerQ active on Qwen (222 t/s) and Mixtral (146 t/s), auto-disabled on GLM/DeepSeek. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
In the VEC flash-attention kernel, skip V dequantization for KV positions where ALL attention weights are below 1e-6. At long context most positions contribute noise, not signal — skipping their V dequant saves compute with zero quality impact. Both half2 and f32 paths are covered. The threshold (1e-6) matches the existing SOFTMAX_FTZ_THRESHOLD behavior where exp() values below this are flushed to zero. Benefit scales with context length — at short context nearly every position contributes so no skip occurs. At 512K+ most positions are skipped during generation. Tested: 10/11 NIAH at 512K (matches pre-sparse-V), 3/3 coherence. Generation at 512K: 25 t/s, prefill: 2535 t/s. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
KGardevoir
pushed a commit
to KGardevoir/llama-cpp-turboquant
that referenced
this pull request
Apr 14, 2026
Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed TheTom#3 (TURBO_D). #1 and TheTom#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
KGardevoir
pushed a commit
to KGardevoir/llama-cpp-turboquant
that referenced
this pull request
Apr 14, 2026
Complete experiment log: #1 4-mag LUT: 15.1 at 8K (BEST, +38%) TheTom#2 Batched extract: 13.7 (+25%) TheTom#3 Inline FA block: 13.5 (I-cache pressure) TheTom#4 Deferred norm: 12.9 (loses ILP) TheTom#5 2-pair half2: 12.0 (ternary overhead) TheTom#6 Select chain: 11.9 (branches kill) TheTom#7 Bit-arithmetic: 11.6 (ALU too heavy) TheTom#8 FMA branchless: 11.4 (ALU still too heavy) TheTom#9 Named-reg ternary: 10.3 (branches worst) TheTom#10 Main (8-LUT): 10.95 (baseline) TheTom#11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai
TheTom
added a commit
that referenced
this pull request
Apr 15, 2026
Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed #3 (TURBO_D). #1 and #2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TheTom
added a commit
that referenced
this pull request
Apr 15, 2026
Complete experiment log: #1 4-mag LUT: 15.1 at 8K (BEST, +38%) #2 Batched extract: 13.7 (+25%) #3 Inline FA block: 13.5 (I-cache pressure) #4 Deferred norm: 12.9 (loses ILP) #5 2-pair half2: 12.0 (ternary overhead) #6 Select chain: 11.9 (branches kill) #7 Bit-arithmetic: 11.6 (ALU too heavy) #8 FMA branchless: 11.4 (ALU still too heavy) #9 Named-reg ternary: 10.3 (branches worst) #10 Main (8-LUT): 10.95 (baseline) #11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai
LogicDaemon
pushed a commit
to LogicDaemon/llama-cpp-turboquant
that referenced
this pull request
May 27, 2026
Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed TheTom#3 (TURBO_D). TheTom#1 and TheTom#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
LogicDaemon
pushed a commit
to LogicDaemon/llama-cpp-turboquant
that referenced
this pull request
May 27, 2026
Complete experiment log: TheTom#1 4-mag LUT: 15.1 at 8K (BEST, +38%) TheTom#2 Batched extract: 13.7 (+25%) TheTom#3 Inline FA block: 13.5 (I-cache pressure) TheTom#4 Deferred norm: 12.9 (loses ILP) TheTom#5 2-pair half2: 12.0 (ternary overhead) TheTom#6 Select chain: 11.9 (branches kill) TheTom#7 Bit-arithmetic: 11.6 (ALU too heavy) TheTom#8 FMA branchless: 11.4 (ALU still too heavy) TheTom#9 Named-reg ternary: 10.3 (branches worst) TheTom#10 Main (8-LUT): 10.95 (baseline) TheTom#11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai
wel97459
pushed a commit
to wel97459/llama-cpp-turboquant
that referenced
this pull request
Jun 4, 2026
Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed TheTom#3 (TURBO_D). TheTom#1 and TheTom#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
wel97459
pushed a commit
to wel97459/llama-cpp-turboquant
that referenced
this pull request
Jun 4, 2026
Complete experiment log: TheTom#1 4-mag LUT: 15.1 at 8K (BEST, +38%) TheTom#2 Batched extract: 13.7 (+25%) TheTom#3 Inline FA block: 13.5 (I-cache pressure) TheTom#4 Deferred norm: 12.9 (loses ILP) TheTom#5 2-pair half2: 12.0 (ternary overhead) TheTom#6 Select chain: 11.9 (branches kill) TheTom#7 Bit-arithmetic: 11.6 (ALU too heavy) TheTom#8 FMA branchless: 11.4 (ALU still too heavy) TheTom#9 Named-reg ternary: 10.3 (branches worst) TheTom#10 Main (8-LUT): 10.95 (baseline) TheTom#11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai
KGardevoir
pushed a commit
to KGardevoir/llama-cpp-turboquant
that referenced
this pull request
Jun 16, 2026
Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed TheTom#3 (TURBO_D). #1 and TheTom#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
KGardevoir
pushed a commit
to KGardevoir/llama-cpp-turboquant
that referenced
this pull request
Jun 16, 2026
Complete experiment log: #1 4-mag LUT: 15.1 at 8K (BEST, +38%) TheTom#2 Batched extract: 13.7 (+25%) TheTom#3 Inline FA block: 13.5 (I-cache pressure) TheTom#4 Deferred norm: 12.9 (loses ILP) TheTom#5 2-pair half2: 12.0 (ternary overhead) TheTom#6 Select chain: 11.9 (branches kill) TheTom#7 Bit-arithmetic: 11.6 (ALU too heavy) TheTom#8 FMA branchless: 11.4 (ALU still too heavy) TheTom#9 Named-reg ternary: 10.3 (branches worst) TheTom#10 Main (8-LUT): 10.95 (baseline) TheTom#11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai
aureliahazel93-commits
added a commit
to aureliahazel93-commits/llama-cpp-turboquant
that referenced
this pull request
Jul 8, 2026
LogicDaemon
added a commit
to LogicDaemon/llama-cpp-turboquant
that referenced
this pull request
Sep 6, 2026
commit bfe452a6d315b776c2e06f7f11c45bd6cd9766b4
Author: LogicDaemon <LogicDaemon@users.noreply.github.com>
Date: Sat Sep 5 13:29:03 2026 +0400
Squashed commit of the following:
commit 10f8aff
Author: thecodacus <thecodacus@gmail.com>
Date: Tue Aug 25 23:39:14 2026 +0530
perf: lower default --cpu-tp-margin to 512 MiB
Measured on a 12GB RTX 3060 with a dense 27B Q4_K_M: tightening the margin
raises both decode and prefill, since every MiB reclaimed moves weights off
the CPU leg. 1024 -> 640 -> 384 MiB gave 7.4 -> 7.8 -> 8.1 t/s decode and
14.4 -> 15.2 -> 16.5 t/s prefill. 512 keeps headroom for larger contexts;
raise it if the GPU runs out of memory.
commit e406c8c
Author: thecodacus <thecodacus@gmail.com>
Date: Tue Aug 25 22:08:09 2026 +0530
feat: add --cpu-tp to use the CPU as a tensor-parallel device
llama.cpp's tensor-parallel meta device already dispatches each member
backend asynchronously before synchronising, but the CPU is excluded from
the device list. Adding it lets the GPU and CPU compute row-slices of every
weight matrix concurrently instead of running one after the other.
The GPU share is sized from the measured free VRAM (minus --cpu-tp-margin)
so VRAM is saturated, and the CPU takes only the remainder. The CPU is
appended last so the GPU subgraph is already in flight when the CPU
backend blocks on its own.
commit d927e7d
Author: Anirban Kar <thecodacus@gmail.com>
Date: Tue Aug 25 15:58:16 2026 +0530
docs: add MoE expert cache usage and tuning guide to README (TheTom#8)
* docs: add MoE expert cache usage and tuning guide to README
* docs: document async CPU split overlap and VRAM headroom rule
commit 0ac3d9b
Author: Anirban Kar <thecodacus@gmail.com>
Date: Sun Aug 23 04:08:21 2026 +0530
feat: overlap CPU cold-expert and GPU hot-expert computation in the MoE cache (TheTom#9)
* feat: overlap CPU cold-expert chain with GPU hot chain (GGML_SCHED_ASYNC_CPU)
The dual-path MoE cache ran its two chains serially: the scheduler
blocks on CPU splits, and the cold chain's input copies drain the whole
CUDA stream. Three changes make them concurrent, gated behind
GGML_SCHED_ASYNC_CPU=1:
- sched: CPU splits execute on a persistent worker thread; the main
loop keeps launching later splits and joins lazily at the first split
that is CPU or reads a CPU-resident input
- graph: cold chain is built before the hot chain so the hot split
follows it and carries no cross-backend inputs (launches while the
worker computes)
- graph: the merge add is pinned to CPU so it does not pull the hot
chain into a split that waits on the cold result
* feat: GGML_SCHED_ASYNC_CPU=2 diagnostic mode (worker without graph restructure)
* fix: mark expert cache pack buffer as weights
Without the weights usage flag the scheduler's assignment pass does not
pin the pack tensors to their backend, so a CPU-assigned consumer (the
pinned merge in async mode) propagates backward and pulls the hot
matmuls onto the CPU, copying pack weights over PCIe every layer.
* feat: make async CPU splits the default, expose --sched-async-cpu flag
Replaces the GGML_SCHED_ASYNC_CPU env var: the overlap is now on by
default and controlled through llama_context_params.sched_async_cpu.
- ggml: ggml_backend_sched_set_async_cpu() setter, env read removed
- llama: context param + cparams plumbing, graph gate reads cparams
- common: --sched-async-cpu / --no-sched-async-cpu (server, cli)
- llama-bench: sched_async_cpu test dimension (--sched-async-cpu 0,1
benches both in one run, column shown as 'sac' when non-default)
* fix: restrict CPU-pinned merge to decode-size batches
For prefill batches the CPU merge and its per-layer activation copies
cost more than the overlap hides (pp256 regressed 307 -> 257 t/s).
Decode and MTP verify batches (n_tokens <= 8) keep the overlap.
commit 8e5bc28
Author: Anirban Kar <thecodacus@gmail.com>
Date: Sat Jul 25 03:54:47 2026 +0530
feat: wire MoE expert cache into laguna graph (TheTom#6)
commit ac743f8
Author: Anirban Kar <thecodacus@gmail.com>
Date: Sat Jul 25 00:41:02 2026 +0530
feat: MoE expert cache — keep hot routed experts resident in VRAM (TheTom#5)
* feat: llama-moe-trace tool + expert-cache policy simulator
Phase 0 of the MoE expert-cache work: capture routed expert ids
(ffn_moe_topk) per layer per decode step via the scheduler eval
callback — backend-agnostic, so CPU-offloaded expert layers are
traced too. Companion simulate.py replays traces against cache
policies (static top-S, LRU, LFU-decay, periodic re-election) and
prints hit-rate vs VRAM-budget curves plus routing-skew stats to
decide whether expert caching is worth building at all.
* fix: moe-trace — common_init_from_params returns ptr with accessors
* fix: moe-trace — include arg.h for common_params_parse
* fix: moe-trace — take output path from MOE_TRACE_OUT env, -o not in this arg set
* fix: moe-trace — topk tensor is a strided view; copy full byte range and index by nb
Sizing the host buffer as n_used*n_tokens under-allocated: ffn_moe_topk
is a view over the argsort output, so nb[1] spans n_expert ids per row
and ggml_backend_tensor_get writes the whole strided range (heap
corruption, malloc corrupted-top-size abort mid-prefill).
* feat: moe-trace simulator v2 — model upload bytes/token and miss/upload cost per policy
* feat: mul_mat_id tolerates id=-1 (skip) on CPU and CUDA fallback path
Groundwork for the hot/cold expert-pack split: an id of -1 means the
routed expert is not owned by this pack. The op contributes a zeroed
output row for that slot so hot-pack and cold-pack results merge
additively. CUDA fast paths (mmvq_id/mmq_id/mmf_id) still assert
non-negative ids and are handled next.
* feat: id=-1 skip support in CUDA mul_mat_id fast paths + tests
mmid.cu gains a generic kernel that zeroes the dst rows of (token,
slot) pairs whose expert id is -1; mmq and mmf launch it after the
shared mm_ids_helper (which already drops unmatched ids from its
compact mapping), and mmvq launches it before its kernels, which now
early-return instead of using a negative id as a channel index (out
of bounds read otherwise). test-backend-ops gains skip_ids variants
of MUL_MAT_ID covering the mmvq/mmq/mmf/fallback dispatch corners.
* fix: skip_ids MUL_MAT_ID test cases were inside an #if 0 block
* fix: mm_ids_helper compact map corruption when ids contain -1
Negative ids inflated nex_prev for every expert, opening a phantom
region at the head of the compact arrays that nothing wrote; the mmq
quantize preprocessing iterates the flat range and read garbage row
indices (illegal memory access). Skipped ids now occupy no compact
position, ids_src1 is sentinel-filled, and the quantize gather/scatter
kernels skip negative entries.
* fix: mmvf id=-1 skip guard (float mul_mat_id vec path)
Same treatment as mmvq: kernels early-return on a negative expert id
instead of using it as a channel index, and the host launcher
pre-zeroes the dst rows of skipped slots. Fixes wrong results (and a
latent out-of-bounds channel read) for f16/f32 MUL_MAT_ID with -1 ids
at small batch sizes.
* fix: zero-init ids_dst before mm_ids_helper — garbage tail entries in the
compact map caused pool-layout-dependent illegal memory accesses when ids
contain -1 (tile-padded reads consumed uninitialized row indices)
* feat: route MUL_MAT_ID with skip-capable ids away from mmq/mmf
Ops whose ids may contain -1 announce it via op_params[0]; the CUDA
dispatch then uses only the paths with verified skip support (mmvq
for small batches, the general gather/scatter path otherwise).
Large-batch mmq with sparse ids shows a pool-layout-dependent fault
that resists isolation (all indexing audits pass; sanitizer and
cuda-gdb unusable on the target driver); since the expert-pack
design only decodes through packs, the safe route costs nothing in
practice. mmq/mmf guards from earlier commits remain as hardening.
* fix: op_params write in test — ggml_set_op_params_i32 is internal-only
* feat: MoE expert cache loader — hot-pack build from routing profile
init_moe_expert_cache() runs post-load: reads a moe-trace profile
(GGML_MOE_CACHE_PROFILE) and slot count (GGML_MOE_CACHE_SLOTS), and for
each CPU-resident MoE layer builds GPU-resident hot packs of the top-S
experts plus i32 remap tables (hot: global->slot or -1; cold: global id
or -1). Graph wiring (build_moe_ffn dual path) is the next commit; these
tensors are unused until then, so this is behavior-neutral.
* feat: MoE expert cache graph wiring — dual hot/cold mul_mat_id in build_moe_ffn
When a layer carries hot packs, routed ids are remapped through
moe_map_hot/cold (get_rows) and each expert projection runs as two
skip-flagged mul_mat_id ops — hot pack on GPU, original tensors for
cold — whose zero-padded outputs sum to the exact single-tensor
result. Engaged for qwen35moe (separate gate/up/down, no expert
scales); layers with merged gate_up, scales, or no packs use the
unchanged path.
* fix: flatten routed ids before map get_rows (batched-gather shape rule)
* fix: topk ids are a strided view - cont before flatten for map lookup
* fix: sched expert-copy path skips id=-1 (hot/cold expert split)
* fix: restructure MoE pack dual path into full FFN chains per residency side
Per-projection hot/cold interleave let the scheduler's split minimizer
assign hot up/down matmuls to CPU, copying the GPU pack weights over
PCIe every layer. Building one unbroken gate-up-swiglu-down chain per
side keeps the hot pack on GPU; skipped (-1) rows stay zero through
swiglu so a single add of the two chain outputs is exact.
* feat: wire MoE expert cache into deepseek2 graph (GLM-4.7-Flash)
* feat: add --moe-cache-profile and --moe-cache-slots flags
Plumbed through llama_model_params; env vars remain as fallback so
existing deployments keep working. Flags allow per-model settings in
multi-model serving.
* docs: fix moe-trace usage comment (output path is MOE_TRACE_OUT env)
commit cd72c28
Author: Anirban Kar <thecodacus@gmail.com>
Date: Fri Jul 24 17:53:41 2026 +0530
fix: restore closing brace lost in dequantize.cuh merge resolution
commit 34f893e
Merge: 40f2036 8f5ab83
Author: Anirban Kar <thecodacus@gmail.com>
Date: Fri Jul 24 17:52:26 2026 +0530
merge: sync upstream master into perf
commit 8f5ab83
Author: Matt Thompson <111157855+boondocklabs@users.noreply.github.com>
Date: Fri Jul 24 03:54:47 2026 -0700
cohere2 moe template parser: enforce JSON schema for text responses if a response schema is provided (ggml-org#26018)
commit 0cea362
Author: Xuan-Son Nguyen <son@huggingface.co>
Date: Fri Jul 24 08:02:23 2026 +0200
vendor: update subprocess.h (ggml-org#26061)
commit 40f2036
Author: Anirban Kar <thecodacus@gmail.com>
Date: Sun Jul 19 04:34:24 2026 +0530
feat: fused turbo MMA flash-attention decode (turbo4/3/2, GQA tensor-core path) (TheTom#4)
* feat(fattn): fused turbo4 MMA decode path (opt-in) — based on the spiritbuun fork MMA approach, adapted to graph-rotated Q
Port buun's fused turbo4 tensor-core (MMA) flash-attention DECODE kernel into
the turbo fork as an OPT-IN path (GGML_TURBO_MMA_FUSED=1, default OFF), routing
turbo4-K==turbo4-V, D in {128,256}, decode (Q->ne[1]<=4) onto the GQA-packed MMA
path so KV is read once per head-group instead of once per query head.
How it works:
- fattn-mma-f16.cuh: add flash_attn_ext_turbo4_load_tile (dequant turbo4 blocks
-> SRAM in the f16 row-major half2 layout) seeded with OUR Lloyd-Max centroids
(turbo-quant.cuh:297, NOT buun's -0.2415xx table). Thread type_K/type_V through
iter/process_tile/global flash_attn_ext_f16 (defaults F16 -> existing f16/q8 MMA
byte-identical). For turbo: force nstages=0 (the cp.async multi-stage path would
read raw turbo bytes as half2), pass RAW byte pitch nb11/nb21 (not /sizeof(half2)),
and dequantize K/V in the load tiles. sizeof(block_turbo4_0)-driven pointer math
(66B block, no rnorm) — never assumes a fixed offset.
- fattn-mma-turbo.cuh + 14 instance files (7 reachable ncols pairs x D in {128,256}):
host launcher reusing the f16 device kernel with type_K/type_V=TURBO4_0 and
need_f16_K/V=false so launch_fattn does NOT pre-convert KV to f16.
- fattn.cu: switch_ncols2 -> per-ncols2 dispatchers (only the 7 compiled instances),
env latch, and the gate (Q untouched — our fork already rotates Q at the graph
level, so NO inline FWHT / src swap, which would double-rotate Q).
Results (Qwen3.6-35B-A3B turbo4, GQA8, D=256, 5090):
depth MMA VEC+PDL buun
8192 213.8 188.5 200 (+6.9% vs buun)
16384 207.6 171.7 193 (+7.5% vs buun)
32768 191.3 141.1 187 (+2.3% vs buun)
MMA holds flat at depth where VEC collapses; the gap vs VEC grows with KV length.
Quality: Mean KLD vs f16 base = 0.008396 (== VEC baseline). The MMA path is NOT
strictly token-identical to VEC — MMA and VEC accumulate the P*V (VKQ) reduction in
f16 with different reduction trees, so a near-tie greedy token can flip (~1/25 on a
hard tie; same irreducible f16-order effect that exists between base f16-MMA and
f16-VEC, and within the VEC-vs-base "same-top-p" 96.3% noise floor). Because strict
token-identity does not hold, the gate DEFAULTS OFF and ships as opt-in; VEC remains
the untouched default/kill-switch.
(cherry picked from commit 5450669510a5cd0a356a863923b057f2552c6fbb)
(cherry picked from commit b3e51cf)
* turbo4 MMA decode: default ON (GQA-packed tensor-core path, quality-neutral; GGML_TURBO_MMA_FUSED=0 kill-switch)
(cherry picked from commit 726234b538194c644be4066e1413b91f9edf12e7)
(cherry picked from commit 38dc665)
* fattn: extend fused MMA decode to turbo3 + turbo2 (parity on turbo4; extends the MMA path to turbo3/2)
The GQA-packed tensor-core decode path was turbo4-only; turbo3/turbo2 fell back to
VEC and collapsed at depth (turbo3 tg32 224->124 from d2048->d32768), losing to
spiritbuun there. Added flash_attn_ext_turbo3_load_tile (3-bit split index: 2 low
bits from qs + 1 high bit from signs) and flash_attn_ext_turbo2_load_tile (plain
2-bit), type-dispatched both the K and V MMA load sites, and routed the gate by
K->type. is_turbo_kv already matches any non-f16 KV, so the kernel needed only the
per-type tile loaders + instances (14 instance files now emit all three types).
Correctness: MMA decode is bit-exact to VEC (Mean KLD @2048 chunks=8:
turbo3 0.020278==0.020278, turbo2 0.041490==0.041490, turbo4 0.009296 unchanged).
Decode tg32 (qwen3.6-35B-A3B, RTX 5090), MMA now flat at depth:
turbo3 2k/8k/16k/32k = 224/225/217/203 (buun 193/193/183/178; was VEC 224/187/160/124)
turbo2 2k/8k/16k/32k = 228/224/217/203 (buun 213/210/204/200; was VEC 229/218/203/183)
turbo3/turbo2 now hold flat at depth (neither fork had them on the MMA path before). GGML_TURBO_MMA_FUSED=0 restores VEC.
(cherry picked from commit 57ad5edd53a7fced0b99fecb99326132cbdb65d0)
(cherry picked from commit 4e223ee)
* fix(fattn): gate turbo2 fused-MMA decode to head_dim 128
Tester @everson isolated a turbo2-only decode regression on RTX 5060 Ti
(Gemma-12B, head_dim 256): turbo2/turbo2 -2.46% at 8K to -1.17% at 200K,
fully restored by GGML_TURBO_MMA_FUSED=0 at every depth, while turbo3 was
neutral and turbo4 exact parity. At 2-bit KV the fused path's GQA-pack
saving is tiny while its dequant/no-pipeline overhead is unchanged, so on
head_dim 256 it is neutral on high-bandwidth GPUs (Ornith-35B and a dense
Gemma-4-12B on RTX 5090, both within noise) and only costs on
bandwidth-limited cards.
turbo2 fused remains a large depth win on dense head_dim 128 (Llama-3.1-8B
on RTX 5090: +7% at 8K, +24% at 32K, +69% at 131K), so this gates turbo2
fused to head_dim 128 only; head_dim 256 turbo2 falls through to VEC, which
is the exact baseline path GGML_TURBO_MMA_FUSED=0 restored. turbo3 and
turbo4 fused are unchanged at both head dims.
Also corrects the stale "default OFF" gate comments to match the
default-ON code.
(cherry picked from commit 539ce5d)
---------
Co-authored-by: TheTom <tturney1@gmail.com>
commit 4cd54c6
Author: Anirban Kar <thecodacus@gmail.com>
Date: Sun Jul 19 04:14:41 2026 +0530
fix: allow mixed turbo/q8_0 KV cache types in CUDA flash-attention selection (TheTom#3)
* fix: allow mixed turbo/q8_0 KV cache types in CUDA flash-attention selection
Without GGML_CUDA_FA_ALL_QUANTS (the default), kernel selection
rejected any K/V type mismatch. A turbo KV setup is almost always
mixed (ctk q8_0 + ctv turbo2/3), so the flash-attention probe failed,
FA was auto-disabled, and context creation either refused the
quantized V cache outright or fell back to the non-FA path whose
KQ buffer (~8.7 GiB at 64k ctx, ubatch 1024) OOMs at load.
Allow exactly the mixed pairs that have FA template instances
compiled in: turbo2/3/4 against each other, q8_0, and f16. Also gate
turbo head-dim geometry (multiples of 64; 128 for turbo4) so shapes
without a compiled instance report unsupported instead of asserting
at dispatch.
Forcing -fa on masked the bug (the probe is skipped); server
deployments use fa auto and hit it on every load.
* feat: accept turbo2/3/4 cache type names in llama-bench
The common arg parser already accepts them for llama-completion and
llama-server; llama-bench has its own name table and rejected -ctk/-ctv
turbo values.
commit 9338207
Author: Anirban Kar <thecodacus@gmail.com>
Date: Sun Jul 19 01:59:58 2026 +0530
CUDA: add Q2_0 support (TheTom#2)
Co-authored-by: Pasha Khosravi <khosravipasha@gmail.com>
commit 170322f
Author: Anirban Kar <thecodacus@gmail.com>
Date: Sat Jul 18 15:27:13 2026 +0530
feat: TurboQuant KV-cache quantization — all backends (TheTom#1)
* feat: TurboQuant KV-cache quantization — core types + CPU backend
Port of the TurboQuant codec (arXiv 2504.19874, reference:
TheTom/llama-cpp-turboquant) onto this fork's base: WHT-rotated polar
quantization for the KV cache.
- new types TURBO2_0/TURBO3_0/TURBO4_0 (KV, 2/3/4-bit blocks of 128)
and TQ3_1S/TQ4_1S (WHT-rotated Lloyd-Max weight formats), renumbered
for this tree's type table
- GGML_OP_TURBO_WHT + ggml_turbo_wht() builder (forward/inverse
Walsh-Hadamard rotation with sign arrays, InnerQ scale hook)
- ggml-turbo-quant.c codec core, rotation data tables, CPU op +
vec_dot paths, type traits, quantize-chunk wiring
- CUDA/Metal/Vulkan kernel sources imported (wired in follow-ups)
- verified: test-turbo-quant round-trip on CPU (cosine 0.986-0.996)
* feat: turbo KV cache — graph wiring and CLI types
- pre-rotate Q (forward WHT) at every cache-backed attention site when
K is a turbo type; no-op otherwise
- inverse WHT on attention output in both FA and non-FA paths of
build_attn_mha (group size derived from K, contiguity enforced)
- register turbo2/3/4 as --cache-type-k/--cache-type-v values
Core scope: no InnerQ scaling, no head-dim padding — turbo KV requires
128-aligned (or 64-group) head dims
* feat: turbo KV cache — Metal backend (WHT kernel + FA dispatch)
Wire the TurboQuant Metal backend host side and kernels on top of the
CPU core already on this branch.
Metal kernels (ggml-metal.metal):
- Inlined turbo-wht.h (WHT sign arrays incl. half4 variants + FWHT
butterfly + rotate fwd/inverse) and the turbo2/3/4 centroids.
- quantize/dequantize_turbo2_0/3_0/4_0 (+_t4) and full-block cache.
- kernel_turbo_wht (O(d log d) Walsh-Hadamard rotation op) and
kernel_turbo4_dequant_f16 bulk prefill dequant.
- kernel_set_rows_turbo/2/4 cache-quantize kernels + instantiations.
- flash_attn_ext / flash_attn_ext_vec instantiations for symmetric and
asymmetric turbo2/3/4 and q8_0<->turbo pairs.
- TurboFlash two-pass kernels (p1/p2), default-disabled via env.
Host side:
- ggml-metal-device.cpp: get_pipeline_turbo_wht; flash-attn pipeline
base names now encode both K and V types (k%s_v%s) so asymmetric K/V
pairs resolve unambiguously. All existing FA kernel host_names in the
shader renamed to match.
- ggml-metal-device.m: TurboQuant compile-time env prep (4-mag LUT auto
select, sparse-V, profile mode); supports_op accepts turbo K types and
asymmetric turbo/q8_0 FA pairs; GGML_OP_TURBO_WHT.
- ggml-metal-ops.cpp: GGML_OP_TURBO_WHT encode case + impl; asymmetric
K/V assert relaxation; TurboFlash gating (use_turbo_flash, extra_tmp
sizing, force-nonvec) + two-pass dispatch; +128 float FA smem for the
turbo block cache.
- ggml-metal-impl.h / -device.h / -ops.h: kargs (turbo_wht,
turbo_flash_p1/p2), FC offsets, pipeline/op declarations.
Adaptations vs the reference tree (older upstream vintage):
- Reference intertwines separate TQ3_1S/TQ4_1S weight-quant work; only
the turbo KV-cache lines are ported here.
- set_rows_turbo dispatch selection is not wired: our upstream refactored
get_pipeline_set_rows/kernel_set_rows_q32 to a different signature, and
it is out of scope for this KV port. supports_op therefore does not
claim CPY/SET_ROWS turbo support, so cache writes fall back to CPU;
the set_rows_turbo shader kernels are present for when dispatch lands.
- No InnerQ scale tensors touched (graph passes nullptr scale).
* feat: turbo KV cache — CUDA backend (set-rows quantize, fattn vec dispatch, dequant)
Port the TurboQuant CUDA backend from the reference implementation, adapted
to this newer upstream vintage:
- set-rows.cu: k_set_rows_turbo2/3/4 kernels (WHT rotation + norm-corrected
PolarQuant KV write, InnerQ calibration hooks) + dispatch. turbo4 rnorm
write dropped (field removed from block_turbo4_0 on this branch).
- dequantize.cuh: dequantize_turbo2/3/4_0 and dequantize_tq3_1s/tq4_1s
device functions.
- convert.cu: turbo + TQ dequant cases in the fp16/fp32 contiguous and
non-contiguous dispatch tables; warp-cooperative TQ4_1S dequant kernel.
- fattn-common.cuh: vec_dot_fattn_vec_KQ_turbo2/3/4_0, dequantize_V_turbo*
and get_vec_dot_KQ / get_dequantize_V dispatch entries.
- fattn-vec.cuh: turbo K types take the float-Q path (Q_q8_1 off), turbo V
uses the ne=4 quantized dequant path; extern decls for the 21 compiled
K/V type combinations. Reference-side perf extras (shared-mem centroid
LUT, sparse-V skip, nthreads tuning) intentionally not ported.
- fattn.cu: turbo types accepted in kv_type_supported, vec kernel cases
registered. Decode goes through the VEC kernel; prefill converts KV to
f16 via the ported nc dequant path.
- fattn-mma-turbo.cuh: reduced to a stub launcher. No typed MMA kernel
exists in the reference either; the pre-existing template instances now
compile against the stub and the dispatcher never selects this path.
- ggml-cuda.cu: GGML_OP_TURBO_WHT dispatch + supports_op, SET_ROWS turbo
gates (head_dim % 64, turbo4 % 128).
- CMakeLists.txt: turbo fattn-vec instances added to the non-FA_ALL_QUANTS
build.
Verified on RTX 3060 (sm_86) with Bonsai-27B-Q1_0, partial offload, fa on:
coherent generation for turbo2/3/4; perplexity over 4 chunks:
f16 KV 7.9486 +/- 0.73, turbo3 KV 8.0261 +/- 0.74 (CUDA) vs 8.0279 (CPU-only).
* feat: turbo KV cache — Vulkan backend (set-rows quantize, turbo3 FA dispatch, WHT op)
- register turbo_wht.comp and set_rows turbo2/3/4 quantize variants in
vulkan-shaders-gen; add dedicated DATA_A_TURBO3_0 flash-attn SPIR-V
(scalar + coopmat1) with turbo3-only K/V bindings
- copy_to_quant: 128-thread per-block set_rows path (normalize, randomized
WHT, Lloyd-Max centroid quantize, subgroup pack), shared WHT tables
across the three turbo widths
- flash_attn_dequant: turbo2/3/4 centroid dequant macros wired into the
FaTypeK/FaTypeV runtime switch (ids 43/44/45); f16 K/V aliases guarded
out under DATA_A_TURBO3_0
- ggml-vulkan: GGML_OP_TURBO_WHT pipeline + dispatch, turbo set_rows
pipelines (required subgroup size 32), FA pipeline selection for
turbo3, supports_op gates (128-elem blocks, turbo3 K/V symmetry,
scalar/coopmat1 only — cm2 falls back to scalar)
* fix: extern initialization warning and C linkage for wht group size
CI builds with -Werror: defining the GGML_API-decorated global with an
initializer warns on every platform. Plain definition in the C TU; the
C++ consumer now declares it extern "C" (it previously referenced a
mangled symbol).
* feat: turbo2/turbo4 FA dequant for Vulkan
- generalize the dedicated turbo flash-attn SPIR-V path from turbo3-only
to all three tiers: DATA_A_TURBO_KV umbrella define in types.glsl gates
out the f16/q4/q5/q8 K/V binding aliases in flash_attn.comp,
flash_attn_cm1.comp and flash_attn_dequant.glsl for any turbo tier
- flash_attn_dequant: per-compilation dequantize4 for DATA_A_TURBO2_0
(2-bit centroids, fp16 norm + 32B idx) and DATA_A_TURBO4_0 (4-bit
nibble idx, fp16 norm + fp16 aux + 64B) alongside the turbo3 one
- vulkan-shaders-gen: register flash_attn_f32_f16_turbo{2,4}_0 scalar +
coopmat1 variants via a per-tier loop
- ggml-vulkan: FA pipeline selection picks the matching turbo SPIR-V for
turbo2/turbo4 k_type (scalar + cm1), supports_op requires symmetric
K/V per tier (turbo2/turbo2, turbo3/turbo3, turbo4/turbo4)
* fix: bump RPC proto patch version for GGML_OP_TURBO_WHT
* fix: export WHT group size via GGML_API for Windows shared builds
* fix: HIP/MUSA portability for turbo kernels
- map cudaMemcpyToSymbol/FromSymbol and __ballot_sync in vendor headers
- pass explicit width to __shfl*_sync (HIP macros take 4 args)
- drop unused warp_id locals flagged by nvcc -Werror
* fix: drop SYCL turbo fattn-vec instances (no turbo dequant support in SYCL fattn)
* fix: size vec_dot buffer by vec_dot_type and handle turbo/tq types in quantize test
- tmp_q2 overflowed for types with vec_dot_type F32 (4 bytes per element)
- skip turbo KV types: they quantize in the rotated domain, round trip
does not reproduce the input by design
- tq3_1s uses the 3-bit and lowbit thresholds
* fix: add turbo fattn-vec instances to HIP and MUSA source lists
* fix: add turbo fattn vec kernels to HIP VGPR ignore list
The gfx908 build of the ncols=2 turbo3 vec kernels lands at 259 total
VGPRs, slightly over the 256 gate. Register them like the other known
high-pressure fattn kernels.
commit 0888217
Merge: 5e7f627 7d56da7
Author: Anirban Kar <thecodacus@gmail.com>
Date: Fri Jul 17 20:02:48 2026 +0530
chore: sync perf with upstream master
commit 5e7f627
Author: Anirban Kar <thecodacus@gmail.com>
Date: Wed Jul 8 23:56:46 2026 +0530
docs(readme): add usage + benchmark instructions for the MoE-offload optimizations
commit 5f83fbb
Author: thecodacus <thecodacus@gmail.com>
Date: Thu Jul 2 12:42:55 2026 +0000
ggml : size prefetch slots per layer and fix fallback use-after-free
Profiling showed the 2-slot rotation stalls ~5.5ms per layer waiting for
the down_exps upload: 3 tensors per MoE layer need 3 slots for a full
layer of lookahead. Default is now 3 (one layer), configurable via the
env var value, degrading to however many slots fit on OOM.
Also fixes a use-after-free: on a failed slot regrow the repointed
staging tensors kept dangling pointers into freed device memory, which
graph reuse carried into later evals. Staging repoints are now restored
right after kernel launch, slots are sized once from the graph max, and
allocation happens before freeing.
Qwen3.6-35B-A3B pp2048 on RTX 3060: 1643 -> 1880 t/s (mainline: 1143).
commit 1163cb3
Author: thecodacus <thecodacus@gmail.com>
Date: Thu Jul 2 12:16:39 2026 +0000
ggml : overlap offloaded expert weight uploads with compute
At large batch sizes virtually every expert is used, so the per-layer
routing-ids readback that mainline waits on buys nothing while forcing a
full device sync per expert tensor (3x per MoE layer). Above a batch
threshold, upload the full expert tensors through a second backend
instance on the same device (own stream) with two event-ordered staging
slots, so uploads for tensor N+1 overlap compute of tensor N.
Qwen3.6-35B-A3B pp2048 on RTX 3060 (-ncmoe 26, ub 2048): 1383 -> 1663 t/s.
Generation output verified token-identical; decode path unaffected.
Opt-in via GGML_SCHED_PREFETCH_EXPERTS=1.
commit 20f5994
Author: thecodacus <thecodacus@gmail.com>
Date: Thu Jul 2 09:44:53 2026 +0000
llama : pin mmap-backed CPU weights for faster H2D uploads
Wire the existing GGML_CUDA_REGISTER_HOST path back up: after model load,
cudaHostRegister the mmap pages backing weights kept in system memory.
Recovers pageable-copy losses when MoE experts are streamed to the GPU
during prefill (n-cpu-moe): Qwen3.6-35B-A3B pp2048 1144 -> 1385 t/s on
RTX 3060. Opt-in via GGML_CUDA_REGISTER_HOST=1, unchanged otherwise.
LogicDaemon
added a commit
to LogicDaemon/llama-cpp-turboquant
that referenced
this pull request
Sep 7, 2026
commit 10f8aff Author: thecodacus <thecodacus@gmail.com> Date: Tue Aug 25 23:39:14 2026 +0530 perf: lower default --cpu-tp-margin to 512 MiB Measured on a 12GB RTX 3060 with a dense 27B Q4_K_M: tightening the margin raises both decode and prefill, since every MiB reclaimed moves weights off the CPU leg. 1024 -> 640 -> 384 MiB gave 7.4 -> 7.8 -> 8.1 t/s decode and 14.4 -> 15.2 -> 16.5 t/s prefill. 512 keeps headroom for larger contexts; raise it if the GPU runs out of memory. commit e406c8c Author: thecodacus <thecodacus@gmail.com> Date: Tue Aug 25 22:08:09 2026 +0530 feat: add --cpu-tp to use the CPU as a tensor-parallel device llama.cpp's tensor-parallel meta device already dispatches each member backend asynchronously before synchronising, but the CPU is excluded from the device list. Adding it lets the GPU and CPU compute row-slices of every weight matrix concurrently instead of running one after the other. The GPU share is sized from the measured free VRAM (minus --cpu-tp-margin) so VRAM is saturated, and the CPU takes only the remainder. The CPU is appended last so the GPU subgraph is already in flight when the CPU backend blocks on its own. commit d927e7d Author: Anirban Kar <thecodacus@gmail.com> Date: Tue Aug 25 15:58:16 2026 +0530 docs: add MoE expert cache usage and tuning guide to README (TheTom#8) * docs: add MoE expert cache usage and tuning guide to README * docs: document async CPU split overlap and VRAM headroom rule commit 0ac3d9b Author: Anirban Kar <thecodacus@gmail.com> Date: Sun Aug 23 04:08:21 2026 +0530 feat: overlap CPU cold-expert and GPU hot-expert computation in the MoE cache (TheTom#9) * feat: overlap CPU cold-expert chain with GPU hot chain (GGML_SCHED_ASYNC_CPU) The dual-path MoE cache ran its two chains serially: the scheduler blocks on CPU splits, and the cold chain's input copies drain the whole CUDA stream. Three changes make them concurrent, gated behind GGML_SCHED_ASYNC_CPU=1: - sched: CPU splits execute on a persistent worker thread; the main loop keeps launching later splits and joins lazily at the first split that is CPU or reads a CPU-resident input - graph: cold chain is built before the hot chain so the hot split follows it and carries no cross-backend inputs (launches while the worker computes) - graph: the merge add is pinned to CPU so it does not pull the hot chain into a split that waits on the cold result * feat: GGML_SCHED_ASYNC_CPU=2 diagnostic mode (worker without graph restructure) * fix: mark expert cache pack buffer as weights Without the weights usage flag the scheduler's assignment pass does not pin the pack tensors to their backend, so a CPU-assigned consumer (the pinned merge in async mode) propagates backward and pulls the hot matmuls onto the CPU, copying pack weights over PCIe every layer. * feat: make async CPU splits the default, expose --sched-async-cpu flag Replaces the GGML_SCHED_ASYNC_CPU env var: the overlap is now on by default and controlled through llama_context_params.sched_async_cpu. - ggml: ggml_backend_sched_set_async_cpu() setter, env read removed - llama: context param + cparams plumbing, graph gate reads cparams - common: --sched-async-cpu / --no-sched-async-cpu (server, cli) - llama-bench: sched_async_cpu test dimension (--sched-async-cpu 0,1 benches both in one run, column shown as 'sac' when non-default) * fix: restrict CPU-pinned merge to decode-size batches For prefill batches the CPU merge and its per-layer activation copies cost more than the overlap hides (pp256 regressed 307 -> 257 t/s). Decode and MTP verify batches (n_tokens <= 8) keep the overlap. commit 8e5bc28 Author: Anirban Kar <thecodacus@gmail.com> Date: Sat Jul 25 03:54:47 2026 +0530 feat: wire MoE expert cache into laguna graph (TheTom#6) commit ac743f8 Author: Anirban Kar <thecodacus@gmail.com> Date: Sat Jul 25 00:41:02 2026 +0530 feat: MoE expert cache — keep hot routed experts resident in VRAM (TheTom#5) * feat: llama-moe-trace tool + expert-cache policy simulator Phase 0 of the MoE expert-cache work: capture routed expert ids (ffn_moe_topk) per layer per decode step via the scheduler eval callback — backend-agnostic, so CPU-offloaded expert layers are traced too. Companion simulate.py replays traces against cache policies (static top-S, LRU, LFU-decay, periodic re-election) and prints hit-rate vs VRAM-budget curves plus routing-skew stats to decide whether expert caching is worth building at all. * fix: moe-trace — common_init_from_params returns ptr with accessors * fix: moe-trace — include arg.h for common_params_parse * fix: moe-trace — take output path from MOE_TRACE_OUT env, -o not in this arg set * fix: moe-trace — topk tensor is a strided view; copy full byte range and index by nb Sizing the host buffer as n_used*n_tokens under-allocated: ffn_moe_topk is a view over the argsort output, so nb[1] spans n_expert ids per row and ggml_backend_tensor_get writes the whole strided range (heap corruption, malloc corrupted-top-size abort mid-prefill). * feat: moe-trace simulator v2 — model upload bytes/token and miss/upload cost per policy * feat: mul_mat_id tolerates id=-1 (skip) on CPU and CUDA fallback path Groundwork for the hot/cold expert-pack split: an id of -1 means the routed expert is not owned by this pack. The op contributes a zeroed output row for that slot so hot-pack and cold-pack results merge additively. CUDA fast paths (mmvq_id/mmq_id/mmf_id) still assert non-negative ids and are handled next. * feat: id=-1 skip support in CUDA mul_mat_id fast paths + tests mmid.cu gains a generic kernel that zeroes the dst rows of (token, slot) pairs whose expert id is -1; mmq and mmf launch it after the shared mm_ids_helper (which already drops unmatched ids from its compact mapping), and mmvq launches it before its kernels, which now early-return instead of using a negative id as a channel index (out of bounds read otherwise). test-backend-ops gains skip_ids variants of MUL_MAT_ID covering the mmvq/mmq/mmf/fallback dispatch corners. * fix: skip_ids MUL_MAT_ID test cases were inside an #if 0 block * fix: mm_ids_helper compact map corruption when ids contain -1 Negative ids inflated nex_prev for every expert, opening a phantom region at the head of the compact arrays that nothing wrote; the mmq quantize preprocessing iterates the flat range and read garbage row indices (illegal memory access). Skipped ids now occupy no compact position, ids_src1 is sentinel-filled, and the quantize gather/scatter kernels skip negative entries. * fix: mmvf id=-1 skip guard (float mul_mat_id vec path) Same treatment as mmvq: kernels early-return on a negative expert id instead of using it as a channel index, and the host launcher pre-zeroes the dst rows of skipped slots. Fixes wrong results (and a latent out-of-bounds channel read) for f16/f32 MUL_MAT_ID with -1 ids at small batch sizes. * fix: zero-init ids_dst before mm_ids_helper — garbage tail entries in the compact map caused pool-layout-dependent illegal memory accesses when ids contain -1 (tile-padded reads consumed uninitialized row indices) * feat: route MUL_MAT_ID with skip-capable ids away from mmq/mmf Ops whose ids may contain -1 announce it via op_params[0]; the CUDA dispatch then uses only the paths with verified skip support (mmvq for small batches, the general gather/scatter path otherwise). Large-batch mmq with sparse ids shows a pool-layout-dependent fault that resists isolation (all indexing audits pass; sanitizer and cuda-gdb unusable on the target driver); since the expert-pack design only decodes through packs, the safe route costs nothing in practice. mmq/mmf guards from earlier commits remain as hardening. * fix: op_params write in test — ggml_set_op_params_i32 is internal-only * feat: MoE expert cache loader — hot-pack build from routing profile init_moe_expert_cache() runs post-load: reads a moe-trace profile (GGML_MOE_CACHE_PROFILE) and slot count (GGML_MOE_CACHE_SLOTS), and for each CPU-resident MoE layer builds GPU-resident hot packs of the top-S experts plus i32 remap tables (hot: global->slot or -1; cold: global id or -1). Graph wiring (build_moe_ffn dual path) is the next commit; these tensors are unused until then, so this is behavior-neutral. * feat: MoE expert cache graph wiring — dual hot/cold mul_mat_id in build_moe_ffn When a layer carries hot packs, routed ids are remapped through moe_map_hot/cold (get_rows) and each expert projection runs as two skip-flagged mul_mat_id ops — hot pack on GPU, original tensors for cold — whose zero-padded outputs sum to the exact single-tensor result. Engaged for qwen35moe (separate gate/up/down, no expert scales); layers with merged gate_up, scales, or no packs use the unchanged path. * fix: flatten routed ids before map get_rows (batched-gather shape rule) * fix: topk ids are a strided view - cont before flatten for map lookup * fix: sched expert-copy path skips id=-1 (hot/cold expert split) * fix: restructure MoE pack dual path into full FFN chains per residency side Per-projection hot/cold interleave let the scheduler's split minimizer assign hot up/down matmuls to CPU, copying the GPU pack weights over PCIe every layer. Building one unbroken gate-up-swiglu-down chain per side keeps the hot pack on GPU; skipped (-1) rows stay zero through swiglu so a single add of the two chain outputs is exact. * feat: wire MoE expert cache into deepseek2 graph (GLM-4.7-Flash) * feat: add --moe-cache-profile and --moe-cache-slots flags Plumbed through llama_model_params; env vars remain as fallback so existing deployments keep working. Flags allow per-model settings in multi-model serving. * docs: fix moe-trace usage comment (output path is MOE_TRACE_OUT env) commit cd72c28 Author: Anirban Kar <thecodacus@gmail.com> Date: Fri Jul 24 17:53:41 2026 +0530 fix: restore closing brace lost in dequantize.cuh merge resolution commit 34f893e Merge: 40f2036 8f5ab83 Author: Anirban Kar <thecodacus@gmail.com> Date: Fri Jul 24 17:52:26 2026 +0530 merge: sync upstream master into perf commit 8f5ab83 Author: Matt Thompson <111157855+boondocklabs@users.noreply.github.com> Date: Fri Jul 24 03:54:47 2026 -0700 cohere2 moe template parser: enforce JSON schema for text responses if a response schema is provided (ggml-org#26018) commit 0cea362 Author: Xuan-Son Nguyen <son@huggingface.co> Date: Fri Jul 24 08:02:23 2026 +0200 vendor: update subprocess.h (ggml-org#26061) commit 40f2036 Author: Anirban Kar <thecodacus@gmail.com> Date: Sun Jul 19 04:34:24 2026 +0530 feat: fused turbo MMA flash-attention decode (turbo4/3/2, GQA tensor-core path) (TheTom#4) * feat(fattn): fused turbo4 MMA decode path (opt-in) — based on the spiritbuun fork MMA approach, adapted to graph-rotated Q Port buun's fused turbo4 tensor-core (MMA) flash-attention DECODE kernel into the turbo fork as an OPT-IN path (GGML_TURBO_MMA_FUSED=1, default OFF), routing turbo4-K==turbo4-V, D in {128,256}, decode (Q->ne[1]<=4) onto the GQA-packed MMA path so KV is read once per head-group instead of once per query head. How it works: - fattn-mma-f16.cuh: add flash_attn_ext_turbo4_load_tile (dequant turbo4 blocks -> SRAM in the f16 row-major half2 layout) seeded with OUR Lloyd-Max centroids (turbo-quant.cuh:297, NOT buun's -0.2415xx table). Thread type_K/type_V through iter/process_tile/global flash_attn_ext_f16 (defaults F16 -> existing f16/q8 MMA byte-identical). For turbo: force nstages=0 (the cp.async multi-stage path would read raw turbo bytes as half2), pass RAW byte pitch nb11/nb21 (not /sizeof(half2)), and dequantize K/V in the load tiles. sizeof(block_turbo4_0)-driven pointer math (66B block, no rnorm) — never assumes a fixed offset. - fattn-mma-turbo.cuh + 14 instance files (7 reachable ncols pairs x D in {128,256}): host launcher reusing the f16 device kernel with type_K/type_V=TURBO4_0 and need_f16_K/V=false so launch_fattn does NOT pre-convert KV to f16. - fattn.cu: switch_ncols2 -> per-ncols2 dispatchers (only the 7 compiled instances), env latch, and the gate (Q untouched — our fork already rotates Q at the graph level, so NO inline FWHT / src swap, which would double-rotate Q). Results (Qwen3.6-35B-A3B turbo4, GQA8, D=256, 5090): depth MMA VEC+PDL buun 8192 213.8 188.5 200 (+6.9% vs buun) 16384 207.6 171.7 193 (+7.5% vs buun) 32768 191.3 141.1 187 (+2.3% vs buun) MMA holds flat at depth where VEC collapses; the gap vs VEC grows with KV length. Quality: Mean KLD vs f16 base = 0.008396 (== VEC baseline). The MMA path is NOT strictly token-identical to VEC — MMA and VEC accumulate the P*V (VKQ) reduction in f16 with different reduction trees, so a near-tie greedy token can flip (~1/25 on a hard tie; same irreducible f16-order effect that exists between base f16-MMA and f16-VEC, and within the VEC-vs-base "same-top-p" 96.3% noise floor). Because strict token-identity does not hold, the gate DEFAULTS OFF and ships as opt-in; VEC remains the untouched default/kill-switch. (cherry picked from commit 5450669510a5cd0a356a863923b057f2552c6fbb) (cherry picked from commit b3e51cf) * turbo4 MMA decode: default ON (GQA-packed tensor-core path, quality-neutral; GGML_TURBO_MMA_FUSED=0 kill-switch) (cherry picked from commit 726234b538194c644be4066e1413b91f9edf12e7) (cherry picked from commit 38dc665) * fattn: extend fused MMA decode to turbo3 + turbo2 (parity on turbo4; extends the MMA path to turbo3/2) The GQA-packed tensor-core decode path was turbo4-only; turbo3/turbo2 fell back to VEC and collapsed at depth (turbo3 tg32 224->124 from d2048->d32768), losing to spiritbuun there. Added flash_attn_ext_turbo3_load_tile (3-bit split index: 2 low bits from qs + 1 high bit from signs) and flash_attn_ext_turbo2_load_tile (plain 2-bit), type-dispatched both the K and V MMA load sites, and routed the gate by K->type. is_turbo_kv already matches any non-f16 KV, so the kernel needed only the per-type tile loaders + instances (14 instance files now emit all three types). Correctness: MMA decode is bit-exact to VEC (Mean KLD @2048 chunks=8: turbo3 0.020278==0.020278, turbo2 0.041490==0.041490, turbo4 0.009296 unchanged). Decode tg32 (qwen3.6-35B-A3B, RTX 5090), MMA now flat at depth: turbo3 2k/8k/16k/32k = 224/225/217/203 (buun 193/193/183/178; was VEC 224/187/160/124) turbo2 2k/8k/16k/32k = 228/224/217/203 (buun 213/210/204/200; was VEC 229/218/203/183) turbo3/turbo2 now hold flat at depth (neither fork had them on the MMA path before). GGML_TURBO_MMA_FUSED=0 restores VEC. (cherry picked from commit 57ad5edd53a7fced0b99fecb99326132cbdb65d0) (cherry picked from commit 4e223ee) * fix(fattn): gate turbo2 fused-MMA decode to head_dim 128 Tester @everson isolated a turbo2-only decode regression on RTX 5060 Ti (Gemma-12B, head_dim 256): turbo2/turbo2 -2.46% at 8K to -1.17% at 200K, fully restored by GGML_TURBO_MMA_FUSED=0 at every depth, while turbo3 was neutral and turbo4 exact parity. At 2-bit KV the fused path's GQA-pack saving is tiny while its dequant/no-pipeline overhead is unchanged, so on head_dim 256 it is neutral on high-bandwidth GPUs (Ornith-35B and a dense Gemma-4-12B on RTX 5090, both within noise) and only costs on bandwidth-limited cards. turbo2 fused remains a large depth win on dense head_dim 128 (Llama-3.1-8B on RTX 5090: +7% at 8K, +24% at 32K, +69% at 131K), so this gates turbo2 fused to head_dim 128 only; head_dim 256 turbo2 falls through to VEC, which is the exact baseline path GGML_TURBO_MMA_FUSED=0 restored. turbo3 and turbo4 fused are unchanged at both head dims. Also corrects the stale "default OFF" gate comments to match the default-ON code. (cherry picked from commit 539ce5d) --------- Co-authored-by: TheTom <tturney1@gmail.com> commit 4cd54c6 Author: Anirban Kar <thecodacus@gmail.com> Date: Sun Jul 19 04:14:41 2026 +0530 fix: allow mixed turbo/q8_0 KV cache types in CUDA flash-attention selection (TheTom#3) * fix: allow mixed turbo/q8_0 KV cache types in CUDA flash-attention selection Without GGML_CUDA_FA_ALL_QUANTS (the default), kernel selection rejected any K/V type mismatch. A turbo KV setup is almost always mixed (ctk q8_0 + ctv turbo2/3), so the flash-attention probe failed, FA was auto-disabled, and context creation either refused the quantized V cache outright or fell back to the non-FA path whose KQ buffer (~8.7 GiB at 64k ctx, ubatch 1024) OOMs at load. Allow exactly the mixed pairs that have FA template instances compiled in: turbo2/3/4 against each other, q8_0, and f16. Also gate turbo head-dim geometry (multiples of 64; 128 for turbo4) so shapes without a compiled instance report unsupported instead of asserting at dispatch. Forcing -fa on masked the bug (the probe is skipped); server deployments use fa auto and hit it on every load. * feat: accept turbo2/3/4 cache type names in llama-bench The common arg parser already accepts them for llama-completion and llama-server; llama-bench has its own name table and rejected -ctk/-ctv turbo values. commit 9338207 Author: Anirban Kar <thecodacus@gmail.com> Date: Sun Jul 19 01:59:58 2026 +0530 CUDA: add Q2_0 support (TheTom#2) Co-authored-by: Pasha Khosravi <khosravipasha@gmail.com> commit 170322f Author: Anirban Kar <thecodacus@gmail.com> Date: Sat Jul 18 15:27:13 2026 +0530 feat: TurboQuant KV-cache quantization — all backends (TheTom#1) * feat: TurboQuant KV-cache quantization — core types + CPU backend Port of the TurboQuant codec (arXiv 2504.19874, reference: TheTom/llama-cpp-turboquant) onto this fork's base: WHT-rotated polar quantization for the KV cache. - new types TURBO2_0/TURBO3_0/TURBO4_0 (KV, 2/3/4-bit blocks of 128) and TQ3_1S/TQ4_1S (WHT-rotated Lloyd-Max weight formats), renumbered for this tree's type table - GGML_OP_TURBO_WHT + ggml_turbo_wht() builder (forward/inverse Walsh-Hadamard rotation with sign arrays, InnerQ scale hook) - ggml-turbo-quant.c codec core, rotation data tables, CPU op + vec_dot paths, type traits, quantize-chunk wiring - CUDA/Metal/Vulkan kernel sources imported (wired in follow-ups) - verified: test-turbo-quant round-trip on CPU (cosine 0.986-0.996) * feat: turbo KV cache — graph wiring and CLI types - pre-rotate Q (forward WHT) at every cache-backed attention site when K is a turbo type; no-op otherwise - inverse WHT on attention output in both FA and non-FA paths of build_attn_mha (group size derived from K, contiguity enforced) - register turbo2/3/4 as --cache-type-k/--cache-type-v values Core scope: no InnerQ scaling, no head-dim padding — turbo KV requires 128-aligned (or 64-group) head dims * feat: turbo KV cache — Metal backend (WHT kernel + FA dispatch) Wire the TurboQuant Metal backend host side and kernels on top of the CPU core already on this branch. Metal kernels (ggml-metal.metal): - Inlined turbo-wht.h (WHT sign arrays incl. half4 variants + FWHT butterfly + rotate fwd/inverse) and the turbo2/3/4 centroids. - quantize/dequantize_turbo2_0/3_0/4_0 (+_t4) and full-block cache. - kernel_turbo_wht (O(d log d) Walsh-Hadamard rotation op) and kernel_turbo4_dequant_f16 bulk prefill dequant. - kernel_set_rows_turbo/2/4 cache-quantize kernels + instantiations. - flash_attn_ext / flash_attn_ext_vec instantiations for symmetric and asymmetric turbo2/3/4 and q8_0<->turbo pairs. - TurboFlash two-pass kernels (p1/p2), default-disabled via env. Host side: - ggml-metal-device.cpp: get_pipeline_turbo_wht; flash-attn pipeline base names now encode both K and V types (k%s_v%s) so asymmetric K/V pairs resolve unambiguously. All existing FA kernel host_names in the shader renamed to match. - ggml-metal-device.m: TurboQuant compile-time env prep (4-mag LUT auto select, sparse-V, profile mode); supports_op accepts turbo K types and asymmetric turbo/q8_0 FA pairs; GGML_OP_TURBO_WHT. - ggml-metal-ops.cpp: GGML_OP_TURBO_WHT encode case + impl; asymmetric K/V assert relaxation; TurboFlash gating (use_turbo_flash, extra_tmp sizing, force-nonvec) + two-pass dispatch; +128 float FA smem for the turbo block cache. - ggml-metal-impl.h / -device.h / -ops.h: kargs (turbo_wht, turbo_flash_p1/p2), FC offsets, pipeline/op declarations. Adaptations vs the reference tree (older upstream vintage): - Reference intertwines separate TQ3_1S/TQ4_1S weight-quant work; only the turbo KV-cache lines are ported here. - set_rows_turbo dispatch selection is not wired: our upstream refactored get_pipeline_set_rows/kernel_set_rows_q32 to a different signature, and it is out of scope for this KV port. supports_op therefore does not claim CPY/SET_ROWS turbo support, so cache writes fall back to CPU; the set_rows_turbo shader kernels are present for when dispatch lands. - No InnerQ scale tensors touched (graph passes nullptr scale). * feat: turbo KV cache — CUDA backend (set-rows quantize, fattn vec dispatch, dequant) Port the TurboQuant CUDA backend from the reference implementation, adapted to this newer upstream vintage: - set-rows.cu: k_set_rows_turbo2/3/4 kernels (WHT rotation + norm-corrected PolarQuant KV write, InnerQ calibration hooks) + dispatch. turbo4 rnorm write dropped (field removed from block_turbo4_0 on this branch). - dequantize.cuh: dequantize_turbo2/3/4_0 and dequantize_tq3_1s/tq4_1s device functions. - convert.cu: turbo + TQ dequant cases in the fp16/fp32 contiguous and non-contiguous dispatch tables; warp-cooperative TQ4_1S dequant kernel. - fattn-common.cuh: vec_dot_fattn_vec_KQ_turbo2/3/4_0, dequantize_V_turbo* and get_vec_dot_KQ / get_dequantize_V dispatch entries. - fattn-vec.cuh: turbo K types take the float-Q path (Q_q8_1 off), turbo V uses the ne=4 quantized dequant path; extern decls for the 21 compiled K/V type combinations. Reference-side perf extras (shared-mem centroid LUT, sparse-V skip, nthreads tuning) intentionally not ported. - fattn.cu: turbo types accepted in kv_type_supported, vec kernel cases registered. Decode goes through the VEC kernel; prefill converts KV to f16 via the ported nc dequant path. - fattn-mma-turbo.cuh: reduced to a stub launcher. No typed MMA kernel exists in the reference either; the pre-existing template instances now compile against the stub and the dispatcher never selects this path. - ggml-cuda.cu: GGML_OP_TURBO_WHT dispatch + supports_op, SET_ROWS turbo gates (head_dim % 64, turbo4 % 128). - CMakeLists.txt: turbo fattn-vec instances added to the non-FA_ALL_QUANTS build. Verified on RTX 3060 (sm_86) with Bonsai-27B-Q1_0, partial offload, fa on: coherent generation for turbo2/3/4; perplexity over 4 chunks: f16 KV 7.9486 +/- 0.73, turbo3 KV 8.0261 +/- 0.74 (CUDA) vs 8.0279 (CPU-only). * feat: turbo KV cache — Vulkan backend (set-rows quantize, turbo3 FA dispatch, WHT op) - register turbo_wht.comp and set_rows turbo2/3/4 quantize variants in vulkan-shaders-gen; add dedicated DATA_A_TURBO3_0 flash-attn SPIR-V (scalar + coopmat1) with turbo3-only K/V bindings - copy_to_quant: 128-thread per-block set_rows path (normalize, randomized WHT, Lloyd-Max centroid quantize, subgroup pack), shared WHT tables across the three turbo widths - flash_attn_dequant: turbo2/3/4 centroid dequant macros wired into the FaTypeK/FaTypeV runtime switch (ids 43/44/45); f16 K/V aliases guarded out under DATA_A_TURBO3_0 - ggml-vulkan: GGML_OP_TURBO_WHT pipeline + dispatch, turbo set_rows pipelines (required subgroup size 32), FA pipeline selection for turbo3, supports_op gates (128-elem blocks, turbo3 K/V symmetry, scalar/coopmat1 only — cm2 falls back to scalar) * fix: extern initialization warning and C linkage for wht group size CI builds with -Werror: defining the GGML_API-decorated global with an initializer warns on every platform. Plain definition in the C TU; the C++ consumer now declares it extern "C" (it previously referenced a mangled symbol). * feat: turbo2/turbo4 FA dequant for Vulkan - generalize the dedicated turbo flash-attn SPIR-V path from turbo3-only to all three tiers: DATA_A_TURBO_KV umbrella define in types.glsl gates out the f16/q4/q5/q8 K/V binding aliases in flash_attn.comp, flash_attn_cm1.comp and flash_attn_dequant.glsl for any turbo tier - flash_attn_dequant: per-compilation dequantize4 for DATA_A_TURBO2_0 (2-bit centroids, fp16 norm + 32B idx) and DATA_A_TURBO4_0 (4-bit nibble idx, fp16 norm + fp16 aux + 64B) alongside the turbo3 one - vulkan-shaders-gen: register flash_attn_f32_f16_turbo{2,4}_0 scalar + coopmat1 variants via a per-tier loop - ggml-vulkan: FA pipeline selection picks the matching turbo SPIR-V for turbo2/turbo4 k_type (scalar + cm1), supports_op requires symmetric K/V per tier (turbo2/turbo2, turbo3/turbo3, turbo4/turbo4) * fix: bump RPC proto patch version for GGML_OP_TURBO_WHT * fix: export WHT group size via GGML_API for Windows shared builds * fix: HIP/MUSA portability for turbo kernels - map cudaMemcpyToSymbol/FromSymbol and __ballot_sync in vendor headers - pass explicit width to __shfl*_sync (HIP macros take 4 args) - drop unused warp_id locals flagged by nvcc -Werror * fix: drop SYCL turbo fattn-vec instances (no turbo dequant support in SYCL fattn) * fix: size vec_dot buffer by vec_dot_type and handle turbo/tq types in quantize test - tmp_q2 overflowed for types with vec_dot_type F32 (4 bytes per element) - skip turbo KV types: they quantize in the rotated domain, round trip does not reproduce the input by design - tq3_1s uses the 3-bit and lowbit thresholds * fix: add turbo fattn-vec instances to HIP and MUSA source lists * fix: add turbo fattn vec kernels to HIP VGPR ignore list The gfx908 build of the ncols=2 turbo3 vec kernels lands at 259 total VGPRs, slightly over the 256 gate. Register them like the other known high-pressure fattn kernels. commit 0888217 Merge: 5e7f627 7d56da7 Author: Anirban Kar <thecodacus@gmail.com> Date: Fri Jul 17 20:02:48 2026 +0530 chore: sync perf with upstream master commit 5e7f627 Author: Anirban Kar <thecodacus@gmail.com> Date: Wed Jul 8 23:56:46 2026 +0530 docs(readme): add usage + benchmark instructions for the MoE-offload optimizations commit 5f83fbb Author: thecodacus <thecodacus@gmail.com> Date: Thu Jul 2 12:42:55 2026 +0000 ggml : size prefetch slots per layer and fix fallback use-after-free Profiling showed the 2-slot rotation stalls ~5.5ms per layer waiting for the down_exps upload: 3 tensors per MoE layer need 3 slots for a full layer of lookahead. Default is now 3 (one layer), configurable via the env var value, degrading to however many slots fit on OOM. Also fixes a use-after-free: on a failed slot regrow the repointed staging tensors kept dangling pointers into freed device memory, which graph reuse carried into later evals. Staging repoints are now restored right after kernel launch, slots are sized once from the graph max, and allocation happens before freeing. Qwen3.6-35B-A3B pp2048 on RTX 3060: 1643 -> 1880 t/s (mainline: 1143). commit 1163cb3 Author: thecodacus <thecodacus@gmail.com> Date: Thu Jul 2 12:16:39 2026 +0000 ggml : overlap offloaded expert weight uploads with compute At large batch sizes virtually every expert is used, so the per-layer routing-ids readback that mainline waits on buys nothing while forcing a full device sync per expert tensor (3x per MoE layer). Above a batch threshold, upload the full expert tensors through a second backend instance on the same device (own stream) with two event-ordered staging slots, so uploads for tensor N+1 overlap compute of tensor N. Qwen3.6-35B-A3B pp2048 on RTX 3060 (-ncmoe 26, ub 2048): 1383 -> 1663 t/s. Generation output verified token-identical; decode path unaffected. Opt-in via GGML_SCHED_PREFETCH_EXPERTS=1. commit 20f5994 Author: thecodacus <thecodacus@gmail.com> Date: Thu Jul 2 09:44:53 2026 +0000 llama : pin mmap-backed CPU weights for faster H2D uploads Wire the existing GGML_CUDA_REGISTER_HOST path back up: after model load, cudaHostRegister the mmap pages backing weights kept in system memory. Recovers pageable-copy losses when MoE experts are streamed to the GPU during prefill (n-cpu-moe): Qwen3.6-35B-A3B pp2048 1144 -> 1385 t/s on RTX 3060. Opt-in via GGML_CUDA_REGISTER_HOST=1, unchanged otherwise.
LogicDaemon
added a commit
to LogicDaemon/llama-cpp-turboquant
that referenced
this pull request
Sep 15, 2026
commit 10f8aff Author: thecodacus <thecodacus@gmail.com> Date: Tue Aug 25 23:39:14 2026 +0530 perf: lower default --cpu-tp-margin to 512 MiB Measured on a 12GB RTX 3060 with a dense 27B Q4_K_M: tightening the margin raises both decode and prefill, since every MiB reclaimed moves weights off the CPU leg. 1024 -> 640 -> 384 MiB gave 7.4 -> 7.8 -> 8.1 t/s decode and 14.4 -> 15.2 -> 16.5 t/s prefill. 512 keeps headroom for larger contexts; raise it if the GPU runs out of memory. commit e406c8c Author: thecodacus <thecodacus@gmail.com> Date: Tue Aug 25 22:08:09 2026 +0530 feat: add --cpu-tp to use the CPU as a tensor-parallel device llama.cpp's tensor-parallel meta device already dispatches each member backend asynchronously before synchronising, but the CPU is excluded from the device list. Adding it lets the GPU and CPU compute row-slices of every weight matrix concurrently instead of running one after the other. The GPU share is sized from the measured free VRAM (minus --cpu-tp-margin) so VRAM is saturated, and the CPU takes only the remainder. The CPU is appended last so the GPU subgraph is already in flight when the CPU backend blocks on its own. commit d927e7d Author: Anirban Kar <thecodacus@gmail.com> Date: Tue Aug 25 15:58:16 2026 +0530 docs: add MoE expert cache usage and tuning guide to README (TheTom#8) * docs: add MoE expert cache usage and tuning guide to README * docs: document async CPU split overlap and VRAM headroom rule commit 0ac3d9b Author: Anirban Kar <thecodacus@gmail.com> Date: Sun Aug 23 04:08:21 2026 +0530 feat: overlap CPU cold-expert and GPU hot-expert computation in the MoE cache (TheTom#9) * feat: overlap CPU cold-expert chain with GPU hot chain (GGML_SCHED_ASYNC_CPU) The dual-path MoE cache ran its two chains serially: the scheduler blocks on CPU splits, and the cold chain's input copies drain the whole CUDA stream. Three changes make them concurrent, gated behind GGML_SCHED_ASYNC_CPU=1: - sched: CPU splits execute on a persistent worker thread; the main loop keeps launching later splits and joins lazily at the first split that is CPU or reads a CPU-resident input - graph: cold chain is built before the hot chain so the hot split follows it and carries no cross-backend inputs (launches while the worker computes) - graph: the merge add is pinned to CPU so it does not pull the hot chain into a split that waits on the cold result * feat: GGML_SCHED_ASYNC_CPU=2 diagnostic mode (worker without graph restructure) * fix: mark expert cache pack buffer as weights Without the weights usage flag the scheduler's assignment pass does not pin the pack tensors to their backend, so a CPU-assigned consumer (the pinned merge in async mode) propagates backward and pulls the hot matmuls onto the CPU, copying pack weights over PCIe every layer. * feat: make async CPU splits the default, expose --sched-async-cpu flag Replaces the GGML_SCHED_ASYNC_CPU env var: the overlap is now on by default and controlled through llama_context_params.sched_async_cpu. - ggml: ggml_backend_sched_set_async_cpu() setter, env read removed - llama: context param + cparams plumbing, graph gate reads cparams - common: --sched-async-cpu / --no-sched-async-cpu (server, cli) - llama-bench: sched_async_cpu test dimension (--sched-async-cpu 0,1 benches both in one run, column shown as 'sac' when non-default) * fix: restrict CPU-pinned merge to decode-size batches For prefill batches the CPU merge and its per-layer activation copies cost more than the overlap hides (pp256 regressed 307 -> 257 t/s). Decode and MTP verify batches (n_tokens <= 8) keep the overlap. commit 8e5bc28 Author: Anirban Kar <thecodacus@gmail.com> Date: Sat Jul 25 03:54:47 2026 +0530 feat: wire MoE expert cache into laguna graph (TheTom#6) commit ac743f8 Author: Anirban Kar <thecodacus@gmail.com> Date: Sat Jul 25 00:41:02 2026 +0530 feat: MoE expert cache — keep hot routed experts resident in VRAM (TheTom#5) * feat: llama-moe-trace tool + expert-cache policy simulator Phase 0 of the MoE expert-cache work: capture routed expert ids (ffn_moe_topk) per layer per decode step via the scheduler eval callback — backend-agnostic, so CPU-offloaded expert layers are traced too. Companion simulate.py replays traces against cache policies (static top-S, LRU, LFU-decay, periodic re-election) and prints hit-rate vs VRAM-budget curves plus routing-skew stats to decide whether expert caching is worth building at all. * fix: moe-trace — common_init_from_params returns ptr with accessors * fix: moe-trace — include arg.h for common_params_parse * fix: moe-trace — take output path from MOE_TRACE_OUT env, -o not in this arg set * fix: moe-trace — topk tensor is a strided view; copy full byte range and index by nb Sizing the host buffer as n_used*n_tokens under-allocated: ffn_moe_topk is a view over the argsort output, so nb[1] spans n_expert ids per row and ggml_backend_tensor_get writes the whole strided range (heap corruption, malloc corrupted-top-size abort mid-prefill). * feat: moe-trace simulator v2 — model upload bytes/token and miss/upload cost per policy * feat: mul_mat_id tolerates id=-1 (skip) on CPU and CUDA fallback path Groundwork for the hot/cold expert-pack split: an id of -1 means the routed expert is not owned by this pack. The op contributes a zeroed output row for that slot so hot-pack and cold-pack results merge additively. CUDA fast paths (mmvq_id/mmq_id/mmf_id) still assert non-negative ids and are handled next. * feat: id=-1 skip support in CUDA mul_mat_id fast paths + tests mmid.cu gains a generic kernel that zeroes the dst rows of (token, slot) pairs whose expert id is -1; mmq and mmf launch it after the shared mm_ids_helper (which already drops unmatched ids from its compact mapping), and mmvq launches it before its kernels, which now early-return instead of using a negative id as a channel index (out of bounds read otherwise). test-backend-ops gains skip_ids variants of MUL_MAT_ID covering the mmvq/mmq/mmf/fallback dispatch corners. * fix: skip_ids MUL_MAT_ID test cases were inside an #if 0 block * fix: mm_ids_helper compact map corruption when ids contain -1 Negative ids inflated nex_prev for every expert, opening a phantom region at the head of the compact arrays that nothing wrote; the mmq quantize preprocessing iterates the flat range and read garbage row indices (illegal memory access). Skipped ids now occupy no compact position, ids_src1 is sentinel-filled, and the quantize gather/scatter kernels skip negative entries. * fix: mmvf id=-1 skip guard (float mul_mat_id vec path) Same treatment as mmvq: kernels early-return on a negative expert id instead of using it as a channel index, and the host launcher pre-zeroes the dst rows of skipped slots. Fixes wrong results (and a latent out-of-bounds channel read) for f16/f32 MUL_MAT_ID with -1 ids at small batch sizes. * fix: zero-init ids_dst before mm_ids_helper — garbage tail entries in the compact map caused pool-layout-dependent illegal memory accesses when ids contain -1 (tile-padded reads consumed uninitialized row indices) * feat: route MUL_MAT_ID with skip-capable ids away from mmq/mmf Ops whose ids may contain -1 announce it via op_params[0]; the CUDA dispatch then uses only the paths with verified skip support (mmvq for small batches, the general gather/scatter path otherwise). Large-batch mmq with sparse ids shows a pool-layout-dependent fault that resists isolation (all indexing audits pass; sanitizer and cuda-gdb unusable on the target driver); since the expert-pack design only decodes through packs, the safe route costs nothing in practice. mmq/mmf guards from earlier commits remain as hardening. * fix: op_params write in test — ggml_set_op_params_i32 is internal-only * feat: MoE expert cache loader — hot-pack build from routing profile init_moe_expert_cache() runs post-load: reads a moe-trace profile (GGML_MOE_CACHE_PROFILE) and slot count (GGML_MOE_CACHE_SLOTS), and for each CPU-resident MoE layer builds GPU-resident hot packs of the top-S experts plus i32 remap tables (hot: global->slot or -1; cold: global id or -1). Graph wiring (build_moe_ffn dual path) is the next commit; these tensors are unused until then, so this is behavior-neutral. * feat: MoE expert cache graph wiring — dual hot/cold mul_mat_id in build_moe_ffn When a layer carries hot packs, routed ids are remapped through moe_map_hot/cold (get_rows) and each expert projection runs as two skip-flagged mul_mat_id ops — hot pack on GPU, original tensors for cold — whose zero-padded outputs sum to the exact single-tensor result. Engaged for qwen35moe (separate gate/up/down, no expert scales); layers with merged gate_up, scales, or no packs use the unchanged path. * fix: flatten routed ids before map get_rows (batched-gather shape rule) * fix: topk ids are a strided view - cont before flatten for map lookup * fix: sched expert-copy path skips id=-1 (hot/cold expert split) * fix: restructure MoE pack dual path into full FFN chains per residency side Per-projection hot/cold interleave let the scheduler's split minimizer assign hot up/down matmuls to CPU, copying the GPU pack weights over PCIe every layer. Building one unbroken gate-up-swiglu-down chain per side keeps the hot pack on GPU; skipped (-1) rows stay zero through swiglu so a single add of the two chain outputs is exact. * feat: wire MoE expert cache into deepseek2 graph (GLM-4.7-Flash) * feat: add --moe-cache-profile and --moe-cache-slots flags Plumbed through llama_model_params; env vars remain as fallback so existing deployments keep working. Flags allow per-model settings in multi-model serving. * docs: fix moe-trace usage comment (output path is MOE_TRACE_OUT env) commit cd72c28 Author: Anirban Kar <thecodacus@gmail.com> Date: Fri Jul 24 17:53:41 2026 +0530 fix: restore closing brace lost in dequantize.cuh merge resolution commit 34f893e Merge: 40f2036 8f5ab83 Author: Anirban Kar <thecodacus@gmail.com> Date: Fri Jul 24 17:52:26 2026 +0530 merge: sync upstream master into perf commit 8f5ab83 Author: Matt Thompson <111157855+boondocklabs@users.noreply.github.com> Date: Fri Jul 24 03:54:47 2026 -0700 cohere2 moe template parser: enforce JSON schema for text responses if a response schema is provided (ggml-org#26018) commit 0cea362 Author: Xuan-Son Nguyen <son@huggingface.co> Date: Fri Jul 24 08:02:23 2026 +0200 vendor: update subprocess.h (ggml-org#26061) commit 40f2036 Author: Anirban Kar <thecodacus@gmail.com> Date: Sun Jul 19 04:34:24 2026 +0530 feat: fused turbo MMA flash-attention decode (turbo4/3/2, GQA tensor-core path) (TheTom#4) * feat(fattn): fused turbo4 MMA decode path (opt-in) — based on the spiritbuun fork MMA approach, adapted to graph-rotated Q Port buun's fused turbo4 tensor-core (MMA) flash-attention DECODE kernel into the turbo fork as an OPT-IN path (GGML_TURBO_MMA_FUSED=1, default OFF), routing turbo4-K==turbo4-V, D in {128,256}, decode (Q->ne[1]<=4) onto the GQA-packed MMA path so KV is read once per head-group instead of once per query head. How it works: - fattn-mma-f16.cuh: add flash_attn_ext_turbo4_load_tile (dequant turbo4 blocks -> SRAM in the f16 row-major half2 layout) seeded with OUR Lloyd-Max centroids (turbo-quant.cuh:297, NOT buun's -0.2415xx table). Thread type_K/type_V through iter/process_tile/global flash_attn_ext_f16 (defaults F16 -> existing f16/q8 MMA byte-identical). For turbo: force nstages=0 (the cp.async multi-stage path would read raw turbo bytes as half2), pass RAW byte pitch nb11/nb21 (not /sizeof(half2)), and dequantize K/V in the load tiles. sizeof(block_turbo4_0)-driven pointer math (66B block, no rnorm) — never assumes a fixed offset. - fattn-mma-turbo.cuh + 14 instance files (7 reachable ncols pairs x D in {128,256}): host launcher reusing the f16 device kernel with type_K/type_V=TURBO4_0 and need_f16_K/V=false so launch_fattn does NOT pre-convert KV to f16. - fattn.cu: switch_ncols2 -> per-ncols2 dispatchers (only the 7 compiled instances), env latch, and the gate (Q untouched — our fork already rotates Q at the graph level, so NO inline FWHT / src swap, which would double-rotate Q). Results (Qwen3.6-35B-A3B turbo4, GQA8, D=256, 5090): depth MMA VEC+PDL buun 8192 213.8 188.5 200 (+6.9% vs buun) 16384 207.6 171.7 193 (+7.5% vs buun) 32768 191.3 141.1 187 (+2.3% vs buun) MMA holds flat at depth where VEC collapses; the gap vs VEC grows with KV length. Quality: Mean KLD vs f16 base = 0.008396 (== VEC baseline). The MMA path is NOT strictly token-identical to VEC — MMA and VEC accumulate the P*V (VKQ) reduction in f16 with different reduction trees, so a near-tie greedy token can flip (~1/25 on a hard tie; same irreducible f16-order effect that exists between base f16-MMA and f16-VEC, and within the VEC-vs-base "same-top-p" 96.3% noise floor). Because strict token-identity does not hold, the gate DEFAULTS OFF and ships as opt-in; VEC remains the untouched default/kill-switch. (cherry picked from commit 5450669510a5cd0a356a863923b057f2552c6fbb) (cherry picked from commit b3e51cf) * turbo4 MMA decode: default ON (GQA-packed tensor-core path, quality-neutral; GGML_TURBO_MMA_FUSED=0 kill-switch) (cherry picked from commit 726234b538194c644be4066e1413b91f9edf12e7) (cherry picked from commit 38dc665) * fattn: extend fused MMA decode to turbo3 + turbo2 (parity on turbo4; extends the MMA path to turbo3/2) The GQA-packed tensor-core decode path was turbo4-only; turbo3/turbo2 fell back to VEC and collapsed at depth (turbo3 tg32 224->124 from d2048->d32768), losing to spiritbuun there. Added flash_attn_ext_turbo3_load_tile (3-bit split index: 2 low bits from qs + 1 high bit from signs) and flash_attn_ext_turbo2_load_tile (plain 2-bit), type-dispatched both the K and V MMA load sites, and routed the gate by K->type. is_turbo_kv already matches any non-f16 KV, so the kernel needed only the per-type tile loaders + instances (14 instance files now emit all three types). Correctness: MMA decode is bit-exact to VEC (Mean KLD @2048 chunks=8: turbo3 0.020278==0.020278, turbo2 0.041490==0.041490, turbo4 0.009296 unchanged). Decode tg32 (qwen3.6-35B-A3B, RTX 5090), MMA now flat at depth: turbo3 2k/8k/16k/32k = 224/225/217/203 (buun 193/193/183/178; was VEC 224/187/160/124) turbo2 2k/8k/16k/32k = 228/224/217/203 (buun 213/210/204/200; was VEC 229/218/203/183) turbo3/turbo2 now hold flat at depth (neither fork had them on the MMA path before). GGML_TURBO_MMA_FUSED=0 restores VEC. (cherry picked from commit 57ad5edd53a7fced0b99fecb99326132cbdb65d0) (cherry picked from commit 4e223ee) * fix(fattn): gate turbo2 fused-MMA decode to head_dim 128 Tester @everson isolated a turbo2-only decode regression on RTX 5060 Ti (Gemma-12B, head_dim 256): turbo2/turbo2 -2.46% at 8K to -1.17% at 200K, fully restored by GGML_TURBO_MMA_FUSED=0 at every depth, while turbo3 was neutral and turbo4 exact parity. At 2-bit KV the fused path's GQA-pack saving is tiny while its dequant/no-pipeline overhead is unchanged, so on head_dim 256 it is neutral on high-bandwidth GPUs (Ornith-35B and a dense Gemma-4-12B on RTX 5090, both within noise) and only costs on bandwidth-limited cards. turbo2 fused remains a large depth win on dense head_dim 128 (Llama-3.1-8B on RTX 5090: +7% at 8K, +24% at 32K, +69% at 131K), so this gates turbo2 fused to head_dim 128 only; head_dim 256 turbo2 falls through to VEC, which is the exact baseline path GGML_TURBO_MMA_FUSED=0 restored. turbo3 and turbo4 fused are unchanged at both head dims. Also corrects the stale "default OFF" gate comments to match the default-ON code. (cherry picked from commit 539ce5d) --------- Co-authored-by: TheTom <tturney1@gmail.com> commit 4cd54c6 Author: Anirban Kar <thecodacus@gmail.com> Date: Sun Jul 19 04:14:41 2026 +0530 fix: allow mixed turbo/q8_0 KV cache types in CUDA flash-attention selection (TheTom#3) * fix: allow mixed turbo/q8_0 KV cache types in CUDA flash-attention selection Without GGML_CUDA_FA_ALL_QUANTS (the default), kernel selection rejected any K/V type mismatch. A turbo KV setup is almost always mixed (ctk q8_0 + ctv turbo2/3), so the flash-attention probe failed, FA was auto-disabled, and context creation either refused the quantized V cache outright or fell back to the non-FA path whose KQ buffer (~8.7 GiB at 64k ctx, ubatch 1024) OOMs at load. Allow exactly the mixed pairs that have FA template instances compiled in: turbo2/3/4 against each other, q8_0, and f16. Also gate turbo head-dim geometry (multiples of 64; 128 for turbo4) so shapes without a compiled instance report unsupported instead of asserting at dispatch. Forcing -fa on masked the bug (the probe is skipped); server deployments use fa auto and hit it on every load. * feat: accept turbo2/3/4 cache type names in llama-bench The common arg parser already accepts them for llama-completion and llama-server; llama-bench has its own name table and rejected -ctk/-ctv turbo values. commit 9338207 Author: Anirban Kar <thecodacus@gmail.com> Date: Sun Jul 19 01:59:58 2026 +0530 CUDA: add Q2_0 support (TheTom#2) Co-authored-by: Pasha Khosravi <khosravipasha@gmail.com> commit 170322f Author: Anirban Kar <thecodacus@gmail.com> Date: Sat Jul 18 15:27:13 2026 +0530 feat: TurboQuant KV-cache quantization — all backends (TheTom#1) * feat: TurboQuant KV-cache quantization — core types + CPU backend Port of the TurboQuant codec (arXiv 2504.19874, reference: TheTom/llama-cpp-turboquant) onto this fork's base: WHT-rotated polar quantization for the KV cache. - new types TURBO2_0/TURBO3_0/TURBO4_0 (KV, 2/3/4-bit blocks of 128) and TQ3_1S/TQ4_1S (WHT-rotated Lloyd-Max weight formats), renumbered for this tree's type table - GGML_OP_TURBO_WHT + ggml_turbo_wht() builder (forward/inverse Walsh-Hadamard rotation with sign arrays, InnerQ scale hook) - ggml-turbo-quant.c codec core, rotation data tables, CPU op + vec_dot paths, type traits, quantize-chunk wiring - CUDA/Metal/Vulkan kernel sources imported (wired in follow-ups) - verified: test-turbo-quant round-trip on CPU (cosine 0.986-0.996) * feat: turbo KV cache — graph wiring and CLI types - pre-rotate Q (forward WHT) at every cache-backed attention site when K is a turbo type; no-op otherwise - inverse WHT on attention output in both FA and non-FA paths of build_attn_mha (group size derived from K, contiguity enforced) - register turbo2/3/4 as --cache-type-k/--cache-type-v values Core scope: no InnerQ scaling, no head-dim padding — turbo KV requires 128-aligned (or 64-group) head dims * feat: turbo KV cache — Metal backend (WHT kernel + FA dispatch) Wire the TurboQuant Metal backend host side and kernels on top of the CPU core already on this branch. Metal kernels (ggml-metal.metal): - Inlined turbo-wht.h (WHT sign arrays incl. half4 variants + FWHT butterfly + rotate fwd/inverse) and the turbo2/3/4 centroids. - quantize/dequantize_turbo2_0/3_0/4_0 (+_t4) and full-block cache. - kernel_turbo_wht (O(d log d) Walsh-Hadamard rotation op) and kernel_turbo4_dequant_f16 bulk prefill dequant. - kernel_set_rows_turbo/2/4 cache-quantize kernels + instantiations. - flash_attn_ext / flash_attn_ext_vec instantiations for symmetric and asymmetric turbo2/3/4 and q8_0<->turbo pairs. - TurboFlash two-pass kernels (p1/p2), default-disabled via env. Host side: - ggml-metal-device.cpp: get_pipeline_turbo_wht; flash-attn pipeline base names now encode both K and V types (k%s_v%s) so asymmetric K/V pairs resolve unambiguously. All existing FA kernel host_names in the shader renamed to match. - ggml-metal-device.m: TurboQuant compile-time env prep (4-mag LUT auto select, sparse-V, profile mode); supports_op accepts turbo K types and asymmetric turbo/q8_0 FA pairs; GGML_OP_TURBO_WHT. - ggml-metal-ops.cpp: GGML_OP_TURBO_WHT encode case + impl; asymmetric K/V assert relaxation; TurboFlash gating (use_turbo_flash, extra_tmp sizing, force-nonvec) + two-pass dispatch; +128 float FA smem for the turbo block cache. - ggml-metal-impl.h / -device.h / -ops.h: kargs (turbo_wht, turbo_flash_p1/p2), FC offsets, pipeline/op declarations. Adaptations vs the reference tree (older upstream vintage): - Reference intertwines separate TQ3_1S/TQ4_1S weight-quant work; only the turbo KV-cache lines are ported here. - set_rows_turbo dispatch selection is not wired: our upstream refactored get_pipeline_set_rows/kernel_set_rows_q32 to a different signature, and it is out of scope for this KV port. supports_op therefore does not claim CPY/SET_ROWS turbo support, so cache writes fall back to CPU; the set_rows_turbo shader kernels are present for when dispatch lands. - No InnerQ scale tensors touched (graph passes nullptr scale). * feat: turbo KV cache — CUDA backend (set-rows quantize, fattn vec dispatch, dequant) Port the TurboQuant CUDA backend from the reference implementation, adapted to this newer upstream vintage: - set-rows.cu: k_set_rows_turbo2/3/4 kernels (WHT rotation + norm-corrected PolarQuant KV write, InnerQ calibration hooks) + dispatch. turbo4 rnorm write dropped (field removed from block_turbo4_0 on this branch). - dequantize.cuh: dequantize_turbo2/3/4_0 and dequantize_tq3_1s/tq4_1s device functions. - convert.cu: turbo + TQ dequant cases in the fp16/fp32 contiguous and non-contiguous dispatch tables; warp-cooperative TQ4_1S dequant kernel. - fattn-common.cuh: vec_dot_fattn_vec_KQ_turbo2/3/4_0, dequantize_V_turbo* and get_vec_dot_KQ / get_dequantize_V dispatch entries. - fattn-vec.cuh: turbo K types take the float-Q path (Q_q8_1 off), turbo V uses the ne=4 quantized dequant path; extern decls for the 21 compiled K/V type combinations. Reference-side perf extras (shared-mem centroid LUT, sparse-V skip, nthreads tuning) intentionally not ported. - fattn.cu: turbo types accepted in kv_type_supported, vec kernel cases registered. Decode goes through the VEC kernel; prefill converts KV to f16 via the ported nc dequant path. - fattn-mma-turbo.cuh: reduced to a stub launcher. No typed MMA kernel exists in the reference either; the pre-existing template instances now compile against the stub and the dispatcher never selects this path. - ggml-cuda.cu: GGML_OP_TURBO_WHT dispatch + supports_op, SET_ROWS turbo gates (head_dim % 64, turbo4 % 128). - CMakeLists.txt: turbo fattn-vec instances added to the non-FA_ALL_QUANTS build. Verified on RTX 3060 (sm_86) with Bonsai-27B-Q1_0, partial offload, fa on: coherent generation for turbo2/3/4; perplexity over 4 chunks: f16 KV 7.9486 +/- 0.73, turbo3 KV 8.0261 +/- 0.74 (CUDA) vs 8.0279 (CPU-only). * feat: turbo KV cache — Vulkan backend (set-rows quantize, turbo3 FA dispatch, WHT op) - register turbo_wht.comp and set_rows turbo2/3/4 quantize variants in vulkan-shaders-gen; add dedicated DATA_A_TURBO3_0 flash-attn SPIR-V (scalar + coopmat1) with turbo3-only K/V bindings - copy_to_quant: 128-thread per-block set_rows path (normalize, randomized WHT, Lloyd-Max centroid quantize, subgroup pack), shared WHT tables across the three turbo widths - flash_attn_dequant: turbo2/3/4 centroid dequant macros wired into the FaTypeK/FaTypeV runtime switch (ids 43/44/45); f16 K/V aliases guarded out under DATA_A_TURBO3_0 - ggml-vulkan: GGML_OP_TURBO_WHT pipeline + dispatch, turbo set_rows pipelines (required subgroup size 32), FA pipeline selection for turbo3, supports_op gates (128-elem blocks, turbo3 K/V symmetry, scalar/coopmat1 only — cm2 falls back to scalar) * fix: extern initialization warning and C linkage for wht group size CI builds with -Werror: defining the GGML_API-decorated global with an initializer warns on every platform. Plain definition in the C TU; the C++ consumer now declares it extern "C" (it previously referenced a mangled symbol). * feat: turbo2/turbo4 FA dequant for Vulkan - generalize the dedicated turbo flash-attn SPIR-V path from turbo3-only to all three tiers: DATA_A_TURBO_KV umbrella define in types.glsl gates out the f16/q4/q5/q8 K/V binding aliases in flash_attn.comp, flash_attn_cm1.comp and flash_attn_dequant.glsl for any turbo tier - flash_attn_dequant: per-compilation dequantize4 for DATA_A_TURBO2_0 (2-bit centroids, fp16 norm + 32B idx) and DATA_A_TURBO4_0 (4-bit nibble idx, fp16 norm + fp16 aux + 64B) alongside the turbo3 one - vulkan-shaders-gen: register flash_attn_f32_f16_turbo{2,4}_0 scalar + coopmat1 variants via a per-tier loop - ggml-vulkan: FA pipeline selection picks the matching turbo SPIR-V for turbo2/turbo4 k_type (scalar + cm1), supports_op requires symmetric K/V per tier (turbo2/turbo2, turbo3/turbo3, turbo4/turbo4) * fix: bump RPC proto patch version for GGML_OP_TURBO_WHT * fix: export WHT group size via GGML_API for Windows shared builds * fix: HIP/MUSA portability for turbo kernels - map cudaMemcpyToSymbol/FromSymbol and __ballot_sync in vendor headers - pass explicit width to __shfl*_sync (HIP macros take 4 args) - drop unused warp_id locals flagged by nvcc -Werror * fix: drop SYCL turbo fattn-vec instances (no turbo dequant support in SYCL fattn) * fix: size vec_dot buffer by vec_dot_type and handle turbo/tq types in quantize test - tmp_q2 overflowed for types with vec_dot_type F32 (4 bytes per element) - skip turbo KV types: they quantize in the rotated domain, round trip does not reproduce the input by design - tq3_1s uses the 3-bit and lowbit thresholds * fix: add turbo fattn-vec instances to HIP and MUSA source lists * fix: add turbo fattn vec kernels to HIP VGPR ignore list The gfx908 build of the ncols=2 turbo3 vec kernels lands at 259 total VGPRs, slightly over the 256 gate. Register them like the other known high-pressure fattn kernels. commit 0888217 Merge: 5e7f627 7d56da7 Author: Anirban Kar <thecodacus@gmail.com> Date: Fri Jul 17 20:02:48 2026 +0530 chore: sync perf with upstream master commit 5e7f627 Author: Anirban Kar <thecodacus@gmail.com> Date: Wed Jul 8 23:56:46 2026 +0530 docs(readme): add usage + benchmark instructions for the MoE-offload optimizations commit 5f83fbb Author: thecodacus <thecodacus@gmail.com> Date: Thu Jul 2 12:42:55 2026 +0000 ggml : size prefetch slots per layer and fix fallback use-after-free Profiling showed the 2-slot rotation stalls ~5.5ms per layer waiting for the down_exps upload: 3 tensors per MoE layer need 3 slots for a full layer of lookahead. Default is now 3 (one layer), configurable via the env var value, degrading to however many slots fit on OOM. Also fixes a use-after-free: on a failed slot regrow the repointed staging tensors kept dangling pointers into freed device memory, which graph reuse carried into later evals. Staging repoints are now restored right after kernel launch, slots are sized once from the graph max, and allocation happens before freeing. Qwen3.6-35B-A3B pp2048 on RTX 3060: 1643 -> 1880 t/s (mainline: 1143). commit 1163cb3 Author: thecodacus <thecodacus@gmail.com> Date: Thu Jul 2 12:16:39 2026 +0000 ggml : overlap offloaded expert weight uploads with compute At large batch sizes virtually every expert is used, so the per-layer routing-ids readback that mainline waits on buys nothing while forcing a full device sync per expert tensor (3x per MoE layer). Above a batch threshold, upload the full expert tensors through a second backend instance on the same device (own stream) with two event-ordered staging slots, so uploads for tensor N+1 overlap compute of tensor N. Qwen3.6-35B-A3B pp2048 on RTX 3060 (-ncmoe 26, ub 2048): 1383 -> 1663 t/s. Generation output verified token-identical; decode path unaffected. Opt-in via GGML_SCHED_PREFETCH_EXPERTS=1. commit 20f5994 Author: thecodacus <thecodacus@gmail.com> Date: Thu Jul 2 09:44:53 2026 +0000 llama : pin mmap-backed CPU weights for faster H2D uploads Wire the existing GGML_CUDA_REGISTER_HOST path back up: after model load, cudaHostRegister the mmap pages backing weights kept in system memory. Recovers pageable-copy losses when MoE experts are streamed to the GPU during prefill (n-cpu-moe): Qwen3.6-35B-A3B pp2048 1144 -> 1385 t/s on RTX 3060. Opt-in via GGML_CUDA_REGISTER_HOST=1, unchanged otherwise.
jasonlnheath
pushed a commit
to jasonlnheath/llama-cpp-turboquant
that referenced
this pull request
Oct 4, 2026
…lection (TheTom#3) * fix: allow mixed turbo/q8_0 KV cache types in CUDA flash-attention selection Without GGML_CUDA_FA_ALL_QUANTS (the default), kernel selection rejected any K/V type mismatch. A turbo KV setup is almost always mixed (ctk q8_0 + ctv turbo2/3), so the flash-attention probe failed, FA was auto-disabled, and context creation either refused the quantized V cache outright or fell back to the non-FA path whose KQ buffer (~8.7 GiB at 64k ctx, ubatch 1024) OOMs at load. Allow exactly the mixed pairs that have FA template instances compiled in: turbo2/3/4 against each other, q8_0, and f16. Also gate turbo head-dim geometry (multiples of 64; 128 for turbo4) so shapes without a compiled instance report unsupported instead of asserting at dispatch. Forcing -fa on masked the bug (the probe is skipped); server deployments use fa auto and hit it on every load. * feat: accept turbo2/3/4 cache type names in llama-bench The common arg parser already accepts them for llama-completion and llama-server; llama-bench has its own name table and rejected -ctk/-ctv turbo values.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR ports TurboQuant3 (turbo3) KV cache compression to CUDA, targeting SM 12.0 (RTX 5090 / Blackwell) with near-parity decode performance vs F16.
What's included
CUDA kernel port (
set-rows.cu,turbo-quant.cuh,turbo-wht.cu/cuh):k_set_rows_turbo3: quantises incoming F32 KV tokens into turbo3 blocks on GPU__shfl_sync(qs) and__ballot_sync(signs) — no atomicsFlash attention integration (
fattn-common.cuh,fattn-vec.cuh,fattn.cu):vec_dot_fattn_vec_KQ_turbo3_0: optimised KQ dot product — elem0/elem1 always share the same turbo3 block, so qs/signs loaded once per pair instead of twicedequantize_V_turbo3_0: ne==4 fast path — single qs byte + single signs byte covers all 4 elements; unrolled float2/half2 outputBug fix — VEC kernel Q/K stride mismatch:
vec_dot_fattn_vec_KQ_turbo3_0originally stepped the outer K loop bynthreads(=8) but Q registers are loaded in blocks ofnthreads * cpy_ne(=32). Threadtat stepsaccessed K element16s+2tbut Q element64*(s/4)+8t+2*(s%4)— matching only whent=0, s%4=0. Fixed by matching the f16 kernel: step bynthreads*cpy_ne, add innerk_KQ_1loop. The MMA kernel (prefill) was unaffected.Quality gate / auto-enable (
llama.cpp,ggml-cuda.cu):ggml_contextoverflow fix for large KV cache allocationsBenchmark results (Qwen3.5 35B A3B Q4_K_M, RTX 5090, tg128)
Memory: 3.47× compression vs F16 (3-bit vs 16-bit KV cache)
NIAH results (single-needle, Kamradt/RULER methodology)
Depth positions 0–100% at each context length. Score = correct needle retrievals / 11.
q8_0 baseline: 11/11 at 4K–64K, 10/11 at 128K, 10/11 at 256K. turbo3 matches or exceeds q8_0 at every length. The single miss at 128K (depth 20%) matches the q8_0 miss — not a turbo3 regression.
1M context tested with YaRN 4× rope scaling (
--rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 262144). q8_0 cannot fit at 1M alongside the model on a 32 GB card; turbo3 KV (~4.5 GB) fits with ~4 GB to spare.Key design choices
grp_norm / recon_norm(not rawgrp_norm) in the half-precision norm field so dequant is a single multiply__launch_bounds__(128)on the quantisation kernel prevents spilling with the large shared memory footprint🤖 Generated with Claude Code