Repository navigation
Conversation
The mixed KV types commit changed the inverse WHT guard from checking k->type to v->type. For MLA, V is a view of K's first 512 elements (v->ne[0]=512, 512%128=0 → group=128). But K was quantized with 64-group WHT (k->ne[0]=576, 576%128≠0 → group=64). The mismatch produced garbage output on GLM-4.7 Flash. Fix: derive group_size from K when K is turbo (since K determines the rotation used during quantization), from V only when K is not turbo. Tested: 16/16 coherence (4 models × 4 KV combos), including GLM-4.7. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
InnerQ equalizes K channel variances before WHT rotation to reduce quantization error on models with anisotropic K distributions. Enabled via TURBO_INNERQ=N env var (N = calibration token count). Pipeline: calibrate per-channel K² stats → compute scales → apply scale to K before WHT (SET_ROWS) → apply 1/scale to Q and V output via ggml_turbo_wht op (scale passed as src[1]). Math: <Q/s, s*K> = <Q, K> preserves dot products. Key design: - d_innerq_active starts at 0 (CUDA zero-init) — kernel never multiplies by uninitialized scales - Auto-disables for 64-group models (group_size < 128) where channel mapping is not 1:1 across WHT groups - Auto-disables when max channel ratio < 1.2 (already balanced) - Cross-TU coordination via turbo-innerq.cu/cuh for scale_inv tensor updates between SET_ROWS and graph-side WHT - Scale_inv tensor stored in KV cache, initialized to identity Also: turbo2 (2-bit) now requires head_dim divisible by 128. At 64-group WHT, the 4-centroid codebook doesn't have enough resolution — DeepSeek (head_dim=192) produced garbage. turbo2 auto-falls back to q8_0 for non-128-aligned heads with a warning. Tested: 18/18 coherence (4 models × 4+ KV combos), InnerQ active on Qwen (222 t/s) and Mixtral (146 t/s), auto-disabled on GLM/DeepSeek. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
In the VEC flash-attention kernel, skip V dequantization for KV positions where ALL attention weights are below 1e-6. At long context most positions contribute noise, not signal — skipping their V dequant saves compute with zero quality impact. Both half2 and f32 paths are covered. The threshold (1e-6) matches the existing SOFTMAX_FTZ_THRESHOLD behavior where exp() values below this are flushed to zero. Benefit scales with context length — at short context nearly every position contributes so no skip occurs. At 512K+ most positions are skipped during generation. Tested: 10/11 NIAH at 512K (matches pre-sparse-V), 3/3 coherence. Generation at 512K: 25 t/s, prefill: 2535 t/s. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PPL benchmarks revealed that 64-group WHT (for non-128-aligned heads) causes catastrophic quality loss on some models: - DeepSeek (head_dim=192): PPL 344,304 vs 9.9 baseline - GLM-4.7 (head_dim=576): PPL 22.79 vs 14.97 baseline (+52%) The 64-group WHT passed NIAH/coherence but the weaker decorrelation (6 stages, 3 groups per 192-dim head) doesn't preserve statistical quality. Straight PolarQuant (no WHT) is even worse. All turbo types now require head_dim divisible by 128. Models with non-128-aligned heads (DeepSeek2, GLM-4.7 Flash) auto-fall back to q8_0 with a warning. The 64-group WHT code remains for future investigation but is not used. PPL summary (Qwen3.5, head_dim=128, wikitext-2): f16: 6.20 | q8_0: 6.18 | turbo3: 6.31 (+2.2%) | turbo2: 6.69 (+8.3%) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
4 centroids, no signs byte. Simpler than turbo3. PPL 6.507 (+6.48% vs q8_0) on Qwen3.5-35B-A3B. Metal shader: quantize, dequant (vec + non-vec), SET_ROWS kernel, FA template instantiations (non-vec nl=2, vec nl=8). Runtime: supports_op, vec/non-vec gating, llama-bench parser. Codex-reviewed: no bugs found, only known limitation is non-128 head_dim tail handling (same as turbo3, covered by q8_0 fallback). Closes #19 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai
- Add mixed-type kernel instantiations for all turbo pairs (turbo2/3/4)
- Both vec and non-vec FA paths, all supported head dims
- Pipeline naming uses k/v prefix format (kturbo3_vturbo4) to avoid
underscore ambiguity in type names
- Relax symmetric K/V assertion to allow turbo mixed pairs
- Update pipeline name construction in both FA pipeline getters
- 90 new kernel instantiations (54 non-vec + 36 vec)
- All 335 FA kernels use unified k{type}_v{type} naming scheme
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Add full asymmetric K/V quantization support for Metal flash attention:
- Pipeline naming uses k{type}_v{type} format for all FA kernels (335 total),
eliminating underscore ambiguity in type names
- 90 turbo × turbo asymmetric instantiations (turbo2/3/4 all combinations)
- 150 q8_0 × turbo asymmetric instantiations (both directions, all head dims)
- Gatekeeper and assertion updated to allow turbo × turbo and q8_0 × turbo pairs
- Zero regression on existing symmetric paths (validated across 4 models, 2 machines)
The q8_0 × turbo kernels fix a silent dispatch failure where mixed q8_0-K + turbo-V
configs would NaN (turbo4-V) or fall to undefined paths (turbo3-V). This enables
the asymmetric quality rescue: q8_0-K + turbo-V recovers near-baseline PPL on
low-bit models where symmetric turbo-K degrades.
Validated on Metal (M2 Pro + M5 Max):
- phi-4-Q8_0: symmetric turbo3 +4.2%, turbo4 +1.7% (no regression)
- Qwen2.5-7B Q4_K_M: q8_0-K + turbo4-V +1.0%, q8_0-K + turbo3-V +2.0% (rescued)
- Qwen3.5-35B MoE, 27B Dense, Mistral-24B: all healthy (no regression)
- Cross-hardware M2/M5 parity confirmed on all tested configs
Closes #27
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
…back)
Instead of falling back to q8_0 for models with non-128-aligned heads,
pad each head to the next multiple of 128 before WHT rotation. The
padded elements are zero and don't affect dot products since WHT
preserves inner products: <WHT(Q_pad), WHT(K_pad)> = <Q, K>.
This keeps turbo compression active on DeepSeek2, GLM-4.7 Flash, and
other non-128 head_dim models. Padding overhead is wasted bits on the
zero-padded quantized elements:
head_dim=192 → 256 (33% overhead, still 3.5x compression)
head_dim=576 → 640 (11% overhead, still 4.1x compression)
Changes:
- llama-kv-cache.cpp: allocate K/V cache at padded dimension, pad
k_cur/v_cur per-head via ggml_pad before set_rows, return padded
views from get_k/get_v
- llama-graph.cpp: pad Q per-head before turbo_wht, extract original
V head_dim from attention output after inverse WHT. All three
build_attn variants (KV, K-only/MLA, ISWA) updated.
PPL (wikitext-2, 512 context, 8 chunks):
DeepSeek (192→256): 11.61 vs 9.90 baseline (+17%)
Previously: 344,304 with 64-group WHT (catastrophic)
GLM-4.7 (576→640): 9.11 vs 14.97 baseline
Qwen3.5 (128): 6.31 unchanged
Tested: 16/16 coherence, NIAH 9-10/11 at 32K, CPU-only works.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GLM-4.7 Flash with turbo3 KV cache pads head_dim 576→640 for WHT.
D=640 had no CUDA FA kernel, falling back to CPU FA at 37 t/s.
Added MMA flash attention configs for D=640 (DKQ=640, DV=512),
using identical tile sizes as D=576 (nbatch_K2=288, nbatch_V2=256).
ncols1 capped at 2 for D=640: at ncols1≥4 the Q shared memory
(ncols × (DKQ/2+4) × 4 = 83KB at ncols=64) exceeds the per-block
limit. ncols1≤2 keeps total shared memory under 80KB.
Changes:
- fattn-mma-f16.cuh: D=640 config entries + extern declarations
(ncols2=16 only, ncols1∈{1,2,4})
- fattn.cu: case 640 in kernel selection + simplified MMA dispatch
(always ncols2=16, ncols1 capped at 2)
- fattn-tile.cuh/cu: D=640 tile configs + dispatch
- 3 MMA template instances + 1 tile instance
Tested:
GLM-4.7 Flash: 192 t/s (was 37), PPL 17.05 (vs 14.97 f16)
Qwen3.5: 222 t/s, PPL 6.31 (unchanged)
DeepSeek: 143 t/s, PPL 11.61 (unchanged)
16/16 coherence, NIAH 9-10/11 at 32K
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mixed turbo3-K/turbo2-V and turbo2-K/turbo3-V had no CUDA FA kernel instances, causing ~11x prefill regression (falling back to CPU FA). Added VEC template instances for both cross-type pairs at D=64/128/256. Updated the mixed-type guard in get_best_fattn_kernel to allow any combination of turbo2, turbo3, and q8_0. Tested: turbo3/turbo2 and turbo2/turbo3 both run at full CUDA VEC speed (~170 t/s prefill, ~221 t/s decode on Qwen3.5 35B). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…bug 1 Ports GGML_TYPE_TURBO4_0 to CUDA using the 4-bit PolarQuant format (16 centroids, nibble-packed, no QJL). Previously turbo4 crashed on CUDA with "cannot run the operation (SET_ROWS)". Changes TURBO4_USE_4BIT default from Metal-only to all backends. The 4-bit format (16 centroids) has better quality than the legacy 3-bit+QJL format and is simpler to implement (no residual projection). Full CUDA stack: - turbo-quant.cuh: 4-bit centroids, midpoints, nearest-centroid, dequant element, per-block quantize - set-rows.cu: k_set_rows_turbo4 kernel (128 threads, WHT rotation, 4-bit quantize, nibble pack via warp shuffle, corrected norm) - dequantize.cuh + convert.cu: turbo4 to f16/f32 - fattn-common.cuh: vec_dot_KQ_turbo4 + dequantize_V_turbo4 - fattn-vec.cuh + fattn.cu: VEC dispatch + all cross-type instances (turbo4×turbo4, turbo4×q8_0, turbo4×turbo3, turbo4×turbo2) - ggml-cpu.c: CPU FA vec_dot for turbo4 PPL (Qwen3.5, wikitext-2): 6.23 (+0.8% vs q8_0) at 3.8× compression Speed: 217 t/s decode (comparable to turbo3 222 t/s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… (issue #28) The block-size divisibility check in llama-context.cpp rejected turbo4 on GLM-4.7 Flash (head_dim=576, QK_TURBO4=128, 576%128≠0) before the KV cache zero-padding code could run. Fix: for turbo types, compute the padded head_dim (ceil to 128) before the divisibility check, matching what llama-kv-cache.cpp actually does. Tested: GLM-4.7 Flash turbo4 loads and runs at 193 t/s. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add V-only layer-adaptive modes to TURBO_LAYER_ADAPTIVE env var:
- Mode 5: first2+last2 V=turbo4, rest V=turbo2
- Mode 6: last8 V=turbo4, rest V=turbo2
- Mode 7 (recommended): first2+last2 V=q8_0, rest V=turbo2
Mode 7 ("Boundary V") protects quality-sensitive boundary layers with
q8_0-V while aggressively compressing middle layers with turbo2-V.
K cache is unchanged (stays at whatever -ctk specifies).
Validated on Metal (M5 Max) across 4 models, 2 context lengths:
- phi-4-Q8_0: 4.784 PPL (vs turbo2 4.835, turbo3 4.742)
- Qwen2.5-7B Q4_K_M: 6.835 (vs turbo2 6.911, turbo3 6.707)
- Qwen3.5-35B MoE: 5.148 (vs turbo2 5.257, turbo3 5.137)
- Qwen3.5-27B Dense: 6.423 (vs turbo2 6.534, turbo3 6.273)
- 8K context: stable, no collapse
- NIAH retrieval: pass
- Speed: no penalty
Effective V compression between turbo2 and turbo3, closer to turbo2
on deeper models. Quality consistently better than uniform turbo2-V.
Usage: TURBO_LAYER_ADAPTIVE=7 llama-server -m model.gguf -ctk q8_0 -ctv turbo2 -fa 1
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
…w-up) state_write_data and state_read_data used hparams.n_embd_k_gqa (576) for ggml_row_size, but turbo types zero-pad to 640. For turbo4 (QK=128), 576 % 128 != 0 → ggml_row_size assertion failure during prompt cache save on llama-server slot reuse. Fix: use k->ne[0] / v->ne[0] (actual padded tensor width) instead of hparams values in all four serialization paths (K write, K read, V write, V read). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…rnels Port TheTom's warp-cooperative turbo3 SET_ROWS kernel and turbo2/turbo3 flash attention templates to HIP/ROCm (7900 XTX, gfx1100). HIP vendor header fixes: - Add cudaMemcpyToSymbol/FromSymbol -> hipMemcpyToSymbol/FromSymbol - Add cudaMemcpyHostToDevice/DeviceToHost mappings - Fix __shfl_sync, __shfl_xor_sync, __shfl_up_sync, __shfl_down_sync to support both 3-arg and 4-arg calls (CUDA allows defaulting width to warpSize, HIP macros required 4 args) - Add __ballot_sync -> __ballot with uint32_t cast (HIP returns 64-bit on wave64 platforms, turbo code expects 32-bit) HIP CMakeLists: - Add turbo3 and turbo2 flash attention template instances (same files as CUDA CMakeLists, were missing from HIP build) Tested: Mistral-Small-24B turbo3 PPL = 5.28 (+2.4% vs F16 baseline 5.16) Previously showed catastrophic PPL ~15000 due to CPU quantize stub bug (fixed by TheTom in 53f1298). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
One norm per rotation group instead of four identical copies.
Eliminates 6 bytes of redundant storage per 128-element group.
turbo3: 3.50 -> 3.125 bits/value, 4.57x -> 5.12x compression
turbo2: 2.50 -> 2.125 bits/value, 6.4x -> 7.53x compression
Zero PPL regression validated across:
- Asymmetric q8_0-K + turbo{2,3}-V
- Symmetric turbo3/turbo3
- Boundary V (LA-V7)
- 3 architectures (dense, Qwen, MoE)
- 3 context lengths (512, 8K, 32K)
- 2 Apple Silicon platforms (M5 Max, M2 Pro)
- NIAH 3/3 pass
+3-7% decode on tested M2 Pro setup. No regression on M5.
Also adds derived NL_TURBO3/NL_TURBO2 macros replacing ~250
hardcoded FA template nl values. Block size is now a one-line
edit in ggml-common.h.
Credit to @AmesianX whose block_size=256 CUDA implementation
prompted this investigation.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The block_size=128 change (adac2c6) broke CUDA quantization: with QK=128, blocks_per_group=1, but the warp-cooperative packing still used blk_base+warp_id, causing warps 1-3 to write OOB. Fix: compute elem_in_block = j % QK_TURBO_N and use it for block pointer (j / QK_TURBO_N) and byte offsets (elem_in_block / 4 for qs, elem_in_block / 8 for signs). Works for both QK=32 and QK=128. Validated on RTX 3090 (sm_86), llama3.1:8b Q4_K_M, q8_0/turbo3: PPL = 7.587 (matches QK=32 baseline exactly).
Sparse V: now enabled by default on all Metal (was M5+ only). Validated across 30+ testers with zero PPL impact. Opt-out: TURBO_SPARSE_V=0. Boundary V: auto-enabled (mode 7) when -ctv turbo2 is set. Protects first 2 + last 2 layers with q8_0-V, rest turbo2-V. 37-91% quality recovery across 4 tested models. Opt-out: TURBO_LAYER_ADAPTIVE=0. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The HIP build was missing 9 turbo cross-type flash attention vec instantiations (turbo4 combos, turbo3/turbo2 cross-types) that were present in the CUDA CMakeLists but not mirrored to the HIP CMakeLists. Also guard the D>=576 tile kernel dispatch with #ifndef GGML_USE_HIP since those instance files are already excluded from the HIP build (they exceed HIP's 65536-byte local memory limit). Tested on: ROCm 6.4.4, gfx1151 (AMD Ryzen AI Max+ 395 / Strix Halo)
Leftover from 1-bit VX experiment. Causes -Werror build failure in CI. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add WHT-rotated weight quantization types: - TQ3_1S (type 44): 3-bit, 8 Lloyd-Max centroids, 4.0 BPW - TQ4_1S (type 45): 4-bit, 16 Lloyd-Max centroids, 5.0 BPW Both use 32-element Randomized Hadamard Transform with dual half-block scales (d0/d1). Quantization: forward RHT → scale search → iterative refinement (6 iter) → pack indices. Metal optimization (V2.1 fused kernel): - Zero threadgroup memory for rotation (was 20KB+ on large models) - Cooperative SIMD rotation via simd_shuffle_xor (registers only) - Single simd_sum at end (not per-block) - NR0=8 rows per threadgroup (amortizes rotation cost) - Memory barriers between rotate/matmul/unrotate dispatches - MoE MUL_MAT_ID support with rotated expert dispatch Config I (recommended): attn+ffn_gate/up=TQ4_1S, ffn_down=Q4_K, boundary 2+2 Validated on Qwen2.5-1.5B, Qwen3.5-27B, Qwen3.5-35B-A3B MoE: - 27-41% model size reduction - +1.3-1.9% PPL (Qwen), 94-102% decode speed - NIAH pass, KLD comparable to turbo3 KV - Llama 3.1 70B shows +25% PPL — needs investigation Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without this barrier, the GPU may start executing the next node's matmul while the unrotate kernel is still modifying src1. This causes data corruption when TQ and non-TQ tensors are mixed within the same layer's attention block. The barrier ensures the unrotate completes before any subsequent operation reads from the same activation buffer. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…uant) Upstream commit 744c0c7 added graph-level Hadamard rotation for KV cache quantization. This conflicts with our kernel-level WHT rotation and causes graph hash table overflow on Phi-4 and potentially other models. Disable by default since TurboQuant already handles rotation at the kernel level (more efficient, no extra graph nodes). Users can re-enable with LLAMA_ATTN_ROT_DISABLE=0 if needed. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…stration - ggml-cuda.cu: add TQ4_1S/TQ3_1S exclusion in ggml_cuda_should_fuse_ mul_mat_vec_q (was missing, causing ABORT in mmvq.cu) - tools/quantize/quantize.cpp: register TQ3_1S/TQ4_1S in allowed types Tested: Qwen2.5-7B TQ4_1S — correct output, PPL 8.82 (+1.1% vs Q2_K), 6510 t/s prefill, 20 t/s decode (cuBLAS dequant-to-f16 path). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two-phase approach: pre-rotate activation once via warp shuffle WHT, then simple mmvq kernel reads pre-rotated values (centroid × scale only, zero WHT per block). Results (Qwen2.5-7B TQ4_1S, RTX 5090): Decode: 20.3 → 69 t/s (3.4x speedup) vs q8_0 (177 t/s): 39% PPL: 8.82 (identical) Comprehensive optimization log (13 versions tested): cuBLAS baseline: 20 t/s V1 per-warp WHT, 4 warps: 60 t/s (3.0x) V3 shmem activation cache: 33 t/s (syncthreads kills it) V5 multi-warp per row: 62 t/s V6 LUT (shmem): 37 t/s (sync overhead) V7 8 warps clean: 62 t/s V8 pre-rotation (2-phase): 69 t/s ← BEST (3.4x) V9 pre-rot + q8_1: 70 t/s (marginal) V10 4-elem/thread: 57 t/s V13 8-elem, 4-thread dot: 45 t/s NR0=2/4/8: all regressed (register spill / cache thrash) The gap to q4_0 (275 t/s) is from dp4a integer intrinsics and packed int32 processing — TQ4_1S requires float centroid lookup which can't use dp4a. The gap to TheTom's Metal (85-99% of q8_0) is from Apple Silicon's cooperative SIMD efficiency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MoE models use MUL_MAT_ID which calls cudaStreamSynchronize in the expert routing dispatch. This is incompatible with CUDA graph capture. TQ4_1S types now disable CUDA graphs for MUL_MAT_ID nodes, matching the existing behavior for non-quantized types. Also: use persistent buffer for activation pre-rotation scratch, with graph-capture-safe check. Tested: Qwen3.5-35B TQ4_1S — 47 t/s decode, PPL 6.42. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ratch Replace two-phase approach (separate pre-rotation kernel + global memory round-trip) with single-phase kernel where all 8 warps cooperatively WHT-rotate activation into shared memory, then each warp processes one row reading from shmem (broadcast reads from L1). Key changes: - No global scratch buffer (eliminates CUDA graph incompatibility) - No separate kernel launch for pre-rotation - Activation stays in shmem (~14-32 KB) instead of global memory - Single __syncthreads between rotation and dot product (NOT in inner loop) - V8 two-phase fallback retained for ncols > 12288 (48 KB shmem limit) This avoids the NR0 regression that killed V3/V6/V11 — those had sync inside the dot product loop. V12's sync is between the two phases. Expected: 30-50% decode improvement on Ampere+ (shmem broadcast eliminates 2x activation bandwidth). Pascal improvement smaller (still bandwidth bound). NEEDS TESTING — apply and benchmark on Ampere/Ada/Blackwell. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add _USE_MATH_DEFINES + M_PI fallback for MSVC (doesn't define M_PI) - Add GGML_API to turbo3_cpu_wht_group_size for DLL export - Move extern declaration to file scope with extern "C" GGML_API linkage to fix C vs C++ name mangling across DLL boundary All changes are no-ops on Linux/Mac. Fixes MSVC build errors: C2065: 'M_PI': undeclared identifier LNK2001: unresolved external symbol turbo3_cpu_wht_group_size Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add missing CUDA->HIP stream capture API mappings (vendors/hip.h) - Add TURBO_IQ_API macro for cross-DLL symbol visibility (Windows + Linux) - Add fileno/isatty POSIX compat macros for clang on Windows No kernel changes needed. signalnine's fused mmvq-tq kernel uses __shfl_xor_sync which maps directly to HIP warp shuffle on RDNA 4. Tested: RX 9070 XT (gfx1201, RDNA 4), Qwen2.5-1.5B Config I. Result: 30% faster decode than Q8_0 (135 vs 104 t/s), +1.8% PPL. Metal regression: clean, no changes to non-Windows/non-HIP paths. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dflash: port #219 onto feature (rebased full CL)
…bedding
A UI dist directory was trusted on the presence of index.html alone. When an
npm PWA-assets build dies partway -- e.g. the native `sharp` step failing on
Termux / very new Node.js, which is common -- it can leave a dist with
index.html but without loading.html, the PWA manifest, service worker, etc.
ui-assets.cmake would then hand that partial dir to llama-ui-embed, which
aborts the entire build with 'missing required asset(s): loading.html ...'.
Add dist_is_complete(): a dist is only usable if it holds the complete asset
set llama-ui-embed requires (index.html, loading.html, manifest.webmanifest,
sw.js, build.json, version.json, bundle*.{js,css}, workbox*.js), matched by
basename recursively to mirror embed.cpp's own check. Gate all three trust
points on it:
- pre-built tools/ui/dist: skip a partial dir, fall through to npm/HF
- HF stamp-skip: only skip the fetch when the cached dist is complete
- final fallback: if provisioning failed and the dist is partial, drop it
and build a no-embedded-UI binary with a clear message instead of aborting
Net effect: a broken/partial UI build (sharp failure, interrupted download)
falls through to the HF-downloaded prebuilt UI, or degrades to a no-UI binary,
instead of hard-failing the whole llama.cpp build. Reported by two users hitting
the 'missing loading.html' abort on Termux.
dist_is_complete verified via cmake -P against partial/complete/empty/nested
fixtures (nested = real vite assets/ subdir layout).
ui: don't trust a partial dist dir (fixes 'missing loading.html' build abort on Termux/sharp failure)
Merges 582 upstream commits bringing deepseek4 arch support (lightning indexer, hyper-connections ops, DSpark/DFlash draft models) while preserving all TurboQuant code: TQ3_1S/TQ4_1S weight quant, turbo2/3/4 KV-cache quant, Metal/CUDA/Vulkan/SYCL kernels, and the turbo rotation/InnerQ KV-cache machinery. Notable conflict-resolution decisions: - ggml type/op enums: kept fork's existing TQ*/TURBO*_0 numeric values, appended upstream's new Q2_0 type and DSV4/lightning-indexer ops. - ggml-cuda/fattn-common.cuh and src/models/models.h had silently gone stale during auto-merge (zero conflict markers but missing upstream content); manually reconciled against upstream/master. - src/models/eagle3.cpp and dflash.cpp: took upstream's structure wholesale (fork's copies pre-dated an upstream internal refactor; no TQ-specific behavior existed in either file). - src/llama-kv-cache.cpp attn_rot_k/v: kept fork's opt-in-by-default policy (env-var gated) instead of upstream's default-on-for-any- quantized-type policy, since auto-rotating already-WHT-rotated turbo KV types would double-rotate; expanded the lightning-indexer force-enable arch list to include DEEPSEEK4/GLM_DSA.
The upstream for-loop -> iterate()+lambda refactor left three continue/break statements untranslated (invalid outside a real loop/switch), and a dead ctx_dft mtmd-draft branch that called server_tokens::process_chunk(), a method upstream removed when it migrated per-chunk multimodal handling into server_slot::process_mtmd_chunk() (which already re-feeds the draft context via common_speculative_process() in its decode callback). - two bare 'continue' -> 'return' (skip only the current slot, same as the original per-slot for-loop iteration skip) - one 'break' -> 'add_ok = false; return;' (the original break exited the whole per-slot for-loop, stopping batch room for every remaining slot this round; iterate() has no early-exit, so this reuses the existing add_ok kill-switch checked at the top of the lambda to replicate that) - removed the now-dead ctx_dft/process_chunk block
…es TQ weights through with-inverse-RHT mul_mm)
…r for unknown archs
…garbage output from strided MLA activations)
…tput) Root-caused via eval-callback node-diffing (CUDA vs CPU-oracle) on DeepSeek-V4-Flash real weights: blk.N.attn_q_a MUL_MAT (TQ3_1S, a 4096->1024 low-rank bottleneck) diverges ~2%/layer on the fused warp-scalar kernel (mul_mat_tq3_1s_multi / tq_prerotate_activation in mmvq-tq.cu), compounding over 61 layers into incoherent generation. A structurally identical TQ3_1S MUL_MAT (hc_attn_fn, different shape) was numerically fine, and test-backend-ops MUL_MAT cases at the exact failing shape (m=1024,n=8,k=4096) pass with random data — so this looks like a reduction-order/cancellation sensitivity specific to real weight/activation distributions rather than an indexing bug. The per-block WHT rotation + centroid dot product was verified bit-for-bit equivalent to the known-correct dequantize_tq3_1s inverse WHT used by the cuBLAS fallback, and moving the pre-rotated activation buffer from half to float (mmvq-tq.cu) — a legitimate precision improvement, kept — made no measurable difference, ruling out simple fp16 truncation as the cause. Confirmed via direct A/B on the CUDA repro: forcing all TQ mul_mats through the cuBLAS/dequant fallback restores coherent output. Fix: route GGML_TYPE_TQ3_1S mul_mat exclusively through the verified dequant+cuBLAS path in ggml_cuda_mul_mat, unconditionally (previously only non-contiguous src1/dst was gated to cuBLAS). TQ4_1S (dp4a and the AMD scalar variant) is untouched — no model on hand exercises it and there's no evidence it shares this bug. Gates: test-backend-ops MUL_MAT CUDA0 1358/1359 (only pre-existing, unrelated tq4_1s k_v=1600 CPU-reference-NaN gap remains; all tq3_1s cases incl. new real-shape regression tests at m=1024/24,n=1..8,k=4096 /16384 pass). llama-perplexity 8 chunks: 18.6043 +/- 1.27 (sane). llama-cli repro produces coherent English instead of token soup. Exact bit-level defect in the fused kernel's accumulation remains unresolved; this is a scoped mitigation, not a full root-cause fix.
DSv4 fixes: converter OOM, Metal escape hatch, server merge fallout, CUDA TQ3_1S correctness
github-actions
Bot
added
documentation
Owner
Author
|
Closing without merging. The fork's main branch already has deepseek4 support (plus DSpark and Metal hyper-connections from ggml-org#25784/ggml-org#26459), and critically it resolved the type-ID collision correctly: Q2_0 stays at upstream's 42 with TURBO2_0 moved to 43. This branch does the opposite (Q2_0 at 47), so merging it would regress main's type IDs and stomp its DSv4 work. The one thing here worth salvaging is the CUDA TQ3_1S correctness fix (fused warp kernel diverges ~2%/layer on real weight distributions; synthetic op-tests pass, found via node-level activation diffing). Will check whether main already has an equivalent and open a small standalone PR if not. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mechanical sync of
upstream/masterinto the fork (591 commits), plus the DSv4 fixes from #256 which are already merged into this branch.Why: the fork had TurboQuant weight types but no
deepseek4architecture; upstream haddeepseek4but no TQ types. Neither alone could run a Config-I quant of DeepSeek-V4-Flash.Merge strategy: please use a merge commit, not squash. Squashing 591 upstream commits would destroy the upstream lineage and make every future sync painful.
Conflict resolution notes
Hotspots resolved by keeping both sides (fork TQ features + upstream's new code): ggml type tables, CPU/CUDA/SYCL/Vulkan quant dispatch,
llama-quant.cpp, KV-cache integration, and the server. Two files auto-merged with zero conflict markers but silently dropped upstream content (fattn-common.cuh,models.h) and were repaired by hand.Upstream assigns
GGML_TYPE_Q2_0 = 42; this fork already uses 42 forTURBO2_0. This sync keeps the fork's IDs and putsQ2_0at 47 so existing TurboQuant KV-cache files stay valid.Consequence: GGUFs written by this branch containing
Q2_0are unreadable by upstream builds and by the pre-merge fork tip (type 47 reads out of range, tensor offsets shift, load fails). Renumbering fixes interop but invalidates existing turbo KV files. Flagging rather than deciding.Testing
test-quantize-fnsexits 0 with TQ3_1S/TQ4_1S/Q2_0 round-tripping.build-cannfails on this branch, but it already failed onfeature/turboquant-kv-cachebefore this work and none of the changed files touch CANN.