Repository navigation
Experiment: interleaved q8_1 activations for the 2-4 token MMVQ (MTP verify) #39
Description
Idea from Strata by @Niko1221 (MIT), originally fork F4 by Eddoursul: STRATA_MMVQ_IL, the interleaved small-batch matvec (MMVQ_IL_TABLE.md).
- The decode and verify windows run their dense projections for 2 to 4 tokens through a kernel that reads an interleaved copy of the q8_1 activations: one load reads the same int of every column.
- A warp takes 1, 2 or 4 matrix rows, from a table per quant type, token count and row-count class.
- It is bitwise equal to the plain kernels. A table cell is enabled only where the worse of an RTX 3060 and an RTX 5070 gains at least 3%. A related multi-column variant (
STRATA_B6_MMVQ_ROWS) measured +0.2-2%.
Why it matters here
- The MTP verify is a 3-token batch (4 with
--spec-draft-n-max 3, which llama : MTP draft vocabulary (--spec-draft-vocab, llama-draft-vocab) #36 makes worthwhile) and takes about 70% of a decode step. Its matmuls are MMVQ withncols_dst3-4. - PR cuda : MMVQ launch parameters for consumer Ampere and Ada #20 already tuned the MMVQ rows per block for sm_86 / sm_89. The interleaved activation layout is the part that is not done.
Experiment
- Prototype in
ggml-cuda/mmvq.cu: aquantize_q8_1variant that also writes the interleaved layout for 2-4 columns, and an MMVQ path that reads it for Q4_K, Q5_K, Q6_K (the types of the production model). - Bit-exact check against the current kernel with test-backend-ops.
- Perf:
test-backend-ops perf -o MUL_MATfor the production shapes (ncols 2-4), then decode t/s with MTP depth 2 and 3,-sm tensoron 4070 + 3060.
Done when
The verify matmuls are measurably faster (Strata's bar: >= 3% per enabled cell), decode t/s improves within interleaved pairs, and the output is bit-identical.
Activity
Piggidragon commentedon Oct 19, 2026
First measurements (2026-10-19)
Where the time goes. nsys, production config (Qwen3.8 27B UD-Q4_K_S, 4070 + 3060
-sm tensor, MTP depth 2, draft vocab list):Kernel-level comparison.
test-backend-ops perfon the 3060 (3 MB L2; the 4070's 36 MB L2 holds these matrices, so its numbers are not usable), 3 columns, the real per-device shapes. The values are time per op against current main's MMVQ; negative is faster.Interleaved q8_1 activations within our MMVQ block structure. One vector load reads the same int of all columns, and the weight block is decoded once. This is Strata's
q8_1_illayout; for IQ4_XS it has to be position-major, because block-major was 70% slower from bad coalescing.One warp per R rows with 4 "virtual warps" (Strata's
native_mmvq_il_kernelstructure). The values are bitwise equal tomul_mat_vec_q. It helps where the K loop does not fill the block (k = 1536 / 2304: 6-9 Q4_K blocks for a block that takes 16 per iteration):With R = 1 it is 15-60% slower everywhere; the table uses the better of R = 2 and R = 4.
Conclusion.
llama/strata-exp(envGGML_CUDA_MMVQ_VW,GGML_CUDA_MMVQ_IL, experiment only).Piggidragon commentedon Oct 20, 2026
Closed: interleaved q8_1 activations gave nothing in our MMVQ (±2%). Warp-per-rows with virtual warps (bitwise equal) is faster only for k <= 2304 (-8..-20% per op), ~0.5% of decode. Not worth the complexity. The prototype (env GGML_CUDA_MMVQ_IL / GGML_CUDA_MMVQ_VW, plus TBO_PROD_SHAPES perf cases) is kept as a WIP commit on the local branch llama/strata-exp.
🤖 Generated with Claude Code