[vpkoskinen]

Title

poly: opt-in ggml Q8_0 KV cache for strata-poly (--kv q8_0, qwen35moe models)

Summary

Adds an opt-in Q8_0 KV cache (K and V) to strata-poly, selected with --kv q8_0, for qwen35moe
models only. At 131072 ctx it roughly halves the attention cache's VRAM (~47% less), freeing room for
more resident experts. --kv fp16 (or --kv omitted) is the default and stays byte-identical to the
previous engine.

Model support — this is not wired up for every architecture yet. The engine runs five families;
--kv q8_0 is implemented for one of them and refused cleanly for the rest:

general.architecture	--kv q8_0
qwen35moe (Tiel, Ornith, Occamy, …)	supported
gemma4, deepseek4, lfm2moe, glm5-next (glm5next)	refused cleanly at load — clear error, non-zero exit, no crash; the model still runs at fp16

Refusal message, e.g. for LFM: --kv q8_0 is only supported for qwen35moe models in v1; this is a 'lfm2moe' model. The check runs in Engine::open() right after reading general.architecture, before
any weights are loaded. Mirroring the cache into another engine (alloc, store, row accounting,
snapshots) is follow-up work, not part of this PR.

The cache uses the ggml Q8_0 layout (QK8_0 = 32: one fp16 scale + 32 int8 codes a block), stored split
as codes: int8[n_kv*head_dim] and scales: fp16[n_kv*(head_dim/32)] a row. This is deliberately not
the other engine's --kv int8 (a different layout) — that option is untouched.

What changed

11 files (9 modified, 2 new):

• src/poly/engine.hpp — enum class KvFmt { kFp16, kQ8_0 } and Options::kv_format (default fp16).
• src/poly/main.cpp — parse --kv fp16|q8_0 (unknown value → clear error, exit 2); #ifdef STRATA_USE_HIP rejects q8_0; usage line.
• src/poly/engine.cpp — Engine::open() restricts --kv q8_0 to qwen35moe and rejects every other
  architecture cleanly (the guard above).
• src/poly/qwen35.hpp — Block gains the q8 handles (k_codes/k_scales/v_codes/v_scales) and a
  bounded q8 dequant workspace.
• src/poly/qwen35.cpp — allocates the q8 buffers (q8-only) under kQ8_0; card_row_bytes() q8 size;
  a n_kv*head_dim % 32 load-time guard; attn_store() stores through store_q8(_at); attn() reads
  through Kvq8{...}; save_card_rows/load_card_rows handle codes+scales (parked conversations).
• include/strata/poly/kernels.hpp — struct Kvq8; store_q8/store_q8_at; q8 parameters on
  attend()/attend_batch().
• src/poly/kernels.cu — store_q8_kernel/store_q8_at_kernel (CUDA-graph-capturable, no host sync);
  the four attention kernels templated on bool Q8 so the fp16 branch is unchanged; a bounded
  dequant_q8_kernel so attend_batch() dequantizes only the current block of keys into a small
  workspace — never the whole prefix.
• CMakeLists.txt — registers poly_kv_q8_test.
• docs/GLM.md, new docs/Q8_KV.md — the --kv fmt option, the model-support scope and the measured
  numbers.

Extra Notes

Measured on an RTX 3070 8 GB (sm_86), Ryzen 9 5900X, CUDA 13.4; 131072 ctx, greedy (temp 0),
--no-draft, 64 tokens; Tiel, Ornith and Occamy (all qwen35moe).

• Scope: the numbers below are for qwen35moe only. No other architecture has been given a q8 path
  in this PR; they refuse --kv q8_0 cleanly (see the table above).
• Default unchanged: --kv fp16 matches the pre-change engine on the fixed prompts (12/12).
• Quality: --kv q8_0 vs --kv fp16 15/15 exact (character agreement 1.0000); an optional
  reference against llama.cpp -ctk q8_0 -ctv q8_0 agreed 2/2.
• Memory: per-token KV −46.88% — Tiel/Ornith 1.46 vs 2.75 GiB, Occamy 1.33 vs 2.50 GiB at 131072.
  With --kv-whole the expert cache grows by ~977–1082 experts.
• Speed: at parity — prompt tok/s −1.4…+0.2%, decode tok/s −0.2…−0.6%.
• Edge cases: context growth past kKvGrow, --kv-whole, a parked/restored conversation, the draft
  block and a vision request all pass under q8_0, and are unchanged under fp16.
• Not tested here: the HIP build — there is no ROCm toolchain on this machine. The option is rejected
  cleanly at startup on HIP (exit 2), and the q8 kernels use only intrinsics the existing poly kernels
  already use; a HIP report from a ROCm machine would close this out. SYCL is unaffected (src/poly is
  not built for it — SYCL is the separate sycl/ engine).
• Left alone: the strata engine and its --kv int8; glm5next; gemma4/deepseek4/lfm2moe (no q8
  yet, refused cleanly); the default path.
• The branch is merged with current main (LFM2.5, gemm_small, deepseek4, …); main's new dim==64
  attention fast paths are guarded so a q8 cache never enters them.
• Unrelated to this change but seen while measuring: an intermittent cudaGraphInstantiate: out of memory at 131072 with the default 256 MiB VRAM reserve; it reproduces on fp16 too, and
  --vram-reserve-mib 512 removes it. (The local run configs use --kv q8_0 --vram-reserve-mib 512;
  those configs are gitignored and are not part of this PR.)

[VecSzn]

Thanks, this is clean work. I ran it on the 5090 and the 4070 laptop:

• Default (no --kv): the same text as main on Qwen3.6 UD-Q4_K_M and LFM2.5 in every run on the 5090, and the same speed on both machines.
• --kv q8_0 on Qwen3.6: repeatable from run to run. The answers part from fp16's after a few hundred characters on two of three prompts and match on the third. Writing after an 8K prompt on the 5090 was 325 to 341 tok/s against 342 to 354 at fp16; on the laptop there was no difference I could measure.
• poly_kv_q8_test passes, the gfx1201 build compiles, and LFM2.5 is refused with your message.

Happy to merge it. One thing first: could you squash the branch into a single commit on top of main? Right now it carries two merge commits (kv_q8_base into main, origin/main into feature/kv_q8), and I keep main's history to one commit per change. After that it goes in as is.

[vpkoskinen]

Squashed down the commits, should merge cleanly.

[VecSzn]

Thanks, that was quick. It's merged on my side and goes out with the next release, which I'm holding until the MiniCPM-V PR is ready so both ship together. The PR will show as merged once I push.

[VecSzn]

Merged, and it's out in 0.1.44.1-poly.5. Thanks again, this one was easy to review.