[TorBere]

Implements the int8 K/V cache for the q36 (qwen35moe) engine's own attention path as --kv (strata's name
for the stores: fp32 default, int8 for both stores, k8v32 / k32v8 for one of the two). The q36 caches
were FP32 (2 KiB per cell per store), so a 128K context ran out of VRAM for expert slots: 0/256 experts in
VRAM, 34 tok/s. This is the same effect issue #2464 reports for the V100 (the q36 decode falling from 154 to
56 tok/s as the context fills; llama.cpp quantizes the cache, this engine does not).

What it does

• --kv int8: K and V stored as int8 codes with one FP16 scale per 64 values (group absmax / 127), ~3.9x
  smaller cache (528 B/cell vs 2048 B at head_dim 256); k8v8 is an alias; k8v32 / k32v8 quantize one
  store only; fp32 (the default) runs bit for bit the old kernels.
• STRATA_Q36_KV=int8 stays as the environment fallback (--kv wins).
• Decode (k_attn_part), prompt chunk (k_kv_f16 -> cuBLAS) and the MTP block dequantize on the fly; the
  verify windows go through the same kernels, the commit/rollback of the cache is unchanged.
• k_attn_prep / k_attn_part / k_kv_f16 are template <bool KI8, bool VI8>; a small kv4() dispatcher
  in Engine launches the right specialization, so kc/vc stay FP32 pointers and the int8 codes + scales
  live in kc8/vc8/ks8/vs8. kKVG = 64; head_dim must be a multiple of it (the host says so and stays
  FP32 otherwise).

Measurements

RTX 3060 12 GB, Qwen3.6-35B-A3B UD-Q4_K_XL (experts 18.75 GiB, streaming), --mtp 2, same machine, two
runs each, 120 tokens per run:

variant	64K	128K	experts in VRAM (64K / 128K)
installed engine (CUDA 13), FP32	50.3 tok/s	35.8 tok/s	18 / 0 of 256
this source, FP32 (default)	50.1 tok/s	33.9 tok/s	18 / 0 of 256
this source, --kv int8	59.7 tok/s	54.8 tok/s	46 / 31 of 256

At 128K the freed VRAM (~4 GiB) brings 31 experts back into the VRAM, which is where the +53% comes from.
The FP32 build reproduces the installed engine (50.1 vs 50.3 / 33.9 vs 35.8), so the difference is the patch,
not the toolchain.

Needle-in-the-haystack, 52.8K-token prompt, 128K mode, max_tokens 250, temperature 0: both FP32 and int8
find the hidden number; the int8 run took 91 s vs 159 s end to end (1.75x). A chat answer at the same
context is qualitatively equal between the two.

--kv k8v32 / k32v8 smoke tested at 64K: each quantizes one store, so they land between FP32 and int8 —
32 experts in VRAM (FP32: 18, int8: 46) and 54.8 / 53.8 tok/s (FP32: 50.1, int8: 59.7).

Not in this PR

--kv q4_0 and k8v4 (PR #21's Hadamard rotation over head_dim 256) are not wired into the q36 kernels
yet; --kv refuses them with a note. The FWHT-256/Q4_0 pieces already exist in the repo
(src/kernels/cuda/kv_q4.cu, include/strata/kernels/kv_q4.hpp), so a follow-up can add them the same way
(rotate K after RoPE and V, rotate the queries, rotate the output back).

How it was tested

• Built from this tree with the setup toolchain (CUDA 12.9, sm_86) into a separate build folder; the
  installed engine was not touched.
• A/B runs against the installed engine and the same source without the patch (fp32 path), same prompts.
• --kv int8 end to end through serve/server.py (the config's args carry --kv int8).
• The q36ref fp32 path is untouched (the FP32 template instantiation is the previous kernel code).

[TorBere]

Update: engine-reported measurements for the int8 KV cache

The decode numbers in the description and in the second commit's message were measured with client-side timing (total request time, and differences between two runs). As built here, the standalone q36 engine has no prompt cache - every request re-reads the whole prompt (reused: 0 on every request) - so those timings include the (re-)prefill, and the two-run difference overestimates the decode rate. The engine reports its phases itself, and that is authoritative and easy to reproduce: GET http://127.0.0.1:8080/metrics returns per request prompt_ms (prefill), decode_ms / decode_tok_s (decode) and reused right after the request finishes.

What the table in the description actually measures: it used a short prompt with the KV cache allocated for 64K / 128K, so it shows the VRAM win (experts back in VRAM: 18 -> 46 at 64K, 0 -> 31 at 128K), not the decode speed at a filled context. A tiny-context run today reproduces the int8 number (~59.5 tok/s at ~50 tokens of context with --mtp 2).

Engine-verified numbers - same machine (RTX 3060 12 GB, Qwen3.6-35B-A3B UD-Q4_K_XL, 128K mode, --kv int8, --mtp 3, temperature 0), prompts actually filled to the context, counters read after warm-up:

context	decode	prefill	drafts accepted
2K	42.4 tok/s	1103 tok/s	70 %
8K	40.9 tok/s	1172 tok/s	77 %
26K	31.9 tok/s	990 tok/s	71 %
64K	20.5 tok/s	675 tok/s	72 %
128K	13.3 tok/s	390 tok/s	71 %

• The second commit's decode figures (55.7 / 48.5 / 35.8) were inflated by the client method. The decay itself is real: roughly -25 % from 2K to 26K, -52 % to 64K, -69 % to 128K.
• The earlier prefill numbers were measured from prompt-read time directly and stand (about 2x the FP32 path at short/mid contexts); the engine figures above are the current ones.
• Reference without MTP (engine-verified): 31.9 / 30.5 / 25.4 / 19.9 tok/s (2K / 8K / 26K / 64K) - MTP adds ~+33 % at 2K but only ~+3 % at 64K, which matters for the decode-vs-context question in [Engine] Qwen3.6-35B-A3B decode falls with context (V100: 109 to 14.6 tok/s at 114K on 0.2.0-beta); llama.cpp holds ~90 #2464.
• The patch itself is unaffected: the FP32 path stays bit-identical, and the VRAM/expert effect is real. An engine-verified before/after of the attention optimization (first vs second commit) will follow.

[christopherrobertbrooks-tech]

V100-PCIe 32 GB (sm_70), the #2464 setup: --kv int8 doubles to triples decode at filled contexts; the fp32 default is unchanged.

Qwen3.6-35B-A3B UD-Q4_K_XL (setup's pinned file), --native --max-context 131072 --prefill auto --mtp 2, every routed expert in VRAM. Built this PR's head (d85b465) and its base on main (296b59d) with CUDA 12.9 for sm_70. Fresh prompts filled to each size, greedy, 256-token answers; decode and prefill from the engine's /metrics (decode_ms, prompt_ms). 3 runs per size per round, rounds in mirrored order (main and fp32 2 rounds, int8 3); medians, rounds agree within 0.5%.

Decode, tok/s:

prompt tokens	main	PR, fp32 (default)	PR, --kv int8	int8 vs fp32
256	153.9	154.1	144.6	-6%
1,000	154.4	154.5	151.7	-2%
4,096	114.7	114.8	135.6	+18%
16,384	56.5	56.5	110.8	+96%
32,768	33.4	33.4	78.2	+134%
65,536	17.3	17.4	52.5	+202%

• Prefill is the same in all three (1,141 / 1,673 / 1,729 / 1,554 / 1,322 / 965 tok/s).
• fp32 is the old engine: same speed, and all 18 greedy texts byte-identical to main.
• Needle (a code hidden at 10 / 50 / 90% depth of ~32.7K- and ~65.5K-token prompts, greedy): 6/6 found with int8, as with fp32 and main.
• int8 texts part from fp32 after 20 to 1,010 characters (of ~1,300), as expected from a quantized cache; they read normally (at 64K both name the last function in the prompt correctly).
• At 16K, int8 (110.8) is now above llama.cpp on the same file (87.3, measured for [Engine] Qwen3.6-35B-A3B decode falls with context (V100: 109 to 14.6 tok/s at 114K on 0.2.0-beta); llama.cpp holds ~90 #2464).

The short-context dip comes from the packed Q8_0 decode copies, not from int8. At --max-context 131072 the fp32 cache leaves too little VRAM, so the engine gives the packed copies back (1.78 GiB); with int8 there is room and it keeps them (V100 in use 27,815 vs 30,021 MiB). On this card the packed copies decode slower than the GGUF layout: the fp32 cache at --max-context 98304 keeps them (texts byte-identical to fp32 again) and gives 134.7 / 135.8 / 104.3 / 53.6 / 32.1 / 16.8 tok/s, so with the copies on both sides int8 is ahead at every size (+7% at 256, +12% at 1,000). The comment in place_experts says decode gets "a little slower" without the copies; on sm_70 it is the other way round (-13% at 256 with them kept). That may deserve its own look; I can test a variant.

[TorBere]

A second data point from a different architecture. RTX 3060 12 GB (sm_86), Qwen3.6-35B-A3B UD-Q4_K_XL, q36 standalone engine with the int8 KV path from #2698, --mtp 3, 128K window, 31/256 experts in VRAM (46/256 with a 64K window). Engine-verified numbers (/metrics prompt_ms / decode_ms), greedy, filled context, MTP drafts accepted ~70-77%:

prompt tokens	2K	8K	26K	64K	128K
decode tok/s	42.4	40.9	31.9	20.5	13.3
prefill tok/s	1103	1172	990	~675	390

Same PC, llama.cpp (LM Studio 2.55 runtime, no drafts): 45.9 / 37.7 / 39.4 / 28.2 @ 2K/8K/26K/64K. And the main engine with a Flash-Next-class model on this box is flat 37-46 all the way to 128K.

So the falloff is not Volta — it shows the same shape on sm_86 (-52% from 2K to 64K here, against -64% from 256 to 16K on the V100). Two notes on the mechanism, from our side:

• The q36 path decodes with dense attention without an indexer — the 3.6-35B weights do not carry one, so the QSA top-k selection of the Flash-Next path is not available for this model. Decode is O(context) per token. llama.cpp runs the same dense attention for this file, but its flash-attention path holds up much better; that is the engineering gap, not a mechanism difference.
• The q36 decode kernel is instruction/address-bound rather than bandwidth-saturated. Evidence: a bit-identical A/B of just (a) uint4 K-code loads and (b) a pointer-walked V loop — less address arithmetic per cell — moved decode by +1.7 / +2.8 / +11.5 / +17.0 % @ 2K/8K/26K/64K. So there is headroom left that needs no different attention scheme. The split-K partial merge (one expf per thread and chunk) also grows with context; we have not profiled it yet.

Two practical traps we hit while measuring, in case they matter for other reports:

• Sampled requests skip MTP entirely. With temperature > 0 (or sampling penalties) the engine decodes one token at a time — spec is greedy-only in the q36 serve loop. On the same state here: greedy 38.7 tok/s (drafts 43/48 accepted) vs temperature 0.6 26.3 tok/s (drafts 0/0). So rates measured with sampling on are not comparable to the short-context --mtp numbers like the V100's 154.
• Conversation continuation never fires. The last 2-4 tokens of a prompt tokenize differently once more text follows them (a BPE boundary effect), and the recurrent GDN state cannot step back — so every turn re-read the whole prompt (~95 s @64k here). Snapshotting the recurrent state 16 tokens before the prompt end (and re-reading that tail) fixed it: a 64K follow-up went from 95.2 s to 1.35 s, with greedy outputs identical to a full re-read.

Our int8 KV branch plus both patches (the attention A/B above, and the conversation checkpoint) are available if they are useful — they sit on top of #2698.

[christopherrobertbrooks-tech]

Thanks for the second data point. I can run both patches on the V100 with the same setup as above (decode at 256 to 64K, the needle check), plus a 64K follow-up turn for the conversation checkpoint. If you push the branch, I'll post the numbers here.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[ggeorgovassilis]

@TorBere I retracted my PR #2742 respecting precedence of yours. Do you intend to resubmit your PR?