Repository navigation
--kv k8v4 produces degenerate repetition when --kv-resident streams (0.1.40) #1188
Description
Summary
On 0.1.40, --kv k8v4 combined with --kv-resident (the combination #711 / #705 newly enabled, and which the 0.1.40 notes say setup now offers) makes the model degenerate into repetition during free generation. The k8v4 format on its own is fine, and the failure is silent: no error or warning in the engine log, and long-context retrieval still works.
Environment
| Strata | 0.1.40 (source + official strata-windows-x64.zip, BUILD.json: archs [75,86,89,120], ptx true, cuda 13.0) |
| OS | Windows 11 Pro Insider Preview, build 29648 (10.0.29648) |
| GPU | NVIDIA GeForce RTX 5080, driver 617.14, sm_120, 16303 MiB |
| CPU / RAM | AMD Ryzen 9 9950X (16 cores) / 61.6 GB |
| Models | IQ3_XXS and stock IQ2_XS (both 2 shards) |
How to reproduce
Config args: --kv k8v4 --kv-resident 16384 --max-context 262144 (the engine clamps the requested 16384 up to its documented 20480 minimum — that is expected, and the log confirms it).
Request, greedy so it is deterministic:
POST http://127.0.0.1:8096/v1/chat/completions
{"model":"strata",
"messages":[{"role":"user","content":"Write the numbers from 1 to 60 in words, separated by commas, with no other text."}],
"max_tokens":400,"temperature":0,"reasoning_effort":"none"}
Actual vs expected
Expected — what int8, q4_0, and k8v4 without streaming all produce, byte for byte identical, ending naturally at 156 tokens:
one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen,
fourteen, fifteen, sixteen, seventeen, eighteen, nineteen, twenty, twenty-one, ...
Actual with k8v4 + --kv-resident at --max-context 262144 — 400 tokens, hit the cap, collapsed into a single token:
one, two, two, three, three, four, four, five, five, six, six, seven, seven,
seven, seven, seven, seven, seven, seven, seven, seven, seven, ...
At --max-context 32768 (streaming still active) it degenerates the same way, just with a different collapse point:
one, two, two, three, three, four, four, five, five, six, six, seven, seven,
eight, eight, nine, eight, nine, nine, ten, ten, eleven, eleven, twelve, twelve,
thirteen, twelve, thirteen, thirteen, ...
The 262144 case is reproducible byte for byte across repeated runs, so this is deterministic, not sampling noise.
Isolation matrix
| # | Model | --kv |
--kv-resident |
--max-context |
Needle recall | Short factual (greedy) | 400-token generation |
|---|---|---|---|---|---|---|---|
| 1 | IQ3_XXS | int8 |
16384 | 262144 | pass | Red, Blue, Yellow (6 tok) |
clean (156 tok) |
| 2 | IQ3_XXS | k8v4 |
16384 | 262144 | pass | verbose (32 tok) | degenerate (400 tok) |
| 3 | IQ3_XXS | k8v4 |
0 (off) | 32768 | pass | Red, Blue, Yellow (6 tok) |
clean (156 tok) |
| 4 | IQ3_XXS | q4_0 |
16384 | 262144 | pass | Red, Blue, Yellow (6 tok) |
clean (156 tok) |
| 5 | IQ2_XS (stock) | int8 |
32768 | 262144 | pass | Red, Blue, Yellow (6 tok) |
clean (156 tok) |
| 6 | IQ2_XS (stock) | k8v4 |
32768 | 262144 | pass | verbose (16 tok) | degenerate (400 tok) |
| 7 | IQ3_XXS | k8v4 |
16384 | 32768 | pass | verbose (16 tok) | degenerate (400 tok) |
Rows 2 and 6 produce the same degenerate string, on two different models (one of them a stock, unmodified quant), so this is not weight-specific.
What the matrix rules out:
- Not a general "4-bit KV costs quality" effect.
q4_0(row 4) quantizes both K and V to 4-bit and its output is byte-identical toint8;k8v4keeps K at INT8 and only rotates V to Q4_0, and it is the one that breaks. - Not the k8v4 format itself. With
--kv-resident 0(row 3) k8v4 is byte-identical to int8. - Not context length. Row 7 breaks at
--max-context 32768with streaming on. - The streaming path is the variable. In every broken row
--kv-residentwas active; in every clean k8v4 row it was off. The log confirms streaming was really engaged, e.g. for row 7:
strata generate: KV streaming: 20480 of 32768 cells per QSA layer in VRAM, the K/V in 0.30 GiB of pinned RAM
and for row 2:
strata generate: KV streaming: 20480 of 262144 cells per QSA layer in VRAM, the K/V in 2.39 GiB of pinned RAM
The memory saving itself is real
For what it is worth, k8v4 does deliver the expected memory reduction, which is why I tried it:
--kv (resident 16384, ctx 262144) |
K/V in pinned RAM | expert slots | VRAM free, everything loaded |
|---|---|---|---|
int8 |
3.09 GiB | 3547 | 524 MiB |
k8v4 |
2.39 GiB (−0.70 GiB, −23%) | 3572 | 558 MiB |
So the format works as advertised for capacity; only the streamed decode path is wrong.
Why it is easy to miss
- No error, no warning, no NaN in the engine log. Output just quietly degrades.
- Retrieval is unaffected. My needle test (a 19,357-token prompt with two unique codes at roughly 10% and 90% depth) passed in all seven configurations, including the broken ones. The damage only shows up in free generation, and mainly once a reply runs past a few dozen tokens — a short chat answer can look merely "more verbose" (see the "Short factual" column) rather than obviously wrong.
Workaround
Keep --kv int8 (or q4_0) when --kv-resident is in use. k8v4 appears to be usable only with streaming off, which at large contexts is not what it is wanted for.
Since the 0.1.40 notes say setup now offers k8v4 together with --kv-resident, users who take that option will get a silently degraded model.
Notes
- I could not run
tools/kv_precision_compare.pyfor a teacher-forced KL number: the 0.1.40 source tree does not contain that file (the bench README underbench/results/2026-09-27-kv-q4/references it). A teacher-forced comparison of k8v4 +--kv-residentagainst int8 would show the divergence properly, if it helps. - These probes are a degeneration check, not a precision benchmark. They separate "grossly broken" from "not grossly broken"; they are not sensitive enough to resolve the subtle perplexity differences the repo's own bench reports for
q4_0. "q4_0 output identical to int8 here" does not mean q4_0 is lossless. - Happy to run any specific test or build if useful — the machine is available.
Activity
taxah92 commentedon Oct 6, 2026
A third data point, from the other side of the architecture gap: on Volta the same probes are clean with streaming ON.
Rig: Tesla V100 SXM2 16 GB (sm_70), CUDA 12.4 distro toolkit, Ubuntu 26.04, EPYC 7532, one GPU.
v0.1.40.1tree plus two local patches: a build guard for CUDA <12.5 invmm.cpp, and the k8v4 batch host-mirror fix from #1169 — neither touches the dequant path, and #1169 is a different failure (stale parked state, not degeneration).Config: IQ3_S,
--max-context 1048576 --rope-scaling yarn --rope-scale 4,--kv k8v4 --kv-resident 32768(streaming engaged:KV streaming: 32768 of 1048576 cells per QSA layer in VRAM, the K/V in 9.56 GiB of pinned RAM),--mtp --spec 3 --suffix-draft 3, no--batch.Your probes verbatim, greedy:
finish=stop, 156 completion tokens — the same count as your clean int8/q4_0 rows; all 60 words, zero consecutive repeats.finish=stop, 6 tokens (red, yellow, blue) — your broken rows report 16–32 verbose.So k8v4 +
--kv-residentshows no degeneration on this rig. Both broken reports so far (#1135: RTX 5090, #1188: RTX 5080) are sm_120; this one is sm_70 — the streamed k8v4 dequant/attention kernel path differs there. If an sm_86/sm_89 streamed rig also comes back clean, the suspect narrows to the Blackwell kernels specifically (or their interaction with the resident ring).merbanan commentedon Oct 6, 2026
Another data point: reproduced on Turing (RTX 2060 SUPER 8 GB, sm_75), with both implementations of the streamed K8V4 path.
Rig: RTX 2060 SUPER 8 GB (sm_75, PCIe 3.0 x8), Ryzen 9 3900X, 60 GB RAM, Linux, CUDA 12.4 toolkit (local build, sm_75). Q2_0,
--max-context 131072 --kv-resident 20480(KV streaming: 20480 of 131072 cells per QSA layer in VRAM),--spec 2with the MTP draft head, one GPU, no--batch.Your probe verbatim, greedy (
temperature 0,reasoning_effort none,max_tokens 400):--kv-resident 20480, 0.1.39 + #705's engine commit (f6e246d)--kv-resident 20480, newmain(#711)--max-context 32768)--kv-resident 20480The broken rows start exactly like yours:
one, two, two, three, three, four, four, five, five, six, six, seven, seven, eight, eight, nine, nine, ten, .... The short "three primary colors" probe was correct (6 tokens,Red, Green, Blue) in all four rows here, so short answers hide it.So far: sm_75 broken, sm_120 broken (yours), sm_70 clean (taxah92's V100). Both the #705 engine code and #711 show it, so it looks like something they share in the streamed hybrid path (the K8V4 block movers / host write-through) rather than one patch. Workaround here:
--kv int8with--kv-resident, which is byte-clean.Happy to run a build or a teacher-forced comparison on this card if it helps.
verify: the batched K/V append writes a streamed K8V4 state's host co…
Niko1221 commentedon Oct 7, 2026
Fixed in 0.1.40.2. The batched K/V append of a verify window skipped the host copy for the K8V4 format, so with --kv-resident the cells a window wrote were missing when a block came back from the host copy, and generation repeated itself. Reproduced here on an RTX 3060 (sm_86) and an R9700 (gfx1201): a count from 1 to 300 printed every number twice with --kv k8v4 --kv-resident, and is clean with the fix (int8 was always clean). The fix is Benjamin Larsson's PR #1264, his authorship kept. Please retest on your card.
verify: the batched K/V append writes a streamed K8V4 state's host co…
verify: the batched K/V append writes a streamed K8V4 state's host co…