What setup chose, and why it is wrong here

setup.py decides KV streaming from system RAM ("the owner's rule", --kv-streaming auto, around the code that prints "the KV cache stays in VRAM (fewer cached experts)"). With this install (iq3_s, --max-context 262144 --kv int8) it wrote --kv-resident 32768: at contexts above 64K the K/V streams from pinned RAM and only 32768 cells per QSA layer stay in VRAM. The reasoning is sound when the KV competes with an expert cache that cannot hold everything. On this PC it cannot:

• all 24576 of 24576 experts are VRAM resident from boot ("R4 hit path ON", zero-doorbell graph);
• after everything loads there are still ~29 GiB VRAM free;
• the whole 262144-token int8 K/V is small enough to live there entirely (observed VRAM use went 60.9 -> 69.7 GiB when I made it fully resident).

So streaming costs PCIe and stalls and buys nothing. The engine log confirms the cost during prompt reads: KV streaming: 47-65 % of block reads hit VRAM, 50-365 MiB read from RAM per request at 40-64K context.

A/B (same machine, same day, same session)

Engine 0.1.43.1 (release build, a0116b2), RTX PRO 6000 Blackwell Workstation Edition (96 GiB, sm_120), Core Ultra 9 285K, 253 GiB RAM, Windows 11 26200, driver 610.74, GPU power capped at 300 W for both arms. Suite: python tools/strata_bench.py --url http://127.0.0.1:8200 --long --runs 2. Only one setting differs: --kv-resident 32768 vs --kv-resident 0 (the latter means "all cells in VRAM"). Medians of two runs each.

Both results folders (results.json + engine log excerpts) can be attached: bench/results/2026-10-13-community-rtx-pro-6000-blackwell-worksta-qwen3-8-flash-next-iq3-s and -2.

Suggested rule (one sentence before code)

When --expert-cache auto fills every slot (all experts resident) and the free VRAM after loading is more than max_context x the int8 K/V per token, keep the KV fully in VRAM regardless of context size — i.e. treat "streaming from 64K" as a consequence of the expert cache's needs, not of the context length alone. The RAM test stays the guard it is today when some experts are not resident.

Not tested

Not tested: contexts above 262144 (where full residency may genuinely not fit), --kv k8v4, multi-GPU splits with this setting, vision-heavy sessions at long context, and whether one intermediate kv-resident value between 32768 and full captures most of the gain on smaller big cards (24-32 GB). Two runs per arm only; the ranges for the chat numbers are wide because batch windows interleave.