[Benchmark] RTX 4090 Laptop 16 GB (sm_89), Strata 0.2.7 / engine 0.1.52: STRATA_ROUTE_PRIOR on a 16 GB card - speed and KL, up to 255K tokens #4345

Reproduction status: Measured locally

What was measured

Whether the 0.2.7 residency-biased routing (STRATA_ROUTE_PRIOR, on by itself only for cards of 13 GiB or less) pays on a 16 GB card, and what it costs in quality. Also 0.1.50 vs 0.1.52 at the same settings (the release notes ask for RTX 40 data).

Hardware: RTX 4090 Laptop GPU 16 GB (sm_89, PCIe 4.0 x16, driver 616.56, WDDM; desktop on the iGPU, 0 MiB used before load), i9-13900HX (8P+16E, no AVX-512), 64 GB DDR5 (48 GB + 16 GB, running at 4800 MT/s), NVMe, Windows 11 build 26300, Legion Performance mode, on AC.
Software: source checkout at v0.2.6 + the prebuilt Windows CUDA 13 engines 0.1.50 and 0.1.52 (v0.2.7 release, checksums verified). ISTA-DASLab Qwen3.8-Flash-Next IQ2_XS and IQ3_XXS; SC117 abliterated IQ3_S (RAM budget 28 GiB). Setup's configs: --max-context 262144 (IQ3_S 131072), --kv int8 --kv-resident 32768, --spec 4 --spec-min-p 0.5, MTP with the en draft vocabulary, --vram-reserve-mib 1280, 15 pool workers, text only (no --vision) unless noted. Windows large pages were refused (error 1314) in every run.

Speed method: one server per arm, started fresh, port 8080. Fresh word-list prompts made from docs/*.md (unique prefix, no cache reuse), temperature 0, thinking off, 512 new tokens; numbers are the engine's own generated in ... (tok/s) line. Each arm = one run of 2K, 2K, 32K, 128K (IQ3_S: 120K). Not interleaved pairs. Identical arms repeated later differed by up to ~18% at 2K and ~7% at 128K, so read single differences below ~10% as noise.

Quality method: tools/quant_quality.py run (teacher-forced, data/quant_quality/v1, 4,995 positions) with --env STRATA_ROUTE_PRIOR=X and --gpu-experts set to the cache size the real server gets (IQ2_XS 6100, IQ3_XXS 4700). Reference: IQ3_XXS with the prior off and --gpu-experts 1500 (no Q8_0 on this PC). So the KL below includes the GPU/CPU rounding difference, and the reference is a quant, not Q8_0.

Decode tok/s (2K / 2K / 32K / 128K)

Prompt reading was the same with and without the prior (~2,400-2,750 tok/s at 128K; IQ3_S with the RAM budget ~1,050-1,340, all in RAM 2,417).

At ~255K tokens (one 255,150-token prompt, 512 new tokens, then a 128-token follow-up on the same chat):

Tried and no consistent gain here: 23 or 7 (+--pool-affinity p-cores) pool workers, --kv-resident 16384. Raising the engine to high priority / no EcoQoS failed (OpenProcess access denied) so it was not tested.

Quality (teacher-forced, vs IQ3_XXS prior off)

IQ2_XS prior 0.5 against IQ2_XS prior off directly: top-1 91.15%, KL 0.074.

Reading

• On this 16 GB card the prior lifts the decode hit rate from ~88% to 95-99%. Decode at 2K-128K against the prior-off runs: IQ2_XS +28-33% at 0.5 and +40-52% at 1.5; IQ3_XXS +29-51% at 1.0. At 255K: IQ2_XS +23% (0.5), IQ3_XXS +44% (1.0).
• 1.5 costs clearly more here than the 12 GB check suggests: IQ2_XS PPL 4.53 -> 4.96, IQ3_XXS at 1.5 lands at plain IQ2_XS quality. 1.0 on IQ3_XXS stays better than plain IQ2_XS (KL 0.164 vs 0.248) while being faster than plain IQ2_XS at every length.
• So a lower default for 13-16 GiB cards (for example 0.5-1.0) might be worth a look on your side; one PC and one run per arm is not enough to set it.
• 0.1.50 vs 0.1.52 with the prior off (sm_89 MMVQ-IL tables on by default in 0.1.52): no difference above the noise here (two runs each).

Limits

One PC, single runs per arm (not interleaved pairs), temperature 0 summaries of docs text; no agent task suite or needle test for the prior; KL reference is IQ3_XXS, not Q8_0; RAM is a mixed 48+16 GB pair at 4800 MT/s.

Raw data (per-request JSON from the benchmark, the quant_quality compare output and per-run engine arguments, the run plan, sanitized configs and nvidia-smi/RAM diagnostics) is kept and can be shared on request.