1M context: identical expert work per window, but the VRAM-touching stages grow 7-8x

Windows, RTX 4080 SUPER (32 GB, driver 616.56, PCIe Gen 4 x16), i7-13790F (8P+16E, AVX-2), 96 GB RAM. Strata 0.1.38, Flash-Next IQ3_S, --kv int8 --kv-resident 32768 --pool-workers 4 --spec 4 --pcie-frac 0.35. Everything is identical between the three runs below except --max-context and the rope settings it requires (262,144 / 524,288 + yarn 2 / 1,048,576 + yarn 4). The expert cache is 11,631 slots / 22.07 GiB in all three (bundled profile, no eviction).

Same prompt, 18 decode windows, 64 tokens, with STRATA_DECODE_TIMING=1, STRATA_VERIFY_PROFILE=1, STRATA_PREFILL_TIMING=1 (ms per window):

stage	262,144	524,288	1,048,576
ms/window	38.47	35.30	69.15
GPU-reach wait	12.94	12.33	41.52
GDN VRAM hits	3.61	3.58	25.35
QSA VRAM hits	1.23	1.23	10.27
GDN PCIe grp	0.28	0.28	1.81
q8 gemv / attention / out-proj	1.26 / 0.74 / 0.93	1.27 / 0.73 / 0.89	1.37 / 0.78 / 1.00
work: experts/layer, VRAM hits/window	2.51, 33.24	2.58, 34.22	2.54, 33.74

On the prompt side, the same prompt read cold: GPU timeline 2,392 -> 2,273 -> 3,880 ms, where qsa proj is 159 -> 145 -> 306 ms while hc read / gdn (314/312 -> 298/290) barely move.

End to end, fresh 43,969-token read + 200 generated: 262,144 = 2,581 ms read (2,652 tok/s) / 103.9 tok/s decode; 524,288 = 2,655 ms / 94.4; 1,048,576 = 4,065 ms (1,684 tok/s) / 46.4.

Reading: the work counters do not change and the expert cache is identical, yet the stages that touch VRAM cost 7-8x more and the host's wait for the GPU grows 3.4x, while pure-compute stages move about 10%. That looks like access cost, not extra work. 524,288 shows none of it (its stage table is within noise of 262,144).

Hypothesis, not measured: the per-layer session tables scale with --max-context (KV residency page table, QSA indexer tables, GDN state), so the same expert addresses may become more expensive to reach (TLB pressure, or a 4x larger session state in VRAM breaking locality). I did not measure TLB or page-fault counters and did not measure the VRAM carve under a fixed --vram-reserve-mib.

Question: is the 1M tier expected to cost this much by design, or is there a knob that recovers it (a different --kv-resident window, --pcie-frac, or separate residency sizing for the KV page table at very large --max-context)?

YaRN is not the cause: at 1,048,576, yarn factor 4 was faster than no scaling at both sample sizes (46.4 vs 39.3 tok/s; 44.2 vs 41.4 on a 64-token sample), with the same expert cache and the same 32,768 resident KV cells. Happy to re-run any of this with extra instrumentation on the same machine.

Hi, well, a data point and a cheaper hypothesis to rule out first, from a 16 GB RTX 5080 on Windows, same engine version (0.1.38).

Is the expert cache set with an explicit --expert-cache N, or auto? Yours stays at 11,631 slots / 22.07 GiB at 262K, 524K and 1M, while the session grows with --max-context. With auto I would expect it to shrink at 1M: on my card the auto sizing gives 2,640 slots at 786,432 and 2,420 at 1,048,576 (free VRAM before the cache 8.00 vs 7.56 GiB, q4_0 KV, --kv-resident 32768). An explicit N is a byte budget, N times the largest blob (generate.cpp line 2943 in 0.1.38), compared only with the free figure read before the slots are written (lines 2945-2946). When that budget is below the free figure at every context, the sized cache comes out identical, as yours does.

Why that matters on Windows. The check made once the slots are actually written is auto-only (the loop breaks on !auto_cache, line 3023). Its comment (lines 2978-2981) gives the reason: under WDDM an allocation is not resident until touched, and "the free figure read before it can be ~1 GB too high", so a cache sized from it can fill the card and the driver then pages. With the driver's default sysmem fallback an over-subscribed allocation does not fail, it goes to system memory (line 2850 says the same). That would match your table: the same work, the stages that read the expert cache ("VRAM hits") 7-8x slower and the wait on the GPU about 3x, pure compute about 10%, and nothing at 524K if it still fits there. I measured the same mechanism today on the prompt side, on main (#765): a 24,576-token chunk with no room put ~9 GiB in Shared Usage (Dedicated +32 MiB), prefill ~6x slower, nothing in the log.

Two quick checks would separate this from TLB or locality:

the Windows GPU process counters for strata.exe at READY, at 524K and at 1M: \GPU Process Memory(*)\Shared Usage and Dedicated Usage. The pinned expert arena is the same at both, so a jump in Shared at 1M points to paging.
NVIDIA Control Panel, "CUDA - Sysmem Fallback Policy" set to "Prefer No Sysmem Fallback" for strata.exe, then 1M again. If it was paging, I would expect 1M to fail at start with the explicit cache, or to come up with a smaller cache under --expert-cache auto and decode closer to the 262K figures.

On the hypothesis list: the GDN state cannot be the part that grows. In src/core/session.cpp, gdn_state_floats(g) has no max_cells term; what scales with --max-context in session_bytes is the QSA state and the QSA buffers.

If the two checks don't settle it, I can try to reproduce your three-context table on the 5080 with STRATA_DECODE_TIMING=1 in the next few days.

Thanks - and this one is a mixed answer: your hypothesis explains what we see, but your first check comes back negative on our machine, so here is both.

It is an explicit N. --expert-cache 8900, which comes out as 11,631 slots / 22.07 GiB with the bundled profile (no calibration, no eviction). So we are on the explicit path and the fit check that runs once the slots are written never runs for us.

What the cache size does at 1,048,576 (yarn factor 4), engine 0.1.39, one 32 GB RTX 4080 SUPER, IQ3_S, fresh 43,969 / 45,670 / 47,956-token prompts, 200 generated each, one engine start per row:

cache	slots	free VRAM at READY (engine's own line)	decode tok/s
explicit 8900	11,631	0 MiB (`LOW`, suggests `--vram-reserve-mib 1212`)	13.7 / 14.7
auto	11,178	217 MiB (`LOW`)	102.1 / 122.3
auto + `--vram-reserve-mib 1500`	10,766	1,075 MiB	88.9 / 96.5 / 101.4
explicit 7000	9,148	4,130 MiB	94.1 / 97.3
the same 11,631-slot cache at 524,288, for reference	11,631	299 MiB	113.2 / 107.9

453 slots (0.86 GiB) separate 13.7 from 102 tok/s, and the slower arm had the higher cache hit rate (91.4 / 94.7% against 90.4 / 93.9%), so this is not misses. At 1,048,576 the size of the explicit budget decides it, and the startup line's free figure is the tell: 0 MiB collapses, 217 MiB is fine.

Your first check does not separate them here, though. We sampled \GPU Process Memory(*)\Shared Usage and Dedicated Usage for strata.exe once a second through all of it:

arm	Shared peak	Dedicated peak	card used peak
524,288, 11,631 slots	55,481 MB	30,062 MB	32,108 MB
1,048,576, 11,631 slots (13.7 tok/s)	62,931 MB	30,425 MB	32,132 MB
1,048,576, auto (102-122 tok/s)	62,409 MB	30,136 MB	31,522 MB
1,048,576, 9,148 slots (94-97 tok/s)	62,345 MB	26,186 MB	27,954 MB

Shared is ~55.5 GB at 262/524K and ~62.4 GB at 1M in every arm; the +6.9 GB is our pinned K/V going from 6.19 to 12.38 GiB. On this machine the counter tracks the deliberately pinned memory (the 50.3 GB expert arena plus the pinned K/V), so it does not move for the arm that is 7x slower, and Dedicated does not separate them either. If the driver is paging here, this counter does not show it to us - I would rather say that than claim the check passed. Your second check (Control Panel, "Prefer No Sysmem Fallback") we have not run.

Your correction accepted. I checked session_bytes: the max_cells terms are qsa_state_bytes and qsa_buffers_bytes, and gdn_state_floats(g) has none, so the GDN state cannot be what grows. The GDN layer's VRAM hits in our stage table is the expert-cache read every layer does, not the GDN state.

What we are doing with it: our own configuration is moving from a fixed 22.07 GiB to auto with --vram-reserve-mib 1500 at 1M (10,766 slots, 1,075 MiB free), and we have corrected the report in PR #780 - its "1M costs 1.6x the read and 2.2x the decode" line was this budget effect, not the context tier.

Thanks for running all of this, and for being so careful with it. Your table is much clearer than my guess: 0 MiB free at READY collapses, 217 MiB is fine, and the higher hit rate in the slow arm rules out misses.

You're right about the counter too, my mistake: if Shared Usage only follows the pinned arena and K/V, it couldn't show this.

One data point from my side, which also takes back the Control Panel test I suggested. On a 16 GB RTX 5080 (UD-IQ4_XS, int8 KV, 786,432, sysmem fallback set to "Prefer No Sysmem Fallback"), explicit --expert-cache 4000 with the reserve going down, one start and one 61K-token request per step:

reserve	cache	nvidia-smi used	decode tok/s	hit rate
770	2,782 slots, 6.26 GiB	15,678 MiB	40.9	59.9%
640	2,837, 6.39 GiB	15,748	36.5	61.2%
512	2,894, 6.52 GiB	15,765	39.4	62.1%
384	2,949, 6.64 GiB	15,779	35.4	61.8%
256	3,004, 6.76 GiB	15,774	34.2	61.9%
128	3,059, 6.89 GiB	15,783	22.4	60.0%
0	3,115, 7.01 GiB	15,807	16.6	63.0%

Every step reached READY with 0 MiB free. The cache grows by 768 MiB while the card's used memory moves by 129 MiB, the hit rate stays flat, and decode falls off past a point. Same shape as yours (one 256-token sample per step, so the upper rows are noisy). So with the fallback off it still starts and degrades instead of failing, and the Control Panel test would not have told you much. Sorry for sending you after it.

Thanks for correcting #780, too.

Thanks for taking the counter back - and the 5080 sweep is more useful than a repeat of ours, because it shows the same ordering on a different card, quantization and context length: as the reserve goes 770 -> 0 the cache grows by 768 MiB while the card's used memory moves only 129 MiB, the hit rate stays flat (59.9 -> 63.0%), and decode falls 40.9 -> 16.6. That is our result at 1,048,576 on the 4080 SUPER with the roles of "slots" and "slack" swapped around: the slack you leave for the rest of the card decided it, and the hit rate did not.

It also answers the Control Panel question better than the test would have: with Prefer No Sysmem Fallback set, your steps still reached READY and degraded instead of failing, so there was nothing for us to see there. No need to apologize - we would have run it and learned less.

I have added your sweep to the documentation in PR #799 as an independent check (with your note that it is one 256-token sample per step), next to the --vram-reserve-mib lever it agrees with.

A second machine (Linux, 2× RTX 5090, layer split) says: at healthy VRAM margins the 1M tier still costs ~4.4% decode at short prompts, and the yarn part of it is free. Data below in case it helps separate "design cost" from "VRAM cliff".

Setup: 2× RTX 5090 (32 GB, layer split auto), EPYC 9654, 377 GB RAM, Linux 6.8, driver 595.71; engine main@6f32ec0 (0.1.39), IQ3_S native pack, --kv int8 --kv-resident 32768 --expert-cache auto --spec 4 --spec-min-p 0.5, experimental projection on (identical in every run). Fresh engine start per config; short Chinese prose prompt (~few hundred tokens), greedy, 768 max tokens; 5 runs per config, first (warmup) dropped, median of the other 4 (run spread ≈ ±3%):

`--max-context`	rope	decode tok/s
131,072	none	165.2
131,072	yarn 4	164.6
524,288	yarn 4	162.7
1,048,576	none	157.7
1,048,576	yarn 4	157.8

Yarn is free here too (165.2≈164.6 and 157.7≈157.8 with the same pool), and the drop tracks the pool size at a prompt short enough that the live KV is negligible. Prefill (6.3K-token prompt) was flat across all five configs (3.5–4.1K tok/s; run-to-run spread dominates). Expert cache hit was 99.6–100% throughout.

On the free-VRAM mechanic from the comments above: with --expert-cache auto and the default --vram-reserve-mib 700 we saw only 699 MiB / 199 MiB free at READY (cache 12,800 slots / 23.08 GiB) — close to the knife edge discussed, but no collapse here. Raising the reserve to 1500 gave 1,197 / 1,071 MiB free and a slightly smaller cache (12,560 slots); decode did not move (157.8). So on this machine the residual cost is not the last-hundred-MiB cliff — it persists with >1 GiB free, at a much smaller magnitude than the 7–8x in the report.

If it is useful I can re-run a 131K vs 1M pair with STRATA_DECODE_TIMING=1 / STRATA_VERIFY_PROFILE=1 to see which stage actually takes the 4.4% (or sweep --kv-resident against pool size). Not urgent for us — 950K-token needles work at 1M and prefill there is ~6.5K tok/s, so the tier is worth using — but if the 4.4% is avoidable we would take it.

setup: bench-data recommendations (tips only), calibrate tries more p…

Thanks - this is the piece we were missing, and it changes how the tier is reported.

The 1M step is a cost, not a cliff. Your 131K -> 1M (165.2 -> 157.7, -4.4%) together with yarn being free of charge is what let us split "a few percent at healthy margins" from the 0 MiB case, instead of one number standing in for both. Before your run our own ladder could not separate them because the tiers moved at the same time as the cache budget.
--vram-reserve-mib is a safety knob, not a speed knob. Your 700 -> 1500 pair (699/199 -> 1,197/1,071 MiB free, decode 157.7 -> 157.8) is the control our sweep was missing: on our card every step of that sweep also changed the cache size, so we could not tell the two effects apart. Your version holds the cache roughly fixed and moves only the reserve.

Both are now cited as third-party measurements, with your handle, in the report and in docs/DETAILS.md (the section on the expert cache being a budget). We also took your 950K needle result at 1M: our report said no recall test had been run at any tier, and that was the weakest row in it.

On your offer: yes please. If you run one pair, the 1M healthy-margin arm against 131K. What we would look for is where the 4.4% sits - expert misses or PCIe traffic that scales with the footprint, against clocks and power - because on our side the only instruments that showed anything were decode tok/s and the cache hit rate: the counter that should have shown the traffic (\GPU Process Memory(*)\Shared Usage and Dedicated Usage) did not separate our arms at all, slow from fast, since it counts the deliberately pinned arena and KV pages.

One thing we cannot settle on our own hardware: our 7x case (0 MiB free at READY, 13.7 against 102 tok/s at 217 MiB) is specific in that Windows pages the over-committed allocation instead of failing. On Linux the same over-commit should fail loudly, which we have not verified. If your setup can produce it, that would tell us whether the mechanism is the paging or something else in that configuration.

Note also that the history cleanup closed our earlier PRs, so this data now lives on the current main: the report is #1351 and the docs section is #1354.

docs: the expert cache is a budget and an over-sized one pages on Win…

serve: a full card with an explicit --expert-cache says so, and what …

docs: troubleshooting for #781 #831, #649, #961 and #654

0.1.40.2: when the card is left with less than 256 MiB free at READY and the cache was set with an explicit --expert-cache N, the log now says so, names the slot count and suggests a smaller N or --expert-cache auto (which checks the free VRAM after the slots are written). It only says it; nothing is changed. Thanks for the numbers.

setup: bench-data recommendations (tips only), calibrate tries more p…

serve: a full card with an explicit --expert-cache says so, and what …

docs: troubleshooting for Niko1221#781 Niko1221#831, Niko1221#649, Ni…

エンジンの小さな修正を移す（プロファイルの版、密な重みのシャード、警告と --help）

Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.

If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.

Navigation Menu

Repository navigation

1M context: identical expert work per window, but the VRAM-touching stages grow 7-8x #781

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

1M context: identical expert work per window, but the VRAM-touching stages grow 7-8x #781

Description

Activity

enkynakamura commented on Oct 4, 2026

1314521gjy commented on Oct 4, 2026

enkynakamura commented on Oct 4, 2026

1314521gjy commented on Oct 4, 2026

gucasbrg commented on Oct 6, 2026

1314521gjy commented on Oct 7, 2026

Niko1221 commented on Oct 7, 2026

Niko1221 commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions