Repository navigation
Conversation
`auto` sizes the VRAM-resident expert tier to every byte the card reports free. On a card that ends up nearly full that is wrong by a lot: the prompt path reads tens of times slower while doing exactly the same work. Under WDDM an over-subscribed allocation does not fail, it pages to system memory and crawls - the engine's own comment at the sizing says so. What reserved against it was `prefill_mib`, which is zero when the prompt path borrows cache slots, which is the default with a profile. So the only protection was `--vram-reserve-mib`, and 2 GiB is not enough. Measured on an Arc Pro B70 (32 GB, driver 32.0.101.8976), the Qwen3.8-Flash-Next IQ3_XXS pack, OpenCL backend, eager, `--prefill 4096`, greedy, 64-token prompt. The same 3,668 gate/up and 3,668 down naive GEMM calls at every cache size - only the cache size and the memory left over change: 17,687 slots (28.82 GiB, `auto` before) 426 s 111 ms per expert GEMM call 15,996 slots (26.03 GiB) 222 s 54 ms 15,382 slots (24.95 GiB) 472 s 122 ms 14,704 slots (23.87 GiB) 9.7 s 2.32 ms 10,698 slots (17.36 GiB) 10.3 s 2.39 ms A cliff, not a slope: every size up to 14,704 slots is stable at ~10 s, every size from 15,382 up takes minutes. The two expert GEMM phases are 95.8% of the GPU timeline in the slow arm, and the phase waiting for the host-to-device staging copies is 34 ms of 957,283 - the bytes are not the problem. Cutting the cache to 14,688 slots puts the same calls at 2.48 ms. Memory during the slow pass peaks at 31.2 GiB dedicated (the card full) and holds ~15 GiB shared; the fast one peaks at 30.08 GiB dedicated and ~3.4 GiB shared. Consistent with WDDM paging, not proven. Not the cause, each measured: the host mirror (35% of the same pack mirrored over PCIe costs nothing, 2.05 ms per call against 2.01 with every expert resident); the prompt length (the cost is fixed per expert call, not per token); subnormals in the dequantized weights or activations (there are none in either pack); `--kv-resident` (7,589 ms against 7,587 ms). The kernel calls and their shapes are identical in every arm. No accuracy cost: the cut and uncut sizing give identical greedy output tokens - eight tokens, `271 71093 271 550 18381 198 12 1510`, from a scrambled 48-token prompt. This pull request adds the knob and changes no default: the share of free device memory `auto` keeps for the session's other buffers is `strata::core::expert_cache_headroom()` in include/strata/core/expert_cache.hpp, STRATA_EXPERT_CACHE_HEADROOM=<fraction> "1/6", "0.17", or "0" for the old sizing read by the CUDA/HIP sizing here. Its default is zero, so every backend keeps the sizing it always had and the default path stays byte-identical. An explicit `--expert-cache N` is the user's own budget and is never cut, and when the headroom would leave no room at all the old sizing stands. The SYCL port takes the same helper with its own default, on the port's branch (Niko1221#1390). docs/EXPERT_CACHE_HEADROOM.md carries the findings, the A/B protocol for a card this has not been measured on, and what each outcome would mean. Not compiled here: this machine has no nvcc or hipcc, so the CUDA/HIP sources carrying the change were not built where it was written. The header compiles (the SYCL build includes it and passes) and the call site is the same statement the SYCL port uses.
…Niko1221#1549) Measured on an RTX 5070 Ti (16 GB, Windows 11, WDDM, driver 610.62), this branch built with CUDA 13.1 and MSVC 19.44 for sm_120, the Coder IQ1_M config, tools/ab_engine.py, three rounds alternating: headroom 0 prompt 2,022 tok/s decode 64.5 tok/s (medians, 6,345-token prompt) headroom 1/6 prompt 2,064 tok/s decode 66.6 tok/s Inside the run-to-run spread; no cliff. The CUDA path already reads the free VRAM after the slots are written and shrinks the cache until --vram-reserve-mib is free, so both arms end within 0.3 GiB of each other. CUDA's default stays 0. generate.cpp with the change compiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he GEMM probe settled (Niko1221#1549) Answers the question the RTX 5070 Ti run left open: the SYCL tree has the same post-write check as CUDA, and it cannot fire on the OpenCL backend. dpct's get_memory_info is a port addition whose own comment says the free query needs ext_intel_free_memory (Level Zero sysman), which OpenCL does not have, and STRATA_DEVICE_FREE_MIB makes it return a setup-time constant instead - 31,744 MiB on a 32 GB card, the same before the cache is allocated and after it. Printed on the B70: 'post-write free 31.00 GiB, want 2048 MiB, cache 17687 slots (28.82 GiB)'. So on Windows/OpenCL the check compares a constant to a threshold and nothing protects the run; on CUDA cudaMemGetInfo is live, the check works, and the headroom is a no-op, which is what the 5070 Ti measurement shows. Also corrects what this doc said the cliff was: sycl/probe/vram_pressure.cpp times the engine's own Gemm::f16 at the prompt path's shapes with the card empty and again after allocating and touching 28 GiB of it - 1,467 us and 1,467 us, 380 and 382 us, so a full card does not slow the GEMM. Under STRATA_PREFILL_SYNC=1 the slowdown is spread across most kernels rather than the expert path (gdn 55x, the QSA projections 42x, the GDN output projection 53x, the two expert GEMMs 39x, the whole prefill 36x). What the GPU is doing with the last 2 GiB is still unexplained, and the doc now says so.
Contributor
Author
|
Superseded by #1890, which carries the same STRATA_EXPERT_CACHE_HEADROOM helper on top of upstream main (this branch's base, fb58e0d, has since been merged upstream as a1c47bc). The knob's default there is 0 everywhere and the real fix on this backend is the live free-VRAM figure, so a second pull request for the helper would only collide with it. |
Maddoxkkm
added a commit
to Maddoxkkm/Strata
that referenced
this pull request
Oct 11, 2026
What the release brought for this box (Niko1221#1709 Windows HIP VRAM floor, the HIP stream-capture fix, the Linux-only PCIe telemetry), the merge table with what was kept and why Niko1221#1439/Niko1221#1841/Niko1221#1656 went, the measurements (0.1.42+local +5.6% at 16K; the four new PRs +4.4% on top), and the ranked shortlist for the next cycle (Niko1221#1297 bisect, Niko1221#1909, Niko1221#1324, Niko1221#1167, Niko1221#1298/Niko1221#1151) with the gfx1100 reading list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
--expert-cache autosizes the VRAM-resident expert tier to every byte the card reports free. On a cardthat ends up nearly full that is wrong by a lot: the prompt path then reads tens of times slower while
doing exactly the same work.
It adds one opt-in setting and changes no default:
the share of free device memory
autokeeps for the session's other buffers instead of experts. Thehelper is
strata::core::expert_cache_headroom()ininclude/strata/core/expert_cache.hpp, read bythe CUDA/HIP sizing in
src/program/generate.cpp.What it leaves alone. The default is zero, so every backend keeps the sizing it always had and the
default path is byte-identical to the last release: with the variable unset, the old expression is
evaluated. An explicit
--expert-cache Nis the user's own budget and is never cut. When the headroomwould leave no room at all, the old sizing stands. Nothing in the prompt, decode or expert paths is
touched - the cache is just sized smaller.
Why, and what it was measured on
An Arc Pro B70 (32 GB, driver 32.0.101.8976), the Qwen3.8-Flash-Next IQ3_XXS pack, OpenCL backend,
eager,
--prefill 4096, greedy, 64-token prompt. The same 3,668 gate/up and 3,668 down naive GEMM callsat every cache size - only the cache size and the memory left over change:
autobefore)A cliff, not a slope: every size up to 14,704 slots is stable at ~10 s and every size from 15,382 up
takes minutes. The two expert GEMM phases are 95.8% of the GPU timeline in the slow arm, and the phase
waiting for the host-to-device staging copies is 34 ms of 957,283 - the bytes are not the problem.
Memory during the slow pass peaks at 31.2 GiB of dedicated memory (the card full) and holds ~15 GiB
shared; the fast one peaks at 30.08 GiB dedicated and ~3.4 GiB shared. That is consistent with WDDM
paging an over-subscribed allocation to system memory, which the engine's own comment at the sizing
describes, but it is not proven.
Ruled out, each measured: the host mirror (35% of the same pack mirrored over PCIe costs nothing, 2.05
ms per call against 2.01 with every expert resident); the prompt length (the cost is fixed per expert
call, not per token); subnormals in the dequantized weights or activations (none in either pack);
--kv-resident(7,589 ms against 7,587 ms). The kernel calls and their shapes are identical in everyarm. No accuracy cost: the cut and the uncut sizing give identical greedy output tokens - eight tokens,
271 71093 271 550 18381 198 12 1510, from a scrambled 48-token prompt.docs/EXPERT_CACHE_HEADROOM.mdhas the full findings, the A/B protocol for a card this has not beenmeasured on, and what each outcome would mean;
docs/TEST_REQUESTS.mdgets a row for it.How I checked it
free_roomis the expression the last release evaluates - no code pathchanges.
1/6set on the B70, the knob reproduces the measurement: 14,688 slots (23.83 GiB) against17,687 (28.82 GiB) unset, with identical output tokens.
nvccorhipcc, so the CUDA/HIP sources in this pull requestwere not built where the change was written. The call site is the same statement the SYCL port makes
against the same helper.
What this is not
This is not a claim that the effect is real on NVIDIA or AMD. It is Windows/WDDM evidence from one
Intel card, offered as a knob so it can be measured elsewhere. If it turns out to be a no-op on CUDA
the default stays zero, and if the cliff is real there the numbers from the A/B are what would justify
turning it on.
Closes the sizing half of #1549.