Expert cache: STRATA_VRAM_FREE=device reads the whole card's free VRAM through NVML (opt-in, Windows + NVIDIA) by Xiaomi2008 · Pull Request #1911 · Niko1221/Strata

Xiaomi2008 · 2026-10-11T02:03:37Z

Title

Issue: none (searched the open issues and pull requests; nothing covers this)

Summary

STRATA_VRAM_FREE=device (opt-in, Windows + NVIDIA): --expert-cache auto reads the free VRAM of the whole card through NVML, and takes the smaller of that and CUDA's own figure.

Under WDDM cudaMemGetInfo answers for the calling process only. Measured on an RTX 5090, driver 610.88: with 6 GiB written by another process, cudaMemGetInfo in a second process still read 30,994 MiB free where NVML read 24,348. So with the image encoder loaded (1,547 MiB here) the cache is sized from a figure that much too high, the check after the slots are written reads less than the reserve, and the cache is shrunk by guesses.

Six starts of one config on 0.1.40.3 and 0.1.41 (Swift 1.5 IQ3_XXS, 128K, the default 700 MiB reserve, image encoder on the GPU) ended with 9,002 to 14,454 slots and 93 MiB to 10 GiB of VRAM free.

The switch is used in three places: the sizing, the check after the slots are written, and the serve check's "VRAM free with everything loaded" figure. nvml.dll is loaded from the system directory at first use and the device is found by its PCI bus id. One line at first use prints both figures.

What changed

src/core/expert_cache.cpp: device_free_bytes() takes the smaller of CUDA's figure and NVML's when the variable is set (a new nvml_device_free, Windows CUDA builds only).
include/strata/core/expert_cache.hpp: the comment on device_free_bytes().
src/program/generate.cpp: the serve check's free figure goes through the same function (one line).
docs/DETAILS.md: one entry with the measurements.

Extra Notes

Measured

This branch (0.1.42 + this change), RTX 5090, Core i7-13700K, 64 GB, Windows 11, Swift 1.5 IQ3_XXS, 128K, image encoder on the GPU, three interleaved server starts each:

	slots	shrink steps	VRAM free with everything loaded
default (700 MiB reserve)	13,543	2	336 / 602 / 602 MiB
`STRATA_VRAM_FREE=device`, `--vram-reserve-mib 1500`	13,102 - 13,130	1	1,107 / 1,225 / 1,288 MiB

At the sizing NVML read 23,695 MiB free where CUDA read 25,458. On 0.1.41 with the switch: one pass, 13,007 to 13,188 slots and 973 to 1,392 MiB free over seven starts.

Why the headroom matters here: a run that filled the card to 0 MiB free decoded 80-115 tok/s where the same forced text ran at 205-230 tok/s with 900 MiB or more free.

What it leaves alone

Unset, device_free_bytes() returns what it did, and the serve check's figure is the same value (min of the same number). I did not run the byte-identity tests; this is from the code path: the new branch is behind the environment variable.
The cache size changes no output bit.
Linux, HIP and SYCL: the new code is compiled for Windows CUDA builds only (STRATA_EC_NVML_FREE).
The shrink loop itself is unchanged.

Not tested

Built for CUDA only: sm_120, CUDA 13.0.88, MSVC 14.50 with nvcc's --allow-unsupported-compiler (this PC has no Visual Studio 2022). HIP and SYCL were not compiled. expert_cache.cpp and generate.cpp are built by both; the HIP build compiles none of the new branch (it sits under !STRATA_EC_NO_VMM), SYCL I did not look at.
One machine, one card, one driver. No multi-GPU run (the device is matched by PCI bus id, which I could only exercise with one card).
Cards where another process holds no VRAM show no difference, by construction.

Docs: one entry in docs/DETAILS.md.

In use, with the rest of a tuned config (context)

0.1.42 built from source, through the server, two interleaved rounds of the same requests on this PC: setup's arguments ended with 458 / 561 MiB of VRAM free with everything loaded, a config with STRATA_VRAM_FREE=device and --vram-reserve-mib 1500 with 1,039 / 1,083 MiB. That config also changes the CPU pool layout and other switches, and its speed table is in #1912; none of the speed difference there is this change, which only decides how much VRAM the cache takes.

🤖 Generated with Claude Code

…M through NVML (opt-in, Windows + NVIDIA) Under WDDM cudaMemGetInfo answers for this process alone: memory another process holds on the card is not in its free figure. Measured on an RTX 5090, driver 610.88: with 6 GiB written by another process, cudaMemGetInfo in a second process read 30,994 MiB free where NVML read 24,348. With the image encoder loaded (1,547 MiB here) `--expert-cache auto` sizes the cache from a figure that much too high, reads less than the reserve once the slots are written and shrinks by guesses. Six starts of one config on 0.1.40.3 and 0.1.41 (IQ3_XXS, 128K, the default 700 MiB reserve) ended with 9,002 to 14,454 slots and 93 MiB to 10 GiB of VRAM free. STRATA_VRAM_FREE=device takes the smaller of CUDA's figure and NVML's (nvml.dll from the system directory, loaded at first use, the device found by its PCI bus id), for the sizing, the check after the slots are written and the serve check's free figure. One line at first use gives both figures. Measured on this branch (0.1.42 + this change), RTX 5090, Core i7-13700K, 64 GB, Windows 11, Swift 1.5 IQ3_XXS, 128K, the image encoder on the GPU, three interleaved server starts each: default (700 MiB reserve) 13,543 slots, two shrink steps, 336 / 602 / 602 MiB free the switch, --vram-reserve-mib 1500 13,102-13,130 slots, one shrink step, 1,107 / 1,225 / 1,288 MiB free (NVML read 23,695 MiB free at the sizing where CUDA read 25,458) On 0.1.41 with the switch: one pass, 13,007-13,188 slots and 973-1,392 MiB free over seven starts. A run that filled the card to 0 MiB free decoded 80-115 tok/s where the same text ran at 205-230 with 900 MiB or more free. Unset, device_free_bytes() returns what it did and the serve check's figure is the same value. The branch is compiled for Windows CUDA builds only (STRATA_EC_NVML_FREE); HIP compiles none of it. Built for CUDA only (sm_120, CUDA 13.0.88, MSVC 14.50 with nvcc's --allow-unsupported-compiler); HIP and SYCL were not compiled. The cache size changes no output bit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…a intera via NVML (STRATA_VRAM_FREE=device) Diff di gh pr diff 1911 (Xiaomi2008, head 44bc51a) applicato senza modifiche. Opt-in, solo Windows + NVIDIA: device_free_bytes() prende il minimo tra cudaMemGetInfo (sotto WDDM conta solo questo processo) e la VRAM libera dell'intera scheda letta da nvml.dll. La scheda NVML si trova dal PCI bus id del device CUDA corrente (dominio:bus:device.0), quindi l'abbinamento non dipende dall'ordine di CUDA_VISIBLE_DEVICES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… reads the whole card's free VRAM through NVML (opt-in, Windows + NVIDIA) Expert cache: STRATA_VRAM_FREE=device reads the whole card's free VRAM through NVML (opt-in, Windows + NVIDIA) Under WDDM cudaMemGetInfo answers for this process alone: memory another process holds on the card is not in its free figure. Measured on an RTX 5090, driver 610.88: with 6 GiB written by another process, cudaMemGetInfo in a second process read 30,994 MiB free where NVML read 24,348. With the image encoder loaded (1,547 MiB here) `--expert-cache auto` sizes the cache from a figure that much too high, reads less than the reserve once the slots are written and shrinks by guesses. Six starts of one config on 0.1.40.3 and 0.1.41 (IQ3_XXS, 128K, the default 700 MiB reserve) ended with 9,002 to 14,454 slots and 93 MiB to 10 GiB of VRAM free. STRATA_VRAM_FREE=device takes the smaller of CUDA's figure and NVML's (nvml.dll from the system directory, loaded at first use, the device found by its PCI bus id), for the sizing, the check after the slots are written and the serve check's free figure. One line at first use gives both figures. Measured on this branch (0.1.42 + this change), RTX 5090, Core i7-13700K, 64 GB, Windows 11, Swift 1.5 IQ3_XXS, 128K, the image encoder on the GPU, three interleaved server starts each: default (700 MiB reserve) 13,543 slots, two shrink steps, 336 / 602 / 602 MiB free the switch, --vram-reserve-mib 1500 13,102-13,130 slots, one shrink step, 1,107 / 1,225 / 1,288 MiB free (NVML read 23,695 MiB free at the sizing where CUDA read 25,458) On 0.1.41 with the switch: one pass, 13,007-13,188 slots and 973-1,392 MiB free over seven starts. A run that filled the card to 0 MiB free decoded 80-115 tok/s where the same text ran at 205-230 with 900 MiB or more free. Unset, device_free_bytes() returns what it did and the serve check's figure is the same value. The branch is compiled for Windows CUDA builds only (STRATA_EC_NVML_FREE); HIP compiles none of it. Built for CUDA only (sm_120, CUDA 13.0.88, MSVC 14.50 with nvcc's --allow-unsupported-compiler); HIP and SYCL were not compiled. The cache size changes no output bit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD

… intorno al codice NVML di Niko1221#1911, il codice resta Seguito della leva 07 (vram-free-nvml): nessun cambio di comportamento, solo le guardie #if che non spegnevano nulla. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Niko1221 · 2026-10-17T07:14:04Z

Thanks. Windows-only and about sizing rather than speed; we have no Windows box to build it on.

…VRAM libera della scheda intera via NVML (STRATA_VRAM_FREE=device), letta anche dalla cache elastica Patch 07 della pila (ora 06): conflitto solo nella documentazione (DETAILS.md), tenute entrambe le voci. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…rough NVML, opt-in) — integ batch 19 [nvml block added beside integ's sealed_write; docs bullet placed in its list]

…iko1221#1911 slice (C1070) + verify.cpp diag_pipelined parked slot (format arg count, dangling .c_str() temporary)

…M through NVML, Windows + NVIDIA, opt-in (#1911) Under WDDM cudaMemGetInfo answers for the calling process: memory another process holds on the card (the image encoder, a browser, another engine) is not in its free figure. The reporter measured, on an RTX 5090, 30,994 MiB free from CUDA where NVML read 24,348 with 6 GiB held elsewhere, so --expert-cache auto sized the cache too large and shrank it by guesses. device_free_bytes() now takes min(CUDA's figure, NVML's figure for the whole card) when STRATA_VRAM_FREE=device is set. nvml.dll is loaded from the system directory at first use (LoadLibraryExW with LOAD_LIBRARY_SEARCH_SYSTEM32), the card is matched by its PCI bus id, and a missing nvml.dll, a missing export or a failed nvmlInit leaves CUDA's figure with one line on stderr. The serve check of "VRAM free with everything loaded" uses the same function. The value can only get smaller, so the switch can only shrink the expert cache. Unset (the default), nothing changes: the NVML code is compiled only for _WIN32 and not for HIP, Metal or the no-VMM build, and the env variable is read once. The serve check's min() is the same value elsewhere (on a unified-memory system device_free_bytes() is never below cudaMemGetInfo's figure). Added to the PR: the NVML block is also left out of a SYCL build (SYCL_LANGUAGE_VERSION), whose CUDA names are a compatibility layer. docs/DETAILS.md: one row in the measured-and-left-off table instead of the PR's long paragraph. Tests: builds on Linux CUDA (the Windows-only block is skipped), SYCL (P1006) and on Windows MSVC (owner PC build of the final tree). Not covered: no run with a model; the Windows gate is written down.

…ports for the conversation cache, prefill events, expert source and batch groups)

Niko1221 · 2026-10-27T15:24:44Z

Thank you. Fixed in 0.2.7: STRATA_VRAM_FREE=device reads the whole card's free VRAM through NVML, Windows + NVIDIA, opt-in. Your name is on the commit.

Release: https://github.com/Niko1221/Strata/releases/tag/v0.2.7

…ool-support Conflicts (14 files): - generate.cpp (12): upstream took Niko1221#1284 (--adapt-decay checked); STRATA_PIPED_MTP (Niko1221#4121) beside the pool's batch-MTP rules (pmtp needs a local split, so never on a pool coordinator, whose last stage may be a worker: st_out/st_launch keep the remote stage, pmtp_verdict / batch_launch_rows only where pmtp is on), the pool worker's draft-in-groups refusal kept, the pool's 96 MiB beside nibble_max_slot_bytes, wide pool rows beside batch_max_t(), the invariant window pool function with pool_pipe, pool_conv_retain closed before ParkSource, the slot penalty windows in the non-pmtp group fill, the pool's batch-wait line and the local-stage profile loop. - prefill.cpp: layer-major prefill (Niko1221#3976) declines on a pool coordinator or worker (remote_next / headless): its walk has no hand-off to other PCs yet. - iq_avx512.cpp: upstream's Singleton rows beside Niko1221#1374's IQ1_S / IQ1_M 512-bit rows (row_dot_any forwards it); iq_kernels.cu: IQ1_S (19) beside the new TQP types 147-149. - conversation_cache.hpp: pool_id beside dev_tag / pending; expert_cache.*: upstream's Niko1221#1911 and nibble code. - serve/server.py: split serving (Niko1221#4000) in the plain (role off) branch beside the pool roles; #pool kept on the URL. - serve/lifeline.py, strata_control/procs.py: upstream's own fix of the watcher token (start time only, /proc first) replaces ours. index.html: the Pool tab beside the i18n tabs. .gitignore both; CMakeLists 0.1.52; README ours. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD

… the IQ1_M rows note follows 0.1.52's Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD

Xiaomi2008 mentioned this pull request Oct 12, 2026

STRATA_EXCHANGE_ROTATE: rotation for packs whose layers differ in expert block size (spares kept per size) #2042

Closed

kbc20f11 mentioned this pull request Oct 19, 2026

Bound Qwen3.6 Windows CUDA placement by dedicated VRAM headroom #3128

Closed

Niko1221 added a commit that referenced this pull request Oct 27, 2026

Merge f-027-gh-eng-k2 (#2526 #1864 #1911 #4005 #3060 #4010 #3809: PR …

675f7fc

…ports for the conversation cache, prefill events, expert source and batch groups)

Niko1221 closed this Oct 27, 2026

Navigation Menu

Repository navigation

Expert cache: STRATA_VRAM_FREE=device reads the whole card's free VRAM through NVML (opt-in, Windows + NVIDIA) - #1911

Expert cache: STRATA_VRAM_FREE=device reads the whole card's free VRAM through NVML (opt-in, Windows + NVIDIA)#1911
Xiaomi2008 wants to merge 1 commit into
Niko1221:mainfrom
Xiaomi2008:pr/vram-free-device

Xiaomi2008 commented Oct 11, 2026 •

edited

Loading

Uh oh!

Niko1221 commented Oct 17, 2026

Uh oh!

Niko1221 commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Xiaomi2008 commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Title

Summary

What changed

Extra Notes

Measured

What it leaves alone

Not tested

In use, with the rest of a tuned config (context)

Uh oh!

Niko1221 commented Oct 17, 2026

Uh oh!

Niko1221 commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Xiaomi2008 commented Oct 11, 2026 •

edited

Loading