[Mirtraxxx]

Addresses #533 for the expert cache, the largest part of the VRAM: the engine gives the cache's VRAM back to other programs and takes it again, while the model, the context and the conversation cache stay loaded.

What it does

• Engine, --expert-cache-release (off by default). The expert cache is allocated with CUDA's virtual memory API (cuMemAddressReserve + cuMemCreate/cuMemMap) instead of cudaMalloc. Two new serve-loop commands:
  • RELEASE: every resident expert is marked not resident (and remembered), then the physical memory is unmapped and freed. The address range stays reserved, so the captured graphs and the experts' pointers stay valid.
  • REFILL: new physical memory is mapped at the same address, and the remembered experts are copied back into the same slots from the host copy. The residency table is restored as it was.
  • A GEN/GENI that arrives while the cache is released refills it first, so a client needs nothing new.
• Server:
  • POST /cache/release and POST /cache/refill answer 409 while a request is running and 400 when the engine can't (e.g. started without the option).
  • --cache-release-idle SECONDS / "cache_release_idle_s" gives the cache back after that many seconds without requests. It adds the engine option itself, and it works like --idle-unload but keeps the model loaded.
• Docs: a paragraph and a table row in DETAILS.md's "Sharing the GPU with other programs".

Measured

RTX 3090 24 GB, Windows 11, driver 616.56, a Qwen3.8-Flash-Next fine-tune at IQ2_XS, --expert-cache auto, MTP on:

VRAM given back	14.5 GiB (10,800 experts) in ~50 ms
refill	1.7-2.5 s
byte check (STRATA_VERIFY_REFILL=1 reads every refilled slot back)	0 of 10,800 slots differed from their host copy, over 3 release/refill cycles
a ComfyUI video render while the model stays loaded	176 s with the cache in VRAM, 128 s with it given back
decoding after a refill	~72 tok/s (a 400-token answer)

serve/test_lifecycle.py gets three tests: the endpoints, the idle wait, and the engine flag being added. test_server, test_lifecycle, test_monitor and test_structured pass (140 tests).

Limits

• Only the expert cache is given back. The dense weights, the KV cache and the engine's buffers stay; on the 3090 the card still held about 7 GiB, the desktop included. A full /unload is still the way to free everything.
• CUDA only. On HIP, when the device has no VMM support, with a layer split, and in the resident RAM mode (some experts only in VRAM), the engine prints why and keeps the cache as before. Not built or tested on HIP or Linux here; the HIP paths are #if-guarded, and CUDA::cuda_driver is only linked in the CUDA build.
• Refill when the VRAM is actually free. On Windows, a refill into a card another program still fills makes WDDM move memory to shared system RAM, and decoding ran at ~10-17 tok/s in the same setup until the other program let go. The engine can't tell that this happened, so deciding when to refill stays with the caller (or with the next request).

The idea and the use case are AXKore's in #533 (a CAD/games workstation). Mine is an LLM sharing a 3090 with ComfyUI.

🤖 Generated with Claude Code

[q8atnight]

Linux data point, and a use you may not have planned for: we use --expert-cache-release to let two Strata processes share one GPU's VRAM over time on a 2× RTX 3090 box (NVLink), IQ3_S, merged onto v0.1.38 (one conflicted file, two trivial keep-both hunks in generate.cpp's options/argv).

Setup: engine B owns GPU0 with --expert-cache-release --shared-expert-arena /dev/shm/...; engine A runs on GPU1 with --peer-device 1 (GPU0 as its peer expert tier). While B is idle it is RELEASEd (and the rest of its GPU state is parked with NVIDIA's cuda-checkpoint), so A's peer tier can take GPU0; when a second request arrives, A drops its peer tier and B gets REFILL.

Measured on Linux (driver 595, CUDA 13), 3090:

• RELEASE: 14.06 GiB (7,417 experts) back in 9-10 ms
• REFILL from the shared host arena (MAP_SHARED /dev/shm, registered by both processes): 1.14 s for the same 14.06 GiB, repeatable over many cycles — ~2× faster than your Windows numbers, as expected without WDDM
• outputs after refill: the exactness gate path (greedy, static residency) is unchanged by the merge (gate PASS with the option compiled in and unused); generations after many release/refill cycles are coherent.

Two notes from that use:

1. A refill can fail with cuMemCreate ... out of memory when another process took a little of the card meanwhile (here: the other engine's CUDA context, ~0.5 GiB, on the shared card). Sizing the cache with --vram-reserve-mib set for that headroom fixed it; maybe worth a sentence in the docs (or a partial refill instead of failing).
2. Nothing in the PR knows about --peer-device (it predates 0.1.38) — release/refill only touch the primary cache, which is exactly what we needed, but a combined process (--peer-device + --expert-cache-release) is untested here.

Thanks — this was the missing primitive for us.

[Mirtraxxx]

Closing: v0.1.39 shipped #533 as --vram-elastic + POST /v1/vram, which does this better (any size instead of all-or-nothing, and it grows back only as far as free VRAM allows). I've moved my own setup (a relay that lends the cache to ComfyUI during renders) over to it and it works well.

@q8atnight thanks for the data points. Your engine B (one GPU, --serve, no peer tier) meets --vram-elastic's conditions: {"reserve_mib": 1048576} shrinks the cache to its floor (the release), and {"reserve_mib": null} (or a number for headroom) grows it back. Because the grow takes only what's free, your note 1 (a refill failing when another context took ~0.5 GiB) should go away. One thing I measured on Windows: growing back to the full size with only ~400 MiB left free made WDDM page the engine (decode 17 vs 64 tok/s), so leaving ~2.5 GiB of headroom with reserve_mib helped there.