[Engine] the per-request #2065 VRAM check and the stall report sample device 0 only - the later card of a layer split can go low with no warning #4412

Area

[Engine] (speed, kernels, models, the server and API)

Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

What happened

The #2065 request-start guard reads free VRAM without a device wrap (generate.cpp ~13413-13429):

size_t vf_b = 0, vt_b = 0;
if (cudaMemGetInfo(&vf_b, &vt_b) == cudaSuccess) { ... }   // no OnDevice: stays on device 0

OnDevice restores the caller's device (on_device.hpp:14-17), and the serve thread's caller device is the primary, so the kVramSafeMib warning, the reserve-suggestion formula, and g_req_vram_free_mib (~1531-1533) are all device-0 values. The stall report then prints that single number as "VRAM free when this request began" (~1547-1552). On a layer split the card that fills first is typically the LATER one (stage layers + drafter + window captures + stage park buffers) - exactly the #2192 environment - yet no per-request warning can ever fire for it, and a stall on that card is reported with the roomier card-0 number.

The same file solved this class twice already, device-aware, and only this check was left behind: the startup check loops all stages per device (#2192, ~12099-12111, "add --vram-reserve-later-mib"), and the #2409 restore check samples main + stage devices (restore_devs loop, ~14136-14153).

Observed context on this machine (2x RTX 4090 24 GiB WDDM, K=22 split, engine 0.1.52, Windows 11): nvidia-smi steady-state free is GPU0 ~1220 MiB vs GPU1 ~1143 MiB (GPU1 consistently tighter), and the engine's own prefill line measured CUDA1 at 734 MiB free ("no VRAM for a super-chunk: layers 22..48 ... (734 MiB free)") - below kVramSafeMib - while no per-request #2065 warning ever mentioned CUDA1.

Expected: per-card sampling and warnings ("CUDA1: only X MiB free at the start of this request ... add --vram-reserve-later-mib N"), and a stall report that prints each sampled card. Actual: device 0 only.

Suggested fix: iterate stages with OnDevice exactly like the #2409 restore check, keep a per-device low-water dedup (the current static low_said_at is also shared), and print the per-card numbers in the stall report.

Smallest steps or command that reproduce it

Static finding (no triggered stall on my machine). What would verify: on a --layer-split rig, drive only the LATER card below kVramSafeMib (1024) at request start (tight --vram-reserve-later-mib plus expert traffic on the later stages) and send a request: no #2065 warning fires for that card although the device-0 check would warn at the same free value; then a stall on the later card prints device 0's number in the stall report.

Reproduction status

unconfirmed static analysis (code quoted above, re-read against the v0.2.7 source). Observed context quoted from this machine: the engine's own prefill line measured CUDA1 at 734 MiB free ("no VRAM for a super-chunk: layers 22..48 ... (734 MiB free)") while no per-request warning ever mentioned CUDA1; nvidia-smi steady-state free GPU0 ~1220 MiB vs GPU1 ~1143 MiB.

Strata version

Terminal and app 0.2.7, engine 0.1.52 (release store build windows-x86_64-cuda-cuda13), source checkout of v0.2.7 for the code citations.

Graphics card, memory and system

2x RTX 4090 24 GiB (both PCIe 4.0 x16, engine probe 26.3-26.4 GB/s), Threadripper 3970X, 128 GiB RAM, Windows 11 (WDDM), driver 610.88.

Model, size and settings

Qwen3.8-Flash-Next Unsloth UD-Q4_K_XL, --max-context 307200 --kv int8 --kv-resident 32768 --spec 4 --pipeline-windows 2 --layer-split auto (K=22) --vram-reserve-mib 1728 --vram-reserve-later-mib 972 --conversation-cache-mib 12288 --conversation-cache-slots 4 --expert-cache auto --prefill auto, STRATA_ROUTE_PRIOR=1.0.