[Engine] CUDA layer split, 0.1.44.1: later-stage (CUDA1) free VRAM drops 2-6 MiB per long cold prompt and does not plateau #3470

Setup

• Strata v0.1.44.1 (tag, built locally: CUDA 13.0 nvcc, CMAKE_CUDA_ARCHITECTURES=120, Release), Linux, driver 595.91.07
• 2x RTX 5060 Ti 16 GB (sm_120), 48 GB RAM
• Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S pack, --layer-split auto (K=23: CUDA0 layers 0-22, CUDA1 23-47 + head + MTP)
• Args: --expert-cache auto --prefill auto --spec 4 --mtp <rt> --max-context 262144 --kv int8 --resident-experts --resident-budget-gib 31.58 --vision --vram-reserve-mib 700 --remote-expert-opt --pcie-frac 0.35 --spec-min-p 0.70
  (plus a control vector, not applied on these requests). CUDA_MODULE_LOADING unset (EAGER forced by generate).
• Startup: CUDA1: 203 MiB of VRAM free with everything loaded - LOW ... add --vram-reserve-later-mib 1265

Observation

Free VRAM on CUDA1 (nvidia-smi -i 1 --query-gpu=memory.free) after each request, fresh process, single client:

• Steps 17-19 repeat an identical prompt (same length, same chunking, same window sizes) and still lose 4, 2 and 6 MiB.
• Flat while idle (an earlier process held 92 MiB overnight).
• No capturing the N-token window lines after the first prompt, so the verify graphs are not being recaptured.
• An earlier 0.1.44.1 process without --resident-budget-gib showed the same trend: 189 MiB at load, 126 after the first long
  prompt, 92 after 118 mixed requests.
• CUDA0: 330 MiB at load, 272 after the first prompt, still 272 after step 19.

Source read (0.1.44.1)

No allocator in src/ grows with prompt length or position:

• verify graphs are keyed by window size (verify.cpp:1881-1887), one extra set for the doorbell path;
• prompt-path buffers live in borrowed cache slots (prefill.cpp:1082-1219);
• the MTP pf_dev_ buffer is a high-water mark of at most one chunk (mtp.cpp:1403);
• KV is preallocated for --max-context (generate.cpp:4446);
• cuBLAS uses fixed 32 MiB workspaces (gemm.cu:441);
• there is no cudaMallocAsync or memory pool.

That leaves state inside the driver or cuBLAS, outside the source.

Mitigation tried: --vram-reserve-later-mib 1265

Same build and args, plus --vram-reserve-later-mib 1265 (the value the startup warning suggests), with --resident-budget-gib
raised from 31.58 to 32.14 so that the experts dropped from the GPU caches still get a RAM copy.

• The first-use drop is the same (about 62 MiB), so the reserve adds about 565 MiB of runway on CUDA1. At the observed
  2-6 MiB per long cold prompt, that is on the order of 100+ long prompts more than before.
• Prefill and cached decode are unchanged.
• Decode after a cold prompt has a median of 41.8 tok/s, at the low end of the earlier cold runs (41.7-52.0). I don't
  read it as a measurable cost of the 310 fewer GPU experts.
• CUDA0 loses slots too, although the reserve only applies to later stages, because it changes the split search
  (generate.cpp:3800-3812, stage_room). The search sees 0.56 GiB less room on CUDA1 and moves one layer to CUDA0
  (K=23 -> 24). CUDA0's own expert cache, sized after the split, then has 0.29 GiB less free: -116 slots. CUDA1 loses
  only 143 slots (not the ~275 the reserve alone would cost) because it no longer holds layer 23.

So for us, --vram-reserve-later-mib is a workable mitigation. It delays the problem but does not stop the drop, and we
also restart the server on a schedule when free VRAM falls below 250 MiB.

Questions

1. Is a slow per-prompt growth on later stages known? What does a full later card do here: stall, OOM, or a fallback?
2. Is there a way to trim it without a restart?
3. Is --vram-reserve-later-mib (placement only, generate.cpp:3808) the intended mitigation?