[Engine] CUDA layer split auto leaves 68 MiB on the last card; verify window capture then fails with out of memory and the engine exits (RTX 3060 + RTX 5060 Ti, 0.2.2) #3702

Area

[Engine]

Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build): engine built from the v0.2.2 tag (011c9e60)

What happened

On an unequal NVIDIA pair (RTX 3060 12 GB + RTX 5060 Ti 16 GB) with "layer_split": "auto" and the default reserves, the split plan leaves 68 MiB free on the last card (CUDA1, the 5060 Ti, which holds the head, the draft layer and the verify). The engine says so at start:

strata serve: CUDA1: 68 MiB of VRAM free with everything loaded - LOW: its window graphs are captured on first use and need room too, and a full card can stall a request; add --vram-reserve-later-mib 1400 to the config's args (or lower --max-context)

Each new verify window size is captured on first use and takes a few MiB on CUDA1 (68 → 64 → 58 → 52 → 10 MiB). The next capture then fails and the engine exits:

• requests fail with verify: instantiate: out of memory, or with the engine stopped unexpectedly (exit code 1); the next request restarts it
• once the engine stopped inside the prompt path instead: prefill gemm: cublasGemmEx f16: cuBLAS status 14
• the engine started 6 times in 11 minutes; 8 requests of a strata_bench.py --long run failed (chat, all three 4K prompts, 16K, agent)

Expected: the auto placement keeps enough room on the later card for its own window graphs, or a failed capture falls back instead of ending the engine. The search reserves kWindowMib = 96 MiB for the verify windows on a later stage (stage_room in src/program/generate.cpp), but the card ended up with 68 MiB free after loading.

Workaround that works here: --vram-reserve-later-mib 1400 (the value the log suggests). With it the same card has 770 MiB free after loading and none of 10 later --long runs failed (5 configurations: both card orders under auto and three manual splits, each cold and warm). It costs expert-cache slots on the last card (auto K=18 → K=19, CUDA1 cache 2,579 → 2,343 slots).

Related: #3470 (later-stage free VRAM shrinking on long prompts, same LOW warning; this report shows what happens when it runs out), #3664 (an MTP capture that ends the engine on out of memory).

Smallest steps or command that reproduce it

1. Config for UD-Q4_K_XL in resident RAM-budget mode with "gpu": [0, 1] (5060 Ti = 0, 3060 = 1 in nvidia-smi), "layer_split": "auto" and no --vram-reserve-* option; the server puts the faster card last (CUDA0 = 3060, CUDA1 = 5060 Ti). Engine args:
   --native <UD-Q4_K_XL shard 1> --resident-budget-gib 38 --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <mtp/rt> --max-context 40960 --kv int8
2. Start the server; the start-up log shows CUDA1: 68 MiB of VRAM free ... LOW.
3. Send a few short chats and a ~4K-token prompt (python tools/strata_bench.py --config <cfg> --long is enough). Within the first minutes a request fails with verify: instantiate: out of memory and the engine exits.

It happened on every start (6 starts in 11 minutes, until the run was stopped). An earlier run on 0.2.0-beta with the same pair also logged CUDA1: 58-70 MiB ... LOW, cuBLAS status 14 twice and engine restarts (not analysed further at the time).

Reproduction status

Reproduced locally: on 0.2.2 (every start, 6 starts). Workaround verified: 0 failed requests in 10 --long runs with --vram-reserve-later-mib 1400.

Strata version

Engine strata 0.1.47, built from the v0.2.2 tag (011c9e60), CUDA 13.3 toolkit, archs 86;120. serve/server.py and tools/strata_bench.py from the same tag; under auto the server puts the faster card last and adds no later-card reserve.

Graphics card, memory and system

• CUDA0: RTX 3060 12 GB (sm_86), PCIe Gen3 x16 (12.3 GB/s H2D)
• CUDA1: RTX 5060 Ti 16 GB (sm_120), PCIe Gen3 x8 (6.9 GB/s H2D); neither card drives a display
• Driver 615.71.09, Fedora 44 (kernel 7.2.11), i7-7800X, 62 GiB RAM

Model, size and settings

Qwen3.8-Flash-Next UD-Q4_K_XL (Unsloth, 4 shards), resident RAM budget 38 GiB (the rest read from an NVMe file tier), context 40,960, --kv int8, MTP on, --spec 4, no pictures. STRATA_ROUTE_TAIL_SKIP default (on).

Logs

Start-up and the first capture sequence, first start (paths shortened): see the attached engine_log_excerpt.txt. The lines that matter:

strata generate: layer split auto: K=18 - predicted 58.9 ms per decode window; the caches hold 4583 of 24576 profiled pairs (~94.6% of the routed mass), best of 48 placements
strata generate: layer split: CUDA1 runs layers 18-47, expert cache 2579 slots (7.50 GiB), 2579 of its 15360 profiled pairs; slot 0 verified
strata serve: 356 MiB of VRAM free with everything loaded
strata serve: CUDA1: 68 MiB of VRAM free with everything loaded - LOW: ... add --vram-reserve-later-mib 1400 to the config's args (or lower --max-context)
strata verify: capturing the 6-token window (68 MiB of VRAM free)
strata verify: capturing the 3-token window (64 MiB of VRAM free)
strata verify: capturing the 1-token window (58 MiB of VRAM free)
strata verify: capturing the 4-token window (52 MiB of VRAM free)
strata verify: capturing the 2-token window (10 MiB of VRAM free)
[engine exits; the client gets "the engine stopped unexpectedly (exit code 1)"]

Client side (strata_bench.py --long, cold):

FAILED: {'type': 'server_error', 'message': 'verify: instantiate: out of memory'}
FAILED: {'type': 'server_error', 'message': 'the engine stopped unexpectedly (exit code 1); the next request restarts it'}

engine_log_excerpt.txt