Repository navigation
[0.1.43] One GPU, default flags: "lending the prompt path its slots failed: ... do not fit" when a request's chunk is smaller than the auto chunk, engine exits (code 1) #2090
Description
Setup. RTX 4070 Ti 12 GB (sm_89), driver 617.14, Windows 11 Pro 26200 (WDDM, the card also drives the
display), Ryzen 9 5900X, 64 GB DDR4. Release engine 0.1.43 (CUDA 13.0 zip, BUILD.json source release), server at
7695859a. Model Qwen3.8-Flash-Next IQ3_S native pack, setup's config plus --calibrate:
--pack iq3_s --native ...IQ3_S-00001-of-00002.gguf --ple-gguf ...IQ3_S-00002-of-00002.gguf --expert-profile expert-profile.bin
--expert-cache auto --prefill auto --spec 4 --mtp rt --max-context 131072 --kv int8 --vision --vram-reserve-mib 700
--pcie-frac 0.35 --spec-min-p 0.70 --adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97
("vision": {"gpu": false}, one request at a time, no --batch.)
What happens. tools/strata_bench.py --long (the new BENCH.bat) failed in 2 of 9 runs. In both, a fresh
prompt whose chunk is smaller than the auto chunk chosen at start fails, the engine exits with code 1, and the
next request restarts it:
[strata] the engine reported an error: lending the prompt path its slots failed: prefill: device buffers for a chunk of 4096 tokens do not fit (34 of 12281 MiB free at the failure; 2589 MiB taken, the failing buffer wanted 40 MiB)
[strata] the engine stopped unexpectedly (exit code 1). ...
Free VRAM is not the limit: in the second case 436 MiB were free and the buffer wanted 30 MiB
(... chunk of 3072 tokens do not fit (436 of 12281 MiB free at the failure; 2207 MiB taken, the failing buffer wanted 30 MiB)). Reading Prefill::init (src/prefill/prefill.cpp, the carve(T, &o) failure), the take that
fails is used + bytes > cap inside the borrowed region, i.e. the lent slots are smaller than the relayout for the
request's own chunk needs.
The start-up sizing decides it (every engine start of the day, from strata-iq3_s.log; the start-up free VRAM
was the same 3.82 GiB each time, the expert cache settles differently after "shrinking"):
| Start | Expert cache | Prompt chunk auto | Prompt path borrows | VRAM free, all loaded | Result |
|---|---|---|---|---|---|
| bench 1 | 1515 slots (no shrink line) | 5120, 199-slot ring | 1316 slots (2.53 GiB) | 180 MiB, "LOW" | 4K prompt (chunk 4096) fails, twice (after the restart the same sizing) |
| bench 1, 3rd start | 1418 slots | 4864 | 1266 slots (2.44 GiB) | 282 MiB | rest of the suite passes |
--vram-reserve-mib 1000 |
1292 slots | 4096 | 1117 slots (2.16 GiB) | 582 MiB | agent turn of 3,080 new tokens (chunk 3072) fails, twice |
| 7 other runs | 1414-1447 slots | 4864 | 1266 slots | 280-282 MiB | all pass, including chunks of 4096 and 3072 |
--expert-cache 1515 |
1628 slots | 5888 | 1463 slots (2.81 GiB) | 0 MiB, "LOW" | passes (prompts 1230 tok/s instead of 1830) |
So the failing pairs are auto 5120 -> request chunk 4096 and auto 4096 -> request chunk 3072, while auto 4864 ->
4096 / 3072 works. A larger --vram-reserve-mib does not help (it only moves the auto chunk). Disabling
--adapt-* did not change it either (0 failures in 2 runs without, 0 in 3 runs with).
Expected. A request chunk below the auto chunk fits in the region lent for the auto chunk (or the engine
falls back without exiting).
Repro. Not deterministic for me. Forcing the chunk does not trigger it, because an explicit chunk is checked at
start: with --prefill 5120 the log says prompt chunk 5120 -> 2560 tokens so its buffers fit in every expert cache, and fresh prompts of ~4,000 / 3,000 / 2,000 tokens (2 each) all pass. It only happened with
--prefill auto, and only when the auto chunk came out other than 4864 (5120 after a cache with no shrink step,
4096 with a 1000 MiB reserve). Maybe the same "fits in every expert cache" check is not applied to the auto chunk
against the request chunk's relayout?
Not run: other models, a second card, Linux.
Benchmark reports (scrubbed by the tool) can be attached on request.
Activity
ischencheng commentedon Oct 12, 2026
Opened #2131 for this. A request at or below 0.1.39's chunk reads with 0.1.39's bigger ring, but the loan was sized for the auto chunk with the smaller byte-budget ring. I got the same error on every start on an L4 (IQ3_XXS) at a few cache sizes, and none with the PR.
Tests: STRATA_TEST_PINNED_SHARE sets the pinned share the prompt ring…
Prompt loan: a request's smaller chunk can need more slots than the a…
Prompt loan: say when a chunk's loan is larger than the auto chunk's (#…
Niko1221 commentedon Oct 12, 2026
Thank you for the table: it is what made this findable. Your numbers point at the cause, and I think I see it in the code, but I have to be honest that I could not reproduce the failure on my boxes.
What I believe happens.
--prefill autopicks the chunk and the ring together: a chunk above the scan's "small" limit gets the byte-budget ring (your 199 slots at 5120), a chunk at or below it gets the older, larger ring (384 slots). The region lent for the prompt path is sized for the auto chunk. When a request's own chunk (4096, 3072) is smaller,lendre-lays it out, but the region start wasmax(auto start, start this chunk needs), which can only ever be the smaller of the two regions. If the smaller chunk's layout (with the bigger ring) needs more than the auto chunk's, the layout runs past the region: exactly yourused + bytes > capwith 34 or 436 MiB free. That the failing pairs are your auto 5120 -> 4096 and 4096 -> 3072 while 4864 -> 4096 passes fits a layout that is close to the region size and flips with the ring.Fix on a branch (not released): the loan now starts where the chunk itself needs it, down to the 128-slot floor; if a chunk cannot be lent even then it is halved for this request and the log says so, instead of exiting. A log line also says when a chunk's loan is larger than the auto chunk's.
Reproduction: with a test hook forcing the pinned-share rule on a 12 GB RTX 3060 (IQ3_XXS, reserve 700-2700 MiB, auto chunks 3328-7168, 4K/3K prompts after a 6K one) every request passes on the old and the new engine, and the new line never fired, so there I never reached your margin. Identity gate and exact prompt-path checks (4K and 20K residual dumps) are bit-identical to before. If you can run the branch build, or send the log of a failing start with
STRATA_TRACE=1(thelent N slots for M tokenslines), I can confirm. Until then a bigger--vram-reserve-mibdoes not help (as you found);--prefill 4096fixed makes the chunk and the lend match.Remote stage: follow 0.1.43.1 on both PCs
Merge upstream main (a0116b2, engine 0.1.43.1): server and bench fixes
Layer split + --batch: the default pipeline wins over --batch-mtp (ST…
Remote stage: a bounded, cancellable reconnect; a worker's chunk; a r…
feat(sycl): port upstream v0.1.43.1 fixes (Niko1221#2090 prompt loan, N…
bdwilton commentedon Oct 13, 2026
Different failure mode of this same change on gfx12 + low-RAM --resident-experts: the larger auto chunk doubles the host-RAM borrow and crashes via swap thrash (not the VRAM fit failure above). Adding it here since the maintainer noted he couldn't reproduce on his boxes — this is a different resource and hardware class, same root change.
Hardware. AMD Radeon AI PRO R9700 (gfx1201), 32 GB VRAM, 30 GB system RAM, Linux. Swift 1.5 IQ3_XXS, --kv int8, --max-context 131072, --resident-experts, --vram-reserve-mib 3072, --spec 4. Engine 0.1.43.1 (built from a0116b2).
What happens. Large input prompts (38K-47K tokens) hang during the prompt read, swap ~20 GB to disk, rack up 1.5M+ major page faults, and get killed by the watchdog / OOM. The same prompts were stable on 0.1.41 and 0.1.42.
The borrow doubled at 0.1.43 on gfx12. docs/DETAILS.md notes a bare --prefill auto may take 16384-token chunks on gfx12. With the #2090 prompt-loan change the prompt-path borrow on this card went:
Engine --prefill resolves to Prompt chunk Prompt-path borrow
0.1.41 / 0.1.42 auto -> 8192 8192 2,068 slots (3.37 GiB)
0.1.43 / 0.1.43.1 auto -> 16384 16384 3,907 slots (6.38 GiB)
That extra ~3 GiB of host RAM during every prompt read, on top of the 21 GB resident expert lock with only 3.5 GB available, is what tips large prompt reads into swap thrash. The stall report shows memory: 5440 MiB resident, 20538 MiB in swap, 3390 MiB RAM available; 1528580 major page faults; the server stdout sits at reading the prompt: 38,706 of 38,711 tokens, 159 s so far and then the engine is killed.
Workaround. --prefill 8192 (or STRATA_GFX12_CHUNK=0 / STRATA_GFX12_DEFAULTS=0) restores the 8192 chunk and the 2,075-slot / 3.37 GiB borrow. With it the same 38-47K prompts run stable: ~95 tok/s decode, 97-98% expert cache hit, zero SSD reads. This matches the maintainer's note above that a fixed --prefill N makes the chunk and the lend match -- but here the binding resource is host RAM, not VRAM, so the fix is to shrink the chunk, not just to match it.
Suggestion. The 16384 chunk is presumably a prompt-speed win on PCs with RAM to spare, so I'm not suggesting reverting the gfx12 default. But --resident-experts's RAM admission (STRATA_RESIDENT_HEADROOM_GIB) doesn't account for the larger gfx12 prompt-path borrow, so a machine that fit at 8192 no longer fits at 16384. Either the resident budget should subtract the gfx12 borrow, or setup/docs should flag --prefill 8192 for low-RAM resident machines.
Not tested. Whether 16384 is faster than 8192 on this card on a PC with enough RAM (only have 30 GB); other gfx12 cards; CUDA/SYCL; --prefill auto:8192 (I used a fixed --prefill 8192); whether --kv-resident changes the picture (not in my config).
K8nsch commentedon Oct 13, 2026
Confirmed fixed in v0.1.43.1 on the RTX 4070 Ti (IQ3_S, 64 GB, Windows 11). 6 cold-start runs of
strata_bench.py --longwithSTRATA_TRACE=1: 0 engine exits (0.1.43: 2 of 9). One run picked auto chunk 5120 – the failing case – and loggedthe loan for a chunk of 4096 tokens (1358 slots) is larger than the auto chunk's (1316 slots): starting it earlier (#2090), then continued normally. Throughput unchanged vs 0.1.43. Thanks for the quick fix!chore(merge): merge upstream v0.1.43.1; drop the Claude Code Fable in…
merge upstream/main (v0.1.43.1 line) into feature/sampling-presets-11…
gfx12: the 16384 auto prompt chunk stays at 8192 when the RAM availab…
36 remaining items
Elastic slots: idle slots trimmed when quiet; STRATA_ELASTIC_SLOTS_GI…
Elastic slots: idle slots trimmed when quiet; STRATA_ELASTIC_SLOTS_GI…
Elastic slots: idle slots trimmed when quiet; STRATA_ELASTIC_SLOTS_GI…
Elastic slots: idle slots trimmed when quiet; STRATA_ELASTIC_SLOTS_GI…
Elastic slots: idle slots trimmed when quiet; STRATA_ELASTIC_SLOTS_GI…
Elastic slots: idle slots trimmed when quiet; STRATA_ELASTIC_SLOTS_GI…
Elastic slots: idle slots trimmed when quiet; STRATA_ELASTIC_SLOTS_GI…
Elastic slots: idle slots trimmed when quiet; STRATA_ELASTIC_SLOTS_GI…
Elastic slots: idle slots trimmed when quiet; STRATA_ELASTIC_SLOTS_GI…
Remote stage: follow 0.1.43.1 on both PCs
Remote stage: a bounded, cancellable reconnect; a worker's chunk; a r…
STRATA_BRANCH_CHECKPOINT_ALIGN round 2: review fixes (slot restores, …
STRATA_BRANCH_CHECKPOINT_ALIGN round 3: mark only what this read save…
gfx12: K-quant experts read the prompt on MMQ by default; bare --pref…
NVIDIA: a bare --prefill auto takes a 32768-token chunk ceiling on ca…