[ischencheng]

Fixes #2090.

With --prefill auto the serve scan sizes the prompt path's loan for the chunk it picked, with the byte-budget ring. A request whose chunk is at most 0.1.39's pick (small in set_ring_budget(..., small)) reads with 0.1.39's ring instead. On a pack with bigger blobs than Q2_0's that ring has more slots (384 against 227 on IQ3_XXS), so the short request can need more of the cache than the loan holds. lend clamps it to the loan's first slot, relayout runs out of room, and the engine exits with "lending the prompt path its slots failed". That is why it only showed with --prefill auto and only at some cache sizes in the report.

The loan now has room for both: max(part_slots(chunk), part_slots(small)). The same helper is used where the loan is sized again (the step-down after a failed init, and relend under --vram-elastic). lend is unchanged, so each request still lends only the slots its own chunk needs. A request that fitted before lends the same slots. Where 0.1.39's chunk needs no more slots than the chosen one, the loan does not change at all.

Measured on Modal, one L4 24 GB (sm_89, Linux), the official IQ3_XXS pack, upstream main 7695859, --prefill auto --max-context 131072 --kv int8. I set --expert-cache N by hand to land on the cache sizes where this happens. Each start gets four fresh prompts whose first part rounds up to a 2048 / 3072 / 4096 / 6144-token chunk.

--expert-cache	auto chunk, ring	the short chunk needs	loan main -> this PR	main	this PR
905, 910, 915	2816, 227 slots	2048: 1059-1060 slots	1023-1024 -> 1059-1060	engine exits, 3 of 3 starts	0 of 3
1088, 1094, 1100	3840, 227	3072: 1288-1289	1252-1254 -> 1288-1289	3 of 3	0 of 3
1284, 1290	4864, 227	4096: 1517	1481-1482 -> 1517	2 of 2	0 of 2
900, 1130, 1296	2816 / 4096 / 5120, 227	fits	1024 / 1309 / 1539, unchanged	0	0

Two more runs of some of the failing rows gave the same result (main 5 of 5 starts, this PR 0 of 5). Main's error is the one in the report, with 13 GB of VRAM free:

lending the prompt path its slots failed: prefill: device buffers for a chunk of 2048 tokens do not fit (13033 of 22563 MiB free at the failure; 1719 MiB taken, the failing buffer wanted 20 MiB)

Replies (temperature 0, thinking off, 64 tokens) to the 2048 / 3072 / 4096 prompts are the same on main and this PR at every cache size where main answered them. The 6144 one changes between starts of main itself on this setup (3 different replies in 4 starts at --expert-cache 900); this PR's reply was one of main's.

src/program/generate.cpp also builds for HIP: compiled it in a ROCm 7.0 build for gfx1201 (that one file, no GPU), with the same 98 warnings as main.

Not tested: Windows, the reporter's IQ3_S pack and 4070 Ti, a HIP run, layer splits, --vram-elastic. sycl/src/program/generate.cpp has the same scan and is not changed here. The resident RAM mode sizes its RAM copy's lend region from plan_lend, which still counts the chosen chunk only, so there a short request's extra slots are read from the model file during that prompt.

[qiuzan2001]

For triage: the failure this PR targets (#2090, "lending the prompt path its slots failed" on one GPU) was fixed in main another way in v0.1.43.1 by 4468e0e and af6affb (the loan starts where the request's chunk needs it, and a chunk that cannot be lent is halved), and the #2090 reporter confirmed it on v0.1.43.1 (RTX 4070 Ti, 6 cold starts with 0 engine exits, against 2 of 9 on 0.1.43). This branch still applies to main 296b59d, so whether its max(chunk, small) sizing adds anything on top of that fix is the open question.

[Niko1221]

Thanks for the table, this is a real crash. It is a fix and not a speed change, so it is outside this round; we would take it as a bug fix.

[ischencheng]

@Niko1221 thanks. Your af6affb in 0.1.43.1 already covers this: the loan starts where the request's chunk needs it, so these configs no longer exit, and the reporter confirmed it. On top of that this branch only makes the lent region 35-36 slots bigger up front in the configs from my table, so I don't think it adds anything. Fine to close it.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.