[cobra91]

Problem and result

A layer split's last stage also runs the drafter, and the drafter's draft head and its logits are allocated when it binds - after the expert caches are sized. One GPU already keeps their room out of its cache (#199); a split's last stage does not, so on a card without a display (the server reserves only 300 MiB there) they had to come out of that reserve, and on two 16 GB cards the engine stopped with "mtp: the draft head does not fit". With this change the last stage's cache leaves room for them, as on one GPU.

Related discussion

From Hardin22's Strata-DualGPU (5a20ec58, part of what #642 described). The same two-16-GB-cards scenario is in our benchmark thread #4470.

Approach

Seven lines: in the per-stage cache sizing loop, subtract mtp.bind_bytes(st.head.row_bytes(), n_vocab) from the last stage's room. Main already declares last_st and uses bind_bytes elsewhere; this is the same subtraction #199 applies on one GPU, applied to the split's last card. No new flags, no behaviour change beyond the placement.

Validation

Engine built from main at 46c152a (MSVC 19.51 / Ninja / CUDA 13.4, arch 89;120). Host: serve/server.py with --pipeline-windows 2, --layer-split auto; rates read from the engine log. Rig: RTX 5080 + RTX 4060 Ti 16 GB (chipset slot), Ryzen 7 9800X3D, 32 GB RAM, Windows 11; Swift 1.5 IQ2_XS, 64K, q4_0 KV, stock MTP draft.

Passed:

• Engine starts, sizes the last stage's cache, and reports 431 MiB of VRAM free on CUDA1 (no --vram-reserve-later-mib); a full bench suite (short + long greedy generations) completes, health 200 throughout.
• Decode 53.0 tok/s sustained on a 2.8K-token answer (main without --pipeline-windows 2 on the same rig: 34-39 tok/s).

Failed:

• none observed.

Not run:

• HIP, SYCL and Metal builds (no hardware).
• The "mtp: the draft head does not fit" error itself: our 4060 Ti drives a display so its reserve is 1800 MiB, enough for the head - the fix is measured by its effect on the cache sizing and by the fork's report of the error on two display-less 16 GB cards.

Limits

The room subtracted is the draft head and its logits only (bind_bytes); the second verifier of --pipeline-windows and its graphs are not in this subtraction (the fork carries a separate change for that, which I can bring if this is welcome).

[cobra91]

Backend note per the guide: the change is host-side arithmetic in the stage cache sizing (no CUDA/HIP-specific code), but I could only build and run the CUDA engine here (MSVC/Ninja/CUDA 13.4, arch 89;120 - validation lists it). If a HIP check is required I will mark this draft, or a maintainer's HIP run would close the gap.