[giostrives]

On 2 x RTX 2080 Ti the layer split was barely faster than one card for decode and about 30% slower for prompts. This branch fixes how the split is set up. On this setup the split is now the faster configuration for both decode and long prompts.

Changes

1. layer split: pin the whole expert arena outside Windows (bb321e5). The 8 GiB pin cap is a WDDM workaround and now applies on Windows only. With the cap on Linux, only the first ~10 layers' misses could take the PCIe share, and the prompt path staged ~80% of the streamed experts through host copies.
2. layer split: every card lends its prompt buffers from its own cache (b578d5e). Each card used to reserve 2048-token buffers for the whole session (~1.5 GiB a card), and prompts ran in 2048-token chunks. Now every card lends its buffers from its expert cache, and all cards read the same chunk, the largest every card can lend.
3. layer split auto: the planner in layer_split.hpp, tested (251fb73). This is layer_split::Planner, with unit tests in src/program/layer_split_test.cpp. It adds a hand-off term per later card and makes the routing-curve exponent a parameter. The commit also fixes the later card's reserve, which counted the drafter twice: it was 1 GiB and is now 128 MiB.
4. bench: the layer split on 2 x RTX 2080 Ti (6bc110a). Harness, raw results and tables in bench/results/2026-09-30-layer-split-2080ti/, plus a section in docs/MULTI_GPU.md. Most of the added lines are here.

Related

The first commit is the same fix as #209 (cudaFuncGetName guarded for CUDA < 12.3). If #209 lands first, I'll drop the commit and rebase. Otherwise this MR carries it, and #209 can be closed as superseded.

Results

Qwen3.8-Flash-Next IQ2_XS, 2 x RTX 2080 Ti 11 GB, i7-7700K, 32K context, int8 KV, --spec 4. Same harness for every row, and every needle was found.

Setup	Decode tok/s	Hits	Prompt 2.7K tok/s	Prompt 8K tok/s	Prompt 16K tok/s	Prompt 29K tok/s
1 GPU	33.9	64%	618	712	795	801
2 GPUs, before (K=31)	35.8	69%	391	497	537	535
2 GPUs, after (auto K=25)	45.3-47.3	80-82%	609-627	826-829	1,120-1,126	1,290

• Decode on this machine is bound by the CPU pool, so the split pays through fewer CPU misses (hit rate 64% → 80-82%).
• The K sweep agrees with auto's pick: K=18 41.4, K=25 45-47, K=31 45.2, K=36 42.2 tok/s.
• On 0.1.27: one card 35.7 tok/s decode, the split 47.3. At 29K, prompts read 832 vs 1,304 tok/s.

Testing

• layer_split_test covers the planner. In the published tree it builds with g++ -std=c++20 -Iinclude src/program/layer_split_test.cpp && ./a.out.
• The full -DSTRATA_BUILD_TESTS=ON build does not work in the published tree, because some sources are not shipped. This is not new to this branch.

[giostrives]

This PR partially overlaps with #216, @gopinath87607 .

What I did extra here is some more tests, removing the memory pin for Linux and adding a planner.

I'm happy to rebase and adapt these changes (removing some of them) after the other PR.

[Niko1221]

Thanks @giostrives, great measurements. #216 covers the per-card loans and the reserve, so please open a new PR on top of it once it lands, with only the full expert-arena pin on Linux and the planner in layer_split.hpp plus its test. Leave out the cudaFuncGetName commit (already in 0.1.28) and the bench JSON (a short summary in the PR or docs is enough). We're closing PRs that wait on changes so the list stays manageable; this is not a no. Please open a new PR when it's ready and link this one.

[Niko1221]

#216 is in 0.1.30. As discussed, a new PR with just the Linux full-arena pin, the planner (layer_split.hpp) and its test is welcome on top of it. One data point for it: on Linux a split still pins at most 8 GiB of the expert arena (generate.cpp, the Windows limit), so on our AMD two-card run only 7 GiB was pinned while a single GPU pins the whole arena.