Layer split auto: extract the planner into layer_split.hpp, pin the whole arena outside Windows

CMakeLists.txt

@@ -650,6 +650,10 @@ if(STRATA_BUILD_TESTS OR STRATA_BUILD_CONVERSATION_TESTS)
   add_executable(bf16_bits_test src/kernels/bf16_bits_test.cpp)
   target_include_directories(bf16_bits_test PRIVATE include)
   add_test(NAME bf16_bits_test COMMAND bf16_bits_test)
+  # the layer split's preparation (layer_split.hpp, header-only, no library to link)
+  add_executable(layer_split_test src/program/layer_split_test.cpp)
+  target_include_directories(layer_split_test PRIVATE include)
+  add_test(NAME layer_split_test COMMAND layer_split_test)
 endif()
 
 add_library(strata_kernels_cpu STATIC src/kernels/cpu/expert.cpp src/kernels/cpu/pool.cpp

docs/MULTI_GPU.md

@@ -58,19 +58,22 @@ The engine flags behind it: `--layer-split K1[,K2..]|auto` and `--split-device D
 devices; default the next visible ones). `--layer-split K --split-device 0` runs both stages on one card sharing
 everything - the bit-exact check of the hand-off, not a speed mode.
 
-**auto** tries every placement (all of them for two or three cards; proportional to the free VRAM beyond that) and
-keeps the one whose caches would hold the most of the expert profile, hottest pairs weighted most; ties go to the
-placement that leaves the fullest card the most room. The startup log prints the choice:
+**auto** tries every placement (all of them for two or three cards; in proportion to speed beyond that) and keeps the
+one with the shortest predicted decode window: each card's layers at its per-layer time (from SMs x clock), plus the
+routed mass no cache would hold (each card's cache takes its own layers' profiled pairs, hottest first, in the VRAM
+its layer range's session leaves), plus one hand-off per later card (`include/strata/program/layer_split.hpp`). The
+startup log prints the choice:
 
 ```
-strata generate: layer split auto: K=19 - the caches hold 11767 of 12288 profiled pairs (fullest device 100%)
-strata serve: layer split: layers 0-18 (CUDA0), 19-47 (CUDA1), one hand-off per window
+strata generate: layer split auto: K=25 - predicted 39.9 ms per decode window; the caches hold 8852 of 24576 profiled pairs (~97.0% of the routed mass)
+strata serve: layer split: layers 0-24 (CUDA0), 25-47 (CUDA1), one hand-off per window
 ```
 
 ## What each card holds
 
-- **every card**: a copy of the dense weights (~3.4 GB for the Coder), its own session state (the KV cache of the full
-  context), its verify window and its prompt-path buffers, and an expert cache for its layers filled from the profile;
+- **every card**: a copy of the dense weights (~3.4 GB for the Coder), the session state of its own layers (their KV
+  cache for the full context), its verify window, and an expert cache for its layers filled from the profile. While a
+  prompt is read, the top slots of that cache hold the card's prompt buffers;
 - **the last card**: also the output head and the draft layer (~0.8 GB);
 - **host RAM**: the expert arena once, shared by all cards (the CPU pool computes whatever no card holds).
 
@@ -92,11 +95,11 @@ into the card that owns the layer.
     as it did without a split: its per-layer round trip costs more than the CPU pool needs for those experts.
 - `--mmap-experts` needs a canonical pack (`experts.bin`), with or without a split; a native (IQ) pack says so at
   start.
-- The prompt path has its own buffers on every card (1.5 GB each at the default 2048-token chunk; `--prefill 1024`
-  halves that) instead of borrowing cache slots as one card does. An explicit `--expert-cache` on the first card is
-  capped to leave room for them.
+- Every card borrows its prompt buffers from its own cache, and all cards read the same chunk: with `--prefill auto`,
+  the largest one every card can lend. Without an expert profile, or with `--no-prefill-borrow`, every card keeps its
+  own buffers for the session instead (2048-token chunks, ~1.5 GB a card).
 - On Windows only 8 GiB of the expert arena is pinned (more, mapped into two GPU contexts, leaves WDDM refusing
-  allocations); the PCIe share covers those layers.
+  allocations); the PCIe share covers those layers. Linux pins all of it, as with one card.
 - Every card needs compute capability 7.5 (RTX 20 or newer). The pre-sm_80 QSA scorer path is fp32 FMAs, so a
   Turing card runs the same kernels instead of the tensor-core prompt attention.
 
@@ -116,6 +119,18 @@ The Coder on an RTX 5080 + RTX 3090 (Ryzen 9 9950X3D), 32K context; details in
   expert, the per-layer GPU time decides.
 - **Correctness:** one GPU is byte-identical to 0.1.20, and the hand-off itself is bit-exact.
 
+Qwen3.8-Flash-Next IQ2_XS (33 GiB of experts, more than both cards hold together) on 2 x RTX 2080 Ti (11 GB, PCIe
+gen3 x16, Core i7-7700K), Linux, 32K context, `--spec 4` with the MTP drafter:
+
+| | Prompt 1.9K / 8K / 16K / 29K tok/s | Decode tok/s | Decode cache hits |
+|---|---|---|---|
+| one 2080 Ti | 466 / 695 / 840 / 836 | 36.7 | 66% |
+| 2 x 2080 Ti, auto (K=25) | 485-490 / 774-781 / 1,116-1,131 / 1,278-1,302 | 48.4-50.3 | 80-81% |
+| same, arena pinned only to 8 GiB | 442 / 696 / 921 / 999 | 44.7 | - |
+
+- **Decode +32-37%.** On this 4-core CPU, decode waits on the CPU pool; the second cache holds ~3,800 more experts.
+- **Prompts gain with length** (+4-5% at 1.9K, +53-56% at 29K): the more chunks, the more the two cards overlap.
+
 **Which cards and in what order:**
 - Put the fastest card first; auto gives it as many layers as its cache allows.
 - Leave out a much slower card when two already hold the model. An RTX 2080 Ti as a third card made the 5080 +

include/strata/program/layer_split.hpp

@@ -0,0 +1,135 @@
+// include/strata/program/layer_split.hpp - the layer split's arithmetic (docs/MULTI_GPU.md), apart from the devices,
+// so src/program/layer_split_test.cpp can check it.
+#pragma once
+
+#include <algorithm>
+#include <cmath>
+#include <cstddef>
+#include <cstdint>
+#include <functional>
+#include <utility>
+#include <vector>
+
+namespace strata::program::layer_split {
+
+/// The most of the host expert arena to pin (0 = all of it).  Under WDDM, pinning all of it into several contexts
+/// left the driver refusing later allocations, so there it stays at 8 GiB.
+inline uint64_t arena_pin_cap(bool several_contexts, bool wddm) {
+    return several_contexts && wddm ? (8ull << 30) : 0;
+}
+
+// --layer-split auto: the predicted time of one decode window, fitted on an RTX 5080 + RTX 3090
+// (bench/results/2026-09-29-layer-split).  Each card's layers cost layer_ms each, every later card a hand-off, and
+// the routed mass no cache holds miss_ms per unit, profile rank r weighing (r+1)^-mass_exp.  A cache takes its layers'
+// pairs hottest first, up to the first that does not fit (as the engine fills it), in the room its session leaves.
+
+/// A card's time per layer and window, from SMs x clock: 0.33 ms on an RTX 5080 (84 SMs, 2.62 GHz).
+inline double layer_ms_estimate(int sms, double ghz) {
+    const double speed = (double) sms * ghz;
+    return 0.33 * (84.0 * 2.617) / (speed > 1.0 ? speed : 1.0);
+}
+
+/// One card of a placement: the VRAM its expert cache and its session may take, and what a layer costs it per window.
+struct Card {
+    int64_t cap_bytes = 0;
+    double layer_ms = 0;
+};
+
+struct Costs {
+    double miss_ms = 190.0;    ///< a unit of routed mass no cache holds (STRATA_SPLIT_MISS_MS)
+    double handoff_ms = 1.0;   ///< one card handing a window to the next (~1.1 ms on 2 x RTX 2080 Ti, PCIe gen3)
+    double mass_exp = 1.2;     ///< the routed mass of profile rank r: (r+1)^-mass_exp
+};
+
+struct Placement {
+    std::vector<int64_t> at;   ///< the first layer of every later card (empty: one card)
+    double ms = 0;             ///< the predicted decode window
+    double held_mass = 0;      ///< the share of the routed mass the caches hold
+    int64_t held = 0;          ///< the profiled pairs they hold
+};
+
+/// The VRAM a card running layers [lb, le) needs for their session (the carve), which its expert cache cannot use.
+using RangeBytes = std::function<int64_t(int64_t lb, int64_t le)>;
+
+class Planner {
+public:
+    /// `profile`: the ranked (layer, expert) pairs, hottest first; `slot_bytes[l]`: the VRAM one expert of layer l
+    /// takes in a cache; `range_bytes`: what a card's layer range takes out of its `cap_bytes` (empty: nothing).
+    Planner(int64_t n_layers, std::vector<std::pair<int32_t, int32_t>> profile, std::vector<int64_t> slot_bytes, Costs c,
+            RangeBytes range_bytes = {})
+        : n_layers_(n_layers), profile_(std::move(profile)), slot_bytes_(std::move(slot_bytes)), costs_(c),
+          range_bytes_(std::move(range_bytes)), mass_(profile_.size()) {
+        for (size_t r = 0; r < profile_.size(); ++r) total_mass_ += (mass_[r] = std::pow((double) r + 1.0, -c.mass_exp));
+    }
+
+    /// The window time of `cards` running from the layers `at` on (at.size() == cards.size() - 1).
+    Placement predict(const std::vector<Card>& cards, const std::vector<int64_t>& at) const {
+        Placement p;
+        p.at = at;
+        const size_t ns = cards.size();
+        auto lb = [&](size_t i) { return i == 0 ? 0 : at[i - 1]; };
+        auto le = [&](size_t i) { return i + 1 < ns ? at[i] : n_layers_; };
+        std::vector<int64_t> room(ns), used(ns, 0);
+        for (size_t i = 0; i < ns; ++i) room[i] = cards[i].cap_bytes - (range_bytes_ ? range_bytes_(lb(i), le(i)) : 0);
+        std::vector<bool> full(ns, false);
+        double mass = 0;
+        for (size_t r = 0; r < profile_.size(); ++r) {
+            const int64_t l = profile_[r].first;
+            size_t st = 0;
+            while (st + 1 < ns && l >= at[st]) ++st;
+            if (full[st]) continue;
+            const int64_t b = slot_bytes_[(size_t) l];
+            if (used[st] + b > room[st]) { full[st] = true; continue; }
+            used[st] += b;
+            mass += mass_[r];
+            ++p.held;
+        }
+        p.held_mass = total_mass_ > 0 ? mass / total_mass_ : 1.0;
+        p.ms = costs_.miss_ms * (1.0 - p.held_mass) + costs_.handoff_ms * (double) (ns > 0 ? ns - 1 : 0);
+        for (size_t i = 0; i < ns; ++i) p.ms += (double) (le(i) - lb(i)) * cards[i].layer_ms;
+        return p;
+    }
+
+    /// The fastest placement: every one for two or three cards (a later card starts at layer 2 or later), the layers
+    /// in proportion to speed beyond.  Ties keep the earlier split point.
+    Placement best(const std::vector<Card>& cards) const {
+        const size_t ns = cards.size();
+        const int64_t L = n_layers_;
+        Placement best;
+        best.ms = 1e300;
+        std::vector<int64_t> at(ns > 0 ? ns - 1 : 0);
+        auto consider = [&]() {
+            Placement p = predict(cards, at);
+            if (p.ms < best.ms) best = std::move(p);
+        };
+        if (ns <= 1) {
+            consider();
+        } else if (ns == 2) {
+            for (int64_t k = 2; k < L; ++k) { at[0] = k; consider(); }
+        } else if (ns == 3) {
+            for (int64_t k1 = 2; k1 + 1 < L; ++k1)
+                for (int64_t k2 = k1 + 1; k2 < L; ++k2) { at[0] = k1; at[1] = k2; consider(); }
+        } else {
+            double total = 0, acc = 0;
+            for (const Card& c : cards) total += 1.0 / c.layer_ms;
+            for (size_t i = 0; i + 1 < ns; ++i) {
+                acc += 1.0 / cards[i].layer_ms;
+                const int64_t lo = i == 0 ? 2 : at[i - 1] + 1, hi = L - (int64_t) (ns - 1 - i);
+                at[i] = std::clamp<int64_t>((int64_t) std::llround(acc / total * (double) L), lo, hi);
+            }
+            consider();
+        }
+        return best;
+    }
+
+private:
+    int64_t n_layers_;
+    std::vector<std::pair<int32_t, int32_t>> profile_;
+    std::vector<int64_t> slot_bytes_;
+    Costs costs_;
+    RangeBytes range_bytes_;
+    std::vector<double> mass_;
+    double total_mass_ = 0;
+};
+
+}  // namespace strata::program::layer_split