Repository navigation
Layer split auto: extract the planner into layer_split.hpp, pin the whole arena outside Windows - #294
giostrives wants to merge 3 commits into
Conversation
A split capped the arena's pinned registration at 8 GiB on every platform. The cap is a WDDM workaround (pinning all of it into two contexts left the driver refusing later allocations on an RTX 5080 + RTX 3090), but on Linux it cost the split the PCIe share of every missed expert past the first 8 GiB of layers, and made its prompt path stage most experts through host copies. One GPU pins all 33 GiB on the same machine. The cap now applies under WDDM only (layer_split::arena_pin_cap, tested). WSL2 is a Linux build on the WDDM driver, but its driver pins only ~1 GiB of host memory in all: the whole registration is refused and the slices stop at that budget, with or without the cap - as with one GPU there today. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-off term The placement's cost model moves out of main() into layer_split::Planner: the same fill (each card's layers' profiled pairs, hottest first, stopping at the first that does not fit), the same search (every placement for two or three cards, in proportion to speed beyond), the same per-layer time from SMs x clock, and the session carve priced per placement (a card's room less session_bytes of its layer range, passed in as range_bytes). For the same inputs it picks the same split points as the inline search did. Each later card now adds a hand-off per window (STRATA_SPLIT_HANDOFF_MS, 1 ms), so the predicted window counts what a card costs beyond its layers; the routing curve's exponent is a parameter (STRATA_SPLIT_MASS_EXP), since it differs between models. 16 checks in layer_split_test, 14 of them on a toy model and three of those for the carve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rompt buffers); 2 x RTX 2080 Ti measured What each card holds and the prompt-buffer limit still described the state before Niko1221#216: every card now holds only its own layers' session and borrows its prompt buffers from its own cache. Adds the IQ2_XS numbers on two RTX 2080 Ti cards, split against one card and against the 8 GiB arena pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I need a web setup |
|
Seed Phrase |
|
I don't know what that comment is, I believe some kind of spam. |
|
Thanks! The pin part shipped in 0.1.31 (#253: whole arena outside WDDM, STRATA_ARENA_PIN_GIB). Please rebase the planner (layer_split.hpp + its test) without arena_pin_cap, with the new hand-off term defaulting to 0 (or show the chosen K is unchanged on the documented rigs). Closing until then. |
|
Okay, rebased and applied the required changes:
Checked on 2 × RTX 2080 Ti (IQ2_XS) with STRATA_SPLIT_HANDOFF_MS at 0 and at 1: both choose K=25 (layers 0-24 / 25-47). Decode is 52.5 tok/s at 0 and 52.8 at 1. Prompts at 0, in tok/s: 604 (2K), 921 (8K), 1257 (16K), 1602 (30K). Needle found at every length. layer_split_test passes (16 checks). 2 x RTX 2080 Ti, IQ2_XS,
On my machine, 1 ms is no worse, and at most slightly better (decode +0.6%, 16K prompt +9%; single runs, so within noise). |
|
@Niko1221 I cannot reopen the PR, let me know if I need to open a new one. Thank you! |
This PR is scavenged from #223, limiting the changes to the memory pinning and multi-gpu planner.
--layer-split autocost model out ofgenerate.cppinto a header-only, unit-testedPlannerChanges
include/strata/program/layer_split.hpp). The placement arithmetic is now inlayer_split::Planner,Card,CostsandPlacement, apart from the devices.generate.cpponly gathers per-card VRAM and SM/clock data and callsplanner.best(cards).STRATA_SPLIT_HANDOFF_MS, default 1.0 ms; about 1.1 ms measured on 2 x RTX 2080 Ti over PCIe gen3). The startup log'spredicted ... ms per decode windowincludes it.STRATA_SPLIT_HANDOFF_MSandSTRATA_SPLIT_MASS_EXPjoinSTRATA_SPLIT_MISS_MS. Defaults are unchanged from the previous hard-coded values (190 ms, exponent 1.2).arena_pin_cap(several_contexts, wddm)caps the pin at 8 GiB only under WDDM with several contexts. Linux now pins the whole arena, as with one card.layer_split_test(registered in CMake) covers the pin cap, per-layer time, the fill-stops-at-first-misfit rule, hand-off terms, placement search for 2, 3 and 4 cards, and the carve pricing.docs/MULTI_GPU.mddescribes the new model, the per-layer sessions and borrowed prompt buffers, and adds a 2 x RTX 2080 Ti measurement.Measurements
Qwen3.8-Flash-Next IQ2_XS (33 GiB of experts, more than both cards hold), 2 x RTX 2080 Ti (11 GB, PCIe gen3 x16, i7-7700K), Linux, 32K context,
--spec 4. Auto chose K=25.Decode is +32-37%. Prompts gain with length: +4-5% at 1.9K, +53-56% at 29K.