A dual-GPU fork for 32 GB PCs (resident mode on a layer split, pipelined windows)

Hi Niko, thanks for Strata, great work. Really.

I have a 5080 + 4060 Ti with 32 GB of RAM, where setup recommends one card because the resident low-RAM mode has no layer split. I made it work on a split and then pipelined the two stages, so the 4060 Ti runs the next window while the 5080 verifies the current one. On my PC that took Swift 1.5 IQ2_XS from 29 tok/s (5080 alone) to 161tok/s on code (200+ peaks), with the same perplexity as 0.1.38 on a teacher-forced comparison.

It's a fork on top of 0.1.38: https://github.com/Hardin22/Strata-DualGPU
What changed and what each part measured: docs/DUAL_GPU.md

Some of it isn't specific to two GPUs, and I'd be glad to send PRs if you want any of it:

the hybrid-CPU pool default (P-cores plus half of the E-cores): on an i9-14900KF, 15 workers decoded much faster than all 23
copying an evicted expert back from the card that owns its layer (resident mode on a split; with one card it doesn't matter)
--vram-reserve-later-mib, a separate reserve for the cards that don't drive a display

No pressure either way. I'm going to post the fork on r/LocalLLaMA and wanted you to hear about it here first.
PS i do believe this can be pushed even further.

Field data from a Windows dual-5090 rig (engine 0.1.38, WDDM): decode is up, prefill is down. Sharing the measurements and what we found in the code, in case it helps the pipelined-split work.

Machine: 2x RTX 5090 — GPU0 Gen5 x16 (H2D probe 49.8 GB/s), GPU1 on a PCH Gen4 x4 slot (7.2 GB/s). qwen3.8-flash-next-iq3_s, 262K context, int8 KV, MTP+spec4, --layer-split 27 (CUDA0 layers 0-26, CUDA1 27-47). 3 rounds each, SHA256-paired requests.

prompt	one GPU	split K=27
4K prefill	4130 tok/s (TTFT 1.03 s)	2188 (1.90 s)
32K	6268 (5.33 s)	4555 (7.29 s)
80K	6189 (13.19 s)	5035 (16.13 s)
250K	5930 (42.7 s)	4828 (52.4 s)
decode geomean	~140 tok/s	+21.6% (code +39.9%)

What the code shows, and what we can rule out:

The link is not the bandwidth cause. The chunk hand-off is hcn_embd4 = 40,960 B/token (prefill.cpp:90,681): 6.5 us/token at our link speeds, 1-3% of prefill. Decode moves 51 KB/token through MAPPED pinned memory with kernel copies (verify.cpp:73,421-428,821-825) and pays nothing.
The 4K case is the clean control: both configs process one 4076-token chunk, and the split has better residency (93% of pairs vs 43%) yet is 1.89x slower per layer. With zero overlap the split should cost the same as one card; it costs nearly twice that, so each stage runs at ~half the single-card per-layer rate.
The prefill hand-off is synchronous: D2H on the compute stream + cudaStreamSynchronize + a next_run.get() barrier, two pinned buffers, depth one chunk (prefill.cpp:2600-2617). For a one-chunk prompt there is no overlap at all.
The split search has no link or prefill term: predict() = miss_ms*(1-held_mass) + layerslayer_ms with layer_ms from SMsGHz only (generate.cpp:2485-2491). The per-card PCIe probes exist but are unused by the search.
Chunk size is a trade, not a lever: auto:16384 on the split lent 5.85 GiB per stage and was worse (32K 8.82 s) than 4096 (7.26 s).
WDDM arena pin: startup says "at most 8 GiB of the expert arena is pinned" under multi-GPU WDDM; docs/MULTI_GPU.md ([Linux][multi-GPU] 8 GiB cudaHostRegister cap causes ~3x Q2_0 prefill slowdown #253) records that cap costing a 4090+3060 split two thirds of its prompt speed. STRATA_ARENA_PIN_GIB=0 and STRATA_SPLIT_OWN=1 are our next A/B rounds.

Asks: (1) async hand-off on the copy stream + a bounded multi-buffer chunk pipeline, with a small first chunk so short prompts pipeline at all; (2) a link/prefill term in the split search; (3) per-stage conversation snapshots so parking works on a split — today cross-session resume is 16.4 s TTFT with 0 reused tokens on the split vs 0.53 s with 80,448 reused on one card.

Also for #644: explicit --layer-split 23/25/27 all started and ran fine here on 0.1.38 with two 5090s — that crash did not reproduce on this rig.

Happy to run further A/B rounds against any branch.

setup: recommend P-cores + half the E-cores as CPU expert workers on …

setup: recommend the hybrid worker count only with more E-cores than …

Thanks for writing this up, and for the measurements in the fork. The first piece is in 0.1.39 as a setup recommendation. On a hybrid CPU, setup now writes --pool-workers into the run config: the performance cores minus the one the host loop uses, plus half of the efficiency cores (15 on an i9-14900K, 9 on an i5-13600K). It only does this when the CPU has more E-cores than P-cores: on an i7-13700KF (8P + 8E) a report here measured all 15 workers faster than 8, so CPUs like that keep the engine's own count. It's an ordinary line you can edit or remove, and a calibration (START-HERE --calibrate) that measured a better count overrides it. With all cores enabled, the engine now also places the workers on the P-cores first, so a smaller pool doesn't land on E-cores. CPUs whose cores are all the same get the same config as before. The other pieces (resident mode on a split, pipelined windows, --vram-reserve-later-mib) are still open.

Merge upstream Niko1221/Strata main (v0.1.39) into ac922

Merge Hardin22/Strata-DualGPU (v0.1.38-dual) onto 0.1.39 with the for…

Thanks! I ported the open pieces from the fork onto main as separate PRs. Each one has its A/B numbers and an exactness check:

Resident RAM mode with a layer split, --vram-reserve-later-mib (#642) #848: the resident RAM mode on a layer split, plus --vram-reserve-later-mib and the setup side, gated on a future engine version. On a 4060 Ti + 5080 with 32 GB of RAM it runs at ~70 tok/s, against 25-29 on the 5080 alone. The greedy output matches --mmap-experts on the same split.
Layer split: pipelined verify windows for one conversation (--pipeline-windows, opt-in) #859: pipelined verify windows for one conversation, opt-in. +16% mean and +30% on copy-heavy edits on top of Resident RAM mode with a layer split, --vram-reserve-later-mib (#642) #848. The greedy output is bit-identical to the serial loop with the same expert caches.
cpu experts: an AVX2 Q8_K activation quantizer (byte-identical to ggml's) #851: an AVX2 Q8_K activation quantizer, byte-identical to ggml's. +2.9% on the split.
cpu kernels: gathered decode on the cores where it is faster, AVX-VNNI rows, IQ3_S one-token kernel #863: the gathered decode on the cores where it pays, AVX-VNNI rows, and IQ3_S's one-token kernel. IQ3_S at one token goes from 0.43 to 0.26 ms per expert on a P-core. Never slower in the bench.

One data point for #639: on the same split (Swift 1.5 IQ3_XXS, 160K, layer split 20, pipelined), --trim-stage-weights grew the caches from 4752 to 5627 slots on the 4060 Ti and from 3948 to 4676 on the 5080. Decode went 90.0 → 96.1 tok/s over two interleaved pairs. No problems in those runs.

An overview of the whole two-GPU stack, now that all the pieces are up. They are separate PRs so each can be reviewed on its own, but they are meant to land together.

What it adds up to. RTX 4060 Ti 16 GB + RTX 5080 16 GB, i9-14900KF, 32 GB of RAM, Swift 1.5 IQ3_XXS at 160K, decode tok/s:

	tok/s
main today, the 5080 alone (setup's choice with 32 GB of RAM)	~29
main today, both cards (`--mmap-experts`, after the file cache warms)	~64
#848 resident RAM mode on the split	~70
+ #859 pipelined windows	~84
+ #876 / #905 async tier beside the pipeline, #904, `--trim-stage-weights`	~118
+ #910	~131

The last two rows use a fine-tuned draft layer and --spec-min-p 0.65; the rows above use the stock one. Every PR has its own same-settings A/B, an exactness check and a stress run.

Two reports from other hardware so far:

adambenhassen ran Resident RAM mode with a layer split, --vram-reserve-later-mib (#642) #848 Layer split: pipelined verify windows for one conversation (--pipeline-windows, opt-in) #859 Asynchronous adaptive expert tier for the resident RAM mode (--adapt-async 1, opt-in) #876 Pipelined windows + async adaptive tier together (stacked on #859 and #876) #905 Two-GPU decode: bounded attention merge (bitwise, opt-in) and pipelined PLE read-ahead / early chain (stacked on #905) #910 Verify window: PDL on sm_90+, graph branches, batched Q4_0 KV append and PLE post-ops (bit-identical) #904 cpu kernels: gathered decode on the cores where it is faster, AVX-VNNI rows, IQ3_S one-token kernel #863 cpu experts: an AVX2 Q8_K activation quantizer (byte-identical to ggml's) #851 on 2× RTX 3090 (Linux). He gets 128-140 tok/s decode, and a 99K prompt reads at 3,052 tok/s against 2,025 on v0.1.39. He also found a stall in Pipelined windows + async adaptive tier together (stacked on #859 and #876) #905; see below.
CC-David-CC built perf: Q8 resident adaptation (+24.5–28.2% rotation gain on RTX PRO 6000) #898 on Asynchronous adaptive expert tier for the resident RAM mode (--adapt-async 1, opt-in) #876: Q8 on an RTX PRO 6000, +24-28%.

Merge order:

The chain: Resident RAM mode with a layer split, --vram-reserve-later-mib (#642) #848 → Layer split: pipelined verify windows for one conversation (--pipeline-windows, opt-in) #859 → Asynchronous adaptive expert tier for the resident RAM mode (--adapt-async 1, opt-in) #876 → Pipelined windows + async adaptive tier together (stacked on #859 and #876) #905 → Two-GPU decode: bounded attention merge (bitwise, opt-in) and pipelined PLE read-ahead / early chain (stacked on #905) #910. Pipelined windows + async adaptive tier together (stacked on #859 and #876) #905 and Two-GPU decode: bounded attention merge (bitwise, opt-in) and pipelined PLE read-ahead / early chain (stacked on #905) #910 are stacked and carry the earlier commits.
Resident RAM mode with a layer split, --vram-reserve-later-mib (#642) #848 and Layer split: pipelined verify windows for one conversation (--pipeline-windows, opt-in) #859 have one textual conflict in the stage_room reserve. Keep both terms: base_reserve + pf + (later ? kWindowMib : kPipeSnapMib) + kPipeWindowMib + drafter.
Independent of the chain: Verify window: PDL on sm_90+, graph branches, batched Q4_0 KV append and PLE post-ops (bit-identical) #904 (bit-identical dense-path launches), cpu experts: an AVX2 Q8_K activation quantizer (byte-identical to ggml's) #851 and cpu kernels: gathered decode on the cores where it is faster, AVX-VNNI rows, IQ3_S one-token kernel #863 (CPU kernels).

Known issue: --pipeline-windows 2 --adapt-async 1 (#905) can stall on stage 1 under long sequential requests (2× RTX 3090). The pipeline alone and the async tier alone are clean there. I'm on it. Until it's fixed, the safe combination is --adapt-async 0, or STRATA_PIPELINE_ADAPT_ASYNC=0.

Everything new is opt-in except #848's resident split (it was refused before) and #910's bitwise attention merge. Happy to rework any of it however you prefer.

setup: the low-RAM mode's resident variant on several GPUs (#642)

The resident RAM mode on a layer split (PR #848) ships in 0.1.40. Please confirm on 0.1.40 with two cards.

@aswin-dot-R

Disclosure: written by an AI assistant on behalf of the owner of the machine below; every number is from runs on that machine.

A two-stage V100 data point for --pipeline-windows, since @aswin-dot-R noted in #859 that the 4-stage V100 rig there cannot take this path. Ours has two stages, so the flag applies.

Rig: 2× Tesla V100-PCIE-32GB (sm_70, no NVLink, PCIe Gen3 ×16), 188 GiB RAM, 2× Xeon E5-2680 v4, openEuler 24.03, CUDA 12.8, driver 580.
Model/config: Swift 1.5 IQ3_XXS, --layer-split 24, experts 100 % resident in VRAM on both cards, --kv int8, --max-context 524288, one conversation, MTP --spec 8. We are not a resident-RAM case, so the #848 half of your request is untested here.

Method: same machine, same config, same serve/ version — only the engine binary differs. The production service is stopped for every run and the cards are otherwise empty. 5 scenarios × 2 rounds, 320 output tokens each. The variants are run in blocks (baseline first), not interleaved pair-by-pair; as a drift check the baseline engine's mean reproduced to within 0.1 tok/s across two sessions ~5 hours apart on the same box.

A) --pipeline-windows 2 on 0.1.39 (the patch from #859 vs stock 0.1.39):

scenario	0.1.39	+ `--pipeline-windows 2`	Δ
list	143.0	159.1	+11.3 %
json	134.0	134.1	+0.1 %
prose	65.7	72.3	+10.0 %
qa	66.9	73.4	+9.7 %
code	113.8	118.1	+3.8 %
mean	104.7	111.4	+6.4 %

Lower than the +16 % in the PR body, which fits your note that it was not measured on NVIDIA. Cost we see: 272 MiB kept out of each card's expert cache (the doc says 160 MiB for two cards). Greedy text is not bit-identical to the serial loop, as documented.

B) 0.1.40 with the official flag, same config, same pair:

scenario	0.1.39 (+#859)	0.1.40 (+#800)	Δ
list	159.2	165.1	+3.7 %
json	134.2	149.0	+10.8 %
prose	72.2	72.7	+0.7 %
qa	73.5	75.3	+2.4 %
code	118.3	125.1	+5.7 %
mean	111.5	117.4	+5.3 %

So 0.1.40 is a further +5.3 % on top of the pipeline on this pair. Three deterministic spot-checks (17*23, a counting question, and a strict "output only ABCDE") passed in every run, including the 0.1.40-only build.

C) PR #800 on Volta, measured with your own parity tool

Built with -DSTRATA_PARITY_PROMPT_ATTN=ON, then qsa_prompt_attn_parity 32768 2048 5:

case	clean 0.1.40 (new vs FP64)	+ #800	the FP32 kernel it replaces
int8, ctx 32768	5.08e-06	2.33e-06	2.37e-06
fp16, ctx 32768	5.29e-06	3.57e-06	1.86e-06
int8, ctx 1500	5.55e-06	2.24e-06	1.79e-06
int8, ctx 2100	5.54e-06	2.74e-06	2.03e-06
`FAILURES`	0	0	—

The tensor-core chain's error lands at the FP32 kernel's own level, as the PR claims. Cost is small: int8 at 32K goes 13.9 → 14.5 ms per chunk on an idle machine, i.e. still 1.76× faster than the FP32 path, and end-to-end decode is unchanged (rows in table B are the same 117.4 with and without #800).

Two notes for docs/NVIDIA_V100.md, which says "A layer split over two V100 was not measured with this kernel":

On this pair the kernel is exercised through a two-card split, and the parity test passes on both cards.
The same page says a split "adds expert-cache room, not prompt speed" because the cards work in turn. That is true of the serial loop, but with --pipeline-windows 2 a single conversation does get faster here — +6.4 % on this pair — so the sentence is worth qualifying once the flag is on.

Happy to run the 10+ interleaved pairs you asked for, or a --batch soak on an all-resident stage (#776), if that is still useful — the box is idle between jobs.

setup: recommend P-cores + half the E-cores as CPU expert workers on …

setup: recommend the hybrid worker count only with more E-cores than …

setup: the low-RAM mode's resident variant on several GPUs (#642)

setup: recommend P-cores + half the E-cores as CPU expert workers on …

setup: recommend the hybrid worker count only with more E-cores than …

setup: the low-RAM mode's resident variant on several GPUs (Niko1221#642

setup: recommend P-cores + half the E-cores as CPU expert workers on …

setup: recommend the hybrid worker count only with more E-cores than …

Written with an AI assistant; every number is from runs on my machine.

Confirming the resident RAM mode on a layer split on 0.1.40.1, two different Ampere cards, Linux.

Rig: RTX 3090 24 GB (layers 0-37) + RTX 3060 12 GB (38-47), Xeon E5-2697 v4 (18 cores, AVX2, no AVX-512), 192 GB DDR4, PCIe probe 12.4 GB/s. Fedora 44, driver 595.80, CUDA 13.3, engine built from source at 82f46a8 (sm_86, GCC 15 as host compiler).
Config: Unsloth UD-IQ4_XS, --layer-split 38, STRATA_STAGE_TRIM=1, --max-context 199999 --kv fp16 --kv-resident 32768, --spec 4, vision on, control vector loaded (off). Log: resident RAM mode: 39.37 GiB of experts in RAM (page-locked), GPU caches 7126 + 3407 slots.

30-minute soak, one client, serial: short prompts (code/prose/SQL at temperature 0.7, plus a greedy one), a ~28K prompt every ~2 min and a 92-127K prompt every ~5 min (past the 32K resident KV window).

156 requests, 0 errors, no !!!! or repetition loops, no engine restart.
Short decode median 71.3 tok/s; first quarter 70.9, last quarter 71.6, so no drift.
28K prompts: 1,060-1,620 tok/s read, 47-57 tok/s decode. 92-127K prompts: 1,700-1,820 tok/s read, 43-60 tok/s decode.

Against 0.1.39 (6f32ec0), same config, medians of 3, decode tok/s: code 69.2 -> 70.1, SQL 69.8 -> 70.2, prose 56.3 -> 52.7. All within run-to-run noise on this box.

One question. At start the engine says the adaptive swaps exchange experts with the GPU cache "(no file reads)". resident RAM: ... 0 blob reads from the file held for the first ~46K exchanges. After that, file reads appeared, starting with a 3.7K-token prompt, and reached 7,394 blobs by the end of the soak (61,041 exchanged). With this much RAM they come from the page cache, so speed did not suffer. Is that expected, for example from the prompt path borrowing CUDA0 cache slots (borrows 1544 CUDA0 cache slots)?

@m4rk4n

@m4rk4n yes, that's expected, and it happens on a 4060 Ti + 5080 split too (42,189 blob reads in a 30-minute soak, details in the 0.1.40 soak comment on #848).

The prompt path borrows cache slots (borrows N CUDA0 cache slots). When it gives them back, the slots are refilled with the experts the GPU held before the prompt. Those are not in the RAM complement, which holds only the experts the GPU caches don't, so they are read from the model files. With your RAM they come from the page cache, as you saw; here the engine counted 41 GB read in 6 minutes while the disk itself read 6.6 GB. --prefill 2048 roughly halves them. Keeping the lent slots' experts in RAM while the prompt runs would remove them; I haven't done that yet.

2-GPU confirmation on 0.1.40 (2x RTX 3090, Linux): the resident RAM mode on the split ran a 30-minute soak clean (346 requests, no degenerate or empty answers, no restart), and --pipeline-windows 2 added +20% decode on top of it. Details and two observations: #1280

Shipped in 0.1.40: the resident split (#848) and the pipeline windows (#859). Thank you for the 161 tok/s code numbers, and BaiQichen2026 for the 2x V100 two-stage --pipeline-windows data, which we keep as bench data. Closing this request.

setup: the low-RAM mode's resident variant on several GPUs (Niko1221#642

setup: 層の分割では、省 RAM モードの判断に後ろのカードの VRAM も数える（upstream 43c8cfa、Niko122…

setup: UD-Q4_K_XL / UD-IQ4_XS keep their RAM budget on a layer split (#…

The setup side is in #1564 in v0.1.42: with an engine that keeps the resident RAM copy on a layer split, setup keeps the RAM budget on several GPUs at any RAM.

setup: 高効率コアの多いハイブリッド CPU では、CPU のワーカーを性能コアと高効率コアの半分にする（upstream ad0df80

setup: UD-Q4_K_XL / UD-IQ4_XS keep their RAM budget on a layer split (N…

Navigation Menu

Repository navigation

A dual-GPU fork for 32 GB PCs (resident mode on a layer split, pipelined windows) #642

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

A dual-GPU fork for 32 GB PCs (resident mode on a layer split, pipelined windows) #642

Description

Activity

wangzc1997 commented on Oct 3, 2026

Niko1221 commented on Oct 4, 2026

Hardin22 commented on Oct 5, 2026

Hardin22 commented on Oct 5, 2026

Niko1221 commented on Oct 6, 2026

BaiQichen2026 commented on Oct 6, 2026

m4rk4n commented on Oct 6, 2026

Hardin22 commented on Oct 6, 2026

andrej-reimer commented on Oct 6, 2026

Niko1221 commented on Oct 7, 2026

Niko1221 commented on Oct 10, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions