layer split: with explicit split points each GPU loads only its own layers' dense weights by JeanP00l · Pull Request #639 · Niko1221/Strata

JeanP00l · 2026-10-03T13:55:49Z

What

With a layer split every GPU keeps a full copy of the dense weights (~3.4 GB for the Coder), although each stage only reads its own layers. This PR lets each GPU load only its layers' dense weights when the split points are explicit (--layer-split 27), and gives the freed VRAM to the expert cache.

The canonical arena of each GPU skips the blk.N.* tensors outside its layer range (read from the pack's index.txt). PLE tensors stay on every GPU.
NativeDense::set_layer_range limits the native projections the next load uploads in the same way.
--layer-split auto and single-GPU runs are unchanged. STRATA_STAGE_TRIM=0 keeps the full copies.
One log line per GPU says which layers it loads, and a later stage logs its free VRAM before its cache is sized.

Measured

2x AMD Instinct MI50 16 GB (gfx906, see #638), Coder IQ1_M, 128K context, --layer-split 27:

	full copies	own layers only
experts held in VRAM	8,819 of 12,288	10,626 of 12,288
VRAM hit rate in decode	95%	98%
decode, 17K-token agent prompt	39.2 tok/s	41.7 tok/s

The trimmed tensors are never read by the GPU that skips them.

Status

Re-run on 2x MI50 with this branch on current main, results in the comment below.

🤖 Generated with Claude Code

…ayers' dense weights Without this every stage of a layer split keeps a full copy of the dense weights (~3.4 GB for the Coder), although it only ever reads its own layers. That VRAM goes to the expert cache instead. - the canonical arena of each GPU skips the `blk.N.*` tensors of layers outside its range (PLE tensors stay everywhere), read from the pack's index.txt; - NativeDense::set_layer_range limits the native projections the next `load` uploads the same way; - only with explicit split points (`--layer-split 27`); `auto` and a single GPU are unchanged; STRATA_STAGE_TRIM=0 keeps the full copies; - one log line per GPU says what it loads, and a later stage logs its free VRAM before the cache is sized. Measured on 2x AMD Instinct MI50 16 GB (gfx906 port, Coder IQ1_M, 128K context, --layer-split 27): experts held in VRAM 8,819 -> 10,626 of 12,288, VRAM hit rate 95% -> 98%, decode on a 17K-token agent prompt 39.2 -> 41.7 tok/s. The trimmed tensors are never read by that GPU.

JeanP00l · 2026-10-03T19:25:21Z

Re-run on the cards with this branch, merged with #638 and #640 (current main 99f3dbd).

Setup: 2x MI50 16 GB, Coder IQ1_M, 131,072-token context, --layer-split 27, --kv int8 --kv-resident 32768, --prefill 4096, --pcie-frac 0, STRATA_ARENA_MMAP=1. Each run is one engine start, a greedy 1,500-token code request and a 4-turn agent conversation (17K-token Claude Code prompt + three follow-ups).

	`STRATA_STAGE_TRIM=0` (full copies)	default (own layers only)
expert cache slots, GPU 0 + GPU 1	6,204 + 4,900 = 11,104	6,912 + 5,376 = 12,288 (all)
decode expert-cache hit rate	99.1%	100.0%
decode, greedy 1,500 tokens	56.5 tok/s	61.0 tok/s (61.0 in a second run)
first turn (17K-token prompt)	47.3 s	46.8 s

Text. The default run's greedy text is the same in both runs. It differs from STRATA_STAGE_TRIM=0's, as expected: which experts sit in VRAM changes, so some go GPU vs CPU and round differently. The trimmed tensors themselves are never read by the GPU that skips them.

Marking ready.

A split loaded the dense weights of all 48 layers on every card, though each card runs only its own layers. Now every card loads them as before, the split is decided, and each card reloads only its own layers' weights before the sessions and the expert caches are made. The VRAM this frees goes to the expert caches. The automatic split counts it when it places the boundary, so this also works with --layer-split auto (upstream's Niko1221#559 and Niko1221#639 do it for explicit split points). --no-trim-stage-weights keeps the old behaviour. The PLE's six layer-1 weights are the only pointers into the old copy taken before the split is known; they are wired again after the reload. The router copy for the file tier's lookahead reads each layer's router from the card that runs it. On a 4060 Ti + 5080 at K=20 this frees 1.9 and 1.3 GiB. Same engine, without and with trimming (code / prose at context 0, code / prose at 32K, 32K prompt): IQ2_XS: 162 / 109, 149 / 100, 1,899 -> 172 / 111, 157 / 104, 2,201 tok/s IQ3_XXS: 141 / 95, 98 / 71, 1,006 -> 144 / 97, 131 / 92, 1,978 tok/s With IQ3_XXS at 160K the experts no card held did not all fit in RAM before, and about 2,400 were read from the SSD per benchmark run; now none are. Teacher-forced over 5,335 tokens, IQ3_XXS with and without trimming picks the same top token at 93.5% of the positions (perplexity 6.95 and 7.03).

…s (PR #639 follow-up) The file tier's routing-aware prefetch (RouterLookahead) copies all layers' ffn_gate_inp.weight from CUDA0's arena. With CUDA0 trimmed to its own layers it found them missing and turned itself off ("the routers are not BF16 in the arena"). CUDA0 now keeps the routers (~2.5 MiB a layer) when --mmap-experts is on; the other stages and the runs without --mmap-experts trim as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…639 follow-up) The trim was measured on 2x MI50 (gfx906) only; nobody has run it on NVIDIA multi-GPU, where CUDA-only paths could still read another stage's dense weights on a card that no longer holds them. The HIP builds trim with explicit split points as the PR does (STRATA_STAGE_TRIM=0 keeps the full copies); the CUDA build keeps the full copies unless STRATA_STAGE_TRIM=1. --layer-split auto and one GPU are unchanged everywhere. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…A_MMAP (PR #640), make_profile --reorder (#587) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

The trim was on by default in the HIP builds. It now waits, as on CUDA, for the PR author's re-run of the release branch on his cards; the default flips after that. --layer-split auto and one GPU were never affected. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…r split, parking with a split Conflicts with 0.1.39: --trim-stage-weights is now the same switch as STRATA_STAGE_TRIM=1 (PR #639's implementation, already in 0.1.39); the #533 VRAM command is taken between batch windows only when no slot is decoding. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Niko1221 · 2026-10-04T12:33:52Z

Thanks. This is in the 0.1.39 branch with two changes:

Routers on CUDA0 with --mmap-experts: the file tier's routing-aware prefetch (RouterLookahead) copies every layer's ffn_gate_inp.weight from CUDA0's arena. On a trimmed CUDA0 it found them missing and turned itself off. CUDA0 now keeps all the routers (~2.5 MiB a layer) when --mmap-experts is on.
Opt-in for now, on HIP and CUDA alike (STRATA_STAGE_TRIM=1): we'd like your re-run of this branch on the MI50s before it becomes the default on HIP, and neither of us has NVIDIA multi-GPU (CUDA-only paths could still read another stage's dense weights). --layer-split auto and one GPU are unchanged everywhere.

It builds for CUDA (Windows and Linux) and HIP (the Windows RDNA engine), and the single-GPU output is unchanged. We can't run the trim itself here (one GPU). Please post the re-run of this branch on the MI50s here when you have it; we'll keep the PR open until then and close it with the release.

Niko1221 · 2026-10-04T12:54:34Z

Merged for 0.1.39 as STRATA_STAGE_TRIM=1: with explicit --layer-split points each GPU loads only its own layers' dense weights. Opt-in for now on AMD and NVIDIA.

Thanks @JeanP00l for the MI50 measurements.

…k's carries Hardin22's two-GPU work (pipelined verify windows, fewer host waits, the asynchronous expert tier, the dense-path launches, resident RAM mode on a split, the later cards' VRAM reserve) on upstream 0.1.39. Where both did the same thing, upstream's is kept: the shared expert on its own stream (cfd3b72), the fused SwiGLU quantization, the row zeroing on the GPU, the per-stage dense weights' loader filter (Niko1221#639), the P-core order (Niko1221#642). Hardin22's late PLE rows serve every stage (upstream's only the all-resident graph); its split host steps (prepare / launch / serve / finish) carry upstream's trace (Niko1221#649) and Niko1221#578's helper hooks. Batch windows (Niko1221#465 / Niko1221#559) keep upstream's paths: no per-row hand-off, side branches or batched PLE inside them. 2c1365e (updates and engines from Hardin22's repository) is left out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ibozulic · 2026-10-04T18:02:00Z

Hi, thanks for including stage trimming in v0.1.39.

Since NVIDIA multi-GPU testing was mentioned as a gap, I wanted to share a data point from my setup:

6 × RTX 3070 8 GB, all connected through PCIe x1 risers
Ubuntu 26.04.1 LTS, CUDA 13.1, locally built for sm_86
Strata v0.1.39, commit 6f32ec0
ISTA-DASLab Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S
Explicit layer split 8,16,24,32,40
131072 context, INT8 KV, --kv-resident 20480
--expert-cache auto, --prefill 3072, STRATA_PREFILL_RING=8
MTP enabled, CPU vision enabled

I compared the same explicit split with trimming disabled and with STRATA_STAGE_TRIM=1. No other configuration settings were changed.

With trimming enabled:

Total resident experts increased from 8422 to 17770.
The last GPU's cache increased from 714 to 2099 slots.
The requested prefill chunk of 3072 became usable; without trimming it was reduced to 1536.

Median results from three repetitions per case, trim off → trim on:

2001-token prompt: prefill 41.7 → 83.6 tok/s; time to first visible token 48.00 → 23.96 s
6231-token prompt: prefill 89.8 → 167.9 tok/s; time to first visible token 69.47 → 37.17 s
12482-token prompt: prefill 114.9 → 241.7 tok/s; time to first visible token 108.70 → 51.73 s
512-token Croatian story: decode 34.5 → 50.7 tok/s

All measured requests had zero prefix reuse. Tests used temperature 0, seed 42 and thinking disabled, with a fixed warmup excluded from the results. Configurations were tested sequentially, not as randomized interleaved A/B runs.

This measures the combined benefit of the larger auto-sized expert cache and the now-usable prefill chunk, not an isolated kernel speedup. I am not claiming bitwise output parity.

Needle retrieval, prefix reuse, tool-call generation, image input and SSE checks passed. An additional 24973-token needle test, beyond the KV-resident window, also returned all three identifiers correctly. Full 128K inference was not tested.

I have saved the exact requests, responses, configurations, timings and logs and can provide a small reproducible sample if useful.

Thanks again — this made a substantial practical difference on my small-VRAM, slow-PCIe setup.

JeanP00l · 2026-10-04T19:38:13Z

Thanks for the careful A/B report and for separating the combined benefit from an isolated kernel speedup. The larger resident expert cache and the now-usable 3,072-token prefill chunk are both part of the measured gain, and your report makes that clear.

The six-card RTX 3070 result adds useful multi-GPU coverage, especially with the PCIe x1 risers. The explicit limits are helpful too: no full 128K inference, image-input test, or claim of bitwise parity.

I also ran a short deterministic inference on the updated gfx906 merge candidate against the existing gfx906 build; the generated code matched (SHA-256 114cf38b24ff3651c135481a4791a9f30597e7c71229d432ce5e2dd3d3956f63). That is only a smoke request, not a substitute for your workload or a full-context comparison.

--trim-stage-weights (PR #559) / STRATA_STAGE_TRIM=1 (PR #639) gives a stage only its own layers' dense weights, but it refuses to run unless --layer-split is written out: `auto` chooses the split points 590 lines after the stages' weights are loaded, so by the time the split is known there is nothing left to trim. Those weights now load after the search. The search still prices each card's arena from the pack index (`WeightTable::pool_bytes` reads the index and allocates nothing), and takes it off the card's measured free VRAM, so the number it sees is the number it has always seen - the split it picks is the split it has always picked. This buys cache, not a different placement. The expert caches are sized later, from live free VRAM, so they collect the freed bytes. On the 4-way IQ3_S rig, `--layer-split auto` (which picks K=10,19,34): CUDA1 3650 -> 4608 expert slots, CUDA2 1241 -> 2485, CUDA3 533 -> 1772 prompt chunk 768 -> 6144 tokens prefill 469.7 -> 1929.7 tok/s on the same 107,082-token prompt CUDA0 still trims only with an explicit --layer-split. Its arena is read out to build the prompt path's PLE tables, 590 lines before the split exists, so moving it is the larger half of this change. It is also the cheap half to leave: measured, it costs 1% (1929.7 against 1950.8 for an explicit `10,19,34 --trim-stage-weights`, where CUDA0's cache is 4598 slots instead of 3017). Co-Authored-By: Claude Code <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

A four-GPU run picked its split from a formula that reads only `layer_ms` (SMs x clock) and shares the layers in proportion to speed; the free VRAM that decides how many experts a stage can actually hold never entered it, so the search could hand the most layers to the card with the least room and never notice. All C(L-3,3) placements are now scored - 15,180 at 48 layers - and 2,068 of them cost nothing extra, because a stage's fill depends only on its own layer range and its own free VRAM: `predict` already hands each ranked pair to the one stage that owns its layer and stops each stage at its first pair that does not fit, so a range is walked once and memoised. The cost model's mass curve changes with it. `mass[r]` was (r+1)^-1.2, fitted to the 5080+3090 sweep - still right on that rig (99.35% predicted at 20,000 held, 99.4% measured) and wrong on IQ3_S, where it claims the top 5,805 pairs hold 94.9% of the routed mass and the rig measures 69.2% over four boots and 14.9M decode lookups. No exponent reconciles them, because the curve is a property of the model's routing and the profile carries only a ranking, with no counts. The mass is now the increment of H(f) = 1 - (1 - f)^b over the held fraction, whose prefixes sum to exactly H(N/total); b = 3 (STRATA_SPLIT_COVER_B overrides it) is chosen to leave the validated regime where it was - at f = 0.8138 it gives the old fit's 99.35% - while at f = 0.59 it gives 93.2% against 92.2% measured. This is the placement search from #584, carried on v0.1.39 and without the weight carve it rode in on: v0.1.39 ships its own carve (#559 / #639), and the search only runs under `--layer-split auto`, where nothing is carved - so `cap[i]` is already net of the dense weights and a stage's room is what `predict` already subtracts, its session. Measured on the 4-way IQ3_S rig (2x RTX 3060 + 2x RTX 5060), same config and same flags in both arms, the binary the only difference: the four-way split moves from K=10,19,34 to K=14,31,41, the prompt chunk the starved cards allow grows 768 -> 1280 tokens, prefill goes 425 -> 597 tok/s (+40%) and decode 44 -> 51 tok/s (+16%), with the decode expert-cache hit 74.7% -> 83.3%. Two boots per arm, and the split is deterministic. The curve and the search are one commit, so that measures the pair, not the search alone. Co-Authored-By: Claude Code <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…s (PR #639 follow-up) The file tier's routing-aware prefetch (RouterLookahead) copies all layers' ffn_gate_inp.weight from CUDA0's arena. With CUDA0 trimmed to its own layers it found them missing and turned itself off ("the routers are not BF16 in the arena"). CUDA0 now keeps the routers (~2.5 MiB a layer) when --mmap-experts is on; the other stages and the runs without --mmap-experts trim as before.

…639 follow-up) The trim was measured on 2x MI50 (gfx906) only; nobody has run it on NVIDIA multi-GPU, where CUDA-only paths could still read another stage's dense weights on a card that no longer holds them. The HIP builds trim with explicit split points as the PR does (STRATA_STAGE_TRIM=0 keeps the full copies); the CUDA build keeps the full copies unless STRATA_STAGE_TRIM=1. --layer-split auto and one GPU are unchanged everywhere.

…A_MMAP (PR #640), make_profile --reorder (#587)

The trim was on by default in the HIP builds. It now waits, as on CUDA, for the PR author's re-run of the release branch on his cards; the default flips after that. --layer-split auto and one GPU were never affected.

…r split, parking with a split Conflicts with 0.1.39: --trim-stage-weights is now the same switch as STRATA_STAGE_TRIM=1 (PR #639's implementation, already in 0.1.39); the #533 VRAM command is taken between batch windows only when no slot is decoding.

--trim-stage-weights (PR #559) / STRATA_STAGE_TRIM=1 (PR #639) gives a stage only its own layers' dense weights, but it refuses to run unless --layer-split is written out: `auto` chooses the split points 590 lines after the stages' weights are loaded, so by the time the split is known there is nothing left to trim. Those weights now load after the search. The search still prices each card's arena from the pack index (`WeightTable::pool_bytes` reads the index and allocates nothing), and takes it off the card's measured free VRAM, so the number it sees is the number it has always seen - the split it picks is the split it has always picked. This buys cache, not a different placement. The expert caches are sized later, from live free VRAM, so they collect the freed bytes. On the 4-way IQ3_S rig, `--layer-split auto` (which picks K=10,19,34): CUDA1 3650 -> 4608 expert slots, CUDA2 1241 -> 2485, CUDA3 533 -> 1772 prompt chunk 768 -> 6144 tokens prefill 469.7 -> 1929.7 tok/s on the same 107,082-token prompt CUDA0 still trims only with an explicit --layer-split. Its arena is read out to build the prompt path's PLE tables, 590 lines before the split exists, so moving it is the larger half of this change. It is also the cheap half to leave: measured, it costs 1% (1929.7 against 1950.8 for an explicit `10,19,34 --trim-stage-weights`, where CUDA0's cache is 4598 slots instead of 3017).

A four-GPU run picked its split from a formula that reads only `layer_ms` (SMs x clock) and shares the layers in proportion to speed; the free VRAM that decides how many experts a stage can actually hold never entered it, so the search could hand the most layers to the card with the least room and never notice. All C(L-3,3) placements are now scored - 15,180 at 48 layers - and 2,068 of them cost nothing extra, because a stage's fill depends only on its own layer range and its own free VRAM: `predict` already hands each ranked pair to the one stage that owns its layer and stops each stage at its first pair that does not fit, so a range is walked once and memoised. The cost model's mass curve changes with it. `mass[r]` was (r+1)^-1.2, fitted to the 5080+3090 sweep - still right on that rig (99.35% predicted at 20,000 held, 99.4% measured) and wrong on IQ3_S, where it claims the top 5,805 pairs hold 94.9% of the routed mass and the rig measures 69.2% over four boots and 14.9M decode lookups. No exponent reconciles them, because the curve is a property of the model's routing and the profile carries only a ranking, with no counts. The mass is now the increment of H(f) = 1 - (1 - f)^b over the held fraction, whose prefixes sum to exactly H(N/total); b = 3 (STRATA_SPLIT_COVER_B overrides it) is chosen to leave the validated regime where it was - at f = 0.8138 it gives the old fit's 99.35% - while at f = 0.59 it gives 93.2% against 92.2% measured. This is the placement search from #584, carried on v0.1.39 and without the weight carve it rode in on: v0.1.39 ships its own carve (#559 / #639), and the search only runs under `--layer-split auto`, where nothing is carved - so `cap[i]` is already net of the dense weights and a stage's room is what `predict` already subtracts, its session. Measured on the 4-way IQ3_S rig (2x RTX 3060 + 2x RTX 5060), same config and same flags in both arms, the binary the only difference: the four-way split moves from K=10,19,34 to K=14,31,41, the prompt chunk the starved cards allow grows 768 -> 1280 tokens, prefill goes 425 -> 597 tok/s (+40%) and decode 44 -> 51 tok/s (+16%), with the decode expert-cache hit 74.7% -> 83.3%. Two boots per arm, and the split is deterministic. The curve and the search are one commit, so that measures the pair, not the search alone.

…s (PR Niko1221#639 follow-up) The file tier's routing-aware prefetch (RouterLookahead) copies all layers' ffn_gate_inp.weight from CUDA0's arena. With CUDA0 trimmed to its own layers it found them missing and turned itself off ("the routers are not BF16 in the arena"). CUDA0 now keeps the routers (~2.5 MiB a layer) when --mmap-experts is on; the other stages and the runs without --mmap-experts trim as before.

…iko1221#639 follow-up) The trim was measured on 2x MI50 (gfx906) only; nobody has run it on NVIDIA multi-GPU, where CUDA-only paths could still read another stage's dense weights on a card that no longer holds them. The HIP builds trim with explicit split points as the PR does (STRATA_STAGE_TRIM=0 keeps the full copies); the CUDA build keeps the full copies unless STRATA_STAGE_TRIM=1. --layer-split auto and one GPU are unchanged everywhere.

…ATA_ARENA_MMAP (PR Niko1221#640), make_profile --reorder (Niko1221#587)

…#639) The trim was on by default in the HIP builds. It now waits, as on CUDA, for the PR author's re-run of the release branch on his cards; the default flips after that. --layer-split auto and one GPU were never affected.

…ned layer split, parking with a split Conflicts with 0.1.39: --trim-stage-weights is now the same switch as STRATA_STAGE_TRIM=1 (PR Niko1221#639's implementation, already in 0.1.39); the Niko1221#533 VRAM command is taken between batch windows only when no slot is decoding.

--trim-stage-weights (PR Niko1221#559) / STRATA_STAGE_TRIM=1 (PR Niko1221#639) gives a stage only its own layers' dense weights, but it refuses to run unless --layer-split is written out: `auto` chooses the split points 590 lines after the stages' weights are loaded, so by the time the split is known there is nothing left to trim. Those weights now load after the search. The search still prices each card's arena from the pack index (`WeightTable::pool_bytes` reads the index and allocates nothing), and takes it off the card's measured free VRAM, so the number it sees is the number it has always seen - the split it picks is the split it has always picked. This buys cache, not a different placement. The expert caches are sized later, from live free VRAM, so they collect the freed bytes. On the 4-way IQ3_S rig, `--layer-split auto` (which picks K=10,19,34): CUDA1 3650 -> 4608 expert slots, CUDA2 1241 -> 2485, CUDA3 533 -> 1772 prompt chunk 768 -> 6144 tokens prefill 469.7 -> 1929.7 tok/s on the same 107,082-token prompt CUDA0 still trims only with an explicit --layer-split. Its arena is read out to build the prompt path's PLE tables, 590 lines before the split exists, so moving it is the larger half of this change. It is also the cheap half to leave: measured, it costs 1% (1929.7 against 1950.8 for an explicit `10,19,34 --trim-stage-weights`, where CUDA0's cache is 4598 slots instead of 3017).

A four-GPU run picked its split from a formula that reads only `layer_ms` (SMs x clock) and shares the layers in proportion to speed; the free VRAM that decides how many experts a stage can actually hold never entered it, so the search could hand the most layers to the card with the least room and never notice. All C(L-3,3) placements are now scored - 15,180 at 48 layers - and 2,068 of them cost nothing extra, because a stage's fill depends only on its own layer range and its own free VRAM: `predict` already hands each ranked pair to the one stage that owns its layer and stops each stage at its first pair that does not fit, so a range is walked once and memoised. The cost model's mass curve changes with it. `mass[r]` was (r+1)^-1.2, fitted to the 5080+3090 sweep - still right on that rig (99.35% predicted at 20,000 held, 99.4% measured) and wrong on IQ3_S, where it claims the top 5,805 pairs hold 94.9% of the routed mass and the rig measures 69.2% over four boots and 14.9M decode lookups. No exponent reconciles them, because the curve is a property of the model's routing and the profile carries only a ranking, with no counts. The mass is now the increment of H(f) = 1 - (1 - f)^b over the held fraction, whose prefixes sum to exactly H(N/total); b = 3 (STRATA_SPLIT_COVER_B overrides it) is chosen to leave the validated regime where it was - at f = 0.8138 it gives the old fit's 99.35% - while at f = 0.59 it gives 93.2% against 92.2% measured. This is the placement search from Niko1221#584, carried on v0.1.39 and without the weight carve it rode in on: v0.1.39 ships its own carve (Niko1221#559 / Niko1221#639), and the search only runs under `--layer-split auto`, where nothing is carved - so `cap[i]` is already net of the dense weights and a stage's room is what `predict` already subtracts, its session. Measured on the 4-way IQ3_S rig (2x RTX 3060 + 2x RTX 5060), same config and same flags in both arms, the binary the only difference: the four-way split moves from K=10,19,34 to K=14,31,41, the prompt chunk the starved cards allow grows 768 -> 1280 tokens, prefill goes 425 -> 597 tok/s (+40%) and decode 44 -> 51 tok/s (+16%), with the decode expert-cache hit 74.7% -> 83.3%. Two boots per arm, and the split is deterministic. The curve and the search are one commit, so that measures the pair, not the search alone.

…s (PR Niko1221#639 follow-up) The file tier's routing-aware prefetch (RouterLookahead) copies all layers' ffn_gate_inp.weight from CUDA0's arena. With CUDA0 trimmed to its own layers it found them missing and turned itself off ("the routers are not BF16 in the arena"). CUDA0 now keeps the routers (~2.5 MiB a layer) when --mmap-experts is on; the other stages and the runs without --mmap-experts trim as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…iko1221#639 follow-up) The trim was measured on 2x MI50 (gfx906) only; nobody has run it on NVIDIA multi-GPU, where CUDA-only paths could still read another stage's dense weights on a card that no longer holds them. The HIP builds trim with explicit split points as the PR does (STRATA_STAGE_TRIM=0 keeps the full copies); the CUDA build keeps the full copies unless STRATA_STAGE_TRIM=1. --layer-split auto and one GPU are unchanged everywhere. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ATA_ARENA_MMAP (PR Niko1221#640), make_profile --reorder (Niko1221#587) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…#639) The trim was on by default in the HIP builds. It now waits, as on CUDA, for the PR author's re-run of the release branch on his cards; the default flips after that. --layer-split auto and one GPU were never affected. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ned layer split, parking with a split Conflicts with 0.1.39: --trim-stage-weights is now the same switch as STRATA_STAGE_TRIM=1 (PR Niko1221#639's implementation, already in 0.1.39); the Niko1221#533 VRAM command is taken between batch windows only when no slot is decoding. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

--trim-stage-weights (PR Niko1221#559) / STRATA_STAGE_TRIM=1 (PR Niko1221#639) gives a stage only its own layers' dense weights, but it refuses to run unless --layer-split is written out: `auto` chooses the split points 590 lines after the stages' weights are loaded, so by the time the split is known there is nothing left to trim. Those weights now load after the search. The search still prices each card's arena from the pack index (`WeightTable::pool_bytes` reads the index and allocates nothing), and takes it off the card's measured free VRAM, so the number it sees is the number it has always seen - the split it picks is the split it has always picked. This buys cache, not a different placement. The expert caches are sized later, from live free VRAM, so they collect the freed bytes. On the 4-way IQ3_S rig, `--layer-split auto` (which picks K=10,19,34): CUDA1 3650 -> 4608 expert slots, CUDA2 1241 -> 2485, CUDA3 533 -> 1772 prompt chunk 768 -> 6144 tokens prefill 469.7 -> 1929.7 tok/s on the same 107,082-token prompt CUDA0 still trims only with an explicit --layer-split. Its arena is read out to build the prompt path's PLE tables, 590 lines before the split exists, so moving it is the larger half of this change. It is also the cheap half to leave: measured, it costs 1% (1929.7 against 1950.8 for an explicit `10,19,34 --trim-stage-weights`, where CUDA0's cache is 4598 slots instead of 3017).

A four-GPU run picked its split from a formula that reads only `layer_ms` (SMs x clock) and shares the layers in proportion to speed; the free VRAM that decides how many experts a stage can actually hold never entered it, so the search could hand the most layers to the card with the least room and never notice. All C(L-3,3) placements are now scored - 15,180 at 48 layers - and 2,068 of them cost nothing extra, because a stage's fill depends only on its own layer range and its own free VRAM: `predict` already hands each ranked pair to the one stage that owns its layer and stops each stage at its first pair that does not fit, so a range is walked once and memoised. The cost model's mass curve changes with it. `mass[r]` was (r+1)^-1.2, fitted to the 5080+3090 sweep - still right on that rig (99.35% predicted at 20,000 held, 99.4% measured) and wrong on IQ3_S, where it claims the top 5,805 pairs hold 94.9% of the routed mass and the rig measures 69.2% over four boots and 14.9M decode lookups. No exponent reconciles them, because the curve is a property of the model's routing and the profile carries only a ranking, with no counts. The mass is now the increment of H(f) = 1 - (1 - f)^b over the held fraction, whose prefixes sum to exactly H(N/total); b = 3 (STRATA_SPLIT_COVER_B overrides it) is chosen to leave the validated regime where it was - at f = 0.8138 it gives the old fit's 99.35% - while at f = 0.59 it gives 93.2% against 92.2% measured. This is the placement search from Niko1221#584, carried on v0.1.39 and without the weight carve it rode in on: v0.1.39 ships its own carve (Niko1221#559 / Niko1221#639), and the search only runs under `--layer-split auto`, where nothing is carved - so `cap[i]` is already net of the dense weights and a stage's room is what `predict` already subtracts, its session. Measured on the 4-way IQ3_S rig (2x RTX 3060 + 2x RTX 5060), same config and same flags in both arms, the binary the only difference: the four-way split moves from K=10,19,34 to K=14,31,41, the prompt chunk the starved cards allow grows 768 -> 1280 tokens, prefill goes 425 -> 597 tok/s (+40%) and decode 44 -> 51 tok/s (+16%), with the decode expert-cache hit 74.7% -> 83.3%. Two boots per arm, and the split is deterministic. The curve and the search are one commit, so that measures the pair, not the search alone.

With STRATA_STAGE_TRIM=1 (PR Niko1221#639) each card loads only its own layers dense weights and the freed VRAM goes to the expert cache. On this box: resident experts 8649 -> 9387 (+8.5%), prompt throughput back to 0.1.40.1 levels (~1854 tok/s at 128K), TTFT 59.5 -> 52.2 s, decode unchanged. Re-ran the full benchmark on the warm cache and needle_bench (6/6). Adds run-262k.sh so the environment is reproducible.

…doc completeness, one audit challenge rejected Pass A (fresh completeness sweep over MULTI_GPU.md, KV_PREFETCH.md, MESSAGE_BOUNDARY_CACHE.md full, DETAILS.md parking section): - --pipeline-windows (speculative window overlap) is switched OFF by --batch itself (MULTI_GPU.md) - the split's batching-surviving pipelining is --batch-groups; stated in Part 1 item 1 (bears on the 4-card plan). - MESSAGE_BOUNDARY_CACHE: the cited reuse numbers carry constraints the entry lacked: opt-in flag, and LAYER-SPLIT EXECUTION IS EXCLUDED - this box's intended shape - plus chunk-geometry FP caveat. - Parking cache: RAM-admission refusal (2,560 MiB floor default), layer-split verification via tools/parking_test.py, and DETAILS.md's own note that HIP and multi-GPU coverage need separate evidence - unvalidated on this box. - Part 2 restore economics: STRATA_KV_PREFETCH=1 (prefill -5.3% at 32K) is the overlap precedent for the H2D restore cost; HIP unvalidated. - Ceiling paragraph: trim-stage-weights attribution split (PR Niko1221#639, not Niko1221#559) and MI50 evidence added (2x MI50 16 GB: 8,819 -> 10,626 VRAM experts, 39.2 -> 41.7 tok/s, MULTI_GPU.md). Pass B (fresh independent verification of all quantitative claims, no cached line numbers): 21 numbered items + spot-checks; wrong-line fixes: fifo is server.py:3133 (not :215); DETAILS.md:967 (not :968) for the 118 MB figure; fit region 4573-4611; G3 loops 1208-1217, 1242-1255; G8 kernel path src/kernels/cpu/ + ISA qualifier (AVX2-only CPUs already run the grouped kernel for IQ3_S, so the rounding caveat is quant- and CPU-dependent); the qsa.hpp quote now uses the header's own words (:74) with the identity line noted at :179-180. The audit's flag that the ~2,300x page-cost ratio was off 4x was CHECKED AND REJECTED: its replacement figure (12,864 B/page) is derivable from nothing; 118 MB / (4 tok x 12,672 B/tok) = 2,328, consistent with kv_q8.hpp:8's per-token-across-12-layers unit, which is the correct unit for a shared node's cross-layer savings. Auditor arithmetic is itself an audit target. Pass C: header updated to record seven passes and the full doc set; coherence read clean (G1-G8 single rows, phase refs, quotes). Signed-off-by: joo@chung.net <joo@chung.net>

…he conversation and in the slots (A0) Ports Project Maya's restore points (#639, needmorevram, 50a3313; ds4_marks.hpp v1.1.42 by mw00 and needmorevram) onto src/dsv41 as a per-file merge, not a commit apply: - ds4_marks.hpp as in Maya v1.1.42 (latest_point, mark_plan, mark_victim, TailCost; TailCost is not wired in yet, #668). - Model::mark(shared) / marks_cap() / check_bytes(): up to STRATA_DS4_RESTORE_POINTS (4, 0 = none, at most 16) restore points besides checkpoints 0 and 1 and the anchor, the least recently used giving way, the shared start last. restore() and best_check() go back to a mark; a slot keeps the marks with their roles (3 mark, 4 shared start), slot identity strata-ds4-session-4; the .meta file takes up to 3 + 16 checkpoint positions. - Replaces Strata's fourth checkpoint (ck[3], the pin=N shared start) by a shared mark: pin=N is now shared=N. - GEN keys marks=<pos>,... tail=<pos> shared=<pos> (pin=N kept): the prompt read stops at each point mark_plan keeps and keeps the state there. - Tests: tests/dsv41/marks_test.cpp (Maya's ds4_marks_test), slot_test for the longer checkpoint list. Credit: Project Maya #639 (needmorevram), #668 / ds4_marks.hpp (mw00), MIT, third_party/project-maya/NOTICE.md.

…red=) (A1) Project Maya #639 (needmorevram) and v1.1.37 / v1.1.39 (mw00), written onto Strata's server: deepseek_restore_points() reads the last user turn, the first user turn (the end of the system prompt and tools) and the tail from the prompt's ids as the template wrote them; Service.prepare puts them in the request's _restore (never a client's own), StrataEngine sampling_keys turns them into marks=<pos>,... tail=<pos> shared=<pos>. The shared start stays Strata's pin=N where the request already has one (the system messages); shared= is asked only without it. STRATA_DS4_RESTORE_POINTS (4, 0 = off) and STRATA_DS4_RESTORE_TAIL (512, 0 = none). serve/test_ds4_restore_points.py. Credit: Project Maya #639 (needmorevram), mw00; third_party/project-maya/NOTICE.md.

This was referenced Oct 3, 2026

hip: AMD Instinct MI50 / MI60 / Radeon VII (gfx906, wave64) as an opt-in build #638

Closed

AMD Instinct MI50 / MI60 / Radeon VII (gfx906, wave64): a working port, numbers, and PRs to upstream it #641

Closed

JeanP00l mentioned this pull request Oct 3, 2026

STRATA_ARENA_MMAP=1: the expert arena as a read-only mapped file (Linux), for small-RAM machines #640

Closed

JeanP00l marked this pull request as ready for review October 3, 2026 19:25

JeanP00l mentioned this pull request Oct 3, 2026

bench: Linux + 2x AMD Instinct MI50 16 GB (gfx906), Coder IQ1_M, 4K to 128K prompt tokens #677

Closed

Niko1221 added a commit that referenced this pull request Oct 4, 2026

docs: the layer split's per-card dense weights (PR #639), STRATA_AREN…

2bc2dc1

…A_MMAP (PR #640), make_profile --reorder (#587) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Niko1221 mentioned this pull request Oct 4, 2026

Several conversations at once: batch slots, a pipelined layer split, per-stage dense weights #559

Closed

Niko1221 closed this Oct 4, 2026

Hardin22 mentioned this pull request Oct 5, 2026

A dual-GPU fork for 32 GB PCs (resident mode on a layer split, pipelined windows) #642

Closed

paulhothersall mentioned this pull request Oct 5, 2026

P40 + 3070 (mixed Pascal/Ampere pair): layer_split: auto vs forced splits on 0.1.39, with hit rates (follow-up to #604) #875

Closed

paulhothersall mentioned this pull request Oct 5, 2026

Proposal (docs only): a short "contributing a change or report" section for AGENTS.md, and a pinned list of test requests. Yes/no? #909

Closed

Niko1221 added a commit that referenced this pull request Oct 6, 2026

docs: the layer split's per-card dense weights (PR #639), STRATA_AREN…

29181ca

…A_MMAP (PR #640), make_profile --reorder (#587)

anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026

docs: the layer split's per-card dense weights (PR Niko1221#639), STR…

d2adc8b

…ATA_ARENA_MMAP (PR Niko1221#640), make_profile --reorder (Niko1221#587)

shrisha108 mentioned this pull request Oct 7, 2026

bench: 2x TITAN RTX (sm_75) at Strata v0.1.40.3, IQ3_S at 262K #1429

Closed

This was referenced Oct 28, 2026

[Engine] layer split: trim every card's dense weights on --layer-split auto too (from Strata-DualGPU, #642) #4522

Open

[Engine] scope: should per-card weight trimming cover --layer-split auto, and on by default? (PR #4522) #4541

Open

rafatxf mentioned this pull request Nov 1, 2026

[Engine] --layer-split auto never trims CUDA0's dense weights: with an explicit split, 2x 2080 Ti gets +945 expert slots on CUDA0 and +6-12% decode #4915

Open

Navigation Menu

Repository navigation

layer split: with explicit split points each GPU loads only its own layers' dense weights - #639

layer split: with explicit split points each GPU loads only its own layers' dense weights#639
JeanP00l wants to merge 1 commit into
Niko1221:mainfrom
JeanP00l:pr-stage-trim

JeanP00l commented Oct 3, 2026 •

edited

Loading

Uh oh!

JeanP00l commented Oct 3, 2026

Uh oh!

Niko1221 commented Oct 4, 2026

Uh oh!

Niko1221 commented Oct 4, 2026

Uh oh!

ibozulic commented Oct 4, 2026

Uh oh!

JeanP00l commented Oct 4, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

JeanP00l commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Measured

Status

Uh oh!

JeanP00l commented Oct 3, 2026

Uh oh!

Niko1221 commented Oct 4, 2026

Uh oh!

Niko1221 commented Oct 4, 2026

Uh oh!

ibozulic commented Oct 4, 2026

Uh oh!

JeanP00l commented Oct 4, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JeanP00l commented Oct 3, 2026 •

edited

Loading