[0.1.43] CUDA layer split (2× RTX 3090): decode windows lose async launch, solo 131.4 → 39–76 tok/s, prefill −35%

Environment

Engine: v0.1.43 (c69adca) — regression present. Good: v0.1.41 (fb58e0d). Not bisected across 0.1.42.
GPUs: 2× RTX 3090 48 GB (sm_86 build), driver 595.71.05, CUDA 13.0.88, Linux 5.15
Model: Qwen3.8-Flash-Next UD-IQ4_XS (3-part GGUF, mmap)
Serving flags (identical for both versions): --pack --native --expert-profile --expert-cache auto --prefill auto --spec 5 --spec-min-p 0.5 --mtp <rt> --max-context 262144 --kv int8 --kv-resident 262144 --batch-mtp --resident-budget-gib 60, layer split auto (2 stages: CUDA0 dense+early layers, CUDA1 rest+expert arena), parallel 4, port 6006

Repro

Serve with the flags above on a 2-GPU layer split, then send one 200k-token prompt, 1024 output tokens, greedy-ish (reasoning off). Compare decode tok/s between v0.1.41 and v0.1.43. Reproduces every run; variance between runs is the tell (below).

Measured impact (same machine, same config, same bench)

Metric	0.1.41	0.1.43	Δ
solo 200k decode	131.4 / 130.8 tok/s (runs: 135.8, 131.4, 130.8 — tight)	39.0 / 57.0 / 59.9 / 76.2 tok/s (2× run-to-run variance)	−42…−70%, unstable
prefill (200k)	3934 / 3914 tok/s	2540–2670 tok/s	−35%
TTFT cold	50.5 s	74–78 s (when prefill regressed)	—
×4 aggregate (batch, cache-aligned)	126.5–129.8	96.3	−25%

Evidence

Decode window launch serialization. STRATA_DECODE_TIMING=1: per-window stage time 0.64 ms (0.1.41) → 29.57–40.60 ms (0.1.43), while the GPU kernels themselves only grew 16.6 → 23 ms. Consistent with the host synchronizing per layer instead of enqueueing the two-stage pipeline ahead — the two-GPU overlap is gone.
dmon during decode: 0.1.41 is a flat 348 W / SM 92% on both cards. 0.1.43 shows the two cards alternating idle (SM 0–34%), power oscillating 200–335 W — classic ping-pong of a broken pipeline.
Page cache thrash: per-request major faults 0 (0.1.41) → 5139–8941 (0.1.43); server log shows lines of the form "OS read 500–1700 MB for 0.0 MB expert reads" — mmap'd GGUF pages being re-read from the OS while the engine believes it issued no expert reads.
Prefill and decode regress by the same magnitude and share the instability → likely one common root cause in the shared launch/scheduling path, not two separate issues.

Ruled out

STRATA_MMVQ_IL=0 (kernels 195d379) — no change (also: we're IQ4_XS, not Q3_K)
Revert of 6b39264 (verify service doorbell) — no change
cgroup memory pressure — 73/180 GB used, no reclaim pressure
Expert residency — --no-prefill-borrow (100% VRAM-resident experts) still reproduces
Profiling overhead — ~10% on solo only, batch unaffected, can't explain 2×

Suspects from the 0.1.41..0.1.43 range (analysis, not verified)

aa7b0cb "CUDA: the default --pcie-frac is refined in the first decode windows from the CPU pool's measured speed (single GPU, no --pcie-frac)" — if the refinement engages on a layer split (where "CPU pool speed" probing may misbehave across stages), it would plausibly stall windows and cause the mmap re-reads in (3). Untested opt-out on our side: STRATA_PCIE_FRAC_DEFAULT=old. Happy to test this first if you think it's plausible.
5ef500d / b97a36a prefill step records from a pinned buffer — candidate for the prefill half, but wouldn't explain decode.
The prefill lend mechanism was independently excluded via --no-prefill-borrow.

Side note: STRATA_SPLIT_MTP_BATCH works

On 0.1.43, STRATA_SPLIT_MTP_BATCH=1 + --batch-mtp on a layer split does function (4 slot drafters on CUDA1, 83.4% acceptance, 7.25 rows of the 8-row window at ×4), but net aggregate was 96.3 vs 129.8 on 0.1.41 — i.e., the feature's gain is currently masked by this regression. Worth re-benchmarking once the regression is fixed.

Layer split + --batch: the default pipeline wins over --batch-mtp (ST…

Thanks for the detailed numbers. Two separate things came out of reproducing this on 2x RTX A4000 (v0.1.41, v0.1.42, v0.1.43 and the 0.1.43.1 branch, IQ3_XXS GGUF in place, --batch 4 --batch-mtp, interleaved pairs).

The x4 aggregate loss is --batch-mtp on a layer split. 0.1.41 refused it there ("--batch-mtp is off"), so --batch-groups auto pipelined two groups, one per stage, and both cards stayed busy. Since 0.1.42 --batch-mtp is accepted on a split and auto then picks one serial group (the start-up line says "--batch-groups auto: 1 group of 4 slots - --batch-mtp does not run in pipelined groups"). That is the alternating idle you see in dmon. x4 aggregate here: 0.1.41 91.8 tok/s, 0.1.43 63.9 (-30%, 0 of 3 pairs faster), 0.1.43 without --batch-mtp 85.8. So dropping --batch-mtp restores the old behaviour. Fixed on the 0.1.43.1 branch: a layer split with --batch and no --batch-groups now keeps the pipelined groups and leaves --batch-mtp off (the start-up line says so); --batch-groups 1 or STRATA_BATCH_MTP_SERIAL=1 keeps --batch-mtp with serial windows. On 2x A4000, 5 interleaved pairs, x4 aggregate: 0.1.41 92.6, 0.1.43 62.3, branch 85.2 (5 of 5 pairs faster than 0.1.43; STRATA_BATCH_MTP_SERIAL=1 gives 64.1); x2: 72.5 / 55.9 / 69.6. Until then pass --batch-groups 2 (it turns --batch-mtp off) or drop --batch-mtp. The adaptive overlap is not involved: it is refused on a layer split, and STRATA_ADAPT_OVERLAP=0 changed nothing here.
The 0 -> 5-9K major faults with "OS read 500-1700 MB for 0.0 MB expert reads" looks like the PLE table switching from direct reads to mmap. Since 0.1.42 the engine probes --ple-io direct for 0.6 s at start and, if it reads under 15000 rows/s, uses mmap instead. Forcing --ple-io mmap on a cold cache reproduces your signature on every version I tried: 32K prefill -33%, faults 1,000 -> 45,000, decode down several percent. Could you grep the 0.1.43 start log for "--ple-io direct measures slow here" (or "PLE direct read probe")? If it is there, starting with --ple-io direct (or STRATA_PLE_PROBE=1) should give the old speed back, and the rows/s number would tell us whether the probe is too eager on your storage.

I could not reproduce the solo 131 -> 39-76 tok/s on the A4000s (solo decode and 32K prefill are the same on 0.1.41, 0.1.43 and the 0.1.43.1 branch), so if it persists after the two changes above, the start-up log of both versions would help.

gfx906: record the v0.1.43.1 build check on T5810

merge upstream/main (v0.1.43.1 line) into feature/sampling-presets-11…

We hit the same 0.1.41 → 0.1.43.1 decode regression on a 4× RTX 3090 layer split, but neither of the two causes you identified applies here — --batch-mtp was never set and the PLE probe kept direct — so this is a clean reproduction of the solo-decode half you could not reproduce on the A4000s.

Rig


GPUs (engine)	4× RTX 3090 / 3090 Ti — `gpu: [0,1,2,3]`, all PCIe Gen4 x16
5th GPU	RTX 5060 Ti (Gen4 x8) — vision encoder only, not in the split
CPU / RAM	AMD EPYC 7F72 (16 vCPU), 64 GiB
Storage	2× Samsung 990 PRO 2TB NVMe (ZFS)
OS / driver / CUDA	Debian 13 in an LXC, kernel 7.0.14-23-pve, driver 595.104.02, CUDA 13.4
Model	Qwen3.8-Flash-Next IQ3_S (GSQ-RCO), native pack, `experts.bin`
Flags	`--serve --batch 8 --batch-groups 4 --layer-split 12,24,36 --trim-stage-weights --remote-expert-opt --spec 4 --spec-min-p 0.5 --mtp mtp/rt --max-context 262144 --kv int8 --expert-cache auto --prefill auto --vision --vram-reserve-mib 700`
`--batch-mtp`	never set — not in the config, and `setup.py` never adds it (no `batch-mtp` match in 0.1.41 or 0.1.43.1)

Versions: good 0.1.41 (fb58e0d, src 62eb4907c92a8cb7), bad 0.1.43.1 (a0116b2, src 7089bf002d14a0e3). Same machine, same config, same engine symlink — swapped by symlink only, no rebuild.

Request-level stats from the engine log

Same production traffic across the restart boundary. 4,416 requests on 0.1.41, 382 on 0.1.43.1.

	0.1.41	0.1.43.1
requests with draft proposals	3,946 (89%)	53 (14%)
requests with zero proposals	470 (11%)	329 (86%)
decode, all requests (median)	156.8 tok/s	82.6 tok/s
decode, drafted requests	161.4 tok/s	135.7 tok/s
decode, zero-proposal requests	81.7 tok/s	81.7 tok/s
draft acceptance (median)	87.7%	77.1%
prefill (median)	418 tok/s	912 tok/s
prompt tokens (median)	126,681	140,736
tokens re-read (median)	150	306

/v1/status on 0.1.43.1: "draft_n": 0, "draft_n_accepted": 0.
On 0.1.41 after rollback: "draft_n": 157, "draft_n_accepted": 124, 138.4 tok/s.

Undrafted decode is identical on both engines (81.7 tok/s). The regression is entirely that drafting stopped running.

Startup log — identical on both engines

strata serve: layer split: layers 0-11 (CUDA0), 12-23 (CUDA1), 24-35 (CUDA2), 36-47 (CUDA3)
strata serve: layer split: 96% of the experts resident, the prompt path's streamed ring 96 slots
strata serve: prompt chunk auto: 8192 tokens, a 96-slot ring
strata serve: 2447 MiB of VRAM free with everything loaded
strata serve: adaptive overlap: off: not on a layer split
strata generate: PCIe probe: 26.3 GB/s host->device (best of 26.3 26.3 26.3 26.3) -> pcie_frac 0.55 (default 0.55)
strata batch (pipelined, 4 groups of 2): ... rows/s (admissions included)

batch (pipelined, 4 groups of 2) appears on both engines (15 occurrences in the 0.1.41 block, and throughout the 0.1.43.1 block). --batch-groups auto: 1 group of 4 slots never appears.

Decode expert cache hit rate: median 99.8% on 0.1.41, 99.9% on 0.1.43.1.

Theory 1 — `--batch-mtp` on a layer split: does not apply

--batch-mtp was never set here.
The log shows the pipelined groups on both engines, not the collapsed serial group.
So 0.1.43.1's fix is already in effect, and the regression is still present.

Theory 2 — the PLE probe switching to mmap: it did not switch

strata generate: WARNING: --ple-io direct measures slow here (11127 rows/s, about 46 MB/s of random 4 KiB reads):
  ... so it is not chosen for you; --ple-io ram needs the RAM for the table
  (kept direct: pass --ple-io mmap or --ple-io ram to change it)

kept direct — no switch, so --ple-io direct / STRATA_PLE_PROBE=1 have nothing to restore.
In 0.1.43.1 the switch is opt-in (STRATA_PLE_AUTO_MMAP=1), src/program/generate.cpp ~3221-3241.
The probe is not in the v0.1.41 source at all: grep -c 'measures slow' → 0 in v0.1.41/src/program/generate.cpp, 3 in v0.1.43.1.
No OS read … MB for 0.0 MB expert reads lines in our log (0 matches), and no major-fault signature.

PLE rows/s on our storage: 11,127 and 11,690 — 74-78% of the 15,000 bar, against ">100k" for a healthy NVMe. 990 PRO on ZFS, LXC. Not the cause of the decode drop; offered as a probe-calibration data point.

What we can reproduce

The part you could not reproduce on the A4000s: solo decode falling, with prefill unaffected (ours prefill is faster on 0.1.43.1). 4× sm_86 at Gen4 x16, no --batch-mtp, on the already-fixed 0.1.43.1.

Caveats: production traffic, not a controlled suite — the two samples differ in workload mix (prompt median 126,681 vs 140,736 tokens, re-read 150 vs 306). The drafting-rate drop (89% → 14%) and the identical undrafted decode (81.7 tok/s) are the solid parts; the 156.8 → 82.6 headline is indicative. bench.sh has not been run on this rig, so there is no baseline.

@Niko1221

@Niko1221 I added a little more of background above ^^^

層の分割と --batch: 既定の組は --batch-mtp に譲らず、組に分けて --batch-mtp を切る（Niko1221#…

Layer split + --batch: the default pipeline wins over --batch-mtp (ST…

@sourceupdev

Both fixes confirmed here — thanks. Follow-up data from a 2× RTX 3090 48 GB layer split (the rig from the issue report), engine v0.1.44.1 (296b59d), same config as the original report (--spec 5 --kv int8 --kv-resident 262144 --batch-mtp --resident-budget-gib 60, parallel 4):

1. The solo regression is gone. 200k×1, 1024 tokens, same bench as the original numbers:

engine	solo 200k decode	prefill	×4 warm aggregate
0.1.41 (`fb58e0d`)	130.8–131.4	3914–3934	126.5–129.8
0.1.43 (`c69adca`)	39–76 (2× run variance)	2540–2670	96.3
0.1.44.1	127.2	4233	123.2 (cold 117.1)

×4 lands ~3–5% under 0.1.41, the same direction and magnitude as your A4000 fix-branch numbers (85.2 vs 92.6). Startup behaves exactly as you described:

strata generate: WARNING: --batch-mtp is off: a layer split with --batch pipelines the slots in groups by default
    and the pipelined groups run no slot drafts (4 streams on 2 GPUs: 25-30% faster than the serial MTP windows);
    --batch-groups 1 or STRATA_BATCH_MTP_SERIAL=1 keeps --batch-mtp with serial windows
strata generate: --batch-groups auto: 2 groups of 2 slots

2. What the solo half actually was on this rig. The 0.1.43 startup line said --batch-groups auto: 1 group of 4 slots - --batch-mtp does not run in pipelined groups, and dmon during a solo 200k decode shows the two cards strictly anti-correlated (one at SM 60–95% while the other sits 0–30%, ~180 W each; 0.1.41 has both at SM 84–99%, ~330 W). Average per-card utilization is exactly half — that is the 2× window-cost increase (22.7 → 51.9 ms/window). So on this rig the solo requests degraded through the same collapsed serial-group topology, and drafting itself was never the problem: the 0.1.43 solo request logged drafts accepted 715 of 992 (72%) vs 0.1.41's 76%. That differs from @sourceupdev's signature (draft_n 0), so the drafting-stop they see is likely a second, separate failure mode.

The 0.1.44.1 decode timing line for the same 200k solo request, now healthy again:

311 windows, avg T 4.05, 3.29 tokens/window, 25.93 ms/window = verify 22.37 (GPU-reach wait 0.00 +
per-layer host 0.00 [plan 0.00 actq 0.00 jobs 0.00 CPU 0.00] + stage 0.71) + commit/emit 0.29 + draft 3.27

3. The PLE probe never fired here. No measures slow here / probe line in any of the 32 sessions in our log (0.1.41 through 0.1.44.1), PLE is always the mmap'd pack table, and the OS read ... for 0.0 MB expert reads lines appear in 0.1.41 sessions too — so that signature alone wasn't diagnostic on this storage. The per-request major-fault spike (0 → 5–9k) did only happen on 0.1.43 steady-state, consistent with your mmap reproduction.

Navigation Menu

Repository navigation

[0.1.43] CUDA layer split (2× RTX 3090): decode windows lose async launch, solo 131.4 → 39–76 tok/s, prefill −35% #2038

Environment

Repro

Measured impact (same machine, same config, same bench)

Evidence

Ruled out

Suspects from the 0.1.41..0.1.43 range (analysis, not verified)

Side note: STRATA_SPLIT_MTP_BATCH works

Rig

Request-level stats from the engine log

Startup log — identical on both engines

Theory 1 — `--batch-mtp` on a layer split: does not apply

Theory 2 — the PLE probe switching to mmap: it did not switch

What we can reproduce

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[0.1.43] CUDA layer split (2× RTX 3090): decode windows lose async launch, solo 131.4 → 39–76 tok/s, prefill −35% #2038

Description

Environment

Repro

Measured impact (same machine, same config, same bench)

Evidence

Ruled out

Suspects from the 0.1.41..0.1.43 range (analysis, not verified)

Side note: STRATA_SPLIT_MTP_BATCH works

Activity

Niko1221 commented on Oct 12, 2026

sourceupdev commented on Oct 13, 2026

Rig

Request-level stats from the engine log

Startup log — identical on both engines

Theory 1 — --batch-mtp on a layer split: does not apply

Theory 2 — the PLE probe switching to mmap: it did not switch

What we can reproduce

sourceupdev commented on Oct 13, 2026

lijiajun1997 commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Theory 1 — `--batch-mtp` on a layer split: does not apply