Repository navigation
--batch on a 100%-resident layer split: engine dies at "captured the batch window over slots 0,1" (zero-doorbell path?) #776
Description
Engine: 0.1.39, Windows, strata-windows-x64.zip Rig: 2× RTX PRO 6000 Blackwell (96 GB), PCIe 5.0 x8/x8 under one switch (PXB), 256
GB RAM Config: --layer-split 24 --expert-cache auto --kv int8 --kv-resident 32768 --max-context 256000 --remote-expert-opt --trim-stage-weights --batch 4 — learned profile, 100% of experts resident on both cards (zero-doorbell graph active on every solo capture)
Repro: engine serves solo requests perfectly (222–280 tok/s, ~7000tok/s prefill). Send 4 concurrent greedy requests (~80-token prompts, 200 tokens out). All four admit and copy into slots (~50 ms each); engine exits 1, last log line: strata verify: captured the batch window over slots 0,1 Same with and without --batch-groups 2. --no-capture and --no-token-graph both refuse batch at init, so there is no non-graphed shape to A/B against.
Theory: the zero-doorbell verify graph (#646, "runs only when every expert of a layer is in VRAM… on big cards was not measured here") combined with batch-window capture over a layer split. Possibly the first rig where both are live. Happy to test builds — full logs attached.
issue-batch-zerodoorbell-excerpt.log
issue-batch-zerodoorbell.md
Activity
qni-live commentedon Oct 4, 2026
Same failure on a second rig, a bisect, and a workaround that works:
STRATA_VERIFY_ALL_RESIDENT=0Same bug here, different hardware: Linux, 2x RTX PRO 4500 Blackwell (32 GB, PCIe 5.0 x16), Threadripper 7960X, ~61 GiB RAM, driver 595.91.07, CUDA 13.4, v0.1.39 built from the tag (sm_120). Swift 1.5 IQ3_XXS, 262144 context,
--kv k8v4,--layer-split 25, MTP,--batch 2 --batch-groups 2 --trim-stage-weights. The startup log sayslayer split: 100% of the experts resident.One request alone is fine (127-133 tok/s). Two at once (8K prompts, 400 tokens out): the first one moves into a slot (
batch: slot 0 takes 7623 tokens), the log showscaptured the batch window over slots 0, thenverify batch: timed out at layer 0; its GPU waits were released and the GPU finished (#267)and the engine exits with code 1. The server restarts it and the next pair does the same.These do not help (all still fail):
--batch-groups 1,--batch 4, no--trim-stage-weights,STRATA_PARALLEL_SOLO=0,STRATA_RING_BYTES=0,--prefill 2048.Bisect (two requests at once, 8K prompt, 300 tokens, pass = both answers complete): the commits of the parallel branch pass: e3978eb (Merge PR #559), 2fbbfa2, bebfca9, 9556298, 831a90f, 1a56d71. The merge commit 1189218 (
Merge branch 'f-139b-par' into f-139-rel) fails, and so does the v0.1.39 tag. So the failure appears when the batch code meets the release line (#646 and the rest), not inside the parallel branch itself. Those parallel-branch commits do not contain cfd3b72 / deee447 / f945515.Workaround:
STRATA_VERIFY_ALL_RESIDENT=0(the switch inverify.cpp, on by default). With it the same config runs, with 2 and with 4 slots, no failed request in any round, and 9 of 9 tool calls valid (alone and two at once). It is the A/B you were missing, @Seen-Tomorrow: the failing shape is the all-resident one. The cost is expected: one request alone decodes about 122 tok/s instead of about 130 (the #646 gain is off).Throughput with the workaround (8K prompts, 400 tokens out, total tok/s incl. reading the prompts, 3 rounds, median; "queue" = v0.1.39 without
--batch):4 slots with only 2 requests is not faster than the queue here, while 2 slots is; I did not look into why. For comparison, before the merge (PR head f65301b on the 0.1.38 base, same machine,
--batch 2 --batch-groups 2), 2 requests gave 84.6 against 66.5 for the queue.Not tested: Windows, images in 0.1.39 batch mode, more than 3 rounds, exactness (solo against solo already differs at temperature 0 in this build, so it was no usable check).
Question: does the zero-doorbell graph need a batch-aware capture, or should
--batchturn it off for now? I can test a fix or a switch on this box (2 cards, Linux).The tests were run by an AI agent (Claude Code) on my machine. I reviewed the results and this report before posting.
0xPreDa commentedon Oct 4, 2026
Your hypothesis is right. When a stage holds every one of its experts, the batch window is recorded as the zero-doorbell graph, which rings no layer. The batch host paths (
run_slot_rows, andbatch_launch/batch_pollwith--batch-groups) still wait for layer 0's ring, so the first batch window times out after 20 s and the engine exits. The solo path already handles this case: it raises the PLE flag once and skips the per-layer loop.I hit the same failure on a 4x RTX 4090 layer split (Linux, 0.1.39, IQ3_S, 100% resident). With the fix in #792,
--batch 4 --batch-groups 4runs without a single engine error: 4 concurrent requests at 81 tok/s each, with the first token in 1.1 s.Seen-Tomorrow commentedon Oct 4, 2026
OP here (the Windows rig from the report). Huge — thank you both for the bisect and the mechanism; STRATA_VERIFY_ALL_RESIDENT is exactly the switch I couldn't find, and "batch host paths wait for a ring the zero-doorbell graph never rings" explains the silent exit-1 perfectly.
One data point neither setup covers: my rig is Windows/WDDM (2× RTX PRO 6000, PCIe 5.0 x8/x8 under one switch), and the batch capture there also carries the WDDM 8 GiB arena-pin cap and the staged hyper-connection path. So #792 is untested on the one platform my report came from.
I'll retest on this box the moment #792 has a build — planned matrix: --batch 4 with and without --batch-groups 2, solo-path regression check (my daily driver is solo/single at 222–280 tok/s, ~7000tok/s prefill and I want the zero-doorbell gain intact there), and the 4-concurrent greedy repro from the original report. Happy to run candidate builds or add logging — the rig stays available.
Also worth noting from the daily-driver side: at 100 % residency the solo path is flawless here (10+ M tokens of real agent work on the split since the report, zero incidents), so whatever #792 changes, the surface is genuinely just the batch host paths.
strata lane: STRATA_VERIFY_ALL_RESIDENT=0 so batch slots stop crashin…
Niko1221 commentedon Oct 6, 2026
Fixed in 0.1.40: --batch now works on a split whose stage is fully resident in VRAM. The batch window used to wait for per-layer doorbells that an all-resident stage never rings; it now raises the PLE flag, lets the graph run to the end and skips the per-layer service (PR #845). Single-GPU behaviour is unchanged. We can only run one GPU here, so a 2-GPU check from someone with a split would help.
meelonjisoo-commits commentedon Oct 6, 2026
2-GPU confirmation from a Linux rig whose stage is 100% VRAM-resident — 0.1.40 fixes it. Plus one nuance on slot reproducibility.
@Niko1221 asked here for a 2-GPU check from someone with a split. This is a Linux one, and it is the all-resident shape that reproduced the original failure.
Rig / shape
-DCMAKE_CUDA_ARCHITECTURES="86;70", 116 cubins),serve/taken from the v0.1.40 tree — a matched pair with its owntools/strata_tokenizer.py, whose md5 changed in this release (a mismatched overlay 500s every request carrying a literal<think>).--layer-split 30 --trim-stage-weights --max-context 174080 --kv q4_0,"parallel": 2, two concurrent ~120K-token agent-shaped prompts. V100 SM clock sampled during the run: median 1380 MHz (full boost), so nothing below is a throttle artefact.9216 of its 9216 profiled pairsresident;--trim-stage-weightsis what gets it there, and it is why this rig reaches the zero-doorbell path.Before 0.1.40 we reproduced the reported symptom verbatim: both slots prefilled, each generated exactly one token, the last engine line was
strata verify: captured the batch window over slots 0, then the engine exited — 400 on one request, 503 on the other.On 0.1.40, with no
STRATA_VERIFY_ALL_RESIDENTworkaround and no local patch:strata verify: captured the batch window over slots 0,1strata batch: 129 windows, avg 1.26 rows, 26.9 ms/windowCORRECTION —
--batch-mtp(#846) does NOT reach a layer split. With the flag on the same config the engine refuses it:So that arm ran as plain batching, and the higher in-slot decode I first reported for it (71.0 vs 59.2 t/s, drafts 78/98 vs 31/47) was run-to-run variance in draft acceptance, not the flag — consistent with the non-reproducibility measured below. My first version of this comment got that wrong; this is the corrected reading. Flagging the single-GPU restriction in case it is not intended to be that wide.
One nuance worth knowing — slot output is not byte-reproducible, even though solo is.
With
temperature: 0:129 windows, avg 1.26 rows; the runs closest to solo loggedavg 1.00 rowsplus a small3 windowsblock (little real sharing).One thing I am not claiming: solo (parallel off) 0.1.40 measured 1932 t/s vs 2041 t/s for 0.1.39 on the same 119,937-token prompt (−5.3%). But my two arms also differed in the resident path (workaround vs zero-doorbell) and I did not randomise order, so I am not calling that a regression. I will re-run it with the path held constant and report if it holds.
meelonjisoo-commits commentedon Oct 6, 2026
Correction to my comment above, since I'd rather flag it than let it stand: I attributed a +20% in-slot decode gain (+ draft acceptance 66% → 80%) to
--batch-mtp(#846). That was wrong.The engine refuses the flag on a layer split:
so that arm ran as plain batching, and the difference was run-to-run variance in draft acceptance — the same non-reproducibility I describe in the other half of that comment. I have edited the comment above to the corrected reading.
The 0.1.40 fix for #776 itself is unaffected — that part reproduced cleanly and repeatedly (both slots admitted,
captured the batch window over slots 0,1, no exit, no 400/503).Noting it in case the single-GPU restriction on
--batch-mtpis wider than intended.qni-live commentedon Oct 6, 2026
Second 2-GPU confirmation: 0.1.40 (tested as v0.1.40.1) fixes
--batchon a fully resident layer split, Linux, 2x Blackwell@Niko1221 asked for a 2-GPU check. This is the rig that reproduced the failure in my first comment above, now on the fixed release.
Rig: Linux (Ubuntu 26.04.1, kernel 7.0.0-38), 2x RTX PRO 4500 Blackwell (32 GB, PCIe 5.0 x16 each, 200 W), Threadripper 7960X, 61 GiB RAM, driver 595.91.07, CUDA 13.4. Engine built from the tag v0.1.40.1 (commit 82f46a8, sm_120). Swift 1.5 IQ3_XXS, 262144 context,
--kv k8v4,--layer-split 25, MTP,--spec 5.What I ran (same config shape as in my first comment,
--batch N --batch-groups 2 --trim-stage-weights, withoutSTRATA_VERIFY_ALL_RESIDENT=0):--batch 2: startup loglayer split: 100% of the experts residentandzero-doorbell graph, the same state that exited withverify batch: timed out at layer 0on 0.1.39. Now 1 to 4 concurrent 8K-prompt requests, 3 rounds each, 400 tokens out, and two concurrent 32K-prompt requests: no engine exit and no failed request in any round.--batch 4: same, no failure (95% of the experts resident there).temperature 0, solo against batch: identical output for 3 of 3 prompts. That is too few prompts to say anything about byte-exactness in general (I read the nuance in the comment above about slots not being byte-reproducible; I did not see it in these 3 prompts, but did not look for it with more).Throughput (8K prompts, 400 tokens out, total tok/s including prompt reading, median of 3 rounds; "queue" = same build without
--batch):This is about the size of the gain I measured on 0.1.39 with the workaround (2 slots +8-14%, 4 slots +20-26%), so the fix does not cost batch speed here. Note the number is total throughput, not decode speed, and the 4-slot arm is not faster than the queue with only 2 requests (same as before).
Solo path: one stream at 1k to 256k prompt tokens, measured on 0.1.40.1 and then on 0.1.39 with the same setup. Prompt reading is 0.3-2.6% faster; the decode difference is within run-to-run spread at most lengths, so I am not claiming a regression or a gain. The 256K prompt (248,489 tokens) ran cleanly. Needle recall 6 of 6 (32K and 128K, depths 10/50/90).
Not tested: Windows, images with
--batch, more than 32K per request with several concurrent requests,presence_penaltyabove 0 in batch mode,--batch-mtp(only 1 GPU in this version). One server start per arm, 3 rounds.I will put the per-run data and scripts into a community-benchmark PR separately.
The tests were run by an AI agent (Claude Code) on my machine. I decided the plan and reviewed the results and this report before posting.