Repository navigation
batch-groups: read a cold prompt beside the decoding slots, stall 11.9 -> 0.8 s, same work 6.4% sooner (opt-in) - #2326
SuperDistance wants to merge 7 commits into
Conversation
|
Measured on a 4-stage split: 4x RTX 5080 16 GB (PCIe Gen3), i9-10920X, 123 GiB RAM, Linux, CUDA sm_120. Qwen3.8-Flash-Next GSQ-RCO IQ3_S, The build is v0.1.44 + this PR + #2050 + a few of our patches. Only one of the two read modes is set per run. All five commits applied on top of #2050; I resolved the overlap in the read loop by keeping your
With What I see on this rig: the joiner's own TTFT stays close to main (11.4 s against 10.5 s), which is the big win over pieces (19.7 s). But the streams barely decode during the read. The engine log says "the slots decoded 36 rows meanwhile (6.1 rows/s; stage 1: 36 windows, 1.4 ms host each; ...)" over a ~5.9 s read, while you saw 85-92 tok/s on 2x 5090. Two of the three reads in our follow-up test also logged "decoded 0 rows" with no slot decoding, which is expected. My guess is that with 4 stages on Gen3 x16 and 16 GB cards (most experts host-side), the prompt's expert traffic and the windows' compete on the same links and the CPU pool. Happy to run a per-stage timing if that helps. For our agent workload we stay on #2050's pieces for now: the follow-up latency matters most to us. Thanks for the work! |
…ead's core on Linux On Linux a thread inherits its creator's CPU mask, and the host thread is pinned to the one core the pool leaves it (aux_cpus.hpp). The read's thread and the doorbell thread were created from it, so they shared that core with the host thread that pumps the windows, and every window waited there for the doorbell of each layer: 6.1 rows/s during a read on 4x RTX 5080, 4 stages (blange48 on Niko1221#2326). Windows places the host with CPU Sets, which new threads do not inherit, so it did not show there. Both threads now move off the host's core and the pool workers' CPUs (on an SMT machine with the default pool: the idle sibling threads; without such CPUs, every CPU but the host's), and the threads that already run on the host's CPU (the prompt path's copy threads, started by the first prompt) move the same way at a read. --aux-cpus places them all when it is on. STRATA_PIPED_CONCURRENT_UNPIN=0 keeps the inherited CPUs (an A/B). Measured on Linux (WSL2, Ubuntu 24.04, 2x RTX 5090, 4 streams, a cold 32K prompt; same binary, UNPIN=0 / default): the slots decoded 25.9 and 23.7 rows/s during the read with the threads kept on the host's CPU (76-81 threads there), and 75.6 and 62.8 with this change; with the n-gram table in RAM (WSL's disk reads it 7x slower than Windows) 110.7 rows/s, as on Windows (107.0). piped_read_join_bench.py (cold32k, followup, two_joiners; table in RAM, 2-3 runs each): 24-37 rows/s during the 30K+ reads kept on the host's CPU, 95-127 with this change, and 925-1,065 stream tokens during the cold 32K join against 454-484. On Windows, STRATA_TEST_CONC_ON_HOST_CPU=1 (a test) puts both threads on the host thread's CPU set first: 18.6 rows/s kept there, 105.4 with this change.
|
@blange48 thank you for testing this so carefully. Your report found a real bug on Linux: a new thread inherits its Your bench on Linux here (WSL2, 2x RTX 5090, 2 stages, all on main ed4773e, 2-3 runs each):
I can't say yet that this is all of what you saw: here the slots fell to 24-37 rows/s during a read, not 6.1, and I For the follow-up, the long read paying ~9 s more (with #2050 too) looks like a re-read from the start after the |
…table The prompt path gathers a chunk's PLE rows on its own thread and the batch windows gather theirs on the main thread, from the same table (ss.ple.table). Today they never overlap - a prompt run's read-ahead ends with the run, and the windows run between runs - but a prompt read beside running windows would race on the reader's tickets and the prefetch slots. Every public method that touches them now holds a recursive mutex (gather holds it across its issue and collect). Uncontended it is a few nanoseconds per call; nothing else changes.
…its own (opt-in, experimental STRATA_PIPED_CONCURRENT) With a layer split and --batch-groups > 1 a prompt read is one sp.run, so every decoding slot stops for the whole read (Niko1221#1906); Niko1221#2050's pieces bound the stop but drain the pipeline at each piece. With STRATA_PIPED_CONCURRENT=1 the read runs on a worker thread while the main thread keeps the groups going through the pipeline (pump) and takes BSTOP and BYIELD meanwhile. The prompt's K/V goes to the main sessions and the slots' to their own; the windows' streams get the device's highest priority on CUDA (STRATA_PIPED_CONCURRENT_PRIO=0 keeps the default). - The slots' doorbells are served on a thread of their own that makes no CUDA call (Verifier::batch_serve, now batch_poll's per-layer service too, and set_external_serve). Two hangs found while building this, read from the stall dumps against the PDB, had Niko1221#31's shape: the main thread waited for the driver's lock in cudaSetDevice while its holder waited for the GPU (a cudaFree in the drafter's prompt pass; cuBLASLt's lazy load of a GEMM module), and a window on that GPU waited for its doorbell. - MtpDrafter::reserve_prefill captures the drafter's prompt graphs and sizes its records before the read. - Prefill::stop_at_chunk: a BYIELD ends the run at its next chunk, also with no slot decoding; the pipeline is drained there and the part read gives way as Niko1221#656 does. A read nobody waits for is not cut. - The reader is joined on every way out (an exception too). Off beside STRATA_PREFILL_HELP and STRATA_SPLIT_MTP_BATCH, which allocate during a read.
…eeps its checks and the watchdog, and waits on a launch From a review of the concurrent read: - With external serving batch_poll still finds a window that will not finish (its graph ended or failed with a layer to ring, or no layer rang for 20 s): the served count is read before the stream query and the ring count after it, at most one query a millisecond; the last ring's time is an atomic. batch_serve reads a window's row count after its served count, so a window launched while the doorbell thread was descheduled is not served with the last one's rows. - Niko1221#1996's pinned upload rings (0.1.43.1) grew a slot at its first larger upload, a pinned free and allocation on the read's thread: the drafter's records and the prompt path's routing tables reserve their slots before the read (PinnedRing::reserve, Prefill::reserve_uploads). - Off, said once, where serving a doorbell makes a CUDA call (--pcie-mode dma, STRATA_FS_SLOTS, the helper caches) or the expert source recycles the buffers it hands out (the GGUF in place, unbuffered, STRATA_IO_PREFETCH); a prompt with pictures is read the usual way. The expert sources' read counters are atomics. The check is a pure function, concurrent_read_off (batch_read.hpp), with a test in batch_read_test. - pump's all-idle branch turned the stall watchdog off when the last slot finished mid-read; it stays on while a prompt is read until the request ends. - Between windows the doorbell thread slept 50 us, a whole timer tick on Windows (up to 15.6 ms): it now waits on a condition each launch notifies (STRATA_TEST_BELL_SLEEP_US=N, a test, restores the sleep for an A/B). With nothing decoding the main thread waits for the read's end or a line instead of sleeping. A pool failure the doorbell thread set is reported before pump clears the flag for the next window. - The read's log line ends with the slots' rate and, per stage, its windows and the host time that set each up. The priority stream's cudaGetLastError runs only under the opt-in.
…ured The opt-in's section: how the read and the windows share the GPUs, the doorbell thread, the give-way at a chunk, where the mode is off. Measured against main (a0116b2) on 2x RTX 5090 in one window (main, this, this, main): the stops, the streams' rate during the reads with the same answers every run, the joining prompts' TTFT, a question during a read, the same randomised load on both, the doorbell thread's first wait as an A/B; 16 GB cards on each side; a weaker host as well in Niko1221#1906's shape; the default path's check; text equality. tools/piped_read_join_bench.py, the client that measured it, is Niko1221#2050's, unchanged (the same file as there).
…p the concurrent read off The flag works only through a helper cache (--expert-cache-device1..3), and on a plain layer split it is inert (MULTI_GPU.md, Niko1221#1447), so the check looks at the helper caches themselves. Niko1221#2275's rig runs a plain split with the flag set, and the mode would have stayed off there for a cache it does not have.
…ead's core on Linux On Linux a thread inherits its creator's CPU mask, and the host thread is pinned to the one core the pool leaves it (aux_cpus.hpp). The read's thread and the doorbell thread were created from it, so they shared that core with the host thread that pumps the windows, and every window waited there for the doorbell of each layer: 6.1 rows/s during a read on 4x RTX 5080, 4 stages (blange48 on Niko1221#2326). Windows places the host with CPU Sets, which new threads do not inherit, so it did not show there. Both threads now move off the host's core and the pool workers' CPUs (on an SMT machine with the default pool: the idle sibling threads; without such CPUs, every CPU but the host's), and the threads that already run on the host's CPU (the prompt path's copy threads, started by the first prompt) move the same way at a read. --aux-cpus places them all when it is on. STRATA_PIPED_CONCURRENT_UNPIN=0 keeps the inherited CPUs (an A/B). Measured on Linux (WSL2, Ubuntu 24.04, 2x RTX 5090, 4 streams, a cold 32K prompt; same binary, UNPIN=0 / default): the slots decoded 25.9 and 23.7 rows/s during the read with the threads kept on the host's CPU (76-81 threads there), and 75.6 and 62.8 with this change; with the n-gram table in RAM (WSL's disk reads it 7x slower than Windows) 110.7 rows/s, as on Windows (107.0). piped_read_join_bench.py (cold32k, followup, two_joiners; table in RAM, 2-3 runs each): 24-37 rows/s during the 30K+ reads kept on the host's CPU, 95-127 with this change, and 925-1,065 stream tokens during the cold 32K join against 454-484. On Windows, STRATA_TEST_CONC_ON_HOST_CPU=1 (a test) puts both threads on the host thread's CPU set first: 18.6 rows/s kept there, 105.4 with this change.
6c9238b to
67bc65c
Compare
|
Re-ran with the fix. The build is v0.1.44.1 + this PR as rebased (6 commits, with 67bc65c) + #2050 + a few of our patches. Same rig: 4x RTX 5080 16 GB (PCIe Gen3), i9-10920X,
With the fix, tool calls (3/3), the 16-agent isolation test (0 leaks) and 16 parked conversations re-opened warm (0.26 s) all pass. The fix shows: stream tokens during the join go from 280 to 535, and the stop and the follow-up both shorten. But the engine log still reports about 7 rows/s for the slots during a read ("read 18037 tokens on a thread of its own in 5434 ms (concurrent, to 42613 of 42620); the slots decoded 37 rows meanwhile (6.8 rows/s; stage 1: 37 windows, 0.9 ms host each; ...)"), against 24-37 on your 2-stage rig. Our follow-up test still prefers pieces (4.0 against 5.4 s), because the follow-up waits for the next chunk boundary of the concurrent read; Overall it is a clear win on our rig: two cold joiners finish about twice as fast and the run's throughput is 48% higher. We will likely move production to it. I can run a per-stage timing if you want to dig into the 7 rows/s; tell me which variables to set. |
|
Found by reading (no Mac here to build): the new |
…calls in its CUDA shim) STRATA_PIPED_CONCURRENT's window streams ask for the device's highest priority through cudaDeviceGetStreamPriorityRange / cudaStreamCreateWithPriority. The Mac engine compiles src/core/verify.cpp too (metal/CMakeLists.txt) with STRATA_GPU_METAL, and metal_compat/cuda_runtime.h declares neither, so the Mac build would not compile. Same guard as the drafter's streams in mtp.cpp: HIP and Metal keep the default priority. CUDA and HIP preprocess to the same code as before. Not built on a Mac (none here). Reported by @qiuzan2001 on Niko1221#2326.
@qiuzan2001 Nice catch! Thanks! Let me get my changes compiled on mac to verify |
2x RTX 2080 Ti (sm_75, 22 GB, layer split) — the mode pays with one stream decoding, costs a lot with threeSummary, first: on this machine the mode does exactly what it says for a decoding stream (a 52 s freeze becomes a 3 s gap), and with one stream decoding the whole run costs 5% more wall clock while the small request is answered 44% sooner — that is the case it is for. With three streams decoding the same run costs 73% more, because the read's rate decays as more slots decode. It does not change aggregate decode throughput in either direction, and the four-stream long-context case that made us open #2275 is untouched by it. Hardware the maintainers don't have: 2x RTX 2080 Ti 22 GB (modded, sm_75), layer split 24, a Broadwell-era Xeon without AVX-512 (expert kernels on AVX2), 123 GiB RAM, CUDA 13.0, driver 595.99.02. Built from this PR's head (63335d4, engine 0.1.44.1): Release, Common settings: A. One stream decoding (the "two users: one chatting, one loading a long prompt" case)
The engine's own lines for that arm: That is the case this PR is for: a stream that would be frozen for 52 s keeps moving, the small question is answered during the read instead of after it, and the run as a whole costs only 5% more wall clock. B. Three streams decoding (same join, same binary)
So the same mechanism costs 2.7x on the read and 73% on the run once three slots are decoding — the read's rate decays as more slots decode, which is more than the decode side gains. What I think this means (offered as a data point, not a theory)
If it helps, I can re-run either arm with |
…ead's core on Linux On Linux a thread inherits its creator's CPU mask, and the host thread is pinned to the one core the pool leaves it (aux_cpus.hpp). The read's thread and the doorbell thread were created from it, so they shared that core with the host thread that pumps the windows, and every window waited there for the doorbell of each layer: 6.1 rows/s during a read on 4x RTX 5080, 4 stages (blange48 on #2326). Windows places the host with CPU Sets, which new threads do not inherit, so it did not show there. Both threads now move off the host's core and the pool workers' CPUs (on an SMT machine with the default pool: the idle sibling threads; without such CPUs, every CPU but the host's), and the threads that already run on the host's CPU (the prompt path's copy threads, started by the first prompt) move the same way at a read. --aux-cpus places them all when it is on. STRATA_PIPED_CONCURRENT_UNPIN=0 keeps the inherited CPUs (an A/B). Measured on Linux (WSL2, Ubuntu 24.04, 2x RTX 5090, 4 streams, a cold 32K prompt; same binary, UNPIN=0 / default): the slots decoded 25.9 and 23.7 rows/s during the read with the threads kept on the host's CPU (76-81 threads there), and 75.6 and 62.8 with this change; with the n-gram table in RAM (WSL's disk reads it 7x slower than Windows) 110.7 rows/s, as on Windows (107.0). piped_read_join_bench.py (cold32k, followup, two_joiners; table in RAM, 2-3 runs each): 24-37 rows/s during the 30K+ reads kept on the host's CPU, 95-127 with this change, and 925-1,065 stream tokens during the cold 32K join against 454-484. On Windows, STRATA_TEST_CONC_ON_HOST_CPU=1 (a test) puts both threads on the host thread's CPU set first: 18.6 rows/s kept there, 105.4 with this change.
…#2050, opt-in STRATA_PIPED_READ_PIECE), on top of #2326's concurrent read Squash of SuperDistance's three commits of PR #2050 (decode between the pieces of a read on the pipelined groups, the piece lines counting the rows the slots decoded meanwhile, the stall watchdog staying on for a read in pieces), applied after #2326 here: the BYIELD give-way is #2326's give_way lambda, the concurrent block runs first and the piece path serves a read that STRATA_PIPED_CONCURRENT leaves to it (pictures, configurations where it is off). Co-authored-by: SuperDistance <43229543+SuperDistance@users.noreply.github.com>
|
Thank you, this is a good piece of work. We ported all six commits with your authorship and ran it on 2x RTX A4000 (Q2_0, layer split, 4 decoding streams, pipelined groups, 3 interleaved rounds): the longest stall of a stream during a cold 32K join goes from 17.7 s to 1.7 s, with two joiners from 26 s to 1.5 s, and a short follow-up asked during a 40K read gets its first token after 5.9 s instead of 18.6 s. The joining prompt pays for it (its first token comes about 25% later, aggregate tok/s is a little higher). Eight repeats of the three scenarios with the mode on gave no hang, stall report or error. With the variable unset everything is identical to main (identity gate on sm_86, sm_60 and gfx1201). It stays opt-in and experimental in 0.1.45; we would like more reports from 3-4 stage splits before we change that. |
|
A 4-stage report on stock 0.1.45 (
Determinism prompts (sequential) were 6/6 identical, unset vs PIECE and unset vs CONCURRENT=1. Our agent mix, one run per arm, took 206.0 s unset, 207.4 s with CONCURRENT=1 and 276.2 s with PIECE. Replies differ between runs even between unset and CONCURRENT=1, which ran the same code path (bench_multiturn 32/48 replies identical; cause not checked). So the PIECE run did different work: 5,577 vs 8,041 output tokens, and 4 vs 10 chat requests (the bench sends them at random intervals, ~25 s mean). For context, our fork's own overlapped read (0.1.44.1, not upstream code; in the builds measured here it never gave way; our own bench, Not tested: |
|
A HIP data point for Rig: 2x RX 6900 XT (gfx1030), ROCm 10.0, Linux, Ryzen 5 5600X, 128 GB DDR4. Flash-Next GSQ-RCO IQ3_S, 262144 context, Stability: 6 arms with the variable on, 136 concurrent reads in total (8 of them with a slot decoding beside the read), up to 57K tokens per read: 0 stalls, 0 hangs, 0 stall reports. Each arm also read a 64K prompt with two codes in it while a short request arrived; both codes were answered correctly in every arm. A long read and a short request (64K prompt, a one-line request sent while it is read):
Cost on a single request: none that I can measure. With the variable on and off (both with Two things worth knowing on this kind of rig:
Developed with an AI coding assistant; every number above was measured on 2x RX 6900 XT (gfx1030), ROCm 10.0, Linux. |
|
Thanks, this is useful, especially the 64K case with a short request beside it. Our data is on the other side of the gate. On 4x RTX 3090 with the native GGUF pack (no On your point 1: we ran Your point 2 (decode slow beside a long read) matches our numbers: during a cold 32K read, the worst stream stopped for 8.4 s without Not tested by us: HIP, the batched draft path, a pack with |
|
@havok2-htwo I tried it: #3103 (opt-in Why the two were exclusive. In The change. Before each concurrent read (next to On 2x RX 6900 XT: 9 arms, 0 stalls; long read + short request, two streams, hand-off and a needle across the yield point all fine (short TTFT 8.0-8.1 s). Prompt read +4.0% at 4K, +1.8% at 32K, +0.9% at 128K (3 interleaved pairs, identical outputs run to run). The catch is What I cannot test, and where your rig would help: the stream-k path that the reservation is actually for. If you can, a run on your 4x 3090 with Developed with an AI coding assistant; every number above was measured on 2x RX 6900 XT (gfx1030), ROCm 10.0, Linux. |
|
Thanks for the pull request. Your change is already in If part of your work did not make it in and you want it considered, open a new pull request on top of |
…osted, and the handover gains the corrections Draft C was written from L-011 and L-012 and all three comments were posted on 2026-10-24 02:08Z: A to Niko1221#2275 (the gate predicate question), B and C to Niko1221#2197 (the seventh sm_75 box, then the mode-8 numbers). No pull request was opened - A asks the maintainers whether the predicate should change before one is offered. Before posting, the threads were checked: Niko1221#2326 is closed (its change is already in main for 0.2.0-beta), Niko1221#2197's last comment is rafatxf questioning which draft path the DFlash2 row actually ran, Niko1221#3724 is still open with no comments. Draft B no longer says EXP_MODE was unmeasured, and Draft A now states plainly that its two arms were the same binary with the variable set or unset, in the gate config rather than the production shape. Handover corrections a reader would otherwise act on wrongly: - S1 is not "measured but not in production": those two lines are in the engine binary that is running now, and concurrent reads happen every day (L-013). The open question is the production-shape trade and whether to turn the variable off, not whether to ship the change. - the sampler's hit-rate column is thinner than the service log's per-request lines; hit rate is judged from the service log from now on (L-012). - MainPID is the python server, not the engine: its environ holds no STRATA_* at all, so reading it reads "the environment did not apply". Read the engine child instead. - mtimes alone cannot pin which source a binary came from (cp -p preserves them); reverse-patch and compare hashes. - the sudo password question was asked together with a new GitHub token and the password itself was not answered, so it stays open. mode 8 was approved for production and the old goal is superseded.
Title
Issue: Related: #2275, #1906 and #1603 (on the pipelined groups every decoding stream stops for a whole prompt read),
and #2050 (the other approach, reads in pieces; this PR does not depend on it).
Motivation
I run Strata as the backend for coding agents on 2x RTX 5090. Every new agent starts with a long cold prompt, and with
--batch-groups> 1 every agent that is already decoding stops until that prompt is read.vLLM and SGLang avoid this with chunked prefill: a new prompt's chunks share batches with the running requests' decode
steps. #2050 brought the time-sliced form of that here (the read in pieces, taking turns with decode windows), but
every turn drains the pipeline, so the new prompt waits longer, and with a coarse chunk one piece is still about 21 s
of reading (superniker on #2050). Strata's prompt path and its decode windows use different kernels and buffers, so
they cannot share one forward pass, but they can run at the same time on the same GPUs. That is this PR.
Summary
With
STRATA_PIPED_CONCURRENT=1(opt-in, experimental) a cold prompt is read on a thread of its own while the slotgroups keep decoding. With the variable unset nothing changes: main's tokens (SHA-256 equal) at main's speed.
On 2x RTX 5090, four streams decoding while a 64K and a 32K cold prompt join:
The whole workload finishes sooner because the GPUs stay busier during a read; on main they sit partly idle while every
stream waits. The price is the joining prompt's own first token, which comes 20-47% later.
How it works
pump) and takesBSTOPandBYIELDmeanwhile.allocated or freed on the read's threads (everything is sized before the read starts).
BYIELDends the read at its next chunk, and the part read waits in a slot, as serve: cooperative prefill preemption at safe chunk boundaries (--prefill-preempt) #656's give-way does.the host thread's one-core mask; Windows' CPU Sets are not inherited).
Where the two rules come from, and what an independent review found
Two hangs turned up while building this, both read from the engine's stall dumps against its PDB. Both had #31's
shape: the main thread waited for the CUDA driver's lock in
cudaSetDevicewhile the lock's holder waited for the GPU(a
cudaFreein the drafter's prompt pass, then cuBLASLt's lazy load of a GEMM module), and a window on that GPUwaited for its doorbell. A doorbell served by a thread that never calls CUDA cannot be held up that way. An independent
review then found five more things, fixed in the third commit: configurations where serving a doorbell does call CUDA
(the mode is now off there), #1996's pinned upload rings growing mid-read (now reserved first), a window that would
never finish going unnoticed, the watchdog switching itself off when the last slot finished mid-read, and a rare race
that could serve a new window with the previous one's row count.
How to review
src/program/generate.cpp,read_part: the block underif (piped && pr_conc && req_imgs.empty()), thenpump'stwo changes (the watchdog during a read, the launch that wakes the doorbell thread).
src/core/verify.cpp:batch_serveisbatch_poll's per-layer service moved out unchanged.concurrent_read_off(batch_read.hpp, tested inbatch_read_test)lists the configurations where the mode stays off.
Results
Measured on 2x RTX 5090 (Swift 1.5 IQ3_S, split 25,
--batch 6 --batch-groups 2), engines built from maina0116b27.reads against 509 (main 293).
overlap.
--batch 2 --batch-groups autoon a real agent client: 9 full prompt re-reads (608 s of reading) in 23 minutes, sibling group decode falls to 4-13 tok/s #1906): the read and the decoding now compete. The read gives upabout a sixth of its speed, the streams keep 22-29% of their rate instead of almost none, and the engine uses 16.6
cores against 6.3. VRAM stays flat.
a 105K conversation.
no error; memory stayed flat.
ed4773eb, batch-groups: a cold prompt read no longer pauses every stream for its whole length, longest pause 12.2 -> 4.2 s on 2x RTX 5090 (opt-in) #2050's bench): a cold 32K join stops the streams 0.7 s against6.1-6.5 s on main and 2.9-3.1 s with batch-groups: a cold prompt read no longer pauses every stream for its whole length, longest pause 12.2 -> 4.2 s on 2x RTX 5090 (opt-in) #2050's 16K pieces, and they emit 925-1,065 tokens during it against 222-229
and 814-860. The first version kept the read's threads on the host thread's core there (6.1 rows/s on blange48's
4x RTX 5080); the sixth commit fixes that.
stall and no hang.
Tables and method
Against #2050 (a cold 59.6K prompt at 6 s and a 33.7K one at 12 s, the same prompts in every arm):
With a second prompt under half the first one's length (72.4K and 28.1K), the long read gives way at its next chunk by
#656's rule, here at 57,344 of 72,368 tokens: the short prompt gained little (7.4 against main's 7.1 s) and the long
one waited for it (17.2 against 8.3 s). A rule that compares what is left to read, as @blange48 carries on #2170, would
skip a give-way that late; that belongs in its own PR.
The same work, sooner. The four streams send the same prompts in every run, greedy, and their answers came out the
same length (3,828-3,899 tokens on main, 3,831-3,865 here). Finishing time is the last stream's last token. Order main,
this mode, this mode, main. Both arms decode at about 330 tok/s before the joins; main then stops the streams for about
12.8 s, while this mode keeps them near 90 tok/s through a 15.7 s read, which puts it 1.5-2 s ahead. GPU busy is
nvidia-smi's utilization every 100 ms during the read period. The gain depends on how much of the time is spentreading; with no reads there is none.
Host contention (the cold 64K prompt read alone, then beside the decoding streams; read speeds from the engine's
line for the whole request):
16 GB cards (
--vram-reserve-mib 16600 --kv-resident 16384): the streams' rate during the reads 0.9 -> 52.9 tok/s,all four done 70.6 -> 63.8 s, GPUs busy 80% -> 92%, the joining prompts' first token 9.5 / 10.1 -> 15.8 / 21.1 s.
A weaker host (16 GB cards, experts from the mapped file, AVX2, 6 pool workers,
--batch 2): a stream's longeststop 11.1 -> 0.41 s beside a 64K read and 20.3 -> 0.64 s beside a 127K one, the prompt's first token +4-7%.
Linux (WSL2, Ubuntu 24.04, the same machine and engine, all builds on main
ed4773ebwith #2050's commits appliedfor its arm;
piped_read_join_bench.py's cold32k, followup and two_joiners; the n-gram table in RAM in every arm, sinceWSL's disk reads it about 7x slower; 2-3 runs each):
The slots decoded 24-37 rows/s during a 30K+ read with the first version and 95-127 with the fix.
Exactness. With the usual settings near-tie tokens flip a little more often than on main (the prompt path borrows
some of the decode's GPU expert slots, and those experts run on the CPU kernels, which round differently). With
--pcie-frac 0 --adapt-swaps 0 --prompt-cache 0 --no-prefill-borrowthis mode's two runs were byte-identical for allthree answers; release's two runs with the same settings still differed. In
batch_interleave_test.py --long 40000every request equaled its solo tokens in two runs of three; the third had a near tie that flips on main as well. With
the variable unset, six greedy answers' SHA-256 equal main's in two engine starts each.
Load. The hour: four streams, cold joins of 4-128K every 4-15 s, short questions, follow-ups, clients leaving
mid-read; GPU0's VRAM grew 82 MiB over 80 minutes (in my local testing an engine grew 44 MiB in its first hours and
then held). My local testing ran v0.1.43 with #2050 and this ported; against the 20 hours before it on #2050's pieces,
the slowest tenth of long reads fell from 15.8 to 7.5 s, but the workload differed, so that is not a controlled A/B.
Costs and limits
BYIELD).--pcie-mode dma, the helper caches (--expert-cache-device1..3), anexpert source that recycles its buffers (the GGUF in place, unbuffered,
STRATA_IO_PREFETCH),STRATA_FS_SLOTS,STRATA_PREFILL_HELPandSTRATA_SPLIT_MTP_BATCH(--remote-expert-optwithout a helper cache does not count).Prompts with pictures are read the usual way.
What changed
src/program/generate.cpp: the concurrent read, the doorbell thread and the log lines;pumpkeeps the watchdog onduring a read and wakes the doorbell thread at a launch.
src/core/verify.cpp,include/strata/core/verify.hpp:batch_serve(shared withbatch_poll) andset_external_serve; the window state as atomics; the windows' streams at high priority under the opt-in.src/prefill/prefill.cpp,include/strata/prefill/prefill.hpp:stop_at_chunk,reserve_uploads.src/core/mtp.cpp,include/strata/core/mtp.hpp:reserve_prefill.include/strata/core/pinned_ring.hpp:PinnedRing::reserve.src/kernels/ngram.cpp: a lock inPleTable.src/core/expert_source.cppand its header: atomic read counters.include/strata/program/batch_read.hpp,tests/program/batch_read_test.cpp:concurrent_read_offand its test.include/strata/platform/aux_cpus.hpp:place_off_hostandrelease_host_inheritors(Linux), the read's threadsoff the host thread's core.
docs/BATCHING.md: the section and the tables.tools/piped_read_join_bench.py: batch-groups: a cold prompt read no longer pauses every stream for its whole length, longest pause 12.2 -> 4.2 s on 2x RTX 5090 (opt-in) #2050's measuring client,unchanged.
Extra Notes
both in, this mode reads where it can run and batch-groups: a cold prompt read no longer pauses every stream for its whole length, longest pause 12.2 -> 4.2 s on 2x RTX 5090 (opt-in) #2050's pieces cover the rest. The pieces suit configurations where this
mode stays off and cost the host less; this mode keeps the streams going and the read at full speed.
296b59d2), where it builds and its unit test passes; the Windows numbers above arefrom
a0116b27, the Linux ones fromed4773eb.batch_read_testpasseson all three. SYCL is not built here.