Repository navigation
Conversation
For the pipelined verify windows of a layer split (next commit). Nothing calls it yet, and nothing changes unless set_force_capture is set before load. - chain_launch: the catch-up round and its steps launched back to back with no host wait, an event after each output (chain_outputs_ready reads them as they land, chain_poll once the chain is done). Its first steps can take given tokens as their input instead of their own picks (teacher forcing: the drafts of a window already in flight), through a one-thread kernel at the end of the round and step graphs that does nothing while its mapped token is -1; the outputs stay the chain's own picks. - prepare_chain / prepare_prefill: capture every graph and size the prefill's input records up front, so neither a later chain nor prefill(sync = false) waits for the device. - set_source_R: the rows a round reads, copied into the bound buffer first (the pipelined loop has one stage-1 verifier per window parity). - with forcing on, the drafter's stream gets the highest priority (CUDA; STRATA_MTP_PRIORITY=0: the default). draft() resets the forced tokens and copies the source rows the same way, so the serial loop's drafts are unchanged.
With a split the two cards take turns on every verify window: stage 0 runs its layers and hands off, then waits while stage 1 runs the rest, the head and the draft. The batch pipeline (--batch-groups) overlaps different conversations; one conversation's decode stayed serial. --pipeline-windows 2 runs window K+1 on stage 0 while stage 1 verifies window K. The guess for K+1 comes from the MTP chain, launched as soon as K's verdict is known and teacher-forced through K+1's drafts: if K is accepted as guessed, K+1 is already half done; if not, stage 0 restores its GDN state from a snapshot, replays K's commit with the real count, and K+1 is rebuilt from the real tokens. --pipeline-windows 1 overlaps only the prompt's short reads through the windows (--short-read). Default 0: nothing changes. What it adds: - Verifier: a non-blocking form of run()/commit() (capture_all, pl_launch, prestage, service, done, pl_finish, pl_commit_async), the batch pipeline's batch_launch/batch_poll pattern for one sequence. The window, its staging, the PLE gather at layer 0, the zero-doorbell graph's flag, the Niko1221#267 release on a timeout and the commit graph are run()'s. Two verifiers per stage (one per window parity) share one stream of their card (set_stream) and each pair has its own hand-off. - set_always_publish: the pipelined verifiers copy the token rows to the host at every layer, because the adaptive tier now marks experts evicted on the host while windows are in flight, before the device residency table follows. - MtpDrafter: chain_launch (the round and its steps back to back, an event after each output, the first steps fed the in-flight window's drafts by a one-thread forcing kernel), prefill without a host sync, 8 rows with 2, the drafter's stream at the highest priority (CUDA only). - the adaptive tier beside the pipeline: evictions are marked first, then the refill copies wait (on the tier's thread) for every window in flight; the RAM side of a swap (exchange, release, unpin) waits for the windows that may still pull the incoming experts over PCIe. - an agreement gate: a speculative window is launched only when the chain's estimate says it is likely kept (STRATA_PIPELINE_THETA, default 0.2); a window known to be wrong gets empty plans for its remaining layers; a lookup window's continuation becomes the speculative window behind it. - VRAM: 160 MiB more kept out of each card's cache for the second verifier, and the two GDN snapshots on the first card. - off, with one log line saying why, for anything it does not support: --batch slots, --peer-device, the helper caches, three or more stages, a split on one GPU, no drafter. Requests with repetition penalties or coupled draft sampling decode serially. Nothing is in flight outside the two pipelined loops, so checkpoints, parking and the serial paths are unchanged. Measured: Swift 1.5 IQ3_XXS at 160K (q4_0 KV, the stock draft layer), RTX 4060 Ti (layers 0-19) + RTX 5080 (20-47), i9-14900KF, 32 GB of RAM, the resident RAM mode on the split (Niko1221#848); greedy, 500 tokens, two interleaved pairs of three rounds, decode tok/s, serial -> pipelined: Python code 90.4 -> 103.1, C code 76.6 -> 81.6, English prose 63.0 -> 71.1, Italian prose 41.6 -> 46.3, a copy-heavy edit (a 5 KB file back with a rename) 90.5 -> 117.9; mean 72.4 -> 84.0 (+16%). With the experts read through the OS file cache (--mmap-experts on that PC) the file reads dominate and it measured no faster. Correctness: with STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0 the greedy text is identical to the serial loop's when both have the same expert caches (4/4 prompts, 500 tokens; the serial run given the pipeline's reserve through --vram-reserve-mib). The same text with STRATA_PIPELINE_THETA=2 (no speculative window), STRATA_PIPELINE_FORCE_MISS=1 (every speculative window wrong, rolled back) with and without DOOM_SKIP, PRESTAGE=0 and SNAP_OVERLAP=0. Against a run without the flag a few experts move from a card to the CPU (the pipeline's VRAM), which rounds them differently, so a near-tie can flip. With the adaptive tier on: 22 varied requests at temperature 0.6 (thinking on and off, five languages, code), and 11 more with STRATA_PIPELINE_FORCE_MISS=2, gave no degenerate answer, no stall. The watchdog prints the pipelined loop's state when it fires; STRATA_PIPELINE_TRACE=<file> writes a timeline.
|
A data point from the shape this PR cannot reach: 4 stages on 4x V100-16GB (sm_70).
We applied the PR cleanly on v0.1.39 as a dry run (10 files, +1601 -33, no rejects) and it coexists with the #845 patch (offset 92 lines), so a 3+ stage version would be testable for us. Say the word and we will run it. |
|
Thanks, both points are useful. The two-stage limit is deliberate for now. With two stages, the speculation only has to guess one window ahead on one card, and rolling back means restoring one card's GDN state. With three or more stages, each extra stage adds another window in flight, another snapshot, and a longer rollback chain. That's where it gets subtle, so I wanted the two-stage case solid first. Your numbers also put a ceiling on it for your rig: 0.24 ms per hand-off and caches already holding 5973 of 6144 pairs on CUDA3. If the bubble is that small and VRAM is that tight, 160 MiB per card plus the snapshots might cost more than they return. On a 4060 Ti + 5080 pair the stage-0 wait was ~10 ms of a ~30 ms window, so there was much more to win. I'll keep your offer in mind. If I generalize it, I'll make the per-card VRAM cost configurable and ping you for the 4x V100 run. |
Tested on 2x gfx1201 (RDNA4, HIP/Linux):
|
| v0.1.39 | + --pipeline-windows 2 |
||
|---|---|---|---|
| decode, 1200 tok (2 runs) | 79.6 / 75.8 | 81.0 / 84.5 | +7% |
| decode, 1500 tok | 80.8 | 93.0 | +15% |
| prompt, 40,920 tok | 1,192 tok/s | 1,316 tok/s | +10% |
| prompt, 232,080 tok | 1,158 tok/s | 1,333 tok/s | +15% |
| needle at 232K (record 7777) | correct | correct | |
| R9700 power while busy | 198 W | 235 W |
The power line is the clearest sign it does what it says: stage 0 stops idling while stage 1 verifies. On the serial split both cards sat ~100 W under what the same R9700 draws on a dense 27B.
Exactness: with the flag the greedy text (temperature 0, seed fixed) diverges from the serial loop at the third word ("respond" vs "answer") - consistent with your note about the 309 MiB kept out of CUDA0's cache moving a few experts to the CPU. With STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0 on both sides the first divergence moves to char 163 of ~4.4K, but I could not make the caches equal: CUDA0 gets 13,582 vs 13,285 slots and the split's CUDA1 cache is auto-sized (4,132 vs 4,145) with no flag to pin it, so I can't confirm or refute bit-exactness on HIP. Answers stayed correct in everything I ran.
Running it as my daily server now; happy to test follow-ups on this pair.
…RATA_PIPELINE_DEBUG=1, exclusive with --adapt-async, merged with batch MTP and the async tier Co-authored-by: Francesco Albano <119073350+Hardin22@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Thanks, taken into 0.1.40 as opt-in (--pipeline-windows, default 0) with your commits. Changes: every STRATA_PIPELINE_* debug hook is read only under STRATA_PIPELINE_DEBUG=1, it excludes --adapt-async, and it is merged with batch MTP and the resident exchange work. It needs two GPUs and #848 to show its gain. We have one GPU, so the release notes ask two-GPU testers for 10 or more interleaved pairs, with and without #848. |
--pipeline-windows runs its own overlapped decode loop with its own hand-offs, so STRATA_SPLIT_OVERLAP now switches itself off when it is set (one log line says so). Rebased onto main 1735d64: the early launch of the next stage keeps Niko1221#871's graph variant (the next stage's refresh_ar() runs before its capture and prelaunch, and not again in its run() when it was prelaunched), resets the plan error word like run()'s launch, and the chain path checks that word after its sync as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…RATA_PIPELINE_DEBUG=1, exclusive with --adapt-async, merged with batch MTP and the async tier Co-authored-by: Francesco Albano <119073350+Hardin22@users.noreply.github.com>
|
Interleaved pairs on 0.1.40 (1735d64), as asked in the release notes. Setup. RTX 4060 Ti 16 GB (stage 0, layers 0-19) + RTX 5080 16 GB (stage 1, layers 20-47), i9-14900KF, 32 GB DDR5, Windows 11. Swift Qwen3.8-Flash-Next IQ3_XXS, 160K context, q4_0 KV, Method. One server per arm, started with With #848 (
Mean 92.1 serial vs 98.5 pipelined: ratio x1.074 mean, x1.055 median, pipelined faster in 7 of 12 pairs. The spread within a pair is large because the adaptive tier keeps moving experts between requests on one server, so the cache a request sees depends on what ran before it. Without #848 ( Pipeline and async tier together. 0.1.40 excludes |
…ehind STRATA_PIPELINE_DEBUG=1, exclusive with --adapt-async, merged with batch MTP and the async tier Co-authored-by: Francesco Albano <119073350+Hardin22@users.noreply.github.com>
--pipeline-windows runs its own overlapped decode loop with its own hand-offs, so STRATA_SPLIT_OVERLAP now switches itself off when it is set (one log line says so). Rebased onto main 1735d64: the early launch of the next stage keeps Niko1221#871's graph variant (the next stage's refresh_ar() runs before its capture and prelaunch, and not again in its run() when it was prelaunched), resets the plan error word like run()'s launch, and the chain path checks that word after its sync as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eline-windows (Niko1221#859) layer split: overlap the stages' hand-off (STRATA_SPLIT_OVERLAP=1, opt-in) Each verify window of a layer split runs its stages strictly in turn: stage 1 finishes, the host syncs its stream, then stages and launches stage 2's graph. Under HIP the sync and the graph launch both sit on the critical path of every window. With STRATA_SPLIT_OVERLAP=1 the hand-off carries a ready word (mapped, one per hand-off buffer): - the writing stage's graph ends with handoff_publish: the payload with volatile stores (RDNA keeps plain stores to mapped memory in L2), a system fence per thread, and the last block raises the flag; - the reading stage's graph starts with wait_flag_ge(flag, 1); - Verifier::run captures the next stage's window on this thread, then stages and launches it on a helper thread (prelaunch) while this thread serves its own layers; it then serves the next stage's layers directly and syncs the earlier stage once the chain is done; - release_gpu_waits also raises the hand-in flag; an early return after the prelaunch releases the next stage. Batch windows and --split-device 0 keep the serial order. Off by default. Measured on 2x RX 7900 XTX (ROCm, Linux), Swift 1.5 IQ3_XXS, 512-token greedy answers, median of 8: decode 53 -> 70.1 tok/s (code prompt), 45 -> 55.9 tok/s (prose prompt); greedy output byte-identical with the flag at 0 and 1; quiz 47/51 both ways. Not measured on NVIDIA. layer split overlap: bound the GPU-side hand-off wait (STRATA_SPLIT_WAIT_MS) Review on Niko1221#936 (justxiami): the reading stage's wait_flag_ge spun with no bound, so a lost publish would leave that stream spinning. - wait_handoff: the reading stage's graph now waits with a bound (default 30 s, STRATA_SPLIT_WAIT_MS, 0 = unbounded, GPU clock as in gpu_stamp). On timeout it sets a mapped error word and returns, so the graph completes; the host checks the word after the window and fails it with an error that names the setting, instead of using a stale hand-off. - The host's per-layer wait (20 s, Niko1221#267) still normally ends a stalled window first; release_gpu_waits already raises the hand-in flag. - Test hook STRATA_TEST_HANDOFF_DROP=N: the first stage's N-th window withholds its flag (a lost publish), to exercise both paths. Only active with STRATA_SPLIT_OVERLAP=1; the default path is unchanged. layer split overlap: the hand-off publish reads the test word only with STRATA_TEST_HANDOFF_DROP set Without the hook the window graph passes no word, so the publish kernel does no extra read of mapped memory per window. layer split overlap: off beside --pipeline-windows (Niko1221#859) --pipeline-windows runs its own overlapped decode loop with its own hand-offs, so STRATA_SPLIT_OVERLAP now switches itself off when it is set (one log line says so). Rebased onto main 1735d64: the early launch of the next stage keeps Niko1221#871's graph variant (the next stage's refresh_ar() runs before its capture and prelaunch, and not again in its run() when it was prelaunched), resets the plan error word like run()'s launch, and the chain path checks that word after its sync as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD
…ehind STRATA_PIPELINE_DEBUG=1, exclusive with --adapt-async, merged with batch MTP and the async tier Co-authored-by: Francesco Albano <119073350+Hardin22@users.noreply.github.com>
--pipeline-windows runs its own overlapped decode loop with its own hand-offs, so STRATA_SPLIT_OVERLAP now switches itself off when it is set (one log line says so). Rebased onto main 1735d64: the early launch of the next stage keeps Niko1221#871's graph variant (the next stage's refresh_ar() runs before its capture and prelaunch, and not again in its run() when it was prelaunched), resets the plan error word like run()'s launch, and the chain path checks that word after its sync as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--pipeline-windows runs its own overlapped decode loop with its own hand-offs, so STRATA_SPLIT_OVERLAP now switches itself off when it is set (one log line says so). Rebased onto main 1735d64: the early launch of the next stage keeps #871's graph variant (the next stage's refresh_ar() runs before its capture and prelaunch, and not again in its run() when it was prelaunched), resets the plan error word like run()'s launch, and the chain path checks that word after its sync as well.
--pipeline-windows runs its own overlapped decode loop with its own hand-offs, so STRATA_SPLIT_OVERLAP now switches itself off when it is set (one log line says so). Rebased onto main 1735d64: the early launch of the next stage keeps Niko1221#871's graph variant (the next stage's refresh_ar() runs before its capture and prelaunch, and not again in its run() when it was prelaunched), resets the plan error word like run()'s launch, and the chain path checks that word after its sync as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
A three-stage sm_70 data point, since the V100 numbers in this thread are either four-stage (the gate refused it) or two-stage. Setup
MethodOne full engine restart per arm; 3 reps x 3 legs per arm; 256 generated tokens; Results (tok/s)
|
Refs #642. These are the pipelined windows from the fork, ported onto main and opt-in (
--pipeline-windows 2; the default 0 changes nothing).With a layer split, one conversation's decode is serial: stage 0 runs its layers and hands off, then waits while the last card runs its layers, the head and the draft. The batch pipeline from #559 overlaps different conversations, so a single conversation gets nothing from it.
--pipeline-windows 2starts window K+1 on stage 0 while stage 1 still verifies window K.STRATA_PIPELINE_THETA).Two commits:
MtpDrafter: a teacher-forced chain as one asynchronous launch. Nothing calls it on its own, and
draft()is unchanged for the serial loop.The pipelined loop: a non-blocking form of the Verifier's
run()/commit()on thebatch_launch/batch_pollpattern, two verifiers per stage, the adaptive tier fenced against the windows in flight, and the VRAM for the second verifier and the snapshots. It turns itself off, with one log line, for anything it does not handle:--batch;--peer-device;--remote-expert-opt;Requests with
penalty_last_nor coupled draft sampling decode serially.Measured
Swift 1.5 IQ3_XXS at 160K (q4_0 KV, stock draft layer). RTX 4060 Ti (layers 0-19) + RTX 5080 (20-47), i9-14900KF, 32 GB of RAM, with the resident RAM mode on the split from #848. Greedy, 500 tokens, two interleaved serial/pipelined pairs of three rounds each, decode tok/s:
--pipeline-windows 2It pays when the windows are GPU-bound. On the same PC without #848 (experts read through the OS file cache,
--mmap-experts), the file reads dominate and it was not measurably faster.Correctness
STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0, the greedy text is identical bit for bit to the serial loop's, as long as both have the same expert caches: 4/4 prompts, 500 tokens. The serial run was given the pipeline's reserve through--vram-reserve-mib.STRATA_PIPELINE_THETA=2(no speculative window at all);STRATA_PIPELINE_FORCE_MISS=1(every speculative window wrong and rolled back), with and withoutSTRATA_PIPELINE_DOOM_SKIP=0;STRATA_PIPELINE_PRESTAGE=0;STRATA_PIPELINE_SNAP_OVERLAP=0.STRATA_PIPELINE_FORCE_MISS=2.Not tested
#if !defined(STRATA_USE_HIP), and the new forcing kernel is plain code.service()only raises the stage flag.Happy to split it differently or change the defaults if you prefer.