Repository navigation
fix: preserve pipeline token equality on 3x P4 (23 -> 28 tok/s) - #1674
Closed
CC-David-CC wants to merge 1 commit into
Closed
CC-David-CC wants to merge 1 commit into
CC-David-CC wants to merge 1 commit into
Conversation
Contributor
|
@CC-David-CC Thanks for finding this. Here is a second CUDA host for the equality check, including the longer contexts on your not-tested list. The rig is 4× GP100 (2× PH402 SKU 200, sm_60) on four stages (
What this does not show: I never ran #1656 without this commit, so I can't say the guard was needed here, only that with it the replies match the serial loop. There was one run per arm. Suffix drafting stayed at its default ( |
This was referenced Oct 9, 2026
This was referenced Oct 9, 2026
Owner
|
Ported with your guard. On 2xA4000, a 2-stage layer split with --pipeline-windows 2 vs 0 and frozen placement: before the guard 3 of 3 greedy answers differ from windows 0, with it 1 of 3 matches and 2 still differ, so the guard helps but the pipelined windows are still not token-equal here. Is there another fix in #1656 that it needs? |
Contributor
Author
|
i misread you yes i will work on it |
Contributor
Author
|
Its exact patch is already upstream as c81f998. Niko’s remaining A4000 mismatch is a separate follow-up. |
lask3802
added a commit
to lask3802/Strata-Link
that referenced
this pull request
Oct 13, 2026
The pipelined decode loop now runs with the worker in its last stage: while the worker and the main process's head verify window N, layers 0 to K-1 already run window N+1 on the guess that N is accepted whole and its bonus is the drafter's. A wrong guess is undone on the main process (GDN snapshot). The worker only ever gets verified windows, in the serial loop's order: no new flag, build or protocol on the worker. Uses Niko1221#1656's N-stage loop (layers 0 to K-1 are its one front stage, the head on the worker's rows its last stage) and Niko1221#1674 (a pipelined verifier leaves a one-token window's state update to the commit). - Verifier: pl_launch of a stage set with set_remote starts remote_run on a job thread of its own instead of blocking; service() launches the window's head graph once the worker's rows are back, and a pl_commit_async made before then is queued behind that launch. The worker's one-way commit goes out after the reply. The last stage has two head verifiers (ver_same and ver_same_b) on CUDA0 beside stage 0, on a stream of their own. - generate: the head stage is the last stage of Niko1221#1656's per-stage bookkeeping (stage_ver_p returns ver_same / ver_same_b), so its stream, odd hand-off, plans, capture, drain, fences and stats go through the same per-stage loops as a local stage's. Layer splits without a remote stage take Niko1221#1656's path unchanged. - When the engine ends after an error while a job still waits for the worker, the verifier shuts the link's socket (StageClient::abort, a third function to set_remote) and joins the job: never detached, so it cannot touch the link or the job after they are gone. - generate: --pipeline-windows is accepted on --remote-stage's main process and refused on a worker (the refusal at start for a remote main is lifted). The check that turns it off for a layer split on one GPU (split_same) and without a second GPU leaves a remote main alone; the automatic default (-1) still resolves to 0 there, so it stays opt-in. 2 x 160 MiB plus the GDN snapshots, 444 MiB in all at K=26, are kept out of CUDA0's expert cache. - docs/REMOTE_STAGE.md: a section on it (what it keeps out of the cache, how to compare its answers with the serial loop's byte for byte, and what that comparison measured), and the refusal list. One sentence each in remote-stage/ENGINEERING.md (decode is sequential unless it is asked for) and MULTI_GPU.md (a remote main takes it on one card). What stays: without --pipeline-windows (or with 0) nothing here runs: a remote main's windows go through remote_run on the request's own thread as before, and the cache keeps its size. Carried over from the same change on an earlier version of this branch (0.1.43, before its watchdog, reconnect and hello commits). The engine code is that change's, unaltered; two textual conflicts with this branch, both kept side by side: StageClient::abort beside the new alive() and the connect with a time-out and a stop flag (stage_link.hpp, stage_link.cpp), and the refusal list in docs/REMOTE_STAGE.md beside the new wording of what is turned off. generate.cpp merged without a conflict. Measured on that earlier build (RTX 3080 main + RTX 2080 Ti worker, 10 GbE, one-token windows; docs/REMOTE_STAGE.md): with the same --expert-cache on both sides, the first replay on a freshly started server gave the same 8 answers byte for byte in 6 of 7 runs (--pipeline-windows 0 three times, 2 four times; the other run 5 of 8, at near-tie tokens), and one server switched between 2, 0, 1 and 2 with forced misses gave them each time. Run on GPUs on this branch afterwards (decode speed and answers: docs/REMOTE_STAGE.md). Checked here: stage_link_test with MSVC /W4 and g++ -Wall -Wextra, and the HIP (gfx1100) build of strata.
lask3802
added a commit
to lask3802/Strata-Link
that referenced
this pull request
Oct 20, 2026
The pipelined decode loop now runs with the worker in its last stage: while the worker and the main process's head verify window N, layers 0 to K-1 already run window N+1 on the guess that N is accepted whole and its bonus is the drafter's. A wrong guess is undone on the main process (GDN snapshot). The worker only ever gets verified windows, in the serial loop's order: no new flag, build or protocol on the worker. Uses Niko1221#1656's N-stage loop (layers 0 to K-1 are its one front stage, the head on the worker's rows its last stage) and Niko1221#1674 (a pipelined verifier leaves a one-token window's state update to the commit). - Verifier: pl_launch of a stage set with set_remote starts remote_run on a job thread of its own instead of blocking; service() launches the window's head graph once the worker's rows are back, and a pl_commit_async made before then is queued behind that launch. The worker's one-way commit goes out after the reply. The last stage has two head verifiers (ver_same and ver_same_b) on CUDA0 beside stage 0, on a stream of their own. - generate: the head stage is the last stage of Niko1221#1656's per-stage bookkeeping (stage_ver_p returns ver_same / ver_same_b), so its stream, odd hand-off, plans, capture, drain, fences and stats go through the same per-stage loops as a local stage's. Layer splits without a remote stage take Niko1221#1656's path unchanged. - When the engine ends after an error while a job still waits for the worker, the verifier shuts the link's socket (StageClient::abort, a third function to set_remote) and joins the job: never detached, so it cannot touch the link or the job after they are gone. - generate: --pipeline-windows is accepted on --remote-stage's main process and refused on a worker (the refusal at start for a remote main is lifted). The check that turns it off for a layer split on one GPU (split_same) and without a second GPU leaves a remote main alone; the automatic default (-1) still resolves to 0 there, so it stays opt-in. 2 x 160 MiB plus the GDN snapshots, 444 MiB in all at K=26, are kept out of CUDA0's expert cache. - docs/REMOTE_STAGE.md: a section on it (what it keeps out of the cache, how to compare its answers with the serial loop's byte for byte, and what that comparison measured), and the refusal list. One sentence each in remote-stage/ENGINEERING.md (decode is sequential unless it is asked for) and MULTI_GPU.md (a remote main takes it on one card). What stays: without --pipeline-windows (or with 0) nothing here runs: a remote main's windows go through remote_run on the request's own thread as before, and the cache keeps its size. Carried over from the same change on an earlier version of this branch (0.1.43, before its watchdog, reconnect and hello commits). The engine code is that change's, unaltered; two textual conflicts with this branch, both kept side by side: StageClient::abort beside the new alive() and the connect with a time-out and a stop flag (stage_link.hpp, stage_link.cpp), and the refusal list in docs/REMOTE_STAGE.md beside the new wording of what is turned off. generate.cpp merged without a conflict. Measured on that earlier build (RTX 3080 main + RTX 2080 Ti worker, 10 GbE, one-token windows; docs/REMOTE_STAGE.md): with the same --expert-cache on both sides, the first replay on a freshly started server gave the same 8 answers byte for byte in 6 of 7 runs (--pipeline-windows 0 three times, 2 four times; the other run 5 of 8, at near-tie tokens), and one server switched between 2, 0, 1 and 2 with forced misses gave them each time. Run on GPUs on this branch afterwards (decode speed and answers: docs/REMOTE_STAGE.md). Checked here: stage_link_test with MSVC /W4 and g++ -Wall -Wextra, and the HIP (gfx1100) build of strata.
kgbrah
pushed a commit
to kgbrah/Strata
that referenced
this pull request
Oct 24, 2026
The pipelined decode loop now runs with the worker in its last stage: while the worker and the main process's head verify window N, layers 0 to K-1 already run window N+1 on the guess that N is accepted whole and its bonus is the drafter's. A wrong guess is undone on the main process (GDN snapshot). The worker only ever gets verified windows, in the serial loop's order: no new flag, build or protocol on the worker. Uses Niko1221#1656's N-stage loop (layers 0 to K-1 are its one front stage, the head on the worker's rows its last stage) and Niko1221#1674 (a pipelined verifier leaves a one-token window's state update to the commit). - Verifier: pl_launch of a stage set with set_remote starts remote_run on a job thread of its own instead of blocking; service() launches the window's head graph once the worker's rows are back, and a pl_commit_async made before then is queued behind that launch. The worker's one-way commit goes out after the reply. The last stage has two head verifiers (ver_same and ver_same_b) on CUDA0 beside stage 0, on a stream of their own. - generate: the head stage is the last stage of Niko1221#1656's per-stage bookkeeping (stage_ver_p returns ver_same / ver_same_b), so its stream, odd hand-off, plans, capture, drain, fences and stats go through the same per-stage loops as a local stage's. Layer splits without a remote stage take Niko1221#1656's path unchanged. - When the engine ends after an error while a job still waits for the worker, the verifier shuts the link's socket (StageClient::abort, a third function to set_remote) and joins the job: never detached, so it cannot touch the link or the job after they are gone. - generate: --pipeline-windows is accepted on --remote-stage's main process and refused on a worker (the refusal at start for a remote main is lifted). The check that turns it off for a layer split on one GPU (split_same) and without a second GPU leaves a remote main alone; the automatic default (-1) still resolves to 0 there, so it stays opt-in. 2 x 160 MiB plus the GDN snapshots, 444 MiB in all at K=26, are kept out of CUDA0's expert cache. - docs/REMOTE_STAGE.md: a section on it (what it keeps out of the cache, how to compare its answers with the serial loop's byte for byte, and what that comparison measured), and the refusal list. One sentence each in remote-stage/ENGINEERING.md (decode is sequential unless it is asked for) and MULTI_GPU.md (a remote main takes it on one card). What stays: without --pipeline-windows (or with 0) nothing here runs: a remote main's windows go through remote_run on the request's own thread as before, and the cache keeps its size. Carried over from the same change on an earlier version of this branch (0.1.43, before its watchdog, reconnect and hello commits). The engine code is that change's, unaltered; two textual conflicts with this branch, both kept side by side: StageClient::abort beside the new alive() and the connect with a time-out and a stop flag (stage_link.hpp, stage_link.cpp), and the refusal list in docs/REMOTE_STAGE.md beside the new wording of what is turned off. generate.cpp merged without a conflict. Measured on that earlier build (RTX 3080 main + RTX 2080 Ti worker, 10 GbE, one-token windows; docs/REMOTE_STAGE.md): with the same --expert-cache on both sides, the first replay on a freshly started server gave the same 8 answers byte for byte in 6 of 7 runs (--pipeline-windows 0 three times, 2 four times; the other run 5 of 8, at near-tie tokens), and one server switched between 2, 0, 1 and 2 with forced misses gave them each time. Run on GPUs on this branch afterwards (decode speed and answers: docs/REMOTE_STAGE.md). Checked here: stage_link_test with MSVC /W4 and g++ -Wall -Wextra, and the HIP (gfx1100) build of strata.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
3x Tesla P4: 23.0 -> 28.6 tok/s with exact output
At 4,096 prompt tokens, prose decode improved from 23.000485 to 28.606868 tok/s (+24.375063%) with two-window pipelining. This is the same-day serial-versus-pipeline comparison on #1656 plus this guard; the pipeline speedup comes from #1656.
Minimal correctness fix
Add
!always_publish_to the one-token self-commit condition inVerifier::record_window()and its matching shortcut inVerifier::commit(). Pipelined verifiers must leave recurrent state updates to the host verdict/snapshot and explicit commit path. Otherwise a one-token window can advance state during verification and then advance it again when the pipeline commits it.The complete code diff is two guards and one explanatory comment in
src/core/verify.cpp. Ordinary serial verifiers retain their existing shortcut. No scheduler, kernel, configuration default, or benchmark files change.Found while testing #1656. The mismatch also reproduced on its exact parent,
fb58e0dbc8399662c0e47c76578c6e878b14f6cf, with a two-stage split, so it predates that PR. The guard is extracted from the earlier three-P4 work in #1154, specifically dd7045d.Measurements
Measured 2026-10-09 on 3x Tesla P4, dual Xeon E5-2697 v3, 251.77 GiB usable RAM, CUDA 12.0, driver 580.178.04. Model: Qwen3.8-Flash-Next-GSQ-RCO IQ3_XXS, int8 KV, split
19,37, 40,960-token context limit, 32,768 resident KV, 27 CPU workers, greedy decoding, MTP--spec 2 --spec-min-p 0.5, suffix drafting off. Static expert placement:STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0.Both arms use the same corrected binary and expert placement, switching
pw=0/pw=2per request. Three measured repetitions per arm; median engine decode throughput below. One initial serial/pipeline code warmup pair excluded. Order reversed for the middle repetition; zero reused prompt tokens throughout. Model loading excluded.4K prose median whole-request latency: 33.755897 -> 31.510369 seconds, a 6.652252% reduction. Decode throughput uses generated tokens divided by engine decode time, not whole-request time. A separate results-only PR will carry all per-run values, ranges, prompts, and reproduction files.
Validation and scope
sm_61build succeeded on pipeline-windows: three or more layer-split stages, +14% decode at 4K / +27% at 32K on 4x RX 7900 XT (opt-in); HIP draft-layer scratch race #1656 head06abdfa50aef6e5068869bc7552fafa5e99b7b9dplus exactly this patch.