Repository navigation
Conversation
…reamed experts no token of the chunk routes to; test A prompt chunk of 1,024 tokens or more streams every non-resident expert of every layer through the ring, and the issuer thread enqueues each copy as soon as the ring has room. The ring is longer than a layer's entries, so a layer's copies are all on the copy stream before its router has run: an expert no token routes to is copied anyway. With the switch on, for a streamed chunk on one GPU with the issuer thread: - the issuer keeps at most STRATA_PREFILL_ROUTED_STREAM_DEPTH copies (default 16) queued ahead of the link: it waits until the copy issued that many copies earlier has landed (cudaEventQuery on its slot's copied event). The ring's own rule is unchanged, so the stream still runs as far ahead of the compute as before; - after each layer's host grouping the compute thread publishes which experts at least one token routes to; - the issuer drops an entry it has not issued yet whose layer is published and whose expert has no token. Entries already issued stay. An entry staged through the Stager's pinned buffers is never dropped (its job queue is in order). Every routed entry is still copied, in the same order, into the same ring slot (k % ring), with the same slot waits and copied event; the consumer's walk is unchanged and never reads an unrouted entry's slot, and the copy stream stays in order, so a group's single wait on its last copy still covers the group. The prompt's products are the same bits. After the read the borrowed slots are refilled from RAM blob by blob into the same slots, independent of what the ring held. On v0.2.6 (4b9e05a) at the release defaults (RTX 3090, PCIe 3.0 x16, 250 W; the same binary with the switch off and on, 4 pairs): a 1,790-token follow-up read on an 18.8K-token conversation took 17.9% less time, fresh reads of 2K, 4K and 32K tokens ran 20%, 8% and 5% faster, and the streamed experts fell by 23%, 16% and 4%. In a deterministic lane the replies are byte-identical to that base at 1K to 32K with the switch off and on, and the switch dropped 42% of the streamed experts at a 1,046-token read and 3% at 31K tokens. Unset, nothing new runs: no gate, no publication, and the default issuer loop is not edited. The fused path, the peer prompt ring, the inline issuer (STRATA_PREFILL_ISSUER=0) and the staged walk are untouched. With STRATA_TRACE=1 each chunk prints the ring entries it copied and dropped. The issuer walk and the route gate are header-only (include/strata/prefill/routed_stream.hpp); routed_stream_test runs the walk itself against a simulated in-order link and a consumer that walks the layers as prefill.cpp does (every routed entry issued into its own slot and read with its own bytes, only unrouted entries of published layers dropped, never more than the depth queued, never past the ring), and on three threads with real atomics. STRATA_PREFILL_COPY_EVENT_EVERY > 1 and STRATA_PREFILL_COPY_WAIT_DEDUPE=1 (Niko1221#4005) keep the default issuer: the routed walk records a copied event for every copy, but it does not publish the records a sparse consumer waits on (a_rec, rec_of) and does not deduplicate release waits, so with either set the consumer would wait forever or the wait policy would be bypassed. routed_stream::eligible holds the rule and routed_stream_test checks it. Those two settings are now read before the switch's own, which moves their two declarations up; nothing else in the default issuer loop changes.
…streamed chunk keeps the default walk; test With the switch set, a streamed chunk keeps the default issuer with STRATA_PREFILL_ISSUER=0, on the peer prompt ring, and with STRATA_PREFILL_COPY_EVENT_EVERY > 1 or STRATA_PREFILL_COPY_WAIT_DEDUPE=1. That was silent, so a set switch read as active where it could not run. The engine now prints one line with the reason (routed_stream::held_off) the first time a streamed chunk is held off. Chunks below the stream floor print nothing, and with the switch unset nothing new runs. routed_stream_test checks that a reason is given exactly when the walk is held off and that it names the setting.
mgendre
changed the title
mgendre
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and result
This PR carries upstream PR 4283 of
Niko1221/StratabyStefano Sinden (GitHub:
bitaccesscomau) into this fork while it is open upstream. Fork only: it is not to beproposed upstream from here, since it is the author's own open PR. Both of the author's commits (4b6d0e7, d8ce2d9 here; 3956b94
and fcb0e39 in upstream PR 4283, its head on 2026-11-01) are cherry-picked unchanged, with the author kept.
What it does, in the author's terms: with
STRATA_PREFILL_ROUTED_STREAM=1(opt-in), a streamed prompt chunk (1,024 tokens ormore, one GPU, the threaded issuer) no longer copies the experts that no token of the chunk routes to. The issuer keeps at most
STRATA_PREFILL_ROUTED_STREAM_DEPTHcopies (default 16) queued ahead of the link; each layer's routing is handed to it once thelayer's router has run; an expert not yet copied that no token routes to is dropped. The routed experts go into the same ring
slots as before, so the results are the same bits. Unset, nothing changes.
The author's measurements are in upstream PR 4283 and in docs/DETAILS.md (RTX 3090, PCIe 3.0 x16).
Related discussion
Upstream: PR 4283 of
Niko1221/Strata(open). Not linked on purpose: this fork does not post to upstream for now.Approach
git cherry-pick 3956b94a fcb0e399ontofork-main(upstream v0.2.9): no conflict. The current version of upstream PR 4283 already keepsthe routed walk off with upstream PR 4005's
STRATA_PREFILL_COPY_EVENT_EVERY > 1orSTRATA_PREFILL_COPY_WAIT_DEDUPE=1(
routed_stream::eligible), so nothing of ours is added.upstream sync and this carry is dropped.
Validation
Passed:
g++-13 -std=c++20 -O2 -Wall -Wextra -Iinclude tests/prefill/routed_stream_test.cpp -pthreadand its run, revision d8ce2d9,Linux:
routed_stream_test: OK, no warning.g++-13 -fsyntax-onlywith the flags of a CUDA Release build:src/prefill/prefill.cppcompiles with the same 2 warnings asfork-main, no new one.Failed:
Not run:
strataat d8ce2d9 succeeded.Limits
deduplicated waits.