Repository navigation
Conversation
…the streamed walk Under WDDM every cudaEventRecord and every cross-stream wait is a submission of its own (~80 us and ~45 us on an RX 6900 XT under Windows 11; ~3 us and ~0 on Linux). The streamed walk's issuer recorded a copied event and waited on the slot's release event for every expert, although the consumer gathers an MMQ group after one wait on its last copy and releases the group with one event. STRATA_PREFILL_COPY_EVENT_EVERY=N records the copied event every N-th entry (and before the issuer waits on the ring or the stager, and on the last entry); the consumer waits on the first recorded entry at or after the one it needs. STRATA_PREFILL_COPY_WAIT_DEDUPE=1 skips the copy stream's wait on a slot's release event when the previous slot (in the same walk, from the second ring round) was released by the same event. Both are off by default; the default path issues the same calls as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Oct 24, 2026
Vop3p
added a commit
to Vop3p/Strata
that referenced
this pull request
Oct 25, 2026
…at --prefill 8192 The same local build and settings as ...-c97e2108 with --prefill 8192 instead of auto (6,144), so the last step is comparable with the package run at 8,192 (...-0e04f3ea): complete, 12/12, prompt 4K 808.5 / 16K 1,125.1 tok/s, decode 43.9 / 45.5 / 45.3. README, results.json, config.json, outputs.jsonl and provenance; engine log left out. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Vop3p
marked this pull request as ready for review
October 25, 2026 03:57
Contributor
Author
|
The Those are macOS 26 SDK symbols, and the runner builds with Marking it ready for review. It has still not run on NVIDIA, SYCL or Metal hardware (CUDA compile only; see "Not run" in the description), and the switches it adds are off by default. |
This was referenced Oct 27, 2026
Niko1221
pushed a commit
that referenced
this pull request
Oct 27, 2026
…he streamed prompt walk (#4005) Under WDDM every cudaEventRecord and cross-stream wait is a submission of its own (the author measured ~80 / ~45 us each on an RX 6900 XT under Windows 11), and the streamed walk paid one record and one wait per expert although a group gathers after one wait on its last copy. Two tester switches, threaded issuer only: - STRATA_PREFILL_COPY_EVENT_EVERY=N (2..64): the copied event is recorded every N-th entry, before the issuer waits and on the last; the compute stream waits on the first recorded entry at or after the one it needs (the copy stream is in order). - STRATA_PREFILL_COPY_WAIT_DEDUPE=1: the copy stream skips its wait on a slot's release event when the previous slot was released by the same event. The same copies into the same slots: the same bytes. Default path (both unset): read line by line. The issuer records copied[slot] and waits on used[used_of[slot]] exactly as before (the counters are bookkeeping, cover() is a no-op). Added to the PR so this holds by construction: wait_copied takes the slot the old code used and, without the sparse switch, waits on copied[slot] at all three consumer sites (the PR derived the slot from the entry number). The record/wait selection is now three pure functions in include/strata/prefill/copy_events.hpp, with a host test (copy_events_test, registered in CMakeLists.txt) that models the issuer and consumer with a full ring and checks that every wait names a live recorded entry at or after the one needed. Tests: copy_events_test passes (z440, g++ -std=c++20). Not yet: a CUDA build of prefill.cpp in this tree (queued), no GPU run. Not covered: the Windows HIP / WDDM speed claim and NVIDIA (the gate is written down). Needs the author's numbers repeated before any default talk.
Niko1221
added a commit
that referenced
this pull request
Oct 27, 2026
…released in the current walk and the ring is longer than one group (#4005) Review fix on the #4005 port. STRATA_PREFILL_COPY_WAIT_DEDUPE=1 skipped the copy stream's wait on a slot's release event when the previous entry had waited on the same event. Two cases break that: - At idx == ring the previous entry (ring - 1) is still in the first round, so its wait was on the event of an earlier walk. The same slot id can be this walk's first flush, which re-records the event after that wait; skipping the wait for slot 0 then lets the next copy overwrite a slot that the flush's gather has not finished reading. - With a ring of one MMQ group (16, the smallest ring) a flush of the whole ring re-records the event right after the previous slot's wait. The rule is now copy_wait_dedupe_ok() in include/strata/prefill/copy_events.hpp: idx > ring, ring > group, same event. copy_events_test models two to three walks with random group sizes and random issuer/consumer interleaving and checks that a refill always waits on a record at or after the flush that released its slot. The same model with the PR's original rule (idx >= ring) finds the stale-event case, so the test can fail. The default path is unchanged (dedupe off). Tests: copy_events_test passes (z440, g++ -std=c++20). Not covered: a GPU run.
Owner
|
Thank you. Fixed in 0.2.7: Opt-in sparse copied events and deduplicated slot waits in the streamed prompt walk. Your name is on the commit. Release: https://github.com/Niko1221/Strata/releases/tag/v0.2.7 |
fragtion
added a commit
to fragtion/StrataPool
that referenced
this pull request
Oct 27, 2026
… forwarded to the workers - The coordinator's CONFIG carries "env": the pool-wide opt-ins set on it (kPoolForwardEnv: Niko1221#4005's STRATA_PREFILL_COPY_EVENT_EVERY / _COPY_WAIT_DEDUPE, Niko1221#3779/Niko1221#3712's STRATA_PARK_ASYNC / STRATA_CKPT_ASYNC / STRATA_CKPT_GATHER, and STRATA_ROUTE_PRIOR). A worker applies each one it has not set itself, before its first prompt, park, checkpoint or verifier reads it, and logs "strata pool: from the coordinator: ...". STRATA_DEFAULTS=026 is not forwarded (a worker reads it before CONFIG) and the VRAM/RAM-sized switches stay per PC. - <config>.pool.json takes "env": {STRATA_*: value} (checked: up to 32, STRATA_* names, short values without ; or newlines), applied to the engine of either split role through pool_env; the Pool tab's save keeps it. - Niko1221#4010's STRATA_PIPE_ADAPT_EVERY start-up line also on a pool coordinator (the pipelined groups drain the workers' stages before a round; the workers already swap between the windows they serve). - Checked for a worker: the async park's device copy is synchronised before the park returns (only the copy to RAM runs on), so a following prompt cannot overwrite what is being copied. - docs/POOL.md: "Opt-in switches for a split" (what each does, pool-wide or per PC, how to set them). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD
riceharvest
added a commit
to riceharvest/strata-b70-only
that referenced
this pull request
Oct 30, 2026
…ion at 0.1.40.1) dpct 2025.3 of the 0.1.40.1 tree (23a38a0) and of 870aedd, normalize.sh, a 3-way merge into sycl/ per docs/INTEL.md (src/dsv41 left out: b70 keeps its own DeepSeek port). Upstream's history restarts at 424b364, whose SYCL copies were still at 0.1.40.1, so the base is the last re-migration, not the merge base. 38 merged files conflicted (about 520 hunks), plus the hand-kept copies (verify, mtp, conversation_state, remote/peer experts, direct_file); every conflict resolved with the port's kernels, launch shapes and B70 defaults kept. In (opt-in unless noted): layer-major prefill (STRATA_PREFILL_LAYER_MAJOR, FP16 expert path on SYCL), PLE residency policy (default as upstream: table in RAM only with table + arena + 8 GiB free), PLE_COMPACT / PLE_COPY_STREAM, DEQUANT_SIDE, KVA, SELECT_LONG (multi-group decode top-k + dx scorer), SELECT_SIMT, QSA_ROWS / slots, SPEC_COST, SPEC_PROB, PIPED_MTP, logit bias / allowed_token_ids / logprobs in the sampler, conversation cache (PARK_ASYNC, CKPT_ASYNC, CKPT_POOL_READY, PARK_VRAM_MIB, Niko1221#1503 Niko1221#2409 Niko1221#3350 Niko1221#4420), expert source file-tier prefetch (IO_PF_*, FS_*), RAM_ADAPT, copy_blobs, graph eviction on out-of-memory (Niko1221#3702), stager waits and copy-wait dedupe (Niko1221#4005), PLE reader fix (Niko1221#3602), cache reserve (Niko1221#4474), QFUSE commit fix (Niko1221#3948), 9-16 token windows (capability; kVerifyMaxT stays 8), Gyro-L TQP5/6/7, EXPDEC, IQ_STAGE_TABLES, GU_SWIGLU, HC_PERSIST, GR_UP_WIDE, B6_MMVQ_ROWS, ROUTE_PRIOR (in strata_kernels now), LOGITS_DUMP, NAN_GUARD, TRACE. Off on SYCL: PDL (pdl.hpp: no Level Zero counterpart), PF_WMMA (gemm_iq_f16_grouped stub), PATTN_SM60, the sm_60..121 / gfx / GP100 tables and kernels, NVML, DFlash2 (--draft dflash2 refused), NVFP4, Qwen3.6, argmax_rows (a dpct major version is not a compute capability), route tail skip default (decode_defaults_arch_ok false). Upstream headers b70 had removed are back in include/strata (host-only). Checked: every changed translation unit compiles (JIT), strata links.
bitaccesscomau
added a commit
to bitaccesscomau/Strata
that referenced
this pull request
Oct 31, 2026
…reamed experts no token of the chunk routes to; test A prompt chunk of 1,024 tokens or more streams every non-resident expert of every layer through the ring, and the issuer thread enqueues each copy as soon as the ring has room. The ring is longer than a layer's entries, so a layer's copies are all on the copy stream before its router has run: an expert no token routes to is copied anyway. With the switch on, for a streamed chunk on one GPU with the issuer thread: - the issuer keeps at most STRATA_PREFILL_ROUTED_STREAM_DEPTH copies (default 16) queued ahead of the link: it waits until the copy issued that many copies earlier has landed (cudaEventQuery on its slot's copied event). The ring's own rule is unchanged, so the stream still runs as far ahead of the compute as before; - after each layer's host grouping the compute thread publishes which experts at least one token routes to; - the issuer drops an entry it has not issued yet whose layer is published and whose expert has no token. Entries already issued stay. An entry staged through the Stager's pinned buffers is never dropped (its job queue is in order). Every routed entry is still copied, in the same order, into the same ring slot (k % ring), with the same slot waits and copied event; the consumer's walk is unchanged and never reads an unrouted entry's slot, and the copy stream stays in order, so a group's single wait on its last copy still covers the group. The prompt's products are the same bits. After the read the borrowed slots are refilled from RAM blob by blob into the same slots, independent of what the ring held. On v0.2.6 (4b9e05a) at the release defaults (RTX 3090, PCIe 3.0 x16, 250 W; the same binary with the switch off and on, 4 pairs): a 1,790-token follow-up read on an 18.8K-token conversation took 17.9% less time, fresh reads of 2K, 4K and 32K tokens ran 20%, 8% and 5% faster, and the streamed experts fell by 23%, 16% and 4%. In a deterministic lane the replies are byte-identical to that base at 1K to 32K with the switch off and on, and the switch dropped 42% of the streamed experts at a 1,046-token read and 3% at 31K tokens. Unset, nothing new runs: no gate, no publication, and the default issuer loop is not edited. The fused path, the peer prompt ring, the inline issuer (STRATA_PREFILL_ISSUER=0) and the staged walk are untouched. With STRATA_TRACE=1 each chunk prints the ring entries it copied and dropped. The issuer walk and the route gate are header-only (include/strata/prefill/routed_stream.hpp); routed_stream_test runs the walk itself against a simulated in-order link and a consumer that walks the layers as prefill.cpp does (every routed entry issued into its own slot and read with its own bytes, only unrouted entries of published layers dropped, never more than the depth queued, never past the ring), and on three threads with real atomics. STRATA_PREFILL_COPY_EVENT_EVERY > 1 and STRATA_PREFILL_COPY_WAIT_DEDUPE=1 (Niko1221#4005) keep the default issuer: the routed walk records a copied event for every copy, but it does not publish the records a sparse consumer waits on (a_rec, rec_of) and does not deduplicate release waits, so with either set the consumer would wait forever or the wait policy would be bypassed. routed_stream::eligible holds the rule and routed_stream_test checks it. Those two settings are now read before the switch's own, which moves their two declarations up; nothing else in the default issuer loop changes.
utombords
pushed a commit
to utombords/Strata
that referenced
this pull request
Nov 1, 2026
…reamed experts no token of the chunk routes to; test A prompt chunk of 1,024 tokens or more streams every non-resident expert of every layer through the ring, and the issuer thread enqueues each copy as soon as the ring has room. The ring is longer than a layer's entries, so a layer's copies are all on the copy stream before its router has run: an expert no token routes to is copied anyway. With the switch on, for a streamed chunk on one GPU with the issuer thread: - the issuer keeps at most STRATA_PREFILL_ROUTED_STREAM_DEPTH copies (default 16) queued ahead of the link: it waits until the copy issued that many copies earlier has landed (cudaEventQuery on its slot's copied event). The ring's own rule is unchanged, so the stream still runs as far ahead of the compute as before; - after each layer's host grouping the compute thread publishes which experts at least one token routes to; - the issuer drops an entry it has not issued yet whose layer is published and whose expert has no token. Entries already issued stay. An entry staged through the Stager's pinned buffers is never dropped (its job queue is in order). Every routed entry is still copied, in the same order, into the same ring slot (k % ring), with the same slot waits and copied event; the consumer's walk is unchanged and never reads an unrouted entry's slot, and the copy stream stays in order, so a group's single wait on its last copy still covers the group. The prompt's products are the same bits. After the read the borrowed slots are refilled from RAM blob by blob into the same slots, independent of what the ring held. On v0.2.6 (4b9e05a) at the release defaults (RTX 3090, PCIe 3.0 x16, 250 W; the same binary with the switch off and on, 4 pairs): a 1,790-token follow-up read on an 18.8K-token conversation took 17.9% less time, fresh reads of 2K, 4K and 32K tokens ran 20%, 8% and 5% faster, and the streamed experts fell by 23%, 16% and 4%. In a deterministic lane the replies are byte-identical to that base at 1K to 32K with the switch off and on, and the switch dropped 42% of the streamed experts at a 1,046-token read and 3% at 31K tokens. Unset, nothing new runs: no gate, no publication, and the default issuer loop is not edited. The fused path, the peer prompt ring, the inline issuer (STRATA_PREFILL_ISSUER=0) and the staged walk are untouched. With STRATA_TRACE=1 each chunk prints the ring entries it copied and dropped. The issuer walk and the route gate are header-only (include/strata/prefill/routed_stream.hpp); routed_stream_test runs the walk itself against a simulated in-order link and a consumer that walks the layers as prefill.cpp does (every routed entry issued into its own slot and read with its own bytes, only unrouted entries of published layers dropped, never more than the depth queued, never past the ring), and on three threads with real atomics. STRATA_PREFILL_COPY_EVENT_EVERY > 1 and STRATA_PREFILL_COPY_WAIT_DEDUPE=1 (Niko1221#4005) keep the default issuer: the routed walk records a copied event for every copy, but it does not publish the records a sparse consumer waits on (a_rec, rec_of) and does not deduplicate release waits, so with either set the consumer would wait forever or the wait policy would be bypassed. routed_stream::eligible holds the rule and routed_stream_test checks it. Those two settings are now read before the switch's own, which moves their two declarations up; nothing else in the default issuer loop changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch:
Vop3p/Strata:prefill-copy-events(one commit onmain7dd5cf1). Not run on NVIDIA, SYCL or Metal hardware (see Not run). No default changes.Problem and result
On Windows (WDDM) every
cudaEventRecordand every cross-streamcudaStreamWaitEventis a submission of its own. A small HIP probe on an RX 6900 XT (Windows 11, driver 32.0.21045.16010, ROCm 10.2.0a20260930 runtime) against the same card model on Linux (ROCm 10.0), 2.79 MB host-to-device copies from registered memory, µs per copy:cudaEventRecordon the copy streamcudaStreamWaitEventon another streamBandwidth is not the gap: back-to-back copies reach 12.5 GB/s (5.5 µs fixed cost per copy) against 14.1 GB/s (7.5 µs) on Linux.
The streamed walk of the prompt path pays one record and one wait per expert on the copy stream: the issuer records
copied[slot]after every copy and waits on the slot's release event before every copy. The consumer, though, gathers an MMQ group of 16 experts after one wait on its last copy and releases the whole group with one event, so 15 of every 16 records are never waited on, and the copy stream waits on the same release event up to 16 times in a row. WithSTRATA_PREFILL_TIMING=1on Windows this shows aswait copy2.1 s of a 4K prompt (Linux: 0.08 s).This PR adds two opt-in switches. Both off (the default), the walk issues exactly the calls it issued before.
STRATA_PREFILL_COPY_EVENT_EVERY=N(2..64): the issuer records the copied event every N-th entry, and also before it waits (ring full, or the stager behind) and on the last entry; the consumer waits on the first recorded entry at or after the one it needs. The copy stream is in order, so a later entry's event covers the earlier copies.STRATA_PREFILL_COPY_WAIT_DEDUPE=1: the copy stream skips the wait on a slot's release event when the previous slot was released by the same event (one flush releases a whole group), from the second ring round of the walk on.Result on the RX 6900 XT under Windows 11, the same build with the switches off vs
EVERY=16+DEDUPE=1, 10 cold-start pairs at the default--prefill auto(6144 on Windows HIP, #1630):EVERY=16 DEDUPE=1(median of 10)Order B SD SD B, five times, a fresh engine per run; each run is the median of two requests per size (
benchmark.py-style synthetic prompts, a new nonce per request). Decode 44.5 / 44.5 (4K) and 44.2 / 46.2 (32K, different continuations). Every setting both arms share:--prefill auto(6144 here),--expert-cache auto,--vram-reserve-mib 1536,--kv int8 --kv-resident 32768,--max-context 65536,--spec 4 --spec-min-p 0.5,--pcie-frac 0.2,STRATA_IO_THREADS=128,STRATA_HIP_PROMPT_F16=1,STRATA_HIP_ADAPT_KERNEL_COPY=1,STRATA_PREFILL_CPU_SHARE=auto,STRATA_HC_PERSIST=1,STRATA_PIPED_CONCURRENT=1,STRATA_PREFILL_STREAM_MIN=2048, a rocBLAS table tuned on this PC; the display on the iGPU. Every run's numbers are in the table at the end.Decode is unchanged (it does not use this path). On Linux the switches change nothing measurable (below).
Related discussion
--prefill 8192/automakes the commit charge jump ~70 GB in one second, the engine dies at the commit limit (0xC0000409); 7500 and below stay flat #1630 (why--prefill autostops at 6144 on Windows HIP) is why the main table is at 6144; a supplement at--prefill 8192is below.Approach
Only the threaded issuer (
STRATA_PREFILL_ISSUERon, the default) and only the streamed walk change; the staged walk (chunks belowstream_all_min()) and the inline issuer keep their per-slot events.Sparse records: the issuer keeps
rec_of[slot](the entry the slot's event marks) anda_rec(one past the last recorded entry).wait_copied(k)spins untila_rec > k, then waits on the firstrin[k, a_rec)withrec_of[r % ring] == r. That slot cannot be re-recorded beforekis given back, because the issuer stays at most one ring ahead of what was given back andkis not given back yet. The issuer records before every wait of its own, so a consumer never waits on an entry nobody will record. All three consumers ofcopied[]in the streamed walk (the group flush, the per-expert path when group gather is off, the fused path) go throughwait_copied.Deduplicated waits: within one walk, from the second ring round on, two consecutive slots that point at the same release event were released by the same flush (a flush releases contiguous slots, and its event is the group's last slot, which is not re-recorded before the issuer reaches it), so the wait for the previous slot already covers this one.
Compatibility: no file format, API or default changes; the switches are read once. They are meant for WDDM; on Linux they measure as no-ops.
Validation
Passed:
setup.pyHIP options (-DSTRATA_PREFILL_MMQ=ON -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON, plus tests), Qwen3.8-Flash-Next IQ3_S pack, reproducible settingsSTRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0 STRATA_PREFILL_CPU_SHARE=0, temperature 0, 4K and 32K prompts x2:main7dd5cf1 vs this branch with the switches off vs this branch withEVERY=16 DEDUPE=1: all 4 outputs identical across the three builds. The confirmation linestrata: prompt copies: a copied event every 16 streamed experts ..., one wait per release eventprinted;STRATA_PREFILL_TIMINGreports ~14,100 of ~20,700 slot waits skipped per 32K prompt.ctest -j 1on gfx1030 (Linux, the branch): 128 passed, 13 skipped (not applicable to gfx1030), 3 failed; the same 3 fail onmain7dd5cf1 with the same output (see Failed).tools\hip\build_windows.bat(gfx1030, ROCm 10.2.0a20260930) builds; the A/B above;strata bench run(standard track) on this branch with the switches on: 21/21 requests, sanity 12/12 (instruction 4/4, retrieval 4/4, tool calls 4/4), prompt 4K 793 / 16K 979 tok/s.strata_prefillcompiles (-DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89).Failed (also on
main7dd5cf1, not caused by this change):ple_parity: needs../../Q2_0/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00002-of-00002.gguf, not on this machine.hip_prefill_mmq_parity: segfault insidelibamdhip64.so.7at the launch ofmmvq_multi_kernel<29,2,true,true>(fromsynthetic_q2).isa_guard_check:iq512x_rowsholds AVX-512 instructions and is not listed; the same class as [Engine] isa_guard_check fails on Linux builds: tq_rows_avx512 (tq_avx512.cpp) is not in WIDE_PUBLIC / WIDE_OBJECTS #3409.Not run:
sycl/CMakeLists.txtcompilesprefill.cpp).Limits
--prefill 8192(3 pairs) the same switches give +35.8% to +37.4% at 4K but only +3.0% to +3.9% at 32K (1,070-1,072 -> 1,103-1,112 tok/s).EVERY=16matchesMMQ_GROUP; other values were not swept.Every run (Windows 11, RX 6900 XT, in run order)
Each cell is the median of the run's two requests of that size. No run failed or was dropped.
Developed with an AI coding assistant; every number above was measured on the RX 6900 XT machines named in it (Windows 11 / i9-10900K and Linux / Ryzen 5 5600X).
🤖 Generated with Claude Code