Repository navigation
Conversation
…mpt's experts A prompt that fits one chunk runs a layer split's stages one after the other, so while one card reads its layers the other idles. --peer-device already has the mechanism for this (PeerPrefill's peer streaming: a share of the experts the primary streams goes over the peer's own PCIe link and is computed there), but it needs P2P and is refused with --layer-split. This makes PeerPrefill usable as a stream-only peer between the stages: - set_stage_helper: each stage's prompt path gets the next stage's (the last one the first's); a stream-only PeerPrefill holds no experts (peer == null), so every use of the peer tier is guarded - per prompt, bind_stage_helper points it at the idle stage's own lent prompt buffers (mixed, Xq, GU, H, Hq, Dm, ring, MMQ context): no VRAM of its own - without P2P the activations, the row table, the group bounds and the result rows go through mapped host memory, read by copy kernels on the device that needs them. Copy-engine uploads queued behind the ring's expert blobs (the grouping tables' reason for mapped memory), which cost ~20 ms per layer and ate the whole gain; cudaMemcpyPeerAsync without P2P is staged by the driver - only for one-chunk prompts on the MMQ path (not the fused kernels, not with --peer-device), with a share that falls with the chunk: min(0.45, 0.5 - T/16384), off below 0.3 (~3.3K tokens) - measured, below - STRATA_PREFILL_HELP=0 turns it off, STRATA_PREFILL_HELP_FRAC fixes the share 2x RTX 3090 (no P2P), UD-Q4_K_XL, layer split auto, 262K context, --adapt-swaps 0, prompt tok/s without / with: 1.1K 496 / 723, 1.5K 674 / 833, 2K 878 / 1128, 2.5K 1017 / 1296, 3K 1260 / 1440, 4K 1605 / 1607, 8K 2215 / 2221; decode unchanged. The output is repeatable (same prompt, same answer) but not bit-identical to STRATA_PREFILL_HELP=0's: the helped rows are computed in another MMQ grouping.
Owner
|
Thanks! Taken into 0.1.39 as opt-in, as you offered: STRATA_PREFILL_HELP=1 turns it on, and STRATA_PREFILL_HELP_FRAC works when it is on. The helped rows round differently, and we keep the default byte-identical. docs/MULTI_GPU.md says so. We have one GPU, so the 2x 3090 numbers are yours. |
eelgaev
added a commit
to eelgaev/Strata-AC922
that referenced
this pull request
Oct 4, 2026
217 upstream commits (0.1.38 -> 0.1.39: Niko1221#646 decode kernels and zero-doorbell verify graph, Niko1221#603 long-context top-k, Niko1221#583 byte-budget ring, Niko1221#598 pipelined layer split, Niko1221#578 helper GPUs, Niko1221#663, Niko1221#540, Niko1221#600 in the experimental build, batch slots Niko1221#465, the SYCL port, older-CPU builds, ...). 23 files conflicted (~70 hunks); every hunk was resolved by hand and checked against both sides. Resolution rules: - The prompt path's stage overlap: upstream's Niko1221#598 (equivalent to our e85f7ee; run() drains the whole chain on every exit), with our release_inherited_pin in the stage threads and our early ids copy for the host grouping (a5a3254) kept. - STRATA_FP16=load's form-aware weights kept wherever upstream's new code assumed BF16: the shared-expert gate (sigmoid+scale fused as upstream), GDN alpha/beta (our kernels; upstream's new gdn_step_commit_kernel added), the hc v3 kernels (upstream's EXACT_T with our dot8w). - verify.cpp: upstream's all-resident fast path, plan copy and Niko1221#578 rows, with our 2-GPU fetch overlap and partner fetch back in the non-resident branch; mapped buffers stay portable (the partner reads them). - Volta prompt attention: upstream's dispatch (the m8n8k4 kernel on sm_70 in this build) with our split hi/lo accumulator chains inside it; the wmma kernel stays opt-in (STRATA_ATTN_WMMA=1 without STRATA_ATTN_M884). - qsa_decode_attn.cu: upstream's (our adf56cc imported the same PR Niko1221#540; upstream's is it plus review fixes, bit-exact). - POWER / aarch64: upstream's new x86 intrinsics in verify.cpp through the portable strata_store_fence / strata_cpu_pause; its AVX2 start-up checks and log line behind x86 guards; stubs for the new CPU probes; the router lookahead keeps portable.cpp's VSX dot on non-x86 (upstream's dispatch would have sent POWER to its scalar fallback); the pool's ThreadAffinity (any CPU count) with our note_host_pin; SMT2 selection with Niko1221#642's sort. - setup.py / tests: both sides (CUDA 12 engine, ISA floor, ARM / non-x86). Checked: - every line either side added since 0.1.38 is in the merge or replaced on purpose (script over all 45 files both sides changed); - all parity tests pass (iq, native_expert on the real shards, mmvq_multi, gr, gdn, gdn_rec, qsa, qsa_topk, elementwise, router_dot, shared_expert, kv_*, sampler, rope, s2/s_gemv, gemm_bf16, ...); - teacher-forced 2K and 32K logs byte-identical to the pre-merge build (4 GPUs, launch config); - 4 GPUs: decode text identical, same windows and acceptance, 28.25 -> 26.9 ms/window (~86 -> ~90 tok/s); prompts 7.8K 3,992 -> 4,038, 65K 7,102 -> 7,172, 123K 7,323 -> 7,334 tok/s; --prefill-ring 320 applies; - 2 GPUs: decode texts identical to the pre-merge binary on three prompts, decode 61.5-71.0 -> 63.5-72.7 tok/s, 65K prompt 3,975 -> 4,083. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZHWKWn7UYZV64KwpGrWuN
Contributor
|
Feedback on a 4-way split (rig: 4 x RTX 5080 16 GB (PCIe Gen3, one card at x8), Linux, IQ3_S, 262K context, int8 K/V with
|
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A prompt that fits one chunk runs a layer split's stages one after the other: while one card reads its layers, the other idles.
--peer-devicealready has the mechanism to use such a card -PeerPrefill's peer streaming (a share of the experts the primary streams goes over the peer's own PCIe link and is computed there) - but it needs P2P and is refused with--layer-split. Most consumer pairs have no P2P (two RTX 3090 here:nvidia-smi topo -p2p rsaysCNS).This makes
PeerPrefillusable as a stream-only peer between the stages:Prefill::set_stage_helper: each stage's prompt path gets the next stage's (the last one the first's). A stream-onlyPeerPrefillholds no experts (peer == nullptr); every use of the peer tier is guarded.bind_stage_helper, per prompt: points it at the idle stage's own lent prompt buffers (mixed,Xq,GU,H,Hq,Dm, its ring, its MMQ context). No VRAM of its own;serve'slend()lays every stage's buffers out at once before the prompt.copy_f32_wide, 16-byte loads). My first version used copy-engine uploads; they queued behind the ring's expert blobs - the same reason the grouping tables live in mapped memory - and cost ~20 ms per layer, which ate the whole gain (thecombinephase went 24 -> 407 ms).cudaMemcpyPeerAsyncwithout P2P is staged by the driver.--peer-device), with a share that falls with the chunk -min(0.45, 0.5 - T/16384), off below 0.3 (~3.3K tokens) - from the sweep below.STRATA_PREFILL_HELP=0turns it off,STRATA_PREFILL_HELP_FRAC=ffixes the share.docs/MULTI_GPU.mddescribes it.The
--peer-devicepath is unchanged (itsctxbecomesrun_ctx, a pointer, so the helper can run in the idle stage's context).Measured
2x RTX 3090 (PCIe 4.0 x16 each, no P2P), EPYC 7413, Unsloth UD-Q4_K_XL, layer split auto, 262K context,
--kv int8 --pcie-frac 0 --ple-io ram --adapt-swaps 0, synthetic prompts (a repeated paragraph, unique per run) over/v1/chat/completions, 3 runs each, same binary:STRATA_PREFILL_HELP=0Decode unchanged. The share sweep behind the rule (
STRATA_PREFILL_HELP_FRAC0.2 / 0.3 / 0.4 against none): 1.5K 749 / 820 / 908 (675), 3K 1,335 / 1,437 / 1,328 (1,262), 4K 1,622 / 1,620 / 1,440 (1,609), 6K 1,811 / 1,660 / 1,471 (2,038), 8K 1,895 / 1,804 / 1,663 (2,230).Output
Repeatable: the same prompt gives the same answer, run after run (
--adapt-swaps 0, four fixed prompts of 1.2-3.1K tokens, twice). It is not bit-identical toSTRATA_PREFILL_HELP=0's: the helped rows are computed in another MMQ grouping and round differently - likeSTRATA_SPLIT_OWN. I made it default-on because the default (adaptive swaps on) is not bit-repeatable across requests anyway and the gain is large for the short prompts agents send; if you prefer it opt-in likeSTRATA_SPLIT_OWN, that is a one-line change and I'm happy to flip it.Not tested: a P2P pair (the helper always takes the mapped route), HIP, three or more stages (the code takes the next stage's card; only two were available).