Skip to content

decode: STRATA_PCIE_BALANCE=1 picks each layer's PCIe count from measured costs (refile of #1038 by @sergqwer, on 0.1.40.3) - #1548

Closed
stuchapin909 wants to merge 1 commit into
Niko1221:mainfrom
stuchapin909:pr/refile-1038
Closed

decode: STRATA_PCIE_BALANCE=1 picks each layer's PCIe count from measured costs (refile of #1038 by @sergqwer, on 0.1.40.3)#1548
stuchapin909 wants to merge 1 commit into
Niko1221:mainfrom
stuchapin909:pr/refile-1038

Conversation

@stuchapin909

@stuchapin909 stuchapin909 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Summary

A refile of #1038 by @sergqwer, which GitHub closed when main's history was rewritten (not a rejection; your note there asked for a new branch from the new main). The commit is theirs, cherry-picked onto 0.1.40.3 with -x, authorship kept. Opt-in: STRATA_PCIE_BALANCE=1 picks each layer's PCIe count from measured CPU and GPU costs; unset, nothing changes. The method, the scope (one GPU, no batch slots, no --pcie-frac) and the original measurements are in #1038.

What changed

Nothing beyond #1038's commit, except one conflict: in include/strata/kernels/verify_kernels.hpp the comment above gpu_stamp changed on main (the PDL sentence) while #1038 added wait_flag_ge_stamped above it. Both are kept: the new declaration, then main's current gpu_stamp comment. wait_flag_ge_stamped launches like wait_flag_ge (no PDL), as on main.

Measured

The case #1038 did not cover: a strong CPU on a slow link, where the probe's share is too high. i9-9920X (Skylake-X, 12 cores, AVX-2 expert kernels), 96 GB DDR4, Windows 11, one RTX 3090 on PCIe 3.0 x16 (probe 12.3 GB/s -> pcie_frac 0.34), built with STRATA_PORTABLE=ON. IQ3_S (GSQ-RCO), expert cache 2,454 slots (4.71 GiB, --vram-reserve-mib 12300, about a 12 GB card), --spec 4 --spec-min-p 0.5 --kv k8v4 --kv-resident 32768 --max-context 262144, one prompt ("Explain how to compute 1719 + 2329 by hand, then give the integer."), greedy, 256 tokens, strata generate --tokens-file, a fresh process per run.

switch unset STRATA_PCIE_BALANCE=1
2026-10-07, #1038's head cherry-picked (alternating with --pcie-frac 0.25) 52.83, 51.90 tok/s 57.04, 57.41 tok/s
2026-10-08, this commit 52.51 tok/s 54.82, 55.75 tok/s
experts per layer read over PCIe 5.55-5.57 4.45-4.78

+9.2% and +5.3% over the probe share in the two sittings. The fitted model printed CPU 162-173 us + 57-65 an expert + 0 a PCIe one, GPU 145-178 us + 146-150 a PCIe expert. The best fixed share on this PC, by hand, is 0.25 (58.0-58.2 tok/s, 4.1 experts a layer over PCIe), so the switch gets most of the way there without tuning. The same tokens in every run.

On this PC's two-card layer split the share makes no measurable difference (automatic vs 0.25, 3 starts each), in line with the switch's one-GPU scope.

Extra Notes

@sergqwer: the change is yours; if you would rather refile it yourself, I will close this one.

🤖 Generated with Claude Code

sergiywith @claude @stuchapin909
decode: STRATA_PCIE_BALANCE=1 picks each layer's PCIe count from meas…
a513394 
…ured costs

The verify window reads pcie_frac of a layer's missed experts over PCIe, a share
the link probe sets; the CPU's speed does not enter it.  With a slow CPU the GPU
spins on the pool's rows: IQ2_XS with one AVX2 pool worker (emulated) runs
16.8-17.2 ms a round at the probe's 0.55.

Opt-in, STRATA_PCIE_BALANCE=1 (one GPU, no batch slots, no --pcie-frac):
- wait_flag_ge_stamped: the flag spin kernels also write the GPU clock as the
  plan arrives, as the PCIe part starts and as the GPU waits for the CPU's rows
  (no extra launch; captured only with the switch on);
- the host times its pool per step; after each window, least squares with
  forgetting fit the pool's a + c n_cpu + d n_pcie (d: the link's DRAM reads
  slowing the pool - on a DRAM-bound pool moving experts to PCIe saves nothing)
  and the GPU's g0 + g n_pcie;
- a layer then reads over PCIe the m of its misses that minimizes the longer
  side; every 64th layer that would keep them all on one side moves one, so
  both fits stay measured.

IQ2_XS, RTX 5090 + 9950X3D, chat to the end of the turn:
- one AVX2 pool worker: 16.77 / 17.17 -> 12.98 / 12.82 ms a round
  (fit: CPU 42-60 us an expert, GPU 28 us a PCIe expert; 0.8 of the misses
  move);
- the 9950X3D's 16 workers: 3 pairs 12.19 -> 12.17 (pool DRAM-bound, few move).
Off (the default) the logits and 32 tokens equal v0.1.39 (2K, fixed cache).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
(cherry picked from commit 4261775)
DarthCY-K added a commit to DarthCY-K/Strata that referenced this pull request Oct 9, 2026
@DarthCY-K
apply upstream PR Niko1221#1548 (on v0.1.41+Niko1221#1107+Niko1221#1367…
9b6123c 
…): decode STRATA_PCIE_BALANCE=1 (refile of Niko1221#1038)
Niko1221 added a commit that referenced this pull request Oct 17, 2026
@Niko1221 sergiywith
decode: STRATA_PCIE_BALANCE=1 picks each layer's PCIe count from meas…
04be241 
…ured costs

The verify window reads pcie_frac of a layer's missed experts over PCIe, a share
the link probe sets; the CPU's speed does not enter it.  With a slow CPU the GPU
spins on the pool's rows: IQ2_XS with one AVX2 pool worker (emulated) runs
16.8-17.2 ms a round at the probe's 0.55.

Opt-in, STRATA_PCIE_BALANCE=1 (one GPU, no batch slots, no --pcie-frac):
- wait_flag_ge_stamped: the flag spin kernels also write the GPU clock as the
  plan arrives, as the PCIe part starts and as the GPU waits for the CPU's rows
  (no extra launch; captured only with the switch on);
- the host times its pool per step; after each window, least squares with
  forgetting fit the pool's a + c n_cpu + d n_pcie (d: the link's DRAM reads
  slowing the pool - on a DRAM-bound pool moving experts to PCIe saves nothing)
  and the GPU's g0 + g n_pcie;
- a layer then reads over PCIe the m of its misses that minimizes the longer
  side; every 64th layer that would keep them all on one side moves one, so
  both fits stay measured.

IQ2_XS, RTX 5090 + 9950X3D, chat to the end of the turn:
- one AVX2 pool worker: 16.77 / 17.17 -> 12.98 / 12.82 ms a round
  (fit: CPU 42-60 us an expert, GPU 28 us a PCIe expert; 0.8 of the misses
  move);
- the 9950X3D's 16 workers: 3 pairs 12.19 -> 12.17 (pool DRAM-bound, few move).
Off (the default) the logits and 32 tokens equal v0.1.39 (2K, fixed cache).

Port of PR #1548 (commit a513394) onto 0.1.44.1. Conflicts with the 0.1.43 tail skip, PCIE_MIN1 and the fused waits resolved: the fused wait+copy launches stay off while the balance stamps are on. Docs line added.

Co-authored-by: sergiywith <sergiywith@gmail.com>
Niko1221 added a commit that referenced this pull request Oct 17, 2026
@Niko1221
docs: STRATA_KV_HOST_REGISTER and our measurements for STRATA_EXCHANG…
39f3670 
…E_ROTATE (#2042) and STRATA_PCIE_BALANCE (#1548)

#1498 pins the KV host copy with cudaHostRegister under WSL; the switch forces either order. Measurements: 5 interleaved pairs on an RTX 3060, an RTX A4000 and a Tesla P100.
@Niko1221

Niko1221 commented Oct 17, 2026

Copy link
Copy Markdown
Owner

Thanks. Ported (authorship credited to sergiywith; the conflicts with the 0.1.42 automatic share, tail skip and the fused waits are resolved by us). Measured with 5 interleaved pairs: RTX A4000 +5.6% decode (5 of 5 pairs), RTX 3060 -1.6% and Tesla P100 -1.3%, prompt speed unchanged. It stays an opt-in switch (STRATA_PCIE_BALANCE=1) and the docs say where it helps.

DarthCY-K added a commit to DarthCY-K/Strata that referenced this pull request Oct 19, 2026
@DarthCY-K
Merge cand-145 into sm120: sm120 becomes the current production sourc…
f9d9c55 
…e (v0.1.45 + Niko1221#1362 + /v1/models context_window); upstream absorbed Niko1221#1107/Niko1221#1367/Niko1221#1548
@Niko1221

Niko1221 commented Oct 19, 2026

Copy link
Copy Markdown
Owner

Thanks for the pull request.

Your change is already in main for 0.2.0-beta, with your credit, so there is nothing left for you to do here. We're closing it because we're resetting the tracker for 0.2.0-beta: there are more than 1,300 open issues and pull requests, and we're closing everything that isn't being worked on right now.

If part of your work did not make it in and you want it considered, open a new pull request on top of main that follows CONTRIBUTING.md, or comment /reopen after fixing the title area prefix and the template.

@Niko1221 Niko1221 closed this Oct 19, 2026
wesraph added a commit to wesraph/Strata that referenced this pull request Oct 30, 2026
@wesraph @claude
serve: STRATA_PCIE_BALANCE=1 on a layer split and the batch slots: ea…
99b0c66 
…ch stage picks its PCIe experts per layer from measured costs; the drafter's load line splits state and buffers

Opt-in, unset the default path is unchanged (no model, no stamps cleared, the fixed --pcie-frac share). The single-card
model (PcieModel, Niko1221#1548) fits the CPU's time per missed expert and the GPU's per PCIe one and picks, per layer, the PCIe
count that minimizes the slower side; it was wired for one GPU without batch slots only. Now each split stage has its
own model (its own link), fed by the batch windows (batch_serve records each step's CPU and PCIe counts and the pool's
time, the launch clears the stamps, batch_poll fits once a window), with --pcie-frac as the start until the fits have
samples. Why: Nsight on 2x RTX 3090 (IQ4_XS decode, three slots at 41K) put 23% of the GPU time in fetch_blobs, a
2.4 MB expert over the x8 link per layer at a median of 176 us, from the configured 0.35 share. Not yet measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@stuchapin909 @Niko1221