[q8atnight]

Part 1 of the peer-tier plan agreed in #204: a second GPU as an opt-in expert-cache tier beside --gpus. Part 2 (the QSA query split) is not included — see the note at the end.

What it does (only with --peer-device N)

• Second expert-cache tier on GPU N: an adaptive cache of its own, filled from the profile after the primary's cache; decode rows come back through the pool's mapped rows (the verify graph is untouched).
• P2P rows in the prompt path: the peer computes its experts' rows of each prompt chunk (activations in / rows out over P2P), with compact group buffers, the rows returned group by group on a second stream, and a streaming ring that lets the peer stream a share of the primary's streamed experts over its own PCIe link.
• Peer rows are scattered straight into the pool's mapped rows (those buffers are now cudaHostAllocPortable, the only unconditional change — no behaviour change without a peer).

New flags: --peer-device N (default -1 = off), --peer-reserve-mib (600), --peer-slots (0 = auto), --peer-adapt-swaps (-1), --peer-prefill-rows (-1 = chunk×K/2, 0 = off).
Env switches, all only read when a peer is active: STRATA_PF_PEER_OUT_PIPE, STRATA_PF_PEER_COMPACT, STRATA_PF_PEER_STREAM (0.35), STRATA_PF_PEER_RING (48), STRATA_PEER_DIRECT, STRATA_PEER_HOT (0.45), STRATA_PEER_HOT_AT.

Rebased on 0.1.28; upstream's --expert-cache-device1..3 tier, the D-5 issuer thread and D-1 attention are left intact.

Exactness (#152 method)

Env trio STRATA_NO_IQ512=1 STRATA_NO_IQ256=1 STRATA_NO_IQ4NL=1 + no-stream-k MMQ (local debug patch), greedy, the three gate prompts (chat / code / math), IQ3_S:

• Default path (no --peer-device) vs plain 0.1.28: byte-identical (PASS).
• Peer on vs single GPU with the same GPU expert set (single --expert-cache 7846 slots; peer run 3896 primary + 3950 peer, static, STRATA_PEER_HOT=0): byte-identical (PASS).

Numbers — 2× RTX 3090 (NVLink), IQ3_S, 0.1.28 base, flashbench --quick (greedy)

setup	prefill 32K (tok/s)	decode 1K	decode 32K
single GPU (auto cache)	2,094	91.5	88.8
--gpu1-experts (--expert-cache-device1 10000)	1,425	88.8	86.4
layer split (--layer-split auto)	936	104.2	94.9
this PR (--peer-device 1, auto)	2,566	106.5	106.7

(Peer run: 8,692 primary + 11,168 peer = 19,860 of 24,576 experts on the GPUs. One run per cell.)

About part 2 (QSA query split)

We ported it too, but on 0.1.28 it does not meet the bar: prefill attention now uses the D-1 tensor-core kernel, which our split cannot divide mid-chunk, so the split falls back to the older kernel — not byte-identical, and slower than this PR alone (2,125 vs 2,566 at 32K). So it's left out. We might come back with a redesign that keeps D-1 on both cards; if so it will be a separate PR against this one.

[Adamyno]

Peer tier on 2x RTX 2080 Ti 22GB (PCIe P2P, no NVLink) — part 1 compiles and runs on sm_75,
but we hit a startup wall and found a workaround worth knowing:

• auto peer sizing (1324-1786 experts) does not start on 22 GB cards: after the auto split,
  CUDA1's expert cache + session leave too few slots to borrow, and the prompt path aborts with
  "the prompt path allocates its own buffers (too few cache slots to borrow)" →
  "device buffers for a chunk of 2048 tokens do not fit". --prefill 1024 + --kv-resident 16384
  still aborts (the chunk shrinks, the fit does not). The Multi-GPU: the session carve and the per-stage prompt loans #216/Multi gpu: Layer split: pinned arena, lent prompt buffers, tested planner (2 x 2080 Ti decode +34%, 29K prompts +61%) #223 buffer lending is exactly
  the missing piece; until it lands, the peer needs headroom reserved for those buffers.
• Workaround that works: --peer-device 1 --peer-slots 256 --peer-reserve-mib 1536.

Measured with that (Qwen3.8-Flash-Next IQ3_XXS, INT8 KV, MTP spec=4, temp 0, warm medians of
3; 13,502- and 30,132-token prompts + 256 generated):

• 1 card: 13k cold 23.5 s / 30k cold 46.0 s; warm decode 5.08 / 5.45 s
• split + peer on the x16 card: 23.2 / 45.8 s — prefill parity; warm decode 4.50 / 4.68 s (−11-14%)
• split + peer on a x8 Gen3 card: 24.6 / 48.2 s — prefill +5%; the link width is the whole
  cost, consistent with your "P2P over PCIe, slower" note

The decode gain is cache coverage (the two cards hold ~16.9k of 24,576 profiled pairs, ~99% of
the routed mass, vs 10.0k slots on one card). A 256-expert peer is deliberately small; auto
sizing needs the buffer fix first.

[q8atnight]

This is exactly the kind of independent data the PR needed — a second rig, and one in a config
we can't test: PCIe P2P without NVLink. Thank you.

Three things that line up cleanly with our 2x3090 box:

1. The startup wall is our known headroom problem, and your workaround is the right one. The
   peer's lazy first-use setup needs a few hundred MiB free; at single-digit-MiB headroom the first
   prompt dies in the prompt-path buffers (we hit exactly this at 23 MiB free on our 24 GB cards
   before pinning --peer-reserve-mib). Our configs carry an explicit reserve (1650 MiB on the
   3090 build) for that reason — auto sizing alone isn't enough on any card once the prompt-path
   chunk + session carve take their share. We'll note the --peer-slots/--peer-reserve-mib
   combo in the PR as the small-card recipe, and the auto path does want Multi-GPU: the session carve and the per-stage prompt loans #216's buffer lending.
2. Your x16 vs x8 split is the link cost made visible. On NVLink the same part 1 measures
   2,566 pp / 107 tg vs single 2,094 / 89 (IQ3_S, 32K, 0.1.28 base) — the P2P rows never wait on
   the link. Your x8 Gen3 result (+5 % prefill cost, gains shrinking) is what we'd expect for the
   row traffic without the bridge; good to have it measured rather than assumed.
3. Coverage explaining the decode gain matches our model — the peer tier is a cache-coverage
   feature; 16.9k/24.6k pairs ≈ 99 % routed mass on your cards tracking our 19k+19k on the 3090s.

One ask if you have the rig for it later: the byte-identity gate (#152 trio + STRATA_MMQ_NO_STREAMK
is our own debug patch so skip that one) with the peer on vs off on your rig would extend the
exactness evidence to sm_75 + PCIe P2P. No rush — the PR is already gated on our side.

[Niko1221]

Thanks! Please rebase on 0.1.31: #216's session carve and 0.1.30's 1024-token streaming conflict with it, and auto sizing on 22 GB cards needs #216's lending. Also: make cudaHostAllocPortable apply only with a peer (it maps the pool into every context, so a layer split pays for it twice, bad under WSL's ~1 GiB budget), and say which combinations with --layer-split / --expert-cache-device1 are refused. A new PR is welcome. Closing this one.

[q8atnight]

Resubmitted on v0.1.36 as #531 (checklist answered there).