[q8atnight]

Resubmit of #229 part 1, rebased on v0.1.36, per your closing comment there. --peer-device N puts an adaptive
expert cache on a second visible GPU beside CUDA0's, holding the ranked pairs the primary cache does not hold.
Activations and result rows cross NVLink or another P2P path; the peer computes its experts' rows for decode
windows and prompt chunks. Without the flag everything is exactly as released.

Checklist from #229

• Rebase past Multi-GPU: the session carve and the per-stage prompt loans #216's session carve and 1024-token streaming: the base is v0.1.36 (both landed in 0.1.30);
  the peer tier is rebased on top and keeps them intact.
• cudaHostAllocPortable only with a peer: core::set_peer_portable(o.peer_device >= 1) runs once after arg
  parsing; verify.cpp, mtp.cpp and the prefill group table add the Portable flag only then. No layer-split or
  single-GPU run pays for it.
• Which --layer-split / --expert-cache-device1 combinations are refused: --peer-device now refuses both
  --layer-split (a different second-GPU mode, use one or the other) and --expert-cache-device1..3 (both would fill
  the same card with an expert cache), with a one-line message each; plus the existing refusals (no --expert-profile /
  expert cache, device not visible). Documented in docs/SECOND_GPU.md and in --help (new lines for the --peer-* flags).
• Auto sizing on 22 GB cards needs Multi-GPU: the session carve and the per-stage prompt loans #216's lending: not in this PR. --peer-reserve-mib and --peer-slots
  are the manual knobs; the docs say so and name Multi-GPU: the session carve and the per-stage prompt loans #216 as the follow-up.

Numbers (2x RTX 3090 + NVLink, IQ3_S, v0.1.36)

v0.1.36	prefill 8K / 32K / 128K (t/s)	decode 8K / 32K / 128K (t/s)	3,000-token story
stock, one 3090	2093 / 2201 / 2019	96.7 / 91.3 / 86.0	88 t/s
+ this PR (--peer-device 1 --peer-reserve-mib 1850)	2401 / 2747 / 2514 (+15 / +25 / +25 %)	115.2 / 104.5 / 99.6 (+19 / +14 / +16 %)	97 t/s

Our private build with the follow-ups (GDN head split etc.) reaches 2657 / 3118 / 3010.

Exactness

Gate: STRATA_IQ_MT_MIN=1 + our local debug STRATA_MMQ_NO_STREAMK=1, --adapt-swaps 0, same expert set
(single --expert-cache 6000 vs dual --expert-cache 3000 + --peer-slots 3950).

• v0.1.36 single reproduces our 0.1.31 single reference (flashbench gate PASS).
• This build without --peer-device == v0.1.36 single (flashbench gate + 19.5K-token long gate PASS): default unchanged.
• This build with --peer-device 1 == v0.1.36 single (both gates PASS): byte-identical.

Independent data

• Adamyno on multi-GPU: second GPU as an opt-in expert-cache tier with P2P rows (--peer-device), part 1 of #204 #229: 2x 2080 Ti (sm_75, PCIe P2P, no NVLink) ran the old part 1; with the
  --peer-slots 256 --peer-reserve-mib 1536 workaround prefill is at parity and decode is -11-14 %, and the
  x8 link kills the gain — auto sizing needs Multi-GPU: the session carve and the per-stage prompt loans #216's lending first.
• Multi-GPU: a --layer-split stage holds only its own layers' weights, and every card's leftover VRAM spills into a helper expert tier (4-way IQ3_S: 9,269 -> 14,172 expert slots, prefill chunk 2,048 -> 4,096) #390: the layer-split camp is converging on the same helper expert tier (9,269 -> 14,172 slots, 4-way).
• Layer split with a small second GPU caps prompt chunk at 512: 6.2x slower prompt reads, same decode #448 (fixed in 0.1.35): the advice there — big card alone, small card as extra expert cache — is what
  --peer-device automates.

Limits / follow-ups

• With a peer the prompt path keeps MMQ; the fused int8 path (Linux prebuilt from CI: CUDA 12.8, sm_80/86/89 + PTX, attached to each release #136) is not combined with the peer's rows yet
  (fused ring sizing switches off when a peer is configured).
• Tier sizing is manual (--peer-reserve-mib, --peer-slots) until Multi-GPU: the session carve and the per-stage prompt loans #216's buffer lending lands.
• Follow-up queued right after this PR: GDN head split over both cards, +6-8 % prefill, byte-identical,
  announced on Multi-GPU on 2x RTX 3090 (NVLink): the peer expert tier (#110) vs the layer split — measurements #204; the branch is already on this fork (preview/gdn-split).

[Niko1221]

Released in 0.1.38 as an opt-in (--peer-device); without the flag the output is unchanged. Three follow-ups of ours: the streamed walk's resident lookup indexes by the model's own expert count (it used 512, which broke the Coder's 256-expert layers on long prompts); a peer session never takes the fused prompt path, even when the peer declines the prompt path, so the ring and buffers match what fused_ring() sized; and the HIP shim gained the peer-access calls (the Windows HIP build stopped on them). Merged under your name (GitHub doesn't show it as merged because it went in by hand). Thank you!

[q8atnight]

Thanks — great to see it in 0.1.38, and the three follow-ups are the right calls. The n_expert indexing one in particular is the kind of bug our exactness gate could never catch (our 512-expert models index the same row), and keeping the fused prompt path off whenever --peer-device is set closes the ring/buffer mismatch before anyone combining it with STRATA_PF_FUSED=1 hits it.

We rebased our own tree onto 0.1.38 tonight (our peer-tier superset kept, your follow-ups taken): the 0.1.38 single-GPU path still matches our 0.1.36 gate reference bit for bit on the 3090s, and the dual path passes the same gate.

Heads-up on the follow-up we have queued: the prompt path can split every GDN layer's heads across the two GPUs on top of the peer tier (opt-in STRATA_PF_GDN_SPLIT=1, default off). With a split-K-matched cuBLASLt product for the half shapes it stays bit-identical to the unsplit path (gate + 20K-token long gate pass both ways) and adds +6–8 % prompt reading on 2× RTX 3090 + NVLink (8K/32K/128K). Now that the peer tier is official, we'll rebase it onto --peer-device itself and open it as a small PR against 0.1.38 — shout if you'd rather it wait for anything in flight.

[pdoyle12]

I'm sure this is a very stupid question, but how do we disable --layer-split so we can try this flag? I've tried every combination of deleting the flag, setting to -1, 0, setting --no-layer-split that I can think of, but I haven't found the right combination. If it's not explicitly set, it's set to auto automatically, and no other setting I can think of turns it off.

Edit: the fix appeared within a minute of my comment. That is some fast work! :)

[q8atnight]

Not a stupid question, and on a stock install there is genuinely no way to switch it off — that is why every combination you tried failed:

• the server adds the split itself whenever the config names more than one GPU
  (serve/server.py: if len(gpu_list(cfg)) > 1 and "--layer-split" not in args: args += ["--layer-split", …]);
• the engine has no "off" value: --layer-split accepts auto or a rising list of layer numbers starting at 2,
  anything else exits with --layer-split K[,K2..]|auto needs --serve, rising K from 2, …. So -1, 0, none can
  never work, and --no-layer-split does not exist.

Still true on 0.1.39 (checked today's serve/server.py) — the quick fix you noticed is fks's #665, which is open, not merged yet.

Workaround that works today (this is how we run it on 2× RTX 3090): keep the config's "gpu" entry to a single
card (or leave it out) so the server never appends the split, and put the peer flags in the args instead:

"args": ["--peer-device", "1", "--peer-reserve-mib", "1850", "…"]

The peer card is picked from the args, so it does not need to be in the "gpu" list. Two things to watch while you are
there: don't combine --peer-device with --expert-cache-device1..3 / --expert-cache-remote (both fill the same
card and you end up with the experts resident twice), and the --peer-* flags have no --help usage lines yet —
docs/SECOND_GPU.md is the reference.

fks's #665 makes this unnecessary by skipping the automatic --layer-split (and --split-skip-if-fits) as soon as
--peer-device is in the args, and updates docs/SECOND_GPU.md — worth merging, since the current failure mode
(--peer-device cannot be combined with --layer-split) looks like the engine refusing your hardware rather than the
server adding a flag you never asked for.

[pdoyle12]

Thank you for the explanation! I pulled in #665 manually and rebuilt, and it's working fine now as you described.

I'll mention in passing that for my rig -- a 5090+3090 with 256 GB of RAM running the Unsloth Q4 quant -- this flag noticeably bumped performance, and I'm seeing ~1275 t/s prefill and ~90 t/s generation, which is frankly astonishing.