[zhedang]

Summary

This PR improves multi-GPU decode performance by making the primary GPU and
expert-helper GPUs cooperate more efficiently.

The optimized v2.2 expert-helper path delivers substantial decode speedups
over both the previous expert-helper path and the automatic layer-split path.

Speedup summary

GPUs / Version	Compared against	Mixed	Code	Math20
RTX 5090 + RTX 4090 / 0.1.38	Original expert-helper	+28%	+63%	+42%
RTX 5090 + RTX 4090 / 0.1.38	Automatic layer split	+17%	+16%	+18%
Dual RTX 4090 / 0.1.37	Original expert-helper	+63%	+132%	—
Dual RTX 4090 / 0.1.37	Automatic layer split	+13%	+34%	—

Enabled with --remote-expert-opt. Disabled by default.

---

Benchmarks

RTX 5090 + RTX 4090 — 0.1.38

Decode throughput, in tokens/s:

Configuration	Mixed	Code	Math20
Original expert helper	135.49	152.56	158.61
Automatic layer split	147.41	213.89	191.06
Optimized expert helper	173.04	248.38	226.00

Dual RTX 4090 — 0.1.37

Decode throughput, in tokens/s:

Configuration	Mixed	Code
Original expert helper	88.07	85.15
Automatic layer split	126.86	146.95
Optimized expert helper	143.35	197.58

---

What changed

• Complementary expert caches
  Keep the primary and helper GPU expert caches complementary, reducing
  duplicated experts and CPU fallback.

• Lower inter-GPU transfer volume
  Reduce weighted expert outputs on helper GPUs before returning results.

• Skip unnecessary CPU quantization
  Avoid CPU activation quantization when all selected experts are served by GPUs.

• Automatic helper-cache sizing
  Add auto sizing based on available VRAM, removing the need to manually tune
  expert counts.

The existing primary/helper GPU layout remains unchanged.

[hipotures]

Dual-4090 IQ3_S results and a helper-ownership dispatch fix

I tested PR head b28121ff6ef117bec2558b3ece7e188dd33a2b7e on the same upstream base, 99f3dbd0b21d1401b3769e0c0d963913607f380b. The local merge had no conflicts.

Summary: I could not reproduce the optimized helper outperforming layer split on our workload. However, I found a dispatch-order interaction worth reviewing: experts already resident on the helper can be assigned to the primary mapped-RAM PCIe path before the helper can claim them. A small local change improved median helper decode throughput by approximately 20–54% in the 256-output context sweep and 79.8% on a separate 2048-output code-maintenance workload. These are workload-specific results, not a general 80% speedup or a claim that helper mode should replace layer split.

Setup and controls

2× RTX 4090 24 GiB, no NVLink, PHB topology; Ryzen 9 7950X3D, 16-vCPU VM, approximately 161 GiB RAM. Model: Qwen3.8-Flash-Next IQ3_S (ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF). GCC 15.2, CUDA 13.4, driver 615.71.09; matching Release/sm_89 build settings.

Common inference settings: MTP --spec 4 --spec-min-p 0.5, --suffix-draft 0, INT8 KV, --kv-resident 32768, maximum context 262144, 15 pool workers, and --prefill auto (8192-token chunk). Greedy sampling, one request at a time, and zero prompt reuse. The steady-decode comparison also explicitly used --prompt-cache 0.

Original helper, PR optimized helper, and the locally patched helper started with the same 8586 primary / 11796 helper physical slots, identical initial resident expert IDs, and zero initial overlap. For this pack, numeric --expert-cache 6567 reproduces the 8586 physical primary slots; passing 8586 as the CLI value would enlarge the byte budget. Layer split is a separate topology comparison, not an equal-layout algorithm-only A/B.

For the steady comparison below, each configuration used a fresh server, the same 4096-token warmup input with exactly 64 generated tokens, then three measured requests with the same 31,400-token input and exactly 2048 output tokens. Equal warmup length does not imply identical generated tokens or identical adaptive-cache state across algorithms. Boundary cache snapshots were disabled in all headline speed runs.

Dispatch issue and local change

In src/core/expert_source.cpp, expert_pool_dispatch_multi() computes the primary PCIe miss quota before calling RemoteExperts::begin(). The miss calculation excludes PeerExperts residents but not remote-helper residents.

In src/core/remote_experts.cpp, RemoteExperts::begin() skips entries whose kind is already different from -1. Consequently, an entry assigned kind = 1 by the primary planner bypasses the helper even when its expert is already cached there. With complementary caches, the helper-resident expert is also excluded from primary adaptive admission, so repeated primary reads from mapped host RAM can persist.

The local change checks current optimized-helper residency, excludes those experts from the primary PCIe miss count, and leaves their entries unclaimed until the existing helper dispatch. Remaining misses retain the existing PCIe quota logic. It does not change model weights, kernels, cache capacities, or original-helper/layer-split ownership decisions. The functional diff is included below.

Steady decode: 31,400 input / 2048 output

Three measured runs per configuration; TG is decode tokens/s after warmup, with MTP enabled and suffix lookup disabled.

Configuration	TG median (min–max)	Decode ms/window	CPU fallback entries, approx.	GPU0 / GPU1 util.	GPU0 PCIe RX, MB/s
Layer split auto, K=25	158.8 (153.3–161.0)	18.9	12,574	45.4% / 54.6%	333
Original helper	76.0 (73.9–80.5)	37.5	75,125	95.0% / 11.2%	4,804
PR optimized helper	75.3 (74.6–78.4)	36.8	3,749	96.9% / 5.3%	7,548
PR optimized helper + local fix	135.4 (125.5–150.9)	23.0	6,307	92.9% / 11.9%	185

On this workload, the submitted optimization did not improve median TG over the original helper (75.3 vs 76.0). The local change improved median TG by 79.8% relative to the PR helper, but remained 14.7% below layer split. The large accompanying reduction in sampled GPU0 PCIe RX supports the dispatch explanation; these are mapped-host-memory reads, not evidence of SSD streaming.

Important qualifications:

• MTP accepted draft tokens per decode window also changed: median 1.885 → 2.119. Generated trajectories differ, so this is an end-to-end measurement, not a fixed-token replay isolating dispatch cost. I would not attribute the entire TG gain to faster verification or decompose it into exact causal factors by multiplying ratios of independently calculated medians.
• The log's ms/window covers the full decode iteration, including verification, drafting, and commit/emit work; it is not verifier-only latency.
• CPU fallback estimates come from rounded per-layer-window log averages × 48 layers × window count, not exact event counters. Fallback increases after the fix, even though TG improves. That observation does not by itself establish why fallback increased.
• GPU/PCIe telemetry was sampled at 1 Hz. The remaining GPU0/GPU1 utilization asymmetry is a profiling lead, not proof of SM saturation, inadequate compute overlap, or the next bottleneck.

Context sweep: 256 output tokens

Each context point used an equal-length 64-output warmup and three measured runs, with matching input token IDs between configurations and zero reuse. Each configuration used one server across the context sweep, retaining adaptation history; these were not independent fresh-server replicates for every cell.

Actual input tokens	Layer split TG	PR helper TG	Patched helper TG	Patch vs PR helper	TTFT: layer split / patched helper
31,400	135.1	95.4	117.5	+23.2%	6.9 / 13.3 s
63,400	126.9	84.1	100.8	+19.9%	12.0 / 26.8 s
127,000	119.2	82.4	125.1	+51.8%	22.7 / 54.2 s
259,500	114.3	76.2	117.7	+54.5%	46.2 / 114.2 s

The patched helper's +4.95% and +2.97% median TG differences relative to layer split at 127K and 259.5K are small observed advantages, not established general wins. At 127K, all three matched comparisons favor the patch (LS 118.3–119.7 vs patched helper 123.5–126.2 tok/s); at 259.5K, the three paired comparisons are mixed. Neither cell establishes statistical equivalence, and neither has independent fresh-server replication or a long-output confirmation at that context size.

Prefill remains about 1.9–2.5× slower with the patched helper across this sweep. At 259,500 input tokens, the approximately 68-second TTFT penalty outweighs the small decode-only advantage for the tested 256-token responses. We are therefore keeping layer split auto for our zero-reuse agent workload.

I have not used the earlier 32K screening result of 95.7 → 119.2 tok/s as a controlled fix-speedup headline: its PR-helper and patched-helper warmups generated 227 and 64 tokens respectively. The final sweep and steady comparison above use equal-length warmups.

Cache behavior and validation limits

Separate cache diagnostics support the complementary-cache mechanism: the PR optimized helper retained zero primary/helper overlap at startup, after warmup, and after run 3. The original helper reached 2073 overlapping experts, or 17.57% of helper capacity. Those diagnostics were separate from the timed runs.

The ten-prompt greedy smoke tests for the PR and local fix showed no observed obvious corruption or NaN/Inf symptoms; the JSON cases remained valid. This is not numerical-parity validation: no comparable teacher-forced KL/top-1 analysis was obtained. CTest recorded 56 passes, 2 skips, and 4 environment-related failures per build (missing fixtures and the mlock limit), not an all-green suite. No OOM or crash occurred in the campaign. Three early-ending measured requests were retained as invalid (63 output tokens, tool_calls) and excluded from fixed-length throughput summaries.

Functional diff

These are the functional changes from the tested local patch. Benchmark-only header context has been removed; no diagnostic instrumentation is included in this diff.

Helper-residency priority patch for PR head b28121f

diff --git a/include/strata/core/remote_experts.hpp b/include/strata/core/remote_experts.hpp
--- a/include/strata/core/remote_experts.hpp
+++ b/include/strata/core/remote_experts.hpp
@@ -38,6 +38,7 @@ public:
                std::string& err);
     bool owns(int64_t index) const { return owned_[(size_t) index] != 0; }
     bool optimized_decode() const { return remote_opt_ != nullptr; }
+    bool has_expert(int64_t layer, int32_t expert) const { return cache_.slot_of(layer, expert) >= 0; }
     bool finish(float* out, std::string& err);
     int64_t resident() const { return cache_.resident(); }
     int64_t computed() const { return computed_; }
diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp
--- a/src/core/expert_source.cpp
+++ b/src/core/expert_source.cpp
@@ -1888,6 +1888,13 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
             if (ids[i] >= 0 && ids[i] < d.n_expert) d.usage[(size_t) d.layers * (size_t) d.n_expert + (size_t) ids[i]] += 1.0f;
     // ---- plan v0.3 P6: the GPU's share, decided and published FIRST so the GPU starts while the CPU works.
     // Distinct experts in routing order; resident ones and the last pcie_num/256 of the missed ones go to the GPU.
+    // Optimized helper residents are already on a GPU: do not classify them as
+    // primary PCIe misses. Keep them unclaimed until RemoteExperts::begin below.
+    const auto optimized_helper_has = [&](int32_t e) {
+        for (int r = 0; r < d.remote_count; ++r)
+            if (d.remote[r]->optimized_decode() && d.remote[r]->has_expert(d.layers, e)) return true;
+        return false;
+    };
     const int64_t n = n_tok * k;
     int32_t kind[kMaxWindowEntries];       // per entry: -1 CPU, 0 VRAM, 1 PCIe
     if (d.plan != nullptr && n <= kMaxWindowEntries && n <= d.plan->cap) {
@@ -1901,7 +1908,7 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
                 distinct[nd++] = i;
                 const int32_t e = ids[i];
                 if (e >= 0 && e < d.n_expert && d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] < 0 &&
-                    !(d.peer != nullptr && d.peer->has(d.layers, e))) ++nmiss;
+                    !(d.peer != nullptr && d.peer->has(d.layers, e)) && !optimized_helper_has(e)) ++nmiss;
             }
         }
         const bool pcie_ok = d.pcie_num > 0 && d.src->pcie_layer(d.layers);
@@ -1923,6 +1930,8 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
                                                                                  : (size_t) slot * (size_t) d.cache_blob));
                 } else if (d.peer != nullptr && d.peer->has(d.layers, e)) {
                     kd = 2;                        // multi-GPU: the second GPU computes it
+                } else if (optimized_helper_has(e)) {
+                    kd = -1;                       // helper claims this row below; not a PCIe candidate
                 } else {
                     if (miss_rank >= nmiss - m && fetches < P.staging_cap && fetches < 64) {
                         const uint8_t* src = d.src->pinned(d.layers, e) ? d.src->blob(d.layers, e) : nullptr;

Does your benchmark exercise the primary mapped-RAM PCIe quota with experts already resident on the helper? Comparing that routing path, the exact command line, and a small reproducible payload would help determine why the submitted helper optimization behaves differently here. The proposed ownership change is the main finding I would like reviewed, independently of whether helper mode ultimately outperforms layer split.

[Niko1221]

Thanks! Taken into 0.1.39. The engine flag stays default-off, and setup now adds --remote-expert-opt to a config on two or more GPUs. To leave it out, pass setup --no-remote-expert-opt or set "remote_expert_opt": false in the config. It acts only with a helper cache (--expert-cache-device1..3), so a layer split runs as before. We have one GPU here: we checked the build, the refusal without --serve and byte-identical default output. Merged with #646, which publishes the routing weights only when a helper needs them, so with the flag the doorbell still sends them. The dual-GPU numbers are yours (dual RTX 4090 +63% mixed / +132% code over the plain helper path).