[ZhongUncle]

The verify window's GPU plan is built with O(n²) scans, and the GPU spin-waits on flag A while it is built: the plan is "decided and published FIRST so the GPU starts while the CPU works", and flag A rises only after it, at P.publish. Plan time is therefore GPU idle time on the flag-A critical path, even where its share of the round is too small to show end-to-end (see Limits). The quadratic parts: a pairwise first-occurrence scan (expert_source.cpp:2462), three full re-scans of the window per distinct expert (kind assignment, VRAM and PCIe group entries), and a std::find dedup of the CPU-miss prefetch (expert_source.cpp:2598).

This PR rewrites all of it as three linear passes with output-identical results. It deliberately does not change what the plan decides — the tiers, the PCIe share and the group order are exactly the old ones. The case for taking it is structural: no behavior change (pinned by a 4,020-case differential test, no GPU needed), and the quadratic term it removes grows with the window, so a bigger --spec widens the saving.

TL;DR

• ~3.1–45.7K branchy iterations per window-layer (3×10 to 12×10 windows, by n(n−1)/2 + 3·distinct·n) collapse to three linear passes
• standalone planner: 2.3–3.2× for the windows seen in the engine A/B (mean ≈3×10, max 6×10), 4.8× near the 128-entry cap
• engine --stats: the dispatch plan figure halves in both rig states seen here (four interleaved pairs); decode unchanged, as expected — the ≈0.3–0.4% gain is below the ±0.7% run-to-run spread
• output-identical, proven without a GPU (4,020/4,020; the reference is diff-verified against the old code) — details in Findings, caveats in Limits

What changes

The planner is extracted as detail::window_gpu_plan (the header's existing detail test-seam pattern) and rewritten:

• one pass groups each entry with its expert's first occurrence — open addressing over the routing ids, 256 slots — replacing the pairwise first_of scan
• one pass over the distinct experts decides the tiers — the peer/helper-GPU questions are answered once per distinct expert instead of once per entry and again per group
• one pass emits the groups with per-group cursors, replacing the three per-distinct-expert re-scans
• the CPU-miss prefetch dedup uses the same open-addressed set instead of std::find

Bounds are the old ones: the call site already guards n <= kMaxWindowEntries (128) and falls back to the simple no-plan loop beyond it, so the 256-slot table never exceeds load ½; out-of-range ids group by the same equality the pairwise scan used and stay kind −1, refused later by the pool loop with the same message. Unchanged by construction: group order (routing order of first occurrence), entry order within a group, counts, kinds.

Measured results

Measured on:

• RTX 3080 20 GB (driver 595.91.07, CUDA 13.2)
• Xeon E5-2680 v4 (no AVX512)
• 62 GB RAM
• Ubuntu 24.04
• Qwen3.8-Flash-Next IQ2_XS pack (48 layers × 512 experts = 24,576 possible pairs)
• --spec 4 --spec-min-p 0.5 + the MTP drafter (verify window up to 6 tokens)

Standalone planner, ns per window-layer (512 experts, ~70% resident, 1,000 pre-generated windows cycled through, 200,000 rounds per arm):

window	old	new	speedup (×)
3×10 (30 entries, 29 avg distinct)	1796	790	2.3
4×10 (40 entries, 39 avg distinct)	2686	1043	2.6
5×10 (50 entries, 48 avg distinct)	3854	1311	2.9
6×10 (60 entries, 57 avg distinct)	5103	1576	3.2
8×10 (80 entries, 74 avg distinct)	8078	2081	3.9
12×10 (120 entries, 107 avg distinct)	15283	3204	4.8

(Per 48-layer window that is 86 → 38 µs at 3×10 up to 734 → 154 µs at 12×10 of GPU wait. The window grows with --spec: 6×10 is this config's maximum, 12×10 is near the 128-entry cap. The gap widens with window size because the old cost grows with distinct × n.)

Engine A/B — dispatch line sections in ms per window-round, mean of two replicated pairs per rig state (four interleaved pairs total; raw lines in the appendix):

section	baseline (slow)	patched (slow)	baseline (fast)	patched (fast)
plan	0.19	0.09	0.14	0.07
activation quantize	0.68	0.68	0.42	0.42
jobs	0.02	0.02	0.01	0.01
run	7.56	7.66	4.87	4.92

(decode: baseline 95.7–97.1, patched 95.8–96.8 tok/s over four runs each — ranges overlap; durations in the appendix.)

Findings

1. The new planner's output is identical to the old one's. tests/core/expert_window_plan_test.cpp drives detail::window_gpu_plan and a reference copy of the O(n^2) planner with the same inputs and compares every positional field — kinds, group ptr/start/start2/dst/tok, the group counts, nmiss, the DMA source list — on 20 targeted corners (the PCIe share's last-m boundary, the staging cap, peer/helper-held experts, out-of-range ids, pcie_mode 0/1/2, per-slot cache offsets) and 4,000 randomized windows: identical in all 4,020. The reference is the replaced block with only the mechanical adaptation to the injected predicates, diff-verified against the base commit; the production predicates are the old inline expressions verbatim.

2. Standalone plan time drops 3.2× at this config's max window, 4.8× near the entry cap. 5,103 → 1,576 ns per window-layer at 6×10 (245 → 76 µs of GPU wait per 48-layer window); the old cost grows with distinct × n, the new one with n — measured across six window sizes from 2.3× at 3×10 to 4.8× at 12×10 (table above; the in-engine figure is finding 3). (6 tokens is the cap at --spec 4 because the default suffix drafter may lengthen a window by 2 — generate.cpp:2091; the log's "rounds of 6" in the appendix confirms it is reached.)

3. In the engine, the dispatch plan figure halves and nothing else moves. 0.19 → 0.09 and 0.14 → 0.07 ms per window-round, replicated in both states; activation-quantize, jobs and run are unchanged within noise state-for-state (the run section's pair deltas are −0.01 to +0.22 ms).

   The engine's absolute numbers sit below the micro-benchmark's 48-layer extrapolation for two reasons. The plan segment times more than the planner — begin_layer, usage accounting, the flag-raising P.publish and the PCIe P.fetch issue share the segment and are untouched, so the ~0.07 ms that remains is those parts plus the now-small planner. And the A/B windows are smaller than the benchmark's headline arms: the logged lookup counts put the mean window at ≈3.3×10 slow-state and ≈2.4×10 fast-state (308,226 lookups ÷ 195 rounds ÷ 48 layers ≈ 33 and 283,774 ÷ 244 ÷ 48 ≈ 24 entries per window-layer — PCIe-served misses excluded; the draft counts agree: 1 + 458/195 ≈ 3.3 and 1 + 355/244 ≈ 2.4 tokens). The per-layer saving (105 µs/round ÷ 48 ≈ 2.2 µs slow-state, 67 ÷ 48 ≈ 1.4 fast-state) is 1.4–2.2× the benchmark's ≈1.0 µs at 3×10 — the expected direction, since the benchmark runs warm (L3-resident working set, regular branches, fixed residency — see Limits) while the engine is cold and irregular, and plan cost is convex in window size, so a spread of real windows costs more than the mean window. The micro-benchmark is the scaling story; the engine figure is the realistic one.

4. Decode tok/s is unchanged, and that is the expected outcome, not hidden work. Within each pair the two arms produce identical progress lines (greedy sampling; every logged counter matches — same rounds, same acceptance, same expert lookups, see the appendix): 195 rounds in the slow state, 244 in the fast, so a round is 27.3 ms slow-state and 21.8 fast-state. The plan saving (105 µs slow-state, 67 fast-state) is an expected ≈0.4% / ≈0.3% decode gain, below the ±0.7% run-to-run spread of the four runs (95.7–97.1 tok/s). The removed plan time is real dead time on the flag-A critical path; it becomes visible end-to-end where its share is larger (bigger --spec windows, slower CPUs — see Limits).

Validation

• What was compared: the extracted planner against a reference copy of the O(n^2) code, same inputs, every positional field (finding 1); engine --stats dispatch lines on four interleaved baseline/patched pairs, 512 generated tokens after an 873-token prompt (finding 3)
• Reference fidelity: diff-verified against the base commit (82f46a8) — see finding 1
• Result: 4,020/4,020 identical; ctest -R expert_window_plan passes in ~0.6 s with no GPU
• Full suite: ctest green except ple_parity and expert_multi_test, which fail identically on the base commit on this host (missing model files, no AVX512)

Limits

• The planner is host code: on a faster CPU the absolute win shrinks, slower CPUs should see more — an expectation from the algorithm, not a measurement on a second rig. Bigger windows do see more, measured (2.3× at 3×10 → 4.8× at 12×10, table above)
• Decode tok/s does not measurably improve on this rig — the expected gain (≈0.3–0.4%) is below the run-to-run spread (finding 4)
• The micro-benchmark cycles 1,000 pre-generated windows per arm (no single window's branches can be learned), but its working set still fits this CPU's L3 and residency is a fixed ~70% — treat the ratios as the scaling story; the engine A/B is the realistic figure
• One model, one quant, one GPU
• The engine baseline binary was built from the pre-rewrite main (1735d64, still fetchable by SHA on GitHub); its build inputs are hash-identical to this PR's base — src 9f4e26be, include d4bf5e4b, cmake 478c0fb6, CMakeLists.txt cd605177 in both commits (the rewrite touched no C++ or build files)

Reproduce

# equivalence, no GPU
ctest -R expert_window_plan

# engine A/B (host with the model; pretokenize via the pack's chat template)
./engine/strata --pack Strata-data/packs/iq2_xs --native <shard1.gguf> --ple-gguf <shard2.gguf> \
  --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto \
  --spec 4 --spec-min-p 0.5 --mtp Strata-data/mtp/rt \
  --max-context 24576 --kv int8 --kv-resident 32768 \
  --tokens-file PROMPT.tokens --max-new 512 --stats
# the dispatch line's plan figure is what this PR reduces; interleave baseline and patched runs

The standalone micro-benchmark is not part of this PR — it is the test's driver with a timing loop; its parameters are in the appendix.

Raw `--stats` lines and the micro-benchmark

Engine A/B, all eight runs — 873-token task prompt, 512 generated tokens, interleaved baseline/patched pairs, slow/fast = the two rig states seen across the four pairs (clock and trajectory both differ between them; supports findings 3 and 4):

baseline r1: dispatch  plan 0.194  activation quantize 0.681  jobs 0.021  run 7.518 ms/round — decode 96.29 tok/s (5317.1 ms)
patched  r1: dispatch  plan 0.089  activation quantize 0.688  jobs 0.021  run 7.737 ms/round — decode 95.78 tok/s (5345.4 ms)
baseline r2: dispatch  plan 0.135  activation quantize 0.419  jobs 0.014  run 4.913 ms/round — decode 96.29 tok/s (5317.4 ms)
patched  r2: dispatch  plan 0.068  activation quantize 0.422  jobs 0.014  run 4.910 ms/round — decode 96.27 tok/s (5318.4 ms)
baseline r3: dispatch  plan 0.135  activation quantize 0.417  jobs 0.014  run 4.822 ms/round — decode 97.11 tok/s (5272.5 ms)
patched  r3: dispatch  plan 0.068  activation quantize 0.418  jobs 0.014  run 4.936 ms/round — decode 96.75 tok/s (5292.2 ms)
baseline r4: dispatch  plan 0.194  activation quantize 0.678  jobs 0.021  run 7.592 ms/round — decode 95.71 tok/s (5349.4 ms)
patched  r4: dispatch  plan 0.089  activation quantize 0.677  jobs 0.021  run 7.586 ms/round — decode 95.96 tok/s (5335.4 ms)

(r1 and r2 baseline differ by 0.3 ms of decode duration — 512/5.3171 = 96.294 and 512/5.3174 = 96.288, both 96.29 at two decimals.)

Two things worth knowing about those lines. The section timer is steady_clock accumulated in double ms (expert_source.cpp:2676), nanosecond-resolution — not a coarse timer. Sampling is greedy and every run within a state is identical in every logged counter (slow: 195 rounds, drafts accepted 318 of 458, lookups 277,710 of 308,226 with 30,516 refused; fast: 244, 268 of 355, 260,078 of 283,774 with 23,696 — each patched run matches its baseline partner), so the pure host segments plan and jobs repeat to three decimals while quantize and run, containing GPU work, jitter normally. Refusals are timing-dependent yet reproduce exactly within a state — apparently the state's timing margins are wide enough that the same experts miss their window in every replica; between states the refusal pattern shifts the logits, hence the different trajectories and round counts.

Standalone planner micro-benchmark — Xeon E5-2680 v4, 512 experts, ~70% resident, 1,000 pre-generated windows cycled through, 200,000 rounds per arm, quiet machine (supports finding 2). Run single-threaded on the otherwise-idle rig with the benchmark core at turbo (~3.2 GHz; frequency polled during a re-run, all six arms reproduced within noise):

window  3x10 =  30 entries,  29 avg distinct: O(n^2)  1796 ns   O(n)  790 ns   2.3x  (~3062 iterations by n(n-1)/2 + 3*distinct*n)
window  4x10 =  40 entries,  39 avg distinct: O(n^2)  2686 ns   O(n) 1043 ns   2.6x  (~5402 iterations)
window  5x10 =  50 entries,  48 avg distinct: O(n^2)  3854 ns   O(n) 1311 ns   2.9x  (~8380 iterations)
window  6x10 =  60 entries,  57 avg distinct: O(n^2)  5103 ns   O(n) 1576 ns   3.2x  (~11973 iterations)
window  8x10 =  80 entries,  74 avg distinct: O(n^2)  8078 ns   O(n) 2081 ns   3.9x  (~20955 iterations)
window 12x10 = 120 entries, 107 avg distinct: O(n^2) 15283 ns   O(n) 3204 ns   4.8x  (~45670 iterations)

Work with Claude Code

[Niko1221]

Thanks for the careful measurement. The plan time halves but is about 0.1 ms of a 7-10 ms window, so decode does not move (your own numbers), and the planner has changed a lot since your base, so a straight port conflicts in three places. We are not taking it as a speed PR. If you rebase it with the differential test we would review it as a cleanup.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.