[Keoian]

Problem and result

The serial decode loop ends an MTP window at the first draft below --spec-min-p. Each draft is judged on its own, so a window can pass the floor token by token while the chance that the whole window is accepted is low (four drafts at p = 0.85 pass a 0.8 floor; all four are accepted about 52% of the time). Every drafted token is verified, and where a verify token is expensive the tail of such a window costs more than it returns.

With STRATA_SPEC_PCUM=base[,deep[,d0,d1]] the window also ends where the product of the drafted tokens' top-1 probabilities falls below a threshold that follows the depth. The drafter stops at the same token. Unset, nothing changes (byte-identical to main on the P100 machine below).

Measured on two machines with Qwen3.8-Flash-Next, the gain is small or negative, because this model's verify cost hardly grows with depth:

• 2x RX 5700 XT (HIP), Q2_0: greedy text byte-identical with the rule on; decode +1.5% when sampling, unchanged greedy.
• 2x Tesla P100 (CUDA), IQ3_XXS: greedy unchanged on short prompts, sampled within noise (-4%), and 4-6% slower at 33K and 110K tokens of context. Greedy text changes with the rule on, as it does on this engine when only --spec-min-p changes (Validation).

Related discussion

#4729 (scope). The idea comes from the depth-scheduled draft cutoff in https://github.com/Keoian/dual-p100-qwen-3-8-27b (branch p100-optimizations, OPTLOG.md attempts 198 and 218). That is a llama.cpp fork serving the dense Qwen3.8-27B, where every verify token costs an attention pass over the whole cache, so its numbers do not apply here. Qwen3.8-Flash-Next's sparse attention keeps the verify cost nearly flat with depth (#4729: a ~1.7-token window took 45.7 ms at 31K and 46.7 ms at 101K), which is why the rule saves little on this model.

Approach

• The threshold is base up to position d0, deep from d1 (16384 / 49152 by default), and a straight line between. deep defaults to base.
• The drafter uses the existing chain-rule hook (mtp.set_chain_rule): float compares and a double product, operation for operation the window rule, so it drafts nothing the window drops.
• Per request, strata_tune takes spec_p_cum (base and deep) and spec_p_cum_deep; serve/server.py passes both through, range 0-1.
• With the variable unset there is no rule and no chain callback, and the window selection runs the old loop.
• docs/DETAILS.md: the switch, and a row in "Opt-in switches measured and left off".

Validation

2x RX 5700 XT (HIP)

Environment: 2x RX 5700 XT 8 GB (gfx1010, layer split 27), i7-3770 (AVX, no AVX2), 32 GB DDR3-1600, Ubuntu with kernel 7.0, ROCm 7.14.0a20260612 (TheRock gfx101X-dgpu), HIP build -DCMAKE_HIP_ARCHITECTURES=gfx1010 -DSTRATA_ISA_FLOOR=avx, base 870aedd. Qwen3.8-Flash-Next ISTA-DASLab Q2_0, --spec 4 --spec-min-p 0.8 --mtp --kv int8, 256-token answers, thinking off.

Passed:

• Build of this branch (4fb2974), ninja strata: 0 errors.

• python -m unittest serve.test_server: 359 tests OK. The new assertions in test_tune_keys fail against main's serve/server.py.

• Exact, on this branch's build: a fixed expert placement (--adapt-swaps 0 --prompt-cache 0, pcie_frac 0), STRATA_SPEC_PCUM=0, 6 greedy prompts (up to 256 tokens). Rule off (spec_p_cum 0) against 0.75: 6 of 6 texts byte-identical.

• Speed. These runs were on my fork's branch, which has this change plus the AVX-only CPU change ([Engine] AVX-only CPUs: a native pack's Q2_0 experts still run on ggml-cpu's vec_dot, not on q2_avx1.cpp #4728), gfx1010 SDWA dot4 and the HC_Q8 split gate ([Engine] HC_Q8: skip the layers' bf16 hyper-connection copies on a layer split too #2427), the same in every arm:

  • Greedy, short prompts (3 prompts x 2 reps per arm): 31.0 tok/s off (12 runs) against 31.3 at best (spec_min_p 0.5 with product 0.7). That is noise.
  • Greedy, one fixed document, arms in ABBA order, 2 rounds: 31K tokens deep 33.5 off against 33.7 with 0.9; 101K deep 32.6 against 32.5.
  • Sampled (temperature 1.0, top_p 0.95, top_k 20, seeded, 9 runs an arm): 29.9 / 30.0 off (two arms) against 30.4 with 0.9 (+1.5%; 7 of 9 pairs faster, 1 equal, 1 slower). Drafts accepted 74% -> 84%.
  Per-run decode tok/s, sampled

  • off, arm 1: 29.0 29.3 31.8 28.9 28.7 31.3 30.4 28.9 30.8
  • off, arm 2: 30.3 28.7 31.8 29.8 28.3 32.1 29.7 27.8 31.7
  • 0.9: 30.6 29.0 32.1 30.4 27.8 32.0 30.7 29.5 31.6

Failed:
none observed.

Not run (this machine):

• CUDA: run on the P100 machine below. SYCL: the SYCL engine builds its own sycl/src/program/generate.cpp, which this PR does not change, so the variable has no effect there (not built).
• The pipelined and batched decode paths: the rule only applies to the serial loop.
• A byte comparison of this branch with the variable unset against main (done on the P100 machine below: identical).

2x Tesla P100 (CUDA)

Environment: 2x Tesla P100-PCIE-16GB (sm_60; the second card on a PCH x4 slot, no P2P), i7-7700 (4 cores, AVX2), 47 GiB RAM, CUDA 12.4, driver 580, Linux. This branch (4fb2974) built with -DSTRATA_ENABLE_CUDA=ON -DSTRATA_EXPERIMENTAL_SM60=ON, base 870aedd. Qwen3.8-Flash-Next ISTA-DASLab IQ3_XXS with a Q8_0 copy of the dense weights (tools/q8_dense_gguf.py), --resident-experts --layer-split 30 --spec 3 --spec-min-p 0.7 --mtp --kv int8 --max-context 262144, STRATA_Q8_SM60=1 STRATA_STAGE_TRIM=1, STRATA_SPEC_PCUM=0 with the rule set per request (strata_tune.spec_p_cum), 256-token answers, thinking off. Six fixed prompts (story, code, explanation, math, list, advice).

Passed:

• CUDA build of this branch for sm_60, ninja strata: 0 errors. One warning: pcum_cur set but not used (src/program/generate.cpp:12334; requests use req_pcum).

• python -m unittest serve.test_server: 359 tests OK.

• Unset path: this branch's build with the variable unset against main's engine (870aedd), fixed expert placement (--adapt-swaps 0 --prompt-cache 0, pcie_frac 0), 6 greedy prompts: 6 of 6 texts byte-identical.

• Speed, production placement (the adaptive expert cache on):

  • Greedy, 6 prompts x 2 rounds per arm (ABC / CBA): off 41.90 tok/s, 0.75 41.83, 0.9 41.58. Drafts accepted 78.4% / 80.4% / 87.3%.
  • Sampled (temperature 1.0, top_p 0.95, top_k 20, seeded, 9 runs an arm): off 43.53 tok/s against 41.64 with 0.9 (0.9 faster in 5 of 9 pairs; single runs range 34.8-54.2). Drafts accepted 74.0% -> 87.3%.
  • Greedy, one fixed document, arms in ABBA order, 2 rounds: 32,879 tokens deep, off 41.61 against 40.02 with 0.9 (-3.8%); 109,525 tokens deep, off 40.12 against 37.83 (-5.7%).
  Per-run decode tok/s, sampled

  • off: 41.6 51.3 34.8 50.8 42.2 46.7 40.3 54.2 37.9
  • 0.9: 43.1 37.2 42.6 46.9 45.1 39.4 42.2 39.1 40.8

Failed:

• Byte-identical greedy text with the rule on. Fixed expert placement (--adapt-swaps 0 --prompt-cache 0, pcie_frac 0), 6 greedy prompts, and every arm in a fresh server, because later requests in a session also vary run to run on this machine. Rule off against main's engine: 6 of 6 identical. Rule on: 0.75 gives 0 of 6, 0.9 gives 1 of 6.
  Control without this PR: main's engine with spec_min_p 0.8 or 0.9 instead of 0.7 gives 0 of 6 and 1 of 6.
  So on this backend greedy text depends on where the draft window is cut, whatever cuts it. The texts part mid-answer into coherent alternatives ("he spotted something bobbing in the churning foam" / "he spotted a glint on the rocks"). The rule changes the text no more than --spec-min-p does, but the byte-identity seen on the RX cards does not hold here.

Not run (this machine):

• The pipelined and batched decode paths.

Limits

On both machines the gain is small or negative for Qwen3.8-Flash-Next, so the switch stays off by default. It should only pay where a verify token's cost grows with depth (a dense model with full attention), which neither machine has measured with Strata.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N976Q4DZqzvqFCT56yvVXy