Repository navigation
"parallel": 2 lowers total throughput on a single 48 GB card (100% expert-cache hits): keep MTP drafts in batch windows? #857
Description
Could batch windows keep MTP drafts? On one card where the experts fit, "parallel": 2 lowers total throughput, because each slot decodes one token per window.
v0.1.39, one RTX PRO 5000 48 GB, Xeon 6148 (AVX2 kernels), Qwen3.8-Flash-Next IQ3_S, --spec 4 --mtp, 262K ctx. Decode expert cache hits 100%. Rates are streamed tokens per second, total across all requests:
no parallel |
parallel: 2 |
|
|---|---|---|
| 1 request | 97 | 64 |
| 2 at once | 103 | 61 |
| 4 at once | 100 | 71 |
strata batch: 800 windows, avg 1.99 rows, 16.71 ms/window = run 15.88 (CUDA0 GPU-reach wait 11.34 + CPU experts 3.05) ...
Happy to test a build.
Activity
fioruccione commentedon Oct 5, 2026
Another data point for this, from a very different rig: two 16 GB cards in a layer split, experts only partly in VRAM. Same conclusion: without MTP in the batch windows,
"parallel": 2moves who waits, but adds no throughput.Rig: v0.1.39 (6f32ec0), Linux, CUDA 12.6 local build, 2× RTX 4060 Ti 16 GB (
layer_split: auto, CUDA1 runs layers 28-47), Threadripper PRO 3975WX 32C, 126 GB DDR4. Unsloth UD-IQ4_XS,--max-context 131072 --kv int8 --kv-resident 32768 --spec 4 --mtp --remote-expert-opt --pcie-frac 0.00 --spec-min-p 0.50, vision on (CPU encoder). Decode expert-cache hits 75-90%.Method: requests sent straight to
serve/server.py(streaming,reasoning_effort: low, temperature 0.7), 300 new tokens each, ~700-1100-token prompts, 3 rounds, with a random prefix on every prompt so the prompt cache cannot help. "Long + short" = an 18,099-token prompt, plus a 73-token question sent 3 s later.parallelparallel: 2parallel: 2+--batch-groups 2Notes:
--batch-groups 2on this 2-GPU split: in the long + short case, the long request stays in its slot at 13.5 tok/s after the short one has finished, and ends at ~35 s instead of ~22 s. It never seems to go back to the solo path with drafts. Is that expected for a pipelined slot ("a pipelined slot is not kept as a conversation cache"), or should the return to solo apply there too?--batch 2on this layer split (two identical cards, experts not 100% resident). So --batch on a 100%-resident layer split: engine dies at "captured the batch window over slots 0,1" (zero-doorbell path?) #776 /--batchwith--layer-splitexits ("verify batch: layer 34 never rang") with a 3070 first and a P40 second; works with the P40 first (CUDA, v0.1.39) #929 do not reproduce here.MTP drafts in the batch windows would be what makes
"parallel"worth turning on for a small office server like this one (about 10 users, rarely more than 2 at once). Happy to test a build on this rig.gucasbrg commentedon Oct 6, 2026
Second rig confirming the headline, plus one caveat this thread should have: at a 1M
--max-contextpool, the slots themselves cost expert cache, which makes"parallel"even less attractive than the numbers above suggest.Rig: 2× RTX 5090 32 GB (layer split auto), EPYC 9654, 377 GB RAM, Linux 6.8, driver 595.71; engine main@6f32ec0 (0.1.39), IQ3_S native, int8 KV,
--kv-resident 32768,--spec 4 --spec-min-p 0.5 --mtp, experimental projection on (identical in all runs),--max-context 1048576 --rope-scaling yarn --rope-scale 4.Decode, Chinese prose prompts, greedy, per-request tok/s and aggregate (wall of the whole burst):
--batch 2--batch 4 --batch-groups 2--batch 2was the worst: the solo→slot STOP-and-move when the second request arrives dominated (two ~270-token answers each took ~11 s). With 4 slots the speeds were as uneven as the other 2-GPU report above, and nothing went back to the solo path after the burst. No engine errors on this split (IQ3_S, unlike #776/#929/#890).The pool-size caveat: with
"parallel": 4the engine reserved slot sessions large enough that the expert cache fell from 12,560 slots / 22.65 GiB to 8,953 / 16.12 GiB (per card; ~1.6 GB/card for 4 slots at a 1M pool) — hits dropped to 94.7–99.4% (+0.2–2.7% over PCIe). The "0.56 GiB per slot at 32K" figure in BATCHING.md is what makes setup say yes; at a 1M pool the per-slot session is multiples of that (the QSA state scales with--max-context, as in #781), so "experts mostly fit" is no longer sufficient — the slots eat exactly the cache that the argument relies on.Why MTP-in-batch would fix this for us: we used to serve a dense 27B on SGLang with continuous batching and speculative decoding together: 4 concurrent = ~113 tok/s each, 2.9× aggregate. Without drafts in the slots, MoE batching only amortizes the shared experts, and our per-request numbers collapse exactly by the draft factor (186/1.8 ≈ 103 ceiling, then ÷ window rows). With drafts in the batch windows, I'd expect ~80–100 per stream at 4-way and 1.8–2.2× aggregate here — that would make
"parallel"worth turning on for a small office like ours.Happy to test a build on this rig — a dual-5090 + 1M-pool + layer-split case seems like a decent boundary test. Also happy to re-run any measurement above with extra instrumentation (
STRATA_DECODE_TIMING=1) if useful.Niko1221 commentedon Oct 6, 2026
Known: with parallel: 2 and 100% cache hits, the batch slots decode one token per window without MTP. 0.1.40 adds --batch-mtp (PR #846, opt-in) which targets exactly this; until you try it, on a single-user card leave parallel unset.
Niko1221 commentedon Oct 7, 2026
Shipped as opt-in in 0.1.40: --batch-mtp runs the MTP draft inside the batch windows, which is what was missing when parallel:2 dropped from 97 to 64 tok/s. Please try it on the RTX PRO 5000 and tell us the numbers. Closing this issue; reopen if it does not help.
bobvious commentedon Oct 7, 2026
Tried
--batch-mtpon 0.1.40.2 (e8ca9af), RTX PRO 5000 48 GB, IQ3_S native, int8 KV,--spec 4, 262K context. Same engine and config for both arms, isolated host. Requests were 3 reps of distinct ~1.5K-token prompts, 400 tokens out, temperature 0.Total tok/s (median of 3):
"parallel": 2+--batch-mtpThere were no crashes; the #1121 fix holds on IQ3_S. Drafts are batched (avg 3.50 rows/window), but windows take 17.6 ms, of which 14.05 ms is "CUDA0 GPU-reach wait", with VRAM expert hits about 100%. So on one GPU it is still below single-slot, and we're staying single-slot. Thanks for the fix.
batch pipeline: a pipelined slot is a cache again, so a request left …
Merge f-141-conc: --batch-groups auto for layer-split batches, #857 f…
Merge upstream v0.1.41 (fb58e0d) into gfx906
batch の pipeline: パイプラインのスロットも会話のキャッシュとしてまた使う（upstream fc84443）
Niko1221#1906: a slot's turn checkpoints move with its conversation i…
batch pipeline: a pipelined slot is a cache again, so a request left …
Merge f-141-conc: --batch-groups auto for layer-split batches, Niko12…
web: two Monitor cards for the metrics 0.1.41 added
web: two Monitor cards for the metrics 0.1.41 added
batch pipeline: a pipelined slot is a cache again, so a request left …
web: two Monitor cards for the metrics 0.1.41 added