[rkcth]

Problem

Batch slots currently verify one token per request, even when solo decoding uses MTP drafts. On a single GPU this can give up much of the solo decode speed as soon as a second request arrives.

Change

With MULTI_CONCURRENCY=TRUE, each active batch slot drafts one token and verifies the current token plus that proposal in a shared window. Each slot keeps private draft KV/state while borrowing the immutable MTP weights and head. The verifier commits only the accepted prefix for each slot; more slots rotate through its eight-row windows. The existing batch path remains the default.

The diff also bounds captured batch-graph layouts, documents the opt-in setting and hardware limits, and removes evaluation-only probes and diagnostics. The comments focus on shared ownership, accepted-prefix state, and slot rotation.

Validation

• HIP build passed on an AMD R9700; python -m unittest serve.test_parallel passed (11 tests).
• Earlier guarded R9700 trials on the same implementation at 8K prompt / 1024 output tokens, matched expert cache and MTP on both sides: two concurrent requests produced identical text and improved aggregate output from 27.48 to 33.63 tokens/s (+22.4%). Three requests produced identical text at 30.72 to 31.60 tokens/s. Four produced identical text but slowed from 30.68 to 29.06 tokens/s. These are single runs, not a cross-hardware throughput claim.
• Earlier guarded trials also covered nine rotating slots, two 115K-token prompts with KV streaming, fixed-seed sampled output, cancellation/replacement, and graph eviction. The PR cleanup removed only test switches and internal notes; it did not get a separate GPU throughput run.

Limits / review focus

Grouped MTP is currently single-GPU only and verifies one proposal per slot. It has not been tested on 16 GB cards or NVIDIA. Per-slot draft state and buffers reduce VRAM available to the expert cache; a smaller card may see less benefit or fail to fit the requested slot count. A longer mixed-workload soak and cross-hardware checks are still needed before enabling this by default.

[bobvious]

NVIDIA result: single RTX PRO 5000 Blackwell 48 GB, Linux, CUDA 13, built on 0.1.39. Model is Qwen3.8-Flash-Next IQ3_S with --spec 4 --mtp. Each request has a distinct ~1.5K prompt and 400 output tokens; 2 runs each.

Total tok/s (last request's TTFT):

concurrent	one at a time	parallel 2 (0.1.39, no MTP)	parallel 2 + MULTI_CONCURRENCY	parallel 4 + MULTI_CONCURRENCY
2	94-96 (5.2 s)	61	84-86 (3.6 s)	79-86 (3.9 s)
4	99-100 (13.3 s)	71	90-93 (11.8 s)	94-96 (8.0 s)

Works and is stable: no errors, both slot drafters loaded, latency clearly better. It recovers most of the batch loss (61 -> 85, 71 -> 95) but doesn't beat one at a time here, where solo verifies up to 4 drafts.

[paulhothersall]

Complementary numbers from a card that holds only 44% of the experts: Tesla P40 24 GB, IQ3_XXS, v0.1.39, --batch 4 (no MTP in slots), 4 to 8 concurrent ~600-token prompts, 200 new tokens each, stream chunks per second (about tokens):

	N=1	N=2	N=4	N=8
one at a time (no --batch): per stream / aggregate	31.5 / 19.5	31.5-32.4 / 19.8	32-34 / 20.3	31-34 / 20.0
--batch 4: per stream / aggregate	30.2 / 18.2	12.8-17.8 / 20.0	6.8-12.4 / 23.6	6.6-11.9 / 23.2

Here batching raises the aggregate by 16% at N=4 and halves the worst first-token wait (32 s to 17 s), while each stream drops to 22-38% of solo. On @bobvious's card (100% hits) parallel 2 lowers the total, so the sign of the aggregate change seems to follow the hit rate. I can run this PR on CUDA/Pascal if that helps (the description says it was validated on HIP).

[poakes28]

NVIDIA result, plus a reproducible crash once the batch-graph cache passes 32 layouts. Single RTX PRO 4500 Blackwell
32 GB (200 W), Ryzen 9 9950X3D, 64 GB DDR5, Ubuntu 26.04, driver 615.71.09, CUDA 13.2. v0.1.39 + this PR (+ #783 for
the throughput rows), Qwen3.8-Flash-Next IQ3_XXS, resident mode (--resident-experts), --spec 4 --mtp, int8 KV,
131K context. Distinct prompts, 512 new tokens each, thinking medium, total tok/s:

	solo	N=2	N=4	N=8	slowest first token, 4 at once
one at a time	133	135	121	—	queued
parallel 4 (0.1.39, no MTP in slots)	108	89	97	96	—
parallel 4 + --kv-resident 20480	131	117	131	127	1.0 s
parallel 4 + --kv-resident 20480 + MULTI_CONCURRENCY=TRUE	133	144	139	138	1.1 s
parallel 2 + --kv-resident 20480 + MULTI_CONCURRENCY=TRUE	139	150	136	132	5.6 s

Same picture as @bobvious's card: it recovers the batch loss and edges past one at a time here (~+5-15 %), mostly a
latency win. Outputs were coherent and matched the other configs' openings. (Most of the gain over the first
parallel 4 row came from --kv-resident: at 131K each slot session was 1.89 GiB, 0.58 GiB with streaming.)

Crash: with MULTI_CONCURRENCY=TRUE the engine exits with code 1 and no error line (the server restarts it, and
the requests in flight get 503) right after it captures its 32nd batch window layout. Last engine log line, e.g.:

strata verify: captured the batch window over slots 2,2,3,3,0,0,1,1

• parallel 4 + MULTI_CONCURRENCY=TRUE, 60 waves of 2-4 staggered requests (48-240 tokens each): fine for ~40
  waves, then the engine exited after exactly 32 captured the batch window over slots ... lines since its start.
• parallel 8 + MULTI_CONCURRENCY=TRUE: exits under 8 concurrent requests (slot rotation reaches new layouts fast).
  Reproduced on v0.1.39 + this PR alone (no perf(cuda): fused decode/verify/MTP kernels and graph launch reductions on 0.1.39 #783).
• Same build, parallel 8, without MULTI_CONCURRENCY: N=4 95 tok/s, N=8 111 tok/s, no crash.
• parallel 2 + MULTI_CONCURRENCY=TRUE: only 2-4 layouts per run, never crashed (incl. 3 x 100K and
  1 x 200K prompts with --kv-resident).

So it looks like the first eviction at graph_limit = 32 in Verifier (src/core/verify.cpp, the
exec_bm_ / commit_bm_ erase + recapture path) rather than anything about 8 slots as such. Happy to test a fix
or a build with more logging around that path.

[rkcth]

I ran a matched upstream v0.1.39 versus PR #846 sweep on both an R9700 and an RTX 4090, using the same IQ3_S model, --spec 4, MTP, 128K context, and one or two requests.
Performance: The PR improved overlapping two-request generation on both cards. On the R9700, the 8K prompt / 512-output pair rose from 29.2 to 35.1 end-to-end tokens/s; at 8K / 1,024 it rose from 38.2 to 48.5. On the 4090, 10 of 11 valid paired cells improved, with a 30% median gain. The 8K / 1,024 pair rose from 11.2 to 18.5 tokens/s; a reverse-order run was 11.7 versus 17.7. Solo and prefill results varied by case and run order, so I would not claim a general gain for either.
Stability concern: An eight-slot HIP run on the R9700 passed 44 distinct batch layouts without a crash. On the 24 GB 4090, the engine exited with code 1 after 12 captures when --vram-reserve-mib 700 left little free VRAM, failing three in-flight requests. Repeating the load with a 2 GiB reserve passed 52 distinct layouts; a 4 GiB reserve passed 48. VRAM grew by roughly 20–25 MiB per new layout, then leveled off after cache eviction. This points to graph-capture headroom as the 4090 trigger, but does not yet establish the cause of the reported RTX PRO 4500 crash at exactly 32 layouts.
I’d suggest checking and logging errors from batch cudaGraphUpload and the following cudaStreamSynchronize, which currently discard their return codes, and accounting for graph-capture memory when setting the VRAM budget. Testing a larger reserve on the PRO 4500 would help distinguish memory pressure from an eviction or CUDA-version issue. I have not measured the throughput cost of a larger reserve, so I would treat it as a diagnostic setting for now.

[rkcth]

Benchmark report · October 5, 2026

Strata PR speed comparison

Upstream v0.1.39 versus the submitted grouped-MTP PR, tested separately on a Radeon R9700 and an RTX 4090. Every table shows upstream → PR in tokens per second.

11 of 12R9700 paired cases improved end-to-end; median gain 32%.

10 of 11Valid RTX 4090 paired cases improved end-to-end; median gain 30%.

How to read this report

Both builds used the same Qwen3.8 Flash Next GSQ-RCO IQ3_S weights, MTP, --spec 4, 128K maximum context, and deterministic prompts. The PR used its grouped-MTP path. Only ratios within the same machine should be interpreted as a software comparison.

• PP is fresh prompt tokens divided by native engine prefill time, excluding cached tokens and queueing.
• TG is generation speed. Solo TG uses the first-to-last streamed-token interval. Paired TG approximates aggregate tokens/s only while both streams were generating; a dash means no useful overlap interval.
• E2E is completed output tokens divided by total request wall time, including prefill and waiting. It is the practical throughput measure here.

Radeon AI PRO R9700

32 GB AMD Radeon AI PRO R9700, HIP build on the AI server

Paired: 11 of 12 paired cases improved end-to-end; median gain 32%. The 512 / 1,024 case was about 5% slower.

Solo: The first-pass solo generation median was about 5% lower with the PR. Solo end-to-end results looked faster, but prompt-processing time varied substantially with run order.

Two simultaneous requests

Prompt / output each	PP	Paired TG	E2E	E2E change
512 / 128	308 → 322	61.1 → 67.1	32.9 → 33.2	+1%
512 / 512	175 → 287	55.8 → 77.6	43.7 → 61.0	+40%
512 / 1,024	169 → 291	57.1 → 50.3	51.8 → 49.3	-5%
2,048 / 128	391 → 644	61.6 → 75.1	17.4 → 26.1	+50%
2,048 / 512	370 → 634	61.6 → 77.0	37.7 → 53.0	+41%
2,048 / 1,024	379 → 633	56.8 → 70.8	45.8 → 60.0	+31%
8,192 / 128	686 → 982	—	9.8 → 13.3	+37%
8,192 / 512	843 → 960	64.0 → 75.1	29.2 → 35.1	+20%
8,192 / 1,024	795 → 1,121	58.2 → 72.3	38.2 → 48.5	+27%
32,768 / 128	833 → 1,091	—	3.1 → 4.0	+29%
32,768 / 512	679 → 964	—	9.6 → 13.2	+37%
32,768 / 1,024	643 → 919	59.4 → 69.3	16.4 → 21.8	+33%

One request

Prompt / output	PP	TG	E2E	E2E change
512 / 128	294 → 305	54.7 → 65.6	30.5 → 34.2	+12%
512 / 512	160 → 318	65.8 → 62.9	45.6 → 51.7	+13%
512 / 1,024	183 → 336	66.1 → 65.1	55.1 → 58.8	+7%
2,048 / 128	368 → 639	71.9 → 60.8	17.5 → 24.1	+38%
2,048 / 512	341 → 585	68.1 → 62.2	38.0 → 43.7	+15%
2,048 / 1,024	368 → 564	68.8 → 62.6	50.2 → 51.3	+2%
8,192 / 128	869 → 1,191	65.2 → 59.0	11.2 → 14.1	+26%
8,192 / 512	829 → 1,169	69.2 → 65.6	29.5 → 34.4	+17%
8,192 / 1,024	861 → 1,198	65.6 → 62.3	40.7 → 43.9	+8%
32,768 / 128	792 → 1,136	67.0 → 64.8	2.9 → 4.1	+40%
32,768 / 512	793 → 1,138	64.7 → 66.4	10.3 → 13.8	+35%
32,768 / 1,024	794 → 1,140	64.9 → 62.7	17.8 → 22.5	+27%

Reverse-order controls

These four cases ran again with the PR first, then upstream, each on a fresh engine. The last number in the case is the request count.

Prompt / output / requests	PP	TG	E2E
2,048 / 512 / 1	637 → 654	46.4 → 58.9	36.0 → 43.3
8,192 / 512 / 2	1,127 → 954	60.4 → 72.4	32.1 → 34.2
8,192 / 1,024 / 2	965 → 927	55.8 → 68.7	39.4 → 45.5
32,768 / 128 / 1	907 → 1,134	56.1 → 66.8	3.3 → 4.1

GeForce RTX 4090

24 GB NVIDIA GeForce RTX 4090, AMD EPYC 7V13 host, driver 595.91.07, CUDA 13.0

Paired: 10 of 11 valid paired cases improved end-to-end; median gain 30%. The 512 / 1,024 case is excluded because one stream stopped early in both builds, twice.

Solo: The first-pass solo generation median was about 14% faster with the PR, although all three 2K solo cells were slower. A reverse-order 2K / 512 control slightly favored the PR instead.

Two simultaneous requests

Prompt / output each	PP	Paired TG	E2E	E2E change
512 / 128	502 → 542	14.3 → 19.1	13.0 → 16.9	+30%
512 / 512	464 → 463	13.6 → 26.0	14.0 → 25.4	+82%
512 / 1,024	Excluded: incomplete output in both builds
2,048 / 128	1,110 → 1,085	16.3 → 17.2	13.2 → 13.7	+3%
2,048 / 512	1,075 → 1,035	17.2 → 16.1	16.8 → 15.6	-7%
2,048 / 1,024	1,007 → 1,006	12.5 → 15.0	13.0 → 15.6	+20%
8,192 / 128	2,647 → 2,748	15.1 → 23.9	11.2 → 15.6	+38%
8,192 / 512	2,769 → 2,880	15.6 → 19.4	14.8 → 18.2	+23%
8,192 / 1,024	3,104 → 3,040	11.1 → 18.5	11.2 → 18.5	+65%
32,768 / 128	2,690 → 2,745	13.6 → 24.6	6.2 → 7.7	+25%
32,768 / 512	2,728 → 2,770	16.5 → 26.6	12.2 → 16.9	+39%
32,768 / 1,024	2,653 → 2,654	8.8 → 16.8	8.5 → 14.8	+74%

One request

Prompt / output	PP	TG	E2E	E2E change
512 / 128	432 → 434	13.6 → 17.7	12.0 → 14.9	+25%
512 / 512	482 → 526	18.6 → 22.4	17.8 → 21.4	+20%
512 / 1,024	526 → 529	19.1 → 23.9	18.7 → 23.3	+25%
2,048 / 128	1,049 → 1,053	17.2 → 13.1	13.4 → 10.8	-19%
2,048 / 512	1,095 → 1,042	19.8 → 17.3	18.4 → 16.2	-12%
2,048 / 1,024	1,042 → 997	18.3 → 16.0	17.6 → 15.5	-12%
8,192 / 128	3,368 → 3,560	17.8 → 18.9	13.0 → 14.0	+8%
8,192 / 512	3,372 → 3,520	17.7 → 24.7	16.3 → 22.2	+36%
8,192 / 1,024	3,688 → 2,903	15.9 → 16.0	15.4 → 15.3	-0%
32,768 / 128	3,346 → 3,398	13.1 → 21.6	6.4 → 8.1	+26%
32,768 / 512	3,377 → 3,439	16.4 → 22.4	12.4 → 15.7	+26%
32,768 / 1,024	3,372 → 3,404	16.9 → 18.0	14.5 → 15.3	+6%

Reverse-order controls

These four cases ran again with the PR first, then upstream, each on a fresh engine. The last number in the case is the request count.

Prompt / output / requests	PP	TG	E2E
2,048 / 512 / 1	841 → 953	15.0 → 15.8	13.9 → 14.7
8,192 / 512 / 2	2,065 → 3,212	11.7 → 16.8	11.0 → 15.8
8,192 / 1,024 / 2	2,106 → 2,563	11.7 → 18.1	11.7 → 17.7
32,768 / 128 / 1	2,510 → 3,354	12.4 → 16.4	5.4 → 7.1

What the comparisons support

• Two-request generation improved on both GPUs. On the R9700, the 8K / 512 and 8K / 1,024 gains persisted when the run order was reversed. On the 4090, those same paired cases also remained faster in reverse-order controls.
• Prefill gains are uncertain. R9700 first-pass PP often looked substantially faster, but a repeated 2K / 512 solo case changed from 341 → 585 PP to 637 → 654 PP. On the 4090, first-pass paired PP was approximately flat overall.
• Solo performance is mixed. The R9700 first-pass solo TG median was about 5% lower with the PR. The 4090 first-pass solo TG median was about 14% higher, but its 2K cases were slower; a repeated 2K / 512 case reversed that apparent loss.
• One 4090 cell is excluded. At 512 / 1,024 with two requests, one stream stopped early in both builds on two attempts. Comparing those wall times as full-length outputs would be misleading.
• Stability is a separate concern. The R9700 HIP build passed 44 distinct eight-slot graph layouts. On the 4090, an eight-slot run with 700 MiB VRAM reserved exited after 12 captures; runs reserving 2 or 4 GiB passed the 32-layout eviction point. Speed was measured with two slots and 700 MiB reserved, so these tables do not establish the throughput cost of a larger reserve.

Each main-sweep cell was observed once, with four selected reverse-order controls per machine. Small differences are less reliable than the repeated, larger concurrency gains. The RTX 4090 rental cost $1.04 in total, including compute, storage, and bandwidth, and was destroyed after testing.

[xdbxdbx]

Found while building on this: with MULTI_CONCURRENCY=TRUE and --spec below 2, the batch_mtp check runs before o.mtp is cleared for --spec < 2. The run goes on with batch_mtp true and slot_mtp empty, and slot_mtp[b]->bind(...) then indexes past the end of the empty vector at start-up. Checking o.spec >= 2 in the batch_mtp condition (or clearing batch_mtp with o.mtp) avoids it.

I made PR #1010 with fixes btw.

[Niko1221]

Thanks, taken into 0.1.40 with changes. It is opt-in (--batch-mtp or STRATA_BATCH_MTP=1). It warns and batches as usual when it cannot run, instead of exiting. A 1-row group keeps the 0.1.39 commit kernel, the batch graph cache is bounded (LRU) only under the flag, and the draft K/V copy uses the drafter's geometry. Without the flag the batch path is 0.1.39's. Your +31 to +39% with 2 to 4 clients is the reason it is in. More NVIDIA numbers are welcome.