[Vop3p]

Summary

STRATA_PIPED_CONCURRENT=1 is off beside STRATA_SPLIT_MTP_BATCH=1 (STRATA_PIPED_CONCURRENT is off beside STRATA_SPLIT_MTP_BATCH), because the batched draft K/V (Prefill::draft_kv) would allocate on the read's thread. This adds an opt-in, STRATA_PIPED_CONCURRENT_SMB=1, that removes the allocation and lets the two run together. Off (the default), nothing changes.

What allocates

Reading Prefill::draft_kv:

• its row buffers are carved from the prompt path's scratch region (m.region): no allocation;
• its two uploads (the bounds, 16 B; the tokens, nb x 4 B) go through up_ring, and are smaller than what reserve_uploads already reserves before the read (T x K x 4 B);
• the Q8_0 projections run through mmq::Context (llama.cpp's MMQ): the context is created at first use, and llama.cpp takes its stream-k fixup buffer from the context's pool, which grows at the first larger request.

So the MMQ pool is the one thing that can allocate. On devices whose MMQ configs do not use stream-k it never does (in the ggml we build with, all 254 RDNA2 configs have stream_k = false); on NVIDIA (Volta and newer) and CDNA it can.

The change

• mmq::Context::reserve_fixup(bytes): takes bytes from the pool and gives them back, so a later run that needs at most that much allocates nothing.
• Prefill::reserve_draft_kv(): creates the MMQ context and grows the pool to the largest fixup, one 128 x 128 tile of floats per SM (nsm x 128 x 128 x 4 B).
• In generate.cpp, with STRATA_SPLIT_MTP_BATCH=1 and STRATA_PIPED_CONCURRENT_SMB=1, it is called on the drafter's stage before each concurrent read, beside the existing mtp.reserve_prefill and sp.reserve_uploads, with nothing in flight. It logs once: STRATA_PIPED_CONCURRENT_SMB=1: the batched draft K/V (STRATA_SPLIT_MTP_BATCH) runs beside the concurrent read; its MMQ pool is grown before each read.
• ConcurrentReadSetup::split_mtp_batch_reserved: concurrent_read_off no longer names STRATA_SPLIT_MTP_BATCH when it is set. batch_read_test covers the new case (checked that it fails when the new assertion is flipped).

Measured

2x RX 6900 XT (gfx1030), ROCm 10.0, Linux. Flash-Next GSQ-RCO IQ3_S, 262144 context, --kv int8 --kv-resident 32768, layer split with part of the experts in RAM, --spec 3 --spec-min-p 0.7, "parallel": 2, --batch-groups 2 --batch-mtp, greedy. This branch's binary, a fresh engine per arm. Both arms have STRATA_PIPED_CONCURRENT=1; "with" adds STRATA_SPLIT_MTP_BATCH=1 STRATA_PIPED_CONCURRENT_SMB=1.

Stability: 9 arms, 0 stalls. In the three full runs (two with, one without), each arm read a 64K prompt while a short request arrived (4 give-ways per arm, short TTFT 8.0-8.1 s), two streams, a hand-off, and a needle test across the yield point (6 of 6 correct).

Prompt read, 3 interleaved pairs (each arm's outputs are token-identical run to run, so these are the same requests):

	4K	32K	128K
without	951-954	1,723-1,724	1,990-1,991
with	990-991 (+4.0%)	1,754 (+1.8%)	2,006-2,008 (+0.9%)

The cost, which is STRATA_SPLIT_MTP_BATCH itself and not this change: the draft layer's K/V come from Q8_1 x Q8_0 MMQ instead of the drafter's own mmvq, so the drafts move (as docs/MULTI_GPU.md says). On the same requests, drafts accepted fell from 87.5% to 85.3% (and 84.5% to 82.4% in the full runs, 6 of 6 comparisons lower), and decode was about 3% lower at 4K and 32K, unchanged at 128K. On our rig the two roughly cancel, so we leave it off. On rigs where the prompt read is a larger share, or where STRATA_SPLIT_MTP_BATCH gains more, it may be worth it.

Not tested

The path this guards: no NVIDIA card here, so the stream-k fixup allocation that this change pre-grows has not been exercised. On gfx1030 the MMQ never takes from the pool, so our runs show the two coexisting, not that the reservation is sufficient on a stream-k device. A run on CUDA with STRATA_PIPED_CONCURRENT=1 STRATA_SPLIT_MTP_BATCH=1 STRATA_PIPED_CONCURRENT_SMB=1 and a few long reads beside a decoding slot would settle it.

---

Developed with an AI coding assistant; every number above was measured on 2x RX 6900 XT (gfx1030), ROCm 10.0, Linux.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.