[gravitomagnetic]

Results-only community benchmark per docs/COMMUNITY_BENCHMARKS.md. No engine changes.

Hardware: RTX 5090 32 GB (Gen5 x16), Ryzen 9 7950X, 122 GB RAM, Ubuntu 26.04.1, driver 595.91.07, source commit 9259cad (v0.1.31), CUDA 13.4.2.

Model/config: Qwen3.8-Flash-Next IQ3_S (split GGUF, sha256 recorded), --max-context 1048576 --rope-scaling yarn --rope-scale 4, --kv int8 --kv-resident 32768, --expert-cache 7500 (auto-fills to 9,804 slots), MTP --spec 4 --spec-min-p 0.5, vision on (no images in requests).

What's measured:

• Cold prefill ~2,500 tok/s at 4k, ~3,300 at 32k, ~3,400 at 128k, ~3,000 at 512k (engine-reported timings).
• Decode 94-105 tok/s with reasoning enabled (temp 1.0 / top_p 0.95 per the HF card); 71-87 tok/s greedy instruct mode.
• Prefix reuse: 500,553 of 500,558 tokens reused in 36 ms on warm repeat.
• Expert cache hit rate 96-98% during 512k runs.
• Needle recall via tools/needle_bench.py: 5/5 found at depth 50%, up to a 1,048,265-token prompt.

Honest caveats (Note A in the README): an apparent ~65 s per-request stall in early drafts was disproven by a controlled probe and re-attributed to serial scheduling behind a co-tenant 208k-token prefill (this machine runs a live 1M-context agent session alongside the benchmark). Client-side TTFT figures are contaminated by that traffic; engine per-request timings are unaffected. The surviving observation — one in-flight 200k+ prefill blocks all other clients for up to ~60 s — may interest whoever works on serve scheduling; happy to open a separate issue with logs if useful.

Files: README.md (report), agent_bench.jsonl (20 runs), needles.json (recall), bench_agent_workload.py (script), strata-iq3_s.json (config, paths genericized).

[j-luwierski]

Hi @Niko1221

I took a closer look at the ~60-65 s blocking observed when another request is doing a large 200K+ prefill, and I think there may be a middle ground between the current strictly serial serving model and full multi-request / multi-slot concurrency.

From what I can see, the current behaviour is expected from the server architecture: "serve/server.py" holds the service FIFO for the whole request, including both prefill and decode. "StrataEngine" also explicitly assumes that requests are serialized, which is why a single stdin/stdout pipe is sufficient.

So with a request like:

A: 208K cold prefill
B: small interactive request

B cannot start until A finishes its complete request. This creates head-of-line blocking, and a long prompt can therefore delay an unrelated interactive request for tens of seconds.

I understand the reasoning in #465 for not implementing full concurrent decoding. Supporting several live sequences at once would require per-slot KV, draft state, checkpoints, scheduling, shared expert-cache handling, etc., and would be a much larger architectural change.

However, I wonder whether a much smaller form of cooperative prefill preemption could be useful.

The prefill path already has natural chunk boundaries:

for (int64_t c0 = 0; c0 < n; c0 += m.T) {
...
}

and near the end of each chunk there is already a synchronization / callback point:

cudaStreamSynchronize(m.cs);

if (on_stage_chunk)
on_stage_chunk(...);

if (on_chunk)
on_chunk(...);

So instead of trying to execute multiple sequences concurrently, the idea would be:

request A:
prefill chunk
prefill chunk
prefill chunk

request B arrives

finish current A chunk
park A
run B
restore A
continue from the next chunk

In other words, the engine would still execute only one sequence at a time. This would be preemption between safe prompt chunks, not cross-request batching.

Conceptually:

current:
A: [======================= 200K prefill =======================][decode]
B: ..............................................................[start]

proposed:
A: [chunk][chunk][chunk]
|
+-- safe yield
B: [prompt][decode]
A: [chunk][chunk]...[decode]

What makes this seem potentially practical is that Strata now already has most of the state save/restore machinery through the conversation cache.

A parked conversation snapshot already contains the important persistent state, including:

• running model state
• conversation checkpoints
• used KV pages
• draft-layer KV

and there is already a "conversation_snapshot_restore(...)" path.

So rather than introducing several permanently active GPU sessions, a long prefill could potentially be suspended at a chunk boundary, its state parked using the existing snapshot infrastructure, a queued request serviced, and then the original request restored.

There are some details that would need care before a chunk can be considered a safe preemption point.

For example, in "Prefill::run()":

ss.ple_prev[0] = prev[0];
ss.ple_prev[1] = prev[1];

is currently committed after the complete prefill. If suspension is allowed between chunks, this state would need to be committed at the boundary before taking the snapshot.

Similarly I think a safe yield point would need to guarantee that all asynchronous work associated with the chunk is finished, including at least:

compute stream
copy stream
expert issuer thread
PLE read-ahead
multi-GPU/stage hand-off, if applicable

before parking the session.

A possible interface could be something like:

enum class PrefillAction {
Continue,
Yield,
Stop,
};

std::function<PrefillAction(int64_t position)> on_chunk_boundary;

with the normal fast path simply returning "Continue".

The scheduler could then behave roughly as:

if no other request is queued:
continue prefill normally

if another request is waiting:
finish the current chunk
make the current session snapshot-safe
park it
service the queued request
restore the previous session
resume at the next prompt chunk

This should avoid changing the decode execution model at all.

One further possibility would be to keep large chunks for throughput while the server is idle, but reduce the scheduling quantum when another request is waiting.

For example, with ~3000 prompt tok/s:

8192 tokens ~= 2.7 s
4096 tokens ~= 1.4 s
2048 tokens ~= 0.7 s

So even without true concurrency, the worst-case additional latency for an interactive request could potentially fall from ~60 s to approximately one prefill chunk plus snapshot/restore overhead.

I would not suggest releasing the existing FIFO directly in the middle of "Prefill::run()", because the current engine owns one live "SessionState" and a second request would overwrite KV/GDN/QSA/PLE/draft state. The important part would be to make the current sequence explicitly suspendable and restore it before continuing.

So the proposal is intentionally narrower than #465:

not:
A + B execute concurrently

but:
A -> suspend -> B -> restore A -> continue

I think this could preserve Strata's current single-sequence execution model while addressing the specific long-prefill head-of-line blocking seen in the 5090 benchmark.

If this direction sounds reasonable, I would be interested in investigating it further and prototyping the smallest version first, probably only allowing preemption during prefill and only at an existing chunk boundary.

[j-luwierski]

#656

[gravitomagnetic]

Update (2026-10-04): re-probed on 0.1.39 — the regression this report documented is closed, and we measured the preemption path @j-luwierski's analysis targets.

Same box, same protocol (unique-salt cold prompts, 3-run medians, engine-reported timings, no other traffic). 0.1.39 source build (commit 6f32ec0, CUDA 13.4, driver 595.91.07, STRATA_NO_LARGEPAGES=1 per #771, ring #583 default-on):

size	0.1.31 (this PR's baseline)	0.1.36 (regressed)	0.1.39
4K	2,560	169.5	2,786
32K	3,281	190.1	3,589
128K	3,379	183.2	3,533

0.1.39 beats the 0.1.31 golden baseline at 32K (+9%) and 128K (+5%). Decode 110–163 t/s.

Parallel requests ("parallel": 2 → --batch 2), first 5090 numbers we've seen:

• Lone-request cost: 2,792 / 3,608 / 3,533 t/s — within 0.2% of solo. The fallback-to-solo-path claim holds on this hardware.
• Head-of-line blocking A/B (the ~60–65 s scenario from @j-luwierski's comment): A = 92,451-token cold prefill (39.0 s wall, 3,404 t/s); B = 92-token request fired 5 s later. Under strict FIFO B waits ~36 s; with BYIELD preemption B completed in 8.2 s, twice, identical reps — ~4.4× better, and A pays only 3.7% prefill rate.
• VRAM budget note: each slot session costs 2.27 GiB carved from the expert cache. On this config (32 GB, 1M ctx YaRN×4, expert-cache 7500) --batch 4 OOMs during native dense upload at load; --batch 2 loads clean. Worth a line in BATCHING.md for 32 GB cards running 1M context.

Full config dump in the PR body; happy to add the 0.1.39 row to the results table if you want this report updated in-place rather than as a comment.

[gravitomagnetic]

Preemption A/B: measured head-of-line blocking vs --batch on RTX 5090 / 1M context (supersedes the preliminary numbers in my 10-04 comment — clean protocol, both legs, engine idle)

Follow-up to @j-luwierski's analysis above and the #465 shipping note. Setup: same box as my bench report (5090 32 GB, driver 595.91.07, 0.1.39 commit 6f32ec0, IQ3_S MoE, 46.84 GiB experts RAM-streamed, 1M ctx YaRN×4, kv int8, STRATA_NO_LARGEPAGES=1). Collision: request A = 125.2K-token cold prefill; request B = 49–92 token ping fired 5 s later. Two B flavors mirroring the real clients on this box: xhigh (agent brain: thinking on, temp 1.0, top_p 0.95, preserve_thinking) and instruct (reflex/voice: enable_thinking: false, temp 0.7, top_p 0.8, presence_penalty 1.5). 2 reps per leg, gateway stopped, engine restarted between legs. B's queue wait = client wall − B's own serve time.

config	B mode	rep 1	rep 2	median
FIFO (no parallel)	xhigh (thinking)	57.1 s	41.4 s	49.2 s
--batch 2	xhigh (thinking)	7.6 s	10.8 s	9.2 s
FIFO (no parallel)	instruct (no-think)	31.5 s	31.5 s	31.5 s
--batch 2	instruct (no-think)	4.7 s	4.8 s	4.8 s

Preemption works: 5.3× (thinking) / 6.5× (no-think) reduction in interactive latency. Under FIFO, B waits out A's prefill and most of its decode (rep 1: A wall 62.1 s, B queue 57.1 s) — exactly the prefill+decode hold @j-luwierski described. Under --batch 2 the serve log shows the BYIELD interleaving directly:

[strata] reading the prompt: 32,768 of 125,220 tokens, 9 s so far
[strata] reading the prompt: 86 of 91 tokens, 5 s so far        ← B, inside A
[strata] reading the prompt: 40,960 of 125,220 tokens, 13 s so far

Cost to the big request: prefill 3,471–3,483 → 3,359–3,405 t/s (−2%); decode 115.4–117.3 → 101.7–104.3 t/s (−13%); A's total wall +2 s.

Verdict for single-big-GPU agent boxes: preemption converts "the agent ignores you for a minute" into "answers in seconds while a 125K prefill runs" — the felt difference whenever a background/voice/daemon client collides with a big agent turn (this box runs a persistent chat daemon + the agent on one engine; those were the collisions). --batch 2 is worth it; --batch 4 OOMs at load on 32 GB (see my comment on #465 for the VRAM budget finding).

Raw data: 4 JSON files (preempt_parallel{1,2}_{xhigh,instruct}) + full terminal and engine logs — happy to upload or PR into bench/results/.

[Niko1221]

Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when main was force-pushed (that is the "base branch was force-pushed" event above). It is not a rejection.

GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new main, cherry-pick your own commits onto it (git fetch upstream && git switch -c my-fix upstream/main && git cherry-pick <your commits>), and open a new PR. Or simply leave it: when we take your change for a release we port it onto the new main ourselves and keep you as its author.

Sorry for the noise, and thank you for the contribution.

[Niko1221]

Thanks for the report. It is in 0.1.40.2 as bench/results/2026-10-02-community-rtx5090-1m-agent, with your authorship kept. Raw dumps and logs were trimmed (see TRIMMED.md in the folder); the README, summaries and configs are unchanged. The row is in bench/results/COMMUNITY.md.