[havok2-htwo]

This PR adds results only: the report folder
bench/results/2026-10-14-community-4x-rtx-3090-0144/
and one bullet in docs/COMMUNITY_BENCHMARKS.md. It changes no code.

Hardware: 4x RTX 3090 24 GB at a 380 W limit each. PCIe: two cards at Gen3 x16, two at Gen4 x8. The rest of
the machine is a Threadripper 3990X (no AVX-512) with 251 GiB RAM and an NVMe RAID0, running Ubuntu 24.04,
driver 580.178.04 and CUDA 13.0.

Model: Unsloth UD-IQ4_XS (shards match the pins in setup.py), with the native-experts pack and the MTP draft
made by setup. The context is 262,144 tokens with --kv int8 --kv-resident 32768, the layer split is 12,25,37,
and all experts are resident in VRAM. Text only.

Configs (same day, same engine arguments, each build with its own serve/server.py):

• stock: v0.1.44 ed4773eb, built from source with setup.py's CUDA flags. It runs 8 slots in 4 groups of 2.
  8 is the most 0.1.44 runs on this split with its default pipelined groups (without --batch-mtp); with
  --batch-mtp and serial windows (--batch-groups 1 or STRATA_BATCH_MTP_SERIAL=1) it accepts more, which we
  did not test. v0.1.44.1 (296b59d2) came out a few hours before the run and was not measured.
• modified: 0.1.43 plus our unmerged patch series, built with -DSTRATA_MMQ_TAB=ON. It runs 16 slots in 4
  groups of 4, with the opt-in switches listed in the README. This is the build we run in production. Source,
  pinned: havok2-htwo/Strata at 08830ddf;
  the same commits as patches 0001-0052 in
  havok2-htwo/StrataQ38FN at 4ece0ca
  plus the last commit 08830ddf (server only).

Aggregate output tok/s (stock / modified), 1 run per arm:

Workload	stock 0.1.44	modified 0.1.43
1 user, ~147-token prompt, 512 out	111.7	116.2
8 users, same	314.4	399.4
16 users, same	292.9 (TTFT p50 8.69 s)	516.7 (0.89 s)
16 users, ~1.2K-token prompt	238.1 (TTFT p50 6.96 s)	406.5 (1.91 s)
16 agents x 3 turns, temperature 0.7	92.0 (follow-up TTFT p50 9.91 s)	151.9 (1.52 s)

The greedy replies were byte-identical between the two builds for all 82 sweep requests. The README also
gives the engine's own timings:

• prompt read rates
• the admission copy into a slot, as the median per phase: 153-197 ms on stock, 1.0-13.2 ms on the modified build
• solo MTP acceptance in the 1-user level, identical in both builds (586 of 948, 61.8 %)
• a note on [0.1.43] CUDA layer split (2× RTX 3090): decode windows lose async launch, solo 131.4 → 39–76 tok/s, prefill −35% #2038: every solo request drafted in both builds, and a request admitted into a batch slot logs one
  strata serve: prompt line with 1 generated token and 0 of 0 drafts

Limitations:

• The 16-user rows mix 16 against 8 slots with everything else that differs. The 8-user row is the closer
  comparison, and there both builds carry 2 rows per group-step. Even there the gap mixes the patches and switches,
  the -DSTRATA_MMQ_TAB=ON build option and the 0.1.43 against 0.1.44 base.
• The modified build used about 26 GiB more system RAM after load (96.2 against 70.4 GiB, whole machine).
• No row isolates a single patch.
• There is one run per arm, in one order (modified first).
• Images, prompts above 20K tokens, more than 16 users, --batch-mtp (also stock above 8 slots in serial
  windows) and v0.1.44.1 are untested.

Two parts of the series (more than 8 slots on a layer split, and the device-to-device slot copy) would come as
separate PRs, each with its own A/B. This report does not measure either alone.

🤖 Generated with Claude Code

[havok2-htwo]

Thank you for porting the series into 0.1.45 and for the R9700 numbers. I have two follow-ups from a run on stock 0.1.45 (477ac13c, built with setup.py's release flags, CUDA 13.0) on 4x RTX 3090 24 GB (PCIe). The model is Qwen3.8-Flash-Next UD-IQ4_XS with the native pack, run with --layer-split 12,25,37 --trim-stage-weights --resident-experts --prefill 512 --spec 4 --kv int8 --kv-resident 28672, 262K context, greedy.

STRATA_SLOT_CKPT_MOVE=1 and tools/batch_groups_cache_test.py. I ran the test unchanged. All arms had the other merged switches on, and they differ only in MOVE and CKPT_POOL. One run per arm:

	merged switches, no MOVE	+ MOVE, no CKPT_POOL	+ MOVE + CKPT_POOL
U (--batch 2 --batch-groups 2, the docstring's args)	PASS	PASS	FAIL
B16P (--batch 16, same cache args)	PASS	FAIL	FAIL

In each of the three FAIL runs, the request list differs from the PASS runs by a single extra line in round 1, and that line comes from how the test counts:

• PASS: (3141, 0), (3142, 0), (3245, 3220), (3246, 3245), (3246, 3221), ...
• FAIL: (3141, 0), (3142, 3141), (3142, 0), (3245, 3220), ...

In the FAIL runs, conversation A started alone on the solo path. When B arrived, A was moved into a batch slot (StrataEngine.generate_batched in serve/server.py, prompt = list(ids) + out # promoted: it continues in a batch slot from here), and the engine logged the same request a second time as prompt + first token: 3142 tokens = 3141 reused + 1 read. The test assumes round 1 is exactly two lines (reqs[2:]). With the extra line, B's real cold read (3142, 0) falls into the checked part, and the test fails. The real round-2/3 requests reuse 99.2 % or more in all six stock runs. The answers are identical round by round between every MOVE-on and MOVE-off pair.

Whether A starts solo in round 1 looks like the arrival race between the test's two threads. In the PASS runs the log has slot 0 takes 3141 tokens before A's prompt line. In the FAIL runs A's solo read ends first and slot 0 takes 3142 tokens follows. It happened in 3 of the 4 runs with MOVE and in 0 of the 2 without. With one run per arm I can't rule out that MOVE makes the solo start more likely. The last line of each run, (3375, 3374), is the same kind of re-log in the other direction: slot back to solo (#857).

Suggested test fix: count one line per request, the read itself, and skip these continuation lines. A request that moves between the solo path and a slot is logged again with its prompt plus the tokens it generated so far, all of them reused but the last one (reused = prompt - 1). No real follow-up in this test reads only one token, since each new turn adds about 25 tokens:

reqs = [(p, r) for p, r in reqs if r != p - 1]   # drop solo<->slot continuation lines of the same request
ok = len(reqs) == 6                              # was >= 6

I have not re-run the changed test. Applied to the saved engine logs of these runs, the changed check passes all six stock runs and six runs of our fork's production build (MOVE on and off, three cache setups). Those six fail the current count with MOVE on and off alike: in our build A started solo in all six, so the same kind of continuation lines appear. I can send this as a small PR if you want it. Whether MOVE then deserves more than "experimental" is your call. This test is all I checked it with, and in every run it reports conversation cache restores: 0: conversations were parked but never restored (the reuse came from the slots giving back), so it never reaches the parked-conversation case your BATCHING.md row warns about.

The merged switches on CUDA. I used the same build and arguments, with a 128 GiB / 448-slot conversation cache (--conversation-cache-mib 131072 --conversation-cache-slots 448) and STRATA_SPLIT_MTP_BATCH=1 STRATA_KV_PREFETCH=1 STRATA_NAN_GUARD=0 in all arms. There was one round per arm, run in the order shown, not your 3 interleaved rounds. The switches arm has every merged switch except MOVE: SLOT_COPY_DIRECT, SLOT_COPY_BATCHSYNC, CKPT_POOL, CKPT_THREADS, ADMIT_MERGE_FIRST, PIPE_BYIELD_NODRAIN, BATCH_SELF_COMMIT, QSA_SLOTS, HANDOFF_PIPE=8, MMQ_GROUP_SORT, and the server's ADMIT_SHORT_FIRST=8192. The load comes from our own scripts. bench_concurrency.py sends a ~147-token (143-152 tokens) or ~1.2K-token (1210-1217) prompt with 512 tokens out and 2 requests per user. bench_multiturn.py runs 16 users x 3 turns over an 8K document. Output tok/s, with TTFT p50 in brackets:

	8 slots	16 slots	16 slots + switches
1 user, short	112.5 (0.34 s)	112.6 (0.34 s)	115.2 (0.26 s)
8 users, short	317.7 (1.84 s)	317.8 (1.61 s)	371.1 (1.17 s)
16 users, short	315.8 (13.02 s)	377.8 (1.87 s)	467.8 (1.01 s)
16 users, 1.2K prompts	240.1 (17.95 s)	276.4 (3.92 s)	335.8 (3.48 s)
16 users x 3 turns, follow-up TTFT	95.9 (13.88 s)	99.3 (3.03 s)	121.8 (2.30 s)

Against your R9700 table (0.1.44.1, IQ3_S, HIP; here 0.1.45, UD-IQ4_XS, CUDA), 16 slots + switches over 8 slots:

Measure	4x RTX 3090	R9700
16 users, short	1.48x	1.56x
16 users, 1.2K prompts	1.40x	1.41x
8 users	1.17x	1.20x
16 x 3 turns, output tok/s	1.27x	1.12x
16 x 3 turns, follow-up TTFT	13.88 to 2.30 s	21.2 to 5.7 s
Share of the 16-user gain from the slot count alone	41 %	38 % (BATCHING.md table; the release notes say about 40 %)

For the single-turn rows the ratios are close. The multi-turn gain is larger here.

Reply identity:

• Short prompts: all 50 replies are identical between 8 and 16 slots, and between 16 slots and 16 slots + switches.
• 1.2K prompts: 31/32 identical between 8 and 16 slots (one 512-token reply differs) and 32/32 with the switches.
• Multi-turn (48 replies per arm): turn 1 is 16/16 identical in both pairs. Turn 2 is 12/16 in both pairs, but in other users (10, 11, 13, 15 for 8 vs 16 slots; 8, 13, 14, 15 for 16 slots vs + switches). Turn 3 is 9/16 and 7/16, 37/48 and 35/48 in total. The 8 vs 16 slot pair has no switch set, so the multi-turn path is not exact here even without the switches. I did not chase which differences, if any, the switches cause.

In this release build STRATA_MMQ_DIRECT=1 does nothing. The engine says STRATA_MMQ_DIRECT=1 is ignored: this build has no pointer-table MMQ (CUDA with STRATA_MMQ_TAB only), as the release notes say. STRATA_MMQ_GROUP_SORT=1 did apply without the tab (prompt MMQ sorts the full expert groups (STRATA_MMQ_GROUP_SORT=1): 48 of 48 layers), so the BATCHING.md row that says both need -DSTRATA_MMQ_TAB=ON is only true of MMQ_DIRECT.

Not tested: a -DSTRATA_MMQ_TAB=ON build of 0.1.45, more than one round per arm, sampled output, a soak, other models, HIP/SYCL/Metal.

The PR itself only carries the 0.1.44 against 0.1.43 + series comparison, which is out of date now that the code is in 0.1.45. Should I close it?

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.