[gopinath87607]

What this changes

STRATA_MTP_ROOM_MIN=TOKENS (--serve with --mtp): a request that reads at least TOKENS prompt tokens (0:
every request) uses the MTP drafter's 512 routed experts as the prompt path's ring while it is read. TOKENS counts the
tokens this request reads, not the whole prompt: a turn that reuses a cached conversation prefix and reads a few
hundred tokens gains nothing from the room. Those experts are 708 MB (675 MiB) that only the decode reads —
record_rest is their only reader; a prompt's passes run the draft layer's dense tensors and its K/V — so the prompt
path's ring is pointed at the block itself: slot i = experts_base + (i - rule) * MAXBLOB (135 slots on a Q8_0
pack, whose largest blob is 4.98 MiB). Opening and closing a room moves the ring count alone
(Prefill::set_ring_room_active): the block is never freed, never allocated again and never moves, so no prompt path
is laid out again, the decode's captured round/step graphs stay valid (nothing is dropped), and what comes back before
the first window is the decode's own bytes (MtpDrafter::experts_reclaim). In the measured shape's successor this is
two 675 MiB DMAs around the read.

The room is opened after the reused prefix is known (read_from) and immediately before lend(), so a relayout still
re-carves with the room; a request that must be re-read from 0 (STRATA_CKPT_REREAD) counts all of its tokens.

Limits, refused with the reason at start: --serve only, --mtp needed, --batch not supported (its slot drafters
share the solo drafter's weights). A layer split is fine (the room goes to the prompt path on the drafter's card),
--pipeline-windows is fine (the room closes before the request's first window), and --prefill-head-alias is fine.

From the Maya checkout (project-maya), local commits 7df369c (STRATA_GLM_ROOM_MIN), d4d0792 (the tail's
slots rounded up to the largest blob) and 5ccf9a4 (the in-place shape, for the MTP config's NextN block). Counting
what a request reads is this port's change, made after the report below.

What it leaves alone

Unset (the default) mtp_room_min is -1 and not one line of the room runs; the prompt path is laid out exactly as
before. The new code is reachable only behind that check. This is by construction, not by a measurement. =0 keeps
the behaviour that was measured: every request takes the rod, on 7-token continuations).

Measurements

• Reported on an 8 GB RTX 2060 SUPER (2026-10-18, merbana branch, not run by us): the
  room works and pays where the ring is the limit. Qwen3.8-Flash-Next IQ3_XXS, --serve --mtp (Q2_0 drafter,
  675 MiB), --resident-budget-gib 40 (fully page-locked), --kv k8v4 --kv-resident 20480, --prefill auto; the
  auto chunk was 6,656 tokens with a 96-slot ring, the in all), chunk unchanged.
  STRATA_MTP_ROOM_MIN=0 against the same build unset (paired A/B, 3 pairs, prompt tok/s): 4K 448.4 -> 527.2 (+17%),
  16K 558.9 -> 658.1 (+17%), 32K 584.0 -> 719.7 (+23%); dec17 (noise). The lend cost
  107 ms per roomed request. STRATA_PREFILL_RING alone showed the ring was the limit (192 slots: +9.8% / +6.2% /
  +2.3%, decode -3.1 to -3.6 tok/s), so this is the case thn logits after a
  15,521-token prompt were bitwise equal to the unset run (two runs per arm, all four identical), the 48-token
  greedy replies matched, and the drafter's acceptance after reclaim was normal (27 of 34).
• The first shape was measured on our Q8_0 4-way rig (202ught no time. The room
  opened, the 675 MiB really went into the ring, the block came back before the first window and the decode ran
  unchanged — but a deeper ring was not what the prompt waiens. That shape laid the
  prompt path out again around the block and cost **3.4–4.0p of the read.
• The in-place shape and the read threshold were measured on our Q8_0 4-way rig (2026-10-19): the room is skipped,
  and its ~206 ms goes with it. Two engines from the sameainst this one), one start
  each, MIN=1024: one 5,042-token chat and three continuations that echoed the answer back. The parent lent the
  room for the whole 5,096-token prompt on continuations thend ~102 ms + reclaim
  ~104 ms); this commit lent nothing. Read time 2,054 -> 1,813 ms, 360 -> 153 ms, 344 -> 152 ms; time to first token
  2.110 -> 1.869 s, 0.409 -> 0.203 s, 0.390 -> 0.199 s. A still opened the room on the 7-token continuations (time to first token 0.432 / 0.415 the difference is the room, not the binary. The cold request still took the room in bver: more than one run per timing, HIP/SYCL,--prefill-head-alias`.
• 7df369c records the room as not yet measured, with its 23 tok/s) on its
  byte-identical 100,535-token read.
• No blanket speedup is claimed. The knob is how you meh the variable set and unset,
  PP line against PP line. The ring gain appears where GB card above) and is nil
  where it is not (our Q8_0 rig). The read threshold's gainend on the ring: it removes
  the room's own DMA from turns that reuse a conversation, ~0.2 s per short turn on our rig.

Checks

• Default path: unset = off by construction (the parsed threshold is -1 and every room call is behind it).
• Threshold: the room opens only when n - read_from >= TOKENS, after read_from is known and before lend(),
  so the prompt path's relayout re-carves with the ring rooes the ring count alone; the
  room's slot pointers are laid out by carve whether or nning it after the head-alias
  rebuild still adds the slots.
• Build: both the parent commit and this commit built fild strata`, CUDA 13.2,
  sm_86 + sm_120) and ran on the Q8_0 4-way rig; the two binaries differ by this commit.
• HIP / SYCL: not compiled here (no toolchain on the machine). generate.cpp carries no per-backend code, so both
  builds compile the same change.
• On a start: the engine prints the read-token threshold, MiB and slots. Per roomed request it prints the tokens
  this request reads (so a reused prefix is visible againstrafter's block ever moved, the
  first shape said so — the in-place shape cannot.

🤖 Generated with Claude Code

[merbanan]

Measured: the room pays on an 8 GB Turing card (+17% to +23% prompt, decode unchanged, bitwise-identical logits)

We ported this PR onto our 0.1.45-based branch and measured it on a card where the ring is the limit. It is now our live default.

Setup

• RTX 2060 SUPER 8 GB (sm_75), PCIe 3.0 x8, Ryzen 9 3900X, 92 GB RAM.
• Qwen3.8-Flash-Next IQ3_XXS, --serve --mtp (Q2_0 drafter, 675 MiB of experts), --resident-budget-gib 40 (fully page-locked), --kv k8v4 --kv-resident 20480, --prefill auto.
• The auto scan's chunk is 6656 tokens with a 96-slot ring. The room adds 303 slots (399 in all), and the chunk stays 6656.

Is the ring the limit here? We ran STRATA_PREFILL_RING alone, paired A/B with 3 pairs, prompt tok/s against 96 slots:

ring	4K	16K	32K	decode
48	-28.4 (-6.3%)	-19.3 (-3.4%)	-26.7 (-4.6%)	~0
192	+44.2 (+9.8%)	+35.2 (+6.2%)	+13.7 (+2.3%)	-3.1 to -3.6

So the ring is a limit here. Taking it from the cache, though, shrinks the auto chunk (6656 -> 5632) and costs decode tok/s after short prompts. The room gets the ring without either cost.

STRATA_MTP_ROOM_MIN=0 vs the same build unset (paired A/B, 3 pairs, prompt tok/s / decode tok/s):

prompt	unset	room	paired d (prompt)	decode paired d
4K	448.4	527.2	+75.8 (+17%)	-0.27 (noise)
16K	558.9	658.1	+94.4 (+17%)	+1.42 (noise)
32K	584.0	719.7	+135.7 (+23%)	-0.17 (noise)

Room cost and correctness

• Lend cost: 107 ms per roomed request. The first roomed request after start paid 24.4 s once, for the 708 MB cudaHostAlloc with 40 GiB already page-locked. Taking that buffer at start would move the cost out of the first request.
• Correctness: the first-token logits after a 15,521-token prompt are bitwise equal to the unset run. We ran it twice per arm, and all four were identical.
• Decode: the 48-token greedy replies matched. The drafter's acceptance after the room was given back was normal (27 of 34), so experts_reclaim restores the bytes intact.

STRATA_PREFILL_KEEP=64 (#2833) on top of the room changed nothing on this card: the chunk stays 6656, and the result was +78.2 / +92.0 / +134.9.

A fix we suggest: count the tokens the request reads, not the whole prompt

As written, the room opens when n >= STRATA_MTP_ROOM_MIN, where n is the whole prompt (ids.size()). In agent use (opencode and the like), almost every request is a long conversation whose prefix is reused. From our live log:

prompt 50326 tokens = 49993 reused + 333 read
prompt 53254 tokens = 53239 reused + 15 read
prompt 53694 tokens = 53594 reused + 100 read

With any threshold below the conversation length, every one of those turns pays the ~107 ms lend/reclaim round trip, 2 x 708 MB over PCIe, to read 15-350 tokens. The ring cannot help a read that small, so on these turns the room is pure overhead.

Fix (our commit ded7b1b8 on merbanan/Strata v15): open the room after the reused prefix is known, and compare the tokens this request actually reads:

• the open block moves from before the head-alias rebuild to just after const int64_t read_from = reread_to > 0 ? 0 : resume;;
• the condition becomes n - read_from >= mtp_room_min;
• the log line prints the tokens to read.

Nothing else moves:

• The loan is still taken later (lend() runs after this point), so "before the loan: a relayout re-carves with the room" still holds.
• room_close("left open") still runs at the top for a cancelled earlier request.
• room_close("prompt done") is unchanged.
• A conversation that must be re-read from 0 (reread_to > 0) counts all of its tokens, as it should.

Effect, with STRATA_MTP_ROOM_MIN=1024:

• a short agent turn or a small tool result skips the room;
• a turn that reads a source file or more (1K+ new tokens) takes it;
• a cold read of a long conversation takes it.

With =0 the behaviour is unchanged.

diff (against our port; the code at both sites is the PR's)

diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index ea35be89..02e0e886 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -11107,19 +11107,6 @@ int main(int argc, char** argv) {
                 return 0;
             };
             if (mtp_room_open && room_close("left open") != 0) return 1;
-            if (mtp_room_slots > 0 && room_path != nullptr && (int64_t) n >= mtp_room_min && mtp.experts_resident()) {
-                const auto room_t0 = Clock::now();
-                if (!mtp.experts_lend(err)) {
-                    std::fprintf(stderr, "strata serve: prompt room: %s: this prompt reads as it always did\n", err.c_str());
-                } else {
-                    room_path->set_ring_room_active(true);   // before the loan: a relayout re-carves with the room
-                    mtp_room_open = true;
-                    std::fprintf(stderr, "strata serve: prompt room: %lld tokens, the prompt path's ring is the drafter's "
-                                         "%llu MiB (%d slots past its own, lent in %.1f ms)\n", (long long) n,
-                                 (unsigned long long) (mtp.experts_bytes() >> 20), mtp_room_slots,
-                                 std::chrono::duration<double, std::milli>(Clock::now() - room_t0).count());
-                }
-            }
             if (o.prefill_head_alias && native_head.loaded()) {
                 const auto head_t0 = Clock::now();
                 native_head.unload();
@@ -11453,6 +11440,21 @@ int main(int argc, char** argv) {
             tr("request", n, geni ? 1 : 0);
             if (use_mtp) mtp.set_prompt_len(n);
             const int64_t read_from = reread_to > 0 ? 0 : resume;
+            // STRATA_MTP_ROOM_MIN: opened here, once the reused prefix is known - the threshold counts the tokens this
+            // request reads (an agent turn on a long conversation reads a few hundred and gains nothing from the room)
+            if (mtp_room_slots > 0 && room_path != nullptr && n - read_from >= mtp_room_min && mtp.experts_resident()) {
+                const auto room_t0 = Clock::now();
+                if (!mtp.experts_lend(err)) {
+                    std::fprintf(stderr, "strata serve: prompt room: %s: this prompt reads as it always did\n", err.c_str());
+                } else {
+                    room_path->set_ring_room_active(true);   // before the loan: a relayout re-carves with the room
+                    mtp_room_open = true;
+                    std::fprintf(stderr, "strata serve: prompt room: %lld tokens to read, the prompt path's ring is the drafter's "
+                                         "%llu MiB (%d slots past its own, lent in %.1f ms)\n", (long long) (n - read_from),
+                                 (unsigned long long) (mtp.experts_bytes() >> 20), mtp_room_slots,
+                                 std::chrono::duration<double, std::milli>(Clock::now() - room_t0).count());
+                }
+            }
             conversations.limit_reuse(read_from);
             pp_total = n;
             pp_from = read_from;

[gopinath87607]

Measured: the room pays on an 8 GB Turing card (+17% to +23% prompt, decode unchanged, bitwise-identical logits)

We ported this PR onto our 0.1.45-based branch and measured it on a card where the ring is the limit. It is now our live default.

Setup

* RTX 2060 SUPER 8 GB (sm_75), PCIe 3.0 x8, Ryzen 9 3900X, 92 GB RAM.

* Qwen3.8-Flash-Next IQ3_XXS, `--serve --mtp` (Q2_0 drafter, 675 MiB of experts), `--resident-budget-gib 40` (fully page-locked), `--kv k8v4 --kv-resident 20480`, `--prefill auto`.

* The auto scan's chunk is 6656 tokens with a **96-slot ring**. The room adds **303 slots** (399 in all), and the chunk stays 6656.

Is the ring the limit here? We ran STRATA_PREFILL_RING alone, paired A/B with 3 pairs, prompt tok/s against 96 slots:
ring 4K 16K 32K decode
48 -28.4 (-6.3%) -19.3 (-3.4%) -26.7 (-4.6%) ~0
192 +44.2 (+9.8%) +35.2 (+6.2%) +13.7 (+2.3%) -3.1 to -3.6

So the ring is a limit here. Taking it from the cache, though, shrinks the auto chunk (6656 -> 5632) and costs decode tok/s after short prompts. The room gets the ring without either cost.

STRATA_MTP_ROOM_MIN=0 vs the same build unset (paired A/B, 3 pairs, prompt tok/s / decode tok/s):
prompt unset room paired d (prompt) decode paired d
4K 448.4 527.2 +75.8 (+17%) -0.27 (noise)
16K 558.9 658.1 +94.4 (+17%) +1.42 (noise)
32K 584.0 719.7 +135.7 (+23%) -0.17 (noise)

Room cost and correctness

* **Lend cost:** 107 ms per roomed request. The first roomed request after start paid 24.4 s once, for the 708 MB `cudaHostAlloc` with 40 GiB already page-locked. Taking that buffer at start would move the cost out of the first request.

* **Correctness:** the first-token logits after a 15,521-token prompt are **bitwise equal** to the unset run. We ran it twice per arm, and all four were identical.

* **Decode:** the 48-token greedy replies matched. The drafter's acceptance after the room was given back was normal (27 of 34), so `experts_reclaim` restores the bytes intact.

STRATA_PREFILL_KEEP=64 (#2833) on top of the room changed nothing on this card: the chunk stays 6656, and the result was +78.2 / +92.0 / +134.9.

A fix we suggest: count the tokens the request reads, not the whole prompt

As written, the room opens when n >= STRATA_MTP_ROOM_MIN, where n is the whole prompt (ids.size()). In agent use (opencode and the like), almost every request is a long conversation whose prefix is reused. From our live log:

prompt 50326 tokens = 49993 reused + 333 read
prompt 53254 tokens = 53239 reused + 15 read
prompt 53694 tokens = 53594 reused + 100 read

With any threshold below the conversation length, every one of those turns pays the ~107 ms lend/reclaim round trip, 2 x 708 MB over PCIe, to read 15-350 tokens. The ring cannot help a read that small, so on these turns the room is pure overhead.

Fix (our commit ded7b1b8 on merbanan/Strata v15): open the room after the reused prefix is known, and compare the tokens this request actually reads:

* the open block moves from before the head-alias rebuild to just after `const int64_t read_from = reread_to > 0 ? 0 : resume;`;

* the condition becomes `n - read_from >= mtp_room_min`;

* the log line prints the tokens to read.

Nothing else moves:

* The loan is still taken later (`lend()` runs after this point), so "before the loan: a relayout re-carves with the room" still holds.

* `room_close("left open")` still runs at the top for a cancelled earlier request.

* `room_close("prompt done")` is unchanged.

* A conversation that must be re-read from 0 (`reread_to > 0`) counts all of its tokens, as it should.

Effect, with STRATA_MTP_ROOM_MIN=1024:

* a short agent turn or a small tool result skips the room;

* a turn that reads a source file or more (1K+ new tokens) takes it;

* a cold read of a long conversation takes it.

With =0 the behaviour is unchanged.
diff (against our port; the code at both sites is the PR's)

hi thanks for testing this pr i applied your suggestion and also i will open different pr for tackling small prompt speeds up currently focusing on the deepseek implementation

[gopinath87607]

Closed because the source fork was deleted. This PR was opened from the gopinath87607/Strata fork, which was deleted on 2026-10-19. Deleting a fork deletes its branches, and GitHub then closes every pull request whose branch lived in it; a closed PR whose source branch is gone cannot be reopened. Nothing was lost: the same work was recreated from a new fork, rebased onto current main, in #3307. Please continue the discussion there.