[jschmied]

What this changes

Qwen 3.5/3.6 on CUDA now pages by default. With TF_QWEN_KV_PAGES unset, every stream takes 256-position pages from the core pool, including the engine's own (TF_QWEN_KV_PAGES_ALL unset). Setting either to 0 keeps today's windows.

So --parallel becomes a cap rather than a reservation. A request holds its prompt and reply in pages, and waits when the pool is full. Two kernel changes keep the paged path at the window path's speed:

• the prompt attention gets a kernel with the page size fixed at 256;

• the decode tiles and the prompt rows look a page up once, rather than once per key or 16-byte chunk.

• Issue: Refs Proposal: one KV admission ledger for unpaged streams: a request reserves its prompt and reply, not a whole window per --parallel stream (#781 follow-up) #1943 (the "Qwen 3.5/3.6 paged by default" step).

• Stacked on: main. Four commits on 071cefe; the branch merges cleanly with ef9fa70.

  • cuda prefill_attention: a paged item's page looked up once, not every 16-byte chunk
  • cuda prefill_attention: hpattn_p256_kernel for pages of 256 positions, with the page size a constant
  • qwen35 attention tiles: a decode fetch looks its page up once per page, not once per key
  • qwen CUDA: streams on pages by default, the engine's own too

• atc overlap --json:

  • It shares families/qwen3_5/cuda_native.zig with fix(cuda): allocate Qwen3.5 scratch lazily #1879 (lazy scratch) and qwen3_5 cuda: a chat's next turn resumes at the previous turn's history (message starts for the planned cache) #1335 (resume at history), and git reports conflicts with both. This PR's lines there are the paging(o.slide) call, two comments and two error texts, so whichever lands second has a small rebase.
  • core/contract/qwen_whole_records.zig overlaps cuda: use MT2 FP8 attention on RTX PRO 4500 #1731 cleanly.

Behaviour:

• Unset: 256-position pages for every stream. 256 divides the 4096-row prompt chunk, as TF_QWEN_KV_SHARE needs.

• Beside --slide: an unset TF_QWEN_KV_PAGES keeps windows, since the learner reads whole windows. Set explicitly beside --slide, it's still refused.

• The profile registry (feat: add stable profile registry and scoped startup reporting #1944) keeps supplying no value, and its startup line follows the new default:

  • a count entry names what its reader takes when unset (unset = 256 for TF_QWEN_KV_PAGES);
  • TF_QWEN_KV_PAGES_ALL takes a new off syntax (on unless exactly 0).

  So TF_QWEN_KV_PAGES=0 and TF_QWEN_KV_PAGES_ALL=0 are reported as overrides, while 256 and 1 are not. docs/profiles.md says so.

Checks on this head

• Head commit: c4eb274.
• Lean checker: 424 on the base, 424 on this head, 0 new.
• Tests:
  • zig build test passes.
  • The CI set passes: test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon.
  • git diff --check is clean.
  • New host tests: the default page size, the window fallback beside --slide and at 0, the registry's unset and off reporting, and the startup line.
• CI: green locally as above.
• Not run: HIP and Metal, which this doesn't reach.

Receipt

• Environment:
  • one GB10 (sm_121), Zig 0.17.0, nvcc 13.0;
  • Qwen3.8-27B MLX 4-bit with its DFlash2 drafter;
  • Qwen3.6-35B-A3B MLX 4-bit with its MTP head;
  • --parallel 4 --context 40960 unless stated.
• Paths reached: CUDA qwen3_5 and qwen3_5_moe. bf16 and fp8 KV cache, drafted and --no-drafts. HIP and Metal are untouched.
• Exactness: I used my own gate harness (a server per arm). It records reply texts plus the first-row logits hash of every prompt pass, through TF_GATE_LOGITS, a local hook that isn't in this diff.
  • Against main (071cefe, windows): the 27B and the 35B give equal texts and logits hashes for 2-turn prompts, a needle at 5k, a chat, and concurrent 1, 2, 4 and 8 against solo. That's at --parallel 8 --context 32768.
  • On the previous base (72f4deb): the same gates are equal for the 27B with fp8 KV and with --no-drafts. Drafted replies equal --no-drafts replies.
  • Overcommitted, 27B at --parallel 16 --context 131072 with the prompt cache off:
    • 16 prompts of about 109k tokens need 1.75M rows against a 1.30M-row pool, so some wait (the last finish 1,306–1,779 s in, the first after 730 s);
    • every reply equals its solo run;
    • short prompts equal solo at c1, c2, c4 and c8;
    • each short prompt's text at max_tokens 16 is the start of its text at 48;
    • main's window path refuses this layout: "--parallel 16 needs 143.38 GiB for its streams at a 131072-token window, and the CUDA memory budget leaves 87.99 GiB: serve --parallel 9".
  • Speed arms: texts and logits are the same across arms and rounds.
• Speed: my own speed harness, fresh servers alternating ABBA. It takes the cold time to first token at 2k, 8k and 32k, a warm turn, decode tok/s at 2k and 16k, and the total tok/s of 4 streams. Main is 071cefe.

Qwen3.6-35B-A3B, 4 rounds	main	this PR	Δ
2k prompt (s)	0.374	0.376	+0.5 %
8k prompt (s)	1.397	1.401	+0.3 %
32k prompt (s)	6.306	6.341	+0.6 %
warm turn (s)	0.690	0.694	+0.7 %
decode at 2k (tok/s)	111.5	111.5	0.0 %
decode at 16k (tok/s)	101.4	100.6	−0.7 %
4 streams (tok/s)	199.3	197.9	−0.7 %

Qwen3.8-27B, 2 rounds	main	this PR
2k prompt (s)	1.602, 1.574	1.571, 1.568
32k prompt (s)	25.92, 25.22	25.23, 25.21
decode at 2k (tok/s)	38.92, 39.11	39.14, 39.20
decode at 16k (tok/s)	36.51, 36.54	36.39, 36.07
4 streams (tok/s)	72.26, 72.38	72.21, 73.00

Main's first 27B round was a cold outlier; its second matches this PR within 0.1 % on prompts.

The remaining cost, from nsys on one 35B 32k prompt (window path against paged, twice):

• the window path's prompt attention varies 5 % between runs (1,122–1,178 ms), while the paged kernel holds at 1,155–1,160 ms. On the means, paged attention costs about 7 ms per 32k prompt;
• the gaps between kernels grow by 5–8 ms, from staging the page tables between chunks;
• in decode, the paged attention partial costs about 6 % more per call (0.0565 against 0.0532 ms). That's about 0.2 % of decode.

So the 35B pays about 0.5 % on prompts and on 4 streams, and the 27B is level. In exchange, a request reserves its prompt and reply instead of a whole window, as the 16-stream run shows.

Kernel contract

• New kernel: hpattn_p256_kernel replaces hpattn_kernel for bf16 pages of 256 positions. It's checked against contiguous rows twice. Its control, a swapped page table, fails as planted.
• Changed kernels: hpattn_kernel and hpattn8_kernel (one page lookup per item), and the decode tiles Bf16Tiles::fetch and Fp8Tiles::fetch (one lookup per page). The addresses change, not the bytes.
• Records: registered usage for the new kernel and the re-measured registers of hpattn_kernel (251 and 233).
• Result: tf-cuda-test qwen-pages on the GB10 passes 26 of 26, with every control RED as planted for bf16 and fp8.
• Chips I could not run: sm_80, sm_86, sm_89, sm_120.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, and concurrent equals solo. Resumed turns (the 2-turn prompts and the warm turn) equal main's resumed turns; I didn't compare a resumed turn with a fresh one separately.
• No precision traded for speed. The bits are unchanged: only addresses move.
• Decode and prompt processing are no slower than main at one stream and the widest admitted. Not met for the 35B: prompts are about 0.5 % slower and 4 streams about 0.7 %, as above. Decode at one stream is level. The 27B is level.

[ashhart]

Thanks @jschmied. An independent review on current main (d8b80bd):
Verdict: CHANGES (one): the new default is slower on the 35B, and a regression on a default path is a no. Then
LAND after our GB10 A/B.

• The kernels read the same bytes and do the same sums.
• The author's receipts show equal texts and logits at 1 to 8 streams.
• The window path stays one setting away.
• But by the author's own numbers the 35B is 0.3 to 0.7% slower on prompts, decode at 16k and 4 streams. The PR's
  own checklist marks "no slower than main" as not met.
• Change: don't make pages the default where they're slower. Either keep the 35B (or any MoE checkpoint where the
  A/B shows a loss) on the window path by default with pages opt-in, or close the gap, and post a cell-by-cell A/B in
  which no default cell is slower than main. The admission gain is real, but it's an opt-in trade, not a default.

Applying it:

• All four commits apply cleanly on main d8b80bd. The lean check stays at 424.
• Checked against the PR's own table and the project's speed checklist.

Nothing was built or run here.

Is paged output the same as windowed output?

The attention reads are exact by construction. Only the address each key and value row is read from changes. The
rows land in the same shared-memory slots, and the softmax and sums are the same code.

• Prompt, hpattn_kernel / hpattn8_kernel (prefill_attention_rows.cuh):
  • An item looks up its first page once (lo, first).
  • A row with lo <= row < lo + PTOK has row / PTOK == lo / PTOK. So first == page_of(row) and
    row - lo == row % PTOK: the same address row_of/page_of gave.
  • Rows outside that page still take row_of/page_of.
• Prompt, new hpattn_p256_kernel:
  • It's the same included body with PTOK = 256 fixed and skip = 0.
  • static_assert(PTOK % HALF == 0) keeps each item's keys in one page, since items start at multiples of HALF.
  • Its base address is PT[k0 / 256] * PS + (hk * 256 + k0 % 256) * D, plus (key − k0) * D. That's the same
    address as above.
  • Masked keys (past limit) read offset 0 and land zeros, as before.
• Decode, Bf16Tiles::fetch and Fp8Tiles::fetch (qwen35_attn_tiles.cuh):
  • Main's kp(pos) is arena + page_base(pos) + lblk + (hoff + pos % ptok) * HD.
  • The new code caches arena + ids[lo / ptok] * stride + lblk + hoff * HD per page and adds (pos − lo) * HD. It's
    re-looked-up whenever a row leaves the cached page, so the address is the same.

The rest of the paged path isn't new. The page pool, the per-stream page runs, the engine stream on pages
(TF_QWEN_KV_PAGES_ALL), DFlash2 and MTP rings in pool lanes, prefix sharing and TF_QWEN_KV_SHARE all exist on
main as opt-ins. This PR flips their default; it doesn't add that logic. So the exactness of the paged path as a whole
rests on what those opt-ins already proved, plus this PR's receipts.

The author's receipts (one GB10; the 27B with DFlash2 and the 35B-A3B with MTP; their own harness and a local
logits hook that isn't in the diff):

• equal texts and first-row logits hashes against main's windows for 2-turn prompts, a 5k needle, a chat, and 1, 2, 4
  and 8 concurrent against solo;
• bf16 and fp8 KV, and drafted equal to --no-drafts;
• an overcommitted 16-stream, 131k-context run where every reply equals its solo run, at a layout main's window path
  refuses.

Not shown: a resumed turn against a fresh one (the author says so), and any chip other than sm_121.

Memory admission

• Each request claims pages for its prompt, max_tokens and one more forward (window_rows) when it's seated
  (cuda_pages.zig:126-149). It claims a run from the top of the pool and KV pages from the bottom.
• A full or fragmented pool returns LaneMemory, and the request waits. A request that could never fit is turned away
  by fits() before it's queued.
• So --parallel becomes a cap, not a reservation. Admission can't over-commit, because pages are claimed up front,
  not grown.
• With the engine's own stream on pages, no window is set aside for it. The plan reserves its pages instead.
• Behaviour change for operators: a request that main's window path would have seated at once can now wait for
  pages. Under heavy load, the time to first token is now set by the pool, not by --parallel. A CHANGELOG line
  should say so.

Defaults and the window path

• TF_QWEN_KV_PAGES unset means 256-position pages (default_page_tokens). =0 keeps a window per stream. Any other
  number is the page size, as on main.
• Beside --slide, an unset value keeps windows, because the learner reads whole windows. A value set explicitly
  beside --slide is still refused (QwenPagesNoLearner, named).
• TF_QWEN_KV_PAGES_ALL is now on unless exactly 0 (pagedOnlyOf). On main only 1 turned it on, so another value
  such as 2 used to mean off and now means on. That's small, but the docs line should say "on unless 0".
• A host test pins all of these: unset, 0, 128 beside --slide, and the _ALL spellings.
• The startup and profile lines report 0 as an override, and 256 and 1 as the defaults (docs/profiles.md).

Speed

Qwen3.6-35B-A3B, 4 rounds (author's)	main	this PR	change
2k / 8k / 32k prompt (s)	0.374 / 1.397 / 6.306	0.376 / 1.401 / 6.341	+0.5 / +0.3 / +0.6% slower
warm turn (s)	0.690	0.694	+0.7% slower
decode at 2k / 16k (tok/s)	111.5 / 101.4	111.5 / 100.6	0.0 / −0.7%
4 streams (tok/s)	199.3	197.9	−0.7%

• The 27B is level within its two rounds.
• The author's profile puts the 35B's cost in page-table staging between prompt chunks (5 to 8 ms per 32k prompt) and
  a 6% dearer paged decode partial (about 0.2% of decode).
• Decision on the default: the default path must be no slower than main at one stream and at the widest admitted
  count, and the MoE misses that by 0.3 to 0.7%. So please keep windows the default for the MoE (paging stays opt-in
  there) until the staging cost is gone, and keep paging on by default for the dense models, where it holds. The
  16-stream 131k admission is a real gain, and it's still there for anyone who opts in.

Contract

• New: hpattn_p256_kernel, registered, with a planted control (a swapped page table) that goes red.
• Changed: hpattn_kernel (registers re-measured, 251 and 233), hpattn8_kernel, Bf16Tiles::fetch and
  Fp8Tiles::fetch.
• qwen_whole_records.zig is updated. The author's sm_121 qwen-pages run passes 26 of 26, with every control red
  for bf16 and fp8. sm_80, sm_86, sm_89 and sm_120 weren't run.

Scope

Only Qwen 3.5/3.6 CUDA files, the shared CUDA prefill-attention kernel (its other instantiations get only P256 = false, so their code is unchanged), and the profile registry and its line. No DeepSeek, Kimi, Flash Next, HIP or
Metal file.

Small points (not blocking)

1. Say in the docs that TF_QWEN_KV_PAGES_ALL is on for any value except 0, or accept only 0 and 1.
2. Add a CHANGELOG line that --parallel is now a cap and requests can wait for pages.
3. The rebase overlaps with fix(cuda): allocate Qwen3.5 scratch lazily #1879 and qwen3_5 cuda: a chat's next turn resumes at the previous turn's history (message starts for the planned cache) #1335 in cuda_native.zig (the author's note): small, but whichever lands second
   rebases.

Device cells

• one GB10 | Qwen3.6-35B-A3B MLX 4-bit with MTP | main against this head with defaults (paged) and with
  TF_QWEN_KV_PAGES=0, fresh servers, greedy and seeded, 1, 2, 4 and 8 streams, drafts on and off: equal
  token_sha256 everywhere, and prompt, decode and 4-stream speed against main
• one GB10 | Qwen3.8-27B MLX 4-bit with DFlash2 | the same, plus fp8 KV, plus a resumed turn against a fresh one
• one GB10 | Qwen3.8-27B | --parallel 16 --context 131072, prompt cache off: every reply equals solo, and requests wait
  instead of being refused
• one GB10 | no model | CUDA kernel contract (qwen-pages and the prefill suite): every new and changed kernel passes,
  and the swapped-page-table control is red

[jschmied]

Thanks @ashhart. Before closing the gap, I ran an A/A on main 68906a4 to separate the paging cost from run-to-run noise. Main's binary ran twice (mainA, mainB). This PR, rebased on 68906a4 plus three fixes (below), also ran twice: with pages (head), and with TF_QWEN_KV_PAGES=0 (headwin, the same binary on windows). Each arm got a fresh server, in ABBA order over 4 rounds. Texts and logits are equal in every arm, round and cell.

35B (MTP). Mean of 4 paired rounds, [min, max]; + means slower:

cell	A/A (mainB vs mainA)	head vs headwin (paging only)	head vs main (both arms)
prompt 2k	+0.27 [−0.27, +0.54]	−0.13 [−1.06, +0.53]	+0.08
prompt 8k	+0.32 [−0.14, +0.72]	+0.13 [−0.50, +0.57]	+0.27
prompt 32k	+0.16 [−0.13, +0.30]	+0.16 [−0.28, +0.73]	+0.31
warm turn	+0.22 [+0.00, +0.44]	+0.65 [+0.29, +1.17]	+0.65
decode 2k	+0.22 [−2.02, +1.27]	+0.30 [−1.18, +1.46]	+0.56
decode 16k	+0.35 [−0.56, +1.19]	+0.67 [−1.02, +1.77]	+1.00
4 streams	−0.34 [−0.86, −0.07]	+0.20 [−0.06, +0.73]	−0.22

What the A/A shows:

• Noise: two runs of the same binary differ by up to about 0.35 % on a cell's mean. The prompt cells are inside that.
• Warm turn (35B): a real paging cost. It's slower in all 4 rounds, by more than the noise.
• Decode at 16k (35B): probably real, but noisy. It swings by ±1 to ±2 % between rounds in every arm.
• 27B (DFlash2): paging costs nothing. Head vs headwin is between −0.62 % and +0.23 % on every cell. Head vs main has small positive cells (warm +0.35 %, decode 16k +0.22 %, 4 streams +0.69 %), but the 27B's own A/A reaches +0.58 % at 4 streams.

Where the 35B's warm-turn cost goes. I compared CPU-sampled nsys traces of one warm turn, paged against windowed: 701.6 against 699.0 ms, so +2.6 ms.

• Zeroing, +1.2 ms.
  • Every time a paged lane opens, mapRun zeroes the lane's log (127.7 MB), taps (24 MB) and drafter ring (80 MB): cuda_cache.zig:190.
  • The window path zeroes these once, at Cache.init (cuda_cache.zig:223-225, cuda_draft.zig:242).
  • The per-open zeroing came in with my 1515357. It only costs on the default path now because this PR makes pages the default.
• Save and restore copies, +0.7 ms. 16 2D copies of 5.24 MB run at 38 GB/s, where the window path makes 40 flat copies of 2.10 MB at 56 GB/s.
• Attention through pages, +0.5 ms.
• Rest, about 0.2 ms.

Choices for the zeroing. I'd like your pick before building one. My proposal is 1:

1. Zero only what's read before it's written (my proposal). For each of the three buffers, establish whether a new stream reads any byte before writing it, and drop the fill for those it never does. This is the most exact choice: it removes the fill on every lane open, not only on warm ones, holds no extra memory, and leaves the pool as it is.
   • Proof: a test mode fills the three buffers with garbage, not zeros, on every open. Logits and texts must then equal the zeroed path on the 35B and the 27B, plain and drafted (MTP, DFlash2), across fresh, warm and resumed streams.
   • Control: a planted read of one unwritten row must break that test.
   • Where a buffer is read first: it keeps its fill, or gets a fill of just the rows that are read.
2. The pool remembers each page's last use. core/kv_pool.zig records which lane and offset last wrote each page of a run. When a lane gets the same run back, as on a warm turn, mapRun skips the fill. Any other run is zeroed as now. It's small, but it only helps when the same run comes back.
3. A fixed run per lane. The log, taps and ring move out of the pool into one buffer per lane that is zeroed once, as on the window path. That removes the fill everywhere, but it holds about 1.4 GB on the 35B whether or not a lane is used, which takes back part of the admission gain.
4. Your first option: keep the 35B (MoE) on the window path by default, with pages opt-in, and land the 27B default now. The 27B has no paging cost.

Expected with 1: the 35B warm turn goes from about +0.65 % to about +0.3 %.

Even with 1, the copies and paged attention leave the 35B warm turn about +0.3 % slower, and decode at 16k needs more rounds to settle. So the cell-by-cell "no default cell slower" bar is likely met only with 4, or with 1 plus the copy work. #2043 tracks that copy work, where a kernel writing to mapped host memory measured 6.4 against 11.3 ms at 32k.

The three fixes the head runs with are local for now, rebased on 68906a4:

• 22f50d006: a decode thread keeps its last page across tiles.
• 3d7c7085d: the paged prompt kernel stages a chunk's page ids in shared memory.
• 183bceff5: save and restore make one 2D copy per page.

I'll push them with whichever choice you pick.

[jschmied]

A correction to my last comment. I measured the zeroing before building anything: a head binary with mapRun's fill removed (cuda_cache.zig:190), against the head and main twice, on the 35B over 4 rounds in ABBA order. Texts and logits are equal in every arm and round.

Mean of 4 paired rounds, [min, max]; + means slower:

cell	no fill vs head	A/A, this run (mainB vs mainA)
prompt 2k	+0.13 [−0.54, +0.53]	+0.34 [−0.27, +0.80]
prompt 8k	+0.09 [−0.29, +0.50]	+0.31 [+0.14, +0.72]
prompt 32k	+0.22 [−0.11, +0.67]	+0.08 [−0.21, +0.37]
warm turn	+0.07 [−0.43, +0.44]	−0.03 [−1.15, +0.88]
decode 2k	+0.02 [−0.97, +1.77]	−0.74 [−2.60, +1.91]
decode 16k	−0.68 [−1.95, +0.13]	+2.00 [+0.33, +4.12]
4 streams	−0.23 [−1.21, +0.41]	+0.84 [+0.05, +1.34]

What this changes:

• Removing the fill gains nothing measurable. The 1.2 ms of memsets in the trace overlap host work, so they're not on the warm turn's critical path.
• My estimate was wrong. I had said the warm turn would go from about +0.65 % to about +0.3 % with the fill gone. It doesn't, so the four zeroing choices don't close the gap, and you don't need to pick one.
• The A/A spread is larger than I assumed. Two runs of main's own binary differ by +2.00 % at decode 16k and +0.84 % at 4 streams. In this session, the head against main (+0.37 % warm, +1.45 % decode 16k) is inside that spread.

Next: before claiming any cell is a paging cost, I'm finding out why two runs of the same binary differ by up to 2 %. I'll post the cell-by-cell table again once the noise floor is explained and tight enough to read a 0.3 % gap.

Separately, I had a static audit run on the three buffers: log, taps and drafter ring. Every read of them is bounded by a counter that the lane resets at open, and covers only bytes the stream has written. The window path already relies on this, since it zeroes them only once at init. I'll keep it for later and won't change it in this PR.

[ashhart]

Thanks @jschmied. Running an A/A was the right step, and so was dropping the zeroing once it measured nothing. Our call is option 4. The 35B's warm turn was slower in all four rounds of your first run, and a slower default is a no. So please:

1. keep the MoE on windows by default with pages opt-in, and make pages the default for the dense 27B, where paging measured no cost;
2. push the three local fixes (22f50d006, 3d7c7085d, 183bceff5);
3. add the CHANGELOG line saying --parallel is now a cap and a request can wait for pages;
4. post the cell-by-cell A/B on the pushed head with an A/A beside it.

Your noise work and #2043 can then make the 35B default its own follow-up.

[jschmied]

Thanks @ashhart. Items 1 to 3 are in the pushed head 4e43c6872:

• the MoE stays on windows by default, with pages opt-in, and the dense 27B pages by default;
• the three fixes;
• the CHANGELOG line.

Here is item 4, the cell-by-cell A/B on that head with an A/A beside it.

Setup:

• Arms: main 68906a4dd twice (mainA and mainB), and the head. Each arm got a fresh server, in ABBA order: 6 rounds, with 3 repeats of every cell per server.
• Box state: the GPU clock was locked at 2300 MHz (nvidia-smi -lgc 0,2300), and the page cache was dropped before every server start.
• Why locked: at stock clocks, the 32k TTFT tracked GPU temperature (r = +0.85) and the clock fell about 2 % as the GPU heated. That was most of the earlier A/A spread on prompts.
• Equality: texts and first-row logits are equal in every arm, round and repeat, on both models.

Mean of 6 paired rounds, [min, max], ±2 SE; + means slower:

Qwen3.6-35B-A3B, MTP (windows by default)

cell	A/A (mainB vs mainA)	head vs main	rounds slower
prompt 2k	−0.23 [−0.79, +0.51] ±0.40	−0.11 [−0.48, +0.49] ±0.31	2/6
prompt 8k	+0.16 [−0.54, +0.76] ±0.48	−0.00 [−0.43, +0.51] ±0.26	3/6
prompt 32k	+0.10 [−0.51, +0.89] ±0.45	+0.05 [−0.31, +0.44] ±0.25	3/6
warm turn	+0.28 [−0.23, +0.90] ±0.34	+0.03 [−0.31, +0.53] ±0.27	3/6
decode 2k	−0.77 [−2.74, +0.89] ±1.06	−0.33 [−1.66, +1.61] ±0.93	2/6
decode 16k	−0.46 [−2.75, +2.16] ±1.32	+0.46 [−1.42, +2.47] ±1.26	3/6
4 streams	−0.04 [−0.96, +1.07] ±0.60	−0.30 [−1.00, +0.25] ±0.39	2/6

Qwen3.8-27B, DFlash2 (pages by default)

cell	A/A (mainB vs mainA)	head vs main	rounds slower
prompt 2k	−0.00 [−0.20, +0.10] ±0.10	+0.18 [−0.03, +0.47] ±0.18	5/6
prompt 8k	+0.01 [−0.15, +0.20] ±0.09	+0.30 [+0.08, +0.70] ±0.19	6/6
prompt 32k	+0.01 [−0.14, +0.16] ±0.09	+0.29 [+0.14, +0.44] ±0.08	6/6
warm turn	−0.06 [−0.31, +0.08] ±0.12	+0.28 [+0.05, +0.50] ±0.12	6/6
decode 2k	+0.08 [−0.71, +1.15] ±0.52	−0.20 [−0.75, +0.07] ±0.24	2/6
decode 16k	+0.06 [−0.09, +0.46] ±0.18	+0.17 [+0.01, +0.28] ±0.08	6/6
4 streams	+0.13 [−0.97, +0.88] ±0.58	+0.22 [−0.51, +1.17] ±0.62	3/6

What it shows:

• 35B: the head is level with main on every cell. Its default path is main's windows.
• 27B: my earlier "no cost" was wrong. With the clock locked, its A/A is tight enough (±0.1 % on prompts) to show a real paging cost:
  • +0.3 % on 8k and 32k prompts and on the warm turn, and +0.17 % at decode 16k;
  • every one slower in all 6 rounds and beyond its A/A.
  • My earlier runs, at stock clocks, couldn't resolve it.
• Against your bar, the 27B default fails too. No default cell may be slower than main.

Proposal: pages opt-in for both models for now. This PR would then:

• make the paged path itself faster;
• add the MoE fallback;
• make --parallel a cap when TF_QWEN_KV_PAGES is set.

Making pages the default would move to #2043, once its copy and attention work closes the 0.3 % (its integrated-GPU save and restore measured 6.4 against 11.3 ms at 32k). Another option is to keep the 27B default and accept 0.3 % for the admission gain, but that's against the rule as you stated it. I can push the opt-in change today, with the CHANGELOG line reworded. Which do you prefer?

One caveat: some of my host builds overlapped the 35B run's middle rounds. Its numbers are level all the same.

[ashhart]

Thanks @jschmied. Locking the clock and dropping the page cache turned the A/A from ±2% into ±0.1% on prompts, and that's what shows the 27B's cost: about 0.3% on 8k and 32k prompts and the warm turn, slower in all six rounds. So yes, pages opt-in for both models for now, as you propose: keep the faster paged path and the MoE fallback here, make --parallel a cap only when TF_QWEN_KV_PAGES is set, and reword the CHANGELOG line. The default can move in its own PR once the copy and attention work closes the gap. One small point: TF_QWEN_KV_PAGES_ALL unset now means on wherever pages are on, while on main it meant off. With pages opt-in, please keep main's meaning for unset, so someone who sets only TF_QWEN_KV_PAGES today gets what they got before. Once that's pushed, we'll run the defaults and the opt-in on our GB10 against main.

[jschmied]

Thanks @ashhart. Pushed as d64e6d04c, three new commits on 4e43c6872:

1. Pages are opt-in for both models. TF_QWEN_KV_PAGES unset or 0 keeps a window each, on the 27B and the 35B alike. The faster paged path and the MoE-aware reader stay, so a per-model default later is one constant (unset_pages).
2. --parallel is a cap only with TF_QWEN_KV_PAGES set. Unset, there's no pool, and the plan reserves a window a stream, as on main.
3. The CHANGELOG line now says pages are opt-in, and that --parallel is a cap with them set.
4. TF_QWEN_KV_PAGES_ALL is on only at exactly 1 again, so unset is off, as on main. The profile registry, its startup line and docs/profiles.md are main's again.

Host tests (test-qwen35-pages-default-host, part of test-cuda-host) cover TF_QWEN_KV_PAGES unset, 0, 1 and a size on a dense and a MoE profile, _ALL unset, and --parallel with pages and without. A planted wrong admit count fails the cap test.

Startup lines on the GB10 (--parallel 4 --context 40960, a drafter each):

35B, default:

CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host): model 19.65 GiB; 4 streams at once, 1.13 GiB each at a 40960-token window, of 86.0 GiB left after a 12.2 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels

35B, TF_QWEN_KV_PAGES=1:

CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host): model 19.65 GiB; 4 streams at once, the engine's own at a 40960-token window (1.13 GiB, in the model's), 3 more taking 189630 pages of 1 rows (3.62 GiB) as they need them, 0 MiB fixed each, of 86.0 GiB left after a 12.2 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels

27B, default:

CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host): model 16.51 GiB; 4 streams at once, 3.46 GiB each at a 40960-token window, of 89.1 GiB left after a 12.2 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels

27B, TF_QWEN_KV_PAGES=1:

CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host): model 16.51 GiB; 4 streams at once, the engine's own at a 40960-token window (3.46 GiB, in the model's), 3 more taking 178296 pages of 1 rows (10.88 GiB) as they need them, 23 MiB fixed each, of 89.1 GiB left after a 12.2 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels

The defaults match main. Both default lines are the same as main 68906a4dd's, and so is registered overrides: none. The rest of the startup log differs only in what changes from run to run: free memory, timings, and the window-cost cache the new binary measured once.

1 means pages of 1 position. The size to use is TF_QWEN_KV_PAGES=256: 741 pages (3.62 GiB) on the 35B, 699 (10.92 GiB) on the 27B.

[ashhart]

Thanks @jschmied. Pages opt-in for both models, --parallel a cap only with them set, and TF_QWEN_KV_PAGES_ALL unset meaning off again is exactly what we asked, and the host tests pin every value. What's left is ours: on a GB10 we'll run the defaults against main for tokens and speed, TF_QWEN_KV_PAGES=256 for tokens with a resumed turn, the 16-stream overcommit run and the pages contract, and land it if they hold.

[ashhart]

Thanks @jschmied. The first GB10 runs are in, both against main with Qwen3.8-27B MLX 4-bit, balanced order:

Arm against main	Prompts, 32,768 tokens	Decode after them	Streams, 1 and 4
Defaults	+0.1%	-0.3% to +0.1%	+0.0% to +0.3%
TF_QWEN_KV_PAGES=256	-0.1% to +0.3%	-0.1% to +0.1%	+0.0% to +0.2%

0 of 20 prompt rows differ in either arm, and both read GREEN. The 16-stream overcommit run and the pages contract are next, and it lands if they hold. It'll need a rebase by then; we'll do that at prep.

[ashhart]

The 16-stream overcommit run on a GB10, main against this PR with the same model, balanced order: GREEN. Aggregate at 16 streams:

Prompt	Temperature	main	this PR	Change
chat	0	150.2	150.9	+0.4%
chat	1	138.4	139.3	+0.7%
code	0	150.9	151.7	+0.5%
code	1	143.2	144.1	+0.6%

The pages contract is the last run, and it's on the box now. It lands if that holds.

[ashhart]

The pages contract on a GB10, this PR against its base: 553 of 553 kernels pass gates 1-4, 359 of 359 planted controls red, and the same 190 limit problems as main, with no limit line new. That was the last run. It goes in the queue, and we'll do the rebase at prep. Thanks @jschmied.