[Bizuayeu]

On two GB10s at TP=2 with --drafter (DFlash2), TF_GLM_CHUNK_SOLO=4096 now serves instead of being refused at load.
Against main with the same drafter, a cold prompt's first token comes 6.1% sooner at 32K tokens and 7.0% sooner at 64K
(0.7 and 1.7% at 2K and 8K), with the same tokens and the same decode speed:

	main, --drafter	this head, --drafter, TF_GLM_CHUNK_SOLO=4096
Cold first request's ttft_s, 2K / 8K / 32K / 64K tokens (bench_cache.py --require-cold)	1.42 / 5.90 / 22.99 / 46.15	1.41 / 5.80 / 21.59 / 42.93
Decode tok/s at one stream, fibonacci-raw / gpu-chat-no-think, sampled (bench_openai.py)	45.08 / 37.13	45.10 / 37.09
The same, greedy	47.96 / 38.92	48.02 / 38.98
Aggregate tok/s at one / four streams of different prompts, sampled (bench_concurrent.py --prompts tools/bench_prompts.json)	58.3 / 71.7	58.2 / 71.1
The same, greedy	68.1 / 74.8	68.1 / 75.5
Startup line: model, KV pages, room left	91.43 GiB, 2,170 pages, 14.6 GiB	93.43 GiB (the 4,096-row prompt buffers and DFlash2's 80 MiB), 1,904 pages, 13.0 GiB

bench_cache.py read upstream's tools/*.py at c085aab04, tokenized by the server's /tokenize and cut into
disjoint 2K, 8K, 32K and 64K-token slices; bench_openai.py and bench_concurrent.py ran 256 tokens, three reps,
medians.

What this changes

TF_GLM_CHUNK_SOLO=4096 lets a lone prompt pass take 4,096 rows, and #1842 refused it beside --drafter, because
DFlash2's prompt tap buffer and promptRun's bound held one 2,048-row chunk. The drafter keeps only the prompt's last
2,048 rows, so its ingest buffers can stay as they are: only taps_prefill, which the forward writes for every row of a
pass, now takes the engine's prompt buffer rows (tapRows: 4,096 with solo, 2,048 without), and promptRun checks a
pass against that buffer. The refusal and its test go. Solo costs 80 MiB more on each rank; without it nothing changes.

This follows @MiaAI-Lab's patch 0095 in MiaAI-Lab/GLM-5.3-Flash-DGX-Sparks-1104,
which found the bound for its 4,096-row grid.

• Issue: Refs glm cuda: TF_GLM_CHUNK_SOLO=4096 beside --drafter is refused by name #1842 (it refused the pair; this serves it).
• Stacked on: main (c085aab04, where the receipt ran; it merges cleanly with main at 8abb30f6f), one commit.
• atc overlap --json (2026-11-05 03:09 UTC, base fb0895099; complete: false, as atc could not read every open pull
  request's files): of the eight open pull requests that share files with this one, seven merge cleanly and GLM CUDA: opt-in context copies priced by the lane host #2146
  conflicts in glm/cuda_native.zig, where it adds BadCopyPolicy's line beside the GlmSoloWithDrafter line this
  removes: whichever lands second keeps BadCopyPolicy's and drops GlmSoloWithDrafter's.

Checks on this head

• Head commit: 7c2f08fcf
• Lean checker on the base and on this head: 425 and 425 problems, 0 new (the 425 are inherited).
• Tests run (Zig 0.17.0, WSL, x86_64): zig build test-glm-prompts-host 8/8 (7/7 on the base; the new test is in this
  step), test-cuda-host 4,793 of 4,816 (23 skipped), test 2,252 of 2,288 (36 skipped; Zig's cache kept the steps
  another run had built from the same inputs),
  test-glm-prompt-settings-host 6/6, test-glm-tp3-host 8/8, test-glm-prompt-arena-host 1/1,
  test-glm-startup-host, test-glm-pages-host, test-golden, compile-cuda-tests (also aarch64),
  compile-config-server: no failures. zig fmt --check build.zig zig and git diff --check: clean.
• CI: green on this head (zig, zig-windows, zig-macos-dev-tools, fmt, benchmark-clients on Ubuntu and Windows,
  release-paths).
• Not run, and why: TP=3 (the three hosts that run it were in another session's use) and one rank (the checkpoint's
  181 GiB does not fit one GB10).

Receipt

• Environment: base c085aab04 (main) and this head, each built with Zig 0.17.0 and nvcc 13.3.73 for sm_121; two GB10s
  (128 GB unified memory each, Ubuntu 24.04.5, kernel 7.0.0-1019-nvidia, driver 580.178.04) at TP=2 over RoCE;
  Bizuayeu/GLM-5.3-Flash-XSFP4 at 88736f91; drafter incoai/GLM-5.3-Flash-DFlash2 at bf582e4e; --context 524288 --parallel 4 (DFlash2 leaves 1M's four streams no room), TF_KV_PAGES unset (pages), TF_GLM_KV_SHARE=16,
  TENSORFOLD_PREFILL_SHARE=0.7, and for the table --prompt-cache-gib 0. With TF_GLM_KV_SHARE the kept prompts stay
  on the pool's pages even so: each prompt's first request was cold (a fresh server, disjoint prompts), and
  bench_cache.py's repeats at 8K to 64K resumed from it.
• Paths and platforms: GLM's CUDA engine with --drafter (DFlash2), on rank 0 and the following ranks; without
  --drafter nothing changes, and with it but without solo the drafter's buffers are main's. Run: TP=2 on two GB10s
  (sm_121), main and this head. Not run: TP=3, other SMs.
• Exactness: bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial on this head
  with solo: every reply at one and four streams equalled its request alone (code and chat, 3 and 12 of 3 and 12), and
  the alone runs equalled "draft": false. bench_cache.py at 2K to 64K: on main and on this head, each prompt's
  repeats and concurrent requests gave its first request's tokens, resuming from it at 8K to 64K (6,144 to 63,488
  tokens cached; a 2K prompt keeps no shorter state), and each prompt's tokens were the same on main and on this head; bench_openai.py's replies were the same on both,
  sampled and greedy. With the prompt cache at its default, prompts of about 7K, 15K and 30K tokens and the decode
  check's count prompt (256 greedy tokens) gave the same token_sha256 drafted and with "draft": false, with solo and
  without; a short request sent 3 s into a 30K-token prompt gave the same tokens as alone, and the long prompt's second
  run, resumed from its kept state (28,672 tokens cached), equalled its first.
• Decode speed before and after: the table above (bench_openai.py --tokens 256 --reps 3 --temperatures 1.0,0).
• Prompt speed before and after: the table above (the first request's ttft_s from bench_cache.py --require-cold).
• Licence of any new weights or drafter: n/a (no weights change; the runs used DFlash2's published weights, CC BY-NC-ND
  4.0).

Raw results

bench_cache.py --require-cold (and its repeats and concurrent requests), main = mu, this head = ou:

{"run": "mu", "tokens": 2048, "qualified": true, "first_token_sha256": "50f9443218e13caab52d4af7fba39d1cc707a6e0ae6752e39cd4fb1d25eb8c1f", "rows": [{"phase": "first", "prompt_tokens": 2048, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 1.422, "equal_first_tokens": null}, {"phase": "repeated", "prompt_tokens": 2048, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 1.396, "equal_first_tokens": true}, {"phase": "repeated", "prompt_tokens": 2048, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 1.391, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 2048, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 2.893, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 2048, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 1.395, "equal_first_tokens": true}]}
{"run": "mu", "tokens": 8192, "qualified": true, "first_token_sha256": "fe0ef359af30fb6de6e26602f88ac9dc76612b1c69c2735efec904385fccacca", "rows": [{"phase": "first", "prompt_tokens": 8192, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 5.905, "equal_first_tokens": null}, {"phase": "repeated", "prompt_tokens": 8192, "cached_tokens": 6144, "cache_state": "reused", "ttft_s": 1.491, "equal_first_tokens": true}, {"phase": "repeated", "prompt_tokens": 8192, "cached_tokens": 6144, "cache_state": "reused", "ttft_s": 1.491, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 8192, "cached_tokens": 6144, "cache_state": "reused", "ttft_s": 3.073, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 8192, "cached_tokens": 6144, "cache_state": "reused", "ttft_s": 1.489, "equal_first_tokens": true}]}
{"run": "mu", "tokens": 32768, "qualified": true, "first_token_sha256": "47c8b22cb55a3736b6cec2795f03938d3187b02a107c401e3ebbb5490fcf3a45", "rows": [{"phase": "first", "prompt_tokens": 32768, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 22.993, "equal_first_tokens": null}, {"phase": "repeated", "prompt_tokens": 32768, "cached_tokens": 30720, "cache_state": "reused", "ttft_s": 1.506, "equal_first_tokens": true}, {"phase": "repeated", "prompt_tokens": 32768, "cached_tokens": 30720, "cache_state": "reused", "ttft_s": 1.512, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 32768, "cached_tokens": 30720, "cache_state": "reused", "ttft_s": 1.505, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 32768, "cached_tokens": 30720, "cache_state": "reused", "ttft_s": 3.11, "equal_first_tokens": true}]}
{"run": "mu", "tokens": 65536, "qualified": true, "first_token_sha256": "137f1eec0b11db8298f32ab74e09dc5d1c9b0438129f2e4c3528b0ade94598dd", "rows": [{"phase": "first", "prompt_tokens": 65536, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 46.152, "equal_first_tokens": null}, {"phase": "repeated", "prompt_tokens": 65536, "cached_tokens": 63488, "cache_state": "reused", "ttft_s": 1.531, "equal_first_tokens": true}, {"phase": "repeated", "prompt_tokens": 65536, "cached_tokens": 63488, "cache_state": "reused", "ttft_s": 1.532, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 65536, "cached_tokens": 63488, "cache_state": "reused", "ttft_s": 1.532, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 65536, "cached_tokens": 63488, "cache_state": "reused", "ttft_s": 3.159, "equal_first_tokens": true}]}
{"run": "ou", "tokens": 2048, "qualified": true, "first_token_sha256": "50f9443218e13caab52d4af7fba39d1cc707a6e0ae6752e39cd4fb1d25eb8c1f", "rows": [{"phase": "first", "prompt_tokens": 2048, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 1.412, "equal_first_tokens": null}, {"phase": "repeated", "prompt_tokens": 2048, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 1.4, "equal_first_tokens": true}, {"phase": "repeated", "prompt_tokens": 2048, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 1.399, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 2048, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 1.405, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 2048, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 2.913, "equal_first_tokens": true}]}
{"run": "ou", "tokens": 8192, "qualified": true, "first_token_sha256": "fe0ef359af30fb6de6e26602f88ac9dc76612b1c69c2735efec904385fccacca", "rows": [{"phase": "first", "prompt_tokens": 8192, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 5.804, "equal_first_tokens": null}, {"phase": "repeated", "prompt_tokens": 8192, "cached_tokens": 6144, "cache_state": "reused", "ttft_s": 1.496, "equal_first_tokens": true}, {"phase": "repeated", "prompt_tokens": 8192, "cached_tokens": 6144, "cache_state": "reused", "ttft_s": 1.499, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 8192, "cached_tokens": 6144, "cache_state": "reused", "ttft_s": 1.498, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 8192, "cached_tokens": 6144, "cache_state": "reused", "ttft_s": 3.086, "equal_first_tokens": true}]}
{"run": "ou", "tokens": 32768, "qualified": true, "first_token_sha256": "47c8b22cb55a3736b6cec2795f03938d3187b02a107c401e3ebbb5490fcf3a45", "rows": [{"phase": "first", "prompt_tokens": 32768, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 21.588, "equal_first_tokens": null}, {"phase": "repeated", "prompt_tokens": 32768, "cached_tokens": 30720, "cache_state": "reused", "ttft_s": 1.514, "equal_first_tokens": true}, {"phase": "repeated", "prompt_tokens": 32768, "cached_tokens": 30720, "cache_state": "reused", "ttft_s": 1.518, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 32768, "cached_tokens": 30720, "cache_state": "reused", "ttft_s": 3.13, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 32768, "cached_tokens": 30720, "cache_state": "reused", "ttft_s": 1.517, "equal_first_tokens": true}]}
{"run": "ou", "tokens": 65536, "qualified": true, "first_token_sha256": "137f1eec0b11db8298f32ab74e09dc5d1c9b0438129f2e4c3528b0ade94598dd", "rows": [{"phase": "first", "prompt_tokens": 65536, "cached_tokens": 0, "cache_state": "cold", "ttft_s": 42.934, "equal_first_tokens": null}, {"phase": "repeated", "prompt_tokens": 65536, "cached_tokens": 63488, "cache_state": "reused", "ttft_s": 1.537, "equal_first_tokens": true}, {"phase": "repeated", "prompt_tokens": 65536, "cached_tokens": 63488, "cache_state": "reused", "ttft_s": 1.54, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 65536, "cached_tokens": 63488, "cache_state": "reused", "ttft_s": 1.54, "equal_first_tokens": true}, {"phase": "concurrent", "prompt_tokens": 65536, "cached_tokens": 63488, "cache_state": "reused", "ttft_s": 3.175, "equal_first_tokens": true}]}

bench_openai.py --tokens 256 --reps 3 --temperatures 1.0,0:

{"run": "mu", "prompt": "fibonacci-raw", "temperature": 1.0, "decode_tps_all": [51.38, 42.47, 45.08], "ttft_s_median": 0.127, "token_sha": ["bb64b44e5544", "ba0b76c1abe6", "e4b32f31655e"]}
{"run": "mu", "prompt": "gpu-chat-no-think", "temperature": 1.0, "decode_tps_all": [36.69, 37.13, 38.0], "ttft_s_median": 0.195, "token_sha": ["9d5725485ba0", "402df752cd35", "00c716572e0e"]}
{"run": "mu", "prompt": "fibonacci-raw", "temperature": 0.0, "decode_tps_all": [48.03, 47.94, 47.96], "ttft_s_median": 0.127, "token_sha": ["5b8af3e0ec61", "5b8af3e0ec61", "5b8af3e0ec61"]}
{"run": "mu", "prompt": "gpu-chat-no-think", "temperature": 0.0, "decode_tps_all": [38.92, 38.91, 38.94], "ttft_s_median": 0.196, "token_sha": ["68b81544a50b", "68b81544a50b", "68b81544a50b"]}
{"run": "ou", "prompt": "fibonacci-raw", "temperature": 1.0, "decode_tps_all": [51.24, 42.6, 45.1], "ttft_s_median": 0.128, "token_sha": ["bb64b44e5544", "ba0b76c1abe6", "e4b32f31655e"]}
{"run": "ou", "prompt": "gpu-chat-no-think", "temperature": 1.0, "decode_tps_all": [36.67, 37.09, 37.92], "ttft_s_median": 0.195, "token_sha": ["9d5725485ba0", "402df752cd35", "00c716572e0e"]}
{"run": "ou", "prompt": "fibonacci-raw", "temperature": 0.0, "decode_tps_all": [47.99, 48.1, 48.02], "ttft_s_median": 0.127, "token_sha": ["5b8af3e0ec61", "5b8af3e0ec61", "5b8af3e0ec61"]}
{"run": "ou", "prompt": "gpu-chat-no-think", "temperature": 0.0, "decode_tps_all": [38.8, 38.98, 38.99], "ttft_s_median": 0.194, "token_sha": ["68b81544a50b", "68b81544a50b", "68b81544a50b"]}

bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial, this head:

{"prompt": "code", "temperature": 0.0, "streams": 1, "failed": 0, "aggregate_tps": 48.1, "per_stream_tps": 48.1, "alone": {"equal": 3, "unequal": 0, "failed": 0}, "serial": {"equal": 1, "unequal": 0, "failed": 0}}
{"prompt": "code", "temperature": 0.0, "streams": 4, "failed": 0, "aggregate_tps": 108.1, "per_stream_tps": 27.6, "alone": {"equal": 12, "unequal": 0, "failed": 0}, "serial": {"equal": 1, "unequal": 0, "failed": 0}}
{"prompt": "chat", "temperature": 0.0, "streams": 1, "failed": 0, "aggregate_tps": 38.8, "per_stream_tps": 38.8, "alone": {"equal": 3, "unequal": 0, "failed": 0}, "serial": {"equal": 1, "unequal": 0, "failed": 0}}
{"prompt": "chat", "temperature": 0.0, "streams": 4, "failed": 0, "aggregate_tps": 116.9, "per_stream_tps": 30.2, "alone": {"equal": 12, "unequal": 0, "failed": 0}, "serial": {"equal": 1, "unequal": 0, "failed": 0}}

bench_concurrent.py --prompts tools/bench_prompts.json --tokens 256 --reps 3 --temperatures 1.0,0 --levels 1,4:

{"run": "main", "prompt": "distinct", "temperature": 1.0, "streams": 1, "failed": 0, "unmeasured": 0, "aggregate_tps": 58.3, "per_stream_tps": 58.3, "steady_tps": null}
{"run": "main", "prompt": "distinct", "temperature": 1.0, "streams": 4, "failed": 0, "unmeasured": 0, "aggregate_tps": 71.7, "per_stream_tps": 21.8, "steady_tps": 82.7}
{"run": "main", "prompt": "distinct", "temperature": 0.0, "streams": 1, "failed": 0, "unmeasured": 0, "aggregate_tps": 68.1, "per_stream_tps": 68.1, "steady_tps": null}
{"run": "main", "prompt": "distinct", "temperature": 0.0, "streams": 4, "failed": 0, "unmeasured": 0, "aggregate_tps": 74.8, "per_stream_tps": 20.9, "steady_tps": 80.7}
{"run": "head", "prompt": "distinct", "temperature": 1.0, "streams": 1, "failed": 0, "unmeasured": 0, "aggregate_tps": 58.2, "per_stream_tps": 58.2, "steady_tps": null}
{"run": "head", "prompt": "distinct", "temperature": 1.0, "streams": 4, "failed": 0, "unmeasured": 0, "aggregate_tps": 71.1, "per_stream_tps": 21.8, "steady_tps": 83.2}
{"run": "head", "prompt": "distinct", "temperature": 0.0, "streams": 1, "failed": 0, "unmeasured": 0, "aggregate_tps": 68.1, "per_stream_tps": 68.1, "steady_tps": null}
{"run": "head", "prompt": "distinct", "temperature": 0.0, "streams": 4, "failed": 0, "unmeasured": 0, "aggregate_tps": 75.5, "per_stream_tps": 20.7, "steady_tps": 81.4}

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency (this head's
  glm_prompt_host.zig on the base fails to build: promptRun takes no bound there and dflash has no tapRows; it
  has no dependency to skip on).
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (Exactness
  above).
• No precision traded for speed. If the bits change, the description says which sums change and why (no arithmetic
  changes: a buffer's rows and a bound).
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter,
  no slower than without it, greedy and sampled (the table above at one and four streams, sampled and greedy; without
  --drafter this head runs main's code).

[ashhart]

Thanks @Bizuayeu, and @MiaAI-Lab for the patch this follows. Drafted equal to plain and concurrent equal to alone, with the cold first token 6.1% and 7.0% sooner at 32K and 64K, makes this an easy one. It's in the queue; we'll add the changelog line.

[ashhart]

Thank you @Bizuayeu. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @Bizuayeu and @MiaAI-Lab! This is in TensorFold 1.2.3, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.3