Repository navigation
glm cuda: TF_GLM_PROMPT_TAIL=1 keeps raw prompts' tail in the unpaged prompt cache, as the KV pages' store does - #2226
Conversation
|
Thanks @Bizuayeu, the growing raw prompt going from 8.68 s to 1.44 s on its repeat, with the same tokens as main's cold run, is a clear win, and finding it while measuring #2220 was a good catch. One question before it lands: raw prompts now add a kept state to the same budget chat states use, and it's on by default for GLM. Could you run chats and raw completions together at the default cache size, for example a few chat turns, then several long raw prompts, then the next chat turn, and post the chat turns' |
Bizuayeu
changed the title
It drops once the raw prompts outgrow the budget, so the tail now waits behind One chat of five turns, about 9.8K tokens a turn, with two raw 32K prompts before turn 4 and five after it, at TP=2
Two raw prompts fit beside the chat's state; with five, the store, which frees the least recently used state first, |
|
Thanks @Bizuayeu, that table answers it: 38,912 cached against 0 on turn 5, 6.9 s against 34 s, because the store frees the least recently used state first. Keeping the tail behind |
|
Thanks @Bizuayeu. Behind |
…y, as TF_GLM_KV_HOST does The setting took any value but empty and 0 as on, the reading of a switch that defaults on, so =false turned it on. It now reads as TF_GLM_KV_HOST does: Settings.read refuses any value but 0 or 1 by name on every rank before the weights load, and the switch sits beside it in core/profile.zig (identical, exact one, glm5_next, ashhart#2226), where cacheRules reads it through core.profile.lookup and capabilities --json lists it with the others.
5641114 to
6ad19aa
Compare
Both are in a third commit,
On this head at TP=2 on two GB10s, a launch each:
|
|
Thanks @Bizuayeu. A profile row and a strict 0-or-1 check on every rank is exactly it. Queued for after the 1.2.3 cut. |
…y, as TF_GLM_KV_HOST does The setting took any value but empty and 0 as on, the reading of a switch that defaults on, so =false turned it on. It now reads as TF_GLM_KV_HOST does: Settings.read refuses any value but 0 or 1 by name on every rank before the weights load, and the switch sits beside it in core/profile.zig (identical, exact one, glm5_next, ashhart#2226), where cacheRules reads it through core.profile.lookup and capabilities --json lists it with the others.
6ad19aa to
16f6b36
Compare
16f6b36 to
0d89383
Compare
…y, as TF_GLM_KV_HOST does The setting took any value but empty and 0 as on, the reading of a switch that defaults on, so =false turned it on. It now reads as TF_GLM_KV_HOST does: Settings.read refuses any value but 0 or 1 by name on every rank before the weights load, and the switch sits beside it in core/profile.zig (identical, exact one, glm5_next, ashhart#2226), where cacheRules reads it through core.profile.lookup and capabilities --json lists it with the others.
… the KV pages' store does The paged store (prompt_radix.zig) keeps a prompt's last page under Rules.tail, for a later turn whose history the request did not name, and GLM's KV pages' store sets it (native/cuda_prompt_pages.zig). The unpaged store had no such rule, and the repeat log never marks a planned family's prompts, so on GLM without TF_GLM_KV_SHARE a /v1/completions prompt kept no state: the same prompt or a longer one never resumed. marksRewind now adds the tail as a mark when Rules.tail is set and the request names no history (history_len 0), snapped like every planned mark to the family's chunk starts. A chat keeps its history's mark alone, so its states do not change. cacheRules reads a family's snap_tail, and GLM sets it.
A raw prompt's tail shares the prompt cache's budget with chat states, and the store frees the least recently used first, so long raw prompts between two chat turns can push the chat's state out. Until that trade is measured, GLM keeps the tail behind a setting: snap_tail names TF_GLM_PROMPT_TAIL, and cacheRules turns the rule on when it is set, not empty and not 0, as TF_FLASHNEXT_REPEAT_CACHE turns Flash Next's repeat rule off.
…y, as TF_GLM_KV_HOST does The setting took any value but empty and 0 as on, the reading of a switch that defaults on, so =false turned it on. It now reads as TF_GLM_KV_HOST does: Settings.read refuses any value but 0 or 1 by name on every rank before the weights load, and the switch sits beside it in core/profile.zig (identical, exact one, glm5_next, ashhart#2226), where cacheRules reads it through core.profile.lookup and capabilities --json lists it with the others.
0d89383 to
16a11eb
Compare
…y, as TF_GLM_KV_HOST does The setting took any value but empty and 0 as on, the reading of a switch that defaults on, so =false turned it on. It now reads as TF_GLM_KV_HOST does: Settings.read refuses any value but 0 or 1 by name on every rank before the weights load, and the switch sits beside it in core/profile.zig (identical, exact one, glm5_next, #2226), where cacheRules reads it through core.profile.lookup and capabilities --json lists it with the others.
|
Thank you @Bizuayeu. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes. |
On two GB10s at TP=2 with the prompt cache off the pool's pages (
TF_KV_SHAREunset),TF_GLM_PROMPT_TAIL=1makes a raw
/v1/completionsprompt keep the state at its last chunk start, so its repeat and a longer prompt resumethere. Unset or
0, as by default, such a prompt keeps none, as on main, and GLM refuses any other value by namebefore the weights load. The tail shares the cache's budget with chat states: at the default size, five raw 32K prompts
between two chat turns pushed the chat's state out, so the tail waits behind the setting until that trade is measured.
Every reply's tokens are main's:
TF_GLM_PROMPT_TAIL=1bench_cache.py --require-cold --require-reuseat 8K / 32K tokens: the repeats' cached tokens, qualifiedThe raw prompts are upstream's
tools/*.pyatc085aab04, tokenized by the server's/tokenize: the 8K and 32Kfixtures are disjoint slices of them, the growing prompt takes the first tokens of a 64K slice, and the chat run's raw
prompts are 32K slices of that one, 4,096 tokens apart, so none shares a prefix with another. The chat's turns are
numbered lines, about 9.8K tokens a turn, each followed by its 32-token greedy reply, thinking off. Main's chat row is
two launches; the first commit alone, the tail always on, also gave 0 at turn 5 in two launches (34.16 and 34.02 s). The
second commit's columns were measured at
56411145a, on main8abb30f6f: the third changes only how the setting isread, and its own short receipt, taken at
6ad19aae1, is under Receipt.What this changes
The paged store keeps a prompt's last page under
Rules.tail, for a later turn whose history the request did not name,and GLM's KV pages' store sets it (
native/cuda_prompt_pages.zig:37). The unpaged store had no tail, and the repeat logmarks no planned family's prompt (
prompt_cache_repeat.zig), so withTF_KV_SHAREunset a/v1/completionsprompt, which names no history, kept no state at all.
marksRewindnow adds the tail as a mark whenRules.tailis set and the request names no history (history_len0).On a planned family it snaps to a chunk start like every other mark, so no pass gains a cut. A chat request names its
history and keeps that mark alone.
cacheRulessetsRules.tailwhen the switch a family'ssnap_tailnames reads1through
core.profile.lookup. GLM namesTF_GLM_PROMPT_TAIL, a row incore/profile.zigbesideTF_GLM_KV_HOST(
identical, exact one,glm5_next), socapabilities --jsonand the startup line list it, andSettings.readrefuses any value but
0or1by name on every rank before the weights load. The KV pages' store still sets its owntail. A kept state costs what a chat's costs, about 0.57 GB a rank at 32K tokens on GLM-5.3-Flash at TP=2, from the
same budget, and the store frees the least recently used state first: in the chat run the five raw prompts' tails
(564 MiB each) pushed turn 4's state out.
bench_cache.py --require-reusefailed on main and on that headalike at these settings).
a189233f8, with glm cuda: TF_KV_SHARE keeps GLM's prompts in the shared prompt store, and GLM's own store goes #2211), three commits: the tail, the setting it waits behind, and the setting readas
0or1through the profile registry. Rebased from6ad19aae1(on9200a3064) once glm cuda: TF_KV_SHARE keeps GLM's prompts in the shared prompt store, and GLM's own store goes #2211 landed: the code isunchanged, the recipe's bullet names
TF_KV_SHAREwhere glm cuda: TF_KV_SHARE keeps GLM's prompts in the shared prompt store, and GLM's own store goes #2211 renamedTF_GLM_KV_SHARE, anddocs/profiles.md's GLMline keeps both sides (
TF_KV_PAGES,TF_GLM_KV_HOST,TF_GLM_KV_HOST_SHARED,TF_GLM_PROMPT_TAIL). The table wasmeasured on
8abb30f6f, before the first rebase.atc overlap --json(2026-11-06 02:48 UTC, basea189233f8;complete: false, as atc could not read every open pullrequest's files in its time): 12 open pull requests share files with this one, and atc's test merges conflict for
five. Merging each with main alone as well (
git merge-tree) shows three that are this branch's: glm:TF_GLM_KV_LIVE_HALF=1keeps the live half of the KDA state alone in a kept prompt state #2181 indocs/recipes/glm-5.3-flash-cuda.md, where each adds a bullet at the same place; core: a bounded ladder of intermediate prompt-cache checkpoints #1953 innative/cuda_snaps.zig'scacheRules, where each adds a family declaration's line aftersnap_fork's; and glm cuda: TF_GLM_GRAPHS=1 replays verify windows as CUDA graphs (#1490) #2279 incore/profile.zig, whereeach adds a switch after
TF_GLM_KV_HOST_SHARED. In each, both stay. qwen3_5 cuda: a chat's next turn can resume at the previous turn's history (opt-in) #2176 and glm cuda: windows to 32 rows, priced by rounds timed at load #2257 conflict with main itself, inthe same files with or without this branch, and the two atc could not test-merge, prompt cache: a refused restore tries the next shorter kept state #2144 and qwen3_5 cuda: a chat's next turn resumes at the previous turn's history (message starts for the planned cache) #1335, conflict in none of
this branch's files. core: a bounded ladder of intermediate prompt-cache checkpoints #1953's checkpoints sit under a chat's history mark; this keeps one for a prompt that names no
history.
Checks on this head
16f6b362d.zig build test7,253 tests, 7,194 passed and 59 skipped,test-cuda-host,test-golden,test-glm-prompts-host8/8,test-glm-prompt-settings-host6/6,test-glm-tp3-host8/8,test-glm-prompt-arena-host1/1,test-glm-startup-host,test-glm-pages-host,compile-cuda-tests(alsoaarch64),
compile-config-server: no failures.zig fmt --check build.zig zigandgit diff --check: clean.6ad19aae1's (only the recipe anddocs/profiles.mddiffer); the receipts below were taken at6ad19aae1and before. At none of them: TP=3 (the threehosts that run it were in another session's use), and the KV pages' store, which this does not change.
Receipt
8abb30f6fand the second commit for the table, and6ad19aae1for the setting's reading, eachbuilt with Zig 0.17.0 and nvcc 13.3.73 for sm_121; two GB10s (128 GB unified memory each, Ubuntu 24.04.5, kernel
7.0.0-1019-nvidia, driver 580.178.04) at TP=2 over RoCE;
Bizuayeu/GLM-5.3-Flash-XSFP4at88736f91;--context 262144 --parallel 4with the cache not named (2.2 to 2.4 GiB by the room each start found; main'sbench_cache.pyand growing-prompt rows came from a--prompt-cache-gib 4start, which on8abb30f6f, before thechange in core kv_pool: an asked prompt cache comes before every window but the engine's own #2220, gave the same 2.4 GiB),
TF_KV_PAGESunset (pages),TF_GLM_KV_SHAREunset; the second commit'scolumns with
TF_GLM_PROMPT_TAILempty (which it read as unset) and set to 1.6ad19aae1(binaryeb8b73e7c4d3bbc5), a launch each: unset, the startup line readsprofile: stable; registered overrides: noneandbench_cache.py --require-cold --require-reuse's 8K repeats runcold, as on main; with
=1it readsregistered overrides: TF_GLM_PROMPT_TAIL=1 (identical, env)and the repeatsresume from 6,144 tokens; with
=falseboth ranks stop before the weights load witherror: TF_GLM_PROMPT_TAIL: 0 or 1. Each reply's tokens were main's first request's, andcapabilities --jsonlists the row (
glm5_next,cuda,identical,one,#2226).snap_tailsetting is on (GLM on CUDA withTF_GLM_PROMPT_TAIL=1) and requests that name no history; chat requests and the KV pages' store are unchanged, andwith the setting unset or
0nothing is. Run: TP=2 on two GB10s (sm_121). Not run: TP=3, other SMs.bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serialon the secondcommit with the setting on: every reply at one and four streams equalled its request alone (code and chat, 3 and 12 of
3 and 12), and the alone runs equalled
"draft": false. Resumed equals fresh:bench_cache.py's repeats andconcurrent requests gave the first request's tokens, which were main's, on every start, resumed from the tail (6,144
and 30,720 tokens cached) with the setting on; the growing prompt and every request of the chat run gave main's
tokens, resumed or cold.
bench_openai.py --tokens 256 --reps 3 --temperatures 1.0,0gave 41.48 / 35.02 tok/s sampled and 44.18 / 35.95greedy on
fibonacci-raw/gpu-chat-no-think, against main's 41.46 / 34.02 and 43.99 / 35.99 (medians).launches: main 23.30 s median (23.00 to 23.50), the first commit 23.20 s (23.02 to 23.37). The chat run's cold raw
prompts gave 23.06 to 23.33 s unset and 23.20 to 23.79 s with the setting (one launch each), main 23.01 to 23.72 s.
Raw results
bench_cache.py --require-cold --require-reuse, the server's first requests, main =m, the second commit unset =su, withTF_GLM_PROMPT_TAIL=1=sn:The setting's reading on
6ad19aae1, a launch each unset (zu), with=1(zn) and with=false(zf): the startupline,
bench_cache.py --require-cold --require-reuseat 8K, the refusals, andcapabilities --json:A raw prompt that grows (the 64K prompt's first 8,192 tokens, then 12,288, then 12,288 again; 64 greedy tokens):
The chat run (the client below; main
mx1mx2, the first committx1tx2, the secondsumunset andsnmwiththe setting, each a fresh launch):
bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial, the second commit withthe setting:
On the first commit, the setting's path: cold 32K raw prompts (
cold.py: three slices 4,096 tokens apart; launches m1,t1, m2, t2 in turn):
bench_openai.py --tokens 256 --reps 3 --temperatures 1.0,0, main =m, the first commit =t:bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial, the first commit:Chat prompts (about 7K, 15K and 30K tokens and a count prompt, 256 greedy tokens, drafted then
"draft": false), main=
m, the first commit =t:The chat run's client
Checklist
without its two lines of fix, the tail mark in
marksRewindandcacheRules'snap_tailline, failedprompt_cache_tail_test's test andcuda_snaps' test inzig build testandtest-cuda-host; the second withoutcacheRules' setting line failedcuda_snaps' test at the setting's on case inzig build test; the third withoutSettings.read's check failedcuda_ranks_test's new test inzig build testandtest-cuda-host(therehealth_test's manifest test also timed out once on its 5 s socket poll, and passed on the head), and withcacheRules' line read as the second commit read it,cuda_snaps' test failed at2inzig build test;profile_test's new test names the row's id, so it does not build without the row; they have no dependency to skipon).
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (Exactnessabove).
changes: which states the cache keeps).
--drafter,no slower than without it, greedy and sampled (unset, this head runs main's path; on, the first commit's, in the
Receipt; the setting keeps one more state for a raw prompt and touches no decode or drafter path).