Repository navigation
glm cuda: TF_KV_SHARE keeps GLM's prompts in the shared prompt store, and GLM's own store goes - #2211
Conversation
39a0f1d to
cbb31ac
Compare
|
Thanks @Bizuayeu, moving GLM onto the shared store with 355 replies on one hash at each TP, and the idle census matching main's, is the careful version of this. Three small things. First, we'll keep the refusal rather than an alias, since it stops before the weights load and names |
cbb31ac to
17c82c4
Compare
|
17c82c4 to
559ccef
Compare
|
Thanks @Bizuayeu, all three are in, and counting the TP=3 resend on both sides was the right check. One small thing from main moving: on today's main this puts |
559ccef to
7940c3d
Compare
7940c3d to
9754b6c
Compare
Done in |
|
Thanks @Bizuayeu, |
9754b6c to
029bb69
Compare
|
Thanks for rebasing this. The two code commits match the ones I pushed: the first differs only in context, where #2288's note now sits above its recipe hunk. One line moved, though: at |
… and GLM's own store goes GLM declares the core page adapter's hooks (share_pool, share_attach), so the server opens the one prompt_radix store for it, as for Spark 2.5, Gemma 4 and Kolibri-1 (ashhart#1657). cuda_prefix.zig goes. What stays GLM's is its side of a pass: which passes the store keeps (the content key), the keeps at the store's marks without table swaps, and a resumed state waiting in its owner for the pass's start on every rank (snap_restore). The rules are the old store's: 512-row pages, the 2,048-row grid, a lookahead of 1 and the tail. share_kept_ppm = 0 keeps GLM's cache as copies, since its states past the pages are each rank's snapshots in the cache's room, so the pool and the cache plan as under TF_GLM_KV_SHARE, and the server opens the store for it at any cache, as GLM's own store kept prompts whatever the cache's size. TF_GLM_KV_SHARE is refused at startup, naming TF_KV_SHARE.
…o 0 and empty start The refusal stopped a server with any TF_GLM_KV_SHARE. 0 and an empty value ask for no store, so they now start as if it were unset; a count or any other value is still refused before the weights load, naming TF_KV_SHARE. cuda_share.zig's host test pins the four cases.
029bb69 to
de7cdcb
Compare
|
Thank you @Bizuayeu. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes. |
GLM's half of #1657:
TF_KV_SHARE=nnow keeps GLM's prompts in the sharedprompt_radixstore over the core KV pool, through the native-CUDA page adapter that Spark 2.5, Gemma 4 and Kolibri-1 use, and GLM's own prefix code (cuda_prefix.zig) goes. With the store on, this head does what main'sTF_GLM_KV_SHARE=16does, at the same marks, pages and memory, with the same tokens. On two GB10s at TP=2 (524k window,PARALLEL=4, the drafts on):TF_GLM_KV_SHARE=16TF_KV_SHARE=16bench_openai)MemAvailableafter load, rank 0 / rank 1 (GiB)Unset, both are main's paths: decode 39.86 tok/s on main and 39.88 on this head, and the cold 136k prompt 100.07 and 98.59 s.
At TP=3 on three GB10s (1M window, the same settings):
TF_GLM_KV_SHARE=16TF_KV_SHARE=16bench_openai)MemAvailableafter load, ranks 0 / 1 / 2 (GiB)Unset at TP=3, main's flat store has no room for the 64k and 128k states, so those messages sent again prefill from the 6,144-row mark on both (32.1 and 66.8 s on main, 31.9 and 66.4 s on this head); decode is 51.40 and 51.16 tok/s, and the cold 136k prompt 80.94 and 80.71 s.
At TP=3 the 128k message sent again took 2.32 s on this head, its first token at 0.86 s against 0.61 s on main's store. That resume is sometimes slow on main's store too. In a count with eight different 128k messages per launch, each sent fresh and then again twice, over two launches of each store (16 first resends a side, the same binaries and settings), the first resend's first token came past 0.7 s twice on each side (0.78 and 0.77 s on this head, 0.94 and 0.79 s on main's store), each time on the launch's first 128k message, and the second resend's once on each (0.85 and 0.73 s, both on the fifth message); the medians were 0.62 and 0.63 s, and each message had one token hash over the four launches.
What this changes
GLM declares the adapter's hooks (
share_pool,share_attach), so the server opens the oneprompt_radixstore over rank 0's pool for it.cuda_prefix.zig,own_prompt_cacheandTF_GLM_KV_SHAREgo.What stays GLM's is its side of a pass, in a small
cuda_share.zig: which passes the store keeps (the content key, with an image's cache key), the keeps at the store's marks, and a resumed state that waits in its owner until the pass starts on every rank (snap_restore), ascuda_prefix.restoredid. A keep doesn't swap the table for pages the store already holds, so at TP no follower's table moves mid-pass; the overlap is held twice until the stream ends, and the tokens don't change.The rules are the old store's: 512-row pages, the planned 2,048-row grid, a lookahead of 1 (the MTP head's row at a mark reads the token after it) and the tail.
share_kept_ppm = 0: GLM's state past the pages is each rank's snapshot in the cache's room (start-up takes n times the state at the window's last mark, as before), so GLM keeps its cache as copies instead of turning the whole cache into pool pages. The pool and the cache then plan as underTF_GLM_KV_SHARE(the start line above), and the server opens the store for GLM at any cache size, including 0, as GLM's own store did.At TP the store and the pages stay rank 0's, and the followers learn what they need from GLM's own steps, as before: how many of the table's first pages are the cache's (so
TF_KV_POISONon a follower leaves them alone) and whether states leave the pages out.The setting:
TF_KV_SHARE=nwithTF_KV_PAGES=on, as for the other three. The profile registry listsglm5_nextunderTF_KV_SHARE, and itsTF_GLM_KV_SHARErow goes. A server still started withTF_GLM_KV_SHAREset to a count, or to anything but 0 or an empty value (which ask for no store and start as if it were unset), is refused before the weights load, namingTF_KV_SHARE, so it doesn't lose its store silently. Below 4 states a stream the start's warning namesTF_KV_SHARE. The recipe,docs/profiles.mdand the CHANGELOG follow.Issue: Part of CUDA: one paged prompt store for every family on the core KV pool (Gemma 4, Kolibri-1, Spark2.5 first) #1657 (GLM's half; Qwen's is yours, on qwen CUDA: a faster opt-in paged path (TF_QWEN_KV_PAGES=256), --parallel a cap with pages set #2011).
Stacked on: main (
a6e25faea), three commits: the move, then from review the refusal letting 0 and an empty value through, and the CHANGELOG line. On CUDA: one paged prompt store for every family on the core KV pool (Gemma 4, Kolibri-1, Spark2.5 first) #1657 I said I'd build on glm cuda: a prompt step takes two chunks while no other stream decodes, each one's exchanges under the other's sites #1423 if it was still queued; its base was 599 commits behind main and had nocore/profile.zig, where the setting's row lives, so this is on main. Against glm cuda: a prompt step takes two chunks while no other stream decodes, each one's exchanges under the other's sites #1423's head the merge conflicts in the same five files as main's own merge with it, from glm cuda: a prompt step takes two chunks while no other stream decodes, each one's exchanges under the other's sites #1423's older base (it moves onto main after glm cuda: TF_GLM_L2PF reads a decode window's next weights into L2 on a side stream, on by default on a GB10 #2162, at layout 19). Against glm cuda: TF_GLM_L2PF reads a decode window's next weights into L2 on a side stream, on by default on a GB10 #2162 it meets in one line ofcuda_engine.zig(besideshare_pages's comment) and in the CHANGELOG's head, where main's own merge with glm cuda: TF_GLM_L2PF reads a decode window's next weights into L2 on a side stream, on by default on a GB10 #2162 conflicts too; whichever lands second rebases.atc overlap --json, run after this rebase ontoa6e25faea(21:48 JST): incomplete (invalidPulls,pullFilesLimit,pullFilesUnavailable,timeLimit). Of what it finished, it found conflicts with Clef-Flash: joint-schema decisions on /v1/systemone (HIP) #2048 and server: DSML calls stream from the invoke name, not the first parameter's closer #2201, each only in the CHANGELOG, as main's own merge with it does, and clean merges with EXL3 Flash Next on CUDA: one rank, offloaded experts, sm_120 gate (1/8) #1441, Vulkan: one submission per batch of dispatches, 30% to 68% less run time on an M1 with the same logits (stacked on #2015) #2016, Vulkan: a dispatch records a barrier only when it conflicts with an earlier one, 3% to 7% less run time on an M1 with the same logits (stacked on #2017) #2038, Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074, hip: build and load the runtime on Windows #2178, Flash Next on CUDA: kept prompt states with drafts across two ranks #2214, engine api: an optional image capability beside speech (#1573) #2248 and test(ci): the heaviest host gates build in ReleaseSafe (#2242) #2249.git merge-treeagainst the heads it didn't finish, and against glm cuda: TF_GLM_PROMPT_TAIL=1 keeps raw prompts' tail in the unpaged prompt cache, as the KV pages' store does #2226, adds conflicts with cluster: serve --cluster brings a GB10 and a Mac up as one cluster over ConnectX and agrees on a disaggregated model's plan, before any tokens run (#1594, step 2a) #2088 (build/cuda_engines.zig, where it changes thetpimport on the line above theengine_apicomment this renames toTF_KV_SHARE), qwen3_5 cuda: a chat's next turn can resume at the previous turn's history (opt-in) #2176 (core/profile.zig, where it adds a row beside theTF_GLM_KV_SHAREone this takes out; itsbuild/cuda_family_host.zigconflict is main's own) and glm cuda: TF_GLM_PROMPT_TAIL=1 keeps raw prompts' tail in the unpaged prompt cache, as the KV pages' store does #2226 (the GLM row ofdocs/profiles.md, where it addsTF_GLM_PROMPT_TAILand this takes outTF_GLM_KV_SHARE; its recipe line namesTF_GLM_KV_SHAREtoo); the rest conflict only where main's own merge with them does (CUDA: GLM's KDA chain and saved-row replay become a shared module #2142 and glm:TF_GLM_KV_LIVE_HALF=1keeps the live half of the KDA state alone in a kept prompt state #2181 among GLM's) or merge cleanly, Unlimited-OCR-2 on CUDA: the family's steps on NVIDIA GPUs, the router's long-context speedup (stacked on #2074) #2077 among them now that this one's test import has moved out of the test block innative/cuda.zigwhere Unlimited-OCR-2 on CUDA: the family's steps on NVIDIA GPUs, the router's long-context speedup (stacked on #2074) #2077 adds its own. The rebases sincecbb31ac48resolved the conflicts with GLM CUDA: opt-in context copies priced by the lane host #2146 (itsTF_GLM_COPIESrow now follows this one's three incuda_native.zig's explain table) and in the CHANGELOG's head. Whichever lands second rebases.Checks on this head
9754b6c99python3 tools/zig/lean_check.py, on the base and on this head: 425 and 425 problems, 0 new.-j4):zig build test545/545 steps (2,259 of 2,295 tests, 36 skipped; it runstest-profile),zig build test-cuda-host313/313 steps (4,877 of 4,900, 23 skipped),test-kv-pool38/38,test-glm-image-cache-host3/3 (now over the adapter),test-glm-snapshots-host21/21,test-glm-pages-host12/12,test-glm-prompt-settings-host6/6,test-glm-tp3-host8/8,test-glm-prompts-host8/8, andtest-glm-startup-host,test-glm-fp8-kv-host,test-glm-admission-host,test-gemma4-shared-hostandcompile-glm-native-openwith every step passing;compile-cuda-tests(x86_64 and-Dtarget=aarch64-linux-gnu),compile-config-server, andzig fmt --check build.zig zig; with-Drerun-tests, as CI runs them,test-kernel-contract22/22 steps (180 of 180 tests) andtest-server-security test-pull-security14/14 steps (29 of 29 tests).glm_snapshot_host.zig: planned, grid 2,048, lookahead 1, min_prompt 0, no warm or fork), the TP and pass tests incuda_pages_test.zigover a store opened with GLM's decls, the image cache over the real adapter,keptPpmandopensStoreinnative/cuda_streams.zig, andfewStatesandmoved(whichTF_GLM_KV_SHAREvalues the start refuses) incuda_share.zig. On main they don't build, ascuda_share.zigand GLM'sshare_pool,share_attachandshare_kept_ppmaren't there (test-glm-pages-host,test-glm-snapshots-hostandtest-glm-image-cache-hostwith this head's test files stop atunable to load "cuda_share.zig").9754b6c99(fmt;zig,zig-windows,zig-macos-dev-tools,release-paths, andbenchmark-clientson Ubuntu and Windows).Receipt
cbb31ac48(this branch's first commit before the rebases; ona6e25faeait is4baed8d62, whose diff changed only in context and in the file that importscuda_streams.zigfor its tests,cuda_kv_pool.ziginstead ofnative/cuda.zig, which main has brought to the lean checker's 600 lines; the two commits after it change only whichTF_GLM_KV_SHAREvalues the start refuses and add the CHANGELOG line) and main8abb30f6f, each built with Zig 0.17.0 (.zig-version) andzig build native -Dnvcc=<nvcc 13.3> -Dsm=121(thetf3-buildcontainer, CUDA 13.3.1); GLM-5.3-Flash XSFP4 at88736f91, whose own MTP head drafts;PARALLEL=4,TF_KV_PAGES=on, serve.sh's window and prompt cache, the drafts on; GB10 (sm_121, Ubuntu 24.04, 128 GB), two joined by one QSFP cable (TP=2, 524k window) and three cabled in a ring (TP=3, 1M window). Launches at each TP: main unset, mainTF_GLM_KV_SHARE=16, this headTF_KV_SHARE=16, this head unset, this headTF_KV_SHARE=16underTF_KV_POISON=1, and this head with--prompt-cache-gib 0, whose replies are all fresh; at TP=2 also this head withTF_KV_SHARE=16andTF_GLM_KV_HOST=1. In each: one chat message of filler text at 2.5k, 4.5k, 6.5k, 64k and 128k tokens sent fresh, again and twice at once (the filler shares its start, so each message after the first resumed from the one before's mark),tools/bench_openai.py,tools/bench_concurrent.py --levels 1,2,4,8 --alone --serialandtools/bench_cache.py --require-coldon a 136k prompt of token ids; the launch without a prompt cache sends the chats only, and theTF_GLM_KV_HOST=1one all but the 136k prompt."draft": falsein each launch that ran them, the five with the 136k prompt at each TP and theTF_GLM_KV_HOST=1one (atPARALLEL=4, four of the 8 wait in the queue).kv pool idle42 times each, and no launch loggedUNEVEN. With the store on, the first idle waspromised 0, free 1024 + store 4 == total 1028and the lastfree 516 + store 512at TP=2 (free 6143 + store 4 == total 6147andfree 5635 + store 512at TP=3), the same on main's store and on this head.TF_KV_POISON=1on every rank the hashes and cells didn't change.TF_GLM_KV_HOST=1beside the store, at TP=2: the same hashes as every other launch, the chats resumed at the same rows as the store without it, all cells equal alone and"draft": false, andkv pool idle39 times withpromised 0, free 1037 + store 4 == total 1041first andfree 793 + store 248last. At start the pool took 4,100 pages of 512 rows (1.50 GiB) on the device and 1,041 pages (6,636 MiB) in pinned host memory,MemAvailableafter load was 13.63 / 14.62 GiB on ranks 0 / 1, and decode 39.83 tok/s against 39.89 without it.tools/bench_openai.py(256 tokens, 3 repeats, temperature 0, the mean of its two prompts' medians): the tables above; this head against main with the store on, +0.18% at TP=2 and −0.34% at TP=3, and unset +0.03% and −0.46%.tools/bench_cache.py --require-cold: the tables above.MemAvailableabove are the same.cbb31ac48started withTF_GLM_KV_SHARE=16stops on both ranks before serving, namingTF_KV_SHARE, andTF_KV_SHARE=4at--parallel 4starts with the warning that names 16. With the review's refusal (the same change as4efafeebf, run oncbb31ac48), 16 still stops both ranks before serving, and 0 and an empty value start.Checklist
a6e25faea; the receipt was taken oncbb31ac48, the first of them before the rebases, and on main8abb30f6f, as the Environment line says.)"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (The gates above, at TP=2 and TP=3.)SUMMARYline is in the receipt. (No GPU kernel is added or changed.)unable to load "cuda_share.zig", and GLM's adapter decls aren't there; none needs a device.)