[ruibeikaa]

Problem and result

Measured on PH402 SKU 200 boards: four GP100 dies, 32 GB HBM2 each (128 GB in all), every expert resident in VRAM, no CPU share, no PCIe share. The change itself is not GPU-specific: it is host bookkeeping plus GDN kernels shared by every CUDA / HIP build, and any layer split that runs --pipeline-windows 2 has the copy it removes.

With --pipeline-windows 2 on a layer split, before every speculative window each front stage copies its GDN state on the side stream beside the window (snap_take), so that a wrong guess can restore it and replay the verified window's commit. For Swift 1.5 at --layer-split 12,24,36 that is 29.4 MB per front stage per window (nine GDN layers), 58.8 MB of reads and writes competing with the window for memory bandwidth. On one GP100 die the window graph was 0.13-0.14 ms slower per stage with the copy running beside it (a window is ~6.85 ms per stage there).

With STRATA_PIPELINE_SNAP_RING=1 nothing is copied. A front stage's state lives in one of three buffers - its session's own and the two snapshot buffers the loop already allocates - and the window kernels read the one a device word names. A speculative commit reads the current buffer and writes a free one, so the state its window read stays where it was: that buffer is the snapshot. An undo replays the verified window's commit in place in that buffer; every other commit is in place, as today. The bytes every window and commit compute are today's (the same kernel bodies on the same inputs); only the address of the state moves. No VRAM is added.

End to end on this branch, four GP100 dies, today's --pipeline-windows 2: decode +0.8% (57.78 -> 58.24 tok/s, two runs per arm in the order off, on, on, off), every reply and every item of the session state identical, also under forced wrong guesses. On my GP100 stack (engine 0.1.47, with a --pipeline-windows 3 loop, details below): decode +0.9..+1.4% (8 prompts token-weighted 60.57 -> 61.15 tok/s, ms per window -0.8..-1.1% on every prompt, 150K context +1.2%), prompt reads unchanged from 8K up, every reply hash identical, and under forced wrong guesses (about 1,230-1,280 undone windows per arm) every stage's state hash identical to the copying loop's - for the 3-deep loop and for today's 2-deep loop, the one main has. Unset (the default) nothing changes.

Related discussion

No scope issue: an opt-in switch inside the existing --pipeline-windows loop, no default or format change. The loop for three or more stages is #1656's.

Approach

• src/kernels/cuda/verify_kernels.cu, include/strata/kernels/verify_kernels.hpp:
  • gdn_conv_l2_multi_ind / gdn_step_norm_multi_ind: the verify path's conv and recurrence (no commit, no q8_1 outputs) on the state at *cur + off, read when the kernel starts. The conv calls gdn_conv_l2_body unchanged; the recurrence is gdn_step_norm_multi_kernel with one more template parameter that adds state = *ind + ind_off at its start.
  • gdn_commit_layers_ring: gdn_commit_layers' two kernel bodies (gdn_step_commit_body_oop: the same statements, the read and the write taking their own base) reading buffer tab[sel[0]] and writing tab[sel[1]], after which one thread sets *cur = tab[sel[1]]. sel is two more words of the verifier's mapped commit words (the device array keeps its 256-byte size), copied in by the commit graph's existing copy_i32_from_mapped.
  • gdn_ring_supported(): false where gdn_commit_layers or the window would not take these paths (STRATA_GDN_SPLIT=1, STRATA_GDN_COMMIT_SPLIT=0, STRATA_DBG_GDN=1).
• src/core/verify.cpp, include/strata/core/verify.hpp: Verifier::set_gdn_ring (only before the verifier's graphs are captured), gdn_ring_ok, the two window kernels and the ring commit wired in when the ring is set, and pl_commit_async(n_keep, err, ring_src, ring_dst). The serial commit() writes 0, 0 (in place in the session's buffer, as today).
• src/program/generate.cpp: the per-stage buffer table and device word, pl_ring_sel (an undo's replay in place in the restored buffer and clear the snapshot records; a speculative commit from the current buffer into one that is neither current nor recorded, recording the old one as the window's snapshot; anything else in place), pl_ring_restore, and pl_ring_settle at the start of a pipelined request and after pl_drain: if a stage's state is not in its session's buffer, that stage's layer range is copied back once (29.4 MB here) and the word reset, so the serial paths, checkpoints, parking and STRATA_STATE_HASH never see the ring. snap_take and the undo's restore copy nothing when the ring is on. Three buffers are enough for the 2-deep loop: two snapshot records at most plus the current buffer.
• Refused, said once on stderr, today's copy kept: without the snapshots beside the window (STRATA_PIPELINE_SNAP_OVERLAP=0), with STRATA_QFUSE, STRATA_FUSE_CONVAB, STRATA_COMMIT_BATCH=0, STRATA_GDN_SPLIT=1, STRATA_GDN_COMMIT_SPLIT=0 or STRATA_DBG_GDN=1, on Metal, or when a verifier's graphs were captured before the ring was set.
• Cost when on: each GDN kernel of a window reads one device word (nine layers x two kernels a stage); a request that ends with a stage's state outside its session's buffer pays one copy of that stage's GDN state.
• src/kernels/gdn_parity.cpp section 7: the ring's kernels against today's.

Validation

Environment note: measured only on PH402 SKU 200 boards - four GP100 dies of 48 SMs and 32 GB HBM2 each (128 GB in all), application clocks 715 / 1050 MHz - with every expert resident in VRAM (no CPU share, no PCIe share, no adaptive tier). The single-die check uses one of those dies. Nothing in the change is specific to that card, but no other card was run.

Rig: Windows 11, driver 581.80 (TCC), CUDA 12.9, VS 2022; build -DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=60, ggml at the pinned 3cf03257f. Model: Swift 1.5 IQ3_S GGUF.

Passed:

• This branch (main 870aedd + this PR), one GP100 die: STRATA_Q8_SM60=1 gdn_parity --selftest: gdn_commit_layers_ring == in place: bitwise (8 cases, 5 layers), ring window kernels == today's: bitwise (16 cases), gdn: 0 failures, gdn_parity OK (the existing sections, including gdn_commit_layers == per-layer commit, pass as on main). Section 7 checks (a) gdn_commit_layers_ring from buffer 0 into buffer 1, and in place in buffer 2, against gdn_commit_layers in place, n_keep 1..8, five layers starting one layer into the buffers: memcmp equal, the source buffer and the layer before untouched, the device word set to the buffer written; (b) the window's conv and recurrence through the word against gdn_conv_l2_multi / gdn_step_norm_multi's verify path, T 1..8, outputs from row 0 and from T/2: memcmp equal.
• Build: sm_60 with STRATA_EXPERIMENTAL_SM60 (strata, gdn_parity), clean.
• End to end, this branch's own revision, four GP100 dies, Swift 1.5 IQ3_S GGUF (dense matrices in Q8_0), MTP, --layer-split 12,24,36, --pipeline-windows 2, --max-context 262144, environment STRATA_Q8_SM60=1 STRATA_MTP_Q8_SM60=1 STRATA_STATE_HASH=1 and nothing else, one fresh engine per run, greedy, six requests (a three-turn conversation of about 3.5K tokens, three decode prompts of 600 tokens out). Every run with the switch logged the GDN snapshot ring.
  • Decode on the three prompts, runs in the order off, on, on, off: 57.84, 58.22, 58.26, 57.73 tok/s: +0.8%.
  • Off against on: 6 of 6 replies and 6 of 6 sessions equal, every STRATA_STATE_HASH item.
  • Forced wrong guesses (STRATA_PIPELINE_DEBUG=1 STRATA_PIPELINE_FORCE_MISS=3, decode 52.3 against 52.7 tok/s): 6 of 6 replies and 6 of 6 sessions equal between off and on, so every undo restored the same state.
  • Default path: this branch with the switch unset against main 870aedd unchanged: 6 of 6 replies and 6 of 6 sessions equal.
• End to end, four GP100 dies, my earlier stack, not this branch's revision: engine 0.1.47 (main 011c9e6) plus about 35 opt-in GP100 commits of mine ([Engine] GP100 (sm_60): 4-bit slots for resident IQ2_S / IQ3_XXS / IQ3_S gate/up experts, decode +9.7% (opt-in, STRATA_EXPERT_NIBBLE=1) #3455, [Engine] GP100 prompt path: a fully resident layer's experts weight-stationary, one launch per product (STRATA_MMQ_WS=1, opt-in) #3752, a variant of [Engine] GP100 (sm_60): the decode down projection staged in shared memory, bitwise expert mode 0 (opt-in, STRATA_DOWN_WS=1) #3867, a --pipeline-windows 3 loop and others not posted), Swift 1.5 IQ3_S, 1M context, --layer-split 12,24,36, --pipeline-windows 3, MTP, one fresh engine per arm, 13 requests per arm (five short prompts, 8K / 26K / 88K-token contexts with 300-400 tokens of decode, 1K / 2K / 6.9K / 30K reads, a 151K read with 800 tokens of decode), two runs:
  • Every reply hash and the draft acceptance (accepted / offered) with the switch equal to the same stack's without this change, 13/13 in both runs (the same build with the switch off matched too, and decoded within 0.2% of it).
  • Decode, one start per arm in the same run: 8 prompts token-weighted 60.57 -> 61.15 tok/s; ms per window, without -> with (without = the mean of three clean runs): code 26.79 -> 26.58, SQL 26.00 -> 25.76, English code 23.53 -> 23.30, story 24.15 -> 23.91, explanation 24.88 -> 24.64, 8K 25.20 -> 24.94, 26K 27.01 -> 26.74, 88K 27.48 -> 27.25; 150K 49.4 -> 50.0 tok/s. The tokens per window are the same in both arms, so the window time is the whole difference. Prompt reads of 8K and more within 0.1%; 1K / 2K reads were ~25 ms (0.8-1.0%) slower with the switch in both runs, which I have not explained.
  • State under forced wrong guesses: STRATA_PIPELINE_FORCE_MISS=3 with two more local debug switches of my stack (not in this PR) that force wrong guesses on the later stages too and hash every stage's GDN, KV, index and PLE state after every request, plus STRATA_STATE_HASH=1, eight decode requests. 3-deep loop: 8/8 equal to the copying loop in one run (1,284 undone windows), 7/8 in the other, where the one difference is the end position of a request whose last window was cut by max_tokens (the count committed there depends on timing and is not comparable; the same request was equal in the other run). Today's 2-deep loop (main's), forced through STRATA_PIPELINE_SWITCH with pw=2: 8/8 equal (1,231 undone windows).
  • VRAM per die unchanged.
• During development, on the stack: cuobjdump of verify_kernels.cu (sm_60) showed the existing kernels' SASS unchanged with the switch's code compiled in; the four instances of gdn_step_norm_multi_kernel, whose template gained the parameter, have the same instruction bodies. Not repeated on this branch.

Failed: none observed.

Not run:

• HIP, SYCL and Metal builds (no such machine here; the ring is refused on Metal and its kernel calls are outside the Metal build), and any CUDA card other than GP100, although the change is not GPU-specific. An sm_86 build.
• A long conversation on this branch's own revision (the run on it is about 3.5K tokens; the 150K context above is my stack).
• A two-stage split (only the four-stage split above), and a model other than Swift 1.5.

Limits

• On when asked only; the copy stays the default.
• Measured only on PH402 dies with every expert resident in VRAM, and the gain there is about 1% of decode: it scales with how much of a window the 29 MB copy slowed, which depends on the card's bandwidth and the stage's GDN layer count.
• The ring needs the batched GDN commit and the window's plain conv / recurrence kernels; with the switches listed above it falls back to the copy.