Repository navigation
[Engine] A third decode window in flight on a layer split (--pipeline-windows 3, opt-in): a host-side scheduler, a slot pool, per-stage undo, an 11-output draft chain - #3760
Open
ruibeikaa wants to merge 10 commits into
Conversation
This was referenced Oct 23, 2026
ruibeikaa
force-pushed
the
pipeline-windows-3
branch
from
October 24, 2026 03:33
93b00c2 to
5569ade
Compare
…source-copy event (opt-in: set_chain_max / set_source_done / n_queue) --pipeline-windows 3 forces the chain through two windows in flight and then wants the next window's row 0 and drafts, so its outputs no longer fit in max_t - 1 = 7. The chain's cap is now its own, kChainMax = 11, and everything here is opt-in: a drafter nobody calls set_chain_max / set_source_done / chain_launch(n_queue) on queues the same operations as before. - Staging: R2 = max(2T, T + kChainMax) rows (2*max_t + 3 at max_t 8). step_row(j) keeps max_t + j - 1 up to j = max_t - 1 (the row that was spare), and puts steps j >= max_t after output 0's row 2*max_t - 1, so a chain of up to 7 outputs stages its cells in the rows it always did. h_out_ / h_prob_ hold 16; chain_tok_ / chain_prob_ / ev_step_ 12; step_exec_ 12. (The staging is three rows larger; nothing reads the new rows unless a chain has more than max_t - 1 outputs.) - set_chain_max(k), before prepare_chain (default max_t - 1): prepare_chain captures steps 1..k-1, the new ones after the old ones; chain_launch refuses n_out > chain_max() as it refused n_out > max_t - 1. - With a chain max past max_t - 1, n_out is clamped so the chain's last cell stays below the drafter's max_cells - 1 (an 11-output chain at the end of the context); n_force follows; chain_n() says what it became. Chains within the default max are not clamped. - chain_launch(..., n_queue): queue the first n_queue outputs only and leave the chain open; chain_extend(k) queues through output k (one step ahead of what the host has read, so the drafter's stream never waits on the host); chain_close() stops it; chain_poll reports an open chain done only once it is closed or fully queued, and chain_outputs_ready reads only the events of steps already queued. - set_source_done(true): each chain_launch records source_done() right after set_source_R's rows land in the bound buffer, for the slot's next window on the last stage to wait on (on the device) before it rewrites them. mtp_chain_parity (new, configured only with -DSTRATA_MTP_PARITY_BASE=<commit>, built only on request): src/core/mtp.cpp and its header at that commit (the one before this change) are copied out of git into the build directory with the class renamed, so the old and the new chain run in one process on the real model files. Old code against new code at n_out <= 7, an 11-output chain's prefix against the same chain at 7, chain_extend step by step (random n_queue, random early closes) against the full queue, source_done against the copied rows, the max_cells clamp, the refused 8th output, and the serial draft(): every token and every probability's bits equal. The model, shard and drafter paths are command-line arguments.
…ne, and its symbolic test (CPU only, not used yet) --pipeline-windows on a layer split keeps windows in flight behind the oldest one not verified; a deeper pipeline (a third window, per-stage undo) needs a scheduler that can be checked without a GPU. src/program/pl_sched.* decides what happens and in which order and calls no CUDA; the engine implements PlIo (polls of its verifiers and drafter, and an exec that turns each abstract operation into the verifier / drafter / snapshot call it names). Three modes: - N2: pw 2, global undo, 2 slots (slot = seq & 1): the pw-2 loop operation for operation, so the engine's translation can be checked against the old loop event by event; - B0: pw 2, per-stage undo, 3 slots: a wrong guess is undone on each front stage as soon as no doomed window runs there, and the fresh window goes on without waiting; - B: pw 3 (a third window in flight), per-stage undo, 4 slots, chains of up to 11 outputs stopped at the point that decides their last new window. Every reply token stays the serial loop's: only the oldest window is emitted, and a wrong guess is undone per stage by restoring the verified window's snapshot and replaying its commit with the real count. pl_sched_test (built on request, CPU only) runs the state machine against a symbolic GPU that tags every buffer with the window that wrote it and checks the hazards directly (a commit or restore of another window's data, a torn snapshot restored, a hand-off overwritten before it is read, the chain's source rows overwritten before its copy, a verifier launched or staged while in flight, a host spin that can never end): a recorded 4-stage decode trace's verdicts, random worlds, exhaustive small worlds with every ending (max_new, EOS, cancel, context edge) at every verdict, and the pw-2 loop transliterated as OldLoop against N2 (the same operations, events and host times). Upstream's pw-2 loop adds three things to the one transliterated - the non-finite check (Niko1221#879), the PCIe auto step and the adaptive thread's CPU pin - none of which changes the scheduling operations, so the transliteration stands. Nothing calls the scheduler yet.
… N2 and B0 (opt-in: STRATA_PIPELINE_UNDO_PER_STAGE=1, --pipeline-windows 3) --pipeline-windows 2 keeps two verifiers per stage (one per parity) and undoes a wrong guess on every front stage at once, after the last doomed window has drained. This adds the engine side of pl_sched: - The slot pool: NS verifiers per stage, a hand-off per boundary and a GDN snapshot per front stage each (slot 0 is `ver`, slot 1 `ver_b`). NS is 2 unless --pipeline-windows 3 (4), STRATA_PIPELINE_UNDO_PER_STAGE=1 (3) or STRATA_PIPELINE_SLOTS=k (at least k, at most 5) asks for more, capped so the process registers at most 16 verifiers. The room kept out of the expert caches grows per extra slot (160 MiB for a verifier, a GDN snapshot); what does not fit at start-up gives the pool back to two slots, said once. - The new loop, a separate branch before the pw-2 loop (which is not touched): PlIoFns turns each scheduler operation into the verifier / snapshot / drafter call it names (prestage, snapshot take/wait/restore, launch, the drafter's source-copy wait, commit, ple_prev, chain launch/extend/close, the adaptive tier's verdict and tick). The verdict emits as the pw-2 loop does, including Niko1221#879 (a window with non-finite logits emits nothing and ends the request, DONE nonfinite) and the PCIe auto step after each verdict; the blocking adaptive thread is pinned to the aux CPUs as there. - Modes per request: N2 (STRATA_PIPELINE_NEWLOOP=1 under STRATA_PIPELINE_DEBUG, testing) is the pw-2 loop operation for operation; B0 (STRATA_PIPELINE_UNDO_PER_STAGE=1) is depth 2 with the per-stage undo and 3 slots. --pipeline-windows 3 runs as B0 until depth 3 is wired; with fewer than three stages it is 2, said once. What a request cannot run in the new loop it runs in the pw-2 loop, said once per reason: fewer slots than the mode needs, the GDN snapshots not beside the window, the draft layer on the first stage (STRATA_MTP_STAGE0: the chain's source rows cross cards, which source_done does not cover), and a first window merged with the prompt's last tokens (STRATA_ADMIT_MERGE_FIRST). - Testing (STRATA_PIPELINE_DEBUG=1): STRATA_PIPELINE_SWITCH takes pw=3, ups=, newloop=, theta3=, force_miss_c=; STRATA_PIPELINE_CHECK_STATE=1 prints one FNV-1a line per stage of the state over the committed tokens at the end of a request, whichever loop decoded it; STRATA_DECODE_TIMING=1 prints a fourth line for the new loop; the watchdog prints the new loop's state and the extra slots' verifiers. - pl_sched.cpp is added to the strata target (CUDA/HIP and Metal). Without --pipeline-windows 3 and the new switches NS is 2 and the request runs the pw-2 loop: the same allocations, reservations, captures and log lines as before.
…ueued so far) An open chain (n_queue, STRATA_MTP_CHAIN_STOP) has chain_n() = the steps queued up to now, so a caller that reads chain_n() right after chain_launch as the chain's n_out sees n_queue and never extends or closes it. chain_cap() is n_out after the max_cells clamp (chain_extend's bound). mtp_chain_parity checks both at an open chain's launch and at the context's end. An accessor only: nothing changes for a caller that does not use it.
…ne state dump - launch_chain takes the chain's outputs from PlIo::chain_cap() (n_out after the clamp); chain_n() is read only once the chain is done (what it really has - fewer for a chain closed early). Reading chain_n() at launch left an open chain (mode B with chain_stop) never extended nor closed: chain_poll stayed 0 and the loop spun. The engine's PlIo forwards chain_cap to MtpDrafter::chain_cap. - The test's mock now answers chain_n() as MtpDrafter does (the outputs queued so far); with the old scheduler code it reports the deadlock in every mode-B exhaustive world that opens a chain. A quarter of the random worlds keep the old reading (Timing::chain_n_engine), which must work too. - Sched::check(Config): what begin() validates, for the engine to ask before a request is committed to the new loop; the slots error names the mode's need. - Sched::dump(): the scheduler's state in one line (windows with seq/slot/fr/fl/com, doomed, undo, held, chain, slot owners) for the engine's error path. Only the new loop (opt-in) calls the scheduler.
…ine; slot-pool captures shrink the pool - The mode of a request is chosen by the slots its depth needs (B 4, B0 3, N2 2) against the pool held; the draft layer on the first stage (STRATA_MTP_STAGE0) still keeps the request on the pw-2 loop. - The scheduler's configuration is built and Sched::check'ed before anything is queued; one it cannot run is said once and the request runs the pw-2 loop (pl_want), which is unchanged. - pl_prepare captures the two parities' slots first (an error as before), then the pool's other slots: a capture that fails there releases their verifiers, hand-offs and snapshots and gives the pool back to two slots, said once (STRATA_PIPELINE_POOL_FAIL= capture forces it, under STRATA_PIPELINE_DEBUG=1). The shrink line says how many MiB stay kept out of the expert caches until a restart. - An error in the new loop prints the scheduler's state (Sched::dump) before the engine ends; the watchdog's slot-pool entries are cleared when the new loop ends or the pool shrinks. All of it is on the new loop's side (--pipeline-windows 3, STRATA_PIPELINE_UNDO_PER_STAGE, STRATA_PIPELINE_SLOTS); with two slots pl_prepare captures what it captured before.
…chain captured, CHAIN_STOP/CHAIN_MAX, fallbacks (--pipeline-windows 3) - --pipeline-windows 3 now runs pl_sched's mode B (the third window, C, launched from the chain's outputs past B's bonus; the chain forced through both guessed windows; four slots) instead of B0. - pl_prepare captures the drafter's step graphs past 7 outputs (set_chain_max) for a pool of 4 slots or more, after the existing graphs; a capture that fails leaves the drafter at 7 and depth 3 runs the pw-2 loop, said once. - STRATA_MTP_CHAIN_MAX (7..11, default 11) and STRATA_MTP_CHAIN_STOP (default 1: a chain is queued only as far as its last new window needs) feed the scheduler. - Depth 3 on fewer than three stages (a switch file's pw=3) runs the pw-2 loop; Sched::check refuses it too. - VRAM: the slot pool's snapshots past the second are kept out of the later front stages' caches (pl_ns > 2 only). - Test hooks STRATA_PIPELINE_POOL_FAIL=init|snap|chain beside capture (under STRATA_PIPELINE_DEBUG=1); a note when 16 verifiers are registered; static_asserts tie pl_sched's bounds to the engine's. - pl_sched: n_force follows the drafter's cell clamp (B0/B); a defensive guard for a chain with no room for D (never fires, asserted by pl_sched_test); the mock refuses a chain longer than the drafter's captured graphs, and a quarter of the random worlds run mode B at chain_max 7..10. Without --pipeline-windows 3 (or STRATA_PIPELINE_SLOTS >= 4) the drafter's chain max stays max_t - 1, nothing more is captured or reserved, and the pw-2 loop decodes as before. Measured on the 0.1.41-based tree, 4x GP100, layer split 12,24,36: decode +3.4% against --pipeline-windows 2, the replies the same.
… emits - the session ends where the reply ends A verdict committed the whole accepted window (keep = a + 1) and the emission loop then cut its output at max_new or at a stop token, so the KV / GDN / indexer / PLE / MTP state - and Sched::end_, the session the next request matches - ran past the reply by however many rows the window had left, and the next request reused a prefix the reply never had. PlIo::emit_count is the engine's own emission rule (max_new, then a stop token): the number of the window's picks the reply keeps. Sched::verdict (and the two fallback loops in generate.cpp) then commit keep = min(T, emitted + 1) rows - the emitted picks plus row 0, which is the token before the window's first pick. A window with nothing to emit commits nothing (a = -1, end_ unchanged). pl_sched_test (f) is the case: long drafts with max_new landing inside a window, B0 and B; it counts the windows cut and re-runs the same worlds with the pre-fix rule, so the case catches the bug, and (d) still has N2 and the old loop line for line.
…pend on the window depth A window commits up to the row it happened to end on, so the same reply can leave a session one position apart, and the speculated windows leave different bytes behind in cells nothing reads back. The next turn's reply then can depend on how much speculation ran. With the switch: the commit ends at the reply's last token, the last page's unclaimed K/V cells and the indexer's spare row are reset, and the drafter's K/V is cleaned (modes 1-3, for diagnosis). The serial and pw-2 loops' trim from the commit before now takes this switch too, so a run without it commits exactly as main does; the new pipelined loop always trims. STRATA_DRAFT_KV_LOG and STRATA_STATE_DUMP are the diagnostics this was found with.
… speculation or on an earlier session [reused, first_needed) and the last 32 cells, so a resumed turn keeps its drafter window. Also fixes the clean's whole-page range (page_size times too far and too long).
ruibeikaa
force-pushed
the
pipeline-windows-3
branch
from
October 31, 2026 07:16
5569ade to
78e4e46
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and result
On a layer split of four stages,
--pipeline-windows 2keeps two speculation windows in flight. On four GP100 dies (two PH402 boards, compute capability 6.0) a window that starts from a verdict takes about 38 ms and a speculated window about 19 ms, and each stage spends much of a window waiting for the one ahead of it. The pipelined loop's draft chain also stops at 7 outputs, which caps what a third window could verify. Deeper speculation inside one window does not pay on this card (every extra verify row costs about 2.2-2.5 ms per stage in my measurements); a third window in flight adds no rows to any window. Today--pipeline-windowsis clamped to 2, so3is accepted and ignored.With
--pipeline-windows 3on three or more stages, three windows are in flight: the last stage verifies the oldest window while the earlier stages run two speculated ones, and a misprediction is undone stage by stage as soon as each stage is free of its doomed window. Measured on this branch on four GP100 dies with every expert resident in VRAM (no CPU or PCIe share), decode goes 57.81 -> 60.41 tok/s (+4.5%; three runs per arm; three decode prompts of 600 tokens), with every reply and every item of the session state identical. On my earlier stack, over eight decode prompts of more shapes, it was +3.5%. The gain needs a fast drafter: with the generic draft kernels it was +0.15%; with the Pascal Q8_0 draft head of #3751 (now on main) it pays (details below). Without--pipeline-windows 3the engine runs today's pw-2 loop with the same allocations, captures and log lines.Related discussion
none; the drafter it pays with is #3751, merged
Approach
Ten commits, in order: seven for the third window, three that make the session a request leaves exact. The branch head builds.
mtp: the draft chain grows to up to 11 outputs, step by step (MtpDrafter::kChainMax, graphs for steps 1..11), with a source-copy event the new loop waits on, andchain_cap()(the outputs a chain may queue). Only a caller ofset_chain_maxabove 7 gets the longer chain, so the output clamp at the context end is unchanged for everyone else. A parity program (mtp_chain_parity, configured only whenSTRATA_MTP_PARITY_BASEnames the commit before this one) checks the chain against the code it replaces.pl_sched(newsrc/program/pl_sched.{hpp,cpp}): the pipelined decode loop as a host-side state machine. It decides which window launches, waits, commits, is undone or is re-staged, and emits operations; it calls no GPU code.pl_sched_test(CPU only,EXCLUDE_FROM_ALL, built with--target pl_sched_test) runs it against a symbolic GPU: a recorded decode trace, random worlds, every small world with every ending, and today's pw-2 loop transcribed operation for operation.generate.cpp, the engine side: a slot pool of depth + 1 slots per stage (a verifier, a hand-off per boundary, a GDN snapshot per front stage), taken from a free list instead ofseq % 2, so a new window never writes a hand-off that a doomed window is still reading; the extra room is kept out of the expert caches at start-up.PlIoFnsturns each scheduler operation into the verifier / snapshot / drafter call it names. The new loop is a separate branch before today's pw-2 loop, which is not changed; it emits a verdict as the pw-2 loop does, including All-logits-NaN degeneration (one repeated token forever) still fires on 0.1.39b / current main — first poison traced to the MoE output rows: garbage bits, varying layer #879 (non-finite logits end the request), the PCIe auto step after each verdict, and the adaptive thread pinned to the aux CPUs and named.Modes:
--pipeline-windows 3runs depth 3 (4 slots);STRATA_PIPELINE_UNDO_PER_STAGE=1runs depth 2 with the per-stage undo (3 slots);STRATA_PIPELINE_SLOTS=ksizes the pool.STRATA_MTP_CHAIN_MAX(7..11, default 11) andSTRATA_MTP_CHAIN_STOP(default 1) tune depth 3's chain. Test hooks (the switch file's new keys,STRATA_PIPELINE_CHECK_STATE, forced misses) needSTRATA_PIPELINE_DEBUG=1.Fallbacks to today's loop, each said once: fewer than three stages (
--pipeline-windows 3is then 2), a pool that does not fit at start-up or whose captures fail (it shrinks back to two slots), the draft layer on the first stage, a first window merged with the prompt's tail, a scheduler configuration that failsSched::check, and a chain that could not be captured at 11 outputs (it stays at 7).Bit-identical by construction: only the oldest window reaches the last stage and only its output is emitted; a mispredicted window is undone on every stage before its replacement reaches that stage; the per-stage undo changes when a stage acts, never the order of operations in a stage's stream.
Default path: with
--pipeline-windows0..2 and none of the new switches the pool is two slots and every request runs the pw-2 loop, with the same reservations and captures. One change reaches it: the drafter's staging rows grow from 2T to max(2T, T + 11) (19 instead of 16 rows at T = 8), so later buffers in its arena move, its mapped output arrays hold 16 entries instead of T, and it keeps 12 step events instead of 8. The bytes read and written and the operations are the same (checked end to end below).Builds:
pl_sched.cppis added to thestratatarget in the main CMake (CUDA/HIP) and inmetal/CMakeLists.txt. The scheduler is host code with no backend-specific parts; SYCL has its owngenerate.cppand is not touched.verify.cppis not touched.A reply that stops inside a window: a verdict committed the whole accepted window and the emission loop then cut its output at
max_newor a stop token, so the state andSched::end_ran past the reply.PlIo::emit_countis the emission rule; the new loop'sSched::verdictcommits only the rows the reply keeps.pl_sched_test(f) is the case.STRATA_STATE_CLEAN(opt-in, default off;4is the one to use, 1-3 are for diagnosis): the session a request leaves does not depend on the window depth. The commit ends at the reply's last token, on the serial and pw-2 loops too; the last page's unclaimed K/V cells and the indexer's spare row are reset; the drafter cells that depend on the speculation or on an earlier session are cleaned, and a resumed turn keeps its drafter window.STRATA_DRAFT_KV_LOGandSTRATA_STATE_DUMPare its diagnostics. Without the switch the serial and pw-2 loops commit exactly as main does.#3751 is what makes depth 3 pay on GP100; it is on main behind
STRATA_MTP_Q8_SM60=1.Validation
Environment note: everything below was measured with every expert resident in VRAM on four GP100 dies (two PH402 SKU 200 boards, 32 GB HBM2 per die, 128 GB in all; the Swift 1.5 IQ3_S pack fits whole: no CPU share, no PCIe share, no adaptive tier). A single card, three stages, or experts in RAM were not measured. HIP, SYCL and Metal were not built.
Measured revisions. The first group under Passed is this branch (main 870aedd, engine 0.1.54, plus the ten commits) with only switches main has. The second group is the earlier measurement on my stack (main e88ee39, engine 0.1.45.3, plus my opt-in GP100 changes), kept because it covers more request shapes.
Rig: four GP100 dies (application clocks 715 MHz memory / 1050 MHz graphics, 140 W cap), Windows 11, driver 581.80 (TCC), CUDA 12.9, VS 2022. Build:
-DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=60, ggml at the pinned 3cf03257f, unpatched. Model: Swift 1.5 IQ3_S (GGUF, dense matrices in Q8_0) with an MTP drafter. For this branch:--serve --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.85 --mtp <drafter> --max-context 262144 --kv int8 --layer-split 12,24,36 --mtp-window 8192 --pipeline-windows 2|3 --short-read 24 --ple-io ram --conversation-cache-mib 16384 --conversation-cache-slots 4, environmentSTRATA_Q8_SM60=1 STRATA_MTP_Q8_SM60=1 STRATA_STATE_HASH=1and nothing else. For my stack, engine arguments, the same on every arm except--pipeline-windows:--serve --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.85 --mtp <drafter> --max-context 1048576 --kv int8 --layer-split 12,24,36 --mtp-window 8192 --pipeline-windows 2|3 --conversation-cache-mib 32768 --conversation-cache-slots 8 --rope-scaling yarn --rope-scale 4. Environment, the same on every arm:STRATA_Q8_SM60=1 STRATA_STAGE_TRIM=1 STRATA_PREFILL_PIPE=70 STRATA_MMQ_RESIDENT_SORT_NE=1 STRATA_DF_BRANCH=1 STRATA_FUSE_DECODE=1 STRATA_FUSE_WAIT=2 STRATA_SIDE_MIXER=2 STRATA_GR_UP75=1 STRATA_B6_MMVQ_ROWS=1 STRATA_ADAPT_LAG=2 STRATA_ADAPT_NOWAIT=1 STRATA_DECODE_TIMING=1, plus the stack'sSTRATA_EXPERT_NIBBLE=1(#3455's 4-bit slots),STRATA_SPLIT_COMMIT_ASYNC=1(an async split commit, not posted) andSTRATA_MTP_Q8_SM60=1(#3751).Passed, this branch:
STRATA_STATE_CLEAN=4: a three-turn conversation (about 3.5K tokens) and three decode prompts of 600 tokens out. Decode on those three, runs in the order 2, 3, 3, 2 and then one more of each: pw 2 57.88, 57.87, 57.67, pw 3 60.40, 60.45, 60.38 tok/s: +4.5%. Every reply hash and everySTRATA_STATE_HASHitem equal across the runs. (The last pair is the final commit; the commit before only moves the serial and pw-2 trim behind the switch, which is on in all of these.)STRATA_STATE_CLEAN: 3 of 3 replies equal, but 1 of 3 sessions (staleafter the read; after the third turnLand the state that follows it, on a reply stopped by its length). WithSTRATA_STATE_CLEAN=4: 3 of 3 replies and 3 of 3 sessions equal, every item. The switch costs no decode time here: 23.30 against 23.33 s at pw 2, 22.40 against 22.33 s at pw 3, for the 1,200 tokens.--pipeline-windows 2with none of its switches, the same six requests: every reply hash and everySTRATA_STATE_HASHitem equal.pl_sched_test(sm_60 build, Windows, CPU only, 370 s):PASS (0 failures). The recorded trace, random worlds and every small world with every ending, in modes N2 (today's loop), B0 and B; today's pw-2 loop against the scheduler's depth-2 mode: 802,830 worlds, every one the same line for line. New case (f), a stop atmax_newinside a window: 0 failures with the fix, and the pre-fix rule run on the same worlds leaves the state past the reply in 35,545 of 140,580 runs (35,291 in the other mode), so the case catches the bug.STRATA_EXPERIMENTAL_SM60=ON(strata pl_sched_test), clean.Passed, my earlier stack (not this revision):
STRATA_DECODE_TIMING): a speculated window 19.02 -> 16.58 ms; a window that starts from a verdict 38.81 -> 37.85 ms; at pw 3, windows held behind one hit 15.19 ms, behind two 17.86 ms.depth3 ... mode B slots 4; no fallback line in any run.STRATA_MTP_Q8_SM60=1, opt-in) #3751 (generic draft kernels; one run per arm, part of a longer ladder of runs): +0.15% (51.73 -> 51.81 tok/s). There the held windows took 19.29 / 20.82 ms instead of 15.19 / 17.86: the chain runs on the last die beside the last stage's verify and reads the 276 MiB draft head every step, which eats what the third window saves. The window counts and the number of windows held behind two hits were within 1% of the runs with [Engine] sm_60: the MTP drafter's draft head and projections on the Pascal Q8_0 GEMV too (STRATA_MTP_Q8_SM60=1, opt-in) #3751.STRATA_STATE_HASHitem (GDN, PLE, tail, pooled, KV) equal. Two kinds of difference, withoutSTRATA_STATE_CLEAN: the drafter's own K/V hash (mtp=) on the six requests with prompts of 26K tokens or more, and the hash of the KV cells past the committed end (stale) on three. Traced since: the drafter's last windows, written by chain steps and not caught up by a round, and cells left from an earlier session. WithSTRATA_STATE_CLEAN=4, on a larger set of 29 requests, content, tokens and every state item were equal, 29 of 29, where 5 of 29 had differed without it.STRATA_STATE_HASHitem equal,mtp=included.Failed: none observed.
Not run: HIP, SYCL and Metal builds (no such machine here; the HIP and Metal
stratatargets now compilepl_sched.cpp, which neither of my builds checks on those toolchains). The sm_86 build of this revision. Faster cards and three-stage splits (only four stages measured).mtp_chain_parityon this branch (it was run on my older tree). Gcc / clang builds ofpl_sched_test.--batch. The draft layer on the first stage and a first window merged with the prompt's tail (both fall back to today's loop, not exercised). Packs other than Swift 1.5 IQ3_S.Limits
--pipeline-windows 3is 2.STRATA_MTP_Q8_SM60=1, opt-in) #3751's draft head (+0.15% without it). I have no number for faster cards, where windows are shorter.generate.cpp, 1,100 in the scheduler and 3,500 in its test.STRATA_STATE_CLEANis off by default, so a plain--pipeline-windows 3can leave a session that differs from depth 2's in cells the next turn does not read. Replies were equal in every run here, with and without it.