Repository navigation
Add opt-in accepted-token adaptive cache barriers - #1779
Closed
CC-David-CC wants to merge 6 commits into
Closed
CC-David-CC wants to merge 6 commits into
CC-David-CC wants to merge 6 commits into
Conversation
…er scratch race --pipeline-windows 2 now runs on any layer split with two or more stages. Every stage but the last is a front stage: the guessed window follows the verified one through them one card behind (an odd verifier and an odd hand-off on every boundary, a pair of GDN snapshots on every front stage, commits and the undo per stage), and only the last stage waits for the verdict. The pipelined prompt read (--pipeline-windows 1) and --adapt-async stay two-stage. HIP: the draft layer's per-row gr_read used the last stage's session scratch (ss.block.gr), which that stage's verifier also uses in lm_head_mix. A chain launched at a verdict and the next window's last stage can run on the card at once after a guess that held, and the window's head then read a clobbered mix. The draft layer now carves its own GrWorkspace. Measured on 4x RX 7900 XT (Flash-Next GSQ-RCO IQ3_S, split 12,24,36, every expert in VRAM, greedy, reasoning off): decode 59.4 -> 67.6 tok/s at 4K and 51.8 -> 65.6 at 32K. Greedy text identical to the serial loop on four prompts with STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0, also with every guess forced wrong. Debug knob STRATA_PIPELINE_SPEC_DEPTH. Two-stage path not re-run on two cards.
CC-David-CC
changed the title
Owner
|
Thanks. There is no speed claim and it is a large change; not taking it for the speed release. |
Contributor
Author
|
want me to close pr ? |
Owner
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Holds expert placement fixed within each accepted-token block, counts accepted routing only, and drains/verifies commits before blocking swaps at the boundary. Rejected draft and speculative rows do not count. Pipelining continues inside each block; the default path stays off.
Refreshed onto current main
61b3fb5d. A positiveSTRATA_ADAPT_BARRIER_TOKENSnow resolves blocking adaptation before three-stage pipeline admission. Previously current main disabled the pipeline before the later async override.Fresh three-P4 / ISTA IQ3_XXS / int8-KV validation: four 30-request matrices pass exact output/finish and accepted-routing/residency-hash comparisons, using intervals 64/17 and swap limits 80/160, with and without the CPU dispatch fix. Serial, pipeline, forced rollback, no-guess and serial-repeat arms follow the same ordered request history. Twelve current-main-versus-feature-off comparisons pass too. CUDA/HIP/SYCL builds pass; HIP/SYCL model execution is not claimed.
Exact comparisons explicitly set
STRATA_IQ_MT_MIN=1andSTRATA_ROUTE_TAIL_SKIP=0. Ordinary rank-7 routing can change target contributions with draft-window composition; two same-binary negative-control matrices remain published. Adaptive repeats are compared at matching request histories using #1813's explicit mode; original static audits remain visible. The barrier does not change ordinary tail-skip defaults.Requires serve mode, a layer split, no PCIe miss sharing and positive adaptive settings. Barrier value unset/0 keeps the existing path. The interval replaces the window-based cadence and forces blocking adaptation even if async was requested. Unsupported remote/helper modes are refused. No fresh universal speedup or arbitrary numerical/quality guarantee is claimed.
Fresh full outputs, manifests, profile, hashes, coverage, build evidence and limits.