Repository navigation
Conversation
…stay inside it The admission checks prompt + max_new <= max_context, but the spec loop opened every window at S_mtp rows, so a request that fits ended with "ran out of context" (rc 2) once p + S passed the end, and printed nothing. The window now takes only the rows the context still holds. MtpDrafter::draft wrote draft j at cell p + a + j for every step, past max_cells near the end, where the page table has no entry; it now drafts only the cells that exist. Far from the end nothing changes.
Owner
|
Reproduced on our box: a 4094-token prompt with --max-context 4096 --max-new 2 --spec 4 gives rc 2 'ran out of context' before the fix and rc 0 after. Ported with your authorship. Serve is not affected for plain requests (its admission stops earlier), the MTP cell cap still applies there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
strata generate --spec Sfails near the end of the context. The admission checksprompt + max_new <= max_context, but the loop still opens a full window there. So a request that fits ends with "ran out of context" (rc 2) and prints nothing. Near the end,MtpDrafter::draftcan also draft into cellmax_cells, past the page table.What changed
src/program/generate.cpp: a window takes only the rows that fit,min(spec, max_context - p). The lookup chain stays inside them.src/core/mtp.cpp: a draft round caps its steps atmax_cells - (p + a). Serve calls the same drafter, so it gets the cap too. The cap binds only at the end of the context.Extra Notes
fb58e0dbagainst this branch:--max-context 4096 --max-new 2 --spec 4: before, rc 2 and no output. After, rc 0 and 2 tokens.--max-new 12 --spec 5 --suffix-draft 0 --mtp, with a probe print: before, a round drafts into cell 4096 (max_cells4096), then rc 2. After, its last cell is 4095, and the run ends with rc 0 and 12 tokens.--max-new 64 --spec 4 --mtp, CPU share off: the same 64 tokens from both builds, twice each.