Repository navigation
prefill: STRATA_PREFILL_KEEP (128) is the slots a loan leaves the cache - #2833
Closed
gopinath87607 wants to merge 1 commit into
Closed
gopinath87607 wants to merge 1 commit into
gopinath87607 wants to merge 1 commit into
Conversation
gopinath87607
force-pushed
the
prefill-keep
branch
2 times, most recently
from
October 17, 2026 02:47
426621d to
1a34d1d
Compare
From the Maya checkout (`project-maya`), local commit `71c1ca7` (2026-10-15): `STRATA_GLM_PREFILL_KEEP` there - the slots a layer of the pool keeps while the pool lends the prompt path - and that name is accepted here too, so a config written for the Maya checkout runs unchanged. There, the floor became a knob because the floor is what binds on the small cards: with the auto lend's percentage cap above it, the number of slots a lend must leave is what decides how big a chunk a part can lend, and lowering it from 13 to 11 (with STRATA_GLM_RESERVE_MB=400) moved that rig's binding part from 6144 to 8192-token chunks on a 99,758-token conversation (its 49,152-token piece 61,762 -> 57,507 ms; the whole five-piece read 161,291 -> 158,938 ms, with a 16,384-token piece that went from three chunks to two losing 3%, 28,716 -> 29,651 ms, since the parts run a chunk as a chain). This engine has the same number in the same role, hard-coded as 128 in twelve places: the chunk scan's old rule, its ring budget and its bisection (plan_lend), the per-stage lend regions, the serve scan's fits_one and ring_room, serve's own loan, the split-autoplan gate, the stage-sizing probe, the smallest cache a start accepts (#496), the vram-elastic reserve check and the vram-elastic relend. All of them read prefill_keep_slots() now, so the number lives once. The clamp is up, not down, as there: a lend has to leave the elastic K/V a region to grow into (kvg_start takes the lend's first slot as its top and turns the K/V off when that region is no larger than its own floor), so a smaller value is clamped to that floor - STRATA_KV_GROW_FLOOR, 128 by default - and the start says so rather than obeying it. Lower both together to lend more of the cache. On this rig the floor is the binding test on the Q8_0 four-way's 691- and 937-slot stages already: it leaves 563 and 809 slots where the 90% cap would leave 621 and 843. Unset, every one of those twelve numbers is 128 as before and nothing prints; what a smaller value buys on a given rig is its owner's A/B - the chunk the start prints and the prompt's ms/token against the same run without it. Co-Authored-By: Claude Code <noreply@anthropic.com>
gopinath87607
force-pushed
the
prefill-keep
branch
from
October 17, 2026 08:14
1a34d1d to
6001911
Compare
Contributor
Author
|
Closed because the source fork was deleted. This PR was opened from the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
STRATA_PREFILL_KEEP=N(128 by default): the expert-cache slots a loan must leave resident while a prompt chunkborrows the cache's tail. On a small cache it is this number, not the auto lend's 90% cap, that picks the chunk. On
the Q8_0 four-way's 691- and 937-slot stages it leaves 563 and 809 slots where the cap would leave 621 and 843.
STRATA_GLM_PREFILL_KEEPis accepted as the same knob.The clamp is up, not down: a lend has to leave the elastic K/V a region to grow into (its floor is
STRATA_KV_GROW_FLOOR, 128 by default), so a smallerNis clamped to that floor and the start says so — lower bothtogether to lend more of the cache and let the chunk grow at the K/V's expense. A larger
Nkeeps more expertsresident for the prompt's own tokens and buys a smaller chunk.
The number is the same one the auto scan, the per-stage lend regions, the serve loan, the smallest cache a start
accepts and the vram-elastic reserve used before; this change routes all of them through one reader of the variable.
From the Maya checkout (
project-maya), local commit71c1ca7(2026-10-15).What it leaves alone
Unset, every use is 128, exactly as before — this is by construction, not a measurement, and this PR's default run
is byte-identical on that path. With the variable set to 128 (its default) the numbers are also unchanged.
Measurements
That commit lowered its pool's floor from 13 to 11 slots a layer (
STRATA_GLM_RESERVE_MB=400) on a four-way split andmeasured, on a 99,758-token conversation, the binding part reach an 8192-token chunk: a 49,152-token piece
61,762 → 57,507 ms (-6.9%) and a whole five-piece read 161,291 → 158,938 ms, on the live split (5060/5060/3060/3060)
— where it was not a free win either (a 16,384-token piece that went from three chunks to two lost 3%,
28,716 → 29,651 ms, since the parts run a chunk as a chain and there is less left to overlap). No number is claimed
for our rig: the four-way Q8_0 runs so far used 128, which is this default.
Checks
src/program/generate.cppcompiles clean against this branch's headers with the build's own flags(
-fsyntax-only; one pre-existing unused-variable warning in unrelated code). The development branch that alsocarries the change built fully with ninja (CUDA 13.2).
generate.cppcarries no per-backend code, so bothbuilds compile the same change.
🤖 Generated with Claude Code