[Engine] Slot placement policy for STRATA_SLOT_CACHE_AFFINITY (2/3, depends on #2527): protection by history length, retention scores, value decay, no wait by andyhrc · Pull Request #2528 · Niko1221/Strata

andyhrc · 2026-10-15T02:05:54Z

Problem and result

#2527 lets the server place batch requests by what the slots hold, behind the interface serve/slot_policy.py. This PR
is the policy behind that interface. It decides which idle slot a request runs in, which history to give up, how long a
history is protected, and where the solo path's conversation is parked.

Plain rules cannot tell which history to give up when a new conversation needs a slot and no free slot is without value.
The one used longest ago may be a 90K-token conversation whose user is reading the answer; the newest may be a side
request's 300 tokens. This policy weighs length and age. Every threshold is a parameter in the server config's "env",
so most other behaviours are a matter of setting them.

Result on our machine (3 x RTX A4000 16 GB, Qwen3.8-Flash-Next IQ3_S, --batch 6 in 3 groups):

A parked 37K-token conversation's next turn after a side request reused 37,352 tokens and took about 1 s, instead of
14 s.
Two conversations taking turns past their protection keep their own slots (table under Validation).
Claude Code's safety classifier queries replace the previous query's slot instead of filling the slots with older
versions of the same transcript.

Related discussion

There is no issue. This PR was opened on 2026-10-14 and closed in the 0.2.0-beta tracker reset; it is rebased on current
main here. This is part 2 of 3 and depends on #2527; the branch includes #2527's commits, so review the policy
commits after those. #2843 (part 3) builds on it. #2589 is the small counterpart for the path this PR does not
change: --batch without pipelined groups. If you prefer other rules, keep #2527 and replace serve/slot_policy.py.
I can open an issue for the scope first if you prefer.

Approach

Variable	Default	What it sets
`STRATA_SLOT_TTL_MAX_S`	120	the protection at 262144 tokens; the whole curve scales with it
`STRATA_SLOT_EXPIRED_KEEP_SCORE`	0.01	an expired history whose current decayed value is at least this goes last
`STRATA_SLOT_IDLE_VALUE_ZERO_S`	1800	the total idle age, including protection, at which value reaches zero; 0 disables decay
`STRATA_SLOT_PARK_VALUE_RATIO`	2	a park takes an expired history worth keeping only when the main session's is worth this many times more; 0 means never
`STRATA_SLOT_PARK_MIN_VALUE`	0.01	a solo conversation whose value at its idle age is below this is not parked
`STRATA_SLOT_PARK_KEPT_SHARE`	0.98	a slot holding this share of the solo conversation already keeps it, so no park
`STRATA_SLOT_SIBLING_SHARE`	0.98	an idle history that shares this share of its own length with the prompt, but not its start, is an older version of the same conversation and is replaced in place, protected or not; 0 is off

Protection by length (piecewise linear):

Tokens 500 2K 8K 32K 50K 100K 262144

Protection 0.1 s 2 s 10 s 45 s 65 s 90 s 120 s
Retention = (n / 50000)^0.7 x (n / (n + 2000))^2 x (0.25 + 0.75 x freshness), where freshness = 1 - age / TTL,
and 0 once expired. The overwrite score is 1 / (1 + retention). A fresh 50K history is worth about 1, a 500-token one
0.002.
Value after expiry = (n / 50000)^0.7 x (n / (n + 2000))^2 x 0.25 x (1 - x^2), where x = clamp((age - TTL) /
(idle_value_zero_s - TTL), 0, 1).
- Age counts from the end of the last actual use and includes the protection period.
- With the default, value is zero at 30 minutes for every length.
- Allocation and parking use the same value. A zero-value history still matches until it is overwritten: the score
  controls replacement and parking, never cache validity. No timer deletes anything.
Blank slots count as expired and worthless: empty, invalid, or holding a superseded copy.
A continued conversation runs in its own slot or in a blank slot of a less loaded group, never in a slot that
holds another conversation. Near matches count as sources: within 2% of the best, one second of the prompt rate, or
4096 tokens.
An older version of the same conversation (STRATA_SLOT_SIBLING_SHARE) is replaced in place. Claude Code's safety
classifier rewrites the end of its transcript every query. Under the rules above, the previous query's slot would still
be protected, so each query would take another slot.
A new conversation takes, in order:
1. what is not worth keeping: a blank slot, or an expired history below STRATA_SLOT_EXPIRED_KEEP_SCORE
2. by group load
3. the lowest value, with a blank slot first at an equal value
A park uses the same order (reclaim_slot).
No wait. When every history is protected, the policy overwrites the least loss at once; losses within 10% are
ranked by group load. An earlier version waited up to 5 s first. With more requests than slots, each wait ran its full
5 s and ended in the same overwrite.
Park. No park when:
- the conversation's value at its idle age is below STRATA_SLOT_PARK_MIN_VALUE, or
- a slot already holds a valid copy of STRATA_SLOT_PARK_KEPT_SHARE of it.
Otherwise the destination is, in order:
1. the conversation's own older copy
2. an older version of it, the most shared first
3. the new-conversation order above, where a history worth keeping counts only when the main session's is worth
  STRATA_SLOT_PARK_VALUE_RATIO times as much
A park never goes into another protected slot. A park keeps the conversation's last use, so it cannot renew its
protection.

Rebase on main (5e8a4cca, v0.2.7): no conflict; the policy is unchanged.

Validation

Passed:

Revision. Branch pr/slot-policy at df1d87bd, on main 5e8a4cca (v0.2.7). The serve tests were run on this head; the GPU runs name their own revision. Ubuntu, AMD EPYC 7543,
3 x RTX A4000 16 GB.
Serve suite. python -m unittest discover -s serve -t . on this branch: 1291 tests OK (38 skipped). Its cases are mainly in
serve/test_slot_alloc.py (the policy's own cases, including the production scenarios below), serve/test_parallel.py
and serve/test_cache_trace.py.
The CI job's form ("Minimal serving runtime": a venv with only requirements.txt, then tools/minimal_server_smoke.py and python -m unittest discover -s serve -p 'test_*.py'): smoke OK, 1291 tests OK (38 skipped). The previous head failed there on serve/test_cache_trace.py's relative imports (fine with -t .); the last commit of [Engine] Keep conversations in pipelined batch slots (1/3): active-row windows, park=, cache-aware placement behind a policy interface #2527 fixes them.
GPU run. Run on the tip of the stack ([Engine] Slot checkpoints for clients that rewrite the end of their history (3/3, depends on #2527 and #2528): STRATA_SLOT_CHECKPOINTS, STRATA_BRANCH_CHECKPOINT, BCHECKS #2843's branch at 58eef307 on v0.2.1.1, with its own engine and serve/server.py; repeated with our v0.2.7 production build, below).
Setup: Qwen3.8-Flash-Next IQ3_S, 200K context, --batch 6 in 3 groups, the deployed settings
(STRATA_SLOT_CACHE_AFFINITY=1, STRATA_SLOT_PARK_MIN_VALUE=0.1, the other thresholds at their defaults). Results:
the agent conversation and the classifier chain kept their own slots through six alternations, alone and with a client decoding beside them. Every agent turn reused its 27,341 tokens, and every classifier query from the third on reused 43,340 tokens ([Engine] Slot checkpoints for clients that rewrite the end of their history (3/3, depends on #2527 and #2528): STRATA_SLOT_CHECKPOINTS, STRATA_BRANCH_CHECKPOINT, BCHECKS #2843's part). With a thinking chat beside four other conversations, the reuse was the same, token for token, as on 0.1.45.
On v0.2.7. Our production build of this stack (plus a few local-only diagnostics, on 5e8a4cca, its own engine and serve/server.py) with the deployed switches repeated the run on 2026-10-28. S: 13.8, 14.5, 1.21, 1.40, 1.65, 1.80 s; B: 0.54, 0.97, 1.09, 1.30, 1.61, 1.75 s; the thinking chat's next turn reused 19,176 tokens in 0.58 s, and the other conversations reused the same tokens as on v0.2.1.3. Throughput (tools/serve_load.py, temperature 0, one run each on two fresh engines, on the build before [Engine] Slot checkpoints for clients that rewrite the end of their history (3/3, depends on #2527 and #2528): STRATA_SLOT_CHECKPOINTS, STRATA_BRANCH_CHECKPOINT, BCHECKS #2843's last commit, which changes only a give-back): 67.6 / 63.0 tok/s with one client (1,024 tokens), 190.7 / 191.5 tok/s total with 6 clients x 2,048 tokens (v0.2.2: 64.3 / 193.7).

Earlier GPU checks on 0.1.44.1 (same machine), not repeated on 0.2.x. Conversation A (37K tokens) was parked;
two clients generated 1,500-token answers in a loop; conversations B (9K) and C (7.9K) took turns about every 30 s,
past their protection. Tokens reused on turns 1-3:

Ordering of expired histories	B	C
by group load first (an earlier version)	0 every turn	0 every turn
+ one still worth keeping goes last (this PR)	all but the ~20 new	0 every turn
+ wait for a cheaper slot about to leave its protection (removed)	all but the ~20 new	all but the ~20 new

This PR accepts the second row: a protected history is not overwritten while an expired one can be. With
STRATA_SLOT_IDLE_VALUE_ZERO_S=150:

a park kept the last use (slot age 27.0 s, not 0)
a main history idle for 160 s was not parked (park_skip, main_low_value)
with six slots worth keeping (0.056 each) and main 12.9K: no park
main 42.1K (0.202) was parked into the lowest-value slot, and its next turn reused 42,092 of 42,119 tokens in 0.7 s

Equivalence check. place() before and after the shared reclaim order chose the same slot in 200,000 random slot
tables.
Production. It has run in production since 2026-10-15.

Failed:

None observed. (The previous revision's note about serve.test_responses.NoLeakedSampler failing on main here no longer applies: the whole suite passes.)

Not run: no throughput or cache-hit measurement over a working day is claimed for the 30-minute default.

Limits

The policy is for pipelined groups (see [Engine] serve: opt-in STRATA_SLOT_EXPIRE_S keeps a long conversation's batch slot from a short side request (next turn 12.8 -> 0.3 s) #2589 for the other path).
The thresholds were chosen for two people with agent clients on 6 slots. They are parameters, not tuned constants.
With a server whose engine lacks slot_padding_safe=1, starting a row above a protected idle history costs that
history's retention too.

🤖 Generated with Claude Code

andyhrc · 2026-10-15T10:44:03Z

Converting this to a draft while we test a revision of the policy on our production machine. Not pushed yet: the
expired-history value now decays with idle time, and a park uses the same reclaim order as a new conversation. I'll
push it and mark this ready again with the results. #2527 goes to draft with it.

🤖 Generated with Claude Code

andyhrc · 2026-10-15T14:48:37Z

Ready again. Pushed since the draft (the description is updated):

the expired-history value now falls as 1 - x² to zero at STRATA_SLOT_IDLE_VALUE_ZERO_S (1800 s by default) of
total idle time, replacing the half-life; a park keeps the conversation's last use instead of renewing it, and a
main-session conversation already worth less than the keep score is not parked;
place() and the park share one reclaim order (reclaim_slot); place() chooses exactly as before (200,000
random slot tables compared). A park may take an expired history worth keeping when the main session's is worth
STRATA_SLOT_PARK_VALUE_RATIO (2) times as much;
the "already kept" check counts only a valid copy (defensive).

GPU checks of each rule are in the description's Results. It has run in production since 2026-10-15.

🤖 Generated with Claude Code

Niko1221 · 2026-10-19T18:00:26Z

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

andyhrc · 2026-10-20T15:49:14Z

/reopen

github-actions · 2026-10-20T15:49:26Z

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.

…s its active slots only, so an idle slot keeps its conversation (opt-in) A pipelined group's batch window spans its slots up to its last active one, and an idle slot below an active one runs as a pad row that writes its state. A conversation that finished in such a slot is gone before its next turn. With STRATA_PIPELINE_ACTIVE_ROWS=1 (a layer split with --batch and more than one group) the window lists only the active slots (strata::program::pipeline_rows::pack, Verifier::batch_launch_rows on CUDA and SYCL): there is no pad row, an idle slot keeps its state, and INFO says slot_padding_safe=1. BT/BDONE name the packed rows' own slots. After a prompt's restore the engine says CACHE_SOURCE <slot or -1> <tokens>: where the reused prefix came from. Unset, every path is the one before. Measured on 3 x RTX A4000 16 GB (layer split, --batch 6 in 3 groups, IQ3_S, 200K context), the same binary with the variable off / on, tools/serve_load.py 6 clients x 2048 tokens, temperature 0, one run each: 177.4 / 173.5 tok/s total, first token p50 2.4 / 3.7 s; one client 67.3 / 66.7 tok/s. tests/core/pipeline_rows_test.cpp (registered in CMake): every active mask of 2, 4 and 8 slots, groups of 1, 2 and 4. Built on CUDA (sm_86, Linux); the SYCL change was not built here (no oneAPI on this machine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…efore a different request replaces it (with STRATA_PIPELINE_ACTIVE_ROWS=1) A request alone runs on the solo path, whose state is the main session. When the next request is another conversation - an agent client's short side request between two turns of a long one - the main session is overwritten and the long conversation's next turn reads its whole history again. New request key park=B on GEN/BGEN (INFO slot_park=1, only with STRATA_PIPELINE_ACTIVE_ROWS=1): when the prompt continues less than half of what the main session holds, the engine first copies the main session's conversation (copy_to_slot, with its last checkpoint) into idle slot B - never the slot it restores from or is admitted to - and says PARKED B <tokens> (0: the copy failed and the slot holds nothing usable). A park= it does not use is logged. Without the key nothing changes. Measured on 3 x RTX A4000 (0.1.43.1 and 0.1.44.1): parking 36-37K tokens took 414-451 ms in five runs; the next turn of a 37,359-token conversation after a side request reused 37,352 tokens (0 without park=) with the same greedy output as before the side request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e slots hold, with the placement policy as an interface (serve/slot_policy.py) With pipelined groups the server placed requests by the groups alone, because the engine kept no pipelined slot as a cache. On a 3 x RTX A4000 server used by two people with Claude Code and Codex, six days of engine log (3,027 prompts of 8K tokens or more) show 16.4M prompt tokens read again because the previous turn's slot had been taken by another conversation, and 15.0M because the previous turn ran on the solo path and the next request replaced it. With STRATA_SLOT_CACHE_AFFINITY=1 and pipelined groups, the server: - asks serve/slot_policy.place() for the slot, with each slot's tokens, last use (from BDONE), validity, the spreading order, the group size, the observed prompt rate and the engine's slot_padding_safe; without that engine capability it tracks from BADM/BT/BDONE which idle slot a later window of its group has padded over; - waits when the policy answers "protected": once per request, at most slot_policy.PROTECTION_WAIT_S, the control lines given back, re-checking every 0.1 s; a cancel leaves every slot as it was; - tracks what the main (solo) session holds and, with INFO slot_park=1, asks slot_policy.park_destination() for a slot and sends park=B when a request is another conversation of at least slot_policy.PARK_MIN_TOKENS; PARKED updates that slot's state, an unused claim is released. /metrics gains live.cache_events (the last 96: allocation, admission with the tokens reused/read, resume and cache source, park, protection waits, expiries), request keys and client addresses (the socket's; X-Forwarded-For is not used) in live and requests[], live.solo_*, and per-slot cache fields; with STRATA_CACHE_TRACE=1 a per-request snapshot of how long a start each slot shared with the prompt (serve/cache_trace.py, lengths only). These /metrics fields are present with the variables off too (additive); generation is unchanged. serve/slot_policy.py is the interface here: place, park_destination, cache_state, superseded_slots, reliable_flags, configure/settings, PROTECTION_WAIT_S, PARK_MIN_TOKENS. Which history to give up is the policy's choice - our policy is the next commit, and another one can replace it without touching the server. With STRATA_SLOT_CACHE_AFFINITY=1 or STRATA_CACHE_TRACE=1 and no policy, NotImplementedError. Tests: serve/test_slot_alloc.py, serve/test_parallel.py, serve/test_cache_trace.py (those that need a policy come with it). Full `python -m unittest discover -s serve -t .` passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… own placement plan slot_last_plan is shared by every request; the admission read it after giving slot_cv back, so a request picking a slot meanwhile could put its own plan there. Seen as a match hint larger than the prompt (126,251 for a 125,512-token prompt). The plan is now kept per request, under slot_cv, when the slot is picked (an admission, or a slot reserved for a read that gives way). Display only: placement unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

live.solo_request_key, solo_cache_source_slot and solo_cache_reused_tokens kept the last solo request's values until the next solo request (only solo_peer_ip was cleared), so a monitor reading them while other requests ran or waited for a slot showed a request that had long ended. They are now cleared with solo_peer_ip, in both places the solo leg ends. Display only: placement unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The admission waited once per request, up to slot_policy.PROTECTION_WAIT_S, when the policy answered "protected" (every idle history still protected), and then asked again allowing an eviction. In production that wait never paid: with more requests than slots, a slot that had just finished held a protected history, and each of 7 waits in 15 minutes (2026-10-15, 3 x RTX A4000, 6 slots, 8 clients) ran the full 5 s and ended in the same eviction, the slot idle meanwhile. The history that the eviction takes is the one with the lowest retention - the least likely to be continued within the wait anyway. Removed: _admission_slot, the per-request protected_since, pick_slot()/place()'s allow_eviction, the protection_wait/protection_end events, PROTECTION_WAIT_S and live.slot_policy.protect_wait_s. A request still waits for a slot only when every slot is busy, re-checked on each release and every 1.0 s as before this change. A read that gives way (Niko1221#656) reserves its slot with the same placement as an admission. The request keeps its own placement plan (choose_slot()). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The server asked the policy for a park destination only when the main session held at least slot_policy.PARK_MIN_TOKENS (8192) tokens. That was a fixed length cut-off beside the policy's own judgement of what a conversation is worth: with the policy's value threshold (STRATA_SLOT_PARK_MIN_VALUE, the next part) it either decided nothing (0.1 already skips conversations under about 18K tokens) or overrode the score. The server now asks whenever the request is another conversation (the first half of what the main session holds differs); whether it is worth a copy is the policy's answer. PARK_MIN_TOKENS and live.slot_policy.park_min_tokens are gone from the interface. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… last turn, not by half of it A park copied the main session's conversation only when the new prompt continued less than half of it (engine: resume * 2 < live; server: the first half differs). Two chats of the same client share the system prompt and the tool definitions - tens of thousands of tokens with agent clients - so a 30K conversation that has 20K of them in common with the next chat counted as the same conversation, and the next chat overwrote its last 10K unparked. The same conversation shares what the main session holds up to its last turn: the last <|im_start|>, where the latest reply begins (the place of the turn checkpoint). A next turn sent back without that reply's thinking, or the reply regenerated, still shares it; a prompt that differs before it is another conversation (or an earlier message edited) and the main session's conversation is parked. The engine checks this itself (--turn-token, default <|im_start|>), says the token in INFO (turn_token=, beside slot_park=1), and the server's claim uses the same test; an engine that does not say it counts any difference. "park=B not used" names the turn and the decision. Checked: the engine compiles (-fsyntax-only, the production build's flags); serve/test_slot_alloc.py covers the shared-system-prompt case, a next turn without the thinking, a regenerated reply and an engine without turn_token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ce and placement hint A request moved between the solo path and a slot is admitted again with its prompt and its reply so far; each admission set the record's cache_match_hint and cache_source_slot, so the record paired the first read's prompt and reuse (Niko1221#1869) with the last leg's hint - more tokens "resumable" than the prompt had (a 207K prompt showed a 214K hint). The values of that first read are kept, as reused0 is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…n serve/ is discovered on its own The "Minimal serving runtime" workflow runs `unittest discover -s serve -p test_*.py` (no top-level directory), where the module has no parent package and its relative imports failed: "attempted relative import with no known parent package". Absolute imports, as the other serve tests use, load either way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ntion scores, one bounded wait, park destinations The policy behind the interface of the previous commit (serve/slot_policy.py). Every threshold is a parameter in the server config's "env": - protection by length: 0.1 s at 500 tokens, 2 s at 2K, 10 s at 8K, 45 s at 32K, 65 s at 50K, 90 s at 100K, 120 s at 262144 (STRATA_SLOT_TTL_MAX_S scales the curve); - retention = (n/50000)^0.7 x (n/(n+2000))^2 x (0.25 + 0.75 x freshness), 0 once expired; overwrite score = 1/(1 + retention); - blank slots (empty, invalid, or a superseded copy) count as expired and worthless; an expired history still matches until it is overwritten; - a continued conversation runs in its slot or a blank slot of a less loaded group - never in a slot that holds another conversation (an earlier version moved two idle Claude Code conversations into each other's slot on every turn; each turn read 60-100K tokens from token 0); near matches (2% of the best, one second of the prompt rate, 4096 tokens) count as sources; - a new conversation takes what is not worth keeping first (blank, or an expired history scoring below STRATA_SLOT_EXPIRED_KEEP_SCORE, default 0.01 - about 2.5K tokens), then by group load, blank before history, the lowest score; - every history protected: one wait of STRATA_SLOT_PROTECT_WAIT_S (default 5 s), then the least loss (losses within 10% by group load); the same wait when only histories worth keeping are left and a cheaper one leaves its protection within it; - park: the conversation's own older copy, else a blank slot of the least loaded group, never another conversation's slot; not under STRATA_SLOT_PARK_MIN_TOKENS (default 8192) or when a slot holds 90% of it already; - on an engine without slot_padding_safe=1, starting a row above a protected idle history costs its retention too. Measured on 3 x RTX A4000 (0.1.44.1, --batch 6 in 3 groups, STRATA_PIPELINE_ACTIVE_ROWS=1): conversation A (37K) parked, two clients generating 1,500-token answers in a loop, conversations B (9K) and C (7.9K) taking turns about every 30 s, past their protection (about 11 and 10 s). Tokens reused on turns 1-3: - expired histories by group load first (an earlier ordering): B 0, C 0 on every turn; - + an expired history worth keeping goes last: B all but the ~20 new tokens, C 0; - + wait for a cheaper slot about to leave its protection (this policy): B and C all but the ~20 new tokens. Park: a 37,359-token conversation's next turn after a side request reused 37,352 tokens and took about 1 s (0 reused, 14 s without), with the same greedy output. Tests: the policy's own in serve/test_slot_alloc.py, serve/test_parallel.py (the wait is bounded, wakes when a protection ends, a cancel leaves every cache in place) and serve/test_cache_trace.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…_EXPIRED_KEEP_SCORE Parking kept the main session's conversation only in its own older copy or a blank slot, so with every slot holding another conversation it was dropped even when one of them was a short expired side chat. The parked conversation is at least STRATA_SLOT_PARK_MIN_TOKENS (8192) long; an expired history below STRATA_SLOT_EXPIRED_KEEP_SCORE (about 2,600 tokens at the default 0.01) is worth less, and place() already overwrites those first. Such a slot is now the last choice for a park (least loaded group, then the lowest score); a history worth keeping is still never taken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Keep the value at expiry, then halve it every STRATA_SLOT_EXPIRED_HALF_LIFE_S seconds (default 120; 0 disables decay). Allocation and main-session parking use the same age-aware value, so a long unused history can eventually yield below the keep threshold. Protection and prefix matching are unchanged. Verify expiry continuity and half-life, long-history reclamation and refresh, matching after decay, bounded waits, busy/excluded park targets and configuration. 97 policy/cache/parallel CPU and mock-engine tests pass; no GPU performance claim.

Use value_at_expiry * (1 - x*x), where x runs from protection expiry to a total idle age of STRATA_SLOT_IDLE_VALUE_ZERO_S (default 1800 seconds, including protection; 0 disables decay). Loss accelerates until value reaches zero. Protection takes precedence if a configured cutoff is shorter than TTL. Record the main session last actual use, check its current value before claiming a park, and copy its timestamp into the parked slot. Score zero does not invalidate history or exclude prefix matches. Existing cache affinity remains opt-in; no native code changed. Verified 103 CPU/mock policy/cache/parallel tests plus a subsequently added main-continuation regression (104 distinct tests). No GPU benchmark, model restart or live activation. Local review only; not pushed.

…already kept park_destination skips the park when a slot holds 90% of the main session's conversation; it now also requires that copy to be reliable, as the own-older-copy choice below it already does. A defensive check: on the current code paths a slot that is marked invalid holds nothing (an admission, a non-finite result and a failed park clear it), and a park needs slot_padding_safe, so it is not triggered today. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… keeping at STRATA_SLOT_PARK_VALUE_RATIO place() and park_destination() now share one reclaim order (reclaim_slot) over idle expired slots: the slots not worth keeping (blank, or below STRATA_SLOT_EXPIRED_KEEP_SCORE) before histories worth keeping; then the group's load; then the lowest current value, a blank slot first at an equal value; then the slot order. place() is unchanged: its key had the blank-slot test before the value, and a blank slot's value is 0, the lowest (200,000 random placements compared, none differ). A park changes in two ways: - a blank slot no longer comes first whatever its group's load: a slot not worth keeping in a less loaded group is taken first, as place() does; - an expired history worth keeping may take the park when the main session's conversation is worth at least STRATA_SLOT_PARK_VALUE_RATIO (default 2) times its current value; 0 keeps every such history. A park still gives up rather than take a protected slot or one below the ratio. The server passes the main session's current value (the one its pre-park check already computes). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…for a park A park copied the main session's conversation into a slot whenever its value at its idle age was at least STRATA_SLOT_EXPIRED_KEEP_SCORE, the threshold that also decides which expired slot histories are worth keeping. With the default 0.01 every conversation of 8K tokens or more was parked until about 26-30 minutes idle, and the copy is not free: 24 production parks of 10K-170K tokens took 0.17-2.56 s each, on the control lines, while no batch window runs. STRATA_SLOT_PARK_MIN_VALUE (default 0.01, so nothing changes unless it is set) is now the park's own threshold; STRATA_SLOT_EXPIRED_KEEP_SCORE keeps deciding slot reclaim only. live.slot_policy shows it. Tests: a 50K conversation idle 25 minutes (value ~0.07) is not parked at 0.1 while slot reclaim keeps 0.01, and is parked idle 20 minutes (~0.13); STRATA_SLOT_PROTECT_WAIT_S=0 admits at once over protected histories. docs/BATCHING.md: the park bullet described an earlier version; it now lists the park's thresholds and destination order, and the value decay. Full serve suite: one failure, test_responses NoLeakedSampler, also without this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

place() answered "protected" in two cases, and the admission waited once per request, up to STRATA_SLOT_PROTECT_WAIT_S (5 s): - every idle history still protected. The wait ended in the overwrite it delayed: the one that loses least is the one with the lowest retention, the least likely to be continued within 5 s. In production (2026-10-15, 6 slots, 8 clients) all 7 such waits ran the full 5 s while requests were queued and a slot stood idle; - only expired histories worth keeping left, and a cheaper protected one ending its protection within the wait (a side request that had just finished). This one kept conversation C in the ABC run; without it C is overwritten while the side request's 1.5K history is still protected - a case of the protection rules, accepted. Both are removed with the mechanism's wait (previous commit): a free slot is always placed - a match, else a reclaim of an expired or blank slot, else the overwrite that loses least. STRATA_SLOT_PROTECT_WAIT_S and live.slot_policy.protect_wait_s are gone. Tests of the waits are removed; the ABC case is kept as a test of the behaviour now. docs/BATCHING.md: the "one bounded wait" rule becomes "no wait", and the ABC table marks the waiting row as removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ld decides alone A park skipped every main-session conversation under STRATA_SLOT_PARK_MIN_TOKENS (8192 tokens) before its value was looked at: a fixed length cut-off beside the score. At the default STRATA_SLOT_PARK_MIN_VALUE (0.01) the score alone skips conversations worth less than about 2.6K tokens at expiry; at 0.1 (production) about 18K, so the cut-off either decided nothing or overrode the score. The constant, its variable and live.slot_policy.park_min_tokens are removed (the server's check went with the mechanism's previous commit). A 5000-token conversation is now parked at the default threshold and a 2000-token one is not (main_low_value); the test says so. docs/BATCHING.md: the park bullet names the value threshold only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…y counts as already kept A park was skipped when a slot already held 90% of the main session's conversation: the missing tail is read again when the conversation comes back. That tail was up to 10%, and the copy it saves is cheap by comparison. Production parks (24, 10K-170K tokens) fit copy time = 13.2 ms per 1K tokens; a tail read on top of a long context runs at about 1,000-1,500 tok/s (a 3.8K chunk after 77K: ~1,100). The two break even at a tail of 1-2% of the conversation (3-4% at 3,000 tok/s): 100K tokens copy in about 1.2 s, while a 10K tail is read again in 7-10 s. The share is now STRATA_SLOT_PARK_KEPT_SHARE, default 0.98; below it the park goes ahead, normally into that slot (the conversation's own older copy), so no further history is given up. live.slot_policy shows it. The long PARK_MIN_VALUE comment is rewrapped to 120 columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… the same conversation is replaced in place An idle history that shares at least this share of its own length with the prompt (or with the main session's conversation, for a park) but is not its start is the same conversation with another end: the client rewrote it. Claude Code's safety classifier does this on every query - it inserts the new tool calls before the closing text of its transcript - so no query continues the last one. Kept side by side, the versions took 4-5 of the 6 slots (2026-10-16) and pushed other conversations out, while a classifier query that ran on the main session was not parked at all (the versions in the slots were worth as much as it, so the value ratio refused them). place(): with no slot continuing the prompt, the version sharing the most is the destination (reason "sibling"), protected or not, before the reclaim order. park_destination(): after its own older copy, before the expired candidates and the value ratio. 0 turns it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

andyhrc mentioned this pull request Oct 15, 2026

[Engine] Keep conversations in pipelined batch slots (1/3): active-row windows, park=, cache-aware placement behind a policy interface #2527

Open

andyhrc force-pushed the pr/slot-policy branch from 85c89c5 to 6955105 Compare October 15, 2026 04:39

andyhrc mentioned this pull request Oct 15, 2026

batch: a slot that gives its whole conversation back to the main session hands over its checkpoints too #2569

Closed

SuperDistance mentioned this pull request Oct 15, 2026

[Engine] serve: opt-in STRATA_SLOT_EXPIRE_S keeps a long conversation's batch slot from a short side request (next turn 12.8 -> 0.3 s) #2589

Open

andyhrc marked this pull request as draft October 15, 2026 10:44

andyhrc mentioned this pull request Oct 15, 2026

Batch slots keep a conversation's turn checkpoints across slot -> solo -> slot moves (#1906) #2384

Closed

andyhrc marked this pull request as ready for review October 15, 2026 14:48

andyhrc force-pushed the pr/slot-policy branch from 968da38 to 54d0bff Compare October 15, 2026 15:35

andyhrc marked this pull request as draft October 15, 2026 15:36

andyhrc force-pushed the pr/slot-policy branch from 54d0bff to a4f01dc Compare October 16, 2026 04:25

This was referenced Oct 17, 2026

[P1][PI][RTX5080] Qualify C1/C2 concurrency for Pi on a single 16 GB RTX 5080 toddballinger/Strata-5080#2

Closed

[P1][CACHE] Qualify full-state 80K–120K Pi follow-up prefix reuse toddballinger/Strata-5080#13

Closed

andyhrc force-pushed the pr/slot-policy branch from 16c5dda to 827f30c Compare October 17, 2026 16:08

andyhrc marked this pull request as ready for review October 17, 2026 18:51

Niko1221 closed this Oct 19, 2026

github-actions Bot reopened this Oct 20, 2026

andyhrc force-pushed the pr/slot-policy branch 2 times, most recently from 92f3fbf to c0276e1 Compare October 20, 2026 18:24

andyhrc mentioned this pull request Oct 20, 2026

[Engine] serve: a slot's live view keeps the request's own clocks after it moves from the solo path (elapsed_s, tok_s) #3492

Open

andyhrc force-pushed the pr/slot-policy branch 2 times, most recently from f8bf92d to 346cb31 Compare October 21, 2026 00:13

andyhrc force-pushed the pr/slot-policy branch from 346cb31 to 2e5e1e9 Compare October 21, 2026 14:40

andyhrc and others added 21 commits October 27, 2026 23:18

andyhrc force-pushed the pr/slot-policy branch from 2e5e1e9 to df1d87b Compare October 27, 2026 17:53

Navigation Menu

Repository navigation

[Engine] Slot placement policy for STRATA_SLOT_CACHE_AFFINITY (2/3, depends on #2527): protection by history length, retention scores, value decay, no wait - #2528

[Engine] Slot placement policy for STRATA_SLOT_CACHE_AFFINITY (2/3, depends on #2527): protection by history length, retention scores, value decay, no wait#2528
andyhrc wants to merge 21 commits into
Niko1221:mainfrom
andyhrc:pr/slot-policy

andyhrc commented Oct 15, 2026 •

edited

Loading

Uh oh!

andyhrc commented Oct 15, 2026

Uh oh!

andyhrc commented Oct 15, 2026

Uh oh!

Niko1221 commented Oct 19, 2026

Uh oh!

andyhrc commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

andyhrc commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem and result

Related discussion

Approach

Validation

Limits

Uh oh!

andyhrc commented Oct 15, 2026

Uh oh!

andyhrc commented Oct 15, 2026

Uh oh!

Niko1221 commented Oct 19, 2026

Uh oh!

andyhrc commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andyhrc commented Oct 15, 2026 •

edited

Loading