[SuperDistance]

Problem and result

I run coding agents on Strata with --batch. Next to their long conversations, clients send short side requests (a title, a summary, a one-line check). When every free slot holds a conversation, pick_slot takes the least recently used one, so a side request can take the slot of a 40K-token conversation that is still going, and that conversation's next turn reads its whole history again: 12.8 s to the first token on 0.2.0-beta (one RTX 5090, below). With the new opt-in STRATA_SLOT_EXPIRE_S=600 the side request takes an expired slot or the one holding the shortest history, and the long conversation's next turn takes 0.34 s. Unset or 0, nothing changes.

Related discussion

#1906 and #1930 (agent sessions reading whole prompts again), #1920, #1938. Next to #2527 / #2528 (andyhrc): their placement policy covers pipelined groups; this is the small counterpart for --batch without pipelined groups (one GPU, or --batch-groups 1). qiuzan2001 noted here that the config's "env" block was ignored: the value is now read from the engine's environment (the server's plus the config's "env"). Rebased onto 0.2.0-beta after the tracker reset (one textual conflict: two functions added at the same place).

Approach

STRATA_SLOT_EXPIRE_S (seconds; off unless set). Set, a new conversation that finds no empty slot takes an expired one first (idle that long or longer: its client has likely moved on), the least recently used among them; else the one holding the shortest history. Unset or 0: the least recently used one, as before (a test checks it picks main's slot on 3,000 random slot tables). A request still goes to the slot holding its own prefix first, and pipelined groups return before this rule. Changes: pick_slot's last min() key, slot_expire_s(env), serve/test_slot_alloc.py, and the sentence on slot choice in docs/BATCHING.md.

Validation

Passed:

• 0.2.0-beta (main e88ee39) + this branch (ca7d91a), Windows 11, Python 3.12: python -m unittest serve.test_slot_alloc 25 OK; the whole serve suite (python -m unittest discover -s serve -t . -p "test_*.py") 899 OK, 21 skipped, against 881 OK, 21 skipped on main.
• One RTX 5090 (Windows 11), engine built from 0.2.0-beta (e88ee39), Swift 1.5 IQ3_S, --batch 3, no pipelined groups, no conversation cache, a fresh engine per arm, STRATA_SLOT_EXPIRE_S=600 in the config's "env" block (main ignores it). A third stream decodes all along, so every request goes through a slot. Conversation A (a 40,250-token prompt) and B (about 1,770 tokens) finish and stay in their slots; then a one-line side request; then A's and B's next turns:

	main	this, STRATA_SLOT_EXPIRE_S=600	this, unset
slot the side request took	A's (40,250 tokens held)	B's (1,781 held)	A's (40,250 held)
A's next turn: first token	12.78 s	0.34 s	12.70 s
B's next turn: first token	1.20 s	1.18 s	1.17 s

• Before the rebase, on 0.1.45 (477ac13), same bench: 14.79 / 14.84 s on main against 0.32 / 0.32 s with 600, and 14.72 s unset. A 21-hour replay of my own agent traffic through the slots (ids, times and token counts only): 17.8 / 20.6 / 21.1% fewer prompt tokens read at 4 / 6 / 8 slots with 600.

Failed: none observed.

Not run: a split with --batch-groups 1 on GPUs; the RAM conversation cache on (replay only: smaller gain); AMD, Linux, Mac (Python-only change, nothing to build).

Limits

Opt-in, because it changes whose conversation loses its slot and is measured on one user's traffic. A long conversation whose client has gone keeps its slot until it expires; a client whose prompts never match its slot leaves histories that stay until they expire.

[qiuzan2001]

serve.test_slot_alloc passes here (Linux). One setup trap: slot_expire_s() reads os.environ of the server process, once, when StrataEngine is defined (SLOT_EXPIRE_S = slot_expire_s()), while the run config's "env" block, where the other STRATA_* switches usually go, is only applied to the engine's environment (child_env, serve/server.py:2822). So "env": {"STRATA_SLOT_EXPIRE_S": "600"} in strata-<model>.json is accepted but changes nothing; only an exported shell variable before START-HERE works. BATCHING.md does say "in the server's environment", but reading it from the config (or from child_env(cfg), as #2528's policy tunables do) would match how users set these. Also, the branch conflicts textually with #2527 (3 files), which reworks slot placement for pipelined groups.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[SuperDistance]

/reopen

[github-actions]

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.