[W1nge]

Summary

With STRATA_PROMPT_REUSE=1, continued chats reuse prompt IDs up to a safe special-token boundary. Adapted from gputier's #567, with credit for the core algorithm.

What changed

Keep at most four prompts, each bounded to 256K characters / 64K tokens. Return fresh lists before image expansion. Protected literal spans, unsupported tokenizers and unsafe boundaries use full encoding. Default encoding stays unchanged; preserve main's piece cache.

Validation

On main 61b3fb5d, python -m unittest serve.test_prompt_encoder serve.test_server tools.test_strata_tokenizer tools.test_tokenizer_plain_spans -q passes all 314 tests, including opt-in/default equivalence, edits, overlapping literals, tools, images and concurrency.

No engine changes. Earlier independent token-ID and latency measurements are in KarlGrier's comment.

[KarlGrier]

Tested this PR (head 94befce) on v0.1.41 (fb58e0db), since the description says full-model validation on current main is outstanding. Server-side only: the three files on a v0.1.41 checkout, the v0.1.41 engine unchanged. RTX 5080 16 GB + RTX 4060 Ti 8 GB (helper expert cache), Ryzen 9 9950X3D, Linux 7.0, Python 3.12, Qwen3.8-Flash-Next GSQ-RCO IQ3_S with its pack's tokenizer.

• Applies to v0.1.41: git apply -3 with no conflicts (tools/strata_tokenizer.py sits beside v0.1.41's piece cache from f346dc6).
• Tests: python -m unittest serve.test_prompt_encoder serve.test_server: 282 run, 1 failure, GpuChoice.test_vision_device. Stock v0.1.41 fails the same test alone on this PC (271 run): it expects the GPU order "0,1" and gets "1,0" from Multi-GPU performance is ~2x lower than Francesco Albano fork despite the changes being merged into v0.1.40 #1352's ordering (test: pin GPU order in the vision device fixture #1632 pins it).
• Same ids: 2,418 prompts (growing, edited and shortened chats, 16 interleaved conversations, tools, special-token text, Unicode, a changed system message or effort, 2,000 random edits), 28,545,777 tokens through one PromptEncoder in server order, compared with a full tok.encode(prompt, parse_special=True): 0 differences. Per turn: a ~33K-token chat 5.8 ms full vs 0.36 ms with reuse; a ~103K-token chat 18.1 vs 18.0 ms (above the 64K-token bound, so it encodes in full, as described).
• Time to first token on cached turns (served, 12 follow-ups per chat size per arm, A B B A, reuse on vs STRATA_PROMPT_REUSE=0, STRATA_PREFILL_CPU_SHARE=0 on both): 31K chat median 0.148 → 0.138 s; 98K chat 0.171 → 0.170 s (bounded, as expected). The engine's own prompt time was within 5 ms in both. With v0.1.41's piece cache a full re-encode is already fast here (18 ms at 103K tokens), so the gain is about 10 ms per turn below the bound.

Not tested: images, Windows, other tokenizers, concurrent clients beyond the PR's own test, and agent-style prompts with tool results.

Measured and drafted with Claude Code.

[Niko1221]

Thank you. The same technique (from #567) is now in our branch through #1852, default-on, with ids identical to a full encode. We did not take this opt-in variant on top.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.