[brocktice]

Why

We serve an agent client whose every chat opens with the same ~79K-token system prompt and tool list, on parallel: 4 with --batch-mtp and --conversation-cache-mib 8192. Its chats overlap with other traffic all day, and every new chat read the whole opening again (20-35 s) although the engine had read the same tokens minutes before.

The cause is in --batch. The root checkpoint (the end of the system prompt, PR #65) lives in the main session's chain, but:

• a slot keeps only its turn checkpoint (the deepest one) at admission;
• a request that continues from a slot clears the main chain and takes over that one checkpoint;
• the next park then drops the older parked copies that still held the root (Conversation cache slots are consumed by successive turns of the same subagent, evicting the parent conversation #342's drop_superseded).

After a few turns of any chat nothing holds the root any more. Solo traffic does not show it: a request that runs alone stays on the main session and its chain stays whole.

What it does

--prompt-cache-roots N (default 0 = off, every path unchanged) keeps a copy of the last N roots the engine made, apart from any conversation (one per distinct system prompt, ~118 MB of RAM each; least recently made or used leaves first). A root's state depends on its tokens alone, so:

• main session: at a request's start, and after a slot or a parked conversation was restored, a kept root goes back into the chain (at its front, where retention pins it) when the session's cells hold its tokens and the prompt starts with them;
• idle slot: a kept root whose tokens an idle slot holds is a source like the slot's own checkpoints: the slot's K/V up to the root is copied back and the root's state restored;
• All-logits-NaN degeneration (one repeated token forever) still fires on 0.1.39b / current main — first poison traced to the MoE output rows: garbage bits, varying layer #879 (non-finite logits) drops the kept roots with everything else.

Because the main chain keeps its root again, parked conversations keep it too, so a new chat also mounts through a parked one.

Tests

conv_cache_test covers the store (kept, refreshed, the least recently used one leaves, other images or control-vector mode do not match, clear).

Real engine: v0.1.43 on one RTX PRO 6000 (Blackwell), Qwen3.8-Flash-Next IQ3_S, parallel: 4, --batch-mtp, --conversation-cache-mib 8192. A 42K-token system prompt; chat A three turns, a request with another system prompt, chat B two turns, chat C. Greedy, the same requests with the flag off and on:

	flag off	flag on
sequential (no overlap)	new chats reuse 42,146 tokens	the same
two other requests always decoding: new chat B (--prompt-cache-roots 2)	0 reused, 11.6 s	42,146 reused, 0.8 s
parking off (--conversation-cache-mib 0), the root only in an idle slot: new chat B (--prompt-cache-roots 8)	0 reused, 7.9 s	42,146 reused, 0.8 s

Answers were identical between the arms except one 600-token answer decoded beside another request; that one also differs between two runs with the flag off (batched greedy decode is not run-to-run identical).

N matters: in the last scenario with N = 2, a third system prompt pushed the 42K root out before chat B came, so we run 8.

In use: v0.1.43 plus this change has run with --prompt-cache-roots 8 on two RTX PRO 6000s for about 8 hours of mixed agent, chat and tool traffic (about 2,800 requests, no errors); new chats of that client now reuse their shared opening.

Builds: the measurements above are from this change on v0.1.43. Rebased on current main, it builds for CUDA (sm_120) and CTest passes (conv_cache_test included) except ple_parity, which needs a Q2_0 model file I do not have. I could not build HIP or SYCL (no toolchain here). The change is host-only C++ in src/program/generate.cpp and conv_cache.hpp; the SYCL generate.cpp includes the header but does not use the new class (the header also compiles alone with g++ -Wall -Wextra).

Default path: with the flag at 0 the store is empty and every new branch is skipped: seed_root, the slot lookup and keep all test enabled() or prompt_cache_roots first, so no checkpoint, copy or log line differs.

Related: #1529 keeps the root in the disk tier's spilled files. This change is the RAM side under --batch and does not touch the disk tier.

Happy to change the shape (a default other than 0, or keeping the roots inside the conversation cache's budget).

🤖 Generated with Claude Code

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.