[code-martin]

Summary

This PR addresses root checkpoint retention for --serve prompt caching, keeping the root system prompt cached across branching chats and new conversations without compromising v0.1.13 prefill throughput:

1. Root & Last Turn Checkpoints Only:

   • Rather than checkpointing every turn (which triggers repeated borrow/refill cycles and large memory copies), the prompt reading loop identifies at most two turn boundaries:
     • urn_first: the first <|im_start|> past token 0 (the root boundary after the system prompt).
     • urn_last: the final <|im_start|> before the assistant turn.
   • Long conversation histories remain batched in full chunks (up to 8192 tokens), preserving v0.1.13 speedup.

2. Root Preservation During Eviction:

   • In checkpoint_at(), when checks.size() > o.prompt_cache, if checks.size() > 2 && checks[0].ids.size() <= checks[1].ids.size(), eviction drops checks[1] (the oldest intermediate turn) instead of checks[0].
   • The root system prompt checkpoint remains resident in checks[0]. Subsequent requests sharing the same system prompt match checks[0] and resume immediately (
     esume == checks[0].ids.size()) without prefilling the system prompt again.

[maedoc]

you might wanna jump in over here #57 where the caching action is

[Niko1221]

Thanks @code-martin - saving a checkpoint at the end of the system prompt was exactly the piece #62 was missing. Since both changed the same eviction loop, I folded your idea into #62 (with credit in the commit), with a minimum of 2,048 tokens so short system prompts don't pay for the extra split (~0.3 s), and a switch (--prompt-cache-root). It's in v0.1.20, so I'm closing this one.