[Mirtraxxx]

What it does

Right now every request starts from an empty session and reads its whole prompt again. In a long chat or an agent
loop, every turn waits for the full prefill. That's the main complaint in the r/LocalLLaMA thread ("throws away the
KV cache every prompt").

With this change the engine keeps the conversation and reads only the new part:

• Live session. If the new prompt starts with exactly the tokens the session already holds, it just continues.
• Checkpoints. Up to 6 are kept in RAM, about 118 MB each. One is taken at the last <|im_start|> of every
  prompt, where the history ends and the new assistant header begins, and one every 16K freshly read tokens. A
  rebuilt chat history (the client re-sends the messages, the template re-renders them) still matches the one
  taken before its last turn, so only the last reply and the new message are read.
• Only the running state is copied. That's the 36 GDN recurrences and conv histories, the 12 QSA indexer tails
  and the PLE history: the same set Verifier::commit rolls back. The KV cache and the pooled indexer keys are
  positional. A query only reads cells below it, and dead for its own block, so stale cells past a rewind point
  are never read and get overwritten.
• Safety. A checkpoint is used only if the prompt starts with exactly its tokens and pictures. Images are
  compared by a hash of their embeddings and grid, so a different picture of the same size never matches.
  Checkpoints whose cells the next request will overwrite are dropped.

Measured

RTX 3090 24 GB, Ryzen 7 5700X, 64 GB DDR4-3000, Windows 11, Swift 1.5 IQ2_XS, --expert-cache auto, MTP --spec 4.

Agent-style session through serve/server.py: OpenAI streaming, one tool, thinking on, reasoning sent back each turn.

Turn	Prompt	Reused	Read	Time to first token
1	19,355	0	19,355	56.2 s
2 (tool result added)	20,882	19,407	1,475	5.6 s
3	21,189	20,877	312	2.9 s
4	21,355	21,184	171	2.6 s

Is it exact?

• STRATA_CKPT_REREAD=1 makes the engine read a checkpoint's tokens again from position 0, in the same chunks the
  request that saved it used, instead of restoring it.
• STRATA_STATE_HASH=1 prints a fingerprint of every part of the session after each request: GDN states, PLE
  history, indexer tails, pooled keys, the KV cells [0, L) of all 12 QSA layers, the MTP K/V, and ple_prev.
• A 4-turn chat (19K-token first prompt, two rebuilt follow-ups, one exact continuation) with --adapt-swaps 0, so
  the VRAM expert set is fixed. Restore and re-read gave identical fingerprints for every component after every
  turn, and identical tokens and draft counts.
• Two runs of the restore path were also identical to each other.

Protocol (backward compatible)

• New stdout lines: RESUME <n> before reading, PP <pos> <n> <ms> <fresh tok/s> per chunk, and REUSED <n> once
  the prompt is read.
• DONE gains <drafts accepted> <drafts offered> <reused> after the existing fields.
• serve/server.py ignores lines it doesn't know. One change: PP lines now count as a heartbeat. They arrive
  every few seconds, which kept resetting the 10 s wait, so a long prompt sent no keep-alives at all: 0 in a
  41-second prefill before this change, 6 after.

Options

• --prompt-cache N: checkpoints kept (default 6; 0 restores the old behaviour).
• --prompt-cache-every N: periodic checkpoint spacing (default 16384; 0 = only at the turn boundary).
• --turn-token ID: the token that opens a turn (default 248045, <|im_start|>).

Limits

• One conversation is cached at a time. Two chats that take turns overwrite each other's state and re-read.
• Small reads still pay the prompt path's fixed cost (lending and refilling the borrowed cache slots): about 1–2.5 s
  even for a few dozen tokens.
• Only tested on Windows / one RTX 3090. Images through the cache are designed for but not tested end to end.

Made with Claude Code (Claude Opus 5.5); tested on the hardware above.

🤖 Generated with Claude Code

[Niko1221]

Honestly this is huge, thank you. I was halfway through my own cache and dropped it for yours, since yours does more (live continuation, pictures, the reread/hash checks). On my 5070 follow-ups went from ~13 s to 0.3 s, and restore vs reread gave the same tokens and the same state hashes on every turn. Out in v0.1.3.