[storm1er]

Overview

Implements a persistent prompt cache that supports hybrid/SWA context
checkpoints, restores automatically across process restarts, and provides
corrupt-entry cleanup and LRU eviction bounded by disk size.

See ggml-org#28092 for initial review of these changes, initially targeting ggml-org/llama.cpp. Created by @yitizi

[ChrisC381]

Tested this on qwen4exp (Qwen3.8-Flash-Next, hybrid + shared-MTP draft, f16 KV, draft-mtp n-max 3) on an 8060S/gfx1151, 128 GB. Caveat: applied as one commit on master 03895887a (your branch head is a merge, so I used the merge-base diff). Can rerun on the branch tip if useful.

Works: eviction writes llama-prompt-cache-*.bin (306 MB) + .bin.meta, and restart logs restored 1 persistent prompt cache entries (292.286 MiB).

Fails: after restart, resending the exact cached prompt still fully re-prefills (prompt_n = 2337, cache_n = 0). By the PR's own test criterion (cache_n + prompt_n == cold prompt_n) that's a miss.

The format looks right for this model class (per-checkpoint offset_dft/size_spec, ctx_dft restore paths). But all three transports I tried on this fork restore bytes that serving then ignores: vanilla slots, slots + the ggml-org#26004 checkpoint appendix, and this disk cache. Same visibility-vs-restored-state class as the qwen4exp draft-MTP bug fixed by the Sep 24-25 QSA wave (#76).

The unit tests use tinygemma3/tinyllama (SWA, pure attention). A hybrid + draft-MTP case should reproduce this: restore succeeds, cache_n stays 0. Repro flags on request.

[ChrisC381]

Correction to my earlier comment: "restored KV invisible to the serving path" was wrong.

Actual mechanism: on hybrid/recurrent models the serving path's rollback guard (pos_min >= pos_min_thold, no usable context checkpoint, force full re-process) fires after any restore that does not bring the slot's context checkpoints back. Vanilla /slots files carry none, so the guard finds an empty checkpoint list and re-prefills everything. Same symptom, different cause.

Three transports, one mechanism:

1. vanilla /slots: no checkpoint payload, fails on hybrid.
2. /slots plus the checkpoint appendix (ggml-org#26004 forward-port, server : preserve context checkpoints across slot save/restore #98): works, prompt_n 2337 → 4 after clobber + restore, journal shows the checkpoint rescue.
3. this PR's disk cache: still re-prefills after restart on our qwen4exp + shared-MTP config, although the meta format does carry per-checkpoint offset_dft/size_dft and offset_spec/size_spec. Given (2) works on the same model and config, the loss is somewhere in this PR's load path, not in the state format.

Test-matrix suggestion: tinyllama and tinygemma3 are SWA/pure-attention and pass while the hybrid path is broken. A hybrid-architecture case would catch this class directly: any qwen4exp variant with --ctx-checkpoints N (a compact quant keeps CI cheap; the full-size model is 112 GB).

[ChrisC381]

Follow-up to my earlier comment: I can narrow the failure I saw.

My failing run was on this PR's head (89fb1c6 on master), so none of the active build's three fixes (no-mmap, save-on-stop, streamed I/O). It was also --parallel 1, so the entry was eviction-written. On that setup the entry did come back after restart (292 MiB, journal confirms), but the exact-match resend still re-prefilled everything (prompt_n 2337, cache_n 0).

I've since seen a report of the active build reusing checkpoints after restart with MTP enabled (117,347 of 125,542 tokens on a 125K doc). That doesn't clash with my result. It narrows the suspect: the format's fine, so the mmap write path or eviction-written entries are where I'd look. Happy to rerun my repro once the fixes land on this branch.

[dzannotti]

Tested on Strix Halo (gfx1151, HIP), this PR at 89fb1c6b1. Sequence: send a ~5k-token prompt P, then an unrelated prompt so P is saved, restart the server with the same --cache-dir, resend P.

model	startup log	P after restart
Qwen3-1.7B-Q8_0 (plain attention)	restored 1 persistent prompt cache entries (547.152 MiB)	cache_n 0, 4987 tokens recomputed (1215 ms, cold 1176 ms)
Qwen3.6-35B-A3B UD-Q3_K_XL (hybrid)	restored 1 ... (287.970 MiB)	cache_n 0, 5075 tokens recomputed (9509 ms, cold 9367 ms)

Entries are restored at startup but never reused, on plain-attention and hybrid models alike, so the main claim does not reproduce.

Issues to address before this can land:

• Readable prompts: *.bin.meta is created 0644 and the cache directory 0755. The meta file holds the serialized prompt tokens (~4 bytes per token), so any local user can read cached prompts. The .bin itself is 0600.
• Unbounded disk use: --cache-dir-max defaults to -1 (no eviction), so remote clients can grow the cache directory without limit.
• Server-thread stalls: saves run synchronously on the server thread (msync(MS_SYNC)), which stalls all slots while an entry is written.

---

Comment generated via automated review of clanker Claude Code.

[dzannotti]

merging this as is, as it's too good not to merge, we can fix the 2 comments forward