[yitizi]

Overview

Implements a persistent prompt cache that supports hybrid/SWA context
checkpoints, restores automatically across process restarts, and provides
corrupt-entry cleanup and LRU eviction bounded by disk size.

Closes #20697

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - the implementation was written by Codex and the tests were written by Claude. I defined the requirements and verified the results.

[ggml-gh-bot]

Hi @yitizi, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[ngxson]

thanks for the PR, actually I already had a half-working version on my side with a slightly different approach, having a chained hash on a block of fixed size token, which allow better detecting reusable prompt from disk. but it's still not perfect, so I want to improve it further

also, I spotted some more refactoring need to be done along the way, so I didn't push my PR

the overall idea from this PR is already contained in my local version and I want to make the code more organized before eventually pushing it. you can keep the PR as-is for now, but note that it will eventually replaced. therefore, I won't dive too deep into this PR, and I won't proceed my review further. thanks for your understanding

cc @ServeurpersoCom for visibility

[sammcj]

Tested on macOS / Metal, which I don't think is covered above.

Build break on macOS. server-context.cpp fails to compile with libc++:

error: use of overloaded operator '<<' is ambiguous
       (with operand types 'basic_ostream<...>' and 'rep' (aka '__int128'))

std::filesystem::file_clock has an __int128 representation on macOS and no operator<< accepts it. One-line fix in server_prompt_cache_key_add_file:

key << "|mtime=" << (ec ? 0 : (long long) modified.time_since_epoch().count()) << "\n";

I have submitted a PR to yours with this fix yitizi#1

Measurements. M5 Max (128GB), Metal, Qwen3.8-Flash-Next 125B-A6B at ~90GB. It is a hybrid model (36 GDN recurrent + 12 attention layers) running with a speculative draft context, so both recurrent and draft state have to survive the round trip. They do.

	prefill
cold, first prompt	39.2 s
after restart, without --cache-disk	39.7 s
after restart, with --cache-disk	0.09 s

Restoring is free at startup: 6.0 s to first listen on a restart against 9.0 s cold. Disk cost was roughly 900 MiB per 33k-token prompt.

One note on the directory layout. On startup a server removes every entry whose cache key does not match its own (the read_metadata -> remove_files path in server-task.cpp). That is correct per-server, but under --models-dir routing the children inherit the parent's argv, so a single --cache-disk path means each model wipes the previous one's cache as it starts. I ended up giving each model its own subdirectory. Might be worth a line in the docs, or deriving the subdirectory from the cache key automatically.

[yitizi]

Tested on macOS / Metal, which I don't think is covered above.

Build break on macOS. server-context.cpp fails to compile with libc++:

error: use of overloaded operator '<<' is ambiguous
       (with operand types 'basic_ostream<...>' and 'rep' (aka '__int128'))

std::filesystem::file_clock has an __int128 representation on macOS and no operator<< accepts it. One-line fix in server_prompt_cache_key_add_file:

key << "|mtime=" << (ec ? 0 : (long long) modified.time_since_epoch().count()) << "\n";

I have submitted a PR to yours with this fix yitizi#1

Measurements. M5 Max (128GB), Metal, Qwen3.8-Flash-Next 125B-A6B at ~90GB. It is a hybrid model (36 GDN recurrent + 12 attention layers) running with a speculative draft context, so both recurrent and draft state have to survive the round trip. They do.

prefill
cold, first prompt 39.2 s
after restart, without --cache-disk 39.7 s
after restart, with --cache-disk 0.09 s
Restoring is free at startup: 6.0 s to first listen on a restart against 9.0 s cold. Disk cost was roughly 900 MiB per 33k-token prompt.

One note on the directory layout. On startup a server removes every entry whose cache key does not match its own (the read_metadata -> remove_files path in server-task.cpp). That is correct per-server, but under --models-dir routing the children inherit the parent's argv, so a single --cache-disk path means each model wipes the previous one's cache as it starts. I ended up giving each model its own subdirectory. Might be worth a line in the docs, or deriving the subdirectory from the cache key automatically.

The macOS build break is fixed — the whole ternary is now wrapped in
static_cast.

Root cause was libc++ rather than macOS as such: libc++ defines
std::filesystem::file_time_type::duration::rep as __int128 and provides no
operator<< for it, while libstdc++ uses int64_t. That is why it never showed
up in my Linux container builds.

To keep it from happening again, I added a check that compiles the server
translation units with both libstdc++ and libc++ on Linux, reusing the real
flags from compile_commands.json with -fsyntax-only. It reproduces your error
before the fix and passes after it, so the whole libc++-vs-libstdc++ class is
now caught locally without needing a Mac. It won't cover Apple-specific ground
— Metal and Accelerate, Apple clang behaviour, the macOS SDK and ld64, or
Metal runtime performance — those still need the macOS runners.

On the multi-model cache directory: my GPU can barely hold a single model, so
that setup never occurred to me. Worth noting it isn't specific to this
implementation — the startup scan deletes mismatched entries outright rather
than skipping them, so any disk-cache design will run into the same thing.

For context — ngxson mentioned he already has similar work locally, so this PR
will most likely end up as a reference implementation showing that a disk cache
is worth having.

Thanks again for catching this.

[yitizi]

Separately — a report on another PR probably applies here too:
#26408 (comment)

@colinrgodsey hit a kernel livelock while saving ~22k tokens of state: the file
is mmap'd MAP_SHARED and llama_state_seq_get_data_ext DMAs straight from VRAM
into the file-backed pages, so the writeback worker and the GPU driver end up
fighting over the same pages at 100% CPU.

This PR does the same thing — server-context.cpp:396 hands the mapping pointer
directly to llama_state_seq_get_data_ext. I haven't hit it myself: he is on
ROCm with a native NVMe ext4, I'm on CUDA under WSL2, so the DMA pinning and
writeback paths differ. I'd rather not rewrite the I/O path blind, so I'm
leaving it as-is here.

@ngxson — flagging it in case your version uses the same mmap-write pattern.
Of everything reported on this PR, it's the only one that can hang the machine.

[lee-b]

On meaningful names, would suggest renaming the flags:

--cache-dir-path <fs-path>
--cache-dir-size-max <size>

Because --cache-disk is a misnomer -- this stores on a filesystem (e.g., /path/to/folder), not a disk device (not e.g., /dev/sdc nor /dev/disk/by-id/wd-black-...). Also because --cache-ram takes a size because that's all you need to know with RAM, whereas --cache-disk/--cache-dir taking a size makes little sense because you expect the disk/directory name to be the parameter.

Some of this may sound nit-picky, but it's important, since:

1. the size and disk/path/dir distinction maintains the principle of least astonishment;

2. actual disk-based caching, to /dev/sdc or something, could possibly be a separate thing, later, for better performance, in the same way that linux has swap files but supports swap disks/partitions for better performance, at which point this would be in conflict and (slightly) hampering that work;

3. Once this is committed to, every user has to live with it forever, or live with it for a time then live with the pain of migrating away from it to a corrected version at some point. Per Tesler's Law, complexity should be paid once, by the system's designers, rather than repeatedly by every user, so the misnomer and incongruity should be addressed now, before merge, not ignored due to the mild inconvenience of adjusting it.

[dimarick]

Livelock reproduced by me on AMD RDNA3 + Vulkan.

Command line $ nice -n +19 ~/llama.cpp/build/bin/llama-server --model Qwen3.8-27b-Dense-Imatrix-IQ3_M.gguf -c 0 -fa 1 --cache-type-k q4_0 --cache-type-v q4_0 --webui-mcp-proxy --ui-mcp-proxy -ngl 99 --load-mode mmap --lazy-mode on --temp 0.3 --top-k 20 --top-p 0.95 --min-p 0.05 --repeat-penalty 1.0 --presence-penalty 0.0 --parallel 4 --no-op-offload --spec-type ngram-mod --spec-ngram-mod-n-match 16 --spec-ngram-mod-n-min 2 --spec-ngram-mod-n-max 128 --kv-unified -t 20 --cache-prompt --reasoning-preserve --cache-reuse 128 --cache-ram 12288 --fit 1 --cache-disk ~/.cache/llama.cpp/kv.cache --cache-disk-max 16384

Reproduced only on generation, never in prefill

--load-mode reproduced on any
other options possible no affect.

Workaround:

--parallel 2 (or more)

while true; do curl -s -X POST http://localhost:8080/completion -H "Content-Type: application/json" -d '{"prompt":"a","n_predict":1,"temperature":0,"cache_prompt":false}' >/dev/null; sleep 60; done

OR any running prompt.

[yitizi]

On meaningful names, would suggest renaming the flags:

--cache-dir-path <fs-path>
--cache-dir-size-max <size>

Because --cache-disk is a misnomer -- this stores on a filesystem (e.g., /path/to/folder), not a disk device (not e.g., /dev/sdc nor /dev/disk/by-id/wd-black-...). Also because --cache-ram takes a size because that's all you need to know with RAM, whereas --cache-disk/--cache-dir taking a size makes little sense because you expect the disk/directory name to be the parameter.

Some of this may sound nit-picky, but it's important, since:

1. the lack of size and disk/path/dir distinction maintains the principle of least astonishment;
2. actual disk-based caching, to /dev/sdc or something, could possibly be a separate thing, later, for better performance, in the same way that linux has swap files but supports swap disks/partitions for better performance, at which point this would be in conflict and (slightly) hampering that work;
3. Once this is committed to, every user has to live with it forever, or live with it for a time then live with the pain of migrating away from it to a corrected version at some point. Per Tesler's Law, complexity should be paid once, by the system's designers, rather than repeatedly by every user](https://lawsofux.com/teslers-law/), so the misnomer and incongruity should be addressed now, before merge, not ignored due to the mild inconvenience of adjusting it.

Thanks for the suggestion — adopted and pushed (--cache-disk → --cache-dir,
--cache-disk-max → --cache-dir-max, env LLAMA_ARG_CACHE_DIR / _MAX).

I went with the shorter form rather than --cache-dir-path / --cache-dir-size-max,
to stay in line with the existing --cache-ram / --cache-reuse / --cache-idle-slots
family, none of which carry a -path or -size-max suffix.

One thing worth noting: this PR will likely be superseded in the end. @ngxson
mentioned he already has a version in development locally, using a different
approach (chained hashes over fixed-size token blocks). This PR is just my own
local implementation.

[michal-zurkowski]

Writing GPU state straight into MAP_SHARED file mappings can livelock the kernel, so use a host buffer with pwrite/pread instead might be better? Especially noticable in systems with UMA like gfx1151 StrixHalo cache files stay in the page cache after writes and reads, which takes memory from the GPU on those kind of systems
Also slot in use is never saved on shutdown, so a --parallel 1server loses the current conversation on every restart or unload of the model. I think preserving that is also important.
--cache-dir-max defaults to unlimited, which i think should at least have some kind of warninig or at least have some kind of check of available/free space so we do not go over some default threshold like 10% and not fully use disk in some cases?
I think --cache-dir silently disables --cache-ram?