Repository navigation
Misc. bug: --cache-reuse no longer seems to be caching prompt prefixes #14113
Description
Name and Version
Affected:
Version at commit: b7a1746
Not affected:
Version at commit: c6a2c9e
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
llama-server
Problem description & steps to reproduce
I had open this bug in oobabooga/text-generation-webui: oobabooga/textgen#7060
The issue being that prompt prefixes were no longer being used in the following requests.
I confirmed that --cache-reuse 1 was being passed on, so that wasn't the issue.
After reverting to the previous version of the WebUI (which ships an older version of llama.cpp), the prompts started to be cached again.
So, this seems to point to being a bug with llama.cpp.
First Bad Commit
It looks like there may have been a commit between c6a2c9e and b7a1746 that broke --cache-reuse, or that changed its behavior.
EDIT - still broken in today's commit: 2bb0467
EDIT 2 - someone seems to have reported the same thing here: #13576 (comment) (although that PR is not yet merged)
There was another recent change, here: 06a92a1
Activity
ggerganov commentedon Jun 11, 2025
Provide repro steps using
llama-server.ghnp5 commentedon Jun 11, 2025
Hello,
Thank you for your reply. I use the
oobabooga/text-generation-webuiproject, which bundles a snapshot ofllama.cpp(the commits I mentioned above),and, from what I saw, this is the command it runs:
The model can be downloaded from: https://huggingface.co/unsloth/gemma-3-12b-it-qat-GGUF/tree/main
I confirm
--cache-reuse 1is present in the parameters, when the cache isn't working.I don't know how exactly a Chat Completion call is made to
llama-server, but basically, if I make two chat-instruct calls, where the first 1000 characters or so are exactly the same, the prompt will be processed from byte 0, when using the affected commits I mentioned above.I'm not providing the
cache_promptparameter, so it should default totrue.I hope this helps!
ggerganov commentedon Jun 11, 2025
The
--cache-reuseparameter is not relevant for this case.SWA models like Gemma could exhibit the problem that you describe, but I am fairly certain that I fixed this in #13833. Unless you can provide repro steps, I would assume this is a problem in the oobabooga project.
ghnp5 commentedon Jun 11, 2025
Thank you. This same Gemma model that I'm using works when I use the
llama-serverversion at c6a2c9eApologies - I'm confused why
--cache-reuseis not relevant. Isn't this (together withcache_prompt: true) what prevents the prompt prefix from being re-processed on subsequent requests that share the same prefix?If I don't include
--cache-reusein the parameters, then the prompt will always be fully processed from the beginning, even on the version that works for me.Please let me know exactly what do you need, for repro steps.
github-actions commentedon Jul 27, 2025
This issue was closed because it has been inactive for 14 days since being marked as stale.
ghnp5 commentedon Aug 5, 2025
@ggerganov - This is still happening when using the version at https://github.com/ggml-org/llama.cpp/tree/90083283ec254fa8d33897746dea229aee401b37
It appears that the fix from #14163 did not fix the cache issue.
Thank you.