Repository navigation
Misc. bug: --cache-reuse no longer seems to be caching prompt prefixes #15082
Description
This is a re-open of #14113
Name and Version
Affected:
Version at commit: b7a1746
Not affected:
Version at commit: c6a2c9e
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
llama-server
Problem description & steps to reproduce
I had open this bug in oobabooga/text-generation-webui: oobabooga/textgen#7060
The issue being that prompt prefixes were no longer being used in the following requests.
I confirmed that --cache-reuse 1 was being passed on, so that wasn't the issue.
After reverting to the previous version of the WebUI (which ships an older version of llama.cpp), the prompts started to be cached again.
So, this seems to point to being a bug with llama.cpp.
First Bad Commit
It looks like there may have been a commit between c6a2c9e and b7a1746 that broke --cache-reuse, or that changed its behavior.
I use the oobabooga/text-generation-webui project, which bundles a snapshot of llama.cpp (the commits I mentioned above),
and, from what I saw, this is the command it runs:
llama-server --model user_data/models/gemma-3-12b-it-qat-UD-Q6_K_XL.guff --ctx-size 32768 --gpu-layers 49 --batch-size 256 --port 60033 --no-webui --threads 10 --threads-batch 10 --rope-freq-scale 0.125 --rope-freq-base 1000000.0 --cache-reuse 1
The model can be downloaded from: https://huggingface.co/unsloth/gemma-3-12b-it-qat-GGUF/tree/main
I confirm --cache-reuse 1 is present in the parameters, when the cache isn't working.
I don't know how exactly a Chat Completion call is made to llama-server, but basically, if I make two chat-instruct calls, where the first 1000 characters or so are exactly the same, the prompt will be processed from byte 0, when using the affected commits I mentioned above.
I'm not providing the cache_prompt parameter, so it should default to true.
I hope this helps!
This is still happening when using the version at https://github.com/ggml-org/llama.cpp/tree/90083283ec254fa8d33897746dea229aee401b37
It appears that the fix from #14163 did not fix the cache issue.
Thank you.
Activity
pwilkin commentedon Aug 5, 2025
I think you need
--swa-fullfor that to work now (it will use more memory though).ggerganov commentedon Aug 11, 2025
Yes, iSWA models require
--swa-fullto be able to do cache reuse. The reason is that this operation require to store the full KV cache - not just the one in the sliding window.ghnp5 commentedon Aug 11, 2025
Hey! Alright - looks like
--swa-fullresolves this problem.When I was testing, the 3rd completion (which was supposed to use the cache from the previous calls) didn't use the cache, for some reason. However, all further attempts successfully used the cache, so it DOES seem to be working. Not really sure what happened on that other one, but as long as most of them use the cache, I'm happy!
Thank you so much!
FSSRepo commentedon Aug 12, 2025
I still have a problem with GPT OSS 20B. When using the web UI, the first chat works fine — it processes the system prompt and generates the response. But on the next question, it takes a long time because it reprocesses the response it already generated before. If the previous response is very long, it takes a long time to generate a token.
ggerganov commentedon Aug 12, 2025
@FSSRepo Could you post the
llama-servercommand that you use?slaren commentedon Aug 12, 2025
I also noticed this with the 120B model. Almost every new message causes a full reprocess of the entire context, which really impacts the usability of the model. To be blunt, it makes the model unusable when keeping MoE layers in the CPU. My assumption is that it is due to differences in the tokenization and what the model generated.
ggerganov commentedon Aug 12, 2025
Rather it's the small SWA window - just 128 tokens. Can you confirm that no extra reprocessing is observed with
--swa-full?slaren commentedon Aug 12, 2025
I will give it a try. My understanding is that SWA makes the issue worse because any differences in the tokenization will cause the entire context to be reprocessed, rather than from the point where the difference starts. However, it should be able to reuse the context if there are no differences. Is that wrong?
ggerganov commentedon Aug 12, 2025
This is generally true, but for reasoning models there is caveat. The thinking of response
Nis not included in requestN+1. So regardless of tokenization differences, if the sum ofthinking + generatedtokens for responseNis larger thann_ubatch + n_swa, we will reprocess.With the
--cache-reuseflag enabled, reprocessing will be reduced for reasoning models because we can reuse all generated tokens. But there is still a problem: if the number ofthinkingtokens in the response is larger thann_ubatch + n_swawe will reprocess too.slaren commentedon Aug 12, 2025
I didn't realize that the reasoning was being stripped from the context in this model.
--swa-fullgreatly reduces the amount of reprocessing on each message, as expected, starting from what I assume is the start previous message, instead of reprocessing the entire context.ggerganov commentedon Aug 12, 2025
You can make it keep the thinking tokens from the webui:
But if I remember correctly this is not recommended and will likely confuse the model.
Note that
--swa-fullis one option, but you can also choose to increasen_ubatch.If we think these are not enough, we can also add a new parameter to explicitly control the SWA cache size independently from
n_ubatch- would be very easy to implement.slaren commentedon Aug 12, 2025
I am skeptical of RoPE shifts because the result is not the same as if the context was regenerated, so it is hard to predict how it will affect the quality of the model output. I suppose it would be possible to keep a checkpoint of the SWA cache state at the end of every message, and it would require a lot less memory than
--swa-full.I already have that option disabled in the WebUI, so if that's the case and also affects this model, then I don't think the cause of the reprocessing is likely to be the stripping of the reasoning.
The messages tend to be rather long because they usually include a lot of code, so I don't think this would be viable, it would increase the size of the compute buffer too much.
ggerganov commentedon Aug 12, 2025
Indeed, there appears to be a bug - I don't think the logic for dropping the thinking tokens works in our WebUI (#15181 (comment)). We should fix this first.
ggerganov commentedon Aug 13, 2025
The #15278 should fix the dropping of the thinking tokens when the WebUI option is enabled (default). I looked if we can make the disabled state of this option work (i.e. keep the context as it is without dropping thinking chunks). I don't see a good way to filter the
harmonytags to make this compatible with the chat completion endpoint. Plus given that the recommendation is to filter old thinking tokens, it's probably not a good idea to keep them anyway. So probably we should avoid this option for now?slaren commentedon Aug 13, 2025
Yeah, this probably shouldn't be an option at all if the model requires it. The problem is not so much having to reprocess the previous response to remove the reasoning, but that when using SWA, this results in the entire context being reprocessed.
ggerganov commentedon Aug 13, 2025
Yes, that's a bummer. SWA KV cache checkpoints are in theory possible - the
llama.cppchanges to support this would be relatively small and easy. But I feel it will be very messy to support this inllama-server- at least I'm having difficulty picturing it.slaren commentedon Aug 13, 2025
I think the logic could be something like this:
drrros commentedon Aug 13, 2025
This doesn't happen for 120b model with branch from #15181 - though not latest version of that branch (i'm on
04e162607a17a14a6df9f5dcf15b090b410484c9)