Repository navigation
Eval bug: Qwen 3.5 Full prompt re-processing on every conversation turn #20225
Description
Name and Version
version: 8182 (05728db)
built with GNU 13.3.0 for Linux x86_64
same issue on release llama-b8233 and 8149
Operating systems
Linux
GGML backends
HIP
Hardware
Ryzen 9600X + Radeon AI PRO R9700
Models
Qwen3.5 27B
Problem description & steps to reproduce
Summary
Qwen 3.5 27B (hybrid attention + Mamba2/SSM architecture) forces full prompt re-processing on every conversation turn due to the recurrent memory's pos_min always exceeding the SWA-based pos_min_thold threshold. This makes multi-turn conversations unusable at scale - a 15k-token conversation takes ~8 minutes per turn instead of seconds.
Steps to Reproduce
- Start the server:
llama-server --host 0.0.0.0 --port 8080 \
-m Qwen3.5-27B.Q8_0.gguf \
-np 1 --batch-size 4096 --ubatch-size 1024 \
-c 131072 -ngl 99 --flash-attn on -v- Send a first message (works fine):
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model":"test",
"messages":[
{"role":"system","content":"You are a helpful assistant."},
{"role":"user","content":"What is the capital of France?"}
],
"max_tokens":20
}'- Send a follow-up message (triggers full re-processing):
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model":"test",
"messages":[
{"role":"system","content":"You are a helpful assistant."},
{"role":"user","content":"What is the capital of France?"},
{"role":"assistant","content":"Paris is the capital of France."},
{"role":"user","content":"What about Germany?"}
],
"max_tokens":20
}'Observed Behavior
Every subsequent conversation turn forces full prompt re-processing:
slot update_slots: id 0 | task 302 | n_past = 3, slot.prompt.tokens.size() = 357, seq_id = 0, pos_min = 356, n_swa = 1
slot update_slots: id 0 | task 302 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id 0 | task 302 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 0 | task 302 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 4096, progress = 0.277714
<~2 minute wait per batch at 30 tok/s>
slot update_slots: id 0 | task 302 | n_tokens = 4096, memory_seq_rm [4096, end)
slot update_slots: id 0 | task 302 | prompt processing progress, n_tokens = 8192, batch.n_tokens = 4096, progress = 0.555427
With a 15k-token conversation at 30 tok/s, each turn takes ~8 minutes because the entire prompt history is re-processed from scratch.
Expected Behavior
Only the new tokens (the latest user message + formatting) should be processed. The common prefix from previous turns should be served from cache. A typical follow-up should take seconds, not minutes.
Root Cause Analysis
The issue is in tools/server/server-context.cpp around line 2286-2378. The logic was designed for SWA (Sliding Window Attention) models but doesn't account for hybrid recurrent models:
1. pos_min_thold calculation assumes SWA semantics
const auto n_swa = std::max(1, llama_model_n_swa(model)); // returns 1 for Qwen 3.5 (no SWA)
const auto pos_min_thold = std::max(0, n_past - n_swa); // e.g., max(0, 31-1) = 30For the recurrent memory, pos_min from llama_memory_seq_pos_min() returns the full sequence length (e.g., 51), because recurrent state encompasses all positions. Since pos_min (51) > pos_min_thold (30), the code enters the checkpoint search/restore path.
2. Checkpoint search condition doesn't match recurrent models
const auto it = std::find_if(
slot.prompt.checkpoints.rbegin(),
slot.prompt.checkpoints.rend(),
[&](const auto & cur) {
return cur.pos_min < pos_min_thold; // designed for SWA, not recurrent
}
);A recurrent checkpoint at pos_min = 66 with pos_min_thold = 30 fails this check (66 < 30 is false), so the checkpoint is considered invalid and the search fails.
3. Checkpoints not created for single-batch prompts
The checkpoint creation code (line ~2577) runs pre-decode, using pos_min/pos_max from the memory. For prompts that fit in a single batch, these values aren't available yet (the batch hasn't been decoded), so pos_min = -1 and no checkpoint is created.
4. Checkpoint erasure invalidates valid recurrent checkpoints
Even when a checkpoint is successfully created (multi-batch prompts), the erasure code at line ~2382 deletes any checkpoint with pos_min > pos_min_thold, which always applies to recurrent models:
for (auto it = slot.prompt.checkpoints.begin(); ...) {
if (cur.pos_min > pos_min_thold) { // always true for recurrent
it = slot.prompt.checkpoints.erase(it); // valid checkpoint destroyed
}
}5. seq_rm always fails for recurrent partial ranges
When llama_memory_seq_rm() is called with a partial range on a hybrid model, the recurrent memory cannot be partially truncated, so seq_rm returns false. The recovery code then tries to restore from a checkpoint, but the position accounting doesn't properly synchronize the recurrent state with the attention KV cache, leading to "Invalid input batch" errors on decode.
Impact
This affects all hybrid/recurrent model architectures (Qwen 3.5, Falcon-H1, Jamba, etc.) in multi-turn conversation scenarios. The issue scales with conversation length - longer conversations waste proportionally more time on redundant re-processing.
Note: the Vulkan backend does not exhibit this issue, suggesting the recurrent memory handling differs between backends.
Related
- PR server: Fix multimodal context checkpointing for hybrid/recurrent models #19747 (Multimodal Context Checkpointing for Hybrid/Recurrent Models) attempted to address parts of this
- PR server : support multi-modal context checkpoints #19849, server : enable multi-modal prompt caching #19877, server : fix ctx checkpoint restore logic #19924, Hybrid model cache: add
--checkpoint-every-nb#20087 (merged checkpoint improvements) partially improved the situation but don't fully resolve it for hybrid models - Issue Eval bug: qwen35moe always forces a full prompt reprocess after each message, 'failed to truncate' #19690 (multimodal checkpoint failures)
- PR kv-cache : add SWA support #13194 (original SWA checkpoint logic, referenced in the warning message)
First Bad Commit
No response
Relevant log output
slot update_slots: id 0 | task 302 | n_past = 3, slot.prompt.tokens.size() = 357, seq_id = 0, pos_min = 356, n_swa = 1
slot update_slots: id 0 | task 302 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id 0 | task 302 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 0 | task 302 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 4096, progress = 0.277714
<~2 minute wait per batch at 30 tok/s>
slot update_slots: id 0 | task 302 | n_tokens = 4096, memory_seq_rm [4096, end)
slot update_slots: id 0 | task 302 | prompt processing progress, n_tokens = 8192, batch.n_tokens = 4096, progress = 0.555427
Activity
NovNovikov commentedon Mar 8, 2026
I can confirm this bug. And it actually got much worse in recent builds. Previously full reprocessing happened only 1 in 3 times, now it happens all the time.
porterchild commentedon Mar 8, 2026
seeing nearly the same behavior, but the cache seems to work ~50% of the time.
was hoping #20087 would solve this, but I'm still seeing huge prompt recomputations from scratch, in Opencode.
Caching works fine as the model takes multiple turns by itself.
But if I add a message, it will sometimes seem to find a checkpoint to recompute from within the (default) bag of <=32, only incurring recomputing the prompt from there. But sometimes (~50% of the time) it recomputes the entire thing, after a lot of
Checking checkpointanderased invalidated context checkpointmessages.example output after an append-only new message:
Details
srv update_slots: all slots are idlesrv params_from_: Chat format: peg-native
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.340 (> 0.100 thold), f_keep = 0.258
srv get_availabl: updating prompt cache
srv prompt_save: - saving prompt with length 42687, total state size = 1150.354 MiB
srv load: - looking for better prompt, base f_keep = 0.258, sim = 0.340
srv update: - cache state: 1 prompts, 3386.305 MiB (limits: 8192.000 MiB, 180224 tokens, 180224 est)
srv update: - prompt 0xc0b1b7090: 42687 tokens, checkpoints: 15, 3386.305 MiB
srv get_availabl: prompt cache update took 8833.97 ms
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> ?min-p -> ?xtc -> ?temp-ext -> dist
slot launch_slot_: id 3 | task 30803 | processing task, is_child = 0
slot update_slots: id 3 | task 30803 | new prompt, n_ctx_slot = 180224, n_keep = 0, task.n_tokens = 32329
slot update_slots: id 3 | task 30803 | n_past = 10995, slot.prompt.tokens.size() = 42687, seq_id = 3, pos_min = 42686, n_swa = 1
slot update_slots: id 3 | task 30803 | Checking checkpoint with [32666, 32666] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [30918, 30918] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [22581, 22581] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [16822, 16822] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [16721, 16721] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [16160, 16160] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [15986, 15986] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [15669, 15669] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [12471, 12471] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [11725, 11725] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [11539, 11539] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [11286, 11286] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [11078, 11078] against 10994...
slot update_slots: id 3 | task 30803 | Checking checkpoint with [10484, 10484] against 10994...
slot update_slots: id 3 | task 30803 | restored context checkpoint (pos_min = 10484, pos_max = 10484, n_tokens = 10485, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 11078, pos_max = 11078, n_tokens = 11079, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 11286, pos_max = 11286, n_tokens = 11287, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 11539, pos_max = 11539, n_tokens = 11540, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 11725, pos_max = 11725, n_tokens = 11726, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 12471, pos_max = 12471, n_tokens = 12472, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 15669, pos_max = 15669, n_tokens = 15670, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 15986, pos_max = 15986, n_tokens = 15987, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 16160, pos_max = 16160, n_tokens = 16161, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 16721, pos_max = 16721, n_tokens = 16722, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 16822, pos_max = 16822, n_tokens = 16823, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 22581, pos_max = 22581, n_tokens = 22582, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 30918, pos_max = 30918, n_tokens = 30919, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 32666, pos_max = 32666, n_tokens = 32667, n_swa = 1, size = 149.063 MiB)
slot update_slots: id 3 | task 30803 | n_tokens = 10485, memory_seq_rm [10485, end)
slot update_slots: id 3 | task 30803 | prompt processing progress, n_tokens = 12533, batch.n_tokens = 2048, progress = 0.387671
slot update_slots: id 3 | task 30803 | n_tokens = 12533, memory_seq_rm [12533, end)
slot update_slots: id 3 | task 30803 | prompt processing progress, n_tokens = 14581, batch.n_tokens = 2048, progress = 0.451019
... recomputes the rest of the prompt
porterchild commentedon Mar 8, 2026
If I'm reading this correctly https://pytorch.org/blog/hybrid-models-meet-sglang-more-than-full-attention/
, maybe this is less a 'bug' and more a 'good cache reuse when using recurrent memory requires a bit more sophistication than simple checkpointing'? idk, just skimmed.
porterchild commentedon Mar 8, 2026
I can't reproduce this with a simple script.
So maybe I'm just having Opencode issues.
I'll leave my comment above in case I succeed in finding a reproducer. Until then, I have no confidence there's a bug.
the cacheing behaves as expected here:
(@williamtwomey I made the prompt a lot bigger because it seems #20087 only creates a recurrent checkpoint every few thousand tokens or so. So a <500 token prompt is less interesting.)
Details
ggerganov commentedon Mar 8, 2026
Yes, it's not a bug. OpenCode mutates the prefix.
Skylux70 commentedon Mar 8, 2026
I can repeat this bug in llama web UI, so it is not only openCode. I got full prompt re-processing problem in every turn after I re-build and install the latest master branch.
yog2745 commentedon Mar 8, 2026
Having the same issue using llama.cpp latest build, it reprocesses the previous prompts all over again with each prompt I provide, incrementally increasing the amount of time needed to process after each prompt.
essentially taking minutes for time for first token output even for a simple few words prompt after a while
ggerganov commentedon Mar 8, 2026
@Skylux70 Most likely you are testing with short prompts (less than 512 tokens).
@yog2745 Provide more details: client, server command, logs
Skylux70 commentedon Mar 9, 2026
@ggerganov No I was not using short prompt, I tested from 10K to nearly 200K + tokens and verified it is happening and tried troubleshooting with Grok 4.2. It is a new bug after I built and installed the latest master branch
ggerganov commentedon Mar 9, 2026
Open an issue and provide logs, server cmd and client
WingsDrafterwork commentedon Mar 9, 2026
Why is this closed?
Is a real bug. is not due to mutated prefix, tested with WebUI, Cline.
Issue persist "forcing full prompt re-processing"
Skylux70 commentedon Mar 9, 2026
Yes, I have no idea why close this bug, it is very easy to repeat.
ggerganov commentedon Mar 9, 2026
@Skylux70 Your client mutates the prefix around token 1044. Run with
LLAMA_SERVER_SLOTS_DEBUG=1and you will see exactly which tokens are being changed.20 remaining items
gtrak commentedon Apr 9, 2026
I'm not going to disable the template fix to test again, but what I do know is I can now run more than 2 sessions concurrently without the cache blowing constantly, and I couldn't before.
hayesmason335-hub commentedon Apr 9, 2026
stragulus commentedon Apr 10, 2026
This fixed it for me as well, a real game changer now that it's actually usable. For anyone stumbling on this bug using opencode (bug in opencode I presume, not here!)
MArtyoma commentedon Apr 10, 2026
It's not an opencode problem. I'm experiencing this with the Pi agent as well (also made fix for Pi with empty user message).
ionizing-plasma commentedon Apr 16, 2026
yup, same thing in my own application. I am hesitant to add an empty user prompt after tool calls, I spent ten months getting things to work properly with exact ordering etc and I don't want to mess with it now lol.
However by adding "--checkpoint-every-n-tokens", "2048", "--ctx-checkpoints", "64" to my launch commands it seems to have improved things slightly?
deepblue96 commentedon Apr 17, 2026
let me quote what fixed it for me in the jinja template:
The fix is really simple one-line change in the template:
from:
{%- if loop.index0 > ns.last_query_index %}
to:
{%- if loop.index0 > ns.last_query_index and reasoning_content %}
Skuxblan commentedon Apr 19, 2026
If somebody still having issue for me this fix didn't work.
{%- if reasoning_content %}Resolved the issue
server: fix context checkpoint restore for hybrid/recurrent models
llama-memory-recurrent: zero freed cells' state on seq_rm
Tip-Tutaki commentedon Jun 18, 2026
Edit: Just noticed with swap: true in groups I get full prompt processing every time. With swap flag removed completely it works fine.
Ironically enough I managed to narrow down how to fix this whilst away from my PC via telegram and qwen 3.6 35b a3b MTP UD Q6KXL.
--cache-ram -1
--cache-reuse 256
llama.cpp/build/bin/llama-server
--port ${PORT}
--model llama.cpp/models/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf
--chat-template-kwargs '{"preserve_thinking":true}'
--ctx-size 131072
--no-mmap
-ot "blk..ffn_down_exps=CPU,blk..ffn_up_exps=CPU,blk..ffn_gate_exps=CPU"
--no-context-shift
--ctx-checkpoints 128
--cache-type-k q8_0
--cache-type-v q8_0
--kv-unified
--cache-reuse 256
--cache-ram -1
--cpu-strict 1
--prio 3
--prio-batch 3
--threads 8
--threads-batch 8
--no-warmup
--temp 0.4
--ubatch-size 2048
--jinja
--reasoning off
--flash-attn on
--parallel 1
--cont-batching
--top-p 0.9
--min-p 0.05
--top-k 20
--spec-type draft-mtp
--spec-draft-n-max 2
--spec-draft-p-min 0.75
--presence-penalty 0.0
--repeat-penalty 1.0
W slot update_slots: id 0 | task 4 | forcing full prompt re-processing due to lack of cache data
W slot update_slots: id 0 | task 4 | erased invalidated context checkpoint (pos_min = 13, pos_max = 13, ...)
I had previously used --cpu-moe with system default --fit-on with good performance but have since found increases from using -ot correctly.
Current pp Tok/s start at 2200-2400 and slowly degrade.
TG Tok/s range between 44 and 88.
Intel 13700k 128gb ddr5 5600mhz
Rtx 5070ti
Undeclocked mem -100
Overclocked core +133 base and boost
fix: prevent full prompt re-processing for hybrid SSM models