Skip to content

Eval bug: Qwen 3.5 Full prompt re-processing on every conversation turn #20225

Closed

Description

@williamtwomey

Name and Version

version: 8182 (05728db)
built with GNU 13.3.0 for Linux x86_64

same issue on release llama-b8233 and 8149

Operating systems

Linux

GGML backends

HIP

Hardware

Ryzen 9600X + Radeon AI PRO R9700

Models

Qwen3.5 27B

Problem description & steps to reproduce

Summary

Qwen 3.5 27B (hybrid attention + Mamba2/SSM architecture) forces full prompt re-processing on every conversation turn due to the recurrent memory's pos_min always exceeding the SWA-based pos_min_thold threshold. This makes multi-turn conversations unusable at scale - a 15k-token conversation takes ~8 minutes per turn instead of seconds.

Steps to Reproduce

  1. Start the server:
llama-server --host 0.0.0.0 --port 8080 \
  -m Qwen3.5-27B.Q8_0.gguf \
  -np 1 --batch-size 4096 --ubatch-size 1024 \
  -c 131072 -ngl 99 --flash-attn on -v
  1. Send a first message (works fine):
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model":"test",
  "messages":[
    {"role":"system","content":"You are a helpful assistant."},
    {"role":"user","content":"What is the capital of France?"}
  ],
  "max_tokens":20
}'
  1. Send a follow-up message (triggers full re-processing):
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model":"test",
  "messages":[
    {"role":"system","content":"You are a helpful assistant."},
    {"role":"user","content":"What is the capital of France?"},
    {"role":"assistant","content":"Paris is the capital of France."},
    {"role":"user","content":"What about Germany?"}
  ],
  "max_tokens":20
}'

Observed Behavior

Every subsequent conversation turn forces full prompt re-processing:

slot update_slots: id  0 | task 302 | n_past = 3, slot.prompt.tokens.size() = 357, seq_id = 0, pos_min = 356, n_swa = 1
slot update_slots: id  0 | task 302 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 302 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 302 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 4096, progress = 0.277714
                                      <~2 minute wait per batch at 30 tok/s>
slot update_slots: id  0 | task 302 | n_tokens = 4096, memory_seq_rm [4096, end)
slot update_slots: id  0 | task 302 | prompt processing progress, n_tokens = 8192, batch.n_tokens = 4096, progress = 0.555427

With a 15k-token conversation at 30 tok/s, each turn takes ~8 minutes because the entire prompt history is re-processed from scratch.

Expected Behavior

Only the new tokens (the latest user message + formatting) should be processed. The common prefix from previous turns should be served from cache. A typical follow-up should take seconds, not minutes.

Root Cause Analysis

The issue is in tools/server/server-context.cpp around line 2286-2378. The logic was designed for SWA (Sliding Window Attention) models but doesn't account for hybrid recurrent models:

1. pos_min_thold calculation assumes SWA semantics

const auto n_swa = std::max(1, llama_model_n_swa(model));  // returns 1 for Qwen 3.5 (no SWA)
const auto pos_min_thold = std::max(0, n_past - n_swa);    // e.g., max(0, 31-1) = 30

For the recurrent memory, pos_min from llama_memory_seq_pos_min() returns the full sequence length (e.g., 51), because recurrent state encompasses all positions. Since pos_min (51) > pos_min_thold (30), the code enters the checkpoint search/restore path.

2. Checkpoint search condition doesn't match recurrent models

const auto it = std::find_if(
    slot.prompt.checkpoints.rbegin(),
    slot.prompt.checkpoints.rend(),
    [&](const auto & cur) {
        return cur.pos_min < pos_min_thold;  // designed for SWA, not recurrent
    }
);

A recurrent checkpoint at pos_min = 66 with pos_min_thold = 30 fails this check (66 < 30 is false), so the checkpoint is considered invalid and the search fails.

3. Checkpoints not created for single-batch prompts

The checkpoint creation code (line ~2577) runs pre-decode, using pos_min/pos_max from the memory. For prompts that fit in a single batch, these values aren't available yet (the batch hasn't been decoded), so pos_min = -1 and no checkpoint is created.

4. Checkpoint erasure invalidates valid recurrent checkpoints

Even when a checkpoint is successfully created (multi-batch prompts), the erasure code at line ~2382 deletes any checkpoint with pos_min > pos_min_thold, which always applies to recurrent models:

for (auto it = slot.prompt.checkpoints.begin(); ...) {
    if (cur.pos_min > pos_min_thold) {  // always true for recurrent
        it = slot.prompt.checkpoints.erase(it);  // valid checkpoint destroyed
    }
}

5. seq_rm always fails for recurrent partial ranges

When llama_memory_seq_rm() is called with a partial range on a hybrid model, the recurrent memory cannot be partially truncated, so seq_rm returns false. The recovery code then tries to restore from a checkpoint, but the position accounting doesn't properly synchronize the recurrent state with the attention KV cache, leading to "Invalid input batch" errors on decode.

Impact

This affects all hybrid/recurrent model architectures (Qwen 3.5, Falcon-H1, Jamba, etc.) in multi-turn conversation scenarios. The issue scales with conversation length - longer conversations waste proportionally more time on redundant re-processing.

Note: the Vulkan backend does not exhibit this issue, suggesting the recurrent memory handling differs between backends.

Related

First Bad Commit

No response

Relevant log output

slot update_slots: id  0 | task 302 | n_past = 3, slot.prompt.tokens.size() = 357, seq_id = 0, pos_min = 356, n_swa = 1
slot update_slots: id  0 | task 302 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 302 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 302 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 4096, progress = 0.277714
                                      <~2 minute wait per batch at 30 tok/s>
slot update_slots: id  0 | task 302 | n_tokens = 4096, memory_seq_rm [4096, end)
slot update_slots: id  0 | task 302 | prompt processing progress, n_tokens = 8192, batch.n_tokens = 4096, progress = 0.555427

Activity

  1. NovNovikov commented on Mar 8, 2026

    @NovNovikov

    I can confirm this bug. And it actually got much worse in recent builds. Previously full reprocessing happened only 1 in 3 times, now it happens all the time.

  2. porterchild commented on Mar 8, 2026

    @porterchild

    seeing nearly the same behavior, but the cache seems to work ~50% of the time.
    was hoping #20087 would solve this, but I'm still seeing huge prompt recomputations from scratch, in Opencode.
    Caching works fine as the model takes multiple turns by itself.
    But if I add a message, it will sometimes seem to find a checkpoint to recompute from within the (default) bag of <=32, only incurring recomputing the prompt from there. But sometimes (~50% of the time) it recomputes the entire thing, after a lot of Checking checkpoint and erased invalidated context checkpoint messages.

    example output after an append-only new message:

    Details srv update_slots: all slots are idle

    srv params_from_: Chat format: peg-native
    slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.340 (> 0.100 thold), f_keep = 0.258
    srv get_availabl: updating prompt cache
    srv prompt_save: - saving prompt with length 42687, total state size = 1150.354 MiB
    srv load: - looking for better prompt, base f_keep = 0.258, sim = 0.340
    srv update: - cache state: 1 prompts, 3386.305 MiB (limits: 8192.000 MiB, 180224 tokens, 180224 est)
    srv update: - prompt 0xc0b1b7090: 42687 tokens, checkpoints: 15, 3386.305 MiB
    srv get_availabl: prompt cache update took 8833.97 ms
    slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> ?min-p -> ?xtc -> ?temp-ext -> dist
    slot launch_slot_: id 3 | task 30803 | processing task, is_child = 0
    slot update_slots: id 3 | task 30803 | new prompt, n_ctx_slot = 180224, n_keep = 0, task.n_tokens = 32329
    slot update_slots: id 3 | task 30803 | n_past = 10995, slot.prompt.tokens.size() = 42687, seq_id = 3, pos_min = 42686, n_swa = 1
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [32666, 32666] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [30918, 30918] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [22581, 22581] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [16822, 16822] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [16721, 16721] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [16160, 16160] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [15986, 15986] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [15669, 15669] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [12471, 12471] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [11725, 11725] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [11539, 11539] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [11286, 11286] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [11078, 11078] against 10994...
    slot update_slots: id 3 | task 30803 | Checking checkpoint with [10484, 10484] against 10994...
    slot update_slots: id 3 | task 30803 | restored context checkpoint (pos_min = 10484, pos_max = 10484, n_tokens = 10485, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 11078, pos_max = 11078, n_tokens = 11079, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 11286, pos_max = 11286, n_tokens = 11287, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 11539, pos_max = 11539, n_tokens = 11540, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 11725, pos_max = 11725, n_tokens = 11726, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 12471, pos_max = 12471, n_tokens = 12472, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 15669, pos_max = 15669, n_tokens = 15670, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 15986, pos_max = 15986, n_tokens = 15987, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 16160, pos_max = 16160, n_tokens = 16161, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 16721, pos_max = 16721, n_tokens = 16722, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 16822, pos_max = 16822, n_tokens = 16823, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 22581, pos_max = 22581, n_tokens = 22582, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 30918, pos_max = 30918, n_tokens = 30919, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | erased invalidated context checkpoint (pos_min = 32666, pos_max = 32666, n_tokens = 32667, n_swa = 1, size = 149.063 MiB)
    slot update_slots: id 3 | task 30803 | n_tokens = 10485, memory_seq_rm [10485, end)
    slot update_slots: id 3 | task 30803 | prompt processing progress, n_tokens = 12533, batch.n_tokens = 2048, progress = 0.387671
    slot update_slots: id 3 | task 30803 | n_tokens = 12533, memory_seq_rm [12533, end)
    slot update_slots: id 3 | task 30803 | prompt processing progress, n_tokens = 14581, batch.n_tokens = 2048, progress = 0.451019
    ... recomputes the rest of the prompt

  3. porterchild commented on Mar 8, 2026

    @porterchild

    If I'm reading this correctly https://pytorch.org/blog/hybrid-models-meet-sglang-more-than-full-attention/
    , maybe this is less a 'bug' and more a 'good cache reuse when using recurrent memory requires a bit more sophistication than simple checkpointing'? idk, just skimmed.

  4. porterchild commented on Mar 8, 2026

    @porterchild

    I can't reproduce this with a simple script.
    So maybe I'm just having Opencode issues.

    I'll leave my comment above in case I succeed in finding a reproducer. Until then, I have no confidence there's a bug.

    the cacheing behaves as expected here:
    (@williamtwomey I made the prompt a lot bigger because it seems #20087 only creates a recurrent checkpoint every few thousand tokens or so. So a <500 token prompt is less interesting.)

    Details
    #!/usr/bin/env python3
    """Qwen3.5 KV cache reuse reproducer."""
    
    import requests, json, time, random, string
    
    SERVER = "http://10.0.0.188:8080/v1/chat/completions"
    NONCE = "".join(random.choices(string.ascii_lowercase, k=12))
    
    filler = " ".join(
        f"The quick brown fox jumps over the lazy dog near the riverbank. "
        f"Paragraph {i} discusses distributed systems architecture including "
        f"consensus algorithms fault tolerance and horizontal scaling."
        for i in range(400)
    )
    
    messages = [
        {"role": "system", "content": f"You are a helpful assistant. [{NONCE}]"},
        {"role": "user", "content": filler},
    ]
    
    print(f"Nonce: {NONCE}\n")
    
    # REQ 1 — cold prompt
    t0 = time.time()
    response = requests.post(
        url=SERVER,
        headers={"Content-Type": "application/json"},
        data=json.dumps({"model": "test", "max_tokens": 200, "messages": messages}),
    )
    print(f"REQ 1 — cold prompt:    {time.time()-t0:.2f}s  (status {response.status_code})")
    
    time.sleep(2)
    
    # REQ 2 — identical (should cache hit)
    t0 = time.time()
    response = requests.post(
        url=SERVER,
        headers={"Content-Type": "application/json"},
        data=json.dumps({"model": "test", "max_tokens": 200, "messages": messages}),
    )
    print(f"REQ 2 — identical:      {time.time()-t0:.2f}s  (status {response.status_code})")
    
    # Append a plausible assistant reply + new user turn
    for i in range(0, 20):
        messages.append({"role": "assistant", "content": "This document discusses distributed systems."})
        messages.append({"role": "user", "content": "What about fault tolerance specifically?"})
    
    time.sleep(2)
    
    # REQ 3 — appended turn (should be fast, but triggers full recompute)
    t0 = time.time()
    response = requests.post(
        url=SERVER,
        headers={"Content-Type": "application/json"},
        data=json.dumps({"model": "test", "max_tokens": 200, "messages": messages}),
    )
    print(f"REQ 3 — appended turn:  {time.time()-t0:.2f}s  (status {response.status_code})")
    
    print(f"\nIf REQ 3 ≈ REQ 1, the bug is confirmed.")
    
    
  5. ggerganov commented on Mar 8, 2026

    @ggerganov
    Member

    Yes, it's not a bug. OpenCode mutates the prefix.

  6. Skylux70 commented on Mar 8, 2026

    @Skylux70

    I can repeat this bug in llama web UI, so it is not only openCode. I got full prompt re-processing problem in every turn after I re-build and install the latest master branch.

  7. yog2745 commented on Mar 8, 2026

    @yog2745

    Having the same issue using llama.cpp latest build, it reprocesses the previous prompts all over again with each prompt I provide, incrementally increasing the amount of time needed to process after each prompt.

    essentially taking minutes for time for first token output even for a simple few words prompt after a while

  8. ggerganov commented on Mar 8, 2026

    @ggerganov
    Member

    @Skylux70 Most likely you are testing with short prompts (less than 512 tokens).

    @yog2745 Provide more details: client, server command, logs

  9. Skylux70 commented on Mar 9, 2026

    @Skylux70

    @ggerganov No I was not using short prompt, I tested from 10K to nearly 200K + tokens and verified it is happening and tried troubleshooting with Grok 4.2. It is a new bug after I built and installed the latest master branch

  10. ggerganov commented on Mar 9, 2026

    @ggerganov
    Member

    Open an issue and provide logs, server cmd and client

  11. WingsDrafterwork commented on Mar 9, 2026

    @WingsDrafterwork

    Why is this closed?
    Is a real bug. is not due to mutated prefix, tested with WebUI, Cline.
    Issue persist "forcing full prompt re-processing"

  12. Skylux70 commented on Mar 9, 2026

    @Skylux70

    Yes, I have no idea why close this bug, it is very easy to repeat.

  13. ggerganov commented on Mar 9, 2026

    @ggerganov
    Member

    slot launch_slot_: id 0 | task 196737 | processing task, is_child = 0
    slot update_slots: id 0 | task 196737 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 36899
    slot update_slots: id 0 | task 196737 | n_past = 1044, slot.prompt.tokens.size() = 36767, seq_id = 0, pos_min = 36766, n_swa = 1

    @Skylux70 Your client mutates the prefix around token 1044. Run with LLAMA_SERVER_SLOTS_DEBUG=1 and you will see exactly which tokens are being changed.

  14. 20 remaining items

  15. gtrak commented on Apr 9, 2026

    @gtrak

    I'm not going to disable the template fix to test again, but what I do know is I can now run more than 2 sessions concurrently without the cache blowing constantly, and I couldn't before.

    Image
  16. hayesmason335-hub commented on Apr 9, 2026

    @hayesmason335-hub
  17. stragulus commented on Apr 10, 2026

    @stragulus

    Fix this error by adding an empty user message after each tool_call or message. This automatically creates the necessary checkpoints.

    Example for OpenCode

    This fixed it for me as well, a real game changer now that it's actually usable. For anyone stumbling on this bug using opencode (bug in opencode I presume, not here!)

  18. MArtyoma commented on Apr 10, 2026

    @MArtyoma

    Fix this error by adding an empty user message after each tool_call or message. This automatically creates the necessary checkpoints.
    Example for OpenCode

    This fixed it for me as well, a real game changer now that it's actually usable. For anyone stumbling on this bug using opencode (bug in opencode I presume, not here!)

    It's not an opencode problem. I'm experiencing this with the Pi agent as well (also made fix for Pi with empty user message).

  19. ionizing-plasma commented on Apr 16, 2026

    @ionizing-plasma

    yup, same thing in my own application. I am hesitant to add an empty user prompt after tool calls, I spent ten months getting things to work properly with exact ordering etc and I don't want to mess with it now lol.

    However by adding "--checkpoint-every-n-tokens", "2048", "--ctx-checkpoints", "64" to my launch commands it seems to have improved things slightly?

  20. deepblue96 commented on Apr 17, 2026

    @deepblue96

    let me quote what fixed it for me in the jinja template:

    The fix is really simple one-line change in the template:

    from:

    {%- if loop.index0 > ns.last_query_index %}

    to:

    {%- if loop.index0 > ns.last_query_index and reasoning_content %}

  21. Skuxblan commented on Apr 19, 2026

    @Skuxblan

    let me quote what fixed it for me in the jinja template:

    The fix is really simple one-line change in the template:

    from:

    {%- if loop.index0 > ns.last_query_index %}

    to:

    {%- if loop.index0 > ns.last_query_index and reasoning_content %}

    If somebody still having issue for me this fix didn't work.

    {%- if reasoning_content %}
    Resolved the issue

  22. Tip-Tutaki commented on Jun 18, 2026

    @Tip-Tutaki

    Edit: Just noticed with swap: true in groups I get full prompt processing every time. With swap flag removed completely it works fine.

    Ironically enough I managed to narrow down how to fix this whilst away from my PC via telegram and qwen 3.6 35b a3b MTP UD Q6KXL.

    --cache-ram -1
    --cache-reuse 256

    llama.cpp/build/bin/llama-server
    --port ${PORT}
    --model llama.cpp/models/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf
    --chat-template-kwargs '{"preserve_thinking":true}'
    --ctx-size 131072
    --no-mmap
    -ot "blk..ffn_down_exps=CPU,blk..ffn_up_exps=CPU,blk..ffn_gate_exps=CPU"
    --no-context-shift
    --ctx-checkpoints 128
    --cache-type-k q8_0
    --cache-type-v q8_0
    --kv-unified
    --cache-reuse 256
    --cache-ram -1
    --cpu-strict 1
    --prio 3
    --prio-batch 3
    --threads 8
    --threads-batch 8
    --no-warmup
    --temp 0.4
    --ubatch-size 2048
    --jinja
    --reasoning off
    --flash-attn on
    --parallel 1
    --cont-batching
    --top-p 0.9
    --min-p 0.05
    --top-k 20
    --spec-type draft-mtp
    --spec-draft-n-max 2
    --spec-draft-p-min 0.75
    --presence-penalty 0.0
    --repeat-penalty 1.0

    W slot update_slots: id 0 | task 4 | forcing full prompt re-processing due to lack of cache data
    W slot update_slots: id 0 | task 4 | erased invalidated context checkpoint (pos_min = 13, pos_max = 13, ...)

    I had previously used --cpu-moe with system default --fit-on with good performance but have since found increases from using -ot correctly.

    Current pp Tok/s start at 2200-2400 and slowly degrade.

    TG Tok/s range between 44 and 88.

    Intel 13700k 128gb ddr5 5600mhz
    Rtx 5070ti
    Undeclocked mem -100
    Overclocked core +133 base and boost

  23. added a commit that references this issue on Aug 18, 2026
    ae289f0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions