[allenzz-dev]

Fixes #28056 (cross-request KV-cache contamination on integrated GPUs — full root-cause analysis and probe evidence there).

What

Two commits:

1. llama : synchronize before mutating graph state in process_ubatch — the core fix. The sync before set_inputs was gated on cparams.pipeline_parallel (always false on a single GPU), and the graph-rebuild path (sched_reset + alloc_graph) had no sync at all. Since graph_compute is fire-and-forget (ggml_backend_sched_graph_compute_async) and prefill ubatches produce no outputs, llama_decode returns with compute still in flight and the next ubatch's set_inputs overwrites input tensors (k_idxs/v_idxs/kq_mask/…) the device is still reading. On integrated GPUs those inputs are read zero-copy from pinned host memory (no protective split-input copy like on dGPUs), so the race scatters in-flight K/V writes to wrong cells and leaves stale K/V from a previous request visible under the new request's mask. The fix makes the synchronize unconditional and hoists it above the can_reuse branch so it also covers the rebuild path.

2. server : do not cache-reuse on memories that cannot erase a mid-range — related hardening found while investigating. A recurrent state cannot represent a partial erase, but llama_memory_recurrent::seq_rm returned true for one while leaving the removed tokens in the state, so the --cache-reuse KV surgery appeared to succeed and stale recurrent state leaked into the new prompt positions on hybrid/recurrent models. This rejects the unrepresentable erase and gates can_cache_reuse on the context being pure attention (seq_rm type PART). Happy to split this into a separate PR if preferred.

Verification

Reproducer (AMD gfx1151 APU / Strix Halo, HIP, Qwen3.6-35B-A3B hybrid, llama-server -b 2048 -ub 2048 -fa on -np 1, deterministic sampling): chat about a distinctive topic → send an unrelated ~21k-token document + questions. Before: 4/4 responses quote the chat verbatim, and per-cell K-cache hash probes show chat-era data scattered into the document's cell range while document chunks fail to land. After: 0/12 contaminated across 3 runs, and the probed cell shows the healthy trajectory (boot value through the chat era, document data lands at its chunk and stays put).

Performance

Zero measured cost on the affected hardware — the async pipeline only queued deeper, the GPU was already saturated:

metric	before	after
21k-token prefill	1230.8 t/s	1231.8 t/s
2k-token prefill	957–970 t/s	957–966 t/s
decode	48.5 t/s	48.4 t/s

A conservative alternative would gate the sync on the integrated/zero-copy condition; unconditional measured free here and also covers any other configuration where graph inputs are device-visible without a copy.

🤖 Generated with Claude Code

[ggml-gh-bot]

Hi @allenzz-dev, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[crusaderky]

I tested this on gfx1150 rather than gfx1151 and on a third model family. It fixes the issue.

Radeon 890M (Strix Point) on Ubuntu 26.04, ROCm 10.0, HIP.
LFM2.5-VL-3B-GGUF:Q8_0, arch lfm2, hybrid short-conv + attention. -c 32768 -ngl 99 --parallel 1 --kv-unified -fa on -ctk q8_0 -ctv q8_0.

[paidaxu]

Tested on NVIDIA Jetson Orin NX (CUDA iGPU): this PR fixes the bug

I hit the same bug on an NVIDIA Jetson. #29450 says NVIDIA testing is still needed, so here are my results.

Setup

• Jetson Orin NX 16 GB, L4T R35.6.0, CUDA 11.4 (sm_87), 25 W mode
• llama.cpp: an older copy bundled with a TinyLLaVA project. It is older than cuda : Disable host buffers on integrated GPUs (#15034) #16308, so integrated = prop.integrated is still active on CUDA.
• Model: TinyLLaVA-Phi-2-SigLIP-3.1B (phi2, Q8_0) + mmproj Q8_0
• llama-cli / llama-mtmd-cli, --temp 0

Problem
With -ngl 33 (all layers on GPU), if the prompt is longer than one ubatch, the output is only GGGGGGGG... from the first token. An image prompt (~750 tokens) and a long text prompt (~700 tokens) both fail. Short prompts work.

Setting (-ngl 33 unless noted)	Output
705-token prompt, default -ub 512 (2 ubatches)	GGGG
same, -ub 1024 or 2048 (1 ubatch)	OK
same, -ngl 32	OK
same, CUDA_LAUNCH_BLOCKING=1	OK
same, GGML_CUDA_DISABLE_GRAPHS=1	GGGG
same, GGML_CUDA_FORCE_CUBLAS=1 or GGML_CUDA_FORCE_MMQ=1	GGGG
same, F16 text model	GGGG

On a 1505-token prompt the result is random-looking across -ub, like a race:

-ub	64	128	256	512	1024	2048
output	GGGG	GGGG	OK	OK	GGGG	OK

llama-eval-callback shows the same values for -ngl 33 and -ngl 32 in all 1874 ops, with no NaN/Inf. So this is not an overflow.

What I found in the scheduler

• With GGML_SCHED_DEBUG=2 and -ngl 33, the CUDA0 split has 0 inputs. GPU nodes in every layer read the graph inputs directly from the CUDA_Host buffer: inp_pos (ROPE), k_idxs / v_idxs (SET_ROWS), and inp_out_ids (GET_ROWS).
• Prefill ubatches have n_outputs = 0, so there is no sync after them. The next ubatch has a different size, so it takes the rebuild path, and set_inputs() overwrites these inputs while the GPU is still running the previous ubatch.
• -ngl 32 works only by luck. The output weight is copied to the GPU in each ubatch, and that copy calls ggml_backend_synchronize().

With this PR
I added the unconditional ggml_backend_sched_synchronize() before the can_reuse check in process_ubatch, built with the same options, and tested with -ngl 33:

• 39/39 runs OK. I tested the 705-token prompt at -ub 512, the 1505-token prompt at -ub 64/128/256/512/1024, and 2 real images at -ub 64/512, 3 times each, plus 9 speed runs.
• Without the patch, the same failing cases still give GGGG (6/6, and the image at -ub 512 failed 3/3).

Speed, median of 3:

Case	prompt before	prompt after	change	gen before	gen after
image (747 tok), -ub 512	4344 ms	4353 ms	+0.19%	7.68 t/s	7.68 t/s
text (1505 tok), -ub 512	6377 ms	6411 ms	+0.53%	7.01 t/s	7.01 t/s
text (1505 tok), -ub 2048	7629 ms	7636 ms	+0.09%	7.00 t/s	7.01 t/s

(Before the patch the image output was GGGG, so that row is timing only.)

So on Jetson, this PR fixes the bug. Prompt processing is 0.1–0.5% slower, and generation speed does not change.

I can run more tests on this board if needed.

I used an AI assistant to help with the English and the analysis. I ran all the tests myself on our lab's Jetson.