server : saving idle slots to the prompt cache stalls all requests for seconds with interleaved KV cells

Name and Version

version: 11387 (8330e96)
built with GCC 13.3.0 for x86_64-linux-gnu, CUDA backend

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server -m Qwen3-8B-Q4_K_M.gguf -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -np 8 -kvu -c 20480 -v

Problem description & steps to reproduce

With unified KV, --cache-idle-slots saves every idle slot on the server thread when a new task starts. Interleaved KV makes a 323-token sequence trigger 23,256 blocking ggml_backend_tensor_get calls. Other requests and streams wait for those saves.

Start the server with the command above.
Start 7 streaming completions and let them generate a few hundred tokens.
Send new requests. The verbose log shows each idle-slot save taking 0.7 to 0.9 s for about 300 tokens.

On an RTX 3070 Ti under WSL2, new requests waited up to 5.4 s. On an RTX 3090 running native Linux, they waited up to 2.1 s.

First Bad Commit

Likely 50e0ad08fb6906fda9ac2e256e43a4bbf9c85639 (#20993). It introduced the default-enabled loop that saves every idle slot when a new task starts. Its parent saved only the selected slot. This is identified from the source diff, not a timing bisect.

Relevant log output

Logs

20.410333 I slot process_sing: id  2 | task -1 | saving idle slot to prompt cache
20.410627 I srv   prompt_save:  - saving prompt with length 357, total state size = 26.675 MiB (draft: 0.000 MiB)
21.338846 D slot process_sing: id  2 | task -1 | __TEST_TAG_CACHE_IDLE_SLOT__
21.339045 I slot process_sing: id  3 | task -1 | saving idle slot to prompt cache

I actually already have a patch ready for this issue that merges nearby device reads. In my tests, it made idle-slot saves 3.7 to 16x faster.

Though I do already have #30077 open, so I plan to wait for that PR before opening this one. If a maintainer would prefer to review this fix sooner, let me know, and I can just open it earlier.

@ilyas-mallah

@ilyas-mallah can you raise PR for this fix?

@gaugarg-nv

Able to repro similar issue with copying out interleaved cells. Used AIPerf agent benchmark with 8 slots - and the slowdown is reproducible pretty easily.

llama-server command with nsys:
nsys.exe profile --trace=cuda,nvtx --cuda-graph-trace=node --sample=none --cpuctxsw=none --force-overwrite=false --output=".\nsight-8slots-20k-q8" llama-server.exe -m "<MODEL_PATH>\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf" --alias llama31-8b --port 8102 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -np 8 -kvu -c 20480 --verbose --log-timestamps --log-file ".\llama-server-8slots-20k-q8.log"

AI perf benchmark pointing at the endpoint:
aiperf.exe profile --url http://127.0.0.1:8102 --endpoint-type chat --model llama31-8b --tokenizer NousResearch/Meta-Llama-3.1-8B-Instruct --tokenizer-revision d10aef7999a2b5ba950ab3974312feeedbfe0b77 --synthetic-input-tokens-mean 512 --synthetic-input-tokens-stddev 0 --output-tokens-mean 512 --output-tokens-stddev 128 --concurrency 8 --request-count 64 --streaming --random-seed 42 --use-server-token-count --export-http-trace --export-level raw --extra-inputs ignore_eos:true temperature:0 seed:42 --ui none --artifact-dir ".\aiperf-8slots-20k-q8"

The AI perf benchmark keeps sending synthetic tasks to the server and after a point around 30 seconds it is forced to move an idle slot out to the CPU. Because of #20993 - it pretty much has to go evict another idle slot as well. That part is fine.

0.30.148.150 D que          post: new task, id = 701/1, front = 0

0.30.152.513 I slot get_availabl: id  0 | task 690 |  - skipping, is_processing = 1
0.30.152.517 I slot get_availabl: id  1 | task -1 |  - checking sim = 0.055 (30/546) > 0.100
0.30.152.517 I slot get_availabl: id  2 | task -1 |  - checking sim = 0.055 (30/546) > 0.100
0.30.152.518 I slot get_availabl: id  3 | task 648 |  - skipping, is_processing = 1
0.30.152.518 I slot get_availabl: id  4 | task 598 |  - skipping, is_processing = 1
0.30.152.518 I slot get_availabl: id  5 | task 530 |  - skipping, is_processing = 1
0.30.152.519 I slot get_availabl: id  6 | task 305 |  - skipping, is_processing = 1
0.30.152.519 I slot get_availabl: id  7 | task 505 |  - skipping, is_processing = 1

0.30.152.520 I slot get_availabl: id  2 | task -1 | selected slot by LRU, n_tokens = 1230, t_last = 29954079
0.30.152.520 I srv  get_availabl: updating prompt cache
0.30.152.645 I srv   prompt_save:  - saving prompt with length 1230, total state size = 81.695 MiB (draft: 0.000 MiB)
0.30.721.686 I srv  get_availabl: prompt cache update took 569.17 ms

0.30.721.866 I slot launch_slot_: id  2 | task 701 | processing task, is_child = 0

0.30.721.867 I slot process_sing: id  1 | task -1 | saving idle slot to prompt cache
0.30.721.984 I srv   prompt_save:  - saving prompt with length 1231, total state size = 81.761 MiB (draft: 0.000 MiB)
0.31.282.243 D slot process_sing: id  1 | task -1 | __TEST_TAG_CACHE_IDLE_SLOT__
0.31.282.257 I slot prompt_clear: id  1 | task -1 | clearing prompt with 1231 tokens

0.31.392.629 D slot process_toke: id  2 | task 701 | n_gen = 1, n_remaining = 774, next token:  2028 'This'
0.31.392.839 D slot process_toke: id  3 | task 648 | n_gen = 52, n_remaining = 386, next token: 84050 ' reluctance'

For slot 3, the client receives adjacent content chunks at 30.3265175 s and 31.5678839 s on the capture timeline.

No kernel activity just memCpy of fragmented kv cache of idle slot.

This is the problematic part. Zoomed in, it is just one memCpy after another when the idle slot is draining itself from GPU to CPU.
Within the inspected kernel-gap portion, 30.326518–31.461958 s:
82,827 cudaMemcpyAsync calls
Corresponding GPU transfers: 82,816 device-to-host + 11 host-to-device = 82,827
Separately: 82,861 cudaStreamSynchronize calls

The problem scales as you decrease the number of slots and keep unified-kv on.
cc: @gaugarg-nv @ORippler

Some sort of gather interleaved items -> bulk copy out of the GPU to CPU makes most sense to me (working on it).

@ilyas-mallah I will try the same bench with your PR and report numbers!

Navigation Menu

Repository navigation

server : saving idle slots to the prompt cache stalls all requests for seconds with interleaved KV cells #30252

Name and Version

Operating systems

Which llama.cpp modules do you know to be affected?

Command line

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

server : saving idle slots to the prompt cache stalls all requests for seconds with interleaved KV cells #30252

Description

Name and Version

Operating systems

Which llama.cpp modules do you know to be affected?

Command line

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Activity

ilyas-mallah commented on Oct 9, 2026

gaugarg-nv commented on Oct 16, 2026

bhargav191098 commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions