Repository navigation
Conversation
…state_read_sinfo llama-server writes its context checkpoints with LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY for both the target and the draft context. llama_kv_cache ignored the flag, so with an MTP draft model every request serialized the draft's whole KV cache (about 2 KB/token, 400-900 MB at 200K context) to the host and read it back on restore. The draft KV needs no checkpoint: common_memory::seq_rm rolls both contexts back. On a 4x CMP 170HX box serving Qwen3.8-Flash-Next at 200K context, time to first token dropped from 3.5 s to under 1 s; checkpoint rewinds still restore correctly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Hi @douyamv, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
ggml-gh-bot
Bot
added
the
draft
github-actions
Bot
removed
the
draft
|
A related case in the same function, in case you'd like to cover it here. The SWA-masked-cell filter in state_write also applies to LLAMA_STATE_SEQ_FLAGS_NONE saves, which the server's --cache-ram prompt cache uses. The prompt cache saves the slot including the generated reply, then restores only the prompt prefix. Cells masked relative to the end of the sequence still fall inside the prefix's window. With --swa-full the server sets n_swa = 0, so it never notices, and the restored prompt silently produces different logits. We hit this with Gemma 4 26B: the drift grows with the reply length, up to ~8 nats on the first token after a 755-token reply. It also reproduces on master with gemma-3-270m, text only. Applying the filter only to partial saves fixes it for us; the restore then matches slot reuse exactly:
Happy to open a separate issue with the repro script if you'd rather keep this PR focused. Our issue was introduced by #23981 |
…te_write/read) Assisted-by: Claude Opus 5
|
Fixes the same issue #29509 (it is not fully correct AFAIK) P.S. Glad to see a familiar face here. Long time. |
It's a small world indeed. :) |
|
Superseded by #30514 |
llama-serverwrites its context checkpoints withLLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLYfor the target and, whenspeculative decoding runs a draft context, for the draft context too.
llama_kv_cache::state_writeandstate_read_sinfoignored the flag, so with an MTP draft model every request serialized the draft's whole KV cache(about 2 KB per token, 400–900 MB at 200K context) to the host and read it back on restore.
A full-attention KV cache is rolled back with
seq_rm(common_memory::seq_rmalready does this for bothcontexts), so a partial checkpoint carries nothing for it. This PR returns early in both functions when the flag is
set and the cache is not SWA (
hparams.is_swa_any()), mirroring the existing gates for the indexer caches.Effect: with Qwen3.8-Flash-Next at 200K context on a 4-GPU box the time to first token of a new turn dropped from
3.5 s to under 1 s; checkpoint rewinds (edited history) still restore correctly (tested by re-sending a prompt whose
last 300 tokens differ and checking the restored position and the generated tokens).
🤖 Generated with Claude Code