llama.cpp server: on a hybrid (recurrent) model the prefill handoff is thrown away — the decoder re-processes the whole prompt after ?action=restore or ?action=adopt #941

Note: filed here because the fork (1bit-MONSTER/llama.cpp) has issues disabled; the code lives in third_party/llama.cpp (tools/server/server-context.cpp, src/llama-memory-hybrid.cpp) and the fix is a fork PR.

Measured on strixhalo, 2026-11-03 ~21:00-22:10, bonsai2-unrot-q4_0.gguf (qwen35 27B: 48 GDN + 16 full-attention layers), engine 22d8047f, llama.cpp e0e2c26f5.

After a prefill handoff, llama-server re-processes the whole prompt. On a hybrid (recurrent) model this makes the
handoff pointless end to end, and it does it on both handoff routes, so it is not specific to the shared-region
work in #181.

(One run, the same three prompts and the same lengths on every row — the two routes were measured on separate boots of
the same build, and the 4 096 point shows the usual box-load spread between them.)

Answers are correct and identical to --device hrx on both routes; the problem is purely that the decoder throws the
handed-over state away and redoes the prefill. For scale: the handoff is 1-44 ms and the wasted prefill is 1.4-16.9 s.

Why — the server's checkpoint test fires, and a handoff installs no checkpoint. llama_memory_hybrid::seq_pos_min
returns max(attn_min, recr_min) (src/llama-memory-hybrid.cpp:172), and after a handoff the recurrent half holds one
cell at the last position, so seq_pos_min == n_past - 1 instead of 0. In launch_slot_
(tools/server/server-context.cpp:3488-3522) that makes pos_min >= pos_min_thold true, the code goes looking for a
context checkpoint, finds none (neither ?action=restore nor ?action=adopt creates one) and takes
do_reset → pos_next = 0; n_past = 0 → full re-process. On a plain llama_kv_cache model seq_pos_min == 0, the
test is false, no checkpoint is needed and the adopted cells are used (Qwen3-0.6B, same harness: prompt_n = 1).

Checked, so the diagnosis is not a guess:

• It is not "hybrid memory cannot use its prompt cache": plain llama-server on the same model and build, two
  identical /completion requests, cache_prompt=true → prompt_n = 161 then prompt_n = 4 (the checkpoint the
  server created during the first request was restored). Caching works when a checkpoint exists.
• It is not the shared-region route: the file route re-evaluates exactly the same token counts.
• It is not a tokenization mismatch: /apply-template + /tokenize of the templated text and the chat path's own
  prompt_n agree (35 tokens, same first id 248045).

Fix (added after auditing the consumers — my first draft of this section was wrong, see the note below). The
state a handoff installs is already in place, so the fix is a checkpoint that carries the position bookkeeping and
no serialized data, pushed by the handoff handlers (SERVER_TASK_TYPE_SLOT_RESTORE and SERVER_TASK_TYPE_SLOT_ADOPT)
right after the cells are in place:

auto & cur = slot->prompt.checkpoints.emplace_back();
cur.update_pos(/*n_tokens=*/ n, /*pos_min=*/ 0, /*pos_max=*/ n - 1);
// deliberately no update_tgt()/update_dft(): the state is already where a restore would put it

The audit that makes this safe rather than plausible:

• slot.prompt.checkpoints has exactly two consumers: the search here (:3493-3506) and the eviction/pruning in
  create_checkpoint (:2400-2423). Nothing else reads the list.
• A data-less entry is already a supported shape: common_prompt_checkpoint::load_tgt returns early when
  data_tgt.empty() (common/common.cpp:2178), so the search's it->load_tgt(...) becomes a no-op and the flow then
  computes pos_next = min(pos_next, max(pos_min + 1, pos_max)) and n_past from the memory as it stands — which is
  exactly right when the memory is already correct.
• The speculative rollback path uses a different object (slot.spec_ckpt, :227, filled at :3159, restored at
  :4044), so it is not affected by entries in slot.prompt.checkpoints at all.
• pos_min = 0 makes the entry eligible under both clauses of the search predicate (cur.pos_min < pos_min_thold || cur.pos_min == 0), and pos_max <= pos_next holds because the handoff covers exactly the prompt.

Update, 2026-11-03 21:55 — the obvious version of that fix was written, built and tried, and it is structurally
wrong for a recurrent model. With the in-place checkpoint pushed, the server did take the handoff (n_past came
out at 271 of 272 tokens instead of 0) and then aborted:

common/common.cpp:1538: failed to remove sequence 0 with p0=271, p1=-1
#6  common_memory::seq_rm(int, int, int) const
#7  server_context_impl::pre_decode()::{lambda(server_slot&)#5}

The server's rule that a checkpoint must leave "at least one token being processed" ([TAG_PROMPT_LOGITS]) is
implemented by removing the tail beyond n_past, and llama_memory_recurrent::seq_rm refuses a partial removal —
a recurrent state is a summary, not per-position data, so there is nothing to trim and re-running the last token would
apply it twice. So on a hybrid model an exact-state handoff cannot be followed by even one token evaluation, and
no checkpoint bookkeeping can change that. (Repro: ~/wt/zc-probes/t1_reuse_check.py; patch attempt kept at
/tmp/zc/941-inplace-checkpoint.patch, not pushed.)

The two designs that can work, both slightly bigger than a checkpoint:

• (A) leave the decoder one token. The provider prefills n-1 tokens and hands over state for [0, n-2]; the
  decoder then evaluates the last prompt token from that state — a legitimate single recurrent step, no rollback, one
  token of decoder compute. Needs the provider (tools/kvshare-prefill) and the adopt count to agree on n-1.
• (B) hand over the logits. The provider already computes the last token's logits (it reports an argmax); let it
  pass them back and have the engine sample the first token without the decoder evaluating anything. Bigger, but it
  is the only version that costs the decoder zero prompt work.

Not fixed here because it wants its own gate and it moves the prompt-cache path for every model: with a data-less
checkpoint in play, the check that matters is that a second request with a diverging prefix still produces the
same text as a fresh one (that is where a stale "in place" claim would show up), plus timings.prompt_n dropping to
1 on the 27B with the answer still identical to --device hrx.

Gate it deserves (proposed, not run): on the 27B, after a handoff the decoder's timings.prompt_n must be small
(not the prompt length) with the answer identical to --device hrx and 0 gfxhub faults; plus a prompt-cache rollback
test (second request with a diverging prefix) that must still produce identical text — that second half is the one
that catches the trap above.

Found while gating #181 (the recurrent half of the shared-region handoff); filed here because the file route has the
same defect and the fix is in this repo. Numbers and reproductions: ~/wt/zc-probes/t1_hybrid_share3.py,
logs in /tmp/zc-gate/, lane record in ~/briefs/TZEROCOPY-STATE.md (2026-11-03T21:45).