[dagnarf]

Fixes the multimodal breakage in DFlash2 speculative decoding (reported on ggml-org#27342: vision requests failing while native MTP works).

Repro (M5 Max, Metal, dflash2 head 5ecbe1ac1)

llama-server -m Qwen3.8-27B-BF16.gguf --mmproj mmproj-model-bf16.gguf \
  --spec-type draft-dflash --spec-draft-model Qwen3.8-27B-DFlash2-BF16.gguf \
  --spec-draft-n-max 3

Any /v1/chat/completions request containing an image returns HTTP 500:

init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 62
 - the tokens for sequence 0 in the input batch have a starting position of Y = 86
 it is required that the sequence positions remain consecutive: Y = X + 1
llama_decode: failed to decode, ret = -1
process: llama_decode(ctx_dft) failed rc=-1 (n_tokens=4, offset=0)
srv  update_slots: decode() failed: failed to process speculative batch

Root cause

mtmd image chunks are decoded into the target context by the mtmd helper, so they never pass through common_speculative_process(). The dflash draft cache is left with a positional hole spanning the image chunk, and the first text ubatch after the image fails the draft KV cache's consecutive-position check. (The process() comment already anticipates this — "skipping the embedding batches leaves a hole in the draft's cache" — but the batches aren't skipped, they simply never arrive.) The same hole can appear when the target reuses a cached prompt prefix the draft cache never saw.

Fix

Detect the per-sequence gap at the top of the dflash process() and seed it with zero-feature encoder injections. Drafted tokens are still verified by the target, so this only degrades draft acceptance right after an image — output stays distribution-exact.

A fuller fix would extract real target features for mtmd chunks (per the existing TODO referencing ggml-org#24669); this is the minimal change that stops the 500 without touching the mtmd path. The LOG_WRN on each hole could arguably be LOG_DBG for cache-reuse-heavy serving — happy to adjust.

Validation (Qwen3.8-27B BF16 + BF16 mmproj + Qwen3.8-27B-DFlash2-BF16, M5 Max)

• Image OCR requests return 200; greedy output byte-identical to a non-speculative run of the same request.
• PDF path (rasterized pages, matching the original breakage report): per-page requests transcribe exactly; a two-image single request (two mtmd chunks → two holes) also returns 200 with greedy output byte-identical to the non-speculative engine.
• Draft acceptance recovers quickly after the image (0.63–1.0 observed on transcription output) despite the zeroed image-region features — the drafter re-anchors on the text tokens flowing through its own context.
• Text-only throughput unchanged.

🤖 Generated with Claude Code

[dagnarf]

Pushed a second commit (1ebcce38f) after live use surfaced a case the first commit didn't cover: large images still failed, with a different error.

A 4000×3000 photo produces a ~3000-row image-embedding chunk that does reach process() (via the post-decode callback on the target's mtmd decode). Its rows carry M-RoPE positions — the temporal component is effectively constant across the image — which the 1D draft cache can't store. The first 2048-row ubatch happens to pass the KV cache checks (while injecting rows at bogus draft positions), and the second ubatch then fails the consecutive-position rule:

process: llama_decode(ctx_dft) failed rc=-1 (n_tokens=913, offset=2048)
slot operator(): failed to decode mtmd chunk, idx = 1755, res = 1
srv send_error: failed to process mtmd chunk

Smaller images "worked" only because their chunk fit in one ubatch — and even then the injected image rows sat at wrong positions inside the draft's sliding window.

The second commit skips embedding batches in process() entirely and lets the zero-fill pass from the first commit cover the hole when the next token batch arrives. That makes the two commits one coherent mechanism: token batches feed real features, everything else (image chunks, reused prefixes) becomes a detected hole seeded with zeros, and the target verifies every drafted token either way.

Re-validated on Qwen3.8-27B BF16 + BF16 mmproj (M5 Max): the 4000×3000 photo returns 200 with greedy output byte-identical to a non-speculative run; small-image OCR and rasterized-PDF transcription remain exact; text-only behavior unchanged.

If a target ever provides linear positions for image chunks (or after ggml-org#24669 lands), the embedding path could be re-enabled behind a position-linearity check — but skip-plus-zero-fill is the correct conservative default for M-RoPE targets today.

🤖 Generated with Claude Code