Repository navigation
spec: properly handle mtmd input for mtp - #30257
Merged
Merged
Conversation
ngxson
marked this pull request as ready for review
October 9, 2026 22:41
ggerganov
approved these changes
Oct 10, 2026
ggerganov
left a comment
ggerganov
left a comment
Member
There was a problem hiding this comment.
This also fixes GLM-5.3-Flash to not crash on image inputs when MTP is enabled.
ckti
pushed a commit
to ckti/llama.cpp-consolidated
that referenced
this pull request
Oct 12, 2026
* spec: properly handle mtmd input for mtp * nits
6 of 7 tasks
CISC
pushed a commit
that referenced
this pull request
Oct 13, 2026
3 of 5 tasks
mclrcha
added a commit
to mclrcha/llama.cpp
that referenced
this pull request
Oct 16, 2026
153 upstream commits (merge-base 1623d8c). Conflicts (6 files) and semantic fixes: - gated_delta_net.cu: fork kernels (column_lanes, indexed state, RDNA4 chunked WY prefill) kept; upstream GDN MMA path (ggml-org#29353, HIP only on RDNA3.5) runs after ours, skipped when the state gather was deferred to the op. - ggml-cuda.cu: n_stable reset kept, warmup reset log behind GGML_CUDA_GRAPH_DEBUG (ggml-org#30327). - llama-batch: upstream state embeddings for MTP hooks (ggml-org#30257) next to the fork embd alias; R57 extended to them: the allocator points at the batch state rows and a ubatch of consecutive rows reads them in place (no copies). - llama-context: upstream sched_new (realloc callback, ggml-org#30329) also used for the second MTP scheduler. - llama-graph: draft LM head and logits top-k kept next to build_hadamard_input / build_logits; the draft head applies the Hadamard input transform too. - llama-model: load_tensors_hadamard, then build_mtp_draft_head. - common: R59 full-ubatch warmup also sets zero state embeddings.
despc
added a commit
to despc/llama.cpp
that referenced
this pull request
Oct 18, 2026
Since upstream ggml-org#30257 the MTP driver hands vision tokens to the draft as embeddings. Two places still assumed token ids, and any prompt with an image aborted the server: - the deferred catch-up stored token ids only (NULL for an image), so the flush failed in get_sub_batch. Batches with embeddings are no longer deferred. - graph_mtp asserted token input. It now takes the embeddings when the ubatch has no tokens, as upstream's head does. Two image prompts on the production launcher: sensible descriptions, draft accepted 200/296 and 183/405. Text output unchanged. Assisted-by: Claude Opus 5.5
gsenkowski
pushed a commit
to gsenkowski/strix-llama.cpp
that referenced
this pull request
Oct 19, 2026
An image chunk of a multimodal prompt reaches the MTP draft context as an
embedding-only batch (no token ids). graph_mtp asserted ubatch.token and
aborted the server ("qwen4exp MTP needs token input").
Follow ggml-org/llama.cpp ggml-org#30257 and ggml-org#30397 (closes ggml-org#30307): accept either
tokens or embeddings, take the token embedding from ubatch.embd when there
are no token ids, and size the embd input by n_embd_inp() with the hidden
state at n_embd_out(), as the other MTP graphs do. The merge of ggml-org#30397 did
not reach this fork's own copy of graph_mtp.
Builds on the CPU backend; not run with a qwen4exp model or an image prompt.
Assisted-by: Claude
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019zrr5zPyKJ6SnVE7BjRcUe
NineBallo
added a commit
to NineBallo/strix-llama.cpp
that referenced
this pull request
Oct 19, 2026
An image chunk of a multimodal prompt reaches the MTP draft context as an
embedding-only batch (no token ids). graph_mtp asserted ubatch.token and
aborted the server ("qwen4exp MTP needs token input"), so every request
containing an image crashed llama-server.
Follow upstream ggml-org#30257 and the merged fix
ggml-org#30397 (closes ggml-org#30307): accept either
tokens or embeddings, take the token embedding from ubatch.embd when there
are no token ids, and size the embd input by n_embd_inp() - the width the
mtmd embeddings arrive at - with the handed-over hidden state still at
n_embd_out(), as the other MTP graphs do.
The fork's own rewrite of qwen4exp.cpp kept the assert through the
upstream syncs, so ggml-org#30397 never reached this copy of graph_mtp.
edwardyoon
added a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 25, 2026
mellum.cpp (upstream ggml-org#30612, applied cleanly during the 399-commit fast-forward) references three infra helpers that were dropped by the 124 conflicted commits. They are added here without cherry-picking the full upstream commits, which would not compile locally: - crop_before_nextn() / crop_after_nextn() on llm_graph_context (ggml-org#30017) - nextn_flags_t / nextn_flags() on llama_model_base (ggml-org#30097) - 2-arg llm_graph_input_embd_h ctor (n_embd, n_embd_state) (ggml-org#30257) The ctor is an overload, not a replacement: the 10 models that call the 1-arg form keep working because n_embd_state defaults to n_embd, so set_input() byte-for-byte matches their prior behavior. set_input() now sizes the hidden-state tensor from n_embd_state (mellum's n_embd_inp() differs from n_embd). Verified: cmake --build --target llama passes (error 0).
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
* spec: properly handle mtmd input for mtp * nits
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Follow-up #29385 and #29601
MTP heads can now properly decode mtmd input, no longer filling it with zeros
Testing: Acceptance rate with image input, Qwen3.5-4B-MTP, invoice, 160 tokens per prompt
Requirements