[MODEL] support qwen3.5 series by JJJYmmm · Pull Request #19468 · ggml-org/llama.cpp

JJJYmmm · 2026-02-09T19:02:07Z

This PR adds model support for the upcoming Qwen3.5 models, including both dense and MoE variants. It has been verified with preview checkpoints from the Qwen Team in both vision and pure text modes (with or without mmproj file).

As mentioned here, this PR was planned to be a follow-up PR after #19435 or #19456 by @pwilkin. However, it seems there is still a lot to do without the real checkpoints (such as vision support, vocab, weight mapping, and forward logic), so I'm sharing my implementation here for reference. 🫡

Some differences compared to #19435 or #19456:

Qwen3.5 uses a different pre-tokenizer
Qwen3.5 uses partial IMRoPE
Qwen3.5 has fixed the qkvzba order—thanks to @ngxson in model: try to improve Qwen3 Next #18683. I saw it last month and we decided to simplify it in the official weights! Now they are split into qkv, z, b, and a.
Qwen3.5 has dense/MoE variants, allowing us to simplify the forward logic (no need to check the mlp type of the current layer).

@pwilkin Sorry for the work overlap, and I really appreciate your work on Qwen3Next! It has been a huge stepping stone for this PR.

Reference HF implementation: huggingface/transformers#43830

pwilkin · 2026-02-09T19:16:18Z

@JJJYmmm no worries, glad if it helped even a little :)

ngxson

I haven't reviewed the actual implementation, but I assume it's mostly qwen3 next + imrope (please correct if I'm wrong)

Don't hesitate to ping if you need helps to add the vision support!

JJJYmmm · 2026-02-09T19:48:39Z

@JJJYmmm no worries, glad if it helped even a little :)

It definitely helped a lot 🙏

ggerganov

The libllama changes are good. I will follow up with a PR to deduplicate and optimize the delta net stuff across the Qwen family.

ggerganov · 2026-02-10T08:04:06Z

+
+    // if head keys and value keys are different, repeat to force tensors into matching shapes
+    if (num_k_heads != num_v_heads) {
+        GGML_ASSERT(num_v_heads % num_k_heads == 0);
+        int64_t repeat_factor = num_v_heads / num_k_heads;
+
+        // repeat interleave: reshape to (repeat part, 1, remaining part), do repeat, then reshape back
+        ggml_tensor * q_reshaped = ggml_reshape_3d(ctx0, q_conv, head_k_dim, 1, num_k_heads * n_seq_tokens * n_seqs);
+        ggml_tensor * k_reshaped = ggml_reshape_3d(ctx0, k_conv, head_k_dim, 1, num_k_heads * n_seq_tokens * n_seqs);
+
+        // Repeat along the third dimension (the new dimension with size 1)
+        ggml_tensor * q_repeated =
+            ggml_repeat_4d(ctx0, q_reshaped, head_k_dim, repeat_factor, num_k_heads * n_seq_tokens * n_seqs, 1);
+        ggml_tensor * k_repeated =
+            ggml_repeat_4d(ctx0, k_reshaped, head_k_dim, repeat_factor, num_k_heads * n_seq_tokens * n_seqs, 1);
+
+        // Reshape back to merge the head and repeat dimensions
+        // From [head_dim, num_k_heads, repeat_factor, n_seq_tokens * n_seqs]
+        // Back to [head_dim, num_k_heads * repeat_factor, n_seq_tokens, n_seqs]
+        q_conv = ggml_reshape_4d(ctx0, q_repeated, head_k_dim, num_k_heads * repeat_factor, n_seq_tokens, n_seqs);
+        k_conv = ggml_reshape_4d(ctx0, k_repeated, head_k_dim, num_k_heads * repeat_factor, n_seq_tokens, n_seqs);
+    }
+


Btw, this is an issue inherited from the Qwen3 Next implementation. Basically we are doing redundant repeats here because the V weights are not arranged correctly during conversion. Ideally this whole section should not be needed and we should only rely on broadcasts.

Basically, we want to broadcast Q and K into V. With the current order of the V heads, we need an interleaved broadcast. However the ggml binary ops perform tiled broadcast. Hence the need for this rearrangement.

If it is too difficult to figure out how to fix this for the new Qwen3.5 models, we'll have to do it in a follow-up PR. But it's going to be a breaking change. Or maybe we can try to gate it with some flag.

I would try to fix it in the qwen3.5 gguf conversion since it is not released.

Since it's a new arch we should not need a flag as such then.

just reorder the related v weights to avoid repeat_interleave🫡 4248c93

Nice. Just make sure results are still as expected after this change.

From here, I think we will be able to remove the explicit repeats - this is something for next PR. Thanks a lot!

Yes, I tested it with the new generated weights, and it works just as before. 🫡

CISC · 2026-02-10T08:28:50Z

@JJJYmmm https://github.com/ggml-org/llama.cpp/actions/runs/21854651667/job/63068757680?pr=19468

…epeat

ngxson

The changes outside of cgraph looks good to me (sorry don't have too much time to look deeper, but I believe the review from @ggerganov and @CISC should already be enough)

Looking forward to the vision support PR then!

JJJYmmm · 2026-02-10T13:35:37Z

The changes outside of cgraph looks good to me (sorry don't have too much time to look deeper, but I believe the review from @ggerganov and @CISC should already be enough)

Looking forward to the vision support PR then!

@ngxson This pr includes the vision part and I test it too! (qwen3.5 uses the same vit as qwen3vl and deepstack was removed temporally)🤗

ngxson · 2026-02-10T13:40:04Z

@JJJYmmm ah yeah right, thanks for the clarification. Yes I already acknowledge that the deep stack is removed (confirmed by transformers implementation)

Look good to me then, thanks again!

ngxson · 2026-02-10T15:50:05Z

It seems to me that all review comments are resolved, and @JJJYmmm also confirmed that the model is still working fine with last commit. So, I'll let @ggerganov to decide when to merge this.

* support qwen3.5 series * remove deepstack for now, and some code clean * code clean * add FULL_ATTENTION_INTERVAL metadata * code clean * reorder v heads for linear attention to avoid expensive interleaved repeat

… support already landed The previous version wrongly concluded llama.cpp had no vision support for Qwen3.5 at all, citing a reverted early PR (#19435) and misreading issue #19917 as an open capability gap. Verified directly against the primary sources this time: - ggml-org/llama.cpp#19468 (merged 2026-02-10) explicitly includes and tests Qwen3.5 vision support. - Issue #19917 is closed -- it reports the standard 'missing --mmproj' hint text, a usage error, not a missing feature. - bartowski/ATH-MaaS_OvisOCR2-GGUF ships a matching mmproj for this exact 0.8B checkpoint. Its GGUF general.architecture is 'qwen35', already in ARCH_TO_FAMILY -- the existing mmproj-upgrade path already promotes it to a vision family with zero new detection code. - The mmproj-upgrade default (lfm2_vl) forces JSON-schema output, wrong for this model's free-text Markdown OCR; vision_ocr is the right family pick -- an operator choice at deploy, not a new family. - The one real llama.cpp correctness bug in this arch's history (#19683) is closed, CUDA-only, and MoE-only -- irrelevant to CPU dense-model serving. Added RUNTIME_NOTES['qwen35'] (cites #19468, warns the mmproj must come from the same checkpoint size -- projector output dimension differs across Qwen3.5 variants) plus a test covering both the plain text and mmproj-upgraded vision verdicts. The two transformers-runtime fixes from the previous commit are unaffected -- they were found while chasing the (ultimately wrong) trust_remote_code theory, but are real, general fixes independent of which family this specific model turned out to need.

…m-code VLMs (#252) * fix(transformers): correct vision detection + honest errors for custom-code VLMs Investigating support for ATH-MaaS/OvisOCR2 (an Ovis-family OCR model on a Qwen3.5 base) surfaced two real gaps in the last-resort transformers serving path, independent of that specific model: - HfTransformersBackend inferred vision capability from which AutoModel constructor happened to succeed (AutoModelForImageTextToText vs a AutoModelForCausalLM fallback). A custom-code VLM whose auto_map registers only under AutoModelForCausalLM (a common, older pattern — the Ovis family included) was silently misclassified as text-only even though its processor is genuinely multimodal. Fixed by deriving vision from the loaded processor's own shape (does it carry an image_processor, the standard ProcessorMixin composition every VLM processor uses regardless of which AutoModel class loaded it). - The chat-completions route only caught ValueError from a backend failure; any other exception (an AttributeError/TypeError from a custom model whose calling convention diverges from the standard apply_chat_template(images=...) contract this shim assumes generically) reached the caller as an unhandled 500 with no actionable message. Now every backend exception comes back OpenAI-shaped, with the real error text. Added a case study to docs/model-onboarding.md documenting why OvisOCR2 needs the transformers_trust_remote_code path rather than a GGUF or a new family: Qwen3.5's hybrid Gated DeltaNet architecture has llama.cpp text-only support as of a few weeks ago (with an open GPU correctness bug), and Ovis's actual vision stack has no llama.cpp implementation at all — so every community GGUF quantization of this repo can only be the text backbone, never a working OCR model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(docs): correct the OvisOCR2 architecture case study The previous version assumed, from the product name alone, that ATH-MaaS/OvisOCR2 shared AIDC-AI's unrelated 'Ovis' architecture (custom AIMv2 vision tower, needing trust_remote_code). The repo's actual config.json (fetched directly) shows architectures: ['Qwen3_5ForConditionalGeneration'], model_type 'qwen3_5', a native vision_config, and no auto_map -- this is ATH-MaaS's own fine-tune of Qwen's native Qwen3.5-VL, not AIDC-AI's Ovis. Corrected conclusion: no trust_remote_code needed; the real gate is transformers version currency -- Qwen3_5ForConditionalGeneration (confirmed as a real, ~2000-line, standard-registration implementation) lives only on transformers' unreleased main branch as of this writing. The llama.cpp vision gap conclusion is unchanged and now more precisely sourced: ggml-org/llama.cpp#19917 ("Qwen3.5 image input is not supported") is a direct match for this exact vision_config, not an inference about an unrelated architecture. The two code fixes in the previous commit (processor-shape vision detection, honest backend error surfacing) are unaffected -- they are general fixes, not specific to which architecture family this model turned out to be. * fix(docs): OvisOCR2 is a rung-1 model, not rung-3 -- llama.cpp vision support already landed The previous version wrongly concluded llama.cpp had no vision support for Qwen3.5 at all, citing a reverted early PR (#19435) and misreading issue #19917 as an open capability gap. Verified directly against the primary sources this time: - ggml-org/llama.cpp#19468 (merged 2026-02-10) explicitly includes and tests Qwen3.5 vision support. - Issue #19917 is closed -- it reports the standard 'missing --mmproj' hint text, a usage error, not a missing feature. - bartowski/ATH-MaaS_OvisOCR2-GGUF ships a matching mmproj for this exact 0.8B checkpoint. Its GGUF general.architecture is 'qwen35', already in ARCH_TO_FAMILY -- the existing mmproj-upgrade path already promotes it to a vision family with zero new detection code. - The mmproj-upgrade default (lfm2_vl) forces JSON-schema output, wrong for this model's free-text Markdown OCR; vision_ocr is the right family pick -- an operator choice at deploy, not a new family. - The one real llama.cpp correctness bug in this arch's history (#19683) is closed, CUDA-only, and MoE-only -- irrelevant to CPU dense-model serving. Added RUNTIME_NOTES['qwen35'] (cites #19468, warns the mmproj must come from the same checkpoint size -- projector output dimension differs across Qwen3.5 variants) plus a test covering both the plain text and mmproj-upgraded vision verdicts. The two transformers-runtime fixes from the previous commit are unaffected -- they were found while chasing the (ultimately wrong) trust_remote_code theory, but are real, general fixes independent of which family this specific model turned out to need. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

support qwen3.5 series

9c94c40

JJJYmmm requested review from CISC and ggerganov as code owners February 9, 2026 19:02

pwilkin mentioned this pull request Feb 9, 2026

[Model] Qwen3.5 support w/o vision, WIP #19456

Closed

pwilkin mentioned this pull request Feb 9, 2026

Migrate Qwen3.5 to IMROPE #19443

Closed

ngxson reviewed Feb 9, 2026

View reviewed changes

Comment thread convert_hf_to_gguf_update.py Resolved

Comment thread src/llama-model.cpp Resolved

Comment thread src/llama-model.cpp Resolved

remove deepstack for now, and some code clean

7512ed7

CISC reviewed Feb 9, 2026

View reviewed changes

Comment thread gguf-py/gguf/tensor_mapping.py Resolved

Comment thread gguf-py/gguf/constants.py Resolved

Comment thread src/llama-model.cpp Resolved

Comment thread src/models/qwen35.cpp Resolved

Comment thread src/models/qwen35.cpp Resolved

CISC reviewed Feb 9, 2026

View reviewed changes

Comment thread convert_hf_to_gguf.py Resolved

Comment thread convert_hf_to_gguf.py Resolved

github-actions Bot added model Model specific examples python python script changes labels Feb 9, 2026

github-actions Bot mentioned this pull request Feb 10, 2026

Reddit News Daily 2026-02-10 gitlawr/reddit-daily-news#151

Open

JJJYmmm added 2 commits February 10, 2026 14:13

code clean

1af4a79

add FULL_ATTENTION_INTERVAL metadata

ac7a89f

ggerganov approved these changes Feb 10, 2026

View reviewed changes

mudler mentioned this pull request Feb 10, 2026

Adding Support for Qwen3.5 mudler/LocalAI#8469

Closed

ggerganov reviewed Feb 10, 2026

View reviewed changes

CISC approved these changes Feb 10, 2026

View reviewed changes

Comment thread convert_hf_to_gguf.py Resolved

Comment thread convert_hf_to_gguf.py Resolved

Comment thread convert_hf_to_gguf.py Resolved

Comment thread src/llama-arch.cpp Resolved

Merge branch 'master' into add_qwen35_open

036f602

JJJYmmm added 2 commits February 10, 2026 16:39

code clean

633436f

reorder v heads for linear attention to avoid expensive interleaved r…

4248c93

…epeat

ngxson approved these changes Feb 10, 2026

View reviewed changes

ggerganov merged commit fc0fe40 into ggml-org:master Feb 10, 2026
79 of 82 checks passed

ggerganov mentioned this pull request Feb 12, 2026

ggml: add GATED_DELTA_NET op #19504

Merged

mythikal03 mentioned this pull request Feb 17, 2026

Eval bug: qwen35moe produces static degenerate output (all "/" tokens) with CUDA, works on CPU-only #19683

Closed

aropb mentioned this pull request Feb 24, 2026

[Feature]: Qwen3.5 support SciSharp/LLamaSharp#1340

Closed

Josh-XT mentioned this pull request Feb 25, 2026

Update for Qwen3.5 Support xorbitsai/xllamacpp-legacy#113

Closed

bereilhp mentioned this pull request Mar 5, 2026

Update llama.cpp to support Qwen 3.5 architecture lgrammel/llama-cpp-provider#26

Closed

fkroener mentioned this pull request Mar 13, 2026

Eval bug: AMD Vulkan vk::DeviceLostError crash sensitive to ubatch-size and context length #20515

Closed

Dnonmi mentioned this pull request Mar 15, 2026

Eval bug: Vulkan Performance Regression, slower than CPU #20603

Closed

unamedkr mentioned this pull request Apr 13, 2026

feat: Qwen3.5-4B full support — hybrid DeltaNet + partial RoPE quantumaikr/quant.cpp#94

Closed

baylitoo mentioned this pull request Aug 26, 2026

fix(transformers): correct vision detection + honest errors for custom-code VLMs baylitoo/slm_benchmark#252

Merged

UnkyMunky mentioned this pull request Oct 5, 2026

Eval bug: Qwen3.6-35B-A3B (qwen35moe) cannot read text in images — sees shapes/colors, misreads letters/digits; same on CPU, all quants, all projectors, latest master #29970

Closed

Conversation

JJJYmmm commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Feb 9, 2026

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JJJYmmm commented Feb 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

ggerganov Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

JJJYmmm Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

CISC Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

JJJYmmm Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

JJJYmmm Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CISC commented Feb 10, 2026

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

JJJYmmm commented Feb 10, 2026

Uh oh!

ngxson commented Feb 10, 2026

Uh oh!

ngxson commented Feb 10, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JJJYmmm commented Feb 9, 2026 •

edited

Loading