[bitalov]

This PR adds support for K2 Horizon, both the dense and MoVA variants. It covers HF-to-GGUF conversion, tokenizer support, model loading, and the inference graph. Everything is built on existing GGML operators, so there are no new kernels.

Along the way, a few supporting changes were needed:

• Numeric indexing in selectattr and rejectattr. The K2 chat template processes tool schemas with dict | items | rejectattr('0', 'equalto', '$ref'), which needs to reach the first element of each key/value pair. The existing filters couldn't do that, so I added support for it.

• A dedicated K2 chat parser. The auto-parser figures out reasoning and tool-call formats by rendering sample conversations. Some of those samples include assistant turns without reasoning_content, and the K2 template rejects them, so discovery fails. The dedicated parser handles K2's reasoning and tool calls, its end-of-turn marker, and the case where tool_choice=required jumps straight to a tool call.

• Template capability detection in common/jinja/caps.cpp. The same missing field can make capability probes fail, which then wrongly reports that tools, object arguments, or parallel calls aren't supported. Now, if a probe fails, it retries once with an empty reasoning_content added to any assistant turns that lack it. This does touch shared capability detection, but when I compared results across 75 templates, only the K2 templates changed.

• Multi-GPU support with -sm tensor. Value-expert weights follow the same split rule as the value weights. Q/K normalization weights are loaded per head, and the MoVA expert sum uses 3D views so the tensor-parallel backend can keep track of the split. None of this requires a change to the GGUF format.

• MoVA save/reload support. The saver now writes the value-expert metadata needed to load saved models back in.

AI usage disclosure: YES - Codex reviewed the changes, made targeted fixes, and ran additional validation.

[ggml-gh-bot]

Hi @bitalov, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[bitalov]

I think I found what's going on (with help from Claude). K2's group norm starts with a reshape, so the view it creates still points to the embeddings, which are on the CPU. That view then ends up in a Meta split, and since the backend expects a Meta buffer there, it asserts.
Adding a ggml_cont() before the reshape fixes it ofc, but I'm not sure it's the right fix. I might be wrong, but I'd expect the scheduler or the Meta backend to handle the transfer and the view, rather than the model code having to deal with it explicitly.
CC @CISC

[CISC]

I think I found what's going on (with help from Claude). K2's group norm starts with a reshape, so the view it creates still points to the embeddings, which are on the CPU. That view then ends up in a Meta split, and since the backend expects a Meta buffer there, it asserts. Adding a ggml_cont() before the reshape fixes it ofc, but I'm not sure it's the right fix. I might be wrong, but I'd expect the scheduler or the Meta backend to handle the transfer and the view, rather than the model code having to deal with it explicitly.

Yeah, this is not the right fix, though, not sure what would be right myself, @ggerganov?

[ggerganov]

Which part of the graph is that?

[bitalov]

Which part of the graph is that?

It happens right at the start of the first layer, before the Q/K/V projections even run. On line 241, the embeddings get passed into the group RMS norm:

cur = k2_horizon_group_rms_norm(ctx0, inpL, layer.attn_norm, hparams.n_norm_groups, hparams.f_norm_rms_eps);

The first thing that helper does is this reshape on line 130:

cur = ggml_reshape_3d(ctx, cur, n_embd / n_groups, n_groups, n_tokens);

[ggerganov]

Hm, not sure. @JohannesGaessler how should we handle CPU buffers reaching the meta backend?

[JohannesGaessler]

I would expect this to be a bug in the ggml backend scheduler. It should not be possible to reach that assert with a tensor allocated on a buffer for which ggml_backend_supports_buft evaluates to false. And ggml_backend_meta_device_supports_buft is implemented in such a way that it should pretty clearly return false for a CPU buffer.

In any case, for this PR I would suggest that you simply add the new arch to llm_arch_supports_sm_tensor in llama-arch.cpp. It seems unlikely to me that this is a bug introduced with the new model arch and the fix should be in a standalone PR.