[frozenblade1224]

Overview

This PR adds support for two models - Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B. It introduces the new architecture used by both models - NemotronH_Omni_Reasoning_V3 - and implements the vision graph for the new architecture in nemotron-omni.cpp. The new NemotronOmniVL class in conversion\nemotron.py will handle the checkpoint conversion for the new models.

Additional information

Conversion

# Super 3.5
python convert_hf_to_gguf.py <path/to/super-3.5-checkpoint> --outfile super-3.5.gguf
python convert_hf_to_gguf.py <path/to/super-3.5-checkpoint> --mmproj --outfile super-3.5-mmproj.gguf

# Bolt 74B A9B
python convert_hf_to_gguf.py <path/to/bolt-74b-a9b-checkpoint> --outfile bolt-74b-a9b.gguf
python convert_hf_to_gguf.py <path/to/bolt-74b-a9b-checkpoint> --mmproj --outfile bolt-74b-a9b-mmproj.gguf

Text-only

llama-cli -m model.gguf -p "Who are you?"

Image

llama-mtmd-cli -m model.gguf --mmproj model-mmproj.gguf --image <path/to/image.jpg> -p "Describe this image."

Video (CLI)

Start llama-mtmd-cli, then enter /video with the video path and your prompt:

llama-mtmd-cli -m model.gguf --mmproj model-mmproj.gguf
/video <path/to/video.mp4>
Summarize the video and describe the key events.

Video (Server API)

Start the server and allow access to the directory containing your video:

llama-server -m model.gguf --mmproj model-mmproj.gguf --media-path <path/to/media-directory> --host 127.0.0.1 --port 8080

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, for code review and supervised authoring

[ruixiang63]

oh wait, what is this change here? We just merged lilicorr, right?

[frozenblade1224]

Not sure why it wasn't caught earlier.

In the LiliCorr PR, LiliCorr calculates the entire candidate lattice even when n_draft < 7. The head's tensors are specifically made to run attention between 56 candidates, so it seemed inadvisable to slice it. In any case, not within in the already-burgeoning scope of this PR.

For that case, doing the loop from 1 to n_block_tokens is wrong; since that will always output 7 draft tokens, even when we have set n_draft to < 7.

That hits this assert later on in llama-context.cpp:2384

GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max);

The change keeps us from passing more than n_max tokens even if the lattice is larger.

[ruixiang63]

Here as well.

[frozenblade1224]

Again, not sure how it passed by earlier!

AFAIK before starting llama-server does a dummy pass with 2 tokens and directly calls the decoder; which causes the LiliCorr head to try to make the lattice with 2 tokens. It can only make the lattice for 8 tokens currently, since we have implemented it that way in #30506.

So the dummy pass fails this assert in (dflash.cpp:664), and the server stops -

GGML_ASSERT(tokens_per_block == block_size);

[ggerganov]

@ngxson Is this good to merge on your end?

[ggerganov]

Will fast-track this to get the model supported in the v0.7.0 release today.