[Bugfix] Fix Nemotron VL image processing

[ducviet00]

Purpose

Correct image processing for Nemotron VL:

• Do not normalize images like InternVL
• Use a correct method to find_closest_aspect_ratio

Test Plan

Using this script to test vLLM

Nemotron VL test script

import requests
from transformers import AutoTokenizer
from PIL import Image

from vllm import LLM, SamplingParams


if __name__ == "__main__":
    model_path = "nvidia/Llama-3.1-Nemotron-Nano-VL-8B-V1"

    tokenizer = AutoTokenizer.from_pretrained(model_path)

    image_url = (
        "https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-VL-8B-V1/resolve/main/images/table.png"
    )
    image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
    image = image.resize((image.width * 2, image.height * 2), Image.BILINEAR)

    question = "<image>\nExtract the table in this image as HTML."

    engine_args = {
        "model": "nvidia/Llama-3.1-Nemotron-Nano-VL-8B-V1",
        "trust_remote_code": True,
        "max_model_len": 8192,
        "limit_mm_per_prompt": {"image": 1, "video": 0, "audio": 0},
    }
    llm = LLM(**engine_args)


    messages = [[{"role": "user", "content": question}]]
    prompts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)


    stop_token_ids = [tokenizer.eos_token_id]
    sampling_params = SamplingParams(temperature=0, max_tokens=8192, stop_token_ids=stop_token_ids)

    inputs = {
        "prompt": prompts[0],
        "multi_modal_data": {"image": image},
    }

    outputs = llm.generate(
        inputs,
        sampling_params=sampling_params,
    )

    for output in outputs:
        print("vLLM output:\n", output.outputs[0].text)

Test Result

On main branch: repeated nonsense:

<table>
 <tr>
    <td>Intel</td>
    <td>Intel</td>
    <td>Intel</td>
    <td>Intel</td>
    <td>Intel</td
....

This branch correctly extracted table from the image and matches HF outputs

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request correctly implements the specific image processing pipeline for Nemotron VL models, addressing a bug in the previous implementation. The changes, which include removing image normalization and adopting a model-specific method for finding the closest aspect ratio, align well with the reference implementation. My review identifies a potential ZeroDivisionError if an image with zero width or height is processed. I've suggested adding input validation to prevent this potential crash, which would improve the robustness of the new logic.

[gemini-code-assist]

The calculation of aspect_ratio can lead to a ZeroDivisionError if orig_height is 0. Similarly, the find_closest_aspect_ratio function called later can also face a division by zero if width * height (area) is zero. It's safer to validate that both orig_width and orig_height are positive at the beginning of this function to prevent potential crashes from invalid image inputs.

Suggested change

	aspect_ratio = orig_width / orig_height
	if orig_width <= 0 or orig_height <= 0:
	raise ValueError("Image dimensions must be positive.")
	aspect_ratio = orig_width / orig_height

[ducviet00]

Hi @simon-mo, could you please re-invite me to the vLLM Slack? I missed the invitation email, and it’s now expired

[DarkLight1337]

Thanks, can you add a correctness test under tests/models/multimodal/generation/test_common.py to avoid future regressions?

[ducviet00]

@DarkLight1337 unfortunately, the config can’t be auto-loaded — it requires patching tests.conftest.HfRunner. I suspect it’s the same issue as OpenGVLab/Mono-InternVL-2B

vllm/tests/models/multimodal/generation/test_common.py

Lines 439 to 440 in c583038

	# FIXME: Config cannot be loaded in transformers 4.52
	# "OpenGVLab/Mono-InternVL-2B",

Minimal reproduce script:

from transformers import AutoConfig

model_path = "nvidia/Llama-3.1-Nemotron-Nano-VL-8B-V1"

# Fails without attn_implementation
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
## TypeError: argument of type 'NoneType' is not iterable

# Works with attn_implementation explicitly set
config = AutoConfig.from_pretrained(
    model_path, trust_remote_code=True, attn_implementation="flash_attention_2"
)
print("WORK!!")

Environment:

uv pip show transformers
Name: transformers
Version: 4.55.0
Location: /data1/workspace/viet.d.hoang/vllm/.venv/lib/python3.12/site-packages
Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm
Required-by: compressed-tensors, vllm, xgrammar

I plan to add Nemotron to the Hugging Face Transformers repo soon, so no worries about that.

[DarkLight1337]

Alright let's just merge this first then, thanks