fix(export): [NVBug 6525534] preserve nested VLM namespaces - #2032
Merged
Conversation
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
📝 WalkthroughWalkthroughThe change filters shadowed reverse weight-renaming rules using registered module namespaces, preventing nested VLM sibling capture and double-prefixing while preserving text-only nested mappings. Regression tests cover both namespace configurations, and the changelog records the fix. ChangesReverse rename scoping
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2032 +/- ##
===========================================
+ Coverage 66.87% 77.91% +11.04%
===========================================
Files 519 519
Lines 59101 59123 +22
===========================================
+ Hits 39521 46066 +6545
+ Misses 19580 13057 -6523
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
meenchen
changed the title
meenchen
marked this pull request as ready for review
July 29, 2026 19:48
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Warning
CodeRabbit couldn't request changes on this pull request because it doesn't have sufficient GitHub permissions.
Please grant CodeRabbit Pull requests: Read and write permission and re-run the review.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CHANGELOG.rst`:
- Around line 88-89: Convert the nested submodel reverse-mappings entry in
CHANGELOG.rst into a separate bullet by adding the changelog list marker at its
start, keeping the existing text unchanged.
In `@tests/unit/torch/export/test_quant_aware_conversion.py`:
- Around line 269-270: Document the optional Transformers dependency before both
local WeightRenaming imports in
tests/unit/torch/export/test_quant_aware_conversion.py at lines 269-270 and
297-298. Add a brief rationale stating that Transformers is optional and the
test is skipped when unavailable; no other import behavior needs to change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 26e031e9-4198-4140-8c3c-f8eaa03b1b2d
📒 Files selected for processing (3)
CHANGELOG.rstmodelopt/torch/export/quant_aware_conversion.pytests/unit/torch/export/test_quant_aware_conversion.py
Comment on lines
+88
to
+89
| Nested submodel reverse mappings are now scoped against registered model namespaces, preventing text-only mappings from capturing an already nested VLM's ``model.visual.*`` namespace or double-prefixing ``model.language_model.*`` (observed on Qwen3.5). | ||
|
|
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Make this a separate changelog bullet.
Without the - marker, this text continues the preceding bug-fix entry instead of creating the new item described by the PR.
Suggested fix
- Nested submodel reverse mappings are now scoped against registered model namespaces, preventing text-only mappings from capturing an already nested VLM's ``model.visual.*`` namespace or double-prefixing ``model.language_model.*`` (observed on Qwen3.5).
+- Nested submodel reverse mappings are now scoped against registered model namespaces, preventing text-only mappings from capturing an already nested VLM's ``model.visual.*`` namespace or double-prefixing ``model.language_model.*`` (observed on Qwen3.5).📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| Nested submodel reverse mappings are now scoped against registered model namespaces, preventing text-only mappings from capturing an already nested VLM's ``model.visual.*`` namespace or double-prefixing ``model.language_model.*`` (observed on Qwen3.5). | |
| - Nested submodel reverse mappings are now scoped against registered model namespaces, preventing text-only mappings from capturing an already nested VLM's ``model.visual.*`` namespace or double-prefixing ``model.language_model.*`` (observed on Qwen3.5). |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@CHANGELOG.rst` around lines 88 - 89, Convert the nested submodel
reverse-mappings entry in CHANGELOG.rst into a separate bullet by adding the
changelog list marker at its start, keeping the existing text unchanged.
Comment on lines
+269
to
+270
| pytest.importorskip("transformers.core_model_loading") | ||
| from transformers.core_model_loading import WeightRenaming |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Document the optional dependency for both local imports.
The conditional placement is valid, but each local import needs a brief comment explaining that Transformers is optional and the test is skipped when unavailable. As per coding guidelines and path instructions, optional in-function imports require this justification.
- tests/unit/torch/export/test_quant_aware_conversion.py#L269-L270: add the rationale before the
WeightRenamingimport. - tests/unit/torch/export/test_quant_aware_conversion.py#L297-L298: add the same rationale before the
WeightRenamingimport.
📍 Affects 1 file
tests/unit/torch/export/test_quant_aware_conversion.py#L269-L270(this comment)tests/unit/torch/export/test_quant_aware_conversion.py#L297-L298
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/unit/torch/export/test_quant_aware_conversion.py` around lines 269 -
270, Document the optional Transformers dependency before both local
WeightRenaming imports in tests/unit/torch/export/test_quant_aware_conversion.py
at lines 269-270 and 297-298. Add a brief rationale stating that Transformers is
optional and the test is skipped when unavailable; no other import behavior
needs to change.
Sources: Coding guidelines, Path instructions
cjluo-nv
approved these changes
Jul 29, 2026
cjluo-nv
left a comment
cjluo-nv
left a comment
Collaborator
There was a problem hiding this comment.
Bot review (gpt-5.6-sol) — DM the bot to share feedback.
The fix is focused and appears correct. It detects broad reverse prefix renames whose destination is already an explicitly registered child namespace, preventing Qwen3.5 VLM sibling capture and duplicate language_model nesting while retaining the rename for standalone text models. The two regression tests cover both sides of that behavior, and the changelog is updated. No licensing text is changed.
Contributor
Author
|
/ok to test 362c1e4 |
kevalmorabia97
added
the
cherry-pick-done
kevalmorabia97
added a commit
that referenced
this pull request
Aug 4, 2026
#2041 #2031 #2061 #1983 #1628 (#2062) ## Cherry-picked PRs - #2022 - #2026 - #2032 - #1981 - #2010 - #2043 - #2042 - #2038 - #2050 - #2041 - #2031 - #2061 - #1983 - #1628 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added recipe-driven ONNX quantization and export for embedding and reranking models. * Added launcher support for inline commands, package requirements, and configurable Docker users. * Added QAD workflows, Nemotron data recipes, pruning, quantization, and evaluation examples. * Added evaluation accuracy and pruning score thresholds. * **Bug Fixes** * Improved ONNX control-flow conversion, compressed NVFP4 export, dataset loading, and multi-GPU DiffusionGemma handling. * **Documentation** * Updated CUDA 12/13 installation guidance and vLLM Blackwell deployment instructions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: vipandya <vipandya@nvidia.com> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com> Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com> Signed-off-by: Meng Xin <mxin@nvidia.com> Signed-off-by: Chenjie Luo <chenjiel@nvidia.com> Signed-off-by: Chenjie Luo <108829653+cjluo-nv@users.noreply.github.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com> Signed-off-by: Gwena Cunha <4861122+gcunhase@users.noreply.github.com> Signed-off-by: Gwenaelle Cunha Sergio <gcunhasergio@nvidia.com> Co-authored-by: vishalpandya1990 <vishalpandya1990@gmail.com> Co-authored-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Wei-Ming Chen <17592131+meenchen@users.noreply.github.com> Co-authored-by: Ajinkya Rasane <131806219+ajrasane@users.noreply.github.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: mxinO <164952785+mxinO@users.noreply.github.com> Co-authored-by: Chenjie Luo <108829653+cjluo-nv@users.noreply.github.com> Co-authored-by: Juhi Mittal <39641197+juhi10071998@users.noreply.github.com> Co-authored-by: Gwena Cunha <4861122+gcunhase@users.noreply.github.com>
yueshen2016
added a commit
that referenced
this pull request
Aug 5, 2026
The bug is filed against 0.46.0 (Committed_ModelOpt_0.46.0) and the fix targets that release, so the entry belongs in the 0.46 section -- next to the related #2032 nested-VLM-namespace fix -- not 0.47. Also notes that Gemma3ForConditionalGeneration is affected identically and fixed by the same change, as found by the cross-architecture regression sweep. Signed-off-by: James Shen <yueshen@nvidia.com>
yueshen2016
added a commit
that referenced
this pull request
Aug 6, 2026
…me (NVBug 6525511) (#2076) ## What does this PR do? **Type of change:** Bug fix Fixes [NVBug 6525511](https://nvbugspro.nvidia.com/bug/6525511) / [OMNIML-5599](https://jirasw.nvidia.com/browse/OMNIML-5599): FP8 PTQ of `llava-1.5-13b` on `transformers>=5.12` produces a checkpoint vLLM refuses to load: ``` ValueError: There is no module or parameter named 'vision_model' in LlavaForConditionalGeneration ``` A cross-architecture sweep run for this PR shows **`google/gemma-3-4b-it` is broken the same way** (884/884 keys mangled) and is fixed by the same change, though no bug was filed for it. ### Root cause transformers collects conversion mappings **recursively** and tags each sub-model's transforms with the sub-module path they belong to, then matches only keys under that prefix (`WeightTransform._scoped_match`: strip `scope_prefix` → match → re-attach): ```python transform.scope_prefix = scope_prefix transform.base_model_prefix = model.base_model_prefix ``` `LlavaForConditionalGeneration` therefore carries the **vision tower's own** `PrefixChange` — "add a `vision_model.` prefix" — scoped to `model.vision_tower`: ``` [5] rev=PrefixChange scope_prefix = 'model.vision_tower' source_patterns = ['^(?:(?!vision_model\.))(.+)$'] target_patterns = ['vision_model.\1'] ``` `_build_reverse_rules` read `rev.source_patterns` / `rev.target_patterns` raw and **discarded `scope_prefix`**. Read as an unscoped regex, that pattern means "any key not already starting with `vision_model.`" — i.e. everything. All 758 llava-1.5-13b tensors were moved under a bogus top-level `vision_model.`: ``` model.language_model.layers.0.self_attn.q_proj.weight -> vision_model.model.language_model.layers.0.self_attn.q_proj.weight lm_head.weight -> vision_model.lm_head.weight ``` This is the same class of defect as #2032 (NVBug 6525534), but that fix's shadowing heuristic cannot catch it: it drops a rule whose target is an **existing** namespace, whereas this rule **invents** one (`vision_model` exists nowhere in the module tree). ### The fix `RenameRule` carries `scope_prefixes`; `_sub_scoped` applies a scoped rule only to keys under one of them — stripping the prefix before the match and re-attaching after, mirroring transformers' own semantics (trying `base_model_prefix + scope_prefix` before `scope_prefix`). The same scoping flows through `build_reverse_name_mapper`, so `exclude_modules` (which lists the BF16 vision tower) stays aligned with the weights. `_drop_shadowed_prefix_renames` skips scoped rules, which are already confined to their subtree. Converter-derived rules (`_expert_leaf_renames`, `_dense_split_rule`) match by **module suffix** rather than an anchored pattern, so they cannot be confined to a subtree the same way. A survey of 9 architectures (LLaVA, LLaVA-Next, Gemma-3, Gemma-4, Qwen2-VL, Qwen3-VL-MoE, Llama-4-Scout, Mixtral, DeepSeek-V2-Lite) found **every `WeightConverter` has `scope_prefix=None`** — transformers only scopes `WeightRenaming`/`PrefixChange` — so the case is unreachable today. Rather than emit rules that could silently reach a sibling namespace if that ever changes, a scoped converter now raises `QuantConversionUnsupportedError` and the caller falls back to in-memory names with a warning. ## Testing **Name-level oracle against the real hub checkpoint.** Exported names must equal the original checkpoint's keys. All 758 `llava-hf/llava-1.5-13b-hf` state-dict keys round-trip exactly: ``` MISSING (hub key not produced): 0 SPURIOUS (name not in hub) : 0 exported: {language_model: 363, vision_tower: 391, multi_modal_projector: 4} hub : {language_model: 363, vision_tower: 391, multi_modal_projector: 4} ``` Pre-fix, 758/758 keys were mangled. This check involves no vLLM. **Cross-architecture regression sweep** — old (unscoped) vs new (scoped) mapping over each model's real state dict: | Result | Models | |---|---| | Identical (no behavior change) | Llama-3.2, Qwen2.5, Qwen3, Mistral, Phi-3, SmolLM2, gpt-oss-20b, DeepSeek-V2-Lite, Mixtral-8x7B, gemma-2-2b, gemma-4-31B, Qwen2-VL-2B | | Differs (fixed) | llava-1.5-7b (686/686), gemma-3-4b-it (884/884) | Note Qwen2-VL carries a scoped rule yet is unchanged — the fix only bites where a scoped rule would have wrongly matched. **End-to-end**, on the exact image from the bug report (`vllm/vllm-openai:v0.26.0`, verified `vllm.__version__ == 0.26.0`, transformers 5.14.1): real FP8 PTQ (`general/ptq/fp8_default-kv_fp8_cast`, `--calib_size 512`) on llava-1.5-13b, then served with the bug's exact `api_server` command. ``` PTQ_EXIT=0 top-level namespaces: {'language_model': 923, 'multi_modal_projector': 4, 'vision_tower': 391} KEYS UNDER BOGUS vision_model.* : 0 occurrences of "no module or parameter named 'vision_model'": 0 Loading weights took 5.58 seconds Model loading took 13.2 GiB and 7.45 seconds ``` The reported failure is gone and weight loading completes. - `pytest tests/unit/torch/export` → **116 passed, 1 skipped** (3 new regression tests) - `pre-commit run --files ...` → all hooks pass ## Additional Information **Out of scope, for whoever picks up the QA ticket.** After this fix the bug's exact repro hits a *different* error: llava-1.5-13b ships `"dtype": "float16"` and vLLM's FP8 kernel requires BF16 output (`RuntimeError: For FP8 input, output must have dtype BF16`). This is not an export defect — it also occurs with `kv_cache_dtype=auto`, i.e. with the FP8 KV cache entirely out of the picture, and the same checkpoint loads and generates correctly under `--dtype bfloat16` (`" Paris. with a population of about 2,249,03"`). QA will need `--dtype bfloat16`. **[NVBug 6525597](https://nvbugspro.nvidia.com/bug/6525597)** (`gemma-4-31B-it`, `assert layer.k_scale > 0.0`) is unrelated to this PR: Gemma4 exposes zero transformers conversions, so this change provably does not touch it (confirmed identical in the sweep above). Re-tested separately on vLLM 0.26.0 with that bug's environment (transformers 5.5.0, TP=1, batch_size 8) it did not reproduce — the server reached `Application startup complete` with zero asserts — but that is tracked outside this PR. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - Did you write any new necessary tests?: ✅ - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved Hugging Face exports for multimodal models with nested prefix conversions. - Ensured vision-tower prefixes apply only within the correct model scope. - Prevented unrelated language-model and head parameters from being incorrectly renamed. - Preserved scoped behavior for module mappings and wildcard exclusions. - Added safer handling for unsupported scoped weight conversions. - **Tests** - Added regression coverage for scoped prefix handling across weights, module names, exclusions, and unsupported conversion scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: James Shen <yueshen@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Type of change: Bug fix
Prevents recursively collected text-submodel reverse mappings from rewriting an already nested multimodal model namespace during unified Hugging Face export.
Transformers reverses the Qwen3.5 text mapping into a broad
^model.->model.language_model.rename. ModelOpt previously applied that rule to every key in the full VLM state dict, movingmodel.visual.*under the language model and nestingmodel.language_model.*twice. This change drops the reverse rule only when its target child namespace is already registered. Standalone text models continue to use the conversion.Usage
# Existing export_hf_checkpoint usage is unchanged.Testing
python -m pytest -q tests/unit/torch/export(112 passed,1 skippedbecause optional Diffusers is not installed)Qwen3_5MoeForConditionalGenerationmeta-device model-tree probe passedpython -m pre_commit run --files modelopt/torch/export/quant_aware_conversion.py tests/unit/torch/export/test_quant_aware_conversion.pyBefore your PR is "Ready for review"
Make sure you read and follow Contributor guidelines and your commits are signed (
git commit -s -S).Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded
trust_remote_code=True,torch.load(..., weights_only=False),pickle, etc.).CONTRIBUTING.md: N/AAdditional Information
No API or dependency changes.
Summary by CodeRabbit
Bug Fixes
Tests