Skip to content

convert_hf_to_gguf: harden NemotronHPuzzle MTP tensor remapping - #5

Closed
rcmorano with Copilot wants to merge 5 commits into
experimental-rocmfpx-branchfrom
copilot/experimental-rocmfpx-branch-another-one
Closed

rcmorano with Copilot wants to merge 5 commits into
experimental-rocmfpx-branchfrom
copilot/experimental-rocmfpx-branch-another-one

Conversation

Copilot AI commented Jul 8, 2026 •

Copy link
Copy Markdown

Overview

Nemotron Labs 3 Puzzle conversion was failing in convert_hf_to_gguf.py after weight indexing because the Puzzle path assumed only mtp.layers.{0,1}.* tensors and did not robustly handle top-level mtp.* tensors.
This change makes MTP remapping resilient to actual Puzzle checkpoint naming and to variable MTP layer counts from config.

  • MTP name handling

    • Added explicit remap support for non-layer tensors:
      • mtp.fc → model.layers.{bid}.eh_proj
      • mtp.pre_fc_norm_embedding → model.layers.{bid}.enorm
      • mtp.pre_fc_norm_hidden → model.layers.{bid}.hnorm
      • mtp.norm → model.layers.{bid}.shared_head.norm
    • Applies shared/top-level MTP tensors across all configured MTP blocks.
  • Dynamic MTP block indexing

    • Replaced fixed bid in (0, 1) assumption with bounds from len(self.mtp_block_configs).
    • Keeps layer-local remapping for eh_proj/enorm/hnorm/final_layernorm and routes remaining mtp.layers.{bid}.* tensors through the existing backbone mapping path.
  • Failure semantics

    • Improved assertion text for invalid MTP block index values to make malformed tensor names/config mismatches easier to diagnose.
# New non-layer remap path (conceptual)
remapper = {
    "mtp.fc": "model.layers.{bid}.eh_proj",
    "mtp.pre_fc_norm_embedding": "model.layers.{bid}.enorm",
    "mtp.pre_fc_norm_hidden": "model.layers.{bid}.hnorm",
    "mtp.norm": "model.layers.{bid}.shared_head.norm",
}

Additional information

The warning about trust_remote_code=True during config loading is expected for this model family and not the conversion failure root cause; the crash path was in Puzzle MTP tensor remapping during export.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES — AI was used assistively to draft and refine targeted converter changes; human review/ownership remains with the submitter. Please ensure compliance with AGENTS.md and CONTRIBUTING.md before submission.

Copilot AI and others added 2 commits July 8, 2026 23:03
Implements support for NVIDIA's NemotronHPuzzleForCausalLM architecture,
a hybrid mamba2/attention/MoE model with per-layer heterogeneous MoE
configuration and a 2-block MTP draft head (attention + MoE sub-blocks).

Key changes:
- Add per-layer n_ff_exp_arr[] and n_expert_used_arr[] to llama_hparams;
  rename scalar fields to n_ff_exp_impl/n_expert_used_impl with accessors
  n_ff_exp(il) and n_expert_used(il) for per-layer lookup with scalar fallback
- Add nextn_predict_per_head for multi-block MTP steps (Puzzle uses 2)
- Add LLM_TYPE_75B_A9B for Puzzle 75B; discriminate from 120B Super via
  per-layer top-k heterogeneity check (both have 88 trunk layers)
- Add llama_model_n_nextn_heads() API so speculative decoding counts
  invokable draft heads rather than raw nextn block count
- Add graph_mtp for NEMOTRON_H_MOE: runs the 2 MTP sub-blocks serially
  (attention blk.n_trunk + MoE blk.n_layer-1) as a single draft step;
  fix MTP context to use plain KV cache (not hybrid) for Puzzle
- Convert: add NemotronHPuzzleModel with block_configs per-layer parsing,
  FP8-dequant support, mtp.safetensors mapping onto blk.88/89, and
  fix _experts AttributeError in NemotronHModel base class
- gguf_writer: make add_expert_feed_forward_length and add_expert_used_count
  accept arrays for per-layer Puzzle encoding
- Add MTP tensor allowlist (NEXTN_EH_PROJ/ENORM/HNORM/SHARED_HEAD_NORM)
  to NEMOTRON_H_MOE in gguf constants

Co-authored-by: rcmorano <55876+rcmorano@users.noreply.github.com>
- Remove redundant `const auto & hp = hparams` alias in load_arch_tensors
- Fix lambda capture in mtp_on_hybrid_nemotron_h filter: capture n_trunk by
  value (not this), filter to il == n_trunk (attention sub-block only)
- Clarify graph_mtp GGML_ASSERT messages to say nemotron_h (not NEMOTRON_H_MOE)
- Add comment explaining num_heads usage for d_inner in NemotronHPuzzleModel

Co-authored-by: rcmorano <55876+rcmorano@users.noreply.github.com>
Copilot AI and others added 3 commits July 9, 2026 09:11
Co-authored-by: rcmorano <55876+rcmorano@users.noreply.github.com>
Co-authored-by: rcmorano <55876+rcmorano@users.noreply.github.com>
Co-authored-by: rcmorano <55876+rcmorano@users.noreply.github.com>
Copilot AI changed the title feat: add NemotronHPuzzleForCausalLM (Nemotron-3-Puzzle-75B-A9B) support convert_hf_to_gguf: harden NemotronHPuzzle MTP tensor remapping Jul 9, 2026
@rcmorano rcmorano closed this Jul 15, 2026
rcmorano pushed a commit that referenced this pull request Jul 18, 2026
* docs: add UltraQuality rankleave32 recipe

* docs: map released models to qualified recipes

---------

Co-authored-by: ciru-ai <290232805+ciru-ai@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants