[planetf1]

Overview

GGUF files converted from newer Granite Switch checkpoints fail to load with the error:

E llama_model_load: error loading model: check_tensor_dims: tensor 'blk.40.attn_norm.weight' not found

The conversion itself succeeds and writes every real decoder layer.

Unfortunate one metadata value is wrong: GraniteSwitchModel assumes the checkpoint's weightless switch owns exactly one KV cache slot and sets block_count = num_hidden_layers - 1.

The newer switch generation (MultiSwitch) owns two slots, so num_hidden_layers counts real layers plus two, and the stored graniteswitch.block_count comes out one too high. So the loader asks for a block that was never written.

This PR derives the block count from the model.layers.N tensors actually present in the checkpoint instead of assuming a fixed offset. That is the count the runtime already expects: granite-switch.cpp sizes the model type from it and adds the router layer beyond n_layer at load time.

Affected are newer-generation Granite Switch checkpoints, for example ibm-granite/granite-switch-4.1-3b-preview. Older SingleSwitch-generation checkpoints convert identically before and after this change.

Fixes #30617

Additional information

• Without the patch, the 3b preview converts but llama-server fails to load it with the error above; with the patch it loads.
• An older SingleSwitch-generation revision of the same repo converts to a byte-identical GGUF on master and with this patch.
• Tensor names and shapes match the set derived from granite-switch.cpp's create_tensor calls.
• The cache-slot count comes from the Granite Switch reference implementation (generative-computing/granite-switch, num_cache_layers), which builds num_hidden_layers - num_cache_layers decoder layers.
• A smaller fix (block_count -= 2 if "ms_code_m" in self.hparams else 1) also works today but breaks again the next time the engine's slot budget changes.

NOTE that the flake pre-commit was skipped (SKIP=flake8) due to dependency issues/crashes

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes. AI has been used to identify the bug when testing models, initial drafts of issue, pr texts and generate the code fix, which then had manual review and minor edits

[taronaeo]

cc: @gabe-l-hart

[gabe-l-hart]

One quick comment NIT, but otherwise this looks like a good quick solution. I confirmed that the updated version of ibm-granite/granite-switch-4.1-3b-preview no longer runs after conversion on master and that this conversion change effectively fixes the hparams and no further c++ side changes are needed to work with the converted file.

[gabe-l-hart]

It's worth noting that ibm-granite/granite-switch-4.1-3b-preview was updated in-place, so older pulls of the model work with the existing conversion, but a fresh pull does not.

[gabe-l-hart]

Related! https://huggingface.co/blog/ibm-granite/granite-switch-4-1 (new Switch with decision adapter, guardian, and ASR baked in).

We'll be working on extending for multimodal adapters soon