[Dorijan10]

Overview

llama-quantize currently ignores --tensor-type overrides when the target type is F32, F16, or BF16. The command exits successfully (code 0), does not print an applying manual override message, and leaves every tensor in the target type. By contrast, --token-embedding-type and --output-tensor-type already work as expected with these targets.

The issue is in llama_tensor_get_type(), where the override loop is nested inside if (ggml_is_quantized(default_type)). This PR moves that loop outside the condition so overrides are checked for all target types. The mixture rules in llama_tensor_get_type_impl remain restricted to quantized targets. tensor_type_fallback() also now runs for every target type; for F32, F16, and BF16, it has no effect because the block size is 1.

This is existing behavior rather than a regression. The condition has been present since --tensor-type was introduced in #12511. Subsequently, #25716 removed the --pure portion of that same condition.

Additional information

The issue can be reproduced using the public Qwen3.5-0.8B-BF16.gguf file from unsloth/Qwen3.5-0.8B-GGUF (revision 6ab4614):

llama-quantize --tensor-type 'blk\.3\.ffn_down=q4_k' Qwen3.5-0.8B-BF16.gguf probe.gguf BF16

On master, the resulting file still contains all 187 tensors in BF16. With this PR, only blk.3.ffn_down.weight is converted from BF16 to Q4_K. The output file is 5,275,648 bytes smaller, exactly matching the difference between that tensor's BF16 and Q4_K sizes. A 32-token greedy completion also matches the BF16 input. When using a quantized target (Q8_0), the output remains byte-identical before and after the change.

For testing, a new local section in test-quant-type-selection applies one override to a mock model for each of F32, F16, BF16, Q8_0, and Q4_K_M. Without the fix, the F32, F16, and BF16 cases fail. With the fix, all five cases pass, and the 11 snapshot models continue to pass as well.

Related work: draft #24408 contains a similar change for ftype COPY, but that change is not included in its split PRs.

Logs

master:

[  52/ 320] blk.3.ffn_down.weight                - [  3584,   1024,      1,      1], type =   bf16, size =    7.000 MiB
llama_model_quantize_impl: quant size  =  1436.03 MiB (16.01 BPW)

this PR:

llama_tensor_get_type: blk.3.ffn_down.weight                - applying manual override: bf16 -> q4_K
[  52/ 320] blk.3.ffn_down.weight                - [  3584,   1024,      1,      1], type =   bf16, converting to q4_K .. size =     7.00 MiB ->     1.97 MiB
llama_model_quantize_impl: quant size  =  1431.00 MiB (15.95 BPW)

test, without and with the fix:

  FAIL  blk.1.ffn_down.weight          F32    override not applied, got f32
  FAIL  blk.1.ffn_down.weight          F16    override not applied, got f16
  FAIL  blk.1.ffn_down.weight          BF16   override not applied, got bf16
  FAIL  tensor type overrides: 2/5 ftypes passed
  PASS  tensor type overrides: 5/5 ftypes passed

Requirements

• I have read and agree with the contributing guidelines.
• AI usage disclosure: YES. I used AI to draft the patch and test. I take full responsibility for the code in this PR.