Eval bug: Nemotron 3.5 Bolt 74B-A9B segfaults on Metal in ggml_metal_op_mul_mat_id #30610

Activity

1. nazeshinjite commented on Oct 15, 2026

   

   nazeshinjite

   on Oct 15, 2026

   More actions

   The null pipeline is the mul_mm_id map0 kernel. Bolt routes to 9 experts per token (n_expert_used = 9), but the map0 kernel is only instantiated for ne20 ∈ {1, 2, 4, 5, 6, 8, 10, 16, 22}:

   llama.cpp/ggml/src/ggml-metal/kernels/mul_mm.metal

   Lines 590 to 598 in b76b133

   	template [[host_name("kernel_mul_mm_id_map0_ne20_1" )]] kernel kernel_mul_mm_id_map0_t kernel_mul_mm_id_map0<1>;
   	template [[host_name("kernel_mul_mm_id_map0_ne20_2" )]] kernel kernel_mul_mm_id_map0_t kernel_mul_mm_id_map0<2>;
   	template [[host_name("kernel_mul_mm_id_map0_ne20_4" )]] kernel kernel_mul_mm_id_map0_t kernel_mul_mm_id_map0<4>;
   	template [[host_name("kernel_mul_mm_id_map0_ne20_5" )]] kernel kernel_mul_mm_id_map0_t kernel_mul_mm_id_map0<5>;
   	template [[host_name("kernel_mul_mm_id_map0_ne20_6" )]] kernel kernel_mul_mm_id_map0_t kernel_mul_mm_id_map0<6>;
   	template [[host_name("kernel_mul_mm_id_map0_ne20_8" )]] kernel kernel_mul_mm_id_map0_t kernel_mul_mm_id_map0<8>;
   	template [[host_name("kernel_mul_mm_id_map0_ne20_10")]] kernel kernel_mul_mm_id_map0_t kernel_mul_mm_id_map0<10>;
   	template [[host_name("kernel_mul_mm_id_map0_ne20_16")]] kernel kernel_mul_mm_id_map0_t kernel_mul_mm_id_map0<16>;
   	template [[host_name("kernel_mul_mm_id_map0_ne20_22")]] kernel kernel_mul_mm_id_map0_t kernel_mul_mm_id_map0<22>;

   ggml_metal_library_get_pipeline_mul_mm_id_map0() looks up kernel_mul_mm_id_map0_ne20_9, which doesn't exist, so it returns a null pipeline. The next line dereferences it in ggml_metal_pipeline_max_theads_per_threadgroup():

   llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp

   Lines 2914 to 2918 in b76b133

   	auto pipeline = ggml_metal_library_get_pipeline_mul_mm_id_map0(lib, ne02, ne20);
   	const size_t smem = pipeline.smem;
   	GGML_ASSERT(ne02 <= ggml_metal_pipeline_max_theads_per_threadgroup(pipeline));

   That matches your backtrace. It also explains why the crash happens during prompt processing: map0 only runs on the batched mul_mm_id path, and single-token decode goes through mul_mv_id, which isn't specialized on ne20.

   The per-layer expert counts (240–496) and the IQ2_XXS/NVFP4 types aren't the trigger. This should reproduce with any quant of Bolt. Nemotron 3.5 Super (top-22) already has a kernel.

   Likely fix: add a <9> instantiation next to the others, as #18667 did for ne20=5. It might also be worth having supports_op (or the map0 getter) reject an uninstantiated ne20, so the next odd top-k falls back instead of segfaulting.

   (I found this by reading the source. I haven't tested the patch on hardware.)

   👍React with 👍3Reacted by Ashe, Vasile Ermicioi and Caladrel
2. LiahMartens commented on Oct 15, 2026

   

   LiahMartens

   on Oct 15, 2026

   More actions

   @nazeshinjite can confirm that works

   👍React with 👍1Reacted by Vasile Ermicioi
3. ashe-wb commented on Oct 15, 2026

   

   ashe-wb

   on Oct 15, 2026

   Author

   More actions

   Confirmed on M1 Max: adding the <9> map0 instantiation fixes the crash. Bolt IQ2_XXS now answers normally.

   👍React with 👍1Reacted by Vasile Ermicioi
4. itsandyking commented on Oct 16, 2026

   

   itsandyking

   on Oct 16, 2026

   More actions

   Confirmed the <9> fix on an M3 Ultra (96 GB), built at f12aaa3. A second problem then shows up with NVIDIA's 24GB GGUF: decode runs at ~6 t/s on Metal.

   The cause is that the file stores the 80 Mamba ssm_in / ssm_out projections as NVFP4. ggml_metal_supports_mul_mat_op returns false for GGML_TYPE_NVFP4, so those matmuls fall back to CPU on every token. The IQ2_XXS experts themselves are fine.

   Workaround: rewrite the file, converting only the NVFP4 tensors to Q8_0. The QAT IQ2_XXS experts, the Q8_0 tensors and the F32 .scale tensors are copied byte-for-byte. The worst conversion error was 0.41% of a tensor's range, and the file grows from 22.0 to 24.2 GB. It takes about 2 minutes with gguf-py:

   import sys
   import numpy as np
   import gguf
   from gguf import GGMLQuantizationType as T
   src, dst = sys.argv[1], sys.argv[2]
   reader = gguf.GGUFReader(src)
   arch = reader.get_field(gguf.Keys.General.ARCHITECTURE).contents()
   writer = gguf.GGUFWriter(dst, arch=arch, endianess=reader.endianess)
   for field in reader.fields.values():
       if field.name == gguf.Keys.General.ARCHITECTURE or field.name.startswith("GGUF."):
           continue
       vt = field.types[0]
       st = field.types[-1] if vt == gguf.GGUFValueType.ARRAY else None
       writer.add_key_value(field.name, field.contents(), vt, sub_type=st)
   converted, worst = {}, 0.0
   for t in reader.tensors:
       if t.tensor_type != T.NVFP4:
           continue
       f = gguf.quants.dequantize(np.asarray(t.data), T.NVFP4)
       q = gguf.quants.quantize(f, T.Q8_0)
       back = gguf.quants.dequantize(q, T.Q8_0)
       rel = float(np.abs(back - f).max() / (np.abs(f).max() or 1))
       worst = max(worst, rel)
       converted[t.name] = q
   print(f"converted {len(converted)} NVFP4 tensors to Q8_0; worst max-abs error {worst:.4%} of tensor range", flush=True)
   for t in reader.tensors:
       if t.name in converted:
           q = converted[t.name]
           writer.add_tensor_info(t.name, q.shape, q.dtype, q.nbytes, T.Q8_0)
       else:
           writer.add_tensor_info(t.name, t.data.shape, t.data.dtype, t.data.nbytes, t.tensor_type)
   writer.write_header_to_file()
   writer.write_kv_data_to_file()
   writer.write_ti_data_to_file()
   for t in reader.tensors:
       writer.write_tensor_data(converted.get(t.name, t.data), tensor_endianess=reader.endianess)
   writer.close()
   print("wrote", dst)

   PYTHONPATH=llama.cpp/gguf-py python3 nvfp4_to_q8.py NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf Bolt-24GB-mamba-q8.gguf

   M3 Ultra, llama-bench -fa 1 -r 3:

   file	pp2048	tg128	tg128 @ d32768
   NVIDIA 24GB (NVFP4 Mamba)	247.8	6.40	—
   same, NVFP4 → Q8_0	604.8	37.74	35.97
   community Q4_K_M from BF16 (45 GiB)	583.6	41.61	39.63

   On a small set of checks (tool call and round trip, strict JSON schema, a code task with hidden tests, a reasoning trap), the rebuilt file passed everything the original did.

   A native Metal NVFP4 mul_mat / mul_mat_id would make this unnecessary. Until then, it may be worth noting on the model card that Apple Silicon users should convert those tensors.
5. Caladrel65 commented on Oct 16, 2026

   

   Caladrel65

   on Oct 16, 2026

   Contributor

   More actions

   Confirmed on an M5 Max as well that the <9> line makes it run. Getting around 11-12 tok/s.

   IMO the unsupported quant type is a separate issue, but one worth creating.
6. arkavo-com

   added 2 commits that reference this issue on Oct 17, 2026

   Add Nemotron 3.5 Bolt 74B-A9B

   ...

   cb7b740

   Add Nemotron 3.5 Bolt 74B-A9B (#754)

   ...

   ec984de
7. nikwen

   added a commit that references this issue on Oct 20, 2026

   metal : fix missing MoE kernel variant for 9 experts per token (#30610)…

   ...

   704366d
8. nikwen

   closed this as completedin #30699on Oct 20, 2026
9. edwardyoon

   added a commit that references this issue on Oct 24, 2026

   metal : fix missing MoE kernel variant for 9 experts per token (ggml-…

   ...

   d720558
10. auRose94

    added a commit that references this issue on Oct 28, 2026

    metal : fix missing MoE kernel variant for 9 experts per token (ggml-…

    ...

    964ff82