Repository navigation
tests : catch vanished rows in the fusion baseline - #30618
Merged
Merged
Conversation
ggerganov
changed the title
ggerganov
marked this pull request as ready for review
October 15, 2026 18:20
github-actions
Bot
added
the
documentation
Member
|
Hmmm, what killed the mellum/qwen4exp fusions? |
Member
Author
|
The qwen4exp fusion died in 889edf4 - it's expected because we use the lightning indexer now. The mellum fusion died in a6b91d8 due to change of how the gating function is determined. It was hardcoded before and now it comes from the hparams. And the dummy model has a different gating function set. Edit: fixed the dummy mellum model to use the proper gating func |
Re-record the MTL fusion baseline with the current build: 276 rows over 131 dummy models. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The report was built from the counters the run produced, and the baseline was only ever used as a lookup table, so a fusion that stopped matching produced no row and was never compared. qwen4exp and mellum lost their fusions this way while the Fusion / metal job stayed green. Report a baseline row that the run does not reproduce as a regression. Scope the pass to the arches that were actually run so --model keeps working. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
mellum reads expert_gating_func from the GGUF since #30578, while the fixture writes sigmoid for every MoE arch, so the Mellum dummy model stopped building the softmax MoE routing fusion. Write softmax for mellum, matching the arch default and the GGUFs that conversion/mellum.py produces. Re-record the MTL baseline: the routing row is back, 277 ok / 0 failed. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
ggerganov
force-pushed
the
gg/fusion-refresh-mtl-baseline
branch
from
October 16, 2026 06:32
7aca860 to
239b017
Compare
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 24, 2026
* tests : update the Metal fusion baseline Re-record the MTL fusion baseline with the current build: 276 rows over 131 dummy models. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : fail test-fusion when a baseline row stops matching The report was built from the counters the run produced, and the baseline was only ever used as a lookup table, so a fusion that stopped matching produced no row and was never compared. qwen4exp and mellum lost their fusions this way while the Fusion / metal job stayed green. Report a baseline row that the run does not reproduce as a regression. Scope the pass to the arches that were actually run so --model keeps working. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : never write inline comments unless the user asks Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : keep the mellum dummy model softmax-gated mellum reads expert_gating_func from the GGUF since ggml-org#30578, while the fixture writes sigmoid for every MoE arch, so the Mellum dummy model stopped building the softmax MoE routing fusion. Write softmax for mellum, matching the arch default and the GGUFs that conversion/mellum.py produces. Re-record the MTL baseline: the routing row is back, 277 ok / 0 failed. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : use emoji in the AI stats disclosure Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : list the disclosure model after the hardware Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : use monochrome symbols in the AI stats disclosure Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : add muse-spark to the Metal fusion baseline Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
* tests : update the Metal fusion baseline Re-record the MTL fusion baseline with the current build: 276 rows over 131 dummy models. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : fail test-fusion when a baseline row stops matching The report was built from the counters the run produced, and the baseline was only ever used as a lookup table, so a fusion that stopped matching produced no row and was never compared. qwen4exp and mellum lost their fusions this way while the Fusion / metal job stayed green. Report a baseline row that the run does not reproduce as a regression. Scope the pass to the arches that were actually run so --model keeps working. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : never write inline comments unless the user asks Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : keep the mellum dummy model softmax-gated mellum reads expert_gating_func from the GGUF since ggml-org#30578, while the fixture writes sigmoid for every MoE arch, so the Mellum dummy model stopped building the softmax MoE routing fusion. Write softmax for mellum, matching the arch default and the GGUFs that conversion/mellum.py produces. Re-record the MTL baseline: the routing row is back, 277 ok / 0 failed. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : use emoji in the AI stats disclosure Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : list the disclosure model after the hardware Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : use monochrome symbols in the AI stats disclosure Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : add muse-spark to the Metal fusion baseline Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Make
test-fusion --checkfail when a fusion stops matching, and re-recordtests/fusion/MTL.csvwith the current build.--checkbuilt its report from the counters the run produced and used the baseline only as a lookup table, so a fusion that stopped matching left no row to compare and theFusion / metaljob stayed green while baseline rows went dead. A baseline row that the run does not reproduce is now reported as a regression, scoped to the arches that were actually run so--modelkeeps working.The baseline is regenerated over the 132 dummy models from
test-llama-archs:maion-coderandmuse-sparkgain their missing rows, two rows that no longer match any graph are dropped, andtest-fusiongoes from 272 ok / 4 failed to 279 ok / 0 failed.The Mellum dummy model writes
expert_gating_func = softmaxso it keeps exercising the softmax MoE routing fusion, matching the arch default and the GGUFsconversion/mellum.pyproduces..pi/gg/SYSTEM.md: never write inline comments unless the user asks, and report the AI stats as monochrome symbols.Requirements
pi:llama.cppMacBook, M5 Max, 128GB