Repository navigation
tests : make MoE router selection independent of the backend noise - #30818
Merged
Merged
Conversation
The router scores are in the range [0, 1), so a random draw can leave the last expert (or expert group) still selected and the first one dropped tied within the noise between the backends. They then route a token to different experts and the logits differ by 4e-3 instead of the usual 5e-7. Beam is the only arch here with a real top-k (8 experts, 2 used) and a group top-k (4 groups, 2 used), so it is the only one that hits this: 2 of 20 seeds failed on an M2 Ultra, the other MoE archs select 2 of 2 and cannot tie. Fill the selection bias with a ramp of step 2 to give every expert a score range of its own, which turns the top-k into a function of the expert index alone. The bias affects only the selection, the weights still come from the unbiased probs, so the MoE output keeps its magnitude. Beam now matches the CPU reference within 4e-7 for all of 160 seeds. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
…table
Both devices report the same description ("Apple M2 Ultra" here), so the
device column of the test table had two indistinguishable rows and a failing
row did not say whether the Metal device or the host CPU produced it.
Add the device name, which is also what -b takes, unless the description
already is the name.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
ggerganov
marked this pull request as ready for review
October 19, 2026 16:29
gagallo7
added a commit
to tetherto/qvac-fabric-llm.cpp
that referenced
this pull request
Oct 22, 2026
…gml-org#30818) A random exp_probs_b router bias lets two expert group scores tie within the backend noise, so a restored seq state flips the top-k and test-llama-context fails beam-moe cp_h/cp_d on the NVIDIA Vulkan CI jobs. Fill it with a ramp as upstream does; not a kernel defect, reproduces on clean upstream b11739. (cherry picked from commit 7879491) Assisted-by: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BGWwt5jSgTYqakvqmLBemV Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 24, 2026
…gml-org#30818) * tests : make MoE router selection independent of the backend noise The router scores are in the range [0, 1), so a random draw can leave the last expert (or expert group) still selected and the first one dropped tied within the noise between the backends. They then route a token to different experts and the logits differ by 4e-3 instead of the usual 5e-7. Beam is the only arch here with a real top-k (8 experts, 2 used) and a group top-k (4 groups, 2 used), so it is the only one that hits this: 2 of 20 seeds failed on an M2 Ultra, the other MoE archs select 2 of 2 and cannot tie. Fill the selection bias with a ramp of step 2 to give every expert a score range of its own, which turns the top-k into a function of the expert index alone. The bias affects only the selection, the weights still come from the unbiased probs, so the MoE output keeps its magnitude. Beam now matches the CPU reference within 4e-7 for all of 160 seeds. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : tell the CPU and the GPU of an Apple SoC apart in the device table Both devices report the same description ("Apple M2 Ultra" here), so the device column of the test table had two indistinguishable rows and a failing row did not say whether the Metal device or the host CPU produced it. Add the device name, which is also what -b takes, unless the description already is the name. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
…gml-org#30818) * tests : make MoE router selection independent of the backend noise The router scores are in the range [0, 1), so a random draw can leave the last expert (or expert group) still selected and the first one dropped tied within the noise between the backends. They then route a token to different experts and the logits differ by 4e-3 instead of the usual 5e-7. Beam is the only arch here with a real top-k (8 experts, 2 used) and a group top-k (4 groups, 2 used), so it is the only one that hits this: 2 of 20 seeds failed on an M2 Ultra, the other MoE archs select 2 of 2 and cannot tie. Fill the selection bias with a ramp of step 2 to give every expert a score range of its own, which turns the top-k into a function of the expert index alone. The bias affects only the selection, the weights still come from the unbiased probs, so the MoE output keeps its magnitude. Beam now matches the CPU reference within 4e-7 for all of 160 seeds. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : tell the CPU and the GPU of an Apple SoC apart in the device table Both devices report the same description ("Apple M2 Ultra" here), so the device column of the test table had two indistinguishable rows and a failing row did not say whether the Metal device or the host CPU produced it. Add the device name, which is also what -b takes, unless the description already is the name. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
test-llama-archs -a beamfails for about one seed in ten (-a beamon an M2 Ultra, 2 of 20 seeds in my run). The failure is deterministic per seed, not a race.Beam is the only arch in the fixture with a real router choice: 8 experts with 2 used, grouped in 4 groups with 2 used. Every other MoE arch selects 2 of 2, so its top-k is a no-op and it cannot disagree with anything.
The grouped router is a discrete decision taken on scores that differ by less than the noise between the backends. For the failing seed, token 17 of
blk.1.ffn_moe_group_topkhas groups 0 and 3 scoring 0.95355 and 0.95353 on the CPU, a margin of 2.6e-05, while the CPU-to-Metal difference in those same scores reaches 9.1e-05. Metal picks group 3 instead of group 0, the second expert of that token becomes a different random matrix, and the final logits end up at 3.8e-03 NMSE instead of the usual 5e-07.Give the selection bias a ramp of step 2 to put each expert in a score range of its own, which makes the top-k a function of the expert index alone. The bias is only added to the scores used for the selection; the weights still come from the unbiased probs, so the MoE output keeps its magnitude.
Additional information
Beam now matches the CPU reference within 4e-07 across 160 seeds, and the previously failing seed 1763288483 passes. No change for the other architectures.
A second commit adds the device name to the device column of the test table. On an Apple SoC the CPU and the Metal device share a description, so a failing row did not say which of the two produced it.
Requirements
pi:llama.cppMacBook, M5 Max, 128GB