Repository navigation
metal : fix moe_reduce fusion race when dst aliases its inputs - #30795
Merged
Merged
Conversation
The fused moe_reduce kernel computes dst = sum(experts[e] * weights[e]) in a single dispatch, reading experts and weights while writing dst. The graph allocator computes tensor liveness on the unfused graph, where the inputs are fully consumed by earlier nodes of the fused group, so it can legally recycle the input memory as dst. In the fused kernel this becomes a threadgroup-vs-threadgroup race that produces rare non-deterministic wrong results (observed with test-llama-archs -a kolibri on the meta backend). Reject the fusion when dst overlaps either input, same guard as the mul_mat_add fusion. The unfused MUL + ADD path is always correct. Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
Contributor
|
Should this happen with |
Member
Author
|
Exactly, the |
6 tasks
github-actions
Bot
added
ggml
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 24, 2026
…org#30795) The fused moe_reduce kernel computes dst = sum(experts[e] * weights[e]) in a single dispatch, reading experts and weights while writing dst. The graph allocator computes tensor liveness on the unfused graph, where the inputs are fully consumed by earlier nodes of the fused group, so it can legally recycle the input memory as dst. In the fused kernel this becomes a threadgroup-vs-threadgroup race that produces rare non-deterministic wrong results (observed with test-llama-archs -a kolibri on the meta backend). Reject the fusion when dst overlaps either input, same guard as the mul_mat_add fusion. The unfused MUL + ADD path is always correct. Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
…org#30795) The fused moe_reduce kernel computes dst = sum(experts[e] * weights[e]) in a single dispatch, reading experts and weights while writing dst. The graph allocator computes tensor liveness on the unfused graph, where the inputs are fully consumed by earlier nodes of the fused group, so it can legally recycle the input memory as dst. In the fused kernel this becomes a threadgroup-vs-threadgroup race that produces rare non-deterministic wrong results (observed with test-llama-archs -a kolibri on the meta backend). Reject the fusion when dst overlaps either input, same guard as the mul_mat_add fusion. The unfused MUL + ADD path is always correct. Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The fused
MOE_REDUCEMetal kernel (kernel_moe_reduce_f32) computesdst = Σ experts[e]·weights[e]in a single dispatch, readingexpertsandweightswhile writingdst. The graph allocator computes tensor liveness on the unfused graph, where the inputs are fully consumed by earlier nodes of the fused group, so it can legally recycle the input memory asdst— in the fused kernel that becomes a threadgroup-vs-threadgroup race inside a single dispatch, producing rare non-deterministic wrong results.Fix: reject the fusion at encode time when
dstoverlaps either input, using the existingggml_metal_fusion_overlaphelper — the same guard pattern as theMUL_MAT_ADDfusion. The unfused MUL + ADD path is always correct.Requirements
pi:llama.cpp