Repository navigation
llama-graph: prevent f16 overflow in MoE down-projection on Metal - #25442
Closed
steelannelida wants to merge 1 commit into
Closed
steelannelida wants to merge 1 commit into
steelannelida wants to merge 1 commit into
Conversation
Metal's MUL_MAT_ID casts src1 to f16, so a SwiGLU activation above 65504 overflows to NaN and the model returns empty. Scale each column of src1 by its L2 norm before the matmul and undo it after; L2 is at least the largest element so the cast stays in range, and it is a per-column identity so no backend output changes.
|
Hi @steelannelida, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
|
Can confirm that this fixes things on my M1 mac |
Member
|
This is not the right fix, but tagging @ggml-org/ggml-metal for visibility. |
Member
|
Yes, it will be fixed with: #26223 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Metal's MUL_MAT_ID casts src1 to f16, so a SwiGLU activation above 65504 overflows to NaN and the model returns empty.
We encountered this in real life with the Laguna XS 2.1 model, which produces large activations in the later layers. We thus apply the classical trick of amax scaling the input.