Repository navigation
ggml-openvino: honour the -1 skip id in MUL_MAT_ID - #363
Merged
Merged
Conversation
Upstream made an id of -1 in MUL_MAT_ID skip the slot and zero its row of the result. The translator passed -1 straight to GatherMatmul or Gather, so 169 of the 743 MUL_MAT_ID op tests on CPU (all skip=1) failed. A skipped slot now looks up a valid expert, and the result is multiplied by a 0/1 mask of the kept slots. For the packed MXFP4 path (Gather and MatMul), clamping the id to expert 0 is enough. GatherMatmul computes a wrong row when a token names the same expert twice, so the general path gives each skipped slot an expert that no other slot of the token uses. TopK indices are never -1, and model graphs take the ids from the router's TopK. Those graphs skip the handling and stay unchanged, so the GPU MoE fusion still matches them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ggml-org#26631 made an id of -1 in MUL_MAT_ID skip the slot and zero its row of the result. It updated the OpenVINO ADD_ID translator but not MUL_MAT_ID, so -1 reached GatherMatmul and Gather as an expert index, and 169 of the 743 MUL_MAT_ID op tests on CPU (all
skip=1) failed.