Repository navigation
metal : add GGML_OP_TOP_K_MASK - #31230
Merged
Merged
Conversation
Port the fused top-k mask from the CUDA backend: a grid-stride kernel fills the output with -inf, then one threadgroup per row copies the mask values of the kept cells over it. Function constants drop the score, blocks and extra paths that a given graph does not use. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
ggerganov
marked this pull request as ready for review
October 27, 2026 06:25
github-actions
Bot
added
ggml
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
* metal : add GGML_OP_TOP_K_MASK Port the fused top-k mask from the CUDA backend: a grid-stride kernel fills the output with -inf, then one threadgroup per row copies the mask values of the kept cells over it. Function constants drop the score, blocks and extra paths that a given graph does not use. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * metal : clean up top-k mask kernel formatting Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
1 task done
yzhou51
pushed a commit
to yzhou51/llama.cpp
that referenced
this pull request
Oct 30, 2026
* metal : add GGML_OP_TOP_K_MASK Port the fused top-k mask from the CUDA backend: a grid-stride kernel fills the output with -inf, then one threadgroup per row copies the mask values of the kept cells over it. Function constants drop the score, blocks and extra paths that a given graph does not use. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * metal : clean up top-k mask kernel formatting Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Port
GGML_OP_TOP_K_MASKfrom the CUDA backend (#30640) to the Metal backend. This is the fused QSA sparse-attention mask: in each output row, a cell keeps itsmaskvalue when it belongs to a live pick (atop_kindex in range, withscore > -infwhenscoreis given, expanded to theblockscells whenblocksis given) or to anextracell; every other cell becomes-inf.The op is a single graph node with 2 dispatches:
kernel_top_k_mask_fill- grid-stride fill of the whole output with-inf, 4 cells per thread when the output layout permits itkernel_top_k_mask- one threadgroup per row, copies themaskcells of the live picks and of the extra cells over the fillAdditional information
The graph builder already emits the fused form (
build_fused_top_k_maskin llama-graph.cpp); with this PR the node resolves on MTL0 instead of falling back to CPU.Requirements
YES
dsh:llama.cppMacBook, M5 Max, 128GB◷ 1.4h · ⚙ 25.3m · ↑ 523k · ↓ 242k · ⇈ 735.3 tok/s · ⇊ 57.4 tok/s · ≡ 619 · ⌨ 49