Skip to content

hexagon: MUL_MAT_ID row skipping optimizations and improvements - #31169

Merged
max-krasnyansky merged 2 commits into
ggml-org:masterfrom
qualcomm:hexagon-matmul-id-row-skipping
Oct 26, 2026
Merged

hexagon: MUL_MAT_ID row skipping optimizations and improvements#31169
max-krasnyansky merged 2 commits into
ggml-org:masterfrom
qualcomm:hexagon-matmul-id-row-skipping

Conversation

@max-krasnyansky

@max-krasnyansky max-krasnyansky commented Oct 26, 2026

Copy link
Copy Markdown
Member

Overview

Quick follow up for #26631
Vectorized row zeroing and added skipping support to fused MUL_MAT_ID_NX kernels.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: Yed, Anti.G for code review and apply updates to all fused paths.

@max-krasnyansky
max-krasnyansky requested a review from a team as a code owner October 26, 2026 05:35
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning Hexagon labels Oct 26, 2026
@max-krasnyansky max-krasnyansky added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Oct 26, 2026
@max-krasnyansky
max-krasnyansky merged commit caa8191 into ggml-org:master Oct 26, 2026
25 checks passed
@ebateni

ebateni commented Oct 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

@max-krasnyansky after this PR, the gemma4-26B generates garbage to some prompts
You are a helpful assistant. Answer concisely. What is the best cookie in the world?<|channel>01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:0

command :
env LD_LIBRARY_PATH=$L GGML_HEXAGON_ARCH=81"CDSP_LIBRARY_PATH=$L;/vendor/dsp/cdsp0" GGML_HEXAGON_DEVICES=HTP0 GGML_HEXAGON_OPFUSION=8191 $P/bin/llama-completion -m /data/local/tmp/models/gemma-4-26B_q4_0-it.gguf -dev HTP0 -c 4096 -ngl 99 -no-cnv --temp 0 -f /data/local/tmp/cookie.txt -n 128 2>&1 | tail -20'

P.S: row split 2c generates sane response

@max-krasnyansky

max-krasnyansky commented Oct 26, 2026

Copy link
Copy Markdown
Member Author

@max-krasnyansky after this PR, the gemma4-26B generates garbage to some prompts You are a helpful assistant. Answer concisely. What is the best cookie in the world?<|channel>01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:0

command : env LD_LIBRARY_PATH=$L GGML_HEXAGON_ARCH=81"CDSP_LIBRARY_PATH=$L;/vendor/dsp/cdsp0" GGML_HEXAGON_DEVICES=HTP0 GGML_HEXAGON_OPFUSION=8191 $P/bin/llama-completion -m /data/local/tmp/models/gemma-4-26B_q4_0-it.gguf -dev HTP0 -c 4096 -ngl 99 -no-cnv --temp 0 -f /data/local/tmp/cookie.txt -n 128 2>&1 | tail -20'

P.S: row split 2c generates sane response

Oh. I didn't hit that in my tests. Will check on this asap.

@ebateni

ebateni commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

@max-krasnyansky after this PR, the gemma4-26B generates garbage to some prompts You are a helpful assistant. Answer concisely. What is the best cookie in the world?<|channel>01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:0
command : env LD_LIBRARY_PATH=$L GGML_HEXAGON_ARCH=81"CDSP_LIBRARY_PATH=$L;/vendor/dsp/cdsp0" GGML_HEXAGON_DEVICES=HTP0 GGML_HEXAGON_OPFUSION=8191 $P/bin/llama-completion -m /data/local/tmp/models/gemma-4-26B_q4_0-it.gguf -dev HTP0 -c 4096 -ngl 99 -no-cnv --temp 0 -f /data/local/tmp/cookie.txt -n 128 2>&1 | tail -20'
P.S: row split 2c generates sane response

Oh. I didn't hit that in my tests. Will check on this asap.

happening only in 1c for me

@ebateni

ebateni commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

@max-krasnyansky after this PR, the gemma4-26B generates garbage to some prompts You are a helpful assistant. Answer concisely. What is the best cookie in the world?<|channel>01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:01:0
command : env LD_LIBRARY_PATH=$L GGML_HEXAGON_ARCH=81"CDSP_LIBRARY_PATH=$L;/vendor/dsp/cdsp0" GGML_HEXAGON_DEVICES=HTP0 GGML_HEXAGON_OPFUSION=8191 $P/bin/llama-completion -m /data/local/tmp/models/gemma-4-26B_q4_0-it.gguf -dev HTP0 -c 4096 -ngl 99 -no-cnv --temp 0 -f /data/local/tmp/cookie.txt -n 128 2>&1 | tail -20'
P.S: row split 2c generates sane response

Oh. I didn't hit that in my tests. Will check on this asap.

happening only in 1c for me

I confirm it was a false alarm in my setup. It is working fine for 1c too

auRose94 pushed a commit to auRose94/llama.cpp that referenced this pull request Oct 28, 2026
@max-krasnyansky
hexagon: MUL_MAT_ID row skipping optimizations and improvements (ggml…
65ae74d 
…-org#31169)

* hex-mmid: vectorize skipped row zeroing and added zeroing to fused MM_ID_NX

* hex-mmid: remove scalar div
@ggerganov ggerganov mentioned this pull request Oct 28, 2026
Merged
@BrewTestBot BrewTestBot mentioned this pull request Oct 28, 2026
Merged
1 task done
edwardyoon pushed a commit to edwardyoon/focus-llama that referenced this pull request Oct 30, 2026
@max-krasnyansky @edwardyoon
hexagon: MUL_MAT_ID row skipping optimizations and improvements (ggml…
c3d2b6e 
…-org#31169)

* hex-mmid: vectorize skipped row zeroing and added zeroing to fused MM_ID_NX

* hex-mmid: remove scalar div
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jhen0409 jhen0409 jhen0409 approved these changes
@lhez lhez lhez approved these changes

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning Hexagon merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@max-krasnyansky @ebateni @jhen0409 @lhez