Repository navigation
hexagon: enable alloc_buffer_n - #30126
Merged
Merged
Conversation
…ons with big MOEs
…vices that need it
github-actions
Bot
added
ggml
Member
Author
|
@lhez for review/ack |
max-krasnyansky
added
the
merge ready
lhez
approved these changes
Oct 8, 2026
jhen0409
approved these changes
Oct 8, 2026
Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
Member
Author
|
@lhez sorry, I applied suggestion and need reapproval |
lhez
approved these changes
Oct 8, 2026
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 8, 2026
* hex-bufs: add support for alloc_buffer_n * hex-bufs: add support for splitting large tensors into separate buffers * hex-bufs: update GGML_HEXAGON_MBUF to accept three values dyn,static,total * hex-bufs: bump dyn. default to 512MB since 128MB causes perf regressions with big MOEs * hex-run: add --no-embd-offload option to simplify command lines on devices that need it * Update scripts/snapdragon/run.py Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com> --------- Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com> (cherry picked from commit 9c2e0e4)
1 task done
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 24, 2026
* hex-bufs: add support for alloc_buffer_n * hex-bufs: add support for splitting large tensors into separate buffers * hex-bufs: update GGML_HEXAGON_MBUF to accept three values dyn,static,total * hex-bufs: bump dyn. default to 512MB since 128MB causes perf regressions with big MOEs * hex-run: add --no-embd-offload option to simplify command lines on devices that need it * Update scripts/snapdragon/run.py Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com> --------- Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
* hex-bufs: add support for alloc_buffer_n * hex-bufs: add support for splitting large tensors into separate buffers * hex-bufs: update GGML_HEXAGON_MBUF to accept three values dyn,static,total * hex-bufs: bump dyn. default to 512MB since 128MB causes perf regressions with big MOEs * hex-run: add --no-embd-offload option to simplify command lines on devices that need it * Update scripts/snapdragon/run.py Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com> --------- Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR update
ggml-hexagonto use the newalloc_buffer_nbackend API which allows us to better control static buffer allocations. Perhaps, static is not quite the right term, it's basically all the weights, kv tensors, etc. The rest (aka dynamic) of the stuff is still allocated via general allocator.The PR also extends
GGML_HEXAGON_MBUFsetting to support three valuesdynamic[,static,total].Dynamic defaults to 512M, static is 1GB as before and total is 0 (entire RAM) same as before as well.
Lower dynamic default resolve the issue on quad-NPU setup where ALLREDUCE has to mmap 4 buffers from all ranks.
Previous 1GB default was problematic in that scenario because it exceeded ~3.5GB limit.
The new static allocator also ensure that the KV tensors are placed into the same buffer so that opbatches that processes those layers don't need to mmap multiple large buffers.
The PR also adds a little helper to the
snapdragon/run.pyscript that simplifies pass tensor override for mapping lm-head to the CPU.Requirements