Skip to content

llama : move no_alloc out of llama_hparams - #31141

Merged
ggerganov merged 4 commits into
masterfrom
gg/extract-hparams-noalloc
Oct 26, 2026
Merged

ggerganov merged 4 commits into
masterfrom
gg/extract-hparams-noalloc

Conversation

@ggerganov

Copy link
Copy Markdown
Member

Overview

no_alloc is a load/context option, not a model hyperparameter. Remove llama_hparams::no_alloc and pass it explicitly instead:

  • memory module constructors take a bool no_alloc parameter right after the model reference; wrappers forward it to their sub-caches
  • llama_memory_params gains a no_alloc field so create_memory threads it without reading it from the model
  • add llama_memory_bufs (llama-memory.h/.cpp): owns the {ggml context, backend buffer} pairs of a memory instance and centralizes what every leaf previously duplicated - buffer allocation, the no_alloc dummy-buffer path (size-0 buffer with the tensors attached so the scheduler won't allocate them), buffer clear, and memory_breakdown() (real buffer sizes, or simulated sizes via ggml_backend_alloc_ctx_tensors_from_buft_size when no_alloc). Used by llama_kv_cache, llama_memory_recurrent and llama_dsv4_comp_state
  • llama_moe_cache takes no_alloc as a parameter and reuses llama_memory_bufs::alloc_buf for its own buffer layout
  • llama_memory_get_buffers() returns an empty vector for a null memory (memory-less archs)

Resolves the TODO added in #31087.

Additional information

  • test-llama-archs: 538 passed
  • test-llama-context over all 135 dummy models generated by test-llama-archs -o: 135/135 passed
  • a no_alloc=true sweep loading every dummy model with LLAMA_LOAD_MODE_NONE: all report a non-zero simulated memory breakdown and only null-base buffers

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES
    • Harness: dsh:llama.cpp
    • Hardware: 2x DGX Spark (RDMA, TP=2)
    • Model: ggml-org/GLM-5.3-Flash-GGUF
    • Stats: ◷ 2.7h · ⚙ 6.7m · ↑ 314k · ↓ 139k · ⇈ 123.9 tok/s · ⇊ 17.8 tok/s · ≡ 688 · ⌨ 43

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
@ggerganov
ggerganov marked this pull request as ready for review October 25, 2026 12:06
@ggerganov
ggerganov requested a review from CISC as a code owner October 25, 2026 12:06
@github-actions github-actions Bot added the testing Everything test related label Oct 25, 2026
@ggerganov
ggerganov merged commit a469940 into master Oct 26, 2026
20 checks passed
@ggerganov
ggerganov deleted the gg/extract-hparams-noalloc branch October 26, 2026 14:21
S95F added a commit to S95F/llama.cpp that referenced this pull request Oct 26, 2026
Upstream moved no_alloc off llama_hparams in ggml-org#31141 and threaded it
through the llama_moe_cache constructor instead.  The merge updated the
signature and the callers, but the host expert pinning added on this
branch still read model.hparams.no_alloc, so the file stopped compiling
as soon as it was touched.  The argument is already in scope right there
and the buffer allocation a few lines above uses it, so use it.

Assisted-by: Claude Opus 5
auRose94 pushed a commit to auRose94/llama.cpp that referenced this pull request Oct 28, 2026
* llama : move no_alloc out of llama_hparams

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))

* cont : clean-up

* cont : clean-up

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))

* tests : add a no_alloc case to test-llama-context

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
HEchternacht added a commit to HEchternacht/llama.cpp that referenced this pull request Oct 28, 2026
…, realtime

Conflicts resolved by hand:
- ggml-cpu.c: keep fork's i02 < 0 skip (MoE hcache uses negative ids)
- vecdotq.cuh: keep fork's PrismML 128-block Q1_0/Q2_0 vec_dot
- server.cpp: take upstream entrypoint refactor, re-add fork /grammar routes
- tests/CMakeLists.txt: take upstream test layout, keep test-kv-mean-center
- llama-context.cpp: hcache gate uses model.no_alloc() (ggml-org#31141)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants