Repository navigation
model: fix "invalid vector subscript" when loading on a full GPU - #29677
Merged
Merged
Conversation
When every device reports zero free memory the split points are normalized by a zero sum, the NaN split sends layers to a device index past the end and loading aborts with vector::_M_range_check.
CISC
approved these changes
Sep 29, 2026
6 tasks done
bernardladenthin
pushed a commit
to bernardladenthin/java-llama.cpp
that referenced
this pull request
Oct 14, 2026
b11632 is master's head of 2026-10-14 (62df1815b), pinned ahead of the tag: JLLAMA_GIT_TAG b11632 with JLLAMA_GIT_COMMIT naming the commit, the first use of the virtual-number pin. Upstream merged the zero-split-sum fallback as ggml-org/llama.cpp#29677 (df18bee11, b11626) on the way; patch 0012 stops applying exactly there -- upstream's line is inside the block it replaced -- so it is dropped, not refreshed, and its runnable guard test_model_split.cpp goes with it: the two functions it linked existed only through the patch, the fixed line sits in a member function with no seam, and the end-to-end route needs two GPU-type devices, which a GPU-less runner cannot provide in one process. The drop note in docs/history/dropped-llama-patches.md records all of that. The other twelve patches apply unchanged at 62df1815b. Besides the fix: cpp-httplib 0.61.0, the RPC transfer logs, --spec-draft-n-ubatch (a new speculative field, not exposed yet), a conv_1d layout fix and a CUDA GDN kernel. Through the production path (FetchContent by commit, the applier, the pin check reporting count 11632 = b11632): ctest 601/601, the model-free native Java tests 27/27 against the linked b11632-62df1815b. Clear JLLAMA_GIT_COMMIT once upstream tags b11632. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017kVXQfPe1ACLAuk29Y4d2H
bernardladenthin
pushed a commit
to bernardladenthin/java-llama.cpp
that referenced
this pull request
Oct 14, 2026
b11632 is master's head of 2026-10-14 (62df1815b), pinned ahead of the tag: JLLAMA_GIT_TAG b11632 with JLLAMA_GIT_COMMIT naming the commit, the first use of the virtual-number pin. Upstream merged the zero-split-sum fallback as ggml-org/llama.cpp#29677 (df18bee11, b11626) on the way; patch 0012 stops applying exactly there -- upstream's line is inside the block it replaced -- so it is dropped, not refreshed, and its runnable guard test_model_split.cpp goes with it: the two functions it linked existed only through the patch, the fixed line sits in a member function with no seam, and the end-to-end route needs two GPU-type devices, which a GPU-less runner cannot provide in one process. The drop note in docs/history/dropped-llama-patches.md records all of that. The other twelve patches apply unchanged at 62df1815b. Besides the fix: cpp-httplib 0.61.0, the RPC transfer logs, --spec-draft-n-ubatch (a new speculative field, not exposed yet), a conv_1d layout fix and a CUDA GDN kernel. Through the production path (FetchContent by commit, the applier, the pin check reporting count 11632 = b11632): ctest 601/601, the model-free native Java tests 27/27 against the linked b11632-62df1815b. Clear JLLAMA_GIT_COMMIT once upstream tags b11632. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017kVXQfPe1ACLAuk29Y4d2H
bernardladenthin
pushed a commit
to bernardladenthin/java-llama.cpp
that referenced
this pull request
Oct 14, 2026
b11632 is master's head of 2026-10-14 (62df1815b), pinned ahead of the tag: JLLAMA_GIT_TAG b11632 with JLLAMA_GIT_COMMIT naming the commit, the first use of the virtual-number pin. Upstream merged the zero-split-sum fallback as ggml-org/llama.cpp#29677 (df18bee11, b11626) on the way; patch 0012 stops applying exactly there -- upstream's line is inside the block it replaced -- so it is dropped, not refreshed, and its runnable guard test_model_split.cpp goes with it: the two functions it linked existed only through the patch, the fixed line sits in a member function with no seam, and the end-to-end route needs two GPU-type devices, which a GPU-less runner cannot provide in one process. The drop note in docs/history/dropped-llama-patches.md records all of that. The other twelve patches apply unchanged at 62df1815b. Besides the fix: cpp-httplib 0.61.0, the RPC transfer logs, --spec-draft-n-ubatch (a new speculative field, not exposed yet), a conv_1d layout fix and a CUDA GDN kernel. Through the production path (FetchContent by commit, the applier, the pin check reporting count 11632 = b11632): ctest 601/601, the model-free native Java tests 27/27 against the linked b11632-62df1815b. Clear JLLAMA_GIT_COMMIT once upstream tags b11632. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017kVXQfPe1ACLAuk29Y4d2H
5 of 6 tasks
2 tasks done
goldzzmj
added a commit
to goldzzmj/llama.cpp
that referenced
this pull request
Oct 23, 2026
the split-point lookup can return n_devices (end iterator) on boundary splits and devices.at() throws out_of_range. the zero-free-memory NaN path is already guarded by ggml-org#29677, so this keeps only the index clamp. refs ggml-org#27454
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 24, 2026
…rg#29677) When every device reports zero free memory the split points are normalized by a zero sum, the NaN split sends layers to a device index past the end and loading aborts with vector::_M_range_check.
auRose94
pushed a commit
to auRose94/llama.cpp
that referenced
this pull request
Oct 28, 2026
…rg#29677) When every device reports zero free memory the split points are normalized by a zero sum, the NaN split sends layers to a device index past the end and loading aborts with vector::_M_range_check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
When a GPU is completely full and llama.cpp loads another model on it (typically a draft model for MTP, EAGLE3 or DFlash loaded after the target on Windows), every device reports zero free memory. The layer split is computed as each device's share of the total free memory, so here it becomes 0/0: the split turns into NaN, layers get sent to a GPU index that doesn't exist, and loading dies with "invalid vector subscript" (MSVC) or vector::_M_range_check (libstdc++).
This has been around since June and keeps coming back under different names: five open reports and three PRs with the same idea, none merged. The fix is one line: when the total is exactly zero, split the layers evenly across devices, so the model loads and a real shortage shows up as a normal allocation error instead of a misleading crash.
Nothing else can change: a device that doesn't report memory already falls back to host memory, which is never zero, and any non zero total, including user -ts values, goes through the exact same division as before. Only configurations that always crashed take the new path, and being a single line it is trivial to revert if an exotic setup ever needs something else.
Tested by forcing the CUDA free memory to zero: master aborts with vector::_M_range_check with -fit on and off, the patch loads the model on the GPU and generates normally.
Additional information
Diagnosis by @jebbadee and @Stevie-O. This supersedes #27647, #27657 and #27887, which implement the same fallback. Also reported in #26913, closed as stale.
Fixes #28964
Fixes #27440
Fixes #27454
Fixes #27717
Fixes #24795
Requirements