Repository navigation
llama: fix early stop in params_fit if ctx is set - #18070
Merged
JohannesGaessler merged 1 commit intoDec 16, 2025
Merged
Conversation
JohannesGaessler
changed the title
Member
|
Also I don't know if this is relevant, but I've had a bug with the autofit of "barely fitting", i.e. the model did fit upon loading, but would crash with "CUDA out of memory error" upon processing the first or second prompt. |
|
This seems to work for me when I specify the context manually: (would it be too much to ask to include single quotes around the arguments so one can just copy-paste them witohut any need to hunt and insert the quotes manually?) Without -c specified the detected suggested reduced context size still does not avoid the cpu hit seemingly: But it's a massive improvement none-the-less. Thank you. |
Contributor
Author
|
@pwilkin what could be happening is that the CUDA pool buffers are growing beyond the margin you're setting. |
Member
|
@JohannesGaessler the thing is, I wasn't setting any buffers. I didn't realize the change was already merged, so I just ran the server without any extra params and I got this behavior. So just indicating that the default settings might be a bit too aggressive. |
|
I tried on a Devstral model and ugh, I don't think this output makes sense? if I try to use this command line, it's accepted, but the load fails: |
Contributor
I also noticed the ability to meet a |
Contributor
Author
|
@verygreen please open a new issue if you find a new bug. That makes keeping track of them much easier from my end. |
ggerganov
approved these changes
Dec 16, 2025
JohannesGaessler
force-pushed
the
llama-fp-fix-ctx
branch
from
December 16, 2025 07:38
be95a9c to
5362340
Compare
JohannesGaessler
changed the title
|
Dense models like desvstral suggested weird parameters is not a separate new issue, I don't get it without this PR (that was not yet merged) (without the PR I get same "this model fits, do nothing" messaging like in #18066) |
Contributor
Author
|
@verygreen your output contains these lines: What is happening here is that due to a bug there is a numerical underflow for the number of layers per GPU, which is why it's trying to assign over 4 billion layers to one of them. This is a bug separate to issues with context size. Please open an issue and post the console output you get, both with and without |
blime4
referenced
this pull request
in blime4/llama.cpp
Feb 5, 2026
Seunghhon
pushed a commit
to Seunghhon/llama.cpp
that referenced
this pull request
Apr 26, 2026
10 of 12 tasks
my-other-github-account
pushed a commit
to my-other-github-account/llama.cpp
that referenced
this pull request
May 15, 2026
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 5, 2026
MrLordCat
referenced
this pull request
in MrLordCat/llama.cpp-rdna-lab
Jul 16, 2026
zommiommy
pushed a commit
to zommiommy/llama.cpp
that referenced
this pull request
Aug 18, 2026
frostyautumnleaf
pushed a commit
to frostyautumnleaf/llama.cpp
that referenced
this pull request
Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Should at least partially fix #18066 .
The early return on context reduction is done too often, in particular it should never be done if the user is explicitly setting a context size.
For a single GPU we can always safely return if we can fit the model via context reduction. However, for multiple GPUs however this is unsafe so we should always go on to fit.