Repository navigation
Eval bug: GGML_ASSERT(n_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed on DS v4 Flash on single GPU #25800
Description
Name and Version
version: 10055 (e8f19cc)
built with GNU 13.3.0 for Linux x86_64
Operating systems
Linux
GGML backends
CUDA
Hardware
Ryzen 9900X + RTX3060
Models
DeepSeek-V4-Flash-UD-IQ2_M from https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF
Problem description & steps to reproduce
Simply try to start it with a large-ish context:
./build/bin/llama-cli -c 65536 -m models/custom/Deepseek-V4-Flash/DeepSeek-V4-Flash-UD-IQ2_M-00001-of-00003.gguf
Starting it with a smaller context works fine, e.g.:
./build/bin/llama-cli -c 32768 -m models/custom/Deepseek-V4-Flash/DeepSeek-V4-Flash-UD-IQ2_M-00001-of-00003.gguf
Since I only have a 3060 (12GB) and it seems limited to large contexts, I suspect something to do with memory pressure on the GPU that is not falling back to the CPU properly (or is just bailing before CUDA can fail to alloc)?
First Bad Commit
Relevant log output
Logs
$ ./build/bin/llama-cli -c 65536 -m models/custom/Deepseek-V4-Flash/DeepSeek-V4-Flash-UD-IQ2_M-00001-of-00003.gguf
Loading model... -/home/eapache/src/llama.cpp/ggml/src/ggml-backend.cpp:1367: GGML_ASSERT(n_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed
\[New LWP 22798]
[New LWP 22797]
[New LWP 22796]
[New LWP 22795]
[New LWP 22794]
[New LWP 22793]
[New LWP 22792]
[New LWP 22791]
[New LWP 22790]
[New LWP 22789]
[New LWP 22788]
[New LWP 22787]
[New LWP 22786]
[New LWP 22785]
[New LWP 22784]
[New LWP 22783]
[New LWP 22782]
[New LWP 22781]
[New LWP 22780]
[New LWP 22779]
[New LWP 22778]
[New LWP 22777]
[New LWP 22776]
[New LWP 22775]
[New LWP 22774]
[New LWP 22773]
[New LWP 22772]
[New LWP 22771]
[New LWP 22770]
[New LWP 22769]
[New LWP 22765]
This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007535168ecadf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7ffcdaa09df0, rem=0x7ffcdaa09df0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
warning: 78 ../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory
#0 0x00007535168ecadf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7ffcdaa09df0, rem=0x7ffcdaa09df0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
78 in ../sysdeps/unix/sysv/linux/clock_nanosleep.c
#1 0x00007535168f9a27 in __GI___nanosleep (req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
warning: 25 ../sysdeps/unix/sysv/linux/nanosleep.c: No such file or directory
#2 0x00007535176aefd5 in cli_server::wait_ready(std::function<bool ()>) () from /home/eapache/src/llama.cpp/build/bin/libllama-cli-impl.so
#3 0x00007535176a9000 in cli_context::init() () from /home/eapache/src/llama.cpp/build/bin/libllama-cli-impl.so
#4 0x00007535176911dc in llama_cli(int, char**) () from /home/eapache/src/llama.cpp/build/bin/libllama-cli-impl.so
#5 0x000075351682a1ca in __libc_start_call_main (main=main@entry=0x5c0c9addd5d0 <main>, argc=argc@entry=5, argv=argv@entry=0x7ffcdaa0d858) at ../sysdeps/nptl/libc_start_call_main.h:58
warning: 58 ../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#6 0x000075351682a28b in __libc_start_main_impl (main=0x5c0c9addd5d0 <main>, argc=5, argv=0x7ffcdaa0d858, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcdaa0d848) at ../csu/libc-start.c:360
warning: 360 ../csu/libc-start.c: No such file or directory
#7 0x00005c0c9addd605 in _start ()
|[Inferior 1 (process 22762) detached]
Aborted (core dumped)
Activity
eapache commentedon Jul 17, 2026
I have the full core dump file locally if more detail is needed. The actual relevant backtrace looks to be
eapache commentedon Jul 17, 2026
Building off of b2dd28a does not have this problem - this is almost certainly a regression from #25585 cc @am17an
edit: confirmed, reverting just 0dc74e3 (the merge of that PR) makes it work again, although of course much more slowly. And without an OOM/alloc failure, so maybe my memory pressure guess was wrong.
eapache commentedon Jul 17, 2026
Full logs (with -v) of reproduction
out.log
ggerganov commentedon Jul 17, 2026
Please attach logs with env
GGML_SCHED_DEBUG=2too.eapache commentedon Jul 17, 2026
out.log.gz
With
GGML_SCHED_DEBUG=2- gzipped as they are 65MB raw.eapache commentedon Jul 17, 2026
At the suggestion of chatgpt: running with
--no-op-offloadsidesteps the bug on master.ggerganov commentedon Jul 17, 2026
Something seems to be forcing the flash attention and the HC ops to the device backend even for the CPU layers:
Not sure this is correct - should be traced why it happens.
am17an commentedon Jul 17, 2026
It may the --fit process, not sure
ggerganov commentedon Jul 17, 2026
The fit process only tries to offload different number of layers. It shouldn't be the root cause. Likely you can reproduce the same problem with
-ngl 39 -fit off.Most likely the CUDA backend decides to offload an op based on this logic:
llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu
Lines 5137 to 5156 in 635cdd5
Maybe
ROPE_BACK? And from there the FA is also offloaded.Additionally, looking at the DSV4 graph, it is doing view operations with the weights. For example:
llama.cpp/src/models/deepseek4.cpp
Line 1081 in 635cdd5
llama.cpp/src/models/deepseek4.cpp
Lines 285 to 286 in 635cdd5
This does not play well with the "tensor info" logic here:
llama.cpp/src/llama-arch.cpp
Lines 609 to 621 in 635cdd5
Generally, a weight should be used directly as it is loaded, without reshaping or viewing it. I recommend to fix that for DSV4 or we will keep seeing such problems like this issue. The fix would require updating the convert process to shape the weights in the right dimensions and using them directly, as declared in the "tensor info" section.
cc @fairydreaming for viz
am17an commentedon Jul 17, 2026
Yes this was done originally to conform with the existing antirez GGUF IIRC
ggerganov commentedon Jul 17, 2026
Yes, this is true for the
wo_atensor. But the HC tensors have the same problem.am17an commentedon Jul 17, 2026
HC tensors can be fixed on conversion, the
wo_aweight I think we can still keep around?ggerganov commentedon Jul 17, 2026
I think #25832 should fix the current issue.
If we update the conversion, it's best to update all tensors. There is no reason to have the
wo_ato be a 2D tensor in the model file if we always reshape it to 3D first.am17an commentedon Jul 17, 2026
The reason is simply to load those existing GGUFs, but we can have if/else for this case?
ggerganov commentedon Jul 17, 2026
Yes, we can have if/else to support both. But my point was that if we change the HC tensors conversion (as you suggested), there is no reason not to change the
wo_atensor conversion too.