Skip to content

Eval bug: CUDA crash on tensor copy #20146

Closed
Closed
Eval bug: CUDA crash on tensor copy#20146
Labels
CUDARelated to the CUDA backendbugSomething isn't working

Description

@pwilkin
pwilkin
opened on Mar 5, 2026

Name and Version

load_backend: loaded BLAS backend from /devel/tools/llama.cpp/build/bin/libggml-blas.so
ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes
Device 1: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from /devel/tools/llama.cpp/build/bin/libggml-cuda.so
load_backend: loaded CPU backend from /devel/tools/llama.cpp/build/bin/libggml-cpu-alderlake.so
version: 8234 (30ea7bce5)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

2x NVidia 5070Ti 16GB

Models

StepFun 3.5 IQ3_XXS

Problem description & steps to reproduce

During normal server operation, command was:

llama-server -c 160000 -ctk q8_0 -ctv q8_0 --host 0.0.0.0 -a syndatis -m IQ3_XXS/Step-3.5-Flash-IQ3_XXS-00001-of-00003.gguf -dio

First Bad Commit

No response

Relevant log output

/devel/tools/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
CUDA error: the launch timed out and was terminated
  current device: 1, in function ggml_backend_cuda_cpy_tensor_async at /devel/tools/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2836
  cudaMemcpyPeerAsync(dst->data, cuda_ctx_dst->device, src->data, cuda_ctx_src->device, ggml_nbytes(dst), cuda_ctx_src->stream())

Not sure if relevant, but this happened after I upgraded drivers to 590 and CUDA version to 13.1

Activity

  1. pwilkin
    added
    bug-unconfirmed
    bugSomething isn't working
    CUDARelated to the CUDA backend
    and removed
    bug-unconfirmed
    on Mar 5, 2026

  2. pwilkin commented on Mar 5, 2026

    @pwilkin
    pwilkin
    on Mar 5, 2026
    MemberAuthor

    dmesg has this:

    [ 3808.581695] NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
[ 3808.583040] NVRM: GPU at PCI:0000:06:00: GPU-3fb04e1a-ffdc-7e9e-0592-544688705e98
[ 3808.583042] NVRM: GPU Board Serial Number: 0
[ 3808.583043] NVRM: Xid (PCI:0000:06:00): 8, pid=4307, name=llama-server, channel 0x00000008

  3. CISC commented on Mar 5, 2026

    @CISC
    CISC
    on Mar 5, 2026
    Member

    Not sure if relevant, but this happened after I upgraded drivers to 590 and CUDA version to 13.1

    I've been using 590 with 13.1 for quite a while without issues...

  4. ggerganov commented on Mar 5, 2026

    @ggerganov
    ggerganov
    on Mar 5, 2026
    Member

    Do you have this commit in your branch: 2cd20b7

  5. pwilkin commented on Mar 5, 2026

    @pwilkin
    pwilkin
    on Mar 5, 2026
    MemberAuthor

    @ggerganov nope, does it possibly fix the issue? (it's not a very reproducible issue, I haven't encountered it since)

  6. ggerganov commented on Mar 6, 2026

    @ggerganov
    ggerganov
    on Mar 6, 2026
    Member

    Ok, I was just making sure that it didn't cause the issue.

  7. pwilkin commented on Mar 6, 2026

    @pwilkin
    pwilkin
    on Mar 6, 2026
    MemberAuthor

    Happened again with said commit included this time.

  8. pwilkin commented on Mar 6, 2026

    @pwilkin
    pwilkin
    on Mar 6, 2026
    MemberAuthor

    Running compute-sanitizer test-backend-ops gives this:

    Failing tests:
  SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=0.100000,max_bias=0.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1023,1023,1,1],mask=0,sinks=1,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=0,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=0,m_prec=f32,nr23=[1,1],scale=0.100000,max_bias=8.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=0,m_prec=f16,nr23=[1,1],scale=0.100000,max_bias=8.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=1,m_prec=f32,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1023,1023,1,1],mask=1,sinks=1,m_prec=f16,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=1,m_prec=f16,nr23=[1,1],scale=0.100000,max_bias=0.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1023,1023,1,1],mask=1,sinks=1,m_prec=f16,nr23=[1,1],scale=0.100000,max_bias=0.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=1,m_prec=f16,nr23=[1,1],scale=1.000000,max_bias=8.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=1,m_prec=f32,nr23=[1,1],scale=0.100000,max_bias=8.000000,inplace=0)
SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=1,m_prec=f16,nr23=[1,1],scale=0.100000,max_bias=8.000000,inplace=0)
  Backend CUDA1: �[1;31mFAIL�[0m

    Also I get the following errors from the sanitizer even though they don't cause failure:

      RMS_NORM_MUL_ROPE(ne=[8192,2,2,1],eps=0.000001,multi_add=1,set_rows=0,mode=0): ggml_backend_cuda_graph_compute: CUDA graph warmup complete
========= Program hit cudaErrorGraphExecUpdateFailure (error 910) due to "the graph update was not performed because it included changes which violated constraints specific to instantiated graph update" on CUDA API call to cudaGraphExecUpdate.
=========     Saved host backtrace up to driver entry point at error
=========         Host Frame: ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) [0x3097666] in libggml-cuda.so
=========         Host Frame: ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) [0x3099631] in libggml-cuda.so
=========         Host Frame: ggml_backend_graph_compute [0x2f6a1] in libggml-base.so.0
=========         Host Frame: ggml_backend_compare_graph_backend [0x351c1] in libggml-base.so.0
=========         Host Frame: test_case::eval(ggml_backend*, ggml_backend*, char const*, printer*) [0x9d294] in test-backend-ops
=========         Host Frame: main [0x1ba16] in test-backend-ops
========= 
========= Program hit cudaErrorGraphExecUpdateFailure (error 910) due to "the graph update was not performed because it included changes which violated constraints specific to instantiated graph update" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========         Host Frame: ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) [0x3098ab1] in libggml-cuda.so
=========         Host Frame: ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) [0x3099631] in libggml-cuda.so
=========         Host Frame: ggml_backend_graph_compute [0x2f6a1] in libggml-base.so.0
=========         Host Frame: ggml_backend_compare_graph_backend [0x351c1] in libggml-base.so.0
=========         Host Frame: test_case::eval(ggml_backend*, ggml_backend*, char const*, printer*) [0x9d294] in test-backend-ops
=========         Host Frame: main [0x1ba16] in test-backend-ops
========= 
�[1;32mOK�[0m
  ROPE(type=f32,ne_a=[80,32,2,1],n_dims=20,mode=0,n_ctx=512,fs=1.000000,ef=0.000000,af=1.000000,ff=0,v=1,inplace=0): ggml_backend_cuda_graph_compute: CUDA graph warmup complete
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
========= Program hit cudaErrorGraphExecUpdateFailure (error 910) due to "the graph update was not performed because it included changes which violated constraints specific to instantiated graph update" on CUDA API call to cudaGraphExecUpdate.
=========     Saved host backtrace up to driver entry point at error
=========         Host Frame: ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) [0x3097666] in libggml-cuda.so
=========         Host Frame: ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) [0x3099631] in libggml-cuda.so
=========         Host Frame: ggml_backend_graph_compute [0x2f6a1] in libggml-base.so.0
=========         Host Frame: ggml_backend_compare_graph_backend [0x3533e] in libggml-base.so.0
=========         Host Frame: test_case::eval(ggml_backend*, ggml_backend*, char const*, printer*) [0x9d294] in test-backend-ops
=========         Host Frame: main [0x1ba16] in test-backend-ops
========= 
========= Program hit cudaErrorGraphExecUpdateFailure (error 910) due to "the graph update was not performed because it included changes which violated constraints specific to instantiated graph update" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========         Host Frame: ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) [0x3098ab1] in libggml-cuda.so
=========         Host Frame: ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) [0x3099631] in libggml-cuda.so
=========         Host Frame: ggml_backend_graph_compute [0x2f6a1] in libggml-base.so.0
=========         Host Frame: ggml_backend_compare_graph_backend [0x3533e] in libggml-base.so.0
=========         Host Frame: test_case::eval(ggml_backend*, ggml_backend*, char const*, printer*) [0x9d294] in test-backend-ops
=========         Host Frame: main [0x1ba16] in test-backend-ops
========= 
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
========= Program hit cudaErrorGraphExecUpdateFailure (error 910) due to "the graph update was not performed because it included changes which violated constraints specific to instantiated graph update" on CUDA API call to cudaGraphExecUpdate.
=========     Saved host backtrace up to driver entry point at error
=========         Host Frame: ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) [0x3097666] in libggml-cuda.so
=========         Host Frame: ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) [0x3099631] in libggml-cuda.so
=========         Host Frame: ggml_backend_graph_compute [0x2f6a1] in libggml-base.so.0
=========         Host Frame: ggml_backend_compare_graph_backend [0x3533e] in libggml-base.so.0
=========         Host Frame: test_case::eval(ggml_backend*, ggml_backend*, char const*, printer*) [0x9d294] in test-backend-ops
=========         Host Frame: main [0x1ba16] in test-backend-ops
========= 
========= Program hit cudaErrorGraphExecUpdateFailure (error 910) due to "the graph update was not performed because it included changes which violated constraints specific to instantiated graph update" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========         Host Frame: ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) [0x3098ab1] in libggml-cuda.so
=========         Host Frame: ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) [0x3099631] in libggml-cuda.so
=========         Host Frame: ggml_backend_graph_compute [0x2f6a1] in libggml-base.so.0
=========         Host Frame: ggml_backend_compare_graph_backend [0x3533e] in libggml-base.so.0
=========         Host Frame: test_case::eval(ggml_backend*, ggml_backend*, char const*, printer*) [0x9d294] in test-backend-ops
=========         Host Frame: main [0x1ba16] in test-backend-ops
========= 
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
�[1;32mOK�[0m

    Details on the softmax errors:

    [SOFT_MAX] ERR = 0.000032182 > 0.000001000   SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=0,sinks=0,m_prec=f32,nr23=[1,1],scale=0.100000,max_bias=0.000000,inplace=0): ggml_backend_cuda_graph_compute: CUDA graph warmup complete
[SOFT_MAX] ERR = 0.000031495 > 0.000001000   SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=0,sinks=1,m_prec=f32,nr23=[1,1],scale=0.100000,max_bias=0.000000,inplace=0): ggml_backend_cuda_graph_compute: CUDA graph warmup complete
[SOFT_MAX] ERR = 0.000074724 > 0.000001000   SOFT_MAX(type=f32,ne=[1023,1023,1,1],mask=1,sinks=0,m_prec=f16,nr23=[1,1],scale=1.000000,max_bias=0.000000,inplace=0): ggml_backend_cuda_graph_compute: CUDA graph warmup reset
[SOFT_MAX] ERR = 0.000088507 > 0.000001000   SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=1,m_prec=f32,nr23=[1,1],scale=0.100000,max_bias=0.000000,inplace=0): ggml_backend_cuda_graph_compute: CUDA graph warmup complete
[SOFT_MAX] ERR = 0.000063644 > 0.000001000   SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=1,m_prec=f16,nr23=[1,1],scale=1.000000,max_bias=8.000000,inplace=0): �[1;31mFAIL�[0m
[SOFT_MAX] ERR = 0.000032019 > 0.000001000   SOFT_MAX(type=f32,ne=[1024,1024,1,1],mask=1,sinks=1,m_prec=f16,nr23=[1,1],scale=0.100000,max_bias=8.000000,inplace=0): �[1;31mFAIL�[0m

  9. am17an commented on Mar 7, 2026

    @am17an
    am17an
    on Mar 7, 2026
    Contributor

    Can you share a repro? It's ok if it happens sporadically, just need a stack trace to debug

  10. pwilkin commented on Mar 8, 2026

    @pwilkin
    pwilkin
    on Mar 8, 2026
    MemberAuthor

    Will try if it happens again.

  11. pwilkin commented on Mar 11, 2026

    @pwilkin
    pwilkin
    on Mar 11, 2026
    MemberAuthor

    BTW, I've reverted to 580 drivers + CUDA 13.0 and the error hasn't reappeared so far.

  12. CISC commented on Mar 11, 2026

    @CISC
    CISC
    on Mar 11, 2026
    Member

    BTW, I've reverted to 580 drivers + CUDA 13.0 and the error hasn't reappeared so far.

    595 and 13.2 is out, YOLO! :)

  13. 4 remaining items

  14. pwilkin commented on Mar 12, 2026

    @pwilkin
    pwilkin
    on Mar 12, 2026
    MemberAuthor

    @ggerganov I have no idea why it's reporting that way (I guess it's reporting old debugging info? I'll clear the build and rebuild) but it's a clean build 8299 (commit d28961d)

  15. ggerganov commented on Mar 12, 2026

    @ggerganov
    ggerganov
    on Mar 12, 2026
    Member

    Most likely it's linking to some stale/old .so. See the output of ldd -r ./bin/llama-server

  16. pwilkin commented on Mar 12, 2026

    @pwilkin
    pwilkin
    on Mar 12, 2026
    MemberAuthor

    Yeah I removed the entire build dir and rebuilt, also changed target arch from 89 to 120 now that I installed CUDA 13. Will see if the problem persists.

  17. pwilkin commented on Mar 14, 2026

    @pwilkin
    pwilkin
    on Mar 14, 2026
    MemberAuthor

    Got the crash again:

    #0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56	in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x0000794a830a013c in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
warning: 49	./nptl/cancellation.c: No such file or directory
#2  __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75	in ./nptl/cancellation.c
#3  0x0000794a8311ca0f in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4  0x0000794a83b67e13 in ggml_print_backtrace () from /devel/tools/llama.cpp/build/bin/libggml-base.so.0
#5  0x0000794a83b67fc6 in ggml_abort () from /devel/tools/llama.cpp/build/bin/libggml-base.so.0
#6  0x0000794991bf2e55 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /devel/tools/llama.cpp/build/bin/libggml-cuda.so
#7  0x0000794991bf723e in ggml_backend_cuda_set_tensor_async(ggml_backend*, ggml_tensor*, void const*, unsigned long, unsigned long) () from /devel/tools/llama.cpp/build/bin/libggml-cuda.so
#8  0x0000794a83b848a3 in ggml_backend_sched_graph_compute_async () from /devel/tools/llama.cpp/build/bin/libggml-base.so.0
#9  0x0000794a838cd350 in llama_context::graph_compute(ggml_cgraph*, bool) () from /devel/tools/llama.cpp/build/bin/libllama.so.0
#10 0x0000794a838cfd13 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /devel/tools/llama.cpp/build/bin/libllama.so.0
#11 0x0000794a838d668f in llama_context::decode(llama_batch const&) () from /devel/tools/llama.cpp/build/bin/libllama.so.0
#12 0x0000794a838d8262 in llama_decode () from /devel/tools/llama.cpp/build/bin/libllama.so.0
#13 0x000064f0d9c25e5e in server_context_impl::update_slots() ()
#14 0x000064f0d9c7564f in server_queue::start_loop(long) ()
#15 0x000064f0d9b75486 in main ()
[Inferior 1 (process 1390545) detached]
Aborted (core dumped)

    This time clean 8299.

    Gonna run a rebuilt 8346 now, see if it helps.

  18. pwilkin commented on Mar 14, 2026

    @pwilkin
    pwilkin
    on Mar 14, 2026
    MemberAuthor

    Still the same:

    CUDA error: the launch timed out and was terminated
  current device: 1, in function ggml_backend_cuda_set_tensor_async at /devel/tools/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2849
  cudaMemcpyAsync((char *)tensor->data + offset, data, size, cudaMemcpyHostToDevice, cuda_ctx->stream())
[New LWP 1423391]
[New LWP 1423390]
[New LWP 1423389]
[New LWP 1423388]
[New LWP 1423387]
[New LWP 1423386]
[New LWP 1423385]
[New LWP 1423369]
[New LWP 1423368]
[New LWP 1423367]
[New LWP 1423366]
[New LWP 1423365]
[New LWP 1423364]
[New LWP 1423363]
[New LWP 1423362]
[New LWP 1423361]
[New LWP 1423360]
[New LWP 1423359]
[New LWP 1423358]
[New LWP 1423357]
[New LWP 1423356]
[New LWP 1423355]
[New LWP 1423354]
[New LWP 1423353]
[New LWP 1423352]
[New LWP 1423351]
[New LWP 1423350]
[New LWP 1423349]
[New LWP 1423348]
[New LWP 1423347]
[New LWP 1423346]
[New LWP 1423345]
[New LWP 1423344]
[New LWP 1423343]
[New LWP 1423342]
[New LWP 1423341]
[New LWP 1423339]
[New LWP 1423338]
[New LWP 1423337]
[New LWP 1423336]
[New LWP 1423329]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
warning: 56	../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56	in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x000078aae64a013c in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
warning: 49	./nptl/cancellation.c: No such file or directory
#2  __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75	in ./nptl/cancellation.c
#3  0x000078aae651ca0f in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4  0x000078aae765fe13 in ggml_print_backtrace () from /devel/tools/llama.cpp/build/bin/libggml-base.so.0
#5  0x000078aae765ffc6 in ggml_abort () from /devel/tools/llama.cpp/build/bin/libggml-base.so.0
#6  0x000078a9f4ff2e15 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /devel/tools/llama.cpp/build/bin/libggml-cuda.so
#7  0x000078a9f4ff718e in ggml_backend_cuda_set_tensor_async(ggml_backend*, ggml_tensor*, void const*, unsigned long, unsigned long) () from /devel/tools/llama.cpp/build/bin/libggml-cuda.so
#8  0x000078aae767c863 in ggml_backend_sched_graph_compute_async () from /devel/tools/llama.cpp/build/bin/libggml-base.so.0
#9  0x000078aae6ccd390 in llama_context::graph_compute(ggml_cgraph*, bool) () from /devel/tools/llama.cpp/build/bin/libllama.so.0
#10 0x000078aae6ccfd53 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /devel/tools/llama.cpp/build/bin/libllama.so.0
#11 0x000078aae6cd67af in llama_context::decode(llama_batch const&) () from /devel/tools/llama.cpp/build/bin/libllama.so.0
#12 0x000078aae6cd8362 in llama_decode () from /devel/tools/llama.cpp/build/bin/libllama.so.0
#13 0x00005bfc77d6501e in server_context_impl::update_slots() ()
#14 0x00005bfc77db480f in server_queue::start_loop(long) ()
#15 0x00005bfc77cb4586 in main ()
[Inferior 1 (process 1423301) detached]
Aborted (core dumped)

  19. pwilkin commented on Mar 14, 2026

    @pwilkin
    pwilkin
    on Mar 14, 2026
    MemberAuthor

    Same stuff as before in dmesg:

    [680309.059273] NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
[680309.060277] NVRM: Xid (PCI:0000:06:00): 8, pid=1390545, name=llama-server, channel 0x00000007

  20. pwilkin commented on Mar 17, 2026

    @pwilkin
    pwilkin
    on Mar 17, 2026
    MemberAuthor

    Still happening. I thought it might be memory pressure, so I ran the server on --cpu-moe only, but this still happens. Maybe some memory leak?

    I'm running with reduced checkpoints now, from 32 to 8, will see if it helps. Any ideas what I could do to debug this further?

  21. am17an commented on Mar 17, 2026

    @am17an
    am17an
    on Mar 17, 2026
    Contributor

    Maybe re-install your OS?

  22. ggerganov commented on Mar 17, 2026

    @ggerganov
    ggerganov
    on Mar 17, 2026
    Member

    From googling a bit, it sounds like a hardware issue.

  23. pwilkin commented on Mar 17, 2026

    @pwilkin
    pwilkin
    on Mar 17, 2026
    MemberAuthor

    Eh, the thing is, I did diagnostics and it doesn't look like a hardware issue. Here's what Claude says after I fed it all the results from logs etc.

    The pattern (both cards, only under load, cool and underutilized power-wise, specific to llama-server) points squarely at llama.cpp issuing a CUDA kernel that the watchdog interprets as a hang — even if the GPU would have eventually completed it.

    The biggest issue here is it's happening on both cards pretty randomly - hard to imagine both (relatively new) cards just broke down. I'll try increasing the watchdog timeout to see if it's an actual issue with the kernels.

  24. am17an commented on Mar 17, 2026

    @am17an
    am17an
    on Mar 17, 2026
    Contributor

    Well unless you can give us a repro, it might be hard to do. You can probably run a release build with debug symbols and share the core dump if that's possible

  25. pwilkin commented on Mar 17, 2026

    @pwilkin
    pwilkin
    on Mar 17, 2026
    MemberAuthor

    Okay, build RelWithDbgInfo and set ulimit -c unlimited, let's see if I can get something.

  26. pwilkin commented on Mar 23, 2026

    @pwilkin
    pwilkin
    on Mar 23, 2026
    MemberAuthor

    Just a FYI: after I increased the watchdog timeout to 60s, the problem hasn't come back, so maybe it was a really long-running kernel.

  27. pwilkin commented on Jul 8, 2026

    @pwilkin
    pwilkin
    on Jul 8, 2026
    MemberAuthor

    If someone wants to know how to do it:

    Add to /etc/modprobe.d/nvidia-graphics-drivers-kms.conf

    options nvidia NVreg_RegistryDwords="RmWatchDogTimeout=60"
  28. pwilkin
    closed this as completedon Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CUDARelated to the CUDA backendbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions