Skip to content

Eval bug: CUDA OOM when loading Qwen 3.6 model in -sm tensor #22476

Closed as not planned
Closed as not planned
Eval bug: CUDA OOM when loading Qwen 3.6 model in -sm tensor#22476
Labels
CUDARelated to the CUDA backendNvidia GPUIssues specific to Nvidia GPUsbug-unconfirmedstaletensor-parallelTensor parallelism feature with meta backend.

Description

@mdierolf
mdierolf
opened on Apr 28, 2026

Name and Version

Docker Image:
ghcr.io/ggml-org/llama.cpp server-cuda 1f0b553ce3a3 7 hours ago 3.61GB

Docker Stack File:

version: '3.8'

services:
  server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    command: -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-IQ4_XS --port 11434 --host 0.0.0.0 --no-mmap -fa on -kvu -sm tensor

Operating systems

Linux

GGML backends

CUDA

Hardware

2xRTX A2000

Models

unsloth/Qwen3.6-35B-A3B-GGUF:UD-IQ4_XS

Problem description & steps to reproduce

With -sm tensor, Llama.cpp crashes on startup, when loading the model on 2xRTX A2000.

The same system works perfectly well without the -sm tensor flag

First Bad Commit

No response

Relevant log output

Logs 
ph3a:1:110 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/direct pointer
ph3a:1:111 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/direct pointer
ph3a:1:110 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/direct pointer
ph3a:1:111 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/direct pointer
ph3a:1:110 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/direct pointer

ph3a:1:107 [1] include/alloc.h:228 NCCL WARN Cuda failure 2 'out of memory'
ph3a:1:107 [1] NCCL INFO transport/p2p.cc:216 -> 1
ph3a:1:107 [1] NCCL INFO transport/p2p.cc:647 -> 1
ph3a:1:110 [1] NCCL INFO transport/p2p.cc:419 -> 1
ph3a:1:110 [1] NCCL INFO transport.cc:34 -> 1
ph3a:1:110 [1] NCCL INFO transport.cc:152 -> 1
ph3a:1:110 [1] NCCL INFO transport/generic.cc:19 -> 1
ph3a:1:110 [1] NCCL INFO group.cc:148 -> 1
ph3a:1:110 [1] NCCL INFO group.cc:75 -> 1 [Async thread]
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 24064 MiB):
  Device 0: NVIDIA RTX A2000 12GB, compute capability 8.6, VMM: yes, VRAM: 12032 MiB
  Device 1: NVIDIA RTX A2000 12GB, compute capability 8.6, VMM: yes, VRAM: 12032 MiB
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build_info: b8953-434b2a1ff
system_info: n_threads = 24 (n_threads_batch = 24) / 48 | CUDA : ARCHS = 500,610,700,750,800,860,890,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
Running without SSL
init: using 47 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort
common_fit_params: fitting params to free memory took 0.00 seconds
llama_model_load_from_file_impl: skipping CPU (AMD EPYC 7443P 24-Core Processor) for tensor parallelism
llama_model_load_from_file_impl: creating a Meta device for tensor parallelism from 2 devices:
llama_model_load_from_file_impl: - device 0: CUDA0 (NVIDIA RTX A2000 12GB)
llama_model_load_from_file_impl: - device 1: CUDA1 (NVIDIA RTX A2000 12GB)
llama_model_load_from_file_impl: using device Meta() (Meta()) (unknown id) - 23860 MiB free
llama_model_loader: loaded meta data with 54 key-value pairs and 733 tensors from /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen35moe
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                     general.sampling.top_k i32              = 20
llama_model_loader: - kv   3:                     general.sampling.top_p f32              = 0.950000
llama_model_loader: - kv   4:                      general.sampling.temp f32              = 1.000000
llama_model_loader: - kv   5:                               general.name str              = Qwen3.6-35B-A3B
llama_model_loader: - kv   6:                           general.basename str              = Qwen3.6-35B-A3B
llama_model_loader: - kv   7:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   8:                         general.size_label str              = 35B-A3B
llama_model_loader: - kv   9:                            general.license str              = apache-2.0
llama_model_loader: - kv  10:                       general.license.link str              = https://huggingface.co/Qwen/Qwen3.6-3...
llama_model_loader: - kv  11:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv  12:                   general.base_model.count u32              = 1
llama_model_loader: - kv  13:                  general.base_model.0.name str              = Qwen3.6 35B A3B
llama_model_loader: - kv  14:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  15:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen3.6-3...
llama_model_loader: - kv  16:                               general.tags arr[str,3]       = ["qwen3_5_moe", "qwen", "image-text-t...
llama_model_loader: - kv  17:                      qwen35moe.block_count u32              = 40
llama_model_loader: - kv  18:                   qwen35moe.context_length u32              = 262144
llama_model_loader: - kv  19:                 qwen35moe.embedding_length u32              = 2048
llama_model_loader: - kv  20:             qwen35moe.attention.head_count u32              = 16
llama_model_loader: - kv  21:          qwen35moe.attention.head_count_kv u32              = 2
llama_model_loader: - kv  22:          qwen35moe.rope.dimension_sections arr[i32,4]       = [11, 11, 10, 0]
llama_model_loader: - kv  23:                   qwen35moe.rope.freq_base f32              = 10000000.000000
llama_model_loader: - kv  24: qwen35moe.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  25:                     qwen35moe.expert_count u32              = 256
llama_model_loader: - kv  26:                qwen35moe.expert_used_count u32              = 8
llama_model_loader: - kv  27:             qwen35moe.attention.key_length u32              = 256
llama_model_loader: - kv  28:           qwen35moe.attention.value_length u32              = 256
llama_model_loader: - kv  29:       qwen35moe.expert_feed_forward_length u32              = 512
llama_model_loader: - kv  30: qwen35moe.expert_shared_feed_forward_length u32              = 512
llama_model_loader: - kv  31:                  qwen35moe.ssm.conv_kernel u32              = 4
llama_model_loader: - kv  32:                   qwen35moe.ssm.state_size u32              = 128
llama_model_loader: - kv  33:                  qwen35moe.ssm.group_count u32              = 16
llama_model_loader: - kv  34:               qwen35moe.ssm.time_step_rank u32              = 32
llama_model_loader: - kv  35:                   qwen35moe.ssm.inner_size u32              = 4096
llama_model_loader: - kv  36:          qwen35moe.full_attention_interval u32              = 4
llama_model_loader: - kv  37:             qwen35moe.rope.dimension_count u32              = 64
llama_model_loader: - kv  38:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  39:                         tokenizer.ggml.pre str              = qwen35
llama_model_loader: - kv  40:                      tokenizer.ggml.tokens arr[str,248320]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  41:                  tokenizer.ggml.token_type arr[i32,248320]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  42:                      tokenizer.ggml.merges arr[str,247587]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  43:                tokenizer.ggml.eos_token_id u32              = 248046
llama_model_loader: - kv  44:            tokenizer.ggml.padding_token_id u32              = 248055
llama_model_loader: - kv  45:                tokenizer.ggml.bos_token_id u32              = 248044
llama_model_loader: - kv  46:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  47:                    tokenizer.chat_template str              = {%- set image_count = namespace(value...
llama_model_loader: - kv  48:               general.quantization_version u32              = 2
llama_model_loader: - kv  49:                          general.file_type u32              = 30
llama_model_loader: - kv  50:                      quantize.imatrix.file str              = Qwen3.6-35B-A3B-GGUF/imatrix_unsloth....
llama_model_loader: - kv  51:                   quantize.imatrix.dataset str              = unsloth_calibration_Qwen3.6-35B-A3B.txt
llama_model_loader: - kv  52:             quantize.imatrix.entries_count u32              = 510
llama_model_loader: - kv  53:              quantize.imatrix.chunks_count u32              = 76
llama_model_loader: - type  f32:  361 tensors
llama_model_loader: - type q8_0:  251 tensors
llama_model_loader: - type q6_K:    4 tensors
llama_model_loader: - type iq3_s:   80 tensors
llama_model_loader: - type iq4_xs:   37 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = IQ4_XS - 4.25 bpw
print_info: file size   = 16.50 GiB (4.09 BPW) 
load: 0 unused tokens
load: printing all EOG tokens:
load:   - 248044 ('<|endoftext|>')
load:   - 248046 ('<|im_end|>')
load:   - 248063 ('<|fim_pad|>')
load:   - 248064 ('<|repo_name|>')
load:   - 248065 ('<|file_sep|>')
load: special tokens cache size = 33
load: token to piece cache size = 1.7581 MB
print_info: arch                  = qwen35moe
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 262144
print_info: n_embd                = 2048
print_info: n_embd_inp            = 2048
print_info: n_layer               = 40
print_info: n_head                = 16
print_info: n_head_kv             = 2
print_info: n_rot                 = 64
print_info: n_swa                 = 0
print_info: is_swa_any            = 0
print_info: n_embd_head_k         = 256
print_info: n_embd_head_v         = 256
print_info: n_gqa                 = 8
print_info: n_embd_k_gqa          = 512
print_info: n_embd_v_gqa          = 512
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 0.0e+00
print_info: n_ff                  = 0
print_info: n_expert              = 256
print_info: n_expert_used         = 8
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 1
print_info: pooling type          = -1
print_info: rope type             = 40
print_info: rope scaling          = linear
print_info: freq_base_train       = 10000000.0
print_info: freq_scale_train      = 1
print_info: n_ctx_orig_yarn       = 262144
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: mrope sections        = [11, 11, 10, 0]
print_info: ssm_d_conv            = 4
print_info: ssm_d_inner           = 4096
print_info: ssm_d_state           = 128
print_info: ssm_dt_rank           = 32
print_info: ssm_n_group           = 16
print_info: ssm_dt_b_c_rms        = 0
print_info: model type            = 35B.A3B
print_info: model params          = 34.66 B
print_info: general.name          = Qwen3.6-35B-A3B
print_info: vocab type            = BPE
print_info: n_vocab               = 248320
print_info: n_merges              = 247587
print_info: BOS token             = 248044 '<|endoftext|>'
print_info: EOS token             = 248046 '<|im_end|>'
print_info: EOT token             = 248046 '<|im_end|>'
print_info: PAD token             = 248055 '<|vision_pad|>'
print_info: LF token              = 198 'Ċ'
print_info: FIM PRE token         = 248060 '<|fim_prefix|>'
print_info: FIM SUF token         = 248062 '<|fim_suffix|>'
print_info: FIM MID token         = 248061 '<|fim_middle|>'
print_info: FIM PAD token         = 248063 '<|fim_pad|>'
print_info: FIM REP token         = 248064 '<|repo_name|>'
print_info: FIM SEP token         = 248065 '<|file_sep|>'
print_info: EOG token             = 248044 '<|endoftext|>'
print_info: EOG token             = 248046 '<|im_end|>'
print_info: EOG token             = 248063 '<|fim_pad|>'
print_info: EOG token             = 248064 '<|repo_name|>'
print_info: EOG token             = 248065 '<|file_sep|>'
print_info: max token length      = 256
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
load_tensors: offloading output layer to GPU
load_tensors: offloading 39 repeating layers to GPU
load_tensors: offloaded 41/41 layers to GPU
load_tensors:    CUDA_Host model buffer size =   515.31 MiB
load_tensors:       Meta() model buffer size =  8295.92 MiB
................................................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
common_init_result: added <|repo_name|> logit bias = -inf
common_init_result: added <|file_sep|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 4
llama_context: n_ctx         = 262144
llama_context: n_ctx_seq     = 262144
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 10000000.0
llama_context: freq_scale    = 1
llama_context:  CUDA_Host  output buffer size =     3.79 MiB
llama_kv_cache:     Meta() KV buffer size =  2560.00 MiB
llama_kv_cache: size = 5120.00 MiB (262144 cells,  10 layers,  4/1 seqs), K (f16): 2560.00 MiB, V (f16): 2560.00 MiB
llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
llama_memory_recurrent:     Meta() RS buffer size =   125.62 MiB
llama_memory_recurrent: size =  251.25 MiB (     4 cells,  40 layers,  4 seqs), R (f32):   11.25 MiB, S (f32):  240.00 MiB
sched_reserve: reserving ...
sched_reserve: resolving fused Gated Delta Net support:
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve:     Meta() compute buffer size =   804.02 MiB
sched_reserve:  CUDA_Host compute buffer size =   520.02 MiB
sched_reserve: graph nodes  = 3729
sched_reserve: graph splits = 2
sched_reserve: reserve took 472.91 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
ph3a:1:1 [1] NCCL INFO cudaDriverVersion 12080
ph3a:1:1 [1] NCCL INFO Bootstrap: Using wan0:184.105.177.19<0>
ph3a:1:1 [0] NCCL INFO NCCL version 2.25.1+cuda12.8
ph3a:1:103 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. Using internal network plugin.
ph3a:1:103 [0] NCCL INFO Failed to open libibverbs.so[.1]
ph3a:1:103 [0] NCCL INFO NET/Socket : Using [0]wan0:184.105.177.19<0> [1]veth4526442:fe80::f3:8ff:fe0f:f598%veth4526442<0>
ph3a:1:103 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
ph3a:1:103 [0] NCCL INFO Using network Socket
ph3a:1:104 [1] NCCL INFO Using network Socket
ph3a:1:103 [0] NCCL INFO ncclCommInitAll comm 0x557f3d32db30 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 41000 commId 0xa62eff62461dc938 - Init START
ph3a:1:104 [1] NCCL INFO ncclCommInitAll comm 0x557f3d4ae310 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId c1000 commId 0xa62eff62461dc938 - Init START
ph3a:1:103 [0] NCCL INFO RAS client listening socket at ::1<28028>
ph3a:1:103 [0] NCCL INFO Bootstrap timings total 0.001098 (create 0.000033, send 0.000122, recv 0.000264, ring 0.000061, delay 0.000000)
ph3a:1:104 [1] NCCL INFO Bootstrap timings total 0.001067 (create 0.000043, send 0.000172, recv 0.000364, ring 0.000054, delay 0.000000)
ph3a:1:103 [0] NCCL INFO comm 0x557f3d32db30 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
ph3a:1:104 [1] NCCL INFO comm 0x557f3d4ae310 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
ph3a:1:103 [0] NCCL INFO Channel 00/04 : 0 1
ph3a:1:103 [0] NCCL INFO Channel 01/04 : 0 1
ph3a:1:103 [0] NCCL INFO Channel 02/04 : 0 1
ph3a:1:103 [0] NCCL INFO Channel 03/04 : 0 1
ph3a:1:104 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 [2] -1/-1/-1->1->0 [3] 0/-1/-1->1->-1
ph3a:1:103 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 [2] 1/-1/-1->0->-1 [3] -1/-1/-1->0->1
CUDA error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
  current device: 0, in function ggml_backend_cuda_comm_allreduce_tensor at /app/ggml/src/ggml-cuda/ggml-cuda.cu:1237
  ncclGroupEnd()
/app/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
ph3a:1:103 [0] NCCL INFO P2P Chunksize set to 131072
ph3a:1:104 [1] NCCL INFO P2P Chunksize set to 131072
ph3a:1:103 [0] NCCL INFO Check P2P Type intraNodeP2pSupport 1 directMode 1
ph3a:1:106 [0] NCCL INFO [Proxy Service] Device 0 CPU core 44
ph3a:1:107 [1] NCCL INFO [Proxy Service] Device 1 CPU core 42
ph3a:1:108 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 8
ph3a:1:109 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 38
ph3a:1:103 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
ph3a:1:103 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
ph3a:1:104 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
ph3a:1:104 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
ph3a:1:103 [0] NCCL INFO CC Off, workFifoBytes 1048576
ph3a:1:103 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so libnccl-net.so. Using internal tuner plugin.
ph3a:1:103 [0] NCCL INFO ncclCommInitAll comm 0x557f3d32db30 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 41000 commId 0xa62eff62461dc938 - Init COMPLETE
ph3a:1:103 [0] NCCL INFO Init timings - ncclCommInitAll: rank 0 nranks 2 total 0.34 (kernels 0.28, alloc 0.05, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
ph3a:1:104 [1] NCCL INFO ncclCommInitAll comm 0x557f3d4ae310 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId c1000 commId 0xa62eff62461dc938 - Init COMPLETE
ph3a:1:104 [1] NCCL INFO Init timings - ncclCommInitAll: rank 1 nranks 2 total 0.34 (kernels 0.28, alloc 0.05, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
ph3a:1:110 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/direct pointer
ph3a:1:111 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/direct pointer
ph3a:1:110 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/direct pointer
ph3a:1:111 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/direct pointer
ph3a:1:110 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/direct pointer

ph3a:1:107 [1] include/alloc.h:228 NCCL WARN Cuda failure 2 'out of memory'
ph3a:1:107 [1] NCCL INFO transport/p2p.cc:216 -> 1
ph3a:1:107 [1] NCCL INFO transport/p2p.cc:647 -> 1
ph3a:1:110 [1] NCCL INFO transport/p2p.cc:419 -> 1
ph3a:1:110 [1] NCCL INFO transport.cc:34 -> 1
ph3a:1:110 [1] NCCL INFO transport.cc:152 -> 1
ph3a:1:110 [1] NCCL INFO transport/generic.cc:19 -> 1
ph3a:1:110 [1] NCCL INFO group.cc:148 -> 1
ph3a:1:110 [1] NCCL INFO group.cc:75 -> 1 [Async thread]
ph3a:1:111 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/direct pointer
ph3a:1:111 [0] NCCL INFO misc/socket.cc:64 -> 3
ph3a:1:111 [0] NCCL INFO misc/socket.cc:80 -> 3
ph3a:1:111 [0] NCCL INFO misc/socket.cc:828 -> 3
ph3a:1:111 [0] NCCL INFO proxy.cc:1209 -> 3
ph3a:1:111 [0] NCCL INFO proxy.cc:1290 -> 3
ph3a:1:111 [0] NCCL INFO proxy.cc:1098 -> 3
ph3a:1:111 [0] NCCL INFO transport/p2p.cc:414 -> 3
ph3a:1:111 [0] NCCL INFO transport.cc:34 -> 3
ph3a:1:111 [0] NCCL INFO transport.cc:152 -> 3
ph3a:1:106 [0] NCCL INFO misc/socket.cc:880 -> 3
ph3a:1:111 [0] NCCL INFO transport/generic.cc:19 -> 3
ph3a:1:111 [0] NCCL INFO group.cc:148 -> 3
ph3a:1:111 [0] NCCL INFO group.cc:75 -> 3 [Async thread]
ph3a:1:1 [0] NCCL INFO group.cc:454 -> 1
ph3a:1:1 [0] NCCL INFO group.cc:573 -> 1
ph3a:1:1 [0] NCCL INFO group.cc:106 -> 1
libggml-base.so.0(+0x1ac36)[0x7fb653959c36]
libggml-base.so.0(ggml_print_backtrace+0x21a)[0x7fb65395a0ba]
libggml-base.so.0(ggml_abort+0x15b)[0x7fb65395a29b]
/app/libggml-cuda.so(_Z15ggml_cuda_errorPKcS0_S0_iS0_+0xb5)[0x7fb649be3ba5]
/app/libggml-cuda.so(+0x1e8273)[0x7fb649be8273]
libggml-base.so.0(+0x457e7)[0x7fb6539847e7]
libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x82f)[0x7fb653977d2f]
libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1)[0x7fb653ab8a31]
libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x112)[0x7fb653abb442]
libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x36f)[0x7fb653ac139f]
libllama.so.0(llama_decode+0xf)[0x7fb653ac2faf]
libllama-common.so.0(_Z23common_init_from_paramsR13common_params+0x3c6)[0x7fb654007576]
/app/llama-server(+0x11924e)[0x557f0078e24e]
/app/llama-server(+0x6d6af)[0x557f006e26af]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7fb65322a1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7fb65322a28b]
/app/llama-server(+0x6e3c5)[0x557f006e33c5]

Activity

  1. mdierolf
    added
    bug-unconfirmed
    on Apr 28, 2026
  2. mdierolf
    changed the title [-]Eval bug: CUDA OOM when fitting Qwen 3.6 model in -sm tensor[/-] [+]Eval bug: CUDA OOM when loading Qwen 3.6 model in -sm tensor[/+] on Apr 28, 2026
  3. taronaeo
    added
    Nvidia GPUIssues specific to Nvidia GPUs
    CUDARelated to the CUDA backend
    tensor-parallelTensor parallelism feature with meta backend.
    on Apr 28, 2026

  4. am17an commented on Apr 28, 2026

    @am17an
    am17an
    on Apr 28, 2026
    Contributor

    cc @JohannesGaessler @gaugarg-nv

  5. gaugarg-nv commented on Apr 28, 2026

    @gaugarg-nv
    gaugarg-nv
    on Apr 28, 2026
    Contributor

    I think there is not enough memory left for NCCL as llama.cpp tries to allocate memory for the full context. Can you try limiting the context by passing -c 4096 in the command line?

  6. github-actions
    added
    stale
    on May 29, 2026

  7. github-actions commented on Jun 12, 2026

    @github-actions
    github-actionsBot
    on Jun 12, 2026 – with GitHub Actions
    Contributor

    This issue was closed because it has been inactive for 14 days since being marked as stale.

  8. github-actions
    closed this as not plannedon Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CUDARelated to the CUDA backendNvidia GPUIssues specific to Nvidia GPUsbug-unconfirmedstaletensor-parallelTensor parallelism feature with meta backend.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions