Skip to content

Eval bug: TP: Stepfun 3.7 does not work with uneven splits (GGML_META_DEBUG=1) #24486

Closed as not planned
#24554
Closed as not planned
Eval bug: TP: Stepfun 3.7 does not work with uneven splits (GGML_META_DEBUG=1)#24486
#24554
Labels
bug-unconfirmedstale

Description

@krampenschiesser
krampenschiesser
opened on Jun 11, 2026

Name and Version

$./llama-cli --version
version: 9598 (fdc3db9b6)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

10x 5060TI 16gb
CUDA 13.0.3

Models

Stepfun 3.7 in Q4_K_M from aessedai, and IQ4_XS from stepfun

Problem description & steps to reproduce

I can run stepfun 3.7 with 8 GPUS:

CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7,8 llama-server -fa 1 -m Step-3.7-flash-IQ4_XS-00001-of-00003.gguf -sm tensor --no-mmap -dio -ngl 99 --fit off --port 10001 -c 4096

however one I add the 9th or even 10th GPU it fails with:

CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7,8,9 llama-server -fa 1 -m Step-3.7-flash-IQ4_XS-00001-of-00003.gguf -sm tensor --no-mmap -dio -ngl 99 --fit off --port 10001 -c 4096 
0.01.345.959 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.01.345.963 I device_info:
0.01.499.142 I   - CUDA0   : NVIDIA GeForce RTX 5060 Ti (15847 MiB, 15695 MiB free)
0.01.662.416 I   - CUDA1   : NVIDIA GeForce RTX 5060 Ti (15847 MiB, 15695 MiB free)
0.01.800.541 I   - CUDA2   : NVIDIA GeForce RTX 5060 Ti (15847 MiB, 15695 MiB free)
0.01.935.820 I   - CUDA3   : NVIDIA GeForce RTX 5060 Ti (15847 MiB, 15695 MiB free)
0.02.066.852 I   - CUDA4   : NVIDIA GeForce RTX 5060 Ti (15847 MiB, 15695 MiB free)
0.02.232.435 I   - CUDA5   : NVIDIA GeForce RTX 5060 Ti (15847 MiB, 15695 MiB free)
0.02.360.044 I   - CUDA6   : NVIDIA GeForce RTX 5060 Ti (15847 MiB, 15695 MiB free)
0.02.492.037 I   - CUDA7   : NVIDIA GeForce RTX 5060 Ti (15847 MiB, 15695 MiB free)
0.02.627.199 I   - CUDA8   : NVIDIA GeForce RTX 5060 Ti (15847 MiB, 15646 MiB free)
0.02.627.219 I   - CPU     : AMD EPYC 9135 16-Core Processor (93628 MiB, 93628 MiB free)
0.02.627.339 I system_info: n_threads = 16 (n_threads_batch = 16) / 32 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.02.627.348 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.02.627.387 I srv          init: running without SSL
0.02.627.421 I srv          init: using 31 threads for HTTP server
0.02.627.530 I srv         start: binding port with default address family
0.02.628.712 I srv  llama_server: loading model
0.02.628.721 I srv    load_model: loading model 'Step-3.7-flash-IQ4_XS-00001-of-00003.gguf'
1.43.305.379 W llama_context: n_ctx_seq (4096) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
1.43.931.608 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
/home/scar-ai/projects/llama.cpp/ggml/src/ggml-backend-meta.cpp:535: GGML_ASSERT(ret.axis != GGML_BACKEND_SPLIT_AXIS_UNKNOWN) failed
#4  0x00007674213474f3 in ggml_print_backtrace () from /home/scar-ai/projects/llama.cpp/build/bin/libggml-base.so.0
#5  0x00007674213476a6 in ggml_abort () from /home/scar-ai/projects/llama.cpp/build/bin/libggml-base.so.0
#6  0x0000767421370fd9 in ggml_backend_meta_get_split_state(ggml_backend_meta_simple_tensor_container&, ggml_tensor const*, bool)::{lambda()#1}::operator()() const () from /home/scar-ai/projects/llama.cpp/build/bin/libggml-base.so.0
#7  0x0000767421369322 in ggml_backend_meta_get_split_state(ggml_backend_meta_simple_tensor_container&, ggml_tensor const*, bool) () from /home/scar-ai/projects/llama.cpp/build/bin/libggml-base.so.0
#8  0x0000767421375287 in ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_meta_simple_tensor_container&, ggml_tensor*) [clone .isra.0] () from /home/scar-ai/projects/llama.cpp/build/bin/libggml-base.so.0
#9  0x000076742137787a in ggml_backend_meta_buffer_init_tensor(ggml_backend_buffer*, ggml_tensor*) () from /home/scar-ai/projects/llama.cpp/build/bin/libggml-base.so.0
#10 0x000076742135dd15 in ggml_gallocr_alloc_graph () from /home/scar-ai/projects/llama.cpp/build/bin/libggml-base.so.0
#11 0x0000767421364291 in ggml_backend_sched_alloc_graph () from /home/scar-ai/projects/llama.cpp/build/bin/libggml-base.so.0
#12 0x00007674206de8e7 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/scar-ai/projects/llama.cpp/build/bin/libllama.so.0
#13 0x00007674206e5875 in llama_context::decode(llama_batch const&) () from /home/scar-ai/projects/llama.cpp/build/bin/libllama.so.0
#14 0x00007674206e7622 in llama_decode () from /home/scar-ai/projects/llama.cpp/build/bin/libllama.so.0
#15 0x0000767420c1cd6c in common_init_from_params(common_params&, bool) () from /home/scar-ai/projects/llama.cpp/build/bin/libllama-common.so.0
#16 0x000076742199255f in server_context_impl::load_model(common_params&) () from /home/scar-ai/projects/llama.cpp/build/bin/libllama-server-impl.so
#17 0x00007674218d8fad in llama_server(int, char**) () from /home/scar-ai/projects/llama.cpp/build/bin/libllama-server-impl.so

First Bad Commit

No response

Relevant log output

Logs

I attached both the successful and failed logs with `

failure.log
success.log

`





Activity
  1. krampenschiesser
    added 
    bug-unconfirmed
     on Jun 11, 2026
  2. krampenschiesser commented on Jun 11, 2026 
    @krampenschiesser
    krampenschiesser
    on Jun 11, 2026
    ContributorAuthor
    From what i can see the blk.0.attn_gate.weight [4 096, 64] is split along axis 1->64.
    
that is nicely divisible by 8 but does not work for divisions with remainders.
    
So this seems to be more of a general issue with uneven splits.
    
Successful:
    

    {blk.0.attn_gate.weight[NONE, 1, {8x1, 8x1, 8x1, 8x1, 8x1, 8x1, 8x1, 8x1}], attn_norm-0[MUL, 1, {8x1, 8x1, 8x1, 8x1, 8x1, 8x1, 8x1, 8x1}]} -> attn_gate-0[MUL_MAT, 0, {8x1, 8x1, 8x1, 8x1, 8x1, 8x1, 8x1, 8x1}]

    

    Failure:
    

    D SPLIT_STATE: {blk.0.attn_gate.weight[NONE, 1, {7x1, 7x1, 7x1, 7x1, 7x1, 7x1, 7x1, 7x1, 8x1}], attn_norm-0[MUL, 1, {7x1, 7x1, 7x1, 7x1, 7x1, 7x1, 7x1, 7x1, 8x1}]} -> attn_gate-0[MUL_MAT, 0, {7x1, 7x1, 7x1, 7x1, 7x1, 7x1, 7x1, 7x1, 8x1}]

    

    Also the next layer has 96 in the 2nd dim: blk.1.attn_gate.weight [4 096, 96]
  3. krampenschiesser commented on Jun 11, 2026 
    @krampenschiesser
    krampenschiesser
    on Jun 11, 2026
    ContributorAuthor
    Tested with 7 gpus and got same failure:
    

    {blk.0.attn_gate.weight[NONE, 1, {9x1, 9x1, 9x1, 9x1, 9x1, 9x1, 10x1}], attn_norm-0[MUL, 1, {9x1, 9x1, 9x1, 9x1, 9x1, 9x1, 10x1}]} -> attn_gate-0[MUL_MAT, 0, {9x1, 9x1, 9x1, 9x1, 9x1, 9x1, 10x1}]

    

    /home/scar-ai/projects/llama.cpp/ggml/src/ggml-backend-meta.cpp:535: GGML_ASSERT(ret.axis != GGML_BACKEND_SPLIT_AXIS_UNKNOWN) failed

  4. krampenschiesser
    changed the title [-]Eval bug: TP: Stepfun 3.7 works in split mode tensor with 8 GPUs but not with 9 or 10 (GGML_META_DEBUG=1)[/-] [+]Eval bug: TP: Stepfun 3.7 works in split mode tensor does not work with uneven splits (GGML_META_DEBUG=1)[/+] on Jun 11, 2026
  5. krampenschiesser
    changed the title [-]Eval bug: TP: Stepfun 3.7 works in split mode tensor does not work with uneven splits (GGML_META_DEBUG=1)[/-] [+]Eval bug: TP: Stepfun 3.7 does not work with uneven splits (GGML_META_DEBUG=1)[/+] on Jun 11, 2026
  6. krampenschiesser commented on Jun 13, 2026 
    @krampenschiesser
    krampenschiesser
    on Jun 13, 2026
    ContributorAuthor
    added a pr for this here: #24554
  7. krampenschiesser commented on Jun 15, 2026 
    @krampenschiesser
    krampenschiesser
    on Jun 15, 2026
    ContributorAuthor
    failure-debug2.log
  8. lee-b commented on Jun 20, 2026 
    @lee-b
    lee-b
    on Jun 20, 2026
    Just not step**. This also affects some qwen3.6 27B quants (at least).
    

    0.01.032.004 I srv    load_model: loading model '/models/llms/mradermacher--Qwen3.6-27B-heretic-heretic2-GGUF/Qwen3.6-27B-heretic-heretic2.Q6_K.gguf'
0.03.084.635 I srv    load_model: [mtmd] estimated worst-case memory usage of mmproj is 3925.03 MiB
0.03.084.649 I common_init_result: fitting params to device memory ...
0.03.084.650 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.03.084.711 W common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort
0.10.570.059 W llama_context: n_ctx_seq (131072) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.11.199.586 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
/app/ggml/src/ggml-backend-meta.cpp:535: GGML_ASSERT(ret.axis != GGML_BACKEND_SPLIT_AXIS_UNKNOWN) failed
libggml-base.so.0(+0x1b1f6)[0x7f13b930d1f6]
libggml-base.so.0(ggml_print_backtrace+0x21a)[0x7f13b930d67a]
libggml-base.so.0(ggml_abort+0x15b)[0x7f13b930d85b]
libggml-base.so.0(+0x46041)[0x7f13b9338041]
libggml-base.so.0(+0x3e005)[0x7f13b9330005]
libggml-base.so.0(+0x47999)[0x7f13b9339999]
libggml-base.so.0(+0x4a2a0)[0x7f13b933c2a0]
libggml-base.so.0(ggml_gallocr_alloc_graph+0x477)[0x7f13b9323ae7]
libggml-base.so.0(ggml_backend_sched_alloc_graph+0x111)[0x7f13b932a161]
libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xeb)[0x7f13b94b01ab]
libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x378)[0x7f13b94b5f98]
libllama.so.0(llama_decode+0xf)[0x7f13b94b7daf]
libllama-common.so.0(_Z23common_init_from_paramsR13common_paramsb+0x41a)[0x7f13b9a5bc3a]
libllama-server-impl.so(_ZN19server_context_impl10load_modelER13common_params+0x60d)[0x7f13ba34d03d]
libllama-server-impl.so(_Z12llama_serveriPPc+0x2fae)[0x7f13ba281d8e]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7f13b9cfe1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7f13b9cfe28b]
/app/llama-server(+0x1315)[0x55e0da09a315]

    

    ** Though Step working is an actual concern too.
  9. github-actions
    added 
    stale
     on Jul 21, 2026
  10. github-actions commented on Aug 4, 2026 
    @github-actions
    github-actionsBot
    on Aug 4, 2026 – with GitHub Actions
    Contributor
    This issue was closed because it has been inactive for 14 days since being marked as stale.
  11. github-actions
    closed this as not plannedon Aug 4, 2026
  12. lee-b commented on Aug 4, 2026 
    @lee-b
    lee-b
    on Aug 4, 2026
    

    This issue was closed because it has been inactive for 14 days since being marked as stale.
    

    

    :/
    

    gotta disable this, or at least make the timeout reflective of actual ticket throughput for all real issues.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Metadata
Assignees
No one assigned
    Labels
    bug-unconfirmedstale
    Type
    No type
    Projects
    No projects
      Milestone
      No milestone
      Relationships
      None yet
      Development
      No branches or pull requests
      Issue actions