[krampenschiesser]

Overview

This fixes #24486 and allows the usage of 4-10 and probably more gpu's. I cannot test with <4 because of too little vram.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes I used llama.cpp and qwen3.5 122b as an advanced grep tool for discovering the codebase all code comes from me (as little as there is).

[krampenschiesser]

I am seeing some problems with tensor splits by going larger with granularity. I will try to implement the discovery of additional tensors during the std::vec initialization. this will include the passthrough of the user data+model into the ggml_backend_meta_get_split_state function so I can ask the model about additional tensors by tensor name.
will post some logs tomorrow regarding the core issue with the graph.

[krampenschiesser]

This is way better.
works in all configurations i could test (4-10 gpus) probably with 2,3 and 10+ too.
Tensor splits look better too.
Should be good for review.
@JohannesGaessler please take a look, i don't know if it is ok to add more model specific exceptions into the llama-model.cpp

[JohannesGaessler]

Sorry for only responding now. I agree that the granularity may need to be adjusted but this value must be derived from hyperparameters, not hard-coded. I don't have the model in question on my machine but based on the compute graph in step35.cpp the granularity should be hparams.n_head(il).

[krampenschiesser]

that's a good idea. I will change that!

[krampenschiesser]

@JohannesGaessler sorry for the late fix. just remembered because laguna is failing with the same thing. so this would fix both.
it seems to be a general pattern as it affects more models, so would it make sense to remove the check for the model arch?

[twnznz]

anecdotally tested ok against b10107
poolside/Laguna-S-2.1-GGUF/laguna-s-2.1-Q8_0.gguf
6x Mi50 32GB - ROCm 7.13.0

[krampenschiesser]

anecdotally tested ok against b10107 poolside/Laguna-S-2.1-GGUF/laguna-s-2.1-Q8_0.gguf 6x Mi50 32GB - ROCm 7.13.0

thank you for the testing! out of curiosity what pp/s tg/s do you get with the mi50 and -sm tensor vs -sm layer

[JohannesGaessler]

I don't think this is the right way to calculate the granularity. The logical interpretation is that this function should return a value to which the tensor dimension is rounded when splitting data between GPUs. That should be hparams.n_head(il), not the GQA ratio. (It may be that for some combinations of models and hardware an incorrect implementation happens to work by chance.)

[krampenschiesser]

So I keep the architecture checks since it is needed for both to run with 4-10 gpus?
Or should i change this differently?

[JohannesGaessler]

The way tensors are named is a lot of the time inconsistent, so either way may need adjustements in the future. My intuition is that removing the check in this PR will have fewer issues in the future.

[twnznz]

anecdotally tested ok against b10107 poolside/Laguna-S-2.1-GGUF/laguna-s-2.1-Q8_0.gguf 6x Mi50 32GB - ROCm 7.13.0

thank you for the testing! out of curiosity what pp/s tg/s do you get with the mi50 and -sm tensor vs -sm layer

Layer:
llama-bench -m poolside/Laguna-S-2.1-GGUF/laguna-s-2.1-Q8_0.gguf -ngl 999 --flash-attn on
ggml_cuda_init: found 6 ROCm devices (Total VRAM: 196512 MiB):
Device 0: AMD Instinct MI60 / MI50, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB
Device 1: AMD Instinct MI60 / MI50, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB
Device 2: AMD Instinct MI60 / MI50, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB
Device 3: AMD Instinct MI60 / MI50, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB
Device 4: AMD Instinct MI60 / MI50, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB
Device 5: AMD Instinct MI60 / MI50, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB

model	size	params	backend	ngl	fa	test	t/s
laguna ?B Q8_0	119.91 GiB	117.56 B	ROCm	999	1	pp512	221.91 ± 0.84
laguna ?B Q8_0	119.91 GiB	117.56 B	ROCm	999	1	tg128	35.52 ± 0.22

build: 54ce507 (10111)

Tensor:
llama-bench -m poolside/Laguna-S-2.1-GGUF/laguna-s-2.1-Q8_0.gguf -ngl 999 --flash-attn on -sm tensor
..

model	size	params	backend	ngl	sm	fa	test	t/s
laguna ?B Q8_0	119.91 GiB	117.56 B	ROCm	999	tensor	1	pp512	368.61 ± 1.80
laguna ?B Q8_0	119.91 GiB	117.56 B	ROCm	999	tensor	1	tg128	35.99 ± 1.10

build: 54ce507 (10111)

So, not much in terms of tg, but a decent bump in pp.
Topology is
numa0-[pex0]-ROCm0 [PCIe 3.0 x16]
numa0-[pex0]-ROCm1 [PCIe 3.0 x16]
| < 2x QPI 9.6GT/s
numa1-[pex1]-ROCm2 [PCIe 3.0 x16]
numa1-[pex1]-ROCm3 [PCIe 3.0 x16]
numa1-[pex2]-ROCm4 [PCIe 3.0 x16]
numa1-[pex2]-ROCm5 [PCIe 3.0 x16]

numa to pex is PCIe 3.0 x16
p2p verified

EDIT: TG at deep context (270k) is MUCH faster; ~25tps (tensor) vs ~7tps (layer)

[krampenschiesser]

EDIT: TG at deep context (270k) is MUCH faster; ~25tps (tensor) vs ~7tps (layer)

That's awesome!

[krampenschiesser]

@CISC @JohannesGaessler how can I continue from here? imho this is a small isolated change to get 2 models to run in tensor parallel and not crash during load. What should I change?

[CISC]

@CISC @JohannesGaessler how can I continue from here? imho this is a small isolated change to get 2 models to run in tensor parallel and not crash during load. What should I change?

I'm not sure exactly what the suggested change is, this will have to wait for feedback from Johannes.

[JohannesGaessler]

Sorry for the long radio silence, I had taken a break from llama.cpp and am currently working through my backlog. Looking at the way the compute graph is being constructed the submitted implementation is correct after all. The wqkv_gate tensor is applied after FlashAttention and before the attention output matrix multiplication as a gate on the FA output. Therefore the granularity needs to be the number of Q heads per K/V head since that is the FA granularity. If you remove the unused variable I'll approve and merge.

[krampenschiesser]

@JohannesGaessler welcome back!
I removed the unused var and rebased

[krampenschiesser]

Maion-coder fails and only on mac, since my adjustments are guarded by a model arch check this is unrelated. probably a rerun needed?

0.04.188.520 I maion-coder          dense any      SOFT_MAX+ARGSORT+GET_ROWS+SUM_ROWS+CLAMP+DIV       2       0       0   7.23e-08   9.14e-07 FAIL

same model again in the llama-context test:

29: Test command: /home/runner/work/llama.cpp/llama.cpp/build/bin/test-llama-context "--models" "/home/runner/work/llama.cpp/llama.cpp/build/tests/test-models/"
29: Working Directory: .
29: Environment variables: 
29:  GGML_SCHED_DEBUG_REALLOC=1
29: Test timeout computed to be: 900
29: models in parallel: 1, threads per model: 4
29: Model                      baseline  seq_rm  state_load  cp_h  cp_d  cp_h_s  cp_d_s  rt    ps    rf    rot   frag  rollback  replay  shared
29: maion-coder-moe.gguf       OK        OK      FAIL        FAIL  FAIL  OK      OK      OK    OK    OK    OK    OK    SKIP      SKIP    SKIP  

[krampenschiesser]

Pushed up another rebase, fingers crossed.