Repository navigation
Conversation
github-actions
Bot
added
Vulkan
|
Did a few more tests with an A16 and a 4090: 4090 + A16-sm layer -sm tensor 2xA16-sm layer -sm tensor So as you can see, the 4090 is held down by the A16 and the boost from tensor parallel there is really small, but on 2xA16, the TG boost is almost double while the PP loss is negligible (almost 90% of the original). |
|
AMD Radeon RX 9070 XT & AMD Radeon AI PRO R9700
|
AMD RX7900XTX (2, 4, 8 GPUs)ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: Found 4 Vulkan devices:
ggml_vulkan: Found 8 Vulkan devices:
And ROCm for comparison:ggml_cuda_init: found 2 ROCm devices (Total VRAM: 49120 MiB):
ggml_cuda_init: found 4 ROCm devices (Total VRAM: 98240 MiB):
ggml_cuda_init: found 8 ROCm devices (Total VRAM: 196480 MiB):
|
There was a problem hiding this comment.
Your way of dealing with the crash seems to be similar to the one proposed here (also AI generated, lol) that basically uses the view src to get around the multi buffer issue. I explain this a bit more in #22197 but I think this only works if the tensor is a view tensor. If it's not a view tensor then we don't have any way of knowing which buffer in the multi buffer the tensor is stored in.
There was a problem hiding this comment.
Yup, valid point. Didn't surface in the small models I've tested, but probably will in any bigger one. Paging @0cc4m here: do you have any qualms about just adding a tensor to sub-buffer map in multi_buffer?
|
OK, bit of oddness on my R9700s compared with the 7900XTX above - prefill takes a significant hit compared with Also, even more weirdness - when run as a server, the first request goes through at 36t/s, but the second falls off to 2t/s: |
Synthetic benchmark (pp4096 / tg128, ~4K context)Without this PR:
With this PR applied:
At short context the PR delivers Real-world big prompt benchmark (88K prompt, 164K context)I've asked the LLM to provide me a resume of a book which content has been sent in the prompt Without this PR:
With this PR applied:
The tensor-mode segfault (fixed by this PR)Without this PR, followed by The tensor decode regression at long contextWith the PR applied, tensor-mode tg drops from 43.6 (short ctx) to 9 (long ctx) — 4.8× worse. Row/layer stay at 37. This appears to be the For reference, |
This comment was marked as off-topic.
This comment was marked as off-topic.
2x Intel Arc Pro B60 (Battlemage, BMG G21)
Observations
|
|
@marksverdhei it's a prototype, don't worry :) when it's been decently tested and cleaned up I'll deslopify it :) |
0cc4m
changed the title
pwilkin
changed the title
pwilkin
changed the title
|
So the damn clanker decided to only implement the proper allreduce for 2 GPUs, because why bother ;) sorry for all the people with 4+ who posted their results, could you please retest with the new commit? (I've tested up to 8 GPUs now for correctness) |
|
Sharing some benchmark results with 2/4/5x Radeon PRO W7900's:
##Vulkan PR TP (2 GPU):
##Vulkan PR TP (4 GPU):
##Vulkan PR TP (5 GPU):
#For comparison with ROCm:
build: 050ee92 (9821)
##ROCm TP RCCL (4 GPU):
##ROCm TP RCCL (5 GPU):
|
|
Could someone else check multi-turn conversations? As noted in my comment above, I'm finding that the first prompt works as expected, but any follow up tanks down to 2t/s. |
|
@AbdullahMPrograms is this after the multiGPU fix already? @netrunnereve I risked a proper fix to the real buffer mapping issue. |
@pwilkin This is with commit: 362fdd2 |
I can reproduce this behavior as well, first prompt: follow up prompt: |
Thank you! At least I know I'm not holding it wrong ;) (EDIT: ...or we both are) |
|
I've also encountered some gibberish in llama-server with TP on 5 GPU's: above on commit e578ca2 |
|
It's faster now on TP=4 and TP=8 (RX7900XTX). Shorter table this time:
And for a multiturn conversation, I don't see any significant degradation. 2nd message: 3rd message: |
Assisted-by: Codex
Assisted-by: Codex
Two hazards in the tensor-parallel ring all-reduce made -sm tensor output non-deterministic with three GPUs: - With n = 3, step t's host-slot region starts at t * cels * 2 bytes, which is not a multiple of 32, so consecutive steps' transfer copies share a 32-byte sector of the imported host memory. On NVIDIA the later copy's leading bytes (up to 10 F16 values at the start of every chunk) end up stale in host memory, even when the copies are ordered by a semaphore. Give every step its own 256-byte-aligned region. n = 2 and n = 4 already had aligned regions for the usual sizes, so their layout is unchanged. - A single-shot all-reduce leaves every device's host slot being read by all peers, but the next ring round only ordered its first transfer after nextd, and only from round 2 on. With n >= 3 a device could overwrite its slot while prevd was still summing it. Wait for all peers on the first ring round after a single-shot all-reduce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With -sm tensor the meta backend drives every device from one thread and calls graph_compute once per subgraph (two per layer), so decode was bound by the CPU cost of submitting, not by the GPUs: ~1500 vkQueueSubmit per token for a 64-layer model on two GPUs, with the GPUs idle ~40% of the time. - Don't split subgraphs by flops while a communicator is active. The heuristic was tuned for whole graphs and turned each small subgraph into ~5 submits. - Fold the small (single-shot) all-reduce into the compute submissions: the last command buffer of a subgraph stays open, the all-reduce appends the gather copy to it and submits it, then opens the next command buffer with the peer waits and the reduce, which the next subgraph is recorded into. Each subgraph now costs one submit per device. The reserved progress value is attached by whichever path submits the open context. - Alternate between two host slots, placed outside the ring's regions, so the gather no longer needs a WAR wait on the peers' previous reduce. - On synchronize, submit the deferred work of every device in the group first, so the tail of the graph runs on all devices in parallel. - Wait for the transfer queue as well as the compute queue before freeing or reallocating communication buffers, and don't leave a stale cap behind when a reallocation fails. qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 32.2 -> 49.7 t/s, 3 GPUs 20.9 -> 40.0, 4 GPUs 15.0 -> 27.6; pp512 on 2 GPUs 2138 -> 2320. Greedy output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…al, wait peers + add from their slots - One exported timeline per device, no backend hooks; comm work recorded in the backend compute ctx - F16 wire above 1 MiB, F32 below; small reductions ride with the next graph submit - Double-buffered, grow-only host slots; zero-size splits zeroed - Remove ring, transfer-queue leg, device staging, shrink, Windows shims - comm_init returns nullptr when unsupported; add GGML_VK_COMM_DISABLE - No flops-based submit splitting for graphs below 1 GFLOP [rebased onto the ring all-reduce fix and the folded all-reduce, which this design replaces; their backend hooks are removed here]
std::aligned_alloc is unavailable on Android bionic below API 28, which upstream switched away from in ggml-org#30828; do the same for the communicator's host slots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c93decc to
468d073
Compare
|
Looks good on my 2x AMD R9700 system:
|
Exchanging SYNC_FD semaphores costs a kernel wake-up on every all-reduce: on NVIDIA the waiting GPU starts ~35 us after its peer has finished, which is most of the idle time in a decode step. OPAQUE_FD semaphores avoid that but must not be imported into another physical device. Use only device-local timeline semaphores instead. Each device signals done[i] once its partial sum is in host memory (after a barrier that makes it available to the host), and a bridge thread signals go[i] from the host once all peers of i are done. The waiting side makes the host memory visible with a host-write barrier, since its submit predates the data. The thread spins for up to 200 us on each wait (GGML_VK_ALLREDUCE_SPIN_US) and then blocks, so it is idle when there is no work. This also takes the per-round fd export, dup and import calls off the main thread and drops the VK_KHR_external_semaphore_fd requirement and the binary semaphore pool. Graphs below 1 GFLOP, such as tensor-parallel decode slices, are no longer split into several submits. qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 37.2 -> 45.4 t/s, 3 GPUs 24.1 -> 35.5, 4 GPUs 16.5 -> 27.1; pp512 unchanged. Greedy output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Here's a quick test on two RX 470s 🚀 Master:
PR:
Layer split:
|
|
Great to see it run on old GPUs too. |
|
New data from my R9700s. SYNC_FD path is back for mesa drivers, it is faster there than the host sync.
|
|
2× AMD Radeon Pro VII (Vega 20, gfx906, 16 GiB HBM2), Xeon E5-2658 v4, 64 GB RAM, Mesa 26.1.6-1 RADV, Vulkan instance 1.4.363 / apiVersion 1.4.354. Build: 48bae8916 (b11890) = pwilkin:vulkan-tp-p2p @ ecfee2e merged into master d327bf1.
-b 4096 -ub 256, -p 4096 f16/q8_0 KV
|
|
@jeffbolznv I think this is in a decent state. Do you want to review it or not? |
|
Don't wait on me, I won't be able to look at it right away. |
0cc4m
added
the
merge ready
Overview
Adds basic Vulkan tensor parallel support:
Only optimized for 2 devices so far, ring-buffer support and/or other optimizations will follow in a subsequent PRs.
Requirements