Repository navigation
Feature Request: Support multiple devices on a single rpc-server #15210
Description
Prerequisites
- I am running the latest code. Mention the version if possible as well.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
Make a signle rpc-server able to handle multiple devices (GPUs). That way it could pass data directly from one device to the next when applicable.
Motivation
Goal: fully utilize the interconnect (PCIe) available on the rpc-server machine.
As of now (version b6084) you must launch one rpc-server instance per device (typically GPU) on your inference server.
If you happen to have 2 or more devices this results in a sub-optimal usage of the available interconnects (e.g. PCIe).
As far as I can see, even when devices on the rpc-server machine host contiguous model layers, all communication between them is still going through the network, to the client (llama-cli) and back.
I guess this was done for the sake of simplicity and ease of implementation.
There's therefore some performance gains to be made by making a single rpc-server able to handle all devices on a machine.
P.S. Thank you so much to everyone who put in the effort to build the rpc-server. It is such a great feature!
Possible Implementation
No response
Activity
rgerganov commentedon Aug 10, 2025
One way to implement this is to introduce a new backend which is able to wrap existing backends, similar to what @slaren proposes for supporting tensor parallelism here. Implementation wise, this should be simpler than tensor parallelism support but it is still non trivial task. I will think about this more and try to come up with some PoC.
thank you for the kind words
slaren commentedon Aug 10, 2025
It may be simpler to allow the server to expose multiple devices, and then just forward the
ggml_backend_tensor_copycalls to the server. This would avoid most of the large transfers between the server and the client.jukofyork commentedon Aug 11, 2025
Yeah, I tried "weaving" RPC through 6 cards over 3 nodes, and even with 10gbit Ethernet I didn't get any real gains (it was a couple of months ago I tried this, so it's possible the RPC code has improved since then though).
rgerganov commentedon Sep 7, 2025
I can think of two ways to implement this:
rpc-serverwill open ports 50052, 50053 and 50054 for each of them. On the client sideggml_backend_rpc_buffer_cpy_tensorwill check if bothsrcanddsttensors are on the same host (not on the same socket as it is now) and forward the call if they are. In this case the RPC protocol remains the same and RPC client usage remains unchanged with the assumption that you cannot start more than one server on the same host (can be enforced if needed). I made a quick and dirty prototype on a Linux host with two NVIDIA cards and it appears to work although I am not sure if CUDA backends are completely thread-safe.@slaren what do you think?
slaren commentedon Sep 8, 2025
@rgerganov sounds about right. I think option 2 would be nicer to use (only need to write one address per server, instead of per device), but it also requires more effort to implement.
rpc : add support for multiple devices
wallentri88 commentedon Sep 30, 2025
I played around with rpc a while ago and, in addition to option 2, made something like this: in function
ggml_backend_sched_compute_splitswhen meeting RPC backend, I iterated until first non-rpc backend, collecting tensors to send to server, compute graphs, and tensors to get back from server (from first non-rpc). And after doing it, I send all this data in one batch. This eliminates:This is, of course, the case when I have one RPC server with multiple devices, didn't think about several RPCs at the moment.
While it sounds good, the draft code in
ggml_backend_sched_compute_splitslooked really awful, so I decided not to share it, haha. I think probably the proper solution would be creating a more abstract RPC backend without exposing devices to main node. So RPC server should doggml_backend_sched_split_graphfor his layers itself. And when we meet this abstract multidevice RPC backend in main nodeggml_backend_sched_compute_splits, we just need to send input tensors and get final output tensors, but not send compute graphs: RPC server should create it by himself because it knows his devices best. Is it even possible, I wonder?rpc : add support for multiple devices (ggml-org#16276)
nguha commentedon Oct 5, 2025
Wow. Great news. Thanks @rgerganov !
rpc : add support for multiple devices (ggml-org#16276)
Fir 1035: Update Llama.cpp to latest and add all the new dependencies…
rpc : add support for multiple devices (#16276)
rpc : add support for multiple devices (ggml-org#16276)
rpc : add support for multiple devices (ggml-org#16276)
rpc : add support for multiple devices (#16276)
rpc : add support for multiple devices (ggml-org#16276)
rpc : add support for multiple devices (ggml-org#16276)