[erwinzhang7]

Overview

Fixes the second half of #26384. Losing an RPC server is not recoverable, but it should not take the whole process down with it. Today it takes down both.

Server, ggml-rpc.cpp:1456 and :1472: a failed graph computation hits GGML_ASSERT, so any backend error kills the worker and closes its socket mid-operation.

Client, RPC_STATUS_ASSERT at ggml-rpc.cpp:30, which expands to GGML_ABORT, reached from 18 call sites: any failed send or receive kills the coordinator. A worker dying for any reason at all, OOM, a crash, a dropped link, a closed laptop lid, takes the coordinator with it.

The Change, The server logs the failed status and returns false, the same shape as the deserialization errors a few lines above it. rpc_serve_client closes that connection and the accept loop keeps the daemon up for the next client. The client marks the endpoint failed and returns GGML_STATUS_FAILED from ggml_backend_rpc_graph_compute. That status is already plumbed the whole way: ggml_backend_sched_compute_splits propagates it, llama_decode turns it into -3, and the server turns that into an HTTP 500 with the slots released. Nothing outside ggml-rpc.cpp changes. The buffers the remote held are gone and every local tensor pointing into them is invalid, so a failed endpoint stays failed. Later operations fail fast rather than pretending to work. This is "fail cleanly", not recovery: there is no reconnect, retry or failover here, and a hung peer still blocks in recv because the sockets have no timeouts. Marking the endpoint follows the has_error latch the Metal backend uses.

Results

Same fault injected in both directions, on the same two machines:

	server	coordinator	client
before	dies, ggml-rpc.cpp:1457: GGML_ASSERT(status == GGML_STATUS_SUCCESS ...) failed	dies on the socket close	connection dropped
after	survives, logs graph computation failed on device 0, status: -1	survives, but still reports /health ok — a defect, see discussion	500 Compute error. immediately

Both halves were exercised separately: a forced GGML_STATUS_FAILED in graph_compute for the server side, and killing the worker mid-generation for the client side. The error surfaces as

graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
process_ubatch: failed to compute graph, compute status: -1
llama_decode: failed to decode, ret = -3
srv update_slots: decode() failed: Compute error.

Normal multi-node generation is unaffected. Measured with llama-bench over the same two machines, against a binary built from the parent commit with only this patch differing:

	pp128	tg32
this branch	698.85 ± 1.27	38.23 ± 0.08
parent	699.27 ± 1.11	38.20 ± 0.08

The added cost on the hot path is one mutex and one hash lookup per graph compute per RPC device, which is not measurable next to the network round trip already being paid.

Also run:

• CI (rpc) (build-rpc.yml, the workflow gated on ggml/src/ggml-rpc/**): green, including ctest -L main
• test-backend-ops: 3/3 backends pass. This does not exercise the RPC backend, since RPC devices are only registered through --rpc in common/arg.cpp and that binary does not link it. The RPC coverage is the two-machine testing above.
• ci/run.sh with GG_BUILD_METAL=1: 100% passed, 0 failed, across both suites (45 and 48 tests)

Additional information

Reproduction

Two Macs, RPC over a Thunderbolt bridge, llama-server on one and ggml-rpc-server on the other with --tensor-split. Killing the worker during generation:

ggml-rpc.cpp:519: Remote RPC server crashed or returned malformed response
E recv failed (bytes_recv=0, size_to_recv=8)
  ggml_abort
  ggml_backend_rpc_buffer_get_tensor
  llama_context::decode  ->  llama_decode
  server_context_impl::decode  ->  update_slots  ->  start_loop

The coordinator is gone, mid-request, along with every other slot it was serving.

No protocol change

There is no wire format change and no version bump. Returning a status byte from GRAPH_COMPUTE would need a major bump rather than a minor one: ggml-rpc.cpp:347 accepts a client newer than the server, so a new client against an old server would send GRAPH_COMPUTE, wait for a response that never comes, and block forever, while the server blocks reading the next command. There are no socket timeouts anywhere in the transport, so that is a permanent silent hang, which is worse than the abort being fixed here. 15d2b46 also removed that response deliberately to save a round trip, and re-adding it would not have prevented the crash above, which came through buffer_get_tensor.

Known limitation

ggml_backend_tensor_get returns void, so one read path has no way to report failure to its caller. It zeroes the destination and marks the endpoint, so the worst case is a single decode returning zeroed output before the next one fails cleanly, rather than stale or partial data. Closing that properly means giving get_tensor a return status, which changes an interface every backend implements and does not belong in a crash fix.

Cost on the hot path is one mutex and one hash lookup per graph compute per RPC device.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[ggml-gh-bot]

Hi @erwinzhang7, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[Kononnable]

Losing an RPC server is not recoverable, but it should not take the whole process down with it. Today it takes down both.

Why should it not take the whole process down? This is a fail fast approach. Server can be restarted automatically if it is running as a service(or under a watchdog) and use selfhealing to mitigate some of the issues. If situation was caused by a transient problem automatic system recovery is possible through restart.

	server	coordinator	client
before	dies, ggml-rpc.cpp:1457: GGML_ASSERT(status == GGML_STATUS_SUCCESS ...) failed	dies on the socket close	connection dropped
after	survives, logs graph computation failed on device 0, status: -1	survives, /health still ok	500 Compute error. immediately

If I'm reading this right the server would return a healthy status while being nonoperational.

[erwinzhang7]

Yeah, you're right wrt the health check and it's a defect this PR introduces.

On fail-fast, I think the two halves differ. The server half replaces a GGML_ASSERT with the
same error return the deserialization paths above it already use; a worker exiting because
one graph failed doesn't restart into a better state, since the failure is per-request. The
client half is where your point bites: a coordinator with a dead endpoint has invalid buffers
and can't recover, so surviving only helps if it serves other models that don't use that
endpoint and it must not claim to be healthy.

Proposed fixes:

1. Fix health here. The state already exists (rpc_endpoint_is_failed in ggml-rpc.cpp) it's
   just static. Needs a small exported query and a check so a failed endpoint reports 503.
   That crosses from ggml into tools/server, which is why I didn't do it unasked.
2. Split: land the server half now, hold the client half until health reporting goes with it.
3. Keep fail-fast on the client side and reduce this to the server half plus a clean exit
   with a diagnostic instead of an abort.

I suggest 1, fallback 2. Either way the "after: /health still ok" row in my description is wrong
and I'll fix it.

[joeydd032995-pixel]

Independent before/after test of this PR, in case it's useful for review.

I built refs/pull/26724/head (a911a9bf4) and its merge-base (9ba73fd1f) as a control, on the same machine with the same model and flags. The merge-base rather than master because otherwise a month of unrelated churn gets attributed to the PR. Confirmed the two builds actually differed before comparing (strings libggml-rpc.so | grep -c "the device is now unusable" → 3 vs 0).

Setup: two ggml-rpc-server processes on loopback, Qwen2.5-0.5B-Instruct Q4_K_M split 26 + 24 layers across them, CPU only, 4-core. kill -9 one server mid-generation, 5 runs each. Every run verified as genuinely mid-generation (tokens already emitted past the prompt echo) with both peers in use.

	commit	aborts	time to detect	exit code
control	9ba73fd1f	5/5	402–417 ms	134 (SIGABRT)
this PR	a911a9bf4	0/5	124–159 ms	0

The control aborts at ggml-rpc.cpp:519: Remote RPC server crashed or returned malformed response; the PR replaces that with the logged error and a clean status return, and detection is also ~2.9× faster. Propagation is intact end to end:

[ggml_backend_rpc_buffer_get_tensor] RPC server 127.0.0.1:NNNNN crashed … the device is now unusable
graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
process_ubatch: failed to compute graph, compute status: -1
llama_decode: failed to decode, ret = -3

Works as described.

One observation on the get_tensor zero-fill

The bound in the description — "at most one decode returning zeroed output before the next one fails cleanly" — holds exactly as stated. What I hadn't expected is that the zeroed decode is user-visible output: the sampler consumes those logits before graph_compute gets a chance to check the latch.

Timeline from one run:

3.579.509  [ggml_backend_rpc_buffer_get_tensor] … the device is now unusable
3.580.306  n_decoded = 14 … next token: 5 '&'          ← 0.8 ms later
3.597.800  graph_compute: … failed with error -1        ← 18 ms later

So every run ends with one anomalous token appended to an otherwise coherent sentence, 5/5:

run	generated text	final token
1	The ocean, a vast and complex ecosystem, is a vital part &	5 '&'
2	The ocean is a vast and beautiful body of water that covers the -	12 '-'
3	The Ocean: A Vast and Intricate Realm … The +	10 '+'
4	The ocean, often referred to as the "sea", is an +	10 '+'
5	The ocean, a vast expanse of water that envelops the *	9 '*'

Always a low token id, which is what all-zero logits produce: equal values leave the top-k filter holding the lowest indices, and those are punctuation in this vocab. The control never produces one, because it aborts first.

I don't think this argues against the approach — a stray token followed by a hard error is plainly better than an uncatchable abort(). It just means the comment at that site, "zero the destination so that partial or stale data is never used", is narrower in practice than it reads: the zeroed data isn't used by ggml, but it does reach the sampler. Might be worth either saying so explicitly in the comment, or having llama_context discard the ubatch when the decode that produced it failed.

Question on the failed-endpoint set

get_failed_endpoints() is insert-only — no erase, no clear — and rpc_endpoint_is_failed is file-static with nothing exposed in ggml-rpc.h. Your own test notes this ("a failed endpoint is latched in a process-wide set that is never cleared").

For a one-shot CLI that's fine. For a long-lived host that wants to reconnect after a worker restarts, it means the endpoint string is permanently unusable and there's no way to ask why. Is a public query and/or a reset intended as follow-up work, or deliberately out of scope here? Happy to test a patch if you add one.

Not tested

Loopback only, no real network hop. Only a hard kill -9 — a graceful shutdown or a silently dropped connection are different paths. I didn't run tests/test-rpc.cpp, only read it. Also worth noting llama-cli exits 0 after this failure, so the error is visible only in the log — but that's pre-existing CLI behaviour, not this PR's: the control build also exits 0 against an unreachable endpoint.

[erwinzhang7]

Thanks for the review @joeydd032995-pixel, and @Kononnable, I've updated the code as suggested.

Changes:

• the client half is rebuilt on the rpc_dispatcher that landed in rpc : implement event and async backend APIs #18626 (abort sites went 18 → 3, all inside work(), and RPC_STATUS_ASSERT is deleted outright)
• the failed-endpoint latch is per-dispatcher instead of a process-wide set; get_dispatcher holds a weak_ptr, so it clears when the connection is recreated
• /health now returns 503 once llama_decode reports its documented fatal range (llama.h:996), so the coordinator stops claiming to be healthy