[pcuenca]

Overview

ACCEL devices are intended to be used alongside a companion CPU. On Mac, they are detected as part of the non-CPU identification loop and added to the list of available devices. Inference later crashes when they are sent an operation (RMS_NORM in my case) that can't run there, and there's no CPU to offload the work to.

Additional information

Before #16276, I think ggml_backend_init_best() was used to select the appropriate device to expose: this considered GPU, IGPU, and lastly CPU. PR #16276 allows multiple accelerators to be exposed, but ACCEL ones are also added to the list; however, they can't work without the CPU.

I think an improved solution would be to:

• Ignore ACCEL during the candidate-selection loop (this PR).
• Add ACCEL alongside CPU in the fallback case (no GPUs/IGPUs exist). I think this is more involved as it would require a combined device to be exposed as a single entity IIUC.

On Mac, the obvious workaround is to specify the metal device in the command-line invocation.

CPU-only platforms that have BLAS or AMX would also expose this issue. Before this PR, inference would crash. After this PR, inference will run on CPU, but not BLAS/AMX.

I wonder if we should also skip IGPUs if GPUs are detected, just like the regular local-inference flow does; this could come in a followup PR perhaps.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: yes (diagnose, code navigation)

[ggerganov]

Can you give the commands that you are using to reproduce the crash? For example using ggml-org/Qwen3.5-0.8B model.

[pcuenca]

Sure, sorry.

Mac 2:

./build-rpc/bin/ggml-rpc-server --host 172.18.3.130 --cache

Mac 1:

./build-rpc/bin/llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF -ngl 99 --rpc 172.18.3.130:50052  --prompt "Who are you?" --single-turn