Misc. bug: OpenVINO GPU aborts on hybrid recurrent models — cache_r_l1 view pre-allocated in an OPENVINO0 buffer that cannot run CPY #28048

Name and Version

CI build via ci/run.sh, so there is no --version string. Configure output instead:

-- llama.cpp version: 0.3.0-dev
-- ggml version: 0.22.0
-- ggml commit:  ca6595eca
OpenVINO 2026.3.1.22476.56d9685302d
built with gcc 13.2.0 for x86_64-linux-gnu (Ubuntu 24.04)

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

Other (Please specify in the next section)

Command line

OpenVINO 2026.3.1.22476.56d9685302d

See steps to repro below.

Problem description & steps to reproduce

Chasing red CI root cause, on an unrelated PR of mine, led me down this path.

The gpu-openvino-low-perf job has been failing on master continuously since 2026-08-28. It reports as test-save-load-state (Timeout) after 1500 s. That is misleading. The test actually dies 4.2 seconds in, on the 6th of 110 models, and then never exits.

The abort is a scheduler placement failure. A view of the recurrent state cache is pre-allocated in the OpenVINO buffer. The OpenVINO backend can't run CPY on that buffer, so ggml_backend_sched_backend_id_from_cur() has nowhere to put the node and aborts.

This isn't new, #20619 reported the same abort in March 2026 on real Qwen3.5 models rather than dummy ones:

pre-allocated tensor (cache_r_l0 (view) (copy of )) in a buffer (OPENVINO0)
  that cannot run the operation (CPY)

Same sched_reserve to graph_reserve stack, at what was then ggml-backend.cpp:809. Several people confirmed it on different iGPUs. The stale bot closed it as not planned on 2026-06-19 while people were still hitting it, and the last word from Intel on 2026-08-10 was that Qwen3.5 is still a work in progress. #22333 is the same abort on cache_r_l0 again, also closed. #27206 covers neighbouring split-boundary problems on hybrid models and is still open.

So the backend limitation is well established. What changed on 2026-08-28 is that it now breaks CI on master, so it is no longer only OpenVINO users who run into it.

1. Requirements

• An Intel GPU with OpenVINO 2026.3.1 (the CI uses an Arc 140V iGPU, GGML_OPENVINO_DEVICE=GPU)
• Ubuntu 24.04

2. Build with OpenVINO

source /opt/intel/openvino/setupvars.sh
cmake -B build -DCMAKE_BUILD_TYPE=Debug -DGGML_OPENVINO=ON -DGGML_SCHED_NO_REALLOC=ON
cmake --build build -j

3. Generate the dummy models, then run the one that fails

./build/bin/test-llama-archs -o ./build/tests/test-models/
./build/bin/test-save-load-state -m ./build/tests/test-models/bailingmoe3-moe.gguf

4. Observe

ggml/src/ggml-backend.cpp:932: pre-allocated tensor (cache_r_l1 (view) (copy of  (view)))
  in a buffer (OPENVINO0) that cannot run the operation (CPY)

The process prints its backtrace, gdb detaches, and then it hangs instead of terminating. Under ctest that eats the whole 1500 s timeout, so the summary says Timeout rather than Subprocess aborted. A 4 second crash ends up hidden inside a 25 minute one, which is why this has been easy to walk past.

5. Where it aborts

This is context construction, not decode. llama_init_from_model() never returns:

#3  ggml_backend_sched_backend_id_from_cur ()  ggml-backend.cpp:932
#4  ggml_backend_sched_split_graph ()          ggml-backend.cpp:1093
#5  llama_context::graph_reserve ()            llama-context.cpp:2453
#6  operator() (probe=..., enabled=true)       llama-context.cpp:513
#7  llama_context::resolve_fused_ops ()        llama-context.cpp:555
#8  llama_context::sched_reserve ()            llama-context.cpp:620
#9  llama_context::llama_context ()            llama-context.cpp:461
#10 llama_init_from_model ()                   llama-context.cpp:3723

First Bad Commit

4e97ac8, tests : run test-save-load-state across all architectures (#27755), landed 2026-08-28T06:45:19Z. The first failing run of this job carries exactly that SHA (run 33149014299, master, 2026-08-28T06:45).

Technically that commit turned the job red. It made test-save-load-state sweep all 110 generated architectures instead of a single model, and that is the first time a hybrid recurrent arch has been constructed on the OpenVINO backend in CI. The backend limitation is older than the commit that exposed it. I have not dug into the backend side, since I can't run it locally.

Relevant log output

main: running save/load tests over 110 models in 'build-ci-debug/tests/test-models/'

================================================================
0.04.174.091 I main: model build-ci-debug/tests/test-models/bailingmoe3-moe.gguf
0.04.182.782 W load: adding 128 dummy tokens
0.04.184.465 I run_save_load_tests_for_model: no prompt provided, generating 100 (n_batch) random tokens
0.04.184.483 I run_save_load_tests_for_model: the input prompt is 100 tokens
ggml/src/ggml-backend.cpp:932: pre-allocated tensor (cache_r_l1 (view) (copy of  (view))) in a buffer (OPENVINO0) that cannot run the operation (CPY)
#2  ggml_abort (file="ggml/src/ggml-backend.cpp", line=932) at ggml/src/ggml.c:268
#3  ggml_backend_sched_backend_id_from_cur (sched=..., tensor=...) at ggml/src/ggml-backend.cpp:932
#4  ggml_backend_sched_split_graph (sched=..., graph=...) at ggml/src/ggml-backend.cpp:1093
#5  llama_context::graph_reserve (n_tokens=2, n_seqs=2, n_outputs=2, split_only=true) at src/llama-context.cpp:2453
#6  operator() (probe=..., enabled=true) at src/llama-context.cpp:513
#7  llama_context::resolve_fused_ops (mctx=..., n_seqs=2) at src/llama-context.cpp:555
#8  llama_context::sched_reserve () at src/llama-context.cpp:620
#9  llama_context::llama_context (model=..., params=...) at src/llama-context.cpp:461
#10 llama_init_from_model (model=..., params=...) at src/llama-context.cpp:3723
#11 test_baseline (...) at tests/test-save-load-state.cpp:62
#12 run_save_load_tests_for_model (model_path=".../bailingmoe3-moe.gguf") at tests/test-save-load-state.cpp:400
#13 main (argc=3, argv=...) at tests/test-save-load-state.cpp:510
[Inferior 1 (process 4181972) detached]

<< process never exits, ctest kills it 1496 s later >>

98% tests passed, 1 tests failed out of 48

The following tests FAILED:
	 28 - test-save-load-state (Timeout)
Errors while running CTest

Full job log: https://github.com/ggml-org/llama.cpp/actions/runs/33289143940/job/99218360170