Repository navigation
Misc. bug: Openvino cannot run gemma on intel core 7 155h #29288
Description
Name and Version
version b11113
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
libllama (core library)
Command line
llama-server -hf unsloth/gemma-4-E4B-it-GGUFProblem description & steps to reproduce
I don't think this will fixed at least not satisfactorily. The NPU in the intel core 7 ultra 155h only supports up to 255 models per process which surpasses the over 700 required for gemma, so weights have to be compiled on the fly for the NPU, which is very slow, or weights could be swapped out on they fly in chunks which would also be slow.
This was heavlily AI assisted: this is my first time touching hardware. i used deepseek v4.1 and the oh my pi harness. i wondered why no one ever talked about using the npu for running ai on laptops, it's because the npu does not support enough models for gemma, or the supported models are too old and not SOTA.
technically openvino does have a huggingface repo https://huggingface.co/collections/OpenVINO/llms-optimized-for-npu and documentation https://docs.openvino.ai/2025/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html on how to run LLMs but that's not really the idea of llama.cpp and they require a lot of setup. and the models on huggingface are outdated maybe this is why?. i am using vulkan now on the GPU instead of the NPU for now. i might try using the openvino documentatoin on running llms next
i am posting this so that no one else wastes 3 days going this route.
After 255 successful zeCommandQueueCreate calls in one context, the next one fails with
L0 zeCommandQueueCreate result: ZE_RESULT_ERROR_UNINITIALIZED, code 0x78000001 - driver is not initialized
while the device and driver are perfectly healthy. The kernel logs the real reason at the same time:
intel_vpu 0000:00:0b.0: [drm] *ERROR* ivpu_cmdq_create(): Failed to allocate command queue ID: -16
-16 is -EBUSY from the command queue ID allocation (IVPU_CMDQ_MIN_ID 1 .. IVPU_CMDQ_MAX_ID 255,
drivers/accel/ivpu/ivpu_drv.h), i.e. the per-context pool of 255 command queue IDs is full. The
reported ZE_RESULT_ERROR_UNINITIALIZED / "driver is not initialized" is misleading: nothing is
uninitialised, the queue budget is exhausted.
This is easy to hit with LLM workloads: the OpenVINO NPU plugin creates one command queue per
compiled model (ZeroCommandQueue::create from Plugin::compile_model,
src/plugins/intel_npu/src/plugin/src/plugin.cpp), and pipelines that compile per-layer or
per-subgraph models hold hundreds of them at once. Concretely, llama.cpp's OpenVINO backend
compiles one OpenVINO model per graph fragment and keeps them alive, so a gemma-3-1b run reaches
~419 compilations for the prefill graph alone and dies on the 256th, after which the backend logs
device NPU is not available, fallback to CPU and silently runs on the CPU.
Environment
- Intel Core Ultra 7 155H (Meteor Lake), NPU37XX, PCI
0000:00:0b.0 - Ubuntu 26.04 LTS, kernel
7.0.0-31-generic,intel_vpu1.0.0 (Scheduler mode: HW) - NPU firmware:
vpu_37xx_v1.bin20260630 (MTL_CLIENT_SILICON-NVR+NN-deployment) - also seen with
20260219; the failure is identical with both - UMD:
intel-level-zero-npu1.38.0 (from thelinux-npu-driverv1.38.0 Ubuntu 24.04 tarball,
used viaLD_LIBRARY_PATH) - and the same with 1.35.0 (snap) - Level Zero loader:
libze_loader.so.1.28.2(Ubuntulibze-dev) - OpenVINO: 2026.4.0 (python wheel and runtime archive; both NPU plugins reproduce)
Minimal reproduction (no llama.cpp needed)
import numpy as np
import openvino as ov
core = ov.Core()
p = ov.opset13.parameter([1, 4], ov.Type.f32, name="x")
model = ov.Model([ov.opset13.relu(p)], [p], "relu_test")
cfg = {"NPU_USE_NPUW": "YES", "NPUW_DEVICES": "NPU", "NPUW_WEIGHTS_BANK": "shared"}
kept = [] # hold them alive: one queue each
for i in range(300):
kept.append(core.compile_model(model, "NPU", cfg))$ LD_LIBRARY_PATH=<umd-1.38.0>:<openvino-runtime> python npu_hold_test.py
compile 255 FAILED while holding 255 compiled models:
L0 zeCommandQueueCreate result: ZE_RESULT_ERROR_UNINITIALIZED, code 0x78000001 - driver is not initialized
Failure at exactly 255, every time. Controls, same driver/loader/device:
- one
compile_model+ oneinferon the NPU: OK (relu([-1,2,-3,4]) == [0,2,0,4]) - 300
compile_modelcalls without holding them alive (each released before the next): OK - 64 infer requests created from one compiled model: OK
So the limit is the number of simultaneously live compiled models in one process.
Reproduction with llama.cpp (the real-world case)
LD_LIBRARY_PATH=<umd-1.38.0>:<openvino-runtime> \
ZE_ENABLE_ALT_DRIVERS=libze_intel_npu.so.1 \
GGML_OPENVINO_DEVICE=NPU \
GGML_OPENVINO_NPU_COMPILE_CONFIG='enable-shave-ddr-access-optimization=true' \
llama-cli -m gemma-3-1b-it-Q4_K_M.gguf -c 256 -n 8 -p 'The capital of France is' -st --no-mmproj
Exception from src/inference/src/cpp/core.cpp:117: (Core::compile_model)
Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/intel_npu/src/plugin/src/plugin.cpp:578: (CompiledModel ctor)
Exception from src/plugins/intel_npu/src/utils/src/zero/zero_wrappers.cpp:321: (ZeroCommandQueue::create)
L0 zeCommandQueueCreate result: ZE_RESULT_ERROR_UNINITIALIZED, code 0x78000001 - driver is not initialized
Number of OpenVINO models compiled for gemma-3-1b-it-Q4_K_M in that setup (counted by
instrumenting the translation entry, same numbers on the CPU device so no NPU needed):
-n (decode steps) |
compilations |
|---|---|
| 1 | 499 |
| 3 | 659 |
| 4 | 738 |
i.e. ~419 for the prefill graph alone (about 210 scheduler fragments x 2 phases), ~80 more per
decode step. All of them are live at once, so the 256th fails during the first prefill.
Expected
- A queue-budget exhaustion should not be reported as
ZE_RESULT_ERROR_UNINITIALIZED/
"driver is not initialized".ZE_RESULT_ERROR_NOT_AVAILABLE(or
ZE_RESULT_ERROR_OUT_OF_HOST_MEMORY) plus a message naming the limit would let consumers react
(retry later, release compiled models, fall back to another device) instead of debugging a
phantom initialisation failure. - Please confirm whether 255 command queues per context is the intended budget for the HW-scheduler
path (Scheduler mode: HW) on NPU37XX, and whether it can be raised or made dynamic. If it is a
hard limit, it needs documenting: with one queue per compiled model it caps the number of
concurrently live compiled models, which is a normal pattern for LLM pipelines. - The
ivpu_cmdq_create()line is logged as*ERROR*but the failure is a resource limit, not a
driver error; it would help to log the current/max queue count with it.
Notes / caveats
- Not a wedged device: the queue IDs belong to the context and are released when the process exits.
A trivial compile works again immediately after a failing run. Reports of an NPU that "needs a
reboot" after such a failure are this same per-process exhaustion: the failing process is the one
that needs more queues than exist. - A module reload (
modprobe -r intel_vpu && modprobe intel_vpu) and a PCI function-level reset
(echo 1 > /sys/bus/pci/devices/0000:00:0b.0/reset, then unbind/bind) both re-initialise the
device cleanly; neither changes the outcome, which is consistent with a per-context budget. - The 255 model compilations in the python reproducer are tiny (
Reluon[1,4]) and take under a
second in total, so the limit is reached long before any memory or time pressure. - Cross-reference for the OpenVINO side: creating one command queue per compiled model makes any
pipeline that keeps many compiled models alive fail here; a shared/pooled command queue per
context, or releasing the queue when the model is released, would avoid the cliff.
First Bad Commit
No response
Activity
alosslessdev commentedon Sep 22, 2026
also here's the ai generated postmortem for qwen: # [Bug]: NPU (NPU37XX): Qwen3.5 GatedDeltaNet hybrid model compiles but hangs at first inference (TDR / ZE_RESULT_ERROR_DEVICE_LOST)
Title
OpenVINO Version
2026.4.0(runtime archive install; NPU plugin + bundled Compiler-in-Plugin)Operating System
Other (Ubuntu 24.04 LTS, kernel
7.0.0-31-generic) - the dropdown has no 24.04 entry.Device used for inference
NPU (Intel Core Ultra 7 155H, Meteor Lake, NPU37XX; PCI
0000:00:0b.0)Framework
PyTorch (original model); executed here as GGUF through llama.cpp's OpenVINO backend.
Model used
unsloth/Qwen3.5-9B-GGUF->Qwen3.5-9B-UD-Q4_K_M.gguf(hybrid GatedDeltaNet / linear-attention + full-attention, Q4_K_M)unsloth/Qwen3.5-0.8B-GGUF->Qwen3.5-0.8B-Q4_K_M.gguf(sameqwen35hybrid architecture, 532 MB) - smallest reproducerQwen/Qwen3-0.6B-GGUFIssue description
A Qwen3.5-class hybrid model that contains GatedDeltaNet (linear-attention) layers compiles on the NPU, but the first inference hangs the NPU and the device is reset by the watchdog (TDR):
The failure is not a size/CMX issue and not a compiler-version issue:
-c 8, prompt "Hi".linux-npu-driverv1.38.0 (both containIE.GatedDeltaNet).Qwen/Qwen3-0.6B-GGUF) runs on the same NPU, so the NPU stack works in general.intel-npu-driverv1.35.0 compiler (no GatedDeltaNet) it fails earlier, at compile time:Cannot create GatedDeltaNet layer ... unsupported opset: extension.Qwen3.5-0.8B-Q4_K_M(532 MB GGUF, sameqwen35GatedDeltaNet tensorsssm_a/ssm_alpha/ssm_beta/ssm_conv1d/ssm_dt/ssm_norm/ssm_out) hangs the NPU the same way, while the plain-attention positive control passes in the same session. The 0.8B blob is far below the 2 GB shave-memory window, so this is not a CMX/size limit. The same 0.8B GGUF runs on the CPU plugin (Prompt 101.8 t/s, Generation 18.6 t/s), so the file and graph are valid.Expected: run correctly on NPU37XX, or fail with a clear "not supported on this platform" error. A device reset is not an acceptable failure mode. GatedDeltaNet / linear-attention hybrid models therefore cannot be used on Meteor Lake NPUs.
Step-by-step reproduction
Environment:
7.0.0-31-generic/opt/intel/openvino_2026.4.0linux-npu-driverv1.38.0 (extracted from the Ubuntu 24.04 release tarball and used viaLD_LIBRARY_PATH; see notes)-DGGML_OPENVINO=ON2b. Smallest GatedDeltaNet model, same flags as the passing positive control (hangs the NPU):
Relevant log output
Qwen3.5-0.8B with the same flags that make the Qwen3-0.6B control pass:
The 0.8B process never recovers and had to be SIGKILLed (timeout hit its hard kill), while the 0.6B control in the same session completed generation normally (Prompt 8.5 t/s, Generation 1.8 t/s). The 0.8B failure also produced
ZE_RESULT_ERROR_UNKNOWN, code 0x7ffffffefromzeCommandQueueExecuteCommandLists/zeFenceHostSynchronizeon an earlier minimal-shape attempt.Kernel (TDR reset):
The recovery counter increments on every attempt (observed reset #0 for the 9B, then #2 and #3 for the 0.8B in a single session), so each try leaves the device reseated.
Earlier failure with
intel-npu-driverv1.35.0 (no GatedDeltaNet in the driver compiler):Evidence that the GatedDeltaNet support exists in the compilers used (so this is an execution issue, not a missing op):
linux-npu-driverv1.38.0 compiler both containIE.GatedDeltaNet,GatedDeltaNetFusion,VPU.GatedDeltaNet.intel-npu-driverv1.35.0 driver compiler does not.Notes / caveats
LD_LIBRARY_PATH+ZE_ENABLE_ALT_DRIVERS(not installed withdpkg), and the firmware was not updated.SplitGatedDeltaNet(CMX-aware sequence splitting) is registered only for NPU50XX (src/vpux_compiler/src/NPU50XX/dialect/VPU/transforms/pipelines.cpp:44); no GatedDeltaNet references exist under NPU37XX/NPU40XX. This is a hint only - the minimal-shape repro (chunk=1) argues against a sequence/CMX explanation.Issue submission checklist
Update 2026-09-20: failure mode changed, rank fix applied, NPU test blocked
After fixing the surrounding gemma issues and installing the NPU plugin built from the
OpenVINO 2026.4.0 source (the runtime-archive plugin fails gemma with a Reshape error), the
GatedDeltaNet failure changed completely: there is no hang and no TDR any more:
Measured inputs: query/key
[1, ?, 2048](rank 3, flattened), value[?, ?, 16, 128](rank 4),gate/beta
[1, ?, 16](rank 3). OpenVINO's op requires query/key/value/state at rank 4(
[B, S, H, D]) and gate/beta at rank 3 ([B, S, H]).Fix applied in
ggml-openvino/openvino/op/gated_delta_net.cpp: reshape rank-3 query/key to[B, S, H_k, D]({0, 0, H_k, -1}, special_zero) before building the op, using the head countalready read from the ggml shape. With that, validation proceeds past the GatedDeltaNet op.
With the NPU healthy again (prebuilt plugin, which enumerates the device reliably), the rank
error is gone and the next validation fails on the head-size check:
ggml provides q/k/v as
[1, T, 16, 128](2048 elements each; state[1, 16, 128, 128]), so theop needs
k_head_size = 128. The OV tensors reaching the op carry 6144 elements (3x) forq/k (value is correct at 2048), i.e. q/k arrive packed/unsliced. Naively splitting 6144 into
heads gives 48, which then violates the GQA check (value heads 16 must be a multiple of q/k
heads). Next step: find why the q/k translators (
q_conv_predelta,k_conv_predelta) emit a3x tensor (likely a packed qkv) and slice them to
[B, T, H, D]before building the op.Also worth reporting upstream: for this fork the NPU plugin built from the 2026.4.0 source makes
gemma work but leaks command queues (
ivpu_cmdq_create(): -EBUSY) and eventually wedges thedevice, while the runtime-archive plugin enumerates fine but fails gemma with a Reshape error.
Update 2026-09-22: graph now validates end-to-end, execution fails on the platform
Two llama.cpp-side bugs found while reproducing this were fixed (they only hid the platform
failure):
gated_delta_net.cppreshapes rank-3 query/key to[B, S, H_k, D]before buildingthe op.
RMS_NORM -> SCALEchains whosesources are views of a packed qkv conv output (3 x 2048 features). The generic view handling
returned such views unchanged (it only expressed a single differing dim, and a packed view
shrinks the source's last dim), so q/k reached the op with 6144 elements instead of 2048.
process_view_input_newinopenvino/utils.cppnow slices those views at the view offset forthe view's element count and reshapes to the view's OV shape.
With both fixes the whole Qwen3.5-0.8B graph translates and compiles with zero shape-inference
failures and the failure moves to execution:
The same graph previously hung the NPU (TDR /
ZE_RESULT_ERROR_DEVICE_LOST); now the graph isvalid and the NPU returns a plain execution error.
IE.GatedDeltaNetis present in bothcompilers used, the graph has no rank/shape problem left, and 0.8B is far inside the 2 GB shave
window, so this is the platform's GatedDeltaNet execution on NPU37XX. The bug report itself is
unchanged: a GatedDeltaNet (linear-attention hybrid) model cannot execute on the Meteor Lake NPU,
and the failure mode should be a clear "not supported on this platform" error rather than a
device reset.
Reproduction note (not an OpenVINO bug): a plugin built from the 2026.4.0 source tree embeds a
RUNPATH pointing into its build tree, so
dlopen("libze_loader.so.1")resolves the Level Zeroloader built next to it. With that loader the plugin cannot initialise the device, llama.cpp logs
device NPU is not available, fallback to CPU(and still generates text). Any result claimed tobe from the NPU has to be checked with
grep -c "not available" <log>; 0 means the NPU ran.Build the plugin with
-DCMAKE_SKIP_BUILD_RPATH=ON -DCMAKE_SKIP_INSTALL_RPATH=ON, or use thereleased plugin.
The command-queue errors noted above (
ivpu_cmdq_create(): -EBUSY) are now understood and arenot a leak: the driver allows 255 command queue IDs per context and the NPU plugin creates one
command queue per compiled model, so a process that keeps more than 255 compiled models alive fails
on the 256th with
zeCommandQueueCreate -> ZE_RESULT_ERROR_UNINITIALIZED(misleading; the queuebudget is exhausted). The IDs are released when the process exits, so the device is not wedged and
no reboot is needed. See the separate report
openvino-bug-gemma-npu-queue-limit.md; it does notchange the GatedDeltaNet issue below, but it does bound what can be verified on the NPU.
Update 2026-09-22 (2): the "executes on the NPU" line above needs re-verification
Two things found today affect how the section above should be read:
were very likely CPU fallbacks (they predate the
grep -c "not available"check). What issolid from that work: two real llama.cpp-side bugs were fixed (rank-3 query/key, packed q/k
views), and with them the whole Qwen3.5 graph translates and compiles with zero shape-inference
failures.
255 live compiled models (see
openvino-bug-gemma-npu-queue-limit.md). Whether Qwen3.5-0.8B'sgraph fits was not measured, so "compiles and executes on the NPU" is unverified.
To re-check properly: build the plugin without a build-tree RUNPATH (or use the released/wheel
plugin), run with
GGML_OPENVINO_DEVICE=NPU, and requiregrep -c "not available" <log>== 0 andno
zeCommandQueueCreatefailure in the log. Until then, the GatedDeltaNet bug report stands onthe original evidence (hang / device reset with the minimal reproducer,
IE.GatedDeltaNetpresentin both compilers, plain-attention control model running on the same NPU).
alosslessdev commentedon Sep 22, 2026
reading https://huggingface.co/collections/OpenVINO/llms-optimized-for-npu is damning, due to the short supported context window of say less than 2k tokens.
alosslessdev commentedon Sep 22, 2026
i just tried using the openvino documentation and intel's libraries do not support a version of transformers that is new enough to support gemma.
Attempting uninstall: tokenizers
Found existing installation: tokenizers 0.23.2
Uninstalling tokenizers-0.23.2:
Successfully uninstalled tokenizers-0.23.2
Attempting uninstall: transformers
Found existing installation: transformers 5.17.0
Uninstalling transformers-5.17.0:
Successfully uninstalled transformers-5.17.0
Successfully installed huggingface_hub-0.36.2 tokenizers-0.22.2 transformers-4.57.6
~$ optimum-cli export openvino -m google/gemma-4-E4B-it --weight-format int4 --sym --ratio 1.0 --group-size -1 gemma-4-E4B-it
NNCF version 2.19 or higher is required to use NNCF-based quantization. Please upgrade your NNCF installation. The current version of NNCF is 2.18.0.
Traceback (most recent call last):
File "/home/asdf/npu-env/lib/python3.14/site-packages/transformers/models/auto/configuration_auto.py", line 1360, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/asdf/npu-env/lib/python3.14/site-packages/transformers/models/auto/configuration_auto.py", line 1048, in getitem
raise KeyError(key)
KeyError: 'gemma4'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/asdf/npu-env/bin/optimum-cli", line 8, in
sys.exit(main())
~~~~^^
File "/home/asdf/npu-env/lib/python3.14/site-packages/optimum/commands/optimum_cli.py", line 219, in main
service.run()
~~~~~~~~~~~^^
File "/home/asdf/npu-env/lib/python3.14/site-packages/optimum/commands/export/openvino.py", line 469, in run
main_export(
~~~~~~~~~~~^
model_name_or_path=self.args.model,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<12 lines>...
# **input_shapes,
^^^^^^^^^^^^^^^^^
)
^
File "/home/asdf/npu-env/lib/python3.14/site-packages/optimum/exporters/openvino/main.py", line 282, in main_export
task = infer_task(
task,
...<6 lines>...
trust_remote_code=trust_remote_code,
)
File "/home/asdf/npu-env/lib/python3.14/site-packages/optimum/exporters/openvino/main.py", line 114, in infer_task
config = AutoConfig.from_pretrained(
model_name_or_path,
...<4 lines>...
trust_remote_code=trust_remote_code,
)
File "/home/asdf/npu-env/lib/python3.14/site-packages/transformers/models/auto/configuration_auto.py", line 1362, in from_pretrained
raise ValueError(
...<8 lines>...
)
ValueError: The checkpoint you are trying to load has model type
gemma4but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.You can update Transformers with the command
pip install --upgrade transformers. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the commandpip install git+https://github.com/huggingface/transformers.gitalosslessdev commentedon Sep 24, 2026
I have found this https://news.ycombinator.com/item?id=49800513 https://github.com/hsfzxjy/npunlock hopefully this lets us run arbitrary code on the NPU, as if it were a GPU with cuda or rocm. This is very promising since this issue is a compiler limitation, and this project here gets rid of it. We could satisfactorily stream weights in and out of the npu like it can be done on a GPU with cuda or rocm.
I would like to touch the npu directly by programming it directly in C and not have to use openvino, because it could add complexity like having to chunk a model and manage context with limited control and at the same time it adds overhead. However ,this would be a new backend. litert llm https://developers.google.com/edge/litert-lm/cli/usage is also promising, but i will have to look into making it work on linux and not just windows.
Zulkifli-Intel commentedon Sep 28, 2026
Hi @alosslessdev
I tested this on a Meteor Lake system (Core Ultra 7 155H) with Ubuntu 24.04, OpenVINO 2026.4.0, and the Intel NPU driver stack 1.32.0.
I was able to reproduce an NPU resource/command-queue exhaustion issue independently using a minimal OpenVINO test. When retaining the compiled models, compilation succeeds up to approximately compile 254 and fails at 255 with:
ZE_RESULT_ERROR_UNINITIALIZED, code 0x78000001 - driver is not initializedAt the same time, the kernel reports:
-16 corresponds to EBUSY. A control test where the compiled models were released between iterations continued through 300 compile calls, so this appears related to the number of simultaneously retained NPU resources rather than a simple limit on the number of compile_model() calls.
I also tested the llama.cpp OpenVINO backend.
For Qwen3.5-0.8B on NPU, the current failure occurs earlier during NPU compilation:
Therefore, I don't think the Qwen3.5 failure can currently be attributed to the command-queue exhaustion observed in the standalone reproducer. It appears to fail earlier due to the NPU VCL/compiler handling of GatedDeltaNet.
I also tested Qwen3-0.6B as a smaller NPU control and Gemma 3 1B. The Qwen3-0.6B test gets through NPU compilation, while Gemma 3 1B encounters a separate f32/f16 tensor type mismatch during inference.
At this point, the confirmed findings are:
• OpenVINO standalone: reproducible NPU command-queue/resource exhaustion at ~255 simultaneously retained compiled models.
• Qwen3.5-0.8B: currently fails earlier with an NPU VCL GatedDeltaNet unsupported-opset error.
I'm still working on this issue and will update the findings here.
Zulkifli-Intel commentedon Oct 6, 2026
Llama.cpp with the OpenVINO backend currently supports only the following Gemma models on the NPU:
Validation was also done on Lunar Lake (Core Ultra 5 238V) with NPU driver 1.35.0, so results on Meteor Lake (155H) may differ.
You may refer to Validated Models for the full list of supported models.
For the ZE_RESULT_ERROR_UNINITIALIZED, code 0x78000001 - driver is not initialized error (the 255 command-queue limit), I'm checking with the dev team and will update here.