Permanent LLMExecutor wedge after CL_OUT_OF_RESOURCES on discrete GPU: servable stays AVAILABLE but every request returns 400 until restart (VLM legacy pipeline, public model)

Describe the bug

Reproduced with the stock openvino/model_server:2026.2-gpu and 2026.3-gpu images and the public OpenVINO/Qwen3.5-9B-int8-ov export — no vendor components involved. Under a sustained agent-style workload, the VLM legacy pipeline enters a permanent failure state on a discrete Intel GPU. One request triggers CL_OUT_OF_RESOURCES inside pipe->generate(); from that point on every subsequent request — including a 60-token one with no tools — fails immediately with

400 {"code":"ovms_400","message":"Mediapipe execution failed. MP status - INVALID_ARGUMENT: CalculatorGraph::Run() failed:
Calculator::Process() for node \"LLMExecutor\" failed: Request processing failed, check its correctness.","type":"upstream_error"}

The process stays alive, /v1/config still reports the servable AVAILABLE, and the llm_executor heartbeat ([legacy_executor.cpp:76] All requests: N;) goes silent. Only restarting the server recovers it.

Two things look wrong on the OVMS side, independent of whatever the driver is doing:

A context-fatal GPU error is not treated as fatal. After CL_OUT_OF_RESOURCES the OCL context/queue is unusable, but the servable is neither marked unavailable nor rebuilt, so it serves errors indefinitely while advertising itself as ready. A load balancer or orchestrator has no way to route around it.
The error is reported as a client error. INVALID_ARGUMENT / HTTP 400 with "check its correctness" points the user at their own request, while the actual cause is a server-side unrecoverable GPU failure. HTTP 503 (or at least a distinct code) would be correct.

To Reproduce

Models repository — the public export, unmodified:

huggingface-cli download OpenVINO/Qwen3.5-9B-int8-ov --local-dir ./repro/ir

./repro/config.json:

{ "model_config_list": [],
  "mediapipe_config_list": [ { "name": "qwen3.5-9b-public", "base_path": "/models/graph" } ] }

./repro/graph/graph.pbtxt (the parts that matter; LLMExecutor node wiring is the standard one):

node_options: {
  [type.googleapis.com / mediapipe.LLMCalculatorOptions]: {
    pipeline_type: VLM,
    tool_parser: "qwen3coder",
    reasoning_parser: "qwen3",
    enable_tool_guided_generation: true,
    models_path: "/models/ir",
    plugin_config: '{"CACHE_DIR":"/opt/cache"}',
    enable_prefix_caching: true,
    cache_size: 2,
    max_num_seqs: 256,
    device: "GPU.1",
  }
}

OVMS launch command

docker run -d --name ovms-pub --device /dev/dri --network host \
  -v $PWD/repro:/models -v $PWD/repro/cache:/opt/cache \
  openvino/model_server:2026.2-gpu \
  --config_path /models/config.json --rest_port 8123 \
  --cache_dir /opt/cache --file_system_poll_wait_seconds 1 --log_level DEBUG

Client — the attached four JSON bodies posted sequentially to /v3/chat/completions, one connection at a time, no think time, in this cycle:

step	prompt tokens	tools	note
`tiny`	148	0	short "generate a conversation title" request
`big01`	~12.0k	46	tool-calling turn, one tool round trip in history
`big02`	~12.6k	47	two round trips
`big03`	~14.3k	47	three round trips, last tool result is a ~2.7 KB structured dump

ENDPOINT=http://127.0.0.1:8123/v3/chat/completions ./wedge-loop.sh 150

All four bodies carry stream: true, stream_options.include_usage: true and chat_template_kwargs: {"enable_thinking": false}.

See error — the loop stops at the first non-200.

With the public export on the stock image (the configuration above): wedged after 25 requests, on the tiny step.

With the same payloads against our own export of the same base model (9 runs total, fresh server process each): wedged after 13, 21, 21, 25, 29, 37, 57, 65 and 89 requests, median 29. Every one of the nine failed on the tiny step — the small tool-less request that immediately follows the large tool-calling turns.

After the failure, the server keeps reporting itself ready:

$ curl -s http://127.0.0.1:8123/v1/config
{"qwen3.5-9b-public":{"model_version_status":[{"version":"1","state":"AVAILABLE","status":{"error_code":"OK","error_message":"OK"}}]}}

$ curl -s -d '{"model":"qwen3.5-9b-public","max_tokens":8,"messages":[{"role":"user","content":"1+1?"}]}' \
       -H 'Content-Type: application/json' http://127.0.0.1:8123/v3/chat/completions
{"error":"Mediapipe execution failed. MP status - INVALID_ARGUMENT: CalculatorGraph::Run() failed: \nCalculator::Process() for node \"LLMExecutor\" failed: Request processing failed, check its correctness."}

Expected behavior

A GPU error that invalidates the inference context should mark the servable unavailable and rebuild the pipeline (or terminate the process so the orchestrator can restart it) instead of serving INVALID_ARGUMENT forever while reporting AVAILABLE.
The health endpoints should reflect it. In the wedged state we measured, every discovery and health endpoint still answers 200 — /v3/models, /v1/config ("state":"AVAILABLE", "error_code":"OK"), /v2/health/ready, /v2/health/live and /v2/models/<name>/ready — while an 8-token inference request returns 400. A readiness probe that returns 200 for a servable that cannot serve anything makes every orchestrator, load balancer and monitor blind to the failure; today the only way to detect it is to send a real inference request.
The response for a server-side fatal error should not be a 4xx that blames the request.
Ideally, the workload above should not provoke the GPU error in the first place.

Logs

Server, at the moment of the first failure (--log_level DEBUG):

[llm_executor][error][legacy_executor.cpp:46] VLM pipeline generation failed: Exception from src/inference/src/cpp/infer_request.cpp:224:
[GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES
[GPU] CL_OUT_OF_RESOURCES exception.
        The CL_OUT_OF_RESOURCES error typically occurs in two cases:
        ...

Immediately after, the executor heartbeat stops:

[llm_executor][info][legacy_executor.cpp:76] All requests: 1
[llm_executor][info][legacy_executor.cpp:76] All requests: 0     <-- last line, then silence

Every later request (including {"messages":[{"role":"user","content":"1+1?"}],"max_tokens":8}) returns the 400 quoted above, with no further ERROR lines.

Kernel side, same instant:

xe 0000:05:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2
xe 0000:05:00.0: [drm] Xe device coredump has been created
xe 0000:05:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data

Configuration

OVMS version: reproduced on both unmodified upstream images (heartbeat line legacy_executor.cpp:76 in both):
- openvino/model_server:2026.2-gpu — wedged after 25 requests
- openvino/model_server:2026.3-gpu (sha256:2a52cd2b…, OpenVINO backend 2026.3.0-22451-8a17657b995-releases/2026/3) — 11 runs, 11 wedges, after 5–13 requests (9,9,9,5,9,5,13,9,9,9,13), same model, same graph, same payloads, identical prompt-token counts
  The exception text is identical apart from the OCL source line (ocl_stream.cpp:395 on 2026.3). On 2026.3 the failure is close to deterministic — the three observed values are exactly the boundary request of cycles 2, 3 and 4 — which makes it the configuration we suggest starting from. We are not claiming 2026.3 is worse than 2026.2: we only have one 2026.2 run with the public export, and the two were not interleaved.
config.json: single mediapipe_config_list entry pointing at the graph above.

Accelerator and driver stack. The workload runs on GPU.1; an integrated GPU is present as GPU.0 and stays idle.

Host:


GPU	Intel discrete, PCI `8086:E223` rev 0 (subsystem `8086:1701`), Battlemage class, 32.5 GB, `arch=v20.2.0`, 256 EUs
Kernel	Linux 6.6.129 x86_64
DRM driver	`xe` (not `i915`)
GuC firmware	`xe/bmg_guc_70.bin`, release version 70.58.0
HuC firmware	not loaded (`status: N/A`) — this workload does not use it

Inside the OVMS container (the image ships its own compute runtime):

package	version
`intel-opencl-icd`	26.18.38308.1-0
`libze-intel-gpu1`	26.18.38308.1-0 (`libze_intel_gpu.so.1.15.38308`)
`intel-ocloc`	26.18.38308.1-0
`intel-igc-core-2` / `intel-igc-opencl-2`	2.34.4
`libze1` (Level Zero loader)	1.27.0-1~~24.04~~ppa2

Model repository: /models/<model>/{ir/,g_gpu.1/graph.pbtxt} plus config.json.
Model: OpenVINO/Qwen3.5-9B-int8-ov (public, Apache-2.0), downloaded unmodified from the Hugging Face hub. The failure was first seen with our own export of the same base model and then reproduced with this public one, so nothing vendor-specific is required to hit it.

Additional context

What we ruled out (each experiment is a full run of the cycle above with one variable changed; "no wedge" means the run completed without a single failure):

change	result
`cache_size` 2 → 8 GB	still wedges (after 101 and 245 requests) — not a capacity problem
remove the `tiny` step, keep only the tool-calling turns	180 requests, no wedge
remove the tool-calling turns, alternate `tiny` with a 16.8k-token tool-less request	240 requests, no wedge
give the boundary request 3 small tools (659 tokens)	240 requests, no wedge
keep the boundary request tool-less but enlarge it to 1.9k tokens	240 requests, no wedge
boundary request the same size but sharing the tool turns' prefix	240 requests, no wedge

So the trigger needs both halves: large tool-schema-heavy prefills alternating with a very small tool-less prefill. Enlarging the KV cache does not remove it.

One more observation that may narrow it down: we built a second payload set with the same token counts, same tool counts and same request structure, but whose text is prose instead of machine data (the real payloads are API schemas plus structured result dumps — short indented lines, opaque ids, timestamps; roughly 10 % punctuation and 13 % digits in the tool results). The prose variant ran 1,440 requests with no wedge; the data-shaped variant wedges in tens of requests. Pushing punctuation density well above the original (18–23 %) also stopped reproducing (600 requests each, twice). That suggests the trigger depends on the shape of the prefill, not only on its length or on the number of tools.

Timing note for anyone reproducing: on our host the public export is slow and erratic per request on both images (big requests 4–40 s, versus ~3–4 s for our own export of the same base model), with a cold plugin cache. This changes how long a run takes but not the outcome.

One more thing you may hit while reproducing: with these payloads (no max_tokens, tool-guided generation on, 46–47 tools) the model occasionally enters a degenerate loop, streaming function_calls tokens for tens of minutes. That is not the wedge — the heartbeat still shows All requests: 1, there is no exception and no engine reset. Our driver caps each request at 300 s to keep the loop moving.

Impact: a single occurrence takes the servable out of service permanently while it continues to advertise itself as ready; recovery requires a restart plus model reload (~3 minutes for us). In production we observed six occurrences, each after 12–129 requests.

Notes: the wedge itself may well be a driver/firmware issue (the engine reset points that way), and we are raising that separately. What we would like from OVMS regardless is the handling: detect CL_OUT_OF_RESOURCES as context-fatal, stop reporting the servable as AVAILABLE, and return a 5xx rather than a 400 that blames the caller.

mock-tiny.json
mock-big01.json
mock-big02.json
mock-big03.json

wedge-loop.sh

Adding a data point: this also reproduces on the newest dev image and with a larger model, so it is not fixed in the current weekly either.

Config (all stock, no vendor components):

Image: openvino/model_server:weekly (dev 2026.4), sha256:de1d2eed43e2a7c6c8e4802788e13c6c0a47371fe4984f9cce98d0d47c88af41, created 2026-08-21T17:33+02:00. OpenVINO 2026.4.0 + GenAI 2026.4.0.0.
Model: OpenVINO/Qwen3.8-27B-int4-ov (the 27B variant, larger than the 9B used here). VLM legacy servable via --model_path ... --task text_generation --pipeline_type VLM --cache_size 8.
Host GPU: same discrete Intel Battlemage 8086:E223 (32 GB), Linux xe driver.

Repro: a sustained multi-turn agent workload (dsh) with deep context and tool-calling turns in history, requests issued sequentially. After a burst of large tool-schema-heavy prefills, a request fails with the exact same trace:

[llm_executor][error][legacy_executor.cpp:46] VLM pipeline generation failed: Exception from src/inference/src/cpp/infer_request.cpp:224:
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:395:
[GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES

...followed by the identical wedge: the servable keeps reporting AVAILABLE, but every later request (down to a trivial 8-token one) returns HTTP 400 Mediapipe execution failed. ... Calculator::Process() for node "LLMExecutor" failed: Request processing failed, check its correctness. until a server restart.

Additional data points beyond the report:

Not fixed in the latest weekly (2026.4 dev) — only GA 2026.2 / 2026.3 were covered here.
Reproduces with the 27B model, confirming it is not specific to the 9B export.

One practical note for anyone reproducing: the kernel-side xe device coredump (/sys/class/drm/card1/device/devcoredump/data) auto-deletes after roughly an hour, so snapshot it immediately after a recurrence if you want the full dump.

Fully agreeing with the OVMS-side handling you propose — treat CL_OUT_OF_RESOURCES as context-fatal (mark servable unavailable / rebuild or restart), and return a 5xx instead of a 400 that blames the caller. Happy to share logs or a captured coredump on request.

Follow-up with kernel-level evidence, plus a more concerning regression in the failure mode.

Captured the xe device coredump (this occurrence created a kernel coredump; earlier occurrences did not):

**** Xe Device Coredump ****
Reason: Timedout job - seqno=540, lrc_seqno=540, guc_id=2, flags=0x20
Process: ovms [464054]
PCI ID: 0xe223 (Arc Pro B70, Battlemage), GT0, IP ver 20.2.0
VM state: [0].error: -19 (ENOENT)

preceded by 11 consecutive:

xe 0000:03:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
xe 0000:03:00.0: [drm] Xe device coredump has been created

So in this case the userspace clFinish CL_OUT_OF_RESOURCES is the tail of a kernel-level GPU page fault (-ENOENT) + job timeout, not an OVMS logic error. The full coredump (504 KB) is available on request.

Update — deterministic trigger isolated, and a correction to my restart note.

With --log_level DEBUG and a packet capture on :8000, the exact trigger is isolated: under a sustained multi-turn agent workload, the failure recurs specifically on the agent's context-compaction / checkpoint request — the agent condenses a long conversation into one large message plus a long system prompt, producing a ~290 KB, tens-of-thousands-of-token prefill, after which GPU generation faults ~5 s later:

15:08:44  POST /v1/chat/completions        content-length: 291691
          messages: [huge system prompt, "automatically generated CHECKPOINT condensing ... free up context ... Continue the task"]
15:08:44  InputProcessor Tokenization      took 402 ms
15:08:49  [llm_executor] VLM pipeline generation failed ... [GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES

So on B70 the practical trigger is a single huge prefill at deep (near-max) context — the compaction request itself.

Correction to the restart claim above: on this occurrence a fresh OVMS restart did recover it (a plain request passed immediately after the restart). The failure then recurred deterministically on the very next compaction/deep-context request. So it is recurring from the same trigger, not a permanent post-restart wedge — each context compaction reliably knocks the servable over again.

Sanitized evidence (coredump reason, dmesg faults, trigger shape; no agent conversation payload or auth material) — gist: https://gist.github.com/snailium/72c510a4a4ab5e166b063a4bb8c52eef

Full artifacts (504 KB xe coredump, packet capture, OVMS DEBUG logs) are retained locally and available on request.

I think I have the same, b60, gemma 4 26b int4, 7gb VRAM was free after model is loaded. Simple query 'how many shutter units do i have' from HA with some tools enabled crashed the OVMS, however I had some test chat messages in openwebUI before that and everything was ok.

ovms           | [2026-09-01 08:17:11.980][155][llm_executor][info][legacy_executor.cpp:76] All requests: 1;
ovms           | [2026-09-01 08:17:13.824][155][llm_executor][info][legacy_executor.cpp:76] All requests: 0;
ovms           | [2026-09-01 08:17:41.486][155][llm_executor][info][legacy_executor.cpp:76] All requests: 1;
open-webui-ov  | 2026-09-01 10:18:01.871 | INFO     | uvicorn.protocols.http.httptools_impl:send:485 - 192.168.2.22:58769 - "GET /_app/version.json HTTP/1.1" 200
ovms           | [2026-09-01 08:18:13.967][155][llm_executor][error][legacy_executor.cpp:46] VLM pipeline generation failed: Exception from src/inference/src/cpp/infer_request.cpp:224:
ovms           | Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:395:
ovms           | [GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES
ovms           |
ovms           | .
ovms           | [2026-09-01 08:18:13.967][155][llm_executor][info][legacy_executor.cpp:76] All requests: 0;
ovms           | WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
ovms           | E0000 00:00:1788250693.968316      56 calculator_graph.cc:898] INVALID_ARGUMENT: CalculatorGraph::Run() failed:
ovms           | Calculator::Process() for node "LLMExecutor" failed: Request processing failed, check its correctness.

@geunhwan

I've also seen such issues with gemma4-26b-a4b-int4 on B60 card. Noticed hangs, crashes and situations as described above. None of it happened on B70.
cc: @geunhwan - is it known issue?

Data point: same failure signature on Arrow Lake integrated graphics (bare metal, i915, not xe)

Adding a data point from a different silicon and driver path than the Arc discrete reports above. This is a bare-metal deployment — the host has no Docker, OVMS runs from the official release package.

Environment

GPU: Intel Arrow Lake-P Graphics [8086:7d51] (rev 03) — integrated, kernel driver i915 (not xe)
OS: Ubuntu 24.04.5 LTS, kernel 7.0.0-31-generic
OVMS: 2026.4.0 (bare-metal release package)
Model: Qwen3-VL-4B-Instruct-int4-ov (public-style OpenVINO IR, int4)
Serving: --target_device GPU.0
Observed — both failure modes from this thread occur on this box

VLM legacy pipeline → permanent wedge (matches this issue Permanent LLMExecutor wedge after CL_OUT_OF_RESOURCES on discrete GPU: servable stays AVAILABLE but every request returns 400 until restart (VLM legacy pipeline, public model) #4469)

[llm_executor][error] Calculator::Process() for node "LLMExecutor" failed:
Request processing failed, check its correctness.
After this, the process stays alive, /v1/config still reports the servable AVAILABLE, [legacy_executor.cpp:76] All requests: N goes silent, and every subsequent request hangs or fails until the server is restarted. Measured incident: 2026-09-15 ~21:40 local, the queue accumulated 7 in-flight requests that never drained, blocking all callers for ~10 minutes until a manual restart.

VLM_CB pipeline → process crash (matches SIGSEGV in continuous batching when a streaming client disconnects mid-generation (block_manager.hpp:633 assertion, GPU, 2026.2.1) #4428)

28 crashes in a single day (2026-09-15), all recorded by systemd as Failed with result 'core-dump'.
Kernel: segfault at 19c ... in libopenvino.so.2026.4.0[421ae4] (on 2026.3.0 the same fault appeared at [723e24] — different address, same signature).
Workload shape: an agent fleet (5 concurrent assistant sessions) calling /v3/chat/completions with images (vision). Requests are occasional, not high-throughput; callers time out and disconnect. This is not a saturation workload — the box is essentially idle between requests, which is why the frequency here (28/day) may be informative compared to the cache-saturation reproducers.

Two questions

The candidate fix in openvino.genai#4332 was published as a Docker image (zegmil/ovms-gpu:wovchena-genai). This host has no Docker — could a bare-metal build (release package / tar.gz) be provided for validation? I can test on the exact configuration above and report back, including whether the i915/integrated path behaves differently from the xe/discrete reports.
Given the servable advertises AVAILABLE while being permanently wedged (point 1 of the original report), is there any health signal an external supervisor should be polling to detect this state, short of a functional probe? We currently restart on a failed functional probe and want to know whether OVMS exposes anything better.

Additional data point from our side — the same signature now reproduces on integrated
Intel graphics, with a different model.

	original report	new
GPU	Arc Pro B70 discrete (`8086:E223`), `xe`	Intel integrated GPU (Core Ultra 7 356H)
Model	`Qwen3.5-9B` int8	`Qwen3.6-35B-A3B` int4 (MoE)

On discrete Battlemage the signature is also not specific to one SKU — it occurs on both
B70 (8086:E223, our reproducer and the coredump posted earlier in this thread) and
B60 (reported above). Together with the integrated GPU above, the affected range is
B70, B60 and integrated.

Under an agent-style workload through the OpenAI-compatible endpoint, every
configuration we have tested reproduces it — no combination of GPU, model or stack
version has been found to be free of it:

GPU	Model	Reproduces
Arc Pro B70 discrete	`Qwen3.5-9B` int8	yes
Arc Pro B70 discrete	`Qwen3.5-9B` int8, public export, stock image	yes
Intel integrated (Core Ultra 7 356H)	`Qwen3.5-9B` int8	yes (4 occurrences over 68 requests)
Intel integrated (Core Ultra 7 356H)	`Qwen3.6-35B-A3B` int4 (MoE)	yes (6 occurrences over 32 requests)

The occurrence counts are given only to indicate how quickly it shows up; they are not a
claim that one configuration is safer than another. Given long enough, every configuration
above fails. Logs available on request.

I rebuilt ovms (pointed it to use most recent sha of openvino at that time), it took hours, but after rebuilding the issue was gone with my gemma4.

@gaynetdinov

@gaynetdinov thanks — your note about rebuilding against a recent OpenVINO SHA is what prompted this
round. We did not repeat your source build (that changes several variables at once), but it pointed us
at the right question: is this already fixed on master? Testing that turned out to be much cheaper
than we expected, because the official weekly image is close enough to serve as a proxy.

Update from our side — we now have a configuration that does not reproduce, which narrows down
where the fix is. This also corrects our previous comment, where we wrote that every
configuration we had tested reproduced the issue; that is no longer true.

All three runs below are on the same machine, same GPU, same IR, same graph, same payloads,
issued back-to-back. The only variable is the server image.

GPU: Intel Arc Pro B60 (8086:E211), Linux xe
Model: OpenVINO/Qwen3.5-9B-int8-ov-equivalent IR, VLM legacy pipeline, cache_size: 2
Workload: a synthetic agent-style cycle (4 requests: 148 / 11,036 / 11,634 / 14,296 prompt tokens),
issued sequentially with no think time, stopping at the first non-200
Sampling: container restarted, wait for AVAILABLE, then start the load immediately

Server image	OpenVINO backend	Result
`2026.2-gpu`	`2026.2.0-21902-90214e5be05-releases/2026/2`	wedges (request #33)
`2026.4.0-gpu`	`2026.4.0-22959-99c81491cc3-releases/2026/4`	wedges (request #9)
`weekly` @ `sha256:fcc16047f4134aa4b76e0439d33222e36c20bc6924a25602fe86feb2484dec96`	`2026.5.0-23099-d015add94c1` (master, 2026-09-11)	1,200 requests, no occurrence (2 × 600, each after a fresh restart)

Notes on how to read this:

The 2026.4.0-gpu image we tested was pulled on 2026-09-18 and reports
OpenVINO Model Server 2026.4.0.869b2186a. The failure signature is identical to the original report:
infer_request.cpp:224 → ocl_stream.cpp:395 → [GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES,
followed by Engine reset: engine_class=ccs + an xe device coredump, after which
/v1/config, /v2/health/ready and /v2/health/live all still return 200 while every
subsequent request — down to an 8-token one — returns HTTP 400 until the server is restarted.
Please do not read the request counts as a severity comparison. The three runs were not under
identical GPU occupancy (the 2026.4 run had ~6.6 GB of unrelated workloads resident on the same
card), so 9 vs 33 is not a meaningful difference. The meaningful part is binary:
both GA branches reproduce, master does not.
The weekly result is two independent 600-request runs with zero occurrences, against a
same-day paired positive control on 2026.2-gpu on the same box that failed at request Simplify Dockerfile and make more use of WORKDIR #33.
Without that control we would not claim the negative result means anything.
We are quoting the weekly image by digest on purpose: that tag rolls, and the image we tested
was built 2026-09-15.

What this suggests: the fix appears to have landed on master after the releases/2026/4
branch point (OpenVINO 22959 → 23099), which would mean no GA release currently contains it.

Request: could the relevant change be identified and backported to releases/2026/4?
As it stands, users on any released version have no upgrade path — the only options are running a
dev image, or an external supervisor that detects the wedge with a functional probe and restarts the
server (health endpoints are useless for this, as noted above).

We also have an xe device coredump from this occurrence on B60 (0xe211, Reason: LR job cleanup,
Process: ovms, 1.5 MB) — the ones posted in this thread so far were from B70. Happy to share it,
or to re-run the same three-image comparison with any specific commit or image you would like tested.

Navigation Menu

Repository navigation

Permanent LLMExecutor wedge after CL_OUT_OF_RESOURCES on discrete GPU: servable stays AVAILABLE but every request returns 400 until restart (VLM legacy pipeline, public model) #4469

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Permanent LLMExecutor wedge after CL_OUT_OF_RESOURCES on discrete GPU: servable stays AVAILABLE but every request returns 400 until restart (VLM legacy pipeline, public model) #4469

Description

Activity

snailium commented on Aug 27, 2026

snailium commented on Aug 27, 2026

gaynetdinov commented on Sep 2, 2026

mzegla commented on Sep 3, 2026

xnxjs commented on Sep 15, 2026

SyueYiLiao commented on Sep 16, 2026

gaynetdinov commented on Sep 16, 2026

SyueYiLiao commented on Sep 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions