can't run qwen3.6 27b in pipeline parallel

Describe the bug
OVMS crashes as soon as I try to run 1 request
Hardware: 2x b580 (24GB)

To Reproduce
Steps to reproduce the behavior:
OpenVINO Model Server 2026.4.0.103456a0a
config.json

{
  "model_config_list": [
    {
      "config": {
        "name": "qwen3-6-27b-int4-ov_model",
        "base_path": "/models/ov/server/qwen3-6-27b-int4-ov",
        "target_device": "HETERO:GPU.0,GPU.1"
      }
    }
  ],
  "mediapipe_config_list": [
    {
      "name": "qwen3-6-27b-int4-ov",
      "graph_path": "/models/ov/server/qwen3-6-27b-int4-ov/graph.pbtxt"
    }
  ]
}

graph.pbxt

input_stream: "HTTP_REQUEST_PAYLOAD:input"
output_stream: "HTTP_RESPONSE_PAYLOAD:output"

node: {
  name: "LLMExecutor"
  calculator: "HttpLLMCalculator"
  input_stream: "LOOPBACK:loopback"
  input_stream: "HTTP_REQUEST_PAYLOAD:input"
  input_side_packet: "LLM_NODE_RESOURCES:llm"
  output_stream: "LOOPBACK:loopback"
  output_stream: "HTTP_RESPONSE_PAYLOAD:output"
  input_stream_info: {
    tag_index: 'LOOPBACK:0',
    back_edge: true
  }
  node_options: {
[type.googleapis.com / mediapipe.LLMCalculatorOptions]: {
    pipeline_type: VLM,
    models_path: "/models/ov/server/qwen3-6-27b-int4-ov/1",
    plugin_config: '{"DYNAMIC_QUANTIZATION_GROUP_SIZE": "32", "PERFORMANCE_HINT": "LATENCY", "KV_CACHE_PRECISION": "u8", "INFERENCE_PRECISION_HINT": "f16", "MODEL_DISTRIBUTION_POLICY": "PIPELINE_PARALLEL", "EXECUTION_MODE_HINT": "PERFORMANCE", "SCHEDULING_CORE_TYPE": "PCORE_ONLY", "ENABLE_CPU_PINNING": false, "DEVICE_PROPERTIES":{"GPU":{"MAX_PROMPT_LEN":2048}}}',
    enable_prefix_caching: false,
    cache_size: 1,
    max_num_batched_tokens: 2048,
    max_num_seqs: 1,
    device: "HETERO:GPU.0,GPU.1",
    tool_parser: "qwen3coder",
}
  }
  input_stream_handler {
    input_stream_handler: "SyncSetInputStreamHandler",
    options {
      [mediapipe.SyncSetInputStreamHandlerOptions.ext] {
        sync_set {
          tag_index: "LOOPBACK:0"
        }
      }
    }
  }
}

log.txt

@geunhwan

@geunhwan Is model qwen3.6 27b supported in HETERO plugin now with A770 cards?

@geunhwan

I managed to reproduce that issue. The crash happens due to plugin config setting "MODEL_DISTRIBUTION_POLICY": "PIPELINE_PARALLEL".
even with single deployment:
ovms --model_repository_path /models --source_model OpenVINO/Qwen3.6-27B-int4-ov --model_name ovms-model --rest_port 9000 --target_device HETERO:GPU.1,GPU.2 --pipeline_type VLM --log_level DEBUG --plugin_config '{"MODEL_DISTRIBUTION_POLICY": "PIPELINE_PARALLEL"}'

Model works without plugin config parameter. Therefore this is not model server issue. The question about support for this configuration is valid, so let's wait for @geunhwan response.

@geunhwan Is model qwen3.6 27b supported in HETERO plugin now with A770 cards?

I just tested on my A770's, it still doesn't work and tries to load the model into a single GPU.

Same here on 2x A770 16GB but on OpenVINO/Qwen3.8-27B-int4-ov model with ovms 2026.4 weekly docker image (de1d2eed43e2 . Model loads OK (model evenly distributed to the two cards with PIPELINE_PARALLEL config) but crashes on the first request without any error message (even with --log_level DEBUG).

But PIPELINE_PARALLEL works on both HETERO: GPU.0, CPU and HETERO: GPU.1, CPU, only crashes on HETERO: GPU.0, GPU.1.

dmesg log on crash:

[  934.026082] ovms[12478]: segfault at 707410173000 ip 0000707510408e09 sp 0000706d10486038 error 4 in libc.so.6[198e09,707510298000+188000] likely on CPU 1 (core 1, socket 0)
[  934.026097] Code: 07 00 01 74 a3 83 f9 c0 0f 87 59 fe ff ff 62 e1 fe 28 6f 4e 01 48 29 fe 48 83 c7 3f 49 8d 0c 10 48 83 e7 c0 48 01 fe 48 29 f9 <f3> a4 62 c1 fe 28 7f 00 62 c1 fe 28 7f 48 01 c3 0f 1f 80 00 00 00

ovms log: ovms-log.txt

Same here on 2x A770 16GB but on OpenVINO/Qwen3.8-27B-int4-ov model with ovms 2026.4 weekly docker image (de1d2eed43e2 . Model loads OK (model evenly distributed to the two cards with PIPELINE_PARALLEL config) but crashes on the first request without any error message (even with --log_level DEBUG).

But PIPELINE_PARALLEL works on both HETERO: GPU.0, CPU and HETERO: GPU.1, CPU, only crashes on HETERO: GPU.0, GPU.1.

dmesg log on crash:
[  934.026082] ovms[12478]: segfault at 707410173000 ip 0000707510408e09 sp 0000706d10486038 error 4 in libc.so.6[198e09,707510298000+188000] likely on CPU 1 (core 1, socket 0)
[  934.026097] Code: 07 00 01 74 a3 83 f9 c0 0f 87 59 fe ff ff 62 e1 fe 28 6f 4e 01 48 29 fe 48 83 c7 3f 49 8d 0c 10 48 83 e7 c0 48 01 fe 48 29 f9 <f3> a4 62 c1 fe 28 7f 00 62 c1 fe 28 7f 48 01 c3 0f 1f 80 00 00 00
ovms log: ovms-log.txt

Guarentee you it’s the same problem that was happening months ago; openvino doesn’t split the model nor the kv cache across the gpus even though it knows there are multiple gpus in the hetero configuration.

Guarentee you it’s the same problem that was happening months ago; openvino doesn’t split the model nor the kv cache across the gpus even though it knows there are multiple gpus in the hetero configuration.

No, in my situation, both GPU got ~9GB VRAM usage with PIPELINE_PARALLEL from xpu-smi, looks like the split works. If you do not specify PIPELINE_PARALLEL, the first GPU will got the full model weight (16GB+) and easily OOM.

I made qwen3.8 27b work on 2x A770 HETERO PIPELINE_PARALLEL with these patches for OVMS and GenAI, all patches are generated by AI (OpenCode Big Pickle), based on current main branch:

OVMS patch:

0000-ovms.patch Fixes the crash on the first request issue. Looks like mainly a threading issue. Ignore the dockerfile modification if it doesn't match your environment.

GenAI patch:

0001-CB-Preallocate-KV-and-linear-attention-cache-to-conf.patch Workarounds CL_EXEC_STATUS_ERROR when prefilling.

0002-CB-LinearAttention-Fix-garbled-generation-on-GPU-sto.patch Fixes garbled outputs (like this: 2343#!/23283##%) starting from the second token after some conversations.
I also have some questions about this patch: AI states that the per-block CPU -> RemoteTensor ROI copies will silently corrupt the zeroed memory block, which I feel strange. Is there some race conditions when copying? Or this only happens on A770? I tried update the compute-runtime from 26.18 to 26.31 but it still generates grabled outputs.

0003-CB-Sum-GPU-memory-across-devices-for-multi-GPU-KV-ca.patch PR #4254 or it will OOM when VRAM of the first GPU is full.

0004-CB-LinearAttention-Preallocate-bounded-LA-state-so-t.patch Workarounds CL_EXEC_STATUS_ERROR (-14) via clWaitForEvents when prefilling, completing the first patch.
Still tensor copying issues. I guess the real problem sits in some other place.

Qwen3.8-27b-ov-int4 now works on my 2x A770 16GB setup, reaching ~192k context with 7GB cache. OVMS arguments are:

--model_name Qwen3.8-27B
--model_path /models/OpenVINO/Qwen3.8-27B-int4-ov
--task text_generation
--pipeline_type VLM_CB
--target_device HETERO:GPU.0,GPU.1
--model_distribution_policy PIPELINE_PARALLEL
--plugin_config '{"GPU_ENABLE_LARGE_ALLOCATIONS": true}'
--cache_size 7
--rest_port 9000

Dynamic cache will result in CL_OUT_OF_RESOURCES when approaching ~160k context.

The performance is good, tested by llama-benchy:

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
Qwen/Qwen3.8-27B	pp512	614.96 ± 86.37		851.18 ± 109.20	849.00 ± 109.20	1003.91 ± 29.98
Qwen/Qwen3.8-27B	tg128	14.91 ± 0.02	15.00 ± 0.00
Qwen/Qwen3.8-27B	pp512 @ d4096	777.57 ± 0.35		5928.36 ± 2.67	5926.18 ± 2.67	6196.09 ± 3.43
Qwen/Qwen3.8-27B	tg128 @ d4096	14.64 ± 0.03	15.00 ± 0.00
Qwen/Qwen3.8-27B	pp512 @ d8192	725.09 ± 7.08		12008.20 ± 117.13	12006.03 ± 117.13	12169.54 ± 35.76
Qwen/Qwen3.8-27B	tg128 @ d8192	13.98 ± 0.00	14.00 ± 0.00
Qwen/Qwen3.8-27B	pp512 @ d16384	640.40 ± 10.08		26394.91 ± 420.48	26392.73 ± 420.48	26551.31 ± 443.40
Qwen/Qwen3.8-27B	tg128 @ d16384	13.16 ± 0.01	14.00 ± 0.00
Qwen/Qwen3.8-27B	pp512 @ d65536	362.65 ± 0.16		182131.29 ± 82.29	182129.11 ± 82.29	182331.79 ± 80.80
Qwen/Qwen3.8-27B	tg128 @ d65536	9.78 ± 0.02	10.00 ± 0.00
Qwen/Qwen3.8-27B	pp512 @ d128000	236.06 ± 0.24		544414.67 ± 543.05	544412.49 ± 543.05	545030.46 ± 473.33
Qwen/Qwen3.8-27B	tg128 @ d128000	7.29 ± 0.02	8.00 ± 0.00

llama-benchy (0.4.0)
date: 2026-09-01 22:00:39 | latency mode: api

The GPU utilizations are under 15% on both cards, so I guess the A770 may have some potentials.

These patches are fully vibe-coded and not ready for a formal PR. I wish they could be a reference for a real fix. OpenVINO is currently the ONLY performant solution on Intel ARC consumer GPUs, specially on old cards like A770. So much thanks to the OpenVINO team!

@HumerousGorgon

I'll add Claude's experience on my rig, if it helps.... (AI content follows, though I can vouch for the testing):

Independent reproduction, confirming this isn't specific to Qwen3.8-27B-int4-ov or to nightly builds — the same bug reproduces on the sibling model Qwen3.6-35B-A3B-int4-ov on a stable release image.

Environment: docker.io/openvino/model_server:2026.4.0-gpu (release, not weekly/nightly), Fedora Linux 44, kernel 7.1.10, 2x Intel Arc A770 16GB, intel-compute-runtime-26.22.38646.6.

Without MODEL_DISTRIBUTION_POLICY: HETERO:GPU.0,GPU.1 alone loads the entire model onto GPU.0 and fails with CL_OUT_OF_RESOURCES (ProgramBuilder build failed, clFinish error -5) — same as already reported here.

With plugin_config: {"MODEL_DISTRIBUTION_POLICY":"PIPELINE_PARALLEL"}: loads successfully to AVAILABLE (model actually split across both cards this time), but segfaults deterministically on the first real inference request — reproduced twice, identical fault offset in libc.so.6 both times:

ovms[2848365]: segfault at 7f1c5809d000 ip 00007f1d882d8f07 sp 00007f14187d8a28 error 4 in libc.so.6[188f07,7f1d88178000+188000]
ovms[2854702]: segfault at 7fea3009e000 ip 00007feb5b0f5f07 sp 00007fe1e97d2a28 error 4 in libc.so.6[188f07,7feb5af95000+188000]

Same libc memcpy-region offset (188f07) both runs, only ASLR base and CPU core differ — looks deterministic, not a race. Disassembly at the fault site is a vectorized rep movsb-style copy, consistent with @HumerousGorgon's trace above (f3 a4 at the same relative position).

New data point: MODEL_DISTRIBUTION_POLICY: TENSOR_PARALLEL is rejected outright at config-validation time, before touching the GPU:

Wrong value TENSOR_PARALLEL for property key MODEL_DISTRIBUTION_POLICY. HETERO plugin only support {PIPELINE_PARALLEL}

So there's no alternate policy to fall back to within the HETERO plugin — PIPELINE_PARALLEL is the only option, and it's the one that crashes.

Given @mzegla's confirmation above that this traces to the GenAI/plugin layer rather than OVMS itself, and @notsyncing's patch set further up — this looks like a real, reproducible defect in OpenVINO GenAI's HETERO + PIPELINE_PARALLEL continuous-batching path, not something model- or build-specific.

Disclosure: the analysis, code, test image and this write-up were produced entirely by Claude (Anthropic's AI model, working through Claude Code). My role was directing and checking the work:

pointed it at this issue and had it review notsyncing's patches for safety and security before applying anything
required confidence before each multi-hour rebuild, which pushed it to trace the crash through the OpenVINO source instead of iterating by trial and error
had the 0002 replacement cross-reviewed by a second model
provided and ran the 2× A770 test hardware, and decided when the machine could be taken out of service for builds
decided the scope: only the zeroing fix goes upstream (notsyncing's patches remain theirs to submit), what gets published, and which GitHub token permissions it could have
reviewed and approved the PR and this comment before they were posted

Please review accordingly, and treat the fix as unverified on anything beyond the hardware below.

Following up on notsyncing's patches — thanks for these, they got us most of the way. Reviewing 0002 turned up a problem, and tracing it found the underlying crash.

Root cause: on HETERO, a GPU RemoteTensor holds one full copy per device. When the hetero plugin copies to or from a host tensor, it offsets device i by i * strides[0] bytes. GenAI's LinearAttentionCacheManager::zero_blocks() resets each new linear-attention state block with a host→device ROI copy from a one-block buffer, so every device after the first reads past the end of that buffer. The reverse direction (device→host) writes past the end, which is the free(): corrupted unsorted chunks heap corruption.

Why 0002's skip isn't enough: remote tensors are allocated with reset=false, and recycled blocks keep the previous sequence's state. With the reset skipped, concurrent requests contaminate each other. On Qwen3.8-27B-int4-ov with greedy decoding, 5 of 8 concurrent answers were wrong: other prompts' answers, </think> loops, repetition.

Fix: openvinotoolkit/openvino.genai#4665 zeroes the state once at allocation through a HETERO-safe padded copy, then resets blocks device→device from a reserved zero block.

Test image, for anyone who wants to try dual-GPU now: ghcr.io/zchef2k/ovms-dual-gpu:2026.5-hetero-v3. It's OVMS 2026.5 with GenAI built from source, ports of notsyncing's 0000/0001/0003/0004, and the #4665 fix in place of 0002. The README, run commands and full patch series are here: https://gist.github.com/zchef2k/14fc14416372ea0d2aeb6ccdaa07ae54

On 2× A770 16 GB, both Qwen3.8-27B-int4-ov and Qwen3.6-35B-A3B-int4-ov now run on HETERO:GPU.0,GPU.1 + PIPELINE_PARALLEL with no crash and no cross-request contamination. Keep --max_num_seqs small (we used 4), since the default of 256 fails the linear-attention memory-budget check. It's an unofficial test build; results from other hardware would be very welcome.

Navigation Menu

Repository navigation

can't run qwen3.6 27b in pipeline parallel #4452

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

can't run qwen3.6 27b in pipeline parallel #4452

Description

Activity

dtrawins commented on Aug 17, 2026

mzegla commented on Aug 18, 2026

HumerousGorgon commented on Aug 23, 2026

notsyncing commented on Aug 24, 2026

HumerousGorgon commented on Aug 24, 2026

notsyncing commented on Aug 24, 2026

notsyncing commented on Sep 2, 2026

zchef2k commented on Oct 10, 2026

zchef2k commented on Oct 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions