OpenVINO: optimize stateful decode and GPU MoE inference - #28638

Merged

ggerganov merged 31 commits into

ggml-org:masterfrom

ravi9:pr-2026-09-09

Sep 15, 2026

Commits on Sep 14, 2026

exclude GPU/NPU failing POOL_2D case

mostafafaheem
authored and
ravi9
committed
cf78650
Fix pool case

mostafafaheem
authored and
ravi9
committed
f72d986
ggml-openvino: fix stateful decode for Gemma-4 per-layer-type head sizes

cavusmustafa
authored and
ravi9
committed
011271a
ggml-openvino: fix MSVC narrowing error in permute

cavusmustafa
authored and
ravi9
committed
70ceed6
ggml-openvino: classify sliding-window layers structurally on interleaved-SWA models

cavusmustafa
authored and
ravi9
committed
4fb04df
ggml-openvino: add GGML_OPENVINO_REQUANT_KQUANT to select a 4-bit requant target

cavusmustafa
authored and
ravi9
committed
0e9794e
ggml-openvino: add GGML_OPENVINO_SPILL_DIR to spill weight buffers to disk

cavusmustafa
authored and
ravi9
committed
c14780c
Stateful Performance: Added pass::KVStateSeqAxis to change KV layout

cavusmustafa
authored and
ravi9
committed
cd327a3
ggml-openvino: fix stateful decode past the sliding-window size

cavusmustafa
authored and
ravi9
committed
80aec82
ggml-openvino: refuse stateful decode that cannot resume from the KV state

cavusmustafa
authored and
ravi9
committed
ca08e4f
ggml-openvino: use the per-layer KV head count for the stateful KV state

cavusmustafa
authored and
ravi9
committed
fecf802
ggml-openvino: apply the KV state relayout to any KV head count

cavusmustafa
authored and
ravi9
committed
6fbe283
ggml-openvino : support ggml_rope_set_offset and simplify op support gating

mostafafaheem
authored and
ravi9
committed
66d2d07
add more cpy cases

mostafafaheem
authored and
ravi9
committed
73ddeb5
reject BF16 cpy on NPU

mostafafaheem
authored and
ravi9
committed
ba4e39d
Remove mul_mat_id fallback, gate large mul_mat_id only for mxfp4

wine99
authored and
ravi9
committed
f304295
ggml-openvino: fuse the MoE expert block into MOECompressed on GPU

cavusmustafa
authored and
ravi9
committed
6645a9e
ggml-openvino: skip GPU MUL_MAT_ID for unbound expert tensors

cavusmustafa
authored and
ravi9
committed
8c24268
ggml-openvino: requantize grouped 8-bit MoE experts on GPU

cavusmustafa
authored and
ravi9
committed
314afa5
Enable special strided CPY for conv state writeback

wine99
authored and
ravi9
committed
db83eee
openvino: support cacheless encoder models on NPU

zhaixuejun1993
authored and
ravi9
committed
e871404
openvino: optimize norm and RoPE translation

zhaixuejun1993
authored and
ravi9
committed
e13baf3
ggml-openvino : simplify op translators and enable IMROPE/NEOX RoPE fusion

mostafafaheem
authored and
ravi9
committed
a59f311
remove unnecessary include and clean up PAD

mostafafaheem
authored and
ravi9
committed
91903c8
fix mulmat bug

mostafafaheem
authored and
ravi9
committed
eec425e
use ov::as_type_ptr instead of std::dynamic_pointer_cast

mostafafaheem
authored and
ravi9
committed
926ef68

Commits on Sep 15, 2026

ggml-openvino: fix mixed-dtype ADD/SWIGLU_CLAMP, gate unsupported ROPE/SOFTPLUS cases

ravi9
authored and
wine99
committed
db8913c
openvino: share compiled models with per-context inference state; fix thread-safety
wine99
committed
88b44d0
ggml-openvino: gate MoE expert-sum ReduceSum shortcut past 8 experts

ravi9
authored and
wine99
committed
a81e3f6
ggml-openvino: gate degenerate m=1,n=1 MUL_MAT on GPU

ravi9
authored and
wine99
committed
3200838
ggml-openvino: make SoftPlus decomposition opt-in native
wine99
committed
5bcd117

Navigation Menu

OpenVINO: optimize stateful decode and GPU MoE inference - #28638

OpenVINO: optimize stateful decode and GPU MoE inference#28638
ggerganov merged 31 commits into
ggml-org:masterfrom
ravi9:pr-2026-09-09

Commits on Sep 14, 2026

exclude GPU/NPU failing POOL_2D case

Fix pool case

ggml-openvino: fix stateful decode for Gemma-4 per-layer-type head sizes

ggml-openvino: fix MSVC narrowing error in permute

ggml-openvino: classify sliding-window layers structurally on interleaved-SWA models

ggml-openvino: add GGML_OPENVINO_REQUANT_KQUANT to select a 4-bit requant target

ggml-openvino: add GGML_OPENVINO_SPILL_DIR to spill weight buffers to disk

Stateful Performance: Added pass::KVStateSeqAxis to change KV layout

ggml-openvino: fix stateful decode past the sliding-window size

ggml-openvino: refuse stateful decode that cannot resume from the KV state

ggml-openvino: use the per-layer KV head count for the stateful KV state

ggml-openvino: apply the KV state relayout to any KV head count

ggml-openvino : support ggml_rope_set_offset and simplify op support gating

add more cpy cases

reject BF16 cpy on NPU

Remove mul_mat_id fallback, gate large mul_mat_id only for mxfp4

ggml-openvino: fuse the MoE expert block into MOECompressed on GPU

ggml-openvino: skip GPU MUL_MAT_ID for unbound expert tensors

ggml-openvino: requantize grouped 8-bit MoE experts on GPU

Enable special strided CPY for conv state writeback

openvino: support cacheless encoder models on NPU

openvino: optimize norm and RoPE translation

ggml-openvino : simplify op translators and enable IMROPE/NEOX RoPE fusion

remove unnecessary include and clean up PAD

fix mulmat bug

use ov::as_type_ptr instead of std::dynamic_pointer_cast

Commits on Sep 15, 2026

ggml-openvino: fix mixed-dtype ADD/SWIGLU_CLAMP, gate unsupported ROPE/SOFTPLUS cases

openvino: share compiled models with per-context inference state; fix thread-safety

ggml-openvino: gate MoE expert-sum ReduceSum shortcut past 8 experts

ggml-openvino: gate degenerate m=1,n=1 MUL_MAT on GPU

ggml-openvino: make SoftPlus decomposition opt-in native