OpenVINO: optimize stateful decode and GPU MoE inference - #28638
Merged
Merged
Conversation
github-actions
Bot
added
documentation
0cc4m
reviewed
Sep 10, 2026
Contributor
|
@ggerganov could you kindly review this PR. |
Assisted-by: Claude Sonnet
…state The stateful path seeds its KV state from ggml's cache when the decode position is ahead of what the state holds. That only works when ggml's cache is a plain prefix, where cell i holds position i. A sliding-window layer keeps just the last n_swa positions and drops the rest, so past the window cell i no longer holds position i and the seeded state is wrong. Slicing the state to the decode position also had no bounds check, so a position past the end surfaced as a bare ov::Exception from the ROI constructor (llama_decode ret = -3, with no reason given at default verbosity). Refuse both cases with a clear message instead, and refuse on the compile path too, where a new model starts with an empty state and so can only serve a sequence from its beginning. Reproducible with llama-bench -d, which restores a saved sequence state rather than recomputing the depth prefill. Assisted-by: Claude Opus 5
The stateful path reinterprets ggml's KV buffer [1, 1, seq, n_heads_kv * head_size] as [1, seq, n_heads_kv, head_size]. The head size is already taken from the tensor's own combined dim, because gemma-4 varies it per layer type, but the head count still came from a model-level scalar that compute_llm_params() overwrites per attention node, so it ended up holding whatever the last layer said. gemma-4 varies the head count per layer too: 12B has 8 x 256 sliding layers and 1 x 512 full layers, 31B has 16 x 256 and 4 x 512. So 40 of 12B's 48 layers were split as 1 x 2048 instead of 8 x 256, and attention read the state with the wrong head split - both models decoded garbage on CPU and GPU. E2B is unaffected, its head count is 1 everywhere. Record the count per layer instead and look it up by the cache_k_l<N> leaf name. Key it by layer, not by layer type: the sliding/full classification comes from cache extents, which tie at a small -c, while the head count does not. The stateful state trim now derives its sequence axis per state for the same reason, since pass::KVStateSeqAxis matches per state on the head count. Assisted-by: Claude Opus 5
pass::KVStateSeqAxis was limited to states with a single KV head, where moving the sequence axis from dim 1 to dim 2 is a pure metadata change. The limit was also based on a measurement showing no gain for a multi-head model, but that was taken at depth 0, which is the one depth where this change does nothing. With several heads the pass does more than move metadata: it drops the reader side transpose of the whole accumulated state, which the graph otherwise redoes every token at a cost that grows with the context length, and replaces it with a transpose of the single new row. Measured on GPU, tg128, alternating arms: gemma-4-12B 6.27 -> 9.11 t/s at depth 8192 (stateless is 7.69, so stateful now wins at depth instead of losing), Llama-3.2-1B 47.8 -> 59.6 t/s. Both are within noise at depth 0, which is why the earlier check saw nothing. The state refill needs the rows copied rather than reinterpreted now: ggml stores [seq][n_heads_kv * head_size], and a relayout state with several heads is a different element order. Without that, a refill would seed wrong data - it is reachable today through llama-bench -d. Assisted-by: Claude Opus 5
Packed QKV views used by mmBERT were rejected by the ROPE support check. This split Q/K RoPE onto CPU, prevented cacheless attention detection, and sent fragmented encoder graphs through the decoder-oriented NPUW path.
Accept packed QKV RoPE views, detect cacheless attention from its mask, and run these models as a single full-sequence prefill without NPUW or a decode graph. Also provide static mask, output index, and mean-pooling shapes and inputs.
Replace the decomposed mean/variance normalization graph with an opset6 MVN operation. This preserves the GGML epsilon placement while allowing OpenVINO plugins to compile normalization as one operation with fewer intermediate tensors.
Cache RoPE sine and cosine outputs in the graph-wide tensor map. Build the cache key from all RoPE parameters and the optional frequency-factor input so compatible Q/K and layer nodes share one subgraph without mixing different RoPE configurations.
Expose NodeContext::put_shared() to publish translator-created outputs for graph-level reuse.
…E/SOFTPLUS cases - translate_add: upcast mismatched operand types (e.g. f16/f32 in fused ADD_ADD) to f32, add, then cast once to the output type. opset1::Add requires matching input types and downcasting first lost precision. - translate_glu_swiglu_clamp: same fix, f16 Swish/Clamp rounding was drifting past the test tolerance. - supports_op: reject ROPE with ne[3] > 1 (multi-sequence) since the cos/sin tables only cover one sequence, and SOFTPLUS on GPU since the OpenVINO GPU kernel overflows to inf for large inputs (CPU is fine). - ci/run.sh: serialize test-backend-ops on OpenVINO GPU; running two workers concurrently crashes the GPU plugin (CL_OUT_OF_RESOURCES).
The ReduceSum shortcut for the MoE expert-plane-sum ADD chain drifts past the 1e-7 test tolerance for >8 experts (f32 accumulation order vs CPU reference), intermittently, like the existing Q4_K/Q5_K NMSE case. Expose is_moe_expert_sum_add() so supports_op can gate on expert count and fall back to CPU for just that reduction op.
CI hit ERR=1.8e-3 (> 5e-4 tolerance) for a scalar-output f32 dot product (m=1,n=1,k=2048); didn't reproduce locally in 8 tries, so likely an internal fp16 accumulation path the GPU plugin picks for this tiny shape. m=1 output dim doesn't occur in real model weights, so gate it.
wine99
force-pushed
the
pr-2026-09-09
branch
from
September 15, 2026 04:36
8d3d693 to
3200838
Compare
github-actions
Bot
added
the
devops
Assisted-by: Codex
dzannotti
added a commit
to halo-box/llama.cpp
that referenced
this pull request
Sep 15, 2026
* upstream/master: (72 commits) HIP: Enable AllReduce for ROCm (ggml-org#27825) opencl: choose the MoE expert matmul by batch size for speculative decoding/MTP (ggml-org#27637) ci: build MUSA for only 1 arch (ggml-org#28944) docs: Rule of thumb for AI review time [no ci] (ggml-org#28945) rpc : hash-cache only weights (ggml-org#28789) cuda: support row-contiguous SUM_ROWS (ggml-org#26308) models : move build_arch_graph() after graph() template specialization (ggml-org#28934) vulkan: support sparse Flash Attention (ggml-org#28105) OpenVINO: optimize stateful decode and GPU MoE inference (ggml-org#28638) opencl: add generic ssm_scan (ggml-org#28881) ci: bump kleidiai runners from 22.04 to 24.04 (ggml-org#28885) metal : add FA kernels for HSK=96, HSV=64 (MiniCPM3) (ggml-org#28599) ci: Bump CUDA Windows x64 builds to 13.4.1 (ggml-org#28930) ci : fix android release (ggml-org#28936) cuda : enable i16 and i32 for DUP (ggml-org#28897) cmake : use PROJECT_SOURCE_DIR instead of CMAKE_SOURCE_DIR (ggml-org#28771) webui: stop re-probing disabled /tools endpoint on every message (ggml-org#28646) ci : reuse build tag name when used instead of safe one (ggml-org#28911) CI: hip-quality-check: ignore spill added in bfdc321 (ggml-org#28909) HIP: fattn-mma: use fp32 accumulation on MFMA devices (ggml-org#28576) ...
quimmedes
pushed a commit
to quimmedes/cafe-llama.cpp
that referenced
this pull request
Sep 16, 2026
) * exclude GPU/NPU failing POOL_2D case * Fix pool case * ggml-openvino: fix stateful decode for Gemma-4 per-layer-type head sizes * ggml-openvino: fix MSVC narrowing error in permute * ggml-openvino: classify sliding-window layers structurally on interleaved-SWA models * ggml-openvino: add GGML_OPENVINO_REQUANT_KQUANT to select a 4-bit requant target * ggml-openvino: add GGML_OPENVINO_SPILL_DIR to spill weight buffers to disk * Stateful Performance: Added pass::KVStateSeqAxis to change KV layout * ggml-openvino: fix stateful decode past the sliding-window size Assisted-by: Claude Sonnet * ggml-openvino: refuse stateful decode that cannot resume from the KV state The stateful path seeds its KV state from ggml's cache when the decode position is ahead of what the state holds. That only works when ggml's cache is a plain prefix, where cell i holds position i. A sliding-window layer keeps just the last n_swa positions and drops the rest, so past the window cell i no longer holds position i and the seeded state is wrong. Slicing the state to the decode position also had no bounds check, so a position past the end surfaced as a bare ov::Exception from the ROI constructor (llama_decode ret = -3, with no reason given at default verbosity). Refuse both cases with a clear message instead, and refuse on the compile path too, where a new model starts with an empty state and so can only serve a sequence from its beginning. Reproducible with llama-bench -d, which restores a saved sequence state rather than recomputing the depth prefill. Assisted-by: Claude Opus 5 * ggml-openvino: use the per-layer KV head count for the stateful KV state The stateful path reinterprets ggml's KV buffer [1, 1, seq, n_heads_kv * head_size] as [1, seq, n_heads_kv, head_size]. The head size is already taken from the tensor's own combined dim, because gemma-4 varies it per layer type, but the head count still came from a model-level scalar that compute_llm_params() overwrites per attention node, so it ended up holding whatever the last layer said. gemma-4 varies the head count per layer too: 12B has 8 x 256 sliding layers and 1 x 512 full layers, 31B has 16 x 256 and 4 x 512. So 40 of 12B's 48 layers were split as 1 x 2048 instead of 8 x 256, and attention read the state with the wrong head split - both models decoded garbage on CPU and GPU. E2B is unaffected, its head count is 1 everywhere. Record the count per layer instead and look it up by the cache_k_l<N> leaf name. Key it by layer, not by layer type: the sliding/full classification comes from cache extents, which tie at a small -c, while the head count does not. The stateful state trim now derives its sequence axis per state for the same reason, since pass::KVStateSeqAxis matches per state on the head count. Assisted-by: Claude Opus 5 * ggml-openvino: apply the KV state relayout to any KV head count pass::KVStateSeqAxis was limited to states with a single KV head, where moving the sequence axis from dim 1 to dim 2 is a pure metadata change. The limit was also based on a measurement showing no gain for a multi-head model, but that was taken at depth 0, which is the one depth where this change does nothing. With several heads the pass does more than move metadata: it drops the reader side transpose of the whole accumulated state, which the graph otherwise redoes every token at a cost that grows with the context length, and replaces it with a transpose of the single new row. Measured on GPU, tg128, alternating arms: gemma-4-12B 6.27 -> 9.11 t/s at depth 8192 (stateless is 7.69, so stateful now wins at depth instead of losing), Llama-3.2-1B 47.8 -> 59.6 t/s. Both are within noise at depth 0, which is why the earlier check saw nothing. The state refill needs the rows copied rather than reinterpreted now: ggml stores [seq][n_heads_kv * head_size], and a relayout state with several heads is a different element order. Without that, a refill would seed wrong data - it is reachable today through llama-bench -d. Assisted-by: Claude Opus 5 * ggml-openvino : support ggml_rope_set_offset and simplify op support gating * add more cpy cases * reject BF16 cpy on NPU * Remove mul_mat_id fallback, gate large mul_mat_id only for mxfp4 * ggml-openvino: fuse the MoE expert block into MOECompressed on GPU * ggml-openvino: skip GPU MUL_MAT_ID for unbound expert tensors * ggml-openvino: requantize grouped 8-bit MoE experts on GPU * Enable special strided CPY for conv state writeback * openvino: support cacheless encoder models on NPU Packed QKV views used by mmBERT were rejected by the ROPE support check. This split Q/K RoPE onto CPU, prevented cacheless attention detection, and sent fragmented encoder graphs through the decoder-oriented NPUW path. Accept packed QKV RoPE views, detect cacheless attention from its mask, and run these models as a single full-sequence prefill without NPUW or a decode graph. Also provide static mask, output index, and mean-pooling shapes and inputs. * openvino: optimize norm and RoPE translation Replace the decomposed mean/variance normalization graph with an opset6 MVN operation. This preserves the GGML epsilon placement while allowing OpenVINO plugins to compile normalization as one operation with fewer intermediate tensors. Cache RoPE sine and cosine outputs in the graph-wide tensor map. Build the cache key from all RoPE parameters and the optional frequency-factor input so compatible Q/K and layer nodes share one subgraph without mixing different RoPE configurations. Expose NodeContext::put_shared() to publish translator-created outputs for graph-level reuse. * ggml-openvino : simplify op translators and enable IMROPE/NEOX RoPE fusion * remove unnecessary include and clean up PAD * fix mulmat bug * use ov::as_type_ptr instead of std::dynamic_pointer_cast * ggml-openvino: fix mixed-dtype ADD/SWIGLU_CLAMP, gate unsupported ROPE/SOFTPLUS cases - translate_add: upcast mismatched operand types (e.g. f16/f32 in fused ADD_ADD) to f32, add, then cast once to the output type. opset1::Add requires matching input types and downcasting first lost precision. - translate_glu_swiglu_clamp: same fix, f16 Swish/Clamp rounding was drifting past the test tolerance. - supports_op: reject ROPE with ne[3] > 1 (multi-sequence) since the cos/sin tables only cover one sequence, and SOFTPLUS on GPU since the OpenVINO GPU kernel overflows to inf for large inputs (CPU is fine). - ci/run.sh: serialize test-backend-ops on OpenVINO GPU; running two workers concurrently crashes the GPU plugin (CL_OUT_OF_RESOURCES). * openvino: share compiled models with per-context inference state; fix thread-safety * ggml-openvino: gate MoE expert-sum ReduceSum shortcut past 8 experts The ReduceSum shortcut for the MoE expert-plane-sum ADD chain drifts past the 1e-7 test tolerance for >8 experts (f32 accumulation order vs CPU reference), intermittently, like the existing Q4_K/Q5_K NMSE case. Expose is_moe_expert_sum_add() so supports_op can gate on expert count and fall back to CPU for just that reduction op. * ggml-openvino: gate degenerate m=1,n=1 MUL_MAT on GPU CI hit ERR=1.8e-3 (> 5e-4 tolerance) for a scalar-output f32 dot product (m=1,n=1,k=2048); didn't reproduce locally in 8 tries, so likely an internal fp16 accumulation path the GPU plugin picks for this tiny shape. m=1 output dim doesn't occur in real model weights, so gate it. * ggml-openvino: make SoftPlus decomposition opt-in native Assisted-by: Codex --------- Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com> Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com> Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
zsogitbe
pushed a commit
to zsogitbe/llama.cpp
that referenced
this pull request
Sep 17, 2026
) * exclude GPU/NPU failing POOL_2D case * Fix pool case * ggml-openvino: fix stateful decode for Gemma-4 per-layer-type head sizes * ggml-openvino: fix MSVC narrowing error in permute * ggml-openvino: classify sliding-window layers structurally on interleaved-SWA models * ggml-openvino: add GGML_OPENVINO_REQUANT_KQUANT to select a 4-bit requant target * ggml-openvino: add GGML_OPENVINO_SPILL_DIR to spill weight buffers to disk * Stateful Performance: Added pass::KVStateSeqAxis to change KV layout * ggml-openvino: fix stateful decode past the sliding-window size Assisted-by: Claude Sonnet * ggml-openvino: refuse stateful decode that cannot resume from the KV state The stateful path seeds its KV state from ggml's cache when the decode position is ahead of what the state holds. That only works when ggml's cache is a plain prefix, where cell i holds position i. A sliding-window layer keeps just the last n_swa positions and drops the rest, so past the window cell i no longer holds position i and the seeded state is wrong. Slicing the state to the decode position also had no bounds check, so a position past the end surfaced as a bare ov::Exception from the ROI constructor (llama_decode ret = -3, with no reason given at default verbosity). Refuse both cases with a clear message instead, and refuse on the compile path too, where a new model starts with an empty state and so can only serve a sequence from its beginning. Reproducible with llama-bench -d, which restores a saved sequence state rather than recomputing the depth prefill. Assisted-by: Claude Opus 5 * ggml-openvino: use the per-layer KV head count for the stateful KV state The stateful path reinterprets ggml's KV buffer [1, 1, seq, n_heads_kv * head_size] as [1, seq, n_heads_kv, head_size]. The head size is already taken from the tensor's own combined dim, because gemma-4 varies it per layer type, but the head count still came from a model-level scalar that compute_llm_params() overwrites per attention node, so it ended up holding whatever the last layer said. gemma-4 varies the head count per layer too: 12B has 8 x 256 sliding layers and 1 x 512 full layers, 31B has 16 x 256 and 4 x 512. So 40 of 12B's 48 layers were split as 1 x 2048 instead of 8 x 256, and attention read the state with the wrong head split - both models decoded garbage on CPU and GPU. E2B is unaffected, its head count is 1 everywhere. Record the count per layer instead and look it up by the cache_k_l<N> leaf name. Key it by layer, not by layer type: the sliding/full classification comes from cache extents, which tie at a small -c, while the head count does not. The stateful state trim now derives its sequence axis per state for the same reason, since pass::KVStateSeqAxis matches per state on the head count. Assisted-by: Claude Opus 5 * ggml-openvino: apply the KV state relayout to any KV head count pass::KVStateSeqAxis was limited to states with a single KV head, where moving the sequence axis from dim 1 to dim 2 is a pure metadata change. The limit was also based on a measurement showing no gain for a multi-head model, but that was taken at depth 0, which is the one depth where this change does nothing. With several heads the pass does more than move metadata: it drops the reader side transpose of the whole accumulated state, which the graph otherwise redoes every token at a cost that grows with the context length, and replaces it with a transpose of the single new row. Measured on GPU, tg128, alternating arms: gemma-4-12B 6.27 -> 9.11 t/s at depth 8192 (stateless is 7.69, so stateful now wins at depth instead of losing), Llama-3.2-1B 47.8 -> 59.6 t/s. Both are within noise at depth 0, which is why the earlier check saw nothing. The state refill needs the rows copied rather than reinterpreted now: ggml stores [seq][n_heads_kv * head_size], and a relayout state with several heads is a different element order. Without that, a refill would seed wrong data - it is reachable today through llama-bench -d. Assisted-by: Claude Opus 5 * ggml-openvino : support ggml_rope_set_offset and simplify op support gating * add more cpy cases * reject BF16 cpy on NPU * Remove mul_mat_id fallback, gate large mul_mat_id only for mxfp4 * ggml-openvino: fuse the MoE expert block into MOECompressed on GPU * ggml-openvino: skip GPU MUL_MAT_ID for unbound expert tensors * ggml-openvino: requantize grouped 8-bit MoE experts on GPU * Enable special strided CPY for conv state writeback * openvino: support cacheless encoder models on NPU Packed QKV views used by mmBERT were rejected by the ROPE support check. This split Q/K RoPE onto CPU, prevented cacheless attention detection, and sent fragmented encoder graphs through the decoder-oriented NPUW path. Accept packed QKV RoPE views, detect cacheless attention from its mask, and run these models as a single full-sequence prefill without NPUW or a decode graph. Also provide static mask, output index, and mean-pooling shapes and inputs. * openvino: optimize norm and RoPE translation Replace the decomposed mean/variance normalization graph with an opset6 MVN operation. This preserves the GGML epsilon placement while allowing OpenVINO plugins to compile normalization as one operation with fewer intermediate tensors. Cache RoPE sine and cosine outputs in the graph-wide tensor map. Build the cache key from all RoPE parameters and the optional frequency-factor input so compatible Q/K and layer nodes share one subgraph without mixing different RoPE configurations. Expose NodeContext::put_shared() to publish translator-created outputs for graph-level reuse. * ggml-openvino : simplify op translators and enable IMROPE/NEOX RoPE fusion * remove unnecessary include and clean up PAD * fix mulmat bug * use ov::as_type_ptr instead of std::dynamic_pointer_cast * ggml-openvino: fix mixed-dtype ADD/SWIGLU_CLAMP, gate unsupported ROPE/SOFTPLUS cases - translate_add: upcast mismatched operand types (e.g. f16/f32 in fused ADD_ADD) to f32, add, then cast once to the output type. opset1::Add requires matching input types and downcasting first lost precision. - translate_glu_swiglu_clamp: same fix, f16 Swish/Clamp rounding was drifting past the test tolerance. - supports_op: reject ROPE with ne[3] > 1 (multi-sequence) since the cos/sin tables only cover one sequence, and SOFTPLUS on GPU since the OpenVINO GPU kernel overflows to inf for large inputs (CPU is fine). - ci/run.sh: serialize test-backend-ops on OpenVINO GPU; running two workers concurrently crashes the GPU plugin (CL_OUT_OF_RESOURCES). * openvino: share compiled models with per-context inference state; fix thread-safety * ggml-openvino: gate MoE expert-sum ReduceSum shortcut past 8 experts The ReduceSum shortcut for the MoE expert-plane-sum ADD chain drifts past the 1e-7 test tolerance for >8 experts (f32 accumulation order vs CPU reference), intermittently, like the existing Q4_K/Q5_K NMSE case. Expose is_moe_expert_sum_add() so supports_op can gate on expert count and fall back to CPU for just that reduction op. * ggml-openvino: gate degenerate m=1,n=1 MUL_MAT on GPU CI hit ERR=1.8e-3 (> 5e-4 tolerance) for a scalar-output f32 dot product (m=1,n=1,k=2048); didn't reproduce locally in 8 tries, so likely an internal fp16 accumulation path the GPU plugin picks for this tiny shape. m=1 output dim doesn't occur in real model weights, so gate it. * ggml-openvino: make SoftPlus decomposition opt-in native Assisted-by: Codex --------- Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com> Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com> Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Requirements