Xuejun/enable npu opt v1 - #305
Open
zhaixuejun1993 wants to merge 3 commits into
Open
Conversation
Groundwork that was sitting uncommitted in the tree, kept separate from the tied-embedding change that follows: - env passthrough for NPU_COMPILER_TYPE, NPUW_FUNCALL_FOR_ALL, NPUW_UNFOLD_IREQS and COMPILATION_NUM_THREADS, plus a generic GGML_OPENVINO_NPU_CONFIG KEY=VALUE escape hatch for bisecting plugin options without a rebuild - GGML_OPENVINO_KV_SCATTER_ELEMENTS: ScatterElementsUpdate instead of ScatterUpdate for the single-row KV write, which is not in-place on NPU - GGML_OPENVINO_REDUCE_COMPILE_MEM: stream requantization in chunks rather than materializing the whole tensor - GGML_OPENVINO_RELEASE_WEIGHTS / GGML_OPENVINO_COMPILE_FROM_IR fail-fast guards for recompiles after host weight buffers are dropped - GGML_OPENVINO_TOKEN_EMBD_I8 and GGML_OPENVINO_NPU_KEEP_Q4_0 opt-ins
For a tied embedding the token_embd table is also the lm_head weight, so one OV constant feeds both a GET_ROWS at layer 0 and a MUL_MAT after the last layer. Those two consumers land in different NPUW partitions, and a value crossing a partition boundary has to be materialized, so gathering rows out of the dequantized table forces the whole table into real memory. That is why compressing the table used to lose: the dequantization traffic cost more than the smaller weights saved. gather_compressed_rows() walks the dequantization chain emitted by make_int4_weights/make_int8_weights, gathers the requested rows out of the leaf Constants, and rebuilds the chain on top of them. The row lookup then reads packed rows and dequantizes only those, and the chain feeding the lm_head matmul is left with a single consumer, so it stays compressed. Only the node types those two builders emit are recognized; anything else falls back to the plain Gather, and the rewrite is skipped for tables under 1M elements and for chains with no low-precision Constant, which leaves the current f16 default bit for bit unchanged. GGML_OPENVINO_TOKEN_EMBD_I4 opts token_embd into group-128 int4, which is only worth doing together with the above. phi4-mini Q4_0 on Panther Lake, -p 512 -n 32 -b 512 -ub 512: arm pp512 tg32 device p50 f16 (default) 344.34 10.70 89.71 ms int4 + packed gather 346.17 13.11 70.59 ms int4, gather not sunk 328.97 8.26 118.11 ms Decode weight traffic goes from 2.90 GB to 1.99 GB per step, and the measured 19.1 ms saving matches the 21.5 ms that predicts at the 42.4 GB/s the device reaches on an f16 matmul of this shape. Accuracy of the int4 embedding is NOT established. llama-perplexity cannot run on this backend (it asks for all-position logits, which the static NPU graph has no shape for), and the one greedy sample that could be compared diverged and read slightly worse. The flag stays opt-in for that reason.
Allow the Q/K ROPE subviews used by Phi-4 so the model remains a single OpenVINO graph, avoiding NPUW f16 interconnect output mismatches. Serialize prefill and decode static model compilation because concurrent NPUW compilation is unstable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several enhancements and configuration options to the OpenVINO integration in
ggml, primarily focused on improving NPU (Neural Processing Unit) support, quantization flexibility, and memory efficiency. The changes expand environment variable controls, add new quantization pathways, and optimize certain operations for better performance on NPUs.Key changes include:
Expanded NPU and Quantization Configuration
Added support for new environment variables to fine-tune NPU compilation and runtime behavior, including
GGML_OPENVINO_NPU_COMPILER_TYPE,GGML_OPENVINO_NPUW_FUNCALL_FOR_ALL,GGML_OPENVINO_NPUW_UNFOLD_IREQS,GGML_OPENVINO_COMPILATION_NUM_THREADS,GGML_OPENVINO_NPU_CONFIG,GGML_OPENVINO_TOKEN_EMBD_I8,GGML_OPENVINO_TOKEN_EMBD_I4,GGML_OPENVINO_NPU_KEEP_Q4_0,GGML_OPENVINO_COMPILE_FROM_IR, andGGML_OPENVINO_KV_SCATTER_ELEMENTS. These allow more granular control over compilation strategies, quantization types, and memory/performance trade-offs. [1] [2] [3]Enhanced quantization logic for embedding tables, allowing selection between F16, int8, or group-128 int4 representations based on new environment variables. This enables memory and performance optimizations, especially for tied embeddings.
Quantization and Memory Optimizations
Updated the
quantize_q4_0function and its usage to support streaming quantization with correct block offset handling, enabling efficient chunked quantization and reducing peak memory usage during compilation. [1] [2] [3] [4] [5] [6] [7]Modified the
get_rowsoperation to gather compressed (packed) embedding rows when possible, avoiding unnecessary materialization of large dequantized tables and improving efficiency for tied embeddings.NPU-Specific Operation Improvements
ScatterElementsUpdateinstead ofScatterUpdatefor key-value cache updates on NPU, reducing unnecessary data copying and improving performance for single-row writes. [1] [2]Bug Fixes and Code Cleanups
Fixed a bug in ROPE operation support checks, ensuring correct dimension comparisons for view operations.
Included missing header for extra OpenVINO helpers in
set_rows.cpp.## OverviewAdditional information
Requirements