Repository navigation
npu: load Ryzen AI OGA NPU model packs on Linux through the engine (pack import -> Q4NX -> our kernels) #671
Description
Goal (owner, 2026-10-30): a Ryzen AI OGA NPU model pack (*_rai_*_npu_*: model.onnx + external data, genai_config.json, descriptors) dropped on a Linux box runs on the NPU through 1bit serve with the engine's own runtime and kernels. The pack supplies the weights, tokenizer and config; the engine supplies everything else. No vendor wheel or shared object is linked, shipped or required, and no vendor-compiled NPU program is executed.
Plan:
1bit npu-pack import <pack-dir> -o <dir>: parse the pack's quantized weights (MatMulNBitstensors, documented com.microsoft contrib op) into the Q4NX container with a layout-version field; self-check dequantises sampled tensors against the reference values bit for bit.- Per-shape full ELFs for the pack model's geometry from our generator (decode M=16 twins, prefill M=128), bit-exact against the CPU reference.
- Gates on the first pack (Qwen2.5-0.5B-Instruct npu_4K): greedy output identical to the vendor stack on three prompts, 12-chunk KLD of our logits vs the vendor stack, decode tok/s at ctx 128/512/2048 and prefill at 128/2048 side by side with the vendor stack measured the same hour, medians of 3.
- A second pack of a different family from the public collection; unsupported pack features filed as issues.
- Later: the ORT execution provider (onnx: ONNX Runtime in-process behind -DONEBIT_ORT, with our plugin EP #647) claims the pack graph; out of scope here.
Related: #647 (ORT EP), #640 (two NPU engines), #650-#653 (NPU feature gaps).
🤖 Generated with Claude Code
Activity
bong-water-water-bong commentedon Oct 31, 2026
Triage:
enhancement. Item 1 is genuinely not started atorigin/main31d52e1:Lane: pi-ogaload. Two facts that bound the first PR:
pack's
model.onnxinitializers plusgenai_config.jsonand writes Q4NX with a layout-version field; theself-check (sample tensors dequantised against the documented com.microsoft
MatMulNBitsformula,bit-exact) is the test, and it needs no device. Item 2's per-shape full ELFs (decode M=16 twins, prefill
M=128) are device work and follow.
8b6895elands ONNX Runtime in-process behind-DONEBIT_ORTwithour plugin EP — that claims the pack graph. This issue claims the weights by re-packing them into our
container, which is what makes a pack runnable on the engine's own kernels today, with no vendor wheel,
shared object or vendor-compiled program anywhere in the path. Keeping them separate is what keeps that
constraint checkable.
One thing I would add to the plan while it is still cheap: a pack that the importer cannot represent
(supported pack features are a moving target) should exit with a named refusal listing the unsupported
attribute, not a partial conversion — item 4 says unsupported features become issues, and that only works if
the importer names them.