Repository navigation
NPU: read all three Q4NX chunk kinds (q4_1, Q4_K, Q8), repack Q4_K into q4_1 - #33
Merged
Merged
Conversation
bong-water-water-bong
enabled auto-merge (squash)
September 24, 2026 10:39
…K into q4_1
npu/q4nx.{h,cpp}: the chunk kind of a tensor (its shape's last dimension), tile and
tensor decoders for q4_1 (5120 B), Q4_K (4736 B) and Q8 (8704 B), and a Q4_K -> q4_1
repack that is exact apart from rounding S*scale and M*min to bf16, so the lane and
dx (which read q4_1) run Q4_K weights unchanged. Tensor::tiles() understands the
3-d [tile rows, tile cols, chunk] shape Qwen3.5-4B uses.
The layouts are 1bit-MONSTER's (npu-infer/src/q4nx_dequant.c, verified there against
the checkpoint's bf16 twin and a CPU forward). tests/npu_q4nx_test.cpp checks ours
bit for bit against its Python decoder (npu-infer/tools/q4nx_dequant.py at
35fafde8f) on one real tile of each kind (tests/golden/q4nx: Qwen3.5-4B qkv_proj and
lm_head, Qwen3-0.6B q_proj; the expected SHA-256 of each 32 x 256 f32 output), and the
repack against its bound: max error 0.00032, 0.92 of the bf16 rounding bound.
docs/npu.md, "Q4NX chunks".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…annot carry a comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
force-pushed
the
npu-q4nx-chunks
branch
from
September 24, 2026 10:40
6d1722f to
be73835
Compare
This was referenced Oct 7, 2026
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 25, 2026
Private HIP add-on pin 744bd713d968 -> 17a882640fa88: NVFP4 + Q2_0 on every family (#33), 2..16-token verify kernels (#32) and the GEAK WMMA rewrite (#37), DFlash2 selector (#34), ZAYA lm_head + one-token kernels (#35), Bonsai ternary verify + PQ2_0 prefill (#36), IQ2_S expert prefill + decode (#38/#39). hw-gate kat goldens (7 models x 9 cases) and KLD bases re-recorded at llama.cpp e7e5cd08 + this pin; check-back 13/13 PASS on the gate box. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Q4NX was fully decoded in 1bit-MONSTER (npu-infer, the Qwen3.5-4B work, merged in #2684): besides the 5120-byte q4_1 tile the engine reads today, Q4NX has a 4736-byte Q4_K chunk and an 8704-byte Q8 chunk. This brings that into the engine.
npu/q4nx.{h,cpp}(new, our notice): chunk kind from the tensor shape's last dimension; tile/tensor decoders for all three; a Q4_K -> q4_1 repack (exact apart from roundingS*scaleandM*minto bf16) so the lane and dx, which read q4_1, run Q4_K weights unchanged. Q8 does not fit q4_1.Tensor::tiles()understands the 3-d[tile rows, tile cols, chunk]shape Qwen3.5-4B uses.code * scale + zeroS * scale * code + M * mind * codeTested (
npu_q4nx, runs in CI): each decoder matches 1bit-MONSTER's verified Python decoder bit for bit on a real tile of each kind (committed intests/golden/q4nx, checked by the SHA-256 of the 32 x 256 f32 output); the repack's max error is 0.00032, 0.92 of its bf16 rounding bound.Note: the Q4_K layout originally came from OpenFlowLM-Next's
q4nx.py(a file-format description, not engine or kernel code).🤖 Generated with Claude Code