CPU reference: GGUF reader, dequant, Qwen3 fp32 forward, golden test by bong-water-water-bong · Pull Request #1 · 1bit-MONSTER/engine

bong-water-water-bong · 2026-09-22T18:59:42Z

The fp32 correctness oracle. Every NPU and GPU backend will be gated against it.

What

engine/core: mmap GGUF v2/v3 reader, F32/F16/BF16/Q8_0 dequant, model config from GGUF metadata (refuses unverified architectures), thread pool.
engine/backends/cpu: Qwen3 decoder in fp32. Loading fails on any missing, misshaped or unused tensor.
tests/: GGUF and dequant unit tests on synthetic data (run in CI); golden_cpu against HF transformers fp32 logits.
tools/golden/make_golden.py, docs/cpu-reference.md.

Results (Strix Halo, Qwen3-0.6B BF16 GGUF vs transformers fp32 eager)

	result	gate
argmax agreement	36/36	all
worst per-position KL	1.3e-10	≤ 1e-6
worst max \|Δlogit\|	1.0e-4	reported

Negative controls: RoPE Normal instead of Neox → 17/36 mismatches, KL 7.9; k-norm skipped → 34/36, KL 24.9. Both fail.

Test plan

ctest locally (unit tests) and on strixhalo with -DENGINE_GOLDEN_MODEL/DIR (3/3 pass)
builds with -Wall -Wextra -Wpedantic, zero warnings (g++ 15.2)
mutation check above

Not in this PR: tokenizer, K-quants, a longer-context golden.

🤖 Generated with Claude Code

The fp32 oracle the NPU and GPU backends will be tested against. - engine/core: mmap GGUF v2/v3 reader, F32/F16/BF16/Q8_0 dequant, model config from GGUF metadata (only verified archs accepted), thread pool. - engine/backends/cpu: Qwen3 decoder in fp32. Load fails on any missing, misshaped or unused tensor. - tests: GGUF and dequant unit tests on synthetic data (run in CI); golden_cpu compares against HF transformers fp32 logits. - tools/golden/make_golden.py writes the golden logits. Qwen3-0.6B, 36 positions teacher-forced: argmax 36/36, worst KL 1.3e-10 (gate 1e-6). Deliberately broken RoPE and k-norm builds both fail the gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…bases our commits third_party/ds4 moves to 1bit-MONSTER/ds4 1bit/main (a7f2e1e): upstream antirez/ds4 main plus our 1BP v5 reader, converter and checker (ds4 #1). bump-ds4.yml now rebases the commits we carry onto upstream main, tags the old tip and opens the pin PR, as bump-llama-vulkan does. 1bit serve recognises a 1BP package by its magic and serves it on ds4, --device auto included. docs/dwarfstar.md and NOTICE follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…bases our commits (#196) third_party/ds4 moves to 1bit-MONSTER/ds4 1bit/main (a7f2e1e): upstream antirez/ds4 main plus our 1BP v5 reader, converter and checker (ds4 #1). bump-ds4.yml now rebases the commits we carry onto upstream main, tags the old tip and opens the pin PR, as bump-llama-vulkan does. 1bit serve recognises a 1BP package by its magic and serves it on ds4, --device auto included. docs/dwarfstar.md and NOTICE follow. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…y_register_host, because the trigger registers with coherence unset Reading the dump further changed what #315 needs most, so the queue's first item is superseded. NEW #1: iree_hal_streaming_memory_register_host (libhrx/src/binding/common/memory.c, called from the in-tree hipHostRegister). It is ABSENT from the dump - no JSON and not among the 103 failures, so it was never captured rather than captured-and-failed. It outranks hrx_allocator_virtual_memory_map because: (1) it decides the COHERENCE DEFAULT for host-registered memory, and the register path has explicit flags (FLAG_HIP_COHERENT 1<<30, FLAG_HIP_NON_COHERENT 1<<31) unlike the pool path; (2) the #315 trigger registers with the flag UNSET - evictonly.hip calls hipHostRegister(b, bytes, hipHostRegisterDefault), flags=0, and the decompiled hipHostRegister (captured, OK) overwrites its own READ_ONLY default with 0 & (...) = 0, so it passes FLAG_DEFAULT = 0, neither coherence bit set - so what the trigger's memory IS depends entirely on what register_host does with 'unspecified'; (3) if that default is non-coherent, the eviction churn operates on non-coherent registered memory and 'coherence does not matter' is wrong for a second independent reason. Order: register_host, then virtual_memory_map/protect/unmap, then the hrx_fence_* and hrx_*_synchronize families already listed. Also asks for the HIP hipHostRegister flag values from the headers so the (flags & 8) branch can be confirmed against the header rather than inferred.

…SISTENT mapping ggml-hrx.cpp:320-338 maps a host-visible buffer ONCE at allocation (hrx_buffer_map READ|WRITE over the whole buffer) and buffer_get then does a plain memcpy out of that mapping; nothing re-maps or invalidates it. So every engine read of a host-visible buffer is a CPU read of a mapping established at allocation. That gives memcpy_stable=0 a third explanation the read-twice control does not separate: read #1 == read #2 != sync means a stale cache line, read #1 != read #2 means the buffer moved - but with a persistent mapping of NON-COHERENT memory, the CPU's own cached view changing between the reads (lines evicted and re-fetched) also yields memcpy_stable=0 with nothing device-side moving. The arm that produced 75/75 was running with the eviction forcer in the same cell, and eviction pressure is exactly what changes which lines are held. The stronger reading in 315-read-twice-result-buffer-moves-under-the-reader.md is corrected. Re-opened narrowly: coherence, as a property of the MAPPING rather than of the allocation flags section 13 tested (hv56=1 still faulted, so a coherent allocation does not remove the fault; whether it removes the mapping's read-to-read divergence is a different, measurable question). Not re-opened: the falsified writeback candidate, whose measurement was a counter rather than a hash comparison. Hands forward: run the twin with GGML_HRX_BUFFER_TRACE=1 in both memory-type cells, which is what 315-two-cell-memory-type-preregistration.md pre-registered.

… box run is authorised) Targets the one measured, unexplained fact left: result_norm changes between two consecutive identical plain reads in a faulting arm (75/75 and 128/128 memcpy_stable=0, three reads three hashes), with the deferred writeback and the persistent-mapping cache explanation both already removed by measurement. Instrument: one edit to the existing twin - a backend_synchronize between read #1 and read #2, reporting hash_after_sync. Configuration: result_norm + BUFFER_TRACE, ub=512, in-regime, eviction forcer, stop at the first fault; diagnostic cell, no arm counts toward a campaign. Readings pre-declared: read#1 != read#2 and hash_after_sync == read#2 -> the writer IS ordered by the stream, so the reads raced work a sync would have covered, which makes GraphExecutor::execute returning without a wait the live candidate. read#1 != read#2 and hash_after_sync != read#2 -> the writer is not ordered by context->stream at all, which kills that candidate and points at a second writer of the arena slot. No divergence -> uninformative, itself new. Falsifier stated. Cost: one instrument edit, one rebuild, one faulting arm (the forcer produced a fault in 1 arm for both previous probes).

Adds a fresh ggml_backend_synchronize(backend) and then a FOURTH plain read of the same tensor at the same call, reported as hash_after_sync with after_sync_stable = (hash_after_sync == memcpy2_hash) on a new [hrx-f32-sync-after] line. Tests whether the writer that moves result_norm between two adjacent plain reads is ordered by the stream the harness already synchronises. Pre-registered reading: #1 != #2 and after_sync == #2 means the writer IS ordered by the stream (a fresh sync stops the movement, so the reads were racing work the sync covers, making GraphExecutor::execute returning without a wait the live candidate); #1 != #2 and after_sync != #2 means the writer is not ordered by context->stream at all, which kills that candidate and points at a second writer of the reused arena slot; #1 == #2 is uninformative. Gated on the same GGML_HRX_TENSOR_FP path as the twin, inert when unset. Refuses to apply unless the read-twice control is present, and the build asserts the markers in source and binary rather than trusting an exit status.

…leaves its buffer Review of #854 (engine#845): the geometry half was right and the activation half was not. The design's K is the container's padded K (1024 for hidden 896), and every launch passes it, while the forward's activation vectors are H floats -- so `npu_gemm`'s `A[k]`/`for i < K` amax scan and `npu_gemm_rows`' `A[m*K + k]` read 128 floats past the end of `x`/`xn`/`o`/`d`/`ctx` on every launch. It was masked only because the container's pad columns are zero, so `a * 0` contributed nothing; the device path also took its int8 activation scale from the amax over the overrun. q4nx_pad_a_f32() (npu/forward/src/q4nx_pack.c) memcpy's each row's real_k floats into a K-wide row and zero-fills the tail, and both GEMM entry points call it once per launch when the design's real_k is below K. The pad columns of the container are zero by construction, so this is the same dot product, not an approximation. real_k <= 0 or >= K is the identity, which keeps the MLA designs (derive_mla does not set real_k) on their existing path. The parameter is explicit at all 42 call sites rather than defaulted, so a design whose real_k is not the activation width cannot silently go back to overrunning: npu_gemm(..., "attn_qkv", g->real_k) Verified on this box, since CI does not compile npu/forward (ONEBIT_NPU): Release (gcc-16), 0.5B container, device-free host GEMM: NPU_INFER_HOST_GEMM=1 NPU_NO_DEVICE_LOCK=1 ./1bit forward-serve -m <c05b> --kernels <m16> -p 8124 -> /health ok, /v1/completions answers, 0 AddressSanitizer reports -fsanitize=address, same request: 0 reports. With the pad disabled again in the same build: ==929485==ERROR: AddressSanitizer: heap-buffer-overflow #0 Q4nxNpuForward::npu_gemm(...) npu/forward/src/q4nx_forward.cpp:2624 #1 Q4nxNpuForward::step(...) npu/forward/src/q4nx_forward.cpp:4209 tests: ctest -R "npu_design_grid|npu_plan|npu_q4nx|npu_forward_loader" -> 5/5 pass, and npu_design_grid_test gains five activation-pad cases whose source is allocated at exactly M*real_k floats with a canary parked immediately after it, so a pad that reads past its input (instead of memcpy-ing the real width) turns the canary and fails the suite. The answer is still wrong on this container -- that is #855's finding, reproduced here independently through a device-free reference forward, and unrelated to this pad.

…o a pack whose hidden is not a multiple of 256 has kernels to find (closes #845) (#854) * npu: derive GEMM K and N on the container's and the kernel's grids (engine#845) A Q4NX q4_1 chunk is 32 rows x 256 columns, so `1bit npu-pack import` stores every projection with K padded up the 256 grid; a full ELF's N is built on the gemv family's column pass (32 x 4 x cols = 512 columns), which is also the grid the M=16 twins are named and built on. The forward derived both from the model's own dims instead, so a Qwen2.5-0.5B OGA pack (hidden 896, intermediate 4864) dequantised a 1024-column tensor at 896 columns -- which the chunk decoder refuses outright, "forward step failed: dequant failed" -- and probed artifact names no kernel has (full_i8_QKV_K896_N1152). q4nx_derive_designs() now pads K with q4nx_pad_k() and N with q4nx_pad_n(), keeps the real widths beside them (real_k, and real_n where it differs), and assemble_B() scatters real rows and zeroes the kernel's n-tile pad. The pad K columns are zero in the container by construction (pack_import writes scale 0, zero 0, codes 0), so a padded K is an exact reading of the container, not an approximation: the extra rows add exactly 0 to every dot product. Both grids are the identity for every model validated before this (hidden 1024/2048/3072/5120/6144, N 4096/6144/14336/17408), which is why the mismatch went unseen. The M=16 twin guard moves from `N % 1024 == 0` to the same 512 grid. The old test refused three of the five ELFs that exist for this geometry (QKV N 1536, GU 9728, head 152064), and those five ran bit-exact over their full width on the array (npu-kernels ogaload/oga-qwen25-0.5b-bench-20261101, K1024 N{1536,1024,9728,1024} _m16), so the guard was stricter than the family that was built. A geometry whose twin is absent still falls back to its M=128 kernel, because the caller probes the disk. Also fixes the container-index walk that made the gate impossible: the importer's own manifest lives under `__metadata__` and repeats every tensor's name and shape, so the flat walk paired those records with the real table's --offset entries and resolved this model's embedding to a `q4_1` descriptor over the next tensor's bytes. parse_json_metadata() now skips the whole object, because metadata is not a tensor table. Verified on this box: * strixhalo, Qwen2.5-0.5B-Instruct_rai_1.8.0_npu_4K imported by `1bit npu-pack import` (291 tensors, 67088 tiles): the derived names are now exactly the five built ELFs, full_i8_{QKV,O,GU,D}_K1024_N{1536,1024,9728,1024}_m16, and `1bit forward-serve` loads and answers it in host-GEMM mode (no device, NPU_INFER_HOST_GEMM=1 NPU_NO_DEVICE_LOCK=1). Its *text* is not right yet -- "!!!!!!!!!!!!" for a three word prompt -- which is item #671 item 5's business (the pack's projections are measurably not the upstream checkpoint's, so the reference is AMD's stack running the pack) and is filed separately rather than hidden here. * the on-grid path is unchanged: the same build serves ~/models/npu/Qwen3-0.6B with sane text (" Paris. The capital of France is also the capital of the country of Belgium.") and its plan.json --check output differs only by the newer generated_by/attention fields main already has. * tests: npu_design_grid (new: 45 checks -- the two grids, the five artifact names, every twin, assemble_B's pad zeroing on O and QKV, and an on-grid model's unchanged names), npu_forward_loader (new case: a manifest that repeats a tensor's name does not shadow it), npu_plan, npu_q4nx, npu_pack_import, npu_pack_import_oga -- 7/7 pass; copyright.py --check rc 0. * npu forward: pad the activation rows to the kernel's K, so no A read leaves its buffer Review of #854 (engine#845): the geometry half was right and the activation half was not. The design's K is the container's padded K (1024 for hidden 896), and every launch passes it, while the forward's activation vectors are H floats -- so `npu_gemm`'s `A[k]`/`for i < K` amax scan and `npu_gemm_rows`' `A[m*K + k]` read 128 floats past the end of `x`/`xn`/`o`/`d`/`ctx` on every launch. It was masked only because the container's pad columns are zero, so `a * 0` contributed nothing; the device path also took its int8 activation scale from the amax over the overrun. q4nx_pad_a_f32() (npu/forward/src/q4nx_pack.c) memcpy's each row's real_k floats into a K-wide row and zero-fills the tail, and both GEMM entry points call it once per launch when the design's real_k is below K. The pad columns of the container are zero by construction, so this is the same dot product, not an approximation. real_k <= 0 or >= K is the identity, which keeps the MLA designs (derive_mla does not set real_k) on their existing path. The parameter is explicit at all 42 call sites rather than defaulted, so a design whose real_k is not the activation width cannot silently go back to overrunning: npu_gemm(..., "attn_qkv", g->real_k) Verified on this box, since CI does not compile npu/forward (ONEBIT_NPU): Release (gcc-16), 0.5B container, device-free host GEMM: NPU_INFER_HOST_GEMM=1 NPU_NO_DEVICE_LOCK=1 ./1bit forward-serve -m <c05b> --kernels <m16> -p 8124 -> /health ok, /v1/completions answers, 0 AddressSanitizer reports -fsanitize=address, same request: 0 reports. With the pad disabled again in the same build: ==929485==ERROR: AddressSanitizer: heap-buffer-overflow #0 Q4nxNpuForward::npu_gemm(...) npu/forward/src/q4nx_forward.cpp:2624 #1 Q4nxNpuForward::step(...) npu/forward/src/q4nx_forward.cpp:4209 tests: ctest -R "npu_design_grid|npu_plan|npu_q4nx|npu_forward_loader" -> 5/5 pass, and npu_design_grid_test gains five activation-pad cases whose source is allocated at exactly M*real_k floats with a canary parked immediately after it, so a pad that reads past its input (instead of memcpy-ing the real width) turns the canary and fails the suite. The answer is still wrong on this container -- that is #855's finding, reproduced here independently through a device-free reference forward, and unrelated to this pad. * npu forward: the head picks its design on the kernel K grid, not on H run_lm_head and run_lm_head_rows chose the head's GEMM design with `designs_[d].K == H`. That is the same rule only while a model's hidden size is already on the q4_1 chunk grid; since this branch pads K to q4nx_pad_k(H), no design has K == H for hidden 896 and the first OGA pack to reach step() failed with forward step failed: LM head: no GEMM design with K=H available although all five designs (and their M=16 twins) had loaded and initialised on the array (measured: one npu-hold ticket, 5 s, 0 amdxdna errors). Both comparisons now use kpad_, the grid the container's hidden width lands on, and both error strings print the grid they looked for instead of "K=H". Identity for every geometry already validated (hidden 896 is the first hidden that is not a multiple of 256 in this family), and the host-GEMM smoke request still answers after the change. Found by the array leg of #845 itself: the code around the lookup was already updated for kpad_ in this branch (tile_elems, with the "kernel's K is the padded width" comment); the lookup was missed. npu/forward is compiled only under ONEBIT_NPU, so CI cannot catch it and the host-GEMM acceptance path never enters this branch. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>

bong-water-water-bong merged commit c14a5ea into main Sep 22, 2026
1 check passed

This was referenced Sep 22, 2026

Remove the CPU reference and tokenizer; port the existing engine instead #5

Merged

Docs: the onebit recipe lives in our Lemonade fork #24

Merged

github-actions Bot mentioned this pull request Sep 25, 2026

Bump llama.cpp to c075cc1: HRX0 runs MoE models above 128 experts (Qwen3.6-35B-A3B) #84

Merged

github-actions Bot mentioned this pull request Sep 28, 2026

DwarfStar from our fork: 1BP packages; bump-ds4 rebases our commits #196

Merged

bong-water-water-bong deleted the cpu-reference branch October 12, 2026 01:30

github-actions Bot mentioned this pull request Nov 4, 2026

os/v3/puppy: vm-check asserts the image's kernel release and that OUR driver is the one a modprobe resolves #989

Merged

Navigation Menu

Repository navigation

CPU reference: GGUF reader, dequant, Qwen3 fp32 forward, golden test - #1

CPU reference: GGUF reader, dequant, Qwen3 fp32 forward, golden test#1
bong-water-water-bong merged 1 commit into
mainfrom
cpu-reference

bong-water-water-bong commented Sep 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bong-water-water-bong commented Sep 22, 2026

What

Results (Strix Halo, Qwen3-0.6B BF16 GGUF vs transformers fp32 eager)

Test plan

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant