npu: the OGA 0.5B pack loads and answers in the engine, but the first greedy token is wrong (flat head over a large-magnitude stream)

The container side of #671 now works end to end: 1bit npu-pack import on Qwen2.5-0.5B-Instruct_rai_1.8.0_npu_4K writes a complete container (291 tensors, 67,088 q4_1 tiles, no refusals — the MLP and final-norm PRs #842/#844), and with #854 the forward derives the geometry its kernels were actually built at, so 1bit forward-serve loads it and answers. What it answers is wrong, and this issue is that, separated from the geometry fix so the two can be judged on their own.

What is observed

Host-GEMM (no device, NPU_INFER_HOST_GEMM=1 NPU_NO_DEVICE_LOCK=1), greedy, on the imported container:

prompt	our output	AMD's stack on the same pack (`~/ogaload/window/oracle.json`, captured 2026-10-31)
`What is the capital of France?` (chat)	`!!!!!!!!!!!!`	`The capital of France is Paris.`
`The capital of France is` (raw completion)	`!!!!!!!!!!!!`	—
`Hello` at `temperature 1.5`	random junk tokens	—

So it is not the chat template (the raw completion fails the same way), and at temperature 0 the pick is token id 0 (!) repeatedly, which is what an argmax over an (almost) flat or all-NaN vector returns.

The forward's own per-layer dump (NPU_INFER_DUMP_LAYERS=1, prompt Hi) says the stack is not zero and not NaN, but it does look inflated:

[LB] m=0 l=22 7.324183 94.5367      (mean|x|, absmax of the residual stream)
[LB] m=0 l=23 7.493761 113.0612
[TB] M=1 start=0 top: 11=10.4926 220=9.1140 13=9.1012 102927=8.9325 14397=8.9212
[mla] pre-norm x: mean|x|=7.282105 absmax=115.964088
[mla] final xn:   mean|x|=5.160123 absmax=36.907848 (n=896)

Top-1 is , (id 11) with 10.49 and the top five sit within 1.6 of each other — a very flat head over a large-magnitude stream. For this model's prompt the first token should be Paris by a wide margin.

What is ruled out (so nobody re-runs it)

The weights are the pack's own bytes. The importer's self-checks re-encode what it parsed: 768/768 attention tiles and 2160/2160 MLP groups come back byte-identical, so the tile decode/layout is not the suspect.
The tile → projection order is measured, not assumed: layer 0's tiles 0,1 carry k_proj.bias[0:128] byte-identically and tiles 2..15 carry q_proj.bias[0:896] byte-identically, which also fixes the row order inside each projection (FACTS §7.16). v_proj is fixed by elimination and matches the checkpoint only numerically — recorded there as an open observation.
The embedding and head are fine: the pack's fp16 embedding is corr 1.0000 / rel 7e-4 against the upstream checkpoint, and the tied head reads that table.
The upstream checkpoint is not a valid reference for the projections — measured on a byte-exact decode (npu-re FACTS §7.15): per-row cosine against the checkpoint sits at the best-of-896 random level (~0.15), no layer/window offset, no Walsh–Hadamard rotation, and neither Gram matrix is preserved. So the only reference for this pack's layer math is AMD's stack running the pack, not Qwen/Qwen2.5-0.5B-Instruct.
The container's format bound is not it: bf16 parameter rounding is 3.34e-3 relative (48.7 % of 32-column groups exact), three orders of magnitude away from what would produce a flat head.
The generic path is not it: the same build serves ~/models/npu/Qwen3-0.6B with sane text ( Paris. The capital of France is also the capital of the country of Belgium.), and that model's geometry is already on both grids, so npu: derive GEMM K and N on the container's and the kernel's grids, so a pack whose hidden is not a multiple of 256 has kernels to find (closes #845) #854 is the identity for it.

What is left, in the order I would take it

The oracle, one short ticket. Re-capture AMD's greedy tokens on the same pack for the same three prompts (~/ogaload/ogaload_dev_run.sh, npu-hold -l ogaload -t 2, 4–7 s inside a ticket) and diff token by token. Divergence at token 1 (which ours already shows) means the very first forward pass differs, so the comparison to make is on layer 0's output, not on text.
Layer 0 against a host reference. One layer of Qwen2.5-0.5B in numpy from the container's own dequantised weights (embedding → rmsnorm → q/k/v (+bias) → RoPE → GQA → o → rmsnorm → gate/up → silu·mul → down) compared against the forward's per-layer dump. That localises the fault to a projection, the bias handling, or the attention math without any device. The exportable dump points already exist (NPU_INFER_DUMP_LATTN/_LAYERS, _LN under ONEBIT_NPU_DEBUG_DUMPS); the missing piece is a report of the intermediate vectors at a chosen layer rather than a running mean.
Only then the golden gate (npu: load Ryzen AI OGA NPU model packs on Linux through the engine (pack import -> Q4NX -> our kernels) #671 item 5): greedy agreement, KLD over 12 chunks, and the speed rows per the ort_run model_benchmark protocol.

What this does not affect

#854 stands on its own: the geometry it fixes is what made forward init failed: dequant failed impossible to get past, and the on-grid path is unchanged (same tests, same text on Qwen3-0.6B). This issue is the numerical gate that comes after it.

Correction to a typo in item 2: the existing dump points are NPU_INFER_DUMP_ATTN, NPU_INFER_DUMP_LN and NPU_INFER_DUMP_LAYERS (the last two under ONEBIT_NPU_DEBUG_DUMPS); NPU_INFER_DUMP_LATTN was a typo. Also filed against the fix that got us here: #854 (geometry) and #842/#844 (the MLP + final norm the container needs).

Triaged: owner pi-ogaload, bucket A. Two additions from the tree, one of which is a precondition the report states
in the present tense and cannot hold yet.

1. At `main` the container that reproduces this cannot be produced

The issue's own chain is two merges short: "the MLP and final-norm PRs #842/#844" are open, and #854 is open, so
1bit npu-pack import at 906f966a2/40d96325 still refuses exactly the tensors that make the container whole:

npu/pack_import.cpp:1158   "the final norm is refused in the plan: the 1,856-byte tail is not an RMSNorm weight"
npu/pack_import.cpp:798    "...while the MLP section ..."          (the re-encode check that #842 relaxes)

So this row is reproducible only on the #842 + #844 + #854 stack, and a lane that starts here at main will get a
refusal, not a wrong token. That is a board-ordering fact, not a criticism of the report — #845 was filed from the same
stack — but it needs to be on the row or the next reader re-derives it.

2. Your item 2 is the one to run first, not item 1

Item 1 (re-capture AMD's oracle) costs an npu-hold ticket and answers a question already answered: the report shows the
divergence at token 1, so the first forward pass already differs, and a token-level oracle cannot localise it further
than "layer 0 or the head". Item 2 (numpy from the container's own dequantised weights) localises it to a projection, the
bias handling or the attention math, needs no device and no ticket, and works while any lane holds the array. Run
item 2 first and let it name the suspect; then the oracle is only needed to confirm the end-to-end text once there is a
candidate.

The dump surfaces it needs already exist (NPU_INFER_DUMP_LATTN, _LAYERS, _LN under ONEBIT_NPU_DEBUG_DUMPS); what is
missing is reporting an intermediate vector at a chosen layer instead of a running mean, which is the small change that
makes item 2 possible.

3. One number in the report is worth re-reading as a clue, not as noise

mean|x|=7.28, absmax=115.96 on the pre-norm residual of a 896-wide model whose embedding is corr 1.0000 against the
checkpoint. A magnitude-ordered stream with a flat head 1.6 wide is not what a wrong scale looks like (that gives a peaky
or saturated head, not a flat one); it looks like two different vectors being combined — the same shape #849 produces,
where a field is read from the wrong half of the nibble. #849 is on q4_1 zero points in the importer, and the
importer's self-checks re-encode tiles, which validates the round trip and says nothing about whether the zero point used
in the forward's dequant is the right one. Worth one line in item 2: print the dequantised first-row values for one
projection and compare against the container's own stored codes, not only against the checkpoint.

Scope note for the row: this is the numerical gate after #854 and does not block it, and #854's own text on Qwen3-0.6B
is unchanged — recorded so nothing here is read as doubting the geometry fix.

npu forward: pad the activation rows to the kernel's K, so no A read …

Facts update — the defect is not the forward, and the upstream checkpoint cannot be the reference. Both are now measured, with scripts (FACTS §7.20, npu-re 6c0f33b, evidence evidence/oga/container-falsified-2026-11-02/).

A device-free reference forward settles "forward or weights"

~/ogaload/oga5/ref_compare.py reads either the imported container (Q4_1 chunks) or the upstream fp16 checkpoint and runs a plain numpy Qwen2 forward. On the checkpoint it reproduces AMD's own oracle answers verbatim ("The capital of France is Paris."), so the forward's semantics have a witness that needs no AMD wheel and no device; read straight from the shipped tiles it also gives lm_head[:512] vs the tied embed_tokens[:512] at corr 1.0000.

On the container's own weights that forward fails exactly like the engine does:

	container	checkpoint
greedy, "What is the capital of France?"	`'forth Raqqa盂00 help suckeraping Raqqa沫…'`	`'The capital of France is Paris.'`
top-5 next	`forth 15.79 · décid 14.80 · fick 14.78`	`The 21.53 · Paris 18.73 · As 18.18`
layer-23 residual	`mean	x

Two independent implementations of the same mathematics failing identically on the same weights means this is the container's contents, not q4nx_forward.cpp (RoPE, the two norms, attention, SiLU).

The checkpoint is not a bit-reference for this pack

It is a per-layer transformed quantisation of the model, so no single correction recovers it: input_layernorm ratio (container/checkpoint) median 1.11 … 1.87 per layer with corr 0.74-0.999, post_ln ratio 0.23 … 1.59, weights q 0.29-0.74 / k 0.37-0.65 / v 0.45-0.87 / o 0.76-0.88 / mlp 0.72-0.91. Consequence for the gates in this issue's siblings: greedy/KLD against the checkpoint cannot be met by any importer; only AMD's stack running this pack is a valid reference.

Ruled out, each with a run: one scalar gain on the layer contributions (8 values, all incoherent); all four container/checkpoint norm swaps (only the pure checkpoint is coherent); gate↔up, q↔o+k↔v and k↔v relabellings; the pack's own sin_cache_cos_cache as the RoPE (it is not a rotation — c²+s² = 6.46 at pos 1 — and using it does not help); and the hybrid pack's model.bin as a third representation (none of our known tensors appear in it in bf16/fp16/f32).

Also measured, and it matters for how much the builder-side checks are worth: the decode and prefill attention payloads are byte-identical (md5 c757351f…, 32 × 34,816 B), so unlike the MLP section there is no second physical layout to cross-check attention against.

The next instrument, and it is one device call

onnxruntime_genai 0.14.0-dev here exposes Generator.get_logits(), so AMD's own next-token vector on this pack is capturable — the value-level reference this issue has been missing. ~/ogaload/oga5/oracle_logits.py does exactly that (3 prompts, one step each, float32 logits to ~/ogaload/window/oracle_logits.npz), batched with the #845 array leg in one npu-hold ticket (~/ogaload/oga5/oga_dev_run.sh, queued 17:47 behind the other lanes). With AMD's logits in hand every hypothesis above becomes a scored comparison instead of a text impression — which is what I will do next rather than guess at another layout.

The container's text is still wrong at #854's head; the pad fix in that PR (activation rows zero-padded to the kernel's K) is a separate, real memory-safety bug and changes none of this.

Item 2, the line the triage asked for: the container's q4_1 decode equals the pack's own tile decode, on every attention projection of layer 0. Device-free (~/ogaload/oga6/{container_vs_pack,all_attn_proj}.py), no ticket, no AMD library in the path. This is the #849-shaped hypothesis ("the zero point used in the forward's dequant is not the right one") tested directly instead of inferred.

What was compared

pack side — the .fconst decode the importer's self-checks already rest on (FACTS §7.13/§7.16), w = (code − zp) · scale, tiles → projections by the measured order k 0-1, q 2-15, v 16-17, o 18-31;
container side — the engine's own chunk decoder (npu/forward/src/q4nx_dequant.c:50), w = code · bf16_scale + bf16_zero, 32×256 tiles, container K padded 896→1024 (only the first 896 columns compared).

projection	rows	codes identical	corr	max\|Δ\|	median\|Δ\|
`k_proj`	128	True	+0.999994	6.8e-03	1.2e-04
`q_proj`	896	True	+0.999995	7.8e-03	1.2e-04
`v_proj`	128	True	+0.999995	2.4e-04	1.5e-05
`o_proj`	896	True	+0.999995	7.3e-04	0.0e+00

The residuals are the container's documented bf16-parameter bound (the q4_1 chunk holds scale and −zp·scale as bf16; 3.34e-3 relative, 52.9 % of groups exact), not a convention error. Two alternatives were run as controls on the same data and both are rejected: zero point dropped → corr 0.366; codes read off by +8 → corr 0.193.

What that settles, and what it leaves

Settled: the importer's value path and the forward's dequant formula agree with the pack's own bytes for the attention projections. So npu: the OGA 0.5B pack loads and answers in the engine, but the first greedy token is wrong (flat head over a large-magnitude stream) #855 is not a q4_1 zero-point/convention defect, and "the values in the container are not the pack's values" is now false as a hypothesis rather than untested.
Left, and it is where I would look next: the pack's contract, not its bytes. config.json's quantization_config is AWQ (name: awq, pack_method: reorder) with the four scaling_layers groups — q/k/v after input_layernorm, o_proj after v_proj, gate/up after post_attention_layernorm, down_proj after up_proj. The two norm-preceded groups can fold the inverse smoothing into the norm gamma (which the pack's norms show: container/checkpoint ratio 1.11–1.87 per layer), and the two linear-preceded groups (o_proj ← v_proj, down_proj ← up_proj) cannot fold it into a norm — it has to be applied to the previous tensor's output channels at runtime, or be folded into that previous matrix's rows. If the pack expects the runtime to apply it and we do not, every layer's attention-output and MLP-output contribution is scaled per-channel by s, which is the shape of the observed failure (residual magnitudes inflate ~8.5× by layer 23 while the head stays flat) and is invisible to every byte-level check in this lane.
A one-line way to test it next, device-free: fit the per-input-channel ratio t[k] between the container's o_proj and the checkpoint's, and check whether t is smooth/structured (an AWQ scale) or noise. If it is structured, the missing compensation is identified — and the pack's own per-op const list (cache/*.meta, restored on 2026-11-02) is where the vector should be named, since FLATMHA/FlatMLP are the ops that would carry it.

Not claimed: no fix, no engine change, no device time. #854's own text on Qwen3-0.6B is untouched, and the array leg (kernel instance naming, main:seq) is being re-run separately — that is a plumbing result and does not speak to this.

Addendum — the next line in my comment above was run, and it rejects the hypothesis it named. Device-free (~/ogaload/oga6/{awq_t_fit,proj_vs_ckpt}.py). Correction posted rather than left standing, because the shop-floor version of this ("o_proj/down_proj need a runtime AWQ compensation") is now measured wrong.

Container vs the upstream checkpoint, layer 0, all seven projections, per-element correlation (and the transposed read as a control):

projection	shape	corr	corr(Wᵀ)	rms container	rms checkpoint
`k_proj`	128×896	+0.054	−0.001	0.1200	0.0806
`v_proj`	128×896	+0.076	+0.004	0.0128	0.0115
`q_proj`	896×896	+0.011	+0.000	0.1600	0.0667
`o_proj`	896×896	+0.030	+0.002	0.0129	0.0131
`mlp.gate_proj`	4864×896	+0.008	+0.000	0.0159	0.0230
`mlp.up_proj`	4864×896	+0.007	+0.000	0.0137	0.0183
`mlp.down_proj`	896×4864	+0.848	+0.000	0.0181	0.0182

Two things follow.

A per-input-channel scale does not repair the attention projections. Fitting t[k] = ⟨W_c[:,k], W_up[:,k]⟩ / ⟨W_up[:,k], W_up[:,k]⟩ (exactly an AWQ smoothing, if that were the relation) moves q_proj from +0.011 to +0.046 and o_proj from +0.030 to +0.038 — and t comes out non-smooth and sign-changing (q_proj range −82…113, median |Δt| 13.7), i.e. noise. On down_proj, by contrast, t is smooth and centred on 1 (0.85…1.38, median |Δt| 0.045) and the fit changes nothing (+0.848 → +0.845), as it should for a matrix that already is the checkpoint's.
No transpose anywhere (corr(Wᵀ) ≤ 0.004 on every tensor), so the container's and the pack's [out, in] orientation is right — worth pinning because a transposed write would have passed every self-check in this lane.

What is left, narrowed

down_proj — the one projection whose tile lives in the MLP section, decoded from the 2,176-byte subvolume with 19 K-blocks — matches the checkpoint at 0.85. The six projections that live in the attention payload (32 × 34,816-byte tiles, whose i0 = 2 + 2·kblock + column-half interleave is what my decode assigns k-blocks with) match at ≈0. My encode/decode pair is a fixed point of that assignment, so it can round-trip byte-exactly while still naming the wrong (k-block, column-half) slot — the self-check cannot see it, and neither can the !-level dumps.

So the suspect I would test next is the subvolume interleave inside the attention tile, not the quantiser and not a smoothing: re-run the known-answer writer (bf16_const_params_shuffle, the instrument that settled §7.13) with weights whose value encodes which k-block and which column-half the writer puts them in — the current ramp encodes (row, k) only, which is exactly the information this failure needs and does not have. If the writer's i0 assignment is a transposition of mine, the six uncorrelated projections become the checkpoint's and #855 has a cause.

Not claimed: no fix, nothing device-side, and #854's Qwen3-0.6B result is untouched.

Correction to my own addendum, 15 minutes later: the k-order suspect is falsified by ground truth, so the tile decode is now pinned end to end. Device-free (~/ogaload/oga6/head_korder.py).

The suspect I named was "my encode/decode pair is a fixed point of the subvolume assignment, so it can round-trip while naming the wrong (k-block, column-half) slot". Two things killed it:

The 11-01 known-answer write pins less than I claimed, but pins the k-block. Its input was
w(k,n) = ((k·3 + n·7) % 16) − 8, whose code pattern has period 16 in k — I verified that the pattern is
invariant under k-shifts of 16, 32, 48, … So "4096/4096 codes" cannot see any reordering with stride ≥ 16 inside a
128-k block. What it does pin is the k-block↔subvolume assignment, through the scales (distinct within every
32-column slot). So "k-block and column-half" was already proven; only the intra-block k order was open.
That open slot is closed against real ground truth, not against my own inverse. The pack ships both
representations of the head: model.onnx.data's lm_head MatMulNBits codes (whose decode is verified at corr
1.0000 against the tied embedding) and the tile at tail+1,856, which is byte-identical to the vendor packer's
output on exactly those arrays (FACTS §7.15). Decoding the shipped tile with the attention-tile convention and
comparing codes against the container's head (which is the ONNX array, untouched by any decoder of mine):
```
 codes identical: True     57,344 / 57,344  (100.00%)
 columns at 100 %: 896 / 896      rows at 100 %: 64 / 64
 best ref column for decoded column 0..31: 0,1,2,…,31   (identity, no permutation)
```
So the intra-block k order, the row order and the column order in my decode are the ONNX arrays' order, and
the hypothesis is dead.

Where that leaves #855 (I am not going to keep proposing mechanisms that the next measurement kills)

Combining the three measurements now on the issue: the container's values are the pack's values (corr 0.999995, codes
identical, per projection), the decode's order is the ONNX order (100 % against an independent ground truth), and the
pack's attention projections are still not the checkpoint's under any per-channel scaling (corr ≈ 0.01-0.03, t fit
rejected). Those three cannot all be "the weights are wrong", so the remaining gap is what the pack's runtime does
with those weights that a plain Qwen2 forward does not — and I do not have an instrument that reads AMD's intermediate
activations, only its final text.

So I am stopping the hypothesis chain here rather than spending another round on it, and leaving the row with what it
now knows:

the weights and their order are right (three independent checks, one of them external);
the checkpoint cannot be the numeric reference for this pack's projections (§7.20 — unchanged);
gate (a) (greedy text vs AMD's stack) is live and cheap: AMD's stack answers The capital of France / The first three prime / The sky is blue reproducibly in a 2-4 s run; our container answers garbage;
what would move it forward is one of: a per-op const table from the pack's own .meta/.ctrlpkt (now on disk since
the 18:40 pack repair) that names an activation-side scale the DD program applies, or an AMD intermediate-activation
dump, which no public API on this box exposes (both logits accessors were checked and return a row that is neither
the sampling row nor a prompt position).

That is a scoping statement, not a fix — recorded so the next reader does not re-run the four hypotheses this lane has
now measured away.

Host-side re-measurement (no device, no npu-hold ticket) that withdraws the "the pack's projections are not the checkpoint's" line in my earlier comments, and narrows #855 to one tensor family.

Scale-invariant, per 32-column-group correlation, container vs Qwen/Qwen2.5-0.5B-Instruct: layers 1-23 score 96.9-100 % of groups above 0.9 with median +0.97..+0.99 for all seven projections (q, k, v, o, gate, up, down). Layer 0 is the only outlier (q 43.5 %, k 55.2 %, both still median +0.89/+0.91). The earlier +0.011 / +0.030 figures were element-wise over whole tensors, which a per-group scale destroys; the codes, row order, column order and group boundaries are right.

The residual is a per-(row, 32-group) affine that multiplies the whole group (b/Z ≈ a/S to 4-5 digits), and it is not our importer: the pack's own attention-tile scale field and the container's are 896/896 values identical.

Non-circular packer test (the 10-31 round trip decoded the pack and re-packed it, i.e. a fixed point of my own inverse): quantising the checkpoint in ONNX form with (max-min)/15 and running matmulnbits_pack_const_float32 on it reproduces 0 % of the shipped tile (codes 2-18 %, zeros ~78 %, scale field 28-45 %). Median step ratio checkpoint/container, layer 1: q 0.765, k 0.786, v 1.160 — tight bands, which is the shape of a clipping quantiser, not different weights.

Ruled out: one scalar gain per tensor (fitting it makes layer 0 sane, mean|x| 11.18 -> 0.171, and blows layer 11 up to 7.08; per-layer gains range 0.22-19.19). And a norm pairing fault: the 48x48 norm matrix is 37/48 on the diagonal, the rest explained by Qwen2.5's norms correlating >0.9 across neighbouring layers.

What is left, stated as one thing: the layer weights are the checkpoint's re-quantised, and the only part of the container that is not explainable that way is the 48 layer norms — container/checkpoint gamma ratio 1.87 (layer 0 input_layernorm) down to 0.23 (layer 2 post_attention_layernorm, spread 2.06), while the final norm is identical to the checkpoint (1.0000) and both AMD packs agree on the layer norms byte-for-byte. Next test is host-only: whether the pack's gammas are the checkpoint's times a per-layer, per-channel vector that also appears in the following projection's columns (a folded RMSNorm-with-gamma), scored against the step ratios a fold must reproduce (q 0.765, k 0.786, v 1.160).

FACTS §7.23 in npu-re (4f4d472, branch ogaload/oga-pack-layout-20261031) with eight device-free scripts under evidence/oga/container-vs-checkpoint-2026-11-02/.

npu forward: pad the activation rows to the kernel's K, so no A read …

npu: derive GEMM K and N on the container's and the kernel's grids, s…

Root cause found and fixed — and it is two defects, both now measured. Host-only for the diagnosis; no device, no ticket.

1. The container was wrong: the importer's zero-point nibbles (#849)

decode_tile() paired the q4_1 zero-point nibbles by halves (i / i+16) while the codes of the same subvolume pair 2i / 2i+1. Fixed in #879 (correlation of zp against the code cluster on the pack's own tile: −0.0227 → +0.9977). With that fix a numpy forward over the container's own weights reproduces AMD's output exactly:

	before #879	after #879
greedy, "What is the capital of France?"	`forth Raqqa盂00 help suckeraping…`	`The capital of France is Paris.`
layer-23 residual	`mean	x

So the earlier conclusion that "the pack's projections are not the checkpoint's" (§7.20) was the symptom: the weights are the checkpoint's inside every 32-column group (median corr +0.99 across all seven projections in layers 1-23 — FACTS §7.23), offset per row by the mis-paired zero point.

2. Then the engine still answered garbage — a second, independent bug: batched prefill row strides

With the container correct, 1bit forward-serve (host-GEMM) still answered 意为： while the reference forward said The capital of France is Paris. Bisected with no device:

1-token prompt → engine and reference agree to 6 digits at every layer (l=0 0.173817 2.3649 vs 0.174 2.4, l=11 identical). So the forward's math is sound per position.
5-token prompt → position 0 matches, positions 1-4 do not. q/k/v (pre-RoPE) and attention()'s output are byte-identical between the batched path and the sequential path at every position; the divergence starts after attention.
Cause: npu: derive GEMM K and N on the container's and the kernel's grids, so a pack whose hidden is not a multiple of 256 has kernels to find (closes #845) #854 pads each design's N to the kernel grid (512 columns), so npu_gemm_rows() writes M rows of the design's N, and the batched prefill read them back at the real width — O[m*H], D[m*H], gv/uv[m*im], ACT[m*2*im]. Row 0 lands right, row m is read m × (padded − real) floats early. On-grid models (Qwen3-0.6B/1.7B/4B, the 27B) are untouched because q4nx_pad_n() is the identity for them, which is why this hid.

Fixed in #885. Result on the default path (no NPU_NO_BATCH), OGA 0.5B container, host-GEMM:

chat "What is the capital of France?" -> "The capital of France is Paris."      <- AMD's oracle string
raw  "The capital of France is"       -> " Paris. The capital of the United States is Washington"

and NPU_BATCH_CHECK=1 (the engine's own batch-vs-sequential oracle) went from seq_argmax=12095 batch_argmax=36589 MISMATCH to agreement.

Why the issue's own ruled-out list could not see either bug

Both are invisible to every byte-level check: the importer's round trip is invariant under a permutation applied on both sides (so 768/768 attention tiles and 2160/2160 MLP groups "passed"), and the forward's per-position math was correct — only the row stride of a batched buffer was wrong, which position 0 never exercises.

Remaining for the gates (item 3 of the issue)

With #879 + #885 (and #842/#844/#854 for the container) the correctness gate is reachable: greedy agreement with AMD's oracle on the three prompts is now demonstrated by hand above; what is not yet done is the automated 12-chunk KLD and the speed rows, and the device leg (npu-hold) has not been re-run since the container became correct. Also open and not mine to close: #842 needs its oga_mlp_gu goldens regenerated (they were generated by the halves-paired decoder), same two lines in its 4,288-byte path.

#855 gate (b) — the 12-chunk KLD rows, with a control. Two parts: the device leg tonight, and then this.

Device leg: refused at load, on purpose (0 faults)

npu-hold -l ogaload -t 3 on the corrected container (~/ogaload/oga8/c849s, 291 tensors, imported by #842+#844+#849) with our own kernels and our own runtime (~/ogaload/oga9/dev_855.sh):

[forward] container K grid: H 896 -> 1024, IM 4864 -> 4864 (pad columns are zero)
[forward] derived 4 designs (fused_gu=1, H=896 NH=14 NKV=2 HD=64 IM=4864)
full ELF ok: uuid=f8891ca2… instance main:seq (1 in this context)
I8Ctx::init_elf: … declares argument 0 (activations) = 245760 B, this launch would pass 131072 B
  -- the kernel's DMA would run past the buffer (M=128 K=1024 N=1536) … refusing before any dispatch.
amdxdna ERRORs during the ticket: 0    IO_PAGE_FAULT during the ticket: 0

Three things that run established:

The 2026-11-02 20:35 IO_PAGE_FAULT class is closed. The launch that produced three AMD-Vi fault bursts then now stops at load with the sizes on both sides and 0 kernel errors, 0 faults. npu forward: refuse a launch whose buffer is narrower than the ELF's declared argument (closes #868) #869's guard is doing its job on the real container.
The remaining blocker is one named item: these forge-built ELFs declare (weights, activations, output) (arg0 = the 245,760-byte per-core weight stream) while I8Ctx drives the dx/lax per-tile family (activations, weights, output). Nothing about the container, the model or the array is in question.
It cost 2 s of device time — the refusal precedes any dispatch.

So gate (a)/(c) on the array stay blocked on the launcher ABI (routes unchanged: teach I8Ctx the gemv launch, or emit the pack's shapes in the family the engine already drives). The engine's greedy text on the corrected container is already AMD's exact oracle string (The capital of France is Paris.), measured on the host-GEMM path.

Gate (b): 12 chunks, container vs the fp16 checkpoint

Reference, stated because it decides what the number means: AMD's own logits are unusable in this build (get_logits() returns a row that is neither the next-token distribution nor any prompt position — 2026-11-02 19:40), and the pack's text now matches the checkpoint token for token, so the checkpoint is the reference. Same forward code over both weight sets; KLD(reference ‖ ours) in nats per position.

chunk	1	2	3	4	5	6	7	8	9	10	11	12	all
mean KLD	0.262	0.315	0.276	0.358	0.244	0.754	0.241	0.296	0.220	0.436	0.281	0.272	0.330
top-1	84.2%	73.7%	77.3%	64.7%	87.5%	47.1%	75.0%	75.0%	82.4%	60.0%	77.8%	56.2%	71.7%

Control (same script, same chunks, pre-#849 container with the mis-paired zero points): mean KLD 12.542 nats, top-1 agreement 0.0%. The metric discriminates by ~38×, so 0.33 nats is the 4-bit quantisation gap, not a residual structural bug — and that is the strongest statement I can make about the importer without AMD's logits.

Script: ~/ogaload/oga9/kl12.py (and dev_855.sh for the device leg), output ~/ogaload/oga9/dev_855.{out,log}.

npu forward: drive the forge q4_1 GEMV ELF family (the pack's own ker…

npu forward: read the batched prefill's GEMM outputs at the kernel's …

Merged: #885 (part 2) and #891 (gemv launcher) are on `main` at `8e8780e5`

Owner authorised the merges this morning. What landed and how it was verified:

PR	content	head reviewed
#885	batched prefill reads each GEMM output at the kernel's N stride — the five sites, plus the split gate/up branch packing `ACT` at the fused stride (the CHANGES item) and the QKV2 split	green at `6f9a9956`, VERDICT: READY twice (`591c51d3`, `6f9a99562`)
#891	the gemv launcher: `I8Ctx` detects the family from the ELF's declared argument widths, uploads the container's own q4_1 tile stream, bf16 activations, f32 readback, per-row fallback	`980e3de2`, VERDICT: READY

#891 was stacked on #885 deliberately and could not be retargeted to main alone: its per-row fallback writes rows at the design's N (out[m*N]), which is exactly the layout #885 taught the five readers to read. On plain main the O reader still indexes O[m*H], so on the pack's O design (N=1024, real 896) every row past 0 would be shifted.

Post-merge verification of `main` itself (not the branches)

Fresh detached worktree off origin/main 8e8780e5, Release + ONEBIT_NPU + ninja + gcc-16: configure rc 0, build rc 0, ctest -R "npu_elf_args|npu_design_grid|npu_forward_loader|npu_pack_import|npu_q4nx" 5/5, tools/copyright.py --check rc 0, and the host-GEMM text on the corrected OGA container is still the exact oracle string, The capital of France is Paris.

What is still open on #855

The device leg only: the NPU runs are blocked on the array, not on this code — npu-serve-keeper launches the frozen owner binary ~/wt/serve-bin/1bit-t2s2-20261102, whose basename does not match the keeper's own pgrep -f "1bit forward-serve.*-p $PORT", so stop_serve() finds nothing and the keeper removes the token while the device stays held. Every lane's ticket fails in 0 s. That is a keeper-side fix (widen the pattern, or give the frozen binary the basename 1bit), reported to pi-bonsai; this lane does not touch :18097 or the keeper.

Until the array frees, the gate rows stand at: (a) greedy text — exact on the host-GEMM path, array not yet measured; (b) 12-chunk KLD vs the fp16 checkpoint — mean 0.3295 nats, top-1 71.7 %, with a structurally-wrong control at 12.54 nats; (c) tok/s — still no valid measurement from us.

Device leg is unblocked and running: the crash was mine, and the next blocker is named

The keeper fix (12:25) means npu-hold tickets are granted again — four taken this session, all alone on /dev/accel/accel0, :18097 untouched, 0 amdxdna errors and 0 IO_PAGE_FAULT in every one. First time this container has run on the array.

1. Found and fixed a heap corruption that #891 introduced — PR #910 (`2baf2324`)

I8Ctx's weight BO is a tile stream on the gemv family, but run_lm_head()'s int8 tiled head still memset KD*ND bytes into whatever BO make_weight_bo() returned: 9,961,472 B into 6,225,920 B on the 0.5B pack's head.

	before	after
first request on the array	`munmap_chunk(): invalid pointer`, process dead	serve answers
ASan build	`heap-buffer-overflow`: WRITE 9,961,472 B at `npu_engine_i8ctx_inc.h:459`, 6144 B before the 38 MB head tile vector	0 reports
head path	none for a non-pb16i container	`lm head M=16 K=1024 N=152064 … LMHEAD_K1024_N152064_m16.elf (q4_1 gemv tiles)`

Two diagnostics were needed to see it, and both are reusable: gdb on the abort gave the free site (run_lm_head), and an -fsanitize=address build of main gave the write site and its size. The non-ASan build dies with no message at all, which is why it looked like a device fault.

2. The remaining blocker, measured rather than guessed: the weight stream order

Text on the array is !!!!. A single-launch known-answer probe (device vs the host dot product of the same freshly assembled layer-0 attn_qkv weights) says why:

[gemv-probe] l=0 attn_qkv K=1024 N=1536 KD=1024 ND=1536
  n=0 dev=-5.635213e+35 host=0.009782292
  n=1 dev=3.072886e+35  host=0.07346858
  n=2 dev=-6.589654e+34 host=-0.01921072
  n=3 dev=nan           host=-0.2516336

The launcher uploads the container's Q4NX tiles unchanged, in row-tile-major / k-chunk-inner order. The forge gemv design expects two other things, both from our own generator:

the tiles repacked Q4NX → DX4m (addons/npu-dx/src/dx4.h: from_q4nx_m) — the codes and the scale/zero tables are re-laid so the matrix unit's A operand reads them directly;
the stream laid out per column, per step, per core (addons/npu-dx/forge/design_gemv.py; bench/forge_bench.cpp replicates it), with these ELFs built as FG_COLS=4 FG_ORDER=kr FG_TPO=1 (~/ogaload/oga_build_elfs.sh) — i.e. TPC = N/32/16 row tiles per core, stream order for c { for step { for r { tile } } }.

So gate (a) needs a launcher-side repack plus that permutation. It is a bounded, testable piece of work (the bench is the reference, and the transform is a pure function of one 5120-byte tile), but it is not this session's leftover hour.

3. ## OWNER — one licence question before I write it

dx4.h and the forge design sources carry "Proprietary and confidential. Distributed only as compiled binaries under the 1bit Binary License", while the engine is public Apache-2.0. The DX4m repack is a documented format (docs/npu-dx.md describes DX4/DX4m), but it is currently implemented only in the private tree. Three ways forward, and this is your call, not mine:

the engine implements the DX4m layout from the docs, as clean-room format code (public, Apache-2.0 header);
the repack stays private and the engine links a small private library for it (keeps the engine free of the format, costs a build dependency);
the forge emits the stream host-side and the engine consumes it opaquely (a private pre-tiling step, no format knowledge in the engine).

Nothing else on #855 is blocked: importer (#879), forward (#885) and the launcher ABI (#891) are merged, and #910 fixes what the launcher broke in the head.

Closed by the merge of #885 (8e8780e5e518a97715b3663adf1e8c6aa656118c), on top of #891 (8c5f1d85 launcher ABI) and #879 (f3d9f6c0, the container's zero points) — all three are in 65460184. Verified here: the batched prefill reads at the design's N — npu/forward/src/q4nx_forward.cpp:3468 (const int nrow = q4nx_pad_n(H)), :3498 (act_w = q4nx_pad_n(2 * im)), :3559 — and the PR's measured acceptance is met (greedy Paris. The capital of the United States is Washington on the batched path, matching the sequential path and AMD's own oracle string).

Note for the record: GitHub closed this on the sentence "this fixes #855 end to end" inside #885, not on a closes line — same keyword parser that falsely closed #674, but here the claim is true, so the close stands.

One residual is not covered by this close: #910 (open) is the same forward's head path — the gemv family running the head as one launch, and the int8 tiled head refusing it. That is a further fix, not a reopen of this bug; it stays a row via its PR.

npu forward: the gemv family runs the head as ONE launch, and the int…

Navigation Menu

Repository navigation

npu: the OGA 0.5B pack loads and answers in the engine, but the first greedy token is wrong (flat head over a large-magnitude stream) #855

What is observed

What is ruled out (so nobody re-runs it)

What is left, in the order I would take it

What this does not affect

1. At `main` the container that reproduces this cannot be produced

2. Your item 2 is the one to run first, not item 1

3. One number in the report is worth re-reading as a clue, not as noise

A device-free reference forward settles "forward or weights"

The checkpoint is not a bit-reference for this pack

The next instrument, and it is one device call

What was compared

What that settles, and what it leaves

What is left, narrowed

Where that leaves #855 (I am not going to keep proposing mechanisms that the next measurement kills)

1. The container was wrong: the importer's zero-point nibbles (#849)

2. Then the engine still answered garbage — a second, independent bug: batched prefill row strides

Why the issue's own ruled-out list could not see either bug

Remaining for the gates (item 3 of the issue)

Device leg: refused at load, on purpose (0 faults)

Gate (b): 12 chunks, container vs the fp16 checkpoint

Merged: #885 (part 2) and #891 (gemv launcher) are on `main` at `8e8780e5`

Post-merge verification of `main` itself (not the branches)

What is still open on #855

Device leg is unblocked and running: the crash was mine, and the next blocker is named

1. Found and fixed a heap corruption that #891 introduced — PR #910 (`2baf2324`)

2. The remaining blocker, measured rather than guessed: the weight stream order

3. ## OWNER — one licence question before I write it

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

npu: the OGA 0.5B pack loads and answers in the engine, but the first greedy token is wrong (flat head over a large-magnitude stream) #855

Description

What is observed

What is ruled out (so nobody re-runs it)

What is left, in the order I would take it

What this does not affect

Activity

bong-water-water-bong commented on Nov 2, 2026

bong-water-water-bong commented on Nov 2, 2026

1. At main the container that reproduces this cannot be produced

2. Your item 2 is the one to run first, not item 1

3. One number in the report is worth re-reading as a clue, not as noise

bong-water-water-bong commented on Nov 2, 2026

A device-free reference forward settles "forward or weights"

The checkpoint is not a bit-reference for this pack

The next instrument, and it is one device call

bong-water-water-bong commented on Nov 2, 2026

What was compared

What that settles, and what it leaves

bong-water-water-bong commented on Nov 2, 2026

What is left, narrowed

bong-water-water-bong commented on Nov 2, 2026

Where that leaves #855 (I am not going to keep proposing mechanisms that the next measurement kills)

bong-water-water-bong commented on Nov 3, 2026

bong-water-water-bong commented on Nov 3, 2026

1. The container was wrong: the importer's zero-point nibbles (#849)

2. Then the engine still answered garbage — a second, independent bug: batched prefill row strides

Why the issue's own ruled-out list could not see either bug

Remaining for the gates (item 3 of the issue)

bong-water-water-bong commented on Nov 3, 2026

Device leg: refused at load, on purpose (0 faults)

Gate (b): 12 chunks, container vs the fp16 checkpoint

bong-water-water-bong commented on Nov 3, 2026

Merged: #885 (part 2) and #891 (gemv launcher) are on `main` at `8e8780e5`

Post-merge verification of main itself (not the branches)

What is still open on #855

bong-water-water-bong commented on Nov 3, 2026

Device leg is unblocked and running: the crash was mine, and the next blocker is named

1. Found and fixed a heap corruption that #891 introduced — PR #910 (2baf2324)

2. The remaining blocker, measured rather than guessed: the weight stream order

3. ## OWNER — one licence question before I write it

bong-water-water-bong commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. At `main` the container that reproduces this cannot be produced

Post-merge verification of `main` itself (not the branches)

1. Found and fixed a heap corruption that #891 introduced — PR #910 (`2baf2324`)