[bong-water-water-bong]

Stacked on #661 (base = npu/bonsai-decode-speed): the arena's harvest #2, integrated. Decode
8.87 → 4.10–4.29 s/token on the H2 container, bit-exact.

The mechanism

The device streams 25.60 GB of int8 weights per decode token. In the layout every container has
today, each 8 KiB B tile is delivered by a 4-level tap sizes=[8,16,8,8] strides=[8N,8,N,1] — 64
runs of 128 contiguous bytes with a row stride of N. Every parameter inside that layout was swept
in the previous pass (tile size, run length, columns, core rows, fifo depth, issue/await
discipline): 2.3–3.1 GB/s, ceiling. Pre-blocking the same bytes into the microkernel's own tile
order ([kb][nb][8][8], tile-n-major) turns each tile into one contiguous 8 KiB burst
(strides=[1024,64,8,1]) and the same phase runs at 8.13 GB/s.

Arena per-phase measurements (byte-exact on 3 seeds, ~/wt/npu-arena/LEADERBOARD.md):

phase	stock	blocked B	×
GU	3.32 GB/s	8.13	2.45
D	3.19	8.25	2.59
QKV	3.35	8.15	2.43
O	3.19	8.15	2.55
Z	3.44	8.36	2.43

What the diff does

• q4nx_weight_layout in the model dir's config.json: absent/"rm" = row-major (every
  earlier container), "b8" = blocked. tools/gguf_to_q4nx.py --weight-layout rm|b8 writes it, so
  a container and the kernel family it needs are decided in one place.
• I8Ctx::bpos(i,j) maps element (i,j) of the padded [KD][ND] image into the blocked buffer;
  packB_blocked() writes through it, so packB, packB_into and the resident lm_head tiles all
  follow. Quantizer, per-column scales and every int8 code are unchanged — the buffer is a
  permutation, so the int32 output is bit-identical. Cross-checked against the arena's
  device-proven ordering over all 32,768 positions of a 128×256 image: 0 mismatches, bijective.
• _b8 artifact naming, because the tap lives in the ELF: a b8 container looks for
  full_i8_<ROLE>_K<K>_N<N>_b8.elf and twins ..._b8_m16.elf; the fused-GU and batched-MoE
  existence probes use the same tag; a missing artifact is a loud "missing full ELF" instead of a
  silently permuted read; init_elf() refuses a b8 pack on a geometry that is not a whole number of
  64×128 tiles.
• No microkernel change, no new design, no new cores. The 14 ELFs (7 roles × {M=128 prefill,
  M=16 twin}) come from the same generator with the contiguous B tap
  (~/wt/bonsai-speed/emit-b8-elfs.sh, gen-b8/n1_blocked.py) — the arena proved the microkernel
  needs no edit.

Numbers on the H2 container (d1/b8.log)

[ph] wall=4098-4286 | layers=3922-4110 (wait 3163-3316, host 537-588, ql 184-201, n=368)
     | head=176-205 (wait 156-189)

	before	after
decode step	8.87 s	4.10–4.29 s
1-token request	9.22 s	4.46 s
4-token request	36.1 s	16.8 s
device wait for 25.60 GB	8.01 s (3.2 GB/s)	3.32 s (7.7 GB/s)
host + quantise	0.85 s	0.80 s

Gates

• NPU_M16_CHECK=1 over the b8 family: checks=7295 mismatched=0 worst int32 maxdiff=0 —
  twin vs M=128 on the same quantized activations and the same weight BO, both families on the
  blocked layout (d1/b8gate.log).
• Greedy text identical to the row-major baseline on 7 prompts and both timing requests
  (d1/b8.out vs d1/after.out): ' Paris', ' Paris.\nThe', ' Paris.',
  '<|im_end|><think>', ' Au.', '\n\n<think>', ' 4', ' George Washington'.
• PPL is unchanged by construction (permutation of the same codes; the gate compares raw int32).

Not in this PR

The remaining 0.80 s/token of host + quantise (0.55 layer host, 0.20 quantise/launch) is now 19%
of the step and is the next lever — the device side alone is 3.32 s, so ≤3.5 s/token needs that host
work down, not another kernel change. Container-payload blocking in the converter (as opposed to the
declared layout + load-time pack) would only save load time, not decode.

The ELF set this needs (_b8) — and why it is not in this PR

The B tap lives inside the ELF, so a container that declares q4nx_weight_layout: "b8" needs a
kernel family built with the contiguous tap. The engine derives the names from the declaration, so
a mismatch cannot be silent: a b8 container asks for ..._b8.elf / ..._b8_m16.elf and a missing
artifact fails with missing full ELF instead of reading permuted weights.

For the H2 Bonsai-2-27B container (~/models/npu/bonsai2-27b-NPU, H=5120, IM=17408, 64 layers,
48 GDN + 16 full-attn) that is 14 ELFs — 7 role geometries × 2 families:

role	K × N	prefill (M=128)	decode twin (M=16)
QKV (full-attn layers)	5120 × 14336	full_i8_QKV_K5120_N14336_b8.elf	..._b8_m16.elf
QKV (GDN layers)	5120 × 10240	full_i8_QKV_K5120_N10240_b8.elf	..._b8_m16.elf
O	6144 × 5120	full_i8_O_K6144_N5120_b8.elf	..._b8_m16.elf
G	5120 × 17408	full_i8_G_K5120_N17408_b8.elf	..._b8_m16.elf
U	5120 × 17408	full_i8_U_K5120_N17408_b8.elf	..._b8_m16.elf
D	17408 × 5120	full_i8_D_K17408_N5120_b8.elf	..._b8_m16.elf
Z (GDN gate)	5120 × 6144	full_i8_Z_K5120_N6144_b8.elf	..._b8_m16.elf

These ELFs stay private — they are not added to the repository (same rule as the rest of the
NPU kernel artifacts: the repo ships the generator and the emitter, not the binaries). They are
rebuilt from source with the pinned toolchain by:

~/wt/bonsai-speed/emit-b8-elfs.sh <out_dir>      # generator gen-b8/n1_blocked.py,
                                                 # microkernel mm_kernel_reference.cc (unchanged)

and dropped into the model dir's kernels directory (<model>/kernels/, or <model>/kernels-m16/
for the twins — the loader searches both). The private Bonsai container directory used for the
numbers above is ~/models/npu/bonsai2-27b-NPU-b8/ (symlinks to the H2 model.q4nx + tokenizer, a
config.json carrying q4nx_weight_layout: "b8", kernels -> b8-kernels/). Because the layout is
declared in the container's own config.json, the same model directory cannot be served with the
wrong family by accident.

[bong-water-water-bong]

Re-stacked on the rebased #661 head (): this branch is now , cherry-picked rather than range-rebased so it carries only the layout change (files=4 on GitHub). Builds clean. Its 4.10-4.29 s/token and the 7,295-launch b8 gate were measured before the rebase; they are unaffected by it (the rebase touched only the FFN-cache ownership in #661), and I will re-confirm on the device after the box comes back.

[bong-water-water-bong]

Re-stacked on the rebased #661 head (9551326). My comment above got its backticks eaten by the shell — the correct text:

• this branch is now 7251733, cherry-picked onto the rebased head rather than range-rebased, so it carries only the layout change (GitHub shows files=4), base = npu/bonsai-decode-speed.
• Builds clean with the pinned toolchain.
• Its numbers (4.10-4.29 s/token, and the 7,295-launch NPU_M16_CHECK gate with mismatched=0) were measured before the rebase and are unaffected by it: the rebase changed only who owns the FFN-cache code in npu: Bonsai-2-27B decode 11.5 -> 8.87 s/token (M=16 twins, no per-token re-dequant) #661. I will re-confirm both on the device in the next window and attach the log here.