bong-water-water-bong · Oct 31, 2026 · Oct 31, 2026 · Oct 31, 2026 · Oct 31, 2026
diff --git a/NOTICE b/NOTICE
@@ -175,11 +175,14 @@ Linked at build time against a system package (not redistributed)
 ONNX Runtime (https://github.com/microsoft/onnxruntime)
   Copyright (c) Microsoft Corporation. All rights reserved.
   License: MIT. Built only with -DONEBIT_ORT=ON (docs/onnx.md): the engine then
-  links the distribution's libonnxruntime 1.23.2 (Ubuntu package
-  libonnxruntime1.23/libonnxruntime-dev 1.23.2+dfsg-6ubuntu1, served OrtApi v23;
-  sha256 of libonnxruntime.so.1.23.2 is recorded in onnx/README.md) and ships
-  libonebit_ep.so, our plugin execution provider, whose only ORT constants are
-  the ones its public headers declare. ONNX Runtime is not modified, and no
+  links libonnxruntime 1.23.2, either the distribution's (Ubuntu package
+  libonnxruntime1.23/libonnxruntime-dev 1.23.2+dfsg-6ubuntu1, served OrtApi v23)
+  or the one we build from upstream with onnx/build_ort.sh (same tag, unmodified
+  apart from one missing #include <cstdint> in core/common/semver.h that GCC 15
+  rejects; MIT, so the notice below covers that build too, and its sources are
+  the published tag). It ships libonebit_ep.so, our plugin
+  execution provider, whose only ORT constants are
+  the ones its public headers declare. ONNX Runtime is not modified in behaviour, and no
   proprietary execution provider ships with it: AMD's VitisAI EP wheel is never
   linked or redistributed here.
   MIT license text, as ONNX Runtime ships it:

diff --git a/docs/onnx.md b/docs/onnx.md
@@ -41,6 +41,11 @@ whose layout is fixed by `ORT_API_VERSION`, so a host built for another version
 the wrong offsets instead of failing. `1bit ort-info` prints both numbers, and the version the
 headers compile against is checked against the version the loaded library serves.
 
+Same version, our own build: `onnx/build_ort.sh` clones tag `v1.23.2`, patches the one `#include
+<cstdint>` GCC 15 needs in `core/common/semver.h`, and installs to `~/vendor/onnxruntime-1.23.2`, so
+`pkg-config` finds it by `PKG_CONFIG_PATH` alone — nothing in `onnx/` changes. The MIT notice covers
+that build too (see `NOTICE`).
+
 ## Building it
 
 ```bash

diff --git a/onnx/BENCHMARK-PARITY.md b/onnx/BENCHMARK-PARITY.md
@@ -0,0 +1,201 @@
+<!--
+Copyright 2026 bong-water-water-bong
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Benchmark parity with AMD's Ryzen AI numbers
+
+AMD's own numbers are the ones to beat, and they are Windows numbers taken with
+`model_benchmark.exe` from the Ryzen AI SDK. We cannot run that tool: it needs Windows 11, and the
+Linux equivalent needs AMD's `libonnxruntime_providers_ryzenai.so`, which ships only inside the
+licence-gated install (our own box has `flm` and XRT but not that EP). So we reproduce the
+**protocol** exactly and publish rows in the same units, each labelled with the device, precision
+and engine it came from.
+
+## The protocol we are matching
+
+Source: <https://ryzenai.docs.amd.com/en/1.3/npu_oga.html> ("Run Benchmark"), Linux equivalent at
+<https://ryzenai.docs.amd.com/en/latest/llm_linux.html>.
+
+```
+.\libs\model_benchmark.exe -i <model_path> -p <model_path>\prompts.txt -l "2048,1024,512,256,128"
+```
+
+| their flag | meaning | their default | ours |
+|---|---|---|---|
+| `-i` | model directory | — | `--model <file.onnx>` |
+| `-p` | prompts file | — | not used; see "prompt" below |
+| `-l` | comma list of prompt lengths | `2048,1024,512,256,128` | `--lens` (same default) |
+| `-g` | tokens to generate | `128` | `--gen` (same default) |
+| `-w` | warmup iterations | `1` | `--warmup` (same default) |
+| `-r` | repeats | `5` | `--repeat` (same default) |
+| `-t` | interval for peak-CPU-utilisation sampling, ms | `250` | not implemented (we report peak RSS only) |
+| `--verbose` | more output | off | not needed |
+
+Stages, and their own definitions, which we copy verbatim:
+
+- **Prompt processing (time to first token)** — prompt tokens divided by the TTFT of one run.
+- **Token generation** — per decoded token, excluding the first (the prefill's own token).
+- **Token sampling** — their sampler's own time; ours is the argmax, so it reports the argmax.
+- **E2E generation** — the whole generation loop, ms.
+- **Peak working set size** — bytes; ours is `VmHWM` from `/proc/self/status`.
+
+Their reference output (Phi-3.5-mini, NPU, int4, 128 prompt / 128 generated) is the only published
+timing they print:
+
+```
+Prompt processing (time to first token): avg (us): 169860  avg (tokens/s): 753.562
+Token generation: avg (us): 20354.1  avg (tokens/s): 49.1301
+E2E generation (entire generation loop): avg (ms): 2755.09
+Peak working set size (bytes): 3543330816
+```
+
+Their model set (11 repos, `-awq-g128-int4-asym-bf16-onnx-ryzen-strix`: Phi-3-mini-4k,
+Phi-3.5-mini, Mistral-7B-v0.3, Qwen1.5-7B-Chat, chatglm3-6b, Llama-2-7b, Llama2-7b-chat,
+Llama-3-8B, Llama-3.1-8B, Llama-3.2-1B, Llama-3.2-3B) publishes no timings on its cards, only
+perplexity — so a per-model comparison table cannot be built from their side. What can be built is
+one protocol, run on hardware we have.
+
+## How we run it
+
+```sh
+ort_run --model <model>-cached.onnx --cached --bench [--lens "2048,1024,512,256,128"] \
+        [--gen 128] [--warmup 1] [--repeat 5] --threads N
+```
+
+- One session, one graph; each repeat restarts from a zero-length cache, which is their repeat loop
+  resetting the generator.
+- Greedy decode (`argmax`), so the token stream is deterministic and reproducibility is checkable.
+- **Prompt**: they read a text file; for timing only the token count matters, so we generate
+  `100 + (i * 7919) % 30000` ids, valid in every model of their set (the smallest vocab, Phi-3, has
+  32064 entries). Prompt text never enters the measurement.
+- Statistics are population stddev over the `-r` repeats (these are all the samples there are).
+
+`--bench` needs a `--cached` graph; on an uncached graph "token generation" would include re-running
+the whole prompt, which is their prefill number wearing a different name.
+
+## Rows we have measured
+
+Qwen3-0.6B, the same weights as the ONNX work in this directory, 128 generated tokens:
+
+| box | device | dtype | engine | prompt 128 | prompt 256 | prompt 512 | prompt 1024 | prompt 2048 |
+|---|---|---|---|---|---|---|---|---|
+| `ryzen` | CPU, 8 threads | fp32 | our ORT 1.23.2 + `ort_run --cached` | 16.73 tok/s | 15.21 | 13.12 | 9.87 | 6.73 |
+| `strixhalo` | CPU, 32 threads | fp32 | same graph, same runner | **23.31** tok/s | 21.21 | 15.04 | 11.93 | 7.90 |
+
+Prompt processing on the same rows (tokens/s):
+
+| box | 128 | 256 | 512 | 1024 | 2048 |
+|---|---|---|---|---|---|
+| `ryzen` | 1102.1 | 1220.0 | 1175.2 | 777.8 | 639.1 |
+| `strixhalo` | 1034.2 | 1295.4 | 1143.5 | 1041.2 | 901.0 |
+
+Token sampling is flat on both (~2510 tok/s on `ryzen`, ~2280 on `strixhalo`) — that is one argmax
+over 151936 logits, and it is not what either box is waiting on. Peak RSS is 8.2 GB on both.
+Raw blocks: `/tmp/bench_qwen3_amd.txt` (ryzen), `/tmp/bench_strixhalo_amd.txt` (strixhalo).
+`ctest` on the strixhalo build: 4/4 passed.
+
+## Round 2: our own cache buffers, and what it did not fix
+
+`run_cached` used to let ONNX Runtime allocate `present` on every step. It now ping-pongs two
+preallocated stacks bound through `OrtIoBinding` (`BindInput "past"`, `BindOutput "present"`,
+`BindOutputToDevice "logits"`), so the only cache traffic is the graph's own `Concat`. Numerics are
+unchanged (ids `13 16 13 16`, mean KL 1.238e-07 against transformers, max|dlogit| 2.12e-05).
+
+`ryzen`, 8 threads, fp32, 128 generated — before → after:
+
+| prompt | decode before | decode after | prefill before | prefill after | peak RSS before | peak RSS after |
+|---|---|---|---|---|---|---|
+| 128 | 16.73 tok/s | 16.70 | 1102.1 | 1051.0 | 8.20 GB | 7.98 GB |
+| 256 | 15.21 | 15.29 | 1220.0 | 1197.6 | | |
+| 512 | 13.12 | 12.66 | 1175.2 | 1072.0 | | |
+| 1024 | 9.87 | 9.89 | 777.8 | 866.3 | | |
+| 2048 | 6.73 | 6.60 | 639.1 | 620.5 | | |
+
+**The allocation-churn hypothesis was wrong.** Everything is inside noise, and the footprint barely
+moved, so the O(n) cost is not `malloc` and not the copy: it is the graph. At a 2048-token prompt the
+prefill alone materialises `logits [1,2048,151936]` (1.2 GB) and the attention intermediates, which is
+where the 8 GB peak comes from. The change is kept because the cache is now a known, bounded quantity
+two stacks wide instead of whatever the arena grew to, but it is not a speed fix and nothing here
+claims it is. The next levers, in order, are graph-side: keep only the last logits row (a
+`logits[:, -1:, :]` slice would delete most of the prefill's lm_head matmul) and profile the 56
+per-layer `Concat`s before touching them.
+
+## NPU row: our engine, the dx route
+
+Qwen3-0.6B through `1bit serve -m <dir> --device npu --npu-opt dx=<kernel dir>`, 64 generated tokens,
+3 repeats after 1 warmup, driven by `onnx/bench_npu.sh`. `strixhalo` (Ryzen AI Max+ 395), int4-weight
+AIE designs, one design per layer:
+
+| prompt tokens | prompt processing | token generation | ms/token | E2E |
+|---|---|---|---|---|
+| 162 | 77.69 tok/s | **72.64 tok/s** | 13.77 | 2.95 s |
+| 332 | 76.02 | 62.56 | 15.98 | 5.24 s |
+| 672 | 71.71 | 56.09 | 17.83 | 9.40 s |
+| 1335 | 63.30 | 47.70 | 20.97 | 22.41 s |
+
+Read against the CPU rows for the same model:
+
+- **Decode: the NPU wins 4–7×** (72.6 tok/s at 162 tokens against 16.7 fp32-CPU; 47.7 at 1335
+  against 6.6) and it degrades far more slowly — 1.5× from 162 to 1335 tokens, where the CPU loses
+  2.5×. That is the KV cache holding, plus kernels that re-read 224 KB/token instead of 2.4 GB.
+- **Prefill: the CPU wins 8–16×** (620–1200 tok/s against 63–78). NPU prompt processing is the weak
+  stage and it is the same gap the A/B harness records against FLM (native ~67 tok/s vs 1269). No row
+  here is a product claim until that closes.
+- The prompt token counts are measured, not requested: the chat template adds its own prefix, so
+  `--lens 1024,512,256,128` lands on 1335/672/332/162. AMD's block prints measured counts too, which
+  is why the column exists.
+- The route reports `prompt_ms` and `predicted_ms`, so `bench_npu_serve.py --mode json` reads those.
+  Its streamed chunks all arrive after the generation finishes, which makes chunk-arrival timing read
+  as 3 µs/token — measured once, discarded, and now the reason `--mode` exists (`stream` is for
+  servers that really stream, like `flm`).
+
+## What this table says
+
+- The numbers are **not** comparable to AMD's published one as a speed claim: theirs is an int4 NPU
+  on Windows, ours is fp32 CPU or our own int4 NPU route on Linux. What is comparable is the
+  definition, so any of our rows drop into their table without translation.
+- Four times the threads buys 1.4× at 128 tokens and 1.2× at 2048: the fp32 CPU graph is
+  bandwidth-bound, streaming a 2.4 GB weight file. 32 threads on `strixhalo` and 8 on `ryzen` land
+  within 20% of each other at long context, which is the tell.
+- The CPU graph's decode degrades with context (16.7 → 6.7 tok/s from 128 to 2048) and round 2 shows
+  it is not the cache buffers. The NPU route shows what the same model looks like when the cache is
+  actually held on device: 72.6 → 47.7 over a comparable range.
+- Prefill is memory-bound on CPU (639 tok/s at 2048 against 1220 at 256, TTFT 3.2 s) and compute-bound
+  on the NPU (63 tok/s, 21 s for the same prompt). Those are two different problems with two
+  different fixes.
+
+## Reproducing
+
+```sh
+# on any box with libonnxruntime and cmake
+cmake -S onnx -B onnx/build -DCMAKE_BUILD_TYPE=Release && cmake --build onnx/build -j
+./onnx/build/ort_run --model ~/wt/onnx-artifacts/qwen3-0.6b-cached.onnx --cached --bench --threads 8
+```
+
+Build the runtime yourself if you want the pin under your control: `onnx/build_ort.sh` (same tag,
+one `#include <cstdint>` patch for GCC 15). Parity against the distro package is checked in
+`onnx/README.md` ("Owning the runtime").
+
+```sh
+# the NPU row, on the box that has the NPU (kernel dir from addons/npu-dx/scripts/build-kernels.sh)
+ONEBIT_DX_KERNELS=~/wt/ross-npu-product/kern/1024x3072x16x8 onnx/bench_npu.sh \
+    --lens 1024,512,256,128 --gen 64 -w 1 -r 3 --mode json
+```
+
+`bench_npu.sh` owns the whole round trip — stage the model dir, start the serve, wait for health,
+drive it, tear it down — and refuses to start if something already answers on its port, because a
+stale server on that port once answered the health check and the bench then timed someone else's
+process. `--mode stream` measures chunk arrival for servers that stream incrementally (`flm`);
+`--mode json` reads the timings our routes report.
diff --git a/onnx/README.md b/onnx/README.md
@@ -38,6 +38,82 @@ sha256sum /usr/lib/x86_64-linux-gnu/libonnxruntime.so.1.23.2
 pkg-config --modversion libonnxruntime
 ```
 
+## Owning the runtime
+
+ORT is MIT, so we build it ourselves and link that: `onnx/build_ort.sh` clones `v1.23.2`, applies
+the one patch GCC 15 needs (`core/common/semver.h` uses `uint32_t` without `<cstdint>`) and installs
+to `~/vendor/onnxruntime-1.23.2`. Nothing in `onnx/` changes — discovery is still `pkg-config`, so
+the switch is two environment variables:
+
+```sh
+export PKG_CONFIG_PATH=$HOME/vendor/onnxruntime-1.23.2/lib/pkgconfig   # find it
+export LD_LIBRARY_PATH=$HOME/vendor/onnxruntime-1.23.2/lib            # run it
+```
+
+That build carries the same three builtin EPs as the distro package — `[0] DnnlExecutionProvider`,
+`[1] XnnpackExecutionProvider`, `[2] CPUExecutionProvider` — because a pin that changes which EPs a
+session can partition to is not a pin. `ort_info` prints the same roster for both.
+
+The numbers do not move: with DNNL and XNNPACK present the cached graph decodes a 256-token prompt at
+62.3 ms/token against 63.7 for the CPU-only build, and the per-step logits against transformers are
+byte-for-byte the same comparison (mean KL 1.238e-07, max|dlogit| 2.122e-05) — i.e. neither EP claims
+this graph's fp32 MatMuls and MLAS still runs them. They are there for parity, not speed.
+
+Two build quirks. ORT pins `DNNL_TAG v3.0.1`, which still declares `cmake_minimum_required` below
+3.5, so the script exports `CMAKE_POLICY_VERSION_MINIMUM=3.5`; and oneDNN installs to its own prefix, so
+`libdnnl.so.3` has to be copied next to `libonnxruntime` or the DNNL provider `.so` loads and the EP is
+simply never registered.
+
+## The cached graph (`--cached`)
+
+The default export re-runs the whole sequence every step, so its cost grows with the answer. With
+`--cached` the graph takes the cache as an input and returns it grown by one position:
+
+| | uncached | `--cached` |
+|---|---|---|
+| inputs | `input_ids`, `attention_mask` | `input_ids` `[1,seq]`, `past` |
+| outputs | `logits` | `logits`, `present` |
+
+`past`/`present` are **one stacked tensor**, not 56 named ones:
+`[2*28, 1, 8, past, 128]` — layer-major, `key` before `value` in each layer pair. `onnx/ort_run.cpp`
+carries that single tensor through the loop; 56 separate names would need a name map and dynShapes
+cannot place a cache axis against a `*kv` argument tree anyway.
+
+There is no `attention_mask` and no `position_ids` input: for one contiguous sequence both follow
+from the cache length, and the traced mask path expands a zero-length `past` axis that ONNX Runtime
+rejects (`Expand … invalid expand shape`). Prefill therefore starts from a cache with `past = 0`.
+
+```sh
+# export, then decode 64 tokens against the cache
+python3 onnx/export_qwen3.py --model-dir ~/models/hf-qwen3-0.6b \
+  --out ~/wt/onnx-artifacts/qwen3-0.6b-cached.onnx --cached
+./onnx/build/ort_run --model ~/wt/onnx-artifacts/qwen3-0.6b-cached.onnx \
+  --ids "15 23 40" -n 64 --threads 8 --cached
+```
+
+```
+  prefill : 3 tokens in 48.3 ms (logits [151936])
+  kv      : 28 layers x 8 heads x 128 (224.0 KB/token)
+  decode  : 4 tokens in 197.5 ms (20.26 tok/s, 49.36 ms/token)
+  ids     : 13 16 13 16
+```
+
+The point of it, measured on a 256-token prompt, `-n 16 --threads 8`, same box:
+
+| | decode ms/token | tok/s |
+|---|---|---|
+| uncached (re-runs 256 positions per step) | 291.22 | 3.43 |
+| `--cached` | 63.66 | 15.71 |
+
+4.6x at 256 tokens, and the gap widens with context because only the cache is re-read instead of the
+whole prefix being recomputed. At a 3-token prompt it is only 1.2x (50.72 vs 60.07 ms/token) — there
+the 2.4 GB weight stream and `lm_head` dominate, not attention. Prefill costs the same either way
+(210 ms for 256 tokens).
+
+Correctness is checked against transformers running the same greedy loop with a real cache:
+`compare_logits.py` over both `--dump-logits` directories agrees to a mean KL of 1.2e-07 nats with
+top-1 matching at every step, so the cached graph is the same function, not just a plausible one.
+
 ## Build and run
 
 ```sh