[bong-water-water-bong]

The CPU path can only get faster by changing precision — round 3 in BENCHMARK-PARITY.md measures
prefill at ~760 GFLOP/s (what fp32 cores do) and decode re-reading 2.4 GB per token. This is the probe
of that candidate, and it lands as evidence rather than a feature, because it does not hold up on
accuracy.

What it does: quantize_cpu.py runs onnxruntime.quantization.quantize_dynamic over the cached
graph — int8 weights, per-channel, MatMulConstBOnly so attention's activation-by-activation matmuls
stay float. The KV cache keeps its fp32 contract, so ort_run --cached runs it unchanged.

Measured (ryzen, Qwen3-0.6B, 8 threads, fp32 graph as reference):

	fp32	int8 dynamic
prefill 512	1141 tok/s	1628 (+43%)
prefill 2048	567 tok/s	860 (+51%)
decode 512	12.98 tok/s	12.28 (−5%)
decode 2048	6.67 tok/s	6.36 (−5%)
peak RSS	6.60 GB	4.68 GB
tokens for 15 23 40	13 16 13 16	13 220 16 24

Mean KL 0.28 nats, max|dlogit| 3.47, top-1 agreement 1/4 — the greedy answer changes at the second
token, so this is not usable for the pack. Three variants measured and worse: quantising attention
matmuls too (KL 1.72), per-tensor scales (KL 0.36), and excluding the tied head explicitly (byte-for-byte
no change — MatMulConstBOnly already leaves it float because its weight arrives through a Transpose).

The error is entirely the uint8 activations, which is what dynamic quantisation does by design. The
accurate route is static QDQ with calibration, or int4 MatMulNBits — which is already what the NPU
side runs. Both are bigger changes than this file, so they stay separate work.

The script docstring carries the whole finding so nobody repeats the three dead ends.

[bong-water-water-bong]

VERDICT: CHANGES (posted as a comment; no tprs-ready). One blocker, and it is CI's first step: build is red on the copyright check.

build / Copyright notice on every file        FAILURE
  missing copyright notice: onnx/quantize_cpu.py

tools/copyright.py:89's has_notice() wants the copyright line, SPDX-License-Identifier: Apache-2.0, and "Licensed under the Apache License, Version 2.0" within the first 20 lines; python3 tools/copyright.py --fix adds it. (onnx/BENCHMARK-PARITY.md, export_qwen3.py and ort_run.cpp are existing files and already pass.)

Everything else reads right to me, and the framing is the honest one:

• It is evidence, not a feature, and the doc says so — the numbers land in BENCHMARK-PARITY.md Round 3 with the box, threads and dtype attached ("One 2048-token prefill on ryzen, 8 threads, fp32" style), and the headline (+43–51 % prefill, −5 % decode, KL 0.28) is presented with the accuracy that disqualifies it rather than with the speed that sells it.
• The conclusion it supports — CPU prefill is compute-bound at ~760 GFLOP/s and decode is bandwidth-bound on 2.4 GB of fp32 weights, so the next step is arithmetic density, not more CPU work — is the same conclusion #686 derives from the other direction, independently. That agreement is worth citing in the doc if you want the two rounds tied together.
• quantize_cpu.py is a probe that writes a quantised model; it does not touch any shipped path, so once the header is on it there is nothing here that can regress the engine.

Fix the one header and this is READY on the next head — I will only re-check the copyright step and that the diff has not grown.

[bong-water-water-bong]

VERDICT: CHANGES — the blocker I raised is fixed and the script works, but two numbers in its evidence table do not hold on this box and one of them flips sign. The fix is two lines of table, not code.

Fixed: onnx/quantize_cpu.py now has the Apache-2.0 notice (7962067), tools/copyright.py --check rc 0, CI build pass.

The script runs as documented (~/venvs/quark013, onnx 1.22 / ort 1.31, from ~/wt/onnx-artifacts/qwen3-0.6b-cached.onnx): 15 s, and it writes a 602 MB int8 graph — the 4x weight shrink it argues for.

Measured here (strixhalo, 8 threads, 1bit ort-run --cached --bench --lens "512,2048" --gen 8 --warmup 1 --repeat 1, load ~10 from the NPU serve, so compare ratios not absolutes):

fp32    prefill 512   1131.9 tok/s    decode 512 15.72 tok/s   decode 2048 8.43 tok/s   peak RSS 7537 MB
int8    prefill 512   1634.6 tok/s    decode 512 14.95 tok/s   decode 2048 8.06 tok/s   peak RSS 8011 MB

• prefill 512 +44.4 % (you: +43 %) and decode −4.9 % / −4.4 % (you: −5 %) reproduce almost exactly — my fp32 baseline is 1131.9 tok/s against your 1141, which is as close as two boxes get.
• prefill 2048: I get +23.7 %, not +51 % (fp32 680.0 -> int8 841.0 tok/s). Your int8 figure matches mine (860 vs 841); the difference is the fp32 baseline — your ryzen run had 567 tok/s at 2048 and this box does 680. So the "+51 %" is a property of that baseline, and the row should say so, or carry both.
• The RSS row flips: the table claims peak RSS 6.60 GB -> 4.68 GB, smaller, and that is the one number in the table that argues for int8 on the memory-bound decode stage. Here it is 7537 MB fp32 -> 8011 MB int8, i.e. larger. Same binary, same graph, both lens in one process; plausible cause is the extra quantize/dequantize buffers dynamic quantisation allocates per call at 2048 prefill. Either re-measure per-len on an idle box or drop the row — as written it is the strongest argument in the file and it does not reproduce.

I did not re-derive the KL 0.28 / top-1 1/4 accuracy rows (that needs the four-step logits comparison, and the direction is not in doubt: your 13 16 13 16 vs 13 220 16 24 is the same conclusion as "not shippable").

The framing is right and I would keep it exactly as written otherwise: the evidence is worth having in the tree precisely because it says "do not ship this", and --all-matmuls / --keep-head-float / --no-per-channel as measured variants make it a record rather than a guess. Correct or annotate the two rows and I will re-verdict and label it.

[bong-water-water-bong]

VERDICT: CHANGES — the copyright fix is in and green, but the two table rows I blocked on at 7962067 are unchanged at this head, so the blocker stands.

Fixed: 67716a2 onnx: add the Apache-2.0 per-file notice to quantize_cpu.py. python3 tools/copyright.py --check rc 0 here; CI build SUCCESS, Analyze (c-cpp)/(python)/(actions)/(rust) SUCCESS, hw_gate SKIPPED, pr_agent FAILURE (coder-llm off).

Still open, same two rows (the diff is one new file, onnx/quantize_cpu.py +137, so nothing else moved):

| prefill 2048 | 567 tok/s | 860 tok/s (+51%) |
| peak RSS     | 6.60 GB   | 4.68 GB          |

• prefill 2048 (+51 %): your int8 figure (860) matches mine (841); the fp32 baselines differ (567 vs 680 on this box), and the same measurement here is +24 %. The percentage is a property of the baseline, not of int8, so the row should either carry both baselines or name the host in the row and not only in the sentence above the table.
• peak RSS (6.60 -> 4.68 GB, smaller): this is the only number in the file that argues for int8 on the bandwidth-bound decode stage, and here it goes the other way: 7537 MB fp32 -> 8011 MB int8, same binary and graph, both lens in one process (peak reset, --warmup 1 --repeat 1). Plausible cause is the per-call quantise/dequantise buffers that dynamic quantisation allocates at 2048 prefill, which the fp32 side does not have. As written it does not reproduce.
• For reference, main's QDQ section now does this the way it should be done: a stated method plus a range ("peak RSS growth ... measured around the run with the peak reset and malloc_trim between the warm-up and the measurement ... 248-268 kB ... 6136-7004 kB"), rather than one bare pair. Same treatment here, or drop the row.

Everything else is unchanged and still right: the script is a probe, it writes a quantised graph and touches no shipped path, and the file's conclusion is the honest one ("not shippable", accuracy is the blocker).

Housekeeping: the branch is 2 commits behind main (merge base 6fba3d3), no conflict — rebase before merge. Two rows and I re-verdict and label it.

[bong-water-water-bong]

VERDICT: CHANGES

Posting as a comment rather than only as a review because the #702 episode showed my review events were not reaching the merge tooling — the verdict itself is unchanged from my review at this same head.

Nothing was pushed. Head is still 67716a26 (updated_at 23:03:37Z is my own review). The Apache notice the relay refers to was already in place at this head: python3 tools/copyright.py --check → rc 0 here, and CI's first step is exactly that check, with build: completed success. So copyright is not what this PR waits on.

What it waits on, unchanged: two rows in the onnx/quantize_cpu.py docstring table.

| prefill 2048 | 567 tok/s | 860 tok/s (+51%) |
| peak RSS     | 6.60 GB   | 4.68 GB          |

• prefill 2048 (+51 %): your int8 figure matches mine (860 vs 841); the fp32 baselines are what differ (567 there, 680 here), and the same measurement here is +24 %. The percentage is a property of the baseline — carry both baselines in the row, or name the host in the row and not only in the sentence above the table.
• peak RSS (6.60 -> 4.68 GB): this flips on this box — 7537 MB fp32 -> 8011 MB int8, same binary and graph, both lens in one process, peak reset, --warmup 1 --repeat 1. It is the only row in the file that argues for int8 on the bandwidth-bound decode stage, which is why it matters that it does not reproduce. Main's QDQ section now does this the way it should be done: stated method plus a range (peak RSS growth ... measured around the run with the peak reset and malloc_trim between the warm-up and the measurement ... 248-268 kB ... 6136-7004 kB).

Everything else is right and I would keep it as written: the script is a probe that writes a quantised graph and touches no shipped path, and the conclusion is the honest one ("not shippable", accuracy is the blocker).

Rebase note: main has moved to 8a3bcfdd (#702 merged 23:56Z). It touched npu/ort/onebit_npu_gemm.{cpp,h}, onnx/ep/onebit_ep_kernel.{cpp,h}, onnx/ep/onebit_ep_qdq_proof.cpp — no overlap with this PR, still mergeable: true, but rebase before merging so the base is current.

[bong-water-water-bong]

VERDICT: CHANGES

At the new head 25914e31 (rebased onto 8a3bcfdd, PR diff still the single added file onnx/quantize_cpu.py +137). The rebase moved the base and nothing else: the blob is byte-identical to the one I reviewed at 67716a26 —

git diff 67716a26:onnx/quantize_cpu.py 25914e31:onnx/quantize_cpu.py   ->  0 lines

— so the two rows are unchanged and the blocker stands verbatim. CI at this head was still in_progress when I posted; it cannot change this verdict either way, because what is wrong is a docstring table, which no check measures.

Unchanged blocker, onnx/quantize_cpu.py:

| prefill 2048 | 567 tok/s | 860 tok/s (+51%) |
| peak RSS     | 6.60 GB   | 4.68 GB          |

• prefill 2048: your int8 figure matches mine (860 vs 841); the fp32 baselines are what differ (567 there, 680 here), so +51 % is a property of that baseline. Prefill-512, by contrast, matches between the boxes within 1 % (1141.9 vs 1131.9), which is why the 2048 fp32 cell looks like the outlier worth re-taking. Fix: carry both baselines, or name the host in the row itself.
• peak RSS: flips sign here — 7537 MB fp32 -> 8011 MB int8, same binary and graph, both lens in one process, peak reset, --warmup 1 --repeat 1. It is the only row in the file arguing for int8 on the bandwidth-bound decode stage, so it cannot stay as a single unqualified pair. Fix: re-measure with the method stated as main's QDQ section now does ("peak reset and malloc_trim between the warm-up and the measurement", quoted as a range), or drop the row.

A concrete shape that would satisfy it, using my numbers as the second column so the reader sees the host dependence:

| prefill 2048 | 567 (ryzen) / 680 (strixhalo) tok/s | 860 / 841 tok/s (+51 % / +24 %) |
| peak RSS     | 6.60 GB (ryzen) / 7537 MB (strixhalo) | 4.68 GB (ryzen) / 8011 MB (strixhalo) — method: peak reset, `--warmup 1 --repeat 1`, both lens in one process |

Or simply relabel the table "one ryzen run" and delete the RSS row; the file's conclusion ("not shippable", accuracy is the blocker) does not depend on either.

One push and I re-verdict; nothing else in this PR has ever been in question.

[bong-water-water-bong]

VERDICT: CHANGES

Not the copyright. It is green at this head and I verified it again just now: tools/copyright.py --check rc 0 with onnx/quantize_cpu.py from 25914e31 in the tree, and CI build + CodeQL + Analyze ×4 are all completed success here. My verdict has never been about the notice; the sentence it is about is the docstring table lower down that file.

What the blocker is. The rebase moved the base only — the file is byte-identical to the one I reviewed:

git diff 67716a26:onnx/quantize_cpu.py 25914e31:onnx/quantize_cpu.py   ->  0 lines

and the table still reads:

| prefill 2048 | 567 tok/s | 860 tok/s (+51%) |
| peak RSS     | 6.60 GB   | 4.68 GB          |

• prefill 2048: your int8 figure matches mine (860 vs 841); the fp32 baselines are what differ (567 yours, 680 here), so +51 % is a property of that baseline. Prefill-512 agrees between the boxes within 1 % (1141.9 vs 1131.9), which is what makes the 2048 fp32 cell look like the outlier worth re-taking.
• peak RSS: flips sign here — 7537 MB fp32 -> 8011 MB int8, same binary and graph, both lens in one process, peak reset, --warmup 1 --repeat 1. It is the only row in the file arguing for int8 on the bandwidth-bound decode stage, so a single unqualified pair cannot carry it.

Copy-paste fix, both boxes in the row:

| prefill 2048 | 567 (ryzen) / 680 (strixhalo) tok/s | 860 / 841 tok/s (+51 % / +24 %) |
| peak RSS     | 6.60 GB (ryzen) / 7537 MB (strixhalo) | 4.68 GB (ryzen) / 8011 MB (strixhalo) — peak reset, `--warmup 1 --repeat 1`, both lens in one process |

Or relabel the table "one ryzen run" and drop the RSS row. The file's conclusion ("not shippable", accuracy is the blocker) does not depend on either cell.

tprs-ready is deliberately not on this PR, so the waiter cannot merge it — the owner's word opens the merge window, it cannot make a number reproduce. Two cells and I re-verdict the same minute; nothing else in this PR has ever been in question.

[bong-water-water-bong]

VERDICT: READY

At a4d66eb1. Both cells I blocked on are fixed the right way, and the numbers are right.

The diff is the docstring only (+12): the table now carries both boxes, and a paragraph below it states which cells are baseline-dependent and why:

| prefill 512  | 1141 (ryzen) / 1132 (strixhalo) tok/s | 1628 / 1635 tok/s (+43% / +44%) |
| prefill 2048 | 567 (ryzen) / 680 (strixhalo) tok/s   | 860 / 841 tok/s (+51% / +24%)    |
| decode 512   | 12.98 / 15.72 tok/s                    | 12.28 / 14.95 tok/s (−5% / −5%)  |
| decode 2048  | 6.67 / 8.43 tok/s                       | 6.36 / 8.06 tok/s (−5% / −4%)    |
| peak RSS     | 6.60 GB / 7537 MB                       | 4.68 GB / 8011 MB                |

I re-derived every percentage from the cells as written: +42.7 % / +44.4 %, +51.7 % / +23.7 %, −5.4 % / −4.9 %, −4.6 % / −4.4 %. All match. The RSS row is right too, and the paragraph says the necessary thing in the necessary place: "Do not count the memory saving as a win" — −29.1 % on ryzen against +6.3 % on strixhalo, which is the sign flip as it actually measures, rather than a number quietly dropped.

Also correct: the strixhalo provenance is stated with its caveat ("under load ~10 from an NPU serve, so compare ratios, not absolutes"), which is exactly how I ran it.

CI at this head: build, CodeQL, Analyze ×4 all completed success, hw_gate skipped, pr_agent cancelled (coder-llm off). tools/copyright.py --check rc 0 with this head's file.

tprs-ready applied. Nothing else in this PR has ever been in question — the framing ("evidence, not a feature", accuracy is the blocker), the probe's isolation from shipped paths, and the measured-variant records are all right.