[bong-water-water-bong]

The QDQ device leg was documented as correctness-only because a device call is dispatch-bound; #702
deferred the model-level number. This PR measures it and records what it is worth, including the fact that
the sign is not what a "speedup" claim would say.

The measurement

Four arms, one graph — the real Quark A8W8 export of qwen3-0.6b.onnx (196 fused groups; the 197th,
lm_head, keeps the host kernel because its weight is a Cast) — Release build of this tree, one
thread, 32-token prompt, 8 generated tokens, quiet box (load 1.17–1.35 for every arm), two runs each, the
second quoted:

arm	prefill 32 tokens	decode ms/token	tok/s
float32, CPU EP	207 / 212 ms	225.6 / 226.0	4.43
A8W8, CPU EP	635 / 645 ms	625.1 / 628.1	1.59–1.60
our EP, host kernel	843 / 845 ms	947.4 / 950.6	1.05–1.06
our EP, array	805 / 809 ms	1073.7 / 1105.1	0.90–0.93

• the array is 1.13–1.16× slower than the host kernel it replaces, 1.7× behind the A8W8 CPU EP, 4.8× behind float32;
• the 32-token prefill is a wash (805/809 against 843/845, 4–5 % ahead of the host leg);
• the kernels are ahead in isolation: 1.54–1.89× per call at a decode step's 36 rows (two blocks per launch,
  32 + 4) and 3.1× at a 32-row block — and that advantage does not survive the model.

The cause is not claimed. Two explanations fit every number here and these measurements do not separate them:
(a) dispatch — 196 calls per pass, two launches each at 33–40 rows, each paying the A upload, two syncs and a
wait (at ~1.3 ms of non-kernel work per call, ~250 ms of a 1074 ms pass); or (b) the host loop is faster on the
graph's real operands than on the probe's — the probe's host column is what makes "kernels are 1.5–1.9× ahead"
true, and if the graph's sparser activations let the aik == 0 skip cut it by 30 %, the host arm's kernel time
falls to ~850 ms/pass and the device's 712 ms is no win at all. Both land on the arms as measured. The next step
is a profile — NPU_C_SYNC_PROF=1 (engine#988) plus a per-call timer inside gemm_q8 — not a redesign. What is
established is the sign, and that the model-level claim must be arm-to-arm, not kernel-to-kernel.

Correctness is unchanged and exact: the device arm generated the same ids as the host arm and the same KL
against float (5.6787 nats both; the A8W8 CPU EP's arm scores 7.2085 with a different continuation), trace
qdq runs=1764 (= 196 × 9 passes), host-rows policy=0, device fallbacks=0, device-busy=0. The
5-token counter-example is the other half of that claim: same binary and designs, 8 tokens in 2159 ms
(3.70 tok/s), trace host-rows policy=1764 and no device call at all — a 5-token prompt never reaches
kDeviceMinRows.

The probe cross-check ties this to the published rows table: at m=32, q 1024×2048 measures
device 1.632 / host 5.166 against the documented 1.566 / 5.062 (4 % and 2 %).

The trap this measurement walked into, because it will happen to the next person

The EP targets carry no -O2 of their own (only ort_run/ort_info do), so a bare
cmake -S onnx -B compiles the host int8 loop unoptimized: one probe call measured 129.6 ms where the
Release build measures 5.0 ms, i.e. a -O0 build inflates the device-against-host ratio by ~25×. The
first take of these arms was such a build and is not quoted. The rule now written into the docs: timing
numbers come from a Release build, correctness gates stay on the unoptimized one (this lane has both
directions on record — unoptimized builds are the stricter gate, and the worse measurement).

Shape of the change

docs/onnx-pack.md (the device-leg bullet: "correctness result" gains its measured model-level qualifier and
the four-arm figures) and docs/onnx.md (a new section, "What the device leg is worth at model level", with
both tables, the accounting, the counter-example, and the statement that the probe's host column is a unit
measurement — it sums to 1209 ms/pass at m=36, above the whole host arm's 947 ms/token, so it must not be
subtracted from the device column). No code change, no behaviour change.

Amended pre-verdict (the head moved from 673c854 to d9d3f59) because the first version of the sentence above
named dispatch as the cause, which the numbers do not support: the probe's host column is inflated by synthetic
operands, so "the kernels are 1.7× ahead" is a synthetic-data result and cannot be used to conclude that per-call
overhead is what eats it. The docs now state both candidates and the profile that would decide.

python3 tools/copyright.py --check → rc 0 (no new files). Every number in both tables is from
~/wt/qdq-model-gate.sh, ~/wt/qdq-model-arms.sh and ~/wt/qdq-kernel-times.sh in one quiet window on
434db3d, three separate short npu-hold -l qdq tickets, device-busy=0 in all of them.

[bong-water-water-bong]

#1023 ready for review at 673c854. Docs-only: the QDQ device leg's model-level number (#702's deferred item) — 1073.7/1105.1 ms/token on the array against 947.4/950.6 for the host kernel it replaces, same ids and same KL, trace qdq runs=1764 host-rows 0 fallbacks 0; plus the -O0 trap that voids any timing from a bare cmake build (~25x on the device-against-host ratio). Every number is from one quiet window (load 1.17-1.35) on 434db3d, three short npu-hold -l qdq tickets, device-busy=0 in all three.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 20b72fa)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 702 - Partially compliant Compliant requirements: Implemented fused A8W8 QDQ path on the device Added gemm_q8_raw in npu/ort for int8 activations and weights Modified onnx/ep/onebit_ep_kernel.cpp to try device path and fall back to host loop Ensured epilogue is shared between device and host paths Fixed ONEBIT_EP_TRACE flag handling bug Fixed failed shapes being rebuilt on every call Non-compliant requirements: No m16 ELFs for q/k/v shapes (deferred to future work) I8Ctx::init_elf does not validate the design (reported as separate issue) Requires further human verification: Verification that the device lock behavior is correct Confirmation that the performance measurements are accurate and reproducible 988 - Partially compliant Compliant requirements: Reduced C syncs to only cover rows the host reads Implemented narrowing of C syncs to c_host_bytes() range Added NPU_NO_C_NARROW=1 to restore whole-BO range for debugging Added NPU_DBG to force whole-BO range for checksumming Added NPU_C_SYNC_PROF=1 to print sync statistics Non-compliant requirements: The A-side flush is unchanged (deferred to future work) The pb16i/gemv routes were not re-gated on the device (deferred to future work) Requires further human verification: Verification that the flush traffic reduction is actually measured and correct Confirmation that the identity pass still works correctly with the changes

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Measurement Documentation The documentation states that the device leg is 1.13–1.16× slower than the host kernel, but the cause is not isolated. The PR adds this measurement to the documentation, but it's important to note that this is a correctness-only result, and the performance characteristics are not fully understood. The documentation should clearly state that this is a measured result, not a claimed performance improvement. **correctness** result — and at model level it is **not yet a speed one**, which is now measured rather than suspected. Same export, a Release build of this tree, a quiet box (load 1.2–1.4), one thread, a 32-token prompt and 8 generated tokens: the array's arm does **1074 / 1105 ms/token against the host kernel's 947 / 951**, i.e. 1.13–1.16× *slower* than the leg it replaces (and 1.7× behind the A8W8 CPU EP at 625 / 628, 4.8× behind float32 at 226); the 32-token prefill is a wash, 805 / 809 against 843 / 845. The kernels are ahead **in isolation** — 1.54–1.89× per call at a decode step's 36 rows and 3.1× at a 32-row prefill block — and that advantage does not survive the model. These measurements do not, by themselves, say which of the two candidate reasons it is: 196 dispatches per pass, each paying BO writes, two syncs and a wait; or the host loop being faster on the graph's real operands than on the probe's synthetic ones (`docs/onnx.md` explains why the second cannot be excluded). What is measured is the sign, not the cause; `NPU_C_SYNC_PROF=1` on one call is the next step. (Those arms, like every number on this page, come from a Release build: the EP targets carry no `-O2` of their own, and an unoptimized host loop inflates the device-against-host ratio by ~25×.) Model-Level Performance Analysis The documentation provides a detailed analysis of why the device kernels are faster in isolation but slower at the model level. It correctly identifies two potential causes: dispatch overhead and the host loop being faster on real data. However, it's important to ensure that the measurements and analysis are accurate and that the documentation clearly explains the implications of these findings for future optimization efforts. **What the device leg is worth at model level, and it is not yet a win.** The rows table above times one shape; this times the graph. Four arms on one graph — the real Quark A8W8 export of `qwen3-0.6b.onnx` (196 fused groups, the 197th being `lm_head`, whose weight is a `Cast`, so it stays float) — a Release build of this tree (the EP targets carry no `-O2` of their own: a bare `cmake -S onnx -B` compiles the host loop unoptimized, which inflates the device-against-host ratio by about 25× — measured, 129.6 ms against 5.0 ms for the same probe call), one thread, a 32-token prompt, 8 generated tokens, quiet box (load 1.17–1.35 for every arm), two runs each and the second quoted: | arm | prefill 32 tokens | decode ms/token | tok/s | |---|---|---|---| | float32, CPU EP | 207 / 212 ms | 225.6 / 226.0 | 4.43 | | A8W8, CPU EP | 635 / 645 ms | 625.1 / 628.1 | 1.59–1.60 | | our EP, host kernel | 843 / 845 ms | 947.4 / 950.6 | 1.05–1.06 | | our EP, **array** | 805 / 809 ms | **1073.7 / 1105.1** | 0.90–0.93 | So at model level the array is **1.13–1.16× slower than the host kernel it replaces**, 1.7× behind the A8W8 CPU EP and 4.8× behind float32; only the 32-token prefill is ahead of the host leg, by 4–5 %, and still 1.3× behind the CPU EP. Correctness is exact where it matters: the device arm generated **the same ids as the host arm** and its KL against float is the same number (5.6787 nats both; the A8W8 CPU EP's own arm scores 7.2085 with a different continuation), and its trace is `qdq runs=1764` (= 196 groups × 9 passes), `host-rows policy=0`, `device fallbacks=0`, `device-busy=0` — every fused group went to the array, and nothing was dressed up as one. The 5-token counter-example is the other half of that claim: same binary, same designs, 8 tokens in 2159 ms (3.70 tok/s), trace `host-rows policy=1764` and no device call at all, because a 5-token prompt never reaches 16 rows. **Why the kernels being 1.5–1.9× ahead does not reach the model.** One probe per design at the decode width (m=36, which is two blocks per launch — 32 rows + 4 — so it measures what a decode step really issues), 20 timed calls each, same build and same window: | shape | groups/pass | device ms/call | host ms/call | device ms/pass | |---|---|---|---|---| | q 1024×2048 | 28 | 3.663 | 5.629 | 102.6 | | k/v 1024×1024 | 56 | 1.670 | 2.874 | 93.5 | | gate/up 1024×3072 | 56 | 5.287 | 8.769 | 296.1 | | o 2048×1024 | 28 | 3.292 | 5.638 | 92.2 | | down 3072×1024 | 28 | 4.560 | 8.626 | 127.7 | | **196 groups** | | | | **device 712.0, host 1209.0 (1.7×)** | At m=32 the same probe reproduces the rows table's q cell — `device 1.632 / host 5.166` against the published `1.566 / 5.062`, i.e. 4 % and 2 % — so the two measurements describe the same kernel. And the host column is a **unit** measurement, not an additive one: it sums to 1209 ms/pass, which is *above* the whole host arm's 947 ms/token. The likely reason is that the probe's operands are synthetic int8 while the graph's are not and the host loop skips `aik == 0` — a hypothesis, not something measured here. Either way the host column is a unit measurement: do not subtract the two rows above to explain the arms, compare arms to arms. **Which means the cause of the model-level loss is not established, only its sign.** Two explanations fit every number on this page, and these measurements do not separate them: 1. **Dispatch.** 196 calls per pass, two launches each at 33–40 rows, each paying the A upload, two syncs and a wait: at ~1.3 ms of non-kernel work per call that is 250 ms of a 1074 ms pass. 2. **The host loop is faster on real data than on the probe's.** The probe's host column (5.629 ms for q at 36 rows) is what makes "the kernels are 1.7× ahead" true in isolation; if the graph's sparser activations let the `aik == 0` skip cut that by 30 %, the host arm's kernel time falls to ~850 ms/pass, the device's 712 ms becomes a far smaller win, and the two arms land where they are measured — the device slightly behind. The bound that *is* rigorous, and it is tight enough to matter: the probe's host column sums to 1209 ms/pass while the whole host arm — kernel plus every other node in the graph — is 947 ms/token, so the graph's real operands make the host loop **at least 1.28× faster** than the probe's synthetic ones. The device's 1.7× kernel-level advantage is therefore at most **1.33×**, and it could be none at all, which is the whole reason the cause is stated as open rather than as dispatch. The next step is a profile, not a redesign: `NPU_C_SYNC_PROF=1` (engine#988) prints the C-sync counters per call, and a per-call timer inside `gemm_q8` separates launch, wait, sync and epilogue. What a *fix* would look like if (1) dominates is one launch covering many projections — a resident weight layout and a pass-level submission, rather than the per-projection ELF this leg uses — which is also why the threshold discussion above stands unchanged: `kDeviceMinRows` is about one call, and the model-level question is about 196 of them.

[bong-water-water-bong]

#1023 head moved 673c854 -> d9d3f59, pre-verdict, and the change is a correction of my own claim: the first version said dispatch costs the model the win. It cannot: the probe's host column that makes 'the kernels are 1.5-1.9x ahead' true is measured on synthetic int8 operands, and its sum (1209 ms/pass at m=36) is above the whole host arm's 947 ms/token — so the same data that shows a kernel-level win is what shows I cannot use it to attribute the model-level loss. The docs now state the two candidates (196 dispatches per pass; or the host loop being faster on real than on synthetic operands), say the measurements do not separate them, and name the profile that would (NPU_C_SYNC_PROF=1 plus a per-call timer inside gemm_q8). The measured claims are unchanged: 1073.7/1105.1 ms/token on the array against 947.4/950.6 for the host kernel, same ids, same KL, trace qdq runs=1764 host-rows 0 fallbacks 0 device-busy 0.

[github-actions]

Persistent review updated to latest commit d9d3f59

[bong-water-water-bong]

Head now 74d1952 (still pre-verdict; this is the last of the correction, not a new claim). Added the one bound the data does support, because it is what makes the open-cause statement quantitative rather than hedged: the probe's host column sums to 1209 ms/pass while the whole host arm — kernel plus every other node — is 947 ms/token, so the graph's real operands make the host loop at least 1.28x faster than the probe's synthetic ones. The device's 1.7x kernel-level advantage is therefore at most 1.33x and could be none. Both candidate causes stay listed; the profile (NPU_C_SYNC_PROF=1 plus a per-call timer in gemm_q8) is what decides.

[github-actions]

Persistent review updated to latest commit 74d1952

[bong-water-water-bong]

#1023 ready for review at 74d1952. Docs-only (docs/onnx-pack.md, docs/onnx.md). Head moved three times before any verdict, all pre-verdict corrections of my own claim (673c854 dispatch-attribution removed -> c92c147 the 1.28x bound -> 74d1952 that paragraph moved beside the two candidates); the numbers themselves did not move. The one red check, pr_agent, is a cancelled run ("higher priority waiting request"), not a finding — the persistent review comment is current at 74d1952. Nothing running, nothing queued, no token or device lock held by this lane.

[bong-water-water-bong]

VERDICT: READY at 74d19520

Docs-only (+79/−6, docs/onnx.md and docs/onnx-pack.md), and the claim is a negative one, so what matters is that its arithmetic holds and that nothing else on the page still contradicts it.

Every derived number re-computes from the printed cells. From the per-shape table and the four arms:

device ms/pass  102.6 + 93.5 + 296.1 + 92.2 + 127.7 = 712.0   (page: 712.0)
host   ms/pass  157.6 + 160.9 + 491.1 + 157.9 + 241.5 = 1209.0 (page: 1209.0)
ratio 1.70x ; array vs host-kernel arm 1073.7/947.4 = 1.133x and 1105.1/950.6 = 1.163x (page: 1.13-1.16x)
array vs A8W8 CPU EP 1.72x ; vs float32 4.76x (page: 1.7x, 4.8x)
bound: host arm 947.4 / device kernel 712.0 = 1.331x (page: at most 1.33x)
trace 196 groups x 9 passes = 1764 (page: qdq runs=1764) ; 1073.7 ms/token = 0.93 tok/s (page: 0.90-0.93)

The reasoning in "Which means the cause ... is not established" is the part I checked hardest, because it is the part that could be overclaiming: it does not assert dispatch is the cause, it gives two explanations that both fit and then states the one bound that is rigorous (host arm total 947 < probe host 1209 => the graph's operands make the host loop >= 1.28x faster than the probe's, so the device's 1.7x kernel advantage is <= 1.33x — 947.4/712.0 = 1.331, and 1209.0/947.4 = 1.276, so "at least 1.28x" is a rounding up by 0.004; with the second run, 1209.0/950.6 = 1.272).

The release-build caveat is structural, and I confirmed it in the build files. onnx/ep/CMakeLists.txt carries no -O flag at all and the host loop (gemm_q8, onnx/ep/onebit_ep_kernel.cpp) is built by it; onnx/CMakeLists.txt only sets -O2 on its own targets (onebit_ort_run, ort_info, …). So a bare cmake -S onnx -B really does build the EP unoptimized and the EP-vs-host comparison inverts. My own attempt to re-derive the "~25x" on a toy int8 loop gave only 4.3x (32x2048x1024, O0 111.72 ms vs O2 25.94 ms — the gap is vectorization, and my layout is not yours), so I am not endorsing the magnitude, only the direction: an -O0 host loop against an -O2 device arm is not a comparison, and printing that sentence next to the table is the right call.

Two things I could not check, and one nit:

• No artifacts for the four arms on this box. Nothing under ~/lb carries a qdq runs= trace or these arms (searched ~/lb for the trace strings; the newest dirs are other lanes'). The tables are the author's run logs only, so "re-run this later" is not currently possible from the repository or the box — if the numbers are to survive as the page of record, the four ort_run invocations and their trace lines belong in the body or a run dir.
• Pre-existing text this PR leaves standing is consistent: docs/onnx-pack.md:295 still says the device leg runs the epilogue "only as a correctness result so far, so it is not where the 1.5x-per-projection argument lands either". Good — the two pages now say the same thing.
• Nit: the per-shape table's host column is explicitly a unit measurement (1209 ms/pass > the 947 ms whole host token), and the next paragraph leans on that. A one-line "do not subtract" note is already there; add "(sums to more than the arm because it is per-unit)" to the table caption too, since the table is what gets quoted.

[bong-water-water-bong]

VERDICT: READY at 20b72fa35

Rebase only: own contribution (merge-base with origin/main..head, +/- lines only) hashes to 3e8ffa947c21 at this head and at 74d19520f, the head I reviewed — 2 files, +79/−6, unchanged. Every number in the two tables was re-computed in that review, and the structural half (onnx/ep/CMakeLists.txt carries no -O flag at all) still holds.

[github-actions]

Persistent review updated to latest commit 20b72fa