[bong-water-water-bong]

Task 4 of the A8W8 QDQ fast path (docs/onnx-pack.md). Based directly on main (9cdf058, all three QDQ
tasks merged). Quark is present on this box (source tree, release/0.13), so this is a real Quark
ONNX A8W8 export — not the fallback.

The export

onnx/quantize_quark_a8w8.py (new) drives Quark's ONNX flow on qwen3-0.6b.onnx. Calibration is 32 ×
512 tokens of local text (no dataset download); Quark's own quark/version.py is hand-written for a
bare source checkout and its deps live in a dedicated pytorch-CPU venv. Two exports were produced:
per-channel weights (the default, and what a transformer needs) and per-tensor (for contrast).

What the EP had to learn to claim a real export

finding	fix
Quark writes QuantizeLinear/DequantizeLinear in the com.microsoft domain	the matcher accepts either spelling (qdq_domain), still refusing any other domain's fused op. Without it: 0 of 254 matmuls claimed
a quantised graph still contains float matmuls the fused path cannot take (attention QK^T has no constant weight)	ONEBIT_EP_QDQ_ONLY=1 claims fused groups and nothing else, so a measurement of a quantised model is not distorted by float matmuls on the host loop
"0 groups" was unexplainable	ONEBIT_EP_QDQ_DEBUG=1 histograms the refusal reason per MatMul (qdq_refused why=… count=…, with the actual op/domain when that is the surprise)
ORT folds a QDQ graph back to float at ORT_ENABLE_ALL	ort-run --basic (the node's comment already explained why the proof uses BASIC)

Proof gains the shapes a real export has: rank-3 activations [1, seq, K], one scale and one zero
point per output column, and a 64×1024×2048 stress case (131072 int8 outputs, bit-exact vs the CPU EP).

Partitioning on the real graph (per-channel export, 4169 nodes after Basic)

	count
DQ → MatMul → Q groups fused into one node each	197 — every projection (28×7 + lm_head)
ONNX nodes those groups replace	788
MatMul left on the CPU EP	57 (28 QK^T + 28 AV are rank-4; 1 rotary matmul has a Cast weight)
everything else on the CPU EP	3381 (activation Q/DQ, LayerNorm, softmax, rope, elementwise)

Measured (single thread, 5-token prompt, 16 tokens, --dump-logits)

graph	tokens/s	ms/token	KL vs float	top-1
float32, CPU EP	7.39	135	—	12095 (" Paris")
A8W8 per-channel, CPU EP	1.43	699	2.19 nats	9625
A8W8 per-channel, our EP	2.09	479	2.81 nats	279
A8W8 per-tensor, CPU EP	3.09	323	6.49 nats	279
A8W8 per-tensor, our EP	2.06	486	6.49 nats	279

• 1.46× the CPU EP on the per-channel export, 0.67× on the per-tensor one. Per-channel DQ has to
  broadcast a scale vector and is expensive on the CPU EP, so removing the whole DQ → float MatMul → Q
  sequence pays off; per-tensor DQ is cheap and MLAS's float MatMul beats this host int8 kernel. Host
  measurement — the device leg is where the speed argument actually lands.
• KL 2.2–2.8 nats is the calibration's fault, not the EP's: 32 documents, MinMax, no
  AdaRound/QuaRot/SmoothQuant. The per-tensor export is worse (6.5 nats) and both paths agree on it
  exactly.
• Correctness, per tensor, via a throwaway probe_dump that dumps every graph output: on the
  per-tensor export our logits are bit-identical to the CPU EP's (KL 0.000, 17/17 decode steps equal).
  On the per-channel export the two agree exactly on the first 38 projection tensors, then differ on
  1 value in 5120 (0.02 %): accumulator −3090, exact value −3.499998 → rounds to −3; the CPU EP's
  float32 accumulation lands on exactly −3.500000 → round-half-to-even → −4. Both are right for their
  own arithmetic; the difference cascades because every layer requantises and this graph has no KV
  cache. That is the whole story behind the 2.81 vs 2.19 KL and the different top-1.

Not in this PR

• The device leg (still host gemm_q8; needs the NPU-runtime entry plus a QDQ-GO window).
• A good quantised model: that is AMD's recipe (real calibration corpus, AdaRound/QuaRot), not a
  measurement of this EP. The export here exists to exercise the EP, and it does.
• Task 5 (docs reconciliation).

Report: ~/briefs/TQDQ-STATE.md.

[bong-water-water-bong]

VERDICT: READY — tprs-ready applied. I re-derived the headline numbers on this box and they hold; the two things I'd still fix are a dead duplicated line and one attribute the matcher does not read (neither changes a result today, details at the end).

Built and ran at f300903: tools/copyright.py --check rc 0, ninja rc 0, ctest -R "onebit_ep|ort_info" 6/6 (2.02 s) including the new onebit_ep_qdq_proof_qdq_only. The four new Quark-shaped cases are the useful part, and all are exactly 0:

case quark         [1,seq,k] activation m=8  k=64   n=48    per-tensor / per-column   0 of 384     |d|=0  fused
case quark-asym    [1,seq,k] activation m=8  k=64   n=48    + zp_x scalar, zp_w per-col 0 of 384   |d|=0  fused
case quark-wide    [1,seq,k] activation m=5  k=1024 n=2048  (Qwen3-0.6B's projection) 0 of 10240   |d|=0  fused
case quark-stress  [1,seq,k] activation m=64 k=1024 n=2048  0 of 131072                 |d|=0  fused
case xaxis-K                                                                            0 of 48     per-node

Note the stress assertion is > 1 LSB, but the run is 0 of 131072, so the body's "bit-exact" is not a tolerance in disguise.

The measurement, re-derived (single thread, 16 tokens, same box, 1bit ort-run --basic, my own 5-token prompt 9707 11 1879 374 13 — different ids from yours, and the box is busier now, so compare ratios more than absolutes):

run	your figure	mine
float32, CPU EP	7.39 tok/s	7.42
A8W8 per-channel, CPU EP	1.43	1.39
A8W8 per-channel, our EP	2.09	1.97
A8W8 per-tensor, CPU EP	3.09	2.88
A8W8 per-tensor, our EP	2.06	1.96
ratio per-channel	1.46x	1.42x
ratio per-tensor	0.67x	0.68x

So the claim that matters — the fused path wins only where per-channel DequantizeLinear is expensive, and loses to MLAS when it is cheap — reproduces, and your honesty about it is the reason the table is worth anything.

The partition arithmetic checks out: 197 = 28x7 + lm_head, 788 = 197x4 (two DQ + MatMul + Q), and 197 + 57 = 254, which is exactly the "0 of 254" of the pre-fix run — the same denominator as the failure it replaces.

qdq_domain() verified independently, because no test covers it. Nothing in the repo spells a Q/DQ in com.microsoft, so I patched my own copy of the proof (local only, not pushed) to put QuantizeLinear/DequantizeLinear in com.microsoft while leaving MatMul in ai.onnx, exactly as Quark writes them:

PROBE_DOMAIN=com.microsoft  ->  case tensor/axis/asym: |d|=0 fused; xaxis-K: per-node; PASS

So the acceptance is real and the "refuse any other domain" side still behaves. Recommend making that a permanent case — a domain field on Case and one CreateNode argument is about ten lines, and right now a regression in qdq_domain() would restore the 0-of-254 failure with every test still green.

Two things to fix, neither blocking:

1. onnx/ort_run.cpp has else if (arg == "--basic") a->basic = true; twice (lines 184 and 187, the second dead). Harmless because the first wins, but it is the line someone will later "move" and be wrong about.
2. qdq_domain()'s comment says "same operator, same inputs, same attributes", and the matcher does not read saturate. I checked what that costs: ORT accepts saturate=0 on a com.microsoft.QuantizeLinear and the EP still fuses it, and with this proof's inputs the result is still |d|=0 — but that is only because s_y is chosen so nothing saturates by construction. On a graph that does clip, saturate=0 would leave ORT's output unclamped while the fused path clamps. Either read the attribute and refuse it, or change that sentence to "same attributes that matter here (saturate is assumed 1, which is what Quark writes)".

The rest reads right to me: ONEBIT_EP_QDQ_ONLY gates the single-node claims and is itself covered by the new ctest; ONEBIT_EP_QDQ_DEBUG copies the reason into the histogram while qdq_why()'s thread-local buffer is alive; the added op && before strcmp(op, "MatMul") fixes a null op that would have been passed to claims_op/can_compute; and --basic sets ORT_ENABLE_BASIC only, with the reason recorded where it is set.

One limitation to state plainly, since it is the whole point of the exercise: this proves the fused path on a real export, not that A8W8 is worth shipping. The KL is the calibration's (32 docs, MinMax), the device leg is still host gemm_q8, and the per-tensor result is a case where the CPU EP wins. None of that is a reason to hold the PR — it is why the numbers in it can be trusted.

[bong-water-water-bong]

Re-verdict at 3dd6d33 (the head moved after my READY): still READY, tprs-ready stays. The new commit is main merged in (3cddcd6, the HRX bump), and the PR's own change is untouched:

$ git diff f300903 3dd6d33 --stat
 registry/architectures.json | 2 +-
 third_party/llama.cpp       | 2 +-          # #694's bump, merged in
$ git diff --stat origin/main 3dd6d33
 7 files changed, 454 insertions(+), 54 deletions(-)   # identical to the head I verified

So the six proof cases, the 1.42x / 0.68x ratio measurement and the qdq_domain check all carry. One thing I noticed while re-checking, and it is now worth more than when I first wrote it: #693 and #694 have both landed, so this PR's onnx/ort_run.cpp and onnx/ep/onebit_ep.cpp are now the only open edits to those files except #686 and #688, which touch ort_run.cpp too. Merge order matters for a clean tree: this one goes after #686 (both edit make_session), and #688 after both.

The two nits from my first review are still open and still non-blocking: the duplicated else if (arg == "--basic") at onnx/ort_run.cpp:184 and :187, and the saturate attribute the matcher does not read.

[bong-water-water-bong]

VERDICT: READY

Gate line for the protocol, at head 3dd6d33 (current). build pass, MERGEABLE, tprs-ready applied.

Recap of what this verdict rests on, so the one-liner is not bare:

• Built and ran: tools/copyright.py --check rc 0, ctest -R "onebit_ep|ort_info" 6/6 (including the new onebit_ep_qdq_proof_qdq_only), and the four new Quark-shaped cases at |d| = 0 LSB — 0 of 384, 0 of 384, 0 of 10240, and 0 of 131072 on the 64x1024x2048 stress case (the > 1 assertion tolerance is slack, not a hidden difference), with xaxis-K still refused to the per-node path.
• Measurement re-derived on this box, single thread, 16 tokens: float32 7.42 tok/s (PR: 7.39), per-channel CPU 1.39 / our EP 1.97, per-tensor CPU 2.88 / our EP 1.96 — 1.42x per-channel, 0.68x per-tensor against the PR's 1.46x and 0.67x.
• qdq_domain() verified independently, because no test covers it: I patched my own copy of the proof to spell QuantizeLinear/DequantizeLinear in com.microsoft (MatMul in ai.onnx), as Quark writes them, and the proof passes with the same fused paths.
• Partition arithmetic closes: 197 = 28x7 + lm_head, 788 = 197x4, 197 + 57 = 254, the same denominator as the "0 of 254" failure this PR fixes.

Two non-blocking nits still open: the duplicated else if (arg == "--basic") at onnx/ort_run.cpp:184/:187, and the saturate attribute the matcher does not read. Merge order: after #686 — both edit make_session.

[bong-water-water-bong]

VERDICT: READY

At head f82d1ca6 (post-update, main merged in), build pass, MERGEABLE, tprs-ready applied.

Content unchanged by the merge, checked not assumed:

$ git log --oneline --no-merges origin/main..f82d1ca6
f300903 onnx: run a real Quark A8W8 export through the fused QDQ path
$ git diff --stat origin/main f82d1ca6
 7 files changed, 454 insertions(+), 54 deletions(-)     # identical to the head I verified
$ git merge-base --is-ancestor origin/main f82d1ca6      # yes

So everything in my review above stands: 6/6 ctests including onebit_ep_qdq_proof_qdq_only, the four Quark-shaped cases at |d| = 0 (0 of 131072 on the stress case), the re-derived 1.42x per-channel / 0.68x per-tensor ratios, the com.microsoft domain check run through my own patched copy of the proof, and the 197 / 788 / 57 / 254 partition arithmetic. The two nits (duplicated --basic line, unread saturate) are still open and still non-blocking.

Merge order note: #686 is ahead of this one and both edit make_session in onnx/ort_run.cpp, so this goes second; #688 is third in that file.

[bong-water-water-bong]

VERDICT: READY

At head 20aada7 (main merged in again after #699 landed), build pass, MERGEABLE, tprs-ready applied.

$ git log --oneline --no-merges origin/main..20aada7
f300903 onnx: run a real Quark A8W8 export through the fused QDQ path
$ git diff --stat origin/main 20aada7
 7 files changed, 454 insertions(+), 54 deletions(-)
$ git merge-base --is-ancestor origin/main 20aada7      # yes

Unchanged content, so my review stands in full: 6/6 ctests with onebit_ep_qdq_proof_qdq_only, four Quark-shaped cases at |d| = 0 LSB including 0 of 131072 on the stress case, the re-derived 1.42x per-channel / 0.68x per-tensor ratios, qdq_domain() verified against a com.microsoft-domain graph, and 197 / 788 / 57 / 254 partition arithmetic. The two nits stay non-blocking.

Merge order: #686 first (both edit make_session in onnx/ort_run.cpp), then this, then #688.