From d101f1e30300db82c5dd8afd9a6480c272e17431 Mon Sep 17 00:00:00 2001 From: Kingdon Barrett Date: Mon, 19 Oct 2026 22:06:33 +0100 Subject: [PATCH 1/4] docs: add QWEN36_METAL_REVIEW.md (claim-by-claim Unsloth comparison review) Spec-Ref: helix-specs@bd4d5000:000684_add-rigor-to-research --- docs/QWEN36_METAL_REVIEW.md | 168 ++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 docs/QWEN36_METAL_REVIEW.md diff --git a/docs/QWEN36_METAL_REVIEW.md b/docs/QWEN36_METAL_REVIEW.md new file mode 100644 index 0000000000..f4e76d54ec --- /dev/null +++ b/docs/QWEN36_METAL_REVIEW.md @@ -0,0 +1,168 @@ +# Qwen3.6 Metal — Review of the Unsloth Comparison and the Phase 2 Roadmap + +> **Reviewer:** Qwen-3.8-Flash-Next (Zed coding agent) · 2026-10-19 +> **Subjects reviewed:** +> 1. `docs/QWEN36_METAL_ROADMAP.md` — the Phase 2 initiatives T8–T11 (this repo), authored by Gemini-3.8-Flash +> 2. `docs/kb/how-to-strata-vs-unsloth-qwen36-metal-benchmarks.md` in **urmanac/nocknock** — the "Standing on the Shoulders of Open-Source Giants" article +> +> **Method.** Every technical claim in both documents was checked against the measured record in this repository (`docs/QWEN36_METAL.md` §6–§10 and the recorded gate numbers, `bench/results/2026-10-18-m2-q36-prefill`), the code (`metal/`), the model card (`unsloth/Qwen3.6-35B-A3B-MTP-GGUF` on Hugging Face), the engine build's release notes (`unslothai/llama.cpp` release `b11724-mix-c208e5f`, published 2026-10-19 16:12 UTC), the fork's pull requests, and the benchmark-session screenshots the maintainer supplied on 2026-10-19. Where a claim could not be checked, this review says so and labels it an assumption. Nothing here is asserted without a measurement or a source; inferences from reading code are marked "static analysis". + +## 0. Summary + +Three findings drive the rest of this review: + +1. **The engine on the other side of the comparison is now identified exactly.** The Unsloth Desktop build is `b11724-mix-c208e5f`: upstream llama.cpp `b11724` plus 20 pinned commits (the release page enumerates every one). **No pinned commit touches Metal compute** — the compute-relevant pins are CUDA (MoE prefill #264, cpy fast path #157, MMVQ batch cap #287), HIP (#149, #158) and Vulkan (#270). On the M2, the benchmark ran **upstream ggml-metal kernels** with the model card's flags. That is good news for attribution and good news for the roadmap: the prefill bar we are chasing is the quality of upstream ggml-metal, and the fork's own CUDA work (#264) shows what "kernel structure" is worth even in that mature engine (+16.4–17.8% on Qwen3.6 prompt processing; decode +0.1%, i.e. tie). +2. **Two of the four Phase 2 initiatives survive review; one is redundant; one attacks a ceiling the 32k experiment proved is not there.** T9 (batched prefill attention) and T10 (batched MoE GEMM) are the right homework. T8 (prefix caching) is homework generated by misreading a fresh request — the feature shipped in T0 (`metal/src/engine/q36_metal.cpp` L2419–2429, documented in `docs/QWEN36_METAL.md` §2.2). T11 (paged context) aims at a wall that the 32,768-token run on the *same 24 GB machine* showed does not exist; it should be reframed from "break the ceiling" to "budget predictably at the ceiling". Two cheap measurement tickets are missing and should come first. +3. **The article's numbers that disagree with the recorded record are misquotes, not measurements.** The GDN-scan and dense-MMA speedup paragraphs quote three different value sets for the same two measurements; only the diagram's values match `docs/QWEN36_METAL.md` §10. Corrected numbers and sources are in Table A. The earlier walk-back commit (`b7806f1`, 2026-10-19) already fixed the MTP attribution and the "static KV table" framing; what remains is listed in §9. + +The maintainer's three research questions get direct answers in §4: why Unsloth prefills faster (dispatch structure, not magic — and the deep-decode numbers show it is *not* falling off a cliff at depth, which is our T6 opening); what we have that they do not (measured, itemized); and whether a combined implementation is possible (yes, with the memory math to back it — the 32k experiment already proved the memory side on their engine). + +--- + +## 1. Verdict labels used below + +| Label | Meaning | +|---|---| +| **Verified** | The claim matches the recorded measurement or source exactly. | +| **Misquoted** | A real recorded measurement, quoted with wrong numbers. Corrected value and source given. | +| **Unverified hypothesis** | Plausible, no recorded source; must be labeled as hypothesis wherever it is repeated. | +| **Refuted** | The evidence contradicts the claim. | +| **Stale baseline** | True when recorded, no longer the current number; the comparison it feeds is wrong until re-measured. | +| **Unconfirmed static analysis** | Read from code, not run. | + +## 2. Table A — the NockNock article, claim by claim + +| # | Claim (as written) | Verdict | Corrected fact and source | +|---|---|---|---| +| 1 | "Reduced GDN recurrence latency from **169 ms to 11.4 ms** at T=512 (**14.8× faster**)" (Layer 1 text) | **Misquoted** | Recorded: `k_gdn_chunk` 167.730 ms → 11.476 ms (prep+cprep+cstate+cout) at T=512 = **14.6×**; at T=2048: 670.919 → 44.942 ms = **14.9×** (`docs/QWEN36_METAL.md` §10 table, build `4836ba99`, Apple M2, 2026-10-18). The article's own diagram says 14.6× — the text contradicts its own diagram and the source. Neither "169 ms" nor "14.8×" matches anything recorded. | +| 2 | "Reduced dense projection latency from **106 ms to 7.1 ms** at T=512 (**14.9×**) and **487 ms to 26.6 ms** at T=2048 (**18.3×**)" (Layer 2 text) | **Misquoted / unsourced** | Recorded: the dense rows ran 110.2 → 7.2 ms per chunk at the attn_qkv shape = **15.3×** (§10, live-session round, build `3dca4b47`). The 487→26.6 ms / 18.3× pair matches nothing recorded anywhere in this repo; the 14.9× factor appears to be lifted from the GDN T=2048 row (row 1), applied to the wrong numbers. | +| 3 | Diagram: "(14.6x faster GDN recurrence)" and "(15.3x faster dense projections)" | **Verified** | Matches §10 exactly. | +| 4 | T8: "Implement token-prefix hashing and state reuse in `q36_metal.cpp`" (roadmap) / "KV Cache Prefix Matching" as new Phase 2 work | **Refuted as scoped — the work shipped in T0** | Prefix matching exists and is documented: `q36_metal.cpp` `run()` L2419–2429 matches `live_tokens` against the request prefix, keeps the S/conv/KV state, and evaluates only the delta when the prefix extends (verified 2026-10-19); `docs/QWEN36_METAL.md` §2.2 "Multi-Turn Prefix Caching". The article's own Test-2 table shows Strata "Cache Hits: 0 (Fresh prefill)" — that was a fresh request, not a resumed session. The honest residual scope is small: measure the existing reuse (one two-request run), keep the common prefix on divergence (today a divergence resets fully — L2424–2428), optionally persist across restarts. | +| 5 | "the session's allocated 8,192-token context buffer was exhausted... likely based on available free system RAM at model load time" (post-walk-back wording) | **Unverified hypothesis (mechanism); policy confirmed (auto-sized, not an engine ceiling)** | The maintainer confirmed Auto was selected, settled at 8,192 under load, and that a **manual 32.8k load succeeded** on the same 24 GB machine (Chrome closed; one yellow warning, no failure). So 8.2k was a conservative auto-budget, not a hard engine limit — the walk-back got that right. But the *mechanism* ("free RAM at load") is undocumented in public: the candidate is the fork's memory-fit pass (`--moe-cache-mib auto`, pinned #251), yet the card's own llama.cpp example hardcodes `-c 8192`, so a card-derived default is equally possible. One engine-log line settles it (§8, item E1). | +| 6 | Decode 31.2 tok/s (shallow) and 23.8 tok/s (2,328 tokens) — MTP as the explanation | **Bounded to one log line; both branches stated** | The walk-back already removed the "proved MTP acceptance ~1.8–2.0×" claim — correct. What remains is a two-branch ambiguity the evidence can settle: the card says MTP needs `-np 1` ("`-np > 1` ... not yet supported with MTP") with the recipe `--spec-type draft-mtp --spec-draft-n-max 2`, and claims "~1.5–2x faster with MTP"; the benchmark session's API page showed **4 slots** — if np=4 reached the engine, MTP was off and 31.2 tok/s is pure kernel efficiency (≈31–37% of the M2's ~83–99 tok/s single-token bandwidth ceiling — a healthy ggml-metal number); if Studio auto-enabled MTP at np=1, the same speed would imply a ~16–24% base efficiency (low for ggml-metal) and the card's 1.5–2× would have to come from somewhere else. The model-load log decides (§8, item E1). Our own in-tree MTP verdict is separate and measured: net-negative on the M2 (§4.2) — but see the calibration note there. | +| 7 | Strata prefill "~25 tok/s"; the gap framed as "~9.5×"/"~10×" | **Stale baseline** | The ~25 tok/s is the bringup token-by-token number. The recorded pre-lever matrix (T3, `docs/QWEN36_METAL.md` §6): 2K 14.7 → **22.3 tok/s** at chunk 512, 8K 10.0 → 12.7 (1.24–1.51×; M2, 2026-10-18). Since then the WY scan, MMA GEMMs and GQA fold have landed but the hours-class matrix was deliberately skipped (§10). The current all-levers prefill is **unrecorded** — re-run the matrix before any multiple is claimed (§7, block 1). | +| 8 | "a single 512-token chunk generates over 20,000 individual Metal command encoder dispatches" | **Unverified count (direction plausible)** | The per-row attention dispatch is real — the quoted code comment at `q36_metal.cpp` L1274–1278 is accurate (verified 2026-10-19). A rough count from the attention side alone (512 rows × 10 QSA layers × several kernels) lands in the same order of magnitude (**static analysis**), but the routed-expert path's count depends on how `push_c_lpr8` (L1023) groups entries, which we have not instrumented. Do not publish a count without the instrumentation ticket (§6). | +| 9 | Diagram: "GQA Attention Fold — Eliminates 8x KV bandwidth" | **Verified factor; overclaimed verb; bounded leverage** | The model card gives Gated Attention **16 Q heads / 2 KV heads** → H/HKV = exactly 8× (also head dim 256, 10 attention layers of 40). T6's rule "read the factor from the geometry, don't assume" is satisfied. Two corrections for the prose: "reduces ~8×", not "eliminates"; and the fold only touches the 10 attention layers — the 30 GDN layers carry a constant-size recurrent state (~31 MB/token total: 30 layers × 32 V-heads × 128² × 2 B), so KV becomes the dominant depth-dependent traffic from roughly 1,500 tokens of context upward (≈20.5 KB per token per 1,000 of context at f16). The lever is real but bounded — and it is exactly the deep-decode lever (§4.1). | +| 10 | "40 layers combining 256 routed MoE experts, Quasi-Sliding Attention, and Gated Delta Net" | **Reconciled — both counts are right** | The card says **40 layers**: layout 10 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE)) = 30 GDN + 10 gated attention; Strata's "41" counts the `blk.40` MTP draft block on top. Define the count once ("40 main + 1 draft") and stop using two. ("Quasi-Sliding Attention" is not a name the card or the upstream docs define — prefer "gated attention (QSA)".) | +| 11 | "14+ custom `.metal` kernels" | **Undercount** | `metal/kernels/q36.metal` alone defines **38** `kernel void k_` functions (counted 2026-10-19), plus `q36_dense.metal`, `qsa*.metal`, `gdn_prompt.metal`. Technically true, strangely modest. | +| 12 | "Unsloth's native Metal implementation"; "batched SIMD-group matrix multiply" | **Mischaracterized hypothesis** | The runtime is llama.cpp's Metal backend inside Unsloth's prebuilt `b11724-mix-c208e5f` (release verified; no Metal pins — see Summary). llama.cpp Metal does run whole-ubatch batched kernels, but attributing the specific mechanism ("SIMD-group matrix multiply") to their quantized GEMM path is unverified until someone reads the fork's `ggml-metal` source at that commit. Label as hypothesis or drop the mechanism name. | +| 13 | Quote: upstream "The Mac engine does not run this model." | **Verified** | `docs/QWEN36.md:19` (checked 2026-10-19). | +| 14 | T9 "Target: ~150–200+ tok/s"; the hand-off's older "1,500–3,000+ tok/s" projection | **Unmeasured projections; the 1,500–3,000 one is unphysical** | The M2 GPU (8 cores, as the article's own hardware note says) is ≈2.9–3.6 TFLOPS fp32. Unsloth's 263.6 tok/s ≈ 2 × 3.1B × 263.6 ≈ 1.6 TFLOP/s ≈ **45–57% of the card's fp32 peak** — already a strong kernel. 1,500 tok/s would need 4.6+ TFLOP/s: beyond the card. Targets must come from the measured ≥5x-on-every-size bar (the §10 MMA precedent), never from prose. | +| 15 | 263.6 tok/s prefill (5,862 in 22.24 s); 23.8 tok/s over 2,328 tokens; 174.2 tok/s (412 in 2.37 s) | **Verified arithmetic; provenance is screenshots** | 5,862/22.24 = 263.4–263.6 ✓; 2,331/9.85 = 236.6 ✓ (the pre-walk-back chunk-2 number); 412/2.37 = 173.8 ✓. The API request records corroborate (120 s total = 22.24 s prefill + 97.8 s generation; 13 s = 2.37 + 10.5). Screenshots are not instrumented telemetry — keep saying so. | +| 16 | The 32k session numbers: prefill **259.9 tok/s**, decode **14.4 tok/s**, then the follow-up **21.2 tok/s** at 6.3k→11.3k depth | **Confounded run + clean recovery (see §3)** | 14.4 was peak contention (CPU ~99%, RAM 94% Warning, swap 1.09 → 5.91 GB in about a minute, 204°F) at only ~2.2k context — shallower than the 23.8-tok/s 8k run. The follow-up recovered to 21.2 tok/s across 5,020 generated tokens at *greater* depth with Chrome open again. Two solid facts and one confound — §3 separates them. | +| 17 | "macOS 26 / Sequoia" (hardware note) | **Internal inconsistency** | macOS 26 is Tahoe; Sequoia is macOS 15. Strata's own benches on the same machine recorded macOS 26.6.2. Pick one name. | +| 18 | Executive summary: "sustained 23.8 tok/s generation ... outperforming Strata's initial bringup prefill (~25 tok/s) by ~10×" | **Category error carried in the frame** | 263.6 (prefill) vs ~25 (prefill) is apples-to-apples only against the stale baseline (row 7); 23.8 (decode) vs 25 (prefill) is not a comparison at all. After the re-baseline, state prefill-vs-prefill and decode-vs-decode separately, with matched batch sizes (row 16's lesson: 136 tok/s on ~450 fresh tokens vs ~260 on 5.8k — prefill rate is strongly batch-size-dependent). | + +## 3. New evidence since the article was written: the 32k experiment + +All numbers in this section are from the maintainer's screenshots (Unsloth Desktop, model card telemetry page, 2026-10-19). They are user-collected evidence, not instrumented telemetry — treat them as strong observations that the reruns below should confirm. + +**The sequence.** After the article's 8.2k truncation, the maintainer raised the context manually to 32.8k on the same 24 GB M2. With Chrome closed, the model loaded with one yellow memory warning and **no failure**. The first request measured prefill **259.9 tok/s** and decode **14.4 tok/s** — while the machine was under peak contention (CPU ~99%, RAM 94% Warning, swap grew 1.09 → 5.91 GB in about a minute, 204°F), and at only ~2.2k of context. A follow-up request (5,850 prompt-cache hits, ~447 fresh tokens at 136.0 tok/s, 5,020 generated) **recovered to 21.2 tok/s sustained from ~6.3k to ~11.3k depth, with Chrome open again**. + +**What this separates:** + +- **Refuted premise:** the 8.2k wall was never an architecture limit. 32k loads and runs on 24 GB on their engine; Strata's split-K + `--kv-f16` already sustain 16k+ on the same machine. T11's "break the ceiling" premise is gone (§5). +- **Depth-independence of prefill:** 259.9 tok/s at a 32k-window session vs 263.6 at 8k — prefill rate is set by kernel/batch structure, not by context depth. This is the cleanest possible confirmation that the prefill gap is a T9/T10 problem and the deep-context problem is a different lever (T6). +- **The confound:** 14.4 tok/s at ~2.2k depth, hot and swapping, is not a "decode at depth" number; it is a "machine under contention" number. It is *shallower* than the 8k run that measured 23.8. Nobody should publish either number as a depth effect. +- **The real story:** 21.2 tok/s sustained across 5,020 tokens from 6.3k→11.3k depth, warm machine, Chrome open — the strongest decode-at-depth number in either camp's record, and it sits at ~26–30% of the M2's bandwidth ceiling at that depth (arithmetic in §4.1). ggml-metal's deep decode does **not** fall off a cliff. That is the number Strata's T6 + `--kv-f16` path has to meet, and the projection says we can (§4.1, with the assumption labeled). +- **The comparison rule this teaches:** the follow-up's prefill read 136.0 tok/s on ~447 fresh tokens (3.29 s) with 5,850 cache hits — versus ~260 on 5.8k tokens. Prefill tok/s depends strongly on batch size; fixed per-request overheads amortize. **Every cross-engine prefill comparison must match batch sizes or state them.** (It also shows, from their side, the same prefix-cache behavior our T0 shipped — see row 4.) + +**Before any 32k decode number is published:** a cool-machine rerun, Chrome closed, temperature noted in both states (204°F contended vs cool), decode tok/s sampled at matched positions (~2k, ~6k, ~11k) on both engines (§7, block 4). + +## 4. The three research questions + +### 4.1 What makes Unsloth's implementation faster than ours? + +**Prefill: dispatch structure.** Their engine evaluates a prompt in whole-ubatch GEMM kernels — one dispatch per matrix per ubatch, flash-attention over the chunk, sorted-batch MoE. Strata's chunk path still dispatches attention per row (a deliberate T2 parity decision, `q36_metal.cpp` L1274–1278) and routes experts via gathered GEMV (`push_c_gemv`/`push_c_lpr8`). Strata's own T3 measurement explains why chunking alone did not close it: at depth, chunked and token-by-token read a 4,098-token prompt in 243 vs 245 s — the binding term is the KV re-read (every query head re-reads the same cells, ~8×), which the parity contract kept identical on both paths. So the gaps decompose cleanly: + +- **Batch structure** (per-row attention + per-entry expert GEMV) → T9 then T10. The fork's pinned #264 proves the MoE part is worth real percentages even in the mature engine (+16.4–17.8% pp on Qwen3.6, decode tie) — and it is **CUDA-only**, so on Metal a good batched MoE GEMM is a differentiator, not catch-up. +- **Fixed per-dispatch overhead** → also attacked by fusing; the 136-vs-260 small-batch observation shows these overheads are real and material. +- **KV traffic at depth** → T6's GQA fold (built, awaiting its M2 measurement) plus `--kv-f16`. + +**Decode at depth: nobody is falling off a cliff — and that defines our target.** Per output token at depth L the engine reads: weights ~0.97–1.16 GB (3.1B active at ~2.5–3.0 bpw), QSA KV (10 layers × 2 × 2 KV heads × 256 dim × 2 B = 20.5 KB per token of context; 232 MB at 11.3k), and the GDN recurrent state (~31 MB/token, depth-independent: 30 layers × 32 V-heads × 128² × 2 B). At 100 GB/s: + +| Depth | Per-token reads | Bandwidth ceiling | Unsloth measured | Efficiency | +|---|---|---|---|---| +| ~0.4k | ~1.0–1.2 GB | ~83–99 tok/s | 31.2 tok/s | ~31–37% | +| ~11.3k | ~1.23–1.42 GB | ~70–81 tok/s | 21.2 tok/s | ~26–30% | + +The same ~25–35% efficiency class at both depths — their deep decode is the same kernels absorbing the KV growth, no cliff. Strata's recorded all-levers deep numbers (12.2–16.3 tok/s client-side at ≤4k depth, fp32 KV, no fold — §10 ad-hoc) sit at ~14–22% of their ceiling at those depths. The fold cuts KV traffic ~16× (8× from the fold on 16/2 geometry × 2× from f16), which raises the 11.3k-depth ceiling to ~76–89 tok/s. **If** Strata's kernels then run at the same ~25–35% efficiency class ggml-metal demonstrates, the projection is **~19–31 tok/s at 11.3k depth — parity with their measured 21.2**. The assumption is labeled: whether the efficiency class carries over to the folded path is exactly what T6's pending A/B measures (today's 14–22% may reflect occupancy or dispatch costs that folding does not remove). This is why T6's measurement jumps to the front of the queue. + +**What it is not (yet proven):** MTP (row 6 — two branches, one log line decides), and fork tuning (no Metal pins; the compute pins are CUDA/HIP/Vulkan). On the M2, we are comparing Strata against **upstream ggml-metal**. + +### 4.2 What have we done that they haven't? + +Measured or verified, item by item: + +1. **Split-K attention past the 8k threadgroup-memory ceiling** with auto-engage above 8,192 cells (T1; 32K boots and generates — M2-verified). +2. **An explicit startup working-set refusal** with an override (`STRATA_Q36M_MEM_OK=1`) — the engine refuses to allocate what the machine does not have, instead of guessing and paging (T1 scope; their Auto *guesses*, conservatively, and the guess is the thing that truncated the article's Test 2). +3. **`--kv-f16`, measured** (§9): 0.34 → 0.17 GB at 8K, 1.34 → 0.67 GB at 32K. Honest inversion worth stating publicly: llama.cpp *already defaults to f16 KV*; our fp32 default exists to keep the bit-parity verify contract, so this flag is us catching up to the conventional default — the differentiator is the fold, not f16. +4. **MTP, implemented in-tree, with an honest measured negative verdict on the M2** (T4/T4.5): 5.0–11.1 tok/s vs the 17.5–21.4 base — the verify window (three rows through 40 layers at ~15–25% acceptance) dominates, so `--mtp` stays strictly opt-in. Two footnotes: (a) the card's "1.5–2× with MTP" is llama.cpp's `draft-mtp` at `-np 1` — a different cost structure (shared embeddings, draft-n-max sweep) our verdict did not measure; the calibration experiment in §7 block 5 is the named path to revisit, and the reviewer's expectation is that a calibrated win in the 1.5–2× range is plausible once the window cost is tuned; (b) the fork pins MTP support for qwen4exp (#240/#241) but not for the Qwen3.6 arch's trimmed sidecar. +5. **The parity culture itself.** 38-stage bit-parity harness, byte-identical defaults, measured error envelopes where reassociation is unavoidable (1e-3 for MMA, ~5e-4 fp16 operand rounding). The fork's own #194 shows the demand exists in their world too: an *opt-in* exact-concurrency mode making output byte-identical regardless of neighbours, at a measured cost where stated (~9–15% on solo decode in their tables; the 35B solo figure 15.6% predates their one-launch expert path and was not re-measured — their numbers, cited as such). **We ship that contract as the default.** For a sovereign-computing audience, reproducibility-by-default is the pitch, stated in the competitor's own terms. +6. **The WY parallel scan and the MPP matrix-unit GEMM donor** (§10: 14.6×/14.9× GDN, 15.3× dense, TTFT −40–44% on short chunks) — mechanisms they do not ship for this model. +7. **Measured deep-context generation** on hardware we own, with the confounds named (this review, §3). + +### 4.3 Is a combined implementation possible? + +**Yes.** The memory math on 24 GB: ~11.4 GB resident weights (UD-Q2_K_XL) + 0.67 GB KV (f16 at 32k, §9 measured) + scratch (~59 MB WY at chunk 512, ~236 MB at 2048, ~40–80 MB MMA arena) + app → **~12.5–13 GB working set, leaving ~10 GB** for Chrome, the IDE and macOS. The 32k experiment proved the memory side empirically on their engine; ours has the same shape plus the explicit refusal gate. The blockers to their prefill speed are kernel dispatch structure (T9/T10), not memory; the deep-context decode gap is T6's pending measurement against T11's ≥6 tok/s-at-16k gate. Nothing about the combined plan requires a bigger machine — it requires the measurements first (§6 order), then the two batch-structure kernels behind parity knobs. + +## 5. Table B — the Phase 2 roadmap, ticket by ticket + +The roadmap's Phase 2 header presents T8–T11 as draft candidates for review. Verdict per ticket, judged against the measured record: + +| Ticket | Verdict | Reasoning and reframe | +|---|---|---| +| **T6** — GQA-folded attention | **Keep — and it is mislabeled in our favor** | The ticket reads as a proposal ("new idea"); the folded kernels are already built on `feature/000681`, host-parity green (38/38, forced-fold ladder 10/10, SK+GQA and MTP+GQA cross-checks 10/10), with only the deep-decode A/B pending. Reframe from "implement" to "**measure + flip decision**". Its value went *up* with the 32k evidence: it is the only lever for decode-at-depth, where Unsloth's 21.2 tok/s @ 6.3k→11.3k currently beats Strata's recorded all-levers numbers. Add the comparative target and the labeled projection from §4.1; the original gates stay unchanged. | +| **T7** — wrap-up (docs, upstream batch PR, maintenance) | **Keep as-is** | Not homework — release admin. Its dependency note (absorbs T6's numbers if T6 lands first) is already correct. | +| **T8** — KV prefix matching | **Refuted as scoped — demote from pillar** | Shipped in T0 (row 4). Residual scope, in value order: (a) measure the existing reuse (one two-request same-prefix run; expected: delta-only evaluation), (b) keep the common prefix on divergence (today's code resets fully — a real, small multi-turn improvement), (c) optional persistence across restarts (a product feature, not a perf lever). | +| **T9** — batched 2D FlashAttention for QSA chunk prefill | **Keep — the most important Phase 2 ticket, two framing fixes** | Attacks the largest measured gap (row 7). Fix 1 — targets: replace "~150–200+ tok/s" prose with the repo's measured ≥5x-on-every-size bar (§6/§10 precedent) set against the re-baselined starting point, with the mechanism named (the per-row dispatch storm; the 136-vs-260 small-batch evidence shows fixed overheads are real). Fix 2 — the parity contract: the ticket says "preserving numerical stability" but fused online-softmax attention cannot preserve T2's bit-identity contract (different accumulation order). State the resolution up front (§7 of the roadmap rewrite): opt-in knob, rel-bound envelope, 10-prompt ladder token-identity, byte-identical default, measured ≥5x bar before any default flip. | +| **T10** — batched MoE GEMM for routed experts | **Keep — second, after T9** | Correct target; the donor exists (`dense_mm.metal` → `q36_dense.metal`; `push_c_dense`/`push_c_lpr8`). Evidence boost: #264's +16.4–17.8% pp on Qwen3.6 (CUDA-only; Metal takes the existing path) makes this a genuine differentiator on Metal. Same parity-contract resolution as T9. Real GEMM shapes: ~8 routed + 1 shared experts over a 2k chunk ≈ 64 rows/expert. | +| **T11** — paged & dynamic context | **Mis-framed — reframe from "break the ceiling" to "budget predictably at the ceiling"** | The 32k evidence refutes the premise (§3): there is no measured wall a paged architecture removes. vLLM-style block tables are speculative infrastructure at these sizes; the fork's own preemption work (#184/#194's planner) exists because pool-full needs *scheduling*, not paging. Honest T11: the explicit working-set gate (already built) + the `--kv-f16` default decision + the controlled 32k comparison (§7 block 4); paging only if a measured need appears (e.g. fragmentation under long multi-session use). Its differentiator story is **predictability against Auto's conservative guess** — the same evidence. | + +## 6. What Phase 2 is missing (and the resulting order) + +1. **Re-baseline first.** Every Phase 2 target is currently set against the stale ~25 tok/s bringup number. First ticket: the fresh all-levers prefill/decode matrix on the M2 (§7 block 1). +2. **Dispatch-count instrumentation.** A one-off counter for Metal dispatches per chunk (per path: per-row attention, MoE GEMV, chunked). It quantifies the storm T9/T10 remove, gives the review a measured number instead of row 8's estimate, and proves the fix worked. The article's "20,000+ dispatches" stays unpublished until this exists. +3. **Decode-at-depth as an explicit comparative workstream.** The heaviest new fact (21.2 tok/s @ 11.3k) makes depth a headline axis; the T6 A/B + T5's deferred deep-decode matrix are that workstream — give it its own line with the comparative target. +4. **Record the MTP decision honestly, not as closed.** Our M2 verdict (window-dominated, net-negative at the shipped mechanism) stands; the calibration experiment (§7 block 5) is the named path to revisit it against the card's 1.5–2× claim. "Not pursuing further *until calibrated*" is the right wording — it is a measured decision with a named reopen condition. + +**Resulting order:** re-baseline → dispatch instrumentation → T6 A/B (flip decision) → T8 reduced to measure + divergence-prefix → T9 → T10 → T11 (reframed). + +## 7. M2 measurement plan (ready to run) + +Every recorded number carries: what, hardware (Apple M2 24 GB), build, date, and machine state (Chrome open/closed, swap in use). Anything not run ships labeled "pending M2 run". + +1. **Re-baseline prefill** — `tools/qwen35/q36_prefill_bench.py`, sizes 2048/4098, chunks 0/512, all levers on (`--prefill auto --kv-f16`, `STRATA_Q36M_GDN_WY=1 STRATA_Q36M_DENSE_MMA=1 STRATA_Q36M_ATTN_GQA=1`), fresh model load per cell. Expected metric: the honest current Strata prefill tok/s, to set against Unsloth's ~260 (batch-size-matched per §3). +2. **T6 GQA-fold A/B** — the `docs/QWEN36_METAL.md` §8 commands (decode tok/s at pos ≈4,100 and the 4,098-token read at `--ctx 16384`, fold on/off). Expected: the actual KV-traffic cut and deep-decode delta; decides the default-flip question. +3. **T8 prefix-cache reuse measurement** — a two-request same-prefix session through the serve protocol; record tokens re-evaluated vs reused (expected: delta-only), plus the divergence-reset behavior. +4. **Controlled 32k decode comparison** — cool machine, Chrome closed, both engines, decode tok/s sampled at matched ~2k/~6k/~11k positions, temperature noted. Separates the 14.4-tok/s confound from any depth effect; Strata's side gives the all-levers deep-decode number against their 21.2. +5. **MTP calibration experiment (optional, the reopen path)** — in-engine `--mtp 2 --draft-vocab N` with a `--spec-draft-n-max`-style window sweep and acceptance telemetry, np=1, to bound the verify-window cost against the card's recipe; decides whether a calibrated 1.2–2× is reachable on the M2 or the verdict stands. + +## 8. Evidence needed from the Unsloth side (for Gemini) + +Two captures settle the two biggest open questions, both one action each: + +- **E1 — the model-load engine log.** llama.cpp prints the speculative configuration and the context/fit decision at startup. One screenshot of the app's model-load log settles: (a) whether MTP was on in the benchmark session (the `spec-type`/`n_draft` lines, and the np question — the API page showed 4 slots), and (b) the Auto-context mechanism (the `n_ctx`/fit lines — the memory-fit pass vs a hardcoded default). Until then, both branches in rows 5–6 stay labeled. +- **E2 — the clean 32k rerun** (block 4's conditions). This is the number that should be in the article if any 32k decode number is. + +Optional: the app's model-config screen showing Auto vs manual context, for the Auto-policy paragraph. + +## 9. Instructions to the NockNock article (Reviewer Number 2) + +For the author of `urmanac/nocknock` → `docs/kb/how-to-strata-vs-unsloth-qwen36-metal-benchmarks.md`. The walk-back commit `b7806f1` (2026-10-19) already fixed the right things — the MTP attribution ("cannot definitively attribute") and the "static 8,192 KV table" framing. The remaining delta, in priority order. None of these require rewriting the article's narrative; each is a bounded correction. + +1. **Fix the kernel-speedup paragraphs to the recorded numbers.** Layer 1 text should read 167.7 → 11.5 ms, **14.6×** at T=512 (and 670.9 → 44.9 ms, 14.9× at T=2048); Layer 2 should read 110.2 → 7.2 ms, **15.3×**. The "487 → 26.6 ms, 18.3×" pair matches no recorded measurement — remove it or produce its source. Source: `docs/QWEN36_METAL.md` §10 (builds `4836ba99`/`3dca4b47`, M2, 2026-10-18/19). +2. **State the engine identity precisely.** It is not "Unsloth's native Metal implementation" — it is llama.cpp's Metal backend in prebuilt `b11724-mix-c208e5f` (upstream b11724 + 20 pins, none touching Metal compute; release page). "Batched SIMD-group matrix multiply" is a mechanism hypothesis — label it or verify it against the fork's `ggml-metal` at that commit. +3. **Capture the model-load log (E1)** and resolve rows 5–6 with it: MTP on or off (the 4-slot API page vs the card's np=1 recipe), and what Auto actually computed. Until then, keep "likely" hedges — the current text still carries one unverified mechanism claim. +4. **Re-run the 32k decode cool (E2)** before publishing any 32k decode number, and correct the 14.4 tok/s row: it was measured under peak contention (CPU ~99%, RAM 94% Warning, swap +4.8 GB, 204°F) at ~2.2k context — shallower than the 23.8-tok/s 8k run. The 21.2-tok/s follow-up at 6.3k→11.3k is the number worth publishing, with its conditions stated. +5. **Match batch sizes in prefill comparisons.** The follow-up measured 136.0 tok/s on ~447 fresh tokens vs ~260 on 5.8k — prefill rate depends on batch size. State the batch size next to every prefill number, or the comparison misleads. +6. **Replace the stale Strata baseline.** "~25 tok/s" is the bringup token-by-token number; the recorded pre-lever matrix reads 22.3 tok/s at 2K (§6), and the all-levers number is pending (our §7 block 1). Until it runs, write "Strata baseline: bringup build, token-by-token prefill (~25 tok/s); all-levers number pending" — or run the re-baseline and use the real number. +7. **Define the layer count once** (40 main layers per the model card: 30 GDN + 10 gated attention; Strata's 41 counts the `blk.40` MTP draft block) and fix "macOS 26 / Sequoia" → macOS 26 (Tahoe), build 26.6.2 as recorded. +8. **Correct "14+ custom .metal kernels"** — `q36.metal` alone defines 38 `kernel void k_` functions (counted 2026-10-19). +9. **Drop or instrument "over 20,000 individual dispatches."** The per-row dispatch mechanism is real (the quoted code comment is accurate — checked at `q36_metal.cpp` L1274–1278), but the count is an estimate; we are adding the instrumentation (our §6 item 2) and will supply a measured number. +10. **Fix the "~10×" frame** in the executive summary: compare prefill to prefill and decode to decode, each against the re-baselined numbers with matched batch sizes (rows 7, 16, 18). +11. **Sync the homework list** when our roadmap lands: the article's Phase 2 summary includes T8 as new work; after the correction it is a measurement of a shipped feature, and T11's premise changes (§5). + +## 10. Working with the author + +This review is written to be iterated on, not applied by force. The plan agreed with the maintainer: we forward this first draft to Gemini; he works through §9's instructions on his side (the evidence items in §8 are all captures he can make in minutes); we fill the "pending M2 run" numbers here as the blocks execute; and both documents converge before either side publishes the next revision. The MTP question is explicitly kept open with a calibration path (§4.2 item 4, §7 block 5) — the maintainer's prior is that the card's 1.5–2× claim holds or is beatable within a calibrated range, and the measurement plan is built to test exactly that. The response article in this repo (`docs/QWEN36_METAL_RESPONSE.md`) is our public-voice version; it links here for the receipts. From fd9620192a7fbd120315a5ddcbfbe5603077b710 Mon Sep 17 00:00:00 2001 From: Kingdon Barrett Date: Mon, 19 Oct 2026 22:07:43 +0100 Subject: [PATCH 2/4] docs: add response article, revise Phase 2 roadmap per the review Spec-Ref: helix-specs@6f160951:000684_add-rigor-to-research --- docs/QWEN36_METAL_RESPONSE.md | 32 +++++++++++++++++ docs/QWEN36_METAL_ROADMAP.md | 66 ++++++++++++++++++++++------------- 2 files changed, 74 insertions(+), 24 deletions(-) create mode 100644 docs/QWEN36_METAL_RESPONSE.md diff --git a/docs/QWEN36_METAL_RESPONSE.md b/docs/QWEN36_METAL_RESPONSE.md new file mode 100644 index 0000000000..de89ad686c --- /dev/null +++ b/docs/QWEN36_METAL_RESPONSE.md @@ -0,0 +1,32 @@ +# Measured Terms: A Reply on Qwen3.6 Metal + +> A response to the Unsloth-vs-Strata comparison in **urmanac/nocknock** (`docs/kb/how-to-strata-vs-unsloth-qwen36-metal-benchmarks.md`), from the people who maintain the engine it tested. Receipts and the claim-by-claim audit: [`docs/QWEN36_METAL_REVIEW.md`](QWEN36_METAL_REVIEW.md). 2026-10-19. + +The comparison article did the community a service: it put two engines on the same 24 GB M2 and published the numbers, including the unflattering ones. We accept the numbers that were measured — the ~263.6 tok/s prefill, the sustained decode, the screenshots' internal consistency all check out. This reply answers the three questions the comparison raises, on measured terms, and corrects the record where the article outran its evidence. + +## What makes their prefill faster + +Dispatch structure, not mystery. Their engine (llama.cpp's Metal backend, in the pinned prebuilt `b11724-mix-c208e5f` — no pinned commit touches Metal compute) evaluates a prompt in whole-batch GEMM kernels. Strata's chunk path still dispatches attention per row — a deliberate choice we made to keep chunked prefill bit-identical to decode, which is the contract this engine ships by default. Our own measurements say the same thing from our side: at depth, chunked and token-by-token read a 4,098-token prompt in 243 vs 245 seconds (`docs/QWEN36_METAL.md` §6), because the binding term is the KV re-read, not the batching. The gaps decompose: batch structure (their win today), fixed per-dispatch overheads, and KV traffic at depth. Two of the three are exactly what our Phase 2 kernels attack. + +One number from their side that we find more interesting than any of ours: pinned fork PR #264 sped up MoE *prompt processing* on CUDA by 16.4–17.8% for Qwen3.6 while leaving decode tied. Kernel structure is where prefill lives — on their flagship backend, on their model, with their own engineers. On Metal their fork pins nothing; the M2 comparison ran against upstream ggml-metal. That is the bar we are measuring ourselves against, and it is a fair one. + +## What we have that they don't + +- **A deep-decode lever, built and waiting on one measurement.** The model's gated-attention layers read each KV cell 8 times (16 query heads share 2 KV heads — read from the model card's published geometry). Our GQA fold reads each cell once, and `--kv-f16` halves what's left (§9: 0.34 → 0.17 GB at 8K, measured on the M2). The bandwidth arithmetic at 11.3k depth puts the ceiling at ~76–89 tok/s with both levers; their measured 21.2 tok/s there sits at ~26–30% of the *un*-folded ceiling. If our kernels reach the same ~25–35% efficiency class their engine demonstrates at both depths, the projection lands at ~19–31 tok/s — parity with the best deep-decode number in either camp's record. The assumption is labeled, and the A/B that tests it is the first thing we run. +- **A memory contract instead of a guess.** Their Auto context-sizing budgeted 8,192 tokens on a 24 GB machine that had room for 32k — the maintainer proved it by loading 32.8k manually with one warning and no failure. Our engine refuses, at startup, to allocate what the machine does not have, with an override for people who know better. Predictability is the feature; the 8.2k truncation in the article was a guess being conservative, not a ceiling. +- **Reproducibility by default.** The fork's own opt-in exact-concurrency mode (#194) exists because people want byte-identical output regardless of neighbouring requests — at a measured cost where they state one (~9–15% on solo decode in their tables). We ship that contract as the default, with a 38-stage bit-parity harness behind it. For sovereign computing, "the same prompt gives the same answer" is not a nice-to-have. +- **An honest negative result.** Our in-tree MTP measured net-negative on the M2 (5.0–11.1 tok/s vs 17.5–21.4 base; the verify window dominates at ~15–25% acceptance — `docs/QWEN36_METAL.md` §7). We publish that. The card's "~1.5–2× with MTP" is a different mechanism configuration (`-np 1`, draft-n-max sweep) our verdict did not measure, and the calibration experiment to test it is named in the review. The expectation on our side: a calibrated win in that range is plausible once the window cost is tuned — that is a testable prior, not a claim. + +## Is the combined implementation possible? + +Yes, and half of it is already proven. The 32k experiment established the memory side on their engine; ours has the same shape: ~11.4 GB weights + 0.67 GB KV at 32k (f16, measured) + ~100–350 MB scratch ≈ **12.5–13 GB working set — about 10 GB left for Chrome, the IDE, and macOS**. The prefill gap is kernel structure (T9: batched flash attention; T10: batched MoE GEMM, with a proven donor kernel and their own CUDA numbers saying what structure is worth). The deep-context gap is T6's measurement. Both engines want the same destination from opposite sides: they have prefill speed and guessy memory budgets; we have explicit budgets, bit-parity, and the deep-context architecture, with the prefill homework written down and gated on measurement. + +## The experiment we'd like to see next + +Long-context inference on this class of machine is thermally bounded — the benchmark session peaked at 204°F with 5.9 GB of swap while measuring 14.4 tok/s, then recovered to 21.2 tok/s at *greater* depth once contention eased. Fewer bytes moved per token is also less energy per token. The hypothesis, testable on hardware both projects own: **tokens per joule at 11.3k depth, folded vs unfolded KV path, cool machine, Chrome closed, `powermetrics` or the menu-bar thermometer as the meter.** If the ~16× KV-traffic cut shows up as sustained clocks at depth as well as tok/s, "runs cool all afternoon with Chrome open" becomes a measured sovereign-computing metric — and it is the one metric neither engine has published yet. + +## Where Unsloth Desktop is simply the right answer + +When you want speed today, don't need reproducibility guarantees, and are happy to let a tool guess your memory budget for you — use it. It is genuinely good at what it does; the numbers in the comparison say so, and we cite them in our own roadmap as the bar. We are building for the case where the guess is the problem: explicit budgets, verified output, measured claims, and hardware the user owns end to end. + +Both documents will be revised together — the audit, the corrections, and the pending M2 numbers live in `docs/QWEN36_METAL_REVIEW.md`, and the roadmap this reply points at (`docs/QWEN36_METAL_ROADMAP.md`) is being updated to match. The next round belongs to the measurements. diff --git a/docs/QWEN36_METAL_ROADMAP.md b/docs/QWEN36_METAL_ROADMAP.md index cba049db49..4783cfabfb 100644 --- a/docs/QWEN36_METAL_ROADMAP.md +++ b/docs/QWEN36_METAL_ROADMAP.md @@ -178,35 +178,53 @@ follow-through paths for hardware the maintainers do not have. --- -## Phase 2: Post-0.2.0 Milestones — Benchmarking Against Unsloth & Uncapped Context +## Phase 2: Post-0.2.0 Milestones — Measured Prefill Parity and Predictable Context -> **Author**: Gemini-3.8-Flash -> **Status**: Draft candidate initiatives for Helix review. -> - [ ] **Review Status**: Pending review by Qwen-3.8-Flash-Next and ticket ordering in Helix. +> **Author**: Gemini-3.8-Flash (original draft); reviewed and revised by Qwen-3.8-Flash-Next, 2026-10-19. +> **Status**: Reviewed — the claim-by-claim audit, the corrected numbers and the evidence requests are in docs/QWEN36_METAL_REVIEW.md (its §5 is this section's review table). +> - [x] **Review Status**: Reviewed by Qwen-3.8-Flash-Next; the tickets below are the post-review versions. -On 2026-10-19, empirical benchmarks were conducted on an Apple M2 (24 GB) comparing Strata's custom Metal Qwen3.6 engine against Unsloth's `Qwen3.6-35B-A3B-MTP-GGUF` (`UD-Q2_K_XL`): -- **Unsloth Highlights**: Achieved ~263.6 tok/s prefill on bulk prompts (5,862 tokens in 22.24s) via batched SIMD-group matrix multiply, and sustained 23.8 tok/s decode generation across 2,328 tokens. Demonstrated instant prefix caching (5,858 cache hits on resume). Note: while the model file contains MTP weights, proposal/acceptance telemetry was uninstrumented, so the exact speed contribution of MTP vs. fused kernel dequantization and memory bandwidth utilization remains an unverified research question. -- **Context Allocation**: In the desktop benchmark run on the 24 GB M2 under active desktop load (Chrome, IDE, etc.), the session context was budgeted at 8,192 tokens (`8.2k / 8.2k`), likely based on available free system RAM at model load time. When the 5,862 prompt tokens combined with 2,328 generated tokens reached 8,190 tokens, a resume attempt could only emit 2 additional tokens before halting mid-sentence with *"Response hit the Max Tokens limit."* (Closing concurrent heavy applications would allow desktop auto-calculators to allocate more headroom, but memory headroom remains variable). -- **Strata Advantage**: Strata's custom Split-K attention and explicit memory management (`--kv-f16` halving KV cache footprint) already sustain continuous generation at 6,000+ context depth (7.2 tok/s decode) up to 16k+ on the same 24 GB hardware without relying on massive unallocated RAM margins. However, Strata's prefill currently runs at ~25 tok/s due to parity-first row-by-row dispatch in `forward_prompt_chunk`. +On 2026-10-19, empirical benchmarks were conducted on an Apple M2 (24 GB) comparing Strata's custom Metal Qwen3.6 engine against Unsloth's `Qwen3.6-35B-A3B-MTP-GGUF` (`UD-Q2_K_XL`). The post-review state of the evidence (every number sourced in the review doc): -The following candidate initiatives form the Phase 2 roadmap to achieve prefill throughput parity with Unsloth while maintaining Strata's deep-context edge: +- **The engine is identified**: the Unsloth Desktop build is `b11724-mix-c208e5f` — upstream llama.cpp `b11724` plus 20 pinned commits, none of which touch Metal compute. On the M2, the comparison ran against upstream ggml-metal kernels (the fork's compute pins are CUDA/HIP/Vulkan only; #264's +16.4–17.8% MoE prefill gain on Qwen3.6 is CUDA-only — on Metal a good batched MoE GEMM is a differentiator, not catch-up). +- **Unsloth highlights (measured, screenshot-derived)**: ~263.6 tok/s prefill on bulk prompts (5,862 tokens in 22.24 s) and ~260 tok/s at a 32k-window session — prefill is depth-independent and batch-size-dependent (136 tok/s on ~447 fresh tokens). Decode: 31.2 tok/s shallow, 23.8 tok/s sustained over 2,328 tokens, 21.2 tok/s sustained 6.3k→11.3k depth on the follow-up. MTP on/off in the benchmark session is unverified (the card's recipe is `-np 1 --spec-type draft-mtp`; the API page showed 4 slots) — one model-load log line settles it. +- **Context allocation**: Auto budgeted 8,192 tokens under desktop load; a manual 32.8k load succeeded on the same machine (one yellow warning, no failure) — 8.2k was a conservative auto-budget, not an engine ceiling. Strata's differentiator is the explicit working-set gate, not a bigger ceiling. +- **Strata's recorded side**: the prefill baseline is stale (~25 tok/s is the bringup token-by-token number; the pre-lever matrix reads 22.3 tok/s at 2K, §6) — T12 re-baselines it before any target is set. Split-K + `--kv-f16` sustain 16k+ today; the GQA fold's A/B (T6) is the pending deep-decode lever. -### T8 — KV Cache Prefix Matching (Prompt Caching / Session Reuse) -- **Observation**: Unsloth reused 5,858 cached tokens with zero compute latency, evaluating only the fresh document delta. -- **Scope**: Implement token-prefix hashing and state reuse in `q36_metal.cpp` so multi-turn sessions and repeated prompts skip evaluating already-computed prefix tokens. -- **Gate**: 100% bit-parity against clean prefill; zero time spent on cached token prefix. +### Execution order (measurement-first; the review doc's §6) -### T9 — Batched 2D FlashAttention for QSA Chunk Prefill -- **Observation**: `push_qsa_attention` currently dispatches single-row decode attention kernels `nt` times per chunk per layer (thousands of Metal command encoder dispatches per chunk) to preserve bit-for-bit equivalence. -- **Scope**: Implement a fused 2D causal FlashAttention kernel on Metal that computes attention over the entire chunk $N \times N$ in a single dispatch while preserving numerical stability. -- **Target**: Scale prompt prefill from ~25 tok/s toward ~150–200+ tok/s on Apple Silicon. +**T12 → T13 → T6's pending A/B → T8 → T9 → T10 → T11.** The kernel tickets' targets are set against T12's numbers, not against prose. -### T10 — Batched MoE GEMM for Routed Experts in Prefill -- **Observation**: Strata routes tokens per-entry via gathered GEMV (`push_c_gemv` / `push_c_lpr8`). -- **Scope**: Group tokens in each chunk by routed expert ID and execute true batched GEMM matrix multiplications across active experts, replacing per-token dispatches with high-occupancy SIMD-group operations. +### T12 — The honest re-baseline (new, first) +- **Scope**: run the §6 matrix with every lever on (`tools/qwen35/q36_prefill_bench.py`, sizes 2048/4098, chunks 0/512, `--prefill auto --kv-f16`, `STRATA_Q36M_GDN_WY=1 STRATA_Q36M_DENSE_MMA=1 STRATA_Q36M_ATTN_GQA=1`), fresh model load per cell; record prefill and decode with hardware, build, date, and machine state. +- **Gate**: the numbers land in docs/QWEN36_METAL.md; the targets below are restated against them. No default flips in this ticket. -### T11 — Paged & Dynamic Context Architecture (Sustained Context > 8.2k) -- **Observation**: Standard desktop runtimes budget context dynamically based on free system RAM at launch, risking truncation under concurrent desktop workloads. -- **Scope**: Formalize Strata's split-K attention, dynamic KV chunking, and `--kv-f16` memory budgeting into an official paged context architecture that guarantees sustained generation across 16k–32k context on 24GB Unified Memory regardless of background application memory pressure. -- **Gate**: Zero truncation errors on prompts exceeding 8k tokens; sustained decode > 6 tok/s at 16k context depth. +### T13 — Dispatch-count instrumentation (new, second) +- **Scope**: a one-off counter for Metal dispatches per chunk, per path (per-row attention, MoE GEMV, chunked) — a stderr counter, no behavior change. +- **Gate**: the measured per-chunk dispatch counts recorded per path; replaces the unverified "20,000+ dispatches" estimate in the comparison narrative; T9/T10 cite before/after counts as their evidence. + +### T8 — Prefix cache: measure the shipped feature, keep the common prefix on divergence (revised) +- **Observation (corrected)**: token-prefix matching and state reuse shipped in T0 (`q36_metal.cpp` L2419–2429; docs/QWEN36_METAL.md §2.2). The comparison article's "Cache Hits: 0 (Fresh prefill)" row was a fresh request, not a missing feature. +- **Scope**: (a) measure the existing reuse — one two-request same-prefix session through the serve protocol (expected: delta-only evaluation); (b) keep the common prefix on divergence — today a divergence resets the state fully; (c) optional, its own decision: persistence across restarts. +- **Gate**: the measured reuse numbers recorded; the divergence case keeps the common prefix and re-evaluates only the divergent tail; the 10-prompt parity unchanged. + +### T9 — Batched FlashAttention for QSA chunk prefill (targets revised) +- **Observation (kept)**: `push_qsa_attention` dispatches the decode path's per-row attention sequence `nt` times per chunk per layer (`q36_metal.cpp` L1274–1278) — the dispatch storm T13 counts. +- **Scope**: a fused causal 2D attention kernel for the chunk path behind an opt-in knob (`STRATA_Q36M_ATTN_FLASH=1`, the env pattern read once per process so the MTP verify window never mixes paths), with a measured rel-bound envelope. +- **Parity contract (the honest version)**: fused online-softmax attention cannot preserve the T2 bit-identity contract — the accumulation order differs by design. The gate set follows the §10 MMA precedent: rel-bound envelope, 10-prompt ladder token-identity (greedy, 128 tokens), byte-identical no-flag default, and the **measured ≥5x-on-every-size bar** (§6) before any default flip. +- **Target**: set by T12's re-baseline and the ≥5x bar — not prose. The mechanism direction is confirmed from both sides: batch structure is where prefill lives (our T3 numbers; their pinned #264 on CUDA). + +### T10 — Batched MoE GEMM for routed experts (donor-reuse) +- **Observation (kept)**: routed experts run gathered GEMV (`push_c_gemv` / `push_c_lpr8`). +- **Scope**: group the chunk's tokens by routed expert ID and run a batched GEMM per active expert, reusing the `q36_dense.metal` donor (`push_c_dense`'s dequant-once, matrix-unit path). Same knob + rel-bound envelope + ladder gates as T9; strictly opt-in until the measured bar clears. +- **Context**: the fork's pinned #264 (+16.4–17.8% pp on Qwen3.6, decode tie) is CUDA-only — Metal takes the existing path, so this is a differentiator opportunity, not catch-up. + +### T11 — Predictable context budgets (reframed) +- **Observation (corrected)**: there is no measured 8.2k wall — the 32k load succeeded on 24 GB on both engines. The original "paged & dynamic context" premise is refuted; vLLM-style paging is speculative infrastructure at these sizes. +- **Scope**: keep and document the explicit working-set gate as the differentiator (refusal with override, already built); decide the `--kv-f16` default on the measured record (§9's numbers are in; the flip decision rides on T6's A/B matrix); run the controlled 32k comparison (cool machine, Chrome closed, decode sampled at matched ~2k/6k/11k positions, both engines). Paging only if a measured need appears (e.g. fragmentation under long multi-session use). +- **Gate**: zero truncation at the gate-approved budget on 24 GB; sustained decode > 6 tok/s at 16k depth measured (T6's A/B feeds this); the comparison table recorded in docs/QWEN36_METAL.md. + +### Decode-at-depth workstream (owned by T6's pending A/B; tracked here for ordering) + +The comparison's heaviest number is 21.2 tok/s sustained at 6.3k→11.3k depth. The bandwidth arithmetic (review doc §4.1): with the fold + `--kv-f16` the 11.3k-depth ceiling rises to ~76–89 tok/s; at the ~25–35% efficiency class their engine demonstrates at both depths, that is ~19–31 tok/s — parity, *if* the efficiency class carries over to the folded path (the labeled assumption the A/B tests; Strata's recorded deep numbers today run ~14–22% of ceiling at ≤4k depth). Ticket: T6 (Phase 1) — its A/B is Phase 2's third gate; T5's deferred deep-decode matrix folds into the same run. The optional tokens-per-joule experiment (energy at depth, `powermetrics`, cool machine) rides on the same A/B — review doc §4.3. From 7258fe88fb1f0726c77c1abcc2ac0da4d594852b Mon Sep 17 00:00:00 2001 From: Kingdon Barrett Date: Mon, 19 Oct 2026 22:08:38 +0100 Subject: [PATCH 3/4] docs: pin scratch-size figures to the recorded values Spec-Ref: helix-specs@42a2b6b8:000684_add-rigor-to-research --- docs/QWEN36_METAL_RESPONSE.md | 2 +- docs/QWEN36_METAL_REVIEW.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/QWEN36_METAL_RESPONSE.md b/docs/QWEN36_METAL_RESPONSE.md index de89ad686c..0e905f808d 100644 --- a/docs/QWEN36_METAL_RESPONSE.md +++ b/docs/QWEN36_METAL_RESPONSE.md @@ -19,7 +19,7 @@ One number from their side that we find more interesting than any of ours: pinne ## Is the combined implementation possible? -Yes, and half of it is already proven. The 32k experiment established the memory side on their engine; ours has the same shape: ~11.4 GB weights + 0.67 GB KV at 32k (f16, measured) + ~100–350 MB scratch ≈ **12.5–13 GB working set — about 10 GB left for Chrome, the IDE, and macOS**. The prefill gap is kernel structure (T9: batched flash attention; T10: batched MoE GEMM, with a proven donor kernel and their own CUDA numbers saying what structure is worth). The deep-context gap is T6's measurement. Both engines want the same destination from opposite sides: they have prefill speed and guessy memory budgets; we have explicit budgets, bit-parity, and the deep-context architecture, with the prefill homework written down and gated on measurement. +Yes, and half of it is already proven. The 32k experiment established the memory side on their engine; ours has the same shape: ~11.4 GB weights + 0.67 GB KV at 32k (f16, measured) + ~100–330 MB scratch depending on chunk size (§10) ≈ **12.5–13 GB working set — about 10 GB left for Chrome, the IDE, and macOS**. The prefill gap is kernel structure (T9: batched flash attention; T10: batched MoE GEMM, with a proven donor kernel and their own CUDA numbers saying what structure is worth). The deep-context gap is T6's measurement. Both engines want the same destination from opposite sides: they have prefill speed and guessy memory budgets; we have explicit budgets, bit-parity, and the deep-context architecture, with the prefill homework written down and gated on measurement. ## The experiment we'd like to see next diff --git a/docs/QWEN36_METAL_REVIEW.md b/docs/QWEN36_METAL_REVIEW.md index f4e76d54ec..1242c42bca 100644 --- a/docs/QWEN36_METAL_REVIEW.md +++ b/docs/QWEN36_METAL_REVIEW.md @@ -104,7 +104,7 @@ Measured or verified, item by item: ### 4.3 Is a combined implementation possible? -**Yes.** The memory math on 24 GB: ~11.4 GB resident weights (UD-Q2_K_XL) + 0.67 GB KV (f16 at 32k, §9 measured) + scratch (~59 MB WY at chunk 512, ~236 MB at 2048, ~40–80 MB MMA arena) + app → **~12.5–13 GB working set, leaving ~10 GB** for Chrome, the IDE and macOS. The 32k experiment proved the memory side empirically on their engine; ours has the same shape plus the explicit refusal gate. The blockers to their prefill speed are kernel dispatch structure (T9/T10), not memory; the deep-context decode gap is T6's pending measurement against T11's ≥6 tok/s-at-16k gate. Nothing about the combined plan requires a bigger machine — it requires the measurements first (§6 order), then the two batch-structure kernels behind parity knobs. +**Yes.** The memory math on 24 GB: ~11.4 GB resident weights (UD-Q2_K_XL) + 0.67 GB KV (f16 at 32k, §9 measured) + scratch (~59 MB WY at chunk 512, ~236 MB at 2048, ~40 MB MMA arena at the attn_qkv shape — §10) + app → **~12.5–13 GB working set, leaving ~10 GB** for Chrome, the IDE and macOS. The 32k experiment proved the memory side empirically on their engine; ours has the same shape plus the explicit refusal gate. The blockers to their prefill speed are kernel dispatch structure (T9/T10), not memory; the deep-context decode gap is T6's pending measurement against T11's ≥6 tok/s-at-16k gate. Nothing about the combined plan requires a bigger machine — it requires the measurements first (§6 order), then the two batch-structure kernels behind parity knobs. ## 5. Table B — the Phase 2 roadmap, ticket by ticket From 8b9c9d9024ba3aff54f8dcc012ac8a46999b9023 Mon Sep 17 00:00:00 2001 From: Kingdon Barrett Date: Mon, 19 Oct 2026 23:16:09 +0100 Subject: [PATCH 4/4] docs: scoreboard, falsifiability, risk register, decides-lines - the convergence pass Spec-Ref: helix-specs@d894740b:000684_add-rigor-to-research --- docs/QWEN36_METAL_RESPONSE.md | 2 +- docs/QWEN36_METAL_REVIEW.md | 81 ++++++++++++++++++++++++++++++++--- docs/QWEN36_METAL_ROADMAP.md | 10 ++--- 3 files changed, 81 insertions(+), 12 deletions(-) diff --git a/docs/QWEN36_METAL_RESPONSE.md b/docs/QWEN36_METAL_RESPONSE.md index 0e905f808d..61a35f8bbc 100644 --- a/docs/QWEN36_METAL_RESPONSE.md +++ b/docs/QWEN36_METAL_RESPONSE.md @@ -29,4 +29,4 @@ Long-context inference on this class of machine is thermally bounded — the ben When you want speed today, don't need reproducibility guarantees, and are happy to let a tool guess your memory budget for you — use it. It is genuinely good at what it does; the numbers in the comparison say so, and we cite them in our own roadmap as the bar. We are building for the case where the guess is the problem: explicit budgets, verified output, measured claims, and hardware the user owns end to end. -Both documents will be revised together — the audit, the corrections, and the pending M2 numbers live in `docs/QWEN36_METAL_REVIEW.md`, and the roadmap this reply points at (`docs/QWEN36_METAL_ROADMAP.md`) is being updated to match. The next round belongs to the measurements. +This exchange has since converged once, end to end: the comparison article applied the full audit in two merged revisions (eleven instructions, then the five polish nits — commits `c179a7e`, `039269b`), and the roadmap in this repo is rewritten measurement-first. What remains on both sides is measurements, not words: the run blocks in the review's §7, the two captures in its §8, and the scoreboard in its §11 — the table that holds every comparative number with its status, ready to be re-checked when the runs land. diff --git a/docs/QWEN36_METAL_REVIEW.md b/docs/QWEN36_METAL_REVIEW.md index 1242c42bca..a689fb537d 100644 --- a/docs/QWEN36_METAL_REVIEW.md +++ b/docs/QWEN36_METAL_REVIEW.md @@ -6,6 +6,8 @@ > 2. `docs/kb/how-to-strata-vs-unsloth-qwen36-metal-benchmarks.md` in **urmanac/nocknock** — the "Standing on the Shoulders of Open-Source Giants" article > > **Method.** Every technical claim in both documents was checked against the measured record in this repository (`docs/QWEN36_METAL.md` §6–§10 and the recorded gate numbers, `bench/results/2026-10-18-m2-q36-prefill`), the code (`metal/`), the model card (`unsloth/Qwen3.6-35B-A3B-MTP-GGUF` on Hugging Face), the engine build's release notes (`unslothai/llama.cpp` release `b11724-mix-c208e5f`, published 2026-10-19 16:12 UTC), the fork's pull requests, and the benchmark-session screenshots the maintainer supplied on 2026-10-19. Where a claim could not be checked, this review says so and labels it an assumption. Nothing here is asserted without a measurement or a source; inferences from reading code are marked "static analysis". +> +> **Revision (2026-10-19, second pass).** The article's two revision rounds are applied and verified — §9.1 records them item by item (PR #69 `c179a7e`: instructions 1–11; PR #70 `039269b`: five polish nits). Added in this pass: §7 now states what each measurement decides and gains the hours-class ablation matrix (block 6); §11 the scoreboard (every comparative number with its status); §12 falsifiability (what would change our mind, with thresholds); §13 the risk register for the kernel tickets; §14 the notation table. ## 0. Summary @@ -130,13 +132,16 @@ The roadmap's Phase 2 header presents T8–T11 as draft candidates for review. V ## 7. M2 measurement plan (ready to run) -Every recorded number carries: what, hardware (Apple M2 24 GB), build, date, and machine state (Chrome open/closed, swap in use). Anything not run ships labeled "pending M2 run". +Every recorded number carries: what, hardware (Apple M2 24 GB), build, date, and machine state (Chrome open/closed, swap in use). Anything not run ships labeled "pending M2 run". Each block states what it decides — a measurement that changes nothing is not on the list. + +1. **Re-baseline prefill** — `tools/qwen35/q36_prefill_bench.py`, sizes 2048/4098, chunks 0/512, all levers on (`--prefill auto --kv-f16`, `STRATA_Q36M_GDN_WY=1 STRATA_Q36M_DENSE_MMA=1 STRATA_Q36M_ATTN_GQA=1`), fresh model load per cell. **Decides:** the base every Phase 2 target is set against. If all-levers prefill at 2K chunk-512 lands under ~2× the pre-lever 22.3 tok/s, investigate lever interaction (occupancy) before building kernels; the ≥5x bar re-anchors to the measured base either way. +2. **T6 GQA-fold A/B** — the `docs/QWEN36_METAL.md` §8 commands (decode tok/s at pos ≈4,100 and the 4,098-token read at `--ctx 16384`, fold on/off). **Decides:** the fold's default and the `--kv-f16` flip. A ≥1.5× deep-decode gain with all gates green schedules the flip behind the roadmap's soak; 1.2–1.5× keeps it opt-in with numbers recorded; <1.2× kills the KV-traffic hypothesis at these depths (§12, item 2) and the projection in §4.1 dies with it. +3. **T8 prefix-cache reuse measurement** — a two-request same-prefix session through the serve protocol; record tokens re-evaluated vs reused (expected: delta-only), plus the divergence-reset behavior. **Decides:** what the narrative may claim about caching (delta-only, measured rather than asserted), and whether the divergence-prefix change proceeds — it does if the 10-prompt parity holds twice (a behavior change on the existing prefix path). +4. **Controlled 32k decode comparison** — cool machine, Chrome closed, both engines, decode tok/s sampled at matched ~2k/~6k/~11k positions, temperature noted. **Decides:** the decode-at-depth table both documents publish (their side pending the cool rerun, §8 E2; ours the matched-position record); retires the 14.4-tok/s confound from the record for good. +5. **MTP calibration experiment (optional, the reopen path)** — in-engine `--mtp 2 --draft-vocab N` with a `--spec-draft-n-max`-style window sweep and acceptance telemetry, np=1, to bound the verify-window cost against the card's recipe. **Decides:** reopen or close. A calibrated ≥1.2× at np=1 on the M2 reopens a window-calibration ticket; otherwise the net-negative verdict stands recorded as final, with the sweep as its evidence. +6. **The hours-class ablation matrix (the stretch)** — `q36_prefill_bench.py` with the levers added one at a time (none → +WY → +MMA → +GQA fold → +f16 KV → all), sizes 2K/4K/8K, chunk 512, fresh load per cell. Purpose: one clean table attributing which lever bought what — today that history lives in separate builds and sessions (the §6 pre-lever matrix, §8's fold commands, §9's f16 numbers, §10's component micro-benches). Run once, cite forever; T7's docs and the response's claims cite this table. **Decides:** the attribution story, and T9's honest starting point at every size. -1. **Re-baseline prefill** — `tools/qwen35/q36_prefill_bench.py`, sizes 2048/4098, chunks 0/512, all levers on (`--prefill auto --kv-f16`, `STRATA_Q36M_GDN_WY=1 STRATA_Q36M_DENSE_MMA=1 STRATA_Q36M_ATTN_GQA=1`), fresh model load per cell. Expected metric: the honest current Strata prefill tok/s, to set against Unsloth's ~260 (batch-size-matched per §3). -2. **T6 GQA-fold A/B** — the `docs/QWEN36_METAL.md` §8 commands (decode tok/s at pos ≈4,100 and the 4,098-token read at `--ctx 16384`, fold on/off). Expected: the actual KV-traffic cut and deep-decode delta; decides the default-flip question. -3. **T8 prefix-cache reuse measurement** — a two-request same-prefix session through the serve protocol; record tokens re-evaluated vs reused (expected: delta-only), plus the divergence-reset behavior. -4. **Controlled 32k decode comparison** — cool machine, Chrome closed, both engines, decode tok/s sampled at matched ~2k/~6k/~11k positions, temperature noted. Separates the 14.4-tok/s confound from any depth effect; Strata's side gives the all-levers deep-decode number against their 21.2. -5. **MTP calibration experiment (optional, the reopen path)** — in-engine `--mtp 2 --draft-vocab N` with a `--spec-draft-n-max`-style window sweep and acceptance telemetry, np=1, to bound the verify-window cost against the card's recipe; decides whether a calibrated 1.2–2× is reachable on the M2 or the verdict stands. +## 8. Evidence needed from the Unsloth side (for Gemini) ## 8. Evidence needed from the Unsloth side (for Gemini) @@ -163,6 +168,70 @@ For the author of `urmanac/nocknock` → `docs/kb/how-to-strata-vs-unsloth-qwen3 10. **Fix the "~10×" frame** in the executive summary: compare prefill to prefill and decode to decode, each against the re-baselined numbers with matched batch sizes (rows 7, 16, 18). 11. **Sync the homework list** when our roadmap lands: the article's Phase 2 summary includes T8 as new work; after the correction it is a measurement of a shipped feature, and T11's premise changes (§5). +**Application status (2026-10-19, verified against the merged article).** The author applied all eleven instructions and merged them into `urmanac/nocknock` main via PR #69 (commit `c179a7e`). Verified in the fetched post-merge text, item by item: the §10-exact speedup numbers with sources; the engine identity (upstream Metal backend, no Metal pins); the two-branch MTP paragraph with the E1 pointer; Test 3 (the 32k experiment) with the contention confound separated from the 21.2 recovery; batch sizes attached to every prefill number; the baselines footnoted as bringup/pending T12; "40 main + 1 draft" and macOS 26 (Tahoe) 26.6.2; 38 kernels; the dispatch estimate labeled "unconfirmed static analysis estimate pending ticket T13 instrumentation"; prefill and decode compared category by category; and the roadmap section rewritten to the measurement-first order with T8's "shipped in T0" clarification. A second round applied five polish nits via PR #70 (commit `039269b`): §4's opening question now states all three batch sizes; Test 2's 7.2 tok/s is footnoted as bringup-era with the recorded equivalents (8 tok/s at pos ~4,100 pre-fold; 12.2–16.3 all-levers at ≤4k depth); T9's envelope reads "measured; the §10 MMA precedent was 1e-3"; the KV footprint reads "a measured 0.34 GB (0.17 at 8k, 0.67 at 32k; §9)"; the summary's baseline range separates the two mechanisms. The article also now carries the bandwidth-efficiency ledger with the same projection and the same labeled assumption as §4.1. **Outstanding on both sides: the two captures in §8 (E1, E2) and the run blocks in §7 — everything else asked has been applied.** + ## 10. Working with the author This review is written to be iterated on, not applied by force. The plan agreed with the maintainer: we forward this first draft to Gemini; he works through §9's instructions on his side (the evidence items in §8 are all captures he can make in minutes); we fill the "pending M2 run" numbers here as the blocks execute; and both documents converge before either side publishes the next revision. The MTP question is explicitly kept open with a calibration path (§4.2 item 4, §7 block 5) — the maintainer's prior is that the card's 1.5–2× claim holds or is beatable within a calibrated range, and the measurement plan is built to test exactly that. The response article in this repo (`docs/QWEN36_METAL_RESPONSE.md`) is our public-voice version; it links here for the receipts. + +The loop has since run once, end to end: both article revisions are applied and verified (§9.1). What remains on both sides is measurements, not words — tracked cell by cell in §11's scoreboard, with the decision rules in §7 and the kernel risks in §13. + +## 11. The scoreboard (the state of every comparative number) + +One table for everything in play. The pending cells are the work; every filled cell is traceable to a section above or a recorded doc §. Re-check this table when the runs land — it is the agreement both documents describe. + +| Metric | Unsloth (measured) | Strata (recorded) | Strata (projected) | Status | +|---|---|---|---|---| +| Prefill, bulk 5.8k tokens | 263.6 tok/s (N=5,862) | bringup ~25 token-by-token; 22.3 tok/s @2K chunk-512 pre-lever | — | theirs screenshot-derived; ours pending T12 | +| Prefill, small batch (~450) | 136.0 tok/s | — | — | pending T12; batch-size rule (§3) | +| Prefill vs context depth | ~260 at 8k and 32k windows alike | — | n/a (structure-bound, §3) | verified twice on their side | +| Decode, shallow (~0.4k) | 31.2 tok/s | 16.6 (article Test 1); 17.5–21.4 (T4/T4.5 gate bases) | — | both sides measured | +| Decode, 8k window | 23.8 tok/s (2,328 tokens) | — | — | pending T12/T6 matrix | +| Decode, 11.3k depth | 21.2 tok/s (5,020 tokens; contention-recovered) | 12.2–16.3 all-levers at ≤4k depth only | ~19–31 with fold+f16 at the 25–35% class (assumption labeled, §4.1) | the headline cell — E2 + T6 A/B | +| Decode, 16k depth | — | 7.2 @ >6k (bringup-era); 8 @ ~4.1k (pre-fold) | T11's gate: >6 sustained, measured | T6 A/B feeds it | +| QSA KV bytes per context token | n/a (f16 default, batched) | fp32 default 41 KB; `--kv-f16` 20.5 KB | fold+f16 2.6 KB — 16× vs today's default | arithmetic from the card's geometry; A/B pending | +| Memory, 32k working set | loads and runs on 24 GB (one yellow warning) | same by math (~12.5–13 GB, §4.3) + the refusal gate | — | their side proven; ours gate-tested at 16k | +| Bus efficiency class | 26–37% across two depths | 14–22% at ≤4k depth (all-levers, fp32 KV) | 25–35% (target) | the carry-over is the A/B question | +| Dispatches per 512-token chunk | n/a (whole-ubatch kernels) | ~20,480 (static analysis, labeled) | measured by T13 | pending | +| MTP in the benchmark session | on/off unresolved — one log line (§8 E1) | net-negative as shipped (5.0–11.1 vs 17.5–21.4 base) | the calibration sweep is optional (§7 block 5) | open by one capture | + +## 12. What would change our mind (falsifiability) + +The projections above are testable. What would kill each, so nobody has to guess where the argument stands after a run: + +1. **"The prefill gap is dispatch structure" (§4.1).** Killed if T13's counts come back small (attention + MoE together under ~5k dispatches per 512-chunk) or if the T9 A/B cuts the count by >10× yet buys <1.5× at every size — then the binding term is not dispatch overhead, and the effort re-points at dequant or bandwidth behavior. +2. **"The ~25–35% efficiency class carries over to the folded path" (§4.1).** Killed if the T6 A/B shows <1.2× deep-decode gain from the fold — KV traffic is then not the binding term at these depths; the GDN state reads and the weight stream get re-examined before more kernel work. +3. **"8.2k was a conservative guess, not a ceiling" (§3).** Survived its test (32k loads on 24 GB on their engine; ours gate-tested at 16k). The residual risk is thermal, and it belongs to the tokens-per-joule run and E2 — not to a paging redesign. +4. **"A calibrated MTP ≥1.2× at np=1 is plausible" (§4.2, the maintainer's stated prior).** Killed if §7 block 5's window sweep cannot clear 1.2× on the M2 — the net-negative verdict then stands recorded as final. +5. **"Fewer KV bytes → measurably cooler sustained sessions" (the response's proposed experiment).** Killed if `powermetrics` shows under ~5% difference in sustained clocks or watts between the folded and unfolded paths at matched tok/s — the energy pitch then comes out of the response. + +## 13. Risk register for the kernel tickets (T9/T10) + +What could stop the two kernel tickets, and the fallback rungs — written down before anyone starts, so a stall has a documented next move instead of a silence: + +| Risk | Why it is real | Mitigation / fallback | +|---|---|---| +| T9: threadgroup memory at head dim 256 | A 512-key tile of K or V at D=256 does not fit where the T1 8k-cell ceiling already bit | Tile rows × keys with a two-pass score stage through scratch (the SK pair's own pattern) | +| T9: the ≥5× bar may only clear at large chunks | Fixed overheads amortize (the 136-vs-260 rule) — a small chunk may never clear 5× | Auto-engage per chunk size (the SK precedent); the bar is per-size, and small chunks keep the current path | +| T9: the rel-bound envelope drifts with chunk length | Softmax reassociation error grows with the summed range | Measure the bound per size in the ablation matrix (§7 block 6); the ladder identity arbitrates; fp32 accumulation keeps it in the MMA's ~1e-3 class (expected, to be measured) | +| T9: full fusion stalls entirely | A 2D flash kernel is the biggest kernel in the queue | Fallback rung: multi-row blocks through the existing row-blocked SK pair — dispatches ÷ B at the same envelope class, no new kernel family; a lower-risk first step on the way to full fusion either way | +| T10: expert imbalance | ~64 rows/expert at a 2k chunk, ~16 at 512 — tail experts get 1–2 rows and a GEMM degenerates to a GEMV | Hybrid dispatch: batched GEMM above a measured row threshold, the existing GEMV below it; the threshold comes from T13's rows-per-expert histogram | +| T10: dequant amortization | The batched win exists only if the per-expert dequant is reused across enough rows | The same histogram decides; below the threshold the GEMV path is already at its floor | +| Both: the parity contract | Any ladder miss on the 10 prompts (twice) | The knob stays off and the default stays byte-identical — the contract is not negotiable; the timeline is | + +## 14. Notation + +| Term | Meaning | +|---|---| +| QSA | The gated-attention layers (10 of 40; 16 query / 2 KV heads, head dim 256) | +| GDN | Gated Delta Net recurrent layers (30 of 40; recurrent state ~31 MB/token total) | +| WY | The parallel prefix-scan form of the GDN recurrence (§10: 14.6–14.9×) | +| SK | Split-K attention (T1; auto-engages past the 8k threadgroup-memory ceiling) | +| MMA | The MPP matrix-unit dense GEMM donor (§10: 15.3×) | +| H / HKV | Query heads / KV heads (16 / 2 on the attention layers — the 8× fold factor) | +| ubatch | llama.cpp's physical batch unit; whole-ubatch kernels are their prefill advantage | +| pp / tg | Prompt processing / text generation (llama.cpp benchmark terms) | +| `--kv-f16` | 16-bit KV cache (§9: 0.34 → 0.17 GB at 8k, 1.34 → 0.67 GB at 32k) | +| TTFT | Time to first token | +| np | llama.cpp parallel slots (the benchmark session's API page showed 4; MTP needs np=1) | +| E1 / E2 | The two evidence captures (§8) | diff --git a/docs/QWEN36_METAL_ROADMAP.md b/docs/QWEN36_METAL_ROADMAP.md index 4783cfabfb..c318f38e6b 100644 --- a/docs/QWEN36_METAL_ROADMAP.md +++ b/docs/QWEN36_METAL_ROADMAP.md @@ -193,14 +193,14 @@ On 2026-10-19, empirical benchmarks were conducted on an Apple M2 (24 GB) compar ### Execution order (measurement-first; the review doc's §6) -**T12 → T13 → T6's pending A/B → T8 → T9 → T10 → T11.** The kernel tickets' targets are set against T12's numbers, not against prose. +**T12 → T13 → T6's pending A/B → T8 → T9 → T10 → T11.** The kernel tickets' targets are set against T12's numbers, not against prose. Every measurement block in the review doc's §7 carries a **Decides** line — what each outcome changes. The kernel tickets' risk register and fallback rungs are the review's §13; the scoreboard that tracks every comparative number (measured, recorded, projected, pending) is its §11. ### T12 — The honest re-baseline (new, first) -- **Scope**: run the §6 matrix with every lever on (`tools/qwen35/q36_prefill_bench.py`, sizes 2048/4098, chunks 0/512, `--prefill auto --kv-f16`, `STRATA_Q36M_GDN_WY=1 STRATA_Q36M_DENSE_MMA=1 STRATA_Q36M_ATTN_GQA=1`), fresh model load per cell; record prefill and decode with hardware, build, date, and machine state. +- **Scope**: run the §6 matrix with every lever on (`tools/qwen35/q36_prefill_bench.py`, sizes 2048/4098, chunks 0/512, `--prefill auto --kv-f16`, `STRATA_Q36M_GDN_WY=1 STRATA_Q36M_DENSE_MMA=1 STRATA_Q36M_ATTN_GQA=1`), fresh model load per cell; record prefill and decode with hardware, build, date, and machine state. The full hours-class ablation matrix — levers added one at a time — is the review's §7 block 6, the stretch that ends the which-lever-bought-what question. - **Gate**: the numbers land in docs/QWEN36_METAL.md; the targets below are restated against them. No default flips in this ticket. ### T13 — Dispatch-count instrumentation (new, second) -- **Scope**: a one-off counter for Metal dispatches per chunk, per path (per-row attention, MoE GEMV, chunked) — a stderr counter, no behavior change. +- **Scope**: a one-off counter for Metal dispatches per chunk, per path (per-row attention, MoE GEMV, chunked) — counted at the single dispatch choke point (the command-encoder wrapper), one stderr line per chunk, no behavior change. The static ~20,480 estimate it replaces sits labeled in the review's §11 scoreboard. - **Gate**: the measured per-chunk dispatch counts recorded per path; replaces the unverified "20,000+ dispatches" estimate in the comparison narrative; T9/T10 cite before/after counts as their evidence. ### T8 — Prefix cache: measure the shipped feature, keep the common prefix on divergence (revised) @@ -212,12 +212,12 @@ On 2026-10-19, empirical benchmarks were conducted on an Apple M2 (24 GB) compar - **Observation (kept)**: `push_qsa_attention` dispatches the decode path's per-row attention sequence `nt` times per chunk per layer (`q36_metal.cpp` L1274–1278) — the dispatch storm T13 counts. - **Scope**: a fused causal 2D attention kernel for the chunk path behind an opt-in knob (`STRATA_Q36M_ATTN_FLASH=1`, the env pattern read once per process so the MTP verify window never mixes paths), with a measured rel-bound envelope. - **Parity contract (the honest version)**: fused online-softmax attention cannot preserve the T2 bit-identity contract — the accumulation order differs by design. The gate set follows the §10 MMA precedent: rel-bound envelope, 10-prompt ladder token-identity (greedy, 128 tokens), byte-identical no-flag default, and the **measured ≥5x-on-every-size bar** (§6) before any default flip. -- **Target**: set by T12's re-baseline and the ≥5x bar — not prose. The mechanism direction is confirmed from both sides: batch structure is where prefill lives (our T3 numbers; their pinned #264 on CUDA). +- **Target**: set by T12's re-baseline and the ≥5x bar — not prose. The mechanism direction is confirmed from both sides: batch structure is where prefill lives (our T3 numbers; their pinned #264 on CUDA). The risk register and the fallback rung (multi-row blocks through the row-blocked SK pair — dispatches ÷ B at the same envelope class) are the review's §13. ### T10 — Batched MoE GEMM for routed experts (donor-reuse) - **Observation (kept)**: routed experts run gathered GEMV (`push_c_gemv` / `push_c_lpr8`). - **Scope**: group the chunk's tokens by routed expert ID and run a batched GEMM per active expert, reusing the `q36_dense.metal` donor (`push_c_dense`'s dequant-once, matrix-unit path). Same knob + rel-bound envelope + ladder gates as T9; strictly opt-in until the measured bar clears. -- **Context**: the fork's pinned #264 (+16.4–17.8% pp on Qwen3.6, decode tie) is CUDA-only — Metal takes the existing path, so this is a differentiator opportunity, not catch-up. +- **Context**: the fork's pinned #264 (+16.4–17.8% pp on Qwen3.6, decode tie) is CUDA-only — Metal takes the existing path, so this is a differentiator opportunity, not catch-up. The imbalance case (tail experts at 1–2 rows, ~64 rows/expert at a 2k chunk but ~16 at 512) keeps the existing GEMV below a measured row threshold — the threshold comes from T13's rows-per-expert histogram (review §13). ### T11 — Predictable context budgets (reframed) - **Observation (corrected)**: there is no measured 8.2k wall — the 32k load succeeded on 24 GB on both engines. The original "paged & dynamic context" premise is refuted; vLLM-style paging is speculative infrastructure at these sizes.