feat(tools): hrx2kineto — HRX dispatch shapes and a TraceLens extension

tools/hyperloom/README.md

@@ -30,6 +30,8 @@ workload is ZAYA1-8B Q4_K_M decode.
 | `lemonade_bench.sh` | Lemonade's own benchmark (`lemonade bench`, the settings of Lemonade's nightly benchmark-regression CI: chat-short + code-short, 1 warmup, 5 runs, ctx 4096) on the fork's `llama-server`, routed through `llamacpp.vulkan_bin` of a private lemond. Lemonade reserves `-dev`, so the device goes in as `LLAMA_ARG_DEVICE` (default HRX0). |
 | `launch.sh` | Starts `hyperloom ... optimize` for ZAYA1-8B, detached. |
 | `zaya1-8b-q4_k_m.reference.json` | Gate reference from the unmodified tree (fork 3b954f0). |
+| `hrx2kineto.py` | Converts an HRX dispatch profile to a Kineto trace (Hyperloom's profile step, TraceLens, rocprof studio). With the fork's shape log it names every dispatch by its ggml op, shapes and layer and doubles as a TraceLens extension (roofline, categories, TraceDiff). |
+| `hrx-tracelens.sh <model> <prompt\|decode> [VAR=value ...]` | Profiles one run, converts it, and writes a TraceLens report (optionally with TraceDiff against another run). |
 
 ## Two benchmark standards
 
@@ -70,4 +72,61 @@ Setup notes:
 - Hyperloom's `install.sh` also installs the scriptable-quality dependencies (lpips pulls in torch);
   the custom workload does not need them.
 
+## HRX profiles in TraceLens
+
+rocprofv3 sees no HRX dispatches (HRX loads its own libhsa and IREE writes the AQL packets), so HRX
+profiles itself (`HRX_PROFILE_MODE=dispatch HRX_PROFILE_FILE=x.prof`) and `iree-profile` reads the
+bundle. `hrx2kineto.py events.jsonl out.trace.json [kernel_trace.csv]` turns the dispatch events into
+Kineto kernel events; that alone gives TraceLens a kernel timeline, every op in "other".
+
+A llama.cpp build with `GGML_HRX_DISPATCH_SHAPE_LOG=<file>` also logs, for every command of every HRX
+command program, the dispatch match that emitted it (covered ggml nodes, tensor types and shapes) and,
+per graph execution, the current graph's node names. With `--shapes <file> --commands <iree-profile
+command jsonl>` the converter then emits one `hrx::<ggml op>` cpu_op per match execution
+(`hrx::mul_mat`, `hrx::mul_mat_id`, `hrx::flash_attn_ext`, `hrx::rms_norm`, ...) with its kernels,
+shapes and the fused op chain, nested under `hrx::step` (one forward pass, across graph splits) and
+`hrx::layer`. FLOPs and bytes use each ggml type's real bits per weight (Q4_K 4.5, Q6_K 6.5625, Q8_0 8.5,
+MXFP4 4.25, ...); a MUL_MAT_ID reads only the routed experts. Passed to TraceLens as
+`--extension_file`, the same file maps these ops to GEMM / GroupedGEMM / SDPA / NORM / RoPE / reduce /
+SSM / elementwise perf models; `--write-arch` writes a gfx1151 arch JSON (WMMA f16/bf16/iu8 1024 and
+iu4 2048 ops/WGP/clk, 20 WGPs, clock from `pp_dpm_sclk`, 256 GB/s) for the roofline columns.
+
+```sh
+GGML_HRX_DISPATCH_SHAPE_LOG=x.shapes.jsonl HRX_PROFILE_MODE=dispatch HRX_PROFILE_FILE=x.prof llama-bench -p 512 -n 0 ...
+iree-profile dispatch --format=jsonl --dispatch_events x.prof > x.events.jsonl
+iree-profile command --format=jsonl x.prof > x.commands.jsonl
+python3 tools/hyperloom/hrx2kineto.py x.events.jsonl x.trace.json --shapes x.shapes.jsonl \
+  --commands x.commands.jsonl --phase prefill --skip 1 --write-arch gfx1151.json --summary
+TraceLens_generate_perf_report_pytorch --profile_json_path x.trace.json --output_xlsx_path x.xlsx \
+  --extension_file tools/hyperloom/hrx2kineto.py --gpu_arch_json_path gfx1151.json \
+  [--comparison_json_path other.trace.json]
+```
+
+`hrx-tracelens.sh` does all of that for one model (prompt: `llama-bench -p 512 -n 0`; decode:
+`llama-server`, 8 tokens; the dispatch ring holds 65,536 events and `llama-bench` generation fails
+under profiling). Reading the numbers: HRX issues independent dispatches without a barrier, so
+concurrent kernels (for example the Q/K/V projections) share the GPU and each one's duration includes
+the others (`--summary` prints the overlapped share); activations that stay in the 32 MB Infinity Cache
+show more than 100% of the DRAM roofline; KV lengths are the padded cache views; prefill attention
+FLOPs include the masked half.
+
 Never put credentials in these files.
+
+## Known limitations (measured 2026-10-05, gfx1151, llama.cpp 522dab4)
+
+The profiling chain has three failure modes that were only found by running it, not by reading it. All three are handled
+so the chain still produces a trace, but two should be fixed rather than tolerated:
+
+1. **`iree-profile` must match the IREE that wrote the profile.** An older tool aborts with
+   `unsupported IREE HAL profile file version`. `hrx-tracelens.sh` now derives the tool from `HRX_BIN`'s own deps
+   build (falling back to `IREE_PROFILE`, then `PATH`) instead of a hard-coded path that may not exist.
+2. **The shape log needs a build carrying `GGML_HRX_DISPATCH_SHAPE_LOG`.** Without it the op-level half is empty.
+   Build the tree you benchmark with that flag, or point `HRX_SHAPELOG_BIN` at one that has it —
+   `custom_strixhalo.sh` warns loudly when the trace describes a different build.
+3. **`hrx2kineto.py`'s op-level match assumes command-buffer order equals the logged program's build order, and the
+   runtime does not guarantee that.** Observed: the dispatch *multiset* matches a program exactly while the *order*
+   differs (`copy_strided*` hoisted ahead of `concat_dim0`; `set_rows` moved), so `map_command_buffers()` reports
+   `matches no logged program` and no trace is written. When that happens `hrx-tracelens.sh` falls back to a
+   **kernel-level** trace (raw dispatch keys), which TraceLens accepts; the roofline is then coarser (op names in
+   "other") but real. The matcher still needs an order-independent key (the profile's `executable_id` /
+   `function_ordinal`, or per-dispatch identity).

tools/hyperloom/custom_strixhalo.sh

@@ -103,5 +103,45 @@ PP=$(guarded bench-pp timeout 900 $B/llama-bench -m "$GGUF" -dev HRX0 -p 512 -n
 [ -z "$TG" ] && fail "llama-bench failed"
 # Lemonade's standard too (reported, not the optimisation metric): scripts/lemonade_bench.sh
 guarded lemonade-bench timeout 900 bash "$(dirname "$0")/lemonade_bench.sh" $B/llama-server "$GGUF" "$OUT/lemonade-bench.json" > /dev/null || echo "lemonade bench failed (see lemonade-bench.log)" >&2
+
+# 5. profile leg. Hyperloom announces it with PROFILE=1 and names the directory in
+#    VLLM_TORCH_PROFILER_DIR / SGLANG_TORCH_PROFILER_DIR (see bypass_scriptable.py).
+#    Its executor then probes <workspace>/torch_trace and globs *.trace.json.gz.
+#    Without this the three candidate dirs are all empty, the profile phase fails
+#    three times, and the whole pass continues in DEGRADED mode with no roofline
+#    to direct the search -- which is exactly what happened on 2026-09-30.
+#    The box lock is already held above (step 2), so BOX_LOCK is cleared for the
+#    helper: a second flock on the same file from a child would deadlock here.
+if [ "${PROFILE:-0}" = "1" ]; then
+  TD=${VLLM_TORCH_PROFILER_DIR:-${SGLANG_TORCH_PROFILER_DIR:-$OUT/torch_trace}}
+  mkdir -p "$TD"
+  TL=$(cd "$(dirname "$0")" && pwd)/hrx-tracelens.sh
+  # The trace MUST describe the binary just benchmarked: Hyperloom's own profile_custom.yaml warns that a trace of a
+  # different run is worthless, and the roofline directs the whole search. Prefer this checkout's build; fall back only
+  # if it cannot trace, and say so loudly, because the roofline is then about a different tree.
+  TB=$B
+  # `grep -c` reads to EOF so `strings` is never killed by SIGPIPE. With `grep -q` under
+  # `set -o pipefail` the pipeline returns non-zero when grep exits on the first match, and a
+  # killed `strings` made this check ALWAYS take the fallback -- which is why every profile leg
+  # failed even on a build that carries the shape log.
+  if [ "$(strings "$TB/libggml-hrx.so.0" 2>/dev/null | grep -c GGML_HRX_DISPATCH_SHAPE_LOG)" = "0" ]; then
+    TB=${HRX_SHAPELOG_BIN:-$HOME/wt/hrx2kineto-src/build/bin}
+    echo "profile leg: WARNING $B lacks GGML_HRX_DISPATCH_SHAPE_LOG; tracing $TB instead -- the trace describes a DIFFERENT build than the one benchmarked" >&2
+  fi
+  echo "profile leg: emitting an HRX trace into $TD from $TB" >&2
+  if [ -x "$TL" ]; then
+    O=$(BOX_LOCK= HRX_BIN=$TB OUT_ROOT="$TD/.." \
+        DECODE_TOKENS=8 bash "$TL" "$GGUF" decode 2>&1 | tail -n 1)
+    if [ -n "$O" ] && [ -f "$O/trace.json" ]; then
+      gzip -c "$O/trace.json" > "$TD/rank_0.trace.json.gz"
+      echo "profile leg: wrote $TD/rank_0.trace.json.gz ($(stat -c%s "$TD/rank_0.trace.json.gz") B)" >&2
+    else
+      echo "profile leg FAILED: hrx-tracelens.sh produced no trace.json" >&2
+    fi
+  else
+    echo "profile leg FAILED: $TL is not executable" >&2
+  fi
+fi
+
 result "$PASSED" "$TG" "$PP" "$PRE" "$DEC" "$WHY"
 echo "decode $TG tok/s, prefill $PP tok/s, gate $PASSED ($WHY)"

tools/hyperloom/hrx-tracelens.sh

@@ -0,0 +1,108 @@
+#!/usr/bin/env bash
+# Copyright 2026 bong-water-water-bong
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# hrx-tracelens.sh <model.gguf> <prompt|decode> [VAR=value ...]
+#   Profile one HRX run (prompt: llama-bench -p 512 -n 0; decode: llama-server, 8 generated tokens),
+#   convert it with hrx2kineto.py (shapes from GGML_HRX_DISPATCH_SHAPE_LOG), and run TraceLens with the
+#   hrx extension + gfx1151 arch JSON. Output: ~/lb/tracelens/<ts>-<model>-<mode>[-TAG]/ (summary.txt =
+#   converter roofline table, report.xlsx + csvs/ = TraceLens, tracediff_top.txt with COMPARE). VAR=value
+#   pairs are exported for the run (e.g. GGML_HRX_DISABLE_DISPATCH=hip.). Env: HRX_BIN (build/bin with the
+#   shape log), TAG, COMPARE=<other
+#   run dir> (adds TraceDiff vs that run's trace), PROMPT_TOKENS (512), DECODE_TOKENS (8), OUT_ROOT
+#   (~/lb/tracelens), BOX_LOCK (flock file shared with other GPU jobs; empty = none), IREE_PROFILE, H2K.
+#   Needs a llama.cpp build with GGML_HRX_DISPATCH_SHAPE_LOG and TraceLens in ~/.cache/tracelens-venv.
+#   Aborts the run at 93 C Tctl; needs MemAvailable >= model size + 11 GB.
+set -uo pipefail
+[ $# -ge 2 ] || { sed -n 17,28p "$0"; exit 2; }
+MODEL=$(readlink -f "$1"); MODE=$2; shift 2
+for kv in "$@"; do export "$kv"; done
+BIN=${HRX_BIN:-$HOME/wt/hrx2kineto-src/build/bin}
+# iree-profile must match the IREE the HRX build wrote its profile with: a stale tool rejects the file with
+# "unsupported IREE HAL profile file version". The deps build a given HRX checkout uses sits beside its
+# bin/, so derive the tool from HRX_BIN; IREE_PROFILE overrides, then PATH.
+IP=${IREE_PROFILE:-}
+if [ -z "$IP" ] || [ ! -x "$IP" ]; then
+  IP=$(dirname "$(readlink -f "$BIN")")/../ggml/src/ggml-hrx/hrx/src/ggml-hrx-deps-build/runtime/src/iree/tools/iree-profile/iree-profile
+  [ -x "$IP" ] || IP=$(command -v iree-profile 2>/dev/null || echo "$IP")
+fi
+H2K=${H2K:-$(dirname "$(readlink -f "$0")")/hrx2kineto.py}
+TL=${TRACELENS_BIN:-$HOME/.cache/tracelens-venv/bin}
+ROOT=${OUT_ROOT:-$HOME/lb/tracelens}
+O=$ROOT/$(date +%Y%m%d-%H%M%S)-$(basename "$MODEL" .gguf)-$MODE${TAG:+-$TAG}
+mkdir -p "$O"; ln -sfn "$O" "$ROOT/latest"
+export LD_LIBRARY_PATH=$BIN
+export IREE_HAL_AMDGPU_LIBHSA_PATH=${IREE_HAL_AMDGPU_LIBHSA_PATH:-/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1}
+export HRX_PROFILE_MODE=dispatch HRX_PROFILE_FILE=$O/run.prof GGML_HRX_DISPATCH_SHAPE_LOG=$O/shapes.jsonl
+K=$(for h in /sys/class/hwmon/*; do [ "$(cat $h/name)" = k10temp ] && echo $h; done)
+avail_gb() { awk '/MemAvailable/ {print int($2/1048576)}' /proc/meminfo; }
+need=$(( $(stat -c %s "$MODEL") / 1073741824 + 11 ))
+[ "$(avail_gb)" -ge $need ] || { echo "MemAvailable $(avail_gb) GB < $need GB, not running" | tee "$O/run.log"
+                                 exit 1; }
+printf '%s\n' "model=$MODEL mode=$MODE bin=$BIN env: $*" > "$O/run.log"
+
+LOCK=${BOX_LOCK-$HOME/.cache/lax-decode/box.lock}
+[ -n "$LOCK" ] && { exec 9>"$LOCK"; flock 9; }
+( while :; do
+    t=$(( $(cat $K/temp1_input) / 1000 ))
+    [ $t -ge 93 ] && { echo "$(date +%T) THERMAL ABORT $t C" >> "$O/run.log"; pkill -f "$BIN/llama-"; }
+    sleep 1
+  done ) & TG=$!
+trap 'kill $TG 2>/dev/null' EXIT
+if [ "$MODE" = prompt ]; then
+  "$BIN/llama-bench" -m "$MODEL" -dev HRX0 -ngl 99 -fa 1 -p "${PROMPT_TOKENS:-512}" -n 0 -r 1 \
+    > "$O/bench.txt" 2> "$O/bench.err"
+  PHASE=prefill
+else
+  P=$((18500 + RANDOM % 400))
+  "$BIN/llama-server" -m "$MODEL" -dev HRX0 -ngl 99 -fa on --host 127.0.0.1 --port $P -c 2048 -np 1 \
+    --no-webui > "$O/server.log" 2>&1 & S=$!
+  for i in $(seq 300); do
+    curl -sf localhost:$P/health >/dev/null && break; kill -0 $S 2>/dev/null || break; sleep 1
+  done
+  curl -s localhost:$P/completion -H 'Content-Type: application/json' \
+    -d "{\"prompt\":\"The history of the city of Paris\",\"n_predict\":${DECODE_TOKENS:-8},\"temperature\":0}" \
+    > "$O/completion.json"
+  kill $S; wait $S 2>/dev/null
+  PHASE=decode
+fi
+kill $TG 2>/dev/null; [ -n "$LOCK" ] && { flock -u 9; exec 9>&-; }
+echo "Tctl $(( $(cat $K/temp1_input) / 1000 )) C after the run" >> "$O/run.log"
+
+"$IP" dispatch --format=jsonl --dispatch_events "$O/run.prof" > "$O/events.jsonl"
+"$IP" command --format=jsonl "$O/run.prof" > "$O/commands.jsonl"
+python3 "$H2K" "$O/events.jsonl" "$O/trace.json" --shapes "$O/shapes.jsonl" --commands "$O/commands.jsonl" \
+  --phase $PHASE --skip 1 --write-arch "$O/gfx1151.json" --summary | tee "$O/summary.txt"
+# FALLBACK: op-level conversion needs each command buffer's dispatch order to equal the logged program's build
+# order. The runtime can reorder dispatches inside a command buffer (multisets match, orders do not), in which
+# case map_command_buffers finds no candidate and writes no trace -- which used to fail the Hyperloom profile
+# phase outright. A kernel-level trace (raw dispatch keys; op names land in "other") is still accepted by
+# TraceLens with the extension and arch, so the phase produces real data instead of failing. Removing this needs
+# an order-independent matcher (see README "Known limitations").
+if [ ! -s "$O/trace.json" ]; then
+  echo "hrx-tracelens: op-level match found no program (runtime reorders command buffers vs the logged programs); falling back to a kernel-level trace" >&2
+  python3 "$H2K" "$O/events.jsonl" "$O/trace.json" 2>&1 | tee -a "$O/summary.txt" >&2
+fi
+CMP=()
+[ -n "${COMPARE:-}" ] && CMP=(--comparison_json_path "$COMPARE/trace.json")
+"$TL/TraceLens_generate_perf_report_pytorch" --profile_json_path "$O/trace.json" \
+  --output_xlsx_path "$O/report.xlsx" --output_csvs_dir "$O/csvs" --extension_file "$H2K" \
+  --gpu_arch_json_path "$O/gfx1151.json" "${CMP[@]}" > "$O/tracelens.log" 2>&1 \
+  || echo "TraceLens failed, see $O/tracelens.log"
+# TraceDiff (this run = trace1, COMPARE = trace2): largest per-step time differences by aligned subtree
+if [ -n "${COMPARE:-}" ] && [ -f "$O/csvs/diff_stats.csv" ]; then
+  python3 "$H2K" --tracediff-top "$O/csvs/diff_stats.csv" --steps-from "$O/trace.json" | tee "$O/tracediff_top.txt"
+fi
+echo "$O"