Repository navigation
hyperloom custom workload: record the device stack (kernel + HSA runtime) with each measurement - #948
Conversation
… each measurement The report named the binary (#933) but not the device stack it ran on. This box rebooted five times in 18 h on 2026-11-03/04 and the tuner launcher applies no kernel gate, so a number could only be tied to a kernel by the prose in ~/briefs/THYPERLOOM-STATE.md: the pair-of-record JSONs carry bench_summary{levers,longctx,placement,pp512_tok_s,runner_type,score_geomean,serve_tok_s,tg128_tok_s} and nothing about the kernel or the HSA runtime. bench_summary.device_stack = {kernel, hsa_runtime, hsa_runtime_sha256}, the runtime being the file IREE_HAL_AMDGPU_LIBHSA_PATH names (whole-file hash, same helper as the build field) or null when there is none -- named by path, so no number is attributed to a runtime it did not load.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 358ad99 (tprs-ready added). build SUCCESS; pr_agent CANCELLED (the expected state with coder-llm off); everything else green.
Run, not read: python3 tests/hyperloom_benchmark_test.py at this head → hyperloom_benchmark_test: OK, rc 0. And device_stack() against this box, loaded the way the report loads it:
{'kernel': '7.3.0-rc4-1bit-perfopt+',
'hsa_runtime': '/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1',
'hsa_runtime_sha256': '21a6682452e159cfcdc6bdc84676f6b7f9c1236f3a2957a97a2e7170889cdfa3'}
That hash is the runtime the c22 package's own 1bit serve prints for the same file (I checked that separately on #946 today), so the field names the right object. The three branches in the test (env set to a real file, to an absent path → None, and the default TheRock path) all behave as the docstring says.
Nothing downstream rejects the new key: Hyperloom's bypass_report.py:90-99 copies bench_summary into the report wholesale whenever it is not None, so an extra sub-dict is carried, not validated. Verified by reading the executor, not by assumption.
Two nits, neither gating:
- The docstring says "the runtime is named by path AND whole-file hash", but it uses
sha256_head(), which hashes the first 64 MiB. Forlibhsa-runtime64.so.1(5.1 MB) that is the whole file, so the value is right; the sentence is what would mislead a later reader if the same helper were pointed at something bigger. "…and its sha256 (a head hash: 64 MiB cap, the whole file for a 5 MB runtime)". - The last assert calls
bench.device_stack()twice (...["hsa_runtime"] is None, bench.device_stack()) — the message re-evaluates it; harmless, just noting it.
Also for merge order: #950 adds its asserts to the same main() region of tests/hyperloom_benchmark_test.py that this PR touches, so the two will conflict textually; whichever lands second needs a trivial rebase.
Closes #947.
Root cause
bench_summary.buildnames the binary (#933) but nothing names the device stack, so the pair-of-record artifacts of the long-context leg are tied to their kernel by prose only. The box rebooted five times in 18 h on 2026-11-03/04 and the tuner launcher applies no kernel gate (only the device-free window harness does), so ahyperloom optimizerun on an unvalidated boot would write numbers into a report that cannot say which stack produced them.placementis not a substitute: it isnullin those same JSONs.Before / after (same file,
bench_summarykeys)hsa_runtimeis the fileIREE_HAL_AMDGPU_LIBHSA_PATHnames (falling back to the box's ROCm-TheRock path when it exists) and isnullwhen there is none — the path is reported, so nothing is attributed to a runtime it did not load.sha256_headis the helperbuild_identityalready uses.Verification
python3 tests/hyperloom_benchmark_test.py→hyperloom_benchmark_test: OK(new asserts:kernel == platform.release(), a stub runtime hashes tohashlib.sha256of its bytes, a missing path yieldsnull).python3 tools/copyright.py --check→ rc=0.No measurement is affected: nothing on the scoring path reads this, and the value is written once per benchmark invocation.
Two files, +41/−1. Not merged — the owner merges in the morning.