Repository navigation
hyperloom custom workload: every measurement writes the same log filenames into RESULT_DIR, so a later arm silently destroys an earlier arm's raw evidence #892
Description
tools/hyperloom/custom/benchmark.py writes its three raw logs to fixed names inside RESULT_DIR, independent of the
measurement's own RESULT_FILENAME:
serve.log—serve_measure()((out_dir / "serve.log").open("w"), andplacement_summary()reads it back)llama-bench.log—run(...)((out_dir / f"{name}.log").write_text(...),nameis the leg, not the measurement)kld.log/kld-record.log—kld()viarun(...)
So two measurements that share one RESULT_DIR — which is the normal shape for a run, and is exactly what a
HL_PAIR A/B pair does — collide on all three. The result JSONs are distinct (<RESULT_FILENAME>.json), so the
numbers survive while the logs that produced them do not, and the loss is silent: nothing errors, the file simply
holds the later arm's text.
Reproducer (no GPU, no model, no server — just the writer):
import pathlib, sys, tempfile
sys.path.insert(0, "tools/hyperloom/custom")
import benchmark as B
with tempfile.TemporaryDirectory() as td:
root = pathlib.Path(td)
B.run(["sh", "-c", "echo FIRST-MEASUREMENT"], "llama-bench", 10, root)
B.run(["sh", "-c", "echo SECOND-MEASUREMENT"], "llama-bench", 10, root)
print(sorted(p.name for p in root.iterdir())) # ['llama-bench.log']
print((root / "llama-bench.log").read_text().strip()) # SECOND-MEASUREMENTfiles in RESULT_DIR: ['llama-bench.log']
llama-bench.log now holds: SECOND-MEASUREMENT
Measured cost, on strixhalo with this workload: the 2026-11-02 19:01 arm overwrote the 18:45 baseline's serve.log,
llama-bench.log and kld.log in ~/hyperloom-ws/window-20261102/. The JSONs and every number quoted in
~/briefs/THYPERLOOM-STATE.md are intact (they were read before the overwrite), but the raw leg output behind them is
gone. The same collision would have hit both halves of a pair, and a driver killed mid-arm (e.g. a reboot) leaves a
partial serve.log that the next arm then overwrites, which is how a partial leg gets mistaken for a complete one.
Workaround in use outside the repo: the harness moves the three files into logs-<result>-<hhmmss>/ after each arm.
That is a workaround for a producer-side defect, and it cannot help a run driven straight through hyperloom optimize.
Activity
bong-water-water-bong commentedon Nov 3, 2026
Triaged: reproduced at
7690a8c3, and the fix (#893) is already upChecked at
origin/main=7690a8c3, not against the report: the three writers are exactly as filed, and they are theonly paths that bypass the per-measurement name —
file:lineat7690a8c3run()tools/hyperloom/custom/benchmark.py:246out_dir / f"{name}.log"—nameis the leg (llama-bench,kld,kld-record), never the measurementserve_measure():328(writer),:361(read-back)out_dir / "serve.log"kld():390→run()kld/kld-recordwhile everything that is per-measurement is keyed on
RESULT_FILENAME(:405-406RESULT_DIR,:454f"{name}.json").That asymmetry is the whole bug: the numbers are named, the evidence behind them is not.
Reproduced, host-only, no model, no GPU — the issue's own scenario against
origin/main'sbenchmark.py, tworun(…, "llama-bench", …)calls in oneRESULT_DIR:One thing the issue does not say, and it is the reason #893's diff has to touch two lines, not one
serve_measure()reads back the same fixed path it wrote::328opensout_dir/"serve.log"for writing and:361returns
placement_summary(out_dir / "serve.log"). So the coupling is not only archive-deep. Sequential arms loseevidence (as filed); two arms sharing one
RESULT_DIRconcurrently would have one arm'splacement_summary()parse theother arm's log, i.e. the wrongness reaches the numbers. I have not seen that happen in
THYPERLOOM-STATE.mdand am notclaiming it did — the point is that any fix which redirects the writer and leaves the reader alone keeps this second failure
mode while looking fixed.
PR #893 does redirect both (
placement_summary(log_dir / "serve.log")), moves the directory creation into one helperlog_dir_for(out_dir, name)→RESULT_DIR/logs-<RESULT_FILENAME>/, and asserts the pair case intests/hyperloom_benchmark_test.py. I ran its test on this box (host-only):Two residuals, both small and neither blocking
logs-<result>-<hhmmss>/after each arm; the producer will writelogs-<RESULT_FILENAME>/. Two schemes for onedirectory it does not control — the mover should be retired when hyperloom custom: keep each measurement's raw logs (they overwrote each other) #893 lands, or the next reader has to know both.
complete one" case the report names as the driver-kill scenario: nothing marks an arm as finished. A
.rc/sentinelwritten at arm end (one line, in the same helper) would make that distinguishable; not requested here, and not needed
for this issue to close.
What happens next: #893 is the fix and it reads as complete from my side (
closes #892in its body, so it closes itself onmerge). The
~/hyperloom-ws/window-20261102logs lost at 19:01 are not recoverable — the JSONs quoted inTHYPERLOOM-STATE.mdare, and that is the trade the fix makes permanent. Labelledbug.hyperloom custom: stop echoing a lever knob the engine never reads (#931