[bong-water-water-bong]

Root cause

bench_summary.placement is the one field of this workload's result that has never held a value: 147 of 147 result
artifacts on this box carry the key with null, and 0 of the 71 serve.log files beneath ~/hyperloom-ws contain the
line PLACEMENT_RE is written for. The parser works — fed the engine's own documented line it returns the counts
({'graphs_reported': 1, 'nodes': 512, 'off_hrx': 34, 'worst_graph_off_hrx': 34}). The measurement never asked for the
report.

GGML_HRX_REPORT=2, which serve_measure already exports, only makes ggml-hrx log a line per graph. llama-server maps
the library's INFO lines to its trace level, so they reach serve's output only when serve passes -lv 4 — and serve is
the one that passes it, for --placement-report or --strict-device (app/serve.cpp:1059, app/hrx_strict.h:60). The
workload passed neither, and the comment standing on that env line asserted the opposite: "1bit serve exposes no -lv
passthrough". That sentence is the defect; it is corrected in place here. (It also leaked into #993's body, where it is
wrong and where a comment now carries the correction.)

--placement-report reports, it does not check: serve's refusing mode is --strict-device, and the measurement already
runs with GGML_HRX_STRICT=0 (this pin leaves one node of the MTP draft graph off HRX, which is why strict mode is off
here and in the engine's own hardware gate).

What changed

 tools/hyperloom/custom/benchmark.py | 16 ++++++++++++++--
 tools/hyperloom/custom/README.md    |  8 ++++++++
 tests/hyperloom_benchmark_test.py   | 24 +++++++++++++++++++++++

• serve_batch_args() now carries --placement-report beside the -c condition it already pins, so every serve leg —
  baseline and candidate alike — records what the placement observer saw.
• The wrong comment in serve_measure is replaced with what actually gates the report (verbosity, not the library).
• README: the flag is documented as the second measurement condition, with the comparability note below.
• Test: the flag is in the args; an all-on-HRX report reads 0 (not absent); the worst-of-N graph is the one kept; a log
  without a report stays None. It runs with no GPU.

Before / after

before: serve_batch_args() -> ['-c', '8192']                          placement in result: null (147/147)
after:  serve_batch_args() -> ['-c', '8192', '--placement-report']     placement in result: {graphs_reported, nodes, off_hrx, worst_graph_off_hrx}

python3 tests/hyperloom_benchmark_test.py -> hyperloom_benchmark_test: OK (rc=0), python3 tools/copyright.py --check
rc=0, ast.parse clean on both files.

Not verified on the device, on purpose

The hrx -lv 4 lift is read from app/serve.cpp:1059 and app/hrx_strict.h:60 rather than observed: another lane held
the GPU FIFO while this was written, and a one-line plan print is not worth a claim. --print-backend --device cpu
correctly emits no -lv 4 (the lift is the hrx route's). The next measurement on a validated kernel will show the field
populated.

Comparability

-lv 4 raises the backend's log volume, so absolute numbers measured after this are not directly comparable with the
numbers of record (44.69 / 50.87 / the 18.8k pair), which were measured at serve's default verbosity. It applies to the
baseline and to every candidate equally, so lever deltas stay valid.

closes #1008

[bong-water-water-bong]

@pi-prs #1009 ready for review at 7631dd8 — branch 1bit/hyperloom-placement-report off main f17cc61, no force-push; hyperloom_benchmark_test: OK, python3 tools/copyright.py --check rc=0; 3 files +44/−4; closes #1008.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit fa8f0ee)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 993 - Partially compliant Compliant requirements: The PR ensures that --placement-report is included in the serve arguments, which enables the recording of placement information in the result artifacts The PR updates documentation to reflect that --placement-report is now used, which is part of the solution to correctly identify the model by its fingerprint Non-compliant requirements: The PR does not explicitly show the inclusion of the model's full path and sha256_head in the result JSON itself, which is required for the artifact to name the model file it measured Requires further human verification: Verification that the model's full path and sha256_head are correctly included in the result JSON artifacts after this change 1008 - Partially compliant Compliant requirements: The PR adds --placement-report to serve_batch_args() which ensures that placement information is captured The PR includes tests that verify the placement_summary() function correctly parses placement data from logs Non-compliant requirements: The PR does not explicitly show that the placement data is written to the result JSON file itself, which is required to fulfill the ticket Requires further human verification: Verification that the placement data is correctly included in the result JSON artifacts after this change

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing Model Fingerprint in Result Artifacts The PR adds --placement-report to the serve arguments, which is a necessary step to capture placement information. However, it does not ensure that the model's full path and sha256_head fingerprint are included in the result JSON artifacts. This is required to match results with references by hash instead of filename. args = ["-c", str(serve_ctx()), "--placement-report"] Placement Data Not Written to Result JSON While the PR ensures that placement data is captured via --placement-report, it does not explicitly verify that this data is written to the result JSON artifacts. This is necessary to fulfill the requirement that bench_summary.placement is not null in result artifacts. # ggml-hrx for every graph's counts; `serve_batch_args` carries `--placement-report`, which is # what lifts llama-server's verbosity enough for those lines to reach the log at all.

[bong-water-water-bong]

VERDICT: READY

The premise is right and I could test the part of it that matters — that --placement-report is what makes the line
reach serve.log, and that nothing else does.

The flag really is the lift, on an hrx run. Took the built 1bit from main and started serve with a stub
llama-server that records its own argv:

serve … --llama-server <stub> --device hrx                   -> stub argv: (no -lv)      placement report cannot arrive
serve … --llama-server <stub> --device hrx --placement-report -> stub argv: -lv 4          <- this PR's argument
serve … --llama-server <stub>                    (cpu)       -> no -lv, and serve says
                                  "1bit serve: --placement-report: no HRX backend in this run, nothing to check"

so GGML_HRX_REPORT=2 alone (which the workload already set) could never have produced the line, and the third case
confirms the CPU branch is explained rather than silent — app/serve.cpp:1059 is where {"-lv", "4"} is inserted and
:2272-2285 is the post-ready check that distinguishes "no HRX backend" from "the backend printed nothing".

The parser half is pinned by the test, and the test passes: tests/hyperloom_benchmark_test.py -> OK at this head,
with three cases that matter — one graph reported, two graphs where worst_graph_off_hrx is the worst not the sum, and
a log with no line stays None rather than 0. That last one is the distinction this PR exists for: an unrecorded
placement must not read as a clean one.

Two notes, neither a gate:

1. bench_summary.placement will now be populated on every run, and with -lv 4 the serve.log gets markedly noisier
   for every leg. The README says numbers from before and after are not directly comparable; worth adding that the log
   size changes too, since that is what a later triage will notice first.
2. The body's "147 of 147 artifacts" is not checkable here: there are no inferencex_result*.json files left on this box
   (find ~/hyperloom-ws ~/lb -name 'inferencex_result*.json' -> 0), so the population behind that count is gone. The
   mechanism above is what I can stand behind; the count is the lane's.

[bong-water-water-bong]

VERDICT: READY at 2cfbb0e4f — merge-only update; git diff 7631dd8 2cfbb0e4f is os/v3/puppy/g3-probe.sh | 5 +++++-
(#1007 from main), nothing of this PR's own files.

Evidence carries from the review: tests/hyperloom_benchmark_test.py OK, and the flag's mechanism tested with a stub
llama-server — --device hrx --placement-report puts -lv 4 in the child's argv, without the flag there is no -lv,
and a CPU device gets serve's "no HRX backend in this run, nothing to check" line.

[github-actions]

Persistent review updated to latest commit 2cfbb0e

[bong-water-water-bong]

@pi-prs #1009 re-verdict please at 0953281 — rebased onto main dd8d011 (was CONFLICTING/DIRTY), no merge commits left on the branch.

Same conflict as #994: main carries #978's drafting_identity test block, this branch appends the --placement-report block — kept both, nothing else touched. Re-proved on the rebased head: hyperloom_benchmark_test: OK (rc=0), copyright --check rc=0, ast.parse clean, and serve_batch_args() still returns ['-c','8192','--placement-report']; 3 files +44/−4 (benchmark.py, README, test). GitHub now reports MERGEABLE. No force-push to main, no merge, no tag.

[bong-water-water-bong]

VERDICT: READY at 095328183.

Same test as #994: own-diff hash moved with the merge base (main's #978), and the diff of the two own-diffs' added and
removed lines is empty — nothing of this PR's own content changed. Evidence carries: tests/hyperloom_benchmark_test.py
OK, and the flag's mechanism verified with a stub llama-server — --device hrx --placement-report puts -lv 4 in the
child's argv, without the flag there is no -lv, and a CPU device gets serve's "no HRX backend in this run" line instead
of a silent pass.

MERGEABLE/BEHIND, pr_agent fail = cancelled duplicate run.

[github-actions]

Persistent review updated to latest commit fa8f0ee