[glennneuber]

The ask: make every OCRBench arm name the machine it ran on, so its seconds per item can't be read against another machine's by accident. extbench.py takes the H26 host profile as HOST_PROFILE and records the hardware in the arm's summary; summarize_extbench.py prints it and warns when arms differ.

Why

SPEC H26 says a published result names its machine, and its own conformance row says "Nothing checks an OCRBench run" (spec/vision-harness-reuse.md, the H26 row). An arm's JSON records host, which is the URL the client called (usually http://127.0.0.1:<port>), and the build. It never records the GPU.

What happened	Evidence
The gemma4:31b quantisation-ladder arms (bf16, q8_0, q4_K_M, mxfp8, nvfp4) were read as a speed comparison from their JSON alone: about 4.9 s/item for every GGUF format, 1.2–2.0 s for mlx-cuda. Nothing in those files says the machine was a shared RTX PRO 6000 Blackwell. That fact is in the prose of ocrbench-quantisation-ladder.md, along with the warning "Speed here is not a throughput claim"	ext_ocr31b_*_ocrbench.json (2026-09-19) against ocrbench-quantisation-ladder.md
The H100 arms do have profiles (bench-runs/ocrbench/host-profile_h100-sm90-*.json), but only by naming convention: an arm doesn't point to its profile, and the footer can't print it	bench-runs/ocrbench/
--timing already refuses to compare arms that sent different prompt_cache_ram, because that moves seconds per item. A different machine moves them more, and nothing warns	summarize_extbench.py, --timing

We would not currently detect an H100 arm and a Blackwell arm rendered into one --timing table.

What this covers

• extbench.py: HOST_PROFILE=<path> reads a host profile (schema host-profile/…) and records summary.hardware: the profile's file name, label, each GPU's name and memory_total_mib, and cloud.machine_type. A path that can't be read as a host profile stops the arm before its first request. Unset, hardware is null.
• summarize_extbench.py: once any rendered file records hardware, the footer adds · hardware: <label> (<GPUs>, <machine type>), and a file without it shows as not recorded (pre-H26 run). --timing warns when the arms ran on different machines or one names none. Files without the field render byte-for-byte as before, so pasted tables (H7) stay valid.
• Tests:
  • test_extbench.py::TestMainLoop: recorded from a profile, null when unset, and a non-profile or missing path stops the arm before any request.
  • test_summarizers.py::TestExtbenchArmsAndSlices: footer naming, unchanged rendering without the field, and the --timing warning for different machines and for a missing one.
• Docs: the H26 "point to it" rule and its conformance row in spec/vision-harness-reuse.md, and HOST_PROFILE in the run command in ocrbench.md.

What this does not cover

• No backfill. Arms from before this change keep hardware absent and render as before, with the warning only under --timing when mixed with a recorded arm. A profile can't be captured after the fact.
• It doesn't capture the profile. That stays tools/host_profile.py on the serving machine (H26: never on the client).
• It warns rather than refuses, for the same reason bundle_campaign.py does: older runs name no machine.

How we'll know

The next OCRBench arm on any host carries summary.hardware, and a --timing render that mixes machines prints the warning. Checked locally: the CI test set in preflight-expectations.yaml passes (test_verdicts, test_bundle_campaign, test_summarizers, test_rescore, test_client, test_extbench, test_prompt_cache_probe), and check_no_names.py finds no names.

Cost

6 files, +156/−5, of which 81 lines are tests. One new optional env var; nothing changes for an arm run without it.

Not now

• Pointing vision_suite.py's own score records at a profile (the same gap, a different harness).
• Requiring HOST_PROFILE for a published arm. That waits until no pre-H26 arms are being re-rendered.

🤖 Generated with Claude Code

[glennneuber]

Reviewed from the CUDA host: correct as far as it goes, and ready to merge. One note below; it doesn't block.

Checked on 916f588fe, which is up to date with main (a4faff320):

• test_extbench and test_summarizers pass: 216 tests, OK.
• load_hardware stops the arm on unreadable JSON or a non-host-profile/ schema before main() sends anything, and hardware is null when HOST_PROFILE is unset. That matches the description.
• In summarize_extbench.py, a file without hardware adds nothing to the footer or --timing, so existing pasted tables render byte-for-byte as before. The only new output appears once some arm records the field.

Note (doesn't block): machines is built from each arm's first run only (got[0][1]), the same way caches is for prompt_cache_ram. The hosts/builds MIXED banner, by contrast, covers every repeat of every arm.

• So an arm whose repeat 2 ran on another machine names only repeat 1's machine in the footer.
• --timing reads only got[0] too, so the timing warning is consistent with what it renders.
• The footer is the gap. If it should cover repeats like the MIXED banner does, collect hardware(s) in the same loop that fills hosts and builds.

From ai-server/mlx-cuda via /collab · 2026-11-03 · build 0.40.3-dynres-12-g8761533 · signed ai-server