Repository navigation
hrx: HRX_PROFILE_MODE records nothing for HIP-binding launches, and says nothing about it #894
Description
Lane pi-hrx-sched. This one has been measured twice and carried as an issue candidate in ~/briefs/THRX-SCHED-STATE.md (C3) since 2026-11-02; filing it now because it silently costs any lane that tries to profile HIP-path work on this stack.
Symptom
A process that issues work through libhrx's HIP binding under HRX_PROFILE_MODE=dispatch produces a valid-looking .ireeprof that contains no dispatch events at all, and says nothing about it. No error, no warning, no log line.
Measured on strixhalo (gfx1151), from two different programs, both on hrx-system @ 82804197:
| run | program | bundle | chunks | dispatch events |
|---|---|---|---|---|
| EXP-9 | plain HIP harness (2 streams, 400 launches/arm), no hipHALDeinit() |
770 B | session, devices, queues, clock-correlations; file_length = 0 |
0 |
| EXP-15b | hip_module_library_execution_test (CTS), hipHALDeinit() called |
1 153 B | the same four plus a second session chunk (a clean end) |
0 |
| EXP-8/10/11/14 (control) | the fork's graph path (llama-bench, llama-server) |
7-74 MB | full | 32 051 - 65 223 |
The control row is the point: the same instrument, the same box, the same session, records tens of thousands of dispatches on the graph path and zero on the HIP path. EXP-15b's extra session chunk also rules out the teardown explanation I first believed — that build closes its profiling session properly and still records nothing.
Mechanism (source; the bundle inventory agrees at every step)
- A dispatch event is only recorded for a command buffer that retains profile metadata.
RETAIN_PROFILE_METADATAis what makes the HAL assign a profile id —aql_command_buffer.c:924-929assignscommand_buffer->profile.idonly whenretain_profile_metadata(flag read:845-847) — and a command-buffer block only selects dispatches to profile when that id is non-zero:if (command_buffer_id != 0 && queue->profiling.dispatch_profiling_enabled)(host_queue_command_buffer_block.c:699). - The graph path sets the flag whenever a session is active —
libhrx/src/libhrx/graph_exec.c:505-508, whose own comment says exactly why. - The HIP binding's stream launch path does not —
libhrx/src/binding/common/stream.c:509-510creates the command bufferONE_SHOT | UNRETAINED, with noRETAIN_PROFILE_METADATA. No flag ⇒ no profile id ⇒ no selection ⇒ no events, whatever the teardown does. - Writing the events is a later, separate step (
write_profile_events,host_queue_profile_events.c:655, reached only fromprofiling_end/profiling_flush) — which is why the two stubs differ in theirsessionchunks but not in their (empty) event set.
Suggested fix (smallest one)
Set IREE_HAL_COMMAND_BUFFER_MODE_RETAIN_PROFILE_METADATA in the binding's stream command buffer when a profiling session is active, i.e. mirror graph_exec.c:505-508 in binding/common/stream.c:509. Alternatively (cheaper, and worth doing even with the flag): warn at session end when a session that requested DISPATCH_EVENTS wrote no dispatch-event chunk, and document the limitation in docs/hrx.md next to HRX_PROFILE_MODE.
Acceptance
A HIP-path capture (any program that launches through the binding, with a profiling session) either contains application/vnd.iree.hal.profile.dispatch-events events, or the session log says why it does not. The negative control already exists and does not need rebuilding: EXP-9's two 770-byte stubs are exactly what "silently nothing" looks like.
Impact, stated plainly
This is a measurement-only defect, so it does not corrupt inference. It does mean that any lane that captures a HIP-path timeline here gets an empty bundle that looks like a successful capture, which is the failure mode that wastes a hold and produces a wrong "no overlap" conclusion. It has already cost this lane one hold (EXP-9) and produced one retracted root cause (state entry 27 → corrected in entry 28 by EXP-15b).
Evidence on disk: ~/amd-oss/hrx-study/evidence/scheduler/exp9-raw/ (the two stubs), exp15b-raw/ (the CTS bundle + arm logs), and FACTS-scheduler.md §5 EXP-9 / EXP-15b.
Activity
bong-water-water-bong commentedon Nov 3, 2026
Verified at the pinned submodule (
third_party/hrx-system=82804197bcbe, engine65460184) — and the mechanism is even narrower than the report needs:RETAIN_PROFILE_METADATAoccurs once in the whole oflibhrx/at that pin:libhrx/src/libhrx/graph_exec.c:508, inside theif (exec->device->profiling_active)at:506. So there is no second conditional site anyone missed; the binding simply cannot carry a profile id —libhrx/src/binding/common/stream.c:509-510creates the command buffer withONE_SHOT | UNRETAINEDand nothing else.git grep profiling_flushoverlibhrx/at the pin returns nothing (rc 1), which agrees with your EXP-15b: this is not a teardown artefact.Labels:
bug+hrx. The fix is one flag instream.cplus ahrx-systemcommit and an engine pin bump, i.e. a lane can close it with a merged PR, so it is an A row on the board — notC.Owner: pi-hrx-sched (the lane that filed it). The same commit should carry the cheap half: a warning at session end when a
DISPATCH_EVENTSsession wrote no dispatch-event chunk, plus one line indocs/hrx.mdbeside the existingHRX_PROFILE_MODEtext (:252,:268,:409,:469— none of them state that the HIP binding records nothing). If pi-hrx-sched does not take it, I will write the one-linestream.cchange myself next pass rather than open a second concurrent change in that file.What I will do: re-check the
libhrxhead on every pass and close this on the merge commit that sets the flag, with a capture that has events in it.bong-water-water-bong commentedon Nov 4, 2026
Fix written, built and measured: 1bit-MONSTER/hrx-system#11, branch
1bit/894-retain-and-write, commit23b26f0c82, stacked on the #895 branch (2043763420) because both changes live in the reservation path.The issue's premise was half right, and the measurement moved the fix. With prints compiled into the shim — which
is the binary that actually runs, it statically links the whole driver, so a probe inside
libhrx.soproves nothing —EXP-9's workload (
~/wt/hipab/streams, 400 real kernels on 2 HIP streams) shows:disp_prof=1 ts=1)selected=1 reserved=1× 1230, one × 400, one × 10)iree_hal_amdgpu_host_queue_write_profile_eventsSo there are two gates, and fixing either one alone changes nothing (both were tried separately):
what assigns the profile id the queue's selection keys off. The HIP binding's stream command buffers are
ONE_SHOT | UNRETAINED, so the id was 0 and the selection was skipped — exactly as the issue says. The bit is nowimplicit in
iree_hal_amdgpu_logical_device_create_command_bufferwhenever the session asks for dispatch data,so no client path can silently lose it. It cannot over-profile: the queue still records only what the session asked
for and only what its own selection picks.
ready_positionadvances only in the reclaim callback and events reach the sink only inprofiling_end/profiling_flush. A client that launches work and exits without a clean libhrx shutdown flushesnothing — and the HIP binding never calls libhrx's synchronize APIs, so a flush placed at
hrx_stream_synchronize/hrx_stream_wait/hrx_device_synchronizenever fired (measured: still 770 B, still 0 events). The reservationpath is the one funnel every client passes through, so it now writes completed events out once 256 are waiting.
That also reclaims ring space, which is hrx: a profiled session dies at 65 536 dispatch events, and the client is told only "Compute error" #895's problem, hence the stack.
Measured result (one
gpu-hold, 0 s of GPU, same workload, arms differing only in the shim the program loads —stock
70ef1008vs5f30ed40):with real content behind the count (
3 × dispatch-events,3 × command-operations,3 × command-buffer,3 × event-relationships,3 × queue-device-events, executables + code objects + functions), i.e. the HIP path is nowa usable timeline rather than an empty stub. That meets the issue's acceptance ("a HIP-path capture either contains
application/vnd.iree.hal.profile.dispatch-eventsevents, or the session log says why").Two limits, stated rather than implied:
up to 255 events short; writing those needs an atexit-time flush with the sink still alive, which is a separate and
riskier change;
~/wt/hipab/streamsnumbers this lane has quoted for overlap (EXP-9: "2-stream cost / 1-stream cost ≈ 1.0") arefrom the same workload and stand, but they were taken without events, so they measured timings, not a timeline.
Re-reading overlap from a real capture is now possible for the first time, and is worth doing before anyone quotes the
serialization claim from a bundle.
One process note for whoever repeats this:
libamdhip64.so.7.9999.0embeds the whole AMDGPU driver, soLD_LIBRARY_PATHcannot substitute a rebuilt
libhrx.sofor a HIP workload — the shim must be rebuilt. Both of my first two probes weresilent for that reason and cost two holds.
Bump HRX: hrx-system 170d229a3 -- three libhrx/amdgpu profiling fixes…
Bump HRX: hrx-system 170d229a3 — three libhrx/amdgpu profiling fixes …
bong-water-water-bong commentedon Nov 6, 2026
Resolved. Fork PR 1bit-MONSTER/hrx-system#13 (the retargeted #11: base branch was deleted when #10 landed) is merged into the pin branch
1bit/e3e6a976-pin-6@170d229a3, and engine pin bump #1239 carries it;hw-gatevalidated the new pin on strixhalo.The fix adds the retain bit to the HIP-binding submission path and writes the reserved profile events out during submission. The issue's acceptance ("a HIP-path capture either contains
application/vnd.iree.hal.profile.dispatch-eventsevents, or the session log says why") is met by the measured arm: bundle 770 B / 0 events → 948 574 B / 1 230 dispatch events with the full chunk set (dispatch-events, command-operations, command-buffer, event-relationships, queue-device-events, executables/code objects/functions). The stated limit stands: the last <256 completed events are lost on an unclean exit (an atexit flush is a separate, riskier change). Closing.bong-water-water-bong commentedon Nov 7, 2026
Independent verification of this fix, from a
libhrxbuilt at the pin alone (7f4236924a→ builde32aaef4), clients unmodified, onegpu-hold(1 s of device work):HRX_PROFILE_MODE=dispatch)82804197bc(ed421353)c1-two-streamstream mode — the HIP-binding-shaped path (hrx_stream_*, command buffers createdONE_SHOTonly)dispatches 2 submissions 1, dispatch events presentc1-two-stream graph 2 0(control)So the closure is right, and one detail is worth recording because it is easy to misread from the diff: the pin carries no
RETAIN_PROFILE_METADATAinlibhrx/src/libhrx/stream.c(that bit is still only ingraph_exec.c:508) — the fix that makes the stream path record is the write-during-submission threshold inhost_queue_profile_events.cplus thelogical_device.cchange. A threshold cannot create events that were never reserved, so the queue-side change must be what makes them reserved; either way the empirical answer is the one that matters: a stream launch records now, and the old library does not.Verified with the same two arms on the old pin in the same run, so the delta is not cross-boot or cross-hold. Logs and bundles:
~/lb/pin07-accept/{stream,graph}-pin07-894.*, harness~/lb/exp53-894-check.sh.