Repository navigation
Description
What happens
A libhrx C-API client that profiles (HRX_PROFILE_MODE=dispatch) and follows the documented acquire/release pairing produces a 770-byte bundle with no dispatch events and no session-end record, even though the HAL did profile the work.
bundle: schema, session(1), device, queue, clock_correlation = 770 B
expected: + session end, command_buffer, command_operation, dispatch_event, queue_device_event
The events are not lost at record time — they are reserved and retired — and they are not lost by the ring either (#895). They are lost at teardown, because nothing ever ends the session.
Mechanism (citation per step, all at pin 82804197bcbe)
hrx_gpu_device_getreturns a borrowed pointer into global state:*device = &g_gpu.devices[index];with nohrx_device_retain—libhrx/src/libhrx/runtime.c:1193-1207. Nothing inhrx_runtime.h's contract for that call says the handle is borrowed.- The device object starts at refcount 1 (
iree_atomic_ref_count_init, same file), whilehrx_device_retain/hrx_device_releaseare the documented pairing (device.c:162-186). - So the natural client pattern — get, …, release, then shutdown — takes the teardown branch on that single release:
if (iree_atomic_ref_count_dec(&device->ref_count) == 1)→device->hal_device = NULL(device.c:177-186), and the HAL device reference is dropped. hrx_gpu_shutdown()then callshrx_gpu_end_all_profiling()→hrx_device_profile_end(), which returnsiree_ok_status()early on!device->hal_device(runtime.c:687-691).iree_hal_device_profiling_endis never called, so the sink gets nothing but the begin-time metadatairee_hal_device_profiling_beginwrote at session start.hrx_gpu_shutdown()returns success. The client is told its profiled run went fine and its capture is empty.
Two callers in one repository, one that knows and one that cannot:
| caller | acquire | profiling reaches iree_hal_device_profiling_end? |
|---|---|---|
engine (ggml/src/ggml-hrx/ggml-hrx.cpp:1425-1428) |
hrx_gpu_device_get(i, &hrx_device) then hrx_device_retain(hrx_device) |
yes — refcount 2, so hrx_device_release at :99 goes 2→1 and hrx_gpu_shutdown() (:103) ends the session against a live handle |
C-API client (c1-two-stream.c:243) |
hrx_gpu_device_get(0, &device) and nothing else |
no — refcount 1, so the release goes 1→0, clears hal_device, and profiling_end no-ops |
Same teardown order in both. The difference is only whether the caller pays for a reference it was never given.
Measured
Instrumented at the HAL (gdb breakpoints on the build-tree libhrx, which keeps .symtab), graph arm vs stream control, HRX_PROFILE_MODE=dispatch:
| stage | graph | stream |
|---|---|---|
HSA-PROFILING-ENABLED enabled=1 |
1 | 1 |
SELECT-DISPATCHES |
1 | 0 |
RESERVE-EVENTS count= |
2 | 0 |
RETIRE-DISPATCH |
4 | 4 |
PROFILING-END / WRITE-EVENTS / QUEUE-WRITE-EVENTS |
0 / 0 / 0 | 0 / 0 / 0 |
| bundle | 770 B | 770 B |
So the record-time half of the chain is complete and the flush is never reached. Confirmed by changing only the client (dropping the release so shutdown runs against a live handle): 770 B → 40 568 B with 2 real dispatch_events, 700 and 704 ns.
And confirmed by fixing the library, with the client untouched (7 arms, one gpu-hold, 2 s of device work; arms alternate stock/fixed/stock/fixed and each arm prints the sha256 of the library ldd resolved inside its own environment):
| arm | libhrx | bundle | content |
|---|---|---|---|
| stock graph | ed421353 (pin) |
770 B | begin-only |
| fixed graph | 4fd4866f |
40 568 B | session 2 (begin+end), command_buffer 1, command_operation 4, dispatch_event 2, queue_device_event 3, executable 1, executable_function 8 |
| stock graph (repeat) | ed421353 |
770 B | begin-only — control reproduces in the same hold |
| fixed graph (repeat) | 4fd4866f |
40 568 B | identical to the first fixed arm |
| stock stream | ed421353 |
770 B | begin-only |
| fixed stream | 4fd4866f |
39 234 B | session 2, queue_device_event 3, 0 dispatch events — correct, that is #894's missing RETAIN_PROFILE_METADATA on the stream path, a different bug, untouched by this fix |
fixed -init probe |
4fd4866f |
1 153 B | session 2 — an init/teardown probe now closes its session too |
Fix
1bit-MONSTER/hrx-system#12 (branch 1bit/profile-end-on-release, base 1bit/e3e6a976-pin-6): end a still-active session in hrx_device_release's refcount-0 branch, before hal_device is cleared, by exposing the existing hrx_device_profile_end() to device.c (11 lines, 3 files, no caller changes). The engine's explicit hrx_device_retain keeps its release out of that branch, so its behaviour is unchanged.
Rejected alternative: make hrx_gpu_device_get retain. It fixes the same defect at the other end and matches the name, but it changes the refcount contract for every caller and makes the engine's existing explicit retain redundant — an API decision for the owners, tracked separately, not a night-shift change.
Acceptance
c1-two-stream graph 2 0unmodified, at the pin + this fix, underHRX_PROFILE_MODE=dispatch: bundle containscommand_operation+dispatch_event+ a session-end record (satisfied above: 40 568 B, 2 events, on this boot).- Nothing changes when profiling is off (no session, no sink) — the branch is a no-op when
profiling_activeis false. - The stream arm still records no dispatch events until hrx: HRX_PROFILE_MODE records nothing for HIP-binding launches, and says nothing about it #894 lands (asserted here as a negative control).
Limits this leaves in place
- The borrowed-handle contract of
hrx_gpu_device_getis still unstated inhrx_runtime.h; this fix makes the release safe, it does not make the getter's semantics obvious. - The end-of-session wait (
iree_hal_device_wait_semaphoreswith an empty list, infinite timeout) now also runs on the release path, so a wedged device that an app releases will wait there instead of hanging later inhrx_gpu_shutdown. Same wait as the existingprofile_endpath, now reached from one more entry point.
Activity
Bump HRX: hrx-system 170d229a3 -- three libhrx/amdgpu profiling fixes…
Bump HRX: hrx-system 170d229a3 — three libhrx/amdgpu profiling fixes …
bong-water-water-bong commentedon Nov 6, 2026
Resolved. Fork PR 1bit-MONSTER/hrx-system#12 merged into the pin branch (
1bit/e3e6a976-pin-6@170d229a3), and the engine pin bump #1239 carries it;hw-gatevalidated the new pin on strixhalo.What the fix does, matching this issue's repair:
hrx_device_release's refcount-0 branch calls the existinghrx_device_profile_end()beforehal_deviceis cleared, so a C-API client that acquires withhrx_gpu_device_get(borrowed, refcount 1) and releases still ends the profiling session. The engine's caller is unaffected: its explicithrx_device_retainkeeps its release out of that branch, and the no-op-when-profiling-off property is preserved.Acceptance measured in the issue holds for the merged change: the client unmodified,
HRX_PROFILE_MODE=dispatch, graph arm bundle 770 B (begin-only) → 40 568 B with session 2 (begin+end), command_buffer 1, command_operation 4, dispatch_event 2, queue_device_event 3; the init/teardown probe closes its session too (1 153 B). The borrowed-handle semantics ofhrx_gpu_device_getremain an API decision, as the issue says — this change makes the release safe, not the getter obvious. Closing.