Skip to content

hrx: a profiled libhrx C-API client loses its whole session at teardown — hrx_device_release clears the handle before hrx_gpu_shutdown can end profiling #1232

Closed

Description

@bong-water-water-bong

What happens

A libhrx C-API client that profiles (HRX_PROFILE_MODE=dispatch) and follows the documented acquire/release pairing produces a 770-byte bundle with no dispatch events and no session-end record, even though the HAL did profile the work.

bundle: schema, session(1), device, queue, clock_correlation   = 770 B
expected: + session end, command_buffer, command_operation, dispatch_event, queue_device_event

The events are not lost at record time — they are reserved and retired — and they are not lost by the ring either (#895). They are lost at teardown, because nothing ever ends the session.

Mechanism (citation per step, all at pin 82804197bcbe)

  1. hrx_gpu_device_get returns a borrowed pointer into global state: *device = &g_gpu.devices[index]; with no hrx_device_retain — libhrx/src/libhrx/runtime.c:1193-1207. Nothing in hrx_runtime.h's contract for that call says the handle is borrowed.
  2. The device object starts at refcount 1 (iree_atomic_ref_count_init, same file), while hrx_device_retain/hrx_device_release are the documented pairing (device.c:162-186).
  3. So the natural client pattern — get, …, release, then shutdown — takes the teardown branch on that single release: if (iree_atomic_ref_count_dec(&device->ref_count) == 1) → device->hal_device = NULL (device.c:177-186), and the HAL device reference is dropped.
  4. hrx_gpu_shutdown() then calls hrx_gpu_end_all_profiling() → hrx_device_profile_end(), which returns iree_ok_status() early on !device->hal_device (runtime.c:687-691). iree_hal_device_profiling_end is never called, so the sink gets nothing but the begin-time metadata iree_hal_device_profiling_begin wrote at session start.
  5. hrx_gpu_shutdown() returns success. The client is told its profiled run went fine and its capture is empty.

Two callers in one repository, one that knows and one that cannot:

caller acquire profiling reaches iree_hal_device_profiling_end?
engine (ggml/src/ggml-hrx/ggml-hrx.cpp:1425-1428) hrx_gpu_device_get(i, &hrx_device) then hrx_device_retain(hrx_device) yes — refcount 2, so hrx_device_release at :99 goes 2→1 and hrx_gpu_shutdown() (:103) ends the session against a live handle
C-API client (c1-two-stream.c:243) hrx_gpu_device_get(0, &device) and nothing else no — refcount 1, so the release goes 1→0, clears hal_device, and profiling_end no-ops

Same teardown order in both. The difference is only whether the caller pays for a reference it was never given.

Measured

Instrumented at the HAL (gdb breakpoints on the build-tree libhrx, which keeps .symtab), graph arm vs stream control, HRX_PROFILE_MODE=dispatch:

stage graph stream
HSA-PROFILING-ENABLED enabled=1 1 1
SELECT-DISPATCHES 1 0
RESERVE-EVENTS count= 2 0
RETIRE-DISPATCH 4 4
PROFILING-END / WRITE-EVENTS / QUEUE-WRITE-EVENTS 0 / 0 / 0 0 / 0 / 0
bundle 770 B 770 B

So the record-time half of the chain is complete and the flush is never reached. Confirmed by changing only the client (dropping the release so shutdown runs against a live handle): 770 B → 40 568 B with 2 real dispatch_events, 700 and 704 ns.

And confirmed by fixing the library, with the client untouched (7 arms, one gpu-hold, 2 s of device work; arms alternate stock/fixed/stock/fixed and each arm prints the sha256 of the library ldd resolved inside its own environment):

arm libhrx bundle content
stock graph ed421353 (pin) 770 B begin-only
fixed graph 4fd4866f 40 568 B session 2 (begin+end), command_buffer 1, command_operation 4, dispatch_event 2, queue_device_event 3, executable 1, executable_function 8
stock graph (repeat) ed421353 770 B begin-only — control reproduces in the same hold
fixed graph (repeat) 4fd4866f 40 568 B identical to the first fixed arm
stock stream ed421353 770 B begin-only
fixed stream 4fd4866f 39 234 B session 2, queue_device_event 3, 0 dispatch events — correct, that is #894's missing RETAIN_PROFILE_METADATA on the stream path, a different bug, untouched by this fix
fixed -init probe 4fd4866f 1 153 B session 2 — an init/teardown probe now closes its session too

Fix

1bit-MONSTER/hrx-system#12 (branch 1bit/profile-end-on-release, base 1bit/e3e6a976-pin-6): end a still-active session in hrx_device_release's refcount-0 branch, before hal_device is cleared, by exposing the existing hrx_device_profile_end() to device.c (11 lines, 3 files, no caller changes). The engine's explicit hrx_device_retain keeps its release out of that branch, so its behaviour is unchanged.

Rejected alternative: make hrx_gpu_device_get retain. It fixes the same defect at the other end and matches the name, but it changes the refcount contract for every caller and makes the engine's existing explicit retain redundant — an API decision for the owners, tracked separately, not a night-shift change.

Acceptance

  • c1-two-stream graph 2 0 unmodified, at the pin + this fix, under HRX_PROFILE_MODE=dispatch: bundle contains command_operation + dispatch_event + a session-end record (satisfied above: 40 568 B, 2 events, on this boot).
  • Nothing changes when profiling is off (no session, no sink) — the branch is a no-op when profiling_active is false.
  • The stream arm still records no dispatch events until hrx: HRX_PROFILE_MODE records nothing for HIP-binding launches, and says nothing about it #894 lands (asserted here as a negative control).

Limits this leaves in place

  • The borrowed-handle contract of hrx_gpu_device_get is still unstated in hrx_runtime.h; this fix makes the release safe, it does not make the getter's semantics obvious.
  • The end-of-session wait (iree_hal_device_wait_semaphores with an empty list, infinite timeout) now also runs on the release path, so a wedged device that an app releases will wait there instead of hanging later in hrx_gpu_shutdown. Same wait as the existing profile_end path, now reached from one more entry point.

Activity

  1. bong-water-water-bong commented on Nov 6, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Resolved. Fork PR 1bit-MONSTER/hrx-system#12 merged into the pin branch (1bit/e3e6a976-pin-6 @ 170d229a3), and the engine pin bump #1239 carries it; hw-gate validated the new pin on strixhalo.

    What the fix does, matching this issue's repair: hrx_device_release's refcount-0 branch calls the existing hrx_device_profile_end() before hal_device is cleared, so a C-API client that acquires with hrx_gpu_device_get (borrowed, refcount 1) and releases still ends the profiling session. The engine's caller is unaffected: its explicit hrx_device_retain keeps its release out of that branch, and the no-op-when-profiling-off property is preserved.

    Acceptance measured in the issue holds for the merged change: the client unmodified, HRX_PROFILE_MODE=dispatch, graph arm bundle 770 B (begin-only) → 40 568 B with session 2 (begin+end), command_buffer 1, command_operation 4, dispatch_event 2, queue_device_event 3; the init/teardown probe closes its session too (1 153 B). The borrowed-handle semantics of hrx_gpu_device_get remain an API decision, as the issue says — this change makes the release safe, not the getter obvious. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghrxThe HRX (HIP/IREE) backend route

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions