Windows HIP, 2× R9700: --peer-device hangs to driver timeout. Launch batching reproduced in a small probe #3767

Hi, a follow-up on the --peer-device hang on Windows (2x Radeon AI PRO R9700, driver 32.0.31041.5005 for ROCm 10.1; it also hung on PRO 26.Q3, ending in a driver timeout with 17 LiveKernelEvent records).

I think I've reproduced the mechanism in a 190-line HIP probe. Two single-thread kernels, one per card, ping-pong a flag through one hipHostMalloc(Coherent | Mapped | Portable) buffer (same address on the CPU and both cards), using system-scope release/acquire, with every wait bounded by time so it can't hang the driver.

• With a timing-free event recorded right after each launch (hipEventCreateWithFlags(hipEventDisableTiming) + hipEventRecord on the same stream), 10,000 round trips complete in both directions, mean 2.8 us.
• Without that, the host launches both kernels and then waits on the initiator's card. The responder's kernel is never submitted: the initiator spins for its full 400 ms bound with no answer. Only when the host synchronizes the responder's card does it start, finds the stale ping and answers it. Host time 450 ms = 400 + 50, both directions, every time.

So on Windows a launch can sit in the runtime's batch indefinitely until something submits that card's work (an event record, a sync, or the batch filling). Any kernel that spins waiting for a kernel on the other card will hang until TDR, unless each card's launch is flushed right after it's made.

If the peer tier exchanges through host memory, recording a timing-free event after every launch the other card waits on should fix it. That's what Radiance's Windows backend does after each collective. If it reads the other card's VRAM directly, note that this driver reports no peer access (hipDeviceCanAccessPeer = 0 both ways), so that path can't work on Windows regardless.

Happy to share the probe source or test a build.