Repository navigation
[release/2.13] [ROCm] Calibrate GPU sleep delays in stream tests (#197541) - #3947
Open
amd-maradosa wants to merge 1 commit into
Open
amd-maradosa wants to merge 1 commit into
amd-maradosa wants to merge 1 commit into
Conversation
The stream-synchronization tests in test/test_autograd.py request their 0.5 s producer delay as a fixed 1e9 GPU cycles (NUM_GPU_CYCLES_IN_ONE_SEC // 2). On gfx11, torch.cuda._sleep counts wall_clock64() ticks at a fixed 100 MHz, so each delay is a single 10 s kernel. On Windows, if anything else needs the GPU while that kernel runs, the GPU watchdog resets the engine and the tests fail with hipErrorLaunchFailure. Compute the delay with get_cycles_per_ms() before building the graph. This branch predates pytorch#193535, so in the tests that still skip CPU inside the test body, the calibration runs after that skip. Pull Request resolved: pytorch#197541 Approved by: https://github.com/jeffdaily (cherry picked from commit d1133c1)
This was referenced Nov 5, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of upstream pytorch#197541 (commit d1133c1) into release/2.13.
Why: the stream-synchronization tests in
test/test_autograd.pyrequest their producer delay as a fixed 1e9 GPU cycles. On gfx11,torch.cuda._sleepcountswall_clock64()ticks at 100 MHz, so each delay is a single 10 s kernel. On Windows, if anything else needs the GPU while that kernel runs, the GPU watchdog resets the engine and the tests fail withhipErrorLaunchFailure; the remaining tests in the process then fail insetUp.This branch predates pytorch#193535, so three of the tests still skip CPU inside the test body; in those, the calibration runs after the skip. Otherwise the changes are identical to upstream.
Tested on Windows with Strix Halo (gfx1151) and torch 2.13.0+rocm10.3.0a20261105:
TestAutogradStreamSynchronizationCUDApassed in 10/10 runs (test_consumer_to_single_producer_case_2_correctness20.3 s -> 1.1 s,test_side_stream_backward_overlap30.0 s -> 1.5 s) with no GPU watchdog events.