Skip to content

test(therock): hold the env lock while spawning the sleeping child - #762

Merged
volen-silo merged 1 commit into
mainfrom
fix/therock-sleep-tests-path-race
Oct 27, 2026
Merged

volen-silo merged 1 commit into
mainfrom
fix/therock-sleep-tests-path-race

Conversation

@volen-silo

Copy link
Copy Markdown
Collaborator

Problem

therock::tests::wait_with_output_bounded_kills_a_slow_child_at_the_deadline and wait_with_output_bounded_waits_unbounded_when_no_timeout_is_given fail intermittently, only in a full cargo test -p rocm --bin rocm run, with:

spawn Start-Sleep: Error { kind: NotFound, message: "program not found" }

They pass when run alone. The failure also makes the pre-push cargo test hook fail on changes that have nothing to do with them (it blocked several pushes in one day).

Cause

Both tests start a sleeping child by program name (powershell on Windows, sleep elsewhere), so the program is found through PATH. render_update_json_installs_progress_guard… replaces the process-wide PATH with its own directory for the length of its body, under PROCESS_ENV_TEST_LOCK. The sleep tests did not take that lock, so a spawn that landed inside that window found no powershell or sleep and panicked. Tests run in parallel in one process, which is why it is intermittent.

Change

spawn_test_sleep (both the Windows and the non-Windows variant) takes PROCESS_ENV_TEST_LOCK for the length of the spawn. The lock is taken poison-tolerantly, so one failing test does not make every later spawn fail. Test code only; nothing outside #[cfg(test)] changes.

Verification

  • New test spawning_the_test_sleep_waits_while_another_test_has_replaced_path holds the lock with PATH emptied, starts the child from another thread, and only then puts PATH back. Without the lock in spawn_test_sleep it fails with the same program not found error; with it, it passes. I removed the lock to confirm that, then restored the change.
  • The two sleep tests and the new one pass 3 runs out of 3 on Windows 11, run together by name.
  • Not run: a full -p rocm --bin rocm pass repeated enough times to show the intermittent failure is gone (it is a race, so a clean run proves little; the deterministic test above is the evidence), the non-Windows variant (sleep) on Linux, and the full cargo test --workspace.

Scope and scenarios

Test-only change with no user-observable behaviour, so there is no scenario to add (AGENTS.md section 3: purely internal changes need none).

Other timing-sensitive tests exist that can fail when the machine is heavily loaded (for example in rocm-dash-tui and rocm-dash-daemon). They are a separate cause, load starvation, and are not touched here.

wait_with_output_bounded_kills_a_slow_child_at_the_deadline and
wait_with_output_bounded_waits_unbounded_when_no_timeout_is_given start a
sleeping child by program name (`powershell` on Windows, `sleep` elsewhere),
so the program is found through PATH. render_update_json_installs_progress_guard
replaces the process-wide PATH with its own directory for the length of its
body, under PROCESS_ENV_TEST_LOCK, which these two tests did not take. When
they ran inside that window the spawn failed with "program not found" and the
test panicked, intermittently and only in a full run, which also made the
pre-push `cargo test` hook fail on unrelated changes.

spawn_test_sleep now takes the same lock for the length of the spawn
(poison-tolerant, so one failing test does not cascade). A new test holds the
lock with PATH emptied, starts the child from another thread, and only then
puts PATH back; without the lock in spawn_test_sleep it fails with the same
"program not found" error.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
volen-silo requested a review from a team as a code owner October 27, 2026 13:25
@volen-silo
volen-silo requested a review from johnl-amd October 27, 2026 13:25

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 3f3cc93e

Review — needs work

Full review of the whole change (one file, +43/-0 lines).
No ticket named; the PR description is a self-contained problem/cause/fix/verification write-up with no tracker reference, and it accurately describes the diff (test-only change inside #[cfg(test)] mod tests, nothing outside it touched).

Blocking

  • E2E tests (Strix Halo, Windows, current) is failing on the PR head commit — CI check on 3f3cc93e (workflow E2E self-hosted), https://github.com/ROCm/rocm-cli/actions/runs/40511966249/job/122216068627
    Confirmed via gh pr checks 762 and gh pr view --json statusCheckRollup: conclusion FAILURE. The job's "Run E2E tests on Strix Halo Windows" step exits 1 and no tests/e2e-cucumber/results/ directory is produced to upload. This is a self-hosted real-hardware Cucumber E2E lane, distinct from the Rust unit-test lanes the diff actually touches — windows-build-and-test (which runs cargo test --workspace --all-targets, single-process-threaded on Windows, i.e. exactly the harness the race in this PR lived in) and Test (affected crates) (nextest) both report SUCCESS on this same commit. I was not able to fetch the failing job's detailed log (the Actions API returned 403 Forbidden for job/run endpoints on this token), so I cannot independently confirm it's an unrelated hardware/infra flake rather than something this change caused; per the review method's rule to take CI status at face value, a failing required check is reported as blocking rather than waved through.
    Confidence 85 (a FAILURE conclusion read directly from CI on the exact commit under review) · architectural · Fix: before merging, establish whether this is a known-flaky self-hosted lane (rerun the job) or an actual regression; this diff gives no obvious mechanism to affect hardware E2E scenarios (it only changes #[cfg(test)] code in apps/rocm/src/therock.rs), which supports "unrelated," but that should be confirmed rather than assumed.

Non-blocking

  • comfyui.rs's spawn_stand_in_process test helper has the identical unguarded-PATH-lookup hazard this PR fixes for therock.rs, pre-existing and untouched here — apps/rocm/src/comfyui.rs:4060 (called at lines 3990, 4019, 4105, 4258, 4528, 4653)
    spawn_stand_in_process spawns "ping" (Windows) / "sleep" (non-Windows) by bare name, resolved through PATH, without holding PROCESS_ENV_TEST_LOCK or any other lock. comfyui.rs, main.rs and therock.rs are all modules of the same rocm crate and therefore the same test binary/process. therock.rs's own comment on PROCESS_ENV_TEST_LOCK (apps/rocm/src/therock.rs:7261-7264) states the rule this violates verbatim: "any test that spawns a bare-name binary (e.g. tar) via PATH lookup must also hold this lock, or it can fail with ENOENT while another test has temporarily narrowed PATH." therock.rs::render_update_json_installs_the_suppression_guard_around_resolution (around line 7728) is the test that actually narrows process-wide PATH, under that same lock. main.rs has its own, separate env-mutation lock (ScopedTestEnv, its own private Mutex), confirmed it never sets PATH (only ROCM_CLI_* config/data/cache vars and a handful of app-specific vars), so it provides no protection here either. This PR closes the race only for therock.rs's own sleeping-child helper; comfyui.rs's equivalent helper remains exposed to the same intermittent "program not found" failure under any single-process threaded run (e.g. windows-build-and-test, cargo llvm-cov, or the e2e-cucumber lib harness named in xtask/src/env_mutation_contract.rs's own comments as other threaded runners).
    Confidence 85 (traced: no lock at any of the 6 call sites; confirmed the only two PATH-mutating tests in the whole crate are in therock.rs; confirmed main.rs's lock never touches PATH) · architectural · Fix: have spawn_stand_in_process take PROCESS_ENV_TEST_LOCK (or a lock shared across the crate) for the length of its spawn, the same shape as this PR's own fix.

Decisions for the author

  • Fixed-duration sleep as the synchronization primitive in the new regression test — tradeoff
    spawning_the_test_sleep_waits_while_another_test_has_replaced_path (apps/rocm/src/therock.rs:9316-9337) coordinates its two threads with std::thread::sleep(Duration::from_millis(300)) rather than a blocking handshake. If the spawned thread is scheduled late enough that it only reaches spawn_test_sleep's .lock() call after the main thread has already restored PATH and dropped the lock, the test still passes but silently stops exercising the "waits for the lock" behavior for that run, rather than failing loudly. This is the same shape as the sleep-based synchronization already used elsewhere in the surrounding tests, so it's consistent with existing practice here rather than a new pattern — flagging only because this particular test's whole purpose is proving the wait, so a silent no-op run is a slightly worse failure mode for it than for the others.

Positive signals

  • lock_process_env_for_spawn narrows the critical section to just the Command::spawn() call rather than the whole test body, and that's correctly reasoned: once spawn() returns, the child's executable has already been resolved from PATH, so nothing afterward needs the lock — this keeps contention with the other nine PROCESS_ENV_TEST_LOCK holders in the file to a minimum.
  • Taking the lock poison-tolerantly (unwrap_or_else(PoisonError::into_inner)) specifically in spawn_test_sleep, rather than changing every lock site in the file, is a deliberate and well-targeted choice: it's exactly the path (spawning a sleeping child for unrelated tests' budget/timeout checks) that must not cascade-fail just because some unrelated env test panicked first and poisoned the shared mutex.
  • The new test is a genuine "fails before the fix, passes with it" regression proof (the author says they verified this manually) rather than a restatement of the production code, matching this repo's own AGENTS.md bar for bug-fix tests.

Deployment notes

None — test-only change behind `#[cfg(test)]; no production code or runtime behavior is touched.

What this covered

Read the whole diff (apps/rocm/src/therock.rs, +43/-0) completely, plus its surrounding #[cfg(test)] mod tests block in full (all nine other PROCESS_ENV_TEST_LOCK call sites, to check the new lock usage actually interoperates with the existing ones and with the static analysis in xtask/src/env_mutation_contract.rs), apps/rocm/src/main.rs's separate ScopedTestEnv/BUILTIN_ENGINE_ENV_LOCK mechanism (to rule out an independent, unguarded PATH race there), apps/rocm/src/comfyui.rs's bare-name process spawns, and AGENTS.md for the applicable test/scenario rules (confirmed this change correctly falls under its "purely internal changes need no scenario" carve-out). Checked git log on the file for related prior flaky-test-fix commits for context. Diffed 24e405a7614576b350dbe47ba7b5cc317f760f33...3f3cc93ce68652da4ff31c44ed9cc290a66f7a74.

This was a single-context pass, not fanned out to sub-workers: the change is one file and 43 added lines, which sits well below the point where splitting the review buys anything — one worker (this one) could, and did, read the whole scope plus the surrounding context (including the two other modules checked for the same hazard class) completely. Not degraded.

Did not run: no Rust toolchain in this sandbox (cargo/rustc not found), so nothing was executed directly; relied on the repo's own CI results instead. Could not fetch the failing job's full step log (Actions API returned 403 Forbidden for job/run endpoints on this token); relied on gh pr checks plus a best-effort fetch of the failure summary page. The "prior changes" pass (do review comments on earlier changes to these files apply here too) could not be run — no mechanism was available here to read historical PR review threads for this file; git log/git blame history was used instead where relevant.
Not reconciled against any discussion on this PR — no review comments or prior rounds were read, per the review method's blind-review rule.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Taking the env lock in both spawn_test_sleep variants closes the race where a test that has swapped PATH makes another test's spawn-by-name fail, and the new regression test exercises exactly that interleaving deterministically. I found no deadlock risk: the existing callers don't hold the lock when they call it.

Two small, non-blocking nits (one inline): the new test locks with .unwrap() rather than the poison-tolerant helper, and it has no guard restoring PATH if it panics mid-test.

CI note: the Strix Halo Windows e2e lane was red when I looked. The diff is #[cfg(test)]-only, so it's most likely the known runner flake, but I haven't confirmed that from the logs, and a few GPU lanes were still pending.

Comment thread apps/rocm/src/therock.rs
#[test]
#[allow(unsafe_code)] // std::env::set_var is unsafe in edition 2024
fn spawning_the_test_sleep_waits_while_another_test_has_replaced_path() {
let holder = PROCESS_ENV_TEST_LOCK.lock().unwrap();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: use the same poison-tolerant lock helper the new spawn_test_sleep uses here. With .unwrap(), one panicking env test poisons the mutex and this test then fails with a poison error instead of exercising the behavior. A small guard that restores PATH on drop would also keep a mid-test panic from leaking an empty PATH into other tests.

@volen-silo
volen-silo added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit bbd80bf Oct 27, 2026
28 of 31 checks passed
@volen-silo
volen-silo deleted the fix/therock-sleep-tests-path-race branch October 27, 2026 15:36
jussielo-amd added a commit that referenced this pull request Oct 28, 2026
sdk_tarball_unpack_never_writes_outside_the_install_root and
sdk_tarball_unpack_extracts_a_benign_tarball_into_the_install_root
extract through extract_tarball_and_discard_archive, which spawns
bare-name tar via PATH, but did not hold PROCESS_ENV_TEST_LOCK. A test
that temporarily narrows PATH (line ~9320) could make a concurrent run
of either property test fail to launch tar with ENOENT -- intermittently,
and only in a full run, which also made the pre-push cargo test hook
fail on unrelated changes (the same failure mode PR #762 fixed for
spawn_test_sleep).

Widen PROCESS_ENV_TEST_LOCK to pub(super) so archive_properties, a
sibling module to tests, can take it, and hold it for the whole of each
property test's run.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Woody-Hu pushed a commit to Woody-Hu/rocm-cli that referenced this pull request Oct 29, 2026
)

* Gate comfyui-01/02/03 on the required Windows lane

PR ROCm#637 made these ComfyUI install scenarios hermetic and Windows-portable,
but no required CI lane actually ran them on Windows: windows-build-and-test
only runs @lifecycle scenarios, and the only Windows lane that did run them
(e2e-selfhosted.yml) is optional and not a required check (ROCm#689).

Add E2E_ALSO_RUN_IDS, a small opt-in mechanism that folds specific
non-@lifecycle scenarios by @id into an E2E_ONLY_LIFECYCLE lane without
retagging them, and point windows-build-and-test's lifecycle step at the
three comfyui ids. Tagging them @lifecycle instead was rejected: it would
silently drop them from nightly.yml's and e2e-selfhosted.yml's full-suite
runs, neither of which sets E2E_INCLUDE_LIFECYCLE.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

* fmt: satisfy cargo fmt for lifecycle_lane_includes

prek's cargo-fmt hook failed CI; rustfmt wraps the condition and two
test call sites differently than written.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

* test(therock): hold the env lock in the SDK archive property tests

sdk_tarball_unpack_never_writes_outside_the_install_root and
sdk_tarball_unpack_extracts_a_benign_tarball_into_the_install_root
extract through extract_tarball_and_discard_archive, which spawns
bare-name tar via PATH, but did not hold PROCESS_ENV_TEST_LOCK. A test
that temporarily narrows PATH (line ~9320) could make a concurrent run
of either property test fail to launch tar with ENOENT -- intermittently,
and only in a full run, which also made the pre-push cargo test hook
fail on unrelated changes (the same failure mode PR ROCm#762 fixed for
spawn_test_sleep).

Widen PROCESS_ENV_TEST_LOCK to pub(super) so archive_properties, a
sibling module to tests, can take it, and hold it for the whole of each
property test's run.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

* test(e2e-cucumber): fail loudly on an unmatched E2E_ALSO_RUN_IDS entry

An id listed in E2E_ALSO_RUN_IDS that matches no scenario's @id was
silently ignored, so a renamed comfyui @id would quietly drop the
Windows lifecycle lane's coverage while CI stayed green. Check every
listed id against the ids actually seen in the suite once the run
completes, and panic naming whichever didn't match.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

* test(xtask): assert the real E2E_ALSO_RUN_IDS env line, not a comment

The Windows lifecycle lane's contract test checked step.contains the
bare var name, which the explanatory comment above env: also
satisfies. Use flattened_values (comments stripped) so only the real
env: entry can pass, and assert the three comfyui ids against its
value rather than the whole step text.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

---------

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants