Repository navigation
test(therock): hold the env lock while spawning the sleeping child - #762
Conversation
wait_with_output_bounded_kills_a_slow_child_at_the_deadline and wait_with_output_bounded_waits_unbounded_when_no_timeout_is_given start a sleeping child by program name (`powershell` on Windows, `sleep` elsewhere), so the program is found through PATH. render_update_json_installs_progress_guard replaces the process-wide PATH with its own directory for the length of its body, under PROCESS_ENV_TEST_LOCK, which these two tests did not take. When they ran inside that window the spawn failed with "program not found" and the test panicked, intermittently and only in a full run, which also made the pre-push `cargo test` hook fail on unrelated changes. spawn_test_sleep now takes the same lock for the length of the spawn (poison-tolerant, so one failing test does not cascade). A new test holds the lock with PATH emptied, starts the child from another thread, and only then puts PATH back; without the lock in spawn_test_sleep it fails with the same "program not found" error. Signed-off-by: Eugene Volen <eugene.volen@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 3f3cc93e
Review — needs work
Full review of the whole change (one file, +43/-0 lines).
No ticket named; the PR description is a self-contained problem/cause/fix/verification write-up with no tracker reference, and it accurately describes the diff (test-only change inside #[cfg(test)] mod tests, nothing outside it touched).
Blocking
E2E tests (Strix Halo, Windows, current)is failing on the PR head commit — CI check on3f3cc93e(workflowE2E self-hosted), https://github.com/ROCm/rocm-cli/actions/runs/40511966249/job/122216068627
Confirmed viagh pr checks 762andgh pr view --json statusCheckRollup: conclusionFAILURE. The job's "Run E2E tests on Strix Halo Windows" step exits 1 and notests/e2e-cucumber/results/directory is produced to upload. This is a self-hosted real-hardware Cucumber E2E lane, distinct from the Rust unit-test lanes the diff actually touches —windows-build-and-test(which runscargo test --workspace --all-targets, single-process-threaded on Windows, i.e. exactly the harness the race in this PR lived in) andTest (affected crates)(nextest) both reportSUCCESSon this same commit. I was not able to fetch the failing job's detailed log (the Actions API returned 403 Forbidden for job/run endpoints on this token), so I cannot independently confirm it's an unrelated hardware/infra flake rather than something this change caused; per the review method's rule to take CI status at face value, a failing required check is reported as blocking rather than waved through.
Confidence 85 (aFAILUREconclusion read directly from CI on the exact commit under review) · architectural · Fix: before merging, establish whether this is a known-flaky self-hosted lane (rerun the job) or an actual regression; this diff gives no obvious mechanism to affect hardware E2E scenarios (it only changes#[cfg(test)]code inapps/rocm/src/therock.rs), which supports "unrelated," but that should be confirmed rather than assumed.
Non-blocking
comfyui.rs'sspawn_stand_in_processtest helper has the identical unguarded-PATH-lookup hazard this PR fixes fortherock.rs, pre-existing and untouched here —apps/rocm/src/comfyui.rs:4060(called at lines 3990, 4019, 4105, 4258, 4528, 4653)
spawn_stand_in_processspawns"ping"(Windows) /"sleep"(non-Windows) by bare name, resolved throughPATH, without holdingPROCESS_ENV_TEST_LOCKor any other lock.comfyui.rs,main.rsandtherock.rsare all modules of the samerocmcrate and therefore the same test binary/process.therock.rs's own comment onPROCESS_ENV_TEST_LOCK(apps/rocm/src/therock.rs:7261-7264) states the rule this violates verbatim: "any test that spawns a bare-name binary (e.g.tar) viaPATHlookup must also hold this lock, or it can fail with ENOENT while another test has temporarily narrowedPATH."therock.rs::render_update_json_installs_the_suppression_guard_around_resolution(around line 7728) is the test that actually narrows process-widePATH, under that same lock.main.rshas its own, separate env-mutation lock (ScopedTestEnv, its own privateMutex), confirmed it never setsPATH(onlyROCM_CLI_*config/data/cache vars and a handful of app-specific vars), so it provides no protection here either. This PR closes the race only fortherock.rs's own sleeping-child helper;comfyui.rs's equivalent helper remains exposed to the same intermittent "program not found" failure under any single-process threaded run (e.g.windows-build-and-test,cargo llvm-cov, or the e2e-cucumber lib harness named inxtask/src/env_mutation_contract.rs's own comments as other threaded runners).
Confidence 85 (traced: no lock at any of the 6 call sites; confirmed the only twoPATH-mutating tests in the whole crate are intherock.rs; confirmedmain.rs's lock never touchesPATH) · architectural · Fix: havespawn_stand_in_processtakePROCESS_ENV_TEST_LOCK(or a lock shared across the crate) for the length of its spawn, the same shape as this PR's own fix.
Decisions for the author
- Fixed-duration
sleepas the synchronization primitive in the new regression test — tradeoff
spawning_the_test_sleep_waits_while_another_test_has_replaced_path(apps/rocm/src/therock.rs:9316-9337) coordinates its two threads withstd::thread::sleep(Duration::from_millis(300))rather than a blocking handshake. If the spawned thread is scheduled late enough that it only reachesspawn_test_sleep's.lock()call after the main thread has already restoredPATHand dropped the lock, the test still passes but silently stops exercising the "waits for the lock" behavior for that run, rather than failing loudly. This is the same shape as the sleep-based synchronization already used elsewhere in the surrounding tests, so it's consistent with existing practice here rather than a new pattern — flagging only because this particular test's whole purpose is proving the wait, so a silent no-op run is a slightly worse failure mode for it than for the others.
Positive signals
lock_process_env_for_spawnnarrows the critical section to just theCommand::spawn()call rather than the whole test body, and that's correctly reasoned: oncespawn()returns, the child's executable has already been resolved fromPATH, so nothing afterward needs the lock — this keeps contention with the other ninePROCESS_ENV_TEST_LOCKholders in the file to a minimum.- Taking the lock poison-tolerantly (
unwrap_or_else(PoisonError::into_inner)) specifically inspawn_test_sleep, rather than changing every lock site in the file, is a deliberate and well-targeted choice: it's exactly the path (spawning a sleeping child for unrelated tests' budget/timeout checks) that must not cascade-fail just because some unrelated env test panicked first and poisoned the shared mutex. - The new test is a genuine "fails before the fix, passes with it" regression proof (the author says they verified this manually) rather than a restatement of the production code, matching this repo's own
AGENTS.mdbar for bug-fix tests.
Deployment notes
None — test-only change behind `#[cfg(test)]; no production code or runtime behavior is touched.
What this covered
Read the whole diff (apps/rocm/src/therock.rs, +43/-0) completely, plus its surrounding #[cfg(test)] mod tests block in full (all nine other PROCESS_ENV_TEST_LOCK call sites, to check the new lock usage actually interoperates with the existing ones and with the static analysis in xtask/src/env_mutation_contract.rs), apps/rocm/src/main.rs's separate ScopedTestEnv/BUILTIN_ENGINE_ENV_LOCK mechanism (to rule out an independent, unguarded PATH race there), apps/rocm/src/comfyui.rs's bare-name process spawns, and AGENTS.md for the applicable test/scenario rules (confirmed this change correctly falls under its "purely internal changes need no scenario" carve-out). Checked git log on the file for related prior flaky-test-fix commits for context. Diffed 24e405a7614576b350dbe47ba7b5cc317f760f33...3f3cc93ce68652da4ff31c44ed9cc290a66f7a74.
This was a single-context pass, not fanned out to sub-workers: the change is one file and 43 added lines, which sits well below the point where splitting the review buys anything — one worker (this one) could, and did, read the whole scope plus the surrounding context (including the two other modules checked for the same hazard class) completely. Not degraded.
Did not run: no Rust toolchain in this sandbox (cargo/rustc not found), so nothing was executed directly; relied on the repo's own CI results instead. Could not fetch the failing job's full step log (Actions API returned 403 Forbidden for job/run endpoints on this token); relied on gh pr checks plus a best-effort fetch of the failure summary page. The "prior changes" pass (do review comments on earlier changes to these files apply here too) could not be run — no mechanism was available here to read historical PR review threads for this file; git log/git blame history was used instead where relevant.
Not reconciled against any discussion on this PR — no review comments or prior rounds were read, per the review method's blind-review rule.
juhovainio
left a comment
There was a problem hiding this comment.
Looks good. Taking the env lock in both spawn_test_sleep variants closes the race where a test that has swapped PATH makes another test's spawn-by-name fail, and the new regression test exercises exactly that interleaving deterministically. I found no deadlock risk: the existing callers don't hold the lock when they call it.
Two small, non-blocking nits (one inline): the new test locks with .unwrap() rather than the poison-tolerant helper, and it has no guard restoring PATH if it panics mid-test.
CI note: the Strix Halo Windows e2e lane was red when I looked. The diff is #[cfg(test)]-only, so it's most likely the known runner flake, but I haven't confirmed that from the logs, and a few GPU lanes were still pending.
| #[test] | ||
| #[allow(unsafe_code)] // std::env::set_var is unsafe in edition 2024 | ||
| fn spawning_the_test_sleep_waits_while_another_test_has_replaced_path() { | ||
| let holder = PROCESS_ENV_TEST_LOCK.lock().unwrap(); |
There was a problem hiding this comment.
Nit: use the same poison-tolerant lock helper the new spawn_test_sleep uses here. With .unwrap(), one panicking env test poisons the mutex and this test then fails with a poison error instead of exercising the behavior. A small guard that restores PATH on drop would also keep a mid-test panic from leaking an empty PATH into other tests.
sdk_tarball_unpack_never_writes_outside_the_install_root and sdk_tarball_unpack_extracts_a_benign_tarball_into_the_install_root extract through extract_tarball_and_discard_archive, which spawns bare-name tar via PATH, but did not hold PROCESS_ENV_TEST_LOCK. A test that temporarily narrows PATH (line ~9320) could make a concurrent run of either property test fail to launch tar with ENOENT -- intermittently, and only in a full run, which also made the pre-push cargo test hook fail on unrelated changes (the same failure mode PR #762 fixed for spawn_test_sleep). Widen PROCESS_ENV_TEST_LOCK to pub(super) so archive_properties, a sibling module to tests, can take it, and hold it for the whole of each property test's run. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
) * Gate comfyui-01/02/03 on the required Windows lane PR ROCm#637 made these ComfyUI install scenarios hermetic and Windows-portable, but no required CI lane actually ran them on Windows: windows-build-and-test only runs @lifecycle scenarios, and the only Windows lane that did run them (e2e-selfhosted.yml) is optional and not a required check (ROCm#689). Add E2E_ALSO_RUN_IDS, a small opt-in mechanism that folds specific non-@lifecycle scenarios by @id into an E2E_ONLY_LIFECYCLE lane without retagging them, and point windows-build-and-test's lifecycle step at the three comfyui ids. Tagging them @lifecycle instead was rejected: it would silently drop them from nightly.yml's and e2e-selfhosted.yml's full-suite runs, neither of which sets E2E_INCLUDE_LIFECYCLE. Signed-off-by: Jussi Elo <jussi.elo@amd.com> * fmt: satisfy cargo fmt for lifecycle_lane_includes prek's cargo-fmt hook failed CI; rustfmt wraps the condition and two test call sites differently than written. Signed-off-by: Jussi Elo <jussi.elo@amd.com> * test(therock): hold the env lock in the SDK archive property tests sdk_tarball_unpack_never_writes_outside_the_install_root and sdk_tarball_unpack_extracts_a_benign_tarball_into_the_install_root extract through extract_tarball_and_discard_archive, which spawns bare-name tar via PATH, but did not hold PROCESS_ENV_TEST_LOCK. A test that temporarily narrows PATH (line ~9320) could make a concurrent run of either property test fail to launch tar with ENOENT -- intermittently, and only in a full run, which also made the pre-push cargo test hook fail on unrelated changes (the same failure mode PR ROCm#762 fixed for spawn_test_sleep). Widen PROCESS_ENV_TEST_LOCK to pub(super) so archive_properties, a sibling module to tests, can take it, and hold it for the whole of each property test's run. Signed-off-by: Jussi Elo <jussi.elo@amd.com> * test(e2e-cucumber): fail loudly on an unmatched E2E_ALSO_RUN_IDS entry An id listed in E2E_ALSO_RUN_IDS that matches no scenario's @id was silently ignored, so a renamed comfyui @id would quietly drop the Windows lifecycle lane's coverage while CI stayed green. Check every listed id against the ids actually seen in the suite once the run completes, and panic naming whichever didn't match. Signed-off-by: Jussi Elo <jussi.elo@amd.com> * test(xtask): assert the real E2E_ALSO_RUN_IDS env line, not a comment The Windows lifecycle lane's contract test checked step.contains the bare var name, which the explanatory comment above env: also satisfies. Use flattened_values (comments stripped) so only the real env: entry can pass, and assert the three comfyui ids against its value rather than the whole step text. Signed-off-by: Jussi Elo <jussi.elo@amd.com> --------- Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Problem
therock::tests::wait_with_output_bounded_kills_a_slow_child_at_the_deadlineandwait_with_output_bounded_waits_unbounded_when_no_timeout_is_givenfail intermittently, only in a fullcargo test -p rocm --bin rocmrun, with:They pass when run alone. The failure also makes the pre-push
cargo testhook fail on changes that have nothing to do with them (it blocked several pushes in one day).Cause
Both tests start a sleeping child by program name (
powershellon Windows,sleepelsewhere), so the program is found throughPATH.render_update_json_installs_progress_guard…replaces the process-widePATHwith its own directory for the length of its body, underPROCESS_ENV_TEST_LOCK. The sleep tests did not take that lock, so a spawn that landed inside that window found nopowershellorsleepand panicked. Tests run in parallel in one process, which is why it is intermittent.Change
spawn_test_sleep(both the Windows and the non-Windows variant) takesPROCESS_ENV_TEST_LOCKfor the length of the spawn. The lock is taken poison-tolerantly, so one failing test does not make every later spawn fail. Test code only; nothing outside#[cfg(test)]changes.Verification
spawning_the_test_sleep_waits_while_another_test_has_replaced_pathholds the lock withPATHemptied, starts the child from another thread, and only then putsPATHback. Without the lock inspawn_test_sleepit fails with the sameprogram not founderror; with it, it passes. I removed the lock to confirm that, then restored the change.-p rocm --bin rocmpass repeated enough times to show the intermittent failure is gone (it is a race, so a clean run proves little; the deterministic test above is the evidence), the non-Windows variant (sleep) on Linux, and the fullcargo test --workspace.Scope and scenarios
Test-only change with no user-observable behaviour, so there is no scenario to add (AGENTS.md section 3: purely internal changes need none).
Other timing-sensitive tests exist that can fail when the machine is heavily loaded (for example in
rocm-dash-tuiandrocm-dash-daemon). They are a separate cause, load starvation, and are not touched here.