Repository navigation
fix(rocm-core): report a timed-out rocm fix step as such (ROCMAI-345) - #1062
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
The fix runners read the probe adapter's collapsed exit code, so a helper stopped at the one-minute bound was reported as "usermod exited 124; group membership NOT changed." That names no timeout, offers no way to recover, and asserts a state nobody checked: the executor stops only the direct child, so usermod under sudo can still finish. Run each step through the bounded executor and report from its typed outcome. A command that exits non-zero or cannot start still says the change was NOT made; one that times out, is signalled or cannot be waited for now says the change may or may not have been made, names a command to check it, and the rocm fix command to re-run. The line and exit code 4 come from one value, and the re-run ids are the catalog's own constants. On Windows, a PowerShell read of the current User/Machine value that did not complete used to be taken as "unset": fix-2 then reported the override as gone and exited 0, and fix-9 skipped its already-set guard and ran setx. Both now stop with exit 4 before planning or prompting, as fix-6 already did. Covered by unit tests over an injected command runner for every outcome of each runner, and by e2e scenarios diagnose-15 (extended), 39 (tightened) and the new 62 (Linux helper never finishes), 63 and 64 (Windows reads fail). Docs, --help and the rocm-doctor skill describe the new exit-4 cases. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
… precisely (ROCMAI-345) diagnose-62 waits out fix-4's real one-minute timeout. Its step ran the CLI synchronously inside the single future the cucumber suite is polled on, so every concurrent scenario stalled for that minute and the TUI scenarios beside it overran their wall-clock caps, failing the hosted E2E lanes. The step now waits for the CLI on spawn_blocking. On Windows, the timeout line of a setx or PowerShell write said "Check with" before a PowerShell expression that cmd cannot run. Step now carries where its check runs, and those lines say "Check in PowerShell with". The Windows runners for fix-2, fix-6 and fix-9 write their own lines through injected stdout sinks, so the unit tests assert what is printed together with the exit code and what was run: a read that exits non-zero stops each fix with exit 4 and nothing written, a failed read prints no plan, and a dry run with a readable scope previews without running setx. diagnose-64 now previews fix-9 with --dry-run instead of applying it. The empty PATH hides PowerShell but not setx.exe, so a regressed read guard could otherwise set the runner's real User-scope HIP_VISIBLE_DEVICES. README, --help, the exit-code docs and the rocm-doctor skill now say exactly which fixes this covers: on Windows, fix-2, fix-6 and fix-9 with --device-index read the current value first, and a failed read exits 4 with no plan, under --dry-run too; the catalog entry printed before it is not a preview. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
7c0de9f to
f4eaca0
Compare
…(ROCMAI-345) The runners.rs entry promised only print, dry-run and consent. On Windows fix-2, fix-6 and fix-9 now read the current env-scope value first and stop with exit 4, before the runner's own preview or any prompt, when that read fails, dry-run included; the entry should say so. "Consent first" becomes "consent before any write", and the report-only arms (fix-2 on Linux, fix-9 without a device index) are named as the exception, as the module header already does. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…MAI-345) The skill is federated from main ahead of a release, and its Phase 0 installs the release-channel CLI. A released CLI reports a timed-out fix step as "<program> exited 124; <what> NOT changed.", so an agent following the skill would tell the user nothing changed when the step may have landed. Treat that line as a timeout: say the change may or may not have landed, give the user the check for that fix (the group list on Linux, the User-scope variable on Windows), and offer the same re-run. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
No blocking correctness problems found; requesting changes for the items below.
To address
- The two wording-only tests in
fix/consent.rs(step_failure_reports_each_outcome_with_its_own_wording,a_failed_read_says_nothing_was_changed) should carry a comment naming the test that proves the behavior behind the message (AGENTS §3). The PR body lists this as a follow-up; it is cheap to do here. - The skill's older-CLI handling is partial:
exited 124is covered, but a signal-killed step on a released CLI (exited -1; ... NOT changed.) is the same may-have-landed case, and older CLIs fold a failed fix-2/fix-9 read into "unset". - The stale comments you list as open (
fix.rsmodule doc, thereport_persistent_overridedoc calling fix-2auto_applicable, the "This PR made..." test comment) are outside the diff, so I could not check them. Please fix them in this PR.
Questions / tradeoffs
- Every Windows read is now bounded at 8s with no retry, and fix-2 reads twice. A cold PowerShell start slower than 8s now stops with exit 4 where fix-2/fix-9 previously proceeded on a collapsed "unset". Safer, but a new user-visible failure mode; is 8s the right bound?
- diagnose-62 waits a real 60s on every lane that runs it.
- diagnose-39/63/64 run only on non-required self-hosted Windows lanes, so the applied fix-9 path is covered by unit tests only.
Positives: the failure line and exit code derive from one value; the runner tests assert exit code, commands run, and stderr together; Windows-gated logic is testable on every host. CI is green apart from a pending CodeQL run. The change matches the timeout / child-failure / recovery-message scope of ROCMAI-345.
| /// and only a command that ran to its own end (or never started) is | ||
| /// reported as having changed nothing. | ||
| #[test] | ||
| fn step_failure_reports_each_outcome_with_its_own_wording() { |
There was a problem hiding this comment.
This test pins the wording of each failure line, which certifies the string rather than the behavior it claims. Add a comment naming the runner tests that prove the behavior behind the line (e.g. render_group_timeout_says_timed_out_and_how_to_check_not_that_nothing_changed for the timeout case and unset_override_user_read_timeout_stops_before_anything_is_reported_or_cleared for the read failure), so a later edit cannot keep the string while the behavior drifts.
There was a problem hiding this comment.
Done in d9e9fa1. The test now carries a comment naming the runner tests that prove each line: render_group_child_failure_says_nothing_changed_and_returns_4 and unset_override_setx_failure_returns_4 (a non-zero exit: NOT changed), render_group_missing_program_says_it_could_not_start (never started), render_group_timeout_says_timed_out_and_how_to_check_not_that_nothing_changed and unset_override_setx_timeout_says_timed_out_and_how_to_check (a timeout: may have changed, how to check, what to re-run), and every_rerun_id_names_a_recipe_with_a_runner (the re-run command exists). It also says that no runner test drives WaitFailed or Signaled, which take the timeout's wording from this match.
| } | ||
|
|
||
| #[test] | ||
| fn a_failed_read_says_nothing_was_changed() { |
There was a problem hiding this comment.
Same as above: this pins "nothing was changed" for a failed read. Point to pin_device_read_timeout_stops_before_setx and a_read_that_exits_non_zero_stops_every_windows_fix_without_writing, which prove that no write follows a failed read.
There was a problem hiding this comment.
Done in d9e9fa1. The comment names pin_device_read_timeout_stops_before_setx, unset_override_user_read_timeout_stops_before_anything_is_reported_or_cleared and a_read_that_exits_non_zero_stops_every_windows_fix_without_writing. 49c7445 adds unset_override_dry_run_with_a_failed_machine_read_stops_without_a_plan, so fix-2's Machine-scope read failure is pinned too.
| and answers in time), offer to re-run the same `rocm fix` command. Do not | ||
| present the printed catalog entry as a preview, and do not hand the user | ||
| its commands (such as `setx HIP_VISIBLE_DEVICES <dGPU-index>`) instead. | ||
| - `… NOT changed.` — the command ran and failed, or could not start. |
There was a problem hiding this comment.
This bullet only special-cases exited 124. A released CLI also prints exited -1; ... NOT changed. for a signal-killed step, which is the same may-have-landed case, and an older CLI's fix-2/fix-9 reads collapse a failed read to "unset". Consider one short passage on how older released CLIs differ, so the agent does not tell the user nothing changed in those cases.
There was a problem hiding this comment.
Done in d9e9fa1, completed in ccfe192 and 49c7445. Step 3 now has one passage on how older released CLIs differ:
- A
NOT changedline sayingexited 124(timed out) orexited -1(killed by a signal) may have landed. The agent says so, gives the check (id -nG <user>for fix-4, or the PowerShellGetEnvironmentVariablecall on Windows) and offers the re-run. - An older CLI takes a failed fix-2/fix-9 read for an unset variable, so a clean result there is not proof. For fix-2 the agent has the user check both the User and Machine scope before calling it done; for fix-9 it checks the User value before applying.
I checked this against the pre-PR code: probe_exec maps a timeout to 124 and a signal to -1, and fix-2/fix-9 used unwrap_or_default() on their reads. reference.md's exit-4 row points at the passage.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 35548d2
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (16 files, +2050/-222). Implements the timeout/read-failure reporting rework named in the title for ROCMAI-345. The description is unusually precise and matches the diff in every detail I cross-checked (exact wording, test names, CI lane names), and it proactively discloses several of the same gaps this review independently found as "Gaps"/"Follow-ups". Ticket ROCMAI-345 named but not retrievable from here (no tracker access in this environment).
Blocking
None
Non-blocking
- A failed/timed-out User-scope
setxin fix-2 can skip the Machine-scope cleanup guidance —crates/rocm-core/src/fix/runners.rs,report_and_clear_override_windows(theif let Err(code) = step.run(&mut run, err) { return code; }branch inside theif !user_val.is_empty()block, before theif !machine_val.is_empty()block that prints the elevated-shell instructions).
When both the User and Machine scopes carry the override and the User-scopesetxfails or times out, the function returns immediately and never reaches the Machine-scope guidance below it — even though the user still has a Machine-scope value to deal with. The function's own doc comment states the opposite principle for the decline case ("Declining just the User scope can't short-circuit... the guidance... still has to print unconditionally"), and this PR added a test for exactly that decline case (windows_decline_with_machine_scope_also_set_still_returns_5) without adding the analogous one for the failure/timeout path. This exact control-flow shape (returnbefore the Machine-scope block) already existed before this PR, so it isn't a new regression or an enlarged blast radius — it's a pre-existing gap in code this PR substantially reworked and tested around.
Confidence 80 · logic · Fix: have the failure path set a flag (as the decline path does) and fall through to the Machine-scope print before returning the failure code; add a test mirroringwindows_decline_with_machine_scope_also_set_still_returns_5for the failure/timeout case. - The new
Step/outcome_phrasemachinery dropsCommandCapture.detailfor aWaitFailed(and, more weakly,SpawnFailed) outcome —crates/rocm-core/src/fix/consent.rs,outcome_phraseandStep::run(which callsself.failure(capture.outcome), discardingcapture.detail).
process/bounded.rspopulatesdetailwith the actual wait/spawn error text for exactly these outcomes (tested: "the wait error is kept in words"), and the codebase's own established convention for surfacing it already exists inhost_gpu/diagnostics.rs::capture_diagnostic_with(capture.detail.as_deref().unwrap_or("unknown error")). The new per-step failure line instead prints a bare "could not be waited for and was stopped" with no underlying cause, where the existing pattern would have surfaced one. This doesn't affect the exit code or the core "NOT changed" vs "may or may not have changed" claim's correctness, just the amount of diagnostic context in a rare failure branch.
Confidence 75 · logic · Fix: threadcapture.detailintooutcome_phrase'sWaitFailed/SpawnFailedarms (e.g. append: {detail}when present), matchinghost_gpu/diagnostics.rs's pattern.
Decisions for the author
- Decline-path consent is untested for 3 of the 4 mutating runners (
render_group_with,path_export_windows,hip_visible_devices_windowsall callconfirm/consentwithout the injectable closure fix-2 uses) — answered-by-intent.
The PR body's "Follow-ups" section already states: "The fix-6 and fix-9 Windows decline paths callconfirmdirectly and are untested; inject consent as fix-2 does. fix-4's decline gate inrender_group_withhas no unit test either." Would otherwise have been a non-blocking, pattern-relevant pre-existing-coverage finding (identical shape to the two findings above — consent wiring untouched by this diff). Confirm this is still the intended order of work.
Positive signals
Step::failurecomputes the printed line and the returned exit code from oneoutcomevalue in one function, so they structurally cannot disagree — a clean, direct satisfaction of "a message about the CLI's own behavior is asserted together with the resulting state."- The four recovery-line fix-ids (
UNSET_OVERRIDE_ID/RENDER_GROUP_ID/PATH_ID/IGPU_DGPU_ID) are defined once incatalog.rsand consumed by both the catalog entries and the runner recovery lines, with a dedicated test (every_rerun_id_names_a_recipe_with_a_runner) that actually callsfind_recipeand asserts a runner exists — not just a shared-constant convention, an enforced one. - The e2e stand-in for a hung
usermodusesexec /bin/sleep 600specifically so the bounded executor's "kill the direct child only" semantics reach the actual sleeping process instead of leaving a 10-minute orphan — a subtle, correctly-reasoned test design choice, explained in its own comment. CommandOutcome's match inStep::failure/outcome_phraseis exhaustive by construction (a new variant is a compile error, not a silent fallthrough), which is exactly what AGENTS.md's "state matrix" guidance asks for.
Deployment notes
Exit-code values are unchanged (0/1/2/3/4/5 contract preserved), but the stderr wording for rocm fix changes materially, and three Windows runners (fix-2-unset-override, fix-6-path, fix-9-igpu-dgpu --device-index N) now fail closed (exit 4) on a slow/failed PowerShell read where they previously proceeded on a silently-collapsed "unset" value. Any downstream automation pattern-matching the old exited 124 text or relying on the old Windows "proceed on unreadable state" behavior will see different output after this ships; the project's own rocm-doctor skill docs already account for this transition for older released CLIs.
What this covered
Read the whole diff (base 4221d665…head 35548d2) across all 16 changed files. Six independent blind fan-out passes (grouped by coupling: fix.rs+fix/consent.rs; fix/runners.rs; fix/catalog.rs+fix/options.rs+apps/rocm/src/cli.rs; the three e2e-cucumber test files; the seven docs/README/skill files; and a combined history+prior-changes+code-comments pass) each read their full file group plus relevant surrounding code and the project's AGENTS.md, completed successfully and independently (fresh contexts, no access to the PR description or discussion). The agent-instruction-adherence pass (the fourth source-reading pass) and the design-review step (Step 7) were done by the driver directly rather than as additional independent fan-out workers, after repeated attempts to spawn them hit this host's shared concurrent-subagent limit (contention from other sessions, not from this review's own fan-out) — their results are folded into the findings above but are not independent of the driver's own judgment.
One process note on blindness: while resolving the PR's head commit and base for the diff, an early metadata call incidentally returned the full PR description before the blind code passes were dispatched. All six fan-out workers were given only file paths and commit SHAs (confirmed by their outputs, which show no awareness of the PR's framing), so the actual defect-finding passes remained blind; the description was then applied properly, and only at the end, per Step 9.
CI: all checks on this head commit (49 of 49 observed) report success. No failing check. Not reconciled against the existing human review thread on this PR (one review exists, from a different reviewer, at this same head commit) — this report was produced independently of it.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
Reviewed the full diff (base e2e5c5bb3d34605888fcab61f4511ab24f215c3c…head 35548d2dcb1b916f2df89e6517212d0b40b0afe6, 16 files) against crates/rocm-core/src/fix/consent.rs, crates/rocm-core/src/fix/runners.rs, crates/rocm-core/src/fix.rs, fix/catalog.rs, fix/options.rs, apps/rocm/src/cli.rs, and the e2e harness changes in tests/e2e-cucumber/.
No blocking correctness problems found. Two non-blocking items posted inline:
fix/runners.rs— a failed/timed-out User-scopesetxin fix-2 returns before the Machine-scope guidance block prints, when both scopes carry the override. Traced this againstorigin/main: the same early-return shape predates this PR (only the message wording and the set of outcomes that trigger it changed), so it's a pre-existing gap this PR reworks and partially tests around (the decline-path equivalent is now tested; the failure-path equivalent is not) rather than a new regression.fix/consent.rs—outcome_phrasedoesn't surfaceCommandCapture.detailfor aWaitFailed(and, more weakly,SpawnFailed) outcome, even thoughprocess/bounded.rspopulates it with the actual underlying error for both. Doesn't affect the exit code or the "changed"/"may not have changed" claim, just diagnostic context in a rare branch.
Exit-code contract is unchanged; the new Step/StepFailure machinery derives the printed line and the returned code from one value, which is a clean way to keep those from disagreeing. Windows-only logic is unit-testable through the injected command-runner/consent closures.
CI: all checks that exercise the code (build, clippy, unit tests, hosted E2E lanes, Windows build/test, CodeQL) report success on the head commit. One self-hosted E2E consolidated report (self-hosted) check is red, but its log shows the failure is a downstream consequence of an upstream GPU lane (E2E tests (MI300X, current)) being cancelled, not a test assertion failing — doesn't look related to this diff.
Note: a separate human review requesting changes (mostly doc/comment follow-ups and a couple of design questions) is already open on this PR; this review was conducted independently of its content.
🤖 Automated by an AI agent
| timeout: RUN_TIMEOUT, | ||
| }; | ||
| if let Err(code) = step.run(&mut run, err) { | ||
| return code; |
There was a problem hiding this comment.
Non-blocking: when the User-scope setx step fails or times out here, the function returns immediately and never reaches the Machine-scope guidance block below (if !machine_val.is_empty() at line 368). On a host where both scopes carry the override, a failed/timed-out User-scope clear means the user never sees the elevated-shell instructions for the Machine scope, even though it still needs clearing.
This early-return shape already existed on main before this PR (same return exit::FAILED; after a non-zero setx exit), so it is not a regression this PR introduces — but this PR adds windows_decline_with_machine_scope_also_set_still_returns_5, which pins the equivalent decline-path fall-through, without an analogous test for the failure/timeout path. Worth fixing alongside, the same way the decline path handles it: record a flag and fall through to the Machine-scope print before returning the failure code.
There was a problem hiding this comment.
Agreed, and thanks for tracing it against main. I've left it out of this PR because it's the pre-existing early-return shape, and fixing it properly means a fall-through like the decline path's plus a failure-path test. It's now in the PR's "Follow-ups (not in this PR)" list.
| "{label} timed out after {}s and was stopped", | ||
| timeout.as_secs() | ||
| ), | ||
| CommandOutcome::WaitFailed => { |
There was a problem hiding this comment.
Non-blocking: outcome_phrase (and Step::failure/Step::run, which only pass capture.outcome into it) never looks at CommandCapture.detail. For CommandOutcome::WaitFailed that means the printed line is the bare "could not be waited for and was stopped", even though process/bounded.rs populates detail with the actual wait error (Some(format!("{error:#}"))) for exactly this outcome. SpawnFailed(kind) is a weaker version of the same gap: the message uses the ErrorKind but not the fuller error.to_string() that run_bounded also puts in detail.
This doesn't affect the exit code or the "NOT changed" vs "may or may not have changed" correctness, just the diagnostic context in a rare failure branch. Threading capture.detail into these two arms (e.g. appending : {detail} when present) would match the existing pattern in host_gpu/diagnostics.rs::capture_diagnostic_with.
There was a problem hiding this comment.
Agreed. It doesn't change the exit code or the changed/not-changed claim, so I've left it out of this PR and added it to "Follow-ups (not in this PR)": append detail when present, as capture_diagnostic_with does.
Conflicts resolved keeping both sides: - fix/options.rs: main moved each exit code's wording into `MEANINGS`; the `failed` description now also says a timed-out or stopped command may have made its change, and that fix-2, fix-6 and fix-9 with a device index read first on Windows, and its comment names the injected-runner tests that prove each case. - diagnose.feature: main took diagnose-62 to -67, so this branch's timeout and Windows read-failure scenarios are now diagnose-68 to -70 (ids unchanged); docs and step comments follow. - manual-testing.md: this branch's section is now 17, after main's 14-16. - testing.md: both paragraphs kept. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…kill (ROCMAI-345) The wording tests in fix/consent.rs pin each failure line against a stand-in step; their comments now name the runner tests that prove what each line claims, and say that no runner test drives a wait failure or a signal. The rocm-doctor skill special-cased only `exited 124`. A CLI released before this change also reports a signalled step as `exited -1` with `NOT changed`, and on Windows takes a failed read for an unset variable: fix-2 can then report the override gone and exit 0, and fix-9 can run setx over a value the user chose. One passage now covers both, with the check to run, and the exit-4 row in reference.md points at it. Also refresh three stale comments: what fix.rs says consent.rs holds, fix-2's class in the report_persistent_override doc, and a PR-relative note on the current_os WSL test. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…CMAI-345) The merge with main moved exit-code wording into `exit::MEANINGS`, and this branch's `failed` entry now says a timed-out or stopped command may have made its change. Regenerate the published catalog so tools read the same text the CLI prints. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…alog (ROCMAI-345) An older CLI swallows a failed read of either scope, so with an override set only in the Machine scope fix-2 can still report it gone. The skill's check for that case now reads both scopes, and since the output does not say which CLI produced it, the skill asks for the check on every Windows fix-2 and fix-9 run. The published `failed` description names the Machine read too, and the skill-eval replays are re-recorded against the new catalog text. Also stop the fix.rs module doc implying the consent.rs helpers are re-exported; only the options.rs names are. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…o do with a set value (ROCMAI-345) The published `failed` description says fix-2 reads its Machine value even under `--dry-run`, but only the User read had a dry-run test. A new test fails the Machine read under `--dry-run` with a User value set and asserts exit 4 with no plan; it goes red when the Machine read failure is swallowed. The skill's older-CLI check told the agent to read both scopes without saying what a set value means. It now says: a User value means re-running fix-2, a Machine value needs an elevated PowerShell, and for fix-9 an existing value is left for the user to change by hand. The lead-in says the Windows case applies to every run, since the output cannot tell an older CLI apart. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
|
@juhovainio, on the parts of your review of 35548d2 that have no inline thread:
Since your approval I merged |
Main landed the fix-step timeout work (#1062), which adds three diagnose scenarios and its own catalog entry, so it collides with this branch on both the feature file and the recorded eval fixtures. tests/e2e-cucumber/features/diagnose.feature: main took `diagnose-68` through `diagnose-70`. This branch's vLLM-extension scenario becomes `diagnose-71` and moves after them, so each number still equals its position. Nothing referenced it by number -- the `diagnose-68` mentions in `diagnose_steps.rs`, `e2e.rs`, `docs/testing.md` and `docs/manual-testing.md` all belong to main's new timeout scenario and are left alone. skills/rocm-doctor/evals/fixtures/*/replies.json: seven conflicts, all in generated files. Resolved by taking main's side and re-running `cargo xtask eval-fixtures`, rather than hand-merging recorded output: the fixtures are a recording of the catalog, so the only correct content is whatever the merged catalog produces. The regeneration re-adds this branch's `fix-23-vllm-extension-load` entry on top of main's, which is the one thing that differs. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Reviewed the full diff (base 9e50eac5…head 49c74452, 24 files) independently via a blind multi-pass review — five fan-out passes (consent.rs+catalog.rs+options.rs+cli.rs; runners.rs; the e2e-cucumber harness changes; the docs/skill/catalog files; and a combined agent-instruction-adherence+history+prior-pattern-consistency+code-comments pass), each reading full files rather than diff hunks only, with the PR description read only afterward to check it against what was found.
No blocking findings.
Independently re-derived both non-blocking items already discussed on this PR's open threads, and confirm they're correctly characterized as pre-existing, not new regressions, and are now explicitly listed in this PR's own "Follow-ups (not in this PR)" section:
report_and_clear_override_windows(crates/rocm-core/src/fix/runners.rs) returns before the Machine-scope guidance block prints when the User-scopesetxfails or times out and both scopes carry the override. Traced againstorigin/maindirectly: this exact early-return shape (fail, thenreturnbefore the Machine-scopeifblock) is present byte-for-byte in the base commit, so this PR carried it forward rather than introducing it. The decline-path equivalent now has a dedicated test (windows_decline_with_machine_scope_also_set_still_returns_5); the failure/timeout-path equivalent does not.outcome_phrase/Step::run(crates/rocm-core/src/fix/consent.rs) don't surfaceCommandCapture.detailforWaitFailed/SpawnFailed, even thoughprocess/bounded.rspopulates that field with the real wait/spawn error text for exactly those outcomes, andhost_gpu/diagnostics.rs::capture_diagnostic_withalready establishes surfacing it as the codebase's convention.
Neither affects the exit-code contract or the correctness of the core "NOT changed" vs. "may or may not have changed" claim — just the completeness of test coverage / diagnostic context in a rare branch. Both are explicitly named in the PR body's "Follow-ups (not in this PR)" and acknowledged by the author in-thread, so I'm treating them as confirmed-and-deferred rather than re-raising them as new findings.
Nothing else survived verification across the five passes: the Windows read-before-write fail-closed behavior (fix-2/6/9, including under --dry-run) is consistently gated before any side effect and before opts.dry_run is even consulted; recovery-line fix-ids are sourced from shared catalog.rs constants with a test tying each one to a real runner-backed recipe; the CommandOutcome match in Step::failure/outcome_phrase is exhaustive by construction; the e2e off-executor helper (run_rocm_with_scenario_env_off_executor) isolates per-scenario state correctly and the exec /bin/sleep 600 stand-in genuinely exercises "kill only the direct child" semantics rather than leaking a process; and doc/skill/catalog wording was checked character-for-character against the actual strings the code produces and matches throughout.
CI: all checks on the current head (49c74452) are green; no failing required check.
Findings were reached blind; reconciled afterward against the existing review threads and the author's replies, which independently reached the same two non-blocking conclusions and already track them as deferred follow-up work.
🤖 Automated by an AI agent
Main landed the fix-step timeout work (#1062), which took `diagnose-68` through `diagnose-70`. This branch's four APU-memory scenarios move to `diagnose-71` through `diagnose-74`, so each number still equals its position. Five references follow them: the ROCMAI-912 scenario list and the `diagnose-74` note in `docs/testing.md`, and three comments in `diagnose_steps.rs`. The other `diagnose-68`/`-69`/`-70` references in those same files -- the fix-4 helper, the one-minute timeout, fix-2 and fix-9 -- belong to main's new scenarios and are left alone. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Why
When a
rocm fixstep did not finish within its one-minute bound, the CLI reported it as an ordinary failure with the change not made. On Linux, afix-4-render-groupwhosesudo usermodhung printedusermod exited 124; group membership NOT changed.and exited 4. That line names no timeout and offers no way to recover, and its claim is unchecked: the bounded executor stops only the process it started (sudo), sousermodunderneath can still complete. Users were told nothing changed when the group may well have been added.Root cause: the fix runners read the probe adapter's collapsed exit code, in which a timeout, a signal and a failed wait all arrive as an ordinary non-zero exit (
124), so the runner could not tell "the command refused" from "the command was stopped and its effect is unknown".The same collapse hid a worse Windows case. A PowerShell read of the current User/Machine value that did not complete was taken as "unset":
fix-2-unset-overridethen printedNo persistent HSA_OVERRIDE_GFX_VERSION found … You're doneand exited 0, on a machine that may still carry the override.fix-9-igpu-dgpu --device-index Nskipped its "already set" guard and went on tosetx, overwriting a device setting the user may have chosen on purpose.What changes
Each runner step now runs on the bounded executor and is reported from its typed outcome. The stderr line and the exit code come from one value, so they cannot disagree.
usermod exited 1; group membership NOT changed.(exit 4)sudo usermod exited 1; group membership NOT changed.(exit 4); the line names the program actually runusermod exited 124; group membership NOT changed.(exit 4)sudo usermod timed out after 60s and was stopped; group membership may or may not have changed. Check with `id -nG <user>`, then re-run `rocm fix fix-4-render-group` if it did not.(exit 4)setxor PowerShell write timed out<setx or PowerShell> exited 124; … NOT changed.(exit 4)setx timed out after 60s and was stopped; User-scope HIP_VISIBLE_DEVICES may or may not have changed. Check in PowerShell with `[Environment]::GetEnvironmentVariable('HIP_VISIBLE_DEVICES','User')`, then re-run `rocm fix fix-9-igpu-dgpu --device-index 1` if it did not.(exit 4); the check is a PowerShell expression, so the line says where to run itNo persistent HSA_OVERRIDE_GFX_VERSION found … You're done(exit 0)Could not read the current User HSA_OVERRIDE_GFX_VERSION: <cause>; nothing was changed.(exit 4), before anything is reported as found or offered for clearingsetxrunCould not read the current User PATH; User PATH NOT changed.(exit 4)Could not read the current User PATH: <cause>; nothing was changed.(exit 4)<cause>is how the read ended, for examplePowerShell could not be started: it was not found,PowerShell exited 1, orPowerShell timed out after 8s and was stopped.--dry-runon Windows: fix-2, fix-6 and fix-9 with--device-indexread the current value before planning, under--dry-runtoo, so a dry run whose read fails now stops with exit 4 and shows no plan (previously fix-9 previewed asetxbuilt on a value that was never read and fix-2 reported the override as gone, both exiting 0). The catalog entryrocm fixprints before the runner still appears; README,--helpand the skill say that it is the fix's generic description (for fix-9, asetx HIP_VISIBLE_DEVICES <dGPU-index>template), not a preview of the change on this machine.Each of those Windows reads runs under the 8 s read bound (
QUERY_TIMEOUT), so a PowerShell that is slow to start cold stops fix-2, fix-6 or fix-9 with exit 4 (fix-2 reads twice, User then Machine). That is deliberate: a read that did not finish is a safe failure that changed nothing, and re-running the fix is the recovery.The
rocm fix <id>named in a recovery line is taken from the same constant as the catalog entry, so it always names a fix that exists. The one other command the runners start, theid -uroot check, still uses the old adapter; it changes nothing on the host.The exit-code contract and
contract_versionare unchanged: exit 4 already meant "attempted, and a command failed"; its description now also covers a timeout and the Windows read failure,--dry-runincluded.Docs updated to match: README,
rocm fix --help,docs/testing.md,docs/manual-testing.md(new section 17, including dry runs of the three Windows fixes under an emptiedPATH),docs/host-probing.md,docs/architecture.md, andskills/rocm-doctor/(reference.md's exit-0 and exit-4 rows, andSKILL.mdstep 3 telling the agent how to act on each kind of exit-4 line: a read failure changed nothing, and a timeout must be checked before claiming either outcome). The skill is published frommainbut installs the release-channel CLI, so step 3 also has one passage on how an older released CLI differs: aNOT changedline reportingexited 124(timed out) orexited -1(killed by a signal) may have landed, with the check to run for each fix and an offer to re-run; and on Windows an older CLI takes a failed fix-2/fix-9 read for "unset" (fix-2 reports the override gone and exits 0, fix-9 runssetx), so on every Windows fix-2 and fix-9 run (the output cannot tell an older CLI apart) the agent has the user check the variable rather than trust a clean result: fix-2 in both the User and Machine scopes, with what to do about a value still set in each, and fix-9 before it is applied.reference.md's exit-4 row points at that passage. The twoSKILL.mdfixture sentences are unchanged.The merge with
mainbroughtexit::MEANINGS, which now holds each exit code's published wording; thefailedentry says a timed-out or stopped command may have made its change and names the Windows reads (fix-2's Machine read included), its comment names the tests proving each case (a new one,unset_override_dry_run_with_a_failed_machine_read_stops_without_a_plan, pins fix-2's Machine read under--dry-run), anddoctor-catalog.jsonand the skill-eval replays are regenerated from it. The wording-only tests infix/consent.rsname the runner tests that prove what each line claims.E2E harness
A step that waits for the CLI to finish used to call it synchronously inside the single future the cucumber suite is polled on. diagnose-68 waits out the real one-minute bound, so every concurrent scenario stalled for that minute and the TUI scenarios beside it overran their wall-clock caps. The step that applies a fix now waits on a blocking thread (
run_rocm_with_scenario_env_off_executorintests/e2e.rs), asprobe_device_memoryalready did.Scope
Not in this change: labelling advisory/print-only fixes and exiting 0 for them. That behaviour was already on
main(FixClass, scenarios diagnose-05/18/27/28/33).Verification
Unit tests in
fix/runners.rsandfix/consent.rsdrive every outcome of each runner through an injected command runner: exited non-zero, not found, spawn failed, timed out, signalled, wait failed, and the Windows reads failing (timed out, not found, exited non-zero) for each read each fix depends on. The Windows runners write their own lines through injected stdout and stderr sinks, so each test asserts the printed lines together with the exit code and what was run: a failed read prints no plan and writes nothing, a dry run with a readable scope prints the plan and runs nosetx, a dry run with a failed read exits 4 with no plan, and a declined fix-2 still prints the Machine-scope instructions. They run on both the Linux and Windows CI jobs. The timeout and read-failure tests were seen to fail with the timeout branch or the read-failure guard reverted to the old behaviour.Scenarios in
diagnose.feature:@id:diagnose-fix-command-timeout-says-change-may-be-partial(diagnose-68, new): a stand-insudorecords its arguments and sleeps past the bound. Asserts the timeout line with exit 4, that neither "NOT changed" nor the success line is printed, and that the helper was asked to add the user to the groups. It waits out the real one-minute bound, adding about 60s to the lane; there is deliberately no test-only switch to shorten it, so the bound under test is the one users get.@id:diagnose-fix-command-failure-reported-on-stderr(diagnose-15, extended): now also asserts the success line is absent and the helper was invoked, so "NOT changed" is checked against an attempted and refused change.@id:diagnose-fix-path-aborts-on-unreadable-user-path(diagnose-39, tightened to the new line).@id:diagnose-fix-override-aborts-on-unreadable-user-scope(diagnose-69, new): an emptyPATHmakes the PowerShell read genuinely fail; asserts exit 4, the read-failure line, and no "already gone" report.@id:diagnose-fix-igpu-aborts-on-unreadable-user-scope(diagnose-70, new): the same failed read underfix-9-igpu-dgpu --device-index 1 --dry-run; asserts exit 4, the read-failure line, that the catalog entry is printed, and that no plan, concretesetx HIP_VISIBLE_DEVICES 1or dry-run marker is. It is a dry run because the emptyPATHhides PowerShell but notsetx.exe, so an applied fix-9 whose guard regressed would set the runner's real User-scope value. diagnose-39 and diagnose-69 cannot write it even then (fix-6 writes through PowerShell; fix-2 would find nothing to clear).Lanes: diagnose-15 and diagnose-68 are
@requires-os:linux @requires-bare-metaland run on the hostedE2E testsandE2E tests (ubuntu-26.04)jobs and the self-hosted Linux lanes. diagnose-39/69/70 are@requires-os:windowsand run only on the self-hostedE2E tests (Strix Halo, Windows, <sdk_version>)lane (e2e-selfhosted.yml, which runs on a pull request touching itsservepaths, as this one does, and is not a required check) and on the nightlyE2E tests (Strix Halo, Windows, incl. nightly-only). The required hostedwindows-releaselane runs only@lifecyclescenarios and itsE2E_ALSO_RUN_IDSlist, which does not include them. The Linux lanes compile them.Scenario numbers:
maintook diagnose-62 to -67 while this was open, so this PR's scenarios are now diagnose-68 to -70; their@id:tags are unchanged.Run locally on Linux at the final head (after merging
main):cargo fmt --all --check;cargo clippy --workspace --all-targets -- -D warnings;cargo test --workspace --all-targets --no-fail-fastbefore the last two commits, thencargo test -p rocm-core --lib, the xtaskeval_fixturestests and theskill-0*scenarios again at the final head (all pass except host-environment failures in modules this PR does not touch:comfyui::source_swapsocket path length,service_controlcontainer engine, twobuild_script::git_watch_pathstests,release_readinessknown issues, and a port-in-usesupervisetest that passed on re-run);cargo xtask check-module-size,check-architecture-doc,check-crate-edges;cargo test -p e2e-cucumber --test feature_naming --test skill_reference;cargo xtask e2e -- -n "(diagnose-15|diagnose-6[2-8]|skill-0[0-9]) "(15 scenarios passed, diagnose-68 included); and the prek pre-commit and pre-push hooks over the branch.cargo xtask smokepassed up to itsrocmdsandbox step, which this host's bubblewrap cannot run (Failed RTM_NEWADDR: Operation not permitted).Gaps:
PATH,setx.exeis still found, so a regressed guard would write the runner's real User environment. The unit tests hold that an applied fix-9 writes nothing after a failed read.setxor PowerShell write; the unit tests cover those rows.opts.yestoconfirm, but the tests inject their own consent closure.run_render_group,run_path_export_windowsand the otherrun_*_windowsglue) are not driven by unit tests; the*_withand inner functions they call are.cargo xtask smoke's sandbox step not run locally (see above).Follow-ups (not in this PR)
confirmdirectly and are untested; inject consent as fix-2 does. fix-4's decline gate inrender_group_withhas no unit test either.write_script_escapes_apostrophe_in_real_path_valuerebuilds the write script itself, so it cannot fail on whatpath_export_windowsactually writes; drive it through the fake host instead.CheckInis paired with its check string by convention, not by type.println!rather than an injected sink, andlet _ = writeln!is repeated rather than factored into a helper.docs/manual-testing.md's fix-2 Windows line still applies with--yesunder an emptiedPATH.diagnose_steps.rsnote that stdin "is piped", anddocs/testing.md's "behindwindows-build-and-test".setxfails or times out, the runner returns before printing the Machine-scope guidance, so a host with both scopes set never sees the elevated-shell instructions. Pre-existing shape; fix it as the decline path does (fall through, then return the failure) with a failure-path test.outcome_phraseignoresCommandCapture.detail, so aWaitFailedline omits the underlying wait error and aSpawnFailedline shows only theErrorKind; append the detail when present, ashost_gpu/diagnostics.rs::capture_diagnostic_withdoes.Checklist
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No rows for ROCMAI-345.)docs/architecture.md(the clap declaration incli.rsand the dispatch wiring inmain.rs/lib.rsare expected, not a violation). (No new subcommand.).rsfile underxtask/srcortests/e2e-cucumber),docs/architecture.mdhas the matching entry (cargo xtask check-architecture-docfails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (No files added; thefix/consent.rsentry describesrun_step/Step/CheckIn, thefix/runners.rsentry the Windows read before planning and its exit 4, and thetests/e2e.rsentry the off-executor helper.)