Skip to content

fix(rocm-core): report a timed-out rocm fix step as such (ROCMAI-345) - #1062

Merged
rominf merged 9 commits into
mainfrom
ROCMAI-345-fix-fix-timeout-reporting
Nov 17, 2026
Merged

rominf merged 9 commits into
mainfrom
ROCMAI-345-fix-fix-timeout-reporting

Conversation

@rominf

@rominf rominf commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Why

When a rocm fix step did not finish within its one-minute bound, the CLI reported it as an ordinary failure with the change not made. On Linux, a fix-4-render-group whose sudo usermod hung printed usermod exited 124; group membership NOT changed. and exited 4. That line names no timeout and offers no way to recover, and its claim is unchecked: the bounded executor stops only the process it started (sudo), so usermod underneath can still complete. Users were told nothing changed when the group may well have been added.

Root cause: the fix runners read the probe adapter's collapsed exit code, in which a timeout, a signal and a failed wait all arrive as an ordinary non-zero exit (124), so the runner could not tell "the command refused" from "the command was stopped and its effect is unknown".

The same collapse hid a worse Windows case. A PowerShell read of the current User/Machine value that did not complete was taken as "unset":

  • fix-2-unset-override then printed No persistent HSA_OVERRIDE_GFX_VERSION found … You're done and exited 0, on a machine that may still carry the override.
  • fix-9-igpu-dgpu --device-index N skipped its "already set" guard and went on to setx, overwriting a device setting the user may have chosen on purpose.

What changes

Each runner step now runs on the bounded executor and is reported from its typed outcome. The stderr line and the exit code come from one value, so they cannot disagree.

How the step ended Before After
exited non-zero, or could not start usermod exited 1; group membership NOT changed. (exit 4) sudo usermod exited 1; group membership NOT changed. (exit 4); the line names the program actually run
Linux fix-4: timed out, killed by a signal, or could not be waited for usermod exited 124; group membership NOT changed. (exit 4) sudo usermod timed out after 60s and was stopped; group membership may or may not have changed. Check with `id -nG <user>`, then re-run `rocm fix fix-4-render-group` if it did not. (exit 4)
Windows fix-2/fix-6/fix-9: a setx or PowerShell write timed out <setx or PowerShell> exited 124; … NOT changed. (exit 4) setx timed out after 60s and was stopped; User-scope HIP_VISIBLE_DEVICES may or may not have changed. Check in PowerShell with `[Environment]::GetEnvironmentVariable('HIP_VISIBLE_DEVICES','User')`, then re-run `rocm fix fix-9-igpu-dgpu --device-index 1` if it did not. (exit 4); the check is a PowerShell expression, so the line says where to run it
Windows fix-2: User/Machine read did not complete No persistent HSA_OVERRIDE_GFX_VERSION found … You're done (exit 0) Could not read the current User HSA_OVERRIDE_GFX_VERSION: <cause>; nothing was changed. (exit 4), before anything is reported as found or offered for clearing
Windows fix-9: User-scope read did not complete guard skipped, setx run same read-failure line (exit 4), before any plan or prompt
Windows fix-6: User PATH read did not complete Could not read the current User PATH; User PATH NOT changed. (exit 4) Could not read the current User PATH: <cause>; nothing was changed. (exit 4)

<cause> is how the read ended, for example PowerShell could not be started: it was not found, PowerShell exited 1, or PowerShell timed out after 8s and was stopped.

--dry-run on Windows: fix-2, fix-6 and fix-9 with --device-index read the current value before planning, under --dry-run too, so a dry run whose read fails now stops with exit 4 and shows no plan (previously fix-9 previewed a setx built on a value that was never read and fix-2 reported the override as gone, both exiting 0). The catalog entry rocm fix prints before the runner still appears; README, --help and the skill say that it is the fix's generic description (for fix-9, a setx HIP_VISIBLE_DEVICES <dGPU-index> template), not a preview of the change on this machine.

Each of those Windows reads runs under the 8 s read bound (QUERY_TIMEOUT), so a PowerShell that is slow to start cold stops fix-2, fix-6 or fix-9 with exit 4 (fix-2 reads twice, User then Machine). That is deliberate: a read that did not finish is a safe failure that changed nothing, and re-running the fix is the recovery.

The rocm fix <id> named in a recovery line is taken from the same constant as the catalog entry, so it always names a fix that exists. The one other command the runners start, the id -u root check, still uses the old adapter; it changes nothing on the host.

The exit-code contract and contract_version are unchanged: exit 4 already meant "attempted, and a command failed"; its description now also covers a timeout and the Windows read failure, --dry-run included.

Docs updated to match: README, rocm fix --help, docs/testing.md, docs/manual-testing.md (new section 17, including dry runs of the three Windows fixes under an emptied PATH), docs/host-probing.md, docs/architecture.md, and skills/rocm-doctor/ (reference.md's exit-0 and exit-4 rows, and SKILL.md step 3 telling the agent how to act on each kind of exit-4 line: a read failure changed nothing, and a timeout must be checked before claiming either outcome). The skill is published from main but installs the release-channel CLI, so step 3 also has one passage on how an older released CLI differs: a NOT changed line reporting exited 124 (timed out) or exited -1 (killed by a signal) may have landed, with the check to run for each fix and an offer to re-run; and on Windows an older CLI takes a failed fix-2/fix-9 read for "unset" (fix-2 reports the override gone and exits 0, fix-9 runs setx), so on every Windows fix-2 and fix-9 run (the output cannot tell an older CLI apart) the agent has the user check the variable rather than trust a clean result: fix-2 in both the User and Machine scopes, with what to do about a value still set in each, and fix-9 before it is applied. reference.md's exit-4 row points at that passage. The two SKILL.md fixture sentences are unchanged.

The merge with main brought exit::MEANINGS, which now holds each exit code's published wording; the failed entry says a timed-out or stopped command may have made its change and names the Windows reads (fix-2's Machine read included), its comment names the tests proving each case (a new one, unset_override_dry_run_with_a_failed_machine_read_stops_without_a_plan, pins fix-2's Machine read under --dry-run), and doctor-catalog.json and the skill-eval replays are regenerated from it. The wording-only tests in fix/consent.rs name the runner tests that prove what each line claims.

E2E harness

A step that waits for the CLI to finish used to call it synchronously inside the single future the cucumber suite is polled on. diagnose-68 waits out the real one-minute bound, so every concurrent scenario stalled for that minute and the TUI scenarios beside it overran their wall-clock caps. The step that applies a fix now waits on a blocking thread (run_rocm_with_scenario_env_off_executor in tests/e2e.rs), as probe_device_memory already did.

Scope

Not in this change: labelling advisory/print-only fixes and exiting 0 for them. That behaviour was already on main (FixClass, scenarios diagnose-05/18/27/28/33).

Verification

Unit tests in fix/runners.rs and fix/consent.rs drive every outcome of each runner through an injected command runner: exited non-zero, not found, spawn failed, timed out, signalled, wait failed, and the Windows reads failing (timed out, not found, exited non-zero) for each read each fix depends on. The Windows runners write their own lines through injected stdout and stderr sinks, so each test asserts the printed lines together with the exit code and what was run: a failed read prints no plan and writes nothing, a dry run with a readable scope prints the plan and runs no setx, a dry run with a failed read exits 4 with no plan, and a declined fix-2 still prints the Machine-scope instructions. They run on both the Linux and Windows CI jobs. The timeout and read-failure tests were seen to fail with the timeout branch or the read-failure guard reverted to the old behaviour.

Scenarios in diagnose.feature:

  • @id:diagnose-fix-command-timeout-says-change-may-be-partial (diagnose-68, new): a stand-in sudo records its arguments and sleeps past the bound. Asserts the timeout line with exit 4, that neither "NOT changed" nor the success line is printed, and that the helper was asked to add the user to the groups. It waits out the real one-minute bound, adding about 60s to the lane; there is deliberately no test-only switch to shorten it, so the bound under test is the one users get.
  • @id:diagnose-fix-command-failure-reported-on-stderr (diagnose-15, extended): now also asserts the success line is absent and the helper was invoked, so "NOT changed" is checked against an attempted and refused change.
  • @id:diagnose-fix-path-aborts-on-unreadable-user-path (diagnose-39, tightened to the new line).
  • @id:diagnose-fix-override-aborts-on-unreadable-user-scope (diagnose-69, new): an empty PATH makes the PowerShell read genuinely fail; asserts exit 4, the read-failure line, and no "already gone" report.
  • @id:diagnose-fix-igpu-aborts-on-unreadable-user-scope (diagnose-70, new): the same failed read under fix-9-igpu-dgpu --device-index 1 --dry-run; asserts exit 4, the read-failure line, that the catalog entry is printed, and that no plan, concrete setx HIP_VISIBLE_DEVICES 1 or dry-run marker is. It is a dry run because the empty PATH hides PowerShell but not setx.exe, so an applied fix-9 whose guard regressed would set the runner's real User-scope value. diagnose-39 and diagnose-69 cannot write it even then (fix-6 writes through PowerShell; fix-2 would find nothing to clear).

Lanes: diagnose-15 and diagnose-68 are @requires-os:linux @requires-bare-metal and run on the hosted E2E tests and E2E tests (ubuntu-26.04) jobs and the self-hosted Linux lanes. diagnose-39/69/70 are @requires-os:windows and run only on the self-hosted E2E tests (Strix Halo, Windows, <sdk_version>) lane (e2e-selfhosted.yml, which runs on a pull request touching its serve paths, as this one does, and is not a required check) and on the nightly E2E tests (Strix Halo, Windows, incl. nightly-only). The required hosted windows-release lane runs only @lifecycle scenarios and its E2E_ALSO_RUN_IDS list, which does not include them. The Linux lanes compile them.

Scenario numbers: main took diagnose-62 to -67 while this was open, so this PR's scenarios are now diagnose-68 to -70; their @id: tags are unchanged.

Run locally on Linux at the final head (after merging main): cargo fmt --all --check; cargo clippy --workspace --all-targets -- -D warnings; cargo test --workspace --all-targets --no-fail-fast before the last two commits, then cargo test -p rocm-core --lib, the xtask eval_fixtures tests and the skill-0* scenarios again at the final head (all pass except host-environment failures in modules this PR does not touch: comfyui::source_swap socket path length, service_control container engine, two build_script::git_watch_paths tests, release_readiness known issues, and a port-in-use supervise test that passed on re-run); cargo xtask check-module-size, check-architecture-doc, check-crate-edges; cargo test -p e2e-cucumber --test feature_naming --test skill_reference; cargo xtask e2e -- -n "(diagnose-15|diagnose-6[2-8]|skill-0[0-9]) " (15 scenarios passed, diagnose-68 included); and the prek pre-commit and pre-push hooks over the branch. cargo xtask smoke passed up to its rocmd sandbox step, which this host's bubblewrap cannot run (Failed RTM_NEWADDR: Operation not permitted).

Gaps:

  • No scenario applies fix-9 through a failed read: under an emptied PATH, setx.exe is still found, so a regressed guard would write the runner's real User environment. The unit tests hold that an applied fix-9 writes nothing after a failed read.
  • No dry-run scenarios for fix-2 or fix-6; the unit tests cover both, with a readable and with a failed read.
  • No scenario times out a Windows setx or PowerShell write; the unit tests cover those rows.
  • fix-2's production consent wiring is untested: its closure forwards opts.yes to confirm, but the tests inject their own consent closure.
  • The production wrappers (run_render_group, run_path_export_windows and the other run_*_windows glue) are not driven by unit tests; the *_with and inner functions they call are.
  • diagnose-39/69/70 not run locally (no Windows host).
  • cargo xtask smoke's sandbox step not run locally (see above).
  • GPU lanes not run locally.
  • diagnose-68 waits the real 60s on every lane that runs it.

Follow-ups (not in this PR)

  • The fix-6 and fix-9 Windows decline paths call confirm directly and are untested; inject consent as fix-2 does. fix-4's decline gate in render_group_with has no unit test either.
  • The existing apostrophe-escaping test write_script_escapes_apostrophe_in_real_path_value rebuilds the write script itself, so it cannot fail on what path_export_windows actually writes; drive it through the fake host instead.
  • CheckIn is paired with its check string by convention, not by type.
  • fix-4 and the Linux rc-file runners still print through println! rather than an injected sink, and let _ = writeln! is repeated rather than factored into a helper.
  • Other e2e steps still wait for the CLI synchronously on the executor.
  • The Windows exit-4 read-failure claim is repeated across the skill and docs rather than stated once.
  • docs/manual-testing.md's fix-2 Windows line still applies with --yes under an emptied PATH.
  • Minor comment wording: the diagnose_steps.rs note that stdin "is piped", and docs/testing.md's "behind windows-build-and-test".
  • The skill's older-CLI passage cannot tell which CLI produced a Windows fix-2/fix-9 result, so it asks for the check every time; once a release ships this behaviour, pin the passage to the versions before it or remove it.
  • No skill eval exercises the exit-4 or older-CLI guidance; it needs output recorded from a real older build first.
  • fix-2 on Windows: when the User-scope setx fails or times out, the runner returns before printing the Machine-scope guidance, so a host with both scopes set never sees the elevated-shell instructions. Pre-existing shape; fix it as the decline path does (fall through, then return the failure) with a failure-path test.
  • outcome_phrase ignores CommandCapture.detail, so a WaitFailed line omits the underlying wait error and a SpawnFailed line shows only the ErrorKind; append the detail when present, as host_gpu/diagnostics.rs::capture_diagnostic_with does.

Checklist

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No rows for ROCMAI-345.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation). (No new subcommand.)
  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry (cargo xtask check-architecture-doc fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (No files added; the fix/consent.rs entry describes run_step/Step/CheckIn, the fix/runners.rs entry the Windows read before planning and its exit 4, and the tests/e2e.rs entry the off-executor helper.)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

The fix runners read the probe adapter's collapsed exit code, so a helper
stopped at the one-minute bound was reported as "usermod exited 124;
group membership NOT changed." That names no timeout, offers no way to
recover, and asserts a state nobody checked: the executor stops only the
direct child, so usermod under sudo can still finish.

Run each step through the bounded executor and report from its typed
outcome. A command that exits non-zero or cannot start still says the
change was NOT made; one that times out, is signalled or cannot be waited
for now says the change may or may not have been made, names a command to
check it, and the rocm fix command to re-run. The line and exit code 4
come from one value, and the re-run ids are the catalog's own constants.

On Windows, a PowerShell read of the current User/Machine value that did
not complete used to be taken as "unset": fix-2 then reported the override
as gone and exited 0, and fix-9 skipped its already-set guard and ran setx.
Both now stop with exit 4 before planning or prompting, as fix-6 already
did.

Covered by unit tests over an injected command runner for every outcome of
each runner, and by e2e scenarios diagnose-15 (extended), 39 (tightened)
and the new 62 (Linux helper never finishes), 63 and 64 (Windows reads
fail). Docs, --help and the rocm-doctor skill describe the new exit-4
cases.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
… precisely (ROCMAI-345)

diagnose-62 waits out fix-4's real one-minute timeout. Its step ran the
CLI synchronously inside the single future the cucumber suite is polled
on, so every concurrent scenario stalled for that minute and the TUI
scenarios beside it overran their wall-clock caps, failing the hosted
E2E lanes. The step now waits for the CLI on spawn_blocking.

On Windows, the timeout line of a setx or PowerShell write said "Check
with" before a PowerShell expression that cmd cannot run. Step now
carries where its check runs, and those lines say "Check in PowerShell
with".

The Windows runners for fix-2, fix-6 and fix-9 write their own lines
through injected stdout sinks, so the unit tests assert what is printed
together with the exit code and what was run: a read that exits non-zero
stops each fix with exit 4 and nothing written, a failed read prints no
plan, and a dry run with a readable scope previews without running setx.

diagnose-64 now previews fix-9 with --dry-run instead of applying it. The
empty PATH hides PowerShell but not setx.exe, so a regressed read guard
could otherwise set the runner's real User-scope HIP_VISIBLE_DEVICES.

README, --help, the exit-code docs and the rocm-doctor skill now say
exactly which fixes this covers: on Windows, fix-2, fix-6 and fix-9 with
--device-index read the current value first, and a failed read exits 4
with no plan, under --dry-run too; the catalog entry printed before it is
not a preview.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ROCMAI-345-fix-fix-timeout-reporting branch from 7c0de9f to f4eaca0 Compare November 13, 2026 11:15
…(ROCMAI-345)

The runners.rs entry promised only print, dry-run and consent. On Windows
fix-2, fix-6 and fix-9 now read the current env-scope value first and stop
with exit 4, before the runner's own preview or any prompt, when that read
fails, dry-run included; the entry should say so. "Consent first" becomes
"consent before any write", and the report-only arms (fix-2 on Linux,
fix-9 without a device index) are named as the exception, as the module
header already does.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…MAI-345)

The skill is federated from main ahead of a release, and its Phase 0
installs the release-channel CLI. A released CLI reports a timed-out fix
step as "<program> exited 124; <what> NOT changed.", so an agent
following the skill would tell the user nothing changed when the step
may have landed. Treat that line as a timeout: say the change may or may
not have landed, give the user the check for that fix (the group list on
Linux, the User-scope variable on Windows), and offer the same re-run.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf marked this pull request as ready for review November 13, 2026 14:46
@rominf
rominf requested a review from a team as a code owner November 13, 2026 14:46
@rominf
rominf requested a review from johnl-amd November 13, 2026 14:46

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking correctness problems found; requesting changes for the items below.

To address

  1. The two wording-only tests in fix/consent.rs (step_failure_reports_each_outcome_with_its_own_wording, a_failed_read_says_nothing_was_changed) should carry a comment naming the test that proves the behavior behind the message (AGENTS §3). The PR body lists this as a follow-up; it is cheap to do here.
  2. The skill's older-CLI handling is partial: exited 124 is covered, but a signal-killed step on a released CLI (exited -1; ... NOT changed.) is the same may-have-landed case, and older CLIs fold a failed fix-2/fix-9 read into "unset".
  3. The stale comments you list as open (fix.rs module doc, the report_persistent_override doc calling fix-2 auto_applicable, the "This PR made..." test comment) are outside the diff, so I could not check them. Please fix them in this PR.

Questions / tradeoffs

  • Every Windows read is now bounded at 8s with no retry, and fix-2 reads twice. A cold PowerShell start slower than 8s now stops with exit 4 where fix-2/fix-9 previously proceeded on a collapsed "unset". Safer, but a new user-visible failure mode; is 8s the right bound?
  • diagnose-62 waits a real 60s on every lane that runs it.
  • diagnose-39/63/64 run only on non-required self-hosted Windows lanes, so the applied fix-9 path is covered by unit tests only.

Positives: the failure line and exit code derive from one value; the runner tests assert exit code, commands run, and stderr together; Windows-gated logic is testable on every host. CI is green apart from a pending CodeQL run. The change matches the timeout / child-failure / recovery-message scope of ROCMAI-345.

/// and only a command that ran to its own end (or never started) is
/// reported as having changed nothing.
#[test]
fn step_failure_reports_each_outcome_with_its_own_wording() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test pins the wording of each failure line, which certifies the string rather than the behavior it claims. Add a comment naming the runner tests that prove the behavior behind the line (e.g. render_group_timeout_says_timed_out_and_how_to_check_not_that_nothing_changed for the timeout case and unset_override_user_read_timeout_stops_before_anything_is_reported_or_cleared for the read failure), so a later edit cannot keep the string while the behavior drifts.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d9e9fa1. The test now carries a comment naming the runner tests that prove each line: render_group_child_failure_says_nothing_changed_and_returns_4 and unset_override_setx_failure_returns_4 (a non-zero exit: NOT changed), render_group_missing_program_says_it_could_not_start (never started), render_group_timeout_says_timed_out_and_how_to_check_not_that_nothing_changed and unset_override_setx_timeout_says_timed_out_and_how_to_check (a timeout: may have changed, how to check, what to re-run), and every_rerun_id_names_a_recipe_with_a_runner (the re-run command exists). It also says that no runner test drives WaitFailed or Signaled, which take the timeout's wording from this match.

}

#[test]
fn a_failed_read_says_nothing_was_changed() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above: this pins "nothing was changed" for a failed read. Point to pin_device_read_timeout_stops_before_setx and a_read_that_exits_non_zero_stops_every_windows_fix_without_writing, which prove that no write follows a failed read.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d9e9fa1. The comment names pin_device_read_timeout_stops_before_setx, unset_override_user_read_timeout_stops_before_anything_is_reported_or_cleared and a_read_that_exits_non_zero_stops_every_windows_fix_without_writing. 49c7445 adds unset_override_dry_run_with_a_failed_machine_read_stops_without_a_plan, so fix-2's Machine-scope read failure is pinned too.

and answers in time), offer to re-run the same `rocm fix` command. Do not
present the printed catalog entry as a preview, and do not hand the user
its commands (such as `setx HIP_VISIBLE_DEVICES <dGPU-index>`) instead.
- `… NOT changed.` — the command ran and failed, or could not start.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bullet only special-cases exited 124. A released CLI also prints exited -1; ... NOT changed. for a signal-killed step, which is the same may-have-landed case, and an older CLI's fix-2/fix-9 reads collapse a failed read to "unset". Consider one short passage on how older released CLIs differ, so the agent does not tell the user nothing changed in those cases.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d9e9fa1, completed in ccfe192 and 49c7445. Step 3 now has one passage on how older released CLIs differ:

  • A NOT changed line saying exited 124 (timed out) or exited -1 (killed by a signal) may have landed. The agent says so, gives the check (id -nG <user> for fix-4, or the PowerShell GetEnvironmentVariable call on Windows) and offers the re-run.
  • An older CLI takes a failed fix-2/fix-9 read for an unset variable, so a clean result there is not proof. For fix-2 the agent has the user check both the User and Machine scope before calling it done; for fix-9 it checks the User value before applying.

I checked this against the pre-PR code: probe_exec maps a timeout to 124 and a signal to -1, and fix-2/fix-9 used unwrap_or_default() on their reads. reference.md's exit-4 row points at the passage.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 35548d2

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (16 files, +2050/-222). Implements the timeout/read-failure reporting rework named in the title for ROCMAI-345. The description is unusually precise and matches the diff in every detail I cross-checked (exact wording, test names, CI lane names), and it proactively discloses several of the same gaps this review independently found as "Gaps"/"Follow-ups". Ticket ROCMAI-345 named but not retrievable from here (no tracker access in this environment).

Blocking

None

Non-blocking

  • A failed/timed-out User-scope setx in fix-2 can skip the Machine-scope cleanup guidance — crates/rocm-core/src/fix/runners.rs, report_and_clear_override_windows (the if let Err(code) = step.run(&mut run, err) { return code; } branch inside the if !user_val.is_empty() block, before the if !machine_val.is_empty() block that prints the elevated-shell instructions).
    When both the User and Machine scopes carry the override and the User-scope setx fails or times out, the function returns immediately and never reaches the Machine-scope guidance below it — even though the user still has a Machine-scope value to deal with. The function's own doc comment states the opposite principle for the decline case ("Declining just the User scope can't short-circuit... the guidance... still has to print unconditionally"), and this PR added a test for exactly that decline case (windows_decline_with_machine_scope_also_set_still_returns_5) without adding the analogous one for the failure/timeout path. This exact control-flow shape (return before the Machine-scope block) already existed before this PR, so it isn't a new regression or an enlarged blast radius — it's a pre-existing gap in code this PR substantially reworked and tested around.
    Confidence 80 · logic · Fix: have the failure path set a flag (as the decline path does) and fall through to the Machine-scope print before returning the failure code; add a test mirroring windows_decline_with_machine_scope_also_set_still_returns_5 for the failure/timeout case.
  • The new Step/outcome_phrase machinery drops CommandCapture.detail for a WaitFailed (and, more weakly, SpawnFailed) outcome — crates/rocm-core/src/fix/consent.rs, outcome_phrase and Step::run (which calls self.failure(capture.outcome), discarding capture.detail).
    process/bounded.rs populates detail with the actual wait/spawn error text for exactly these outcomes (tested: "the wait error is kept in words"), and the codebase's own established convention for surfacing it already exists in host_gpu/diagnostics.rs::capture_diagnostic_with (capture.detail.as_deref().unwrap_or("unknown error")). The new per-step failure line instead prints a bare "could not be waited for and was stopped" with no underlying cause, where the existing pattern would have surfaced one. This doesn't affect the exit code or the core "NOT changed" vs "may or may not have changed" claim's correctness, just the amount of diagnostic context in a rare failure branch.
    Confidence 75 · logic · Fix: thread capture.detail into outcome_phrase's WaitFailed/SpawnFailed arms (e.g. append : {detail} when present), matching host_gpu/diagnostics.rs's pattern.

Decisions for the author

  • Decline-path consent is untested for 3 of the 4 mutating runners (render_group_with, path_export_windows, hip_visible_devices_windows all call confirm/consent without the injectable closure fix-2 uses) — answered-by-intent.
    The PR body's "Follow-ups" section already states: "The fix-6 and fix-9 Windows decline paths call confirm directly and are untested; inject consent as fix-2 does. fix-4's decline gate in render_group_with has no unit test either." Would otherwise have been a non-blocking, pattern-relevant pre-existing-coverage finding (identical shape to the two findings above — consent wiring untouched by this diff). Confirm this is still the intended order of work.

Positive signals

  • Step::failure computes the printed line and the returned exit code from one outcome value in one function, so they structurally cannot disagree — a clean, direct satisfaction of "a message about the CLI's own behavior is asserted together with the resulting state."
  • The four recovery-line fix-ids (UNSET_OVERRIDE_ID/RENDER_GROUP_ID/PATH_ID/IGPU_DGPU_ID) are defined once in catalog.rs and consumed by both the catalog entries and the runner recovery lines, with a dedicated test (every_rerun_id_names_a_recipe_with_a_runner) that actually calls find_recipe and asserts a runner exists — not just a shared-constant convention, an enforced one.
  • The e2e stand-in for a hung usermod uses exec /bin/sleep 600 specifically so the bounded executor's "kill the direct child only" semantics reach the actual sleeping process instead of leaving a 10-minute orphan — a subtle, correctly-reasoned test design choice, explained in its own comment.
  • CommandOutcome's match in Step::failure/outcome_phrase is exhaustive by construction (a new variant is a compile error, not a silent fallthrough), which is exactly what AGENTS.md's "state matrix" guidance asks for.

Deployment notes

Exit-code values are unchanged (0/1/2/3/4/5 contract preserved), but the stderr wording for rocm fix changes materially, and three Windows runners (fix-2-unset-override, fix-6-path, fix-9-igpu-dgpu --device-index N) now fail closed (exit 4) on a slow/failed PowerShell read where they previously proceeded on a silently-collapsed "unset" value. Any downstream automation pattern-matching the old exited 124 text or relying on the old Windows "proceed on unreadable state" behavior will see different output after this ships; the project's own rocm-doctor skill docs already account for this transition for older released CLIs.

What this covered

Read the whole diff (base 4221d665…head 35548d2) across all 16 changed files. Six independent blind fan-out passes (grouped by coupling: fix.rs+fix/consent.rs; fix/runners.rs; fix/catalog.rs+fix/options.rs+apps/rocm/src/cli.rs; the three e2e-cucumber test files; the seven docs/README/skill files; and a combined history+prior-changes+code-comments pass) each read their full file group plus relevant surrounding code and the project's AGENTS.md, completed successfully and independently (fresh contexts, no access to the PR description or discussion). The agent-instruction-adherence pass (the fourth source-reading pass) and the design-review step (Step 7) were done by the driver directly rather than as additional independent fan-out workers, after repeated attempts to spawn them hit this host's shared concurrent-subagent limit (contention from other sessions, not from this review's own fan-out) — their results are folded into the findings above but are not independent of the driver's own judgment.

One process note on blindness: while resolving the PR's head commit and base for the diff, an early metadata call incidentally returned the full PR description before the blind code passes were dispatched. All six fan-out workers were given only file paths and commit SHAs (confirmed by their outputs, which show no awareness of the PR's framing), so the actual defect-finding passes remained blind; the description was then applied properly, and only at the end, per Step 9.

CI: all checks on this head commit (49 of 49 observed) report success. No failing check. Not reconciled against the existing human review thread on this PR (one review exists, from a different reviewer, at this same head commit) — this report was produced independently of it.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff (base e2e5c5bb3d34605888fcab61f4511ab24f215c3c…head 35548d2dcb1b916f2df89e6517212d0b40b0afe6, 16 files) against crates/rocm-core/src/fix/consent.rs, crates/rocm-core/src/fix/runners.rs, crates/rocm-core/src/fix.rs, fix/catalog.rs, fix/options.rs, apps/rocm/src/cli.rs, and the e2e harness changes in tests/e2e-cucumber/.

No blocking correctness problems found. Two non-blocking items posted inline:

  1. fix/runners.rs — a failed/timed-out User-scope setx in fix-2 returns before the Machine-scope guidance block prints, when both scopes carry the override. Traced this against origin/main: the same early-return shape predates this PR (only the message wording and the set of outcomes that trigger it changed), so it's a pre-existing gap this PR reworks and partially tests around (the decline-path equivalent is now tested; the failure-path equivalent is not) rather than a new regression.
  2. fix/consent.rs — outcome_phrase doesn't surface CommandCapture.detail for a WaitFailed (and, more weakly, SpawnFailed) outcome, even though process/bounded.rs populates it with the actual underlying error for both. Doesn't affect the exit code or the "changed"/"may not have changed" claim, just diagnostic context in a rare branch.

Exit-code contract is unchanged; the new Step/StepFailure machinery derives the printed line and the returned code from one value, which is a clean way to keep those from disagreeing. Windows-only logic is unit-testable through the injected command-runner/consent closures.

CI: all checks that exercise the code (build, clippy, unit tests, hosted E2E lanes, Windows build/test, CodeQL) report success on the head commit. One self-hosted E2E consolidated report (self-hosted) check is red, but its log shows the failure is a downstream consequence of an upstream GPU lane (E2E tests (MI300X, current)) being cancelled, not a test assertion failing — doesn't look related to this diff.

Note: a separate human review requesting changes (mostly doc/comment follow-ups and a couple of design questions) is already open on this PR; this review was conducted independently of its content.


🤖 Automated by an AI agent

timeout: RUN_TIMEOUT,
};
if let Err(code) = step.run(&mut run, err) {
return code;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: when the User-scope setx step fails or times out here, the function returns immediately and never reaches the Machine-scope guidance block below (if !machine_val.is_empty() at line 368). On a host where both scopes carry the override, a failed/timed-out User-scope clear means the user never sees the elevated-shell instructions for the Machine scope, even though it still needs clearing.

This early-return shape already existed on main before this PR (same return exit::FAILED; after a non-zero setx exit), so it is not a regression this PR introduces — but this PR adds windows_decline_with_machine_scope_also_set_still_returns_5, which pins the equivalent decline-path fall-through, without an analogous test for the failure/timeout path. Worth fixing alongside, the same way the decline path handles it: record a flag and fall through to the Machine-scope print before returning the failure code.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and thanks for tracing it against main. I've left it out of this PR because it's the pre-existing early-return shape, and fixing it properly means a fall-through like the decline path's plus a failure-path test. It's now in the PR's "Follow-ups (not in this PR)" list.

"{label} timed out after {}s and was stopped",
timeout.as_secs()
),
CommandOutcome::WaitFailed => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: outcome_phrase (and Step::failure/Step::run, which only pass capture.outcome into it) never looks at CommandCapture.detail. For CommandOutcome::WaitFailed that means the printed line is the bare "could not be waited for and was stopped", even though process/bounded.rs populates detail with the actual wait error (Some(format!("{error:#}"))) for exactly this outcome. SpawnFailed(kind) is a weaker version of the same gap: the message uses the ErrorKind but not the fuller error.to_string() that run_bounded also puts in detail.

This doesn't affect the exit code or the "NOT changed" vs "may or may not have changed" correctness, just the diagnostic context in a rare failure branch. Threading capture.detail into these two arms (e.g. appending : {detail} when present) would match the existing pattern in host_gpu/diagnostics.rs::capture_diagnostic_with.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. It doesn't change the exit code or the changed/not-changed claim, so I've left it out of this PR and added it to "Follow-ups (not in this PR)": append detail when present, as capture_diagnostic_with does.

Conflicts resolved keeping both sides:

- fix/options.rs: main moved each exit code's wording into `MEANINGS`;
  the `failed` description now also says a timed-out or stopped command
  may have made its change, and that fix-2, fix-6 and fix-9 with a
  device index read first on Windows, and its comment names the
  injected-runner tests that prove each case.
- diagnose.feature: main took diagnose-62 to -67, so this branch's
  timeout and Windows read-failure scenarios are now diagnose-68 to -70
  (ids unchanged); docs and step comments follow.
- manual-testing.md: this branch's section is now 17, after main's 14-16.
- testing.md: both paragraphs kept.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…kill (ROCMAI-345)

The wording tests in fix/consent.rs pin each failure line against a
stand-in step; their comments now name the runner tests that prove what
each line claims, and say that no runner test drives a wait failure or
a signal.

The rocm-doctor skill special-cased only `exited 124`. A CLI released
before this change also reports a signalled step as `exited -1` with
`NOT changed`, and on Windows takes a failed read for an unset variable:
fix-2 can then report the override gone and exit 0, and fix-9 can run
setx over a value the user chose. One passage now covers both, with the
check to run, and the exit-4 row in reference.md points at it.

Also refresh three stale comments: what fix.rs says consent.rs holds,
fix-2's class in the report_persistent_override doc, and a PR-relative
note on the current_os WSL test.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…CMAI-345)

The merge with main moved exit-code wording into `exit::MEANINGS`, and
this branch's `failed` entry now says a timed-out or stopped command may
have made its change. Regenerate the published catalog so tools read the
same text the CLI prints.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…alog (ROCMAI-345)

An older CLI swallows a failed read of either scope, so with an
override set only in the Machine scope fix-2 can still report it gone.
The skill's check for that case now reads both scopes, and since the
output does not say which CLI produced it, the skill asks for the check
on every Windows fix-2 and fix-9 run. The published `failed` description
names the Machine read too, and the skill-eval replays are re-recorded
against the new catalog text.

Also stop the fix.rs module doc implying the consent.rs helpers are
re-exported; only the options.rs names are.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…o do with a set value (ROCMAI-345)

The published `failed` description says fix-2 reads its Machine value
even under `--dry-run`, but only the User read had a dry-run test. A
new test fails the Machine read under `--dry-run` with a User value set
and asserts exit 4 with no plan; it goes red when the Machine read
failure is swallowed.

The skill's older-CLI check told the agent to read both scopes without
saying what a set value means. It now says: a User value means
re-running fix-2, a Machine value needs an elevated PowerShell, and for
fix-9 an existing value is left for the user to change by hand. The
lead-in says the Windows case applies to every run, since the output
cannot tell an older CLI apart.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

@juhovainio, on the parts of your review of 35548d2 that have no inline thread:

  • Stale comments (item 3): fixed in d9e9fa1.
    • The fix.rs module doc now lists what fix/consent.rs holds: run_step, Step, CheckIn, StepFailure and read_failure.
    • report_persistent_override says fix-2 is FixClass::Auto on Windows and FixClass::PrintOnly on Linux/WSL.
    • The "This PR made…" test comment now states the invariant instead.
  • The 8 s read bound: kept on purpose. A slow cold PowerShell start now stops with exit 4 and changes nothing, and re-running is the recovery. The old behaviour, reading the variable as unset and carrying on, could report the override as gone, or run setx over a value the user had set.
  • diagnose-68 (was 62) waiting a real 60 s: also deliberate. The scenario tests the bound users actually get, so there is no test-only switch to shorten it. The wait runs off the cucumber executor, so concurrent scenarios are not held up.
  • Windows scenarios on non-required lanes: diagnose-39/69/70 run only on the self-hosted Strix Halo Windows lane. The applied fix-9 path is covered by unit tests, which run on the required Windows CI job. The PR body states this gap.

Since your approval I merged main in, without a rebase or force-push. Main had taken diagnose-62–67, so this PR's scenarios are now diagnose-68/69/70 and the manual-testing section is §17.

@rominf
rominf added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit ca170c9 Nov 17, 2026
46 checks passed
@rominf
rominf deleted the ROCMAI-345-fix-fix-timeout-reporting branch November 17, 2026 14:12
r0x0r added a commit that referenced this pull request Nov 17, 2026
Main landed the fix-step timeout work (#1062), which adds three diagnose
scenarios and its own catalog entry, so it collides with this branch on both
the feature file and the recorded eval fixtures.

tests/e2e-cucumber/features/diagnose.feature: main took `diagnose-68` through
`diagnose-70`. This branch's vLLM-extension scenario becomes `diagnose-71`
and moves after them, so each number still equals its position. Nothing
referenced it by number -- the `diagnose-68` mentions in `diagnose_steps.rs`,
`e2e.rs`, `docs/testing.md` and `docs/manual-testing.md` all belong to main's
new timeout scenario and are left alone.

skills/rocm-doctor/evals/fixtures/*/replies.json: seven conflicts, all in
generated files. Resolved by taking main's side and re-running
`cargo xtask eval-fixtures`, rather than hand-merging recorded output: the
fixtures are a recording of the catalog, so the only correct content is
whatever the merged catalog produces. The regeneration re-adds this branch's
`fix-23-vllm-extension-load` entry on top of main's, which is the one thing
that differs.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff (base 9e50eac5…head 49c74452, 24 files) independently via a blind multi-pass review — five fan-out passes (consent.rs+catalog.rs+options.rs+cli.rs; runners.rs; the e2e-cucumber harness changes; the docs/skill/catalog files; and a combined agent-instruction-adherence+history+prior-pattern-consistency+code-comments pass), each reading full files rather than diff hunks only, with the PR description read only afterward to check it against what was found.

No blocking findings.

Independently re-derived both non-blocking items already discussed on this PR's open threads, and confirm they're correctly characterized as pre-existing, not new regressions, and are now explicitly listed in this PR's own "Follow-ups (not in this PR)" section:

  1. report_and_clear_override_windows (crates/rocm-core/src/fix/runners.rs) returns before the Machine-scope guidance block prints when the User-scope setx fails or times out and both scopes carry the override. Traced against origin/main directly: this exact early-return shape (fail, then return before the Machine-scope if block) is present byte-for-byte in the base commit, so this PR carried it forward rather than introducing it. The decline-path equivalent now has a dedicated test (windows_decline_with_machine_scope_also_set_still_returns_5); the failure/timeout-path equivalent does not.
  2. outcome_phrase/Step::run (crates/rocm-core/src/fix/consent.rs) don't surface CommandCapture.detail for WaitFailed/SpawnFailed, even though process/bounded.rs populates that field with the real wait/spawn error text for exactly those outcomes, and host_gpu/diagnostics.rs::capture_diagnostic_with already establishes surfacing it as the codebase's convention.

Neither affects the exit-code contract or the correctness of the core "NOT changed" vs. "may or may not have changed" claim — just the completeness of test coverage / diagnostic context in a rare branch. Both are explicitly named in the PR body's "Follow-ups (not in this PR)" and acknowledged by the author in-thread, so I'm treating them as confirmed-and-deferred rather than re-raising them as new findings.

Nothing else survived verification across the five passes: the Windows read-before-write fail-closed behavior (fix-2/6/9, including under --dry-run) is consistently gated before any side effect and before opts.dry_run is even consulted; recovery-line fix-ids are sourced from shared catalog.rs constants with a test tying each one to a real runner-backed recipe; the CommandOutcome match in Step::failure/outcome_phrase is exhaustive by construction; the e2e off-executor helper (run_rocm_with_scenario_env_off_executor) isolates per-scenario state correctly and the exec /bin/sleep 600 stand-in genuinely exercises "kill only the direct child" semantics rather than leaking a process; and doc/skill/catalog wording was checked character-for-character against the actual strings the code produces and matches throughout.

CI: all checks on the current head (49c74452) are green; no failing required check.

Findings were reached blind; reconciled afterward against the existing review threads and the author's replies, which independently reached the same two non-blocking conclusions and already track them as deferred follow-up work.

🤖 Automated by an AI agent

r0x0r added a commit that referenced this pull request Nov 17, 2026
Main landed the fix-step timeout work (#1062), which took `diagnose-68`
through `diagnose-70`. This branch's four APU-memory scenarios move to
`diagnose-71` through `diagnose-74`, so each number still equals its position.

Five references follow them: the ROCMAI-912 scenario list and the
`diagnose-74` note in `docs/testing.md`, and three comments in
`diagnose_steps.rs`. The other `diagnose-68`/`-69`/`-70` references in those
same files -- the fix-4 helper, the one-minute timeout, fix-2 and fix-9 --
belong to main's new scenarios and are left alone.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants