Skip to content

test(e2e): pin the doctor commands' contracts, failing where they are broken - #216

Merged
volen-silo merged 3 commits into
mainfrom
test/e2e-doctor-command-contracts
Aug 12, 2026
Merged

volen-silo merged 3 commits into
mainfrom
test/e2e-doctor-command-contracts

Conversation

@volen-silo

Copy link
Copy Markdown
Collaborator
  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. This PR fixes no bug; it adds rows.

Summary

examine, diagnose and fix make promises that nothing was checking. This adds five scenarios and a harness tag. Three of the five are expected to fail — they assert the behaviour we want and record the gap between that and today, rather than pinning whatever the code happens to do now.

What fails, and why that is the point

examine --json is not the equal of the human report. It answers before the CLI has loaded its paths or config, so every CLI-side fact is out of reach. Scenario 7 fails naming them:

the machine-readable form withholds what the readable one states:
  effective_default_engine (the human report says "lemonade")
  legacy_rocm_status (the human report says "not_detected")
  managed_runtimes (the human report says "0")
  config_dir (the human report says "/tmp/rocm-e2e-lgmswL/config")

Eleven facts in total have no field in the JSON. A caller cannot learn from --json which engine this host will serve on, whether an existing ROCm install was found, or what is provisioned. The two forms also serialise different types, so where they overlap the names differ — os_family/os_version against os/arch.

The same form disagrees with the human one about the GPU. This is not new information: capability.rs:325 says the harness parses the human text because of it, and records that on a real MI300X the JSON reported has_amd_gpu:false on a machine that has one. That workaround is load-bearing — every host capability this suite resolves comes from scraped text. Scenario 8 pins it, xfailed only where a GPU is actually present. The mock lane and a WSL2 box both correctly agree there is none, so an unconditional row would read as a stale XPASS and fail those lanes on unrelated PRs.

I have not reproduced the MI300X half myself; the comment in the tree is the evidence. The GPU lanes in this PR are the first real test of it — if scenario 8 passes on MI300X, the comment is stale and I want to know.

--framework does not exist on the command line. The probe can be scoped in the library, but no call site passes anything but the default. Scenario 10 fails with error: unexpected argument '--framework' found. Skipping is the variant pinned, because its outcome is identical on every host — asserting that a named framework was probed would depend on what happens to be installed.

What passes, and is not dressed up as more

Two scenarios are regression cover. Nothing suggests either behaviour is broken, and inventing an xfail to improve the ratio would put a falsehood in expectations.toml.

  • examine reports what it finds without failing. Its outcome says whether it managed to look, not whether it liked what it saw. The mock lane is where this means something: it is the one lane with nothing to find.

  • rocm fix will not change anything without agreement. The only gate before the CLI edits a machine, and it had no end-to-end coverage. The scenario hands the CLI a home directory it owns, so the file the fix would append to is one the test reads back and proves untouched. Of the four AUTO recipes only fix-9 reaches that gate on a host with nothing installed — fix-2 never calls it on Linux, fix-4 exits early once the user is in the groups, fix-6 exits early with "no ROCm install found". The note on MUTATING_FIX_ID records that.

    Worth a reviewer's eye: probing this turned up that the recipe appends to $HOME/.bashrc, and the piped harness does not isolate HOME. Run without the planted home it names the developer's real dotfile as the target. It refuses without --yes so nothing is written, but the scenario pins HOME and $SHELL so it can never reach a runner's files.

@requires-bare-metal

Two diagnose scenarios need the catalog to produce a match, and on WSL2 it never does — that platform uses /dev/dxg and the Windows host driver, so the bare-metal catalog is deliberately not run rather than emit checks that cannot apply. They failed on every local WSL2 run.

They are skipped, not xfailed. A row in expectations.toml means "known BUG", and this behaviour is designed and separately unit-tested, so a row would tell every later reader the opposite. @requires-os:linux cannot express it either — WSL2 reports an os_family of linux, so it matches. The new tag resolves to Skip when the host probe reports WSL, in the same shape as the existing @requires-no-gpu inverse tag, and there is a test asserting @requires-os:linux still runs on WSL2 — if that ever starts skipping, the two tags have collapsed and this one is redundant.

Test plan

Run on a WSL2 host, which is the only place @requires-bare-metal can be verified end to end:

  • cargo xtask e2e — 38 scenarios, 33 passed, 5 xfail, 0 XPASS, 0 unexpected failures. The two diagnose scenarios drop out of the report; previously they failed every run.
  • Confirmed the skip is not masking a working scenario: rocm diagnose --symptom HSA_STATUS_ERROR_INVALID_ISA --json on this host returns out_of_scope set and zero matches. The premise is genuinely absent.
  • cargo test -p e2e-cucumber --lib — 62 passed, including four new resolver tests.
  • cargo fmt --all --check, workspace clippy --all-targets -D warnings, and clippy -p e2e-cucumber --test e2e -D warnings all clean.

Not verified locally: every scenario here has only ever run on WSL2. The mock lane, MI300X, Strix Ubuntu and Strix Windows are all first runs, and @requires-bare-metal has never resolved on a non-WSL host. Reading the per-lane report.json rather than the job conclusions is the point of this PR's CI.

🤖 Generated with Claude Code

@volen-silo
volen-silo requested a review from a team as a code owner August 12, 2026 08:25
Two diagnose scenarios need the catalog to produce a match, and on WSL2 it
never does: that platform uses /dev/dxg and the Windows host driver, so the
bare-metal Linux catalog is deliberately not run rather than emit checks
(render group, /dev/kfd, modprobe amdgpu) that cannot apply. The scenarios
therefore have no premise there and failed on every local WSL2 run.

They are skipped, not xfail'd. An expectations.toml row means "known BUG",
and this behaviour is designed and separately unit-tested, so a row would
tell every later reader the opposite. `@requires-os:linux` cannot express it
either — WSL2 reports an os_family of linux, so it matches.

Adds `@requires-bare-metal`, resolving to Skip when the host probe reports
WSL, in the same shape as the existing `@requires-no-gpu` inverse tag.

Verified on a WSL2 host: both scenarios drop out of the report and the suite
reports no unexpected failures. `rocm diagnose --json` there confirms the
premise is genuinely absent -- out_of_scope set, zero matches -- so the skip
is not masking a working scenario.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… broken

Three surfaces these commands promise had no end-to-end coverage. Writing it
turned two of them into evidence of defects rather than guards.

`examine --json` is not the equal of the human report, and scenario 7 fails
saying so: it answers before the CLI has loaded its paths or config, so the
engine this host will serve on, whether an existing ROCm install was found,
what is provisioned and where all have no field in it at all. The harness has
been working around this for a while -- capability.rs parses the human text
precisely because the JSON form could not be trusted, and records that on a
real MI300X it reported no AMD GPU on a machine that has one. Scenario 8 pins
that second half, xfailed only where a GPU is actually present; the mock lane
and a WSL2 box both correctly agree there is none, and an unconditional row
would read as a stale XPASS there.

Scenario 10 fails because the framework probe can be scoped in the library but
no call site passes anything but the default, so `--framework` does not exist
on the command line. Skipping is the variant pinned here: its outcome is the
same on every host, where asserting a named framework was probed would depend
on what happens to be installed.

The other two are regression cover, not findings, and are not dressed up as
more than that. `examine`'s outcome says whether it managed to look rather
than whether it liked what it saw -- the mock lane is where that means
something, being the one lane with nothing to find. And `rocm fix` refuses to
change anything without agreement: that scenario hands the CLI a home
directory it owns, so the file the fix would append to is one the test can
read back and prove untouched. Of the four AUTO recipes only fix-9 reaches
that gate on a host with nothing installed; the note on MUTATING_FIX_ID
records why the other three do not.

`examine` now records its exit code in the shared step, which nothing did.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
The row went in scoped to any host with a detected GPU family, on the
reasoning that a disagreement about GPU detection needs a GPU to disagree
about. CI disproved the wider half: it xfailed on MI300X (gfx943) as
intended, and XPASSed on Strix Halo (gfx1151) on both the Ubuntu and the
Windows runner, where the two forms agree. Two independent lanes, so this is
measured rather than inferred.

A stale XPASS fails the lane, so leaving it would have made both Strix lanes
red on every unrelated PR.

The narrowing also says something about the defect. `Examination` enumerates
GPUs through lspci and rocminfo, while the human report reads the kfd
topology out of sysfs; where those binaries are absent -- as on the Instinct
runner -- only the JSON form comes up empty. Recorded on the ticket.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

@r0x0r r0x0r left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Test-only change (no production code touched), green across every CI lane — including the ones that are the whole point: mock, MI300X, Strix Ubuntu, and Strix Windows.

Verified the load-bearing claims against the tree rather than the narrative:

  • Scenario 8 consent gate (fix-9-igpu-dgpu). In crates/rocm-core/src/fix.rs, the Linux path requires --device-index (the test supplies 1), resolves the rc file via shell_rc_file() which honors $SHELL (the test pins SHELL=/bin/bash, so .bashrc, matching fix_rc_file), prints Plan: append … to <rc_file> (satisfies the "must name the file" assertion), then confirm() non-interactively without --yes prints Non-interactive shell and --yes not passed; refusing to apply. and returns 5. That one string satisfies both the --yes and refusing to apply assertions, exit 5 is documented as "user declined", and append_line is only reached after confirmation — so the planted .bashrc is genuinely untouched. The "only fix-9 reaches the gate on a bare host" claim is consistent with fix-2/4/6 exiting early.

  • @requires-bare-metal. Tag, parsing, resolver branch, and the four unit tests are coherent. requires_os_linux_does_not_stand_in_for_requires_bare_metal earns its place — WSL2's os_family = "linux" is exactly why @requires-os:linux can't express this — and bare_metal_skip_is_not_recorded_as_a_known_bug correctly enforces Skip-precedes-xfail so a designed limitation never gets logged as a defect.

  • xfail rows are honestly scoped. EAI-7998's GPU-disagreement row is narrowed to therock_family = "gfx94*", which the PR says was measured (xfail on gfx943, XPASS on gfx1151) rather than assumed; green CI confirms it xfailed as expected on MI300X, so the capability.rs scraped-text workaround comment is still accurate. The two unconditional rows (JSON field gaps, missing --framework) are platform-independent, matching where those fields are printed only in the human report path.

One note for later, non-blocking: the gfx94* GPU row is now load-bearing — if the JSON has_amd_gpu path is ever fixed, that row must be removed or it becomes a masking XPASS. The MI300X half of scenario 8 was honestly flagged as first-run-in-CI, and it behaved as expected.

No correctness, scope, or leak concerns.

@volen-silo
volen-silo added this pull request to the merge queue Aug 12, 2026
@juhovainio

Copy link
Copy Markdown
Collaborator

Reviewed this one closely since it makes a lot of specific factual claims (a capability.rs comment, the exact JSON schema of examine --json, exit codes, which fix recipes reach the confirmation gate) and I wanted to check them against the actual code rather than take the PR body at its word. They all held up:

  • The GPU-detection disagreement is real and documented where the PR says (a comment in the harness's capability.rs explaining why it parses human text instead of --json for GPU detection, following the MI300X observation).
  • examine --json's schema genuinely lacks fields the human report has, has no --framework flag, and status really does carry "wsl" as one of its values (I checked Examination::probe() and the CLI's arg definitions directly).
  • Exit code 5 for a declined mutating fix is correct, and run_rocm_with_env / shell_rc_file behave as the comments describe.
  • The narrowing of the GPU-disagreement xfail from gfx* to gfx94* in the third commit is a good example of catching an overly broad expectation from real CI signal (XPASS on Strix Halo) rather than leaving it flaky.

One small inaccuracy worth a look, not blocking: the comment on MUTATING_FIX_ID in diagnose_steps.rs says fix-4 (render-group) "exits early once the user is already in the groups." Looking at run_render_group in crates/rocm-core/src/fix.rs, there's no such check — it only exits early for a missing $USER/$LOGNAME, missing usermod, or no sudo/root, so on a normal CI host it would also reach the confirmation gate. Doesn't affect the test (it only ever invokes fix-9 directly), just the rationale in the comment.

Looks safe to approve pending final validation — this reflects a diff-only read, not the CI results themselves (which I can see are already green here).

Merged via the queue into main with commit 63639db Aug 12, 2026
23 checks passed
@volen-silo
volen-silo deleted the test/e2e-doctor-command-contracts branch August 12, 2026 11:51
rominf added a commit that referenced this pull request Aug 12, 2026
The tag this branch originally added, `@requires-no-wsl`, landed on main
as `@requires-bare-metal` (#216) with the same resolution. The tag itself
is gone from here; two prose references still named the old one.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf added a commit that referenced this pull request Aug 12, 2026
The tag this branch originally added, `@requires-no-wsl`, landed on main
as `@requires-bare-metal` (#216) with the same resolution. The tag itself
is gone from here; two prose references still named the old one.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants