Repository navigation
test(e2e): pin the doctor commands' contracts, failing where they are broken - #216
Conversation
Two diagnose scenarios need the catalog to produce a match, and on WSL2 it never does: that platform uses /dev/dxg and the Windows host driver, so the bare-metal Linux catalog is deliberately not run rather than emit checks (render group, /dev/kfd, modprobe amdgpu) that cannot apply. The scenarios therefore have no premise there and failed on every local WSL2 run. They are skipped, not xfail'd. An expectations.toml row means "known BUG", and this behaviour is designed and separately unit-tested, so a row would tell every later reader the opposite. `@requires-os:linux` cannot express it either — WSL2 reports an os_family of linux, so it matches. Adds `@requires-bare-metal`, resolving to Skip when the host probe reports WSL, in the same shape as the existing `@requires-no-gpu` inverse tag. Verified on a WSL2 host: both scenarios drop out of the report and the suite reports no unexpected failures. `rocm diagnose --json` there confirms the premise is genuinely absent -- out_of_scope set, zero matches -- so the skip is not masking a working scenario. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… broken Three surfaces these commands promise had no end-to-end coverage. Writing it turned two of them into evidence of defects rather than guards. `examine --json` is not the equal of the human report, and scenario 7 fails saying so: it answers before the CLI has loaded its paths or config, so the engine this host will serve on, whether an existing ROCm install was found, what is provisioned and where all have no field in it at all. The harness has been working around this for a while -- capability.rs parses the human text precisely because the JSON form could not be trusted, and records that on a real MI300X it reported no AMD GPU on a machine that has one. Scenario 8 pins that second half, xfailed only where a GPU is actually present; the mock lane and a WSL2 box both correctly agree there is none, and an unconditional row would read as a stale XPASS there. Scenario 10 fails because the framework probe can be scoped in the library but no call site passes anything but the default, so `--framework` does not exist on the command line. Skipping is the variant pinned here: its outcome is the same on every host, where asserting a named framework was probed would depend on what happens to be installed. The other two are regression cover, not findings, and are not dressed up as more than that. `examine`'s outcome says whether it managed to look rather than whether it liked what it saw -- the mock lane is where that means something, being the one lane with nothing to find. And `rocm fix` refuses to change anything without agreement: that scenario hands the CLI a home directory it owns, so the file the fix would append to is one the test can read back and prove untouched. Of the four AUTO recipes only fix-9 reaches that gate on a host with nothing installed; the note on MUTATING_FIX_ID records why the other three do not. `examine` now records its exit code in the shared step, which nothing did. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
The row went in scoped to any host with a detected GPU family, on the reasoning that a disagreement about GPU detection needs a GPU to disagree about. CI disproved the wider half: it xfailed on MI300X (gfx943) as intended, and XPASSed on Strix Halo (gfx1151) on both the Ubuntu and the Windows runner, where the two forms agree. Two independent lanes, so this is measured rather than inferred. A stale XPASS fails the lane, so leaving it would have made both Strix lanes red on every unrelated PR. The narrowing also says something about the defect. `Examination` enumerates GPUs through lspci and rocminfo, while the human report reads the kfd topology out of sysfs; where those binaries are absent -- as on the Instinct runner -- only the JSON form comes up empty. Recorded on the ticket. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
989c017 to
bfb2b83
Compare
r0x0r
left a comment
There was a problem hiding this comment.
Approving. Test-only change (no production code touched), green across every CI lane — including the ones that are the whole point: mock, MI300X, Strix Ubuntu, and Strix Windows.
Verified the load-bearing claims against the tree rather than the narrative:
-
Scenario 8 consent gate (
fix-9-igpu-dgpu). Incrates/rocm-core/src/fix.rs, the Linux path requires--device-index(the test supplies1), resolves the rc file viashell_rc_file()which honors$SHELL(the test pinsSHELL=/bin/bash, so.bashrc, matchingfix_rc_file), printsPlan: append … to <rc_file>(satisfies the "must name the file" assertion), thenconfirm()non-interactively without--yesprintsNon-interactive shell and --yes not passed; refusing to apply.and returns 5. That one string satisfies both the--yesandrefusing to applyassertions, exit 5 is documented as "user declined", andappend_lineis only reached after confirmation — so the planted.bashrcis genuinely untouched. The "only fix-9 reaches the gate on a bare host" claim is consistent with fix-2/4/6 exiting early. -
@requires-bare-metal. Tag, parsing, resolver branch, and the four unit tests are coherent.requires_os_linux_does_not_stand_in_for_requires_bare_metalearns its place — WSL2'sos_family = "linux"is exactly why@requires-os:linuxcan't express this — andbare_metal_skip_is_not_recorded_as_a_known_bugcorrectly enforces Skip-precedes-xfail so a designed limitation never gets logged as a defect. -
xfail rows are honestly scoped. EAI-7998's GPU-disagreement row is narrowed to
therock_family = "gfx94*", which the PR says was measured (xfail on gfx943, XPASS on gfx1151) rather than assumed; green CI confirms it xfailed as expected on MI300X, so thecapability.rsscraped-text workaround comment is still accurate. The two unconditional rows (JSON field gaps, missing--framework) are platform-independent, matching where those fields are printed only in the human report path.
One note for later, non-blocking: the gfx94* GPU row is now load-bearing — if the JSON has_amd_gpu path is ever fixed, that row must be removed or it becomes a masking XPASS. The MI300X half of scenario 8 was honestly flagged as first-run-in-CI, and it behaved as expected.
No correctness, scope, or leak concerns.
|
Reviewed this one closely since it makes a lot of specific factual claims (a capability.rs comment, the exact JSON schema of
One small inaccuracy worth a look, not blocking: the comment on Looks safe to approve pending final validation — this reflects a diff-only read, not the CI results themselves (which I can see are already green here). |
The tag this branch originally added, `@requires-no-wsl`, landed on main as `@requires-bare-metal` (#216) with the same resolution. The tag itself is gone from here; two prose references still named the old one. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The tag this branch originally added, `@requires-no-wsl`, landed on main as `@requires-bare-metal` (#216) with the same resolution. The tag itself is gone from here; two prose references still named the old one. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. This PR fixes no bug; it adds rows.Summary
examine,diagnoseandfixmake promises that nothing was checking. This adds five scenarios and a harness tag. Three of the five are expected to fail — they assert the behaviour we want and record the gap between that and today, rather than pinning whatever the code happens to do now.What fails, and why that is the point
examine --jsonis not the equal of the human report. It answers before the CLI has loaded its paths or config, so every CLI-side fact is out of reach. Scenario 7 fails naming them:Eleven facts in total have no field in the JSON. A caller cannot learn from
--jsonwhich engine this host will serve on, whether an existing ROCm install was found, or what is provisioned. The two forms also serialise different types, so where they overlap the names differ —os_family/os_versionagainstos/arch.The same form disagrees with the human one about the GPU. This is not new information:
capability.rs:325says the harness parses the human text because of it, and records that on a real MI300X the JSON reportedhas_amd_gpu:falseon a machine that has one. That workaround is load-bearing — every host capability this suite resolves comes from scraped text. Scenario 8 pins it, xfailed only where a GPU is actually present. The mock lane and a WSL2 box both correctly agree there is none, so an unconditional row would read as a stale XPASS and fail those lanes on unrelated PRs.I have not reproduced the MI300X half myself; the comment in the tree is the evidence. The GPU lanes in this PR are the first real test of it — if scenario 8 passes on MI300X, the comment is stale and I want to know.
--frameworkdoes not exist on the command line. The probe can be scoped in the library, but no call site passes anything but the default. Scenario 10 fails witherror: unexpected argument '--framework' found. Skipping is the variant pinned, because its outcome is identical on every host — asserting that a named framework was probed would depend on what happens to be installed.What passes, and is not dressed up as more
Two scenarios are regression cover. Nothing suggests either behaviour is broken, and inventing an xfail to improve the ratio would put a falsehood in
expectations.toml.examinereports what it finds without failing. Its outcome says whether it managed to look, not whether it liked what it saw. The mock lane is where this means something: it is the one lane with nothing to find.rocm fixwill not change anything without agreement. The only gate before the CLI edits a machine, and it had no end-to-end coverage. The scenario hands the CLI a home directory it owns, so the file the fix would append to is one the test reads back and proves untouched. Of the four AUTO recipes onlyfix-9reaches that gate on a host with nothing installed —fix-2never calls it on Linux,fix-4exits early once the user is in the groups,fix-6exits early with "no ROCm install found". The note onMUTATING_FIX_IDrecords that.Worth a reviewer's eye: probing this turned up that the recipe appends to
$HOME/.bashrc, and the piped harness does not isolateHOME. Run without the planted home it names the developer's real dotfile as the target. It refuses without--yesso nothing is written, but the scenario pinsHOMEand$SHELLso it can never reach a runner's files.@requires-bare-metalTwo diagnose scenarios need the catalog to produce a match, and on WSL2 it never does — that platform uses
/dev/dxgand the Windows host driver, so the bare-metal catalog is deliberately not run rather than emit checks that cannot apply. They failed on every local WSL2 run.They are skipped, not xfailed. A row in
expectations.tomlmeans "known BUG", and this behaviour is designed and separately unit-tested, so a row would tell every later reader the opposite.@requires-os:linuxcannot express it either — WSL2 reports anos_familyoflinux, so it matches. The new tag resolves toSkipwhen the host probe reports WSL, in the same shape as the existing@requires-no-gpuinverse tag, and there is a test asserting@requires-os:linuxstill runs on WSL2 — if that ever starts skipping, the two tags have collapsed and this one is redundant.Test plan
Run on a WSL2 host, which is the only place
@requires-bare-metalcan be verified end to end:cargo xtask e2e— 38 scenarios, 33 passed, 5 xfail, 0 XPASS, 0 unexpected failures. The two diagnose scenarios drop out of the report; previously they failed every run.rocm diagnose --symptom HSA_STATUS_ERROR_INVALID_ISA --jsonon this host returnsout_of_scopeset and zero matches. The premise is genuinely absent.cargo test -p e2e-cucumber --lib— 62 passed, including four new resolver tests.cargo fmt --all --check, workspaceclippy --all-targets -D warnings, andclippy -p e2e-cucumber --test e2e -D warningsall clean.Not verified locally: every scenario here has only ever run on WSL2. The mock lane, MI300X, Strix Ubuntu and Strix Windows are all first runs, and
@requires-bare-metalhas never resolved on a non-WSL host. Reading the per-lanereport.jsonrather than the job conclusions is the point of this PR's CI.🤖 Generated with Claude Code