diff --git a/tests/e2e-cucumber/expectations.toml b/tests/e2e-cucumber/expectations.toml index 80335b1d3c..359b77502e 100644 --- a/tests/e2e-cucumber/expectations.toml +++ b/tests/e2e-cucumber/expectations.toml @@ -7,8 +7,12 @@ # host capability probe (see src/capability.rs) and is NOT encoded here. # # Resolution at runtime (see tests/e2e-cucumber/src/expectation.rs): -# 1. A scenario whose @requires-gpu / @requires-engine can't be satisfied on -# this host is SKIPPED (not-applicable) — never listed here. +# 1. A scenario whose @requires-gpu / @requires-engine / @requires-bare-metal +# can't be satisfied on this host is SKIPPED (not-applicable) — never listed +# here. Reach for a tag, not a row, whenever the scenario simply has no +# premise on that host: a row here says "known BUG", so using one for +# designed-in behaviour (e.g. diagnose not running its bare-metal catalog on +# WSL2) tells every later reader the opposite of the truth. # 2. Otherwise, the FIRST matching condition below marks it expected-fail # (xfail); a scenario that then PASSES is an XPASS — a stale entry to be # removed, UNLESS the row is `flaky = true`, which tolerates either result @@ -165,3 +169,32 @@ bug = "EAI-7423" reason = "Lemonade managed serve reaches ready then shuts down immediately, so the benchmark never reaches a live model." serve_timeout_secs = 90 flaky = true + +# --- EAI-7998: `examine --json` is not the equal of the human report. It +# answers before the CLI has loaded its paths or config, so eleven facts the +# text form states have no field in it at all. Platform-independent — the +# missing fields are missing everywhere. --- +[["examine-machine-readable-report"]] +when = {} +bug = "EAI-7998" +reason = "`examine --json` withholds CLI-side facts (engine, install status, managed runtimes, paths) that the human report states." + +# --- EAI-7998: the same form also disagrees with the human report about GPU +# detection — but only on Instinct. Measured, not assumed: this row first ran +# scoped to `gfx*`, which xfailed correctly on MI300X (gfx943) and XPASSed on +# Strix Halo (gfx1151), where the two forms agree. Narrowed to the family where +# it actually reproduces; the mock lane and a WSL2 box correctly agree there is +# no GPU at all, so they stay expected-pass. --- +[["examine-both-forms-agree-on-gpu"]] +when = { therock_family = "gfx94*" } +bug = "EAI-7998" +reason = "On Instinct the JSON form reports has_amd_gpu:false while the human report names the gfx target." + +# --- EAI-7193: the framework probe can be scoped in the library, but no call +# site passes anything but the default, so `--framework` does not exist on the +# command line and the choice is unreachable. Platform-independent (pure +# argument parsing). Remove this row when the flag lands. --- +[["examine-can-skip-framework-probing"]] +when = {} +bug = "EAI-7193" +reason = "`examine` has no --framework flag, so the framework probe cannot be scoped from the command line." diff --git a/tests/e2e-cucumber/features/diagnose.feature b/tests/e2e-cucumber/features/diagnose.feature index c0611c99eb..6497b1f088 100644 --- a/tests/e2e-cucumber/features/diagnose.feature +++ b/tests/e2e-cucumber/features/diagnose.feature @@ -10,7 +10,13 @@ Feature: Diagnosing failures and listing fixes # dependent. They assert the SHAPE of a diagnosis (a scored match with an id # and a plan) and the query/refusal contracts. - @id:diagnose-matches-known-symptom + # @requires-bare-metal: these two need the catalog to actually produce a match. + # On WSL2 the catalog is deliberately not run at all — that platform uses + # /dev/dxg and the Windows host driver, so bare-metal Linux diagnoses would be + # false positives — which leaves these scenarios with no premise there. That is + # designed behaviour with its own unit test, not a bug, so they are skipped + # rather than xfail'd. `@requires-os:linux` would not do it: WSL2 is linux. + @id:diagnose-matches-known-symptom @requires-bare-metal Scenario: 1 - Diagnosing a recognised failure reports a likely cause and a fix Given a user who hit a known ROCm failure When the user asks the CLI to diagnose that symptom @@ -23,7 +29,7 @@ Feature: Diagnosing failures and listing fixes When the user asks the CLI to diagnose that symptom in machine-readable form Then the CLI always points to somewhere the problem can be reported - @id:diagnose-json-has-match-flag + @id:diagnose-json-has-match-flag @requires-bare-metal Scenario: 3 - A diagnosis is available in machine-readable form for tooling Given a user who hit a known ROCm failure When the user asks the CLI to diagnose that symptom in machine-readable form @@ -56,3 +62,19 @@ Feature: Diagnosing failures and listing fixes Given a user who refers to a cause by its position in the diagnosis When the user asks the CLI to apply that fix Then the CLI refuses and explains that a position is not a fix-id + + # The one gate standing between `rocm fix` and an edited machine, and until now + # it had no end-to-end coverage. The scenario gives the CLI a home directory it + # owns, so the file the fix would edit is one the scenario can read back: the + # refusal must not depend on what is in the runner's dotfiles, and a regression + # here must not be able to reach them. + # Linux-only because the assertion is "the file is untouched": on Windows the + # same recipe persists through `setx` into the user environment, which the + # suite cannot plant or read back safely. The gate itself is shared code, so + # this still guards it — just not the Windows persistence step. + @id:fix-requires-agreement-before-changing-anything @requires-os:linux + Scenario: 8 - A fix that changes the machine is not applied without agreement + Given a user who has chosen a fix that would change the machine + When the user asks the CLI to apply it without agreeing to the change + Then the CLI refuses and explains that it needs agreement + And the file the fix would have changed is untouched diff --git a/tests/e2e-cucumber/features/examine.feature b/tests/e2e-cucumber/features/examine.feature index 9f24f7eb2c..a80143890c 100644 --- a/tests/e2e-cucumber/features/examine.feature +++ b/tests/e2e-cucumber/features/examine.feature @@ -50,3 +50,49 @@ Feature: GPU detection and system inspection And the inspection names that install's version And the inspection does not claim nothing is installed And the inspection suggests setting up a CLI-managed install + + # Expected to FAIL. The machine-readable form is a separate code path, not a + # re-rendering of the human one: it answers before the CLI has loaded its paths + # or config, so every CLI-side fact is out of reach. Eleven things the human + # report states have no field in it at all — among them which engine this host + # will serve on and whether an existing ROCm install was found. Tooling reads + # this form; it should not be the weaker of the two. + @id:examine-machine-readable-report + Scenario: 7 - What the inspection tells a tool matches what it tells a person + When the user inspects the system both for reading and for scripting + Then the machine-readable form states everything the readable one does + + # Expected to FAIL on Instinct. The harness parses the human text rather than + # this form precisely because of this defect, and says so in capability.rs — + # on a real MI300X the machine-readable form reported no AMD GPU on a machine + # that has one. That workaround makes the disagreement load-bearing: every + # host capability the suite resolves comes from scraped text because this form + # could not be trusted. + # + # It is narrower than "any host with a GPU": Strix Halo (gfx1151) agrees, + # MI300X (gfx943) does not. The expectations row is scoped to match. + @id:examine-both-forms-agree-on-gpu + Scenario: 8 - Both forms of the inspection agree about the GPU + When the user inspects the system both for reading and for scripting + Then both reports agree on whether this machine has an AMD GPU + And both reports agree on whether this platform is in scope + + # `examine` is an inspector: the outcome says whether it managed to look, not + # whether it liked what it saw. Finding no GPU is a finding, not a failure. + # This is the mock lane's to prove — it is the one lane with nothing to find. + @id:examine-reports-without-failing + Scenario: 9 - Inspecting a machine reports what it finds without failing + When the user inspects the system + Then the inspection completes successfully + And it states a verdict for this machine + + # Expected to FAIL until the framework choice is reachable from the command + # line. The library can already scope the probe, but every caller passes the + # default, so the user cannot. Leaving the frameworks out is the variant worth + # pinning here: the outcome is identical on every host, whereas asserting that + # a *named* framework was probed would depend on what happens to be installed. + @id:examine-can-skip-framework-probing + Scenario: 10 - The user can leave the frameworks out of the inspection + When the user inspects the system without probing frameworks + Then the inspection reports that it skipped the frameworks + And it still states a verdict for this machine diff --git a/tests/e2e-cucumber/src/expectation.rs b/tests/e2e-cucumber/src/expectation.rs index 0e2164f2a7..4d8b48f7d3 100644 --- a/tests/e2e-cucumber/src/expectation.rs +++ b/tests/e2e-cucumber/src/expectation.rs @@ -26,6 +26,7 @@ const REQUIRES_ENGINE_PREFIX: &str = "requires-engine:"; const REQUIRES_OS_PREFIX: &str = "requires-os:"; const REQUIRES_GPU_TAG: &str = "requires-gpu"; const REQUIRES_NO_GPU_TAG: &str = "requires-no-gpu"; +const REQUIRES_BARE_METAL_TAG: &str = "requires-bare-metal"; const SERVE_TIMEOUT_PREFIX: &str = "serve-timeout:"; const NIGHTLY_TAG: &str = "nightly"; const LIFECYCLE_TAG: &str = "lifecycle"; @@ -58,6 +59,18 @@ pub struct ScenarioDecl { /// GPU — the inverse of `requires_gpu`. This is how the no-GPU fail-fast path /// gets live coverage: it runs on the GitHub-hosted mock job. pub requires_no_gpu: bool, + /// `@requires-bare-metal`: the scenario's premise is a host running the + /// in-tree amdgpu driver, so it does not hold under WSL2 — which uses + /// `/dev/dxg` and the Windows host driver instead. `rocm diagnose` skips its + /// bare-metal catalog there *by design* rather than emitting Linux diagnoses + /// that cannot apply, so a scenario needing a catalog match has no premise on + /// WSL2. This is a SKIP and not an xfail row: the behaviour is deliberate and + /// separately unit-tested, and recording it as a known bug would tell every + /// later reader the opposite. + /// + /// `@requires-os:linux` cannot express this — WSL2 reports an os_family of + /// `linux`, so it matches. + pub requires_bare_metal: bool, /// Engine the scenario pins via `@requires-engine:` (if any). pub requires_engine: Option, /// OS the scenario requires via `@requires-os:` (e.g. "linux"), if any — @@ -94,6 +107,7 @@ impl ScenarioDecl { let mut id = None; let mut requires_gpu = false; let mut requires_no_gpu = false; + let mut requires_bare_metal = false; let mut requires_engine = None; let mut requires_os = None; let mut serve_timeout_secs = None; @@ -117,6 +131,8 @@ impl ScenarioDecl { requires_gpu = true; } else if tag == REQUIRES_NO_GPU_TAG { requires_no_gpu = true; + } else if tag == REQUIRES_BARE_METAL_TAG { + requires_bare_metal = true; } else if tag == NIGHTLY_TAG { nightly = true; } else if tag == LIFECYCLE_TAG { @@ -129,6 +145,7 @@ impl ScenarioDecl { id, requires_gpu, requires_no_gpu, + requires_bare_metal, requires_engine, requires_os, serve_timeout_secs, @@ -323,8 +340,9 @@ pub struct PlatformManifest<'a> { /// /// 1. Not-applicable → `Skip`: a `@nightly` scenario when nightly isn't included, /// a `@merge-queue` scenario outside the merge queue, a `@requires-gpu` -/// scenario on a host with no AMD GPU, a `@requires-os:` scenario on a -/// different OS, or a scenario whose effective engine can't start. +/// scenario on a host with no AMD GPU, a `@requires-bare-metal` scenario on +/// WSL2, a `@requires-os:` scenario on a different OS, or a scenario whose +/// effective engine can't start. /// 2. First matching `expectations.toml` condition → `ExpectXfail`. /// 3. Otherwise → `ExpectPass`. /// @@ -370,6 +388,11 @@ pub fn resolve( reason: "requires a host with no AMD GPU; this host has one".to_owned(), }; } + if decl.requires_bare_metal && cap.is_wsl { + return Expectation::Skip { + reason: "requires a bare-metal host; this one is WSL2".to_owned(), + }; + } if let Some(os) = &decl.requires_os && !os.eq_ignore_ascii_case(&cap.os_family) { @@ -474,6 +497,19 @@ mod tests { effective_serve_engine: "lemonade".into(), platform_slug: "strix-halo".into(), }, + // A WSL2 dev box: `os_family` is `linux` and a GPU is visible through + // /dev/dxg, so nothing but `is_wsl` distinguishes it from bare metal. + // That is precisely why `@requires-os:linux` cannot stand in for + // `@requires-bare-metal`. + "wsl2" => HostCapability { + os_family: "linux".into(), + is_wsl: true, + gfx_target: Some("gfx1151".into()), + has_amd_gpu: true, + available_engines: vec!["lemonade".into(), "vllm".into()], + effective_serve_engine: "lemonade".into(), + platform_slug: "wsl2".into(), + }, _ => HostCapability { os_family: "other".into(), is_wsl: false, @@ -528,6 +564,14 @@ serve_timeout_secs = 90 assert!(d.lifecycle); } + #[test] + fn bare_metal_tag_parses_in_both_shapes() { + assert!(decl(&["id:x", "requires-bare-metal"]).requires_bare_metal); + assert!(decl(&["@id:x", "@requires-bare-metal"]).requires_bare_metal); + // Absent by default, so no existing scenario changes meaning. + assert!(!decl(&["id:x", "requires-gpu"]).requires_bare_metal); + } + #[test] fn serve_timeout_tag_parses_seconds() { let d = decl(&["id:serve-large-model-inference", "serve-timeout:2400"]); @@ -683,6 +727,59 @@ serve_timeout_secs = 90 )); } + #[test] + fn requires_bare_metal_skips_on_wsl_and_runs_everywhere_else() { + let m = Expectations::default(); + let d = decl(&["id:diagnose-matches-known-symptom", "requires-bare-metal"]); + assert!(matches!( + resolve(&d, &cap("wsl2"), &m, false, false, false), + Expectation::Skip { .. } + )); + // Bare-metal hosts of every shape still run it — including the mock lane, + // which is where these scenarios earn their keep as a required check. + for host in ["mock", "mi300x", "strix-ubuntu", "strix-windows"] { + assert_eq!( + resolve(&d, &cap(host), &m, false, false, false), + Expectation::ExpectPass, + "{host} is bare metal and must still run the scenario" + ); + } + } + + #[test] + fn requires_os_linux_does_not_stand_in_for_requires_bare_metal() { + // The reason the tag has to exist: WSL2 reports os_family "linux", so an + // `@requires-os:linux` scenario runs there. If this ever starts skipping, + // the two tags have collapsed into one and the new one is redundant. + let m = Expectations::default(); + let d = decl(&["id:some-linux-scenario", "requires-os:linux"]); + assert_eq!( + resolve(&d, &cap("wsl2"), &m, false, false, false), + Expectation::ExpectPass + ); + } + + #[test] + fn bare_metal_skip_is_not_recorded_as_a_known_bug() { + // WSL2 out-of-scope is deliberate product behaviour, so it must resolve to + // Skip even when the scenario also carries an xfail row — otherwise the + // report would call a designed-in limitation a defect. + let m = Expectations::parse( + r#" +[["diagnose-matches-known-symptom"]] +when = {} +bug = "EAI-0000" +reason = "unrelated open bug" +"#, + ) + .unwrap(); + let d = decl(&["id:diagnose-matches-known-symptom", "requires-bare-metal"]); + assert!(matches!( + resolve(&d, &cap("wsl2"), &m, false, false, false), + Expectation::Skip { .. } + )); + } + #[test] fn vllm_pinned_scenario_skips_where_vllm_cannot_start() { let m = Expectations::default(); diff --git a/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs b/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs index fdb231f3e0..fac3fc4ba4 100644 --- a/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs @@ -22,6 +22,39 @@ const KNOWN_SYMPTOM: &str = "HSA_STATUS_ERROR_INVALID_ISA"; /// dry-run, which would make the assertion host-dependent. const PREVIEW_FIX_ID: &str = "fix-1-arch"; +/// A recipe that would really change the machine, used to prove the CLI asks +/// first. Of the four AUTO recipes this is the only one that reaches the +/// confirmation gate on a host with nothing installed: `fix-2-unset-override` +/// never calls it on Linux, `fix-4-render-group` exits early once the user is +/// already in the groups, and `fix-6-path` exits early with "no ROCm install +/// found". This one needs only `--device-index`, which the scenario supplies. +const MUTATING_FIX_ID: &str = "fix-9-igpu-dgpu"; + +/// Contents planted in the scenario's own shell rc file. The assertion is that +/// this survives the run byte for byte. +const PLANTED_RC: &str = "# planted by the e2e suite; the fix must not touch this\n"; + +/// The home directory handed to the fix under test, inside the scenario's +/// isolated root. `fix-9` appends to a shell rc file under `$HOME`, and the +/// piped harness otherwise lets the CLI inherit the runner's real one — so +/// without this the scenario would read (and a regression could edit) the +/// dotfiles of whoever is running the suite. +fn fix_home(world: &E2eWorld) -> std::path::PathBuf { + world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .join("fix-home") +} + +/// The rc file `fix-9` resolves to under [`fix_home`]. `shell_rc_file` picks +/// `.zshrc` when `$SHELL` names zsh, so the scenario pins `$SHELL` to bash and +/// this stays `.bashrc` regardless of the runner's login shell. +fn fix_rc_file(world: &E2eWorld) -> std::path::PathBuf { + fix_home(world).join(".bashrc") +} + // ── Given ────────────────────────────────────────────────────────── #[given("a user who hit a known ROCm failure")] @@ -44,6 +77,14 @@ async fn user_named_unknown_fix(world: &mut E2eWorld) { world.model_name = Some("fix-does-not-exist".to_string()); } +#[given("a user who has chosen a fix that would change the machine")] +async fn user_chose_mutating_fix(world: &mut E2eWorld) { + let rc_file = fix_rc_file(world); + std::fs::create_dir_all(fix_home(world)).expect("failed to create the scenario's home dir"); + std::fs::write(&rc_file, PLANTED_RC).expect("failed to plant the shell rc file"); + world.model_name = Some(MUTATING_FIX_ID.to_string()); +} + #[given("a user who refers to a cause by its position in the diagnosis")] async fn user_named_diagnosis_position(world: &mut E2eWorld) { // Quoted deliberately: unquoted, the shell treats `#1` as a comment and the @@ -94,6 +135,23 @@ async fn user_applies_fix(world: &mut E2eWorld) { world.cli_rc = Some(rc); } +#[when("the user asks the CLI to apply it without agreeing to the change")] +async fn user_applies_fix_without_agreeing(world: &mut E2eWorld) { + let fix_id = world.model_name.clone().expect("no fix id set"); + let home = fix_home(world).display().to_string(); + // No `--yes`, and the harness pipes stdin, so this is the non-interactive + // case the gate exists for. `--device-index` is what carries `fix-9` past + // its own "tell me which GPU" branch and up to the gate. + let (stdout, stderr, rc) = crate::run_rocm_with_env( + world, + &["fix", &fix_id, "--device-index", "1"], + &[("HOME", home.as_str()), ("SHELL", "/bin/bash")], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + // ── Then ─────────────────────────────────────────────────────────── #[then("the CLI reports a likely cause with a suggested fix")] @@ -299,3 +357,46 @@ async fn assert_position_argument_corrected(world: &mut E2eWorld) { "the refusal must name the identifier to use instead:\n{combined}" ); } + +#[then("the CLI refuses and explains that it needs agreement")] +async fn assert_refuses_without_agreement(world: &mut E2eWorld) { + let output = world.cli_output.as_ref().expect("no fix output"); + // The refusal has to say *why* and how to proceed. A bare non-zero exit + // reads as a broken fix rather than a deliberate stop. + assert!( + output.contains("--yes"), + "the refusal must name what to pass to proceed:\n{output}" + ); + assert!( + output.contains("refusing to apply"), + "the refusal must say it did not apply the fix:\n{output}" + ); + // Distinct from the unknown-id refusal (2), so a script can tell "you did + // not agree" apart from "no such fix". + assert_eq!( + world.cli_rc, + Some(5), + "declining to apply is its own outcome, not an error:\n{output}" + ); +} + +#[then("the file the fix would have changed is untouched")] +async fn assert_rc_file_untouched(world: &mut E2eWorld) { + let rc_file = fix_rc_file(world); + let output = world.cli_output.as_ref().expect("no fix output"); + // The fix names its target before asking. If it ever stops naming this file + // the scenario would be reading back a file the CLI never intended to edit, + // and would pass without proving anything. + assert!( + output.contains(&rc_file.display().to_string()), + "the fix must name {} as what it would change:\n{output}", + rc_file.display() + ); + let after = std::fs::read_to_string(&rc_file).expect("the planted rc file should still exist"); + assert_eq!( + after, + PLANTED_RC, + "declining the fix must leave {} byte-for-byte unchanged", + rc_file.display() + ); +} diff --git a/tests/e2e-cucumber/tests/e2e/examine_steps.rs b/tests/e2e-cucumber/tests/e2e/examine_steps.rs index 09349befd6..9489614ff8 100644 --- a/tests/e2e-cucumber/tests/e2e/examine_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/examine_steps.rs @@ -34,8 +34,11 @@ async fn user_lists_engines(world: &mut E2eWorld) { #[when("the user inspects the system")] async fn user_inspects_system(world: &mut E2eWorld) { - let (stdout, _, _) = crate::run_rocm(world, &["examine"]); + // The exit code is recorded because `examine`'s contract is partly about it: + // it reports whether the inspection ran, not whether it liked what it found. + let (stdout, _, rc) = crate::run_rocm(world, &["examine"]); world.cli_output = Some(stdout); + world.cli_rc = Some(rc); } #[when("the user asks for help")] @@ -202,3 +205,210 @@ async fn assert_suggests_managed_runtime(world: &mut E2eWorld) { "expected guidance to install sdk:\n{output}" ); } + +// ── Machine-readable inspection ──────────────────────────────────── + +/// The verdict field `examine --json` carries, and the one the harness's own +/// capability probe and the ROCm Doctor skill both read. Asserting the field is +/// present and non-empty — rather than pinning a value — keeps this a contract +/// test: the set of verdicts is host-dependent and grows over time. +const VERDICT_FIELD: &str = "status"; + +fn parsed_json(world: &E2eWorld) -> serde_json::Value { + let output = world.cli_output.as_ref().expect("no command was run"); + serde_json::from_str(output) + .unwrap_or_else(|e| panic!("`examine --json` did not emit valid JSON ({e}):\n{output}")) +} + +#[when("the user inspects the system both for reading and for scripting")] +async fn user_inspects_both_ways(world: &mut E2eWorld) { + let (human, _, _) = crate::run_rocm(world, &["examine"]); + let (json, _, rc) = crate::run_rocm(world, &["examine", "--json"]); + // Both are needed by the comparison step; the human form goes in the stderr + // slot rather than adding a World field for one scenario. + world.cli_stderr = Some(human); + world.cli_output = Some(json); + world.cli_rc = Some(rc); +} + +#[when("the user inspects the system without probing frameworks")] +async fn user_inspects_skipping_frameworks(world: &mut E2eWorld) { + let (stdout, stderr, rc) = + crate::run_rocm(world, &["examine", "--framework", "skip", "--json"]); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +/// Facts the human report states that a tool has at least as much right to. +/// Each entry is the label the text form prints, paired with the field names the +/// machine-readable form could reasonably carry it under — it is free to name +/// things its own way, so the assertion is that *some* field carries the fact, +/// not that the two schemas match key for key. +/// +/// Deliberately not the full list of eleven: these are the ones a caller cannot +/// work around. Which GPU was found, which engine this host will serve on, +/// whether an existing ROCm install was detected, and what is provisioned. +const FACTS_A_TOOL_ALSO_NEEDS: &[(&str, &[&str])] = &[ + ( + "detected_gfx_target", + &["detected_gfx_target", "gfx_target"], + ), + ("effective_default_engine", &["effective_default_engine"]), + ("legacy_rocm_status", &["legacy_rocm_status"]), + ("managed_runtimes", &["managed_runtimes"]), + ("config_dir", &["config_dir"]), +]; + +/// Whether the human report printed a `