Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 35 additions & 2 deletions tests/e2e-cucumber/expectations.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,12 @@
# host capability probe (see src/capability.rs) and is NOT encoded here.
#
# Resolution at runtime (see tests/e2e-cucumber/src/expectation.rs):
# 1. A scenario whose @requires-gpu / @requires-engine can't be satisfied on
# this host is SKIPPED (not-applicable) — never listed here.
# 1. A scenario whose @requires-gpu / @requires-engine / @requires-bare-metal
# can't be satisfied on this host is SKIPPED (not-applicable) — never listed
# here. Reach for a tag, not a row, whenever the scenario simply has no
# premise on that host: a row here says "known BUG", so using one for
# designed-in behaviour (e.g. diagnose not running its bare-metal catalog on
# WSL2) tells every later reader the opposite of the truth.
# 2. Otherwise, the FIRST matching condition below marks it expected-fail
# (xfail); a scenario that then PASSES is an XPASS — a stale entry to be
# removed, UNLESS the row is `flaky = true`, which tolerates either result
Expand Down Expand Up @@ -165,3 +169,32 @@ bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the benchmark never reaches a live model."
serve_timeout_secs = 90
flaky = true

# --- EAI-7998: `examine --json` is not the equal of the human report. It
# answers before the CLI has loaded its paths or config, so eleven facts the
# text form states have no field in it at all. Platform-independent — the
# missing fields are missing everywhere. ---
[["examine-machine-readable-report"]]
when = {}
bug = "EAI-7998"
reason = "`examine --json` withholds CLI-side facts (engine, install status, managed runtimes, paths) that the human report states."

# --- EAI-7998: the same form also disagrees with the human report about GPU
# detection — but only on Instinct. Measured, not assumed: this row first ran
# scoped to `gfx*`, which xfailed correctly on MI300X (gfx943) and XPASSed on
# Strix Halo (gfx1151), where the two forms agree. Narrowed to the family where
# it actually reproduces; the mock lane and a WSL2 box correctly agree there is
# no GPU at all, so they stay expected-pass. ---
[["examine-both-forms-agree-on-gpu"]]
when = { therock_family = "gfx94*" }
bug = "EAI-7998"
reason = "On Instinct the JSON form reports has_amd_gpu:false while the human report names the gfx target."

# --- EAI-7193: the framework probe can be scoped in the library, but no call
# site passes anything but the default, so `--framework` does not exist on the
# command line and the choice is unreachable. Platform-independent (pure
# argument parsing). Remove this row when the flag lands. ---
[["examine-can-skip-framework-probing"]]
when = {}
bug = "EAI-7193"
reason = "`examine` has no --framework flag, so the framework probe cannot be scoped from the command line."
26 changes: 24 additions & 2 deletions tests/e2e-cucumber/features/diagnose.feature
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,13 @@ Feature: Diagnosing failures and listing fixes
# dependent. They assert the SHAPE of a diagnosis (a scored match with an id
# and a plan) and the query/refusal contracts.

@id:diagnose-matches-known-symptom
# @requires-bare-metal: these two need the catalog to actually produce a match.
# On WSL2 the catalog is deliberately not run at all — that platform uses
# /dev/dxg and the Windows host driver, so bare-metal Linux diagnoses would be
# false positives — which leaves these scenarios with no premise there. That is
# designed behaviour with its own unit test, not a bug, so they are skipped
# rather than xfail'd. `@requires-os:linux` would not do it: WSL2 is linux.
@id:diagnose-matches-known-symptom @requires-bare-metal
Scenario: 1 - Diagnosing a recognised failure reports a likely cause and a fix
Given a user who hit a known ROCm failure
When the user asks the CLI to diagnose that symptom
Expand All @@ -23,7 +29,7 @@ Feature: Diagnosing failures and listing fixes
When the user asks the CLI to diagnose that symptom in machine-readable form
Then the CLI always points to somewhere the problem can be reported

@id:diagnose-json-has-match-flag
@id:diagnose-json-has-match-flag @requires-bare-metal
Scenario: 3 - A diagnosis is available in machine-readable form for tooling
Given a user who hit a known ROCm failure
When the user asks the CLI to diagnose that symptom in machine-readable form
Expand Down Expand Up @@ -56,3 +62,19 @@ Feature: Diagnosing failures and listing fixes
Given a user who refers to a cause by its position in the diagnosis
When the user asks the CLI to apply that fix
Then the CLI refuses and explains that a position is not a fix-id

# The one gate standing between `rocm fix` and an edited machine, and until now
# it had no end-to-end coverage. The scenario gives the CLI a home directory it
# owns, so the file the fix would edit is one the scenario can read back: the
# refusal must not depend on what is in the runner's dotfiles, and a regression
# here must not be able to reach them.
# Linux-only because the assertion is "the file is untouched": on Windows the
# same recipe persists through `setx` into the user environment, which the
# suite cannot plant or read back safely. The gate itself is shared code, so
# this still guards it — just not the Windows persistence step.
@id:fix-requires-agreement-before-changing-anything @requires-os:linux
Scenario: 8 - A fix that changes the machine is not applied without agreement
Given a user who has chosen a fix that would change the machine
When the user asks the CLI to apply it without agreeing to the change
Then the CLI refuses and explains that it needs agreement
And the file the fix would have changed is untouched
46 changes: 46 additions & 0 deletions tests/e2e-cucumber/features/examine.feature
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,49 @@ Feature: GPU detection and system inspection
And the inspection names that install's version
And the inspection does not claim nothing is installed
And the inspection suggests setting up a CLI-managed install

# Expected to FAIL. The machine-readable form is a separate code path, not a
# re-rendering of the human one: it answers before the CLI has loaded its paths
# or config, so every CLI-side fact is out of reach. Eleven things the human
# report states have no field in it at all — among them which engine this host
# will serve on and whether an existing ROCm install was found. Tooling reads
# this form; it should not be the weaker of the two.
@id:examine-machine-readable-report
Scenario: 7 - What the inspection tells a tool matches what it tells a person
When the user inspects the system both for reading and for scripting
Then the machine-readable form states everything the readable one does

# Expected to FAIL on Instinct. The harness parses the human text rather than
# this form precisely because of this defect, and says so in capability.rs —
# on a real MI300X the machine-readable form reported no AMD GPU on a machine
# that has one. That workaround makes the disagreement load-bearing: every
# host capability the suite resolves comes from scraped text because this form
# could not be trusted.
#
# It is narrower than "any host with a GPU": Strix Halo (gfx1151) agrees,
# MI300X (gfx943) does not. The expectations row is scoped to match.
@id:examine-both-forms-agree-on-gpu
Scenario: 8 - Both forms of the inspection agree about the GPU
When the user inspects the system both for reading and for scripting
Then both reports agree on whether this machine has an AMD GPU
And both reports agree on whether this platform is in scope

# `examine` is an inspector: the outcome says whether it managed to look, not
# whether it liked what it saw. Finding no GPU is a finding, not a failure.
# This is the mock lane's to prove — it is the one lane with nothing to find.
@id:examine-reports-without-failing
Scenario: 9 - Inspecting a machine reports what it finds without failing
When the user inspects the system
Then the inspection completes successfully
And it states a verdict for this machine

# Expected to FAIL until the framework choice is reachable from the command
# line. The library can already scope the probe, but every caller passes the
# default, so the user cannot. Leaving the frameworks out is the variant worth
# pinning here: the outcome is identical on every host, whereas asserting that
# a *named* framework was probed would depend on what happens to be installed.
@id:examine-can-skip-framework-probing
Scenario: 10 - The user can leave the frameworks out of the inspection
When the user inspects the system without probing frameworks
Then the inspection reports that it skipped the frameworks
And it still states a verdict for this machine
101 changes: 99 additions & 2 deletions tests/e2e-cucumber/src/expectation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ const REQUIRES_ENGINE_PREFIX: &str = "requires-engine:";
const REQUIRES_OS_PREFIX: &str = "requires-os:";
const REQUIRES_GPU_TAG: &str = "requires-gpu";
const REQUIRES_NO_GPU_TAG: &str = "requires-no-gpu";
const REQUIRES_BARE_METAL_TAG: &str = "requires-bare-metal";
const SERVE_TIMEOUT_PREFIX: &str = "serve-timeout:";
const NIGHTLY_TAG: &str = "nightly";
const LIFECYCLE_TAG: &str = "lifecycle";
Expand Down Expand Up @@ -58,6 +59,18 @@ pub struct ScenarioDecl {
/// GPU — the inverse of `requires_gpu`. This is how the no-GPU fail-fast path
/// gets live coverage: it runs on the GitHub-hosted mock job.
pub requires_no_gpu: bool,
/// `@requires-bare-metal`: the scenario's premise is a host running the
/// in-tree amdgpu driver, so it does not hold under WSL2 — which uses
/// `/dev/dxg` and the Windows host driver instead. `rocm diagnose` skips its
/// bare-metal catalog there *by design* rather than emitting Linux diagnoses
/// that cannot apply, so a scenario needing a catalog match has no premise on
/// WSL2. This is a SKIP and not an xfail row: the behaviour is deliberate and
/// separately unit-tested, and recording it as a known bug would tell every
/// later reader the opposite.
///
/// `@requires-os:linux` cannot express this — WSL2 reports an os_family of
/// `linux`, so it matches.
pub requires_bare_metal: bool,
/// Engine the scenario pins via `@requires-engine:<e>` (if any).
pub requires_engine: Option<String>,
/// OS the scenario requires via `@requires-os:<os>` (e.g. "linux"), if any —
Expand Down Expand Up @@ -94,6 +107,7 @@ impl ScenarioDecl {
let mut id = None;
let mut requires_gpu = false;
let mut requires_no_gpu = false;
let mut requires_bare_metal = false;
let mut requires_engine = None;
let mut requires_os = None;
let mut serve_timeout_secs = None;
Expand All @@ -117,6 +131,8 @@ impl ScenarioDecl {
requires_gpu = true;
} else if tag == REQUIRES_NO_GPU_TAG {
requires_no_gpu = true;
} else if tag == REQUIRES_BARE_METAL_TAG {
requires_bare_metal = true;
} else if tag == NIGHTLY_TAG {
nightly = true;
} else if tag == LIFECYCLE_TAG {
Expand All @@ -129,6 +145,7 @@ impl ScenarioDecl {
id,
requires_gpu,
requires_no_gpu,
requires_bare_metal,
requires_engine,
requires_os,
serve_timeout_secs,
Expand Down Expand Up @@ -323,8 +340,9 @@ pub struct PlatformManifest<'a> {
///
/// 1. Not-applicable → `Skip`: a `@nightly` scenario when nightly isn't included,
/// a `@merge-queue` scenario outside the merge queue, a `@requires-gpu`
/// scenario on a host with no AMD GPU, a `@requires-os:<os>` scenario on a
/// different OS, or a scenario whose effective engine can't start.
/// scenario on a host with no AMD GPU, a `@requires-bare-metal` scenario on
/// WSL2, a `@requires-os:<os>` scenario on a different OS, or a scenario whose
/// effective engine can't start.
/// 2. First matching `expectations.toml` condition → `ExpectXfail`.
/// 3. Otherwise → `ExpectPass`.
///
Expand Down Expand Up @@ -370,6 +388,11 @@ pub fn resolve(
reason: "requires a host with no AMD GPU; this host has one".to_owned(),
};
}
if decl.requires_bare_metal && cap.is_wsl {
return Expectation::Skip {
reason: "requires a bare-metal host; this one is WSL2".to_owned(),
};
}
if let Some(os) = &decl.requires_os
&& !os.eq_ignore_ascii_case(&cap.os_family)
{
Expand Down Expand Up @@ -474,6 +497,19 @@ mod tests {
effective_serve_engine: "lemonade".into(),
platform_slug: "strix-halo".into(),
},
// A WSL2 dev box: `os_family` is `linux` and a GPU is visible through
// /dev/dxg, so nothing but `is_wsl` distinguishes it from bare metal.
// That is precisely why `@requires-os:linux` cannot stand in for
// `@requires-bare-metal`.
"wsl2" => HostCapability {
os_family: "linux".into(),
is_wsl: true,
gfx_target: Some("gfx1151".into()),
has_amd_gpu: true,
available_engines: vec!["lemonade".into(), "vllm".into()],
effective_serve_engine: "lemonade".into(),
platform_slug: "wsl2".into(),
},
_ => HostCapability {
os_family: "other".into(),
is_wsl: false,
Expand Down Expand Up @@ -528,6 +564,14 @@ serve_timeout_secs = 90
assert!(d.lifecycle);
}

#[test]
fn bare_metal_tag_parses_in_both_shapes() {
assert!(decl(&["id:x", "requires-bare-metal"]).requires_bare_metal);
assert!(decl(&["@id:x", "@requires-bare-metal"]).requires_bare_metal);
// Absent by default, so no existing scenario changes meaning.
assert!(!decl(&["id:x", "requires-gpu"]).requires_bare_metal);
}

#[test]
fn serve_timeout_tag_parses_seconds() {
let d = decl(&["id:serve-large-model-inference", "serve-timeout:2400"]);
Expand Down Expand Up @@ -683,6 +727,59 @@ serve_timeout_secs = 90
));
}

#[test]
fn requires_bare_metal_skips_on_wsl_and_runs_everywhere_else() {
let m = Expectations::default();
let d = decl(&["id:diagnose-matches-known-symptom", "requires-bare-metal"]);
assert!(matches!(
resolve(&d, &cap("wsl2"), &m, false, false, false),
Expectation::Skip { .. }
));
// Bare-metal hosts of every shape still run it — including the mock lane,
// which is where these scenarios earn their keep as a required check.
for host in ["mock", "mi300x", "strix-ubuntu", "strix-windows"] {
assert_eq!(
resolve(&d, &cap(host), &m, false, false, false),
Expectation::ExpectPass,
"{host} is bare metal and must still run the scenario"
);
}
}

#[test]
fn requires_os_linux_does_not_stand_in_for_requires_bare_metal() {
// The reason the tag has to exist: WSL2 reports os_family "linux", so an
// `@requires-os:linux` scenario runs there. If this ever starts skipping,
// the two tags have collapsed into one and the new one is redundant.
let m = Expectations::default();
let d = decl(&["id:some-linux-scenario", "requires-os:linux"]);
assert_eq!(
resolve(&d, &cap("wsl2"), &m, false, false, false),
Expectation::ExpectPass
);
}

#[test]
fn bare_metal_skip_is_not_recorded_as_a_known_bug() {
// WSL2 out-of-scope is deliberate product behaviour, so it must resolve to
// Skip even when the scenario also carries an xfail row — otherwise the
// report would call a designed-in limitation a defect.
let m = Expectations::parse(
r#"
[["diagnose-matches-known-symptom"]]
when = {}
bug = "EAI-0000"
reason = "unrelated open bug"
"#,
)
.unwrap();
let d = decl(&["id:diagnose-matches-known-symptom", "requires-bare-metal"]);
assert!(matches!(
resolve(&d, &cap("wsl2"), &m, false, false, false),
Expectation::Skip { .. }
));
}

#[test]
fn vllm_pinned_scenario_skips_where_vllm_cannot_start() {
let m = Expectations::default();
Expand Down
Loading