Skip to content

fix(diagnose): correct fix-9's auto flag and test the rocm-doctor contract - #149

Merged
juhovainio merged 16 commits into
mainfrom
test/rocm-doctor-skill-contract
Oct 2, 2026
Merged

juhovainio merged 16 commits into
mainfrom
test/rocm-doctor-skill-contract

Conversation

@volen-silo

@volen-silo volen-silo commented Jul 29, 2026 •

Copy link
Copy Markdown
Collaborator

What

skills/rocm-doctor/ is a published Agent Skill, and this repo is its source
of truth. The skill is a thin driver: the probe, the closed 15-mode catalog and
the fixes all ship inside the rocm binary, so it is versioned with the binary
and lives beside it. The amd/skills catalog federates skills from the repos
that own them, which is how the published copy stays in step.

What the skill owns is a contract: which fix-ids exist, which the CLI applies
itself, which machines each is for, what a diagnosis carries, and where a report
goes when nothing matched. Nothing tested that seam. diagnose.feature
deliberately asserts only the shape of a diagnosis so it stays
host-independent, so a renamed fix-id or a 16th failure mode would merge green
and silently break the published skill.

Changes

Contract feature — rocm_doctor_skill.feature (6 scenarios) parses
reference.md's catalog table, its auto-applicable prose, the fields and
thresholds it names, the verdicts it enumerates and the trackers it lists, then
diffs each against what the CLI reports. Query-only: no GPU, no serve, no
download, no mutation, so it runs on the blocking mock lane with no capability
tags. Reading a document as test data is a deliberate, narrow exception to the
suite's black-box rule — nothing is imported from the codebase, and the document
is the thing under test.

rocm-core unit tests — exit codes 1, 4 and 5 were asserted nowhere and
no test reached a runner that writes to disk. The two that looked like coverage
did not: the dry-run test picks fix-2, whose Linux runner takes no
FixOptions and never prompts, and the e2e preview scenario picks print-only
fix-1, which returns before any runner.

One CLI correction — route_when_no_match matched on lemonade / ollama /
lm-studio, which Examination::probe can never report, so the CLI advertised
routing it cannot do.

This originally read "two corrections". The second was fix-9's
auto_applicable flag, which check_9_igpu_dgpu_collision reported as false
on Linux while fix::RECIPES said true. main landed the same correction
independently while this branch was in review, so rebasing dropped it from this
diff: auto_applicable is unchanged context here, and only the comment above it
differs. What this branch still adds for it is the regression test that keeps
the two sides from drifting apart again.

Skill corrections — the skill gated on "is matched empty?", which
DiagnoseReport::has_match exists to prevent; it promised that every
auto-applicable fix honours --dry-run and confirms before mutating, which is
false for fix-2-unset-override on Linux (its runner reports and stops); the
installer was labelled "Linux / macOS" when install.sh is Linux x86_64 only;
and the nightly-only install claim was stale.

Licence config — skills/rocm-doctor/** is excluded: SKILL.md must open
with YAML frontmatter for the skill loader, which pushes any header out of
hawkeye's detection window, and catalog skills carry no headers.

Scope

Claims proven elsewhere are not restated. Two earlier scenarios were dropped for
that reason. skills/** joins the heavy paths filter, since reference.md is
now a fixture.

Verification

  • cargo xtask e2e — 81 scenarios, 79 pass; the 2 failures are the expected
    WSL2 diagnose xfails (0 unexpected)
  • cargo test --workspace --all-targets --exclude e2e-cucumber — clean
  • cargo clippy --locked --workspace --all-targets and
    --locked -p e2e-cucumber --test e2e — clean
  • hawkeye check and all prek hooks — clean, and verified positively:
    stripping README.md's or rocm-cli-assistant's header still exits 1, so the
    new exclusion is not swallowing the check
  • The routing assertion verified to fail when it should, not just to pass:
    restoring "lemonade" => "lemonade" makes it fail, then reverted

Known coverage gap

The fix-9 auto_applicable flag is user-observable — render_report_text
adds a flags: line when it is set, and Fix derives Serialize, so the JSON
carries it too. The correction itself now comes from main (see above), but the
regression test this branch adds for it still has no Gherkin scenario behind it,
which AGENTS.md asks for. This is a gap I could not close rather than one I
overlooked:
check_9_igpu_dgpu_collision only fires on an Examination carrying an APU
and a discrete AMD GPU, the suite drives the real binary against the real
host, no --from-examination-style input exists to hand it a synthetic one, and
no CI lane has both. It is covered by a rocm-core regression test that runs
diagnose for both OS families and asserts every emitted fix agrees with the
catalog.

Not verified locally: scenario 6 rejecting a matching symptom. This host is
WSL2, where the catalog is out of scope and never runs, so has_match is false
either way. That case bites on the no-GPU Linux lane.

Unrelated pre-existing flake seen while verifying: dash-gen-tps-* assert
against a wall-clock validity window and fail intermittently on full runs — on
main as well as here, and never in isolation. Not touched by this PR.

@volen-silo
volen-silo requested a review from a team as a code owner July 29, 2026 10:34
@volen-silo
volen-silo marked this pull request as draft July 29, 2026 11:21

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 987dd9a2. Checked out locally: fix::tests 18/18 pass, all 22 CI checks green, and I re-verified the description's load-bearing claims (exit codes 1/4/5 were indeed unasserted before; the 15-mode catalog, OS scopes and auto-flags in reference.md match RECIPES exactly).

The refactor is the good part — splitting consent_without_prompt out of confirm and taking the rc path plus an injected consent verdict in pin_device_in_rc_file makes the mutation path genuinely testable without touching $HOME or stdin, and it's behaviour-preserving. The comments explaining why the two tests that looked like coverage weren't are worth keeping.

Three things I'd want resolved before this merges, then some smaller ones.

Dependent on an unmerged upstream PR, but not draft

The premise is that skills/rocm-doctor/ is byte-verbatim with amd/skills#87 — but #87 is still open and can still change in review. Merging first can land a mirror that never matches what upstream finally accepts, and the drift job fails open so nothing will say so. Suggest draft until #87 lands, then mirror the merged files.

That also sequences the next point for free.

The mirror ships documentation that's already known to be wrong

You flag it in the PR body, so this is agreement rather than a new find — but it interacts badly with the framing. The point of the change is that the documented contract becomes executable and can't silently break; a reader will reasonably assume the whole file is under test. It isn't: the new scenarios cover the catalog, and SKILL.md's install instructions are untested and currently false (install.sh:310-319 is Linux x86_64 only and hard-fails otherwise, against "Linux / macOS"; the nightly-only claim is likewise stale).

Fixing it upstream is the right layer. Landing #87 with the correction first and mirroring the corrected file avoids importing a known-wrong doc.

Scenario 2's pinned set duplicates an existing unit test, and its rationale doesn't hold

skill_steps.rs:264-266 justifies the hard-coded AUTO_APPLICABLE with "both files name these four ids in prose". SKILL.md contains no fix-ids at all — grep 'fix-[0-9]' skills/rocm-doctor/SKILL.md is empty. Only reference.md names them (line 43), and the assertion doesn't read that prose either: it compares the CLI to a Rust array, which is exactly what exactly_the_four_known_fixes_are_auto (fix.rs:1018) already asserts, same ids, same order.

So it's a duplicate by the PR's own "claims already proven elsewhere are not restated" standard. Either drop it, or make it do the thing the comment describes and parse the prose line out of reference.md.

Smaller

  • world.model_name is reused to carry a symptom string (skill_steps.rs:184) and a fix-id (:196). The PR already adds a dedicated skill_reference field; these deserve the same rather than borrowing an unrelated name.
  • pinning_without_a_device_index_is_not_applicable returns early on non-Linux, so it's a vacuous green on the Windows lane. #[cfg(target_os = "linux")] states that instead of passing silently.
  • Both parsers panic! on an unrecognised row (skill_steps.rs:96, :123). Loud failure is defensible, but it means a cosmetic change to rocm fix output surfaces as a panic rather than an assertion diff naming the two sets.
  • Scratch-dir cleanup is remove_dir_all(...).ok() after the assertions, so a failing test leaves debris under .rocm-work/. Minor, and the no-tempfile convention is a fair reason.
  • !skills/rocm-cli-assistant/*.md is non-recursive, and a new top-level directory with tracked markdown goes unchecked silently. AGENTS.md documents the hazard, which is the right mitigation — worth being sure that note is the first thing anyone editing licenserc.toml sees.
  • The drift job's if: github.event_name == 'pull_request' skips the merge queue. Harmless while it's advisory, but it means the last state before merge is never compared.

On the licence change specifically: I checked it positively rather than trusting the green check, since a passing hawkeye check only proves nothing is failing. Stripping README.md's header makes hawkeye flag it, so !/*.md anchoring works and the mirror is genuinely exempt — and no currently-tracked markdown lost coverage in the blanket-to-enumeration swap.

@volen-silo
volen-silo force-pushed the test/rocm-doctor-skill-contract branch from 529f774 to 0564a0e Compare August 19, 2026 10:04
@volen-silo
volen-silo marked this pull request as ready for review August 19, 2026 11:03
rominf
rominf previously requested changes Sep 2, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this at 04fe5aa5. The idea is good — making a published skill's documented contract executable, rather than letting it rot silently, is exactly the right instinct, and the doc-as-fixture parsing in skill_steps.rs is nicely done. The ten new rocm-core unit tests are real coverage, and I confirmed the scenarios actually run (plain untagged, skill_steps.rs is registered in e2e.rs).

But the branch is 38 commits behind main and that has caused real damage, plus the mirror premise no longer holds. My recommendation is: rebase, re-mirror all three files from upstream, and fix upstream first for the three places the doc is now wrong — then this is close.

The blocking set:

  1. licenserc.toml reverts main's HawkEye v7 migration (#300). It's the only merge conflict, and after rebase the exemption mechanism it edits no longer exists, so the mirrored .md files would be header-required and the blocking check would fail.
  2. The drift job fetches a path that 404s. The one check whose job is catching mirror rot can never fire — and it fails open, so nobody will notice.
  3. The mirror is not byte-verbatim any more. Upstream's SKILL.md has gained a Scope gate section (a substantive safety instruction: decline non-AMD/WSL2 setups and give no troubleshooting) that this copy doesn't have.
  4. A new unit test asserts exit 3 where main deliberately changed the branch to return 0 — and fix.rs auto-merges without conflict, so the merged tree gets two tests asserting different values for the same call. Guaranteed red after rebase, with no conflict marker to warn you.

Then five smaller ones about the tests not checking what they claim, and two nits.

Also worth flagging on the PR body: amd/skills#87 merged on 2026-08-13, so "Depends on amd/skills#87, which is still open… the upstream files 404" and the draft rationale are both stale — and the PR is not in draft. The dependency is discharged; the body should say so.

Sequencing thought: #250 renames rocm diagnose → rocm doctor and changes render_report_text's signature. This PR promotes the pre-consolidation surface to a tested, published contract — and the new feature only checks CLI-against-doc, never doc-against-CLI-surface, so it won't flag the mismatch. I'd land #250 first, then rebase this with the mirror re-fetched after upstream is corrected.

CI: E2E tests (Strix Halo, Windows) is red, but the branch is 38 commits stale so that result doesn't mean much until rebase.

Method note: parts of this were read by parallel helper passes that ran degraded (sequential, no fan-out). Everything above I verified myself — including diffing the mirror against upstream, running git merge-tree, and querying the upstream repo's contents API.

Comment thread licenserc.toml Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread skills/rocm-doctor/SKILL.md Resolved
Comment thread crates/rocm-core/src/fix.rs Resolved
Comment thread skills/rocm-doctor/SKILL.md Resolved
Comment thread skills/rocm-doctor/reference.md Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
@volen-silo
volen-silo force-pushed the test/rocm-doctor-skill-contract branch from 04fe5aa to 9eb0eab Compare September 8, 2026 10:30
@volen-silo
volen-silo marked this pull request as draft September 8, 2026 10:31
@volen-silo volen-silo changed the title test(e2e): mirror the rocm-doctor skill and test its CLI contract test(e2e): publish the rocm-doctor skill and test its CLI contract Sep 8, 2026

@rominf rominf left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 9eb0eabc.

Went through all 9 threads from my previous pass and checked each fix against the actual current code rather than taking the replies at face value — all genuinely hold:

  • licenserc.toml is back on the v7 schema with skills/rocm-doctor/** excluded, and the resurrected v6 file is gone.
  • The dead amd/skills drift-check job is deleted outright (the repo is now the source of truth, so there's nothing to drift-check).
  • SKILL.md/reference.md are re-synced: the Scope gate, Prerequisites, and the has_match gating are all present, in both directions (the reverse field-coverage assertion actually catches a doc omission when I mentally removed has_match from it).
  • The conflicting device-index test in fix.rs is dropped in favor of main's, and reference.md's exit-3 row is corrected to "negative only."
  • The dead Lemonade/Ollama/LM Studio routing arms are gone from diagnose.rs, and the routing table now lives in the skill's own hand-off, checked against the CLI's real reachable set.
  • The diagnose.rs comment is honest now about the framework list being hand-maintained rather than promising a guarantee the test doesn't provide.
  • skill_steps.rs parses the "auto-applicable" prose out of reference.md instead of pinning a redundant array, and scenario 6's has_match assertion is live (verified via the WSL no-catalog path) with a dedicated skill_symptom field instead of overloading model_name.

I resolved all 9 threads.

One new thing turned up on a fresh pass over the current diff: the new rocm_doctor_skill.feature has no entry in FEATURE_KEYS (tests/e2e-cucumber/tests/feature_naming.rs), so cargo test -p e2e-cucumber --test feature_naming fails — reproduced locally, feature_files_and_declared_keys_agree panics with "rocm_doctor_skill.feature has no key in FEATURE_KEYS — add one so its scenarios are indexed and its ids are qualified like every other feature." That's what's turning Test (affected crates), windows-build-and-test, and E2E tests red right now. Left as an inline comment below.

Everything else checked out: cargo check/cargo build on the touched crates, the full rocm-core unit suite (48/48), clippy with -D warnings, and the real rocm_doctor_skill.feature run against a built rocm binary (all 6 scenarios pass) — which also confirms the fix-9 auto_applicable correction is a genuine, previously-unnoticed bug fix backed by a real regression test, not just a doc tweak.

rominf
rominf previously requested changes Sep 8, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marking the verdict properly — my review just above should have been request-changes rather than a plain comment (my error, not a change of opinion). Nothing new here.

The blocker is the one inline comment on tests/e2e-cucumber/features/rocm_doctor_skill.feature: the file has no FEATURE_KEYS entry, so feature_files_and_declared_keys_agree fails and Test (affected crates), windows-build-and-test and E2E tests are red. Everything from the previous round is resolved and verified.

rominf
rominf previously requested changes Sep 8, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at e6fe813d. Both things I flagged last round are fixed, and I owe you a correction on one of them.

Correction: I attributed all three red checks to the missing FEATURE_KEYS entry. That was wrong. Only Test (affected crates) and windows-build-and-test came from that. E2E tests failed for the separate reason you found — scenario 6's has_match == false assertion. And that assertion was there because I asked for it in my previous review, on the reasoning that out_of_scope was the only host-dependent branch. It isn't: the catalog scores host state, so fix-5-amdgpu-load reaches 90 on the hosted Linux runner whatever symptom it is handed. Your diagnosis is right and mine was wrong. Sorry for the round trip.

Confirmed fixed at this head:

  • rocm_doctor_skill.feature is in FEATURE_KEYS and the six scenarios are indexed skill-01..skill-06, so the three sibling naming tests now actually run against it instead of skipping it vacuously.
  • The has_match claim moved into rocm-core against a constructed Examination. sub_threshold_causes_leave_has_match_false_and_route_upstream plus the companion that flips the flag is the right shape — it pins the thing you actually mean without depending on what the runner happens to look like.

All checks are green on e6fe813d as of now, including the E2E tests and windows-build-and-test lanes.

Three things still open, inline below. The first matters most: the skill doc makes a promise the CLI does not keep on Linux, and the whole point of this PR is that the doc and the CLI agree.

Comment thread skills/rocm-doctor/reference.md Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
@volen-silo
volen-silo requested a review from rominf September 8, 2026 13:04
rominf
rominf previously requested changes Sep 10, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything from my last two rounds holds — I re-verified each against the code rather than the replies. licenserc is back on the v7 schema, has_match is documented and gated with a real two-way field equality rather than an allow-list, the exit-3 row is right, the unreachable routing arms really are unreachable (I traced every producer of Examination.framework), the auto set is parsed from prose, skill_symptom is a dedicated field that something actually consumes, and FEATURE_KEYS is wired. The report_persistent_override and pin_device_in_rc_file extractions are behaviour-preserving, and the fix-2 dotfile byte-identity test is the strongest thing in the PR.

The blocker is that the premise the whole design rests on isn't true yet, and I only found it by going and looking at the catalog rather than reading the PR text. Details inline, but in short: amd/skills does not federate this repo, rocm-doctor sits in its staging/ directory which federation never touches, and the two copies have already diverged — while the drift-check that would have caught that was deleted last round on the grounds that federation replaces it.

Second, fix-9 repeats the fix-2 overpromise we corrected in the last round, in the same PR that promoted it.

The rest are smaller: several contract restatements that the parsers never read (including SKILL.md, which no step opens at all), a workflow filter that no longer matches the invariant it claims, the commit type, and two comment-accuracy points.

Nothing here is about the quality of the test scaffolding, which is good. It's about the guarantees being claimed for it being wider than what it checks.

Comment thread AGENTS.md Outdated
`rocm` binary, so it is versioned with the binary and lives here. The
[`amd/skills`](https://github.com/amd/skills) catalog federates it: a nightly
job vendors the folder from `main` and opens a pull request there. So edit it
here, and never edit the catalog's copy — the next federation run overwrites

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This instruction points at a sync that doesn't happen, and I'd treat it as blocking because of what was removed on the strength of it.

I checked the catalog directly rather than going from the PR text. amd/skills/.github/federation.json declares exactly one source — AMD-AGI/TraceLens, skill tracelens-analysis-orchestrator. There is no ROCm/rocm-cli entry. And rocm-doctor isn't under amd/skills/skills/ at all; it's in amd/skills/staging/rocm-doctor/, which federate-skills.yml never touches, since it vendors only the sources declared in federation.json. The two copies have already drifted apart.

So "never edit the catalog's copy — the next federation run overwrites it" tells a future agent to rely on something that won't run. That matters more than a stale sentence: last round the amd/skills drift-check job was deleted outright, on the reasoning that both copies stop mattering once federation replaces the mirror. Federation doesn't currently replace it, so this PR removes the only detector and substitutes a mechanism that isn't wired up. Right now nothing keeps the published copy honest.

Two ways out, either fine by me: land the federation.json entry and promote staging/rocm-doctor first, and reference that from here; or say plainly that the catalog copy is hand-synced from staging today, name who syncs it, and keep a drift detector until federation is live. The same claim appears in licenserc.toml, the e2e README, and the feature header, so they'd all move together.

Separately, line 191 says federation doesn't carry the skill's evals/ folder — but skills/rocm-doctor/ in this PR has no evals/ at all. That's phantom detail in a file loaded on every task; either drop the sentence or land the folder.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and took the second path rather than landing a federation.json entry from this PR. AGENTS.md's federation bullet now says plainly that amd/skills is still Phase-1 incubation for this skill — no automated sync, federation.json only declares TraceLens, and rocm-doctor sits in staging/rocm-doctor outside any job's coverage — and that the rocm-cli team hand-syncs staging/rocm-doctor from here whenever it changes materially, rather than describing a job that doesn't run. Also dropped the phantom evals/ sentence — this skill has no evals/ folder. I did not restore a drift-check job or touch amd/skills from this PR; left open for you to re-verify the wording matches what you found live.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I did not touch, flagging rather than silently leaving inconsistent: licenserc.toml's comment still says the folder is "vendored verbatim into the amd/skills catalog" — same stale claim, out of this PR's assigned scope, so it's still there for a follow-up.

Comment thread crates/rocm-core/src/diagnose.rs Resolved
## Closed catalog (15 failure modes)

| id | OS | Failure mode | Typical signal | Auto-fix |
| --- | --- | --- | --- | --- |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is a second statement of the OS scope, and nothing reads it.

assert_same_os_scope compares the CLI against parse_reference_catalog, which reads the table's OS column only. So re-scoping a fix in the CLI and updating the table leaves this summary line stale and every scenario still green — while the feature header promises that "a re-scoped OS … cannot merge here and silently break the published skill".

This is the same double-statement problem you fixed for the auto set with documented_auto_prose; it just wasn't applied to the OS claim. Parsing this line the same way would close it.

Two more in the same family, since they share a root cause. The counts are unparsed — ## Closed catalog (15 failure modes) above, and SKILL.md's "The other 11 are print-only" — so adding a sixteenth recipe plus a table row leaves all six scenarios green with both numbers stale, which contradicts the header's "or a 16th failure mode" clause. And SKILL.md is never opened by any step: reference_md_path() is the only path constructed, yet the Given is worded "the skill as it is published" and the e2e README says this feature is the only thing catching the document drifting. SKILL.md independently restates the auto four, the routing table and the per-fix field list, none of it compared to anything.

Either extend the parsers to cover these, or narrow the header and README claims to what they actually check. I'd rather have a smaller promise than a wider one that isn't kept.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took the narrowing path rather than extending the parsers, on all three points: the feature header now says explicitly that the guarantee covers the table's cells and the auto-applicable prose line — what skill_steps.rs actually parses — and that the OS-scope summary line, the "(N failure modes)" heading count, and SKILL.md's "the other M are print-only" line are free-standing prose that restates those facts and is not parsed, so it can go stale even while the table stays correct. Same caveat added to tests/e2e-cucumber/README.md's description of this feature. Did not extend the parsers to cover SKILL.md, and did not change what skill_steps.rs checks — left open for you to confirm the narrowed claim matches what's actually verified.

Comment thread .github/workflows/ci.yml Resolved
Comment thread crates/rocm-core/src/diagnose.rs Resolved
Comment thread AGENTS.md Resolved
@volen-silo volen-silo changed the title test(e2e): publish the rocm-doctor skill and test its CLI contract fix(diagnose): correct fix-9's auto flag and test the rocm-doctor contract Sep 11, 2026
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Pushed a round addressing this review pass:

  • fix-9's "one exception" doc gap: extended both reference.md and SKILL.md to name fix-9 alongside fix-2, and surfaced the --device-index N requirement in check_9_igpu_dgpu_collision's own diagnosis notes so a JSON consumer sees it too, not just the docs.
  • Reworded the dead None arm in skill_steps.rs's assert_route_is_documented into a loud panic, since documented_cli_routes only ever returns Some in practice.
  • Added 'skills/**' to e2e-selfhosted.yml's heavy filter so it matches ci.yml.
  • Reworded AGENTS.md so "catalog" isn't overloaded between the external amd/skills catalog and the failure-mode catalog, and attributed the failure catalog's checkers/OS-scoping to diagnose.rs alongside fix.rs.
  • Rewrote the federation description to match what's actually true today: no automated sync for rocm-doctor, staging/rocm-doctor is outside any job's coverage, and the rocm-cli team hand-syncs it during Phase-1 incubation. Left that thread open for you to re-verify the wording.
  • Narrowed the feature header and README claims to what skill_steps.rs actually parses (the catalog table and auto-applicable prose), and called out that the OS-scope summary line and the failure-mode/print-only counts are unparsed prose that can drift. Left that thread open too, since it's the same kind of judgment call.
  • Retitled the PR so the squash-merge commit type is fix, matching the real CLI behavior change (fix-9's auto flag, the removed routing arms), rather than test.

Also confirmed the four already-fixed threads from the previous round still hold (feature naming, the fix-2 exception paragraph, the duplicated doc-comment line, and the fix-9 flag-flip coverage gap, which is disclosed in the PR body under "Known coverage gap").

Verification: cargo test --workspace --all-targets --exclude e2e-cucumber clean, cargo clippy --workspace --all-targets --exclude e2e-cucumber -- -D warnings and the e2e-cucumber crate's clippy both clean, cargo fmt --all --check clean, prek hooks clean (license-headers hook skipped locally — hawkeye isn't installed in this environment, it normally runs in CI's dedicated job). Ran the full rocm_doctor_skill.feature (6/6 scenarios) and the broader diagnose.feature — all pass except the 2 pre-existing WSL2-environment diagnose xfails, unrelated to this round's changes.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 9ef3777

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Publishes the rocm-doctor skill from this repo and makes its reference.md a live fixture: six e2e scenarios parse the published document and diff it against what rocm fix / rocm diagnose / rocm examine actually report, plus two CLI corrections and unit coverage for the fix consent/exit-code contract. Needs work — the contract testing is genuinely good, but one graded dataset encodes the behaviour the skill now forbids, and one of the two CLI corrections has no test that fails on its revert. Verified: read all eleven commits and every changed file; confirmed the catalog table's 24 rows, the four auto-applicable ids, the OS recap line, the exit-code table and the routing targets all match RECIPES and diagnose.rs; confirmed list_recipes() emits every recipe regardless of host OS and DiagnoseReport serialises exactly the six documented keys (so scenarios 1–4 are real, host-independent assertions that do fail on catalog drift — not vacuous); confirmed skills/** reaches the heavy filter that gates the e2e job in ci.yml, and that every new action pin is a full SHA with a version comment. No build, test or suite run was performed here, and no mutation or revert experiment was run — the revert conclusions below are read directly off the test bodies, which decide them without execution. Checks at review time: 23 success, 1 failure, 4 pending. Blocking: 2 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

skills/rocm-doctor/evals/evals.json (the rocm-under-wsl2 entry) — the graded dataset asserts the exact opposite of the skill it grades.
This range makes WSL2 a first-class platform family: reference.md:77-115 gives it seven fix-wsl-* entries, reference.md:146 says "WSL2 used to sit here and no longer does", and SKILL.md:47-52 is emphatic — "WSL2 is in scope and does not stop this gate… Do not decline a WSL2 user or route them away untouched — run the workflow as for any other platform", repeated at SKILL.md:195-196 and SKILL.md:200-202. The eval left behind says the reverse: its note calls WSL2 "a separate platform the scope gate declines rather than troubleshoots", its expected_behavior is "State plainly that WSL2 is out of scope for this skill and why" and "Give no troubleshooting for it", and its unexpected_behavior is "Run rocm examine, rocm diagnose, or rocm fix" and "Offer any WSL2 troubleshooting steps". A skill that follows SKILL.md correctly fails this case, and a skill that passes it violates SKILL.md. This is the "pins a behaviour that is itself wrong" class: the dataset would certify the defect. The head commit compounds it by claiming the dataset is "written and checked to make switching [routing/behavioral] on a workflow change and nothing more" — switching them on today would grade conformant behaviour as wrong. The same commit says "four prompts the skill must answer and two near misses"; the file has five positives, because this entry moved from skill_should_trigger: false to true and only the flag was updated, not the body. Fix: rewrite this entry's note and expected/unexpected behaviours to the in-scope workflow (diagnose normally, WSL-scoped entries only), and correct the count in the commit message.

crates/rocm-core/src/diagnose.rs:2894 — routing_targets_cover_every_framework_the_probe_reports passes unchanged if the routing deletion it is presented as pinning is reverted.
Commit 5232431 removes the unreachable lemonade / ollama / lm-studio arms from route_when_no_match (and those plus amdgpu-install from upstream_tracker) and states "Both are pinned: … another pins the reachable target set." The test iterates only ["skipped", "pytorch", "llama-cpp", "unknown"], so restoring "lemonade" => "lemonade" to the match changes nothing it observes and the assertion still yields ["llama-cpp", "pytorch", "rocm-core"]. Scenario 6 does not cover the gap either: it asserts only that whatever target comes back is documented, and reference.md:132-134 still lists those three apps in its own routing table. So the regression the change exists to prevent — the CLI advertising routing the probe can never reach — has no failing test. Fix is one line: add "lemonade", "ollama", "lm-studio" to the loop array. With the arms absent they route to rocm-core and the existing assert_eq! is unchanged; if an arm is re-added the target set grows and the test fails loudly with the message it already carries.

Non-blocking

  • skills/rocm-doctor/SKILL.md:3-6 and skills/rocm-doctor/skill-card.md — the frontmatter description ("on Linux or Windows") and the card's Description ("Linux and Windows") both omit WSL2, so the surface a runtime routes on excludes the platform the body insists is in scope.
  • crates/rocm-core/src/diagnose.rs:2932 — every_diagnosis_agrees_with_the_fix_catalog_on_auto_applicability guards only on !report.matched.is_empty(); narrowing check_9_igpu_dgpu_collision's trigger would leave it green while covering nothing, so assert fix-9-igpu-dgpu is among the matched for each OS.
  • .github/workflows/e2e-selfhosted.yml:105 — skills/** is added to the heavy filter, but no job in that workflow reads needs.changes.outputs.heavy (every lane gates on serve), so the added comment's claim that a skill-only edit "must therefore run this job too" does not hold there; the fixture check does run, via ci.yml.
  • skills/rocm-doctor/reference.md:21 — "Always exits 0" for rocm diagnose is not quite true: a failed --distro probe propagates an error and exits 1, and that flag is also missing from the documented invocation just above.
  • tests/e2e-cucumber/features/rocm_doctor_skill.feature:11 says "a 16th failure mode" and SKILL.md:180 omits skipped from the probe's framework values that reference.md:130 lists — both stale against the 24-entry catalog and the sibling document (the PR body's "closed 15-mode catalog" is stale the same way).

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Went through all seven findings against the code rather than taking the labels at face value. Every one of them is real. Fixed the ones that earned it in dcd30f81.

The WSL2 items are one defect with three surfaces, not two findings. Bringing WSL2 into the catalog updated the skill's body but not the parts that decide whether the body is ever read, nor the dataset grading it:

  • SKILL.md frontmatter and skill-card.md both said "Linux and Windows". That is the surface a runtime routes on, so a WSL2 user could fail to load the skill at all and never reach the five places the body insists WSL2 is in scope. Both now say WSL2. Left the symptom list alone: the description is 962 characters against the conventional 1024 cap, and "WSL2" in the platform clause is the routing token that matters.
  • The rocm-under-wsl2 eval is rewritten to grade the in-scope workflow. Your read of the cause was exact: skill_should_trigger had been flipped to true without the body following. The flag was right, the expected and unexpected behaviours were left over from when the platform was declined. It now also checks that the CLI's own scoping holds, since a bare-metal Linux remediation reaching a WSL2 host means the agent went around it.

I would rank the frontmatter one above the eval rather than below it. The eval grades nothing today, because only structural runs and routing/behavioral need an API key this repo does not have. The frontmatter has live routing effect right now.

The routing test finding was correct and the suggested fix was the right one. I confirmed it by experiment rather than by reading: restored "lemonade" => "lemonade" to route_when_no_match and the test now fails with the message it already carried, left: ["lemonade", "llama-cpp", "pytorch", "rocm-core"], then reverted. Before the change it passed with the arm restored, exactly as you said.

On severity. Neither of the two you marked blocking actually blocks a merge, for the reasons above. They were worth fixing because they are cheap and true, not because of the label. Flagging that because the distinction matters for anyone triaging from the headline.

Deferred, all verified as real: the fix-9-igpu-dgpu assertion guard, the dead heavy filter comment in e2e-selfhosted.yml, reference.md's "always exits 0" against --distro, and the stale "16th failure mode" plus the missing skipped value. Not doing the commit-message count correction: that commit now sits inside a merge commit, so fixing the wording means rewriting an eleven-commit branch for a number in prose.

Unrelated to this PR, but worth someone's attention: the Strix Halo WSL2 lane is red here and was red on the previous commit too. It 404s fetching llama-b9752-bin-ubuntu-rocm-7.14-x64.tar.gz. The b9752 tag is live but only ships a rocm-7.13 build, so the backend resolver is asking upstream for something never published. That will redden the lane on every PR until the pin moves.

@volen-silo
volen-silo dismissed stale reviews from rominf, rominf, rominf, and rominf September 25, 2026 09:34

Addressed

@volen-silo
volen-silo added this pull request to the merge queue Oct 1, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 1, 2026
volen-silo and others added 16 commits October 2, 2026 06:04
Exit codes 1, 4 and 5 were asserted nowhere, and no test reached a runner
that writes to disk. The two that looked like coverage did not: the dry-run
test picks fix-2, whose Linux runner takes no FixOptions and never prompts,
and the e2e preview scenario picks print-only fix-1, which returns before
any runner runs.

Split consent_without_prompt out of confirm, and let pin_device_in_rc_file
take the rc path plus an injected consent verdict, so the mutation path is
reachable from a test without touching $HOME or stdin. Behaviour is
unchanged. Ten tests then cover the consent gate, both mutation outcomes,
and the dry-run short-circuit.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
check_9_igpu_dgpu_collision reported auto_applicable: false on Linux while
fix::RECIPES -- what `rocm fix` dispatches on -- says true. It was the only
OS-split checker whose branches diverged, and an agent branches on that
field, so it was told fix-9 was print-only while `rocm fix` would happily
apply it.

route_when_no_match also matched on lemonade / ollama / lm-studio, and
upstream_tracker carried those plus amdgpu-install, but Examination::probe
only ever reports skipped, pytorch, llama-cpp or unknown. Those arms were
unreachable, so the CLI advertised routing it cannot do. Routing an app that
merely appears in the symptom text belongs to the caller, not the probe.

Both are pinned: one test runs diagnose for each OS family and asserts every
emitted fix agrees with the catalog, another pins the reachable target set.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
The rocm-doctor skill is a thin driver: the probe, the closed 15-mode
catalog and the fixes all ship inside the `rocm` binary. What the skill owns
is a contract -- which fix-ids exist, which the CLI applies itself, which
machines each is for, and what a diagnosis carries. Because it describes
this binary, this repo is its source of truth; the catalog it is published
through vendors the folder from here.

Nothing tested that seam. diagnose.feature deliberately asserts only the
shape of a diagnosis so it stays host-independent, so a renamed fix-id or a
16th failure mode would merge green and silently break the published skill.

rocm_doctor_skill.feature parses reference.md's catalog table, its
auto-applicable prose, the fields and thresholds it names, the verdicts it
enumerates and the trackers it lists, then diffs each against what the CLI
really reports. Every scenario is a query -- no GPU, no serve, no download,
no mutation -- so they run on the blocking mock lane and need no capability
tags. Reading a document as test data is a deliberate, narrow exception to
the suite's black-box rule: nothing is imported from the codebase, and the
document is the thing under test.

The skill folder is excluded from the licence-header check: SKILL.md must
open with YAML frontmatter for the skill loader, which pushes any header out
of hawkeye's detection window. Verified in both directions -- the folder is
exempt, and stripping a header elsewhere still fails the check.

skills/** joins the `heavy` paths filter, since reference.md is now a
fixture and a skill-only edit has to run the e2e job.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
The skill told an agent to route upstream "when `matched` is empty", which
`DiagnoseReport::has_match` exists precisely to prevent: several checkers
open with a nonzero base score for a merely potentially relevant situation,
so a healthy host returns a non-empty `matched` of sub-threshold entries. An
agent following that instruction proposes a fix for a machine with nothing
wrong and never routes upstream. Both skill files now name `has_match` and
gate on it.

That gap was invisible to the contract feature because the field check ran
one way only -- documented ⊆ emitted -- so a field the CLI emits and the
document never mentions could not fail anything. It is now an equality, and
`rocm diagnose` emits exactly the six keys the reference names.

The route check had the same shape. It compared the CLI's target against
every tracker in the Framework routing section, including the three the
skill hands over itself, so a documented-but-unreachable CLI target passed
silently. The section now separates the two lists and the CLI is held to its
own.

Scenario 6's Given was inert: `route_when_no_match` is populated whether or
not anything matched, so the scenario passed identically for a symptom that
did match. It now asserts `has_match` is false first. Where the catalog is
out of scope the check still holds, since the catalog is not run at all; a
matching symptom can only be produced on a host the catalog covers, so the
no-GPU Linux lane is where it bites.

Finally, `route_when_no_match`'s comment claimed a test would flag a newly
added framework. That test iterates a hand-maintained list and reads nothing
from `examine.rs`, so it would not. Reworded to name the three places that
must be edited together.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… holds still

Two CI failures, both mine.

`rocm_doctor_skill.feature` had no `FEATURE_KEYS` entry, so
`feature_files_and_declared_keys_agree` panicked and took `Test (affected
crates)`, `windows-build-and-test` and `E2E tests` down with it. Adds the key
`skill` and renames the six scenarios to `skill-01`..`skill-06`, which the
sibling naming tests require once a file is indexed. The `@id:` tags already
carried the prefix. I missed this locally by running the workspace suite with
`--exclude e2e-cucumber` -- the crate that holds the test.

Scenario 6 then asserted `has_match == false` on the grounds that its symptom
carried no catalog keyword. That premise does not survive a real host:
`diagnose` scores several checkers from host state alone, and the
GitHub-hosted Linux runner ships /etc/modprobe.d/blacklist-radeon-instinct.conf
with amdgpu unloaded, which is `fix-5-amdgpu-load` at score 90 no matter what
symptom is passed. No symptom can hold that still, so the assertion only
encoded a runner's state. It passed on WSL2 because the catalog is skipped
there entirely, which is why it looked host-independent.

The claim moves to `rocm-core`, where the Examination is constructed instead
of probed: a host whose only causes are sub-threshold has a NON-empty `matched`
and `has_match` false, and the route still names somewhere to go -- the exact
shape that makes gating on emptiness wrong. A companion test asserts a matching
symptom flips the flag, so the assertion is not free.

What is left in the feature is the half only it can check: whatever target the
CLI hands back is one the published document names. The scenario is renamed to
say that, rather than to claim a premise it does not establish.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
…Linux

`reference.md` told an agent that auto fixes "print the exact command, honor
--dry-run, refuse on a non-interactive shell without --yes, and confirm before
mutating". That is false for fix-2 on Linux. `run_unset_override` splits by
platform: the Windows arm takes `FixOptions` and does exactly that, while
`run_unset_override_linux` takes no options at all -- it reports where the
override is set and which rc files carry it, then stops, because editing a
user's dotfiles is not its business. So there is no prompt to answer, nothing
for --dry-run to preview, and --yes is never read. An agent following the doc
would tell a Linux user a change was previewed that was never going to happen.

`auto_applicable` is a flat bool on a const recipe, and making it OS-aware
would ripple through the listing, the diagnosis JSON and the catalog row, so
the flag keeps its meaning and the docs stop overpromising: the Auto-fix column
is now defined as what `rocm fix` actually reports -- the CLI has a runner and
will carry it out itself -- and the mutation contract moves to prose that names
fix-2-on-Linux as the exception.

Pinned so it cannot drift back. `auto_applicable_recipes_have_a_runner` cannot
catch this, because fix-2 does have a runner. The rc-file scan is split into
`report_persistent_override`, taking its candidate paths the way this PR
already made `pin_device_in_rc_file` take its rc path -- deriving them from
$HOME would mean mutating a process-global in a parallel test suite. The test
asserts the files are byte-identical afterwards rather than asserting an exit
code, since a runner that started stripping the line would still return 0;
confirmed it fails when the runner is made to mutate.

Also drops a stale duplicate doc line left on `upstream_tracker` when the
framework-keyed arms were removed.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
…ding

fix-9-igpu-dgpu mutates on both Linux and Windows, gated only by
--device-index, not by platform like fix-2's documented exception.
Note both in reference.md and SKILL.md, and surface the same caveat
in the CLI's own diagnosis notes so an agent reading a diagnosis (not
just the docs) sees it too.

AGENTS.md's federation description no longer matches amd/skills:
there is no automated nightly vendoring job for rocm-doctor, and the
skill sits under staging/ which no federation job covers yet. Rewrite
the section to say what actually happens today (hand-sync during
Phase-1 incubation) instead of describing a job that doesn't exist,
and attribute the failure catalog's checkers/OS-scoping to
diagnose.rs alongside fix.rs.

Narrow the e2e suite's and README's claims to what skill_steps.rs
actually parses: the catalog table and the auto-applicable prose line
are checked; the OS-scope summary line and the failure-mode/print-only
counts are not, and can drift without failing anything here. Also
reword the dead None arm in assert_route_is_documented into a loud
panic, since documented_cli_routes only ever returns Some in practice.

Add skills/** to e2e-selfhosted.yml's heavy path filter so it matches
ci.yml: a skill-only edit should trigger the e2e job that uses it as a
fixture.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
`fix-17-torch-dlpack` landed in `crates/rocm-core` but never reached
`reference.md`, so the doc-vs-CLI scenarios this branch introduces
(skill-01..03) failed on every E2E lane: `rocm fix` offered an id the
published catalog did not name.

Add the table row, correct the mode count and the Linux-only recap, and
bump SKILL.md's print-only count. Call out the `fix-16` gap explicitly --
the id is a reserved handle, so a reader counting rows against the
heading would otherwise read the jump as a missing entry.

The counts and recap line are free-standing prose that no step parses;
only the table row is what turned the lanes green.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
…m family

WSL2 stopped being off-catalog: it is now a platform family of its own with
seven entries, and four cross-platform fixes widened to reach it. The skill
still described the old world, so `reference.md` named 16 of the CLI's 23
remediations and scoped four of them wrongly -- the doc-vs-CLI scenarios
caught both.

Add the seven `fix-wsl-*` rows, replace the `both` OS cell with the explicit
family list, and retire the WSL2 out-of-scope entry in favour of a note on why
it still gets its own entries rather than inheriting the bare-metal ones.

SKILL.md needed more than a count. Its Scope gate told an agent to decline a
WSL2 user outright and offer no troubleshooting whatsoever, which is now the
opposite of what the CLI does; the gate narrows to the GPU vendor, and WSL2 is
called out as explicitly in scope in the four places that excluded it. The
`out_of_scope` field is documented as what it now means -- no entries for this
platform family at all, so nothing was checked.

Signal cells are written from what each checker actually keys on rather than
from its rationale: fix-wsl-5 fires on the distro release, not on a glibc
error, and fix-wsl-6 on the host reporting no AMD adapter.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
`fix-19-shm-too-small` landed in the CLI catalog while this branch was in
review, so rebasing onto main put the published skill one entry behind
what `rocm fix` offers. The contract scenarios caught it: skill-01,
skill-02 and skill-03 all failed on the same missing row.

Add it with the scope and flag the CLI actually reports -- linux/wsl,
print-only -- and carry the three free-standing prose restatements that
nothing parses: the heading count, the OS-scope summary line, and
SKILL.md's print-only count.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
* ci: grade the rocm-doctor skill with skillscope

The skill's prose is now tested against the binary, but nothing checked the
parts an agent runtime reads before it ever gets that far: the frontmatter
that decides whether the skill loads at all, and the links it follows into
reference.md. A `name` that disagrees with its folder, or a link into a
renamed file, fails silently -- the agent simply never uses the skill, or
follows the link, finds nothing, and improvises.

skillscope is AMD's harness for exactly this. Only its `structural` command
runs here: no agent, no API key, and no network beyond the install, so it can
gate a merge without ever being wrong for a reason of its own. It checks the
frontmatter, the `skill-card.md` sections, every internal reference, and the
dataset's coverage bar.

`evals/evals.json` is that dataset -- four prompts the skill must answer and
two near misses it must decline, over the scope gate, the consent rule and
the Phase 0 probe. One case pins the correction this series just made: a
vague report where nothing clears the threshold must be routed upstream on
`has_match`, not diagnosed from the sub-threshold entries still sitting in
`matched`.

Federation does not carry `evals/`, so this dataset is ours and the catalog
keeps its own.

`routing` and `behavioral` stay off: both need an authenticated `claude` CLI
and an ANTHROPIC_API_KEY this repo does not have. They are the half of the
question that asks whether the skill actually fires, so the dataset is
written and checked to make switching them on a workflow change and nothing
more.

skills/rocm-cli-assistant/ is deliberately out of scope. Despite living under
skills/ it is not a published skill: main.rs embeds it verbatim into the chat
system prompt with include_str!, so giving it the frontmatter this check
requires would inject that frontmatter into a live prompt.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

* ci: correct skill-evals claims and close its coverage gaps

Address review feedback: the AGENTS.md/ci.yml text claimed skill-evals
already blocks merges, when it isn't a required status check yet, and
claimed the pinned skillscope action fully covers supply-chain drift,
when the composite's own steps and the uvx-resolved harness aren't
pinned. Both are now stated accurately instead of asserted.

Also: the changes-job `skills` filter now matches its siblings by
covering all of `.github/workflows/**` instead of just `ci.yml`; a new
guard step fails the job if a skills/ directory shows up that isn't
graded or excluded by design, so a second skill can't go ungraded
silently; and the WSL2 eval case is reshaped to `skill_should_trigger:
true` (the skill loads to decline it, per its own scope gate) with a
fresh negative case restoring the two-negative floor.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>

---------

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Bringing WSL2 into the catalog updated the skill's body but not the two
places that decide whether the body is ever read, nor the dataset that
grades it.

The frontmatter description and the skill card both said Linux and
Windows. That is the surface a runtime routes on, so a WSL2 user could
fail to load the skill at all and never reach the five places the body
insists WSL2 is in scope.

The `rocm-under-wsl2` eval asserted the opposite of what the skill now
requires: decline the platform, run nothing, offer no troubleshooting.
Its `skill_should_trigger` flag had been flipped to true without the
body following, so a skill obeying SKILL.md failed the case and one
passing it violated SKILL.md. Rewrite it to grade the in-scope
workflow, and to check the CLI's own scoping holds -- a bare-metal
Linux remediation reaching a WSL2 host means the agent went around it.

Separately, `routing_targets_cover_every_framework_the_probe_reports`
did not pin what its commit said it did. The loop carried only the four
frameworks the probe reports, so restoring an unreachable `lemonade`
arm to `route_when_no_match` left the observed target set unchanged and
the test green. Add the three removed names: absent arms route them to
`rocm-core` and nothing changes, a restored arm grows the set and fails.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
The WSL2 eval's expectation said the CLI returns "the `fix-wsl-*`
family" on that platform. It does not: twelve catalog entries reach the
`wsl` family, because `fix-1`, `fix-2`, `fix-6` and `fix-8` are scoped
to all three and `fix-19` to Linux and WSL. The entry's own prompt --
`torch.cuda.is_available()` false -- is exactly what `fix-1` and `fix-8`
fire on, so an agent surfacing them would have been graded as violating
an expectation while doing what the skill asks. Drop the enumeration and
let the instruction stand on what it actually means: stay on the entries
the CLI returned, whatever they are.

`every_diagnosis_agrees_with_the_fix_catalog_on_auto_applicability`
carried a doc comment describing fix-9's flag as a self-contradiction
this branch repaired. `main` landed that correction independently while
this branch was in review, so the rebase left the claim without a diff
behind it. Reword it to say what the test guards going forward, which is
what a reader needs from it anyway.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
fix-9 said the same thing twice. The detected-targets note carried an
appended "--device-index is required" sentence and the notes list carried
a second, longer one saying exactly that; render_report_text prints every
note on its own line, so `rocm diagnose` showed two bullets repeating each
other on Linux and on Windows alike. Keep the standalone note, which also
explains what the AUTO flag does and does not promise, and drop the
appended copy. A test now pins the caveat to exactly one rendered note
line, and pins the notes to be distinct, on both OS branches.

The skill contract's catalog reader rewrote an os-scope cell of "both"
to "linux/windows". No row spells it that way, so it was dead, but it was
dead code that could only ever do harm: reintroduced for a fix scoped to
linux, windows and wsl, it would have dropped "wsl" and made the document
and the CLI agree when they did not -- the one mismatch the contract
exists to catch. Take the cell verbatim instead, so an invented spelling
surfaces as the os-scope diff it is. A new guard test rejects any cell
that is not a "/"-joined list of the platforms `rocm fix` prints, and it
runs in the ordinary `cargo test` set rather than behind the E2E suite.

The ungraded-skills CI guard iterated `skills/*/` under `set -euo
pipefail` with no nullglob. Verified against an empty tree: it reported
"skills/* is neither graded ..." and exited 1, naming a skill that does
not exist. With `shopt -s nullglob` the loop does not run, which is the
right answer for nothing to grade, and an ungraded directory is still
caught.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Two doc comments had merged. The block explaining what
`routing_targets_cover_every_framework_the_probe_reports` pins ended up
on `sub_threshold_causes_leave_has_match_false_and_route_upstream`,
because a later commit inserted the has_match tests between the comment
and its function and the two blocks ran together with no separator.

It reads as false prose on one test, which has no framework list below
it and pins no reachable set, and as a missing warning on the other:
that paragraph is the only record that the list is hand-maintained and
will not notice a fifth framework added to the probe, which is exactly
what a maintainer needs when they touch it.

Also make the paragraph honest about the list it now describes. Three
entries were added to catch a re-added unreachable arm, so the list is
no longer only the probe's own values.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
An `else if confirm(...)` with no matching `else` let a declined
(or, without --yes, a non-interactive) User-scope clear fall through
silently. Nothing printed, nothing ran, and the function still
returned 0 at the bottom -- indistinguishable from "nothing to clear"
or "cleared successfully". skills/rocm-doctor/reference.md documents
5 as "user declined"; an agent driving this fix through the CLI had
no way to tell a no from a yes.

A decline still can't return 5 on the spot, though: if the Machine
scope is also set, its elevated-shell guidance has to print
unconditionally regardless of what happened with the User scope, and
an early return would skip it. So the decline is recorded in a flag
and checked once after both scopes have had their say.

Pulled the scope-dependent logic into
report_and_clear_override_windows, taking the two scope values and
consent as parameters instead of reading them from ps_env_scope/
confirm directly -- the same shape pin_device_in_rc_file already
uses, and for the same reason: it lets the decline path be exercised
by a test without a real Windows host.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
@volen-silo
volen-silo force-pushed the test/rocm-doctor-skill-contract branch from 71128f2 to d011945 Compare October 2, 2026 06:09
@volen-silo
volen-silo enabled auto-merge October 2, 2026 06:13
@volen-silo
volen-silo added this pull request to the merge queue Oct 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 2, 2026
@juhovainio
juhovainio added this pull request to the merge queue Oct 2, 2026
Merged via the queue into main with commit 9930f6c Oct 2, 2026
@juhovainio
juhovainio deleted the test/rocm-doctor-skill-contract branch October 2, 2026 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants