Repository navigation
fix(diagnose): correct fix-9's auto flag and test the rocm-doctor contract - #149
Conversation
rominf
left a comment
There was a problem hiding this comment.
Reviewed at 987dd9a2. Checked out locally: fix::tests 18/18 pass, all 22 CI checks green, and I re-verified the description's load-bearing claims (exit codes 1/4/5 were indeed unasserted before; the 15-mode catalog, OS scopes and auto-flags in reference.md match RECIPES exactly).
The refactor is the good part — splitting consent_without_prompt out of confirm and taking the rc path plus an injected consent verdict in pin_device_in_rc_file makes the mutation path genuinely testable without touching $HOME or stdin, and it's behaviour-preserving. The comments explaining why the two tests that looked like coverage weren't are worth keeping.
Three things I'd want resolved before this merges, then some smaller ones.
Dependent on an unmerged upstream PR, but not draft
The premise is that skills/rocm-doctor/ is byte-verbatim with amd/skills#87 — but #87 is still open and can still change in review. Merging first can land a mirror that never matches what upstream finally accepts, and the drift job fails open so nothing will say so. Suggest draft until #87 lands, then mirror the merged files.
That also sequences the next point for free.
The mirror ships documentation that's already known to be wrong
You flag it in the PR body, so this is agreement rather than a new find — but it interacts badly with the framing. The point of the change is that the documented contract becomes executable and can't silently break; a reader will reasonably assume the whole file is under test. It isn't: the new scenarios cover the catalog, and SKILL.md's install instructions are untested and currently false (install.sh:310-319 is Linux x86_64 only and hard-fails otherwise, against "Linux / macOS"; the nightly-only claim is likewise stale).
Fixing it upstream is the right layer. Landing #87 with the correction first and mirroring the corrected file avoids importing a known-wrong doc.
Scenario 2's pinned set duplicates an existing unit test, and its rationale doesn't hold
skill_steps.rs:264-266 justifies the hard-coded AUTO_APPLICABLE with "both files name these four ids in prose". SKILL.md contains no fix-ids at all — grep 'fix-[0-9]' skills/rocm-doctor/SKILL.md is empty. Only reference.md names them (line 43), and the assertion doesn't read that prose either: it compares the CLI to a Rust array, which is exactly what exactly_the_four_known_fixes_are_auto (fix.rs:1018) already asserts, same ids, same order.
So it's a duplicate by the PR's own "claims already proven elsewhere are not restated" standard. Either drop it, or make it do the thing the comment describes and parse the prose line out of reference.md.
Smaller
world.model_nameis reused to carry a symptom string (skill_steps.rs:184) and a fix-id (:196). The PR already adds a dedicatedskill_referencefield; these deserve the same rather than borrowing an unrelated name.pinning_without_a_device_index_is_not_applicablereturns early on non-Linux, so it's a vacuous green on the Windows lane.#[cfg(target_os = "linux")]states that instead of passing silently.- Both parsers
panic!on an unrecognised row (skill_steps.rs:96,:123). Loud failure is defensible, but it means a cosmetic change torocm fixoutput surfaces as a panic rather than an assertion diff naming the two sets. - Scratch-dir cleanup is
remove_dir_all(...).ok()after the assertions, so a failing test leaves debris under.rocm-work/. Minor, and the no-tempfileconvention is a fair reason. !skills/rocm-cli-assistant/*.mdis non-recursive, and a new top-level directory with tracked markdown goes unchecked silently. AGENTS.md documents the hazard, which is the right mitigation — worth being sure that note is the first thing anyone editinglicenserc.tomlsees.- The drift job's
if: github.event_name == 'pull_request'skips the merge queue. Harmless while it's advisory, but it means the last state before merge is never compared.
On the licence change specifically: I checked it positively rather than trusting the green check, since a passing hawkeye check only proves nothing is failing. Stripping README.md's header makes hawkeye flag it, so !/*.md anchoring works and the mirror is genuinely exempt — and no currently-tracked markdown lost coverage in the blanket-to-enumeration swap.
529f774 to
0564a0e
Compare
rominf
left a comment
There was a problem hiding this comment.
I reviewed this at 04fe5aa5. The idea is good — making a published skill's documented contract executable, rather than letting it rot silently, is exactly the right instinct, and the doc-as-fixture parsing in skill_steps.rs is nicely done. The ten new rocm-core unit tests are real coverage, and I confirmed the scenarios actually run (plain untagged, skill_steps.rs is registered in e2e.rs).
But the branch is 38 commits behind main and that has caused real damage, plus the mirror premise no longer holds. My recommendation is: rebase, re-mirror all three files from upstream, and fix upstream first for the three places the doc is now wrong — then this is close.
The blocking set:
licenserc.tomlreverts main's HawkEye v7 migration (#300). It's the only merge conflict, and after rebase the exemption mechanism it edits no longer exists, so the mirrored.mdfiles would be header-required and the blocking check would fail.- The drift job fetches a path that 404s. The one check whose job is catching mirror rot can never fire — and it fails open, so nobody will notice.
- The mirror is not byte-verbatim any more. Upstream's
SKILL.mdhas gained a Scope gate section (a substantive safety instruction: decline non-AMD/WSL2 setups and give no troubleshooting) that this copy doesn't have. - A new unit test asserts exit 3 where
maindeliberately changed the branch to return 0 — andfix.rsauto-merges without conflict, so the merged tree gets two tests asserting different values for the same call. Guaranteed red after rebase, with no conflict marker to warn you.
Then five smaller ones about the tests not checking what they claim, and two nits.
Also worth flagging on the PR body: amd/skills#87 merged on 2026-08-13, so "Depends on amd/skills#87, which is still open… the upstream files 404" and the draft rationale are both stale — and the PR is not in draft. The dependency is discharged; the body should say so.
Sequencing thought: #250 renames rocm diagnose → rocm doctor and changes render_report_text's signature. This PR promotes the pre-consolidation surface to a tested, published contract — and the new feature only checks CLI-against-doc, never doc-against-CLI-surface, so it won't flag the mismatch. I'd land #250 first, then rebase this with the mirror re-fetched after upstream is corrected.
CI: E2E tests (Strix Halo, Windows) is red, but the branch is 38 commits stale so that result doesn't mean much until rebase.
Method note: parts of this were read by parallel helper passes that ran degraded (sequential, no fan-out). Everything above I verified myself — including diffing the mirror against upstream, running git merge-tree, and querying the upstream repo's contents API.
04fe5aa to
9eb0eab
Compare
There was a problem hiding this comment.
Reviewed at 9eb0eabc.
Went through all 9 threads from my previous pass and checked each fix against the actual current code rather than taking the replies at face value — all genuinely hold:
licenserc.tomlis back on the v7 schema withskills/rocm-doctor/**excluded, and the resurrected v6 file is gone.- The dead
amd/skillsdrift-check job is deleted outright (the repo is now the source of truth, so there's nothing to drift-check). SKILL.md/reference.mdare re-synced: the Scope gate, Prerequisites, and thehas_matchgating are all present, in both directions (the reverse field-coverage assertion actually catches a doc omission when I mentally removedhas_matchfrom it).- The conflicting device-index test in
fix.rsis dropped in favor of main's, andreference.md's exit-3 row is corrected to "negative only." - The dead Lemonade/Ollama/LM Studio routing arms are gone from
diagnose.rs, and the routing table now lives in the skill's own hand-off, checked against the CLI's real reachable set. - The
diagnose.rscomment is honest now about the framework list being hand-maintained rather than promising a guarantee the test doesn't provide. skill_steps.rsparses the "auto-applicable" prose out ofreference.mdinstead of pinning a redundant array, and scenario 6'shas_matchassertion is live (verified via the WSL no-catalog path) with a dedicatedskill_symptomfield instead of overloadingmodel_name.
I resolved all 9 threads.
One new thing turned up on a fresh pass over the current diff: the new rocm_doctor_skill.feature has no entry in FEATURE_KEYS (tests/e2e-cucumber/tests/feature_naming.rs), so cargo test -p e2e-cucumber --test feature_naming fails — reproduced locally, feature_files_and_declared_keys_agree panics with "rocm_doctor_skill.feature has no key in FEATURE_KEYS — add one so its scenarios are indexed and its ids are qualified like every other feature." That's what's turning Test (affected crates), windows-build-and-test, and E2E tests red right now. Left as an inline comment below.
Everything else checked out: cargo check/cargo build on the touched crates, the full rocm-core unit suite (48/48), clippy with -D warnings, and the real rocm_doctor_skill.feature run against a built rocm binary (all 6 scenarios pass) — which also confirms the fix-9 auto_applicable correction is a genuine, previously-unnoticed bug fix backed by a real regression test, not just a doc tweak.
rominf
left a comment
There was a problem hiding this comment.
Marking the verdict properly — my review just above should have been request-changes rather than a plain comment (my error, not a change of opinion). Nothing new here.
The blocker is the one inline comment on tests/e2e-cucumber/features/rocm_doctor_skill.feature: the file has no FEATURE_KEYS entry, so feature_files_and_declared_keys_agree fails and Test (affected crates), windows-build-and-test and E2E tests are red. Everything from the previous round is resolved and verified.
rominf
left a comment
There was a problem hiding this comment.
Re-reviewed at e6fe813d. Both things I flagged last round are fixed, and I owe you a correction on one of them.
Correction: I attributed all three red checks to the missing FEATURE_KEYS entry. That was wrong. Only Test (affected crates) and windows-build-and-test came from that. E2E tests failed for the separate reason you found — scenario 6's has_match == false assertion. And that assertion was there because I asked for it in my previous review, on the reasoning that out_of_scope was the only host-dependent branch. It isn't: the catalog scores host state, so fix-5-amdgpu-load reaches 90 on the hosted Linux runner whatever symptom it is handed. Your diagnosis is right and mine was wrong. Sorry for the round trip.
Confirmed fixed at this head:
rocm_doctor_skill.featureis inFEATURE_KEYSand the six scenarios are indexedskill-01..skill-06, so the three sibling naming tests now actually run against it instead of skipping it vacuously.- The
has_matchclaim moved intorocm-coreagainst a constructedExamination.sub_threshold_causes_leave_has_match_false_and_route_upstreamplus the companion that flips the flag is the right shape — it pins the thing you actually mean without depending on what the runner happens to look like.
All checks are green on e6fe813d as of now, including the E2E tests and windows-build-and-test lanes.
Three things still open, inline below. The first matters most: the skill doc makes a promise the CLI does not keep on Linux, and the whole point of this PR is that the doc and the CLI agree.
rominf
left a comment
There was a problem hiding this comment.
Everything from my last two rounds holds — I re-verified each against the code rather than the replies. licenserc is back on the v7 schema, has_match is documented and gated with a real two-way field equality rather than an allow-list, the exit-3 row is right, the unreachable routing arms really are unreachable (I traced every producer of Examination.framework), the auto set is parsed from prose, skill_symptom is a dedicated field that something actually consumes, and FEATURE_KEYS is wired. The report_persistent_override and pin_device_in_rc_file extractions are behaviour-preserving, and the fix-2 dotfile byte-identity test is the strongest thing in the PR.
The blocker is that the premise the whole design rests on isn't true yet, and I only found it by going and looking at the catalog rather than reading the PR text. Details inline, but in short: amd/skills does not federate this repo, rocm-doctor sits in its staging/ directory which federation never touches, and the two copies have already diverged — while the drift-check that would have caught that was deleted last round on the grounds that federation replaces it.
Second, fix-9 repeats the fix-2 overpromise we corrected in the last round, in the same PR that promoted it.
The rest are smaller: several contract restatements that the parsers never read (including SKILL.md, which no step opens at all), a workflow filter that no longer matches the invariant it claims, the commit type, and two comment-accuracy points.
Nothing here is about the quality of the test scaffolding, which is good. It's about the guarantees being claimed for it being wider than what it checks.
| `rocm` binary, so it is versioned with the binary and lives here. The | ||
| [`amd/skills`](https://github.com/amd/skills) catalog federates it: a nightly | ||
| job vendors the folder from `main` and opens a pull request there. So edit it | ||
| here, and never edit the catalog's copy — the next federation run overwrites |
There was a problem hiding this comment.
This instruction points at a sync that doesn't happen, and I'd treat it as blocking because of what was removed on the strength of it.
I checked the catalog directly rather than going from the PR text. amd/skills/.github/federation.json declares exactly one source — AMD-AGI/TraceLens, skill tracelens-analysis-orchestrator. There is no ROCm/rocm-cli entry. And rocm-doctor isn't under amd/skills/skills/ at all; it's in amd/skills/staging/rocm-doctor/, which federate-skills.yml never touches, since it vendors only the sources declared in federation.json. The two copies have already drifted apart.
So "never edit the catalog's copy — the next federation run overwrites it" tells a future agent to rely on something that won't run. That matters more than a stale sentence: last round the amd/skills drift-check job was deleted outright, on the reasoning that both copies stop mattering once federation replaces the mirror. Federation doesn't currently replace it, so this PR removes the only detector and substitutes a mechanism that isn't wired up. Right now nothing keeps the published copy honest.
Two ways out, either fine by me: land the federation.json entry and promote staging/rocm-doctor first, and reference that from here; or say plainly that the catalog copy is hand-synced from staging today, name who syncs it, and keep a drift detector until federation is live. The same claim appears in licenserc.toml, the e2e README, and the feature header, so they'd all move together.
Separately, line 191 says federation doesn't carry the skill's evals/ folder — but skills/rocm-doctor/ in this PR has no evals/ at all. That's phantom detail in a file loaded on every task; either drop the sentence or land the folder.
There was a problem hiding this comment.
Agreed, and took the second path rather than landing a federation.json entry from this PR. AGENTS.md's federation bullet now says plainly that amd/skills is still Phase-1 incubation for this skill — no automated sync, federation.json only declares TraceLens, and rocm-doctor sits in staging/rocm-doctor outside any job's coverage — and that the rocm-cli team hand-syncs staging/rocm-doctor from here whenever it changes materially, rather than describing a job that doesn't run. Also dropped the phantom evals/ sentence — this skill has no evals/ folder. I did not restore a drift-check job or touch amd/skills from this PR; left open for you to re-verify the wording matches what you found live.
There was a problem hiding this comment.
One thing I did not touch, flagging rather than silently leaving inconsistent: licenserc.toml's comment still says the folder is "vendored verbatim into the amd/skills catalog" — same stale claim, out of this PR's assigned scope, so it's still there for a follow-up.
| ## Closed catalog (15 failure modes) | ||
|
|
||
| | id | OS | Failure mode | Typical signal | Auto-fix | | ||
| | --- | --- | --- | --- | --- | |
There was a problem hiding this comment.
This line is a second statement of the OS scope, and nothing reads it.
assert_same_os_scope compares the CLI against parse_reference_catalog, which reads the table's OS column only. So re-scoping a fix in the CLI and updating the table leaves this summary line stale and every scenario still green — while the feature header promises that "a re-scoped OS … cannot merge here and silently break the published skill".
This is the same double-statement problem you fixed for the auto set with documented_auto_prose; it just wasn't applied to the OS claim. Parsing this line the same way would close it.
Two more in the same family, since they share a root cause. The counts are unparsed — ## Closed catalog (15 failure modes) above, and SKILL.md's "The other 11 are print-only" — so adding a sixteenth recipe plus a table row leaves all six scenarios green with both numbers stale, which contradicts the header's "or a 16th failure mode" clause. And SKILL.md is never opened by any step: reference_md_path() is the only path constructed, yet the Given is worded "the skill as it is published" and the e2e README says this feature is the only thing catching the document drifting. SKILL.md independently restates the auto four, the routing table and the per-fix field list, none of it compared to anything.
Either extend the parsers to cover these, or narrow the header and README claims to what they actually check. I'd rather have a smaller promise than a wider one that isn't kept.
There was a problem hiding this comment.
Took the narrowing path rather than extending the parsers, on all three points: the feature header now says explicitly that the guarantee covers the table's cells and the auto-applicable prose line — what skill_steps.rs actually parses — and that the OS-scope summary line, the "(N failure modes)" heading count, and SKILL.md's "the other M are print-only" line are free-standing prose that restates those facts and is not parsed, so it can go stale even while the table stays correct. Same caveat added to tests/e2e-cucumber/README.md's description of this feature. Did not extend the parsers to cover SKILL.md, and did not change what skill_steps.rs checks — left open for you to confirm the narrowed claim matches what's actually verified.
|
Pushed a round addressing this review pass:
Also confirmed the four already-fixed threads from the previous round still hold (feature naming, the fix-2 exception paragraph, the duplicated doc-comment line, and the fix-9 flag-flip coverage gap, which is disclosed in the PR body under "Known coverage gap"). Verification: |
c7a236c to
88b35c6
Compare
88b35c6 to
d6096b9
Compare
siloteemu
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 9ef3777
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Summary
Publishes the rocm-doctor skill from this repo and makes its reference.md a live fixture: six e2e scenarios parse the published document and diff it against what rocm fix / rocm diagnose / rocm examine actually report, plus two CLI corrections and unit coverage for the fix consent/exit-code contract. Needs work — the contract testing is genuinely good, but one graded dataset encodes the behaviour the skill now forbids, and one of the two CLI corrections has no test that fails on its revert. Verified: read all eleven commits and every changed file; confirmed the catalog table's 24 rows, the four auto-applicable ids, the OS recap line, the exit-code table and the routing targets all match RECIPES and diagnose.rs; confirmed list_recipes() emits every recipe regardless of host OS and DiagnoseReport serialises exactly the six documented keys (so scenarios 1–4 are real, host-independent assertions that do fail on catalog drift — not vacuous); confirmed skills/** reaches the heavy filter that gates the e2e job in ci.yml, and that every new action pin is a full SHA with a version comment. No build, test or suite run was performed here, and no mutation or revert experiment was run — the revert conclusions below are read directly off the test bodies, which decide them without execution. Checks at review time: 23 success, 1 failure, 4 pending. Blocking: 2 · Non-blocking: 5.
🚫 Blocking (must fix before merge)
skills/rocm-doctor/evals/evals.json (the rocm-under-wsl2 entry) — the graded dataset asserts the exact opposite of the skill it grades.
This range makes WSL2 a first-class platform family: reference.md:77-115 gives it seven fix-wsl-* entries, reference.md:146 says "WSL2 used to sit here and no longer does", and SKILL.md:47-52 is emphatic — "WSL2 is in scope and does not stop this gate… Do not decline a WSL2 user or route them away untouched — run the workflow as for any other platform", repeated at SKILL.md:195-196 and SKILL.md:200-202. The eval left behind says the reverse: its note calls WSL2 "a separate platform the scope gate declines rather than troubleshoots", its expected_behavior is "State plainly that WSL2 is out of scope for this skill and why" and "Give no troubleshooting for it", and its unexpected_behavior is "Run rocm examine, rocm diagnose, or rocm fix" and "Offer any WSL2 troubleshooting steps". A skill that follows SKILL.md correctly fails this case, and a skill that passes it violates SKILL.md. This is the "pins a behaviour that is itself wrong" class: the dataset would certify the defect. The head commit compounds it by claiming the dataset is "written and checked to make switching [routing/behavioral] on a workflow change and nothing more" — switching them on today would grade conformant behaviour as wrong. The same commit says "four prompts the skill must answer and two near misses"; the file has five positives, because this entry moved from skill_should_trigger: false to true and only the flag was updated, not the body. Fix: rewrite this entry's note and expected/unexpected behaviours to the in-scope workflow (diagnose normally, WSL-scoped entries only), and correct the count in the commit message.
crates/rocm-core/src/diagnose.rs:2894 — routing_targets_cover_every_framework_the_probe_reports passes unchanged if the routing deletion it is presented as pinning is reverted.
Commit 5232431 removes the unreachable lemonade / ollama / lm-studio arms from route_when_no_match (and those plus amdgpu-install from upstream_tracker) and states "Both are pinned: … another pins the reachable target set." The test iterates only ["skipped", "pytorch", "llama-cpp", "unknown"], so restoring "lemonade" => "lemonade" to the match changes nothing it observes and the assertion still yields ["llama-cpp", "pytorch", "rocm-core"]. Scenario 6 does not cover the gap either: it asserts only that whatever target comes back is documented, and reference.md:132-134 still lists those three apps in its own routing table. So the regression the change exists to prevent — the CLI advertising routing the probe can never reach — has no failing test. Fix is one line: add "lemonade", "ollama", "lm-studio" to the loop array. With the arms absent they route to rocm-core and the existing assert_eq! is unchanged; if an arm is re-added the target set grows and the test fails loudly with the message it already carries.
Non-blocking
skills/rocm-doctor/SKILL.md:3-6andskills/rocm-doctor/skill-card.md— the frontmatterdescription("on Linux or Windows") and the card's Description ("Linux and Windows") both omit WSL2, so the surface a runtime routes on excludes the platform the body insists is in scope.crates/rocm-core/src/diagnose.rs:2932—every_diagnosis_agrees_with_the_fix_catalog_on_auto_applicabilityguards only on!report.matched.is_empty(); narrowingcheck_9_igpu_dgpu_collision's trigger would leave it green while covering nothing, so assertfix-9-igpu-dgpuis among the matched for each OS..github/workflows/e2e-selfhosted.yml:105—skills/**is added to theheavyfilter, but no job in that workflow readsneeds.changes.outputs.heavy(every lane gates onserve), so the added comment's claim that a skill-only edit "must therefore run this job too" does not hold there; the fixture check does run, viaci.yml.skills/rocm-doctor/reference.md:21— "Always exits 0" forrocm diagnoseis not quite true: a failed--distroprobe propagates an error and exits 1, and that flag is also missing from the documented invocation just above.tests/e2e-cucumber/features/rocm_doctor_skill.feature:11says "a 16th failure mode" andSKILL.md:180omitsskippedfrom the probe's framework values thatreference.md:130lists — both stale against the 24-entry catalog and the sibling document (the PR body's "closed 15-mode catalog" is stale the same way).
|
Went through all seven findings against the code rather than taking the labels at face value. Every one of them is real. Fixed the ones that earned it in The WSL2 items are one defect with three surfaces, not two findings. Bringing WSL2 into the catalog updated the skill's body but not the parts that decide whether the body is ever read, nor the dataset grading it:
I would rank the frontmatter one above the eval rather than below it. The eval grades nothing today, because only The routing test finding was correct and the suggested fix was the right one. I confirmed it by experiment rather than by reading: restored On severity. Neither of the two you marked blocking actually blocks a merge, for the reasons above. They were worth fixing because they are cheap and true, not because of the label. Flagging that because the distinction matters for anyone triaging from the headline. Deferred, all verified as real: the Unrelated to this PR, but worth someone's attention: the Strix Halo WSL2 lane is red here and was red on the previous commit too. It 404s fetching |
Addressed
Exit codes 1, 4 and 5 were asserted nowhere, and no test reached a runner that writes to disk. The two that looked like coverage did not: the dry-run test picks fix-2, whose Linux runner takes no FixOptions and never prompts, and the e2e preview scenario picks print-only fix-1, which returns before any runner runs. Split consent_without_prompt out of confirm, and let pin_device_in_rc_file take the rc path plus an injected consent verdict, so the mutation path is reachable from a test without touching $HOME or stdin. Behaviour is unchanged. Ten tests then cover the consent gate, both mutation outcomes, and the dry-run short-circuit. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
check_9_igpu_dgpu_collision reported auto_applicable: false on Linux while fix::RECIPES -- what `rocm fix` dispatches on -- says true. It was the only OS-split checker whose branches diverged, and an agent branches on that field, so it was told fix-9 was print-only while `rocm fix` would happily apply it. route_when_no_match also matched on lemonade / ollama / lm-studio, and upstream_tracker carried those plus amdgpu-install, but Examination::probe only ever reports skipped, pytorch, llama-cpp or unknown. Those arms were unreachable, so the CLI advertised routing it cannot do. Routing an app that merely appears in the symptom text belongs to the caller, not the probe. Both are pinned: one test runs diagnose for each OS family and asserts every emitted fix agrees with the catalog, another pins the reachable target set. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
The rocm-doctor skill is a thin driver: the probe, the closed 15-mode catalog and the fixes all ship inside the `rocm` binary. What the skill owns is a contract -- which fix-ids exist, which the CLI applies itself, which machines each is for, and what a diagnosis carries. Because it describes this binary, this repo is its source of truth; the catalog it is published through vendors the folder from here. Nothing tested that seam. diagnose.feature deliberately asserts only the shape of a diagnosis so it stays host-independent, so a renamed fix-id or a 16th failure mode would merge green and silently break the published skill. rocm_doctor_skill.feature parses reference.md's catalog table, its auto-applicable prose, the fields and thresholds it names, the verdicts it enumerates and the trackers it lists, then diffs each against what the CLI really reports. Every scenario is a query -- no GPU, no serve, no download, no mutation -- so they run on the blocking mock lane and need no capability tags. Reading a document as test data is a deliberate, narrow exception to the suite's black-box rule: nothing is imported from the codebase, and the document is the thing under test. The skill folder is excluded from the licence-header check: SKILL.md must open with YAML frontmatter for the skill loader, which pushes any header out of hawkeye's detection window. Verified in both directions -- the folder is exempt, and stripping a header elsewhere still fails the check. skills/** joins the `heavy` paths filter, since reference.md is now a fixture and a skill-only edit has to run the e2e job. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
The skill told an agent to route upstream "when `matched` is empty", which `DiagnoseReport::has_match` exists precisely to prevent: several checkers open with a nonzero base score for a merely potentially relevant situation, so a healthy host returns a non-empty `matched` of sub-threshold entries. An agent following that instruction proposes a fix for a machine with nothing wrong and never routes upstream. Both skill files now name `has_match` and gate on it. That gap was invisible to the contract feature because the field check ran one way only -- documented ⊆ emitted -- so a field the CLI emits and the document never mentions could not fail anything. It is now an equality, and `rocm diagnose` emits exactly the six keys the reference names. The route check had the same shape. It compared the CLI's target against every tracker in the Framework routing section, including the three the skill hands over itself, so a documented-but-unreachable CLI target passed silently. The section now separates the two lists and the CLI is held to its own. Scenario 6's Given was inert: `route_when_no_match` is populated whether or not anything matched, so the scenario passed identically for a symptom that did match. It now asserts `has_match` is false first. Where the catalog is out of scope the check still holds, since the catalog is not run at all; a matching symptom can only be produced on a host the catalog covers, so the no-GPU Linux lane is where it bites. Finally, `route_when_no_match`'s comment claimed a test would flag a newly added framework. That test iterates a hand-maintained list and reads nothing from `examine.rs`, so it would not. Reworded to name the three places that must be edited together. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
… holds still Two CI failures, both mine. `rocm_doctor_skill.feature` had no `FEATURE_KEYS` entry, so `feature_files_and_declared_keys_agree` panicked and took `Test (affected crates)`, `windows-build-and-test` and `E2E tests` down with it. Adds the key `skill` and renames the six scenarios to `skill-01`..`skill-06`, which the sibling naming tests require once a file is indexed. The `@id:` tags already carried the prefix. I missed this locally by running the workspace suite with `--exclude e2e-cucumber` -- the crate that holds the test. Scenario 6 then asserted `has_match == false` on the grounds that its symptom carried no catalog keyword. That premise does not survive a real host: `diagnose` scores several checkers from host state alone, and the GitHub-hosted Linux runner ships /etc/modprobe.d/blacklist-radeon-instinct.conf with amdgpu unloaded, which is `fix-5-amdgpu-load` at score 90 no matter what symptom is passed. No symptom can hold that still, so the assertion only encoded a runner's state. It passed on WSL2 because the catalog is skipped there entirely, which is why it looked host-independent. The claim moves to `rocm-core`, where the Examination is constructed instead of probed: a host whose only causes are sub-threshold has a NON-empty `matched` and `has_match` false, and the route still names somewhere to go -- the exact shape that makes gating on emptiness wrong. A companion test asserts a matching symptom flips the flag, so the assertion is not free. What is left in the feature is the half only it can check: whatever target the CLI hands back is one the published document names. The scenario is renamed to say that, rather than to claim a premise it does not establish. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
…Linux `reference.md` told an agent that auto fixes "print the exact command, honor --dry-run, refuse on a non-interactive shell without --yes, and confirm before mutating". That is false for fix-2 on Linux. `run_unset_override` splits by platform: the Windows arm takes `FixOptions` and does exactly that, while `run_unset_override_linux` takes no options at all -- it reports where the override is set and which rc files carry it, then stops, because editing a user's dotfiles is not its business. So there is no prompt to answer, nothing for --dry-run to preview, and --yes is never read. An agent following the doc would tell a Linux user a change was previewed that was never going to happen. `auto_applicable` is a flat bool on a const recipe, and making it OS-aware would ripple through the listing, the diagnosis JSON and the catalog row, so the flag keeps its meaning and the docs stop overpromising: the Auto-fix column is now defined as what `rocm fix` actually reports -- the CLI has a runner and will carry it out itself -- and the mutation contract moves to prose that names fix-2-on-Linux as the exception. Pinned so it cannot drift back. `auto_applicable_recipes_have_a_runner` cannot catch this, because fix-2 does have a runner. The rc-file scan is split into `report_persistent_override`, taking its candidate paths the way this PR already made `pin_device_in_rc_file` take its rc path -- deriving them from $HOME would mean mutating a process-global in a parallel test suite. The test asserts the files are byte-identical afterwards rather than asserting an exit code, since a runner that started stripping the line would still return 0; confirmed it fails when the runner is made to mutate. Also drops a stale duplicate doc line left on `upstream_tracker` when the framework-keyed arms were removed. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
…ding fix-9-igpu-dgpu mutates on both Linux and Windows, gated only by --device-index, not by platform like fix-2's documented exception. Note both in reference.md and SKILL.md, and surface the same caveat in the CLI's own diagnosis notes so an agent reading a diagnosis (not just the docs) sees it too. AGENTS.md's federation description no longer matches amd/skills: there is no automated nightly vendoring job for rocm-doctor, and the skill sits under staging/ which no federation job covers yet. Rewrite the section to say what actually happens today (hand-sync during Phase-1 incubation) instead of describing a job that doesn't exist, and attribute the failure catalog's checkers/OS-scoping to diagnose.rs alongside fix.rs. Narrow the e2e suite's and README's claims to what skill_steps.rs actually parses: the catalog table and the auto-applicable prose line are checked; the OS-scope summary line and the failure-mode/print-only counts are not, and can drift without failing anything here. Also reword the dead None arm in assert_route_is_documented into a loud panic, since documented_cli_routes only ever returns Some in practice. Add skills/** to e2e-selfhosted.yml's heavy path filter so it matches ci.yml: a skill-only edit should trigger the e2e job that uses it as a fixture. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
`fix-17-torch-dlpack` landed in `crates/rocm-core` but never reached `reference.md`, so the doc-vs-CLI scenarios this branch introduces (skill-01..03) failed on every E2E lane: `rocm fix` offered an id the published catalog did not name. Add the table row, correct the mode count and the Linux-only recap, and bump SKILL.md's print-only count. Call out the `fix-16` gap explicitly -- the id is a reserved handle, so a reader counting rows against the heading would otherwise read the jump as a missing entry. The counts and recap line are free-standing prose that no step parses; only the table row is what turned the lanes green. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
…m family WSL2 stopped being off-catalog: it is now a platform family of its own with seven entries, and four cross-platform fixes widened to reach it. The skill still described the old world, so `reference.md` named 16 of the CLI's 23 remediations and scoped four of them wrongly -- the doc-vs-CLI scenarios caught both. Add the seven `fix-wsl-*` rows, replace the `both` OS cell with the explicit family list, and retire the WSL2 out-of-scope entry in favour of a note on why it still gets its own entries rather than inheriting the bare-metal ones. SKILL.md needed more than a count. Its Scope gate told an agent to decline a WSL2 user outright and offer no troubleshooting whatsoever, which is now the opposite of what the CLI does; the gate narrows to the GPU vendor, and WSL2 is called out as explicitly in scope in the four places that excluded it. The `out_of_scope` field is documented as what it now means -- no entries for this platform family at all, so nothing was checked. Signal cells are written from what each checker actually keys on rather than from its rationale: fix-wsl-5 fires on the distro release, not on a glibc error, and fix-wsl-6 on the host reporting no AMD adapter. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
`fix-19-shm-too-small` landed in the CLI catalog while this branch was in review, so rebasing onto main put the published skill one entry behind what `rocm fix` offers. The contract scenarios caught it: skill-01, skill-02 and skill-03 all failed on the same missing row. Add it with the scope and flag the CLI actually reports -- linux/wsl, print-only -- and carry the three free-standing prose restatements that nothing parses: the heading count, the OS-scope summary line, and SKILL.md's print-only count. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
* ci: grade the rocm-doctor skill with skillscope The skill's prose is now tested against the binary, but nothing checked the parts an agent runtime reads before it ever gets that far: the frontmatter that decides whether the skill loads at all, and the links it follows into reference.md. A `name` that disagrees with its folder, or a link into a renamed file, fails silently -- the agent simply never uses the skill, or follows the link, finds nothing, and improvises. skillscope is AMD's harness for exactly this. Only its `structural` command runs here: no agent, no API key, and no network beyond the install, so it can gate a merge without ever being wrong for a reason of its own. It checks the frontmatter, the `skill-card.md` sections, every internal reference, and the dataset's coverage bar. `evals/evals.json` is that dataset -- four prompts the skill must answer and two near misses it must decline, over the scope gate, the consent rule and the Phase 0 probe. One case pins the correction this series just made: a vague report where nothing clears the threshold must be routed upstream on `has_match`, not diagnosed from the sub-threshold entries still sitting in `matched`. Federation does not carry `evals/`, so this dataset is ours and the catalog keeps its own. `routing` and `behavioral` stay off: both need an authenticated `claude` CLI and an ANTHROPIC_API_KEY this repo does not have. They are the half of the question that asks whether the skill actually fires, so the dataset is written and checked to make switching them on a workflow change and nothing more. skills/rocm-cli-assistant/ is deliberately out of scope. Despite living under skills/ it is not a published skill: main.rs embeds it verbatim into the chat system prompt with include_str!, so giving it the frontmatter this check requires would inject that frontmatter into a live prompt. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com> * ci: correct skill-evals claims and close its coverage gaps Address review feedback: the AGENTS.md/ci.yml text claimed skill-evals already blocks merges, when it isn't a required status check yet, and claimed the pinned skillscope action fully covers supply-chain drift, when the composite's own steps and the uvx-resolved harness aren't pinned. Both are now stated accurately instead of asserted. Also: the changes-job `skills` filter now matches its siblings by covering all of `.github/workflows/**` instead of just `ci.yml`; a new guard step fails the job if a skills/ directory shows up that isn't graded or excluded by design, so a second skill can't go ungraded silently; and the WSL2 eval case is reshaped to `skill_should_trigger: true` (the skill loads to decline it, per its own scope gate) with a fresh negative case restoring the two-negative floor. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com> --------- Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Bringing WSL2 into the catalog updated the skill's body but not the two places that decide whether the body is ever read, nor the dataset that grades it. The frontmatter description and the skill card both said Linux and Windows. That is the surface a runtime routes on, so a WSL2 user could fail to load the skill at all and never reach the five places the body insists WSL2 is in scope. The `rocm-under-wsl2` eval asserted the opposite of what the skill now requires: decline the platform, run nothing, offer no troubleshooting. Its `skill_should_trigger` flag had been flipped to true without the body following, so a skill obeying SKILL.md failed the case and one passing it violated SKILL.md. Rewrite it to grade the in-scope workflow, and to check the CLI's own scoping holds -- a bare-metal Linux remediation reaching a WSL2 host means the agent went around it. Separately, `routing_targets_cover_every_framework_the_probe_reports` did not pin what its commit said it did. The loop carried only the four frameworks the probe reports, so restoring an unreachable `lemonade` arm to `route_when_no_match` left the observed target set unchanged and the test green. Add the three removed names: absent arms route them to `rocm-core` and nothing changes, a restored arm grows the set and fails. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
The WSL2 eval's expectation said the CLI returns "the `fix-wsl-*` family" on that platform. It does not: twelve catalog entries reach the `wsl` family, because `fix-1`, `fix-2`, `fix-6` and `fix-8` are scoped to all three and `fix-19` to Linux and WSL. The entry's own prompt -- `torch.cuda.is_available()` false -- is exactly what `fix-1` and `fix-8` fire on, so an agent surfacing them would have been graded as violating an expectation while doing what the skill asks. Drop the enumeration and let the instruction stand on what it actually means: stay on the entries the CLI returned, whatever they are. `every_diagnosis_agrees_with_the_fix_catalog_on_auto_applicability` carried a doc comment describing fix-9's flag as a self-contradiction this branch repaired. `main` landed that correction independently while this branch was in review, so the rebase left the claim without a diff behind it. Reword it to say what the test guards going forward, which is what a reader needs from it anyway. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
fix-9 said the same thing twice. The detected-targets note carried an appended "--device-index is required" sentence and the notes list carried a second, longer one saying exactly that; render_report_text prints every note on its own line, so `rocm diagnose` showed two bullets repeating each other on Linux and on Windows alike. Keep the standalone note, which also explains what the AUTO flag does and does not promise, and drop the appended copy. A test now pins the caveat to exactly one rendered note line, and pins the notes to be distinct, on both OS branches. The skill contract's catalog reader rewrote an os-scope cell of "both" to "linux/windows". No row spells it that way, so it was dead, but it was dead code that could only ever do harm: reintroduced for a fix scoped to linux, windows and wsl, it would have dropped "wsl" and made the document and the CLI agree when they did not -- the one mismatch the contract exists to catch. Take the cell verbatim instead, so an invented spelling surfaces as the os-scope diff it is. A new guard test rejects any cell that is not a "/"-joined list of the platforms `rocm fix` prints, and it runs in the ordinary `cargo test` set rather than behind the E2E suite. The ungraded-skills CI guard iterated `skills/*/` under `set -euo pipefail` with no nullglob. Verified against an empty tree: it reported "skills/* is neither graded ..." and exited 1, naming a skill that does not exist. With `shopt -s nullglob` the loop does not run, which is the right answer for nothing to grade, and an ungraded directory is still caught. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Two doc comments had merged. The block explaining what `routing_targets_cover_every_framework_the_probe_reports` pins ended up on `sub_threshold_causes_leave_has_match_false_and_route_upstream`, because a later commit inserted the has_match tests between the comment and its function and the two blocks ran together with no separator. It reads as false prose on one test, which has no framework list below it and pins no reachable set, and as a missing warning on the other: that paragraph is the only record that the list is hand-maintained and will not notice a fifth framework added to the probe, which is exactly what a maintainer needs when they touch it. Also make the paragraph honest about the list it now describes. Three entries were added to catch a re-added unreachable arm, so the list is no longer only the probe's own values. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
An `else if confirm(...)` with no matching `else` let a declined (or, without --yes, a non-interactive) User-scope clear fall through silently. Nothing printed, nothing ran, and the function still returned 0 at the bottom -- indistinguishable from "nothing to clear" or "cleared successfully". skills/rocm-doctor/reference.md documents 5 as "user declined"; an agent driving this fix through the CLI had no way to tell a no from a yes. A decline still can't return 5 on the spot, though: if the Machine scope is also set, its elevated-shell guidance has to print unconditionally regardless of what happened with the User scope, and an early return would skip it. So the decline is recorded in a flag and checked once after both scopes have had their say. Pulled the scope-dependent logic into report_and_clear_override_windows, taking the two scope values and consent as parameters instead of reading them from ps_env_scope/ confirm directly -- the same shape pin_device_in_rc_file already uses, and for the same reason: it lets the decline path be exercised by a test without a real Windows host. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
71128f2 to
d011945
Compare
What
skills/rocm-doctor/is a published Agent Skill, and this repo is its sourceof truth. The skill is a thin driver: the probe, the closed 15-mode catalog and
the fixes all ship inside the
rocmbinary, so it is versioned with the binaryand lives beside it. The
amd/skillscatalog federates skills from the reposthat own them, which is how the published copy stays in step.
What the skill owns is a contract: which fix-ids exist, which the CLI applies
itself, which machines each is for, what a diagnosis carries, and where a report
goes when nothing matched. Nothing tested that seam.
diagnose.featuredeliberately asserts only the shape of a diagnosis so it stays
host-independent, so a renamed fix-id or a 16th failure mode would merge green
and silently break the published skill.
Changes
Contract feature —
rocm_doctor_skill.feature(6 scenarios) parsesreference.md's catalog table, its auto-applicable prose, the fields andthresholds it names, the verdicts it enumerates and the trackers it lists, then
diffs each against what the CLI reports. Query-only: no GPU, no serve, no
download, no mutation, so it runs on the blocking mock lane with no capability
tags. Reading a document as test data is a deliberate, narrow exception to the
suite's black-box rule — nothing is imported from the codebase, and the document
is the thing under test.
rocm-core unit tests — exit codes
1,4and5were asserted nowhere andno test reached a runner that writes to disk. The two that looked like coverage
did not: the dry-run test picks
fix-2, whose Linux runner takes noFixOptionsand never prompts, and the e2e preview scenario picks print-onlyfix-1, which returns before any runner.One CLI correction —
route_when_no_matchmatched on lemonade / ollama /lm-studio, which
Examination::probecan never report, so the CLI advertisedrouting it cannot do.
This originally read "two corrections". The second was fix-9's
auto_applicableflag, whichcheck_9_igpu_dgpu_collisionreported asfalseon Linux while
fix::RECIPESsaidtrue.mainlanded the same correctionindependently while this branch was in review, so rebasing dropped it from this
diff:
auto_applicableis unchanged context here, and only the comment above itdiffers. What this branch still adds for it is the regression test that keeps
the two sides from drifting apart again.
Skill corrections — the skill gated on "is
matchedempty?", whichDiagnoseReport::has_matchexists to prevent; it promised that everyauto-applicable fix honours
--dry-runand confirms before mutating, which isfalse for
fix-2-unset-overrideon Linux (its runner reports and stops); theinstaller was labelled "Linux / macOS" when
install.shis Linux x86_64 only;and the nightly-only install claim was stale.
Licence config —
skills/rocm-doctor/**is excluded:SKILL.mdmust openwith YAML frontmatter for the skill loader, which pushes any header out of
hawkeye's detection window, and catalog skills carry no headers.
Scope
Claims proven elsewhere are not restated. Two earlier scenarios were dropped for
that reason.
skills/**joins theheavypaths filter, sincereference.mdisnow a fixture.
Verification
cargo xtask e2e— 81 scenarios, 79 pass; the 2 failures are the expectedWSL2
diagnosexfails (0 unexpected)cargo test --workspace --all-targets --exclude e2e-cucumber— cleancargo clippy --locked --workspace --all-targetsand--locked -p e2e-cucumber --test e2e— cleanhawkeye checkand allprekhooks — clean, and verified positively:stripping
README.md's orrocm-cli-assistant's header still exits 1, so thenew exclusion is not swallowing the check
restoring
"lemonade" => "lemonade"makes it fail, then revertedKnown coverage gap
The fix-9
auto_applicableflag is user-observable —render_report_textadds a
flags:line when it is set, andFixderivesSerialize, so the JSONcarries it too. The correction itself now comes from
main(see above), but theregression test this branch adds for it still has no Gherkin scenario behind it,
which
AGENTS.mdasks for. This is a gap I could not close rather than one Ioverlooked:
check_9_igpu_dgpu_collisiononly fires on anExaminationcarrying an APUand a discrete AMD GPU, the suite drives the real binary against the real
host, no
--from-examination-style input exists to hand it a synthetic one, andno CI lane has both. It is covered by a
rocm-coreregression test that runsdiagnosefor both OS families and asserts every emitted fix agrees with thecatalog.
Not verified locally: scenario 6 rejecting a matching symptom. This host is
WSL2, where the catalog is out of scope and never runs, so
has_matchis falseeither way. That case bites on the no-GPU Linux lane.
Unrelated pre-existing flake seen while verifying:
dash-gen-tps-*assertagainst a wall-clock validity window and fail intermittently on full runs — on
mainas well as here, and never in isolation. Not touched by this PR.