Repository navigation
docs(skills): say where rocm-doctor's routing and behavioral are graded - #798
Conversation
juhovainio
left a comment
There was a problem hiding this comment.
Looks good to me. I checked the new text against amd/skills: federation.json does list rocm-cli/skills/rocm-doctor, the federate workflow runs nightly and records the source commit and content hash in .federated.json, and rocm-doctor is not in routing_room. The old AGENTS.md and ci.yml claims were stale and the replacements are accurate. CI is green.
Nothing blocking. Smaller notes:
- The time-bound facts (what is graded where, what is not yet in
routing_room) now appear in both AGENTS.md anddocs/skill-quality.md, so they can drift apart. Two inline comments below cover the ones I'd tighten. - "Reading a run" and "Reading a change" overlap a little; they could be merged.
- Pre-existing, not from this PR: AGENTS.md suggests
skillscope@v0.1.0for local repro while amd/skills pins v0.1.3.
| been graded, and not otherwise. | ||
|
|
||
| Running `routing` and `behavioral` from this repository's own `skill-evals` job as well, | ||
| with the credential `amd/skills` uses, is planned as a separate change. Until it lands, |
There was a problem hiding this comment.
"is planned as a separate change" reads as a commitment, and I couldn't find it recorded in the ticket. Either confirm it is intended, or reword to something like "is not done yet".
|
|
||
| It needs the skill in the room. A routing run installs the skills in `routing_room` | ||
| side by side and grades which one fires, so a skill that is not listed there is not | ||
| routed. `rocm-doctor` was not listed when this was checked, which is why no routing |
There was a problem hiding this comment.
This "not listed when checked" fact is repeated in AGENTS.md. Consider giving it one canonical home and linking from the other, so the two don't drift when routing_room changes.
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 294cc6a
Review — needs work
Full review of the whole change (3 files, +176/-16: .github/workflows/ci.yml, AGENTS.md, new docs/skill-quality.md).
Refs ROCMAI-501 (retrieved from Jira): a tracking task whose "Done when" checklist (pins recorded per run, a kept series, a measured run-to-run spread, scheduled graded runs) is explicitly not implemented here — the PR's own text says so ("None of those are in this change"), so this is correctly scoped as a partial, informational step toward that ticket rather than a claim to close it.
Blocking
- AGENTS.md and the
ci.ymlcomment both overstate thatroutingalready grades this skill, contradicting the PR's own more careful text two paragraphs later and the new doc it points to —AGENTS.md:327-328,.github/workflows/ci.yml:1326-1328
AGENTS.md:327-328reads: "Those bump pull requests are also whereroutingandbehavioralgrade the skill" — grouping both graders as currently running.ci.yml:1326-1328's comment says the same: "They run in amd/skills on every federation bump of this skill". ButAGENTS.md:365(added by this same PR, same section) says the opposite: "routingdoes not yet [grade the skill], because the skill is not in that repo'srouting_room" — and the newdocs/skill-quality.md("Where it is graded" / "What is not available yet") is explicit that no routing figure exists forrocm-doctortoday. So within this one PR, one passage says routing runs on every bump and another says it doesn't run at all — not "runs but has no figure yet," but doesn't execute. A reader of the first bullet alone (which is how AGENTS.md introduces the topic) would conclude routing coverage already exists, which is false by the PR's own later text. This is the same category of staleness the PR exists to fix, now reintroduced in a different spot.
Confidence 85 · logic · Fix: in both locations, say plainly thatbehavioralgrades the skill on every bump whileroutingdoes not yet (not inrouting_room), instead of grouping the two together as already running.
Non-blocking
None
Decisions for the author
- The same time-bound facts (what's graded where, what
routing_roomomits) are now stated in bothAGENTS.mdanddocs/skill-quality.md— tradeoff
Keeping a short summary inAGENTS.mdmakes it self-contained for a reader who never opens the new doc; routing all detail todocs/skill-quality.mdand keepingAGENTS.mdto a bare pointer would remove the duplication but cost that self-containedness. The Blocking finding above shows the duplication has already drifted once in this very diff, which is evidence for tightening this rather than a hypothetical risk. Confirm the current balance (short summary + pointer) is still the intended tradeoff, and consider whether the summary side needs a lighter touch given how easily it drifts from the detailed source.
Positive signals
- The new
docs/skill-quality.mdis unusually tight for a 157-line doc: no padding, no fragile exhaustive enumerations (it describes case counts qualitatively rather than hardcoding numbers that will drift), and it is internally consistent across "Where it is graded," "Errored cases," "What pins a run," and "What is not available yet" — all of which agree with each other on the one point the edited files above got wrong. - Commit history is three small, well-described, atomic commits ("say where...", "say grading... is planned, not ruled out", "say which... is ours to change") — each a clear incremental correction, good commit hygiene.
- The PR description is unusually rigorous: it dates every external claim, states exactly what remains missing and defers it to follow-up PRs against
amd/skillsrather than scope-creeping this change, and explicitly says what was and wasn't run (cargo test --workspace, clippy,smoke_local.pyskipped with a stated reason).
Deployment notes
None — documentation and a workflow comment only; no step, trigger, permission, or binary behaviour changed.
What this covered
Read completely: all three changed files' full diffs plus surrounding context (AGENTS.md §7 in full, the skill-evals job and its comment block in ci.yml, the entire new docs/skill-quality.md), against base 6ecb30a1…head 294cc6ac. Two independent fan-out workers each reviewed a file grouping and their findings were consolidated here; the ai-instruction-quality skill was activated and applied to the AGENTS.md edits (its "no contradictory directives" Tier-1 check independently corroborates the Blocking finding above). Impact analysis: searched the repo for remaining stale references to the old framing (Phase-1, staging/rocm-doctor, hand-sync) — none found outside the files already updated; confirmed all of the new doc's internal links/paths resolve; confirmed the ci.yml change is comment-only (no steps/triggers/permissions touched); spot-checked the doc's "14/14 individual expectations" figure against the actual skills/rocm-doctor/evals/evals.json fixture — matches exactly. Read CI for the reviewed commit directly: all 23 checks green, including Skill checks (skillscope), Markdown links resolve (lychee), and License header check.
Could not verify: the underlying factual claims about the external amd/skills/amd/skillscope repositories (actual federation.json contents, routing_room membership, the nightly federate-skills workflow's behavior, the cited 28 Oct run results) — this review is scoped to ROCm/rocm-cli only and has no access to those repositories, so these are reported as internally consistent with this repo's own files, not as independently confirmed external facts.
Did not run: a dedicated "prior changes" pass (no earlier review threads on these exact files were examined beyond what git history shows). This PR already carries one prior human approving review; per this review's own ground rules it was not read for content, only confirmed to exist, and is not reconciled against here.
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 84c73ca
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (3 files, +174/-17: .github/workflows/ci.yml, AGENTS.md, new docs/skill-quality.md), at base 6ecb30a1…head 84c73ca.
Refs ROCMAI-501 (retrieved from Jira): a tracking ticket whose "Done when" checklist (a routing figure, a kept series, pins per run, a measured run-to-run spread, scheduled graded runs, publication beside ROCMAI-500) is explicitly not implemented here — the PR's own text says so ("None of those are in this change") — and the ticket's own latest comment (by this PR's author) already corrects the ticket's stale premise and links to this PR. This change corrects a stale documentation claim and records how the two existing figures are read; it does not claim to close the ticket.
Blocking
None
Non-blocking
None
Decisions for the author
- The same time-bound facts (what is graded where, what
routing_roomomits) are stated in both a short summary (AGENTS.md, theci.ymlcomment) and the detailed source (docs/skill-quality.md) — tradeoff
Keeping a short summary inAGENTS.md/ci.ymlmakes each self-contained for a reader who never opens the new doc; routing all detail todocs/skill-quality.mdand leaving the other two as a bare pointer would remove the duplication but cost that self-containedness. This PR's own commit history (ecd9f2c→2cd7dc0→294cc6a→84c73ca) shows the two drifted apart once already within this very diff and the last commit, "keep what runs where in one place," was written specifically to pull them back in line — concrete evidence this duplication is a real, demonstrated risk rather than a hypothetical one. At the current head the three files were independently re-checked (two blind fan-out workers plus a direct read here) and found consistent with each other. Confirm the current balance (short summary + pointer) is the intended tradeoff going forward, and whether the summary side needs an even lighter touch given how easily it drifted.
Positive signals
- Every external claim about
amd/skills(.github/federation.jsonlistingROCm/rocm-cli→skills/rocm-doctor, the nightlyfederate-skillscron and its.federated.jsonfields,evals.yml'srouting_roomomittingrocm-doctor, the weekly schedule forcingmode: references, theskillscopepin) was independently verified by fetching and reading that live repository directly, not assumed from this PR's say-so — and all of it matched exactly, including matching the correction posted on ROCMAI-501 itself. - Commit history is four small, well-described, atomic commits where the last one explicitly exists to fix a contradiction an earlier commit in the same PR introduced — visible evidence the author re-read their own draft rather than trusting it, which is exactly the class of defect this method looks hardest for.
- The new
docs/skill-quality.mdis unusually tight for a 155-line doc: no padding, no hardcoded figures that will go stale (case counts are described qualitatively), explicit "What pins a run" and "What is not available yet" sections, and every statement aboutamd/skillsis dated with the exact files to re-read. - The PR description is unusually rigorous: it dates every external claim, states exactly what remains missing and defers it to follow-up PRs against the correct external repo rather than scope-creeping this change, and says plainly what verification was and wasn't run and why.
Deployment notes
None — documentation and a workflow comment only; confirmed the ci.yml change touches only a comment block (no step, trigger, or permission in the skill-evals job changed).
What this covered
Read completely: AGENTS.md §7 in full, the entire skill-evals job block in .github/workflows/ci.yml (comment and job YAML together), and all of the new docs/skill-quality.md. Two independent blind fan-out workers reviewed this in parallel — one the three files as one tightly-coupled group (shared contract: all three must agree on what runs where), one running the agent-instruction-adherence (via ai-instruction-quality, activated against the AGENTS.md diff), history, prior-changes and code-comments passes — plus consolidation, impact analysis and design review here. Impact analysis: grepped the full repository tree for stale terms this PR retires (Phase-1 incubation framing, staging/rocm-doctor, hand-sync, the old "no such secret" framing) — the only remaining hits are unrelated, pre-existing uses of ANTHROPIC_API_KEY for a different feature (the dash chat seam's own credential handling), confirmed by reading those call sites directly. Verified every internal markdown link/path the diff touches resolves in the PR's tree. Read CI for the reviewed commit directly: all 23 checks green, including Skill checks (skillscope), Markdown links resolve (lychee), License header check (hawkeye), and Commit signatures + sign-off.
Did not run: cargo test -p xtask / cargo test -p rocm advised were not independently re-executed here — CI's green status for the reviewed commit was taken at face value instead, per this method's rule for already-reported checks. Prior-changes pass was partial: author tooling in that worker's sandbox could query the forge for other PRs touching these files (none found revising this section before) but could not reliably check broader review-thread history beyond git log, which is the expected limit of that pass. Not reconciled against the existing "nowycondro" review at commit 294cc6ac — this review was conducted fully blind of it. That review's one blocking finding (a contradiction between AGENTS.md/ci.yml claiming routing already grades this skill on every bump, versus the same section and the new doc saying it does not) is addressed by this head's final commit, 84c73ca ("keep what runs where in one place") — confirmed independently here, not by reading that review's text.
AGENTS.md section 7 and a comment on the skill-evals job said amd/skills has no automated federation for rocm-doctor, and that routing and behavioral cannot run because this repo holds no credential. Both are out of date: amd/skills federates ROCm/rocm-cli nightly, opens a bump pull request when skills/rocm-doctor changes, and grades it there with its own credential. behavioral runs for the skill today; routing does not, because the skill is not in that workflow's routing_room. Fix both statements, and add docs/skill-quality.md: what the two figures are and how each counts an errored case, what pins a run, how to read one, what is not available yet (a routing figure, a kept series, a scheduled graded run, a measured run-to-run spread), and how a change to what is graded is made. Statements about amd/skills are dated and name the files to re-read. The ci.yml edit is a comment only. No scenario is needed: nothing a CLI user can observe changes. Refs: ROCMAI-501 Signed-off-by: Eugene Volen <eugene.volen@amd.com>
docs/skill-quality.md said routing and behavioral run in amd/skills because this repo holds no credential, which reads as a decision never to run them here. Running them from the skill-evals job with the credential amd/skills uses is still planned as a separate change; say so, and that the page describes what runs today. Refs: ROCMAI-501 Signed-off-by: Eugene Volen <eugene.volen@amd.com>
… not docs/skill-quality.md said a change to how rocm-doctor is graded is ours to propose, which blurred two things. This repo owns the skill and its dataset and changes them here. It does not own the amd/skills workflows that grade them, which belong to that repo's maintainers, so a routing_room entry, a repeated run to measure the spread, or an uploaded report are requests to them, or pull requests they review and may decline. Say so. Refs: ROCMAI-501 Signed-off-by: Eugene Volen <eugene.volen@amd.com>
Review of the previous commits found AGENTS.md and the ci.yml comment saying that routing and behavioral both grade this skill on every federation bump, while the same change and docs/skill-quality.md say routing does not. The two had drifted within one diff. Stop restating the time-bound facts outside docs/skill-quality.md. AGENTS.md and the ci.yml comment now say only that amd/skills grades the skill and point to the doc for what runs there, which is the one place kept current. Also say that grading from this repo's own skill-evals job is "not done yet" rather than "planned", since that intent is not recorded in the ticket, and fold the short "Reading a run" section into "What pins a run", which it overlapped. Refs: ROCMAI-501 Signed-off-by: Eugene Volen <eugene.volen@amd.com>
84c73ca to
93fb7d2
Compare
|
Rebased onto current I re-read the |
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 93fb7d2
Review — needs work
Full review of the whole change (3 files, +168/-10: .github/workflows/ci.yml, AGENTS.md, new docs/skill-quality.md), at base df1548a2…head 93fb7d27.
Refs ROCMAI-501 (retrieved from Jira): a tracking ticket asking for two skill-grading figures to be recorded, with pins, a kept series, a measured run-to-run spread, and a regular schedule. This change does not claim to close it — it corrects a stale documentation claim and writes down how to read the figures that already exist elsewhere, and its own text says the remaining items are not implemented here. That matches the ticket's state and the scope this PR's description claims for itself.
Blocking
- The new doc contradicts itself on whether a figure can move with nothing underlying it changed —
docs/skill-quality.md:13-15,docs/skill-quality.md:122-125
Lines 13-15 state a flat rule: "A figure can only move when the skill's prose, its dataset (evals.json), the grading model or the harness changes, never because of anything a user did." Lines 122-125, under "What is not available yet," say the opposite about an unchanged skill: "The judge is non-deterministic and is the same model family as the run, so two runs of an unchanged skill can differ. Until the spread has been measured on an unchanged commit, no single movement can be called a regression." A reader who takes the opening sentence at face value — which is the first thing the doc tells them, and the sentence most likely to be remembered — would treat any two runs of an unchanged skill as directly comparable, which is exactly the mistake the doc's own later section warns against. The two passages need to agree: either the opening list gains judge noise as a cause of movement, or its claim is scoped to "a deliberate figure" rather than "a figure."
Confidence 85 · logic · Fix: add judge non-determinism to the opening paragraph's list of causes, or narrow its claim so it doesn't read as excluding run-to-run noise.
Non-blocking
None
Decisions for the author
AGENTS.mdand theci.ymlcomment both still restate the same one-line reason ("need a credential this repo does not hold") thatdocs/skill-quality.mdalso states — tradeoff
This phrase is unchanged by this PR's final commit (confirmed by diffing the two preceding commits against head) and appears identically in all three files, so it isn't itself a site of drift — the detailed, externally-controlled specifics (whetherrouting/behavioralrun on every bump,routing_roommembership) are the ones this PR's last commit moved solely into the doc, and that cleanup is complete. Keeping the short, stable reason in all three places buys self-containedness for a reader who never opens the doc; folding it into a bare pointer would remove the duplication but cost that. Worth a deliberate call given this exact section has needed more than one correction pass before: confirm the current split (stable reason restated, volatile specifics centralized) is the intended line, not just where the diff happened to stop.docs/skill-quality.mdhas no automated check guarding its own internal consistency — non-blocking-improvement
This repo already treats related text as a fixture that a test holds to the code (thereference.mdcatalog table, thecontract_versionand print-only-count sentences inSKILL.md, perAGENTS.md§7). The new doc has no analogous check, so the kind of self-contradiction flagged above can only be caught by a human or model re-reading the whole page, not by CI. Not a reason to block this PR, but worth considering if this doc grows.
Positive signals
- The branch's own final commit ("keep what runs where in one place") demonstrably fixes a real contradiction its own earlier commits introduced — diffing the penultimate commit against head shows AGENTS.md and the
ci.ymlcomment dropped a specific claim thatroutingandbehavioralboth "grade the skill" on every bump, replacing it with the vaguer, accurate "amd/skills grades the skill instead" — visible evidence the author re-read their own draft rather than trusting it. - The doc's illustrative "14/14 individual expectations" figure was checked directly against
skills/rocm-doctor/evals/evals.jsonand matches exactly (5 qualifying cases, 14 expectations total) — a concrete claim that holds up, not just a plausible-sounding one. - Impact analysis found no leftover references to the retired framing ("hand-sync", a
staging/copy, "no such secret") anywhere else in the repository, and the new doc's placement (cross-linked rather than added to the Sphinxtoctree) matches how this repo's other internal engineering docs are already discovered. - The PR description dates every external claim, states plainly what remains missing and routes it to the external repo that owns those workflows rather than scope-creeping this change, and says what verification was and wasn't run and why.
- All four commits carry a sign-off and a verified signature.
Deployment notes
None — documentation and a workflow comment only; the ci.yml edit touches only a comment block, confirmed no step, trigger, or permission in the skill-evals job changed.
What this covered
Read completely: all of AGENTS.md (full file, with §7 as the focus), the entire skill-evals job block in .github/workflows/ci.yml (comment and job YAML together), and all of the new docs/skill-quality.md, against base df1548a2…head 93fb7d27. Three independent blind fan-out workers each cloned the repository read-only (fetch only, no checkout of the PR branch): one ran the coupled-group code pass over all three files; one ran the agent-instruction-adherence, history, and prior-changes passes (confirming via git log that this exact AGENTS.md/ci.yml area has required correction before, and that this branch's own earlier commits drifted and were then self-corrected); one ran impact analysis (repo-wide grep for stale leftovers, the evals.json fixture check, doc-discoverability check, commit-quality check). Findings were consolidated and re-verified directly against the file text and a direct diff of the branch's penultimate commit against its head. Read CI for the reviewed commit directly: all 23 checks green, including Skill checks (skillscope), Markdown links resolve (lychee), License header check (hawkeye), and Commit signatures + sign-off; the Read the Docs build for this commit also succeeded.
Did not run as a distinct pass: "code comments" — the only touched file carrying guidance-bearing comments is ci.yml, and its comment block was read in full as part of the code pass rather than as a separate pass. Not independently re-executed: the cargo nextest runs the PR description cites; CI's green status for this commit was taken at face value instead. Not reconciled against any existing discussion on this PR — conducted fully blind of it, per this method's rules.
Why
AGENTS.md§7 and a comment on theskill-evalsjob inci.ymldescribe a setup thatno longer exists. They say
amd/skillshas no automated federation forrocm-doctor(the skill is hand-synced into
staging/), and thatroutingandbehavioralcannotrun because this repo has no credential, with the dataset left ready for the day they
can.
Checked against
amd/skillstoday, both are out of date:.github/federation.jsonthere declaresROCm/rocm-cliwithskills/rocm-doctor,and the nightly
federate-skillsworkflow opens a "Bumprocm-doctor" pull requestwhenever this folder changes on
main. The vendored folder's.federated.jsonrecordsthe source commit and a content hash.
evalsworkflow grades that pull request with skillscope, with that repo's owncredential. On the 28 Oct bump
(run)
behavioralran forrocm-doctoron Linux and Windows (5/5 cases, 14/14 expectations,opus, efforthigh) androutingwas skipped, because the skill is not in that workflow'srouting_room.So the figures ROCMAI-501 asks for are partly being produced already, by a pipeline this
repo does not describe anywhere. Anyone reading
AGENTS.mdtoday would conclude nothinggrades the skill, and would build a second pipeline to do it.
What changes
AGENTS.md§7: the source-of-truth bullet now describes the federation bump, andthe paragraph about
routingandbehavioralsays only thatamd/skillsgrades theskill and points to
docs/skill-quality.mdfor what runs there..github/workflows/ci.yml: the comment onskill-evalssays the same. Commentonly; no step, trigger or permission changes.
docs/skill-quality.md(new): what the two figures are (first-run success rate androuting accuracy, with the harness's own rules for how each counts), how an errored case
is counted (it differs between the two graders), what pins a run and how to read one,
what runs where, what is not available yet, and how a change to what is graded is made. It is dated where a
statement is about
amd/skills' workflow, with the files to re-read.What it says is missing
Written down so it is not rediscovered: no routing figure for
rocm-doctor; no keptseries, since the run's JSON report is not uploaded as an artifact; no scheduled graded
run, since the weekly schedule there is references-only on purpose; and no measured
run-to-run spread. The doc proposes how each would be closed, as pull requests to
amd/skills. None of those are in this change.Verification
prekhooks pass on the three files, including the license-header check and themarkdown-link check (
lychee). Leak scan of the diff is clean.amd/skillswas read from that repository on 2026-10-30:federation.json, the vendored.federated.json,evals.yml(routing_room, theskillscope pin, the weekly
referencesmode) and the run above.main:cargo nextest run -p xtask(330 passed, including theworkflow-contract tests that parse
ci.yml) andcargo nextest run -p rocm --bin rocm -E 'test(advised)'(20 passed, the scanner that checks everyrocmcommand inREADME.md,docs/andskills/), on Windows.cargo test --workspace, clippy andsmoke_local.py. Nothing in Rust, scripts orworkflow behaviour changed (the
ci.ymledit is a comment), so CI is the authoritativecheck here.
Scenario / regression test
None, per AGENTS.md §3: this is documentation and a workflow comment, with no change to
what a user of the CLI can observe.
Refs ROCMAI-501.