Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
3018871
test(rocm-core): cover the fix consent and exit-code contract
volen-silo Sep 8, 2026
4c2df68
fix(diagnose): correct fix-9's auto flag and drop unreachable routing
volen-silo Sep 8, 2026
2832b0d
test(e2e): publish the rocm-doctor skill and test its CLI contract
volen-silo Sep 8, 2026
ff0b9f3
fix(doctor-skill): gate on has_match, and check the contract both ways
volen-silo Sep 8, 2026
d877bbd
test(e2e): index the doctor feature, and pin has_match where the host…
volen-silo Sep 8, 2026
2eea19c
docs(doctor-skill): stop promising a dry run fix-2 never performs on …
volen-silo Sep 8, 2026
cdb0b7b
fix(rocm-doctor): correct fix-9 scope, federation, and test-claim wor…
volen-silo Sep 11, 2026
b0c0753
docs(doctor-skill): document fix-17 so the catalog matches the CLI
volen-silo Sep 11, 2026
5134f6a
docs(doctor-skill): bring the published catalog onto the WSL2 platfor…
volen-silo Sep 11, 2026
7545482
docs(doctor-skill): document fix-19 so the catalog matches the CLI
volen-silo Sep 25, 2026
f219d17
ci: grade the rocm-doctor skill with skillscope (#356)
volen-silo Sep 25, 2026
d00d86a
fix(doctor-skill): carry WSL2's in-scope move to the surfaces it missed
volen-silo Sep 25, 2026
b3e70f6
fix(doctor-skill): stop the WSL2 eval and the drift test over-claiming
volen-silo Sep 25, 2026
ca06fae
fix(doctor-skill): drop a repeated note and two silent traps
volen-silo Sep 25, 2026
e07358f
fix(diagnose): reattach a test doc comment to the test it describes
volen-silo Sep 25, 2026
d011945
fix(rocm-core): fix-2's Windows arm never returns 5 on decline
volen-silo Sep 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ jobs:
lint: ${{ steps.filter.outputs.lint || steps.all.outputs.forced }}
tpn: ${{ steps.filter.outputs.tpn || steps.all.outputs.forced }}
docs: ${{ steps.filter.outputs.docs || steps.all.outputs.forced }}
skills: ${{ steps.filter.outputs.skills || steps.all.outputs.forced }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

Expand Down Expand Up @@ -136,6 +137,11 @@ jobs:
# The containerised control-channel lane; its Dockerfile matches
# none of the extension globs above.
- 'tests/remote-ssh/**'
# skills/rocm-doctor/reference.md is an e2e FIXTURE: the
# rocm_doctor_skill.feature scenarios parse its catalog table and
# compare it to `rocm fix`. A skill-only edit must therefore run
# the E2E job, which gates on `heavy`.
- 'skills/**'
Comment thread
volen-silo marked this conversation as resolved.
Resolved
- 'install*'
# Pinned-key consistency compares docs/keys/* against the installers,
# so a canonical-key-only change must trigger the heavy job that runs
Expand Down Expand Up @@ -173,6 +179,13 @@ jobs:
- '.readthedocs.yaml'
- '.github/workflows/**'

# Published Agent Skills, graded by skillscope. The dataset and the
# skill's prose are one unit — a prompt can stop matching because the
# description changed — so any edit under skills/ re-runs the check.
skills:
- 'skills/**'
- '.github/workflows/**'

- name: Force full run off pull requests
id: all
if: github.event_name != 'pull_request'
Expand Down Expand Up @@ -1143,6 +1156,88 @@ jobs:
if: needs.changes.outputs.lint == 'true'
run: cargo xtask powershell-lint --shell powershell

# Grades the published Agent Skills under skills/ with AMD's skillscope
# harness. Only the `structural` command runs: it needs no agent, no API key
# and no network beyond the install, so a red run is never wrong for a
# reason of its own. It is advisory today, not blocking: `Skill checks
# (skillscope)` (this job's `name:`, not the `skill-evals` job id) is not a
# required status check in branch protection, so a failing run does not
# stop a merge. An admin must add that exact context to make it blocking.
# It checks the frontmatter an agent runtime
# parses (a `name` that disagrees with the folder makes the skill unloadable),
# the dataset's coverage bar, and every internal markdown reference — a link
# into a renamed file is not cosmetic here, because the agent follows it,
# finds nothing, and improvises.
#
# `routing` and `behavioral` are deliberately NOT run: both need an
# authenticated `claude` CLI and an ANTHROPIC_API_KEY, and this repo has no
# such secret. Turning them on is a caller change to skillscope's
# `skill-evals.yml` reusable workflow once one exists — they are the half of
# the question that asks whether the skill actually fires, so this job is not
# a substitute for them.
#
# skills/rocm-cli-assistant/ is deliberately out of scope. Despite its path it
# is not a published skill: apps/rocm/src/main.rs embeds it verbatim into the
# chat assistant's system prompt with include_str!, so giving it the YAML
# frontmatter this check requires would inject that frontmatter into a live
# prompt.
skill-evals:
name: Skill checks (skillscope)
runs-on: ubuntu-latest
timeout-minutes: 10
needs: changes
# A manual E2E dispatch only exercises the E2E jobs; skip the rest.
if: github.event_name != 'workflow_dispatch' && needs.changes.outputs.skills == 'true'
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# The `skillscope` step below names skills/rocm-doctor explicitly, so a
# third skills/<name>/ folder added later would pass this job green
# while never being graded -- the same silent-non-grading failure this
# job exists to catch, just one level deeper. Fail loudly instead: any
# directory under skills/ that isn't graded above or excluded by design
# (rocm-cli-assistant -- see AGENTS.md #7) forces a deliberate edit here.
- name: Guard against an ungraded skills/ directory
run: |
set -euo pipefail
# Without nullglob, a skills/ with no subdirectories leaves the
# pattern unexpanded and the loop runs once on the literal
# `skills/*/`, reporting `skills/* is neither graded ...` -- a
# made-up skill name, from a tree that has no ungraded skill in it.
# With it the loop simply does not run, which is the right answer for
# "nothing to grade".
shopt -s nullglob
known="rocm-doctor rocm-cli-assistant"
for dir in skills/*/; do
Comment thread
volen-silo marked this conversation as resolved.
Resolved
name="${dir%/}"
name="${name#skills/}"
case " $known " in
*" $name "*) ;;
*)
echo "::error::skills/$name is neither graded by skill-evals nor excluded by design ($known). Add it to the skillscope step's 'skills:' input, or add it to this guard's exclusion list with the reason, before merging." >&2
exit 1
;;
esac
done

# The action is only a launcher: it sets up Python and uv, resolves which
# build of the harness to run, and execs it. It imports nothing from
# skillscope, so pinning it to a release SHA cannot break on a payload
# version it predates. What the pin does NOT cover: the composite's own
# steps (setup-python, setup-uv, setup-node) are on their own moving
# tags, and the harness itself is resolved at runtime via
# `uvx --from git+.../skillscope@<ref>` with no lockfile. Unlike this
# repo's usual SHA pin (AGENTS.md #6), this one does not reach the code
# that actually runs. install-claude stays off; structural needs no
# agent.
- name: Check skill structure, dataset, and references
uses: amd/skillscope@6dde8e8a34ad5456d3c2ff3418bc81c2cd9d5d69 # v0.1.0
with:
command: structural
skills: skills/rocm-doctor
args: --skill-files skill-card.md --skill-sections Description,Owner,License
install-claude: "false"

license-headers:
name: License header check (hawkeye)
runs-on: ubuntu-latest
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/e2e-selfhosted.yml
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,11 @@ jobs:
- '**/*.ps1'
- '**/*.feature'
- 'tests/e2e-cucumber/**'
# skills/rocm-doctor/reference.md is an e2e FIXTURE: the
# rocm_doctor_skill.feature scenarios parse its catalog table and
# compare it to `rocm fix`. A skill-only edit must therefore run
# this job too, same as ci.yml's `heavy` filter.
- 'skills/**'
- 'install*'
- 'docs/keys/**'
- '.github/workflows/**'
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ active-implementation-log.md
current-user-asks.md
__pycache__/
/.hawkeye-bin/
# skillscope writes its JSON run reports here.
/.skillscope/
/.claude
/plans/
/workspace/
Expand Down
57 changes: 57 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ When changing assistant-adjacent behavior, keep consistency with:

- `docs/llm-tool-use.md`
- `skills/rocm-cli-assistant/SKILL.md`
- `skills/rocm-doctor/SKILL.md` and `skills/rocm-doctor/reference.md`

Required consistency points:

Expand All @@ -224,6 +225,62 @@ Required consistency points:
- avoid invented shell/package-manager commands in assistant behavior paths
- preserve built-in assistant constraints and no-CPU-fallback policy

### `skills/rocm-doctor/` — published from here, and a test fixture

`skills/rocm-cli-assistant/SKILL.md` is compiled into the binary
(`include_str!` in `apps/rocm/src/main.rs`). `skills/rocm-doctor/` is different
on two counts, and both change how you edit it:

- **This repo is its source of truth.** The skill is a thin driver over the
`rocm` binary, so it is versioned with the binary and lives here.
[`amd/skills`](https://github.com/amd/skills) is still in Phase-1
incubation for this skill: it carries no automated federation for
`rocm-doctor` yet — `.github/federation.json` there only declares
`AMD-AGI/TraceLens` as a source, and this skill instead sits under
`staging/rocm-doctor`, outside any job's coverage. Until federation picks it
up, the rocm-cli team hand-syncs `staging/rocm-doctor` from this folder
whenever it changes materially. So edit it here, and never edit the
`amd/skills` copy directly — the next hand-sync overwrites it.
- **`reference.md` is an e2e fixture.** `tests/e2e-cucumber/features/rocm_doctor_skill.feature`
parses its closed-catalog table and compares it to what `rocm fix` reports.
The failure catalog itself is authoritative in `crates/rocm-core/src/fix.rs`
(the `RECIPES` list) and `crates/rocm-core/src/diagnose.rs` (each mode's
checker and OS scoping) — adding, renaming, or re-scoping a failure mode
means changing the CLI **first**, then the two docs. That feature is what
catches you if you forget.

The folder is excluded from `licenserc.toml`: `SKILL.md` must open with YAML
frontmatter for the skill loader, and skills published this way carry no
license headers of their own. The licence is stated in `skill-card.md`
instead.

Two checks gate it, and they cover different things:

- **`skill-evals` (skillscope, advisory today)** — the frontmatter an agent
runtime parses, the `evals/evals.json` coverage bar (at least 3 prompts that
should trigger the skill and 2 near misses that should not), the
`skill-card.md` sections, and every internal markdown link. It is not yet a
required status check in branch protection: an admin must add its exact
context, `Skill checks (skillscope)` (the job's `name:`, not the
`skill-evals` job id), before a red run actually blocks a merge. Reproduce
it locally with
`uv tool install git+https://github.com/amd/skillscope@v0.1.0`, then
`skillscope structural --skills-dir 'skills/rocm-doctor' --skill-files
skill-card.md --skill-sections Description,Owner,License`. Add `--external`
to check the outbound URLs too; CI does not, because a rate-limited host is
not a broken link.
- **`rocm_doctor_skill.feature` (e2e, blocking)** — whether the prose still
describes the binary, as above.

Neither grades whether the skill actually *fires*. That is skillscope's
`routing` and `behavioral`, which need an authenticated `claude` CLI and an
`ANTHROPIC_API_KEY` this repo does not have. The dataset is written and checked
so those can be switched on without further work.

`skills/rocm-cli-assistant/` is **not** in scope for skillscope: it is embedded
verbatim into the chat system prompt with `include_str!`, so the YAML
frontmatter a published skill needs would end up inside that prompt.

## 8) Verification Matrix For This Repo

Minimum quality gate before upstream-ready status:
Expand Down
Loading