Skip to content

ci: run public skill acceptance tests - #4

Open
bruno-archastro wants to merge 1 commit into
ArchAstro:mainfrom
bruno-archastro:bruno/archdev-skill-ci
Open

ci: run public skill acceptance tests#4
bruno-archastro wants to merge 1 commit into
ArchAstro:mainfrom
bruno-archastro:bruno/archdev-skill-ci

Conversation

@bruno-archastro

Copy link
Copy Markdown

Review on ArchCode

Problem and author intent

ArchDev's public skills include executable acceptance scripts, but the repository's only automated checks exercise the Unix and Windows installers. The current lifecycle is:

skill package changes → pull request runs installer smoke tests only → green checks despite the skill test never running

That leaves npx skills packaging, cold bootstrap, and domain workflow regressions outside the merge gate. The intended outcome is one automatic check that discovers every public skill package and executes its matching acceptance test without maintaining a second hard-coded skill list.

What changed

  • Added scripts/run-skill-tests.sh, which discovers skills/*/SKILL.md in deterministic order and runs tests/<skill>-skill.sh for each package.
  • The runner fails if a discovered skill has no matching test or if that test is not executable. Repositories with no skills pass, allowing this CI foundation to land independently before the domain PRs.
  • Added tests/skill-test-runner.sh to prove discovery, execution through a separate shell process, paths containing spaces, fail-closed missing coverage, and the no-skills bootstrap state.
  • Added the Skill Acceptance workflow for pull requests, pushes to main, and manual dispatches. It runs the runner proof followed by all discovered domain tests with read-only repository permissions.

Scope

Backend/tooling-only. This changes public repository CI and does not affect the ArchDev runtime or frontend.

Risk assessment

Low. The new workflow is additive and has read-only permissions. The main risk is CI time as more skills are added; tests run sequentially to avoid shared npx and home-directory contention.

User impact

No direct CLI behavior changes. Contributors receive a failing pull-request check when a public skill is missing coverage or its install/bootstrap workflow regresses.

Testing

Canonical end-to-end proof

tests/skill-test-runner.sh creates a synthetic public skill repository, crosses the runner-to-acceptance-script process boundary, and asserts the script's externally visible log output. It then injects an uncovered skill and verifies that the runner exits nonzero with the expected test path before proving that an empty pre-skill repository remains bootstrap-safe.

Commands run

  • tests/skill-test-runner.sh — passed
  • scripts/run-skill-tests.sh on this branch's no-skills state — passed
  • scripts/run-skill-tests.sh against the Tasks and Jobs branch — both acceptance scripts passed
  • scripts/run-skill-tests.sh against the pending Rooms branch — Rooms acceptance passed
  • Bash syntax checks and git diff --check — passed
  • Workflow YAML parse — passed

Automatic execution path

.github/workflows/skill-acceptance.yml runs on every pull_request, every push to main, and workflow_dispatch. Its Public skill acceptance job first runs tests/skill-test-runner.sh, then scripts/run-skill-tests.sh; there are no path, environment, or platform skip conditions. The domain scripts use Unix bootstraps and therefore run on Ubuntu. Existing installer CI continues to provide separate Windows coverage.

Follow-ups and known issues

This branch intentionally contains no domain skill packages so it can merge independently. As Tasks, Jobs, or Rooms land, the same workflow discovers their committed tests without additional workflow edits.

Agent session: 01a080c2-a41b-758e-a30c-753f9b03b9c9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant