Repository navigation
fix(pack): certify manifestless marketplace skills - #3064
Daniel Meppiel (danielmeppiel) merged 10 commits into
Conversation
Treat a missing remote apm.yml as certifiable only after SKILL.md is verified at the same resolved repository, ref, and subdirectory. This fixes the HTTP 404 outcome collapse while preserving fail-closed behavior for missing repositories, refs, and package paths.\n\nCloses microsoft#3055\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Require non-empty Agent Skills name and description frontmatter before certifying a remote SKILL.md, and cover repository, ref, path, authentication, and network failure controls.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved builder findings and documentation/changelog updates must be addressed before approval.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Adds fail-closed certification for GitHub/GHES manifestless marketplace skills by validating SKILL.md when apm.yml is absent.
Changes:
- Adds
manifestlessmetadata status and fallback probing. - Adds regression tests and architecture-linter checks.
- Updates documentation, usage guidance, and changelog.
| File | Summary / review notes |
|---|---|
tests/unit/marketplace/test_metadata_enrichment.py |
Adds certification and failure-path coverage. |
tests/unit/marketplace/test_builder.py |
Adds GHES and raw/REST fallback tests. |
src/apm_cli/marketplace/builder.py |
Implements manifestless certification. Moderate findings require REST ordering and additional malformed, invalid, authentication, and network failure coverage. |
scripts/architecture_linter/checks/marketplace_tag_and_version.py |
Enforces metadata-owner invariants. |
packages/apm-guide/.apm/skills/apm-usage/commands.md |
Synchronizes usage guidance. |
docs/src/content/docs/reference/cli/pack.md |
Documents the new status; a nit remains for stale normative references. |
docs/src/content/docs/producer/releasing-from-any-ci.md |
Clarifies exit-code behavior. |
docs/src/content/docs/producer/publish-to-a-marketplace.md |
Documents manifestless publishing. |
CHANGELOG.md |
Records the fix; the entry should use the PR number rather than the issue number. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> # Conflicts: # CHANGELOG.md
Cover malformed frontmatter and missing, non-string, empty, or blank name and description values through the raw and REST metadata paths. Assert failed, uncertifiable outcomes and pack --check-clean exit 4 so arbitrary remote content cannot be certified silently. Clarify the existing fail-closed contract in the pack reference and usage guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mohammed Alkindi (MohammedAlkindi)
left a comment
There was a problem hiding this comment.
_rest_url falls back to https://{effective_host}/api/v3, which drops the old https://api.github.com default for github.com. With _host_info unset, a token, and raw 404s for both files, _fetch_remote_metadata now requests https://github.com/api/v3/repos/acme/my-tool/contents/apm.yml; on main the same probe hits api.github.com. Keeping the github.com default inside _rest_url would restore it.
Preserve current main and contributor history before validating the bounded marketplace certification repair. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f79e464c-180e-4be3-9056-0773aa679a1a
APM Review Panel:
|
| Persona | B | R | N | Takeaway |
|---|---|---|---|---|
| auth-expert | 0 | 0 | 0 | API-base fallback for unclassified github.com host correctly restored; no auth bypass or credential risk introduced. |
| test-coverage-expert | 0 | 0 | 0 | Prior host_info=None REST fallback gap now covered by parametrized unit test over classified/unclassified x manifest/skill; mutation evidence confirms guard is exercised; no new gaps. |
B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.
Recommendation
All code findings from the initial lean panel are resolved in this delta cycle. The auth-expert and test-coverage-expert both return zero findings; the API-host fallback is restored and regression-trapped with mutation-verified tests. Exact-head technical CI is now observed green. The maintainer retains the independent final merge decision. No follow-up items remain on the code side.
Folded in this run: 3085cacccd2830c34b4708f777fa27df7a67af6e restores the GitHub REST default, adds classified/unclassified manifest/skill cases, and synchronizes the pack reference and packaged guide. Accepted upstream guide and README changes are preserved.
Copilot signal reviewed: comment 4080862807 was legitimate and already addressed by the contributor in 1c27e37be83ada0d3d3d57a407389cbafff6be4f; those malformed/non-string/blank frontmatter cases passed in this run. No scope deferrals remain.
Exact-head local validation: 9f2a28118deb38c7ce339133b1578f7e2af91755, tested with main f65ac26356353304f6a333ee738636be7b9206b4.
uv run --extra dev pytest tests/unit/marketplace/test_metadata_enrichment.py tests/unit/marketplace/test_builder.py tests/integration/test_architecture_marketplace_metadata_enrichment.py tests/integration/test_architecture_marketplace_remote.py -q --basetemp=/Users/danielmeppiel/.copilot/session-state/f79e464c-180e-4be3-9056-0773aa679a1a/files/pr-workers/pr-3064-evidence/merge-refresh/pytest: 221 passed in 30.85s.- All seven current canonical CI lint gates and
bash scripts/lint-architecture-boundaries.shexited 0. GNUggrepsupplied the unchanged CI expressions on macOS. - Deterministic exact-base/head owner detection and version-2 completion semantic verification passed for the existing marketplace metadata-certifiability owner.
- The prior mutation proof is retained: guard and regression-test file bytes are unchanged. Removing the GitHub-specific API default had produced two failures; restoring it passed all four combinations. The unchanged tests were rerun within the fresh 221-test suite.
Observed-green CI: all six exact-head technical workflows completed successfully and their real jobs were inspected: CodeQL, docs, CI, required-check aggregation, NOTICE, spec conformance.
History and authority: normal push only; no force push, new review request, repository-policy change, check bypass, or PR merge. The maintainer retains the final landing decision.
Main subsequently advanced to 4037af112df9727df33ab0ac12fe0740001e0a53. A clean read-only merge preview and live mergeability check were recorded separately; the tested head and base above have not been relabeled.
Full per-persona findings
auth-expert
No findings.
test-coverage-expert
No findings.
This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.
Generated by autopilot-pr-merge-worker. This comment is AI-generated and may contain errors.
Restore the github.com API default when host classification is absent. Cover both manifest and manifestless authenticated REST responses, including exact-host/path/ref and credential assertions. Addresses both panel follow-ups and human review 5499353923; removing the restored guard makes both new unclassified cases fail. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f79e464c-180e-4be3-9056-0773aa679a1a
Preserve the latest accepted guide changes without altering the reviewed marketplace implementation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f79e464c-180e-4be3-9056-0773aa679a1a
Preserve the latest accepted upstream documentation without changing the reviewed marketplace fix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f79e464c-180e-4be3-9056-0773aa679a1a
Preserve reviewed metadata fallback and upstream MCP, manifest, documentation, and dependency updates. Automatic merge; no conflict edits or history rewriting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f79e464c-180e-4be3-9056-0773aa679a1a
Preserve upstream changes without altering reviewed manifestless certification behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f79e464c-180e-4be3-9056-0773aa679a1a
ee08bb5
into
microsoft:main
…3251) Restore the release-gate integration suite (nightly + tag) to green: - runtime: route runtime setup/version/uninstall children through external_process_env so a frozen binary's bundled LD_LIBRARY_PATH / DYLD_* never leak into the setup script's python3 (ensurepip failed under the PyInstaller binary). - tests: align fixtures with contracts merged since v0.33.0 -- grok-bot skills_rel_root/supports on MagicMock targets (#3106), Cursor native flat hooks (#3149), frozen installs that never rewrite the lockfile (#2384), manifestless SKILL.md probe (#3064), unknown apm.yml key warning (#3222), resolved Claude config path (#3148), and the build-ID owner guard mutation case (#3113). - flakes: skip transient git maintenance files when snapshotting a bare fixture repo; widen multiprocess lock-test timeouts for slow runners. Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 761bf207-57da-41d0-bd3f-ffecbb97b83a

fix(pack): certify manifestless marketplace skills
TL;DR
apm pack --check-cleannow certifies GitHub-hosted manifestless skills using semantic version ranges whenSKILL.mdwith valid frontmatter is verified at the resolved ref and subdirectory. This resolves an HTTP 404 outcome collapse where the expected absence ofapm.ymlfor valid skills was treated as an uncertifiable remote metadata outage. Missing repositories, deleted refs, absent package paths, and authentication or network errors remain fail-closed.Important
Fixes #3055.
Problem (WHY)
<subdir>/apm.yml.apm.ymlis intentionally absent and returns HTTP 404.failed, causingapm pack --check-clean --dry-runto exit4(marketplace_metadata_uncertifiable) even when the artifact had zero drift.The regression is grounded in deterministic behavior because “Grounding outputs in deterministic tool execution transforms probabilistic generation into verifiable action.”
Approach (WHAT)
manifestlessmetadata enrichment status.src/apm_cli/marketplace/builder.pyapm.yml, probeSKILL.mdat the same resolved repository, ref, and subdirectory.SKILL.mdhas valid Agent Skills YAML frontmatter with non-emptynameanddescriptionstrings.src/apm_cli/utils/yaml_io.py::loads_frontmatterfailedwhenSKILL.mdis also absent, unparseable, or fails authentication/network access.Implementation (HOW)
src/apm_cli/marketplace/builder.pymanifestlessto_METADATA_STATUSESand_CERTIFIABLE_METADATA_STATUSES. In_fetch_remote_metadata, whenapm.ymlreturns 404, probesSKILL.mdat the same resolved path and parses frontmatter withloads_frontmatter(). Valid frontmatter yieldsMetadataEnrichmentOutcome(pkg.name, "manifestless").scripts/architecture_linter/checks/marketplace_tag_and_version.py_check_metadata_enrichmentwith invariant assertions requiring the marketplace metadata owner to parse manifestless skills before certification.docs/src/content/docs/reference/cli/pack.mdmanifestlessstatus in themetadata_enrichmentvocabulary table and details the 404 fallback verification flow.docs/src/content/docs/producer/publish-to-a-marketplace.mdapm.ymlis certifiable whenSKILL.mdexists at the resolved path.docs/src/content/docs/producer/releasing-from-any-ci.md--check-cleanexit code 4 documentation to clarify manifestless acceptance.packages/apm-guide/.apm/skills/apm-usage/commands.mdapm packmetadata vocabulary and rules.CHANGELOG.md[Unreleased]linked to #3055.tests/unit/marketplace/test_metadata_enrichment.pyobra/superpowerswith version range^4.0.0), along with negative regressions for missing skill path, auth failure, network failure, missing repository, and unmatched version.tests/unit/marketplace/test_builder.pyDiagrams
Legend: The dashed elements indicate the new
SKILL.mdverification probe andmanifestlesscertifiable status whenapm.ymlis absent.flowchart LR subgraph Probe[Remote Metadata Probe] M["Fetch apm.yml"] S["Probe SKILL.md"] end subgraph Classify[Classification] F1["status: fetched"] M1["status: manifestless"] E1["status: failed"] end M -->|"200 OK"| F1 M -->|"404 Not Found"| S S -->|"Valid frontmatter"| M1 S -->|"404 or invalid"| E1 M -->|"Auth or network error"| E1 classDef new stroke-dasharray: 5 5; class S,M1 new;Trade-offs
SKILL.mdinstead of accepting bare 404s. Chose to explicitly verifySKILL.mdwith valid frontmatter rather than treating any HTTP 404 onapm.ymlas certifiable, preserving the fail-closed guarantee against deleted repos, invalid refs, or non-existent package subdirectories.SKILL.md; other Git hosts (GitLab, ADO) continue to require fixeddescriptionandversionfields on the marketplace package entry.nameanddescriptionto be certifiable, preventing empty files or API directory listing responses from masquerading as valid skills.Benefits
apm pack --check-clean --dry-runexits 0 on valid manifestless skills using semantic version ranges.apm.ymlfiles or lock to exact version pins.scripts/architecture_linter/checks/marketplace_tag_and_version.py.Validation
uv run --extra dev pytest tests/unit/marketplace/test_metadata_enrichment.py -q:uv run --extra dev pytest tests/unit/marketplace/test_builder.py -k "manifestless or TestFetchRemoteMetadataGHEHost" -q:Canonical CI lint mirror
uv run --extra dev ruff check src/ tests/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/:uv run --extra dev ruff format --check src/ tests/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/:uv run --extra dev python -m pylint --disable=all --enable=R0801 --min-similarity-lines=10 --fail-on=R0801 src/apm_cli/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/:bash scripts/lint-auth-signals.sh:Scenario Evidence
apm pack --check-clean --dry-runsucceeds (exit 0) for a manifestless marketplace skill using a semantic version range whenSKILL.mdis present.tests/unit/marketplace/test_metadata_enrichment.py::test_pack_check_clean_certifies_manifestless_skill_with_version_range(regression-trap for #3055)SKILL.mdat an absentapm.ymlpath remains uncertifiable and exits 4 withmarketplace_metadata_uncertifiable.tests/unit/marketplace/test_metadata_enrichment.py::test_pack_check_clean_rejects_missing_manifestless_skill_pathtests/unit/marketplace/test_metadata_enrichment.py::test_pack_check_clean_rejects_metadata_access_failurestests/unit/marketplace/test_metadata_enrichment.py::test_pack_check_clean_rejects_invalid_remote_or_refSKILL.mdvia Contents API and certifies manifestless skills, but rejects directory responses.tests/unit/marketplace/test_builder.py::TestFetchRemoteMetadataGHEHost::test_metadata_fetch_ghes_certifies_manifestless_skilltests/unit/marketplace/test_builder.py::TestFetchRemoteMetadataGHEHost::test_metadata_fetch_does_not_certify_directory_named_skill_mdHow to test
apm.ymlwith a manifestless skill entry using a version range (e.g.,obra/superpowers, subdirskills/brainstorming, version^4.0.0).apm packto generate the initial marketplace artifact.apm pack --check-clean --dry-runand observe that it exits0with statusmanifestlessinstead of failing with[x] Marketplace clean check failed: remote metadata unavailable(exit 4).apm pack --check-clean --dry-runexits4.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com