Skip to content

chore(setup): drop the comdev-checkout check from setup-isolated-setup-verify - #1185

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:verify-drop-comdev-check
Sep 9, 2026
Merged

chore(setup): drop the comdev-checkout check from setup-isolated-setup-verify#1185
potiuk merged 1 commit into
apache:mainfrom
potiuk:verify-drop-comdev-check

Conversation

@potiuk

@potiuk potiuk commented Sep 9, 2026

Copy link
Copy Markdown
Member

Summary

  • Removes check 9 (comdev MCP checkout on main, origin is apache/comdev)
    from setup-isolated-setup-verify. That checkout is a personal working copy —
    which branch it sits on, and whether origin names the canonical repo or a
    contributor's fork, are not properties of the secure agent setup the skill
    exists to certify. A contributor carrying a topic branch off their own fork
    was being reported as a broken install.
  • The skill's evals had already settled this: every case under
    tools/skill-evals/evals/setup-isolated-setup-verify/ asserts exactly eight
    checks, numbered 1–8 and ending at project-root coverage, with no comdev case
    anywhere. SKILL.md and its fixtures were out of step; they now agree.
  • Check 9 was last, so checks 1–8 keep their numbers. The prompt-injection
    preamble was written around check 9's parsing of untrusted git output, so
    rather than lose the guardrail with the check it now points at the untrusted
    machine output the remaining checks do parse: the git worktree list --porcelain output read by check 8, settings-file contents, command stderr.

Also updated so nothing claims a check that no longer exists: the canonical list
in docs/setup/secure-agent-setup.md (which the skill defers to) drops the
matching item, and tools/ponymail/tool.md / tools/apache-projects/tool.md no
longer say verify asserts the checkout is on main and not behind.
setup-isolated-setup-update still surfaces the behind-origin/main warning
and prints the pull command — that check keeps living there.

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixture note below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • prek passes on all four changed files — including check-doc-sync
    (declared skill counts) and skill-and-tool-validate, the two hooks that
    would catch a stale "nine checks" claim in the frontmatter.
  • For Python packages touched: n/a
  • For Groovy bridges touched: n/a
  • For skill changes: eval suite not run under --cli. The runner's
    default mode is print-only; all 12 fixtures across the three steps load
    and render cleanly, but this PR carries no live-CLI eval result. Happy to
    run it if a reviewer wants it before merge.
  • For skill behaviour changes: no fixture change needed — the existing
    fixtures already assert eight checks and never covered comdev, so this
    change makes SKILL.md match its regression tests rather than requiring new
    ones. case-6-injection-attempt still exercises the preamble's guardrail.
  • Other:

RFC-AI-0004 compliance

  • HITL — no new mutation
  • Sandbox — strictly a removal; no new host access. Drops the one check
    that resolved a path outside the project from MCP config and shelled git
    against it.
  • Vendor neutralitycheck-placeholders passes; no placeholder prose changed
  • Conversational + correctable — override path unchanged
  • Write-access discipline — skill stays read-only
  • Privacy LLM — n/a

Linked issues

None.

Notes for reviewers

The judgement call worth a look is the prompt-injection preamble. It could have
been deleted along with check 9, since check 9's untrusted git output was its
stated subject. I retargeted it instead — check 8 parses git worktree list --porcelain, which carries attacker-influenceable branch names and paths, so
the guardrail still has a real referent. Say the word if you'd rather see it
dropped.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KP5vkDusxrhi77Nx9xp9C1

…p-verify

Check 9 asserted that the local `apache/comdev` checkout backing the
ponymail and apache-projects MCP servers has an `apache/comdev` `origin`
and sits on `main`. That checkout is a personal working copy; which
branch it happens to be on, and whether `origin` names the canonical
repo or a contributor's fork, are not properties of the secure agent
setup this skill exists to certify. A contributor carrying a topic
branch off their own fork was being reported as a broken install.

The skill's own evals had already settled this: every case under
`tools/skill-evals/evals/setup-isolated-setup-verify/` asserts exactly
eight checks, numbered 1-8 and ending at project-root coverage, with no
comdev case anywhere. SKILL.md and its fixtures have been out of step;
they now agree, so no fixture changes were needed.

Check 9 was last in the list, so checks 1-8 keep their numbers. The
prompt-injection preamble was written specifically around check 9's
parsing of untrusted git output (branch names, commit subjects, remote
strings). Rather than lose the guardrail along with the check, it now
points at the untrusted machine output the remaining checks do parse:
the `git worktree list --porcelain` output read by check 8,
settings-file contents, and command stderr.

The canonical list in `docs/setup/secure-agent-setup.md`, which the
skill defers to, drops the matching item. The ponymail and
apache-projects tool docs no longer claim verify asserts the checkout is
on `main` and not behind; `setup-isolated-setup-update` still surfaces
the behind-`origin/main` warning and prints the pull command, which is
where that check belongs.

Generated-by: Claude Code (Opus 5)
@potiuk
potiuk merged commit 22eaa73 into apache:main Sep 9, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant