Skip to content

fix(coverage-ledger): verify local-check artifacts exist on disk - #25

Open
alter wants to merge 1 commit into
cloudflare:mainfrom
alter:fix/coverage-ledger-artifact-existence
Open

alter wants to merge 1 commit into
cloudflare:mainfrom
alter:fix/coverage-ledger-artifact-existence

Conversation

@alter

@alter alter commented Sep 17, 2026 •

Copy link
Copy Markdown

Closes #21

validateChecks in skills/security-audit/validate-coverage-ledger.cjs only checked that a
local check's artifact path was lexically owned by the checking agent (isOwnedArtifactPath:
matches agents/<agent_id>/artifacts/... and is a safe relative path). It never confirmed the
file the path names actually exists. A covered, blocked, or candidate coverage unit could
pass validation while its local check evidence was never produced, exactly the repro in the
issue: a syntactically valid artifact path with no file behind it passes with "PASS: 1
coverage units valid".

Added validateArtifactExists, called for every local method check once its path already
passes the existing ownership check. It resolves the artifact against the ledger's own
directory (the /coverage-ledger.json convention already documented in
RECONNAISSANCE.md and VALIDATION-AND-REPORTING.md, same directory the README's
~/security-audit-skill//run- default describes), and rejects when:

  1. the path doesn't resolve to anything ("local check artifact does not exist")
  2. it resolves to a symlink ("local check artifact must not be a symlink")
  3. it resolves to something other than a regular file, like a directory ("local check
    artifact must be a regular file")
  4. its real path, after resolving symlinks, escapes the owning agent's real artifacts
    directory ("local check artifact resolves outside the owning agent's artifacts
    directory"). This catches a symlinked ancestor directory smuggling the artifact outside
    its owner's tree even when the artifact itself isn't a symlink.

Diagnostic format matches the issue's own suggestion:
$[0].local_checks[0].artifact: local check artifact does not exist.

validateDocument, collectUnitErrors, validateAttempts, and validateChecks all take a new
optional baseDir parameter that only the CLI (run(file), via
path.dirname(path.resolve(file))) supplies. Direct callers of validateDocument that don't
have a real ledger directory, like the existing unit tests exercising isolated schema rules,
keep today's shape only behavior. That's deliberate, not an oversight: hundreds of existing
tests build ledgers in memory without ever writing artifact files to disk, and forcing every
one of them to also manage real fixtures would turn a focused validator correctness fix into
an unrelated test suite rewrite. The CLI path, the only one that runs for real per
RECONNAISSANCE.md's "run after seeding, after every parent update, and before Phase 6", now
always gets the real check.

Tested:

  1. Four new tests in validate-coverage-ledger.test.cjs, each driving the real CLI end to end
    via spawnSync, matching the file's existing CLI test convention: missing artifact,
    directory instead of file, symlink artifact, and the accepting case with a real file in
    place. Confirmed all three rejecting cases pass against unmodified code (the bug
    reproduces, PASS: 1 coverage units valid when it should fail) before the fix, and fail
    correctly after.
  2. A fifth new test confirms validateDocument called directly, no baseDir, matching every
    pre-existing unit test in the file, keeps its current shape only behavior. That's the
    compatibility guarantee the design above depends on.
  3. node --test skills/security-audit/*.test.cjs: all 70 tests pass, including the pre-existing
    cross check that findings and coverage ledger validator share aligned helpers.
  4. No lint or build step in this repo, plain Node.js .cjs scripts, no package.json.

validateChecks only checked that a local check's artifact path was
lexically owned by the checking agent (agents/<id>/artifacts/...); it
never confirmed the file actually exists. A covered/blocked/candidate
unit could pass validation while pointing at local evidence that was
never produced.

The CLI now resolves each local-check artifact against the ledger's
own directory (the documented <output-dir>/coverage-ledger.json
convention) and rejects a missing artifact, a non-regular file, or a
symlink, and rejects an artifact whose real path escapes the owning
agent's artifacts directory. validateDocument/validateChecks keep
today's shape-only behavior when called without a base directory, so
existing unit tests that check schema rules in isolation are
unaffected.

@agammann agammann left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The symlinked-ancestor case described in the PR still passes this check. If agents/<agent>/artifacts itself is a symlink to a directory outside the run, lstatSync(fullPath) reports the target file as regular, then both realFull and realAgentArtifactsDir resolve into that outside directory. The prefix test succeeds because it compares the file to the already-escaped artifacts directory. Could this also verify that the resolved artifacts directory stays inside the resolved run/agent tree (or reject a symlinked ancestor)? A fixture with artifacts as a directory symlink should reproduce it.

coffeegrind123 added a commit to coffeegrind123/security-audit-skill that referenced this pull request Nov 11, 2026
Upstream main has not moved since 2026-09-14, but 79 pull requests are
open. Each was read at the diff level, tested in its own worktree, and
checked for conflicts; the best of each duplicate group is ported.

Validators: cloudflare#143 archived verifier owner reuse, cloudflare#87 shared artifacts
within one attempt, cloudflare#102/cloudflare#103 duplicate JSON keys, cloudflare#111 1,000-field
objects, cloudflare#127 ordinary class must match assignment, cloudflare#104 self-excluded
ordinary block, cloudflare#141 verbatim quotes checked with --target, cloudflare#45 line
upper bound, cloudflare#84 --terminal rejects open units, cloudflare#145 identity-checked
open on Windows (POSIX still fails closed), cloudflare#48 retained-artifact
check (skills/ part only). Overlapping changes were merged so both
behaviours and both test sets survive.

Method: cloudflare#125 critic reasons through handoff, cloudflare#132 deployment controls
become units, cloudflare#83 carried-record gates, cloudflare#93 abort cleanup, cloudflare#23
streaming tool calls, cloudflare#50 config-dependent reachability stays
needs_validation, cloudflare#58 sequential verifier fallback, cloudflare#96 verifier no
longer sees the hunter's verdict, severity or fix.

Execution safety (cloudflare#76 neutralized git, symlink-safe copies; cloudflare#95
sanitized repo ID and prior-run compatibility; cloudflare#108 untrusted-content
prompt block; cloudflare#129 isolation built beside the host) moves into
EXECUTION-SAFETY.md so SKILL.md stays under its token budget.

Companions: cloudflare#101 cryptography, cloudflare#116 Android, cloudflare#35 edge isolate state.

Skipped: duplicates (cloudflare#82 cloudflare#106 cloudflare#91 cloudflare#70 cloudflare#138 cloudflare#131 cloudflare#75 cloudflare#63 cloudflare#97 cloudflare#123 cloudflare#25
cloudflare#26 cloudflare#16 cloudflare#40 cloudflare#42 cloudflare#142), validator-weakening fallbacks (cloudflare#59 cloudflare#113 cloudflare#99),
README/LICENSE churn, rebrands, scope creep, and cloudflare#85 cloudflare#90 cloudflare#8 cloudflare#89 cloudflare#32.

UPGRADE.md records this sync procedure.

Co-authored-by: Rohankollabathula67 <rohankollabathula67@gmail.com>
Co-authored-by: lab 1207 <hajarevaibhav527@gmail.com>
Co-authored-by: Rudy Celekli <rudy@gradiahq.com>
Co-authored-by: Mohit Rawat <rawatmohit17906@gmail.com>
Co-authored-by: MAHER AL-AQIL <18458259+maqalaqil@users.noreply.github.com>
Co-authored-by: CodeInf <jiuqiaoy@gmail.com>
Co-authored-by: pucedoteth <119044801+pucedoteth@users.noreply.github.com>
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Co-authored-by: Pizo <pizofreude@proton.me>
Co-authored-by: ZhiHe-ma <302089060+ZhiHe-ma@users.noreply.github.com>
Co-authored-by: shivam <juyaljii610@gmail.com>
Co-authored-by: Dante <duanjl.china@gmail.com>
Co-authored-by: Francisco <francisco@theysay.me>
Co-authored-by: Roman Dolgov <alter@users.noreply.github.com>
Co-authored-by: bodapatisaikrishna <saikrishnabodapati@gmail.com>
Co-authored-by: alvessh <117163985+alvessh@users.noreply.github.com>
Co-authored-by: itinance <hhuebel@itinance.com>
Co-authored-by: ChoYugyeong <cho4639@gmail.com>
Co-authored-by: Shaun Liebenberg <shaun@liebies.co.za>
Co-authored-by: JAI <jai19kharb@gmail.com>
Co-authored-by: ghostzeronz-coder <ghostzeronz-coder@users.noreply.github.com>
Co-authored-by: Aman Mohamed Ali <69119173+spidervirus@users.noreply.github.com>
Co-authored-by: Sandeep.dhami <sandeep84397@gmail.com>
Co-authored-by: jastfan <duohello640@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coverage validator accepts nonexistent local-check artifacts

2 participants