Skip to content

fix: validate on-disk existence and integrity of local-check artifacts - #26

Open
anvithsg2004 wants to merge 3 commits into
cloudflare:mainfrom
anvithsg2004:fix/coverage-artifact-validation
Open

anvithsg2004 wants to merge 3 commits into
cloudflare:mainfrom
anvithsg2004:fix/coverage-artifact-validation

Conversation

@anvithsg2004

@anvithsg2004 anvithsg2004 commented Sep 17, 2026 •

Copy link
Copy Markdown

Fixes #21

Summary

Previously, validate-coverage-ledger.cjs only validated the lexical format and ownership prefix (agents/<agent-id>/artifacts/...) of local_checks[].artifact paths. It did not verify whether referenced artifacts actually existed on disk in the retained output directory.

As a result, units marked as covered, candidate, or blocked could pass validation while pointing to phantom or non-existent local check evidence.

This PR extends validate-coverage-ledger.cjs to enforce strict on-disk validation of all local-check artifacts, verifying their existence, file type, symlink status, and directory containment, while adding a comprehensive regression test suite.


How We Solved It

  1. On-Disk Artifact Validation (validateArtifactFile):

    • Existence Check: Uses fs.lstatSync (without following symlinks) to verify the artifact file exists. If missing (ENOENT / ENOTDIR), it produces:
      $[...].artifact: local check artifact does not exist
    • Symlink Rejection: Rejects symlink artifacts using stat.isSymbolicLink():
      $[...].artifact: local check artifact must not be a symlink
    • Regular File Enforcement: Ensures the artifact is not a directory, FIFO, socket, or device using !stat.isFile():
      $[...].artifact: local check artifact must be a regular file
    • Boundary & Realpath Containment: Resolves fs.realpathSync of the artifact against the real path of the agent's artifacts directory (baseDir/agents/<agent_id>/artifacts/). If directory-level symlinks or path traversal attempt to escape this subtree, it produces:
      $[...].artifact: local check artifact must reside inside agent artifacts directory
  2. Pipeline Option Plumbing & Historical Checks:

    • Extended validateChecks(value, location, errors, options = {}) to trigger filesystem checks whenever options.baseDir is provided and lexical checks pass.
    • Passed options into validateAttempts so that archived history in unit.attempts[].local_checks[] also enforces artifact integrity.
    • Updated collectUnitErrors and validateDocument(ledger, options = {}) to thread options throughout the document traversal.
  3. CLI Directory Resolution & Backward Compatibility:

    • In CLI mode (run()), baseDir automatically defaults to the ledger file's parent directory (path.dirname(path.resolve(file))), which naturally aligns with the <output-dir>/coverage-ledger.json layout used throughout the audit workflow.
    • Added CLI arguments --output-dir <path> (and --base-dir <path>) supporting both space-separated and --output-dir=<path> syntax.
    • Added --schema-only flag to allow purely structural validation when desired.
    • Kept programmatic validateDocument(ledger) calls without baseDir schema-only, preserving backward compatibility for existing in-memory tests.
  4. Comprehensive Test Suite (validate-coverage-ledger.test.cjs):
    Added extensive positive, negative, and edge-case tests:

    • Missing artifacts report exact diagnostic paths ($[0].local_checks[0].artifact: local check artifact does not exist).
    • Valid promoted regular files pass cleanly with 0 errors.
    • Directories passed as artifacts are rejected (must be a regular file).
    • Symlinks and directory-escaping symlinks are rejected.
    • Historical checks in attempts[] are validated on disk.
    • 0-byte empty files and deeply nested files inside the artifacts directory are verified and accepted.
    • Unicode (NFC-normalized) filenames are handled correctly.
    • Multi-unit ledgers and multi-check units accurately report errors with exact index precision ($[N].local_checks[M]).
    • CLI integration tests verify exit codes (1 on missing artifact, 0 on valid artifact, --output-dir custom resolution, and --schema-only).

Verification

All existing and newly added tests pass cleanly:

node --test skills/security-audit/validate-coverage-ledger.test.cjs

@anvithsg2004 anvithsg2004 changed the title Validate local check artifact integrity fix: validate on-disk existence and integrity of local-check artifacts Sep 17, 2026
sheduxdev added a commit to sheduxdev/security-audit-skill that referenced this pull request Oct 22, 2026
sheduxdev added a commit to sheduxdev/security-audit-skill that referenced this pull request Oct 22, 2026
…f validating only the first file

Conflict in validate-coverage-ledger.cjs resolved by rejecting a second positional path inside parseArgs, since PR cloudflare#84/cloudflare#26 own the CLI argument parsing.
athallajovian9-cyber added a commit to JovianOps/security-audit-skill that referenced this pull request Oct 26, 2026
athallajovian9-cyber added a commit to JovianOps/security-audit-skill that referenced this pull request Oct 26, 2026
coffeegrind123 added a commit to coffeegrind123/security-audit-skill that referenced this pull request Nov 11, 2026
Upstream main has not moved since 2026-09-14, but 79 pull requests are
open. Each was read at the diff level, tested in its own worktree, and
checked for conflicts; the best of each duplicate group is ported.

Validators: cloudflare#143 archived verifier owner reuse, cloudflare#87 shared artifacts
within one attempt, cloudflare#102/cloudflare#103 duplicate JSON keys, cloudflare#111 1,000-field
objects, cloudflare#127 ordinary class must match assignment, cloudflare#104 self-excluded
ordinary block, cloudflare#141 verbatim quotes checked with --target, cloudflare#45 line
upper bound, cloudflare#84 --terminal rejects open units, cloudflare#145 identity-checked
open on Windows (POSIX still fails closed), cloudflare#48 retained-artifact
check (skills/ part only). Overlapping changes were merged so both
behaviours and both test sets survive.

Method: cloudflare#125 critic reasons through handoff, cloudflare#132 deployment controls
become units, cloudflare#83 carried-record gates, cloudflare#93 abort cleanup, cloudflare#23
streaming tool calls, cloudflare#50 config-dependent reachability stays
needs_validation, cloudflare#58 sequential verifier fallback, cloudflare#96 verifier no
longer sees the hunter's verdict, severity or fix.

Execution safety (cloudflare#76 neutralized git, symlink-safe copies; cloudflare#95
sanitized repo ID and prior-run compatibility; cloudflare#108 untrusted-content
prompt block; cloudflare#129 isolation built beside the host) moves into
EXECUTION-SAFETY.md so SKILL.md stays under its token budget.

Companions: cloudflare#101 cryptography, cloudflare#116 Android, cloudflare#35 edge isolate state.

Skipped: duplicates (cloudflare#82 cloudflare#106 cloudflare#91 cloudflare#70 cloudflare#138 cloudflare#131 cloudflare#75 cloudflare#63 cloudflare#97 cloudflare#123 cloudflare#25
cloudflare#26 cloudflare#16 cloudflare#40 cloudflare#42 cloudflare#142), validator-weakening fallbacks (cloudflare#59 cloudflare#113 cloudflare#99),
README/LICENSE churn, rebrands, scope creep, and cloudflare#85 cloudflare#90 cloudflare#8 cloudflare#89 cloudflare#32.

UPGRADE.md records this sync procedure.

Co-authored-by: Rohankollabathula67 <rohankollabathula67@gmail.com>
Co-authored-by: lab 1207 <hajarevaibhav527@gmail.com>
Co-authored-by: Rudy Celekli <rudy@gradiahq.com>
Co-authored-by: Mohit Rawat <rawatmohit17906@gmail.com>
Co-authored-by: MAHER AL-AQIL <18458259+maqalaqil@users.noreply.github.com>
Co-authored-by: CodeInf <jiuqiaoy@gmail.com>
Co-authored-by: pucedoteth <119044801+pucedoteth@users.noreply.github.com>
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Co-authored-by: Pizo <pizofreude@proton.me>
Co-authored-by: ZhiHe-ma <302089060+ZhiHe-ma@users.noreply.github.com>
Co-authored-by: shivam <juyaljii610@gmail.com>
Co-authored-by: Dante <duanjl.china@gmail.com>
Co-authored-by: Francisco <francisco@theysay.me>
Co-authored-by: Roman Dolgov <alter@users.noreply.github.com>
Co-authored-by: bodapatisaikrishna <saikrishnabodapati@gmail.com>
Co-authored-by: alvessh <117163985+alvessh@users.noreply.github.com>
Co-authored-by: itinance <hhuebel@itinance.com>
Co-authored-by: ChoYugyeong <cho4639@gmail.com>
Co-authored-by: Shaun Liebenberg <shaun@liebies.co.za>
Co-authored-by: JAI <jai19kharb@gmail.com>
Co-authored-by: ghostzeronz-coder <ghostzeronz-coder@users.noreply.github.com>
Co-authored-by: Aman Mohamed Ali <69119173+spidervirus@users.noreply.github.com>
Co-authored-by: Sandeep.dhami <sandeep84397@gmail.com>
Co-authored-by: jastfan <duohello640@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coverage validator accepts nonexistent local-check artifacts

1 participant