Skip to content

Validate retained coverage artifacts and fix platform-aware tests - #48

Open
ZhiHe-ma wants to merge 2 commits into
cloudflare:mainfrom
ZhiHe-ma:fix/retained-artifact-validation
Open

ZhiHe-ma wants to merge 2 commits into
cloudflare:mainfrom
ZhiHe-ma:fix/retained-artifact-validation

Conversation

@ZhiHe-ma

@ZhiHe-ma ZhiHe-ma commented Sep 20, 2026 •

Copy link
Copy Markdown

A coverage unit could pass validation even when its local-check artifact did not exist. The coverage CLI now verifies retained files for both live checks and archived attempts, so missing evidence cannot support a successful coverage validation.

Fixes #21.

Changes

  • Resolve artifact paths relative to the ledger directory, with --output-dir for exported ledgers.
  • Require existing, single-link regular files under each check owner's artifact tree; reject symlinks/junctions, non-directory parents, and special files without reading artifact contents.
  • Preserve the structural-only library API; validateDocument(ledger, { outputDir }) enables filesystem checks.
  • Correct seven findings CLI tests that assumed OS support for protected file opening, and explicitly test fail-closed behavior on unsupported platforms.
  • Add a Node.js 22/24 CI matrix across Linux, macOS, and Windows, plus workflow documentation and an implementation/validation record.

Validation

node --test skills/security-audit/validate-findings.test.cjs skills/security-audit/validate-coverage-ledger.test.cjs

  • Windows / Node.js 24.19.0: 59 passed, 23 platform-dependent skips, 0 failures.
  • Linux / official Node.js 24 Alpine container: 80 passed, 2 unsupported-platform-only skips, 0 failures. The container had networking disabled, a read-only source mount, temporary writable storage, and resource limits.
  • All 82 tests passed in at least one applicable environment across these runs.
  • Skill structure validation, workflow YAML checks, JavaScript syntax checks, and git diff --check passed.
  • macOS and Node.js 22 are configured in CI but were not run locally.

AI Assistance Disclosure

AI tools were used to assist with implementation review, test development, documentation, and reasoning about edge cases. All submitted changes, security assumptions, test results, and validation claims were reviewed and verified by the contributor. AI-generated output was not treated as independent evidence of correctness or as a substitute for the validation described above.

The artifact check assumes a trusted, stable, parent-owned output tree. It verifies file presence/type, not evidence contents, and does not replace race-safe artifact promotion or independent finding verification. Native Windows CLI file opening continues to fail closed when the required OS protections are unavailable.

coffeegrind123 added a commit to coffeegrind123/security-audit-skill that referenced this pull request Nov 11, 2026
Upstream main has not moved since 2026-09-14, but 79 pull requests are
open. Each was read at the diff level, tested in its own worktree, and
checked for conflicts; the best of each duplicate group is ported.

Validators: cloudflare#143 archived verifier owner reuse, cloudflare#87 shared artifacts
within one attempt, cloudflare#102/cloudflare#103 duplicate JSON keys, cloudflare#111 1,000-field
objects, cloudflare#127 ordinary class must match assignment, cloudflare#104 self-excluded
ordinary block, cloudflare#141 verbatim quotes checked with --target, cloudflare#45 line
upper bound, cloudflare#84 --terminal rejects open units, cloudflare#145 identity-checked
open on Windows (POSIX still fails closed), cloudflare#48 retained-artifact
check (skills/ part only). Overlapping changes were merged so both
behaviours and both test sets survive.

Method: cloudflare#125 critic reasons through handoff, cloudflare#132 deployment controls
become units, cloudflare#83 carried-record gates, cloudflare#93 abort cleanup, cloudflare#23
streaming tool calls, cloudflare#50 config-dependent reachability stays
needs_validation, cloudflare#58 sequential verifier fallback, cloudflare#96 verifier no
longer sees the hunter's verdict, severity or fix.

Execution safety (cloudflare#76 neutralized git, symlink-safe copies; cloudflare#95
sanitized repo ID and prior-run compatibility; cloudflare#108 untrusted-content
prompt block; cloudflare#129 isolation built beside the host) moves into
EXECUTION-SAFETY.md so SKILL.md stays under its token budget.

Companions: cloudflare#101 cryptography, cloudflare#116 Android, cloudflare#35 edge isolate state.

Skipped: duplicates (cloudflare#82 cloudflare#106 cloudflare#91 cloudflare#70 cloudflare#138 cloudflare#131 cloudflare#75 cloudflare#63 cloudflare#97 cloudflare#123 cloudflare#25
cloudflare#26 cloudflare#16 cloudflare#40 cloudflare#42 cloudflare#142), validator-weakening fallbacks (cloudflare#59 cloudflare#113 cloudflare#99),
README/LICENSE churn, rebrands, scope creep, and cloudflare#85 cloudflare#90 cloudflare#8 cloudflare#89 cloudflare#32.

UPGRADE.md records this sync procedure.

Co-authored-by: Rohankollabathula67 <rohankollabathula67@gmail.com>
Co-authored-by: lab 1207 <hajarevaibhav527@gmail.com>
Co-authored-by: Rudy Celekli <rudy@gradiahq.com>
Co-authored-by: Mohit Rawat <rawatmohit17906@gmail.com>
Co-authored-by: MAHER AL-AQIL <18458259+maqalaqil@users.noreply.github.com>
Co-authored-by: CodeInf <jiuqiaoy@gmail.com>
Co-authored-by: pucedoteth <119044801+pucedoteth@users.noreply.github.com>
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Co-authored-by: Pizo <pizofreude@proton.me>
Co-authored-by: ZhiHe-ma <302089060+ZhiHe-ma@users.noreply.github.com>
Co-authored-by: shivam <juyaljii610@gmail.com>
Co-authored-by: Dante <duanjl.china@gmail.com>
Co-authored-by: Francisco <francisco@theysay.me>
Co-authored-by: Roman Dolgov <alter@users.noreply.github.com>
Co-authored-by: bodapatisaikrishna <saikrishnabodapati@gmail.com>
Co-authored-by: alvessh <117163985+alvessh@users.noreply.github.com>
Co-authored-by: itinance <hhuebel@itinance.com>
Co-authored-by: ChoYugyeong <cho4639@gmail.com>
Co-authored-by: Shaun Liebenberg <shaun@liebies.co.za>
Co-authored-by: JAI <jai19kharb@gmail.com>
Co-authored-by: ghostzeronz-coder <ghostzeronz-coder@users.noreply.github.com>
Co-authored-by: Aman Mohamed Ali <69119173+spidervirus@users.noreply.github.com>
Co-authored-by: Sandeep.dhami <sandeep84397@gmail.com>
Co-authored-by: jastfan <duohello640@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coverage validator accepts nonexistent local-check artifacts

1 participant