Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The skill runs a structured audit in six phases:
5. **Independent record verification** -- fresh agents verify final source claims. Material replacements receive another independent verifier.
6. **Target-neutral reporting** -- derive `REPORT.md`, `FINDINGS-DETAIL.md`, and `NEEDS-VALIDATION.md` from the verified records and coverage ledger.

The parent runs `validate-coverage-ledger.cjs` after creating the ledger and after each later ledger update. It runs `validate-findings.cjs` in Phase 4 and again after every Phase 5 replacement.
The parent runs `validate-coverage-ledger.cjs` after creating the ledger and after each later ledger update, and with `--terminal` in Phase 4 and after every Phase 5 replacement so a complete claim cannot retain open assignments. It runs `validate-findings.cjs` in Phase 4 and again after every Phase 5 replacement.

The verdicts are distinct: `confirmed` has a complete source trace and bounded observed result, `needs_validation` has an exact unresolved fact and no severity, and `rejected` records a disproved candidate.

Expand Down
4 changes: 2 additions & 2 deletions skills/security-audit/HUNTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ Every assigned coverage ID appears exactly once in `units`. A `covered` unit nee

## Parent consolidation and ledger update

The parent validates each unit result, maps it to exactly one assigned `coverage_id`, and updates only that ledger unit. Reject duplicate or absent IDs, unsafe unit or check agent IDs, source checks with artifacts, and local artifacts that trusted parent-side code did not promote into the check owner's artifacts subtree. Copy the unit's `reviewed_paths`, its `checks` into the unit's `local_checks`, linked artifact paths, candidate fingerprints, and unresolved facts into the ledger. Retain each hunter's `hardening` list in a parent bookkeeping field on the relevant units (outside the semantic fields) so Phase 6 can report it. A failed, malformed, or unsupported conclusion leaves that unit `planned` for reassignment. Untouched budget/profile units become unassigned `deferred` units with empty evidence and a reason; do not hide partial evidence in `deferred`. Run `validate-coverage-ledger.cjs` after the update; an invalid ledger cannot drive another assignment. This per-unit contract allows one hunter to close one unit while returning a candidate or blocker for another.
The parent validates each unit result, maps it to exactly one assigned `coverage_id`, and updates only that ledger unit. Reject duplicate or absent IDs, unsafe unit or check agent IDs, source checks with artifacts, and local artifacts that trusted parent-side code did not promote into the check owner's artifacts subtree. Copy the unit's `reviewed_paths`, its `checks` into the unit's `local_checks`, linked artifact paths, candidate fingerprints, and unresolved facts into the ledger. Retain each hunter's `hardening` list in a parent bookkeeping field on the relevant units (outside the semantic fields) so Phase 6 can report it. A failed, malformed, unsupported, or missing conclusion leaves that unit `planned` for reassignment. A hunter that never returns (killed, hung, crashed, or timed out with no result) is a missing conclusion: restore `planned` with a null owner and empty evidence, and keep the current `wave`. Do not append `in_progress` to `attempts`; that archive holds only evidence-bearing `blocked`, `covered`, or `candidate` states. The coverage critic cannot issue `reassign_id` for an `in_progress` unit. In `quick`, allow one retry assignment inside the same single wave; if that retry also never returns, mark the unit `deferred` with reason `hunter_never_returned` and do not launch a second wave. Untouched budget/profile units become unassigned `deferred` units with empty evidence and a reason; do not hide partial evidence in `deferred`. Run `validate-coverage-ledger.cjs` after the update; an invalid ledger cannot drive another assignment. This per-unit contract allows one hunter to close one unit while returning a candidate or blocker for another.

Consolidate candidate entries by fingerprint and then by root cause. One root cause that exposes several entry paths or effects is one candidate with the strongest complete trace. Related but independent missing controls use separate fingerprints. Record duplicate fingerprints in the relevant ledger unit and do not send duplicate candidates to validation.

Expand Down Expand Up @@ -244,7 +244,7 @@ Immediately after each hunter wave, spend the reserved invocation on one fresh `

The critic checks for unmapped entry points, unchecked parallel paths, missing lifecycle modes, selected companion classes without a unit, unjustified exclusions, units closed without paths/checks, and prior `needs_validation` or changed-source gaps that no unit addresses. It proposes coverage, not findings. `stop` is the critic's own assessment: `true` only when it accepts no `missing_units` and no `reassign_ids`; the parent's loop condition below, not `stop` alone, decides whether another wave runs. For each fingerprint in `resolved_prior_leads`, the parent marks the linked unit or prior-lead entry resolved and records the critic's source-backed reason.

The parent rejects proposed units outside the review scope or source/local boundary, derives canonical IDs for accepted units, and deduplicates them against current units. A prior same-source completed unit may supply evidence; prior `deferred`, `blocked`, `out_of_scope`, or changed-source units become current work and never suppress an accepted unit. Fail rather than merge a canonical ID collision. For each legitimate `reassign_id` with live `blocked`, `covered`, or `candidate` evidence, append that exact terminal record to the unit's `attempts` with the critic's source-backed `reassignment_reason`. Preserve its owner, checks, artifacts, fingerprints, and unresolved facts in that archive. Increment the live `wave`; the next hunter must be a fresh owner and receives an `in_progress` unit with empty live evidence. The hunter's terminal result writes only its new evidence into the live fields. Never copy an archived owner's checks or artifacts into the new live attempt. Sort IDs and validate the ledger before another assignment. In `standard` and `deep`, when the post-wave critic reports no accepted `missing_units` or legitimate `reassign_ids` and no `planned` units remain, spend the separately reserved invocation on a distinct final-clean critic. Complete coverage only when that critic also returns no accepted work. If it finds work, queue it and repeat the wave, post-wave critic, and final-clean process. If time or resources force an early stop, mark every untouched unit `deferred`, preserve the critic's reason, and disclose the gap in the report. Never use a silent wave or agent cap as evidence of complete coverage.
The parent rejects proposed units outside the review scope or source/local boundary, derives canonical IDs for accepted units, and deduplicates them against current units. A prior same-source completed unit may supply evidence; prior `deferred`, `blocked`, `out_of_scope`, or changed-source units become current work and never suppress an accepted unit. Fail rather than merge a canonical ID collision. For each legitimate `reassign_id` with live `blocked`, `covered`, or `candidate` evidence, append that exact terminal record to the unit's `attempts` with the critic's source-backed `reassignment_reason`. Preserve its owner, checks, artifacts, fingerprints, and unresolved facts in that archive. Increment the live `wave`; the next hunter must be a fresh owner and receives an `in_progress` unit with empty live evidence. The hunter's terminal result writes only its new evidence into the live fields. Never copy an archived owner's checks or artifacts into the new live attempt. Sort IDs and validate the ledger before another assignment. In `standard` and `deep`, when the post-wave critic reports no accepted `missing_units` or legitimate `reassign_ids` and no `planned` or `in_progress` units remain, spend the separately reserved invocation on a distinct final-clean critic. Resolve a stuck assignment with the missing-conclusion rule before treating the wave as closed. Complete coverage only when that critic also returns no accepted work. If it finds work, queue it and repeat the wave, post-wave critic, and final-clean process. If time or resources force an early stop, mark every untouched unit `deferred`, preserve the critic's reason, and disclose the gap in the report. Never use a silent wave or agent cap as evidence of complete coverage.

The run profile bounds this loop. A `quick` run has exactly one hunter wave followed by exactly one final critic pass. Add each accepted `missing_unit` to the current ledger and mark it `deferred` with reason `quick_profile_final_critic`. For each legitimate evidence-bearing `reassign_id`, archive the live terminal state in `attempts`, increment `wave`, and set the live state to unassigned `deferred` with empty evidence and reason `quick_profile_final_critic`. Do not launch a second hunter wave or another critic. In a scoped run, the critic still reports out-of-scope gaps it notices, but the parent records them as `out_of_scope` with the critic's reason instead of assigning them. The early-stop rule above is the same mechanism: `quick` is a pre-declared early stop, not evidence of complete coverage.

Expand Down
2 changes: 1 addition & 1 deletion skills/security-audit/RECONNAISSANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,6 @@ Enforce this state table exactly:

Canonical agent IDs match `^[a-z0-9][a-z0-9_-]{0,63}$` and are not Windows device names. Lowercase is mandatory, so one ledger cannot contain case-fold aliases. The unit `agent_id` records the assignment owner. Every check records its own `agent_id` and nonempty `reviewed_paths`; the unit-level `reviewed_paths` is exactly their union. A source-only check uses `artifact: null`. A local check requires a regular file promoted only by trusted parent-side code under exactly `agents/<check.agent_id>/artifacts/`. This lets hunter and verifier checks coexist in one unit. Scratch paths, output-root files, symlinks, special files, and another check owner's artifacts are not evidence.

The ledger is the coverage claim. An architecture summary, agent count, or generic "auth reviewed" sentence is not coverage evidence. Phase 2 closes units only from the paths and checks in a hunter's structured result.
The ledger is the coverage claim. An architecture summary, agent count, or generic "auth reviewed" sentence is not coverage evidence. Phase 2 closes units only from the paths and checks in a hunter's structured result. `in_progress` is a live assignment, not a terminal coverage state: a hunter that never returns must be restored to `planned` (or `deferred` with reason `hunter_never_returned` after the `quick` retry). A final `--terminal` coverage-ledger check rejects remaining `in_progress` and `planned` units.

Run `node <skill-dir>/validate-coverage-ledger.cjs <output-dir>/coverage-ledger.json` after seeding, after every parent update, and before Phase 6. The validator rejects input beyond 5 MiB, 64 nesting levels, 10,000 units, 1,000 entries in a nested collection, or 500,000 traversed values, and caps reported validation errors at 100. In practice the 5 MiB byte limit holds roughly 2,000-5,000 realistic units, so it binds before the 10,000-unit cap. Fix every error before assigning work or making a coverage claim.
2 changes: 1 addition & 1 deletion skills/security-audit/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ In full audit mode, follow all six phases in order:
5. **Independent record verification** — use fresh agents to verify final source claims and reconcile corrections or state changes.
6. **Target-neutral report** — derive `REPORT.md`, `FINDINGS-DETAIL.md`, and `NEEDS-VALIDATION.md` from the final records, with no live-probe instructions.

Do not end the run before one of exactly two terminal states: (a) all Phase 6 artifacts are written and both validators pass, or (b) `run_status: "incomplete"` is recorded with its exact reason and the gap is disclosed in the report. Never stop mid-phase.
Do not end the run before one of exactly two terminal states: (a) all Phase 6 artifacts are written, both validators pass, and the coverage ledger was last checked with `--terminal` (zero `in_progress` and zero `planned` units), or (b) `run_status: "incomplete"` is recorded with its exact reason and the gap is disclosed in the report. Remaining `blocked` and `deferred` units are disclosed as partial coverage, not a silent complete-coverage claim. Never stop mid-phase.

## Anti-patterns

Expand Down
8 changes: 4 additions & 4 deletions skills/security-audit/VALIDATION-AND-REPORTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,10 +112,10 @@ Run:

```sh
node <skill-dir>/validate-findings.cjs <output-dir>/findings.json
node <skill-dir>/validate-coverage-ledger.cjs <output-dir>/coverage-ledger.json
node <skill-dir>/validate-coverage-ledger.cjs <output-dir>/coverage-ledger.json --terminal
```

Fix every structural and semantic error before continuing. The findings validator rejects input beyond 5 MiB, 1,000 top-level findings, or 64 nesting levels, and caps reported error output at 100 messages. Validator success proves format and ledger consistency only.
Fix every structural and semantic error before continuing. The findings validator rejects input beyond 5 MiB, 1,000 top-level findings, or 64 nesting levels, and caps reported error output at 100 messages. Phase 4 and later ledger checks that underwrite a complete run must pass `--terminal`, which rejects remaining `in_progress` and `planned` units. Mid-run ledger updates omit `--terminal` so a live assignment can still validate. Validator success proves format and ledger consistency only.

### Phase 5: Verify the final records with fresh eyes

Expand Down Expand Up @@ -144,7 +144,7 @@ Each verifier returns exactly one JSON object: `{"decision":"verified","fingerpr

Do not apply a Phase 5 replacement as final when it promotes a record to a stronger verdict, including any promotion to `confirmed`, or materially changes the root cause, trace, execution input or observed result, demonstrated impact, or severity. Give that complete replacement to a new independent verifier that did not hunt, perform Phase 3 validation, or propose the Phase 5 replacement. The new verifier rechecks the current source and independently reproduces any decisive local result under the execution boundary, then returns `verified` or another replacement. Apply a material replacement only after this fresh verification. If another material replacement results, repeat with a fresh verifier. If budget or independence is unavailable, remove the disputed record from `findings.json`, keep its ledger unit as an unresolved candidate, and set `run_status: "incomplete"` with an exact `incomplete_reason`. Non-material wording or repository-line corrections may be applied directly when they do not change meaning or evidence.

After every applied replacement, rerun both validators and update linked ledger decisions. If a final verifier identifies a separate root cause, assign a new fingerprint and send it through independent candidate validation before inclusion. Set `run_status: "complete"` only when every ledger candidate has an independent final disposition and every retained record passes Phase 5.
After every applied replacement, rerun both validators and update linked ledger decisions. If a final verifier identifies a separate root cause, assign a new fingerprint and send it through independent candidate validation before inclusion. Set `run_status: "complete"` only when every ledger candidate has an independent final disposition, every retained record passes Phase 5, and the coverage ledger has no `in_progress` or `planned` units. Remaining `blocked` and `deferred` units must be counted in the coverage summary and named as partial-coverage limits; they do not by themselves make the run incomplete.

Do not verify only `confirmed` records. A misleading `needs_validation` handoff wastes owner time and can preserve a false premise.

Expand All @@ -162,7 +162,7 @@ Write:
4. Each confirmed finding: repository source location, lower-trust principal, target-native bounded reproduction, conditions, actual result, impact, priority rationale, and smallest source fix.
5. A separate `NEEDS VALIDATION` table. Give each lead's title, repository trace, exact blocker, bounded local next step, and safe owner-observed deployment check. Do not assign severity or call it a confirmed vulnerability.
6. Separate hardening notes and positive source patterns.
7. Coverage summary from the ledger: covered, candidate, blocked, and deferred counts, plus important exclusions and the final critic result.
7. Coverage summary from the ledger: covered, candidate, blocked, deferred, and in_progress counts, plus important exclusions and the final critic result. An `in_progress` count above zero means an assignment is still open and the run is not complete.

Do not describe rejected records as findings. Mention their fingerprints only when they explain a prior disagreement or coverage decision.

Expand Down
31 changes: 25 additions & 6 deletions skills/security-audit/validate-coverage-ledger.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

/**
* Validates coverage-ledger.json and its canonical coverage IDs.
* Usage: node validate-coverage-ledger.cjs <path-to-coverage-ledger.json>
* Usage: node validate-coverage-ledger.cjs <path-to-coverage-ledger.json> [--terminal]
*/

const fs = require("node:fs");
Expand Down Expand Up @@ -758,7 +758,8 @@ function readFileWithinLimit(file) {
}
}

function validateDocument(ledger) {
function validateDocument(ledger, options = {}) {
const terminal = options.terminal === true;
const errors = createErrorList();
if (!Array.isArray(ledger)) {
errors.push("$: expected a top-level array");
Expand All @@ -778,6 +779,12 @@ function validateDocument(ledger) {
for (let index = 0; index < ledger.length && errors.length < LIMITS.validationErrors; index++) {
const unit = ledger[index];
errors.push(...collectUnitErrors(unit, index));
if (terminal && isObject(unit) && unit.status === "in_progress") {
errors.push(`$[${index}].status: terminal ledger cannot include in_progress units`);
}
if (terminal && isObject(unit) && unit.status === "planned") {
errors.push(`$[${index}].status: terminal ledger cannot include planned units`);
}
if (errors.length >= LIMITS.validationErrors) break;
if (!isObject(unit) || typeof unit.coverage_id !== "string") continue;

Expand Down Expand Up @@ -805,9 +812,21 @@ function validateDocument(ledger) {
return errors;
}

function run(file) {
function parseCliArgs(argv) {
const args = argv.slice(2);
let file;
let terminal = false;
for (const arg of args) {
if (arg === "--terminal") terminal = true;
else if (file === undefined) file = arg;
}
return { file, terminal };
}

function run(argv = process.argv) {
const { file, terminal } = parseCliArgs(argv);
if (!file) {
console.error("Usage: node validate-coverage-ledger.cjs <path-to-coverage-ledger.json>");
console.error("Usage: node validate-coverage-ledger.cjs <path-to-coverage-ledger.json> [--terminal]");
return 1;
}

Expand Down Expand Up @@ -837,7 +856,7 @@ function run(file) {

let errors;
try {
errors = validateDocument(ledger);
errors = validateDocument(ledger, { terminal });
} catch {
console.error("Failed to validate coverage ledger: unexpected validation error");
return 1;
Expand Down Expand Up @@ -869,4 +888,4 @@ module.exports = {
validateDocument,
};

if (require.main === module) process.exit(run(process.argv[2]));
if (require.main === module) process.exit(run());
Loading