Skip to content

Rework the audit workflow, findings contract, and validators end to end - #12

Merged
literally-dan merged 1 commit into
mainfrom
harden-audit-workflow-and-validators
Sep 10, 2026
Merged

literally-dan merged 1 commit into
mainfrom
harden-audit-workflow-and-validators

Conversation

@literally-dan

Copy link
Copy Markdown
Collaborator

What this does

One squashed commit covering a full rework of the skill:

  • Findings contract — findings.json uses three verdict branches (confirmed, needs_validation, rejected) with complete traces, bounded local evidence, and severity capped at demonstrated impact.
  • Deterministic coverage — reconnaissance seeds a coverage ledger with canonical collision-checked unit IDs, append-only attempt provenance, per-state field invariants, and prior-run rules that revalidate unchanged confirmations and turn deferred/blocked/out-of-scope prior units into current work. Critic-inclusive budget reservation and an explicit incomplete-run state replace silent coverage claims.
  • Six new domain companions (cloud/deployment, data isolation/lifecycle, desktop/mobile/local IPC, protocols/RPC/messaging, resource exhaustion, supply chain/release) plus deepened existing four, all sharing one section schema, copied verbatim into hunter prompts.
  • Execution safety — target-controlled execution requires an OS-enforced sandbox (no external network, allowlisted empty environment, resource limits, scratch-only writes); missing controls retain the lead as needs_validation. One canonical artifact-promotion procedure, embedded byte-identical in hunter and verifier prompts. Output defaults outside the target repository.
  • Validators — validate-findings.cjs hardened (prose vs exact-payload separation, unsafe-path rejection, pre-parse structural limits, no-follow input, fatal UTF-8, capped sanitized errors) and new zero-dependency validate-coverage-ledger.cjs, with 65 tests including a cross-validator consistency test.
  • Prompting conventions — reviewed against opencode skill format, Anthropic authoring practice, and the OpenAI GPT-5 prompting guide: conflicts resolved, explicit stop conditions and terminal states, depth-bounded exploration, schema branches included in subagent prompts with literal return envelopes.

Verification

  • 65/65 validator tests pass; links, whitespace, and diagnostics clean.
  • Multiple independent review rounds (workflow coherence, validator code with adversarial fixtures, packaging, wording) ending in accepted verdicts.
  • Two live end-to-end audits of an internal repository under gpt-5.6-sol: zero provider errors, both validators passed on generated artifacts, and the run correctly self-reported budget-limited partial coverage with named unvalidated fingerprints.

Findings contract: findings.json now uses three verdict branches -
confirmed, needs_validation, and rejected - with complete traces,
bounded local evidence, and severity capped at demonstrated impact.

Coverage: reconnaissance seeds a deterministic coverage ledger with
canonical collision-checked unit IDs, append-only attempt provenance,
explicit per-state field invariants, and prior-run carry rules that
revalidate unchanged confirmations and turn deferred, blocked, and
out-of-scope prior units into current work. Coverage-critic waves,
critic-inclusive budget reservation, and an explicit incomplete-run
state replace silent coverage claims.

Ten domain companions (AI/LLM, web protocol and auth, client-side,
memory safety and binary, cloud and deployment, data isolation and
lifecycle, desktop/mobile/local IPC, protocols/RPC/messaging, resource
exhaustion, supply chain and release) share one section schema and are
copied verbatim into hunter prompts alongside the ordinary attack
classes.

Safety: target-controlled execution requires an OS-enforced sandbox
with no external network, an allowlisted empty environment, resource
limits, and scratch-only writes; missing controls retain the lead as
needs_validation. Artifact promotion is a single canonical procedure -
descriptor-anchored no-follow walks, regular-file and size checks,
exclusive creation - stated once in SKILL.md and embedded byte-identical
in the hunter and verifier prompts. Audit output defaults outside the
target repository.

Validators: validate-findings.cjs is hardened (visible-prose vs exact
payload separation, platform-unsafe path rejection, canonical Set-based
uniqueness, pre-parse structural limits, no-follow input, fatal UTF-8,
sanitized capped error output) and a new zero-dependency
validate-coverage-ledger.cjs enforces the ledger contract with the same
input and output safety. Both ship with test suites (65 tests) and a
cross-validator consistency test.

Prompting: instruction conflicts resolved, explicit stop conditions and
terminal states, exploration depth bounds, schema branches included in
subagent prompts with literal return envelopes, and malformed-result
handling that preserves verifier independence.

Validated end to end with a live budget-limited audit run against an
internal repository: both validators passed and the run reported honest
partial coverage.
liuchongai1998

This comment was marked as duplicate.

@liuchongai1998 liuchongai1998 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L

devsart95 pushed a commit to devsart95/security-audit-skill that referenced this pull request Nov 18, 2026
…w-and-validators

Rework the audit workflow, findings contract, and validators end to end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants