Skip to content

docs: redact target secrets and run the validators in the sandbox - #153

Open
johncgoat wants to merge 1 commit into
cloudflare:mainfrom
johncgoat:docs/redact-secrets-sandbox-validators
Open

johncgoat wants to merge 1 commit into
cloudflare:mainfrom
johncgoat:docs/redact-secrets-sandbox-validators

Conversation

@johncgoat

Copy link
Copy Markdown

Refs #148 (the "Secrets in reports" hardening note).

Problem

1. Real secrets found in the target can be copied verbatim into the outputs. ATTACK-CLASSES.md sends the "Obvious things" agent looking for hardcoded passwords, API keys, tokens, -----BEGIN blocks, .env, *.pem and credentials.json, and it also asks for "secrets that were committed then removed (still in history)". When the agent finds one, nothing says how to record it. SKILL.md covers only the ambient environment during reproduction ("Launch from an empty environment rather than trying to redact one after execution"), and promote the minimum non-secret result applies only to artifacts. trace, evidence, payloads, observed_result, REPORT.md, FINDINGS-DETAIL.md, and the hunter-to-parent and parent-to-verifier hand-offs have no rule at all. An audit of a repository with a leaked key can therefore produce a second copy of that key in every report file and agent transcript.

2. The validators are run with host node on target-derived input. RECONNAISSANCE.md, SKILL.md (workflow step 4) and VALIDATION-AND-REPORTING.md Phase 4 all tell the parent to run node <skill-dir>/validate-*.cjs <output-dir>/...json with no execution context. Both files carry strings that came from the target: titles, scopes, payloads, observed output, and file names. The validators are careful: they use only fs, path and util, open with O_NOFOLLOW, cap input at 5 MiB, and never write, spawn, or touch the network. Even so, this is the only step in the workflow that hands target-influenced bytes to an interpreter outside the OS-enforced sandbox the skill requires everywhere else. It also runs in the parent's full environment, credentials included.

Change

Docs only. 4 files, +14/−4.

  • SKILL.md › Universal execution safety: a new paragraph says a real secret found in the target is never written, in whole, in part, or in any reversible encoding, into findings.json, the ledger, the three report files, scratch or artifacts, or any prompt or hand-off. Records keep only the type, the repository-relative location (path and line, or commit and path for history), and a short value hash: sha256: plus the first 12 hex digits. The hash is omitted for values under 16 characters, because a short hash of a short value can be brute-forced. The text calls it a "value hash" so it is not confused with the record fingerprint. A hunter or verifier result that contains a value is malformed and is re-run, matching how the skill already handles malformed results. Liveness is never tested. Validity and rotation are owner-observed facts.
  • SKILL.md › new ### Validator execution: both validators run inside the same OS-enforced sandbox as target code. That means no network, an empty environment, the skill directory and the input file read-only, no writable path, and the same limits. A pass counts only if the parent saw it come out of that sandbox. If no sandbox is available, the validators still run, and REPORT.md and run-metadata.json must say so. Workflow step 4 points to this section.
  • HUNTING.md: the hunter prompt repeats the secret rule in its own words, since hunters receive that file and not SKILL.md.
  • VALIDATION-AND-REPORTING.md: the record rules restate the secret rule for records and derived reports, and the Phase 4 Run: line points to Validator execution.
  • RECONNAISSANCE.md: the ledger-validator command points to Validator execution.

Before / after

Before (main @ c1c8a8c), the only redaction language in the skill:

$ git grep -n -i redact -- skills/
skills/security-audit/PROTOCOLS-RPC-AND-MESSAGING.md:50: ... Review policy and redaction at the failure path ...   (a hunting hint about targets)
skills/security-audit/SKILL.md:84: ... Launch from an empty environment rather than trying to redact one after execution.

and the validator instruction:

VALIDATION-AND-REPORTING.md:111  Run:
VALIDATION-AND-REPORTING.md:114  node <skill-dir>/validate-findings.cjs <output-dir>/findings.json

After: an evidence entry for a committed key looks like this:

{"file": "config/prod.env", "line": 7,
 "description": "AWS access key pair committed in plaintext (type: aws-access-key; value hash sha256:3f1c9a0b7e22). Not tested for liveness; owner to confirm validity and rotate."}

and Phase 4 reads: "Run both validators as Validator execution in SKILL.md requires (inside the target-code sandbox, not with the host node)".

Overlap with open PRs

I read #149, #147, #151, #141, #129, #96, #75 and #108 before writing this, and grepped every open PR diff for redact and for guidance on where the validators run.

Validator test change?

None. Validators can't tell a real secret from a documented dummy such as AKIAIOSFODNN7EXAMPLE, so a schema or validator check would either miss real values or reject legitimate test vectors. This is a rule for agents, enforced at hand-off. If you want a mechanical check later, I can follow up with a small high-precision check (PEM private-key blocks only, for example) and tests, as a separate PR.

Verification

  • Docs-only. No validator, schema, or test file changed.

  • node --test from skills/security-audit/, run in node:24 pinned by digest (node@sha256:9d266751ca3d7fdbe651c878ffd5ad0a42fed49a18c283509267c97f025b7496, v24.22.3). The container ran with --network none --read-only --cap-drop ALL, uid 65534, and the repo mounted :ro:

    main @ c1c8a8c   tests 65  pass 65  fail 0  skipped 0
    this branch      tests 65  pass 65  fail 0  skipped 0
    
  • git diff --check is clean.

Limitations

These are instruction changes and were not exercised in an end-to-end agent audit.

This PR was prepared with AI assistance (Claude Code). The verification output above is from local runs.

🤖 Generated with Claude Code

Add a universal rule that a real secret found in the target is never
copied into findings.json, the ledger, the reports, scratch or artifact
files, or any agent hand-off. Records carry only the secret type, its
repository-relative location, and a short SHA-256 value hash (omitted
for values under 16 characters). The hunter prompt repeats the rule.

Add a Validator execution section: findings.json and the coverage
ledger carry target-derived text, so run both validators inside the
same OS-enforced sandbox as target code (no network, empty environment,
read-only inputs, no writable path, the same limits) instead of host
node, and disclose any run outside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant