Repository navigation
Conversation
Add a universal rule that a real secret found in the target is never copied into findings.json, the ledger, the reports, scratch or artifact files, or any agent hand-off. Records carry only the secret type, its repository-relative location, and a short SHA-256 value hash (omitted for values under 16 characters). The hunter prompt repeats the rule. Add a Validator execution section: findings.json and the coverage ledger carry target-derived text, so run both validators inside the same OS-enforced sandbox as target code (no network, empty environment, read-only inputs, no writable path, the same limits) instead of host node, and disclose any run outside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #148 (the "Secrets in reports" hardening note).
Problem
1. Real secrets found in the target can be copied verbatim into the outputs.
ATTACK-CLASSES.mdsends the "Obvious things" agent looking for hardcoded passwords, API keys, tokens,-----BEGINblocks,.env,*.pemandcredentials.json, and it also asks for "secrets that were committed then removed (still in history)". When the agent finds one, nothing says how to record it.SKILL.mdcovers only the ambient environment during reproduction ("Launch from an empty environment rather than trying to redact one after execution"), andpromote the minimum non-secret resultapplies only to artifacts.trace,evidence,payloads,observed_result,REPORT.md,FINDINGS-DETAIL.md, and the hunter-to-parent and parent-to-verifier hand-offs have no rule at all. An audit of a repository with a leaked key can therefore produce a second copy of that key in every report file and agent transcript.2. The validators are run with host
nodeon target-derived input.RECONNAISSANCE.md,SKILL.md(workflow step 4) andVALIDATION-AND-REPORTING.mdPhase 4 all tell the parent to runnode <skill-dir>/validate-*.cjs <output-dir>/...jsonwith no execution context. Both files carry strings that came from the target: titles, scopes, payloads, observed output, and file names. The validators are careful: they use onlyfs,pathandutil, open withO_NOFOLLOW, cap input at 5 MiB, and never write, spawn, or touch the network. Even so, this is the only step in the workflow that hands target-influenced bytes to an interpreter outside the OS-enforced sandbox the skill requires everywhere else. It also runs in the parent's full environment, credentials included.Change
Docs only. 4 files, +14/−4.
SKILL.md› Universal execution safety: a new paragraph says a real secret found in the target is never written, in whole, in part, or in any reversible encoding, intofindings.json, the ledger, the three report files, scratch or artifacts, or any prompt or hand-off. Records keep only the type, the repository-relative location (path and line, or commit and path for history), and a short value hash:sha256:plus the first 12 hex digits. The hash is omitted for values under 16 characters, because a short hash of a short value can be brute-forced. The text calls it a "value hash" so it is not confused with the recordfingerprint. A hunter or verifier result that contains a value is malformed and is re-run, matching how the skill already handles malformed results. Liveness is never tested. Validity and rotation are owner-observed facts.SKILL.md› new### Validator execution: both validators run inside the same OS-enforced sandbox as target code. That means no network, an empty environment, the skill directory and the input file read-only, no writable path, and the same limits. A pass counts only if the parent saw it come out of that sandbox. If no sandbox is available, the validators still run, andREPORT.mdandrun-metadata.jsonmust say so. Workflow step 4 points to this section.HUNTING.md: the hunter prompt repeats the secret rule in its own words, since hunters receive that file and notSKILL.md.VALIDATION-AND-REPORTING.md: the record rules restate the secret rule for records and derived reports, and the Phase 4Run:line points to Validator execution.RECONNAISSANCE.md: the ledger-validator command points to Validator execution.Before / after
Before (main @ c1c8a8c), the only redaction language in the skill:
and the validator instruction:
After: an evidence entry for a committed key looks like this:
{"file": "config/prod.env", "line": 7, "description": "AWS access key pair committed in plaintext (type: aws-access-key; value hash sha256:3f1c9a0b7e22). Not tested for liveness; owner to confirm validity and rotate."}and Phase 4 reads: "Run both validators as Validator execution in
SKILL.mdrequires (inside the target-code sandbox, not with the hostnode)".Overlap with open PRs
I read #149, #147, #151, #141, #129, #96, #75 and #108 before writing this, and grepped every open PR diff for
redactand for guidance on where the validators run.validate-environment.cjsto check the sandbox, which complements this. Require verbatim quotes in trace/evidence and check them with --target (#134) #141 and Harden the audit workflow against target-influenced results #96 change the validator arguments (--target). This PR changes only the prose around the command, so whichever merges first leaves a one-line conflict on theRun:line.Validator test change?
None. Validators can't tell a real secret from a documented dummy such as
AKIAIOSFODNN7EXAMPLE, so a schema or validator check would either miss real values or reject legitimate test vectors. This is a rule for agents, enforced at hand-off. If you want a mechanical check later, I can follow up with a small high-precision check (PEM private-key blocks only, for example) and tests, as a separate PR.Verification
Docs-only. No validator, schema, or test file changed.
node --testfromskills/security-audit/, run innode:24pinned by digest (node@sha256:9d266751ca3d7fdbe651c878ffd5ad0a42fed49a18c283509267c97f025b7496, v24.22.3). The container ran with--network none --read-only --cap-drop ALL, uid 65534, and the repo mounted:ro:git diff --checkis clean.Limitations
These are instruction changes and were not exercised in an end-to-end agent audit.
This PR was prepared with AI assistance (Claude Code). The verification output above is from local runs.
🤖 Generated with Claude Code