Skip to content

fix(code-review): keep findings outside the diff - #315

Merged
wongk merged 6 commits into
mainfrom
fix/iss-12737-keep-out-of-diff-findings
Oct 20, 2026
Merged

wongk merged 6 commits into
mainfrom
fix/iss-12737-keep-out-of-diff-findings

Conversation

@wongk

@wongk wongk commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Campaign chunk C1 of 3 (sequential): keep findings outside the diff.

Implements PLN-2214 · PR 1 for ISS-12737 (plan). code-review 3.10.5 → 3.11.0.

What changes

The validator used to discard every finding on a file outside the diff (DISCARD_FILE_NOT_CHANGED) before verification, at any priority. Code outside the diff is often exactly what a change breaks. In 34 local review runs, this discarded 6 findings that were all caused by their diff, three of them P1 ("stale mocks crash every BranchesPage test").

  • Validator: a finding on a file outside the diff is kept and tagged out_of_diff. It goes through the normal 0.5 confidence floor, not the 0.80 out-of-hunk floor. A diff-scope finding with no file is now discarded as DISCARD_MISSING_FILE.
  • Verifier owns causation: every out_of_diff finding is verified. It bypasses the verify cache and does not use the 50 in-diff verification slots. The verifier searches the whole patch for the change that caused it, and rejects with the new rejection_class: not_caused_by_diff if there is none.
  • Cache: BHA findings outside the diff are filed under the diff file their evidence[] cites, or under every file of their partition. They are dropped on a cache hit once their own file is in the diff.
  • Posting: on GitHub, an out_of_diff finding is posted inline at the diff line its evidence[] cites. Otherwise it appears in the summary only. It is labelled "caused by this change" only when the verifier upheld it, and "causation unconfirmed" otherwise.
  • Prompts: the reviewer template, Bug Hunter B and fast-path prompts now say findings outside the diff are in scope when this diff caused them, and must cite the causing line in evidence[]. The shared prompt spells out that shape.
  • Schema docs: SCHEMA.md documents out_of_diff, kept_out_of_diff, not_caused_by_diff and the previously missing evidence_contradicted. discarded_file_not_changed is removed; it had no consumers.

Acceptance ledger

30 criteria: 27 met, 3 met differently, 0 not met.

Deviations from the plan

  • Comment anchoring: post-comments is only given --findings, so it has no diff data. To place an out_of_diff finding inline, it reads the PR's files from GitHub, and only when the run has such a finding. PR 2's deleted-side (side=LEFT) posting should reuse this lookup.
  • Schema: out_of_hunk_kept was never in code_review_schema.py, so both it and out_of_diff were added there as optional booleans.
  • Presenter label: the presenters had no companion-change label to copy. Out-of-diff findings get their own label, and out-of-hunk findings are unchanged.
  • Old-vs-new comparison: ran over the 31 of 34 local runs that have the validator's inputs.

Review

Two /code-review:deep passes, both proven against this worktree's commits:

  • First pass: APPROVED, 9 verified findings, all MEDIUM or LOW.
    • Fixed 7: stale cache served after the file was fixed; verifier priority inversion; misleading "caused by this change" label; undefined evidence[] shape; partial patch reads; a duplicated regex; a stale start.md note.
    • Pushed back 1: a hard cap on out-of-diff verifications. It would let findings surface unverified, which this PR exists to prevent. The starvation half is fixed.
    • Declined 2:
      • Consolidating partitions.json parsing, which already existed before this PR.
      • Keeping not_caused_by_diff through escalation, which changes a contract shared by every rejection class; the label fix covers the visible harm.
  • Re-review of the fixes (incremental): APPROVED, 0 findings. It ran as a single fast-path reviewer because the incremental diff was small.
  • Caveats:
    • Both reviews ran the plugin helpers from this branch, not the installed plugin, because CLAUDE_PLUGIN_ROOT was empty.
    • In the first pass, signal extraction returned no signals, so the Impact Analyzer was never considered.
    • Signal extraction reused a stale cached result because its cache key uses the literal string HEAD rather than a commit. That is a pre-existing plugin bug, not part of this PR.

Verification

  • uv run pytest plugins/code-review/tools/python/ -q: pass (1,418 after the fixes).
  • uv run ruff check .: pass.
  • uv run pyright: 0 errors.
  • Counterfactuals: 12 mutations of code_review_helpers.py; every new test went red under its mutation.
  • Old vs new validate over 31 local review runs: exactly the 6 findings from ISS-12737 flip from discarded to kept, and nothing else changes.

Still owed by later chunks

  • C2 · PR 2: route deleted code to the cross-file reviewers.
  • C3 · PR 3: moved-file context. Built only if a dry run over recent merged PRs finds moved-and-edited files.

wongk and others added 6 commits October 20, 2026 10:24
PLN-2214 PR 1 (ISS-12737): the validator keeps and tags out_of_diff
findings instead of discarding them; verify-prepare always verifies
them; cache, post-comments, presenters and prompts handle them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wongk
wongk merged commit 977f429 into main Oct 20, 2026
10 checks passed
@wongk
wongk deleted the fix/iss-12737-keep-out-of-diff-findings branch October 20, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant