Skip to content

fix(code-review): key coverage-critic cache on the diff too - #193

Open
mikeangstadt wants to merge 11 commits into
mainfrom
fix/iss-9674-coverage-critic-cache-key
Open

mikeangstadt wants to merge 11 commits into
mainfrom
fix/iss-9674-coverage-critic-cache-key

Conversation

@mikeangstadt

Copy link
Copy Markdown
Collaborator

Stacked on #192 (base is fix/iss-8961-review-signal-cache-key, not main). Retarget to main after #192 merges. ISS-9674.

Summary

  • This is ISS-8961 in the coverage_critic/ namespace, and it was safe only by accident. coverage_critic_cache_key was (coverage_plan_initial_hash, signals_hash, diff_tip, prompt_hash, available_reviewers_hash). _build_coverage_critic_input returns (main_input, diff_summary) where diff_summary is literally _build_signal_input(diff_data, intent_summary=None) — the bundle the critic reads — and it appeared in no key component. As in fix(code-review): key signal cache on the diff, not the worktree #192, diff_tip is a ref name and never a commit id, so nothing in the tuple varied with the diff, while the cache directory is the pooled worktree's basename.
  • The trap this closes. The namespace did not misfire only because signals_hash hashes extract_signals.json, which carries a wall-clock generated_at. The key therefore changed every run and the cache never hit. Anyone looking at a namespace with a 0% hit rate would find that timestamp, remove it, measure a real speedup, and ship a critic that serves one review's coverage plan to a different review — with no test failing. That is the whole reason this is a separate, filed ticket rather than a footnote.
  • The key now includes diff_summary_hash (signal_input_hash over that bundle), and cmd_coverage_critic_prepare builds the agent input before computing the key rather than after. The manifest reports diff_summary_hash beside its sibling component hashes.
  • Corrects the key docstring's false claim that "All five are content-addressed" (diff_tip is not), and the SCHEMA.md coverage-critic row, which documented the path as <diff_tip>.json when the code writes <key>.json and listed three of the five components. These are the two doc errors fix(code-review): key signal cache on the diff, not the worktree #192 deliberately left on the adjacent table row to keep that PR to one subject.

Breaking changes

None — additive. One-time invalidation of the coverage_critic/ namespace, which in practice never hit. coverage_critic_cache_key() gains a positional argument; it has no callers outside this module and its tests.

UI Feature Flag

[ui-flag:N/A] — tooling-only change in a Python helper; no UI surface in this repo.

Test plan

  • test_pooled_lane_with_a_different_diff_misses_prior_cache is the regression test: one shared cache directory, and plan_initial, signals, prompt, roster, and diff_tip all held constant so that only the diff varies — which is precisely what the old key could not see.
  • Counterfactual: dropping diff_summary_hash back out of the key payload turns 3 tests red, including that one. Restored: green.
  • Key composition pinned to a frozen literal digest, not re-derived through the builder under test.
  • test_cache_hit_serves_directly no longer re-derives the key to seed its fixture; it prepares once and seeds under the key the manifest reports — the false-green shape caught in fix(code-review): key signal cache on the diff, not the worktree #192's review.
  • diff_summary_hash is normalized in the golden harness like its sibling hashes: it is fixture-coupled rather than wall-clock, so pinning it in goldens would force a hand-updated digest on every fixture diff edit. Composition is pinned by the unit test instead.
  • ruff check clean; uv run pyright 0 errors; uv run pytest plugins/ 2119 passed, 0 failed.

- ISS-8961: the signals/ namespace key was (diff_tip, taxonomy_hash,
  prompt_hash), and diff_tip is a ref name -- literally "HEAD" for a
  local branch review -- so no component varied with the diff. The
  cache dir is ~/.claude/cr-cache-global-repo-<worktree basename>, so
  two reviews from one pooled directory shared directory and key, and
  the second was served the first's extracted signals.
- Add signal_input_hash: SHA-256 over the canonical serialization of
  the agent input bundle extract-signals-prepare builds, which is
  exactly what the extraction agent reads. Key is now (diff_tip,
  input_hash, taxonomy_hash, prompt_hash).
- Build the bundle before computing the key rather than only on a miss.
- A bundle with no changed files has no diff identity: emit an empty
  cache_key plus cache_bypass_reason "no-changed-files", which
  suppresses the read and (via consolidate's empty-key guard) the write.
- Correct the --diff-tip help, which claimed the flag was a SHA.

Testing: pytest plugins/ (3 pre-existing failures unrelated to this
change: TestOverrideCache/TestPR114ReviewFixes fail on clean main
because their override fixture is dated 2026-05-29 and the overrides
TTL is 90 days). Counterfactual: reverting code_review_helpers.py turns
TestPooledWorktreeSignalCacheIsolation red with lane 2's manifest
reading status "cache_hit". ruff check; pyright.

Risks: cached extractions written under the old key scheme are misses
after this lands, so the first review in each cache directory re-runs
one Haiku extraction.
- Close a false green in the pooled-worktree regression test: it varied
  the changed PATH as well as the content, so a key hashing only the
  file list still passed. Add a lane pair holding path, status, and
  line counts identical and varying only the excerpt -- the shape two
  reviews of one branch actually take as it is amended.
- Delegate signal_input_hash to _stable_json_hash, the module's existing
  owner of the deterministic cache-key JSON rule, instead of a
  byte-identical second copy of it.
- Pin the bundle's run-to-run stability: a run-varying field added to
  _build_signal_input would make every review a permanent miss, and the
  golden harness cannot catch it because it normalizes cache_key.
- Emit input_hash in the manifest beside taxonomy_hash and prompt_hash,
  so the one component that actually varies is visible when diagnosing
  a key move. Normalized in the golden harness like its siblings.
- State the real reason the no-changed-files bundle bypasses the cache.
  It is not that such a bundle cannot be keyed -- it is the shape a
  degraded parse-diff takes, and caching it would persist a degraded
  extraction for the namespace TTL. Name the cost too: a genuinely
  empty review now re-dispatches its extraction every run.

Testing: uv run pytest plugins/ -- 2113 passed, 3 pre-existing failures
(override-TTL fixture dated 2026-05-29 vs a 90-day TTL, red on main).
ruff check; uv run pyright 0 errors. Counterfactual: restoring
origin/main's helper turns 13 tests red, including both pooled-lane
tests. Mutation: narrowing the key to the file list alone is now killed
by test_same_file_different_content_is_not_served_a_stale_hit.

Risks: manifest gains an additive input_hash field; no consumer reads
it, and the golden harness normalizes it.
Mechanics repair, not an expectation change. The behaviour under test
is unchanged and still asserted: a valid, within-TTL override
short-circuits verification, routes through consolidate as RE_ASSERTED,
and lands in the by-reviewer stats. No assertion was weakened, widened,
skipped, or deleted.

Three tests hardcoded asserted_at "2026-05-29T22:00:00+00:00" while
_override_is_expired sweeps the overrides namespace against the real
wall clock with a 90-day TTL. The fixture was therefore a time bomb: it
went red on 2026-08-27 with no code change, and the suite has been red
on main ever since (main last ran CI 2026-08-17, green). A test that
fails on a calendar date rather than a behaviour change was never
pinning what it claimed to.

Both sides of the comparison are now literals -- the fixture timestamp
and the clock -- so the gap stays a constant 3 days forever. The TTL
gate itself is untouched, and the tests whose subject IS the TTL keep
offsetting from the real clock on purpose.

Testing: uv run pytest plugins/ -- 2116 passed, 0 failed (first green
run since 2026-08-27). Mutation: forcing _override_is_expired to return
True turns all three repaired tests red, so the pin did not neuter the
gate they depend on. ruff check; uv run pyright 0 errors.

Risks: none identified. Test-only change.
ISS-9674, the coverage-critic twin of ISS-8961.

- coverage_critic_cache_key was (coverage_plan_initial_hash,
  signals_hash, diff_tip, prompt_hash, available_reviewers_hash).
  _build_coverage_critic_input returns (main_input, diff_summary) where
  diff_summary is literally _build_signal_input(diff_data, None) -- the
  bundle the critic reads -- and it was in no key component. diff_tip is
  a ref name, never a commit id, so nothing varied with the diff.
- It did not misfire only because signals_hash hashes
  extract_signals.json, which carries a wall-clock generated_at, so the
  key changed every run and the namespace never hit. That made the
  obvious optimization -- strip generated_at so the cache finally hits --
  a change that would serve one review's coverage plan to another review
  from the same pooled worktree, with no test failing.
- Add diff_summary_hash to the key; build the input before the key
  rather than after; report the hash in the manifest beside its
  siblings.
- Correct the docstring's false "All five are content-addressed" claim
  and the SCHEMA.md row (wrong path shape, 3 of 5 components listed).
- Reseed the cache-hit test from the key prepare reports instead of
  re-deriving it through the builder under test.

Testing: uv run pytest plugins/ -- 2119 passed, 0 failed. Counterfactual:
dropping diff_summary_hash from the payload turns 3 tests red, including
test_pooled_lane_with_a_different_diff_misses_prior_cache, which holds
plan, signals, prompt, roster and diff tip constant and varies only the
diff. ruff check; uv run pyright 0 errors.

Risks: one-time invalidation of the coverage_critic namespace, which in
practice never hit anyway.
@shafty023

Copy link
Copy Markdown
Collaborator

Beginning a code-review. Please allow time for me to finish before merging the PR

@shafty023

Copy link
Copy Markdown
Collaborator

Beginning a code-review. Please allow time for me to finish before merging the PR

mikeangstadt and others added 4 commits November 3, 2026 12:41
main's #321 already keys the signals cache on the agent input bundle and
drops diff_tip from the key; this branch keeps that key shape and retains
its additive pieces (input_hash in the manifest, the no-changed-files
cache bypass). The override-clock pin is dropped in favour of main's
_iso_days_ago fixtures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…674-coverage-critic-cache-key

main's #321 already keys the coverage_critic cache on the diff summary
(without diff_tip), passes the summary to the critic via extra_inputs, and
seeds a prior entry in its changed-diff regression. This branch keeps that
shape and retains diff_summary_hash in the manifest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from fix/iss-8961-review-signal-cache-key to main November 3, 2026 18:56
mikeangstadt added a commit that referenced this pull request Nov 3, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikeangstadt

Copy link
Copy Markdown
Collaborator Author

Review request: Lint, Type Check and Tests pass on the head, there are 0 unresolved threads, and the branch merges cleanly with main. The ruleset needs one approving review. Members of @closedloop-ai/engineering, please take a look.

…-critic-cache-key

# Conflicts:
#	CHANGELOG.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants