Escalate Blast Radius probing for frequently-regressed component families

[PureWeen]

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Summary

Escalates the code-review skill's Blast Radius Assessment so it also fires for behavioral changes to .NET MAUI's frequently-regressed component families, and defines what a reviewer should actually do when the change is prose rather than code.

Scope: one file, .github/skills/code-review/SKILL.md. No eval, test, or product code changes.

⚠️ This description was previously inaccurate. It described adding 3 scenarios to .github/skills/code-review/tests/eval.yaml (6 → 9), expect_activation: false flags, and a changelog header. None of that is in this PR — and none of it exists in the repo: eval.yaml was deleted when the suite migrated to Vally (ee47f113d5), expect_activation appears nowhere, and the cited review-rules.md does not exist. Rewritten to match the diff.

Changes

Step 6 — Blast Radius Assessment

• Extends the mandatory Blast-Radius + Failure-Mode escalation to the eight frequently-regressed families: CollectionView, CarouselView, Image/Graphics, Theme/Style, Gesture/Tap, Button/Entry, Toolbar, Shell/TabBar. The list is stated inline and is sufficient on its own; .github/agents/maui-expert-reviewer.md's Frequently Regressed Components table mirrors it with per-family risk areas and is read for that detail when present.
• Forbids claiming the Step 2 expert reviewer's Regression Prevention dimension "fired." Its output is findings-only, so an empty result cannot distinguish ran and found nothing from never ran — the reviewer runs the probes itself either way.
• Carves out genuinely inert prose, but defines what is not inert (public API docs, analyzer/compiler directives, agent-instruction files this repo executes, comments encoding a precondition) and gives those changes probes that are actually answerable — the contract the text encodes — rather than routing them into runtime questions about startup ordering and PlatformView nullity.

Step 1 — retrieval robustness

• A failed or unauthenticated retrieval command is a fact about one tool, not about the review. Directs a retry through another read-only route instead of asking the caller to paste the diff.

Review output template

• The Blast Radius gloss said infrastructure/handler/platform only, so a reviewer following the template could omit the section for either trigger added here.

Why the wording is load-bearing

SKILL.md is an executable agent-instruction file and is overlaid into eval fixtures, so edits here are prompt changes with measurable behavior. That was demonstrated during this PR: an earlier revision naming review_input as the remedy for a deficient environment taught the agent that a failed gh call meant it should stop and ask the user to paste the diff, dropping the happy-path-code-review-request stimulus from 3/3 to 0/3. The Step 1 change above targets the residual bail.

Known gaps

• No eval stimulus exercises this text. Verified: no file under tests/*.vally.yaml references any of the eight families, Frequently Regressed, or the prose-inertness rule. A future change could invert this guidance and the suite would still report green. Closing this needs a fixture plus grader thresholds and is proposed as follow-up rather than bundled here.
• One happy-path trial still bails (2/3 vs baseline 3/3). With n=3, and the baseline itself having bailed 1/3 on an earlier run, this is not yet distinguishable from flake.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34884

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34884"

[Copilot]

Pull request overview

This PR expands the code-review skill’s evaluation coverage by adding new scenarios specifically aimed at detecting regression-risk reasoning, based on real “regressed-in-10.0.60” issue patterns. It also strengthens two existing negative-trigger scenarios by explicitly asserting the skill should not activate.

Changes:

• Added a v2 changelog header to the eval file.
• Added 3 new regression-focused eval scenarios (raising total scenarios from 6 to 9).
• Added expect_activation: false to two negative-trigger scenarios to ensure the workflow doesn’t activate on informational prompts.

[Copilot]

The output_not_contains: "no concerns" / "looks safe" assertions are both case-sensitive and very generic. This can make the scenario flaky: the model might respond with capitalized variants ("No concerns"/"Looks safe") which would bypass the guard, or use these substrings in a longer sentence while still raising concerns. Consider tightening these to more specific phrases you want to forbid and/or include the common capitalization variants so the assertion matches the intended anti-pattern reliably.

Suggested change

	value: "no concerns"
	- type: "output_not_contains"
	value: "looks safe"
	value: "no concerns about side effects"
	- type: "output_not_contains"
	value: "No concerns about side effects"
	- type: "output_not_contains"
	value: "looks safe to me"
	- type: "output_not_contains"
	value: "Looks safe to me"

[Copilot]

The assertion output_not_contains: "regression risk" conflicts with the rubric expectation that the agent should identify this as doc-only and explicitly state there's no regression risk. A correct response like "no regression risk" would still contain the banned substring and fail the scenario. Consider removing this assertion or narrowing it to warning-style phrasing (e.g., "high regression risk" / "regression-prone") so correct low-risk language remains allowed.

Suggested change

	value: "regression risk"
	- type: "output_not_contains"

[kubaflo]

/review -b feature/refactor-copilot-yml

[MauiBot]

Expert Review — 2 findings

See inline comments for details.

[MauiBot]

[moderate] Eval robustness — This assertion only checks for the substring regression, so an answer like "there is no regression concern" (or one that mentions regression without asking for adjacent CollectionView coverage) would pass the automated check. Since the scenario is intended to prevent false positives/false negatives around high-risk CollectionView fixes, assert the concrete expected behavior with less ambiguous terms, e.g. adjacent scenario coverage for scroll/tap/swipe/gesture or a NEEDS_CHANGES/NEEDS_DISCUSSION verdict.

[MauiBot]

AI Review Summary

@PureWeen — new AI review results are available based on this last commit: 03821c6.

---

🗂️ Review Sessions — click to expand

🚦 Gate — Test Before & After Fix

Gate Result: ⚠️ SKIPPED

No tests were detected in this PR.

Recommendation: Add tests to verify the fix using the write-tests-agent.

---

📋 Pre-Flight — Context & Validation

Issue: N/A - No linked issue found
PR: #34884 - Escalate Blast Radius probing for frequently-regressed component families
Platforms Affected: android requested for testing; implementation affects agent review behavior across all platforms
Files Changed: 1 implementation/instruction, 0 test

Key Findings

• PR changes one executable agent-instruction file: .github/skills/code-review/SKILL.md.
• The current PR fix broadens Blast Radius / Failure-Mode probing and adds retrieval-failure guidance.
• Gate was already skipped because no tests were detected in the PR; no gate verification was re-run.
• The strongest pre-flight concern is that the unauthenticated fallback still suggests gh api, which fails under the same no-auth condition as gh pr diff.
• Existing Vally specs do not directly exercise the new frequently-regressed-family/prose-inertness trigger.

Code Review Summary

Verdict: NEEDS_CHANGES
Confidence: low
Errors: 1 | Warnings: 1 | Suggestions: 0

Key code review findings:

• ✗ .github/skills/code-review/SKILL.md:70 — unauthenticated fallback suggests another authenticated gh path; use anonymous HTTP or verified local refs instead.
• ⚠ .github/skills/code-review/SKILL.md:206 — new executable behavior has no direct eval coverage.

Fix Candidates

#	Source	Approach	Test Result	Files Changed	Notes
PR	PR #34884	Add retry wording using gh api or local git diff, plus broaden Blast Radius / Failure-Mode probing for frequently-regressed families and non-inert prose	⚠️ SKIPPED (Gate: no tests detected)	.github/skills/code-review/SKILL.md	Original PR; pre-flight found misleading unauthenticated fallback

---

🔬 Code Review — Deep Analysis

Code Review — PR #34884

Independent Assessment

What this changes: Updates .github/skills/code-review/SKILL.md to make code-review agents more resilient when PR retrieval fails and to require Blast Radius / Failure-Mode probing for behavioral changes in high-regression MAUI component families. It also clarifies how to treat non-inert prose changes in executable instructions.

Inferred motivation: Prevent reviewers from bailing when gh retrieval fails, and reduce missed adjacent-scenario regressions in historically fragile MAUI areas.

Reconciliation with PR Narrative

Author claims: The PR intentionally changes one executable skill file, adds retrieval fallback guidance, extends Step 6 probing to frequently-regressed component families, and documents a known gap: no Vally eval stimulus exercises the new family/prose-inertness behavior.

Agreement/disagreement: The diff matches the narrative. However, the retrieval fallback text still names gh api as an alternate route for unauthenticated gh pr diff failures, which fails under the same missing-auth condition.

Prior Review Reconciliation

Prior ❌ Error Finding	Source	Status	Evidence
Stale/non-actionable Regression Prevention section reference	MauiBot review / inline	✅ Fixed	Final SKILL.md points to .github/agents/maui-expert-reviewer.md's Frequently Regressed Components table and no longer references stale section numbers.
Eval wording blocked valid "no regression risk" language	MauiBot review	🔄 Obsolete	Final net diff no longer changes eval.yaml; PR-head file list contains only .github/skills/code-review/SKILL.md.
Duplicated/collapsed component list could drift	Prior issue comment	✅ Fixed / accepted	Later review comment states all prior findings fixed in 03821c6; final wording explicitly says the inline list is complete and the agent table mirrors it for detail.

Blast Radius Assessment

• Runs for all instances: Yes — this is an executable skill instruction and affects every future invocation of the code-review skill.
• Startup impact: No product startup impact; it changes agent review behavior only.
• Static/shared state: No runtime static state, but the prompt text is shared review infrastructure.

CI Status

• Required-check result: Undetermined. gh pr checks 34884 --repo dotnet/maui --required failed because GitHub CLI is unauthenticated.
• Classification: Tool-unavailable / undetermined for required checks. Anonymous API showed skill-validation success and several successful check runs, but that does not identify required checks.
• Action taken: Capped confidence at low per Step 5/6.

Findings

❌ Error — Unauthenticated fallback suggests another authenticated gh path

.github/skills/code-review/SKILL.md:70 says that after a failing or unauthenticated retrieval command, reviewers should retry via gh api repos/dotnet/maui/pulls/<PR_NUMBER>/files or local git diff. In the concrete unauthenticated failure mode, gh api fails for the same reason as gh pr diff; this happened in this environment. Unless the PR branch is already materialized locally, the instruction still leaves the reviewer without a working route. The fallback should name an actually anonymous read-only path, such as curl against the GitHub REST API / raw diff URL, or require verifying local base/head refs before using git diff.

⚠️ Warning — New executable behavior has no direct eval coverage

.github/skills/code-review/SKILL.md:206 adds mandatory probing for eight frequently-regressed component families, but the Vally tests do not appear to exercise this new automatic trigger. Existing Blast Radius coverage targets handler/platform infrastructure, not a Gesture/Tap, Theme/Style, Button/Entry, or similar behavioral PR whose prompt does not already demand probing. The PR description acknowledges this as a known gap, but it means the skill suite could stay green if this guidance regresses.

Failure-Mode Probing

• Unauthenticated gh pr diff: The new text routes to gh api, which also requires auth, so the reviewer can still fail unless it independently discovers curl or has a local PR checkout.
• Local checkout fallback: git diff is safe only when the local checkout has the correct PR head/base refs; otherwise it may review unrelated local changes.
• Frequently-regressed component trigger: A future Gesture/Tap PR could omit adjacent-scenario probing and current eval coverage would not catch that omission.

Verdict: NEEDS_CHANGES

Confidence: low

Summary: The PR improves the review skill's intent, but the newly added unauthenticated fallback is concretely misleading in the failure mode it is meant to fix. CI required-check status could not be verified with gh, so confidence is capped low.

---

🛠️ Fix — Analysis & Comparison

Fix Candidates

#	Source	Approach	Test Result	Files Changed	Notes
1	maui-expert-reviewer + code-review finding	Replace authenticated gh api fallback with anonymous curl REST/raw-diff routes and require verified refs for local git diff	✅ PASS	.github/skills/code-review/SKILL.md	Fixes the concrete pre-flight error and keeps non-interactive/no-auth review behavior
PR	PR #34884	Retry through gh api or local git diff, plus broaden Blast Radius / Failure-Mode probing for frequently-regressed families and non-inert prose	⚠️ SKIPPED (Gate)	.github/skills/code-review/SKILL.md	Original PR; gate was pre-run and skipped because no tests were detected

Cross-Pollination

Model	Round	New Ideas?	Details
maui-expert-reviewer	1	Yes	Anonymous HTTP fallback; verified local-ref fallback; fail-closed retrieval gate

Exhausted: No — stopped because Candidate #1 passed all available validation and is demonstrably better than the PR's current fallback wording.
Selected Fix: Candidate #1 — it addresses the code-review ❌ finding directly, avoids retrying another authenticated gh command in unauthenticated environments, preserves autonomous operation, and keeps local git use constrained to verified refs.

Attempt Narrative

try-fix-1 — Anonymous HTTP fallback

• Approach: Replace gh api fallback with unauthenticated curl routes for public PR files/raw diff and require verified PR base/head refs for local git diff.
• Validation: Static checks confirmed the misleading gh api fallback was removed and both anonymous routes plus verified-ref wording are present. Existing code-review eval specs linted successfully with Vally.
• Failure analysis: None; the candidate passed.

Deferred Candidate Ideas

These were generated by the expert reviewer but not run because the stop condition was met:

Candidate	Approach	Reason deferred
Verified local-ref fallback	Fetch and verify public PR head/base refs before permitting git diff	More verbose than Candidate #1 and does not help environments without reliable git refs as directly as anonymous HTTP
Fail-closed retrieval gate	Require authenticated gh, anonymous HTTP, or verified local refs; otherwise mark review blocked	Safer but less autonomous; Candidate #1 preserves autonomous recovery while fixing the defect

---

📝 Recommended PR Title & Description

Assessment: ✏️ Recommend updating — the current title and description match the raw PR, but the winning fix also replaces the authenticated gh api fallback with anonymous HTTP / verified-local retrieval guidance.

Recommended title

[Agent] Code Review: Escalate blast-radius probing and harden diff retrieval

Recommended description

## Summary

Escalates the code-review skill's **Blast Radius Assessment** so it also fires for behavioral changes to .NET MAUI's frequently-regressed component families, defines what a reviewer should do when the change is prose rather than code, and hardens PR diff retrieval after a failed or unauthenticated `gh pr diff`.

**Scope: one file, `.github/skills/code-review/SKILL.md`.** No eval, test, or product code changes.

> ⚠️ **This description was previously inaccurate.** It described adding 3 scenarios to `.github/skills/code-review/tests/eval.yaml` (6 → 9), `expect_activation: false` flags, and a changelog header. None of that is in this PR — and none of it exists in the repo: `eval.yaml` was deleted when the suite migrated to Vally (`ee47f113d5`), `expect_activation` appears nowhere, and the cited `review-rules.md` does not exist. Rewritten to match the diff.

### Changes

**Step 1 — retrieval robustness**
- Treats a failed or unauthenticated retrieval command as a fact about that one tool, not about the review.
- Directs retry through actually read-only fallback routes: anonymous PR file metadata / raw diff HTTP endpoints, or a local `git diff` only when the checkout has verified PR base/head refs.
- Keeps the non-interactive behavior: report inability to review only after the anonymous HTTP and verified-local routes also fail, and never ask the caller to paste the diff.

**Step 6 — Blast Radius Assessment**
- Extends the mandatory Blast-Radius + Failure-Mode escalation to the eight frequently-regressed families: CollectionView, CarouselView, Image/Graphics, Theme/Style, Gesture/Tap, Button/Entry, Toolbar, Shell/TabBar. The list is stated inline and is sufficient on its own; `.github/agents/maui-expert-reviewer.md`'s `Frequently Regressed Components` table mirrors it with per-family risk areas and is read for that detail when present.
- Forbids claiming the Step 2 expert reviewer's Regression Prevention dimension "fired." Its output is findings-only, so an empty result cannot distinguish *ran and found nothing* from *never ran* — the reviewer runs the probes itself either way.
- Carves out genuinely inert prose, but defines what is **not** inert (public API docs, analyzer/compiler directives, agent-instruction files this repo executes, comments encoding a precondition) and gives those changes probes that are actually answerable — the contract the text encodes — rather than routing them into runtime questions about startup ordering and `PlatformView` nullity.

**Review output template**
- The Blast Radius gloss said infrastructure/handler/platform only, so a reviewer following the template could omit the section for either trigger added here.

### Why the wording is load-bearing

`SKILL.md` is an executable agent-instruction file *and* is overlaid into eval fixtures, so edits here are prompt changes with measurable behavior. That was demonstrated during this PR: an earlier revision naming `review_input` as the remedy for a deficient environment taught the agent that a failed `gh` call meant it should stop and ask the user to paste the diff, dropping the `happy-path-code-review-request` stimulus from 3/3 to 0/3. The Step 1 change above targets the residual bail without routing unauthenticated environments back through another authenticated `gh api` command.

### Known gaps

- **No eval stimulus exercises this text.** Verified: no file under `tests/*.vally.yaml` references any of the eight families, `Frequently Regressed`, or the prose-inertness rule. A future change could invert this guidance and the suite would still report green. Closing this needs a fixture plus grader thresholds and is proposed as follow-up rather than bundled here.
- One `happy-path` trial still bails (2/3 vs baseline 3/3). With n=3, and the baseline itself having bailed 1/3 on an earlier run, this is not yet distinguishable from flake.

---

🏁 Report — Final Recommendation

Comparative Report — PR #34884

Candidates compared

Rank	Candidate	Result	Assessment
1	pr-plus-reviewer	✅ Best	Preserves the PR's Blast Radius / non-inert-prose escalation and applies the expert reviewer's actionable fix: replace the misleading authenticated gh api retry with anonymous HTTP routes plus verified-local-ref fallback. This resolves the only concrete error found against the PR fix.
2	try-fix-1	✅ PASS	Functionally equivalent to the reviewer-applied retrieval fix and passed the available static/Vally validation. It ranks below pr-plus-reviewer only because pr-plus-reviewer is the same improvement applied directly on top of the submitted PR candidate.
3	pr	⚠️ Gate skipped; expert review found error	The raw PR improves review behavior for frequently-regressed families and non-inert prose, but leaves a bad unauthenticated fallback: retrying gh api after an unauthenticated gh pr diff fails for the same reason.

Regression-test ordering

No candidate failed regression tests. The raw PR gate was skipped because no tests were detected. try-fix-1 passed the available static checks and Vally validation recorded in STEP 5a; pr-plus-reviewer applies the same substantive retrieval correction as try-fix-1.

Winning candidate

Winner: pr-plus-reviewer

pr-plus-reviewer is the best candidate because it keeps the PR's useful executable-instruction changes while addressing the expert reviewer's single actionable defect. It is not weaker than any passing try-fix candidate, and it avoids selecting the raw PR while the unauthenticated fallback remains incorrect.

---

🧭 Next Steps — review latest findings

No alternative fix was selected for this run. Review the session findings and CI results before merging.

[kubaflo]

Could you please check the ai's suggestions?

[kubaflo]

/review -b feature/enhanced-reviewer -p android

[kubaflo]

Could you please check the ai's suggestions?

[kubaflo]

🤖 Multi-model code review — Request changes

Three models reviewed this PR independently (Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro), then cross-pollinated.

Consensus: NEEDS_CHANGES (light) — the regression-scenario intent is good and the SKILL.md addition is fine; the concerns are all about eval assertion quality + coordination, none about code safety.

Good 👍

The new "Regression risk check" step in SKILL.md and the version header comments are accurate, the YAML parses, and the new fields (expect_activation: false, output_contains) are consistent with the convention already used across the sibling skill eval files — they're not stray/no-op fields.

Assertion quality (inline)

1. Self-contradicting assertion (line 152) — the docs-only "should NOT warn" scenario asserts output_not_contains: "regression risk", but its rubric wants the agent to say "no regression risk" — the ideal answer contains that substring and would false-fail.
2. Gameable (line 112) — output_contains: "regression" passes even if the agent says "not a regression"; substring matches are direction-blind.
3. Prompt-echo (line 135) — output_contains: "side effect" just echoes a word already in the prompt.

Two coordination notes

• No CI runs these evals. There is no skill-validation workflow or skill-validator in the repo, and copilot-evaluate-tests only reviews src/** PR test diffs — so eval.yaml scenarios are currently unverified (nothing catches issues like #1 above). Worth wiring up a runner, or at least noting these are aspirational.
• Format migration in flight. PR #35942 (draft) migrates this same eval.yaml to the new Vally format — these additions to the legacy format will likely need re-doing. Consider landing against Vally, or confirm the sequencing.

_{Independent verdicts: Opus 4.8 — LGTM (med, "tighten assertions") · GPT-5.5 — NEEDS_DISCUSSION (high) · Gemini 3.1 Pro — NEEDS_DISCUSSION (high). (One model asserted a skill-validation.yml runner exists — I verified it does not, so that claim is excluded.) Consensus: safe & additive, but tighten the assertions and coordinate with the Vally migration.}

[kubaflo]

✏️ Correction to my earlier review

I need to correct a factual error in my review above. I stated "there is no skill-validation workflow or skill-validator in the repo, so eval.yaml scenarios are currently unverified" and excluded a model's claim to the contrary. That was wrong.

.github/workflows/skill-validation.yml does exist on main — it runs static checks on every PR touching .github/skills/**, plus an LLM evaluation of the skill eval.yaml scenarios (via the dotnet/skills validator binary), automatically for contributor PRs or when a contributor comments /evaluate-skills. My mistake: I verified the file's existence against this PR's branch head, which predates the workflow (it 404s on c5d204e9…), instead of against main. A rebase onto current main will pick up the runner — and credit to the reviewer model whose "it runs" claim I wrongly dismissed.

This makes the assertion-quality findings more important, not less: since these scenarios actually execute in CI, the self-contradicting output_not_contains: "regression risk" (line 152) would genuinely false-fail the validator on the correct "no regression risk" output. The three inline findings stand; please treat them as real eval failures rather than hypotheticals.

_{Posted by the multi-model review workflow — self-correcting the record.}

[github-actions]

Skill Validation Results

@PureWeen — new skill validation results are available based on this last commit: 03821c6.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

✅ Skill Validation Results — 03821c6 · Escalate Blast Radius probing for frequently-regressed component families · 2026-07-27T21:03:57Z

✅ Static Checks Passed

Skills: 23 | Eval specs linted: 12

Full lint output

── .github/skills/agentic-labeler/tests/eval.vally.yaml
npm warn deprecated prebuild-install@7.1.3: No longer maintained. Please contact the author of the relevant native addon; alternatives are available.
✔ .github/skills/agentic-labeler/tests/eval.vally.yaml is valid
── .github/skills/analyze-sessions/tests/eval.vally.yaml
✔ .github/skills/analyze-sessions/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/eval.capability.vally.yaml
✔ .github/skills/code-review/tests/eval.capability.vally.yaml is valid
── .github/skills/code-review/tests/eval.inline-findings.vally.yaml
✔ .github/skills/code-review/tests/eval.inline-findings.vally.yaml is valid
── .github/skills/code-review/tests/eval.producer-trace.vally.yaml
✔ .github/skills/code-review/tests/eval.producer-trace.vally.yaml is valid
── .github/skills/code-review/tests/eval.trim-aot.vally.yaml
✔ .github/skills/code-review/tests/eval.trim-aot.vally.yaml is valid
── .github/skills/code-review/tests/eval.vally.yaml
✔ .github/skills/code-review/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/hermeticity.vally.yaml
✔ .github/skills/code-review/tests/hermeticity.vally.yaml is valid
── .github/skills/evaluate-pr-tests/tests/eval.vally.yaml
✔ .github/skills/evaluate-pr-tests/tests/eval.vally.yaml is valid
── .github/skills/pr-review/tests/eval.gh-auth.vally.yaml
✔ .github/skills/pr-review/tests/eval.gh-auth.vally.yaml is valid
── .github/skills/try-fix/tests/eval.vally.yaml
✔ .github/skills/try-fix/tests/eval.vally.yaml is valid
── .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml
✔ .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml is valid

✅ LLM Evaluation Passed

5/5 eval suite(s) met threshold

Suite	Score	Threshold	Verdict
code-review-capabilities	0.81	0.60	✅
code-review-inline-findings-write	1.00	0.70	✅
code-review-producer-trace	0.93	0.85	✅
code-review-trim-aot	0.97	0.90	✅
code-review-regressions	0.92	0.60	✅

Harness hermeticity (negative control)

✅ Hermetic — the negative-control stimulus correctly came back unauthenticated (anonymous core rate limit; no GitHub token leaked into the agent env).

📊 code-review — eval report

Eval Results

Timestamp: 2026-07-27T21:03:40.297Z

code-review-capabilities [claude-opus-4.6] (/home/runner/work/maui/maui/.github/skills/code-review/tests/eval.capability.vally.yaml)

Capability suite for the code-review skill — behaviorial properties measured against real PRs (tool-call ordering, structured-output shape, API-misuse refusal, blast-radius reasoning, prior-review surfacing, CI-status interpretation).

Stimulus	Skills	Model	Graders	Pass Rate	pass@k	pass^k	Duration (median)	Tokens (median)	Turns (median)	Tool Calls (median)	Verdict
anti-pattern-never-approve-via-api	code-review (3×)	claude-opus-4.6	✅ output-not-contains 3/3 ✅ output-matches 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	2m 36s	701,039	17	21 calls (median) total across 3 trials: bash: 42, web_fetch: 13, view: 6, skill: 3, stop_bash: 1</details>	✅ 1
blast-radius-infra-changes-get-probed	—	claude-opus-4.6, claude-sonnet-4.6	❌ output-matches 2/3 ❌ prompt 2/3	2/3	100.0%	29.6%	3m 26s	1,153,239	23	31 calls (median) total across 3 trials: web_fetch: 48, bash: 42, view: 1, task: 1</details>	🟡 2
ci-hard-gate-skipping-required-checks-block-lgtm	—	claude-opus-4.6	✅ output-not-contains 3/3 ✅ output-matches 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	2m 17s	399,411	10	15 calls (median) total across 3 trials: web_fetch: 34, bash: 9</details>	✅
happy-path-code-review-request	code-review (3×)	claude-opus-4.6	✅ output-matches 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	3m 06s	887,925	21	31 calls (median) total across 3 trials: bash: 90, view: 4, skill: 3</details>	✅
independence-first-diff-before-description	code-review (3×)	claude-opus-4.6	✅ output-matches 3/3 ❌ prompt 2/3	3/3	100.0%	100.0%	2m 31s	416,374	12	15 calls (median) total across 3 trials: bash: 48, web_fetch: 5, skill: 3</details>	✅ 3
negative-trigger-describe-changes-query	—	claude-opus-4.6	✅ output-not-contains 3/3 ❌ prompt 0/3	3/3	100.0%	100.0%	28.4s	84,935	4	3 calls (median) total across 3 trials: web_fetch: 6, bash: 3</details>	✅
negative-trigger-informational-query	—	claude-opus-4.6	✅ output-not-contains 3/3 ❌ prompt 0/3	3/3	100.0%	100.0%	30.7s	84,969	4	3 calls (median) total across 3 trials: web_fetch: 6, bash: 3</details>	✅
prior-review-reconciliation-surfaces-prior-findings	code-review (3×)	claude-opus-4.6, claude-sonnet-4.6	✅ output-matches 3/3 ❌ prompt 0/3	3/3	100.0%	100.0%	3m 53s	1,553,582	26	37 calls (median) total across 3 trials: web_fetch: 66, bash: 57, view: 4, skill: 3, read_agent: 3, task: 1</details>	✅ 4
verdict-consistency-errors-block-lgtm	code-review (3×)	claude-opus-4.6	✅ output-not-contains 3/3 ✅ output-matches 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	2m 08s	454,149	12	12 calls (median) total across 3 trials: bash: 26, web_fetch: 15, skill: 3</details>	✅ 5

Model: claude-opus-4.6 | Judge: claude-opus-4.6 | Executor: copilot-sdk

---

code-review-inline-findings-write [claude-opus-4.6] (/home/runner/work/maui/maui/.github/skills/code-review/tests/eval.inline-findings.vally.yaml)

Regression guard for the expert-review phase: after producing file:line findings on a diff, the agent must WRITE them as a JSON array to the inline-findings.json path the pipeline reads from disk — it must NOT declare the write prohibited or substitute a chat-only dump. Missing the file silently drops every inline comment.

Stimulus	Environment	Graders	Pass Rate	pass@k	pass^k	Duration (median)	Tokens (median)	Turns (median)	Tool Calls (median)	Verdict
regression-writes-inline-findings-to-disk	git: 48c7d8711d6d6befd0297336c6fb8958cfcfc3bd Git: worktree @ 48c7d8711d6d6befd0297336c6fb8958cfcfc3bd from .</details>	✅ output-matches 5/5 ✅ inline-findings-judge 5/5	5/5	100.0%	100.0%	1m 41s	387,705	13	16 calls (median) total across 5 trials: bash: 76, view: 9, glob: 2, create: 2</details>	✅ 6

Model: claude-opus-4.6 | Judge: claude-opus-4.6 | Executor: copilot-sdk

---

code-review-producer-trace [claude-opus-4.6] (/home/runner/work/maui/maui/.github/skills/code-review/tests/eval.producer-trace.vally.yaml)

Verifies that code-review independently traces a token matched from external output to its out-of-diff producer and catches an over-restrictive gate.

Stimulus	Environment	Skills	Graders	Pass Rate	pass@k	pass^k	Duration (median)	Tokens (median)	Turns (median)	Tool Calls (median)	Verdict
producer-trace-console-token-over-cap	3 files · 1 skill Files: fixtures/producer-trace/change.diff → review-input/change.diff, fixtures/producer-trace/src/.github/skills/review-test-failures/scripts/Gather-TestFailureContext.ps1 → review-input/src/.github/skills/review-test-failures/scripts/Gather-TestFailureContext.ps1, fixtures/producer-trace/src/eng/devices/run-windows-devicetests.cmd → review-input/src/eng/devices/run-windows-devicetests.cmd Skills: ..</details>	code-review (5×)	✅ output-matches 5/5 ❌ tool-calls 4/5 ✅ producer-trace-judge 5/5	4/5	100.0%	32.8%	1m 41s	244,288	9	9 calls (median) total across 5 trials: bash: 23, view: 16, skill: 5, glob: 1</details>	🟡 7

Model: claude-opus-4.6 | Judge: claude-opus-4.6 | Executor: copilot-sdk

---

code-review-trim-aot [claude-opus-4.6] (/home/runner/work/maui/maui/.github/skills/code-review/tests/eval.trim-aot.vally.yaml)

Hermetic capability suite for code-review Trim and NativeAOT guidance. The reviewer must trace annotation chains and distinguish a documented analyzer limitation from a reachable dynamic-code path.

Environment: 3 files

Stimulus	Environment	Graders	Pass Rate	pass@k	pass^k	Duration (median)	Tokens (median)	Turns (median)	Tool Calls (median)	Verdict
annotated-helper-structurally-isolates-dynamic-registration	5 files Files: fixtures/trim-aot/shared/src/Core/src/Handlers/HybridWebView/HybridWebViewHandler.cs → review-input/src/Core/src/Handlers/HybridWebView/HybridWebViewHandler.cs, fixtures/trim-aot/shared/src/Core/src/Hosting/MauiHandlersCollectionExtensions.cs → review-input/src/Core/src/Hosting/MauiHandlersCollectionExtensions.cs, fixtures/trim-aot/shared/src/Core/src/RuntimeFeature.cs → review-input/src/Core/src/RuntimeFeature.cs, fixtures/trim-aot/case-b/change.diff → review-input/change.diff, fixtures/trim-aot/case-b/src/Controls/src/Core/Hosting/AppHostBuilderExtensions.cs → review-input/src/Controls/src/Core/Hosting/AppHostBuilderExtensions.cs</details>	✅ output-matches 5/5 ❌ tool-calls 4/5 ✅ annotated-helper-judge 5/5	4/5	100.0%	32.8%	1m 24s	204,380	9	12 calls (median) total across 5 trials: view: 29, bash: 29</details>	🟡 8
reachable-suppression-must-be-rejected	5 files Files: fixtures/trim-aot/shared/src/Core/src/Handlers/HybridWebView/HybridWebViewHandler.cs → review-input/src/Core/src/Handlers/HybridWebView/HybridWebViewHandler.cs, fixtures/trim-aot/shared/src/Core/src/Hosting/MauiHandlersCollectionExtensions.cs → review-input/src/Core/src/Hosting/MauiHandlersCollectionExtensions.cs, fixtures/trim-aot/shared/src/Core/src/RuntimeFeature.cs → review-input/src/Core/src/RuntimeFeature.cs, fixtures/trim-aot/case-c/change.diff → review-input/change.diff, fixtures/trim-aot/case-c/src/Controls/src/Core/Hosting/AppHostBuilderExtensions.cs → review-input/src/Controls/src/Core/Hosting/AppHostBuilderExtensions.cs</details>	✅ output-matches 5/5 ✅ tool-calls 5/5 ✅ reachable-suppression-judge 5/5	5/5	100.0%	100.0%	1m 13s	137,842	6	11 calls (median) total across 5 trials: bash: 31, view: 22</details>	✅
scoped-suppression-requires-annotation-chain-proof	5 files Files: fixtures/trim-aot/shared/src/Core/src/Handlers/HybridWebView/HybridWebViewHandler.cs → review-input/src/Core/src/Handlers/HybridWebView/HybridWebViewHandler.cs, fixtures/trim-aot/shared/src/Core/src/Hosting/MauiHandlersCollectionExtensions.cs → review-input/src/Core/src/Hosting/MauiHandlersCollectionExtensions.cs, fixtures/trim-aot/shared/src/Core/src/RuntimeFeature.cs → review-input/src/Core/src/RuntimeFeature.cs, fixtures/trim-aot/case-a/change.diff → review-input/change.diff, fixtures/trim-aot/case-a/src/Controls/src/Core/Hosting/AppHostBuilderExtensions.cs → review-input/src/Controls/src/Core/Hosting/AppHostBuilderExtensions.cs</details>	✅ output-matches 5/5 ✅ tool-calls 5/5 ✅ scoped-suppression-judge 5/5	5/5	100.0%	100.0%	1m 00s	113,254	5	8 calls (median) total across 5 trials: bash: 25, view: 17</details>	✅

Model: claude-opus-4.6 | Judge: claude-opus-4.6 | Executor: copilot-sdk

---

code-review-regressions [claude-opus-4.6] (/home/runner/work/maui/maui/.github/skills/code-review/tests/eval.vally.yaml)

Regression-detection corpus for the code-review skill. Each stimulus presents the diff of a PR that was later confirmed to have introduced a real, p/0-class regression in a shipping MAUI release. The eval asserts the reviewer would have surfaced the regression risk had they reviewed the PR pre-merge.

Stimulus	Environment	Graders	Pass Rate	pass@k	pass^k	Duration (median)	Tokens (median)	Turns (median)	Tool Calls (median)	Verdict
gradient-alpha-forced-opaque	git: 48c7d8711d6d6befd0297336c6fb8958cfcfc3bd · 1 file Git: worktree @ 48c7d8711d6d6befd0297336c6fb8958cfcfc3bd from . Files: ../SKILL.md → review-input/code-review-SKILL.md</details>	✅ tool-calls 5/5 ✅ output-matches 5/5 ✅ regression-judge 5/5	5/5	100.0%	100.0%	2m 12s	571,661	19	26 calls (median) total across 5 trials: bash: 101, view: 29</details>	✅
native-collection-null-overlays	git: dcd44b30fb4a95319b1a33cce1ab1ffd7b3a16d9 · 1 file Git: worktree @ dcd44b30fb4a95319b1a33cce1ab1ffd7b3a16d9 from . Files: ../SKILL.md → review-input/code-review-SKILL.md</details>	✅ tool-calls 5/5 ✅ output-matches 5/5 ✅ regression-judge 5/5	5/5	100.0%	100.0%	1m 19s	179,659	8	13 calls (median) total across 5 trials: bash: 49, view: 15, glob: 2</details>	✅
navigatedto-latch-suppresses-reentry	git: 8ee24cfe4c38038cec62e09dacc182815310c97d · 1 file Git: worktree @ 8ee24cfe4c38038cec62e09dacc182815310c97d fro
…(truncated — see artifacts)…

🔍 Full results and investigation steps

Footnotes

1. Trial durations: 2m 27s – 3m 45s ↩

2. ⚠️ Flaky (33% minority outcome). Grader breakdown: output-matches passed 2/3 trials, prompt passed 2/3 trials ↩

3. Grader breakdown: prompt passed 2/3 trials ↩

4. Trial durations: 3m 44s – 7m 58s ↩

5. Trial durations: 1m 51s – 2m 56s ↩

6. Trial durations: 1m 19s – 2m 25s ↩

7. ⚠️ Flaky (20% minority outcome). Grader breakdown: tool-calls passed 4/5 trials ↩

8. ⚠️ Flaky (20% minority outcome). Grader breakdown: tool-calls passed 4/5 trials ↩

[kubaflo]

Note

🤖 AI-generated action (on behalf of @kubaflo). This branch was rebased onto main and force-pushed to finish the PR.

What changed and why. This PR had drifted ~3 months behind main and was superseded on both files, so a straight conflict-resolve wasn't the right move:

• eval.yaml scenarios — dropped. The skill-eval suite was migrated to the hermetic Vally corpus (eval.vally.yaml), which deliberately replaced synthetic inline-diff scenarios with frozen, worktree-pinned real regression commits (e.g. gradient-alpha-forced-opaque → the actual regressed-in-10.0.60 commit). Re-adding the synthetic output_contains scenarios would reintroduce exactly the brittleness that migration removed — and regression detection is already covered there.
• SKILL.md note — kept, but rewritten. The original referenced the old Section 21/22 layout, which no longer exists (SKILL.md was restructured to Step 1‑6 / Blast Radius / Failure-Mode Probing). The durable value is preserved as a "Regression risk check" in Step 3 that now routes high-regression component families to the current Blast Radius / Failure-Mode Probing (Step 6) and the expert reviewer's Regression Prevention dimension (Step 2). It also folds in the intent of the dropped eval scenarios: the adjacent-scenario side-effect insight, and the guard that documentation-only changes must not attract regression scrutiny.

Result: a clean +2/−0 change that rebases cleanly and is now mergeable. Heads-up @PureWeen — your original authorship is preserved via Co-authored-by.

Decisions here (salvage vs. port vs. close) were made by @kubaflo; this note is for transparency.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.