test: enable evidence-based AutoTest analysis

[wenyt (wenytang-ms)]

Summary

• resolve the latest @vscjava/vscode-autotest release once per workflow run, share that exact version across every matrix job and aggregate analysis, and print the installed version; subsequent runs still resolve the latest release without a permanent version pin
• run every matrix case with --analysis-mode case so successful cases receive false-pass audits and failed cases receive evidence-based root-cause analysis
• preserve the existing aggregate-job scope: scheduled runs and full-matrix workflow dispatches with an empty test_plan; targeted dispatches retain per-platform case analyses but skip aggregation, and pull-request runs remain excluded from aggregation
• use --report-only for aggregate analysis so failed test cases do not prevent successful summary generation
• retain lightweight case analyses and evidence manifests in the aggregate artifact while each matrix artifact keeps the full evidence bundle

Dropdown verification scope

The java-webview-migration wording change corrects the expected state for the existing step-level LLM screenshot check: an already-expanded dropdown may legitimately produce unchanged screenshots, but it must still show a runtime entry. This does not add a deterministic DOM assertion or make case-level analysis affect verdicts.

The deterministic-coverage concern is valid and remains a known limitation: with LLM verification disabled or unavailable, this particular wait step does not independently prove that the dropdown is expanded and non-empty. AutoTest's current verifyWebview API only supports text contains/notContains; a page-text check would not prove the popup state. Completing that coverage requires a proper DOM-verification capability in AutoTest and a corresponding plan update once available. This PR does not add unsupported YAML fields or claim that the gap is fixed.

Validation

• validated test-plans/java-basic-editing.yaml with @vscjava/vscode-autotest@latest (0.7.27)
• dispatched java-basic-editing across Windows, Linux, and macOS: https://github.com/microsoft/vscode-java-pack/actions/runs/34821757397
• all three case analyses identified the same root cause with 0.9 confidence: bundled Lombok 1.18.39-4050 is incompatible with JDT Core 3.48.0.v20260911-1206, causing NoSuchFieldError in lombok.eclipse.EclipseAST.buildMethod
• aggregate analysis correctly classified later code-action, organize-imports, rename, and Explorer failures as cascading failures

The referenced run used commit 9712832, with targeted aggregation temporarily enabled to exercise the analysis pipeline. Its E2E jobs failed because they reproduced the existing Lombok/JDT product issue; the aggregate analysis job completed successfully. Commit e9b9683 subsequently restored the original inputs.test_plan == '' guard at the maintainer's request. On the current PR head, a targeted dispatch still produces per-platform case analyses and evidence, but does not run the aggregate job.

[Copilot]

🟡 Changes recommended

Targeted workflow-dispatch runs currently skip the aggregate analysis job.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Enables evidence-based AutoTest case and aggregate analysis in the E2E workflow.

Changes:

• Installs and reports the latest AutoTest CLI version.
• Adds case-level and report-only aggregate analysis.
• Preserves analysis summaries, manifests, and evidence artifacts.

File summaries

File	Description
.github/workflows/e2e-autotest.yml	Updates AutoTest installation, execution, aggregation, and artifact retention.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

Resolve AutoTest version consistency and add a deterministic dropdown verification assertion.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

test-plans/java-webview-migration.yaml:185

• This step still has only a natural-language verify description; it does not add a deterministic assertion for either the expanded state or the presence of an option. Since verify is contextual rather than pass/fail authority, the new case-analysis mode can still rely on a screenshot/model judgment for this regression. Please add a stable webview/DOM-level verifier (or another deterministic check supported by AutoTest) for the dropdown state and runtime entry, as the surrounding webview steps do.

.github/workflows/e2e-autotest.yml:398

• The aggregate job resolves @latest independently from the matrix jobs. If a new AutoTest release is published while the workflow is running, the analyzer can be a different version from the CLI that produced the downloaded result artifacts, causing format or analysis incompatibilities. Resolve the version once and pass that exact version to both jobs (or pin it for the run).

          npm install -g @vscjava/vscode-autotest@latest
          npm list -g @vscjava/vscode-autotest --depth=0

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[wenyt (wenytang-ms)]

Follow-up on the latest review:

• AutoTest version consistency: addressed in 144ad70. discover resolves @latest once, validates the returned version, and shares it with all matrix jobs and analyze. A later release cannot change the analyzer version halfway through the run. This is run-level consistency, not a permanent pin; the next workflow run resolves @latest again. The existing PR and targeted-run aggregate exclusions remain unchanged.
• Dropdown verification: the deterministic-coverage gap is valid, but the current AutoTest verifyWebview API only supports text checks, which cannot establish the expanded popup state and a visible option. The existing step-level LLM re-check does evaluate the state when enabled; case-level analysis itself remains advisory. The PR description now explicitly documents this limitation, including the lack of a deterministic assertion when LLM verification is disabled/unavailable. The gap is not claimed as fixed, and no unsupported or merely cosmetic assertion was added. A proper fix needs DOM-verification support in AutoTest followed by a plan update.

[Copilot]

🟢 Approval recommended

No unresolved blocking issues were identified in the reviewed changes.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite