Repository navigation
Conversation
Select changed cases and required feature, tool, and OS coverage using Responses v1. Plan compatible images, provide dedicated-platform recommendations, and execute Azure groups sequentially with fixed-baseline replay for failures. Skip complete case deletions and automatic draft PR validation, and document execution and reporting behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved defects in test selection, suite loading, result aggregation, and head/baseline comparability can omit tests or falsely report successful validation.
8 open issues
What changed in this PR
This PR replaces smoke-test fallbacks with targeted validation plans, connecting changed LISA code to required tests and compatible images. It also adds failure-triggered baseline comparisons while preserving raw test results.
Changes:
- Select changed tests deterministically and enforce feature, tool, and OS coverage requirements.
- Execute compatible Azure groups sequentially and publish recommendations for dedicated platforms.
- Compare failed head cases against a fixed ancestor, skip automatic draft validation, and document execution limits.
Suggested validation, not executed during this review:
Key Test Cases:
smoke_test|verify_reboot_in_platform
Impacted LISA Features:
None directly; changes affect validation orchestration and test selection.
Tested Azure Marketplace Images:
- canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest
| File | Description |
|---|---|
lisa/microsoft/runbook/ai_selected.yml |
Allows explicitly selected stable and preview cases. |
docs/run_test/run.rst |
Links the new PR validation guide. |
docs/run_test/pr_validation.rst |
Documents selection, execution, comparison rules, and limitations. |
.github/scripts/validation_report.py |
Renders dedicated-platform recommendation tables. |
.github/scripts/validation_plan.py |
Builds metadata-checked plans using reviewed images. |
.github/scripts/run_selected_cases.py |
Executes groups and compares failed cases with baseline results. |
.github/scripts/ai_select_cases.py |
Adds required coverage, deletion handling, and Responses API selection. |
.azuredevops/azure-pipelines.yml |
Integrates plans, baseline runs, reporting, draft filtering, and a six-hour timeout. |
Review effort: Balanced
Comment on lines
+69
to
+72
| if case.get("classname") == name | ||
| or ( | ||
| case.get("classname") == suite | ||
| and re.sub(r" \([^()]*\)$", "", case.get("name", "")) == method |
Comment on lines
+157
to
+179
| keys = ( | ||
| "vm_size", | ||
| "osdisk_size_in_gb", | ||
| "maximize_capability", | ||
| "marketplace", | ||
| "security_profile", | ||
| "disk_controller_type", | ||
| "data_disk_count", | ||
| "data_disk_type", | ||
| "data_disk_size", | ||
| "nic_count", | ||
| "network_data_path", | ||
| ) | ||
| for path in directory.rglob("arm_template_parameters.json"): | ||
| parameters = json.loads(path.read_text(encoding="utf-8")) | ||
| nodes = parameters.get("nodes", {}).get("value", []) | ||
| configuration = sorted( | ||
| json.dumps({key: node.get(key) for key in keys}, sort_keys=True) | ||
| for node in nodes | ||
| ) | ||
| signatures.add( | ||
| hashlib.sha256(json.dumps(configuration).encode("utf-8")).hexdigest() | ||
| ) |
Comment on lines
+257
to
+261
| "keep_environment:no", | ||
| "-v", | ||
| "wait_delete:True", | ||
| "-v", | ||
| "concurrency:1", |
Comment on lines
+342
to
+346
| base = execute_group( | ||
| str(executable), | ||
| baseline_directory / "source", | ||
| base_directory, | ||
| runbook, |
Comment on lines
+130
to
+139
| for path in sorted(set(case_files.values())): | ||
| module = importlib.import_module(path.removesuffix(".py").replace("/", ".")) | ||
| if Path(module.__file__ or "").resolve() != (repo_root / path).resolve(): | ||
| raise ValueError( | ||
| f"Imported testsuite does not match the repository: {path}" | ||
| ) | ||
| registered = get_cases_metadata() | ||
| result: Dict[str, TestCaseMetadata] = {} | ||
| for name, path in case_files.items(): | ||
| module_name = path.removesuffix(".py").replace("/", ".") |
Comment on lines
+320
to
+327
| if head_statuses == {"passed"}: | ||
| execution_result = "Passed" | ||
| elif head_statuses.intersection({"failed", "error"}): | ||
| execution_result = "Failed" | ||
| elif head_statuses == {"skipped"}: | ||
| execution_result = "Skipped" | ||
| else: | ||
| execution_result = "Incomplete" |
| changed_lines, | ||
| ) | ||
| related_cases: List[str] = [] | ||
| for changed_file_path in sorted(changed_testsuite_files): |
| "-v", | ||
| "enable_tests:False", | ||
| "-v", | ||
| "auth_type:token", |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace smoke-test fallback behavior with targeted PR validation and explicit coverage requirements.
Validation
Scope and limitations