Repository navigation
test: realign daily release smoke tests with 0.37 contracts (#3885) - #3891
Daniel Meppiel (danielmeppiel) merged 5 commits into
Conversation
- retained-lock-keys mutation targets install/orphans.py, where the expected-path selector moved in #3879/#3882 - locked-skill-subset rule requires skill_subset=sorted(self.skill_subset) as one fragment; _replay_pairing(self.skill_subset, ...) from #3769 satisfied the old split needles, so the mutation went undetected - install --dry-run bootstrap previews targets and writes no apm.yml (#3865) - pack docs pointer is the install-patterns page (#3865) - finalize imports _rich_success from the console owner (#3879) - audit JSON contract tests parse stdout; the target: deprecation warning goes to stderr and is emitted once per process Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ly-release-smoke-3885
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Two integration tests can fail on narrow terminals because Rich wraps the preview text and documentation URL in ways their assertions do not handle.
2 open issues
What changed in this PR
Realigns daily release smoke tests with the intentional 0.37 contracts and recent internal refactors, addressing the Linux failures tracked in #3885 without changing product code under src/.
Changes:
- Update dry-run bootstrap expectations, marketplace documentation links, and finalize mock targets.
- Parse audit JSON and SARIF from stdout so stderr warnings do not break structured-output checks.
- Refresh the retained-lock mutation target and tighten the locked skill-subset architecture guard.
| File | Description |
|---|---|
tests/integration/test_wave7_marketplace_install_coverage.py |
Assert bootstrap previews and that dry-run creates no manifest. |
tests/integration/test_install_marketplace_phase4w3.py |
Patch the console helper where finalize looks it up. |
tests/integration/test_audit_primitive_contract.py |
Parse structured audit output from stdout. |
tests/integration/test_architecture_retained_lock_keys.py |
Point the mutation at the extracted orphan-selection module. |
tests/integration/marketplace/test_pack_ux_e2e.py |
Expect the current marketplace installation documentation path. |
scripts/architecture_linter/checks/install_policy_skill_and_lock.py |
Require the skill-subset keyword and its source together. |
CHANGELOG.md |
Record the smoke-test and architecture-guard fixes. |
Review effort: Balanced
Rich wraps CliRunner output at the terminal width, so on a narrow console the dry-run target preview and the pack docs URL split across lines and the assertions fail although the output is correct. Pass a wide COLUMNS per invoke, as other integration tests do. Addresses Copilot inline on test_pack_ux_e2e.py:362 and test_wave7_marketplace_install_coverage.py:1013. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
8381e63 to
a76a789
Compare
APM Review Panel:
|
| Persona | B | R | N | Takeaway |
|---|---|---|---|---|
| Python Architect | 0 | 2 | 0 | The single-fragment needle is the right minimal fix and the test realignments follow existing owners. One gap remains: target_subset has no static guard in to_dependency_ref. |
| Test Coverage Expert | 0 | 2 | 0 | Rewrites still trap the 0.37 contracts and the linter needle is caught by its mutation; the non-dry-run bootstrap target-persist promise is covered only by a mocked unit test. |
B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.
Top 4 follow-ups
- [Python Architect] Add a static guard for
target_subsetreconstruction inLockedDependency.to_dependency_ref(src/apm_cli/deps/lockfile.py:1032) toRULE_LOCKED_SUBSET, using a single-fragment needle that_replay_pairingcannot mask. -- Same masking risk this PR fixes for skill_subset: today a mutation totarget_subset=Nonewould pass the linter. - [Test Coverage Expert] Add a non-dry-run integration test confirming that bootstrap persists the selected target to
apm.yml. -- The realigned wave7 test correctly stopped asserting persistence under--dry-run, so the real persist promise is now covered only by a mocked unit test. - [Test Coverage Expert] Decide whether to raise the click floor in pyproject to
>=8.2, or document the stdout/stderr split assumption. -- uv.lock pins click 8.2.1 so CI is stable, but about 319 test files already depend on the 8.2 stream split. - [Python Architect] If
RULE_LOCKED_SUBSETchanges again, replace literal-prefix needles with an AST keyword check on theto_dependency_refcall. -- The current needle guards one exact spelling, not the contract.
Recommendation
Ship once CI on a76a789 is green; the prior head was green and the only delta is the COLUMNS=240 hardening, verified locally. The PR meets the #3885 done-when: the targeted Linux smoke tests pass and the locked-skill-subset mutation is caught again. After it lands, track the target_subset static-guard follow-up as the highest-signal item.
Full per-persona findings
Python Architect
- [recommended] No static guard covers target_subset reconstruction in LockedDependency.to_dependency_ref, and it has the same masking risk the PR fixes for skill_subset at
scripts/architecture_linter/checks/install_policy_skill_and_lock.py:229
The sibling keyword at src/apm_cli/deps/lockfile.py:1032 has no needle in RULE_LOCKED_SUBSET; a mutation to target_subset=None would pass the linter today, and a naive self.target_subset needle would be masked by the same _replay_pairing call. The scope record limits this PR to realigning existing guards, so this belongs in a follow-up rather than a fold.
Suggested: Add a target_subset=sorted(self.target_subset) fragment plus a MutationCase.
Proof (missing at):tests/integration/test_architecture_semantic_rule_mutations.py - [recommended] The new needle is a literal prefix of the mutation's old string, so the rule guards one exact spelling rather than the underlying contract at
scripts/architecture_linter/checks/install_policy_skill_and_lock.py:229
It catches the documented mutation and fails loudly rather than silently; the cost is false positives on benign refactors. A keyword-scoped AST check would enforce the contract directly, but only if the rule churns again.
Suggested: No change needed in this PR; switch to an ast-based keyword check if the needle needs editing again.
Test Coverage Expert
- [recommended] Dropping the persisted-apm.yml assertion from the wave7 integration test leaves target persistence on bootstrap covered only by a mocked unit test. at
tests/integration/test_wave7_marketplace_install_coverage.py:995
The old test ran with --dry-run and asserted the wrong contract; the rewrite is correct. Persistence on a real install is covered only by tests/unit/test_install_command.py::test_install_no_apm_yml_persists_target_flag (mocked). Not a regression from this PR.
Suggested: Optional follow-up: a non-dry-run integration test asserting targets: [copilot] in the written apm.yml.
Proof (missing at):tests/integration/test_wave7_marketplace_install_coverage.py::TestInstallAutoBootstrap::test_no_apm_yml_with_target_flag_persists_targets-- proves: apm install --target copilot in an empty directory records the chosen target in apm.yml. [devx,portability-by-manifest] - [recommended] The audit tests' result.stdout fix is correct only on Click >= 8.2, but pyproject allows click>=8.0.0. at
tests/integration/test_audit_primitive_contract.py:77
On Click 8.0/8.1 CliRunner mixes stderr into stdout by default. uv.lock pins click 8.2.1, so CI is fine; the assumption is already relied on elsewhere.
Suggested: Raise the click floor to >=8.2 or note the assumption.
Proof (unknown):tests/integration/test_audit_primitive_contract.py::test_shared_settings_command_inventory_is_visible_and_non_failing-- proves: apm audit --format json emits parseable JSON on stdout even when a deprecation warning is written to stderr. [devx]
This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.
Generated by autopilot-pr-review-worker. This comment is AI-generated and may contain errors.

Description
The 2026-11-19 scheduled CI/CD Pipeline run (44455839414 at dc9972d) failed 9 tests. I reproduced 6 locally on main HEAD f8ef817. Five fail because tests are stale after intentional 0.37 changes. The sixth is an architecture rule that stopped catching its own mutation. This PR changes no product code under
src/.test_architecture_retained_lock_keys[selector-rederives-depth]commands/_helpers.pytoinstall/orphans.py(#3879/#3882). The rule already scans the new file.install/orphans.pytest_architecture_semantic_rule_mutations[install-deployment-locked-skill-subset-reconstruction]skill_subset=andself.skill_subsetas separate needles._replay_pairing(self.skill_subset, ...)(#3769) satisfies the second one, soskill_subset=Nonepassed.scripts/architecture_linter: requireskill_subset=sorted(self.skill_subset)as a single fragmenttest_wave7_marketplace_install_coverage::...test_no_apm_yml_with_target_flag_persists_targetsinstall --dry-runno longer writesapm.ymlWould create apm.yml,Would set targets: copilot) and that noapm.ymlis written; renamed..._previews_targetsmarketplace/test_pack_ux_e2e::...test_catalog_lists_both_profiles_with_docs_pointerconsumer/installing-from-marketplaces/#install-patterns. The unit test was already updated.test_install_marketplace_phase4w3::TestFinalizePhase::test_bare_success_emitted_without_loggerfinalize.pyimports_rich_successfromutils/console(#3879), so the old patch target never firedapm_cli.install.phases.finalize._rich_successtest_audit_primitive_contract(shared settings x2, copilot sessionStart)target:triggers the deprecation warning (#3801). The warning goes to stderr, but Click'sresult.outputincludes stderr. It is emitted once per process, so which test fails depends on order.result.stdoutfor the JSON contracttest_windows_installer_launchers::test_windows_installer_exposes_stable_executableapm self-updateran the aka.msinstall.ps1under Windows PowerShell, andExpand-ArchivehitCannot find an overload for "ExtractToDirectory". The same step passed on the 11-18 schedule and on today's dispatch run 44462718723 at f8ef817.Issue and approved scope
Issue: Fixes #3885
Human scope-approval comment: #3885 (comment) (review contact Daniel Meppiel (@danielmeppiel)). Only tests and the architecture linter change, no product code.
Does this PR complete the issue, or what remains? It fixes every Linux failure in the run. The Windows installer failure is an external-script flake (above). If it comes back, #3885 will reopen on the next scheduled run.
Type of change
Testing
These integration tests only run fully on main/schedule. I ran these locally on macOS:
.venv/bin/python -m pytest -p no:xdist -o addopts="" tests/integration/test_wave7_marketplace_install_coverage.py::TestInstallAutoBootstrap tests/integration/marketplace/test_pack_ux_e2e.py::TestVendorNeutralCatalog tests/integration/test_audit_primitive_contract.py tests/integration/test_install_marketplace_phase4w3.py::TestFinalizePhase tests/integration/test_architecture_semantic_rule_mutations.py tests/integration/test_architecture_retained_lock_keys.py: passed (260 passed). Before the fix, 6 of these failed on f8ef817.test_audit_primitive_contractcase in its own process ([True],test_copilot_session_start_prompt_text_is_scanned), to check order-dependence: passed..venv/bin/python -m pytest -p no:xdist -o addopts="" tests/unit/scripts/test_architecture_runner.py "tests/integration/test_architecture_semantic_rule_mutations.py::test_semantic_rule_catches_its_mutation[install-deployment-locked-skill-subset-reconstruction]" tests/integration/test_architecture_retained_lock_keys.py: passed (75 passed).uvx ruff@0.15.12 checkandruff format --checkonsrc/ tests/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ scripts/windows_native_symlink_probe_entry.py: passed.pylint --enable=R0801 ... --fail-on=R0801: passed (10.00/10).bash scripts/lint-auth-signals.sh: passed.bash scripts/lint-architecture-boundaries.sh: passed (exit 0).Spec conformance (OpenAPM v0.1)
docs/src/content/docs/specs/openapm-v0.1.mdupdated(new/changed
<a id="req-XXX"></a>anchor + prose + Appendix Crow).
docs/src/content/docs/specs/manifests/openapm-v0.1.requirements.ymlupdated.
@pytest.mark.req("req-XXX")test undertests/spec_conformance/added or extended.CONFORMANCE.{md,json}regenerated viauv run --extra dev python -m tests.spec_conformance.gen_statementand committed.