Skip to content

test: realign daily release smoke tests with 0.37 contracts (#3885) - #3891

Merged
Daniel Meppiel (danielmeppiel) merged 5 commits into
mainfrom
danielmeppiel-fix-daily-release-smoke-3885
Nov 19, 2026
Merged

Daniel Meppiel (danielmeppiel) merged 5 commits into
mainfrom
danielmeppiel-fix-daily-release-smoke-3885

Conversation

@danielmeppiel

@danielmeppiel Daniel Meppiel (danielmeppiel) commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The 2026-11-19 scheduled CI/CD Pipeline run (44455839414 at dc9972d) failed 9 tests. I reproduced 6 locally on main HEAD f8ef817. Five fail because tests are stale after intentional 0.37 changes. The sixth is an architecture rule that stopped catching its own mutation. This PR changes no product code under src/.

Failing test Cause Fix
test_architecture_retained_lock_keys[selector-rederives-depth] The expected-path selector moved from commands/_helpers.py to install/orphans.py (#3879/#3882). The rule already scans the new file. Point the mutation at install/orphans.py
test_architecture_semantic_rule_mutations[install-deployment-locked-skill-subset-reconstruction] The rule checked skill_subset= and self.skill_subset as separate needles. _replay_pairing(self.skill_subset, ...) (#3769) satisfies the second one, so skill_subset=None passed. scripts/architecture_linter: require skill_subset=sorted(self.skill_subset) as a single fragment
test_wave7_marketplace_install_coverage::...test_no_apm_yml_with_target_flag_persists_targets #3865: project-scope install --dry-run no longer writes apm.yml Assert the preview (Would create apm.yml, Would set targets: copilot) and that no apm.yml is written; renamed ..._previews_targets
marketplace/test_pack_ux_e2e::...test_catalog_lists_both_profiles_with_docs_pointer #3865: the pack docs pointer moved to consumer/installing-from-marketplaces/#install-patterns. The unit test was already updated. Assert the new path
test_install_marketplace_phase4w3::TestFinalizePhase::test_bare_success_emitted_without_logger finalize.py imports _rich_success from utils/console (#3879), so the old patch target never fired Patch apm_cli.install.phases.finalize._rich_success
test_audit_primitive_contract (shared settings x2, copilot sessionStart) The fixture's target: triggers the deprecation warning (#3801). The warning goes to stderr, but Click's result.output includes stderr. It is emitted once per process, so which test fails depends on order. Parse result.stdout for the JSON contract
test_windows_installer_launchers::test_windows_installer_exposes_stable_executable Environmental flake: the v0.28.0 binary's apm self-update ran the aka.ms install.ps1 under Windows PowerShell, and Expand-Archive hit Cannot find an overload for "ExtractToDirectory". The same step passed on the 11-18 schedule and on today's dispatch run 44462718723 at f8ef817. No change

Issue and approved scope

Issue: Fixes #3885

Human scope-approval comment: #3885 (comment) (review contact Daniel Meppiel (@danielmeppiel)). Only tests and the architecture linter change, no product code.

Does this PR complete the issue, or what remains? It fixes every Linux failure in the run. The Windows installer failure is an external-script flake (above). If it comes back, #3885 will reopen on the next scheduled run.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Testing

  • Tested locally
  • All existing tests pass
  • Added tests for new functionality (if applicable)

These integration tests only run fully on main/schedule. I ran these locally on macOS:

  • .venv/bin/python -m pytest -p no:xdist -o addopts="" tests/integration/test_wave7_marketplace_install_coverage.py::TestInstallAutoBootstrap tests/integration/marketplace/test_pack_ux_e2e.py::TestVendorNeutralCatalog tests/integration/test_audit_primitive_contract.py tests/integration/test_install_marketplace_phase4w3.py::TestFinalizePhase tests/integration/test_architecture_semantic_rule_mutations.py tests/integration/test_architecture_retained_lock_keys.py: passed (260 passed). Before the fix, 6 of these failed on f8ef817.
  • Each test_audit_primitive_contract case in its own process ([True], test_copilot_session_start_prompt_text_is_scanned), to check order-dependence: passed.
  • On HEAD merged with main 6683116: .venv/bin/python -m pytest -p no:xdist -o addopts="" tests/unit/scripts/test_architecture_runner.py "tests/integration/test_architecture_semantic_rule_mutations.py::test_semantic_rule_catches_its_mutation[install-deployment-locked-skill-subset-reconstruction]" tests/integration/test_architecture_retained_lock_keys.py: passed (75 passed).
  • Lint contract on HEAD merged with main 6683116: uvx ruff@0.15.12 check and ruff format --check on src/ tests/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ scripts/windows_native_symlink_probe_entry.py: passed. pylint --enable=R0801 ... --fail-on=R0801: passed (10.00/10). bash scripts/lint-auth-signals.sh: passed. bash scripts/lint-architecture-boundaries.sh: passed (exit 0).
  • Not run locally: the Windows installer test (Windows-only); I reasoned from the CI logs.

Spec conformance (OpenAPM v0.1)

  • Spec edit: docs/src/content/docs/specs/openapm-v0.1.md updated
    (new/changed <a id="req-XXX"></a> anchor + prose + Appendix C
    row).
  • Manifest edit: docs/src/content/docs/specs/manifests/openapm-v0.1.requirements.yml
    updated.
  • Test edit: a @pytest.mark.req("req-XXX") test under
    tests/spec_conformance/ added or extended.
  • CONFORMANCE.{md,json} regenerated via
    uv run --extra dev python -m tests.spec_conformance.gen_statement
    and committed.
  • N/A -- this PR does not change OpenAPM-observable behaviour.

- retained-lock-keys mutation targets install/orphans.py, where the
  expected-path selector moved in #3879/#3882
- locked-skill-subset rule requires skill_subset=sorted(self.skill_subset)
  as one fragment; _replay_pairing(self.skill_subset, ...) from #3769
  satisfied the old split needles, so the mutation went undetected
- install --dry-run bootstrap previews targets and writes no apm.yml (#3865)
- pack docs pointer is the install-patterns page (#3865)
- finalize imports _rich_success from the console owner (#3879)
- audit JSON contract tests parse stdout; the target: deprecation
  warning goes to stderr and is emitted once per process

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 05:25
@danielmeppiel Daniel Meppiel (danielmeppiel) added this to the 0.37.0 milestone Nov 19, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two integration tests can fail on narrow terminals because Rich wraps the preview text and documentation URL in ways their assertions do not handle.

2 open issues
What changed in this PR

Realigns daily release smoke tests with the intentional 0.37 contracts and recent internal refactors, addressing the Linux failures tracked in #3885 without changing product code under src/.

Changes:

  • Update dry-run bootstrap expectations, marketplace documentation links, and finalize mock targets.
  • Parse audit JSON and SARIF from stdout so stderr warnings do not break structured-output checks.
  • Refresh the retained-lock mutation target and tighten the locked skill-subset architecture guard.
File Description
tests/​integration/​test_wave7_marketplace_install_coverage.py Assert bootstrap previews and that dry-run creates no manifest.
tests/​integration/​test_install_marketplace_phase4w3.py Patch the console helper where finalize looks it up.
tests/​integration/​test_audit_primitive_contract.py Parse structured audit output from stdout.
tests/​integration/​test_architecture_retained_lock_keys.py Point the mutation at the extracted orphan-selection module.
tests/​integration/​marketplace/​test_pack_ux_e2e.py Expect the current marketplace installation documentation path.
scripts/​architecture_linter/​checks/​install_policy_skill_and_lock.py Require the skill-subset keyword and its source together.
CHANGELOG.md Record the smoke-test and architecture-guard fixes.

Review effort: Balanced

Resolve the CHANGELOG conflict with #3886 by taking main's file: the
concise rewrite drops test-only entries (as it did for #3877), so this
test-only PR carries no CHANGELOG line.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Rich wraps CliRunner output at the terminal width, so on a narrow
console the dry-run target preview and the pack docs URL split across
lines and the assertions fail although the output is correct. Pass a
wide COLUMNS per invoke, as other integration tests do. Addresses
Copilot inline on test_pack_ux_e2e.py:362 and
test_wave7_marketplace_install_coverage.py:1013.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel
Daniel Meppiel (danielmeppiel) force-pushed the danielmeppiel-fix-daily-release-smoke-3885 branch from 8381e63 to a76a789 Compare November 19, 2026 06:01
@danielmeppiel

Copy link
Copy Markdown
Collaborator Author

APM Review Panel: ship_with_followups

Restores the daily release smoke signal by realigning five stale integration tests and one masked architecture-linter needle with the intentional 0.37 contracts, without touching product code under src/.

panel-mode=lean; personas=python-architect,test-coverage-expert,apm-ceo (test-coverage-expert ad hoc: the PR exclusively rewrites regression-trap assertions)

cc Daniel Meppiel (@danielmeppiel) Sergio Sisternes (@sergio-sisternes-epam) -- a fresh advisory pass is ready for your review.

This is a hygiene PR, and the panel agrees it is the right one. The daily scheduled smoke run is the release-confidence signal, and run 44455839414 had stopped meaning anything because the tests asserted pre-0.37 contracts. The five realignments each move an assertion to its current owner: the dry-run bootstrap preview, the pack docs pointer path, the finalize _rich_success patch target, audit JSON parsed from stdout, and the retained-lock-keys mutation target. None of them weakens what the test promises. python-architect confirms the single-fragment needle skill_subset=sorted(self.skill_subset) is the minimal correct fix: the old split needle was satisfied by _replay_pairing(self.skill_subset, ...), so the linter passed while the reconstruction mutation went undetected. test-coverage-expert confirms the rewritten assertions still trap the 0.37 contracts and that the documented mutation is caught again.

Delta since the PR opened: the two Copilot inline findings about Rich wrapping were valid and are folded in a76a789 with a per-invoke COLUMNS=240 (at COLUMNS=40 both tests failed before the fold and pass after). The CHANGELOG conflict with #3886 was resolved by taking main's concise rewrite, which leaves out test-only entries, so this PR carries no CHANGELOG line. The #3885 scope record is now linked from the PR body.

There is no disagreement on the diff. Every remaining finding is about coverage next to the change, not in it, and each was deferred against the #3885 scope record (realign existing guards, nothing under src/). The deferred items are not equal: the target_subset gap has the exact shape of the bug this PR fixes, on the lockfile reconstruction surface, so it is the top follow-up.

Aligned with: Portable by manifest -- restores the static guard on locked skill_subset reconstruction (the sibling target_subset keyword is still unguarded). Governed by policy -- the architecture-boundary linter is a real gate on the documented mutation again. OSS community driven -- picks up valid Copilot inline findings with before/after proof and leaves the requested CODEOWNER review path unchanged. Pragmatic as npm -- test-only, no src/ changes, no migration.

Panel summary

Persona B R N Takeaway
Python Architect 0 2 0 The single-fragment needle is the right minimal fix and the test realignments follow existing owners. One gap remains: target_subset has no static guard in to_dependency_ref.
Test Coverage Expert 0 2 0 Rewrites still trap the 0.37 contracts and the linter needle is caught by its mutation; the non-dry-run bootstrap target-persist promise is covered only by a mocked unit test.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 4 follow-ups

  1. [Python Architect] Add a static guard for target_subset reconstruction in LockedDependency.to_dependency_ref (src/apm_cli/deps/lockfile.py:1032) to RULE_LOCKED_SUBSET, using a single-fragment needle that _replay_pairing cannot mask. -- Same masking risk this PR fixes for skill_subset: today a mutation to target_subset=None would pass the linter.
  2. [Test Coverage Expert] Add a non-dry-run integration test confirming that bootstrap persists the selected target to apm.yml. -- The realigned wave7 test correctly stopped asserting persistence under --dry-run, so the real persist promise is now covered only by a mocked unit test.
  3. [Test Coverage Expert] Decide whether to raise the click floor in pyproject to >=8.2, or document the stdout/stderr split assumption. -- uv.lock pins click 8.2.1 so CI is stable, but about 319 test files already depend on the 8.2 stream split.
  4. [Python Architect] If RULE_LOCKED_SUBSET changes again, replace literal-prefix needles with an AST keyword check on the to_dependency_ref call. -- The current needle guards one exact spelling, not the contract.

Recommendation

Ship once CI on a76a789 is green; the prior head was green and the only delta is the COLUMNS=240 hardening, verified locally. The PR meets the #3885 done-when: the targeted Linux smoke tests pass and the locked-skill-subset mutation is caught again. After it lands, track the target_subset static-guard follow-up as the highest-signal item.


Full per-persona findings

Python Architect

  • [recommended] No static guard covers target_subset reconstruction in LockedDependency.to_dependency_ref, and it has the same masking risk the PR fixes for skill_subset at scripts/architecture_linter/checks/install_policy_skill_and_lock.py:229
    The sibling keyword at src/apm_cli/deps/lockfile.py:1032 has no needle in RULE_LOCKED_SUBSET; a mutation to target_subset=None would pass the linter today, and a naive self.target_subset needle would be masked by the same _replay_pairing call. The scope record limits this PR to realigning existing guards, so this belongs in a follow-up rather than a fold.
    Suggested: Add a target_subset=sorted(self.target_subset) fragment plus a MutationCase.
    Proof (missing at): tests/integration/test_architecture_semantic_rule_mutations.py
  • [recommended] The new needle is a literal prefix of the mutation's old string, so the rule guards one exact spelling rather than the underlying contract at scripts/architecture_linter/checks/install_policy_skill_and_lock.py:229
    It catches the documented mutation and fails loudly rather than silently; the cost is false positives on benign refactors. A keyword-scoped AST check would enforce the contract directly, but only if the rule churns again.
    Suggested: No change needed in this PR; switch to an ast-based keyword check if the needle needs editing again.

Test Coverage Expert

  • [recommended] Dropping the persisted-apm.yml assertion from the wave7 integration test leaves target persistence on bootstrap covered only by a mocked unit test. at tests/integration/test_wave7_marketplace_install_coverage.py:995
    The old test ran with --dry-run and asserted the wrong contract; the rewrite is correct. Persistence on a real install is covered only by tests/unit/test_install_command.py::test_install_no_apm_yml_persists_target_flag (mocked). Not a regression from this PR.
    Suggested: Optional follow-up: a non-dry-run integration test asserting targets: [copilot] in the written apm.yml.
    Proof (missing at): tests/integration/test_wave7_marketplace_install_coverage.py::TestInstallAutoBootstrap::test_no_apm_yml_with_target_flag_persists_targets -- proves: apm install --target copilot in an empty directory records the chosen target in apm.yml. [devx,portability-by-manifest]
  • [recommended] The audit tests' result.stdout fix is correct only on Click >= 8.2, but pyproject allows click>=8.0.0. at tests/integration/test_audit_primitive_contract.py:77
    On Click 8.0/8.1 CliRunner mixes stderr into stdout by default. uv.lock pins click 8.2.1, so CI is fine; the assumption is already relied on elsewhere.
    Suggested: Raise the click floor to >=8.2 or note the assumption.
    Proof (unknown): tests/integration/test_audit_primitive_contract.py::test_shared_settings_command_inventory_is_visible_and_non_failing -- proves: apm audit --format json emits parseable JSON on stdout even when a deprecation warning is written to stderr. [devx]

This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.


Generated by autopilot-pr-review-worker. This comment is AI-generated and may contain errors.

@danielmeppiel
Daniel Meppiel (danielmeppiel) merged commit b3df642 into main Nov 19, 2026
27 checks passed
@danielmeppiel
Daniel Meppiel (danielmeppiel) deleted the danielmeppiel-fix-daily-release-smoke-3885 branch November 19, 2026 06:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ci] Daily release smoke is failing

2 participants