Repository navigation
ci(release): shard Linux integration, slim macOS, promote verified tags - #3771
Conversation
Extracts the three release-CI wins from #3582 for #3581: - Linux x86_64/arm64 integration is split across runners by measured duration (scripts/ci_sharding.py, .github/ci-durations seeds, integration-plan job), keeping xdist groups and heavy modules whole. Architecture contracts run once on x86_64 and share their baselines across workers; network-dependent tests get their own retrying lane. - macOS keeps one job per architecture: arm64 runs units plus lifecycle_smoke/windows_compat/platform_sensitive, Intel runs windows_compat/platform_sensitive. - scripts/release_plan.py picks push/full/promoted. A tag promotes only when its SHA has a green schedule/repository_dispatch run on main, and fails closed otherwise; the override is the repository variable APM_RELEASE_FULL_VALIDATION_TAG. create-release requires a pushed tag and every release-bound lane; only promoted mode may skip integration. PR-time and merge-queue changes from #3582 are out of scope. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The network lane can count as promotion evidence while its tests are skipped due to a missing opt-in, and grouped-test timing lookups miss measured durations, potentially selecting too few shards.
2 open issues
What changed in this PR
Restructures APM's release pipeline to reduce validation time through parallel Linux testing, focused macOS coverage, and evidence-based tag promotion. This implements the release-focused portion of #3581; PR and merge-queue changes remain in #3582.
Changes:
- Adds duration-balanced Linux shards, a separate network lane with bounded retries, and shared architecture-test baselines.
- Limits macOS validation to platform-sensitive subsets while retaining one job per architecture.
- Introduces push, full, and promoted release modes, with supporting contracts and documentation.
| File | Description |
|---|---|
tests/workflow_contracts.py |
Resolves environment references across workflow scopes. |
tests/utils/xdist_shared.py |
Adds locked, session-shared fixture caching. |
tests/unit/test_xdist_shared.py |
Covers shared caching, lock timeout, and source hashing. |
tests/unit/test_windows_unit_diagnostics.py |
Updates diagnostics contracts for the Windows build job. |
tests/unit/test_windows_signing_contract.py |
Retargets signing contracts to the Windows build job. |
tests/unit/test_platform_contract_workflow.py |
Checks lane selection, sharding, macOS capacity, and publication gates. |
tests/unit/test_ado_live_workflow_contract.py |
Updates live-ADO exclusion checks for the new lanes. |
tests/unit/scripts/test_release_plan.py |
Tests release modes and promotion-evidence requirements. |
tests/unit/scripts/test_ci_sharding.py |
Tests partitioning, duration weighting, planning, and recording. |
tests/integration/test_unix_installer_path_integration.py |
Marks installer-path tests as platform-sensitive. |
tests/integration/test_sparse_cone_symlink_repair.py |
Marks symlink-repair tests as platform-sensitive. |
tests/integration/test_self_update_release_lifecycle.py |
Marks self-update lifecycle tests as platform-sensitive. |
tests/integration/test_mixed_case_materialization_lifecycle.py |
Marks case-sensitive materialization tests for platform coverage. |
tests/integration/test_lockfile_symlink_write_e2e.py |
Marks lockfile symlink tests as platform-sensitive. |
tests/integration/test_integrators_hooks_execution.py |
Marks hook-execution tests as platform-sensitive. |
tests/integration/test_audit_unrecorded_unicode_lifecycle.py |
Marks Unicode audit lifecycle tests as platform-sensitive. |
tests/integration/test_architecture_semantic_rule_mutations.py |
Shares semantic-rule baselines across workers. |
tests/integration/test_architecture_owner_rule_mutations.py |
Shares owner-rule baselines across workers. |
tests/integration/test_architecture_install_compound_mutations.py |
Removes forced single-worker grouping. |
tests/integration/test_architecture_contract_compound_mutations.py |
Removes forced single-worker grouping. |
tests/integration/test_architecture_confirmed_bypass_mutations.py |
Removes forced single-worker grouping. |
tests/integration/test_architecture_compile_inventory_mutations.py |
Removes forced single-worker grouping. |
tests/integration/test_architecture_bundle_format.py |
Removes forced single-worker grouping. |
tests/integration/conftest.py |
Classifies architecture contracts before marker selection. |
scripts/test-integration.sh |
Adds failed-test retries and module-based pytest invocation. |
scripts/release_plan.py |
Selects release modes and verifies promotion evidence. |
scripts/ci-shard-durations.sh |
Selects nightly duration data with committed-seed fallback. |
scripts/ci_sharding.py |
Implements duration-aware shard planning and recording. |
pyproject.toml |
Registers architecture and platform-sensitive markers. |
docs/src/content/docs/contributing/integration-testing.md |
Documents release modes and platform test selections. |
CHANGELOG.md |
Records the release-CI changes. |
.github/workflows/build-release.yml |
Wires platform builds, integration lanes, and promotion gates. |
.github/instructions/cicd.instructions.md |
Updates contributor guidance for the release topology. |
.apm/instructions/cicd.instructions.md |
Updates the corresponding APM instruction source. |
Review effort: Balanced
A v* tag without promotion evidence now runs full validation in the tag run and then publishes, instead of failing at Release Plan. A tag whose SHA has a green schedule/repository_dispatch run on main is still promoted without re-running the suites. A failed evidence lookup falls back to full validation. The APM_RELEASE_FULL_VALIDATION_TAG override is no longer needed and is removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Advisory review: PR #3771Release CI pipeline sharded and gated by tag-promotion script, cutting wall-clock time while preserving fail-closed validation. panel-mode=lean; personas=supply-chain-security-expert SummaryThe fail-closed design is sound: a tag publishes only when a green
Recommended follow-ups (curated)
Ship recommendationship_with_followups. There are no blocking findings, and the PR body reports passing tests. Track the follow-ups above in a small post-merge PR. CODEOWNERS review requests ( cc Daniel Meppiel (@danielmeppiel)
|
…summary Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- The network lane sets APM_RUN_INTEGRATION_TESTS=1, so its requires_network_integration tests run instead of skipping. - ci_sharding resolves a grouped test's duration with or without the @<group> suffix that xdist loadgroup adds, so the planner (plain collection) and the shards agree on measured weights. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
APM Review Panel:
|
| Persona | B | R | N | Takeaway |
|---|---|---|---|---|
| Supply Chain Security Expert | 0 | 1 | 0 | Publication integrity is sound. Main-branch provenance for tags without evidence now rests on the tag ruleset. |
| Test Coverage Expert | 0 | 2 | 0 | Strong mutation-style coverage. Both gaps are folded on this head. |
| Python Architect | 0 | 0 | 0 | Single-owner mode authority is preserved, and the topology is correct for every tag path. |
B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.
Top 1 follow-ups
- [Supply Chain Security Expert] Add a main-ancestry check (compare API or merge-base) for tags, so that only commits reachable from
maincan publish. -- This restores the provenance invariant inside the pipeline instead of relying only on the tag ruleset.
Recommendation
Merge now to unblock the v0.36.0 tag. The main-ancestry guard is hardening, not a regression fix; track it as an issue before the next release cycle. Every other panelist finding is resolved on this head.
Full per-persona findings
Supply Chain Security Expert
- [recommended] The no-evidence tag path removes the main-branch provenance invariant from the release pipeline, at
scripts/release_plan.py:79
Anyv*tag without evidence now gets full validation and is published. The tested binary keeps its integrity, but "released code went through main's review" now depends on the tag ruleset.
Suggested: Check ancestry withgit merge-base --is-ancestoror the compare API, or pin the tag ruleset in a contract test.
Test Coverage Expert
- [recommended] Nothing pinned the "Require every integration lane" step in
Integration Tests Summary. Folded:_assert_integration_summary_requires_every_laneplus four mutation tests. - [recommended] The "Evidence: none:" step-summary diagnostic was not asserted on the no-evidence
main()path. Folded:test_main_writes_outputs_and_summarynow asserts it.
Python Architect
No findings.
This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.
Generated by autopilot-pr-review-worker. This comment is AI-generated and may contain errors.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolves the CHANGELOG [Unreleased] conflict with #3771: keep both entries, Changed before Fixed per Keep a Changelog. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* chore: release v0.36.1 Bump apm-cli to 0.36.1 in pyproject.toml and the apm-cli entry of uv.lock (no uv lock re-resolve), and cut the [Unreleased] CHANGELOG block into [0.36.1] - 2026-11-18 with upgrade notes. The #3771 and #3776 notes move into [0.36.0], where they shipped. Lint mirror (ruff check, ruff format --check, pylint R0801, auth-signals) is clean on main 70554b6. Post-merge: run the pre-tag gate on the merge commit, then tag v0.36.1. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(changelog): do not claim ADO issues fixed before a live-org check Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(changelog): close the Azure DevOps issues in the 0.36.1 entry Reverts the previous wording change at @danielmeppiel's direction: #3752, #3753 and #3758 are closed by #3790. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(changelog): note SSE removals and the COPILOT_HOME skill scope in 0.36.1 upgrade notes Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(changelog): drop the 0.36.1 upgrade-notes paragraph Patch releases carry upgrade details in their entries, as before. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* docs(changelog): make 0.36.0 onwards legible and concise Rewrites [Unreleased], [0.36.1] and [0.36.0] as one short line per user-visible change, with Highlights and a Breaking changes and upgrade notes list per version, grouped PRs and internal-only entries dropped (#3637, #3771, #3794, #3796, #3877). Credits, PR references, the 0.36.1 issue closures and the live Azure DevOps caveat are kept. Sections from 0.35.1 down are unchanged. The changelog instructions now describe this style. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * chore(lock): record the new changelog instructions hash Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs: note the shared .mcp.json and VS Code Local mode in 0.37 upgrade notes Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(changelog): restore accuracy qualifiers flagged in review Restores the static env caveat for .mcp.json, the pack default-format qualifier, Copilot CLI scope for COPILOT_HOME, the 0.36.1 frozen-lockfile recovery step, and the bundle install --target qualifier. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Description
Named release fix (Lane L freeze exemption, approved by Daniel Meppiel (@danielmeppiel) on 2026-11-17). It extracts the three biggest release-CI wins from #3582 into one PR scoped to
build-release.ymland its scripts and tests. Releasing does not change: push av*tag. What changes is how much work that tag run does, and how fast every other run is.How to release
manual-build-releasedispatch onmain), the tag run is promoted. It builds the 5 binaries, smoke-tests them, runs release validation and publishes in about 5 min.Pipeline modes
flowchart TD E{"Trigger"} -->|"push to main"| P["push<br/>unit tests + Linux/Windows binaries<br/>no macOS, no publish"] E -->|"nightly schedule or<br/>repository_dispatch"| F["full<br/>every lane, no publish<br/>green run = evidence for its SHA"] E -->|"v* tag"| L{"green full run on main<br/>for this exact SHA?<br/>(lookup error = no)"} L -->|"yes"| PR["promoted<br/>build + smoke + release validation<br/>then publish (~5 min)"] L -->|"no"| FT["full validation in this run<br/>then publish (~35 min)"]Full-run topology
Each platform's lanes start as soon as that platform's binary is built. There is no whole-matrix barrier, and macOS still runs exactly two jobs.
flowchart LR RP["Release Plan"] --> IP["Integration Shard Plan<br/>(durations: nightly artifact,<br/>else committed seed)"] RP --> BX["Build & Test<br/>linux x86_64"] RP --> BA["Build & Test<br/>linux arm64"] RP --> BW["Build & Test<br/>windows"] RP --> MI["Build & Validate<br/>macOS x86_64"] RP --> MA["Build & Validate<br/>macOS arm64"] IP --> IX IP --> IA BX --> IX["x86_64 shards 1..N<br/>hermetic + architecture contracts"] BX --> NW["network lane<br/>retries failed tests (max 2)"] BA --> IA["arm64 shards 1..M<br/>hermetic"] BW --> IW["Windows integration"] BX --> RV["Release Validation<br/>linux x86_64 / arm64"] BA --> RV BW --> RVW["Release Validation<br/>windows"] IX & IA & NW & IW --> S["Integration Tests Summary<br/>requires every lane,<br/>publishes durations"] S & RV & RVW & MI & MA --> CR{"Create GitHub Release<br/>pushed tag only"} CR --> PY["PyPI, docs, GH-AW compat"]Promoted tag (fast path)
What changed
scripts/ci_sharding.py, the seeds in.github/ci-durations/and theIntegration Shard Planjob keep xdist groups and heavy modules whole. Architecture-linter contracts (architecture_contract) run once on x86_64 and share their baselines across workers (tests/utils/xdist_shared.py). Network-dependent tests run in their own lane, which retries only failed tests. On the current corpus the planner produces 7 x86_64 shards and 3 arm64 shards of about 7.5 min of test time each. The nightly run publishes measured durations for the next plan.lifecycle_smoke or windows_compat or platform_sensitive. Intel runswindows_compat or platform_sensitive. Seven OS-, filesystem- or shell-sensitive integration modules are now markedplatform_sensitive. The platform-agnostic remainder runs once on Linux.scripts/release_plan.pypickspush,fullorpromoted. Promotion evidence is only a greenscheduleorrepository_dispatchrun ofbuild-release.ymlonmainfor the exact SHA. This is stricter than ci: faster PR, merge-queue and release CI (per-platform fan-out, tag promotion, duration sharding) #3582, which also accepted a main push plus a merge-queue run; main's merge queue covers Linux x86_64 integration only. There is noworkflow_dispatch(test_security_workflow_contracts.py).Builds are now one job per platform (
build-linux-x86,build-linux-arm,build-windows) and share their steps through a YAML anchor. Check names are unchanged:Build & Test (...),Build & Validate (macOS ...),Release Validation (...),Create GitHub Release. The only required check,gate, is untouched, and no ruleset requiresbuild-release.ymlchecks.Expected timings (from the #3582 rehearsals; post-merge numbers will be posted here):
Issue and approved scope
Issue: Refs #3581
Human scope-approval comment: #3581 (comment), which supersedes #3581 (comment) after the "keep releasing on a plain tag" change. Context: #3581 (comment)
Does this PR complete the issue, or what remains? Partial. The PR-time (
ci.yml) and merge-queue (ci-integration.yml,merge-gate.yml) changes stay in #3582, which remains open.Type of change
Testing
Guardrails:
test_lane_selection_partitions_the_non_live_corpus: the lanes partition the non-live corpus, checked by marker algebra.test_macos_capacity_is_two_jobs_with_no_fan_out: at most two macOS jobs.create-releaseneeds a pushed tag and every release-bound lane, and only promoted mode may skip integration.test_release_plan.py: strict evidence; push and merge-queue runs never promote; a lookup failure never promotes; an unpromoted tag validates fully and still publishes.test_ci_sharding.py: every node lands in exactly one shard, with xdist groups and modules whole.Validation (one process, no xdist):
pytest tests/unit/scripts/test_release_plan.py tests/unit/scripts/test_ci_sharding.py tests/unit/test_xdist_shared.py -p no:xdistpytest tests/unit/test_platform_contract_workflow.py tests/unit/test_windows_signing_contract.py tests/unit/test_windows_unit_diagnostics.py tests/unit/test_ado_live_workflow_contract.py tests/unit/test_security_workflow_contracts.py tests/unit/test_windows_compat_gate_workflow.py tests/unit/test_docs_contract_workflow.py tests/unit/test_guardrailing_hero_topology.py tests/unit/test_windows_native_symlink_workflow.py tests/unit/integration/test_integration_wiring_invariants.py tests/unit/scripts/test_lifecycle_smoke_sharding.py -p no:xdistpytest tests/unit/scripts/test_release_plan.py tests/unit/test_platform_contract_workflow.py tests/unit/test_security_workflow_contracts.py -p no:xdist(current head)not livevs hermetic + network, and hermetic vs arm64 +architecture_contractruff check/ruff format --check(0.15.12, CI paths)pylint --enable=R0801 ...bash scripts/lint-auth-signals.shactionlint .github/workflows/build-release.ymlSpec conformance (OpenAPM v0.1)