Repository navigation
ci: faster PR, merge-queue and release CI (per-platform fan-out, tag promotion, duration sharding) - #3582
ci: faster PR, merge-queue and release CI (per-platform fan-out, tag promotion, duration sharding)#3582Daniel Meppiel (danielmeppiel) wants to merge 21 commits into
Conversation
- Windows Compatibility Gate runs the windows_compat family with -n 4 --dist loadgroup (913 tests ran serially on a 4-vCPU runner and set PR time-to-gate). loadgroup keeps xdist_group affinity. - Linux unit shards use -n 4 instead of -n 2; coverage floors unchanged. - ci.yml, spec-conformance.yml and codeql.yml cancel an in-flight pull_request run when the PR gets a newer push. Every other event gets a unique per-run group, so merge_group, push, schedule and dispatch runs are never cancelled or queued. docs.yml keeps its ref-keyed group for deploys and cancels only PR build runs. - Contract tests pin the gate scheduling flags and the PR-only cancellation semantics. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Lifecycle shards failed after ~7.5 min in "Install Unix shell startup test runtimes" when a mirror stalled; a healthy install takes ~10 s. scripts/ci-apt-install.sh now bounds each apt-get call (60 s), so a stalled primary falls over to the #3570 fallback mirror, and the step gets timeout-minutes: 5. Release binary uploads, one of which hung a v0.35.1 tag job for 11.5 min, get timeout-minutes: 5 and one retry. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Lint job's R0801 duplication check takes ~3.5 min on pylint 4.0.9. On the same tree, 4.1.3 takes ~15 s locally and reports the identical result (10.00/10, no findings; a synthetic 14-line duplicate is still flagged). pylint's other requirements are unchanged, so only the two lock entries move; URLs, hashes and sizes are taken from PyPI metadata (precedent #3563). Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Workflow contract tests are incorrectly marked as unit tests despite filesystem access, and the fallback URL assertion can pass without verifying the active URL’s components.
2 open issues
What changed in this PR
This PR implements Phase 1 of #3581 to shorten APM’s CI feedback cycle and reduce reruns, without changing required-check names or test selections.
Changes:
- Runs Windows compatibility tests and Linux unit shards on four workers, and cancels superseded PR runs.
- Bounds apt operations and adds time-limited release artifact upload retries.
- Updates pylint and astroid, with regression contracts for concurrency, parallelism, and apt failover.
| File | Description |
|---|---|
uv.lock |
Updates pylint to 4.1.3 and astroid to 4.3.4. |
tests/unit/test_workflow_concurrency_contract.py |
Adds contracts for PR-only cancellation and concurrency groups. |
tests/unit/test_windows_compat_gate_workflow.py |
Verifies four-worker scheduling while preserving test selection. |
tests/unit/scripts/test_ci_apt_install.py |
Tests healthy installs, timeout failover, and bounded failure. |
scripts/ci-apt-install.sh |
Bounds apt calls and attempts package recovery before failover. |
CHANGELOG.md |
Records the CI performance changes. |
.github/workflows/spec-conformance.yml |
Adds concurrency cancellation for superseded PR runs. |
.github/workflows/docs.yml |
Limits in-progress cancellation to PR builds. |
.github/workflows/codeql.yml |
Adds PR-scoped cancellation and unique non-PR groups. |
.github/workflows/ci.yml |
Adds concurrency controls, four-worker tests, and an apt step timeout. |
.github/workflows/ci-integration.yml |
Caps the shell-runtime installation step at five minutes. |
.github/workflows/build-release.yml |
Adds bounded artifact upload retries and an apt step timeout. |
Review effort: Balanced
Under xdist the Windows gate failed twice, deterministically, on test_fresh_clone_audit_preserves_identity_across_cold_repeated_and_warm_runs[git]: "Unable to prepare Git clone configuration probe". xdist nests each worker's tmp_path one level deeper (popen-gwN), which pushes the Git cache probe past Windows MAX_PATH; serial runs pass with about 10 characters to spare. Use a short --basetemp on C:, where 8.3 aliases are enabled. Measured -n 4 on the Linux unit shards in run 43179405432: per-worker collection added 40-60 s, execution gained 0-25 s, and both shards got slower than -n 2. Revert to -n 2 and record the measurement. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Concurrency contracts read workflow YAML from disk, so they are component tests. The apt failover test now compares the active URIs entry by parsed scheme, host and path instead of a substring. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Fold the remaining CI assessment options into this PR (maintainer decision on #3581): - Option 7a: the architecture mutation files no longer pin all their tests to one xdist worker. The owner and semantic baselines are computed once per session and shared across workers through a file lock keyed by a source hash (tests/utils/xdist_shared.py). - Option 7b: tests/integration/test_architecture_* are classified architecture_contract at collection. They run once on Linux x86_64 (release, merge queue, and path-gated PR shards); arm64 and macOS lanes deselect them. - Option 8: scripts/ci_sharding.py balances whole xdist groups across shards by measured duration, plans the shard count from a target per-shard wall time, and records durations so the nightly run can publish fresh ones. Lifecycle, unit, merge-queue and release lanes use it; scripts/lifecycle_smoke_sharding.py delegates to it. - Option 10a/10b/10c: build-release.yml starts with a release plan. Pushes to main build and unit-test; schedule/dispatch runs validate everything and become promotion evidence; a tag promotes a SHA with evidence (build, smoke, release validation) and fails closed without it, unless vars.APM_RELEASE_FULL_VALIDATION_TAG names it. Builds are per platform, so each platform's integration starts as soon as its own binary exists. macOS Intel leaves the push path. - macOS capacity: still exactly one Intel and one ARM job per run, now running the platform-sensitive subset (lifecycle_smoke, windows_compat, platform_sensitive) instead of the full corpus. - Option 5b: network-dependent tests run in their own lane that retries only failed tests (at most twice). - Option 9: the merge queue builds once, plans duration-balanced hermetic shards, adds the network lane and the full Windows unit suite (three shards), and runs release validation in parallel; merge-gate requires "Windows Unit Tests" on merge_group. - Option 4: PR unit, lifecycle and architecture shards run on ubuntu-24.04-arm (A/B on this PR's runs). TEMP-REHEARSAL triggers let two rehearsal branches exercise the full, promoted and merge-queue paths without a tag; they are removed before merge. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ps it upload-artifact skips hidden paths by default, so .shard-plan/*.json matched nothing and the plan job failed after a correct plan. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Equal-weight sharding scattered each module's tests across both unit shards, exposing pre-existing cross-file state leaks (test_mcp_registry_module, test_console_utils) and raising every shard's coverage footprint. Shards now keep each module together, as the previous pytest-split chunking did, and split only modules heavier than half a shard's share. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- .github/ci-durations/unit.json seeds unit shard balance from this PR's arm64 run 43207447758 (count-balanced shards measured 232 s vs 346 s of worker time). Seed files keep tests slower than 0.5 s and fold the rest into module totals (26,496 entries -> 1,082, 71 KB). - macOS Intel no longer repeats the architecture-agnostic Python unit suite (macOS arm64 runs it) and runs the filesystem/shell/path slice (windows_compat or platform_sensitive) instead of the lifecycle family, which arm64 covers. Still one job per macOS architecture. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Update the integration-testing guide and the CI/CD instructions (source in .apm/, deployed copy and lockfile hashes refreshed; apm audit --ci passes) for the per-platform release fan-out, promotion evidence, the network lane, the merge-queue Windows unit shards and the reduced macOS subsets. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- .github/ci-durations/linux-{x86_64,arm64}.json come from rehearsal
run 43209892067's measured shard durations (11,505 and 10,072 tests,
compacted). With them the planner splits the owner-rule module whose
whole-module placement timed out merge-queue shard 1 in run
43209891804 (count-weighted bootstrap).
- The shard-count planner bounds a shard's wall by its longest serial
unit (an xdist group or one test), not by a whole module, whose tests
spread across the shard's workers.
Refs #3581
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run 43213842057 split the owner-rule module test by test across all seven x86_64 shards, so every worker repeated its warm-up and shards took 398-627 s against equal estimates. Heavy modules are now cut into the fewest in-order chunks of at most a quarter of one shard's share, and the x86_64 seed averages runs 43209892067 and 43213842057. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Merge-queue shards run with --cov, which costs about 1.5x the release lanes (18,407 vs 12,083 worker-seconds), so shards planned from release durations ranged 9.3-13.6 min in run 43217427126. The merge queue now plans from .github/ci-durations/merge-queue.json, measured in that run, with up to 12 shards (11 at the 420 s target). Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seeds .github/ci-durations/merge-queue.json from rehearsal runs 43217427126 and 43224024360 (18,407 and 18,638 worker-seconds). Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The rehearsal branches exercised the full release path (runs 43209892067, 43213842057, 43217428603, 43224024332), the merge-queue lanes (43213841924, 43217427126, 43224024360), promotion with evidence (43222627589) and fail-closed promotion without it (43217158518). Pin the triggers so a rehearsal branch cannot linger: build-release pushes from main and v* tags only, ci-integration runs on merge_group only. Refs #3581 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…e1-speedups # Conflicts: # .github/workflows/build-release.yml # uv.lock
|
Extracted into #3771 (named release fix, approved 2026-11-17), with no rebase of this branch:
Still only in this PR: the |
…gs (#3771) * ci(release): shard Linux integration, slim macOS, promote verified tags Extracts the three release-CI wins from #3582 for #3581: - Linux x86_64/arm64 integration is split across runners by measured duration (scripts/ci_sharding.py, .github/ci-durations seeds, integration-plan job), keeping xdist groups and heavy modules whole. Architecture contracts run once on x86_64 and share their baselines across workers; network-dependent tests get their own retrying lane. - macOS keeps one job per architecture: arm64 runs units plus lifecycle_smoke/windows_compat/platform_sensitive, Intel runs windows_compat/platform_sensitive. - scripts/release_plan.py picks push/full/promoted. A tag promotes only when its SHA has a green schedule/repository_dispatch run on main, and fails closed otherwise; the override is the repository variable APM_RELEASE_FULL_VALIDATION_TAG. create-release requires a pushed tag and every release-bound lane; only promoted mode may skip integration. PR-time and merge-queue changes from #3582 are out of scope. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(changelog): note release CI speed-up Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci(release): keep "push a tag" as the release trigger A v* tag without promotion evidence now runs full validation in the tag run and then publishes, instead of failing at Release Plan. A tag whose SHA has a green schedule/repository_dispatch run on main is still promoted without re-running the suites. A failed evidence lookup falls back to full validation. The APM_RELEASE_FULL_VALIDATION_TAG override is no longer needed and is removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * chore(apm): refresh lockfile hash for cicd instructions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(ci): pin the integration summary's every-lane gate and fallback summary Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci(release): opt the network lane in; match grouped durations either way - The network lane sets APM_RUN_INTEGRATION_TESTS=1, so its requires_network_integration tests run instead of skipping. - ci_sharding resolves a grouped test's duration with or without the @<group> suffix that xdist loadgroup adds, so the planner (plain collection) and the shards agree on measured weights. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Docs re-sweep (#3800) coordination: one fix overlaps this PR's hunk in - - E2E tests only run on releases (minimizing API usage)
+ - E2E tests run in the merge queue and promotion runs, not on every PR push (minimizing API usage)On main, E2E runs in |
|
Closing as superseded. This PR now conflicts with What this PR contributed, and where it landed
Result: first-attempt green PR Not landed from this PR. Each is tracked or dropped in #3581:
Written by automation (CI speedups session) on Daniel Meppiel (@danielmeppiel)'s direction. |


TL;DR
CI is faster at every stage, and every change was exercised on real runs. PR time-to-
gatedrops from 9.5 to 5.0–5.9 min. The merge queue now runs every lane and takes 13.8–16.6 min (was 18.5 min p50). A full release validation drops from 81–85 min to 31–35 min. A release tag whose commit already passed full validation is promoted in 4.9 min instead of re-running everything for about 82 min. macOS still runs exactly one Intel and one Apple Silicon job, each doing less work.Fixes #3581
Important
mainis red for a reason unrelated to this PR: the Linux arm64 binary build fails the glibc floor check added in #3553 (GLIBC_2.35 … libgcc_s.so.1, e.g. run 43211571466). The release rehearsals added one rehearsal-only commit that skipped that check on arm64. That commit is not in this PR.Issue and approved scope
Issue: Fixes #3581
Human scope approval: Daniel Meppiel (@danielmeppiel)'s direction is recorded in #3581. That covers Phase 1, plus the later decision to fold Options 4–10 into this PR, test all of them, and not fan out macOS runners (comment). Both are automation-written records of a session mandate, not human-authored approval comments.
Does this PR complete the issue? Yes, with one exception. Option 6 depends on #3580, which was reverted on main, so it is a follow-up. The table below lists every option and what changed in its scope.
Problem (WHY)
gatetook 9.5 min at p50.Approach (WHAT)
-n 4 --dist loadgroup; cancel superseded PR runs; time-boxed apt installs and uploads; pylint 4.1.3ubuntu-24.04-armarchitecture_contractmarker; those tests run once on Linux x86_64 (release, merge queue, path-gated PR shards) and are deselected elsewherescripts/ci_sharding.py: balances shards by measured duration, keeps xdist groups and modules whole, plans the shard count; nightly publishes durations, with committed seeds as fallbackWindows Unit Tests(3 shards);EXPECTED_CHECKSupdatedrelease-planmodes: push / full / promoted. Tag promotion needs evidence and fails closed without it. Per-platform builds; macOS Intel no longer runs on push; no dispatch needed when evidence existsImplementation (HOW)
.github/workflows/build-release.yml—release-planrunsscripts/release_plan.py. Builds are now separate jobs per platform:build-linux-x86,build-linux-armandbuild-windows, sharing steps through a YAML anchor so the check names are unchanged.integration-plansizes the Linux shards while the binaries build. The integration lanes areintegration-linux-{x86,arm}(sharded),integration-networkandintegration-windows;Integration Tests Summaryaggregates them and publishes durations. Release validation runs per platform.create-releaserequires a pushed tag and every release-bound lane, and allows a skipped integration only in promoted mode.(lifecycle_smoke or windows_compat or platform_sensitive). Intel runs(windows_compat or platform_sensitive)and skips the architecture-agnostic unit suite. Promoted tags only build, smoke-test and run release validation.scripts/release_plan.py— evidence means a green schedule orrepository_dispatchrun on this SHA, or a green main push plus a green merge-queue run. The override is the repository variableAPM_RELEASE_FULL_VALIDATION_TAG=<tag>, not aworkflow_dispatchinput:test_security_workflow_contracts.pyforbidsworkflow_dispatchon this secret-bearing workflow..github/workflows/ci-integration.yml,merge-gate.yml— see Option 9. Release validation now runs in parallel with integration.EXPECTED_CHECKS(merge_group only) addsWindows Unit Tests..github/workflows/ci.yml— unit and lifecycle shards run on arm64 and are balanced by duration. A new non-requiredArchitecture Contractsjob (3 shards) runs when a PR touchessrc/, the linter or its tests.scripts/ci_sharding.py,scripts/lifecycle_smoke_sharding.py,scripts/ci-shard-durations.sh,.github/ci-durations/*.json— the planner, the partition and the duration recorder. Seeds are compacted: tests over 0.5 s are kept and the rest fold into module totals (71–397 KB). The merge queue has its own coverage-inclusive seed.scripts/test-integration.sh— runspython -m pytestso repository plugins load, and supportsPYTEST_RETRY_FAILED=N(exit code 1 only,--last-failed,--cov-append).tests/utils/xdist_shared.py;architecture_contract, applied bypytest_itemcollected;platform_sensitiveon 7 integration files.test_platform_contract_workflow.pybecame invariants: lane fan-out, a marker-algebra check that the lanes partition the non-live corpus, the macOS two-job cap, and publication gates. Newtest_merge_queue_workflow_contract.py,test_release_plan.pyandtest_ci_sharding.py.effective_envnow resolves${{ env.X }}.Diagrams
Legend: release jobs per platform. Each platform's lanes start when its own binary is built, and only a pushed tag can publish.
flowchart LR P["release-plan<br/>push / full / promoted"] --> BX["build-linux-x86"] P --> BA["build-linux-arm"] P --> BW["build-windows"] P --> MI["macOS Intel job<br/>(full, promoted)"] P --> MA["macOS arm64 job<br/>(full, promoted)"] P --> IP["integration-plan<br/>(full only)"] BX --> IX["x86_64 shards 1..N<br/>hermetic + architecture"] BX --> NW["network lane<br/>retries failed tests"] BA --> IA["arm64 shards 1..M<br/>no architecture"] BW --> IW["Windows integration"] IP --> IX IP --> IA BX --> RV["release validation"] BA --> RV BW --> RVW["release validation (Windows)"] IX & IA & NW & IW --> S["Integration Tests Summary<br/>publishes durations"] S & RV & RVW & MI & MA --> CR{"create-release<br/>pushed tag only"}Legend: how a pushed tag decides between promotion, the one-tag override, and refusal.
flowchart TD T["tag pushed"] --> E{"green full run on this SHA?<br/>or green main push + merge queue?"} E -->|"yes"| PR["promoted: build + smoke + release validation<br/>no repeated unit or integration"] E -->|"no"| O{"vars.APM_RELEASE_FULL_VALIDATION_TAG == tag?"} O -->|"yes"| F["full validation in this run, then publish"] O -->|"no"| X["fail closed: nothing builds or publishes"]Trade-offs
test_mcp_registry_module,test_console_utils). Heavy modules are cut into a few contiguous chunks instead, because per-worker warm-up made per-test scatter slower.--cov): 1.46, 2.10 and 1.63. The cause is xdist tails and per-runner fixture warm-up, not total work, which the plan predicts to within 2%. Nightly durations keep refining the estimate.workflow_dispatchfor this secret-bearing workflow.build-release.ymlpushes tomainandv*andci-integration.ymltomerge_group.Benefits
gate: 9.5 → 5.0 min on the final head (5.6–5.9 on earlier arm64 runs).Testing
Real CI runs (all green unless noted)
ci.yml+gate, final head5db80c906Release Planrefused; every downstream job skipped.shard-plandir), 43209891804 (count-weighted shard timeout), 43213842057 (arm64 #3553)Baselines: nightly 43152428915 (84.8 min); assessment p50s (dispatch 81.5 min, merge queue 18.5 min,
gate9.5 min).7a evidence: nightly x86 lane had 1,384 architecture nodes passed; the rehearsal had 1,387 passed (3 new from main merges) with 0 outcome differences on the shared nodes.
Local validation
Scenario evidence
test_lane_selection_partitions_the_non_live_corpus(marker algebra over every marker combination)test_integration_lanes_fan_out_per_platform,test_macos_capacity_is_two_jobs_with_no_fan_outtest_only_verified_tags_publish,test_tag_without_evidence_fails_closed, run 43217158518test_ci_sharding.py,test_lifecycle_shards_collect_exact_required_unionHow to test
5db80c906: everything is green, andArchitecture Contractsshards appear.Release Planlogsmode=promoted … evidence=…/43217428603, and integration, release creation and PyPI are skipped.integration-durations.Spec conformance (OpenAPM v0.1)
Type of change
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com