Skip to content

ci: faster PR, merge-queue and release CI (per-platform fan-out, tag promotion, duration sharding) - #3582

Closed
Daniel Meppiel (danielmeppiel) wants to merge 21 commits into
mainfrom
danielmeppiel-ci-phase1-speedups
Closed

Daniel Meppiel (danielmeppiel) wants to merge 21 commits into
mainfrom
danielmeppiel-ci-phase1-speedups

Conversation

@danielmeppiel

@danielmeppiel Daniel Meppiel (danielmeppiel) commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

TL;DR

CI is faster at every stage, and every change was exercised on real runs. PR time-to-gate drops from 9.5 to 5.0–5.9 min. The merge queue now runs every lane and takes 13.8–16.6 min (was 18.5 min p50). A full release validation drops from 81–85 min to 31–35 min. A release tag whose commit already passed full validation is promoted in 4.9 min instead of re-running everything for about 82 min. macOS still runs exactly one Intel and one Apple Silicon job, each doing less work.

Fixes #3581

Important

main is red for a reason unrelated to this PR: the Linux arm64 binary build fails the glibc floor check added in #3553 (GLIBC_2.35 … libgcc_s.so.1, e.g. run 43211571466). The release rehearsals added one rehearsal-only commit that skipped that check on arm64. That commit is not in this PR.

Issue and approved scope

Issue: Fixes #3581

Human scope approval: Daniel Meppiel (@danielmeppiel)'s direction is recorded in #3581. That covers Phase 1, plus the later decision to fold Options 4–10 into this PR, test all of them, and not fan out macOS runners (comment). Both are automation-written records of a session mandate, not human-authored approval comments.

Does this PR complete the issue? Yes, with one exception. Option 6 depends on #3580, which was reverted on main, so it is a follow-up. The table below lists every option and what changed in its scope.

Problem (WHY)

  • The PR critical path was a serial Windows gate, and Linux shards were half-used. gate took 9.5 min at p50.
  • The release ran the whole corpus on single runners. Linux x86 integration took 68.7 min and macOS arm64 73.5 min (nightly 43152428915). Every Linux lane waited for the Windows build. One architecture xdist group ran on one worker for 2,506 s.
  • Tags re-tested commits that were already validated. v0.35.1 took about 3 h across a dispatch rehearsal, the tag run and reruns.
  • The merge queue lacked Windows unit tests. 77 of 91 red pushes on main included Windows Build & Test, which only ran after merge.
  • [!] Network flakes and naive sharding. A network failure failed a 50-min lane, and file-count shards were up to 9x imbalanced.

Approach (WHAT)

Option Change Status
1, 3, 5a/c, 2 (Phase 1) Windows gate on -n 4 --dist loadgroup; cancel superseded PR runs; time-boxed apt installs and uploads; pylint 4.1.3 done, tested
4 PR unit, lifecycle and architecture shards on ubuntu-24.04-arm done, A/B below
5b Network-dependent tests in their own lane, which re-runs only failed tests (at most twice) done
7a Architecture mutation tests no longer pinned to one xdist worker; module baselines shared once per session (file lock keyed by source hash) done
7b architecture_contract marker; those tests run once on Linux x86_64 (release, merge queue, path-gated PR shards) and are deselected elsewhere done
8 scripts/ci_sharding.py: balances shards by measured duration, keeps xdist groups and modules whole, plans the shard count; nightly publishes durations, with committed seeds as fallback done; ratio target partly met
9 (workflow side) Merge queue builds once, plans hermetic shards, adds the network lane and Windows Unit Tests (3 shards); EXPECTED_CHECKS updated done; no ruleset change
10a/b/c/d release-plan modes: push / full / promoted. Tag promotion needs evidence and fails closed without it. Per-platform builds; macOS Intel no longer runs on push; no dispatch needed when evidence exists done
6 Path-owned families shared with preflight follow-up (#3580 reverted)

Implementation (HOW)

  • .github/workflows/build-release.yml — release-plan runs scripts/release_plan.py. Builds are now separate jobs per platform: build-linux-x86, build-linux-arm and build-windows, sharing steps through a YAML anchor so the check names are unchanged. integration-plan sizes the Linux shards while the binaries build. The integration lanes are integration-linux-{x86,arm} (sharded), integration-network and integration-windows; Integration Tests Summary aggregates them and publishes durations. Release validation runs per platform. create-release requires a pushed tag and every release-bound lane, and allows a skipped integration only in promoted mode.
  • macOS — still one Intel and one arm64 job, with no matrix. arm64 runs unit tests plus (lifecycle_smoke or windows_compat or platform_sensitive). Intel runs (windows_compat or platform_sensitive) and skips the architecture-agnostic unit suite. Promoted tags only build, smoke-test and run release validation.
  • scripts/release_plan.py — evidence means a green schedule or repository_dispatch run on this SHA, or a green main push plus a green merge-queue run. The override is the repository variable APM_RELEASE_FULL_VALIDATION_TAG=<tag>, not a workflow_dispatch input: test_security_workflow_contracts.py forbids workflow_dispatch on this secret-bearing workflow.
  • .github/workflows/ci-integration.yml, merge-gate.yml — see Option 9. Release validation now runs in parallel with integration. EXPECTED_CHECKS (merge_group only) adds Windows Unit Tests.
  • .github/workflows/ci.yml — unit and lifecycle shards run on arm64 and are balanced by duration. A new non-required Architecture Contracts job (3 shards) runs when a PR touches src/, the linter or its tests.
  • scripts/ci_sharding.py, scripts/lifecycle_smoke_sharding.py, scripts/ci-shard-durations.sh, .github/ci-durations/*.json — the planner, the partition and the duration recorder. Seeds are compacted: tests over 0.5 s are kept and the rest fold into module totals (71–397 KB). The merge queue has its own coverage-inclusive seed.
  • scripts/test-integration.sh — runs python -m pytest so repository plugins load, and supports PYTEST_RETRY_FAILED=N (exit code 1 only, --last-failed, --cov-append).
  • Tests and markers — tests/utils/xdist_shared.py; architecture_contract, applied by pytest_itemcollected; platform_sensitive on 7 integration files.
  • Contract tests — the pinned flag strings in test_platform_contract_workflow.py became invariants: lane fan-out, a marker-algebra check that the lanes partition the non-live corpus, the macOS two-job cap, and publication gates. New test_merge_queue_workflow_contract.py, test_release_plan.py and test_ci_sharding.py. effective_env now resolves ${{ env.X }}.

Diagrams

Legend: release jobs per platform. Each platform's lanes start when its own binary is built, and only a pushed tag can publish.

flowchart LR
  P["release-plan<br/>push / full / promoted"] --> BX["build-linux-x86"]
  P --> BA["build-linux-arm"]
  P --> BW["build-windows"]
  P --> MI["macOS Intel job<br/>(full, promoted)"]
  P --> MA["macOS arm64 job<br/>(full, promoted)"]
  P --> IP["integration-plan<br/>(full only)"]
  BX --> IX["x86_64 shards 1..N<br/>hermetic + architecture"]
  BX --> NW["network lane<br/>retries failed tests"]
  BA --> IA["arm64 shards 1..M<br/>no architecture"]
  BW --> IW["Windows integration"]
  IP --> IX
  IP --> IA
  BX --> RV["release validation"]
  BA --> RV
  BW --> RVW["release validation (Windows)"]
  IX & IA & NW & IW --> S["Integration Tests Summary<br/>publishes durations"]
  S & RV & RVW & MI & MA --> CR{"create-release<br/>pushed tag only"}
Loading

Legend: how a pushed tag decides between promotion, the one-tag override, and refusal.

flowchart TD
  T["tag pushed"] --> E{"green full run on this SHA?<br/>or green main push + merge queue?"}
  E -->|"yes"| PR["promoted: build + smoke + release validation<br/>no repeated unit or integration"]
  E -->|"no"| O{"vars.APM_RELEASE_FULL_VALIDATION_TAG == tag?"}
  O -->|"yes"| F["full validation in this run, then publish"]
  O -->|"no"| X["fail closed: nothing builds or publishes"]
Loading

Trade-offs

  • macOS gets less work, not more jobs. The Intel unit suite and lifecycle family moved to arm64, and the platform-agnostic tests moved to Linux. Never more than 2 macOS jobs run per workflow run.
  • Shards keep modules whole. Scattering tests exposed existing cross-file state leaks (test_mcp_registry_module, test_console_utils). Heavy modules are cut into a few contiguous chunks instead, because per-worker warm-up made per-test scatter slower.
  • Balance target (max/min < 1.3) is only partly met. arm64 shards: 1.09–1.13. x86_64 release shards: 1.11, 1.26 and 1.48. Merge-queue shards (with --cov): 1.46, 2.10 and 1.63. The cause is xdist tails and per-runner fixture warm-up, not total work, which the plan predicts to within 2%. Nightly durations keep refining the estimate.
  • The override is a repository variable, not an input. This keeps the existing ban on workflow_dispatch for this secret-bearing workflow.
  • Rehearsal used branch triggers that are now removed. A contract test pins build-release.yml pushes to main and v* and ci-integration.yml to merge_group.

Benefits

  1. PR time-to-gate: 9.5 → 5.0 min on the final head (5.6–5.9 on earlier arm64 runs).
  2. Full release validation: 84.8 → 31–35 min. Linux x86 integration drops from 68.7 min to shards of 7–11 min; macOS arm64 from 73.5 to about 31 min; Intel from 36.6 to 9–12 min.
  3. Promoted tag: about 82 min → 4.9 min. A tag without evidence is refused before anything builds.
  4. The merge queue covers every lane, including full Windows unit tests, in 13.8–16.6 min.
  5. The architecture owner-rule module span drops from 2,506 s on one worker to 118–168 s per shard, with identical results.

Testing

  • Tested locally
  • All existing tests pass (CI on the final head; see runs below)
  • Added tests for new functionality
Real CI runs (all green unless noted)
Path Runs Wall
PR ci.yml + gate, final head 5db80c906 43232616695 / 43232616650 gate 5.03 min
PR on arm64 (A/B, Option 4) 43207447758, 43208932799, 43210340135, 43213838711 gate 5.92, 5.57, 5.92, 5.80
PR on x86_64 (Phase 1) 43181971921, 43183414217 gate 7.17, 6.77
Full release rehearsal 43217428603, 43224024332, 43229679061 31.2, 35.2, 32.1 min
Merge-queue rehearsal 43213841924, 43217427126, 43224024360, 43229679228 16.6, 15.7, 15.4, 13.8 min
Promoted tag path (evidence: 43217428603) 43222627589 4.9 min; integration and publishing skipped
Fail-closed tag path 43217158518 Release Plan refused; every downstream job skipped
Earlier iterations (bugs fixed in-branch) 43202913457 (hidden .shard-plan dir), 43209891804 (count-weighted shard timeout), 43213842057 (arm64 #3553) —

Baselines: nightly 43152428915 (84.8 min); assessment p50s (dispatch 81.5 min, merge queue 18.5 min, gate 9.5 min).

Job (baseline → run 4 / 5 / 6) Baseline Rehearsal
Linux x86 integration 68.7 7 shards, 6.9–11.0
Linux arm64 integration 34.9 3 shards, 7.5–8.9
macOS arm64 73.5 31 / 35 / 32
macOS Intel 36.6 9.3 / 12.3 / 11.0
Windows unit (merge queue) after merge only 3 shards, 3.0–4.4
PR lifecycle shard (max) 6.3–6.8 (x86) 4.4–4.8 (arm64)

7a evidence: nightly x86 lane had 1,384 architecture nodes passed; the rehearsal had 1,387 passed (3 new from main merges) with 0 outcome differences on the shared nodes.

Local validation
$ uv run --extra dev ruff check src/ tests/ scripts/...      -> All checks passed!
$ uv run --extra dev ruff format --check ...                 -> 2066 files already formatted
$ python -m pylint --disable=all --enable=R0801 ... src/apm_cli/ ... -> 10.00/10
$ bash scripts/lint-auth-signals.sh                          -> [+] auth-signal lint clean
$ bash scripts/lint-architecture-boundaries.sh               -> exit 0
$ apm audit --ci                                             -> All 10 check(s) passed
$ actionlint (changed workflows)                             -> clean
$ pytest -n 4 <contract + script suites>                     -> 581 passed, 3 skipped
$ pytest tests/quality/test_ci_topology.py                   -> 144 passed (in isolation)
$ BASE_REF=origin/main bash tests/spec_conformance/mode_b_detector.sh -> no critical-path diff

Scenario evidence

Scenario Test Principle
Lanes partition the non-live corpus; macOS subsets stay inside it test_lane_selection_partitions_the_non_live_corpus (marker algebra over every marker combination) Govern: no silently dropped tests
No whole-matrix barrier; at most two macOS jobs test_integration_lanes_fan_out_per_platform, test_macos_capacity_is_two_jobs_with_no_fan_out Capacity, speed
Only verified pushed tags publish test_only_verified_tags_publish, test_tag_without_evidence_fails_closed, run 43217158518 Secure release
Shards keep groups and modules whole and cover every node exactly once test_ci_sharding.py, test_lifecycle_shards_collect_exact_required_union Determinism

How to test

  • Check this PR's checks on 5db80c906: everything is green, and Architecture Contracts shards appear.
  • Open run 43222627589: Release Plan logs mode=promoted … evidence=…/43217428603, and integration, release creation and PyPI are skipped.
  • Open run 43217158518: the refusal message names both remedies.
  • After merge, compare the first nightly's wall time with 43152428915, and check that it uploads integration-durations.

Spec conformance (OpenAPM v0.1)

  • N/A -- this PR does not change OpenAPM-observable behaviour.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

- Windows Compatibility Gate runs the windows_compat family with
  -n 4 --dist loadgroup (913 tests ran serially on a 4-vCPU runner and
  set PR time-to-gate). loadgroup keeps xdist_group affinity.
- Linux unit shards use -n 4 instead of -n 2; coverage floors unchanged.
- ci.yml, spec-conformance.yml and codeql.yml cancel an in-flight
  pull_request run when the PR gets a newer push. Every other event gets
  a unique per-run group, so merge_group, push, schedule and dispatch
  runs are never cancelled or queued. docs.yml keeps its ref-keyed group
  for deploys and cancels only PR build runs.
- Contract tests pin the gate scheduling flags and the PR-only
  cancellation semantics.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Lifecycle shards failed after ~7.5 min in "Install Unix shell startup
test runtimes" when a mirror stalled; a healthy install takes ~10 s.
scripts/ci-apt-install.sh now bounds each apt-get call (60 s), so a
stalled primary falls over to the #3570 fallback mirror, and the step
gets timeout-minutes: 5. Release binary uploads, one of which hung a
v0.35.1 tag job for 11.5 min, get timeout-minutes: 5 and one retry.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Lint job's R0801 duplication check takes ~3.5 min on pylint 4.0.9.
On the same tree, 4.1.3 takes ~15 s locally and reports the identical
result (10.00/10, no findings; a synthetic 14-line duplicate is still
flagged). pylint's other requirements are unchanged, so only the two
lock entries move; URLs, hashes and sizes are taken from PyPI metadata
(precedent #3563).

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 07:59
Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Workflow contract tests are incorrectly marked as unit tests despite filesystem access, and the fallback URL assertion can pass without verifying the active URL’s components.

2 open issues
What changed in this PR

This PR implements Phase 1 of #3581 to shorten APM’s CI feedback cycle and reduce reruns, without changing required-check names or test selections.

Changes:

  • Runs Windows compatibility tests and Linux unit shards on four workers, and cancels superseded PR runs.
  • Bounds apt operations and adds time-limited release artifact upload retries.
  • Updates pylint and astroid, with regression contracts for concurrency, parallelism, and apt failover.
File Description
uv.lock Updates pylint to 4.1.3 and astroid to 4.3.4.
tests/​unit/​test_workflow_concurrency_contract.py Adds contracts for PR-only cancellation and concurrency groups.
tests/​unit/​test_windows_compat_gate_workflow.py Verifies four-worker scheduling while preserving test selection.
tests/​unit/​scripts/​test_ci_apt_install.py Tests healthy installs, timeout failover, and bounded failure.
scripts/​ci-apt-install.sh Bounds apt calls and attempts package recovery before failover.
CHANGELOG.md Records the CI performance changes.
.github/​workflows/​spec-conformance.yml Adds concurrency cancellation for superseded PR runs.
.github/​workflows/​docs.yml Limits in-progress cancellation to PR builds.
.github/​workflows/​codeql.yml Adds PR-scoped cancellation and unique non-PR groups.
.github/​workflows/​ci.yml Adds concurrency controls, four-worker tests, and an apt step timeout.
.github/​workflows/​ci-integration.yml Caps the shell-runtime installation step at five minutes.
.github/​workflows/​build-release.yml Adds bounded artifact upload retries and an apt step timeout.

Review effort: Balanced

Under xdist the Windows gate failed twice, deterministically, on
test_fresh_clone_audit_preserves_identity_across_cold_repeated_and_warm_runs[git]:
"Unable to prepare Git clone configuration probe". xdist nests each
worker's tmp_path one level deeper (popen-gwN), which pushes the Git
cache probe past Windows MAX_PATH; serial runs pass with about 10
characters to spare. Use a short --basetemp on C:, where 8.3 aliases
are enabled.

Measured -n 4 on the Linux unit shards in run 43179405432: per-worker
collection added 40-60 s, execution gained 0-25 s, and both shards got
slower than -n 2. Revert to -n 2 and record the measurement.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel Daniel Meppiel (danielmeppiel) changed the title ci: Phase 1 speed-ups (parallel Windows gate, PR concurrency cancel, unit shard cores, bounded apt, pylint 4.1.3) ci: Phase 1 speed-ups (parallel Windows gate, PR concurrency cancel, bounded apt/uploads, pylint 4.1.3) Nov 12, 2026
Concurrency contracts read workflow YAML from disk, so they are
component tests. The apt failover test now compares the active URIs
entry by parsed scheme, host and path instead of a substring.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Fold the remaining CI assessment options into this PR (maintainer
decision on #3581):

- Option 7a: the architecture mutation files no longer pin all their
  tests to one xdist worker. The owner and semantic baselines are
  computed once per session and shared across workers through a file
  lock keyed by a source hash (tests/utils/xdist_shared.py).
- Option 7b: tests/integration/test_architecture_* are classified
  architecture_contract at collection. They run once on Linux x86_64
  (release, merge queue, and path-gated PR shards); arm64 and macOS
  lanes deselect them.
- Option 8: scripts/ci_sharding.py balances whole xdist groups across
  shards by measured duration, plans the shard count from a target
  per-shard wall time, and records durations so the nightly run can
  publish fresh ones. Lifecycle, unit, merge-queue and release lanes
  use it; scripts/lifecycle_smoke_sharding.py delegates to it.
- Option 10a/10b/10c: build-release.yml starts with a release plan.
  Pushes to main build and unit-test; schedule/dispatch runs validate
  everything and become promotion evidence; a tag promotes a SHA with
  evidence (build, smoke, release validation) and fails closed
  without it, unless vars.APM_RELEASE_FULL_VALIDATION_TAG names it.
  Builds are per platform, so each platform's integration starts as
  soon as its own binary exists. macOS Intel leaves the push path.
- macOS capacity: still exactly one Intel and one ARM job per run, now
  running the platform-sensitive subset (lifecycle_smoke,
  windows_compat, platform_sensitive) instead of the full corpus.
- Option 5b: network-dependent tests run in their own lane that
  retries only failed tests (at most twice).
- Option 9: the merge queue builds once, plans duration-balanced
  hermetic shards, adds the network lane and the full Windows unit
  suite (three shards), and runs release validation in parallel;
  merge-gate requires "Windows Unit Tests" on merge_group.
- Option 4: PR unit, lifecycle and architecture shards run on
  ubuntu-24.04-arm (A/B on this PR's runs).

TEMP-REHEARSAL triggers let two rehearsal branches exercise the full,
promoted and merge-queue paths without a tag; they are removed before
merge.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ps it

upload-artifact skips hidden paths by default, so .shard-plan/*.json
matched nothing and the plan job failed after a correct plan.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Equal-weight sharding scattered each module's tests across both unit
shards, exposing pre-existing cross-file state leaks
(test_mcp_registry_module, test_console_utils) and raising every shard's
coverage footprint. Shards now keep each module together, as the previous
pytest-split chunking did, and split only modules heavier than half a
shard's share.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- .github/ci-durations/unit.json seeds unit shard balance from this
  PR's arm64 run 43207447758 (count-balanced shards measured 232 s vs
  346 s of worker time). Seed files keep tests slower than 0.5 s and
  fold the rest into module totals (26,496 entries -> 1,082, 71 KB).
- macOS Intel no longer repeats the architecture-agnostic Python unit
  suite (macOS arm64 runs it) and runs the filesystem/shell/path slice
  (windows_compat or platform_sensitive) instead of the lifecycle
  family, which arm64 covers. Still one job per macOS architecture.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Update the integration-testing guide and the CI/CD instructions (source
in .apm/, deployed copy and lockfile hashes refreshed; apm audit --ci
passes) for the per-platform release fan-out, promotion evidence, the
network lane, the merge-queue Windows unit shards and the reduced
macOS subsets.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- .github/ci-durations/linux-{x86_64,arm64}.json come from rehearsal
  run 43209892067's measured shard durations (11,505 and 10,072 tests,
  compacted). With them the planner splits the owner-rule module whose
  whole-module placement timed out merge-queue shard 1 in run
  43209891804 (count-weighted bootstrap).
- The shard-count planner bounds a shard's wall by its longest serial
  unit (an xdist group or one test), not by a whole module, whose tests
  spread across the shard's workers.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run 43213842057 split the owner-rule module test by test across all
seven x86_64 shards, so every worker repeated its warm-up and shards
took 398-627 s against equal estimates. Heavy modules are now cut into
the fewest in-order chunks of at most a quarter of one shard's share,
and the x86_64 seed averages runs 43209892067 and 43213842057.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Merge-queue shards run with --cov, which costs about 1.5x the release
lanes (18,407 vs 12,083 worker-seconds), so shards planned from release
durations ranged 9.3-13.6 min in run 43217427126. The merge queue now
plans from .github/ci-durations/merge-queue.json, measured in that run,
with up to 12 shards (11 at the 420 s target).

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seeds .github/ci-durations/merge-queue.json from rehearsal runs
43217427126 and 43224024360 (18,407 and 18,638 worker-seconds).

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The rehearsal branches exercised the full release path (runs
43209892067, 43213842057, 43217428603, 43224024332), the merge-queue
lanes (43213841924, 43217427126, 43224024360), promotion with evidence
(43222627589) and fail-closed promotion without it (43217158518). Pin
the triggers so a rehearsal branch cannot linger: build-release pushes
from main and v* tags only, ci-integration runs on merge_group only.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…e1-speedups

# Conflicts:
#	.github/workflows/build-release.yml
#	uv.lock
@danielmeppiel Daniel Meppiel (danielmeppiel) changed the title ci: Phase 1 speed-ups (parallel Windows gate, PR concurrency cancel, bounded apt/uploads, pylint 4.1.3) ci: faster PR, merge-queue and release CI (per-platform fan-out, tag promotion, duration sharding) Nov 12, 2026
@danielmeppiel

Daniel Meppiel (danielmeppiel) commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

Extracted into #3771 (named release fix, approved 2026-11-17), with no rebase of this branch:

  • scripts/ci_sharding.py, scripts/ci-shard-durations.sh, .github/ci-durations/linux-{x86_64,arm64}.json, and the integration-plan, sharded integration-linux-{x86,arm}, integration-network and Integration Tests Summary jobs in build-release.yml
  • the architecture_contract marker, the shared architecture baselines (tests/utils/xdist_shared.py, plus dropping xdist_group from the architecture mutation modules), the platform_sensitive markers, and the PYTEST_RETRY_FAILED / python -m pytest support in scripts/test-integration.sh
  • slim macOS (one job per architecture) and the per-platform build-linux-x86 / build-linux-arm / build-windows jobs
  • scripts/release_plan.py (push/full/promoted). Two changes: only a green schedule or dispatch full run counts as evidence (main push plus merge queue does not), and a tag without evidence validates fully in its own run before publishing (no fail-closed refusal, no override variable), so releasing stays a plain tag push.
  • the contract tests: test_platform_contract_workflow.py, test_release_plan.py, test_ci_sharding.py, test_xdist_shared.py, and the job-ID updates in the signing, diagnostics and ADO-live contracts

Still only in this PR: the ci.yml PR-time changes (Windows gate -n 4, arm64 unit/lifecycle shards, Architecture Contracts job, concurrency), the merge-queue changes in ci-integration.yml and merge-gate.yml (including Windows Unit Tests), lifecycle_smoke_sharding.py on ci_sharding, the ci-apt-install.sh time-boxing, the merge-queue.json and unit.json seeds, and the pylint bump. After #3771 merges, rebasing here should drop the extracted files.

Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 17, 2026
…gs (#3771)

* ci(release): shard Linux integration, slim macOS, promote verified tags

Extracts the three release-CI wins from #3582 for #3581:

- Linux x86_64/arm64 integration is split across runners by measured
  duration (scripts/ci_sharding.py, .github/ci-durations seeds,
  integration-plan job), keeping xdist groups and heavy modules whole.
  Architecture contracts run once on x86_64 and share their baselines
  across workers; network-dependent tests get their own retrying lane.
- macOS keeps one job per architecture: arm64 runs units plus
  lifecycle_smoke/windows_compat/platform_sensitive, Intel runs
  windows_compat/platform_sensitive.
- scripts/release_plan.py picks push/full/promoted. A tag promotes only
  when its SHA has a green schedule/repository_dispatch run on main, and
  fails closed otherwise; the override is the repository variable
  APM_RELEASE_FULL_VALIDATION_TAG. create-release requires a pushed tag
  and every release-bound lane; only promoted mode may skip integration.

PR-time and merge-queue changes from #3582 are out of scope.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(changelog): note release CI speed-up

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* ci(release): keep "push a tag" as the release trigger

A v* tag without promotion evidence now runs full validation in the tag
run and then publishes, instead of failing at Release Plan. A tag whose
SHA has a green schedule/repository_dispatch run on main is still
promoted without re-running the suites. A failed evidence lookup falls
back to full validation. The APM_RELEASE_FULL_VALIDATION_TAG override is
no longer needed and is removed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* chore(apm): refresh lockfile hash for cicd instructions

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* test(ci): pin the integration summary's every-lane gate and fallback summary

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* ci(release): opt the network lane in; match grouped durations either way

- The network lane sets APM_RUN_INTEGRATION_TESTS=1, so its
  requires_network_integration tests run instead of skipping.
- ci_sharding resolves a grouped test's duration with or without the
  @<group> suffix that xdist loadgroup adds, so the planner (plain
  collection) and the shards agree on measured weights.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 18, 2026
Build on the duration-aware sharding introduced by the release slice of #3581/#3582. Keep full release validation and scan reporting intact.

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Copy link
Copy Markdown
Collaborator Author

Docs re-sweep (#3800) coordination: one fix overlaps this PR's hunk in docs/src/content/docs/contributing/integration-testing.md, so I'm leaving it to this PR:

- - E2E tests only run on releases (minimizing API usage)
+ - E2E tests run in the merge queue and promotion runs, not on every PR push (minimizing API usage)

On main, E2E runs in merge_group (ci-integration.yml), and release tags promote without re-running it. The nearby Speed vs Confidence bullets are already corrected in #3850, outside your hunk.

@danielmeppiel

Copy link
Copy Markdown
Collaborator Author

Update: #3849 and #3850 are folded into #3848. The coordination notes above still apply. Where they reference #3849 or #3850, read #3848.

@danielmeppiel

Copy link
Copy Markdown
Collaborator Author

Closing as superseded. This PR now conflicts with main, and its goal has landed in smaller PRs that each had their own hosted evidence. Thank you: most of what shipped started here.

What this PR contributed, and where it landed

Contribution from #3582 Landed in
scripts/ci_sharding.py (duration-weighted, keeps xdist groups whole), .github/ci-durations/ seeds, scripts/ci-shard-durations.sh, nightly duration publishing #3771
Linux x86_64/arm64 integration sharding, one macOS job per architecture, tag promotion via scripts/release_plan.py #3771
Parallel Windows push validation, plus the warning that packed hooks are not xdist-safe in spec conformance (led to 3 serial spec shards) #3830
Python CodeQL on the standard ARM runner #3834
Parallel Windows Compatibility Gate on PRs. The -n 4 run (1.8x speedup, extra collection time, needed a MAX_PATH --basetemp workaround) is why #3863 splits across runners and keeps tests serial within each shard #3863
Duration-balanced Lifecycle Smoke and Build & Test shards on PRs #3863

Result: first-attempt green PR gate went from a median of 11:54 (p90 13:16) to a median of 7:20 (p90 8:05). That's 14 runs since #3863 merged on 2026-11-18.

Not landed from this PR. Each is tracked or dropped in #3581:

  • PR-only concurrency cancellation (item 2): still open.
  • pylint 4.1.3 bump (item 5): still open. Lint is not one of the checks gate waits on, so this speeds up author feedback, not time-to-gate.
  • apt time-boxing (item 4): dropped. The last 60 PR CI runs had 0 reruns, and 123 apt steps took a median of 11s (max 26s) with no failures.
  • Sharing architecture-mutation baselines across xdist workers (7a) and merge-queue sharding: no merge-queue runs since 2026-09-09, so not on any path today.

Written by automation (CI speedups session) on Daniel Meppiel (@danielmeppiel)'s direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: speed up PR, merge-queue and release CI (Phase 1 + assessment Options 4-10)

2 participants