Skip to content

ci: speed up PR, merge-queue and release CI (Phase 1 + assessment Options 4-10) #3581

Closed

Description

Scope

Phase 1 ("this week") of the CI performance assessment: workflow-level, reversible changes that shorten PR time-to-gate and remove the top rerun cause. No test is added to or removed from any required selection.

  1. Windows Compatibility Gate in parallel. Run the windows_compat family with -n 4 --dist loadgroup (ci.yml). It runs 913 tests serially today on a 4-vCPU runner and is the PR critical path. loadgroup keeps the existing xdist_group affinity for HOME-mutating integration tests.
  2. Supersede stale PR runs. Add concurrency with cancel-in-progress limited to pull_request to ci.yml, spec-conformance.yml and codeql.yml, and make docs.yml cancel only PR build runs. Non-PR events (merge_group, push to main, tags, schedule, dispatch) get a unique per-run group, so they are never cancelled or queued. merge-gate.yml is unchanged.
  3. Linux unit shards use all 4 vCPUs (-n 2 -> -n 4). Coverage floors (60% per shard, 80% combined) are unchanged.
  4. Bounded apt install. scripts/ci-apt-install.sh time-boxes each apt-get call (60 s) so a stalled mirror falls over to the existing fix(ci): bypass unresolvable Azure apt mirror when installing fish and zsh #3570 fallback mirror instead of hanging. The step gets timeout-minutes: 5. Release binary uploads in build-release.yml get timeout-minutes: 5 plus one retry.
  5. pylint 4.0.9 -> 4.1.3 (astroid 4.0.4 -> 4.3.4) in uv.lock, hand-edited from PyPI metadata (precedent fix(deps): lock transitive setuptools 84.1.0 (GHSA-xmj3-9gf7-h52w) #3563). The dependency closure is unchanged apart from these two entries.
  6. Update the CI contract tests that pin these flags, and add regression tests for the new concurrency and apt-timeout behavior.

Evidence (baseline, from the assessment)

  • PR time-to-gate: p50 9.5 min, p90 10.4 min (44 first-try ci.yml PR runs, 42906426462..43090781426). Windows Compatibility Gate p50 9.0 min; Build & Test shard p50 up to 5.0 min (fin); Lint p50 4.1 min.
  • Windows gate run 43090781426 / job 130766958026: 913 selected, 901 passed in 370 s serially. No test over 15 s.
  • Lint: the pylint R0801 step is 206 s p50. Locally on the same tree, pylint 4.0.9 takes 178 s and 4.1.3 takes 24.5 s, both 10.00/10.
  • 57 of 241 ci.yml PR runs (2026-10-28..11-11) were superseded while still running. ci.yml has no concurrency block.
  • 49 of 161 green PR runs needed a rerun. In 8 of 12 sampled ones, both Lifecycle shards failed in "Install Unix shell startup test runtimes" after about 7.5 min (jobs 130638784255, 130625494893, 130607581342, 130591026938). A healthy install takes about 10 s.
  • v0.35.1 tag run 43092150223: macOS x86 failed at "Upload binary as workflow artifact" after an 11.5 min hang. Normal uploads take under 11 s.

Done when

  • One PR lands items 1-6, and its CI passes, including the Windows gate selecting the same 913 tests.
  • The PR records measured time-to-gate and the durations of the Windows gate, Build & Test shards and Lint, compared with the baseline above.
  • CI contract tests pin the new flags and the PR-only cancellation semantics.

Out of scope / follow-ups

  • Spec conformance -n 4: not xdist-safe yet (test_packed_hook_resolution_reqs fails under -n 4 locally). Follow-up.
  • .deb caching for apt, moving Linux PR lanes to ubuntu-24.04-arm, path-owned test families, durations-based sharding, the architecture long-pole fix, merge-queue changes and release restructuring (assessment Phases 2-3).
  • No change to required-check names, EXPECTED_CHECKS, or the branch ruleset.

Human direction and review contact

Review contact: Daniel Meppiel (@danielmeppiel).

In the CI-assessment session, the maintainer asked to "present the CI speed up work ... with its issue and pr open", with maximum speed and throughput as the priority. This issue is an automation-written record of that direction, not a human-authored GitHub approval comment.

Activity

  1. danielmeppiel commented on Nov 12, 2026

    @danielmeppiel
    CollaboratorAuthor

    Scope update from #3582 (automation-written; review contact Daniel Meppiel (@danielmeppiel)):

    • Item 3 (Linux unit shards -n 2 -> -n 4) was measured and dropped. In run 43179405432, per-worker collection of about 13k tests rose from 51-73 s to 87-113 s, while execution varied more between runs (97-254 s) than between settings. Both shards were net slower. ci.yml keeps -n 2 with a comment recording this. The unit-shard speed-up moves to balanced, durations-based sharding (assessment Option 8).
    • Added to item 1: the Windows gate uses --basetemp=C:\apm-pt. xdist's extra popen-gwN path level pushed one Git clone probe in test_audit_workspace_identity_lifecycle[git] past Windows MAX_PATH; it failed deterministically twice. Follow-up: git_clone_env probes are not long-path-safe on Windows.
    • Measured on the final head: time-to-gate went from 9.5 to 7.1 min, the Windows gate from 9.0 to 5.65 min, and Lint from 4.1 to 1.0 min (CI 43181971921, Merge Gate 43181971591).
  2. danielmeppiel commented on Nov 12, 2026

    @danielmeppiel
    CollaboratorAuthor

    Scope update: remaining assessment options folded into #3582

    Maintainer decision (Daniel Meppiel (@danielmeppiel), CI session): "those changes must be folded into the current PR (CI improvements) and I want all of them tested, not just the quick ones. Fanning out to more macos runners has the danger of capacity on github, because sometimes they're not available." This comment was written by automation to record that direction. It is not a human-authored approval comment.

    Added scope

    Option Change
    7a Architecture mutation tests are no longer pinned to a single xdist worker. Their baselines are computed once per session and shared between workers through a file lock keyed by a source hash.
    7b tests/integration/test_architecture_* get the architecture_contract marker. They run once on Linux x86_64 (release, merge queue, and PR runs that touch relevant paths), and arm64/macOS lanes deselect them.
    8 scripts/ci_sharding.py shards by measured duration and keeps each xdist group on one shard. It plans the shard count dynamically and stores durations; nightly publishes them, and a committed seed is the fallback. Target: slowest shard / fastest shard < 1.3.
    10a/10b/10c/10d A release plan job decides each run's mode. Tags promote a SHA that already has evidence (a green nightly or dispatch run, or a green main push plus a green merge-queue run) and fail closed without it. The manual override is vars.APM_RELEASE_FULL_VALIDATION_TAG. Release jobs fan out per platform, Linux integration is sharded, and macOS Intel no longer runs on push.
    macOS Each run keeps one Intel and one ARM job, never more. Both run only the platform-sensitive subset, not the full corpus.
    4 PR unit and lifecycle shards move to ubuntu-24.04-arm, with an A/B comparison on #3582's runs.
    5b A separate network lane runs requires_github_token / requires_network_integration tests and retries only failed tests, at most twice.
    9 The merge queue builds once and runs sharded hermetic integration, the network lane, and the full Windows unit suite in 3 shards. EXPECTED_CHECKS for merge_group adds Windows Unit Tests. No ruleset or settings changes.

    Testing: temporary push triggers on the rehearsal branches danielmeppiel-ci-release-rehearsal (full release path plus merge-queue lanes) and danielmeppiel-ci-promote-rehearsal (promoted-tag path). Publishing jobs require a pushed tag, so they cannot run on these branches. The triggers are removed before final review.

    Out of scope / follow-up: Option 6 (path-owned families, plan.py shared with preflight) depends on #3580.

  3. changed the title [-]ci: Phase 1 speed-ups (Windows gate parallelism, concurrency cancel, unit shard cores, rerun causes, pylint bump)[/-] [+]ci: speed up PR, merge-queue and release CI (Phase 1 + assessment Options 4-10)[/+] on Nov 12, 2026
  4. 7 remaining items

  5. danielmeppiel commented on Nov 17, 2026

    @danielmeppiel
    CollaboratorAuthor

    Decision: approve
    Area: project
    Scope: Release CI only, extracted from #3582 into one PR touching build-release.yml: (1) Linux x86_64 and arm64 integration sharded across runners by measured duration (scripts/ci_sharding.py, .github/ci-durations seeds, integration-plan job) keeping xdist groups and modules whole; (2) slim macOS, one job per architecture, arm64 runs units plus lifecycle_smoke or windows_compat or platform_sensitive, Intel runs windows_compat or platform_sensitive; (3) tag promotion via scripts/release_plan.py push/full/promoted modes that fails closed without green full-validation evidence, override repo variable APM_RELEASE_FULL_VALIDATION_TAG, no workflow_dispatch.
    Done when: Lanes partition the non-live corpus (marker-algebra contract), at most two macOS jobs, test_release_plan.py and test_ci_sharding.py pass, create-release requires a pushed tag and every release-bound lane and skips integration only in promoted mode, existing check names kept, CI green, and a post-merge manual-build-release run is green.
    Out of scope: PR-time ci.yml changes, merge-queue ci-integration.yml and merge-gate.yml changes, Windows gate parallelism, arm64 PR runners, ruleset changes, and the rest of #3582.
    Review contact: Daniel Meppiel (@danielmeppiel)

  6. danielmeppiel commented on Nov 17, 2026

    @danielmeppiel
    CollaboratorAuthor

    Context for the scope record above (posted by the CI speed captain session on Daniel Meppiel (@danielmeppiel)'s instruction, 2026-11-17 15:58 CET, URGENT named release fix that lands through the Lane L freeze before v0.36.0).

    The record covers only the three release-CI wins from #3582: duration-sharded Linux integration, slim macOS (one job per architecture), and fail-closed tag promotion. #3582 stays open for the PR-time and merge-queue work.

    One deliberate tightening versus #3582: promotion evidence is only a green schedule or repository_dispatch run of build-release.yml on main for the tag's exact SHA. A green main push plus a green merge-queue run does not count, because the current merge queue runs Linux x86_64 integration only.

    Release flow after merge: dispatch manual-build-release on the main SHA, wait for it to finish green, then push the tag. Without that evidence the tag run fails at Release Plan; the one-tag override is the repository variable APM_RELEASE_FULL_VALIDATION_TAG.

  7. danielmeppiel commented on Nov 17, 2026

    @danielmeppiel
    CollaboratorAuthor

    Decision: approve
    Area: project
    Scope: Release CI only, extracted from #3582 into #3771: (1) Linux x86_64 and arm64 integration sharded across runners by measured duration (scripts/ci_sharding.py, .github/ci-durations seeds, integration-plan job) keeping xdist groups and modules whole; (2) slim macOS, one job per architecture, arm64 runs units plus lifecycle_smoke or windows_compat or platform_sensitive, Intel runs windows_compat or platform_sensitive; (3) tag promotion via scripts/release_plan.py push/full/promoted modes: releasing stays "push a v* tag"; a tag whose SHA has a green schedule or repository_dispatch run on main is promoted (build, smoke, release validation, publish), any other tag runs full validation in the tag run before publishing; no workflow_dispatch.
    Done when: Lanes partition the non-live corpus (marker-algebra contract), at most two macOS jobs, test_release_plan.py and test_ci_sharding.py pass, create-release requires a pushed tag and every release-bound lane and skips integration only in promoted mode, nothing publishes without a green full validation of the exact SHA, existing check names kept, CI green, and a post-merge manual-build-release run is green.
    Out of scope: PR-time ci.yml changes, merge-queue ci-integration.yml and merge-gate.yml changes, Windows gate parallelism, arm64 PR runners, ruleset changes, and the rest of #3582.
    Review contact: Daniel Meppiel (@danielmeppiel)

  8. danielmeppiel commented on Nov 17, 2026

    @danielmeppiel
    CollaboratorAuthor

    Context: the record above supersedes issuecomment-6952677210 for #3771. On Daniel Meppiel (@danielmeppiel)'s direction, releasing stays a plain v* tag push. A tag without promotion evidence now validates fully in its own run before publishing, instead of failing closed, so the APM_RELEASE_FULL_VALIDATION_TAG override is gone. The publication invariant still holds: nothing publishes without a green full validation of the exact SHA.

  9. danielmeppiel commented on Nov 19, 2026

    @danielmeppiel
    CollaboratorAuthor

    Reconciliation, 2026-11-19

    The PR-time goal of this issue has landed. One small item remains, so this stays open. #3582 is closed as superseded.

    Measured time-to-gate (first-attempt green PR runs):

    p50 p90
    Baseline in this issue (Oct–Nov) 9.5 min 10.4 min
    Just before this round (corpus growth) 11:54 13:16
    After #3863 (14 runs since 2026-11-18 09:28Z) 7:20 8:05

    Single-run pair on the same day: PR #3848 took 12:29 (Merge Gate 44265921414). #3863 took 6:54 (Merge Gate 44274663541). Per check: Windows Compatibility Gate 12:18 -> 5:53, Lifecycle Smoke 8:49 -> 6:26, slowest Build & Test shard 7:36 -> 6:39.

    Scope items

    Item Status
    1. Parallel Windows Compatibility Gate Done in #3863. Three serial, duration-balanced shards on parallel windows-latest runners, behind a fan-in that keeps the required name. Every test ID runs exactly once: hosted shards selected 414 + 288 + 368 = 1,070, the same as the control selection. Done on runners rather than with -n 4, so no MAX_PATH --basetemp workaround is needed.
    2. PR-only concurrency cancellation Remaining. This frees runners but does not shorten any single run.
    3. Linux unit shards -n 4 Dropped earlier (measured net slower). Shards are now balanced by measured durations instead (#3863).
    4. Bounded apt install Dropped on evidence. The last 60 PR CI runs had 0 reruns, and 123 apt steps took a median of 11s (max 26s) with no failures. The mirror fallback and per-request timeouts already cover the 2026-11 failure mode.
    5. pylint 4.0.9 -> 4.1.3 Remaining. Lint is not one of the checks gate waits on, so this shortens author feedback (pylint step about 4.7 min), not time-to-gate. It needs a hand-edited uv.lock.
    6. Contract tests Done for everything landed: tests/unit/test_windows_compat_gate_workflow.py, tests/quality/test_ci_topology.py, tests/unit/scripts/test_ci_sharding.py.
    Added scope 7b, 8, 10a–d Done in #3771.
    Added scope 7a, merge-queue sharding Not on any current path: no merge_group runs since 2026-09-09. Reopen if the merge queue comes back.

    Done when: items 2 and 5 land with their contract tests, or the maintainer drops them. Neither changes time-to-gate.

    Written by automation (CI speedups session) on Daniel Meppiel (@danielmeppiel)'s direction. This is not a human approval comment.

  10. danielmeppiel commented on Nov 19, 2026

    @danielmeppiel
    CollaboratorAuthor

    Closing. #3889 has merged at 6683116, which completes this issue's final scope.

    • Item 5 (pylint 4.0.9 -> 4.1.3, astroid 4.3.4): done in build(deps): bump pylint 4.0.9 -> 4.1.3 (astroid 4.3.4) #3889. The hosted Code duplication guardrail (pylint R0801) step went from 282 s (#3848 Lint) to 20 s. R0801 findings are unchanged: 0 at the CI threshold, and the same 58 duplicate blocks with identical file:line ranges at --min-similarity-lines=6.
    • Item 2 (PR-only concurrency cancellation): dropped deliberately on the maintainer's direction. It saves runner minutes, not developer time, and would need a hosted rehearsal.
    • Outcome for time-to-gate: median 11:54 / p90 13:16 before this round, 7:20 / 8:05 after ci: shard the Windows gate and balance PR shards by measured duration #3863 (see the reconciliation comment above). Everything else is accounted for there.

    Written by automation (CI speedups session) on Daniel Meppiel (@danielmeppiel)'s direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pillar/foundationsPillar: reliability, platforms, CI, code healthstatus/acceptedHuman scope approval; verify the issue's approval record and review contact before work.

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions