Repository navigation
Closed
Labels
pillar/foundationsPillar: reliability, platforms, CI, code healthPillar: reliability, platforms, CI, code healthstatus/acceptedHuman scope approval; verify the issue's approval record and review contact before work.Human scope approval; verify the issue's approval record and review contact before work.
Milestone
Description
Scope
Phase 1 ("this week") of the CI performance assessment: workflow-level, reversible changes that shorten PR time-to-gate and remove the top rerun cause. No test is added to or removed from any required selection.
- Windows Compatibility Gate in parallel. Run the
windows_compatfamily with-n 4 --dist loadgroup(ci.yml). It runs 913 tests serially today on a 4-vCPU runner and is the PR critical path.loadgroupkeeps the existingxdist_groupaffinity for HOME-mutating integration tests. - Supersede stale PR runs. Add
concurrencywithcancel-in-progresslimited topull_requestto ci.yml, spec-conformance.yml and codeql.yml, and make docs.yml cancel only PR build runs. Non-PR events (merge_group, push to main, tags, schedule, dispatch) get a unique per-run group, so they are never cancelled or queued. merge-gate.yml is unchanged. - Linux unit shards use all 4 vCPUs (
-n 2->-n 4). Coverage floors (60% per shard, 80% combined) are unchanged. - Bounded apt install.
scripts/ci-apt-install.shtime-boxes each apt-get call (60 s) so a stalled mirror falls over to the existing fix(ci): bypass unresolvable Azure apt mirror when installing fish and zsh #3570 fallback mirror instead of hanging. The step getstimeout-minutes: 5. Release binary uploads in build-release.yml gettimeout-minutes: 5plus one retry. - pylint 4.0.9 -> 4.1.3 (astroid 4.0.4 -> 4.3.4) in
uv.lock, hand-edited from PyPI metadata (precedent fix(deps): lock transitive setuptools 84.1.0 (GHSA-xmj3-9gf7-h52w) #3563). The dependency closure is unchanged apart from these two entries. - Update the CI contract tests that pin these flags, and add regression tests for the new concurrency and apt-timeout behavior.
Evidence (baseline, from the assessment)
- PR time-to-
gate: p50 9.5 min, p90 10.4 min (44 first-try ci.yml PR runs, 42906426462..43090781426). Windows Compatibility Gate p50 9.0 min; Build & Test shard p50 up to 5.0 min (fin); Lint p50 4.1 min. - Windows gate run 43090781426 / job 130766958026:
913 selected, 901 passed in 370 s serially. No test over 15 s. - Lint: the pylint R0801 step is 206 s p50. Locally on the same tree, pylint 4.0.9 takes 178 s and 4.1.3 takes 24.5 s, both 10.00/10.
- 57 of 241 ci.yml PR runs (2026-10-28..11-11) were superseded while still running. ci.yml has no
concurrencyblock. - 49 of 161 green PR runs needed a rerun. In 8 of 12 sampled ones, both Lifecycle shards failed in "Install Unix shell startup test runtimes" after about 7.5 min (jobs 130638784255, 130625494893, 130607581342, 130591026938). A healthy install takes about 10 s.
- v0.35.1 tag run 43092150223: macOS x86 failed at "Upload binary as workflow artifact" after an 11.5 min hang. Normal uploads take under 11 s.
Done when
- One PR lands items 1-6, and its CI passes, including the Windows gate selecting the same 913 tests.
- The PR records measured time-to-
gateand the durations of the Windows gate, Build & Test shards and Lint, compared with the baseline above. - CI contract tests pin the new flags and the PR-only cancellation semantics.
Out of scope / follow-ups
- Spec conformance
-n 4: not xdist-safe yet (test_packed_hook_resolution_reqsfails under-n 4locally). Follow-up. .debcaching for apt, moving Linux PR lanes toubuntu-24.04-arm, path-owned test families, durations-based sharding, the architecture long-pole fix, merge-queue changes and release restructuring (assessment Phases 2-3).- No change to required-check names,
EXPECTED_CHECKS, or the branch ruleset.
Human direction and review contact
Review contact: Daniel Meppiel (@danielmeppiel).
In the CI-assessment session, the maintainer asked to "present the CI speed up work ... with its issue and pr open", with maximum speed and throughput as the priority. This issue is an automation-written record of that direction, not a human-authored GitHub approval comment.
Metadata
Metadata
Assignees
Labels
pillar/foundationsPillar: reliability, platforms, CI, code healthPillar: reliability, platforms, CI, code healthstatus/acceptedHuman scope approval; verify the issue's approval record and review contact before work.Human scope approval; verify the issue's approval record and review contact before work.
Type
Projects
- StatusShow more project fieldsTodo
Activity
docs(changelog): note Phase 1 CI speed-ups
ci: keep Windows gate under MAX_PATH; restore unit shards to -n 2
danielmeppiel commentedon Nov 12, 2026
Scope update from #3582 (automation-written; review contact Daniel Meppiel (@danielmeppiel)):
-n 2->-n 4) was measured and dropped. In run 43179405432, per-worker collection of about 13k tests rose from 51-73 s to 87-113 s, while execution varied more between runs (97-254 s) than between settings. Both shards were net slower. ci.yml keeps-n 2with a comment recording this. The unit-shard speed-up moves to balanced, durations-based sharding (assessment Option 8).--basetemp=C:\apm-pt. xdist's extrapopen-gwNpath level pushed one Git clone probe intest_audit_workspace_identity_lifecycle[git]past Windows MAX_PATH; it failed deterministically twice. Follow-up:git_clone_envprobes are not long-path-safe on Windows.gatewent from 9.5 to 7.1 min, the Windows gate from 9.0 to 5.65 min, and Lint from 4.1 to 1.0 min (CI 43181971921, Merge Gate 43181971591).test: classify workflow contracts as component; parse apt fallback URL
ci: per-platform release fan-out, duration sharding, tag promotion
danielmeppiel commentedon Nov 12, 2026
Scope update: remaining assessment options folded into #3582
Maintainer decision (Daniel Meppiel (@danielmeppiel), CI session): "those changes must be folded into the current PR (CI improvements) and I want all of them tested, not just the quick ones. Fanning out to more macos runners has the danger of capacity on github, because sometimes they're not available." This comment was written by automation to record that direction. It is not a human-authored approval comment.
Added scope
tests/integration/test_architecture_*get thearchitecture_contractmarker. They run once on Linux x86_64 (release, merge queue, and PR runs that touch relevant paths), and arm64/macOS lanes deselect them.scripts/ci_sharding.pyshards by measured duration and keeps each xdist group on one shard. It plans the shard count dynamically and stores durations; nightly publishes them, and a committed seed is the fallback. Target: slowest shard / fastest shard < 1.3.vars.APM_RELEASE_FULL_VALIDATION_TAG. Release jobs fan out per platform, Linux integration is sharded, and macOS Intel no longer runs on push.ubuntu-24.04-arm, with an A/B comparison on #3582's runs.requires_github_token/requires_network_integrationtests and retries only failed tests, at most twice.EXPECTED_CHECKSfor merge_group addsWindows Unit Tests. No ruleset or settings changes.Testing: temporary
pushtriggers on the rehearsal branchesdanielmeppiel-ci-release-rehearsal(full release path plus merge-queue lanes) anddanielmeppiel-ci-promote-rehearsal(promoted-tag path). Publishing jobs require a pushed tag, so they cannot run on these branches. The triggers are removed before final review.Out of scope / follow-up: Option 6 (path-owned families,
plan.pyshared with preflight) depends on #3580.[-]ci: Phase 1 speed-ups (Windows gate parallelism, concurrency cancel, unit shard cores, rerun causes, pylint bump)[/-][+]ci: speed up PR, merge-queue and release CI (Phase 1 + assessment Options 4-10)[/+]ci: keep the shard plan in a visible directory so upload-artifact kee…
ci: keep test modules whole across shards unless too heavy
ci: seed unit durations, compact duration files, trim macOS Intel work
docs: describe duration-sharded lanes, tag promotion and macOS scope
ci: seed integration durations; plan walls from serial units only
ci: cut heavy modules into contiguous chunks; refresh x86 durations
ci: give merge-queue shards their own coverage-inclusive durations
7 remaining items
danielmeppiel commentedon Nov 17, 2026
Decision: approve
Area: project
Scope: Release CI only, extracted from #3582 into one PR touching build-release.yml: (1) Linux x86_64 and arm64 integration sharded across runners by measured duration (scripts/ci_sharding.py, .github/ci-durations seeds, integration-plan job) keeping xdist groups and modules whole; (2) slim macOS, one job per architecture, arm64 runs units plus lifecycle_smoke or windows_compat or platform_sensitive, Intel runs windows_compat or platform_sensitive; (3) tag promotion via scripts/release_plan.py push/full/promoted modes that fails closed without green full-validation evidence, override repo variable APM_RELEASE_FULL_VALIDATION_TAG, no workflow_dispatch.
Done when: Lanes partition the non-live corpus (marker-algebra contract), at most two macOS jobs, test_release_plan.py and test_ci_sharding.py pass, create-release requires a pushed tag and every release-bound lane and skips integration only in promoted mode, existing check names kept, CI green, and a post-merge manual-build-release run is green.
Out of scope: PR-time ci.yml changes, merge-queue ci-integration.yml and merge-gate.yml changes, Windows gate parallelism, arm64 PR runners, ruleset changes, and the rest of #3582.
Review contact: Daniel Meppiel (@danielmeppiel)
danielmeppiel commentedon Nov 17, 2026
Context for the scope record above (posted by the CI speed captain session on Daniel Meppiel (@danielmeppiel)'s instruction, 2026-11-17 15:58 CET, URGENT named release fix that lands through the Lane L freeze before v0.36.0).
The record covers only the three release-CI wins from #3582: duration-sharded Linux integration, slim macOS (one job per architecture), and fail-closed tag promotion. #3582 stays open for the PR-time and merge-queue work.
One deliberate tightening versus #3582: promotion evidence is only a green
scheduleorrepository_dispatchrun ofbuild-release.ymlonmainfor the tag's exact SHA. A green main push plus a green merge-queue run does not count, because the current merge queue runs Linux x86_64 integration only.Release flow after merge: dispatch
manual-build-releaseon the main SHA, wait for it to finish green, then push the tag. Without that evidence the tag run fails atRelease Plan; the one-tag override is the repository variableAPM_RELEASE_FULL_VALIDATION_TAG.danielmeppiel commentedon Nov 17, 2026
Decision: approve
Area: project
Scope: Release CI only, extracted from #3582 into #3771: (1) Linux x86_64 and arm64 integration sharded across runners by measured duration (scripts/ci_sharding.py, .github/ci-durations seeds, integration-plan job) keeping xdist groups and modules whole; (2) slim macOS, one job per architecture, arm64 runs units plus lifecycle_smoke or windows_compat or platform_sensitive, Intel runs windows_compat or platform_sensitive; (3) tag promotion via scripts/release_plan.py push/full/promoted modes: releasing stays "push a v* tag"; a tag whose SHA has a green schedule or repository_dispatch run on main is promoted (build, smoke, release validation, publish), any other tag runs full validation in the tag run before publishing; no workflow_dispatch.
Done when: Lanes partition the non-live corpus (marker-algebra contract), at most two macOS jobs, test_release_plan.py and test_ci_sharding.py pass, create-release requires a pushed tag and every release-bound lane and skips integration only in promoted mode, nothing publishes without a green full validation of the exact SHA, existing check names kept, CI green, and a post-merge manual-build-release run is green.
Out of scope: PR-time ci.yml changes, merge-queue ci-integration.yml and merge-gate.yml changes, Windows gate parallelism, arm64 PR runners, ruleset changes, and the rest of #3582.
Review contact: Daniel Meppiel (@danielmeppiel)
danielmeppiel commentedon Nov 17, 2026
Context: the record above supersedes issuecomment-6952677210 for #3771. On Daniel Meppiel (@danielmeppiel)'s direction, releasing stays a plain
v*tag push. A tag without promotion evidence now validates fully in its own run before publishing, instead of failing closed, so theAPM_RELEASE_FULL_VALIDATION_TAGoverride is gone. The publication invariant still holds: nothing publishes without a green full validation of the exact SHA.ci(release): shard Linux integration, slim macOS, promote verified ta…
ci: parallelize Windows push validation and spec conformance (#3830)
ci: shard the Windows gate and balance PR shards by measured duration (…
danielmeppiel commentedon Nov 19, 2026
Reconciliation, 2026-11-19
The PR-time goal of this issue has landed. One small item remains, so this stays open. #3582 is closed as superseded.
Measured time-to-
gate(first-attempt green PR runs):Single-run pair on the same day: PR #3848 took 12:29 (Merge Gate 44265921414). #3863 took 6:54 (Merge Gate 44274663541). Per check: Windows Compatibility Gate 12:18 -> 5:53, Lifecycle Smoke 8:49 -> 6:26, slowest Build & Test shard 7:36 -> 6:39.
Scope items
windows-latestrunners, behind a fan-in that keeps the required name. Every test ID runs exactly once: hosted shards selected 414 + 288 + 368 = 1,070, the same as the control selection. Done on runners rather than with-n 4, so no MAX_PATH--basetempworkaround is needed.concurrencycancellation-n 4Lintis not one of the checksgatewaits on, so this shortens author feedback (pylint step about 4.7 min), not time-to-gate. It needs a hand-editeduv.lock.tests/unit/test_windows_compat_gate_workflow.py,tests/quality/test_ci_topology.py,tests/unit/scripts/test_ci_sharding.py.merge_groupruns since 2026-09-09. Reopen if the merge queue comes back.Done when: items 2 and 5 land with their contract tests, or the maintainer drops them. Neither changes time-to-
gate.Written by automation (CI speedups session) on Daniel Meppiel (@danielmeppiel)'s direction. This is not a human approval comment.
build(deps): bump pylint 4.0.9 -> 4.1.3 (astroid 4.3.4) (#3889)
danielmeppiel commentedon Nov 19, 2026
Closing. #3889 has merged at 6683116, which completes this issue's final scope.
Code duplication guardrail (pylint R0801)step went from 282 s (#3848 Lint) to 20 s. R0801 findings are unchanged: 0 at the CI threshold, and the same 58 duplicate blocks with identicalfile:lineranges at--min-similarity-lines=6.gate: median 11:54 / p90 13:16 before this round, 7:20 / 8:05 after ci: shard the Windows gate and balance PR shards by measured duration #3863 (see the reconciliation comment above). Everything else is accounted for there.Written by automation (CI speedups session) on Daniel Meppiel (@danielmeppiel)'s direction.