Skip to content

ci: shard the Windows gate and balance PR shards by measured duration - #3863

Merged
Daniel Meppiel (danielmeppiel) merged 2 commits into
mainfrom
danielmeppiel-ci-windows-gate-sharding
Nov 18, 2026
Merged

Daniel Meppiel (danielmeppiel) merged 2 commits into
mainfrom
danielmeppiel-ci-windows-gate-sharding

Conversation

@danielmeppiel

@danielmeppiel Daniel Meppiel (danielmeppiel) commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Halve PR time-to-gate by removing the three long poles on the required path.

First-attempt PR run gate created -> success
Before: PR #3848, CI 44265921274 / Merge Gate 44265921414 12:29
After: this PR, CI 44274663751 / Merge Gate 44274663541 6:54

Recent baseline across PRs: median 11:54, p90 13:16. These are single runs, not new medians.

Required check Before (#3848) After (this PR)
Windows Compatibility Gate 12:18 (one runner) 5:53 (shards 4:19 / 5:46 / 5:41 + fan-in)
Lifecycle Smoke (Linux) 8:49 (shards 8:41 / 6:15) 6:26 (shards 6:20 / 5:10 / 5:52 + fan-in)
Build & Test, slowest shard 7:36 (pytest 442 s vs 236 s) 6:39 (pytest 378 s vs 263 s)

Build & Test runner variance is visible: on the previous head's run the same partition took 251 s vs 224 s (4:35 slowest job), when every required check finished 5:54 after start.

Why the old run was slow:

Required check Before Why
Windows Compatibility Gate 10:43-12:18 1,068 tests serially on one runner (pytest 10:02)
Lifecycle Smoke Shard 1 8:41 count-balanced shards: 492 s vs 350 s
Build & Test Shard 2 7:36 pytest-split naive split: 442 s vs 236 s

Changes (all on standard runners, no new dependencies, no ruleset edits):

  1. Windows Compatibility Gate runs as three serial shards on parallel windows-latest runners. Execution is unchanged (no xdist, so no MAX_PATH --basetemp workaround). A fan-in job keeps the required name Windows Compatibility Gate and fails on any missing, skipped, cancelled or failed shard. Seed: .github/ci-durations/windows-compat.json, per-test times averaged over serial job 134549967146 and this PR's first sharded run.
  2. Lifecycle Smoke goes from two to three shards, balanced by the existing measured linux-x86_64.json seed. The count-weighted scripts/lifecycle_smoke_sharding.py is retired in favour of the canonical scripts/ci_sharding.py; its composite-label test moves to test_ci_sharding.py.
  3. Build & Test keeps two shards and its check names, balanced by .github/ci-durations/linux-unit.json (local full-suite measurement) instead of pytest-split's naive split. Per-shard coverage stays above the 60% floor (75.00% / 75.22%).
  4. apm.lock.yaml records the redeployed tests.instructions.md hash (edited in .apm/ and redeployed).

EXPECTED_CHECKS is unchanged. Seeds are weights, never allowlists.

Partition proof (0 missing / 0 overlap vs control)

Local collection, control vs union of shards:

Lane Control Shards Overlap Missing Extra
Windows gate (-m windows_compat) 1,070 312 / 225 / 533 0 0 0
Lifecycle (required expression) 697 231 / 234 / 232 0 0 0
Build & Test 28,362 14,796 / 13,566 0 0 0

Hosted: the Windows shards selected 414 + 288 + 368 = 1,070 = the control selection. At runtime ci_sharding.partition fails closed unless the shards are nonempty and partition the selection exactly.

Issue and approved scope

Issue: Refs #3581

Human scope-approval comment: the maintainer (Daniel Meppiel (@danielmeppiel)) directed this bounded follow-up in the coordinating session ("but the ci session was working on additional gains (e.g. parallelizing windows compatibility gate job)"). No public scope-record comment is cited for this follow-up; this is an automation-written record, not a human approval comment.

Does this PR complete the issue, or what remains? Partial. It implements item 1's goal (Windows gate parallelism) via cross-runner sharding instead of in-runner xdist, and adds durations-based sharding for the PR lanes. Concurrency cancellation, apt time-boxing and the pylint bump remain open in #3581 / #3582.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Testing

  • Tested locally

  • All existing tests pass

  • Added tests for new functionality (if applicable)

  • pytest tests/unit/test_windows_compat_gate_workflow.py tests/unit/scripts/test_ci_sharding.py: 59 passed

  • pytest tests/quality/test_ci_topology.py: 189 passed

  • apm audit --ci: all 10 checks passed

  • Hosted: all 26 PR checks pass on head 35c1a45 (3 skipped by design)

  • Lint contract (ruff check, ruff format --check, pylint R0801, lint-auth-signals, lint-architecture-boundaries): all clean

Spec conformance (OpenAPM v0.1)

  • N/A -- this PR does not change OpenAPM-observable behaviour.

Run the Windows Compatibility Gate as three serial, duration-balanced
shards on parallel windows-latest runners behind a fan-in that keeps the
required check name and fails on any missing, skipped, cancelled or failed
shard. Split Lifecycle Smoke into three shards and balance both lifecycle
and Build & Test shards by measured durations with scripts/ci_sharding.py,
retiring the count-weighted lifecycle sharder. Seeds are weights, not
allowlists; every lane still partitions its selection exactly once.

Refs #3581

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ction

Average the Windows compatibility per-test durations over two hosted runs
to damp runner variance, and record the redeployed tests.instructions.md
content hash so apm audit --ci and the integrity unit test stay green.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel
Daniel Meppiel (danielmeppiel) marked this pull request as ready for review November 18, 2026 09:26
Copilot AI balanced review requested due to automatic review settings November 18, 2026 09:26
@danielmeppiel
Daniel Meppiel (danielmeppiel) merged commit 1bf22af into main Nov 18, 2026
32 checks passed
@danielmeppiel
Daniel Meppiel (danielmeppiel) deleted the danielmeppiel-ci-windows-gate-sharding branch November 18, 2026 09:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the CI sharding changes.

No open issues 🎉

What changed in this PR

This PR advances the CI performance work in #3581 by parallelizing the Windows compatibility gate and using measured durations to balance required test lanes, while preserving required-check names and test selections.

Changes:

  • Splits Windows compatibility testing into three serial shards on parallel runners, with an aggregate requiring all shards to succeed.
  • Moves lifecycle smoke testing to three duration-balanced shards and switches the two Linux unit shards to the shared sharding plugin.
  • Updates workflow contracts, contributor documentation, and deployed instruction hashes.
File Description
tests/​unit/​test_windows_compat_gate_workflow.py Verifies Windows shard configuration and aggregate failure handling.
tests/​unit/​scripts/​test_lifecycle_smoke_sharding.py Removes tests for the retired lifecycle-specific sharder.
tests/​unit/​scripts/​test_ci_sharding.py Adds composite xdist-group affinity coverage.
tests/​quality/​test_ci_topology.py Updates lifecycle contracts and partition checks for three shards.
scripts/​lifecycle_smoke_sharding.py Removes the lifecycle-specific sharding plugin.
docs/​src/​content/​docs/​contributing/​integration-testing.md Documents duration-based sharding and updated local commands.
apm.lock.yaml Refreshes deployed test-instruction hashes.
.github/​workflows/​ci.yml Configures duration-balanced lanes and required shard aggregates.
.github/​instructions/​tests.instructions.md Updates deployed guidance for the sharded Windows gate.
.github/​ci-durations/​windows-compat.json Adds measured Windows compatibility scheduling weights.
.apm/​instructions/​tests.instructions.md Updates source guidance for the sharded Windows gate.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants