Skip to content

ci: run Windows push tests and spec conformance in parallel - #3830

Merged
Daniel Meppiel (danielmeppiel) merged 1 commit into
mainfrom
danielmeppiel-push-ci-optimization
Nov 18, 2026
Merged

Daniel Meppiel (danielmeppiel) merged 1 commit into
mainfrom
danielmeppiel-push-ci-optimization

Conversation

@danielmeppiel

@danielmeppiel Daniel Meppiel (danielmeppiel) commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

One hosted comparison met both elapsed-time targets without changing CodeQL or skipping any test. The main-push rehearsal ended in 6:21, including both CodeQL lanes. The actual PR spec conformance gate ended in 2:26.

First-attempt green workflow Historical median Comparable before After Less elapsed vs before
Build-bearing main push, first creation to last finish of build + CodeQL 15:03 13:14 build, 6:10 scan 6:21 build, 5:02 scan 52%
Spec conformance, including generated artifacts and final gate 6:34 6:06 2:26 60%

Note

These are single after runs, not new medians. On the nine historical green code pushes, unchanged CodeQL took 7:43 median, so it can exceed the 7:31 target on its own. The spec figure is not the time for the overall PR gate.

On main pushes, start the installer and two duration-balanced Windows unit shards together. The Windows binary has four explicit prerequisites; a missing, skipped, failed or cancelled producer blocks both the build and its artifact upload:

flowchart LR
    I[Windows installer] --> B[Build and upload one Windows binary]
    U1[Windows units 1/2] --> B
    U2[Windows units 2/2] --> B
    P[Release plan: push] --> B
Loading

On qualifying PRs and merge groups, run the three spec shards serially within isolated runners. The unchanged Spec conformance gate name requires all three plus a separate checkout that runs the orphan check, Mode B detector, successful full collection, generation and clean diff.

Trade-off: extra jobs on existing standard runners and repeated collection buy lower elapsed time. The seeds are weights, never allowlists. Windows uses per-module elapsed time between successive xdist results on each worker from a green Windows run; spec uses a local serial pytest profile. No larger runners, concurrency cancellation, CodeQL changes, ruleset edits, or new dependencies. Full/promoted release paths still use the original build/test steps.

Issue and approved scope

Issue: Refs #3581. Credit to Daniel Meppiel (@danielmeppiel) and #3582 for the duration-aware sharding work and the warning about packed hooks under xdist. This uses the already-landed scripts.ci_sharding plugin; it does not duplicate #3582’s PR gate or release changes, close either item, or depend on v0.36.1.

Human scope-approval comment: the maintainer requested this bounded implementation in the coordinating session. No public scope-record comment for this follow-up is cited; the older #3771 approval was narrower.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Testing

  • Tested locally
  • All existing tests pass
  • Added tests for new functionality (if applicable)

Rehearsal boundary: the before push is the PR’s base 2be1328f7a. The branch-only harness checked out candidate d391bd67f0 for every push-build job, excluded all release/publication jobs, and changed only the push ref filter on the existing CodeQL workflow for the rehearsal branch. No release/tag was triggered. Production codeql.yml is byte-for-byte unchanged.

Scenario evidence and how to reproduce
  • No missing Windows tests: compared node IDs in the serial control log to the first and second hosted execution logs. All 27,857 original IDs ran exactly once: 0 missing, 0 overlap, 3 added (the new workflow contract cases). Unit shard sizes: 14,207 and 13,653. The installer ran in parallel; the single Windows build started after all three finished. Both runs emitted the same three artifact names: apm-linux-x86_64, apm-linux-arm64, apm-windows-x86_64.
  • No missing spec execution: independent full vs sharded collections: 161 + 170 + 250 = 581, exact union and no overlap. All six packed-hook cases ran together. The three hosted shards reproduced the control outcomes: 576 passed, 2 skipped, 3 xfailed. The statement check passed separately. To inspect the partition, run uv run --frozen --extra dev python -m pytest --collect-only -q with the respective suite and shard arguments from the workflow.
  • No false green: uv run --frozen --extra dev pytest -q tests/unit/test_push_ci_gates.py tests/unit/test_shared_apm_workflow_contract.py — 62 passed. The test executes each actual gate, substituting failure, cancellation, skip and missing result for each producer. Windows cannot get past step 1 to sign/upload; conformance cannot pass on generation alone.
  • Original contracts: uv run --frozen --extra dev pytest -q tests/unit/test_{push_ci_gates,windows_unit_diagnostics,windows_signing_contract,platform_contract_workflow}.py tests/unit/scripts/test_{ci_sharding,release_plan}.py — 152 passed. Serial spec profile: 576 passed, 2 skipped, 3 xfailed in 387.15s.
  • actionlint -shellcheck= .github/workflows/{build-release,spec-conformance}.yml — passed; local CI lint mirror passed. No claim of a full release qualification.

Spec conformance (OpenAPM v0.1)

If this PR changes behaviour that an OpenAPM v0.1 req-XXX covers,
confirm the three-step ritual in the
development guide:

  • Spec edit: docs/src/content/docs/specs/openapm-v0.1.md updated
    (new/changed <a id="req-XXX"></a> anchor + prose + Appendix C
    row).
  • Manifest edit: docs/src/content/docs/specs/manifests/openapm-v0.1.requirements.yml
    updated.
  • Test edit: a @pytest.mark.req("req-XXX") test under
    tests/spec_conformance/ added or extended.
  • CONFORMANCE.{md,json} regenerated via
    uv run --extra dev python -m tests.spec_conformance.gen_statement
    and committed.
  • N/A -- this PR does not change OpenAPM-observable behaviour.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Build on the duration-aware sharding introduced by the release slice of #3581/#3582. Keep full release validation and scan reporting intact.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The hosted Windows rehearsal is still in progress, so a maintainer must confirm the changed dependency gating and successful artifact production on the new push and unchanged release paths before landing.

No open issues 🎉

What changed in this PR

This PR extends APM's duration-based CI sharding to reduce Windows push validation and spec conformance elapsed time, while preserving prerequisite checks and the existing full/promoted release paths.

Changes:

  • Runs the Windows installer and two unit shards concurrently, requiring all three and the push plan to succeed before building one Windows binary.
  • Splits spec execution across three isolated runners, retaining Spec conformance gate as the combined execution and statement-check signal.
  • Adds duration seeds, gate regression tests, shared test utilities, and documentation.
File Description
tests/​workflow_contracts.py Centralizes the native Bash execution helper.
tests/​unit/​test_windows_unit_diagnostics.py Updates diagnostics expectations for full validation.
tests/​unit/​test_shared_apm_workflow_contract.py Reuses the shared Bash helper.
tests/​unit/​test_push_ci_gates.py Checks producer failure handling and shard execution contracts.
tests/​unit/​test_platform_contract_workflow.py Updates the original installer step's full-mode condition.
docs/​src/​content/​docs/​contributing/​integration-testing.md Documents parallel validation and duration-seed semantics.
.github/​workflows/​spec-conformance.yml Adds three execution shards and a combined conformance gate.
.github/​workflows/​build-release.yml Separates Windows push tests from the gated binary build.
.github/​ci-durations/​windows-unit.json Supplies Windows unit-test duration weights.
.github/​ci-durations/​spec-conformance.json Supplies spec conformance duration weights.

Review effort: Balanced

@danielmeppiel
Daniel Meppiel (danielmeppiel) merged commit 9df9a3c into main Nov 18, 2026
32 checks passed
@danielmeppiel
Daniel Meppiel (danielmeppiel) deleted the danielmeppiel-push-ci-optimization branch November 18, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants