Repository navigation
ci: run Windows push tests and spec conformance in parallel - #3830
Merged
Daniel Meppiel (danielmeppiel) merged 1 commit intoNov 18, 2026
Merged
Conversation
Daniel Meppiel (danielmeppiel)
requested a review
from Sergio Sisternes (sergio-sisternes-epam)
as a code owner
November 18, 2026 04:30
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The hosted Windows rehearsal is still in progress, so a maintainer must confirm the changed dependency gating and successful artifact production on the new push and unchanged release paths before landing.
No open issues 🎉
What changed in this PR
This PR extends APM's duration-based CI sharding to reduce Windows push validation and spec conformance elapsed time, while preserving prerequisite checks and the existing full/promoted release paths.
Changes:
- Runs the Windows installer and two unit shards concurrently, requiring all three and the push plan to succeed before building one Windows binary.
- Splits spec execution across three isolated runners, retaining
Spec conformance gateas the combined execution and statement-check signal. - Adds duration seeds, gate regression tests, shared test utilities, and documentation.
| File | Description |
|---|---|
tests/workflow_contracts.py |
Centralizes the native Bash execution helper. |
tests/unit/test_windows_unit_diagnostics.py |
Updates diagnostics expectations for full validation. |
tests/unit/test_shared_apm_workflow_contract.py |
Reuses the shared Bash helper. |
tests/unit/test_push_ci_gates.py |
Checks producer failure handling and shard execution contracts. |
tests/unit/test_platform_contract_workflow.py |
Updates the original installer step's full-mode condition. |
docs/src/content/docs/contributing/integration-testing.md |
Documents parallel validation and duration-seed semantics. |
.github/workflows/spec-conformance.yml |
Adds three execution shards and a combined conformance gate. |
.github/workflows/build-release.yml |
Separates Windows push tests from the gated binary build. |
.github/ci-durations/windows-unit.json |
Supplies Windows unit-test duration weights. |
.github/ci-durations/spec-conformance.json |
Supplies spec conformance duration weights. |
Review effort: Balanced
Daniel Meppiel (danielmeppiel)
deleted the
danielmeppiel-push-ci-optimization
branch
November 18, 2026 06:33
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
One hosted comparison met both elapsed-time targets without changing CodeQL or skipping any test. The main-push rehearsal ended in 6:21, including both CodeQL lanes. The actual PR spec conformance gate ended in 2:26.
Note
These are single after runs, not new medians. On the nine historical green code pushes, unchanged CodeQL took 7:43 median, so it can exceed the 7:31 target on its own. The spec figure is not the time for the overall PR
gate.On main pushes, start the installer and two duration-balanced Windows unit shards together. The Windows binary has four explicit prerequisites; a missing, skipped, failed or cancelled producer blocks both the build and its artifact upload:
flowchart LR I[Windows installer] --> B[Build and upload one Windows binary] U1[Windows units 1/2] --> B U2[Windows units 2/2] --> B P[Release plan: push] --> BOn qualifying PRs and merge groups, run the three spec shards serially within isolated runners. The unchanged
Spec conformance gatename requires all three plus a separate checkout that runs the orphan check, Mode B detector, successful full collection, generation and clean diff.Trade-off: extra jobs on existing standard runners and repeated collection buy lower elapsed time. The seeds are weights, never allowlists. Windows uses per-module elapsed time between successive xdist results on each worker from a green Windows run; spec uses a local serial pytest profile. No larger runners, concurrency cancellation, CodeQL changes, ruleset edits, or new dependencies. Full/promoted release paths still use the original build/test steps.
Issue and approved scope
Issue: Refs #3581. Credit to Daniel Meppiel (@danielmeppiel) and #3582 for the duration-aware sharding work and the warning about packed hooks under xdist. This uses the already-landed
scripts.ci_shardingplugin; it does not duplicate #3582’s PR gate or release changes, close either item, or depend on v0.36.1.Human scope-approval comment: the maintainer requested this bounded implementation in the coordinating session. No public scope-record comment for this follow-up is cited; the older #3771 approval was narrower.
Type of change
Testing
Rehearsal boundary: the before push is the PR’s base
2be1328f7a. The branch-only harness checked out candidated391bd67f0for every push-build job, excluded all release/publication jobs, and changed only the push ref filter on the existing CodeQL workflow for the rehearsal branch. No release/tag was triggered. Productioncodeql.ymlis byte-for-byte unchanged.Scenario evidence and how to reproduce
apm-linux-x86_64,apm-linux-arm64,apm-windows-x86_64.161 + 170 + 250 = 581, exact union and no overlap. All six packed-hook cases ran together. The three hosted shards reproduced the control outcomes: 576 passed, 2 skipped, 3 xfailed. The statement check passed separately. To inspect the partition, runuv run --frozen --extra dev python -m pytest --collect-only -qwith the respective suite and shard arguments from the workflow.uv run --frozen --extra dev pytest -q tests/unit/test_push_ci_gates.py tests/unit/test_shared_apm_workflow_contract.py— 62 passed. The test executes each actual gate, substituting failure, cancellation, skip and missing result for each producer. Windows cannot get past step 1 to sign/upload; conformance cannot pass on generation alone.uv run --frozen --extra dev pytest -q tests/unit/test_{push_ci_gates,windows_unit_diagnostics,windows_signing_contract,platform_contract_workflow}.py tests/unit/scripts/test_{ci_sharding,release_plan}.py— 152 passed. Serial spec profile: 576 passed, 2 skipped, 3 xfailed in 387.15s.actionlint -shellcheck= .github/workflows/{build-release,spec-conformance}.yml— passed; local CI lint mirror passed. No claim of a full release qualification.Spec conformance (OpenAPM v0.1)
If this PR changes behaviour that an OpenAPM v0.1
req-XXXcovers,confirm the three-step ritual in the
development guide:
docs/src/content/docs/specs/openapm-v0.1.mdupdated(new/changed
<a id="req-XXX"></a>anchor + prose + Appendix Crow).
docs/src/content/docs/specs/manifests/openapm-v0.1.requirements.ymlupdated.
@pytest.mark.req("req-XXX")test undertests/spec_conformance/added or extended.CONFORMANCE.{md,json}regenerated viauv run --extra dev python -m tests.spec_conformance.gen_statementand committed.
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com