CI's tests and tests-no-submodule jobs take about an hour, so every landing and every fix push waits an hour: split the shell suite, shard the rest, and give changes that touch only docs or specs a short path #300

Activity

1. brettheap

   added this to the v1.2.0 milestone on Oct 21, 2026
2. brettheap commented on Oct 22, 2026

   

   brettheap

   on Oct 22, 2026

   ContributorAuthor

   More actions

   CLAIMED — lane openRepoTools-3, session 83619980-06bb-4469-ae73-2a8c5e61981a@Eagle, 2026-10-22T12:29:38Z, for #300

   Logged in lanes/log/openRepoTools-3.md at 9c15492ab1ccd1b3e127b55f4b80f1a7d6d1a180.

   The three reads (lane-collision-protocol Rule 1):

   1. existing `CLAIMED —` comments on opensoft/openRepoTools#300:
      none
   2. `gh pr list --repo opensoft/openRepoTools --state all --search "#300"`, kept where the row or its body names `#300`:
      none
   3. `git ls-remote --heads git@github.com:opensoft/openRepoTools` branches naming `300` as a whole token:
      none
   
3. brettheap commented on Oct 22, 2026

   

   brettheap

   on Oct 22, 2026

   ContributorAuthor

   More actions

   Lane openRepoTools-3 on #300, 2026-10-22T13:1xZ: a measured plan. CI is slow because the suite runs serially on one runner, not because of queueing or setup. Shards that keep the gate's check names cut a run from about 71 min to about 29 (step 1). Splitting the shell suite into chunks then gets it to about 11 (step 2). This is analysis only. Nothing was pushed, dispatched or re-run, and the build waits for the cutover and Brett's word.

   Where the time goes. These are medians, with ranges in brackets, over the 15 all-green tests.yml runs on ubuntu-24.04. They come from the 40 newest runs, 2026-10-21 14:59Z to 2026-10-22 12:21Z.

   part	tests	tests-no-submodule
   queue wait	4 s [2–5]	4 s [2–5]
   setup	9 s [6–11]	6 s [3–13]
   pytest step	70.2 min [56.3–78.0]	58.4 min [48.5–77.5]
   ↳ of which the shell suite	1720 s (41%)	1286 s (43%)
   ↳ of which the other ~3000 tests	2477 s	1732 s

   • The whole run, from its creation to its last job, takes 70.9 min [57.4–78.3]. The queue never passed 5 s, even at the peak of 28 concurrent jobs of this workflow (2026-10-22 08:49:53Z).
   • 26.04. 22 of the 38 completed tests jobs ran on ubuntu-26.04. Every one came from a feat/198-* branch whose tests.yml predates CI and the release workflow run on ubuntu-24.04, because ubuntu-latest is now Ubuntu 26.04, where setup-python has no Python 3.9 and the TERM-ignoring kill test fails #296 and still says ubuntu-latest. All 22 failed, taking 86 min [65–103], so the table leaves them out. That is Ubuntu 26.04's uutils timeout -k never sends its KILL to the process group, so a bounded command can wait out a child that survives TERM, and setup-python has no Python 3.9 there; CI runs on ubuntu-24.04 until both are handled #297's subject.
   • The slowest units, as the median of 4 green runs, with the share of the 4196 s pytest step:
     • test_supervised_restart_regressions.py, 661 s (16%; probably 700–750, because module times are split out of -q progress lines);
     • the shell section "Amendment 18 Addendum 1", 327 s (8%);
     • test_lanes_index.py, 267 s (6%);
     • test_status_command.py, 146 s;
     • the shell sections "Amendment 17" and "Amendment 14", 139 s and 138 s.

   The options.

   option	tests wall time	verdict
   A. the shell suite in its own job	~41.5 min	covered by B
   B. A, plus the rest in 3 shards by module (step 1)	~29 min, bounded by the shell suite (each pytest shard ~13.8 min)	do first: 4 → 12 jobs per run, at about the same ~131 job-minutes
   C. B, plus the shell suite in 3 chunks and 4 pytest shards (step 2)	~11 min	do second: 18 jobs; needs a refactor of the 17,469-line shell suite
   D. pytest -n 4 (xdist)	≥ 29 min, because the shell suite is one test	reject: a new dependency, and timing-bound tests on loaded vCPUs
   E. cache the setup	−3–5 s per job	not worth it: setup is 9 s
   F. drop the shell shard from tests-no-submodule	0 wall time	optional, with step 2. The shell suite has no upstream/ path, so this saves 21 runner-minutes per run, but narrows that job's claim
   G. slow tests only nightly	e.g. −11 min	reject for the gate: landings would test less than they land
   H. faster or self-hosted runners	~0, because the suite is serial	reject: a self-hosted runner on a public repo runs fork code on Eagle
   I. a short path for docs-only PRs	~1 min for those PRs	skip: pointless after the cutover, when docs/, openspec/ and specs/ move to the spec leg
   J. cancel superseded pull_request runs	0	optional: it frees runners (12 runs of one PR overlapped between 07:30 and 10:05Z on 10-22), but any new label would cancel a run and restart it

   What step 2 involves.

   • About 34 names are defined in one section and used by later ones, from 10 sections; the chief ones are Amendment 7's $E, $LOGD, $WS_S and $CLONE2. They move to the preamble.
   • On-disk coupling between sections (rows and logs one section writes and a later one reads) stays unmeasured until each chunk runs alone.
   • The best contiguous 3-way split is sections 1–14, 15–19 and 20–28, at 625, 616 and 478 s.
   • test_lane_helpers_suite_timeout.py calls test_the_lane_helper_suite_passes() with no arguments in six places, so that signature must stay callable.

   Step 1 as drafted. The patch is below: 4 files, +299/−17.

   • tests.yml.
     • tests becomes the matrix job tests-shard, over shell, py-1, py-2 and py-3, with fail-fast: false. tests-no-submodule becomes tests-no-submodule-shard the same way.
     • Each shard still runs exactly python3 -m pytest tests -q (or -q -rs), with OPENREPOTOOLS_CI_SHARD naming its shard.
   • The check names stay. New one-step jobs named tests and tests-no-submodule each needs: its shard job and runs under if: always(). Each is green only when the combined result is success. The always() is there because a job skipped by a failed needs: would read as a passing required check.
   • tests/ci_shards.py holds the plan. Every test lands in exactly one shard: the first shard with a name that matches it, or else py-3. A new module is therefore never left out.
   • tests/conftest.py.
     • It pops OPENREPOTOOLS_CI_SHARD in pytest_configure, so a pytest that a test starts never inherits a shard.
     • It refuses an unknown shard, with exit 4.
     • It deselects the other shards' tests.
     • It prints --durations=25, the data for rebalancing.
   • tests/test_tests_workflow.py checks that:
     • the workflow has exactly the four gate jobs and the two shard jobs;
     • the only if: is the gatherers' always();
     • each gatherer's script, when run, is green only for success, and red for failure, cancelled, skipped and an empty result;
     • each matrix equals ci_shards.SHARDS;
     • every name in the plan is a real module or test.
   • Landing does not change. release.yml's gate (about lines 549–594) and bin/land-ort3.sh read tests, tests-no-submodule, guard-launch-mode and package by exact name. Queue mode also needs the run's conclusion to be success, and that requires every shard. A flaky shard re-runs alone, in 14–29 min instead of 70. The shard rows are extra checks. Never make them required, because they move with the plan.

   Verified locally. All of it ran one pytest process at a time, through tests/run.sh, with every lane variable unset.

   • actionlint is clean, and the pinned repo_shape.load_yaml parses the draft.
   • Collection finds 3024 tests. The plan splits them 1 / 750 / 1320 / 953.
   • An unknown shard exits 4.
   • Passing: test_tests_workflow.py (45 tests), test_repo_hygiene.py (237) and test_test_roots.py (40).
   • It composes with T010's code-leg tests.yml patch. Applied to main's tests.yml (blob 46feb68) in either order, the two patches give identical files, and both checks pass on the result.
   • Not done: any CI run of the draft. Two behaviours are GitHub's documented ones and were tested here only offline: needs.<job>.result over a matrix, and how a skipped required check reads.

   The order of work.

   • Before the cutover: no change to tests.yml here. Main is frozen at e2, and the Gate B documents quote the patched file's job lines.
   • After the cutover, in opensoft/openRepoTools-code:
     1. Step 1. Dispatch it about three times on its branch to check for flakes and the estimates, then land it.
     2. The same matrix in the assembly's composed.yml, with a gatherer named exact-pins. That job should refuse a matrix that differs from the pinned leg's SHARDS.
     3. Step 2, which meets this issue's target of under ~20 min.
     4. Optionally: F; J; a timeout-minutes per shard (there is none today, so the 360-min default applies); and a re-measure once Ubuntu 26.04's uutils timeout -k never sends its KILL to the process group, so a bounded command can wait out a child that survives TERM, and setup-python has no Python 3.9 there; CI runs on ubuntu-24.04 until both are handled #297 settles 26.04.

   Files on Eagle:

   • the patch: /workspace/projects/.lane-worktrees/openRepoTools-3/briefs/300-ci-work/300-phase1-draft.patch (sha256 267575f3…);
   • the measurements, scripts and the 30-unit table: /workspace/projects/.lane-worktrees/openRepoTools-3/briefs/300-ci-work/measurements/.

   The step 1 draft patch (local commit ea29556, never pushed)

   From ea29556ebf35da19231dfd6e6f62effc38d0a89c Mon Sep 17 00:00:00 2001
   From: openRepoTools lane 3 draft <ci@openrepotools.invalid>
   Date: Thu, 22 Oct 2026 13:01:09 +0000
   Subject: [PATCH] DRAFT (local only, never pushed): #300 phase 1, the suite in
    shards under the gate's names
   
   ---
    .github/workflows/tests.yml  |  74 ++++++++++++++++++++++--
    tests/ci_shards.py           |  96 ++++++++++++++++++++++++++++++
    tests/conftest.py            |  37 ++++++++++++
    tests/test_tests_workflow.py | 109 +++++++++++++++++++++++++++++++----
    4 files changed, 299 insertions(+), 17 deletions(-)
    create mode 100644 tests/ci_shards.py
   
   diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
   index 46feb68..d3948f0 100644
   --- a/.github/workflows/tests.yml
   +++ b/.github/workflows/tests.yml
   @@ -30,9 +30,11 @@ name: tests
    # (`opened`, `synchronize`, `reopened`). It was put there so that the
    # coordinator applying `ready` to an already-open PR would reach `tests-macos`'s
    # `if:`; that job is gone, so no job here reads a label any more, and `labeled`
   -# stays as it was. `on:` is shared by the whole workflow and no job has an
   -# `if:`, so EVERY TRIGGER RUNS ALL FOUR JOBS, `guard-launch-mode`, `tests`,
   -# `tests-no-submodule` and `package`, and so does ANY label landing on a PR.
   +# stays as it was. `on:` is shared by the whole workflow and no job is
   +# skipped by an `if:`, so EVERY TRIGGER RUNS ALL FOUR GATE JOBS,
   +# `guard-launch-mode`, `tests`, `tests-no-submodule` and `package`, with the
   +# shards `tests` and `tests-no-submodule` gather (#300), and so does ANY label
   +# landing on a PR.
    # `push` covers `main` and every `release/v*` branch (#228): the release
    # workflow builds only a commit whose push or dispatch run of those four is
    # green, and a patch release's commit is on its release branch, not on main.
   @@ -74,8 +76,22 @@ jobs:
              GIT_COMMITTER_EMAIL: ci@openrepotools.invalid
            run: bash tests/run.sh -k 'guard_launch_mode or repo_hygiene'
    
   -  tests:
   +  # THE SUITE IN SHARDS (#300). One serial run took 70 min (median of 15
   +  # green runs, 56 to 78), and the shell suite alone 29 of them. Each matrix
   +  # entry is a runner of its own running the same `pytest tests -q`, and
   +  # `OPENREPOTOOLS_CI_SHARD` makes `tests/conftest.py` keep only that shard's
   +  # tests of `tests/ci_shards.py`, a plan in which every test is in exactly
   +  # one shard. `fail-fast: false`, so one red shard does not hide another's
   +  # result. `tests/run.sh`'s lock is a workstation's: each shard here is its
   +  # own machine, and calls pytest directly, as this job always did.
   +  tests-shard:
        runs-on: ubuntu-24.04
   +    strategy:
   +      fail-fast: false
   +      matrix:
   +        shard: [shell, py-1, py-2, py-3]
   +    env:
   +      OPENREPOTOOLS_CI_SHARD: ${{ matrix.shard }}
        steps:
          # PINNED BY COMMIT, not by tag — openRepoShape's own rule, and the same
          # two commits every job here pins. A tag can be moved and a commit
   @@ -120,6 +136,27 @@ jobs:
          - name: tests
            run: python3 -m pytest tests -q
    
   +  # THE GATE'S `tests`, UNDER ITS OWN NAME (#300): green only when every shard
   +  # above is. `bin/land-ort3.sh`, the other lanes' landing scripts and
   +  # release.yml's gate read the job by this name, so the name stays and the
   +  # work moved. `always()`, because a job whose `needs:` failed is SKIPPED,
   +  # and a skipped required check reads as passing: run, it is red instead.
   +  tests:
   +    needs: tests-shard
   +    if: always()
   +    runs-on: ubuntu-24.04
   +    steps:
   +      - name: every shard of the suite passed
   +        env:
   +          SHARDS: ${{ needs.tests-shard.result }}
   +        run: |
   +          set -euo pipefail
   +          echo "tests-shard: $SHARDS"
   +          if [ "$SHARDS" != success ]; then
   +            echo "::error::a shard of the suite did not pass: tests-shard is $SHARDS"
   +            exit 1
   +          fi
   +
      # THE SKIP PATH, PROVED. A fork's first command is `git clone` without
      # `--recurse-submodules` and then `pytest`, and if that goes RED the fork is
      # one nobody finishes. So one job checks out with NO submodules and asserts
   @@ -129,9 +166,17 @@ jobs:
      #
      # `pytest -q` exits 0 on skips and 1 on failures, so the job's own status IS
      # the assertion; `-rs` prints the skip reasons, which is what a person
   -  # reading this job wants to see.
   -  tests-no-submodule:
   +  # reading this job wants to see. IT RUNS IN THE SAME SHARDS (#300), so
   +  # together they still run the whole suite without the submodule, and
   +  # `tests-no-submodule` below gathers them under the gate's name.
   +  tests-no-submodule-shard:
        runs-on: ubuntu-24.04
   +    strategy:
   +      fail-fast: false
   +      matrix:
   +        shard: [shell, py-1, py-2, py-3]
   +    env:
   +      OPENREPOTOOLS_CI_SHARD: ${{ matrix.shard }}
        steps:
          - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262  # v4
            with:
   @@ -155,6 +200,23 @@ jobs:
          - name: the suite skips rather than fails
            run: python3 -m pytest tests -q -rs
    
   +  # THE GATE'S `tests-no-submodule`, as `tests` above gathers its shards.
   +  tests-no-submodule:
   +    needs: tests-no-submodule-shard
   +    if: always()
   +    runs-on: ubuntu-24.04
   +    steps:
   +      - name: every shard of the skip path passed
   +        env:
   +          SHARDS: ${{ needs.tests-no-submodule-shard.result }}
   +        run: |
   +          set -euo pipefail
   +          echo "tests-no-submodule-shard: $SHARDS"
   +          if [ "$SHARDS" != success ]; then
   +            echo "::error::a shard of the skip path did not pass: tests-no-submodule-shard is $SHARDS"
   +            exit 1
   +          fi
   +
      # THE openrepotools PACKAGE, BUILT, CHECKED AGAINST THIS COMMIT, INSTALLED
      # AS A PERSON INSTALLS IT AND TESTED (#228, contracts/workflows.md). The
      # release workflow builds only a commit whose push or dispatch run of all
   diff --git a/tests/ci_shards.py b/tests/ci_shards.py
   new file mode 100644
   index 0000000..1906205
   --- /dev/null
   +++ b/tests/ci_shards.py
   @@ -0,0 +1,96 @@
   +# SPDX-License-Identifier: Apache-2.0
   +"""THE CI SHARD PLAN (opensoft/openRepoTools#300): which shard of
   +`.github/workflows/tests.yml` runs which test.
   +
   +READ ONLY WHEN `OPENREPOTOOLS_CI_SHARD` NAMES A SHARD, which only the
   +`tests-shard` and `tests-no-submodule-shard` jobs set, one shard per matrix
   +entry. `tests/conftest.py` then deselects every test of the other shards. A
   +workstation run, `tests/run.sh` and the assembly's composed job set nothing,
   +so they run the whole suite in one process, exactly as before.
   +
   +TOTAL BY CONSTRUCTION. Every test belongs to exactly one shard: the first
   +shard below one of whose names matches it, or else `REST`. A module this plan
   +does not name, such as a new one, runs in `REST` and is never left out. A shard
   +that comes to select nothing makes its job exit 5, "no tests collected", which
   +is red: a plan whose names no longer match is fixed, not ignored.
   +
   +A NAME IS A MODULE OR A TEST, read against the node id with its directory
   +dropped: `test_x.py` takes every test of that module, and `test_x.py::test_y`
   +takes that test and each of its parametrized cases.
   +
   +BALANCED ON MEASURED TIME, the medians of four green `tests` jobs on
   +`ubuntu-24.04` between 2026-10-21 and 2026-10-22 (#300 plan v1): the shell
   +suite alone is about 1720 s, and each `py-` shard about 830 s. Rebalance from
   +the `--durations` table each shard's log prints.
   +"""
   +
   +from __future__ import annotations
   +
   +from typing import Dict, Tuple
   +
   +#: THE SHELL SUITE ON ITS OWN: one test, about 41% of the `tests` job.
   +SHELL = "shell"
   +
   +#: The shard every test not named below runs in.
   +REST = "py-3"
   +
   +#: Each shard and the names it takes, matched in this order.
   +PLAN: Dict[str, Tuple[str, ...]] = {
   +    SHELL: (
   +        "test_lane_helpers_suite.py::test_the_lane_helper_suite_passes",
   +    ),
   +    "py-1": (
   +        "test_supervised_restart_regressions.py",
   +        "test_triad_lane_tooling.py",
   +        "test_lane_worktrees_report.py",
   +        "test_lane_helpers_suite.py",
   +        "test_lane_helpers_suite_timeout.py",
   +        "test_handoff_skill_72_80.py",
   +        "test_repo_hygiene.py",
   +        "test_package_stamp.py",
   +        "test_command_builtin_errexit_292.py",
   +        "test_run_wrapper.py",
   +        "test_skill_shell_blocks.py",
   +        "test_claude_restart_check.py",
   +        "test_install_by_rename.py",
   +    ),
   +    "py-2": (
   +        "test_lanes_index.py",
   +        "test_lifecycle_fail_open_113_114_115.py",
   +        "test_install_payload_source.py",
   +        "test_wip_init_command.py",
   +        "test_triad_lane_tooling_handoff.py",
   +        "test_release_workflow.py",
   +        "test_tests_workflow.py",
   +        "test_lane_start_hardening_138.py",
   +        "test_park_resume_commands.py",
   +        "test_install_hook_gate.py",
   +        "test_register_hardening_120_122.py",
   +        "test_rename_publication_atomic_158.py",
   +        "test_wrapper_hardening_138.py",
   +        "test_register_scan_hardening_138.py",
   +        "test_handoff_skill_98_99_100.py",
   +        "test_handoff_skill_commit_block.py",
   +    ),
   +    REST: (),
   +}
   +
   +#: Every shard's name, in the plan's order, as the matrix lists them.
   +SHARDS: Tuple[str, ...] = tuple(PLAN)
   +
   +
   +def key_of(nodeid: str) -> str:
   +    """A node id with its directory dropped: `tests/test_x.py::test_y[a/b]`
   +    is `test_x.py::test_y[a/b]`. Only the path part loses its directory, so a
   +    parametrized id holding a `/` keeps it."""
   +    path, sep, rest = nodeid.partition("::")
   +    return path.rsplit("/", 1)[-1] + sep + rest
   +
   +
   +def shard_of(key: str) -> str:
   +    """The one shard that runs the test whose `key_of` is `key`."""
   +    for shard, names in PLAN.items():
   +        for name in names:
   +            if key == name or key.startswith(name + "::") or key.startswith(name + "["):
   +                return shard
   +    return REST
   diff --git a/tests/conftest.py b/tests/conftest.py
   index 8341124..a05da11 100644
   --- a/tests/conftest.py
   +++ b/tests/conftest.py
   @@ -515,6 +515,43 @@ def pytest_configure(config):
        refusal = ROOTS.refusal()
        if refusal:
            raise pytest.UsageError(refusal)
   +    # THE SHARD IS THIS PROCESS'S, NEVER A CHILD'S: it leaves the environment
   +    # here, before any test runs, so a pytest a test starts (the suite has
   +    # several) collects what that test asked for and not one shard of it.
   +    shard = os.environ.pop(ENV_CI_SHARD, "")
   +    if shard:
   +        import ci_shards
   +        if shard not in ci_shards.SHARDS:
   +            raise pytest.UsageError(
   +                f"{ENV_CI_SHARD}={shard!r} names no shard of tests/ci_shards.py; "
   +                f"its shards are {', '.join(ci_shards.SHARDS)}")
   +        config.stash[_CI_SHARD] = shard
   +        # The table a rebalance of tests/ci_shards.py reads, in this
   +        # process's own report only.
   +        if config.option.durations is None:
   +            config.option.durations = 25
   +
   +
   +#: THE CI SHARD (opensoft/openRepoTools#300). `tests.yml`'s shard jobs set it
   +#: to one shard of `tests/ci_shards.py`, and the run keeps only that shard's
   +#: tests, reporting the rest as deselected. Unset - every workstation run,
   +#: `tests/run.sh` and the composed job - nothing is deselected.
   +ENV_CI_SHARD = "OPENREPOTOOLS_CI_SHARD"
   +_CI_SHARD = pytest.StashKey[str]()
   +
   +
   +def pytest_collection_modifyitems(config, items):
   +    shard = config.stash.get(_CI_SHARD, "")
   +    if not shard:
   +        return
   +    import ci_shards
   +    keep, drop = [], []
   +    for item in items:
   +        mine = ci_shards.shard_of(ci_shards.key_of(item.nodeid)) == shard
   +        (keep if mine else drop).append(item)
   +    if drop:
   +        config.hook.pytest_deselected(items=drop)
   +        items[:] = keep
    
    
    def pytest_report_header(config):
   diff --git a/tests/test_tests_workflow.py b/tests/test_tests_workflow.py
   index 030880c..b06bceb 100644
   --- a/tests/test_tests_workflow.py
   +++ b/tests/test_tests_workflow.py
   @@ -77,6 +77,12 @@ PINNED_READER = UPSTREAM / "scripts" / "repo_shape.py"
    #: a gate that finds nothing.
    JOB_IDS = ("guard-launch-mode", "tests", "tests-no-submodule", "package")
    
   +#: THE SHARDS TWO OF THEM GATHER (#300): each gate job on the left is a
   +#: one-step job that `needs:` the matrix job on the right and is green only
   +#: when every shard of it is. The work moved; the names the gate reads did not.
   +SHARD_JOBS = {"tests": "tests-shard",
   +              "tests-no-submodule": "tests-no-submodule-shard"}
   +
    #: The two commits every job pins (contracts/workflows.md).
    CHECKOUT = "actions/checkout@11d5960a326750d5838078e36cf38b85af677262"
    SETUP_PYTHON = "actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065"
   @@ -101,7 +107,7 @@ EXISTING_JOBS = {
                "bash tests/run.sh -k 'guard_launch_mode or repo_hygiene'",
            ],
        },
   -    "tests": {
   +    "tests-shard": {
            "checkout": {"submodules": True, "fetch-depth": 1},
            "python": "3.x",
            "runs": [
   @@ -115,7 +121,7 @@ EXISTING_JOBS = {
                "python3 -m pytest tests -q",
            ],
        },
   -    "tests-no-submodule": {
   +    "tests-no-submodule-shard": {
            "checkout": {"submodules": False, "fetch-depth": 1},
            "python": "3.x",
            "runs": [
   @@ -833,16 +839,97 @@ def test_tests_yml_its_other_triggers_are_unchanged(workflow):
    
    
    @NEEDS_UPSTREAM
   -def test_tests_yml_has_exactly_the_four_gate_jobs(workflow):
   -    """The release gate's four ids (FR-046) and no fifth. None has a
   -    `name:`, so each check run is named by its id, and none has an `if:`,
   -    so every trigger runs all four."""
   +def test_tests_yml_has_exactly_the_four_gate_jobs_and_their_shards(workflow):
   +    """The release gate's four ids (FR-046), the two shard jobs two of them
   +    gather, and nothing else. None has a `name:`, so each check run is named
   +    by its id. No job is skipped by an `if:`: the only `if:` is the two
   +    gatherers' `always()`, which makes them RUN when a shard failed, since a
   +    skipped job reads as a passing required check."""
        jobs = jobs_of(workflow)
   -    assert sorted(jobs) == sorted(JOB_IDS), (
   -        f"the jobs are {list(jobs)}, not {list(JOB_IDS)}")
   -    for job_id in JOB_IDS:
   -        assert "name" not in jobs[job_id], f"`{job_id}` has a `name:`"
   -        assert "if" not in jobs[job_id], f"`{job_id}` has an `if:`"
   +    want = sorted([*JOB_IDS, *SHARD_JOBS.values()])
   +    assert sorted(jobs) == want, f"the jobs are {list(jobs)}, not {want}"
   +    for job_id, job in jobs.items():
   +        assert "name" not in job, f"`{job_id}` has a `name:`"
   +        if job_id in SHARD_JOBS:
   +            assert job.get("if") == "always()", (job_id, job.get("if"))
   +        else:
   +            assert "if" not in job, f"`{job_id}` has an `if:`"
   +
   +
   +@NEEDS_UPSTREAM
   +@pytest.mark.parametrize("gate", sorted(SHARD_JOBS))
   +def test_tests_yml_a_gatherer_needs_its_shards_and_runs_one_check(workflow, gate):
   +    """`tests` and `tests-no-submodule` each need their shard job alone, on
   +    the runner every job uses, and hold one step that reads that job's
   +    combined result from its environment and nothing else."""
   +    job = jobs_of(workflow)[gate]
   +    assert job.get("needs") == SHARD_JOBS[gate], job.get("needs")
   +    assert job.get("runs-on") == "ubuntu-24.04", job.get("runs-on")
   +    steps = job.get("steps") or []
   +    assert len(steps) == 1 and "uses" not in steps[0], steps
   +    assert steps[0].get("env") == {
   +        "SHARDS": f"${{{{ needs.{SHARD_JOBS[gate]}.result }}}}"}, steps[0].get("env")
   +
   +
   +@NEEDS_BASH
   +@NEEDS_UPSTREAM
   +@pytest.mark.parametrize("gate", sorted(SHARD_JOBS))
   +@pytest.mark.parametrize("result,code", [("success", 0), ("failure", 1),
   +                                         ("cancelled", 1), ("skipped", 1),
   +                                         ("", 1)])
   +def test_tests_yml_a_gatherer_is_green_only_when_every_shard_is(
   +        workflow, gate, result, code, tmp_path):
   +    """The gatherer's own script, RUN under `bash -e` as the runner runs it,
   +    with each combined result GitHub can give a matrix job: green on
   +    `success` alone. A skipped or cancelled shard is not a passing one."""
   +    script = jobs_of(workflow)[gate]["steps"][0]["run"]
   +    (tmp_path / "step.sh").write_text(script, encoding="utf-8")
   +    env = {"PATH": os.environ["PATH"], "SHARDS": result}
   +    proc = subprocess.run(["bash", "-e", str(tmp_path / "step.sh")], env=env,
   +                          capture_output=True, text=True, timeout=60, check=False)
   +    assert proc.returncode == code, shown(proc)
   +
   +
   +@NEEDS_UPSTREAM
   +@pytest.mark.parametrize("shard_job", sorted(SHARD_JOBS.values()))
   +def test_tests_yml_a_shard_job_runs_every_shard_of_the_plan(workflow, shard_job):
   +    """Each shard job's matrix is `tests/ci_shards.py`'s shards, in its
   +    order, and each entry names its shard to conftest by the environment:
   +    the plan puts every test in one shard, so the matrix runs every test."""
   +    import ci_shards
   +    job = jobs_of(workflow)[shard_job]
   +    strategy = job.get("strategy") or {}
   +    assert strategy.get("fail-fast") is False, strategy
   +    assert strategy.get("matrix") == {"shard": list(ci_shards.SHARDS)}, strategy
   +    assert job.get("env") == {
   +        "OPENREPOTOOLS_CI_SHARD": "${{ matrix.shard }}"}, job.get("env")
   +
   +
   +def test_ci_shard_plan_names_only_tests_this_suite_has():
   +    """A name in `tests/ci_shards.py` that matches no module, or no test of
   +    its module, is a plan gone stale: its tests went to `REST` unseen. And
   +    every module the suite has falls in exactly one shard."""
   +    import ci_shards
   +    modules = {path.name: path.read_text(encoding="utf-8")
   +               for path in (REPO / "tests").glob("test_*.py")}
   +    stale = []
   +    for shard, names in ci_shards.PLAN.items():
   +        for name in names:
   +            module, _, test = name.partition("::")
   +            if module not in modules or (
   +                    test and not re.search(rf"^def {re.escape(test)}\(",
   +                                           modules[module], re.M)):
   +                stale.append(f"{shard}: {name}")
   +    assert not stale, f"names that match nothing: {stale}"
   +    for module in modules:
   +        assert ci_shards.shard_of(module + "::test_x") in ci_shards.SHARDS
   +    assert ci_shards.PLAN[ci_shards.SHELL] == (
   +        "test_lane_helpers_suite.py::test_the_lane_helper_suite_passes",)
   +    assert ci_shards.shard_of(ci_shards.key_of(
   +        "tests/test_lane_helpers_suite.py::test_the_lane_helper_suite_passes")) == "shell"
   +    assert ci_shards.shard_of(ci_shards.key_of(
   +        "tests/test_lane_helpers_suite.py::test_the_wrapper_looks_for_what_the_suite_prints")) == "py-1"
   +    assert ci_shards.shard_of("test_new_module_nobody_planned.py::test_y") == ci_shards.REST
    
    
    @NEEDS_UPSTREAM
   -- 
   2.43.0
   

   Lane: openRepoTools-3