Skip to content

CI's Linux test job takes 46–72 minutes a run, because its Python tests and the shell suite run one after the other #77

Open

Description

@brettheap

tests/test_lane_helpers.sh now takes 45 minutes on the macOS job and crossed its wrapper's 2400 s bound on feat/lane-the-word at b9018c1:

tests-macos   03:18:03 -> 04:03:37   1 failed, 654 passed in 2722.13s (0:45:22)
              subprocess.TimeoutExpired: Command '['bash', '…/tests/test_lane_helpers.sh']'
              timed out after 2400 seconds
tests         03:17:59 -> 03:34:46   green, the same 655 cases

Every other case in that macOS job passed. #45 raised the bound to 3600 s so the cap is not the thing failing — that is not the fix, it is the cap moving out of the fix's way.

What is actually slow. The suite is ONE bash file, run serially, that does real work per case: a bare repository and two clones seeded in a mktemp -d, several thousand real git invocations, a sleep process per liveness fixture, and a growing seed register rewritten and pushed by dozens of sections. It is 2345 assertions today and every amendment adds a section — 1500 at 92eab6c, 1678 after #41, 2102 after #52 and #47, 2345 with #45's own. macOS is roughly 2.7× the Linux time for the same file, which is where the cap bit first.

Directions worth measuring before choosing one:

  • Where the time actually goes. Nobody has profiled it. A PS4-stamped run, or a section-by-section timer printed beside each == … == heading, would say whether the cost is git process spawns, the sleep-backed liveness fixtures, the register pushes, or the per-case sandbox churn — and that answer decides everything below it.
  • One sandbox per SECTION rather than one per file. Sections are already fenced by their == heading == lines; if the seeding is the cost, sections that seed nothing could share.
  • Splitting the file so pytest can run sections in parallel (pytest -n is not in this repository's dependencies today, and adding one is its own decision) — the wrapper is deliberately ONE test, and that choice is documented in tests/test_lane_helpers_suite.py; splitting it means keeping the bash suite's own ok/FAIL vocabulary while giving pytest more than one bit, which the file's own comment explains the cost of.
  • git itself: commit.gpgsign=false, gc.auto=0, core.fsmonitor=false, protocol.version=2 and a shared template directory in the sandbox's $HOME/.gitconfig are cheap to try and easy to measure.

Wants: a measurement first, a direction second. Filed from the lane running #45, whose merge of Amendment 17's and Amendment 18's sections is what crossed the line.

Activity

  1. brettheap commented on Sep 15, 2026

    @brettheap
    ContributorAuthor

    CLAIMED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-15T04:06:34Z, for #77

    Logged in lanes/log/openRepoTools-3.md at 846bb5b9eda35e5bf4bfe38a4049b1b2f1b5e890.

    The three reads (lane-collision-protocol Rule 1):

    1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
       none
    2. `gh pr list --repo opensoft/openRepoTools --state all --search "#77"`, kept where the row or its body names `#77`:
       none
    3. `git ls-remote --heads git@github.com:opensoft/openRepoTools` branches naming `77` as a whole token:
       none
    
  2. brettheap commented on Sep 16, 2026

    @brettheap
    ContributorAuthor

    RELEASED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-16T00:36:27Z, for #77

  3. brettheap commented on Sep 16, 2026

    @brettheap
    ContributorAuthor

    CLAIMED — lane openRepoTools-1, session df82f553-7803-49c4-8ddf-3f9acdcb8ee0@Eagle, 2026-09-16T01:07:10Z, for #77

    Logged in lanes/log/openRepoTools-1.md at 2346a217e3c871082141313c73c8c88bd8359579.

    The three reads (lane-collision-protocol Rule 1):

    1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    6:CLAIMED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-15T04:06:34Z, for opensoft/openRepoTools#77
    13:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    26:RELEASED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-16T00:36:27Z, for opensoft/openRepoTools#77
    2. `gh pr list --repo opensoft/openRepoTools --state all --search "#77"`, kept where the row or its body names `#77`:
    83	One binding per lane: the record says host/os/container, liveness is pronounced only from inside that binding, and a second place ASKS for the handoff before it binds (Amendment 18 acts 1-10)	feat/amendment-18-one-binding	OPEN
    45	`lane`: one word lists the lanes and binds the one you pick — available goes through the launcher in its own directory and profile, LIVE HERE attaches and never starts a second process, bound elsewhere names where; and `restart` leaves the person's PATH in the same breath (Amendment 18 Addendum 1 and Addendum 2)	feat/lane-the-word	MERGED
    3. `git ls-remote --heads git@github.com:opensoft/openRepoTools` branches naming `77` as a whole token:
       none
    
  4. brettheap commented on Sep 16, 2026

    @brettheap
    ContributorAuthor

    Lane: openRepoTools-1 — measurement only, no edits; the suite is another lane's in flight.

    How this was measured

    One instrumented run of tests/test_lane_helpers.sh, reproducing exactly what tests/test_lane_helpers_suite.py does (same env-var pop list — LANES_FILE/EDIT/REPO/PATH/LANE/WORKSPACE_ROOT/REPOS_TSV*, AGENT_PROTOCOL_ROOT, OPENREPOTOOLS_BIN_DIR, PROJECTS_ROOT — plus the same GIT_AUTHOR_*/GIT_COMMITTER_* defaults; also unset two more LANES_* vars this interactive shell happened to carry ambiently, which the suite re-pins itself anyway), cwd at the repo root, run directly with bash (never python3 -m pytest).

    Shim: a git and a sleep script placed first on PATH (real binaries resolved via command -v before prepending the shim dir, so no recursion). Each shim reads $EPOCHREALTIME (bash 5.2, no process spawn needed to time), invokes the real binary as a child (not a literal exec, since an exec can't log an end time — noted as a deliberate, negligible-cost deviation from the letter of the brief), appends <start> <end> <cwd> <argv> to a log, and exits with the real exit code. Verified correct on a smoke-test repo before the real run. STDOUT was piped through while IFS= read -r line; do printf '%s %s\n' "$EPOCHREALTIME" "$line"; done for arrival timestamps. Guard + lock exactly per AGENTS.md (pgrep -f '^python3 -m pyt''est' host-wide, then flock ${TMPDIR:-/tmp}/openrepotools-pytest.lock).

    One run only (no second run: the biggest cost turned out to need no bash -x to find — see below). Section boundaries come from the suite's own == … == echo lines (20 of them) as they actually arrived in stdout, not from source line numbers.

    Overhead/contention caveats, stated plainly:

    • Shim overhead. A same-host microbenchmark just now (300× git rev-parse HEAD in a scratch repo): direct = 0.546s (1.82ms/call), shimmed = 1.225s (4.08ms/call) → ≈2.3ms/call shim overhead. Over 50,419 real calls that's ≈116s of the totals below that is my instrumentation, not the suite — small next to the numbers that matter but not zero.
    • Workstation contention. This box runs several lanes' suites concurrently. My guard correctly waited 33m24s (01:23:16Z→01:56:40Z) for other python3 -m pytest runs to clear, then blocked a further 33m53s inside flock (01:56:40Z→02:30:33Z) for another lane's holder — that wait is excluded from every number below (the suite's own start/end markers bracket only the bash tests/test_lane_helpers.sh invocation itself). Separately, at least three long-lived bash tests/test_lane_helpers.sh processes from other lanes (two later confirmed orphaned, ppid=1) were observed running on this workstation around the guard/lock window; I could not confirm whether they were still alive during my actual measurement window (02:30:33Z–03:26:16Z is the ~4-hour gap in which my own agent process was killed by a usage-limit reset and could not observe the host). Read the wall-clock figures as a plausible upper bound on isolated cost; git-spawn counts, per-call git seconds and the subcommand mix are far less sensitive to contention (they're sums of this run's own measured intervals), and relative proportions across sections/subcommands should still be informative.
    • A methodology bug caught and fixed in review: a later, unrelated microbenchmark (run ~2h after the suite finished, reusing the same shim dir on purpose) left 300 rows in the shared git-call log with timestamps after suite_end; an unguarded "clamp to last bucket" fallback in the analysis script silently misattributed them to the post-footer row. Fixed by dropping any shim-log row outside [suite_start, suite_end] before bucketing (0 such rows remained in-range afterward). Flagged here because it's exactly the kind of silent error this kind of report should not hide.

    Suite totals

    Wall (suite start → process exit) 3342.68s (55m43s)
    — of which: section content (start → last stdout line) 1469.13s (24m29s)
    — of which: post-footer, zero-stdout gap (last line → process exit) 1873.55s (31m14s), 56.0% of total wall
    git spawns 50,419
    git seconds (sum of real git execution time) 180.44s — 5.4% of total wall, 12.3% of content wall
    sleep spawns 5
    sleep seconds 2.63s — negligible
    unaccounted, within content wall only (bash/sed/awk/jq churn + my ≈116s shim overhead) 1286.06s — 87.6% of content wall

    Exit code was 1: 2541 passed, 1 failed, 0 skipped, 0 pending. The one failure —

    FAIL without --confirm nothing is asked, which is today's behaviour
         expected [0], got [5]
    

    — is tests/test_lane_helpers.sh:2985, inside the == Amendment 11 hotfix …== section. It is a clean exit-code mismatch (not a timeout/race shape), everything else in a 2542-assertion run passed, and correctness is out of scope for this profiling task — flagging it for the owning lane rather than diagnosing it further.

    Per-section table

    section wall (s) git# git s sleep s
    (preamble, before first section) 0.35 18 0.17 0.05
    lane-start 25.40 1890 6.38 0.00
    lane-start: Amendment 6 16.07 1207 2.78 0.00
    lane-end 9.00 569 2.93 0.00
    Amendment 7: the per-lane object log 83.90 7081 25.11 0.00
    Addendum 5: R17–R23 24.11 2681 8.16 0.00
    Addendum 6: R24–R26 11.91 885 3.97 0.00
    Addendum 7: R29–R30 4.92 703 2.15 0.00
    Addendum 8: R33–R35 12.69 906 3.72 0.00
    Amendment 8: swap and restart 16.27 1126 2.75 0.00
    Amendment 11 hotfix (R-A11-1, --name) 105.20 5962 23.21 2.58
    Amendment 11: record's fields, clause (h) 81.41 4788 18.69 0.00
    Amendment 11: cross-lane write clause (d) 15.64 712 2.84 0.00
    Amendment 11 decision 7: lane <name> 14.89 502 1.59 0.00
    Amendment 11 decision 6: lanes 181.75 2985 9.03 0.00
    Amendment 12: name guard and the lock 37.42 2481 6.95 0.00
    Amendment 15: one name under any case 56.08 3633 11.74 0.00
    Amendment 17: the handoff is the swap 68.14 4326 21.23 0.00
    Amendment 18 Addendum 1: lane, numbered pick, three branches 657.22 5509 15.40 0.00
    Amendment 13: row is state, log is history 39.89 1626 8.62 0.00
    the workstation seam 6.87 829 3.01 0.00
    (after the last stdout line — EXIT trap: kill 4 liveness fixtures, rm -rf "$SANDBOX") 1873.55 0 0.00 0.00

    Two content sections are wildly disproportionate to their own git-call count — Amendment 18 Addendum 1 (657s wall for 5509 calls / 15.4 gitS — 10x the per-call overhead of, say, Amendment 7's 7081 calls / 83.9s) and Amendment 11 decision 6 lanes (182s wall for 2985 calls / 9.0 gitS). Together they're 57% of all content wall from 2 of 20 sections. Could be genuinely heavier per-call work (numbered-pick/branch resolution across real worktrees is plausible for the first one), could be where workstation contention happened to concentrate — this run can't distinguish the two; worth an isolated re-measurement of just these two sections if the owning lane wants a firm answer.

    Top git subcommands

    subcommand count % of calls seconds % of git time
    show 19,992 39.6% 52.08 28.9%
    rev-parse 16,023 31.8% 38.74 21.5%
    remote 2,806 5.6% 3.54 2.0%
    diff 2,577 5.1% 7.49 4.2%
    cat-file 2,015 4.0% 5.48 3.0%
    ls-tree 1,671 3.3% 4.86 2.7%
    ls-remote 1,541 3.1% 9.80 5.4%
    fetch 1,175 2.3% 16.20 9.0%
    commit 500 1.0% 4.98 2.8%
    add 499 1.0% 2.05 1.1%
    push 484 1.0% 13.20 7.3%
    pull 470 0.9% 15.97 8.9%
    log 301 0.6% 3.33 1.8%
    rev-list 224 0.4% 1.20 0.7%
    init 38 0.08% 0.53 0.3%
    clone 5 0.01% 0.58 0.3%

    show+rev-parse alone are 71.4% of every git spawn in the suite (36,015 of 50,419) and 50.3% of git time — almost certainly the assertion helpers (is/has/hasnt) reading committed content and resolving refs to compare against expected strings. Per-call cost is telling: show/rev-parse average ≈2.5ms; fetch/pull/push average 12–34ms (10–15x more per call) — the network-shaped plumbing is intrinsically heavier even against local bare repos, but there are two orders of magnitude fewer of them.

    What the numbers say

    1. By far the biggest lever, and not one of the four listed: the post-footer rm -rf "$SANDBOX" teardown, ≈31m14s, 56% of total wall. All 2542 assertions finish and the full pass/fail footer prints at content-end; the process itself doesn't exit for another 31 minutes with zero further stdout, zero further git calls, zero further sleep — only the EXIT trap's 4 (near-instant) kills and one rm -rf of a sandbox that accumulated files across 50,419 git calls remain. This host's /tmp is a deeply-layered container overlayfs (100+ lowerdir entries, overlay on / and /tmp alike) on top of WSL2 — exactly the shape of filesystem where bulk unlink of many small files is disproportionately slow, and load average: 4.15 at time of writing confirms real concurrent host load too. Estimated saving: most of that ~31 minutes, by backing the sandbox with tmpfs instead of the default disk-backed mktemp -d location. A tmpfs exists here (/dev/shm, but only 64MB — too small to size this without measuring the sandbox itself, which was already deleted by the time this was investigated); a CI image could mount one sized to the sandbox's actual footprint. This isn't literally issue #77's "one sandbox per section" idea, but it's the same diagnosis from a different angle: it's not the number of sandboxes, it's what medium the one (or many) live on and get torn down from. Splitting into per-section sandboxes without addressing the storage medium would just spread the same total unlink cost around rather than cut it.

    2. Second lever: the bash/sed/awk/jq churn around the git calls, not the git calls themselves — ≈1286s, 87.6% of content wall. Git execution is only 12.3% of content time even before subtracting shim overhead. This is what direction 2 ("splitting for parallel pytest") would actually attack, since it's CPU-bound per-assertion scripting that should parallelize across sections about as well as the sections turn out to be independent of each other's state (not measured here — a code-structure question, not a timing one). Estimated saving: proportional to parallelism achieved (e.g., 4-way parallel content sections ≈ content wall/4 plus fixed overhead, if state independence holds) — the more promising of the two real levers to chase after the sandbox fix.

    3. Git config tweaks (direction 3): checked and not worth it. The sandbox sets no gpgsign/gc.auto/fsmonitor/protocol.version/template-dir — nor does this operator's global config (git config --global --get on all four returns "not set"). commit already averages ≈10ms (500 calls, 4.98s total) confirming gpgsign is already off in practice; protocol.version=2 has been git's default since 2.26 (this box runs 2.43.0), so setting it explicitly is a no-op; gc.auto's default 6700-loose-object threshold is never plausibly reached by these tiny per-test repos (500 commits total across the whole suite, spread over many repos); a shared template dir would save, at absolute most, the 0.53s spent across all 38 git init calls. Even zeroing out ALL git time entirely saves only 5.4% of total wall. Do not spend effort here.

    Not worth bothering with at all: the sleep fixtures (direction 4). 5 real sleep calls, 2.63s total. The sleep 3000 & liveness fixtures (4 of them) are backgrounded and killed at cleanup before they ever complete naturally — they cost the suite nothing (confirmed: 0 appear in the completed-calls log). The only two non-trivial ones are lanes-edit.sh:762 (sleep 1) and :1434 (sleep 3, cut short at ~0.53s — consistent with a deliberate SIGTERM-interrupt test in the hung-push/mutex code, not a bug). This whole direction is a non-issue.

    macOS-vs-Linux scaling, since the issue asks specifically: process-spawn-bound cost (the ≈50K git forks and the surrounding bash/sed/awk/jq forks, together ≈94.6% of the wall time excluding the teardown gap) is exactly the class that plausibly scales with macOS's documented ~2.7x fork/exec penalty — consistent with the issue's 45–50min macOS vs ~17min Linux baselines. sleep does not scale that way (it's already ~0 cost, backgrounded, everywhere). The post-footer bulk-teardown cost is a third, distinct class that this run puts front and center: it scales with the filesystem's bulk-unlink performance (APFS on macOS vs. whatever backs $TMPDIR on the Linux runner), which is a different axis than fork/exec and not necessarily the same 2.7x — worth its own measurement on macOS specifically, since if APFS handles bulk unlink of many small files poorly too, it could be an even larger fraction of the 45–50 minute macOS figure than the 56% measured here.


    Measured on one Linux workstation run under real contention (see caveats above); raw shim logs (50,419 git-call rows, per-section boundaries) available on request from lane openRepoTools-1's scratch if useful for a follow-up.

  5. brettheap commented on Oct 6, 2026

    @brettheap
    ContributorAuthor

    RELEASED — lane openRepoTools-1, session df82f553-7803-49c4-8ddf-3f9acdcb8ee0@Eagle, 2026-10-06T17:40:59Z, for #77 — Brett Heap's word of 2026-10-06 (release 77 to lane 3): the measurement this lane owed is on the issue (issuecomment-5692549523: 56% of Linux wall is the exit-trap sandbox rm -rf; git spawns 5%; sleeps nil; teardown first; then parallel sections); the fix is the suite owner's; lane openRepoTools-3 claims it

  6. brettheap commented on Oct 6, 2026

    @brettheap
    ContributorAuthor

    CLAIMED — lane openRepoTools-3, session 83619980-06bb-4469-ae73-2a8c5e61981a@Eagle, 2026-10-06T17:42:32Z, for #77

    Logged in lanes/log/openRepoTools-3.md at 1396e6ed8f2b5c37b049f42450d11a4ec41dff98.

    The three reads (lane-collision-protocol Rule 1):

    1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    6:CLAIMED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-15T04:06:34Z, for opensoft/openRepoTools#77
    13:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    26:RELEASED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-16T00:36:27Z, for opensoft/openRepoTools#77
    33:CLAIMED — lane openRepoTools-1, session df82f553-7803-49c4-8ddf-3f9acdcb8ee0@Eagle, 2026-09-16T01:07:10Z, for opensoft/openRepoTools#77
    40:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    41:6:CLAIMED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-15T04:06:34Z, for opensoft/openRepoTools#77
    42:13:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    43:26:RELEASED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-16T00:36:27Z, for opensoft/openRepoTools#77
    166:RELEASED — lane openRepoTools-1, session df82f553-7803-49c4-8ddf-3f9acdcb8ee0@Eagle, 2026-10-06T17:40:59Z, for opensoft/openRepoTools#77 — Brett Heap's word of 2026-10-06 (release 77 to lane 3): the measurement this lane owed is on the issue (issuecomment-5692549523: 56% of Linux wall is the exit-trap sandbox rm -rf; git spawns 5%; sleeps nil; teardown first; then parallel sections); the fix is the suite owner's; lane openRepoTools-3 claims it
    2. `gh pr list --repo opensoft/openRepoTools --state all --search "#77"`, kept where the row or its body names `#77`:
    83	One binding per lane: the record says host/os/container, liveness is pronounced only from inside that binding, and a second place ASKS for the handoff before it binds (Amendment 18 acts 1-10)	feat/amendment-18-one-binding	MERGED
    103	`--install` writes a receipt of the files it placed, and a retirement reads that before it reads a header substring	feat/install-receipt	MERGED
    45	`lane`: one word lists the lanes and binds the one you pick — available goes through the launcher in its own directory and profile, LIVE HERE attaches and never starts a second process, bound elsewhere names where; and `restart` leaves the person's PATH in the same breath (Amendment 18 Addendum 1 and Addendum 2)	feat/lane-the-word	MERGED
    3. `git ls-remote --heads git@github.com:opensoft/openRepoTools` branches naming `77` as a whole token:
       none
    
  7. brettheap commented on Oct 6, 2026

    @brettheap
    ContributorAuthor

    Deferred under Brett Heap's close-out direction of 2026-10-06 (no new scope until the existing PRs land). The writer that was about to start found, before writing anything:

    • No usable tmpfs on Eagle: /dev/shm is 64 MB and mounted noexec, and the suite runs scripts from its sandbox, so a tmpfs-backed TMPDIR is not a lever here; faster (deferred) teardown and parallel sections are. Any tmpfs check must prove it can execute a script there, not only look at free space.
    • The empty worktree .claude/worktrees/suite-speed (branch feat/shell-suite-wall-time, no commits of its own, upstream unset) is left in place for whoever resumes this.

    Lane: openRepoTools-3

  8. brettheap commented on Oct 13, 2026

    @brettheap
    ContributorAuthor

    RELEASED — lane openRepoTools-3, session 83619980-06bb-4469-ae73-2a8c5e61981a@Eagle, 2026-10-13T18:24:13Z, for #77 — Brett Heap 2026-10-11 gave #28-#89 to lane openRepoTools-5 (its RULED line of 16:19:19Z); lane 3 holds no live work on this number

  9. brettheap commented on Oct 18, 2026

    @brettheap
    ContributorAuthor

    CLAIMED — lane openRepoTools-5, session 0d83dca2-c18c-4bb0-99ce-896e6cafea02@Eagle, 2026-10-18T20:16:18Z, for #77

    Logged in lanes/log/openRepoTools-5.md at 47eaf6e99a713a2f6e374696c8a72e17a3e388ae.

    The three reads (lane-collision-protocol Rule 1):

    1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    6:CLAIMED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-15T04:06:34Z, for opensoft/openRepoTools#77
    13:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    26:RELEASED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-16T00:36:27Z, for opensoft/openRepoTools#77
    33:CLAIMED — lane openRepoTools-1, session df82f553-7803-49c4-8ddf-3f9acdcb8ee0@Eagle, 2026-09-16T01:07:10Z, for opensoft/openRepoTools#77
    40:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    41:6:CLAIMED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-15T04:06:34Z, for opensoft/openRepoTools#77
    42:13:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    43:26:RELEASED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-16T00:36:27Z, for opensoft/openRepoTools#77
    166:RELEASED — lane openRepoTools-1, session df82f553-7803-49c4-8ddf-3f9acdcb8ee0@Eagle, 2026-10-06T17:40:59Z, for opensoft/openRepoTools#77 — Brett Heap's word of 2026-10-06 (release 77 to lane 3): the measurement this lane owed is on the issue (issuecomment-5692549523: 56% of Linux wall is the exit-trap sandbox rm -rf; git spawns 5%; sleeps nil; teardown first; then parallel sections); the fix is the suite owner's; lane openRepoTools-3 claims it
    173:CLAIMED — lane openRepoTools-3, session 83619980-06bb-4469-ae73-2a8c5e61981a@Eagle, 2026-10-06T17:42:32Z, for opensoft/openRepoTools#77
    180:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    181:6:CLAIMED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-15T04:06:34Z, for opensoft/openRepoTools#77
    182:13:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    183:26:RELEASED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-16T00:36:27Z, for opensoft/openRepoTools#77
    184:33:CLAIMED — lane openRepoTools-1, session df82f553-7803-49c4-8ddf-3f9acdcb8ee0@Eagle, 2026-09-16T01:07:10Z, for opensoft/openRepoTools#77
    185:40:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    186:41:6:CLAIMED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-15T04:06:34Z, for opensoft/openRepoTools#77
    187:42:13:1. existing `CLAIMED —` comments on opensoft/openRepoTools#77:
    188:43:26:RELEASED — lane openRepoTools-3, session 4135b2c9-34a6-47e3-a156-5adebb632fae@Eagle, 2026-09-16T00:36:27Z, for opensoft/openRepoTools#77
    189:166:RELEASED — lane openRepoTools-1, session df82f553-7803-49c4-8ddf-3f9acdcb8ee0@Eagle, 2026-10-06T17:40:59Z, for opensoft/openRepoTools#77 — Brett Heap's word of 2026-10-06 (release 77 to lane 3): the measurement this lane owed is on the issue (issuecomment-5692549523: 56% of Linux wall is the exit-trap sandbox rm -rf; git spawns 5%; sleeps nil; teardown first; then parallel sections); the fix is the suite owner's; lane openRepoTools-3 claims it
    214:RELEASED — lane openRepoTools-3, session 83619980-06bb-4469-ae73-2a8c5e61981a@Eagle, 2026-10-13T18:24:13Z, for opensoft/openRepoTools#77 — Brett Heap 2026-10-11 gave #28-#89 to lane openRepoTools-5 (its RULED line of 16:19:19Z); lane 3 holds no live work on this number
    2. `gh pr list --repo opensoft/openRepoTools --state all --search "#77"`, kept where the row or its body names `#77`:
    83	One binding per lane: the record says host/os/container, liveness is pronounced only from inside that binding, and a second place ASKS for the handoff before it binds (Amendment 18 acts 1-10)	feat/amendment-18-one-binding	MERGED
    103	`--install` writes a receipt of the files it placed, and a retirement reads that before it reads a header substring	feat/install-receipt	MERGED
    45	`lane`: one word lists the lanes and binds the one you pick — available goes through the launcher in its own directory and profile, LIVE HERE attaches and never starts a second process, bound elsewhere names where; and `restart` leaves the person's PATH in the same breath (Amendment 18 Addendum 1 and Addendum 2)	feat/lane-the-word	MERGED
    3. `git ls-remote --heads git@github.com:opensoft/openRepoTools` branches naming `77` as a whole token:
       none
    
  10. brettheap commented on Oct 19, 2026

    @brettheap
    ContributorAuthor

    Lane openRepoTools-5, 2026-10-19T16:31Z: step 1a's first numbers, from a green Linux CI log (#261 landed as b707bd7; #265, step 1b, makes the wrapper print them on a pass).

    This is draft PR #265's tests job: run 39276800512, job 118143094255, at head 458f831 over main 55aae16. The job reported 1958 passed, 51 skipped in 2695.34s overall, and the shell suite is one of those tests.

    wall time per section, longest first (whole seconds; opensoft/openRepoTools#77):
       191 s   15%  Amendment 18 Addendum 1: the word `lane`, the numbered pick, the three branches
        99 s    8%  Amendment 11 hotfix: R-A11-1, --name on every branch, unknown refused
        99 s    8%  Amendment 14: the derived index is never read by an act
        98 s    8%  Amendment 17: the handoff is the swap
        82 s    6%  Amendment 11: the record's fields, and clause (h)'s reads
        74 s    6%  Amendment 7: the per-lane object log
        65 s    5%  Amendment 11 decision 6: `lanes`
        63 s    5%  Amendment 18: ONE BINDING PER LANE — host/os/container, the ask, the wait, the force
        59 s    4%  the managed-owner seam: managed ledger owns enrolled lanes; this tooling owns legacy
        46 s    3%  Amendment 12: the name guard and the lock
        42 s    3%  openRepoTools#91: the crash-consistent lifecycle and the worktree inventory
        38 s    3%  opensoft/openRepoTools#163: lane-end's close-out gate, and lane-worktrees add
        35 s    2%  lane-start
        33 s    2%  Amendment 15: a lane name is one name under any case
        31 s    2%  Amendment 16: a lane is renamed by one word, in one commit
        21 s    1%  Addendum 5: R17–R23
        19 s    1%  lane-start: Amendment 6 — resume the row's recorded session
        17 s    1%  Amendment 13: the row is current state, the log is history
        15 s    1%  Amendment 19: a closed or dormant lane leaves the listing
        13 s    1%  Amendment 11: the cross-lane write clause (d) rule 1 forbids
        13 s    1%  Amendment 8: swap and restart
        11 s    0%  Addendum 8: R33–R35
        10 s    0%  Addendum 6: R24–R26, and the lifecycle reads
        10 s    0%  Amendment 11 decision 7, as Amendment 18 Addendum 2 leaves it: `lane <name>`
         9 s    0%  lane-end
         8 s    0%  the workstation seam: unset, every writer reads the host
         5 s    0%  Addendum 7: R29–R30
         1 s    0%  opensoft/openRepoTools#162: bytecode never rides a workspace commit
         0 s    0%  (preamble, before the first heading)
      1207 s  100%  the whole run, from the top of this file to this table
    teardown: rm -rf of the sandbox took 1 s
    

    What the numbers say, measurement only:

    • On Linux CI the shell suite took 1207 s (about 20 min) of the job's 45. In The shell suite's wrapper prints the suite's timing table and teardown line on a pass, where CI's log can read them (#77, step 1b) #265's first CI run (head d14e482) it took 1466 s.
    • The teardown is not the cost on CI. rm -rf of the sandbox took 1 s, and 0 s in the first run. The 56% that lane 1 measured in the exit trap was on WSL's filesystem, so that finding is WSL-only.
    • No single section dominates. The longest is Amendment 18 Addendum 1 at 191 s (15%), and the top four together are 39%. A real speed-up would need a split of the suite, not a single fix.
    • A parallel run would need a pytest -n dependency, or sharding the sections into separate jobs. That is the direction question this issue left for Brett Heap after the measurement. Lane 5 starts nothing on it without his word.

    Lane: openRepoTools-5

  11. changed the title [-]The shell suite takes 45 minutes on macOS — the cap it just crossed is not the fix, its duration is[/-] [+]CI's Linux test job takes 46–72 minutes a run, because its Python tests and the shell suite run one after the other[/+] on Oct 23, 2026
  12. brettheap commented on Oct 23, 2026

    @brettheap
    ContributorAuthor

    Retitled on Brett Heap's word of 2026-10-23T15:16:44Z, given by multiple choice: "pytest-xdist -n 3 (Recommended)" for the first step, and "Later, after measuring (Recommended)" for splitting the shell suite. The macOS cap this issue was filed about is solved. What is left is the Linux job's wall. CI on 2026-10-19 and 2026-10-20 measured 46–72 min a run, with the Python tests about 55% of it and the shell suite about 45%, run one after the other.

    The steps, in order:

    1. Run the Python tests three-wide under pytest-xdist (pinned, CI only), with the shell suite's test started first. Lane openRepoTools-5 builds this as a v1.2.0 branch, fix/ci-python-tests-three-wide-77.
    2. Measure three runs. Only if the wall still hurts, split the shell suite into section groups inside its one test.

    Dropped: the git-config levers. All git time was 5.4% of the suite's wall (180 of 3343 s), and their saving is about zero. Local tests/run.sh stays serial.

    Lane: openRepoTools-5

  13. brettheap commented on Oct 23, 2026

    @brettheap
    ContributorAuthor

    Three wide, measured. The commit is a79e05d on lane openRepoTools-5's branch fix/ci-python-tests-three-wide-77: pytest-xdist 3.8.0, -n 3 --dist loadfile --no-loadscope-reorder, with the shell suite's module handed out first. It ran three times on the branch, and all four jobs were green each time.

    Run tests tests-no-submodule Shell suite's own clock pytest's wall in tests
    40008603941 36m37s 32m01s 2175 s 2185 s
    40013612982 38m18s 36m44s 2272 s 2282 s
    40013631641 25m42s 35m09s 1516 s 1524 s

    For comparison, two serial tests runs the same day took 50m23s (run 40001227949) and 72m13s (run 40008392449, which overlapped runs 2 and 3).

    What the runs show:

    • The shell suite is the whole critical path. In every run its own clock is within 10 s of pytest's wall, so the Python files finish inside it.
    • Contention costs part of the saving. Beside two other workers the suite runs slower than it does alone: 1516–2272 s here, against the 1208–1815 s measured serially on 2026-10-19 and 2026-10-20. That is why three wide saves about a quarter to a half of the job, and not more.
    • A warning to watch. Each run printed three DeprecationWarnings from Python 3.14's forkpty() in tests/test_run_wrapper.py: "This process is multi-threaded, use of forkpty() may lead to deadlocks". An xdist worker runs execnet threads. No run hung.

    The next step is Brett Heap's, per his ruling of 2026-10-23T15:16:44Z ("Later, after measuring"). The question is whether to shard the shell suite into section groups inside its one test. That is the only lever left on the critical path.

    Lane: openRepoTools-5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions