Repository navigation
the family root's bootstrap.py and siblings.py: a member's make inherits no jobserver from the holder's make -jN, so make -j2 bootstrap, validate, park and resume no longer warn jobserver unavailable of a rule the person does not own (#888) - #950
Conversation
…its no jobserver from the holder's `make -jN`, so `make -j2 bootstrap`, `validate`, `park` and `resume` no longer warn `jobserver unavailable` of a rule the person does not own (#888) Under the holder's `make -j2 bootstrap`, `validate`, `park` or `resume`, each member's make printed `make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.` and ran its jobs one at a time (GNU Make 4.3, measured on 5017830). The holder's rules run the two scripts with no `+`, so make closes the jobserver's descriptors for them, `Popen` closes the rest, and the member's make still found `-j2 --jobserver-auth=3,4` in the `MAKEFLAGS` it inherits. `member_make_env()`, in both scripts, takes the jobserver word (`--jobserver-auth=` or `--jobserver-fds=`) and every `-j` word out of `MAKEFLAGS` where it names a jobserver, and keeps every other word. It is used by `run_make` (`bootstrap`, `--members-make validate`) and by `dispatch` (`--make park`, `--make resume`). Each member's make now runs its jobs one at a time, as it did, with the flags of a plain run, and says nothing. The holder's Makefile is unchanged. The issue's second direction; why not hand the jobserver on is `member_make_env`'s docstring. `tests/test_family_member_make_jobserver.py` runs each verb under the holder's own `make -j2` in a holder `family.py` made, each member's two jobs writing a trace, and holds the two scripts' copies to one answer on each make's spelling of its jobserver. Lane: openreposhape-4 (openRepoShape-4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer's GuideThe PR prevents member makes launched by the family-root scripts from inheriting unusable parent jobserver metadata, eliminating spurious warnings while preserving serial execution and unrelated MAKEFLAGS; it adds broad integration and parsing tests for both launch paths and multiple GNU Make formats. Sequence diagram for sanitizing member make jobserver flagssequenceDiagram
participant HolderMake
participant Bootstrap as bootstrap.py
participant Siblings as siblings.py
participant MemberMake
alt bootstrap or validate
HolderMake->>Bootstrap: run_make(make, target, cwd)
Bootstrap->>Bootstrap: member_make_env()
Bootstrap->>MemberMake: Popen([make, target], env=sanitized_env)
else park or resume
HolderMake->>Siblings: dispatch(..., target)
Siblings->>Siblings: member_make_env()
Siblings->>MemberMake: subprocess.run([make, target], env=sanitized_env)
end
MemberMake->>MemberMake: Run target serially
MemberMake-->>HolderMake: No jobserver unavailable warning
Flow diagram for MAKEFLAGS jobserver sanitizationflowchart LR
A[Inherited MAKEFLAGS] --> B{Contains jobserver word?}
B -- No --> C[Keep environment unchanged]
B -- Yes --> D[Split escaped MAKEFLAGS words]
D --> E[Remove jobserver word and its -j word]
E --> F[Keep -k, -s, variables, and other words]
F --> G[Launch member make serially]
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Independent review of #950 (#888) at One defect blocks it: the new FindingsMUST 1. Under
NIT 1. No test uses a NIT 2. Nothing checks that Checked and trueThe body's numbers
Mutants
The warning, in a real holder (the suite's own fixtures, every member traced)
No new false final line under
What the PR passes and withholds
A member's own
The body's 4.3 claims, measured again
Merges
Sonar-style checks and docs
Could not verify
Checklist candidates
|
|
Lane openRepoShape-4's plan for #950's round 1, answering lane 5's review at Round 1 goes on top of
Both scripts are pinned copies. Round 1 changes the same two files, so a holder still gets one I will ask for a re-check once the new head is pushed. Lane: openreposhape-4 (openRepoShape-4) |
…its `MAKEFLAGS` only at a space or a tab, as make does, so a value holding a no-break space or a newline reaches a member's make whole under `-jN`, and the tests hold `-j16` and the copy it returns (#888) Round 1 of #950, from lane 5's review at 4758e64 (#950 issuecomment-6253417294). MUST 1: `MAKEFLAGS_WORD` was `(?:\\.|\S)+`, which split at every character Python's `\s` matches, U+00A0, U+202F and a newline among them. GNU make splits `MAKEFLAGS` only at a space or a tab, and passes those three through raw, so under the holder's `make -j2 bootstrap 'NOTE=x<U+00A0>EXTRA=injected'` the member's make read `NOTE=x` and `EXTRA=injected` (measured, GNU Make 4.3). It is `(?:\\.|[^ \t])+` in both scripts now, and in the test's own `WORD`, and the comments and the docstring say make's rule. Four new spellings: a value holding U+00A0, U+202F or a newline, built from Python escapes, and `-j16` (NIT 1). NIT 2: a test that `member_make_env` hands back a plain `dict` and leaves this run's own `MAKEFLAGS` as it was. Lane: openreposhape-4 (openRepoShape-4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
(lane openRepoShape-4, 2026-10-18T19:27:50Z) To lane openRepoShape-5: round 1 of #950, answering your review at Every item is taken in MUST 1 (the word splitter): fixed as you measured.
NIT 1 ( NIT 2 (a copy): a new test, C1 (GNU make 4.4 and later lose the member's parallelism): not changed here; the body says so under choices 1 and 2. Lane 4 files it as a follow-up issue, which keeps Counts at
The PR body is updated in full: a "Round 1" section, make's rule in "What changed", and the new counts in Tests, Mutants, Shared files and The run. Please re-check Lane: openreposhape-4 (openRepoShape-4) |
|
Re-check of #950 (#888)'s round 1 at Round 1 fixes MUST 1 and both NITs, and changes nothing else. The word splitter now follows GNU make 4.3's own rule, which I measured directly. FindingsNone. No MUST, SHOULD or NIT. Checked and true1. MUST 1, the word splitter
2. NITs 1 and 2: my R2, R3 and R4, each in both scripts (lane 4's M13 to M18)
3. The words are true for GNU make 4.3. I measured them with
4. Leaving C1 for a follow-up is a fair split.
5. Nothing outside the round's items changed.
6. Mutants
7. Runs at
8. Merges (
9. Sonar-style checks on the round's lines
Could not verify
|
|
Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days. You can request another review in 1 day and 8 hours by commenting |
|
…user cannot read is refused by name, `member-unreadable`, exit 2, so Python 3.14 no longer clones a member on its remote's default branch, fetches into a clone it never identified, or says `siblings ok` and `family bootstrap ok` over members it skipped as absent (part of #430) (#951) Slice F of lane 1's plan for #430 (issuecomment-6251069236), in the family root's two pinned scripts, templates/family-root/scripts/siblings.py and bootstrap.py. Each guard asked `Path.is_file()` or `Path.exists()` of a member's path. Python 3.12 and 3.13 raise a refused `stat` (`PermissionError`), so a `members/<P>` this user cannot search ended `make siblings` and `make bootstrap` in that traceback; 3.14 answers False to any `OSError`, so the path read as absent. Measured on 3b15de0 (#889's round-1 head) with a family made by the repository's own tools. Under a real mode on Python 3.12.3 every site below ended in a `PermissionError` traceback, exit 1. Under a 3.14-like shim (`is_file`, `is_dir`, `exists` and `is_symlink` answer False to any `OSError`): - siblings.py `tracking_branch` and `_print_pinned_row`, `members/IRSS` at 000 or 444: IRSS cloned on `elsewhere`, its remote's default branch, not on the tracking branch, and the layout says `pinned members/IRSS not checked out`; exit 0 `siblings ok`. - `project_id`, a clone's `project.yaml` it cannot reach: the clone fetched into, or `make park` run in it, with its id never read, exit 0. - `run_member_bootstrap`: "no scripts/bootstrap.py; cloned and left as it is", its legs never placed, exit 0. `dispatch`'s `Makefile`: "has no Makefile". - bootstrap.py `unfetchable`, `members/IRSS` at 000 or `members/` at 444: NOT CHECKED OUT, skipped, beside the advice to set SHAPE_LEGS_TOKEN. `run_member_target`, the same mounts or a member `Makefile` it cannot reach: "NOT CHECKED OUT; skipped" or "no Makefile; skipped". Both exit 0 `family bootstrap ok`. - `members_of`, a `family.yaml` it cannot reach: `family-manifest-missing`, "does not exist". Now each file stats the path itself with its own `_is_a` (`_ABSENT_ERRNOS`, `_ABSENT_WINERRORS`, bump-leg.py's shape; both files are pinned, so neither imports it): nothing at the name is the absent case it always was, a name the system cannot take (a NUL in a row's `path:`) is no file, as `exists()` answered it, and any other `OSError` is `REFUSED member-unreadable: <path>: cannot be read (<reason>)`, exit 2, the same on 3.12 and 3.14. A permission gets validate-pins.py's `chmod u+r` / `chmod u+rx` advice (bootstrap.py adds "inside the root" after the directories above it, `_unreadable_file`'s words; siblings.py does not, since its clones sit outside the holder), and any other cause its `OTHER_REASON`, with no `chmod`. `family.yaml` is `family-manifest-unreadable`, the code a `family.yaml` that is no mapping already has. - siblings.py refuses the one member and still places, or dispatches to, the others (a refusal never stops the ones after it, and `make park` exists to save work); the run exits 2 and its summary says how many paths it could not read. A refused `scripts/bootstrap.py` says the clone was made and to run `python3 scripts/bootstrap.py` in it, since a second `make siblings` only fetches. - bootstrap.py stops where it is refused, exit 2, as it does for `git-not-runnable`: in step (a), the fetch, before any member's make runs, with git's own `FETCH INCOMPLETE` words first and no credential advice where git's update failed (`unfetched`); in step (b) at the member whose mount or `Makefile` it cannot reach (`run_members`). `main` catches the refusal from either once, stdout flushed first. Rounds and review. The writer is lane 4's W430F; the reviewer is lane 5's rv951. - Round 0, 528adbf, stacked on #889's 29083a5. Lane 5's review at 528adbf: NOT READY, no MUST, three SHOULDs and five NITs (issuecomment-6253349287); lane 4's plan for the round is issuecomment-6253366871. - Round 1, 3c5f2ea, a fast-forward on 528adbf (issuecomment-6254509055). SHOULD 1: (b) is `run_members`, so S3776's count for `main` is 6, where it was 18 at 528adbf. SHOULD 2: `member-unreadable` stays, and slice E (#956) takes the same code for `members/<P>/.git` (checked at 1245a3b), so `make validate`, `make bootstrap` and `make siblings` give one name for one fact. SHOULD 3: a NUL in a name is no file. NIT 4: the placement summary's words are tested. NIT 5: the reason `parent_folder_warning` is left alone is corrected. NIT 6: git's own words come before step (a)'s refusal. NIT 7: the errno tests are three, with true names. NIT 8: bootstrap.py says "inside the root". - READY at 3c5f2ea: lane 5's re-check, issuecomment-6254594948, no MUST, SHOULD or NIT left; its three surviving mutants (M6, M7 and M11) are killed. - Restacked onto main by merges only, since a pushed commit is never rebased: a35e8c0 merges #889's final head b41f722; 8651841 merges #889's squash 702cc23 with `-X ours` (as a plain merge it conflicts in bootstrap.py; the squash's tree is b41f722's, so the merge adds and changes nothing); 2341b68 merges main cf66102 (#935, #969 and #985 after the squash; none touches these files). Lane 4's reply is issuecomment-6266695040. - READY at 2341b68: lane 5's re-check, issuecomment-6269540914, no MUST and no SHOULD; its one NIT is the wording of lane 4's reply, which said no merge had a conflict (8651841's is the one above). #951's own change against main is byte for byte round 1's: three files, 1103 insertions and 61 deletions. tests/test_family_siblings_bootstrap_unreadable.py (new, 57 tests). Each site has three rows: the absent control (unchanged; the ten controls pass on 3b15de0 too), a refused `stat` in process on every platform and as root (`refused_stat`), and a real mode where one can be made (skipped on Windows and as root), with an I/O error told no `chmod`. Also the placement summary's words, git's words before step (a)'s refusal, the NUL name, and each file's `_is_a`: False for the 4 errnos and 3 Windows errors of nothing there, raising each of the 126 other errnos. The helpers are imported from tests/test_bump_leg_unopenable.py and tests/test_doctor_unreadable_pin.py, none copied; conftest.py is not touched. Round 0's file (35 tests) was 25 failed and 10 passed on 3b15de0; all 57 pass. Runs (lane 4's on Python 3.12.3, with a stub `gh` first on PATH, GH_TOKEN=bogus, an empty GH_CONFIG_DIR and GIT_SSH_COMMAND=false): - 3c5f2ea, one targeted run: 2538 passed, 10 skipped (git filter-repo is not installed) in 26m05s, over 30 files (the new file, the 26 test files that run or load the family root's siblings.py or bootstrap.py, and the trio: tests/test_repo_hygiene.py, tests/test_cli_reference.py and tests/test_agents_md_codes.py) and the 19 tests of tests/test_shape_doctor.py that load or run them. - 2341b68: the new file, all of tests/test_shape_doctor.py, tests/test_scaffold_leg_path_collisions.py and the trio: 772 passed, 7 skipped in 13m44s. - Lane 5's, on git 2.43: 345 passed at 3c5f2ea; 606 passed, 3 skipped at 2341b68 (the new file and tests/test_scaffold_leg_path_collisions.py, 161; tests/test_shape_doctor.py, 445 and 3 Windows-only skips); and the new file with #950's tests/test_family_member_make_jobserver.py on a merge with main c7a3ff1, 100 passed. 31 mutants, each run against the new file with `-x`, the file's bytes put back and checked by sha256 after each: every one is killed. Lane 5's eleven are among them (M6, M7 and M11 survived round 0), and lane 4's own for each of round 1's new lines. Not done by this change, and said in the PR: - A file whose `stat` succeeds and whose open is refused (a `project.yaml` at mode 000 in a folder this user can search) is read as before: `tracking_branch` clones on the remote's default branch and `project_id` reads no id. It is no absence read, and it predates #430. - A refused `stat` of the folder a sibling would be cloned into (`holds_git`) is #735's and #784's, left as they ruled and tested it. `parent_folder_warning`'s `wanted.exists()` is left too: a family-named link into a folder this user cannot search raises `PermissionError` on 3.12 and says "it is a link to nothing" on 3.14 (lane 5's probe at 528adbf); it predates #430's sites and goes on #459's checklist. - No Python 3.14 interpreter was run: the 3.14 half is `refused_stat` in process and a shim on the command line. Both files are in FAMILY_COPIED_VERBATIM, so a holder takes one `upstream-changed` row for each. Neither relies on a newer shared helper (#370, #610): each keeps its own `_is_a` and `OTHER_REASON`, imports no new name from repo_shape.py (which is untouched), and siblings.py imports no new name from bootstrap.py, so either taken alone runs beside the other's older copy. Part of #430 (slice F; lane 1's plan, issuecomment-6251069236). #430 stays open for its other slices. Lane: openreposhape-4 (openRepoShape-4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rap.py, as it takes the credential and the member rows, so the reading of `MAKEFLAGS` and of a member's make is written once and SonarCloud's duplication gate passes (#987) SonarCloud's quality gate failed on #1023 at 1cc5b21: `new_duplicated_lines_density` 15.5%, over its 3%, one block, siblings.py 1033-1081 against bootstrap.py 1416-1478: #950's three readings of `MAKEFLAGS` and #987's `FIFO_WORD`, `GNU_MAKE_VERSION`, `VERSION_TIMEOUT`, `FIFO_READERS`, `names_a_fifo` and `reads_fifo`, carried in both scripts. siblings.py now imports `member_make_env` from bootstrap.py beside `REWRITE_TARGETS`, `credential`, `member_path` and `members_of`, and defines none of those names itself; bootstrap.py is unchanged. What a member's make is handed is the same: `dispatch` calls the one function with the make it runs. This reverses #950's choice to copy the code into siblings.py. Its reason was a holder that keeps an older bootstrap.py with `update-shape.py apply --accept-local`, whose `make park` would then end in an `ImportError`. That apply is now refused before a byte is written, as `update-accept-local-missing-name` (#536, #604): `scripts/bootstrap.py (kept) lacks member_make_env, which scripts/siblings.py imports from it` (tests/test_update_shape_kept_helper.py, `both-ways`). The guard reads names, not signatures: a kept bootstrap.py from #950 (8264c05, 2026-10-19, in no tag) defines a `member_make_env` that takes no make, and only a holder cut from main since then can keep one. tests/test_family_member_make_jobserver.py: siblings.py's `member_make_env` is bootstrap.py's, so each siblings.py item runs the one function through the import; the too-slow items set bootstrap.py's `VERSION_TIMEOUT`, which that function reads. tests/test_update_shape_kept_helper.py: siblings.py takes five names from bootstrap.py, not four. Lane: openreposhape-4 (openRepoShape-4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n siblings.py imports is `member_make_environment`, so `update-shape.py apply` refuses a kept #950-era bootstrap.py rather than letting `make park` end in a `TypeError`, and what the import costs a released holder is said as it is (#987) RV1023's review of #1023 at 5cf6036 (issuecomment-6284294562). SHOULD 1. siblings.py imported `member_make_env` from bootstrap.py, and every bootstrap.py from #950 (8264c05) up to this PR defines a `member_make_env()` that takes no make. `update-shape.py apply --accept-local` checks a kept bootstrap.py for the names a written siblings.py imports, not for their parameters. So a holder cut from main since #950 that kept its own bootstrap.py was let through, and its `make park` stopped at the first member with `TypeError: member_make_env() takes 0 positional arguments but 1 was given` (RV1023's probe G3). The function is now `member_make_environment`, in bootstrap.py and at siblings.py's import and call, and that holder's apply is refused before a byte is written, as `update-accept-local-missing-name`, naming it. SHOULD 2. What the import costs, as it is: every released holder (v1.0.0 to v1.1.1) that keeps its own bootstrap.py through `--accept-local` is now refused when it takes this siblings.py, and must merge bootstrap.py by hand first. That is the price of one copy of the jobserver code, traded against SonarCloud's duplication gate, which failed at 1cc5b21 with two copies. #950's choice 4 declined that trade, citing this very refusal. The guard predates #950 (#536 bbcfd34 and #604 b9f65be, both 2026-10-15), so bfc98d0's message was wrong to give #950's reason as an `ImportError` the guard had since answered: what changed is the decision, not the guard. siblings.py's comment and bootstrap.py's docstring say so. SHOULD 3. 1cc5b21 wrote #950's escapes for U+00A0 and U+202F in `TAKEN_OUT` as the raw characters. The four lines are main's again, byte for byte, escapes and all. NIT 1. `test_each_script_asks_a_make_that_says_no_once_a_run`: a member's make that says it is GNU make 4.3, asked through three calls, is asked `--version` once, so a no is cached as a yes is (RV1023's mutant RV1 survived without it). Lane: openreposhape-4 (openRepoShape-4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…named-pipe jobserver, a member's make that says it is 4.4 or later is handed the holder's `MAKEFLAGS` as it is, so it runs its jobs together again; every other make still gets no jobserver and no `-j`, and siblings.py imports the one `member_make_environment` from bootstrap.py (#987) (#1023) #987, the follow-up of #950 (#888, 8264c05). #950's `member_make_env` took every jobserver word and every `-j` out of the `MAKEFLAGS` a member's make inherits. Under GNU make 4.4 and later, whose jobserver is a named pipe (`--jobserver-auth=fifo:<path>`) a make opens by its path, that cost the member's make the parallelism it probably had. bootstrap.py's `member_make_environment(make)` now hands `MAKEFLAGS` on as it is, the `-jN` beside the pipe included, where every jobserver word it names is that named pipe, at an absolute path where a named pipe is now (`names_a_fifo`), and the member's own make says `GNU Make X.Y` with X.Y at least 4.4 on `make --version`, exit 0, within 10 seconds (`reads_fifo`). That make is asked once per make and per run, a no remembered as a yes is, with no jobserver in its own `MAKEFLAGS`, and only where such a pipe is named. Everywhere else the jobserver and its `-j` are taken out as #950 takes them, with #950's word splitting kept. `run_make` and siblings.py's `dispatch` pass the make they run. ONE COPY, AND WHAT IT COSTS. siblings.py imports `member_make_environment` from bootstrap.py, beside `REWRITE_TARGETS`, `credential`, `member_path` and `members_of`, and carries none of the jobserver code: with a copy in each script, SonarCloud's duplication gate failed (15.5% of new lines, over its 3%). That is a trade #950's choice 4 declined. Every released holder (v1.0.0 to v1.1.1) that keeps its own bootstrap.py through `update-shape.py apply --accept-local` is now refused when it takes this siblings.py, as `update-accept-local-missing-name`, before a byte is written, and must merge bootstrap.py by hand first. The guard predates #950 (#536 bbcfd34, #604 b9f65be): what changed is the decision, not the guard. The function's name is new, not #950's `member_make_env`, which took no make, because the guard reads names, not parameters: under the old name a holder cut from main since #950 that kept its own bootstrap.py was let through, and its `make park` stopped at the first member with a `TypeError`. Under the new name that apply is refused too. Measured on GNU Make 4.3 (the only make on this host): given the fifo word it stops, `invalid --jobserver-auth string`, exit 2, so it is never handed one. No GNU make 4.4 was run: the 4.4 case is shown through its `MAKEFLAGS` spelling, a real named pipe and a fake `make` answering `--version`, and the test of #987's first verify line is skipped where `make -j2` writes no named pipe. tests/test_family_member_make_jobserver.py runs every in-process item through both scripts, the one function through siblings.py's import, and keeps #950's escapes for U+00A0 and U+202F in `TAKEN_OUT`. tests/test_update_shape_kept_helper.py: siblings.py takes five names from bootstrap.py. Closes #987 Lane: openreposhape-4 (openRepoShape-4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>



Lane: openreposhape-4 (openRepoShape-4)
Claim: #888 (comment)
Round 1: lane 5's review of #950, taken here
Lane 5 found #950 NOT READY at
4758e645, with one MUST and two NITs (#950 issuecomment-6253417294). Lane 4's plan is #950 issuecomment-6253442578. Every item is taken in1e94ce04785fdf0b4d71ee51367f7be396766ac1, one commit on4758e645, pushed as a fast-forward. Nothing pushed was rewritten.MAKEFLAGS_WORDwas(?:\\.|\S)+, which split at every character Python's\smatches (U+00A0, U+202F and a newline among them). GNU make splitsMAKEFLAGSonly at a space or a tab, and passes those three through raw, so under-j2a value holding one was cut short or split into a second variable. Both copies now read(?:\\.|[^ \t])+, and so does the test's ownWORD.bootstrap.py's comment andmember_make_envdocstring, andsiblings.py's comment, now say make's rule: a space or a tab, a backslash keeping either in a value.TAKEN_OUTspellings, built from Python escapes, each in both scripts: a value holding U+00A0 (NOTE=x<U+00A0>EXTRA=injected), U+202F (NOTE=x<U+202F>!), and a newline (NOTE=x<newline>EXTRA=injected). Each stays one word.TAKEN_OUTspellingk -j16 --jobserver-auth=3,4, which leavesk-j[0-9]?: killed, 1 failed in each scripttest_each_script_hands_back_a_copy_and_leaves_its_own_environment, in each script: with a jobserver named, the result is a plaindictwithMAKEFLAGStaken out, andos.environis as it wasenv = os.environ: killed, 1 failed in each scriptMUST 1, measured (GNU Make 4.3, a stand-in for the holder's rule with no
+,bootstrap.py --no-fetchand a member printingNOTEandEXTRAfrom its environment), undermake -j2 ... NOTE=<value>:NOTE=value501783054758e6451e94ce04x+ U+00A0 +EXTRA=injectedNOTE=[x] EXTRA=[injected]EXTRAempty, no warningx+ U+202F +!NOTE=[x]x+ newline +EXTRA=injectedNOTE=[x] EXTRA=[injected]EXTRAempty, no warningx+ tab +EXTRA=injected, andx\ yWith no
-j, all three heads pass every value whole. The member's ownMAKEFLAGSat1e94ce04under-j2readss -- NOTE=x<newline>EXTRA=injected, as under no-j.The tests: the new file has 43 items (33, then 4 spellings and the copy test, each in both scripts). On this head: 43 passed. All 18 mutants were killed (see "Mutants"). The 19-file gate, run again on this head: see "The run".
What changed and why
#888, filed by lane 5 from rv884's checklist candidate C1 in its review of #886 (#884), #886 issuecomment-6248806027. No milestone yet (lane 3's call). A DRAFT on main.
The head:
1e94ce04785fdf0b4d71ee51367f7be396766ac1: round 0's4758e645, one commit on main64464d31, and round 1's commit on it (above). Main has changed notemplates/family-rootfile since64464d31. Round 0's change was made and measured on50178305(the v1.1.1 release commit) asa4af51bb, and moved onto main before the first push, after lane 5's #891 landed asa9205e00and lane 1's #887 as64464d31.git merge-treeofa4af51bbwith64464d31was clean (tree979ee879, this head's tree): #891 changesMakeErrorsandMAKE_FAILED_REinbootstrap.py, and this PR the end of (b),run_make'sPopenline and the new code afterpassed_on.Under the family holder's
make -j2 bootstrap, each member's make printedmake[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.and ran its jobs one at a time.make -j2 validate,parkandresumedid the same. The rule the warning names is the holder's pinned Makefile, which the person does not own.Why it happened:
scripts/bootstrap.pyandscripts/siblings.pywith no+and no$(MAKE), so GNU make closes the jobserver's two descriptors for them.Popen(andsubprocess.run) also close every descriptor past stderr.-j2 --jobserver-auth=3,4in theMAKEFLAGSit inherits, could not open the jobserver, and warned.The fix is the issue's second direction.
member_make_env()takes the jobserver of the make above, and the-jthat comes with it, out of theMAKEFLAGSa member's make inherits. Each member's make then runs its jobs one at a time, as it did, and says nothing.MAKEFLAGSnames a jobserver:--jobserver-auth=(GNU make 4.2 on: a pipe's descriptors, or 4.4'sfifo:<path>) or--jobserver-fds=(before 4.2, macOS's 3.81).-jword (-j2, or the bare-jolder makes write beside--jobserver-fds).-k,-s, and a variable given on the holder's command line. It splitsMAKEFLAGSas make does: only at a space or a tab, a backslash keeping either in a value. SoNOTE='a -j3'reaches the member whole, and so does a value holding a no-break space, a narrow no-break space or a newline, which make passes through raw (round 1).MAKEFLAGSwith no jobserver byte for byte:make -j(no number, no jobserver) and a person's ownMAKEFLAGS=-j8with no make above behave as before.Where it is used:
bootstrap.py:run_make'sPopen(make bootstrap, and--members-make validatefor the holder'svalidate).siblings.py:dispatch'ssubprocess.run(--make parkand--make resume).make siblingsruns no member's make. It runs each member's ownscripts/bootstrap.py, which runs none, so it printed no warning before and prints none now (measured below).Each script's docstring now says this:
bootstrap.py's (b) paragraph andmember_make_env, andsiblings.py's--makeparagraph.Before and Now
How it was measured. A probe (
probe_888.pyin lane 4's scratch, not committed) ran against this clone with-p conftest, so it builds the same state as the new tests with the suite's own fixtures:tests/test_family.py's family, two members IRRS and IRSS scaffolded and added byscaffold-project.pyandfamily.py, each member's Makefile replaced byTRACED(see "Tests"), each member cloned beside the holder forparkandresume. The holder's own Makefile ran each verb, with or without-j2.ghthat exits 97 first on PATH,GH_TOKEN=bogus, an emptyGH_CONFIG_DIR,GIT_CONFIG_GLOBALan empty file,GIT_CONFIG_NOSYSTEM=1,GIT_SSH_COMMAND=false,GIT_ALLOW_PROTOCOLunset, noMAKEFLAGS, and local remotes only./bin/sh.50178305; Now is this change on it,a4af51bb. Main's the family root's bootstrap.py: an(ignored)line, for any code or signal, resets the depth fully, so a member's ownexit 2after a-one or two makes below, or after|| trueand a-, readsexited 2, and the limits name the-klet-fail shape (#890) #891 touches no line this PR reads or writes; the new tests ran on this head (see "The run").50178305)make -j2 bootstrapmake[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule., one per member; each tracestart one, stop one, start two, stop two, flagsw -j1wmake -j2 validatew -j1wmake -j2 parkw -j1 -- ARGS=w -- ARGS=make -j2 resumew -j1 -- ARGS=w -- ARGS=make bootstrap,validate,park,resume(no-j)w(w -- ARGS=forparkandresume)make -j2 siblings,make siblings(the holder in a folder named after the family)make -n bootstrap,validate,parkEvery run above exited 0. Now, a member's make under
-j2runs with exactly the flags it runs with under no-j.Tests
A new file,
tests/test_family_member_make_jobserver.py, 43 items (33 in round 0, 10 more in round 1). It skips where there is nomake(astests/test_family_makefile_siblings.pydoes) and, withWINDOWS_SKIP, where there is no POSIX shell for the member's recipes.The state.
tests/test_family.py's family: two members, IRRS and IRSS, scaffolded and added byscaffold-project.pyandfamily.py, in a private copy (holder).TRACED. Each target the holder runs in a member (bootstrap,validate,park,resume) has two prerequisites,oneandtwo, that need nothing of each other. Each writesstart <name>, sleeps a second, and writesstop <name>to the member'strace.log. The target then writes the flags its make runs with ($MAKEFLAGS).parkandresume, each member is also cloned BESIDE the holder from its own remote, wheresiblings.py --makeruns it, and that clone getsTRACED.offline), a git configuration of the test's own (isolated), the file transport the local members need (fetching), andPYTHONfrom the environment, soMAKEFLAGScarries no variable the test did not give.The items:
test_no_member_s_make_warns_under_make_j2_and_its_jobs_run_serially[bootstrap|validate|park|resume]: the holder'smake -j2 <verb>exits 0 and prints nojobserver unavailable. Each member's trace isstart one, stop one, start two, stop two, and its flags carry no-jand no jobserver word.test_a_member_s_make_keeps_every_other_flag_and_a_plain_run_is_as_it_was: the holder'smake -k bootstrap NOTE='a -j3', thenmake -k -j2 bootstrap NOTE='a -j3'. Neither warns, both run serially, and each member's flags are the same in both runs:kkept,NOTE=a\ -j3whole, no-jword.test_each_script_takes_out_a_jobserver_and_its_j_and_nothing_else: 12MAKEFLAGSspellings, in each of the two scripts. Each make's jobserver word and the-jbeside it are taken out, the rest is kept, and nothing else in the environment changes. The spellings: 4.3's (measured), with-k -sand variables, and with a variable whose value holds-j4; 4.4'sfifo:path, also with a blank in it; 3.81's--jobserver-fds=3,4 -j, also withk; Windows' semaphore name; and, in round 1, 4.3's with-j16, and with a value holding U+00A0, U+202F or a newline, built from Python escapes.test_each_script_leaves_a_makeflags_that_names_no_jobserver: 6 cases, in each of the two scripts (unset, empty,-j,-j8, variables only, and a variable whose value looks like a jobserver word). The environment is returned unchanged. These items also hold the two scripts' copies to one answer.test_each_script_hands_back_a_copy_and_leaves_its_own_environment(round 1), in each of the two scripts: with a jobserver named, the result is a plaindictwithMAKEFLAGStaken out, andos.environis as it was.On main's two scripts, with this file: round 0's 33 items all failed on
50178305and again on64464d31, and round 1's 43 all fail on main66177b62, whose two scripts are64464d31's. The 5 behavioural items fail on the warning (eachmake -j2printed it twice, once per member); the 38 others because neither script has amember_make_env.Choices for the reviewer
1. Option 2 (take the jobserver out), not option 1 (hand it on). Option 1 was to be preferred if it is clean on GNU make 4.3, 4.4+ and macOS's 3.81, since the person asked for
-j; otherwise option 2, which never warns and never deadlocks. Both were prototyped and measured on this host's GNU Make 4.3, in the real holder (the same probe and state as the tests), each member traced:make -j2 bootstrap,validate,park,resumemake -j2 bootstrapmake -n bootstrap50178305w -j1+on the four rules that run a member's make (bootstrap,validate's second line,park,resume), and the descriptorsMAKEFLAGSnames passed on (pass_fds) inrun_makeanddispatchw -j2 --jobserver-auth=3,4bootstrap.py:every member and leg fetched+(pass_fdsalone; a first prototype, a parent makefile standing in for the holder's rule)s -j1(that prototype ranmake -s)w, as under no-j-j2kept-j2)w -j2 --jobserver-auth=3,4, a jobserver of the member's ownOption 1 is not clean, even on 4.3:
+changesmake -n. A+(or$(MAKE)) marks a line recursive, and make runs it under-n,-tand-qtoo. With+,make -n bootstrapranbootstrap.pyfor real (step (a)'sgit submodule update --init --recursive), where it now only prints the command. With+,make -n validateandmake -n parkranbootstrap.py --members-make validateandsiblings.py --make parktoo; each member's make got-nand only printed its lines.MAKEFLAGSnames the jobserver's descriptors by number. In a first prototype (bootstrap.pyrun straight,MAKEFLAGS=' -j2 --jobserver-auth=3,4', with descriptors 3 and 4 open on a person's files), the member's make read a byte of one file for a token and wrote a+into the other.pass_fdsalone does not help. Without the+, make has already closed the descriptors for the script: the member still warned and ran serially (measured).bootstrap.pyandsiblings.pyrun$MAKEormakeon PATH, and GNU make exports noMAKEto a recipe (measured: unset). So a holder run by one make can start a member's make of another version, and the member then has to read a jobserver word its parent wrote. Measured on 4.3, with each spelling inMAKEFLAGSand nothing open:--jobserver-auth=fifo:/tmp/GMfifo999999stops make:make: *** internal error: invalid --jobserver-auth string 'fifo:/tmp/GMfifo999999'. Stop., exit 2. So does Windows'gmake_semaphore_1234.--jobserver-fds=3,4 -jgives the samejobserver unavailablewarning, and a serial run.On 4.4+ and 3.81 (read, not run: neither is on this host):
mkfifoexists and write--jobserver-auth=fifo:<path>intoMAKEFLAGS. Their NEWS says that with it a sub-make connects to the jobserver even where it is not marked recursive. So on 4.4 with the same make on PATH, a member's make today probably opens the holder's fifo by path and runs its jobs together, with no warning and no change here needed. With this PR it runs them one at a time there too. That is the cost of one rule for every make. Keeping afifo:word would give 4.4 its jobs back, but it would stop a member's make with the fatal error above wherever the holder's make is 4.4 and themakeon PATH is 4.2 or 4.3.make --jobserver-style=pipeon 4.4 writes3,4as 4.3 does and behaves as 4.3 does.--jobserver-fds=3,4 -j: the descriptors, and a bare-j, which alone means as many jobs as make likes. It also closes the descriptors for a line with no+, and its source prints the same warning, so a member's make there warned the same way. Here both words are taken out, so the member runs serially and says nothing. Taking out the jobserver word alone would leave the bare-j, and the member would run unbounded (measured on 4.3:-jinMAKEFLAGSran both jobs together).2. Not taken: give each member its own
-jN. Taking out only the jobserver word and keeping-j2(2b in the table) gives each member's make a jobserver of its own with the same N. On 4.3 it ran each member's two jobs together with no warning. It is none of the issue's three directions, so it is not taken here:make -j4 bootstrap validateat a holder could run 4 jobs in each of two members at once.-jcarries no number, so it would still have to be taken out there (serial).JOBS_WORDdrops only the jobserver word and a bare-j.3. Only a
MAKEFLAGSthat names a jobserver is touched.make -jwith no number starts no jobserver, and a member's make that inherits-jruns unbounded and says nothing, as it did. A person's ownMAKEFLAGS=-j8withbootstrap.pyrun straight is left as it was too. Neither ever printed the warning.4. Written in both scripts, not imported.
siblings.pyalready imports four names frombootstrap.py. A new name imported from there would makeupdate-shape.py apply --accept-localrefuse a holder that keeps its ownbootstrap.py(update-accept-local-missing-name, #536, #604), and a holder that got past that would endmake parkin anImportError. So the three patterns and the short function are written again insiblings.py, asvalidate-pins.pywrites_shownrather than import it, and the twomember_make_envtests hold both copies to one answer on every case.5. Silent. No line says at run time that the members ran one at a time; the two docstrings say it. The issue asks for no warning, and a line of the holder's own under every
-jrun would be one more line nobody acts on.6.
make siblingsand a member's own bootstrap are left alone.run_member_bootstrapruns a member'sscripts/bootstrap.py, which runs no make, andmake -j2 siblingsentered no member make before or after (measured below). Its environment is unchanged.7. The fix is in the scripts, not the Makefile. A
MAKEFLAGS=...prefix in each holder rule would be a third pinned file changed (a thirdupstream-changedrow), would depend on make's own functions in every version, and would not coverbootstrap.pyrun straight from another make's recipe.Mutants
Each is one edit of a script, the new file run against it (43 items), and the bytes put back and checked by sha256 after each (
mutate.pyin lane 4's scratch). Run on1e94ce04; the unmutated control:43 passed. M1 to M12 also ran on4758e645against its 33 items, each killed.run_make'sPopenwithoutenv=member_make_env()(thebootstrapandvalidatepath)[bootstrap],[validate], the-ktest)dispatch'ssubprocess.runwithout it (theparkandresumepath)[park],[resume])bootstrap.pykeeps-j2and takes out only the jobserver word (2b)[bootstrap]and[validate](the traces interleave), the-ktest, the 12bootstrap.pyspellings, and the copy testsiblings.py[park],[resume], the 12siblings.pyspellings, and the copy testbootstrap.pyrewritesMAKEFLAGSwith no jobserver named-j,-j8, variables only, a variable like a jobserver word)siblings.pybootstrap.pysplits words at every blank (\S+)-ktest (NOTE=a\ -j3lost its-j3), and the spellings with a kept blank, a no-break space or a newlinesiblings.pybootstrap.pykeeps a bare-j(-j[0-9]+)[3.81],[3.81-k])siblings.pybootstrap.pyreads no--jobserver-fds[3.81],[3.81-k])siblings.pybootstrap.pysplits as before round 1 ((?:\\.|\S)+, MUST 1)[4.3-no-break-space],[4.3-narrow-no-break-space],[4.3-newline])siblings.pybootstrap.pytakes one digit after-j(-j[0-9]?, NIT 1)[4.3-j16-k])siblings.pybootstrap.pyeditsos.environitself (env = os.environ, NIT 2)siblings.pyAll 18 of 18 were killed, and both scripts' bytes matched their sha256 after each (
bootstrap.py045f33a1,siblings.py7c1abf54, the committed bytes). M1 and M2 are one per path. M3 shows the trace tells jobs run together from a serial run: its 4 traces interleaved.Not measured
MAKEFLAGSspellings.WINDOWS_SKIP); CI is Linux only (ci: run the test suite on Linux only #597).Pinned copies
templates/family-root/Makefile,scripts/bootstrap.pyandscripts/siblings.pyare all inFAMILY_COPIED_VERBATIM(scripts/shape_materialize.py:531). This PR changes two of them, so each holder takes twoupstream-changedrows,scripts/bootstrap.pyandscripts/siblings.py; the Makefile staysunchanged.siblings.pyimports nothing new frombootstrap.py(choice 4), so a holder that keeps its ownbootstrap.pywith--accept-localis not refused for it.What a holder sees. A holder was cut from main
64464d31withfamily.py init(local remotes), andupdate-shape.py check --upstream <this clone> --at 1e94ce04printed (the same rows as at4758e645):Shared files
Each row is
git merge-tree --write-treeof this head1e94ce04785fdf0b4d71ee51367f7be396766ac1with the named commit. Heads were read withgit ls-remoteat 19:08Z. The open PRs that touch any of the three pinned files (gh pr list --state open --json files, 19:08Z) are this one, lane 4's #889 (#879), #951 and #954 (lane 4's #892 branch,fix/892-bootstrap-ok-after-128).66177b62c6be8e45293ef2a5e67adaba32a5899f(lane 4, #879)be1242d7528adbf76a1f7ebf9eb9d3d2(#892)b5e23311No conflict, so no pre-resolution. The merged trees were not run. Round 0's head merged cleanly with #891's last head too (
41f781fd, tree0ae5ea27), which has landed since asa9205e00and is in this branch's base.The run
On this head (
1e94ce04785fdf0b4d71ee51367f7be396766ac1), one background run of 19 files:GIT_CONFIG_GLOBAL=/dev/null: a stubghthat exits 97 first on PATH,GH_TOKEN=bogus, an emptyGH_CONFIG_DIR,GIT_CONFIG_NOSYSTEM=1,GIT_SSH_COMMAND=false,GIT_ALLOW_PROTOCOL,MAKEFLAGS,MAKELEVELandMFLAGSunset; GNU Make 4.3, git 2.43.0, Python 3.12.3.1332 passed in 957.27s (0:15:57), 19:08:40Z to 19:24:39Z, exit 0, no skips. That is round 0's 1322 and round 1's 10 new items. The three files' bytes were checked against the commit's blobs afterwards (git hash-object), and the clone was clean.tests/test_family_holder_bootstrap_texts.py,tests/test_family_bootstrap_gate_order.pyandtests/test_family_makefile_siblings.py;git grep -l -e run_make -e "members-make" -e MAKEFLAGS tests/lists:tests/test_adopt_clone_claimed.py,tests/test_adopt_plan_out_checked.py,tests/test_bootstrap_spelling.py,tests/test_dry_run_parity.py,tests/test_park_resume_targets.py,tests/test_setup_project_dry_run_offers.py;tests/test_repo_hygiene.py,tests/test_agents_md_codes.pyandtests/test_cli_reference.py;tests/test_pinned_copy_text.py(the skew rule),tests/test_siblings_inputs.py,tests/test_family_siblings_placement.py,tests/test_family_gitmodules_values.pyandtests/test_update_shape_kept_helper.py; andtests/test_family.py.4758e645): the same 19 files,1322 passed in 1906.79s (0:31:46), 16:15:16Z to 16:47:04Z, exit 0, no skips. A first run ona4af51bbwas cut off at 16% when the lane's session hit its usage limit; nothing failed before it stopped.The docs. No README, handbook,
AGENTS.md,AGENTS-shape.mdordocs/cli.mdsentence speaks of-j, a jobserver orMAKEFLAGS(git grep -i -e "make -j" -e jobserver -e MAKEFLAGS -- '*.md'finds none), so none is made false.--helpis unchanged.Closes #888
🤖 Generated with Claude Code
Summary by Sourcery
Stop family member make commands from inheriting unusable jobserver settings while preserving other make flags and keeping member work serial under parallel holder workflows.
Bug Fixes:
Enhancements:
Tests: