Repository navigation
repo_shape.py, validate-manifest.py: a bracket in a plain flow scalar is refused and a name holding a lone surrogate is a finding, so pins: [a, b]] is not the pin b] and name: "a\ud83db" no longer kills make validate (#301, #280) - #390
Conversation
…[`, `]`, `{` or `}` is refused, and a display name holding a lone surrogate is a finding, so `pins: [a, b]]` no longer reads as the pin `b]` and `name: "a\ud83db"` no longer stops `make validate` on a traceback while `bump-leg.py` commits it as `a?b` (#301, #280)
#301. In YAML a plain scalar inside a flow collection ends at any of
`[ ] { } ,`. `_split_flow` counted brackets to find its commas and never
asked whether the count went below zero or ended above it, so a stray
bracket was kept as text: `k: [a]]` read `['a]']`, `k: [x [a, b]]` read
`['x [a, b]']`, and since #284 `k: {a"]:}` read the key `a"]`. PyYAML
refuses each. `_refuse_flow_indicator` now refuses a plain item or value
(`_flow_value`) and a plain key (`_split_key_flow`) of a flow collection
that holds one, as a `YamlError` (`REFUSED yaml-unparsable`). A quoted
scalar's brackets, and a bracket in a block plain scalar, read as before.
#280. The reader keeps the escape `\udXXXX` on purpose: `double_quoted`
writes a non-UTF-8 path byte that way into an adoption plan (#199), and
PyYAML's Python reader reads it the same. A display name is text a person
chose, and no UTF-8 stream holds a surrogate: `validate-manifest.py`'s
`manifest ok` line raised `UnicodeEncodeError`, so `make validate` and
`shape-doctor.py` met a traceback, while `bump-leg.py` and `update-shape.py
apply` passed the project and the bump's commit subject read `... in a?b`.
`_name_finding` now reports a name holding any of U+D800..U+DFFF as
`manifest-name`, and all four launchers stop at that finding.
Evidence: all ten #301 rows and every #280 row reproduce on c7c02fa and are
refused on this commit; on the seeded PyYAML-oracle fuzz (seed 2490250,
785,646 documents) no document c7c02fa read the way PyYAML reads it changes,
67,437 documents PyYAML refuses are refused instead of read, and 1,058 that
c7c02fa misread (PyYAML's `#` leniency and single-pair flow mappings) are
refused instead of misread; every tracked YAML file loads the same.
Tests: tests/test_yaml_flow_brackets.py (new), the surrogate tests in
tests/test_manifest_validator.py and tests/test_validator_child_encoding.py
(a real bump, `update-shape.py apply`, the doctor and `make validate`), and
the NIT-A count in tests/test_yaml_quote_boundaries.py's docstring.
Lane: openreposhape-5 (openRepoShape-5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
(lane openRepoShape-5, 2026-10-12T15:19Z) @codex review |
|
Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days. You can request another review in 8 hours and 10 minutes by commenting |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Reviewer's GuideThe PR hardens YAML parsing by refusing bracket-bearing plain scalars in flow collections, preventing both invalid-document acceptance and known misreads, and hardens manifest validation by reporting lone-surrogate display names as actionable findings so all validator callers fail cleanly rather than crashing or committing corrupted output. Flow diagram for YAML flow-scalar validationflowchart TD
A[Parse flow collection] --> B{Scalar is quoted?}
B -->|Yes| C[Parse brackets as text]
B -->|No| D{Contains [, ], {, or }?}
D -->|No| E[Parse plain scalar]
D -->|Yes| F[Raise YamlError]
F --> G[load_yaml returns REFUSED yaml-unparsable]
Flow diagram for manifest surrogate validationflowchart TD
A[Read manifest name] --> B{Non-empty string?}
B -->|No| C[Return manifest-name finding]
B -->|Yes| D{Contains U+D800 to U+DFFF?}
D -->|No| E[Continue validation]
D -->|Yes| F[Return manifest-name finding with escaped name]
C --> G[Validator caller fails cleanly]
F --> G[Validator caller fails cleanly]
G --> H[Prevent UnicodeEncodeError or corrupted commit]
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
… into fix/adopt-record-submodule-placement Brings in f7f4b14 (#366), cb7063f (#390), 5678f68 (#357) and 53bee5b (#289) on top of 3ef059c, which 563ea16 had merged. Merged by sha, no rebase. One conflict, kept on both sides: - tests/test_manifest_validator.py, the import block. cb7063f adds `import os`, `import subprocess` and `import sys`; this branch had already added `import os` and `import sys`, so only `subprocess` was in conflict. The merge keeps all three, once each, as cb7063f's message says it would, with this branch's `from conftest import REPO, git, run_script` and `from repo_shape import parse_yaml, path_spelling_fault` after them. `git merge-tree --write-tree` of the two parents gives the same tree in every file but that one. Checked: - Every top-level definition of the eleven modules either side changed is the changing side's text, byte for byte. The only ones both sides changed are that import block and shape-doctor.py's module docstring, where the merged docstring differs from main's by exactly this branch's two hunks, and from this branch's by exactly 5678f68's one. - No name is bound twice by the merge. scripts/shape_materialize.py imports `YamlError` and `parse_yaml` twice, but main 3ef059c does that too. - Each module's names are the base's plus what each side added. Lane: openreposhape-1 (openRepoShape-1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…escaped Main has moved past this branch's base (cd1fc79), and #390 (cb7063f, the #280 validator half) has landed, so a lone-surrogate name now ends the run at `FINDING manifest-name` instead of validate-manifest.py's own traceback. The merge is clean and touches neither bootstrap.py template. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t/scripts/bootstrap.py: `spelled` escapes the controls, the surrogates, the line and paragraph separators and the bidirectional controls and nothing else, so a Persian name with its `\u200c`, an emoji sequence with its `\u200d`, and a name with `\u3000`, `\xa0` or `\xad` print byte for byte as they did before #391 (#391) Review of the first head found that `spelled` wrote as an escape every character `str.isprintable` refuses. That is wider than a control, a surrogate or a line separator: it also refuses every space but ASCII space, every format character and every unassigned code point. So a valid name that `validate-manifest.py` accepts, and prints raw a few lines below, was escaped by bootstrap's own first line: `\u200c` (the zero-width non-joiner a Persian name needs), `\u200d` (the joiner of an emoji sequence), `\u3000` (the ideographic space between a Japanese family name and a given name), `\xa0`, `\xad` and `\u200f`. The `git submodule update --init <path>` command printed for a leg path holding one of them was no longer right to paste, and what was escaped moved with the interpreter's Unicode database. The test is now the category (the coordinator's decision; Brett may reverse it). `spelled` writes the escape Python spells for EXACTLY these characters: - the controls (Cc); - the surrogates (Cs); - the line and paragraph separators (Zl, Zp); - the bidirectional controls `\u061c`, `\u200e`, `\u200f`, `\u202a` to `\u202e` and `\u2066` to `\u2069`, so a name cannot reorder the text around it on a terminal. Every other character prints as it did before #391. The four categories are fixed ranges, so the answer no longer depends on the Unicode database. The two sets are module-level constants in both copies, and the identity test compares them with the two functions. `unicodedata` is the standard library's. Tests: `tests/test_bootstrap_spelling.py` pins each newly raw character (`\u200c` in a Persian name, `\u200d` in an emoji sequence, `\u3000` in a Japanese full name, `\xa0`, `\xad`, and a zero-width space with a byte-order mark) as the bytes a plain `print` wrote on the base, in a scaffolded project and in a family holder, and with the validators' own `manifest ok:` line beside it. `\u202e`, `\u200f` and `\u2028` are pinned as escaped. Every code point is swept once against a set written from the specification, which is itself checked against `unicodedata.category`. The lone-surrogate rows now assert `FINDING manifest-name` unconditionally, since #280 landed in #390 (cb7063f), and the `make bootstrap` row asserts there is no `Traceback`. Reverting the set to `isprintable` fails 59 of the new rows. Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…plan-manifest-invalid` joins `_entry_findings` after #255's values, and this branch's `plan-root-symlink` stays after the source's own submodules origin/main moved 92e29d9 -> dd8d0ea (twelve commits), merged by sha: - f0a8bf6 (#330 via #381), scripts/family.py; - 87d2eb7 (#333 via #379), update-shape.py; - cd1fc79 (#327 via #375), shape-doctor.py; - 45e01f4 (#348 via #396), tests/test_agents_md_codes.py: every code-shaped name AGENTS.md spells is a code a tool prints; - 3ef059c (#258 and #261 via #314), adopt-project.py, scaffold-project.py and scripts/shape_materialize.py: the project's YAML reader and its `validate-manifest.py` are asked about the manifest's values before anything exists; `check` reports `plan-manifest-invalid` through `_manifest_findings` in `_entry_findings`; `_repository_urls` refuses `adopt-name-too-long`; - f7f4b14 (#366 via #399), scripts/family.py; - cb7063f (#301 and #280 via #390), scripts/repo_shape.py and the template's validate-manifest.py: a bracket or brace in a plain flow scalar is refused; - 5678f68 (#278, #297 and #298 via #357), shape-doctor.py; - 53bee5b (#289 via #352), scripts/bump-leg.py; - f5cb616 (#369 via #398), scripts/shape_materialize.py: the docstrings of `force_stage` and `stage_written`; - a92e60b (#326 via #400), scripts/family.py; - dd8d0ea (#292 via #353), adopt-project.py: `_elector` keeps a `user.name` byte that is not UTF-8, `_remote_repository` reads an origin URL that is not UTF-8 as none, and `adopt-no-org` is asked before `checked_value`. scripts/shape_materialize.py, scripts/repo_shape.py and every other file merge with no conflict. adopt-project.py conflicts in two hunks of `_entry_findings`, each resolved by keeping both sides: 1. The docstring. Main's order, with #261's manifest findings after #255's values and before #190's relative urls; this branch's symlinks stay after the source's own submodules. 2. The return. Main's `_manifest_findings(plan)` call, and the findings in main's order: each entry's path (#196), coverage, the legs and what the root keeps where the shape writes, the source's own submodules (#166), this branch's `_root_symlink_findings`, #255's values, #261's manifest findings, and #190's relative urls last. The remediation and the NOTEs are main's. How each of main's commits that reach this branch's code meets it: - 3ef059c: `_manifest_findings` asks the validator the template ships about the values `execute` writes; it reads no symlink and no path the shape writes. Its materializer half adds the validator's helpers, `refuse_unstorable_names` and `MANIFEST_VALIDATOR`, and holds `election_date` to the validator's `DATE_RE`; it leaves `place`, `_materialize`, `copy_tree`, `template_files` and `_refuse_unwritable` as they were. In `execute`, its `adopt-name-too-long` (in `_repository_urls`) comes before #255's clone-URL refusal, `_refuse_what_check_finds` and this branch's `core.symlinks` probe, all before any leg. Its new paragraph of the module docstring follows the one 6b53a6d corrected; both merge with no conflict. - f5cb616: docstrings only. - dd8d0ea: `_remote_repository`, `_checked_plan_inputs`' organisation and `_elector`, none of them this branch's. - cb7063f: the plans `check` and `execute` read go through this reader. The plans `plan` writes for this branch's tests hold their entries as block mappings, each path `double_quoted` where it needs to be, and the rehearsal below read every one of them. Checked on the merged tree: - AST (fin183g/names.py, base 92e29d9): no top-level name is bound twice in adopt-project.py, scaffold-project.py or scripts/repo_shape.py, and none is loaded but bound nowhere. scripts/shape_materialize.py binds `YamlError` and `parse_yaml` twice, as main dd8d0ea does (1f90a0f's import and 3ef059c's, the same objects); the merge adds none. - Both sides intact (fin183g/sides_intact.py): the only lines that differ from either side's own change are those of the two hunks. - A rehearsal of this merge (tree d21f8b4, the same as this one) passed tests/test_adopt_root_symlinks.py with main's test_adopt_{manifest_values,git_reads,relative_submodule_url}.py, test_scaffold_manifest_values.py, test_agents_md_codes.py and test_yaml_flow_brackets.py: `440 passed`. Lane: openreposhape-1 (openRepoShape-1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ncode and spells each value it prints, so `name: "a\ud83db"` or `日本` under `PYTHONIOENCODING=ascii` no longer ends `make bootstrap` in a traceback (#391) (#456) `templates/assembly-root/scripts/bootstrap.py` `main` opened with `print(f"bootstrap: {name} ({root})")`, before any step ran, and `templates/family-root/scripts/bootstrap.py` with the same line (`:180`, with a literal dash in it). - `name: "a\ud83db"` reads as a string holding a lone surrogate, which no UTF-8 stdout can write. `scripts/bootstrap.py` exited 1 with `UnicodeEncodeError`, `make bootstrap` exited 2, and `validate-manifest.py`, the validator that names the problem, was never reached. - `日本` under `PYTHONIOENCODING=ascii` ended the same way. - The family root's line holds a literal `—`, so under that stdout it raised for EVERY name (`InkRouter` included). The assembly root's own wording has the same dash in two lines and a middle dot in the register readout. - A raw ESC or CR in a name is encodable on every stdout and reached the terminal as the control it is (`bootstrap: a^[b^Mc (...)`). Both files are shape copies, digest-pinned in the project that holds them, and a project carries `scripts/repo_shape.py` and no `shape_materialize.py`, so neither can call #214's helper. Both already import `Refusal`, `find_repo_root` and `load_yaml` from that `repo_shape.py`. `scripts/repo_shape.py`, the pinned shared library every project and every family holder carries beside its `bootstrap.py`, gains two functions and the two sets one of them reads (a new section at its end, `scripts/repo_shape.py:2524`). Both roots' `bootstrap.py` import the two functions on the `from repo_shape import` line they already had, and define neither: - `escape_unencodable_stdout()` is the first statement of each root's `main`. A `strict` or `surrogateescape` stdout becomes `backslashreplace`, the handler stderr already has: what the stdout holds is written as it was, what it cannot hold as `\u65e5`. Any other handler is left alone and the encoding is never changed. It is `validate-pins.py`'s guard from #295 (`validate-pins.py:851`), which is inline there and cannot be imported. - `spelled(value)` writes the escape Python spells (`\x1b`, `\r`, `\n`, `\u2028`, `\ud83d`, `\u202e`) for EXACTLY four kinds of character, chosen by Unicode category and not by `str.isprintable`: - the controls (Cc: C0, DEL and C1); - the surrogates (Cs); - the line and paragraph separators (Zl, Zp: U+2028 and U+2029); - the bidirectional controls: U+061C, U+200E, U+200F, U+202A to U+202E and U+2066 to U+2069. Every other character prints as it did before this change: the zero-width non-joiner (U+200C) a Persian name needs, the zero-width joiner (U+200D) of an emoji sequence, the ideographic space (U+3000) of `山田` U+3000 `太郎`, the no-break space (U+00A0), the soft hyphen (U+00AD), any other format, space or unassigned character, and a backslash. The four categories are fixed ranges, so the answer does not move with the interpreter's Unicode database; `unicodedata` is the standard library's. Helper decision: `repo_shape.double_quoted` / `_escaped` (`scripts/repo_shape.py:347`, `:371`) was read first and still cannot serve: it quotes and escapes `"` and `\`, so a plain printable name would not stay byte-identical. The two new helpers now live beside it, in the same file, the one library both roots import from. `shape_materialize.escape_what_stdout_cannot_encode` is in no project; `shape_advisory.printable` flattens to `?` and is in no copy list. Every print of a value the script read from the manifest, a register row or the disk is spelled: - assembly root: the name and the root (first line); a leg's role, path and tracking branch in step (a); a register row's holder, act, objects, state and expiry, the register's path, and the reader's reason when it cannot be read. - family root: the name and the root (first line), a member's project and directory name, `--members-make` (heading and failure line). The member's own command is still handed the RAW target. A plain name prints exactly what it printed before: the new tests spell out the whole output of a plain project and a plain holder, and that golden also passes on the base. So does every valid name that holds one of the characters kept raw above (see the tests). **Files.** - `scripts/repo_shape.py`: `escape_unencodable_stdout()`, `spelled()`, `ESCAPED_CATEGORIES` and `ESCAPED_BIDI_CONTROLS`, in a new section after `die()`; `import unicodedata`; one bullet in the module docstring's "WHAT THIS MODULE OWNS". - `templates/assembly-root/scripts/bootstrap.py` and `templates/family-root/scripts/bootstrap.py`: both names on the existing `from repo_shape import` line (the family root's becomes the parenthesised form); `escape_unencodable_stdout()` first in `main`; every print of a value read from the manifest, a register row or the disk goes through `spelled`. Against origin/main each differs in nothing else. - `tests/test_bootstrap_spelling.py` (new). `docs/cli.md` is unchanged: no `--help` text moved. **Fix round 1 (review of 118c85a).** The first head's `spelled` wrote as an escape every character `str.isprintable` refuses. Independent review (its S1) showed that is wider than controls, surrogates and line separators: it also refuses every space but ASCII space, every format character and every unassigned code point, so a valid name that `validate-manifest.py` accepts and prints raw on the very next lines was escaped by this script's first line (a Persian name with its ZWNJ, an emoji sequence with its ZWJ, `山田` U+3000 `太郎`, a no-break space, a soft hyphen). The printed `git submodule update --init <path>` command was then wrong to paste for such a leg path, and what was escaped moved with the interpreter's Unicode database (Python 3.9 ships Unicode 13, 3.12 ships 15). The claim "prints byte for byte" was false for those names. **The coordinator decided, and Brett may reverse it:** escape by category, not by `str.isprintable`, the four kinds listed above. Everything else prints raw, as on the base. The bidirectional controls stay escaped on purpose, so a name cannot reorder the text around it on a terminal. The alternative the review offered, and this does not take, is to keep `isprintable`, correct the claims, and pin ZWNJ and U+3000 as escaped by a test row. The other round-1 changes: the lone-surrogate rows now assert `FINDING manifest-name` and no `Traceback` unconditionally, since #280 landed in #390 (cb7063f), and the `make bootstrap` row asserts there is no `Traceback` either; the docstrings, the test module's docstring and the unit rows say what is escaped and what is not. The branch has merged origin/main (53bee5b, which carries #390); the merge touched neither bootstrap file. **Fix round 2 (CI and SonarCloud).** - **CI** (2733c6a). The test module loaded each root's `bootstrap.py` through `importlib`, which wrote `templates/*/scripts/__pycache__/bootstrap.cpython-*.pyc`, and `tests/test_scaffold_leg_path_collisions.py`, which reads every file of the assembly template as UTF-8, then failed on it. `load()` now sets `sys.dont_write_bytecode` around the load, and a regression test asserts that loading either root, from the tree and from a copy in an empty directory, leaves no file behind. - **SonarCloud** (d0aea3e). The quality gate failed on duplication, 14.7% on new code against 3%: `spelled()` and `escape_unencodable_stdout()`, with the two sets, were written out once in each root, and a test held the two copies identical. The premise for the copies, that a root cannot import anything of this repository's, was false: both bootstraps already import from `scripts/repo_shape.py`, which every project and every family holder carries. **Brett Heap decided** to move the two helpers into `repo_shape.py` and have both roots import them, with no Sonar exclusion. The code moved byte for byte (the AST of each function and set, docstrings aside, equals both old copies); only the docstring sentences the move made untrue are reworded. The new section, `import unicodedata` after `import sys`, and the docstring bullet after `Refusal` sit where none of the open pull requests that also edit `repo_shape.py` edits: #409, #453 and #394 each merge with this head with no conflict, in that file or any other (`git merge-tree --write-tree` against each branch's head, 2026-10-12). #416 landed as 580d7df without touching `repo_shape.py`. - The branch merged origin/main ec8e09a (e4b0386) first; that merge touched none of this pull request's files. `git merge-tree --write-tree` of this head with current main is clean (2026-10-12). **How it was tested.** `tests/test_bootstrap_spelling.py` (new, 205 tests at this head, local bare remotes, no `gh`). Answered in bytes, stdout decoded strictly in the encoding the run was given (stderr with `replace`), under `utf-8:strict`, `utf-8:surrogateescape`, `cp1252:surrogateescape`, `cp1252:strict`, `ascii:strict` and `utf-8:replace`: - name rows in a real scaffolded project and a real family holder: - spelled: plain, lone surrogate, ESC and CR, `日本`, newline/tab/DEL, U+2028, the right-to-left override (U+202E) with the right-to-left mark (U+200F), an emoji, a quote and a backslash, two lone surrogates; - kept raw, as on the base: a no-break space (on all six stdouts, the cp1252 ones hold it), a Persian name with its ZWNJ, an emoji sequence with its ZWJ, `山田` U+3000 `太郎`, a soft hyphen, a zero-width space with a byte-order mark. For each, the first line is compared in BYTES with what the plain `print` it replaced wrote (assembly and family), and a full run with the validators shows the validator's own `manifest ok: <name>` line raw beside it; - every other print above: a leg's role, path and tracking branch (and the four lines that name a branch, one with a remote-tracking ref elsewhere), a register row, a register's path and the reader's reason, the root's own directory name in both roots, a member row and `--members-make` (with `MAKE` the interpreter, so a member runs only if the RAW target reached it); - the run reaches the validators: a lone-surrogate name prints `FINDING manifest-name`, `make bootstrap` exits 2 and `bootstrap FAILED: manifest (exit 1)`, with no `Traceback` and no frame in `bootstrap.py`; - `spelled` and `escape_unencodable_stdout` directly, in `repo_shape` itself, once rather than once per root: every code point swept once against a set written from the specification (explicit ranges), which is itself checked against `unicodedata.category`; that each root's `from repo_shape import` names both and that neither root defines or assigns either function or either set (an `ast` check of the file); that the name each root's module exposes IS `repo_shape`'s object; that the reconfigure is `main`'s first statement; and that loading a root writes no bytecode under `templates/`. The end-to-end rows run a real scaffolded project and a real family holder, and each carries `scripts/repo_shape.py`, so they run the moved code where it ships. Results (`Python 3.12.3`, Linux; head d0aea3e = the move, on e4b0386 = the merge of origin/main ec8e09a): | run | result | |---|---| | `test_bootstrap_spelling.py`, `test_scaffold_leg_path_collisions.py`, `test_stdout_encoding.py`, `test_template_quoted_values.py`, `test_template_values_single_pass.py`, `test_repo_hygiene.py`, `test_agents_md_codes.py` | 684 passed, 1 skipped (`tests/test_stdout_encoding.py:286`, Windows-only) | | the five `test_yaml_*.py`, the six `test_update_shape*.py`, `test_scaffold_e2e.py`, `test_family.py`, `test_validate_family_unnamed_mounts.py`, `test_cli_reference.py`, `test_windows_paths.py` | 706 passed, 3 skipped (`tests/test_update_shape_add_symlink.py:831`, NTFS junctions) | | mutation: `repo_shape.spelled` escapes nothing | 61 failed, 144 passed: 29 unit rows (every escape row, the code-point sweep, the non-string and one-line rows) and 32 end-to-end rows in a scaffolded project and a family holder (the name lines, a leg's role and path, a branch, a register row and a register's path, the root's directory, the member rows) | | `test_bootstrap_spelling.py` again, the mutation reverted | 205 passed | Earlier heads, not re-run on this one: on af26b19 the then 260-test file gave 197 failed, 63 passed against the base (cd1fc79) templates, 59 failed with both copies' `spelled` reverted to `str.isprintable`, and caught all 12 mutations of the category sets; reverting each hunk of the first round's fix, one at a time, still failed a new test (34 mutations, listed in the report); on 118c85a the existing bootstrap, scaffold, adopt, update-shape and doctor tests gave 496 passed, 1 skipped, and 305 passed. macOS and Windows have not run here; CI runs them on this head. Every new test reads and writes bytes and compares against `project.resolve()`. What the rest of the run says (the lone-surrogate and `make bootstrap` rows assert it at this head): a lone-surrogate name prints `bootstrap: a\ud83db (<root>)`, reaches step (b), and ends ``` FINDING manifest-name: name is 'a\ud83db', which holds U+D83D, a lone surrogate: no UTF-8 file, commit message or console can hold it. ... bootstrap FAILED: manifest (exit 1) ``` exit 1 (`make bootstrap` 2), with no `Traceback` on a pipe, a TTY or `make`. **What is left out.** - The validators' own stdout. `validate-manifest.py`'s `manifest ok: <name>` still prints through a stdout with no handler: a valid `日本` under `ascii` ends in ITS traceback (the issue's "Not in this issue"), and an ESC or CR name is accepted and the line writes the raw control. - A manifest value that is passed to git rather than printed: `tracking_branch: "a\ud83db"` (or a leg's `branch:`) raises from `subprocess` inside step (a), before the validators; a NUL does too (`ValueError: embedded null byte`). Reproduced on origin/main 53bee5b. `validate-manifest.py` never reads either key, so it accepts the manifest (`manifest ok`, exit 0) and NO tool names the value. Not a `print`; follow-up below. - stderr: `print(str(exc), file=sys.stderr)` (Refusal text built by `repo_shape`, multi-line, manifest text `repr`-escaped; stderr cannot raise) and the family's `FETCH INCOMPLETE` text (git's own words). - `templates/family-root/scripts/siblings.py`, which carries the SAME first line twice (`:676` in `dispatch_all`, `:906` in `_run_placement`) with the same plain `print`, and prints member rows raw. `make siblings`, `make park` and `make resume` on `name: "a\ud83db"` still end in a `UnicodeEncodeError` traceback (reproduced on origin/main), and under `PYTHONIOENCODING=ascii` they end so for every name (the literal dash). It already imports from `repo_shape`, where both helpers now are, so they are two more names on its existing import line. Follow-up below. - Behaviour that differs from the base, all deliberate: - a control, a surrogate, a line or paragraph separator or a bidirectional control in a value now prints as its escape where the base wrote it raw, so a Hebrew or Arabic name that holds an explicit U+200E, U+200F or U+061C prints `\u200f` for it; - a root path holding a byte that is not UTF-8 prints `\udcXX` where the root `bootstrap` shim's #214 handler wrote the raw byte back; - a name holding the four characters `\x1b` prints as one holding ESC does; - the wording's dash and middle dot print `\u2014` and `\xb7` on a stdout that cannot hold them, instead of raising. - The digest: no digest of either bootstrap or of `repo_shape.py` is recorded in this repository. The scaffold, `family.py init` and adopt compute each `files:` row from the bytes they write, and `update-shape.py` compares upstream blobs at two commits. A project scaffolded from origin/main 580d7df, checked against this head (d0aea3e), reads `upstream-changed` for `scripts/bootstrap.py` and `scripts/repo_shape.py` and nothing else (assembly root: 2 changed, 9 unchanged; family holder: 2 changed, 8 unchanged). `apply` copies both and re-pins, the project's own validators say `11 copied shape file(s) match their digests` (the holder's, 10), each copy equals its upstream blob, and `scripts/bootstrap.py` runs. Run for real, local remotes only. - Compatibility: a project that keeps a locally edited `scripts/repo_shape.py` (`check` reads it `both`, and `update-shape.py apply --accept-local scripts/repo_shape.py` keeps the project's bytes) while taking the new `bootstrap.py` meets an `ImportError` the next time it bootstraps: `scripts/bootstrap.py` ends in `cannot import name 'escape_unencodable_stdout' from 'repo_shape'`, exit 1. Following `check`'s own instruction avoids it: `check` says to merge upstream's bytes into the edited file by hand, commit it, and only then `apply --accept-local`, and a merge that takes the new section brings both names. A project that skips the merge gets the error and nothing reports the break after `apply`: its validators still pass, because they import neither name; `check` then reads everything unchanged; and the project's `validate.yml` never runs `bootstrap.py`, so its CI stays green. The first command to fail is the one run after `git clone`. Guarding it is an item on #458's checklist (#458 (comment)): `apply` would refuse a kept `--accept-local` copy that lacks a name another copied shape file imports from it. It is the same coupling as the names the bootstraps already import from that file. Run for real on a project scaffolded from 580d7df. Follow-ups found by lane 5's review of this branch, each reproduced on origin/main and filed separately: - `siblings.py`'s two first-line prints (and its member rows): `make siblings`, `make park` and `make resume` on a lone-surrogate `family.yaml` name end in a `UnicodeEncodeError` traceback (#454). - A lone surrogate or a NUL in `tracking_branch` (or a leg's `branch:`) ends `make bootstrap` in a subprocess traceback in step (a), and `validate-manifest.py` accepts the manifest (#455). - #454 and #455 (open, off the milestone) and the two #459 checklist items moved from #463 and #464 can import `spelled` and `escape_unencodable_stdout` from `repo_shape.py`, wherever they print a value. Reviewed adversarially at 118c85a (CHANGES NEEDED: 0 blockers, 2 should-fixes, 2 nits); fix round 1 (af26b19) was re-verified APPROVE WITH NITS (0/0/2). Fix round 2 stopped the test module writing bytecode into `templates/` (2733c6a) and moved the two helpers into `scripts/repo_shape.py` on Brett Heap's decision (d0aea3e); its independent re-verification read APPROVE WITH NITS (0/0/3), all three about this text and settled here. CI is green on Linux, macOS and Windows, with SonarCloud passing, at d0aea3e (main ec8e09a merged in). Closes #391 Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d `open_glass` share one `SHAPE_PIN_SOURCE_<PRODUCT>`, so `--pin` refuses the pair and `validate-pins.py` notes a declared one and reads the variable for neither (#275) (#409) `pin_source_env_name` turns every character outside `[A-Z0-9_]` into `_`, so `open-glass`, `open_glass` and `open.glass` are three legal product names, with three pin files (`contracts/<product lowercased>-pin.yaml`), and one variable, `SHAPE_PIN_SOURCE_OPEN_GLASS`. On ab623dd (this branch's main parent; the same rows reproduce on c7c02fa), with the issue's own probe against local bare remotes and a `gh` that exits 97: | step | ab623dd | this head (6494714) | |---|---|---| | `scaffold-project.py --pin open-glass@<A> --pin open_glass@<B>` | exit 0, three repositories | exit 2 `scaffold-pin-collides`, nothing created | | `validate-pins.py`, no source, on a project that already declares the pair | exit 0, both skips offer `SHAPE_PIN_SOURCE_OPEN_GLASS` | exit 0, one `note neutral-pin-source-collides`; the skips offer `--pin-source <product>=<path>` and a sibling checkout only | | `SHAPE_PIN_SOURCE_OPEN_GLASS=<open-glass>` | `neutral-pin-source-unreadable` for `open_glass`, exit 1 | exit 0, the note; the variable is read for neither pin, so each pin is read from its own source or skipped | | `SHAPE_PIN_SOURCE_OPEN_GLASS=<open_glass>` | `neutral-pin-source-unreadable` for `open-glass`, exit 1 | the same | | `--pin-source open-glass=... --pin-source open_glass=...` | exit 0 | both pins recompute, exit 0, the note | **What changed.** Every variable name is kept: an operator may have set one and the docs name them. What is done instead: - `scripts/repo_shape.py`: `pin_source_env_collisions(products)` is the one pure answer, `{variable: [products]}` for each variable two or more distinct products share. Distinct means distinct PIN FILES (`pin_product_key`, `lower()`, which is what `contracts/<product lowercased>-pin.yaml` is named by), so a product typed twice or in two cases is #241's file collision and never this one. `pin_source_env_shared` is the set of those keys. - `scaffold-project.py`: `_refuse_colliding_pins` (left where it is) names each variable's `--pin` arguments as typed after #241's file sentences, in the same `scaffold-pin-collides` refusal. Dry run and real run say the same no, before any process starts and ahead of the leg-path refusal. #241's wording and remediation are unchanged when only a file collides. A NEW pair is still refused, so the situation is not created anew. - `templates/assembly-root/scripts/validate-pins.py`: one NOTE per variable, `neutral-pin-source-collides`, naming every product and the variable, the keyed `--pin-source <product>=<path>` that reads each pin, and that which declaration stays is the human's decision. It is never a finding and never moves the exit code. A product whose variable is shared does not read it (rule 2 of `resolve_neutral_pin_source`), and its skip hint offers the keyed `--pin-source <product>=<path>` and not the variable or the bare form. A variable one product has to itself reads and is offered exactly as before. - `scripts/repo_shape.py`, the two chain readers: `link_pins_from_trees` does not read a variable two links of one chain share, and the `declared-unverified` hint stops naming it. Only a hand-written chain can have two such links, since scaffold takes every link but the first from the naming policy's alphanumeric `open<Product>` form. Each reader of `pin_source_env_name`, decided: | site | decision | |---|---| | `scaffold-project.py --pin` | refuse (above) | | `validate-pins.py` rule 2 and the skip hint | a note, and the variable is neither read nor offered for a shared product | | `repo_shape.py` `resolve_link_source` (via `link_pins_from_trees`) | skip the variable when another link of the chain shares it | | `repo_shape.py` `_chain_link_findings` hint | omit the variable for a shared link | | `adopt-project.py --pin`, `plan`, `check`, `execute` | not in this PR, see below | **Fix round 1 (from the review of 34dbd86).** **The design change: a declared pair is a NOTE and not a finding. This is the lane coordinator's decision, and Brett may reverse it.** The first round made `neutral-pin-source-collides` a FINDING. Main through ab623dd scaffolds such a project (`--pin open-glass@A --pin open_glass@B` exits 0), and `update-shape.py apply` runs the NEW `validate-pins.py` over the rewritten root, so for a project already declaring the pair the finding ended every apply as `REFUSED update-validators-red`, rolled back, for this update and every later one, until a person deleted a true declaration of descent. AGENTS.md tells an agent never to edit "a pinned file, a digest or a manifest to make a row go green", and the finding's remediation was exactly that, so the person was locked out. The coordinator chose the review's option (b): it is a NOTE now. The guard stays (the variable is read for neither pin), each pin still verifies from its own source, scaffold still refuses a new pair, and `validate-pins.py` says the rest in one line. To put the finding back, restore 34dbd86's `report.finding(...)` call in `_report_pin_source_collisions` (it is not a word swap for `report.advise`: `advise` takes one string, `finding` a code and a detail), give the finding the remediation 34dbd86 had, and turn the tests that expect the note (and `test_a_project_declaring_a_colliding_pair_takes_a_shape_update`) back into exit 1; the first round's versions of all of them are on 34dbd86. - **Proved on a real old project.** Main (ab623dd) scaffolded the pair into local bare remotes; `update-shape.py apply` to the first round's head 34dbd86 exited 2 `REFUSED update-validators-red` and rolled back (`git branch` shows only `main`); to this head it exits 0, the project's validator passes and prints the note, and the doctor's `pins` row is `ok` before and after. The doctor prints `passes` for a validator that exits 0 and does not echo its stdout, so the note itself is visible in `update-shape.py apply`'s output and in `validate-pins.py`'s, and the verdict moves for no reason but the update (`COMPLIANT, SHAPE BEHIND` to `COMPLIANT`). It is now a test: `test_a_project_declaring_a_colliding_pair_takes_a_shape_update`, over local stand-ins beside the project, so no `gh` is asked. - **Collisions by the variable.** Pairs were decided by `casefold()`, but the pin file is `lower()`. `openß`/`openss`, `openſ`/`opens`, `openfi`/`openfi` and `openς`/`openσ` (the letters are U+00DF, U+017F, U+FB01 and U+03C2 beside U+03C3) share a variable and have two pin files, and were never found. They are now, from the pin-file fold (`pin_product_key`). A pool of 31 names is checked pairwise against `neutral_pin_path` and `pin_source_env_name` themselves, and no single product is ever a collision. They are reachable only from a hand-written `project.yaml`. - **Windows.** The guard's only test needed the `#!` `gh` stub and was skipped on Windows. `test_a_shared_variable_answers_for_neither_pin_with_no_gh` puts a checkout of each pin beside the project, names one in the variable, and needs no `gh`: on ab623dd it is `neutral-pin-source-unreadable`, here each pin reads its own sibling. The hint and the non-ASCII rows are tested in-process, with no skip and no filesystem. - **The hint.** For a product two pins share, the skip hint offers the keyed `--pin-source <product>=<path>`: the bare form answers for every pin with no more specific source, so with two pins it gives the other the finding this issue is about. - **Trailers and title.** The trailers of the earlier commits (a30091f and 34dbd86 name Sonnet 5.5) are settled in this message, which carries Opus 5.5. The PR title now names `repo_shape.py`. - `scaffold-project.py` is not touched this round: `fix/adopt-pin-writes-pin-file` moves that code, and the change would only conflict with it. **Left out.** - `adopt-project.py` is untouched. Its half arrives with #273's PR, which moves scaffold's pin machinery into a shared module and routes adopt through `pin_source_env_collisions`. Once it lands, the one place that still keys by `casefold()` (`pin_variable_collisions` in `scripts/shape_materialize.py`) should read `pin_product_key`; it cannot differ for an ASCII `--pin`, which is all `PIN_ARG_RE` takes. - `scaffold-project.py`'s own `_colliding_pin_variables` also keys its argument lists by `casefold()` and one docstring there still says the project's validator reports the pair "as a finding"; both go with the code when #273 moves it. #312's `PinSources` keys `--pin-source` by `casefold()` too, so `openß=` and `openss=` are one key there; that is #312's. - A product in `neutral_product_pins:` and a chain link in `naming.referent_chain` that share a variable are read by two validators in one process, and neither sees both sets. Not detected here. - The doctor does not print a passing validator's stdout, so its `pins` row does not carry the note. Not changed: it would add a file to this PR. - No `--help` text changed, so `docs/cli.md` is untouched. `README.md` is at its line cap and its blob is pinned by `docs/handbook.html`, so the one README sentence this wants needs a regenerated handbook and waits for its own change. - `validate-pins.py` and `repo_shape.py` ship into projects, so every project's `shape-pin.yaml` reads `upstream-changed` for both until `update-shape.py apply`, which now succeeds for a project that declares such a pair. **Evidence.** All on the head (6494714), with git-filter-repo 2.47.0 on PATH, a `gh` that cannot be reached (a stub first on PATH, `GH_TOKEN` bogus, an empty `GH_CONFIG_DIR`), and the repository-local `user.name` blanked at command-line scope, as the standard's checkout sets one that two scaffold tests read. - `tests/test_pin_source_env_collisions.py` is 137 tests and passes on this head. Laid over 34dbd86, 50 fail and 87 pass: the note and update tests, the non-ASCII rows, the hint, the sibling guard, and `pin_product_key` itself (which 34dbd86 does not have); the scaffold refusals, the chain readers and the controls hold on both. - Targeted, run on ad6f569, whose tree differs from this head in `tests/test_pin_source_env_collisions.py` only (the last edits made one comparison read the committed blob and wrapped one line): the new file, `test_pin_validator`, `test_neutral_pin_validator`, `test_cli_reference`, `test_manifest_validator`, `test_naming_policy` and `test_repo_hygiene` **445 passed**; the scaffold pin, collision, referent-chain, leg-path and e2e files **245 passed**; `tests/test_update_shape*.py` **218 passed, 3 skipped** (junctions, NTFS only); `tests/test_shape_doctor.py` **254 passed, 3 skipped** (the PowerShell line, a Windows host only). - On this head itself: the new file, `test_repo_hygiene`, `test_cli_reference`, `test_pin_validator` and `test_neutral_pin_validator` **244 passed**. - This head merged with origin/main a983e6a, in a throwaway worktree: the new file, `test_pin_validator`, `test_neutral_pin_validator`, the scaffold pin, collision, referent-chain and e2e files, the whole doctor file and `test_cli_reference` **590 passed, 3 skipped** (the PowerShell line); the new file alone, in its final form, **137 passed**. origin/main had moved twice since; the merge-tree below is against it. - Twelve mutation probes, each hunk reverted alone: all caught. The note-to-finding one is caught by the update test (a committed mutant, since the test clones the repository's HEAD) and the guards by the sibling test, run with every test that needs a `#!` stub deselected as a Windows runner would; the hint one by an in-process test. - Nothing ran on macOS or Windows before this PR's CI. **Merge-tree, as computed during the fix round on 2026-10-12** (every sha and head is the one it had then; main and the other branches have moved since). `git merge-tree --write-tree` of this head (6494714) against origin/main 1f90a0f is clean, and so is the merge of origin/main a983e6a, which the merged-tree runs above used. #312, #309, #270, #268, #228, #245 and #351 have landed. The rest were computed on ad6f569 (the same tree but for one test file) against a983e6a. Open PRs touching this branch's files, all clean against the head: #314, #355 and #373 (`scaffold-project.py`), #343 (`validate-pins.py`), #390 (`repo_shape.py`) and #394 (`repo_shape.py`, which conflicts with origin/main by itself). The other 25 open PRs and the lane-5 branches `fix/adopt-check-link-and-manifest`, `fix/scaffold-free-text-values`, `fix/family-init-values`, `docs/adopt-execute-exits`, `fix/yaml-flow-brackets-surrogates` and `fix/adopt-pin-writes-pin-file` (#273's, 7a94b6b, also clean for this head) are clean too. #356, #382 and `fix/adopt-record-submodule-placement` conflict with origin/main and not with this head; #358, `fix/adopt-late-failure-notice` and `fix/adopt-plan-quote-yaml-keywords` conflict with both, with origin/main by themselves, so they are not this branch's. Neither are the five PRs below #200 that origin/main has not landed (#94, #96, #98, #100 and #161), which conflict with it too. The PR set is read from the refs under `refs/pull/` that origin/main's history has no `(#n)` subject for, not from GitHub, so a closed one may be in it. **One conflict, with `fix/link-source-values` (#338, 40762ca) in `scripts/repo_shape.py`, four hunks.** Not resolved in either branch; take both sides: (1) `resolve_link_source`'s signature, `default_source=None, env_shared: bool = False, *, tried: list | None = None`; (2) its docstring, both paragraphs; (3) `link_pins_from_trees`'s docstring, both paragraphs; (4) its call, `default_source, pin_product_key(name) in shared, tried=asked)` followed by #338's `if tried is not None: tried[name] = asked`. The auto-merged `resolve_link_source` body already composes: a shared variable is never an `env` candidate and so never appears in #338's `tried`. **Known limits after the re-verification (APPROVE WITH NITS, 0 blockers, 0 should-fixes, 3 nits, at 6494714).** For the four non-ASCII pairs this change now detects (`ß`/`ss`, `ſ`/`s`, `fi`/`fi`, `ς`/`σ`), the keyed `--pin-source <product>=<path>` form does NOT separate the two pins: #312's source lookup folds both keys into one, so one pin can get a false `unreadable` finding. A sibling checkout does separate them. Only a hand-written `project.yaml` can declare such a pair. The doctor's `pins` row still prints only "passes" for a validator that passes, so it does not carry the collision note; that needs `shape-doctor.py`, and is left for a follow-up. The wording for restoring the finding is under "Fix round 1" above: it is 34dbd86's `report.finding(...)` call, not a word swap. Reviewed adversarially at 34dbd86 (CHANGES NEEDED: 0 blockers, 2 should-fixes, 5 nits). The first should-fix was the finding itself, which locked a project that already declares the pair out of every `update-shape.py apply`; the second was a description that had gone stale. Fix round 1 (6494714) made the pair a NOTE, which is the coordinator's decision and one Brett may reverse (the way back is above), keyed collisions by the pin file's own fold, gave the guard a test that runs on Windows, and made a shared product's skip hint offer the keyed `--pin-source`. Its re-verification at 6494714 found every finding handled: APPROVE WITH NITS (0 blockers, 0 should-fixes, 3 nits: the keyed form and the non-ASCII pairs, the doctor's silence about the note, and the wording for restoring the finding, each stated above). CI is green on Linux, macOS and Windows, with SonarCloud passing, at b70a07d (main a73b4e9 merged in). Its merge with main 5a25387 passed the targeted set (the six pin and scaffold test files, `tests/test_validate_pins_unreadable_workflow.py`, `tests/test_ruleset_hint_tracking_branch.py` and `tests/test_shape_doctor.py`): 850 passed, 3 skipped. Closes #275 Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… is `PRODUCT=PATH` only when its key names a chain link, and a link source that cannot be entered is a WARNING with the system's reason, not a traceback (#338) (#453) **What was wrong.** Two faults, one in the parser and one in the lookup. - **The parser.** `_link_sources_from_args` split every `--link-source` value at its first `=`. A link tree under a directory named `a=b`, passed bare, became a source keyed to a link `a` that no chain names. Nothing asked for it, so the link stayed `declared-unverified`, exit 0. The WARNING then offered `--link-source openXdox=<path>` to the person who had passed exactly that. A mistyped key failed the same silent way. - **The lookup.** `repo_shape.link_manifest_path` asked `Path.is_dir()` and then `is_file()`. Before Python 3.14 those swallow only ENOENT, ENOTDIR, EBADF and ELOOP. So each of these ended the naming gate in a traceback, exit 1: - a keyed source under a parent this user cannot enter; - `SHAPE_PIN_SOURCE_<LINK>` pointing there; - a checkout `../openXdox` at mode 000, which every `--project` run asks for; - a 300-character name. **What changed.** **How a value is read** (`LinkSources` in `scripts/validate-repository-naming.py`). This mirrors #312's `PinSources`, with a chain link where that file has a declared pin: - The text before the first `=` is a LINK only when it casefolds to a link of a chain this run reads. Those are the entries of `--referent-chain`, and of each leg's `naming.referent_chain:` under `--project`. Both are read before any leg is classified. - The keyed reading wins even over a directory spelled like the whole value. `./` reads that directory. - Any other value is a bare PATH, taken whole. **What a lookup says** (`scripts/repo_shape.py`): - `link_manifest_lookup` asks `stat()` inside a guard, as `_digest_from_source` does in #312. It returns the system's reason when a source cannot answer, or says which of four plain cases it is (`no such file or directory`, `it holds no project.yaml`, `its project.yaml is not a file`, `it is neither a directory nor a file`), and it never raises. It replaces `link_manifest_path`, which is gone: nothing calls it. `link_declarations` gives the reason a manifest could not be read, and `declared_neutral_pins`, which it left with no caller, is removed (fix round 1). - `resolve_link_source` and `link_pins_from_trees` take a keyword-only `tried`. It records each source asked, in order, with its reason. Nothing in `repo_shape` prints. - `link_pins_from_trees` also takes a keyword-only `with_reasons` (fix round 2). Asked, it returns a `LinkPins`, a dict equal to the plain mapping that also remembers why a tree it reached could not be read. Only the scaffold asks. **What the validator prints** (stderr, after each name's WARNINGs): - `WARNING link-source-unreadable: <path> (<how>) could not answer for <link>: <reason>`. This covers a `--link-source` asked for a link that holds a pin, a `SHAPE_PIN_SOURCE_<LINK>` likewise, and a checkout beside the project that exists and still cannot answer. A keyed value that also names a directory gets a second line saying how to ask for that directory with `./`. A bare value holding `=` says it was read as a PATH. - `NOTE --link-source <typed>: not used — <why>.` for a value no such link asked for. The why is one of five: a later value replaced it; no name reads that link's own declaration (a keyed value); no name records a chain at all; no name reads any chain link's own declaration (a bare value, where chains exist and none is read); or every link that is read had a more specific source. - Three kinds of source are never named: a variable or a checkout asked only for a chain's LAST link (no one ever reads that link's own declaration); a variable or a checkout for a chain no classification reads (fix round 1: a spec leg's chain, a name with no referent, a chain whose shape fails; a `--link-source` value for such a chain is the NOTE); and a checkout that is absent or holds no manifest. - `--quiet` drops both, with the WARNINGs. So `make bootstrap` and the doctor's `naming` row print nothing new. `--help`, the module docstring and `docs/cli.md` (regenerated) say all of this. **Fix round 1 (lane 5's review of 40762ca: 0 blockers, 2 should, 4 nits).** Second commit, 4b21263. Each review finding, in order: - **A link manifest whose `neutral_product_pins:` is not a list of names is a tree that cannot answer** (SHOULD-1). `link_declarations` iterated `data.get(key) or []` as it stood, against the `link_pins_from_trees` docstring this change wrote ("nothing a source does raises out of here"). `5` ended the naming gate, the scaffold and the project's `validate-manifest.py` in `TypeError: 'int' object is not iterable`. A scalar `openDox` was read as the characters `D`, `e`, `n`, `o`, `p`, `x`, so the chain broke, and under `--project` that is a FINDING. A mapping was read as its keys, so `openDox: yes` DECLARED `openDox`. Each is now `(None, why)`, as `not a mapping` is: declared-unverified, with the reason named ("`neutral_product_pins` is not a list", "holds an entry that is not text"). An absent or null value, an empty list, and a blank entry (`null`, `''`) in a list, declare nothing, as ever. The fix is in the shared function, so the scaffold and `validate-manifest.py` are reached too. **Three behaviours change on purpose here, and the tests of fix round 2 pin each:** - **A falsy value is no longer an empty list.** `neutral_product_pins: 0`, `false` or `''` was read as no pin at all (`or []`), so a chain through the tree was BROKEN, and under `--project` that is a FINDING. It is now a tree that cannot answer: the link is declared-unverified, a WARNING, exit 0. - **An entry that is not text is no longer read as text.** `[openInk, 0]` and `[openInk, false]` dropped the entry as a blank, and `[openInk, true]` read it as the text `True`, so a chain through the list VERIFIED. Each is now declared-unverified, with a WARNING saying the list "holds an entry that is not text". That is stricter than the link tree's own `validate-manifest.py`, which takes `str(pin)` of an entry (`_declared_pins`, `templates/assembly-root/scripts/validate-manifest.py:85`): a chain is better left unverified than verified through a name nobody wrote. - **The scaffold no longer refuses `chain-not-declared` over a pins value that is a scalar.** With `--pin-source` a tree whose `neutral_product_pins:` is the scalar `openInk`, the scaffold used to refuse (`REFUSED chain-not-declared: ... openXdox declares I, e, k, n, o, p, not openInk`, exit 2, from the characters). It is now the declared-unverified WARNING and the scaffold goes on, exit 0, which is its documented rule for a tree that cannot answer: "a warning, never a refusal, because the check is offline". The warning says the tree was reached and could not be read (fix round 2, below). - **The reach wording is corrected** (SHOULD-2): "Who else this reaches" below, and the docstring of the test that made the claim. What stops crashing is the naming and manifest steps. `make validate`, `validate.yml` and `make bootstrap` still end in a traceback on the sibling row before Python 3.14, at a file this change does not touch (below). On 3.14 the sibling is passed over without a word there instead, which is #430. - **The mutation gaps** (nit). Six of the review's ten mutations survived. Five now die, each by a new test: a `project.yaml` that is a directory; a device or a FIFO as the source; the order A, B, A; a value holding `=` with no chain at all; and every failure of `SHAPE_PIN_SOURCE_<LINK>` for a link that holds a pin (a path that is not there, a directory with no manifest, a file that is no manifest), not only a permission. The sixth, the `typed in self.used` test in `unread_lines`, could not be killed because it was dead: a value is read as one link's keyed source or is the one bare PATH, so every lookup asks it of the same path and it is used or unread, never both. That was asserted over 293 tests and the test is removed. - **A chain no classification reads** (nit). `classify` consults a link's own declaration only for a name that claims a referent, with a recorded chain whose shape holds. For any other chain (one recorded on a spec leg; one on a name with no `open<Product>` to reach; one that does not begin at a held pin or does not end at a referent of the name) a source was still asked, so a path that was not there was reported as `link-source-unreadable` and a good tree was counted used, each for a question nobody put. `NamingPolicy.reads_link_declarations` now says when `resolve_referent` reads a chain, and `LinkSources.link_pins` asks it first: such a chain asks nothing, and a value for it is the NOTE that no name reads that link's declaration. A test runs `resolve_referent` with a mapping that records whether it was read, over every way a chain is and is not, and holds the new method to it. - **Dead code and stale sentences** (nit). `declared_neutral_pins` is removed: `git grep` over the tree, every pushed branch and every sibling worktree finds no caller. `link_manifest_lookup`'s docstring no longer says "replaces" over a name that is gone. This description no longer says `tests/test_naming_policy.py` takes `SEARCH_DENIED` from #312: it defines a mark of its own, with the same condition and its own reason text. **Fix round 2 (lane 5's re-review of 4b21263: 0 blockers, 0 should, 3 nits).** Third commit, 2ac911c. Each finding, in order: - **The three behaviours the body did not state are stated and tested** (nit 1). Round 1's not-a-list fix changed three behaviours on purpose, each listed under SHOULD-1 above, and the re-review found two of them had no test. R1 (`if declared is None` read as `if not declared`, which makes `0`, `false` and `''` declare nothing again) and R2 (`if pin is None or pin == ""` read as `if not pin`, which drops a `0` or `false` entry as a blank) both passed all 210 cases of `tests/test_naming_policy.py` at 4b21263. Six rows join `NOT_A_LIST_OF_NAMES`, which runs at the CLI and in process: `0`, `false` and `''` as the value, and `0`, `false` and `true` among the names. R1 now fails 9 cases and R2 fails 6. - **The scaffold's warning is true for a tree it reached** (nit 1, the third behaviour). Over a `--pin-source` whose `neutral_product_pins:` is not a list, round 1 turned the `chain-not-declared` refusal into a declared-unverified WARNING that says the tree "is not reachable here (a sibling checkout, SHAPE_PIN_SOURCE_<LINK>, or --link-source <LINK>=<path> would let it be read)". `--pin-source` did reach the tree, and only its declaration could not be read, so that blamed a path that was right. Now `link_pins_from_trees` takes a keyword-only `with_reasons`; asked, it returns a `LinkPins`, a dict equal to the plain mapping that also keeps `unreadable`, `{link: why}` for each link whose source answered and whose manifest could not be read. `_chain_link_findings` words such a link "which was reached but could not be read as a declaration (<why>)", in a branch of its own, and only the scaffold asks. The new text is asserted for each of the eleven not-a-list rows, with `scaffold-project.py --dry-run` run as a subprocess. - **Every other path's text is the bytes it was.** The sentence for a link no source reached is the same code in the same lines, and four tests hold it: the generic warning for a plain mapping, for an empty `LinkPins` and for an absent tree; the scaffold's own old warning for a clone that holds no `project.yaml`; the naming validator's two lines (its warning, and the `link-source-unreadable` line that names the reason); and `validate-manifest.py`'s. The naming validator and `validate-manifest.py` do not ask for reasons, so they print what they printed. - **One more scaffold path changes with it, on purpose.** A tree the scaffold reached and could not read as a declaration for any other reason (a manifest that does not parse, or is not a mapping) used to say "not reachable here" too, and was no truer. It now says what it is, with the system's reason, and a test for each of the two (a manifest that is not a mapping, one that does not parse) holds it, at the scaffold's command line and in process: narrowing the wording to the not-a-list reasons fails both. - `_declaration` still returns its four values, so nothing else in `scaffold-project.py` moved. - **A qualified chain link in the spy test** (nit 2). `_chain_read_cases` had no case whose chain link is qualified, so `reads_link_declarations` could take the chain as typed and not through `_chain_links` (R3) and pass every test. With `--referent-chain testorg/openXdox,openDox` that turns a verified chain into a declared-unverified one. `("codexDox", {"openXdox"}, ("testorg/openXdox", "openDox"), True)` is a case now, and R3 fails it. - **The description** (nit 3): the ten corrections of the re-review are applied (the reason a lookup returns, the fifth NOTE reason, the third kind of silent source, the SHOULD-1 bullet, "before Python 3.14", the `validate-pins.py` bullet, the 3.14.7 rows, the merge-tree section, the filed numbers, and the project's own `validate-manifest.py`). The mutation proofs: all three survivors were reproduced first at 4b21263 (210 passed each) and each dies at 2ac911c. Six mutations of the new code each die too: the warning ignoring the reasons (12 cases fail), the lookup recording none (12), the scaffold not asking (11), `with_reasons` on by default (3), an absent tree called reached (3), and the reached sentence used for a tree nothing reached (17). **Where it departs from #312, and why.** - **An explicit source that cannot answer stays a WARNING, exit 0.** #312 made an unreadable `--pin-source` a finding. Three reasons for the difference: - Exit 1 here means "a name is wrong", and this module has always said an unread link tree is a warning. - `SHAPE_PIN_SOURCE_<PRODUCT>` also answers `validate-pins.py`'s digest question, from a checkout that need not hold a manifest. - The lookup falls through to the next source, which may still verify the link. - **The fall-through is kept**, as it always was. With `--pin-source`, a keyed value is final. - **The lines go to stderr, spelled `WARNING`/`NOTE` as this file spells them.** #312's notes are ` note` lines on stdout, where that file's report lives. **Who else this reaches.** The guard is in `repo_shape.py`, so the same `link_pins_from_trees` lookup stops crashing for these callers too: - **The project's own `validate-manifest.py`.** It reaches the lookup through `SHAPE_PIN_SOURCE_<LINK>` or a sibling checkout, and now gives its usual declared-unverified WARNING instead of the traceback. - **`scaffold-project.py --referent-chain`.** A `SHAPE_PIN_SOURCE_<LINK>` under a locked parent no longer kills the plan, and `--pin-source` verifies the link. Neither of these prints the reason a source could not be entered; the scaffold says it only for a tree it reached and could not read (fix round 2, above). A `--pin-source` the scaffold cannot enter is still a traceback: that is #339 (`pin_digest_from_source`), and this PR does not touch it. **What this does not reach, and what the doctor says.** - **The project's gates still end in a traceback on the sibling row.** The NAMING and MANIFEST steps of `make validate`, `validate.yml` and `make bootstrap` stop crashing for a checkout `../openXdox` at mode 000. Those targets also run `validate-pins.py`, whose `resolve_neutral_pin_source` asks `(sibling / ".git").exists()` with no guard (`templates/assembly-root/scripts/validate-pins.py:463` on main), so before 3.14 a sibling at mode 000 still ends them in a `PermissionError`. That file is #312's, and `fix/pin-source-env-collisions` (PR #409, since landed), `fix/adopt-pin-writes-pin-file` and open PRs #343 and #419 touched it when this was written, so this PR does not edit it. The same line is listed for Python 3.14 by #430 (lane 2), where the helper that would cure both halves is asked for; the pre-3.14 traceback reproduction is added there as a comment, not filed apart. A `SHAPE_PIN_SOURCE_<LINK>` under a locked parent is different: `validate-pins.py` makes that a `neutral-pin-source-unreadable` FINDING (#312), and that stays. - **The doctor's `naming` row** runs the STANDARD's copy of the validator, so it stops crashing at once. **Its `manifest` row** runs the project's OWN `validate-manifest.py` on the project's own `repo_shape.py`, so it changes only once the project has applied this change. Over an existing project the doctor says `INVALID (manifest, pins)` until `update-shape.py apply` lands the two `upstream-changed` rows below. - **The verdict.** For `SHAPE_PIN_SOURCE_<LINK>` under a locked parent, on a project that carries these copies, `INVALID (naming, manifest, pins)` becomes `INVALID (pins)`, and `pins` is #312's finding for that locked variable, which is right. For a sibling at mode 000 the `pins` that remains is the traceback above, not a finding. The two copied scripts this change edits (`scripts/repo_shape.py` and `scripts/validate-repository-naming.py`) are copied into every project and pinned there. A project takes the change through `update-shape.py` (two `upstream-changed` rows), and a family holder takes the `repo_shape.py` row. Neither file has a second copy in this repository. **Evidence.** - **Issue rows, main ab623dd** (`repro338.sh`, a `gh` stub first on PATH). Under 3.10.21 and 3.12.3, rows 4-7 (keyed under a mode-000 parent, `SHAPE_PIN_SOURCE_OPENXDOX` there, a sibling at mode 000, a 300-character name) are a traceback, exit 1. Rows 2-3 (`a=b`, the typo) are a silent declared-unverified, exit 0. Under 3.14.7 row 1 verifies, and rows 2-7 are the silent declared-unverified WARNING, exit 0. - **Issue rows, this PR.** Under 3.10.21, 3.11.16, 3.12.3, 3.12.14, 3.13.15, 3.14.7 and 3.15.0rc2 (first commit), all 7 rows exit 0 with no traceback, and the output is byte-identical across versions. `a=b` verifies. The other five rows each name their source with its reason. Re-run on the fix-round head, 4b21263, under 3.10.21, 3.12.3 and 3.14.7: byte-identical to the first commit's run. Re-run on the fix-round-2 head, 2ac911c, under the same three: all 7 rows exit 0 with no traceback, the output is byte-identical across the three versions, and it is identical to the re-review's runs of the previous head over main. - **Reach on main.** Main gives a `PermissionError` traceback, exit 1, for `scaffold --dry-run`, the project's `validate-manifest.py`, `validate-repository-naming.py --project` and its `--quiet` form, each with `SHAPE_PIN_SOURCE_OPENXDOX` under a locked parent. This PR gives exit 0 for all four. `validate-pins.py` on the sibling row is the exception said above. - **New tests.** 17 cases (11 functions) in the first commit; 16 fail on ab623dd. The fix round adds 31 cases in `tests/test_naming_policy.py`, and 17 of them fail on 40762ca: the five shapes of pins that are not a list (each at the CLI, in-process and, for the scalar, under `--project`), and the chains nobody reads. The other 14 guard what a mutation could break (below) or the side the fix must not move (an absent, null or blank pin list still answers). - **Targeted run on 4b21263**, files `test_naming_policy`, `test_manifest_validator`, `test_scaffold_referent_chain_links`, `test_cli_reference`, `test_repo_hygiene`, `test_neutral_pin_validator`, `test_scaffold_e2e`, `test_shape_doctor`, `test_git_output_decoding` and `test_pin_validator`: **765 passed, 3 skipped**. The 3 skips are the PowerShell-only rows. (40762ca: 734 passed, 3 skipped.) `render-cli-reference.py --check` is ok. - **Mutation probes, first commit.** Reverting the stat guard, the parse, keyed-over-directory, the NOTE lines, the chain's-end restriction, the sibling-absence filter, the manifest reason, the `--quiet` drop, the "read as a PATH" sentence, the `./` line, the project-chain read, the WARNING lines or the env reporting each fails at least one new test. - **Targeted runs on 2ac911c (fix round 2).** Under 3.12.3, `test_naming_policy` (238 cases: 210 and 28 new), `test_cli_reference`, `test_manifest_validator`, `test_scaffold_referent_chain_links`, `test_repo_hygiene` and `test_setup_project_py`: **468 passed**; and `test_scaffold_e2e`, `test_scaffold_pin_and_reuse`, `test_scaffold_leg_path_collisions`, `test_scaffold_pin_collisions`, `test_neutral_pin_validator`, `test_git_output_decoding` and `test_pin_validator`: **312 passed**. Under 3.10.21 and under 3.14.7, `test_naming_policy`, `test_manifest_validator` and `test_scaffold_referent_chain_links`: **321 passed** each. `render-cli-reference.py --check` is ok, and `ast.parse(feature_version=(3, 9))` passes on the three changed files. - **Mutation probes, fix round.** The review's ten, re-run on 4b21263: MW1, MW2, MW3, MW5 and MW11 now die (they survived), and MW4, MW7, MW12 and MW13 still die; MW6 is the dead test, removed. (Fix round 2 kills the three the re-review found, R1 to R3, and six more: see above.) Nine more for the new code (the list check, the entry check, the null and blank tolerances, `reads_link_declarations` always true, without the shape check, without the held pins, and the validator ignoring it): eight die. The ninth, `reads_link_declarations` ignoring whether the name has a referent, survives as an equivalent mutant: with no referent `_chain_shape_problem`'s last question always reports a problem, so the explicit early return is redundant, and is kept because `resolve_referent` has the same first line. The spy test holds the two together. - **Windows.** The mode-000 rows skip on Windows and as root, using a `SEARCH_DENIED` mark this file defines, with the same condition as #312's. The new FIFO and `/dev/null` rows skip where those do not exist. `a=b` and the 300-character name run everywhere. On Windows the long-name row asserts the line, not POSIX's reason. None of it had been run on Windows or macOS when this was written; the last paragraph gives the CI result. **Merge-tree.** Recomputed at 19:04Z with `git merge-tree --write-tree` of 2ac911c against `origin/main` as it was then, 53bee5b, and against every one of the other 37 pushed branches that touches a file this PR changes (a scan of each head against its merge-base with main): - **Clean:** origin/main 53bee5b (it carries #314 and #390); #394 `fix/adopt-record-submodule-placement` 563ea16 (`scripts/repo_shape.py` and `validate-manifest.py`) and #406 `fix/validate-family-unnamed-mounts` 369af2e (`scripts/repo_shape.py`); #397 abb27e6, #400 7062a47, #401 409f10b, #416 7cca975, #452 e2df9a8 and #395 1ef3310 (`docs/cli.md`); #355 6fdbe3d, #373 741cbf2 and #395 (`scaffold-project.py`). - **Conflict, four hunks each, all in `scripts/repo_shape.py`:** `fix/pin-source-env-collisions` (PR #409) 6494714, and `fix/adopt-pin-writes-pin-file` 421e4e5 (no PR). Both also touch `scaffold-project.py`, and that merges clean with this PR's one edit there. Fix round 2's new hunks merge with neither: #409's change to the "not reachable here" sentence (the `offered` text, when a variable is shared) and the reached branch sit in different lines of `_chain_link_findings`. The four hunks, each taking both sides: 1. the `resolve_link_source` signature: `default_source=None, env_shared: bool = False, *, tried: list | None = None`; 2. its docstring: both paragraphs; 3. the `link_pins_from_trees` docstring and the lines after it: both paragraphs, then `found`, this PR's `unreadable` and #409's `links` and `shared`; 4. the call: `resolve_link_source(name, root, keyed_sources, default_source, pin_product_key(name) in shared, tried=asked)` (the same spelling on both branches), followed by `if tried is not None: tried[name] = asked`. Resolved that way in a throwaway and committed there, the merge with 6494714 passed **496 tests** (`test_naming_policy`, `test_pin_source_env_collisions`, `test_neutral_pin_validator`, `test_scaffold_referent_chain_links`, `test_manifest_validator`, `test_cli_reference`), and the merge with 421e4e5 passed **559 tests** (the same files and `test_pin_source_unreadable`). `render-cli-reference.py --check` is ok on both, pyflakes is clean, and the grammar parses as 3.9's. Neither branch carried the resolution when this was measured. #409 has since landed, and the merge of main into this branch (the last paragraph) takes those four hunks both ways. The earlier counts (468 and 434 from the first fix round, 430 from the re-review) are those runs' own, at their heads. **Left out.** Each of these was found by lane 5's reviews of this change, and each is filed or folded: - **A bare `--link-source` answers for every link with no more specific source** (#441). In a chain of three or more links, one bare tree is read as two links' declarations, and a chain that holds is reported broken, a FINDING for a name with no direct pin on its referent. Pre-existing, the same on main; `validate-pins.py`'s `--pin-source` has the same documented rule and was not examined. - **An empty path after a key (`--link-source openXdox=`) is `Path("")`, the current directory** (#442). Run from a project root, the project's own `project.yaml` is read as the link's declaration. Pre-existing; `validate-pins.py`'s `--pin-source openXdox=` has the same shape (there a finding naming `.`). - **`validate-pins.py`'s `(sibling / ".git").exists()` is unguarded** (the sibling row above), folded into lane 2's #430, which already covers the Python 3.14 half: the pre-3.14 traceback and its reproduction are added there as a comment. #312's file. - **The project's OWN manifest** with `neutral_product_pins: 5` ends in a `TypeError` in two gates before either can say anything about it (#443): `_pins_from_project` in the naming gate (`scripts/validate-repository-naming.py:302` on main), and `_declared_pins` in the project's own `validate-manifest.py` (`templates/assembly-root/scripts/validate-manifest.py:85`), which ends it before its own `manifest-neutral-product-pins` finding (`_neutral_product_pins_type_finding`, `:524` on main) can print. Reproduced on 3ef059c. That is the project's own declaration, read by the project's own gates, and not a link source, so it is outside this change; "read by the project's own gates" covers two crashing gates. Also not done here: - **Python 3.9 (the floor) was not run** because it is not installed here. The grammar of the three changed files was checked with `ast.parse(feature_version=(3, 9))`. Windows and macOS were not run here; CI ran them (the last paragraph). Reviewed adversarially at 40762ca, the branch's first commit (CHANGES NEEDED: 0 blockers, 2 should-fixes, 4 nits). The head stopped every crash in the issue's table, and the should-fixes were a docstring claim the code did not keep for a `neutral_product_pins:` that is not a list, and reach claims that overstated what the project's gate does on the sibling row. Fix round 1 (4b21263) resolved every finding, and the re-review of 4b21263 read APPROVE WITH NITS (0 blockers, 0 should-fixes, 3 nits: three behaviour changes this description had not stated, two of them untested; a spy-test case for a qualified chain link; and description corrections). Fix round 2 (2ac911c) resolved those three, and its re-verification of 2ac911c read APPROVE WITH NITS (0 blockers, 0 should-fixes, 2 nits: no test of the scaffold's wider wording for a manifest that is not a mapping or does not parse, now pinned by two tests that each fail when the wording is narrowed, and a cross-reference in this description that pointed the wrong way, now corrected). Main (9736424, #409's squash) was merged into the branch as ced8fac. `scripts/repo_shape.py` conflicted in four hunks; each takes both sides, as the review of 2ac911c resolved it: `resolve_link_source(..., default_source=None, env_shared: bool = False, *, tried=None)`, both docstrings kept, `link_pins_from_trees` opening with `found`, `unreadable`, `links` and `shared`, and the call passing `pin_product_key(name) in shared` with `tried=asked`. Nothing else conflicted. After the merge, `tests/test_pin_source_env_collisions.py` and the targeted set passed (667 and 312), and `render-cli-reference.py --check` printed `[ok]`. CI is green on Linux, macOS and Windows, with SonarCloud passing, at 938a131 (main 48585e6 merged in). Its merge with main 994be39 passed the targeted set (the naming, manifest, scaffold, pin and setup test files): 983 passed. Closes #338 Lane: openreposhape-5 (openRepoShape-5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>



Lane: openreposhape-5 (openRepoShape-5)
Claim: #301 (comment)
Claim: #280 (comment)
Two reader-side defects the lane filed, in one commit (21fbea4) on
fix/yaml-flow-brackets-surrogates, cut from main c7c02fa.What changed
#301,
scripts/repo_shape.py. In YAML a plain scalar inside a flow collection ends at any of[ ] { } ,._split_flowcounts brackets to find its commas and never asked whether the count went below zero or ended above it, so a stray bracket was kept as text:pins: [a, b]]read['a', 'b]'];k: [x [a, b]]read['x [a, b]'];k: [it's, a]reads two items,k: it's fine # notedrops its comment, and an unclosed"abcor a mid-key[is refused, not misread (#249, #250) #284,k: {a"]:}read the keya"].PyYAML refuses each. A new
_refuse_flow_indicatorrefuses a PLAIN item or value (called from_flow_value) and a PLAIN key (called from_split_key_flow) of a flow collection that holds[,],{or}. The refusal is aYamlError, soload_yamlturns it intoREFUSED yaml-unparsableand names the scalar:a plain scalar in a flow collection holds `]`, which ends it in YAML: 'b]'. These still read as before:k: ["a]", {"b}": "c]"}]);k: a[b],- see [1]);No other function changed.
#280,
templates/assembly-root/scripts/validate-manifest.py._name_findingnow reports a display name holding any of U+D800 to U+DFFF asmanifest-name:The finding goes to stderr with the name escaped by
repr, before themanifest okline could raise.Why the #280 refusal is in the validator and not in the reader
The brief offered two places. The evidence rules out the reader:
The writer emits these escapes on purpose.
double_quoted('caf\udce9')is"caf\udce9". That is howadopt-project.py's plan keeps a path byte that is not UTF-8 (adopt-project.py plan writes a plan its own loader cannot parse when a source name holds a line break (the macOSIcon\rfile), socheckrefuses the whole plan asyaml-unparsablewith no entry to edit #199), andcheckmust read it back.tests/test_adopt_entry_paths.pyholds that round trip (120 passed withtest_adopt_plan.pyon this head). Lane 1's pending An adopted assembly records nothing about which leg a human chose for a source submodule, so shape-doctor.py cannot verify a kept gitlink's placement and can only note it; adopt should write the plan's submodule answers into the assembly #189 branch (70f4d29,submodule_record) also writes\udce9escapes intoproject.yaml'ssubmodules:block, and its docstring relies on the reader resolving them. A reader refusal would break both.PyYAML 6.0.3 agrees with the reader. The pure-Python
SafeLoaderreadsname: "a\ud83db"as the lone surrogate. Only libyaml (CSafeLoader) refuses it, and it refuses even the escape pair"\ud83d\ude00", which the Python loader reads as two surrogates, as this reader does.The defect is in a value, not in the YAML. A display name is text a person chose, and no UTF-8 stream holds a surrogate. On c7c02fa (
probe280.py, local bare remotes only):scripts/validate-manifest.pyUnicodeEncodeErrortracebackmanifest-namefindingmake validateshape-doctor.py --root .INVALID (manifest), the row quotesUnicodeEncodeError: ...INVALID (manifest), the row quotesFINDING manifest-name: ...bump-leg.pymanifest ok: a\ud83db, and COMMITS the subjectBump spec leg to <sha> in a?b(errors="replace")bump-leg-validators-red, every byte rolled back, 0 commitsupdate-shape.py applymanifest ok: a\ud83db, commitsupdate-validators-red, restored, 0 commitsSo all four launchers now agree, and each names the problem.
Readers and callers this changes
repo_shape.py: a flow collection with an unmatched
]or}is read, not refused, sok: {a"]:}reads{'a"]': None}since #284 (a quote toggle used to swallow the:}) andk: [a]]reads['a]'], where PyYAML refuses both #301: everyparse_yaml/load_yamlcaller, and only for a document whose flow collection holds a plain scalar with a bracket. Such a document was read, or misread, before and is refused now. Every tracked YAML file loads the same on c7c02fa and on this head (14/14 raw, 18/18 rendered templates). So do the 6 YAML files of a scaffolded project (project.yaml, fourcontracts/*.yamlandvalidate.yml): 6/6 on a c7c02fa scaffold and 6/6 on a 21fbea4 scaffold. No writer emits such a scalar:[](adopt'spins: [...]andallow_empty_legs: [...],naming_block'salso_matchesandreferent_chain);double_quotedquotes any value holding a bracket.validate-manifest.py accepts a display name holding a lone surrogate (
name: "a\ud83db") and then raisesUnicodeEncodeErrorprinting it, somake validateand shape-doctor.py fail on a traceback instead of a finding while bump-leg.py and update-shape.pyapplypass the same project once #254 lands #280: the project's ownscripts/validate-manifest.py(a digest-pinned shape copy; it reaches existing projects throughupdate-shape.py apply, which reports itupstream-changed). Every caller of that copy now stops at the finding:make validate;validate.yml;shape-doctor.py'smanifestrow;bump-leg.pyandupdate-shape.py apply(their red-validator refusal and rollback);make validate, which runs each member's.bootstrap.pyruns the same copy in its step (b), but on such a name it dies at its own firstprint, before step (b) (see Left out).With scaffold-project.py and adopt-project.py ask the project's YAML reader and validate-manifest.py about the manifest's values before anything exists, so
--id null,--tracking-branch trueand a blank--elected-byare refused (#258, #261) #314 merged as well, the scaffold's pre-check asks this validator about--name. In the tree of this head and scaffold-project.py and adopt-project.py ask the project's YAML reader and validate-manifest.py about the manifest's values before anything exists, so--id null,--tracking-branch trueand a blank--elected-byare refused (#258, #261) #314 alone,scaffold-project.py --name $'caf\xe9' --dry-runis refusedscaffold-manifest-invalidwith this finding before anything is created (checked at scaffold-project.py and adopt-project.py ask the project's YAML reader and validate-manifest.py about the manifest's values before anything exists, so--id null,--tracking-branch trueand a blank--elected-byare refused (#258, #261) #314's head 4b8da3d). On origin/main, where scaffold-project.py:--elected-by,--reference,--nameand the elector's fallback are refused before anything exists when no file can hold them, so a byte that is not UTF-8 no longer creates repositories before a traceback (#282) #351 has landed and scaffold-project.py and adopt-project.py ask the project's YAML reader and validate-manifest.py about the manifest's values before anything exists, so--id null,--tracking-branch trueand a blank--elected-byare refused (#258, #261) #314 has not, that--nameis already refused, asscaffold-unwritable-value.Evidence
Issue rows. All ten repo_shape.py: a flow collection with an unmatched
]or}is read, not refused, sok: {a"]:}reads{'a"]': None}since #284 (a quote toggle used to swallow the:}) andk: [a]]reads['a]'], where PyYAML refuses both #301 documents reproduce on c7c02fa and are refused on this head. 8 are documents PyYAML refuses. 2 (k: {a:}#}: b},k: {}#":}) PyYAML reads only by taking a#with no space before it for a comment, and they are refused by design, ask: "a"#cis. Every validate-manifest.py accepts a display name holding a lone surrogate (name: "a\ud83db") and then raisesUnicodeEncodeErrorprinting it, somake validateand shape-doctor.py fail on a traceback instead of a finding while bump-leg.py and update-shape.pyapplypass the same project once #254 lands #280 row is in the table above.Seeded oracle fuzz (
fuzz_oracle.py). Corpus: reviewA's, seed 2490250, 785,646 documents in six contexts. PyYAML 6.0.3 is the oracle only.Only two kinds of document change:
#leniency, and 21 are a single-pair mapping in a flow sequence (k: [a:[]]), which the subset never read as a mapping.Nothing c7c02fa read correctly changes, nothing is newly read, and there are 0 crashes. Against the issue's two-line
depth < 0sketch, run on the same corpus, this head refuses everything the sketch refuses, plus 21,631 more: 21,608 invalid documents (the opener class, such ask: [a[]andk: [x [a, b]]) and 23 misreads. The fuzz ran on 38738cf, whosescripts/andtemplates/are byte-identical to 21fbea4's.Tests fail on c7c02fa. With the four test files copied onto c7c02fa: 45 failed, 191 passed. The failures are 31 bracket rows, 10 validator rows and the 4 launchers.
Mutation probes. Each hunk was reverted or weakened on its own, and each mutant fails new tests:
Tests
tests/test_yaml_flow_brackets.py, with the ten issue rows, sixteen more positions, the refusal's wording, a file refusal, fifteen still-read rows and the writer round trip.tests/test_manifest_validator.py: five surrogate spellings, each under a pipe and underPYTHONIOENCODING=utf-8, plus four still-ok controls on either side of the range.tests/test_validator_child_encoding.py: a real bump,update-shape.py apply, the doctor andmake validate, each onname: "a\ud83db".tests/test_yaml_quote_boundaries.py: its docstring saidk: "a"#cwas the only PyYAML disagreement. There are eight; seven areBRACKET_IN_A_KEYrows refused by design (repo_shape.py: a flow collection with an unmatched]or}is read, not refused, sok: {a"]:}reads{'a"]': None}since #284 (a quote toggle used to swallow the:}) andk: [a]]reads['a]'], where PyYAML refuses both #301's NIT-A).Targeted runs on 21fbea4, Linux, Python 3.12.3, git-filter-repo 2.47.0, each in the background:
test_yaml_flow_brackets,test_yaml_flow_quotes,test_yaml_flow_mappings,test_yaml_quote_boundaries,test_yaml_block_scalar_keys,test_load_yaml_refusals,test_adopt_plan_escapes,test_manifest_validator,test_pin_validator,test_neutral_pin_validator,test_template_quoted_values): 462 passed;test_validator_child_encoding,test_stdout_encoding,test_repo_hygiene,test_cli_reference: 165 passed, 1 skipped (test_stdout_encoding.py:286, a Windows-only default);test_adopt_plan,test_adopt_entry_paths: 120 passed, 1 skipped (test_adopt_plan.py:393,SHAPE_MEDXEHR_CLONEunset);test_shape_doctor: 242 passed, 3 skipped (test_shape_doctor.py:5330, PowerShell on a Windows host only);rendersubstitutes in one pass and refuses only a placeholder its table lacks, so--elected-by '{{SPEC_PATH}}'is no longer recorded asspecand{{NOPE}}no longer refuses after the legs are pushed (#260) #316:test_template_quoted_values,test_template_values_single_pass,test_yaml_flow_brackets,test_yaml_quote_boundaries,test_manifest_validator: 412 passed;--id null,--tracking-branch trueand a blank--elected-byare refused (#258, #261) #314 (ab66bc0):test_scaffold_manifest_values,test_adopt_manifest_values,test_manifest_validator,test_yaml_flow_brackets: 293 passed.The full suite and the macOS and Windows legs were not run here.
Merge-tree (
git merge-tree --write-tree --name-only)Re-run at 2026-10-12T15:12Z, against origin/main d82a83e and the heads named here.
rendersubstitutes in one pass and refuses only a placeholder its table lacks, so--elected-by '{{SPEC_PATH}}'is no longer recorded asspecand{{NOPE}}no longer refuses after the legs are pushed (#260) #316 (34ecbc2), adopt-project.py: a relative.gitmodulesurl the plan keeps in a leg is resolved from the source's remote and from the leg's, and where the two differcheckfinds it andexecuterefuses before any leg exists, so a recursive clone no longer fails afteradoption verified(#190) #245 (head a0272b2, landed at 14:55Z as 161c6c9) and scaffold-project.py:--elected-by,--reference,--nameand the elector's fallback are refused before anything exists when no file can hold them, so a byte that is not UTF-8 no longer creates repositories before a traceback (#282) #351 (landed as d82a83e). adopt-project.py: a relative.gitmodulesurl the plan keeps in a leg is resolved from the source's remote and from the leg's, and where the two differcheckfinds it andexecuterefuses before any leg exists, so a recursive clone no longer fails afteradoption verified(#190) #245's one change toscripts/repo_shape.pyis inremote_key, away from the YAML reader, and scaffold-project.py:--elected-by,--reference,--nameand the elector's fallback are refused before anything exists when no file can hold them, so a byte that is not UTF-8 no longer creates repositories before a traceback (#282) #351 touchesscaffold-project.pyonly;--id null,--tracking-branch trueand a blank--elected-byare refused (#258, #261) #314 (4b8da3d). With it merged, the--namerefusal above still holds (checked at 4b8da3d);fix/adopt-late-failure-notice(9f1778d) andfix/adopt-plan-quote-yaml-keywords(d38e143);fix/pin-source-env-collisions(34dbd86, scripts/repo_shape.py:pin_source_env_nameturns every character outside[A-Z0-9_]into_, so the neutral productsopen-glass,open_glassandopen.glassshare oneSHAPE_PIN_SOURCE_OPEN_GLASS, and a project that pins two of them gets aneutral-pin-source-unreadableFINDING (exit 1) for whichever one the variable does not name #275);manifest kindsrow puts back a shape or leg pin undercontracts/that does not load with the git commandpin_restorepicks from its own state, shows a copy the shape pin has a row for or a neutral product pin with a read-onlygit diff, and leaves any other file to a person whatever it is called, so its NEXT no longer tells the reader to delete the pin (#291) #358.manifest kindsrow puts back a shape or leg pin undercontracts/that does not load with the git commandpin_restorepicks from its own state, shows a copy the shape pin has a row for or a neutral product pin with a read-onlygit diff, and leaves any other file to a person whatever it is called, so its NEXT no longer tells the reader to delete the pin (#291) #358 (ebffb08), inshape-doctor.py;fix/adopt-root-kept-symlinks(6c44591), inadopt-project.py.--id null,--tracking-branch trueand a blank--elected-byare refused (#258, #261) #314, adopt-project.py:checkfinds, andexecuterefuses before any leg exists, a planfollow_ups:that is not a list of text and anelected_on:thatelection_datedoes not read, with or without areference:, sofollow_ups: 5is no longer a traceback andelected_on: somedayis no longer refused under a flag neither command takes (#293, #294) #356 (5a5011d), adopt-project.py:checkreports the split branch an earlier run left asexecute'sadopt-branch-exists, and a branch in git's way of it asplan-split-branch-taken, whichexecutenow refuses before any leg exists, so a source branchadoptno longer has both legs made beforegit filter-repofails (#320) #382 (7e55d00) and lane 1'sfix/adopt-plan-quote-yaml-keywordsinadopt-project.py, and lane 1'sfix/adopt-late-failure-noticeintests/conftest.py.fix/adopt-record-submodule-placement(70f4d29, lane 1, An adopted assembly records nothing about which leg a human chose for a source submodule, so shape-doctor.py cannot verify a kept gitlink's placement and can only note it; adopt should write the plan's submodule answers into the assembly #189), in one hunk, the import block oftests/test_manifest_validator.py. Both branches add to it: that branch addsimport osandimport sys, and this one addsimport subprocessbetween them, so the resolution is all three. Itsrepo_shape.pyandvalidate-manifest.pychanges auto-merge. Not resolved in either branch.Review
An independent opus review at 21fbea4 approved it with nits: 0 blocker, 0 should-fix, 5 nit. All five are in text (this body, the writer's report and the commit message), none in the code or the tests. Four are fixed in this body; the fifth, the commit message's spelling of the surrogate escape, is for the squash message. Its own seeded structured fuzz (seed 7301280: 120,000 nested flows in five block contexts, 600,000 documents, PyYAML 6.0.3 as the oracle only) found 0 changed reads: no document c7c02fa read the way PyYAML reads it changed, and nothing crashed. An instrumented reader returned no plain scalar holding
[ ] { }inside a flow on this head, against 76,955 on c7c02fa. The writer's seed 2490250 counts reproduced row for row, and all 13 mutants it tried were killed. 664 passed on this head merged with origin/main ab623dd (16 test files; 361 on the head alone, 8 files). The one conflict, with lane 1's #189 branch, is thetests/test_manifest_validator.pyimport block, and resolves to all three imports. Three follow-ups it confirmed are not in this change: the bootstrap's raw name print and the flow entries still read (both under Left out), and a block value that opens with]or}, which c7c02fa and this head both read where PyYAML refuses.Left out
A plain flow entry that opens with an indicator, and an empty flow entry. Most of what this head still reads where PyYAML refuses is one of two classes:
:,#,?or-(k: {:":}reads{':"': None},k: [:]; 6,286 flow documents);k: [,a]reads['a'],k: [a,,b]reads['a', 'b']; 3,858).Neither is a bracket, and repo_shape.py: a flow collection with an unmatched
]or}is read, not refused, sok: {a"]:}reads{'a"]': None}since #284 (a quote toggle used to swallow the:}) andk: [a]]reads['a]'], where PyYAML refuses both #301 does not claim them.A single-pair mapping in a flow sequence.
k: [a: b]still reads as the string'a: b', where PyYAML reads[{'a': 'b'}]. Only the ones holding a bracket are now refused.templates/assembly-root/scripts/bootstrap.py. It prints the display name raw (bootstrap: <name> (<root>)) before its validators run. On a project named"a\ud83db",make bootstrapstill dies there withUnicodeEncodeError, on c7c02fa and on this head.elected_byandreference. A surrogate in either is still accepted. Neither is printed by the validator, so nothing raises; the scaffold side of such values is scaffold-project.py writes--elected-by,--referenceand--namebetween literal double quotes with no escaping and asks none of them whether a file can hold it, so a line break leaves a project.yaml its own validators refuse, a backslash is recorded as another character (CORP\nickas CORP, a line feed, ick), and a byte that is not UTF-8 dies inwrite_lfafter the legs are pushed #282.The escape pair
"\ud83d\ude00". It is still two surrogates, not one character, as PyYAML's Python loader reads it. The finding asks for the character itself or its one\Uescape instead of the reader inventing a joining rule.Closes #301
Closes #280
🤖 Generated with Claude Code
Summary by Sourcery
Refuse ambiguous flow YAML and surface invalid manifest names consistently across validation tools.
Bug Fixes:
Tests: