adopt-project.py and repo_shape.py: a name that holds a default-ignorable character, which a terminal draws as nothing, is spelled by its code point in `plan`'s lines and in the `unsafe-value` refusal, so a branch `ma<U+034F>in` no longer prints as `main` (#1128) by brettheap · Pull Request #1171 · opensoft/openRepoShape

brettheap · 2026-10-22T18:35:25Z

Stacked on #1120 (base fix/1035-source-branch-shown, head c55e55e3): it lands after #1120, and only after the v1.2.0 tag.

What was wrong

_shown printed a name as it is wherever str.isprintable() passed it, and _spelled_character wrote such a character as it is. str.isprintable() passes 267 of Unicode's Default_Ignorable_Code_Point characters, which a terminal draws as nothing: 263 combining marks, among them U+034F and the variation selectors, and the four Hangul fillers, among them U+3164. Git takes each in a branch or a path, so a source on ma<U+034F>in printed in plan's source line (#1120's) as (main). checked_value's unsafe-value refusal names the value with repr, which printed U+3164 raw on stderr the same way. RV1120 found both (#1120 issuecomment-6319321596).

What changed

scripts/repo_shape.py:
- DEFAULT_IGNORABLE_RANGES: the property's 27 lines of DerivedCoreProperties.txt, joined into 17 ranges, 4174 code points. The table is the same in Unicode 14.0.0, 16.0.0 and 17.0.0, and is written into the code with a citation, not asked of unicodedata. CI runs Python 3.14 and the bench 3.12, and their unicodedata differ.
- DEFAULT_IGNORABLE, one character of the table as a pattern.
- code_point_escaped, taken out of _escaped, which writes exactly what it wrote before.
- visible_repr: repr with each default-ignorable character written as its code-point escape.
- checked_value's unsafe-value refusal names the value with visible_repr: 'ma\u3164in', where it said 'ma<U+3164>in'.
adopt-project.py:
- _shown spells a name that holds a default-ignorable character, whatever str.isprintable() says of it: `ma\u034fin`.
- _spelled_character writes one as its code-point escape, which the plan's reader reads back as the character.
- _shown's signature, and its output for every other character, are unchanged, so adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120's and adopt-project.py: plan written to and cloning <repository> (read only) into name the caller's --out and --work-dir with _shown, as the other lines of the report name a path, so a bidi control, a line feed or a tab in either no longer reaches the terminal as it was typed (#1115) #1154's rows still pass.

scripts/repo_shape.py is a shape copy that every root digest-pins, so this change shows every project an upstream-changed row, as every change to it does. The new names are used inside the file itself: checked_value calls visible_repr. No pinned copy calls checked_value (validate-family.py only names it in a comment); the change is in scripts/repo_shape.py because checked_value lives there, so its refusal cannot change without the upstream-changed row. No pinned script imports a new name, so the skew rule in test_pinned_copy_text.py holds, and its list of names a kept repo_shape.py may lack, NEWER_THAN_A_KEPT_LIBRARY, now holds the four, so it refuses a pinned script that imports one. No sentence in README.md, AGENTS.md or the handbook describes these helpers.

Tests

tests/test_adopt_shown_default_ignorables.py (new, 48 rows, ASCII only; every character outside ASCII is written as an escape):

The table against DerivedCoreProperties.txt, listed again line by line in the test, so no row depends on the Python's Unicode data.
The compiled DEFAULT_IGNORABLE against the table, over all 0x110000 code points: not one more, not one fewer (RV1171's N1).
All 4174 code points through _shown, _spelled_character and checked_value's refusal.
A value that holds three, m U+034F a U+3164 in U+FE0F, through checked_value's refusal and _shown: each is written as its escape and none is raw (RV1171's S1).
The issue's characters (U+034F, U+3164, U+FE0F, and the controls U+200B, U+2060 and U+180E, which were spelled already) through _shown, through a read back of the plan, and end to end through plan. In plan, they appear in the source line, the tracking-branch NOTE, a path in the table, and the unsafe-value refusal on stderr.
The rows RV1120 asked for on adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120's source line (adopt-project.py: follow-ups from RV1120's review of #1120 (#1035): _shown lets 267 invisible characters through, and two cases the tests do not pin #1128 items 2 and 3):
- feat/caf\u00e9 prints as it is;
- a branch named `main`, backticks included, is spelled `\x60main\x60`;
- the branch that is not UTF-8 is spelled `caf\udce9` under a utf-8:surrogateescape stdout, where main wrote the byte 0xE9 itself.

tests/test_adopt_spelled_unambiguous.py: its two sweeps of every character take the default-ignorables apart and hold each to its escape.

tests/test_pinned_copy_text.py: NEWER_THAN_A_KEPT_LIBRARY holds the four new names, so the skew rule refuses a pinned script that imports one (RV1171's N3).

Counts for the new file:

Code	Result
main `961640c68`	28 failed, 20 passed
#1120 alone `c55e55e3` (the 10 `source`-line and item 2-3 rows)	3 failed, 7 passed
this head	48 passed

On main, every failure is for the right reason:
- the table is missing;
- 267 characters come back raw from _shown, _spelled_character and the refusal, and the three in one value are raw in both;
- the compiled pattern is missing too;
- the raw bytes are in the source line, the NOTE, the table and stderr;
- (`main`) is printed as it is;
- caf\xe9 is on stdout.
The 20 rows that pass on main are the controls: the three format characters, the printable names and feat/caf\u00e9.
On adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120 alone, the 3 failures are U+034F, U+3164 and U+FE0F in the source line. Items 2 and 3's three rows pass on adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120 alone: they pin adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120's code, which is right in each case. They kill RV1120's mutants E (_spelled for every branch outside ASCII) and N (str.isprintable() alone, with no backtick rule).

Sweep

Run with /usr/bin/python3 3.12.3 (Unicode 15.0.0), against #1120's head c55e55e3, whose helpers are main's:

Of the 4174 default-ignorable code points, 267 pass str.isprintable() (263 Mn, 4 Lo). Raw on the base: 267. Raw on this head: 0.
Of the 148731 printable code points that are not default-ignorable, none is spelled on this head.
For every code point that is not default-ignorable, _shown, _spelled_character and visible_repr give the same output as the base's _shown, _spelled_character and repr: 0 differ.

Mutants

Each ran against the new file, test_adopt_spelled_unambiguous.py and #1120's test_adopt_plan_source_branch_shown.py (212 rows). Each file was restored afterwards and its sha256 checked.

Mutant	Result
The table empty	killed: the pattern no longer compiles, so every row errors
The table's pattern matches nothing	killed, 20 rows fail
The range U+115F-U+1160 dropped	killed, 3 rows fail
The range U+1D173-U+1D17A dropped	killed, 1 row fails (the table row)
The escape removed from `_spelled_character`	killed, 18 rows fail
`checked_value` back on `repr`	killed, 4 rows fail
The compiled pattern runs one past each range, the table unchanged	killed, 1 row fails (the compiled-pattern row)
`visible_repr` escapes only the first match (`count=1`)	killed, 1 row fails (the three-in-one row)
A pinned script (`validate-family.py`) imports `visible_repr`	killed, 1 row fails (the skew rule's `validate-family` row); passes on the old list

Gate

On a scratch merge commit of origin/main 038dafba9, #1120's c55e55e3 and this head (39cb15180), in the landing gate's environment with a stub gh:

94 test files, 9806 ids, in 6 shards: every test file that mentions _shown, _spelled or unsafe-value, with test_tracking_branch_ref.py, test_pinned_copy_text.py, test_adopt_plan_escapes.py, test_stdout_encoding.py, test_non_utf8_name_callers.py, test_path_guard_hygiene.py, test_repo_hygiene.py and test_scaffold_leg_path_collisions.py. 9777 passed, 29 skipped, 0 failed.
test_shape_doctor.py, 448 ids in 8 shards from mapfile lists: 445 passed, 3 skipped, 0 failed.
On a merge of main, adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120, this head and adopt-project.py: plan written to and cloning <repository> (read only) into name the caller's --out and --work-dir with _shown, as the other lines of the report name a path, so a bidi control, a line feed or a tab in either no longer reaches the terminal as it was typed (#1115) #1154, the rows of adopt-project.py: plan written to and cloning <repository> (read only) into name the caller's --out and --work-dir with _shown, as the other lines of the report name a path, so a bidi control, a line feed or a tab in either no longer reaches the terminal as it was typed (#1115) #1154, of adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120 and of this PR: 166 passed, 1 skipped.
Fix round 1 (RV1171's S1, N1 and N3), test-only, so no doctor shards: on a scratch merge 331dc93c7 of origin/main 961640c68 and 8162b874e, test_adopt_shown_default_ignorables.py, test_adopt_spelled_unambiguous.py, test_pinned_copy_text.py, test_adopt_plan_source_branch_shown.py, test_stdout_encoding.py, test_path_guard_hygiene.py and test_repo_hygiene.py: 496 passed, 1 skipped, 0 failed.

Merge-trees

git merge-tree of this head with origin/main is clean. So is each of the 13 other open PRs that touch adopt-project.py or scripts/repo_shape.py, against both this head and the scratch merge: #1154, #1153, #1143, #1141, #1126, #1125, #1123, #1122, #1121, #1119, #1118, #1106 and #1105.

Fix round 1: git merge-tree of 8162b874e with origin/main and with #1105 (the one other open PR that touches tests/test_pinned_copy_text.py) is clean, and so is the scratch merge with #1105.

Out of scope: filed separately by the lane

The other helpers that test only str.isprintable(), or use repr, let the same characters through. They are:

the repr-based _shown copies in validate-repository-naming.py, validate-manifest.py and validate-pins.py, and repo_shape.shown_path;
shape_materialize.py's --leading unsafe-value refusal;
repo_shape.double_quoted, which writes them raw into the plan file;
adopt-project.py's _worded, _line_shown, unwritable_values and _refuse_an_unwritable_clone_url.

They are filed separately by the lane as #1165 (see #1165), and are untouched here.

🤖 Generated with Claude Code
Lane: openreposhape-4 (openRepoShape-4)

Summary by Sourcery

Make terminal-invisible Unicode characters unambiguous wherever branch and path names are displayed or refused.

New Features:

Spell Unicode default-ignorable characters in displayed names and plan output using explicit code-point escapes.
Render unsafe-value refusals with visible representations that expose default-ignorable characters.

Bug Fixes:

Prevent branch and path names containing terminal-invisible characters from being misread as different names in plans, notes, tables, and error messages.

Enhancements:

Centralize the Unicode Default_Ignorable_Code_Point ranges and escaping behavior in repo_shape.py while preserving existing output for other characters.

Tests:

Add exhaustive coverage for all 4,174 default-ignorable code points, helper behavior, plan output, refusals, and Unicode table consistency.
Extend character-spelling sweeps to account for default-ignorable characters.

…able character, which a terminal draws as nothing, is spelled by its code point in `plan`'s lines and in the `unsafe-value` refusal, so a branch `ma<U+034F>in` no longer prints as `main` (#1128) `_shown` printed a name as it is wherever `str.isprintable` passed it, and that passes 267 of Unicode's Default_Ignorable_Code_Point characters: 263 combining marks, among them U+034F and the variation selectors, and the four Hangul fillers, among them U+3164. `checked_value` named the value it refused with `repr`, which prints the same 267 as they are. - scripts/repo_shape.py: `DEFAULT_IGNORABLE_RANGES`, the property's 17 joined ranges (4174 code points, the same in Unicode 14.0.0, 16.0.0 and 17.0.0), written as a fixed table so that Python 3.12 and 3.14 agree; `DEFAULT_IGNORABLE`; `code_point_escaped`, taken out of `_escaped`, which is unchanged; `visible_repr`; and `checked_value`'s `unsafe-value` refusal names the value with `visible_repr`. - adopt-project.py: `_shown` spells a name that holds one, and `_spelled_character` writes one as its code-point escape, which the plan's reader reads back as the character. - tests/test_adopt_shown_default_ignorables.py (new, 46 rows), with the rows RV1120's review asked for on #1120's `source` line: a printable branch outside ASCII prints as it is, a branch named `main` between backticks is spelled, and the branch that is not UTF-8 is spelled under a `utf-8:surrogateescape` stdout too. - tests/test_adopt_spelled_unambiguous.py: its two sweeps of every character take the default-ignorables apart and hold each to its escape. Stacked on #1120 (see #1035). Lane: openreposhape-4 (openRepoShape-4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sourcery-ai · 2026-10-22T18:38:19Z

Reviewer's Guide

This PR prevents terminal-invisible Unicode Default_Ignorable_Code_Point characters from being displayed raw in adoption plans or unsafe-value diagnostics. It adds a pinned Unicode range table and shared escaping/representation helpers, applies them to plan rendering and refusal messages, and validates all 4,174 code points alongside existing printable, backtick, and surrogate-escape behavior.

Sequence diagram for visible unsafe-value diagnostics

sequenceDiagram
    participant Caller
    participant checked_value
    participant visible_repr
    participant DEFAULT_IGNORABLE
    Caller->>checked_value: checked_value(what, value)
    checked_value->>visible_repr: visible_repr(text)
    visible_repr->>DEFAULT_IGNORABLE: Match default-ignorable characters
    DEFAULT_IGNORABLE-->>visible_repr: Matching characters
    visible_repr-->>checked_value: repr with code-point escapes
    checked_value-->>Caller: Refusal with visible value

Flow diagram for escaping default-ignorable characters

flowchart LR
    Name[Branch or path name] --> Shown[_shown]
    Shown --> Ignorable{DEFAULT_IGNORABLE match}
    Ignorable -->|no| Raw[Display unchanged]
    Ignorable -->|yes| Spelled[_spelled_character]
    Spelled --> Escape[code_point_escaped]
    Escape --> Plan[Plan contains readable code-point escape]

File-Level Changes

Change	Details	Files
Centralize Unicode Default_Ignorable_Code_Point detection and code-point escaping so visually empty characters cannot be mistaken for ordinary branch or path names.	Added a version-stable, cited 4,174-code-point range table and regex rather than relying on interpreter-specific Unicode data. Extracted code-point escaping and added visible_repr for safe diagnostic representations. Updated unsafe-value refusals to use visible_repr without changing other escaping behavior.	`scripts/repo_shape.py`
Make adoption-plan display and serialization unambiguous for default-ignorable characters while preserving existing handling for printable names, backticks, and surrogate escapes.	Updated _shown to reject names containing default-ignorables before returning them raw. Updated _spelled_character to emit code-point escapes that the plan reader round-trips. Imported the shared detection and escaping helpers.	`adopt-project.py`
Add exhaustive regression coverage for Unicode data, helper behavior, plan output, and compatibility with prior branch-display rules.	Added table validation and sweeps covering all 4,174 default-ignorable code points plus representative controls and printable names. Covered source lines, tracking-branch notes, path tables, unsafe-value stderr, YAML round-tripping, backticks, and non-UTF-8 branch names. Extended unambiguous-spelling sweeps to assert default-ignorables are escaped.	`tests/test_adopt_shown_default_ignorables.py` `tests/test_adopt_spelled_unambiguous.py`

Possibly linked issues

adopt-project.py: follow-ups from RV1120's review of #1120 (#1035): _shown lets 267 invisible characters through, and two cases the tests do not pin #1128: The PR directly addresses the issue by escaping default-ignorables in output and refusals and adding the specified regression tests.

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment
@sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment
@sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

brettheap · 2026-10-23T02:52:03Z

RV1171: independent review of #1171 at 159815d7: NOT READY

Independent adversarial review for lane 4. The code is right everywhere I measured: the table is Unicode's, and only the 267 characters the issue names change, in every helper. One SHOULD keeps it from READY. The refusal's promise to escape EACH default-ignorable character has no row with two in one value, and a visible_repr that escapes only the first passes all 210 rows. It takes one row to fix. There are also five NITs.

Heads and merge

PR head 159815d7, its base adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120 c55e55e3, current origin/main 961640c68.
git merge-tree --write-tree origin/main 159815d7: exit 0, tree 42742bf2e. The merged tree differs from main only in the PR's four files and adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120's tests/test_adopt_plan_source_branch_shown.py.
Every gate, probe and mutant below ran on the scratch merge commit 9de8cbdd9 (main + this head), unless it says otherwise.

1. The table

I fetched DerivedCoreProperties.txt for Unicode 14.0.0, 16.0.0 and 17.0.0 from unicode.org.
Each version lists the property on 27 lines. Joined where they touch, they make 17 ranges and 4174 code points, which equals each file's own Total code points.
DEFAULT_IGNORABLE_RANGES equals the joined ranges exactly in all three versions. The ranges are sorted and neither overlap nor touch.
The test's DERIVED_CORE_PROPERTIES equals the 27 lines of each file, in order.
The compiled DEFAULT_IGNORABLE matches exactly those 4174 of the 0x110000 code points (Python 3.12.3, Unicode 15.0.0).

2. The sweep: every code point, #1120's head against this head

I ran six outputs for each of the 1114112 code points c:

_shown("ma" + c + "in") and _shown(c);
_spelled_character(c) and _spelled("ma" + c + "in");
the checked_value refusal's detail;
repr on the base against visible_repr on this head.

Output	Differ	Outside the table
each of the six	267	0

The 267 are exactly the default-ignorables str.isprintable() passes: 263 Mn and 4 Lo.
On this head, no output holds a raw default-ignorable character. Each of the 4174 comes out as its code-point escape, exactly: `ma\u034fin`, 'ma\u034fin', \U000e0100.
No printable character outside the table is spelled: 0 of the 148731 printable code points outside it. That agrees with the body's figure.
The merge is the same: the sweep of 9de8cbdd9 is byte-identical to the sweep of 159815d7.
The escape is unambiguous and reads back:
- visible_repr stays injective. repr writes a literal backslash as two, so 'ma\u034fin' arises only from U+034F.
- Each of the 4174, as _spelled("ma" + c + "in") copied between double quotes, reads back through parse_yaml as the name: 0 failures (the new rows do this for six).

3. The refusal's new wording, the readers, and the pinned copy

The change: unsafe-value used to print U+3164 raw, as 'ma + U+3164 + in'. It now prints 'ma\u3164in'. For every character outside the table the text is byte for byte the same (the sweep above).
No test relies on the old text. Outside this PR's own new lines, no tracked file names any of the 267, raw or escaped. The only raw default-ignorables in the repository are U+200D (tests/test_adopt_pins.py:956, :1021) and U+FEFF (tests/test_shape_advisory.py:727). Both are format characters repr already escaped, so their output does not move.
No doc or reader relies on it. No doc quotes the refusal, and nothing parses the split commit's message or a _shown name back.
- setup-project.py's own refusal (refuse_value, about :522) uses ascii() and is untouched.
- The v1.2.0 notes' known limit (docs/release-notes/v1.2.0.md, about :1596-1603) names adopt-project.py: follow-ups from RV1120's review of #1120 (#1035): _shown lets 267 invisible characters through, and two cases the tests do not pin #1128. It stays true for v1.2.0, which this PR lands after.
The skew rule holds. test_no_pinned_script_imports_what_an_older_library_lacks passes (gate below).
- git grep finds the four new names only in adopt-project.py, scripts/repo_shape.py and tests/test_adopt_spelled_unambiguous.py.
- No pinned script (validate-pins, validate-repository-naming, validate-family, validate-manifest, siblings, either bootstrap.py) imports one.
scripts/repo_shape.py changes, so every root sees upstream-changed scripts/repo_shape.py at its next update-shape.py check, as the body says. See N2 on the reason the body gives.

4. End to end: `plan` with a stub `gh` and local sources

The setup: each source has a branch ma<c>in beside main and a root file ma<c>in.md, planned with --tracking-branch main. A source whose only branch is ma<c>in gives the tracking-branch NOTE, and the unsafe-value refusal when planned with no --tracking-branch.
The characters: U+034F, U+3164 and U+FE0F, with U+200B as the control.

Tree	Raw bytes of the character in stdout/stderr (3 surfaces x 3 characters)
#1120 `c55e55e3`	9 of 9
this head	0
the merge `9de8cbdd9`	0

The lines on this head, one character each:

source     .../probe-pr/U034F/s1 @ a876039c7162 (`ma\u034fin`), 3 files, 1 commits
`ma\u3164in.md`                    spec       medium       1 extension-majority
NOTE the tracking branch is `main`, which is no branch of the source (its branches: `ma\ufe0fin`; the source's default branch is `ma\ufe0fin`). ...
REFUSED unsafe-value: --tracking-branch (not given, so the source's default branch) is 'ma\u034fin', which is not a value this tool will put on a `git` or `gh` command line

The refusal exits 2 and writes no plan.
U+200B is spelled on both trees.
check on those plans prints none of the characters raw.
The plan file still holds them raw (default_branch: "ma + U+034F + in"), because double_quoted writes them. That is validate-repository-naming.py, validate-manifest.py, validate-pins.py, repo_shape.py, shape_materialize.py and adopt-project.py: the helpers that spell a name for a line or for the plan test str.isprintable() or use repr, so a combining mark, a variation selector or a Hangul filler in a name prints raw, and ma U+034F in reads as main #1165's item 3.

5. Mutants

Setup: each ran on the merge, against the new file, test_adopt_spelled_unambiguous.py and adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120's test_adopt_plan_source_branch_shown.py. The baseline was 210 passed.
Restored: each file was restored afterwards, and its sha256 checked.

Mutant	Result
M1: the range U+3164 dropped	killed, 9 failed
M2: the range U+E0000-U+E0FFF dropped (holds the printable U+E0100-U+E01EF)	killed, 3 failed
M3: a range's end one short, U+FE00-U+FE0E	killed, 9 failed
M4: a range's end one past, U+180B-U+1810 (U+1810 is a printable digit)	killed, 1 failed: the table row alone
M10: a range's start one past, U+17B5-U+17B5	killed, 3 failed
M5: `_spelled_character` back to raw	killed, 18 failed
M6: `checked_value` back to `{text!r}`	killed, 4 failed
M6b: `visible_repr` returns `repr(str(value))`	killed, 4 failed
M7: `_shown` without the `DEFAULT_IGNORABLE` test	killed, 10 failed
M8: `_shown` asks `.match` where it asked `.search`	killed, 10 failed
M9: the escape applied before `repr` (`'ma\\u034fin'`)	killed, 7 failed
M11: the compiled pattern runs one past each range, the table unchanged	survives, 210 passed
M12: `visible_repr` escapes only the first match (`count=1`)	survives, 210 passed

M12 measured outside the tests:
- this head refuses m U+034F a U+3164 in U+FE0F as 'm\u034fa\u3164in\ufe0f';
- M12's form is 'm\u034fa + U+3164 + in + U+FE0F + ', with two characters raw.
M11: this head's compiled pattern equals the table (4174), so M11 is a gap in the tests, not in the code.

6. Merge-trees against the open PRs

Each of the 19 open PRs that touch adopt-project.py or scripts/repo_shape.py merges cleanly (exit 0) with this head and with the scratch merge. The heads fetched are the heads gh pr list gave.

Besides #1120 itself, five change a line that names _shown, checked_value or isprintable: #1180, #1178, #1177, #1154 and #1125. None of them changes the helpers. #1180 adds an isprintable() check of its own (N5).

7. Gate

Linux, the landing gate's environment, a stub gh, on 9de8cbdd9.

The files:

the new file, test_adopt_spelled_unambiguous, test_pinned_copy_text and test_stdout_encoding;
every test file that names unsafe-value or checked_value, which is 40 files and includes adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120's;
test_repo_hygiene, test_path_guard_hygiene (tests/test_path_guard_hygiene.py and shape-doctor.py: a bare is_file/is_dir/exists guard in a shipped tool fails by file and line unless its _is_a or a reasoned allowlist covers it, and a project.yaml linked into an unsearchable folder is yaml-unreadable (#430) #1111's rule) and test_scaffold_leg_path_collisions.

That is 46 files and 5062 ids, in 8 shards. test_shape_doctor.py is 448 ids in 8 shards. Every shard ran from a bash script with mapfile, every list non-empty, every collected id in exactly one shard, no timeout under 600 s.

Shards	Result
g01-g08 (5062 ids)	4880 passed, 182 skipped, 0 failed
d01-d08 (448 ids)	441 passed, 7 skipped, 0 failed
the 160 ids those runs skipped because `git-filter-repo` was not on PATH (156 of the 182 and 4 of the 7), run again with it on PATH	159 passed, 1 skipped, 0 failed

Per shard: g01 614/19s, g02 592/41s, g03 582/51s, g04 590/43s, g05 617/16s, g06 632/1s, g07 622/11s, g08 631/0s (passed/skipped).
d01-d08 per shard: 56, 56, 56, 56, 56, 52/4s, 56, 53/3s.
The re-run's one skip is the row's own (test_leg_path_yaml_scalars.py:581: "the templates read 2026 back ... so the rule has nothing to refuse").
Net: 5035 of the 5062 gate ids and 445 of the 448 doctor ids passed, 0 failed anywhere. The doctor's 3 skips match the body's. I did not re-run the other 26 gate skips, and their reasons were not recorded (the shards ran with -rfE).

Findings

S1 (SHOULD): no row puts two default-ignorable characters in one value, so visible_repr's "each" is unpinned (M12 survives).

What the PR promises: "with each default-ignorable character written as its code-point escape", and the refusal is the surface adopt-project.py: follow-ups from RV1120's review of #1120 (#1035): _shown lets 267 invisible characters through, and two cases the tests do not pin #1128 names.
What the rows cover: every one holds one character per value: the 4174 sweep, the six characters, the end-to-end rows.
So a regression goes unseen: a visible_repr that escapes only the first (M12, count=1) passes all 210, and it brings adopt-project.py: follow-ups from RV1120's review of #1120 (#1035): _shown lets 267 invisible characters through, and two cases the tests do not pin #1128's defect back for a value holding two. Measured above, its refusal of m U+034F a U+3164 in U+FE0F leaves U+3164 and U+FE0F raw on stderr.
Fix: one row. Give checked_value (and, for symmetry, _shown) a value holding the three issue characters together. Assert that the refusal holds 'm\u034fa\u3164in\ufe0f', that _shown gives `m\u034fa\u3164in\ufe0f`, and that neither holds a raw byte of any of the three.

N1 (NIT): the compiled pattern is never checked against the table (M11 survives; M4 dies only on the table row).

Why the sweeps miss it: test_adopt_spelled_unambiguous.py's two sweeps take their expected value from DEFAULT_IGNORABLE itself. A pattern that matches more than the table therefore spells extra characters, and every row still passes.
The cost is small: that over-spells, which is the safe direction.
Fix: in test_the_table_is_unicodes, add assert [code for code in range(0x110000) if repo_shape.DEFAULT_IGNORABLE.match(chr(code))] == IGNORABLE. This head passes it (measured: 4174, equal). It fits the S1 round.

N2 (NIT): the body's reason for the shape bump is not so.

The claim: "a root's validate-family.py calls checked_value".
What the file holds: templates/family-root/scripts/validate-family.py imports no checked_value (its repo_shape import, about :183-188) and names it only in a comment (about :1761).
No pinned copy calls it. Its callers are adopt-project.py, scaffold-project.py, scripts/bump-leg.py, scripts/family.py, scripts/shape_materialize.py, setup-project.py and update-shape.py, so inside a root the new code never runs.
The placement is still right. checked_value lives in repo_shape.py, so its refusal cannot change without the upstream-changed row. That makes this the exception to the rule shape_materialize.py keeps ("IT LIVES HERE AND NOT IN scripts/repo_shape.py", about :1602, :2526, :2980, :4749).
Fix: say that in the body.

N3 (NIT): the skew rule's list does not grow with this PR's names.

NEWER_THAN_A_KEPT_LIBRARY (tests/test_pinned_copy_text.py, about :782) still holds only shown_path and host_shell_dialect.
A kept root repo_shape.py older than this PR has none of DEFAULT_IGNORABLE_RANGES, DEFAULT_IGNORABLE, code_point_escaped and visible_repr.
validate-repository-naming.py, validate-manifest.py, validate-pins.py, repo_shape.py, shape_materialize.py and adopt-project.py: the helpers that spell a name for a line or for the plan test str.isprintable() or use repr, so a combining mark, a variation selector or a Hangul filler in a name prints raw, and ma U+034F in reads as main #1165 is the change most likely to want one in a pinned validator. Add the four here or in validate-repository-naming.py, validate-manifest.py, validate-pins.py, repo_shape.py, shape_materialize.py and adopt-project.py: the helpers that spell a name for a line or for the plan test str.isprintable() or use repr, so a combining mark, a variation selector or a Hangul filler in a name prints raw, and ma U+034F in reads as main #1165, so the skew rule refuses that import.

N4 (NIT, for #1165): two docstrings no longer hold for the 267.

_spelled's "THE SPELLING IS THE PLAN'S OWN": the plan file holds them raw (double_quoted, validate-repository-naming.py, validate-manifest.py, validate-pins.py, repo_shape.py, shape_materialize.py and adopt-project.py: the helpers that spell a name for a line or for the plan test str.isprintable() or use repr, so a combining mark, a variation selector or a Hangul filler in a name prints raw, and ma U+034F in reads as main #1165 item 3), while the spelling escapes them.
unwritable_values: it says its value is spelled "as unsafe-value spells one", but it still uses !r.
Both move with validate-repository-naming.py, validate-manifest.py, validate-pins.py, repo_shape.py, shape_materialize.py and adopt-project.py: the helpers that spell a name for a line or for the plan test str.isprintable() or use repr, so a combining mark, a variation selector or a Hangul filler in a name prints raw, and ma U+034F in reads as main #1165.

N5 (NIT, outside this PR): #1180 adds another member of #1165's class.

Where: adopt-project.py: check and execute run with GIT_NO_LAZY_FETCH=1 and refuse a partial-clone source that lacks an object its branches, tags or HEAD reach as source-partial-clone, so neither fetches into the source any more (#1145) #1180 (2c8082a2a) adds _refetch, which quotes a remote for a person to paste wherever remote.isprintable() and not remote.startswith("-").
What happens: a remote named ma U+034F in passes that test and is printed raw.
Note it on validate-repository-naming.py, validate-manifest.py, validate-pins.py, repo_shape.py, shape_materialize.py and adopt-project.py: the helpers that spell a name for a line or for the plan test str.isprintable() or use repr, so a combining mark, a variation selector or a Hangul filler in a name prints raw, and ma U+034F in reads as main #1165 or adopt-project.py: check and execute run with GIT_NO_LAZY_FETCH=1 and refuse a partial-clone source that lacks an object its branches, tags or HEAD reach as source-partial-clone, so neither fetches into the source any more (#1145) #1180.

Out of scope, as the brief says

#1165's helpers are untouched by this PR, and I did not review them.

Lane: openreposhape-4 (openRepoShape-4)

…a value that holds three default-ignorable characters is refused and shown with each one escaped, the compiled `DEFAULT_IGNORABLE` is held to the table, and the skew rule knows the four new names (#1128) RV1171's review of #1171 (issuecomment-6356224053), the SHOULD and two of the NITs. - tests/test_adopt_shown_default_ignorables.py (46 rows, now 48): - S1: `m` U+034F `a` U+3164 `in` U+FE0F through `checked_value`'s `unsafe-value` refusal and through `_shown`: each is written as its escape and none is raw. Every other row held one character to a value, so a `visible_repr` that escaped only the first match passed all of them. - N1: the compiled `DEFAULT_IGNORABLE` matches exactly the table's code points over all 0x110000. The sweeps take their expected value from the pattern, so a pattern one past each range passed them. - tests/test_pinned_copy_text.py: `DEFAULT_IGNORABLE_RANGES`, `DEFAULT_IGNORABLE`, `code_point_escaped` and `visible_repr` join `NEWER_THAN_A_KEPT_LIBRARY`, so the skew rule refuses a pinned script that imports one. Stacked on #1120 (see #1035). Test-only. Lane: openreposhape-4 (openRepoShape-4) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

sonarqubecloud · 2026-10-23T03:08:18Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

brettheap · 2026-10-23T07:47:59Z

RV1171r2: independent re-check of #1171 at 8162b874: READY

Independent adversarial re-check for lane 4, after fix round 1. RV1171's SHOULD (S1) and its NITs N1, N2 and N3 are resolved, and each is measured below. N4 and N5 are left for #1165, as RV1171 said. I found no MUST and no SHOULD. One new NIT (N6) is about tracking, not about the code.

Heads and merge

PR head 8162b874, one commit on 159815d7; gh pr view says draft, open, CLEAN. I did not change its state.
origin/main is 961640c68 (fetched again just before posting).
git merge-tree --write-tree origin/main 8162b874: exit 0, tree 37444fdc4. The merged tree differs from main in six files only: this PR's five (adopt-project.py, scripts/repo_shape.py, tests/test_adopt_shown_default_ignorables.py, tests/test_adopt_spelled_unambiguous.py, tests/test_pinned_copy_text.py) and adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120's tests/test_adopt_plan_source_branch_shown.py.
Gate and merge checks ran on my scratch merge commit 9429f8799 (main 961640c68 + 8162b874, same tree). The mutants ran on the PR head itself.

1. Scope of the fix round

git diff 159815d7 8162b874 touches two files: tests/test_adopt_shown_default_ignorables.py (+35) and tests/test_pinned_copy_text.py (+8/-2). Nothing else; no source file moved.
Both are ASCII in what the commit adds: 0 of the 41 added lines hold a byte outside ASCII, and the new test file holds none at all. One precision on the brief's "both are ASCII only": tests/test_pinned_copy_text.py already holds 9 lines with non-ASCII characters (at lines 204, 291-300, 424, 461-505), the same 9 on origin/main and on 159815d7. The fix round adds none.

2. The four findings

S1 (SHOULD): resolved. test_each_of_three_default_ignorables_in_one_value_is_escaped takes m U+034F a U+3164 in U+FE0F.

It asserts checked_value refuses it as unsafe-value, that the detail holds is 'm\u034fa\u3164in\ufe0f', exactly, that _shown returns `m\u034fa\u3164in\ufe0f` exactly, and that none of the three raw characters is in the detail or in the shown text.
M12 (count=1) now fails it, and so do two mutants I added: count=2 (M13) and "only the last match" (M14). See the table.

N1 (NIT): resolved. test_the_compiled_pattern_matches_the_table_and_nothing_else lists every code point of 0x110000 that DEFAULT_IGNORABLE.match takes and asserts it equals the table's 4174, as RV1171 asked.

It reads only re and the file's own table, so it depends on no unicodedata. It is not among the slowest five rows of the gate (all at or under 1.7 s).
M11 now fails it, and fails nothing else.

N2 (NIT): resolved. The PR body now says: "No pinned copy calls checked_value (validate-family.py only names it in a comment); the change is in scripts/repo_shape.py because checked_value lives there, so its refusal cannot change without the upstream-changed row."

I checked it against the tree: git grep checked_value -- templates finds one hit, the comment at templates/family-root/scripts/validate-family.py:1761. No file in templates/ and no other pinned script imports it. Its callers are the root scripts RV1171 listed.
The body counts the new file's rows as 48 (the file collects 48) and lists the S1 and N1 rows. Its table of counts is right: I ran the new file against origin/main (with adopt-project.py: plan's source line names the source's branch as _shown names a path, so a bidi control, a line boundary or a byte that is not UTF-8 in it is spelled and no longer reaches the terminal raw (#1035) #1120's helper file beside it) and got 28 failed, 20 passed, and both new rows are among the failures.

N3 (NIT): resolved. NEWER_THAN_A_KEPT_LIBRARY now holds shown_path, host_shell_dialect, DEFAULT_IGNORABLE_RANGES, DEFAULT_IGNORABLE, code_point_escaped and visible_repr, with a comment naming #1128.

I tested that the list does the work. I made validate-family.py import visible_repr from repo_shape. With this head's test file, test_no_pinned_script_imports_what_an_older_library_lacks[validate-family] fails (assert 'visible_repr' not in {...}). With 159815d7's test file the same mutant passes all 7 skew rows. So the four names make the difference.

N4 and N5: left for #1165, as RV1171 said. N4 (_spelled's docstring and unwritable_values's !r) moves with #1165, whose body already names unwritable_values. N5 is #1180's _refetch; see N6.

3. Mutants

Run on the PR head (8162b874, unmerged), against the two changed test files and tests/test_adopt_spelled_unambiguous.py: 280 rows. The baseline is 280 passed. Each mutant was applied to scripts/repo_shape.py with RV1171's own definitions for M11 and M12, then the file was restored with git checkout and its sha256 checked. All four tracked files read back OK after each run.

Mutant	Result
M11: the compiled pattern runs one past each range (`high + 1`), the table unchanged	killed, 1 failed: `test_the_compiled_pattern_matches_the_table_and_nothing_else`; 279 passed
M12: `visible_repr` escapes only the first match (`count=1`)	killed, 1 failed: `test_each_of_three_default_ignorables_in_one_value_is_escaped`; 279 passed
M13: `visible_repr` escapes only the first two (`count=2`), mine	killed, the same row
M14: `visible_repr` escapes only the last match, mine	killed, the same row
Skew: `validate-family.py` imports `visible_repr`	killed, 1 failed (the `validate-family` skew row); passes with the old list

Both M11 and M12 fail for the right reason. M12's refusal read 'm\u034fa + U+3164 + in + U+FE0F + ', with two characters raw, so the exact-detail assertion failed. M11's matched list was 17 items longer than the table, one for each range.

4. Gate

Linux, the landing gate's environment, a stub gh, /usr/bin/python3 3.12, on 9429f8799.

Files	Result
`test_adopt_shown_default_ignorables` (48), `test_adopt_spelled_unambiguous` (145), `test_pinned_copy_text` (87), `test_repo_hygiene` (101)	381 passed
`test_adopt_plan_source_branch_shown`, `test_stdout_encoding`, `test_path_guard_hygiene` (the rest of W1128c's seven)	115 passed, 1 skipped (`test_stdout_encoding.py:285`: "the default this is about is Windows' own")

That is 496 passed, 1 skipped, 0 failed: W1128c's count, exactly. RV1171's full gate on 159815d7 passed and the fix round touches tests only, so I ran no doctor shards and no full gate.

5. CI on `8162b874`

Run 39893261504 (head sha 8162b874): tests success (1h39m46s), tests-git-bash success, package success, SonarCloud Code Analysis success. The check-runs API for the commit agrees.

New finding

N6 (NIT, tracking): #1165 does not yet name N5's site. #1165's body names unwritable_values (N4) but has no mention of #1180's _refetch, which quotes a remote for a person to paste wherever remote.isprintable() and not remote.startswith("-"); a remote named ma U+034F in passes and prints raw. RV1171 said to note it on #1165 or #1180. A line on #1165 naming _refetch keeps it from being missed. No change to this PR.

I found nothing else. The fix round changes tests only, and the code RV1171 measured at 159815d7 is unchanged.

Lane: openreposhape-4 (openRepoShape-4)

brettheap added this to the v1.2.1 milestone Oct 23, 2026

Navigation Menu

Repository navigation

adopt-project.py and repo_shape.py: a name that holds a default-ignorable character, which a terminal draws as nothing, is spelled by its code point in `plan`'s lines and in the `unsafe-value` refusal, so a branch `ma<U+034F>in` no longer prints as `main` (#1128) - #1171

brettheap commented Oct 22, 2026 •

edited

Loading

Uh oh!

sourcery-ai Bot commented Oct 22, 2026

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

brettheap commented Oct 23, 2026

Uh oh!

sonarqubecloud Bot commented Oct 23, 2026

Uh oh!

brettheap commented Oct 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

brettheap commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was wrong

What changed

Tests

Sweep

Mutants

Gate

Merge-trees

Out of scope: filed separately by the lane

Summary by Sourcery

Uh oh!

sourcery-ai Bot commented Oct 22, 2026

Reviewer's Guide

Sequence diagram for visible unsafe-value diagnostics

Flow diagram for escaping default-ignorable characters

File-Level Changes

Possibly linked issues

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

brettheap commented Oct 23, 2026

Heads and merge

1. The table

2. The sweep: every code point, #1120's head against this head

3. The refusal's new wording, the readers, and the pinned copy

4. End to end: plan with a stub gh and local sources

5. Mutants

6. Merge-trees against the open PRs

7. Gate

Findings

Out of scope, as the brief says

Uh oh!

sonarqubecloud Bot commented Oct 23, 2026

Quality Gate passed

Uh oh!

brettheap commented Oct 23, 2026

Heads and merge

1. Scope of the fix round

2. The four findings

3. Mutants

4. Gate

5. CI on 8162b874

New finding

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

brettheap commented Oct 22, 2026 •

edited

Loading

4. End to end: `plan` with a stub `gh` and local sources

5. CI on `8162b874`