[brettheap]

Lane: openreposhape-5 (openRepoShape-5)
Claim: #254 (comment)

What was wrong

A real bump-leg.py (not --dry-run) and update-shape.py apply run the
PROJECT'S own scripts/validate-pins.py and scripts/validate-manifest.py as
child processes after they rewrite the pin. A child takes its stdout
configuration from its environment (PYTHONIOENCODING, then the locale), not
from the reconfigure its parent did in #214, and the validators are
digest-pinned shape copies that print the display name with a plain print.
So on a stdout whose code page cannot hold the name (PYTHONIOENCODING=cp1252,
or the cp1252 with surrogateescape that Windows gives any stdout that is not
the console), validate-manifest.py raised UnicodeEncodeError at
manifest ok: <name>, exited 1, and the tool refused and rolled back a bump
the pins had no quarrel with. The refusal blamed the validators.

What changed

• scripts/shape_materialize.py: two helpers build the child's environment,
  and each tool passes the one it needs as env=.
  • captured_child_environment() sets PYTHONIOENCODING=utf-8:backslashreplace.
  • inherited_child_environment(stream=None) sets the stream's OWN canonical
    encoding with its own handler where that handler never raises
    (replace, ignore, backslashreplace, xmlcharrefreplace,
    namereplace), and backslashreplace for every other one.
• scripts/bump-leg.py run_validator CAPTURES the output. It passes
  env=captured_child_environment() and reads the pipe with
  encoding="utf-8", errors="replace", where shape_materialize.run() and 26 other git and gh calls read what the child wrote as UTF-8 whatever the locale, so an adopt-project.py execute whose leg push is rejected with a message naming a non-ASCII path ends REFUSED pushing the spec leg and the ruleset hint under LC_ALL=C PYTHONUTF8=0, not a UnicodeDecodeError traceback (#217) #230 had errors="replace" and
  no encoding=.
• update-shape.py run_validator lets the child INHERIT stdout. It passes
  env=inherited_child_environment().
• tests/test_git_output_decoding.py: the bump-leg.py run_validator row
  leaves EXEMPT, because that call now names both encoding and errors.
• tests/test_validator_child_encoding.py, 51 new tests.

Decisions, and the evidence for each

Captured case: the child writes UTF-8, and the tool reads UTF-8. #230 gave
run_validator errors="replace" and no encoding=, "so that codec is the
one that reads it back". That was right only while the child wrote in the
locale's codec. Once the child is told UTF-8, the locale's codec misreads it:
'日本'.encode('utf-8') read as ascii is six U+FFFD, and read as cp1252 is
'\xe6\u2014\xa5\xe6\u0153\xac'. So encoding="utf-8" is
explicit. Dropping it fails
test_a_real_bump_reads_the_validators_as_utf_8_whatever_the_locale_is
(mutant bump-decode-with-locale). The words then reach the person through
the tool's own stdout, which #214 made safe: manifest ok: \u65e5\u672c on
a cp1252 stdout, manifest ok: 日本 on UTF-8 and cp932.

An operator's PYTHONIOENCODING is replaced for the captured pipe and
respected for the inherited one. The operator's value describes the stream
their own words land in. The captured pipe is the tool's, so respecting
cp1252:strict there would put the failure back one pipe away from the
console. The inherited fd IS the operator's stream, so the child is told what
that stream is. The tests run under cp1252:strict and cp1252:surrogateescape
(the brief's two), plus cp1252:replace (an operator's own lenient handler:
?? before and after), cp932:strict (a Japanese Windows) and utf-8:strict.

Inherited case: the stdout's own encoding, never UTF-8, and built-in handler
names only. UTF-8 bytes on a cp1252 pipe would be a different mistake. A
child cannot use a handler only its parent registered:
PYTHONIOENCODING=cp1252:openreposhape.surrogateescape-then-backslashreplace
still STARTS the child and fails with LookupError at the first character it
cannot encode. test_the_handler_a_child_would_not_know_is_never_passed_on
runs that child and asserts both halves. At the point the child is built,
main() has already turned a strict stdout into backslashreplace and a
surrogateescape one into the registered handler, so the helper maps
strict, surrogateescape, surrogatepass and the registered name to
backslashreplace, and keeps a handler that never raises.

One difference, and it is small: a byte of a file name that was not UTF-8,
read in as U+DC80..U+DCFF, is written by a validator as the text \udcff
where the tool writes the byte back. A validator prints one only for a path it
lists.

Evidence

Fails on 60d08a1: with the new test files copied into a throwaway worktree at
60d08a1, 42 failed, 12 passed (the 12 are controls that pass either way:
the five --dry-run bumps, cp1252:replace and utf-8:strict for both tools,
cp932:strict for apply, and #230's own two C-locale tests). The decisive
lines:

• bump, cp1252:strict and cp1252:surrogateescape: exit 2, REFUSED bump-leg-validators-red: scripts/validate-manifest.py exited 1 after the rewrite, UnicodeEncodeError: 'charmap' codec can't encode characters in position 13-14.
• apply, the same two: exit 2, REFUSED update-validators-red: after the rewrite, scripts/validate-manifest.py refused.
• the guard in test_git_output_decoding.py flags run_validator once its
  EXEMPT row is gone.

Passes on the head (5757e29):

• the new file (51), test_stdout_encoding.py (48), test_bump_leg.py (23),
  test_update_shape.py (77), test_update_shape_add.py (17),
  test_git_output_decoding.py (3), test_repo_hygiene.py (50) and
  test_cli_reference.py (13): 281 passed, 1 skipped (the existing
  Windows-only default-stdout test);
• test_shape_doctor.py: 181 passed;
• test_family.py and test_update_shape_add_ignored.py: 102 passed;
• the head merged with main d89fffa, on the files above plus
  test_update_shape_pr_base.py: 392 passed, 1 skipped.

One extra finding, fixed by the same two changes. Under LC_ALL=C PYTHONUTF8=0 (an ASCII stdout; UTF-8 mode has to be switched off by hand, as
it is on by default in a C locale) validate-pins.py raises on the …
(U+2026) it prints, so on 60d08a1 a real bump of a project named Atlas was
refused as well: exit 2, REFUSED bump-leg-validators-red: scripts/validate-pins.py exited 1, UnicodeEncodeError: 'ascii' codec can't encode character '\u2026'. On the head the same run exits 0. This is why the
C-locale bump test fails on 60d08a1 at the PINS validator.

Emulation caveat. The cp932:strict BUMP also fails on 60d08a1 when the
test runs on Linux, but only as an artefact: the child wrote cp932 and a UTF-8
locale cannot read it, where a Japanese Windows' own locale could. It is not
the defect and is not cited as one. The Windows-specific runs here are
emulations (PYTHONIOENCODING=cp1252:surrogateescape); I did not run anything
on Windows, and these tests first run on Windows in this PR's tests-windows
job. The two C-locale tests are POSIX-only (skipif os.name == "nt") and skip
where the platform reads UTF-8 under LC_ALL=C PYTHONUTF8=0.

Mutation probes. Eleven hunks reverted one at a time in a throwaway
worktree, the whole new file run each time: all eleven killed.

hunk reverted	failed	first killers
captured_child_environment sets nothing	6	both captured_child unit tests, bump cp1252:strict, cp1252:surrogateescape, cp932:strict, the C-locale bump
captured value utf-8 with no handler	2	the two captured_child unit tests
bump-leg.py drops env=	5	the three bump cp1252/cp932 runs, the C-locale bump, the AST guard
bump-leg.py drops encoding="utf-8"	1	the C-locale bump
update-shape.py drops env=	4	apply cp1252:strict, cp1252:surrogateescape, the C-locale apply, the AST guard
inherited handler passed verbatim	14	strict/surrogateescape/surrogatepass unit and child cases, apply cp1252:surrogateescape, the C-locale apply
inherited always backslashreplace	11	ignore/namereplace/replace/xmlcharrefreplace cases, apply cp1252:replace
replace dropped from the never-raises set	5	the replace cases, apply cp1252:replace
inherited encoding forced to utf-8	27	every non-UTF-8 case
inherited encoding not canonicalised	3	ANSI_X3.4-1968, latin-1, the no-stdout case
inherited drops the no-stdout guard	1	test_a_stdout_that_is_not_there_leaves_the_environment_alone

Byte-identical where nothing was wrong. A real bump and an apply, from
60d08a1 and from the head, on the same project, under utf-8:strict,
utf-8:surrogateescape and no PYTHONIOENCODING, named Atlas and named
日本: twelve pairs, stdout, stderr and exit code identical once the run
directory and the shas are masked.

Sibling merge-trees (git merge-tree --write-tree HEAD <branch>, local
tips of lane 5's branches and every open branch that touches bump-leg.py,
update-shape.py or shape_materialize.py): clean against
fix/yaml-quote-boundaries, fix/yaml-block-key-nul,
fix/yaml-flow-mapping-keys, fix/scaffold-referent-chain-links,
fix/adopt-check-unsafe-path, fix/adopt-plan-quote-yaml-keywords,
fix/adopt-record-submodule-placement, fix/adopt-template-values-utf8,
fix/bump-leg-rollback-every-failure, fix/bump-leg-tracked-workflows-only,
fix/leg-path-yaml-scalars, fix/update-shape-add-dangling-symlink,
fix/update-shape-pr-base-branch and fix/update-shape-pr-create-failure.
One conflicts: fix/adopt-root-kept-symlinks (adopt-project.py and
scripts/shape_materialize.py), and it conflicts the same way with origin/main
itself, so it is not this change.

Left out

• The shape copies a project runs itself. scripts/bootstrap.py,
  scripts/validate-manifest.py and the family root's siblings.py are
  digest-pinned and print the display name raw, so make bootstrap and make validate on a project named 日本 fail under a cp1252 stdout at their own
  print, before any child is run (checked: bootstrap.py raises at
  bootstrap: <name> (<root>)). That is a shape change, and it reaches
  projects through update-shape.py.
• shape-doctor.py's run_validator already sets PYTHONIOENCODING=utf-8
  for the same validators and reads them as UTF-8, and it is unchanged. A
  display name holding a lone surrogate (name: "a\ud83db") reaches it
  today. The project's own validate-manifest.py accepts that name, then
  raises printing it, so make validate and the doctor fail on a traceback
  (INVALID (manifest)). After this change, a real bump and apply print
  manifest ok: a\ud83db and pass. That is pre-existing, and the fix belongs
  in the validator or the doctor, not in these two callers: validate-manifest.py accepts a display name holding a lone surrogate (name: "a\ud83db") and then raises UnicodeEncodeError printing it, so make validate and shape-doctor.py fail on a traceback instead of a finding while bump-leg.py and update-shape.py apply pass the same project once #254 lands #280.

Review

An adversarial review of 5757e29 found no blocker. Its one should-fix was a
false sentence in this body's "Left out", corrected above and filed as #280.

• It ran the inherited case on Python 3.10, 3.12, 3.13, 3.14 and 3.15rc2,
  including UTF-8 mode and the C locale. In each 28-run cell the head had 0
  failures; main had 8, or 12 under LC_ALL=C PYTHONUTF8=0.
• It compared 30 real runs between main and the head under UTF-8: stdout and
  stderr were identical.
• The head merged with c3f4987 passed: 264 passed, 1 skipped.

Three nits are kept as they are:

• The inherited child's stderr. It follows the parent's STDOUT encoding,
  because PYTHONIOENCODING sets both streams. With a console stdout and a
  redirected stderr, the child writes UTF-8 to a cp1252 stderr.
• errors="replace" in bump-leg.py's run_validator. It is defensive.
  The child now always writes UTF-8, so a strict mutant survives every test.
• The \udcff text. This is the difference stated above.

Fix round 1 (tests-windows)

The first head, 5757e29, failed tests-windows: 5 failed and 2477 passed, and macOS and Linux passed. All five failures were test_an_apply_on_a_project_named_in_japanese_lands_under_any_stdout. Its assert_the_update_landed compared the applied copy with the upstream clone's WORKING-TREE file. On Windows that file is CRLF, because the shared test helper commit_upstream_changes rewrites it with text-mode write_text. Git stores the blob as LF, which is what apply writes. The test now compares with the committed blob (git show <b>:<path>), as test_update_shape.py already does, and no product code changed.

• Emulation on Linux. The helper was made to write CRLF: the old assertion failed 6 apply cases, and the new one passed them.
• Targeted runs. The seven targeted files gave 274 passed and 1 skipped on the head merged with main b30595c (97c4280).
• A candidate follow-up. commit_upstream_changes leaves the upstream clone's working-tree copy CRLF on Windows for the rest of its module. Writing the appended bytes in binary would remove the trap at its source, but the helper is shared and outside this change.

Closes #254

🤖 Generated with Claude Code

[brettheap]

(lane openRepoShape-5, 2026-10-11T23:42Z) @codex review

[sourcery-ai]

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 13 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[sourcery-ai]

Reviewer's Guide

The PR prevents project validators from failing solely because a child process inherits an incompatible stdout configuration: captured validators are forced to emit and be decoded as UTF-8, while inherited validators receive the actual stdout encoding with a child-known non-raising handler, backed by extensive cross-platform regression tests.

Sequence diagram for safe validator output during bump and apply

sequenceDiagram
    participant Tool as bump-leg.py or update-shape.py
    participant Validator as Project validator
    participant Stdout as Tool stdout

    alt Captured validator output
        Tool->>Validator: subprocess.run(env=captured_child_environment())
        Validator-->>Tool: UTF-8 output
        Tool->>Tool: Decode with encoding="utf-8", errors="replace"
        Tool->>Stdout: Print escaped validator output
    else Inherited validator output
        Tool->>Validator: subprocess.run(env=inherited_child_environment())
        Validator->>Stdout: Write using stdout encoding and safe handler
    end

Loading

File-Level Changes

Change	Details	Files
Standardize the encoding environment for validator child processes according to whether output is captured or inherited.	Add a UTF-8/backslashreplace environment for captured validator output. Add an environment derived from the inherited stream's canonical encoding and a child-known non-raising handler. Preserve the parent environment while selectively overriding PYTHONIOENCODING and leave it unchanged when stdout is unavailable or unusable.	scripts/shape_materialize.py
Make bump-leg validator execution explicitly UTF-8 end-to-end.	Pass the captured-child environment to project validators. Decode captured stdout and stderr as UTF-8 with replacement while retaining defensive error handling.	scripts/bump-leg.py
Make update-shape validator execution safe for the tool's actual stdout.	Pass the inherited-child environment when validators write directly to the parent's stdout. Ensure validators use built-in handlers rather than parent-only custom handlers.	update-shape.py
Add regression coverage for validator encoding behavior across locales, code pages, handlers, and real tool workflows.	Test real bump and apply operations for Japanese project names under cp1252, cp932, UTF-8, and POSIX C-locale configurations. Test helper environment construction, child process output, canonicalized codec names, handler fallback, absent stdout, and preservation of unrelated environment variables. Add an AST guard ensuring validator-launching tools pass an explicit environment. Update the existing decoding guard to recognize explicit encoding and error arguments.	tests/test_validator_child_encoding.py tests/test_git_output_decoding.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#254	Ensure that a real bump-leg.py operation can run the project's validators and succeed for display names such as 日本 under a cp1252 or otherwise incompatible stdout, rather than being refused and rolled back as bump-leg-validators-red.	✅
#254	Ensure that update-shape.py apply can run the project's validators and succeed for display names such as 日本 under an incompatible stdout, rather than being refused and rolled back as update-validators-red.	✅
#254	Configure validator child processes in the two tools with encoding settings appropriate to how their output is handled: UTF-8 with matching decoding for captured bump output, and the parent's canonical stdout encoding with a child-known non-raising error handler for inherited apply output.	✅

Possibly linked issues

• bump-leg.py (a real bump) and update-shape.py apply run the project's own validators as children that cannot print its display name on a cp1252 stdout, so a project named 日本 is refused and rolled back as bump-leg-validators-red or update-validators-red under PYTHONIOENCODING=cp1252:surrogateescape, and the refusal blames the validators #254: The PR directly fixes issue bump-leg.py (a real bump) and update-shape.py apply run the project's own validators as children that cannot print its display name on a cp1252 stdout, so a project named 日本 is refused and rolled back as bump-leg-validators-red or update-validators-red under PYTHONIOENCODING=cp1252:surrogateescape, and the refusal blames the validators #254 by configuring validator child environments and matching captured-output decoding.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.7% Duplication on New Code

See analysis details on SonarQube Cloud