[brettheap]

Lane: openreposhape-1 (openRepoShape-1)
Claim: #527 (comment)

What

execute gives git rm each path the split moves to a leg or drops as the bytes the source's tree holds, under any locale. This is line X1 of #457, filed as #527, and it carries out the 2026-10-12 ruling: "pass git the exact bytes the tree holds, under any locale".

_mount_the_legs used to pass each path to subprocess as a str. On POSIX, subprocess encodes a str argument with the filesystem encoding, and that is the locale's codec when Python is not in UTF-8 mode. Under LC_ALL=C PYTHONUTF8=0 that codec is ASCII. So every path holding a character outside ASCII raised UnicodeEncodeError (a byte that is not UTF-8 did not: Source.tree keeps it as a surrogate, and surrogateescape already gave git that byte back): plan and check passed, both legs were made and pushed, and execute exited 2.

Why (repro on main)

Measured on 5e59e3b3 in a clean worktree. Its adopt-project.py, scripts/ and tests/conftest.py are byte-identical to bfed143's (the v1.0.0 release commit). For example, adopt-project.py has sha256 12768bc4… at both.

Every run used a local source and --local-remote-dir, with a stub gh first on PATH that logged no call. plan, check and execute all ran under LC_ALL=C PYTHONUTF8=0, where Python reports the filesystem encoding ascii. The source also held README.md, docs/guide.md and src/app.py.

source path and its answer	plan	check	execute	leg repositories made
café.md (NFC) to spec	0	0, plan ok	2, REFUSED mounting the legs. UnicodeEncodeError: 'ascii' codec can't encode character '\xe9'	both
café.md (NFD: e + U+0301) to spec	0	0	2, the same for '\u0301'	both
both spellings to spec	0	0	2	both
文書.md to code	0	0	2	both
café.md dropped	0	0	2	both
a directory Übersicht/ to spec, the plan written in UTF-8 mode	0	0	2, for '\xdc'	both
control: café.md kept in the root	0	0	0	both
control: ASCII names only	0	0	0	both

On this branch (39da3adf), every row above gives execute exit 0 and adoption verified under the same locale. The same eight sources in UTF-8 mode also give exit 0.

How

• _path_argument(path), beside _git_path, which it undoes. It returns what subprocess must be handed so that git receives the bytes the tree holds.
  • On POSIX it returns bytes: the path encoded as _git_path decoded it, UTF-8 with surrogateescape. So the NFC and the NFD spelling of a name stay two paths. A byte that is not UTF-8, which Source.tree keeps as a surrogate, is that byte again.
  • On Windows it returns the str, as before (see Windows below).
• _git_rm(assembly, path) runs the same command as before, git --literal-pathspecs rm -r -q -- <path>, with the path from _path_argument.
  • run takes only text, so _git_rm asks check_program of the command as the person reads it.
  • A git that refuses raises the same CommandFailed that run raises, naming that command. What execute reports after both legs exist is unchanged.
• _mount_the_legs calls _git_rm for each path, in the same order, .gitmodules first.
• Only paths read from git need this. A path read from the disk (Path, os.readlink, Path.resolve) is text that os.fsdecode made, and the os.fsencode that subprocess applies to it gives the same bytes back in any locale. The docstring says so.

Every git call execute makes after the first leg exists, and which of them hands git a path of the source's tree:

where	what reaches git	the same defect?
_extract_leg	the path list git filter-repo --paths-from-file reads, written as bytes (write_paths_listing, #199)	no
_mount_the_legs, git rm	each moved or dropped path, as a str	yes: fixed here
_refuse_an_unmountable_path, _index_entries, git submodule add, git config -f .gitmodules submodule.<path>.url	the two mount paths, which are checked_values held to SAFE_ARG_RE (ASCII)	no
_stage_the_split, git add -f	the shape's own paths (ASCII), or for a file written through a symlink the plan keeps in the root, the path Path.resolve() gives, which round-trips through os.fsencode	no. Measured on main: .github -> Ünterlagen kept in the root adopts and verifies under the C locale.
env_commit	the split message, on stdin as UTF-8 (#196)	no
_verify	trees by ls-tree -z and blobs by object id; no path on the command line	no

Against GitHub, gh pr create --body also carries the split message, which names every moved path. That is gh, not git, and today it cannot be reached under this locale (see Not in this PR).

Tests

New: tests/test_adopt_git_rm_bytes.py, 4 tests. All four fail on 5e59e3b3, each with UnicodeEncodeError at the first git rm of a name outside ASCII. The paths are:

• moved: résumé.md (NFC), café.md (NFD), a directory Übersicht/ and 文書.md;
• dropped: naïve.md and caf<0xE9>.txt, a byte that is not UTF-8;
• kept in the root: each moved name's twin in the other spelling, and a file Übersicht.md.

The tests:

• test_the_mount_hands_git_rm_each_path_as_the_bytes_the_tree_holds runs _mount_the_legs in a child interpreter under LC_ALL=C PYTHONUTF8=0. Its arguments are ASCII: hex for the directories, JSON for the paths. It then compares the assembly's index, as bytes, with exactly the paths kept plus .gitmodules, spec and code. It needs no git filter-repo, so it runs in both modes.
• test_a_source_with_names_outside_ascii_adopts_under_a_c_locale, with 2 cases, runs plan, check and execute from the command line.
  • every-command-under-the-c-locale is the issue's run, with every name above except the directory and Übersicht.md.
  • a-directory-planned-in-utf-8-mode covers Übersicht/. plan cannot plan that directory under the C locale (see Not in this PR), so the plan is written in UTF-8 mode, and check and execute run under the C locale.
  • Each asserts adoption verified, each leg's file names as bytes, and that each kept twin is in the split and each moved or dropped name is not.
• test_a_path_git_refuses_is_refused_as_the_command_reads (added at 3ba005fc, from r527's review) runs _mount_the_legs in the same child for missing-café.md, a path the tree does not hold.
  • It asserts that git's refusal raises CommandFailed naming the command as the person reads it, git --literal-pathspecs rm -r -q -- missing-café.md. The path is text there, not the bytes git was handed.
  • It also asserts the first three lines of the REFUSED block (the message, the command and the directory it ran in) execute prints after both legs exist, and git's own fatal: pathspec 'missing-café.md' did not match any files line.
  • It needs no git filter-repo, so it runs in both modes.

Each test skips on Windows, then asks a child interpreter for its filesystem encoding under LC_ALL=C PYTHONUTF8=0 and skips where that is UTF-8 (macOS): neither has such a locale to reproduce the failure in. The source's files are written by conftest's make_source_repo, the plan is edited as bytes, and only names are compared, as the bytes git records.

Runs. Every run used /usr/bin/python3 -m pytest, a stub gh first on PATH, a bogus GH_TOKEN and an empty GH_CONFIG_DIR, and the stub logged no call. Each log names its head with dirty 0 and ends EXIT=0.

• Environment. The runs at 3ba005fc and on its merge also set GIT_CONFIG_GLOBAL=/dev/null and GIT_ALLOW_PROTOCOL=file. The earlier runs at 39da3adf and 7184443d used this machine's own global git config.
• The full set is every file that names adopt-project, adopt_project or conftest's ADOPT, every tests/test_adopt_*.py, tests/test_shape_doctor.py and tests/test_scaffold_leg_path_collisions.py. It ran in five chunks per mode.
• The neighbouring set is r527's 23 files: the new file, test_adopt_entry_paths, late_failure, leg_path_collisions, mount_ignored, submodules, ignored_shape_files, clone_template, question_and_submodule_spelling, plan_escapes, template_values, e2e, execute_refusals, filter_repo_failure, root_symlinks, spec_only and relative_submodule_url, plus test_git_output_decoding, test_stdout_encoding, test_windows_paths, test_repo_hygiene, test_shape_doctor and test_scaffold_leg_path_collisions. It ran in four chunks per mode.
• Why the full set was not re-run at 3ba005fc. The full 71-file sets were not run again there, because the code is unchanged: the fix round changed only the new test file, and adopt-project.py is the same as at 39da3adf.

head	files	mode	passed	skipped	failed
e1f43d71 (this branch now: r527's round-2 test edit, no code change)	1, the new file	git filter-repo on PATH	4	0	0
e1f43d71	1	hidden (PATH=/usr/local/bin:/usr/bin:/bin, stub first)	2	2	0
e1f43d71	1, tests/test_repo_hygiene.py	git filter-repo on PATH	53	0	0
3ba005fc	23, the neighbouring set	git filter-repo on PATH	1,835	4	0
3ba005fc	23	hidden (PATH=/usr/local/bin:/usr/bin:/bin, stub first)	1,670	169	0
9df9da62, 3ba005fc merged into main a2243f1f (tree 666a3c0c, never pushed)	3: the new file, test_adopt_late_failure.py, test_adopt_entry_paths.py	git filter-repo on PATH	169	0	0
9df9da62	3	hidden	158	11	0
39da3adf (the fix commit)	70, the full set	git filter-repo on PATH	5,693	13	0
39da3adf	70	hidden	5,378	328	0
7184443d, 39da3adf merged into main 2247dbff (tree 769f3675, never pushed)	71	git filter-repo on PATH	5,704	13	0
7184443d	71	hidden	5,389	328	0

f2fbab6d changes only one docstring, which now cites #527's review, and the file passes there: 4 passed, 0 skipped, 0 failed with git filter-repo on PATH.

git merge-tree --write-tree 39da3adf 2247dbff is clean. Main has since moved to 9f809390 (#553, #561, #562, #579 and #567, which changes adopt-project.py away from this PR's hunks); git merge-tree --write-tree 39da3adf 9f809390 is clean too (tree d6c24338).

At this branch's head e1f43d71 (2026-10-15 17:27Z), main is 3f768a66. Since a2243f1f, #581 (fb786c11) changed adopt-project.py and tests/conftest.py, 26c1600 changed scripts/shape_materialize.py, and #578 (3f768a66) changed tests/conftest.py. git merge-tree --write-tree e1f43d71 3f768a66 is clean (tree 1ddd1560), and this PR's CI runs on that merge.

Earlier: at this branch's head 3ba005fc (fetched 2026-10-15 13:44Z), main is a2243f1f: #558, #556, #580, #560 and #557 came after 9f809390, and none of them changes adopt-project.py, tests/conftest.py or scripts/shape_materialize.py. git merge-tree --write-tree 3ba005fc a2243f1f is clean (tree 666a3c0c). In hidden mode the two command-line cases skip, and both child tests run and pass.

Revert table. Each row reverts the change, or changes one decision of _path_argument or _git_rm, alone. Each was run in a scratch checkout of 3ba005fc, over the 4 new tests. No existing test was changed.

row	what it takes away	fails
none	nothing	0 of 4
R1: _mount_the_legs runs run(["git", ..., "--", path]) as on main	the fix	4 of 4
M1: _path_argument returns the str on POSIX too (Windows' rule everywhere)	the bytes	4 of 4
M2: it encodes with strict, not surrogateescape	a byte that is not UTF-8	2: the bytes test and every-command-under-the-c-locale (surrogates not allowed)
M3: it returns os.fsencode(path), the locale's codec	the tree's encoding	4 of 4
M4: it normalizes to NFC first	NFC and NFD as two paths	2: the bytes test (the NFD café.md stays and its NFC twin is removed) and every-command-under-the-c-locale
M5: CommandFailed names the bytes command: TypeError, a traceback after both legs are pushed	the readable command	1: only the new test (TypeError: sequence item 6: expected str instance, bytes found)
M6: a git that refuses is ignored	the refusal	1: only the new test (nothing raised)
all of adopt-project.py as on 5e59e3b3	everything	4 of 4 (the new test with UnicodeEncodeError)

The bytes test is test_the_mount_hands_git_rm_each_path_as_the_bytes_the_tree_holds. The new test, test_a_path_git_refuses_is_refused_as_the_command_reads, fails on R1, M1, M3, M5, M6 and the last row, and passes on M2 and M4, which change nothing about its path: missing-café.md is NFC and holds no byte that is not UTF-8.

Windows

• The str goes to subprocess on Windows, as it always did. There subprocess hands CreateProcessW the command line as UTF-16, which no locale codec touches, and git for Windows turns it into the UTF-8 its index holds.
• A bytes argument would not work there for every path. Python's Windows subprocess turns each bytes argument back into text with os.fsdecode (list2cmdline, for arg in map(os.fsdecode, seq), CPython 3.12's Lib/subprocess.py). That raises on a byte that is not UTF-8. So nothing changes on Windows: _path_argument returns its argument unchanged there.
• The new tests skip on Windows. The locale trick does not apply there. The existing adoption tests that tests-windows runs go through _git_rm on the str branch, which hands subprocess the same argument as before. I did not run them on a Windows host.
• Newline emulation. This is a pytest plugin kept outside the repository. Under it, Path.write_text and text-mode stdin write \r\n. Under it the 3 new tests ran with tests/test_adopt_entry_paths.py and tests/test_adopt_submodules.py, at 39da3adf and again on 7184443d: 249 passed, 0 skipped, both times. The plugin translated 1,104 write_text calls each time. At 3ba005fc the 4 new tests ran under it alone: 4 passed, with 32 write_text calls translated.
• Nothing in the new file compares "...\n" in bytes a helper wrote. Every path it creates is legal on NTFS. The byte 0xE9 name is created only where the test runs, which is never on Windows.

Not in this PR

Each is measured on main under LC_ALL=C PYTHONUTF8=0. Each is the same kind of defect, text where git or gh needs bytes, but none is the git rm of #527, and each fails before any leg exists or cannot be reached today. They are reported for #457's checklist.

• plan: cross_leg_references hands git grep the spec directory (-e) and the code entries (pathspecs) as text. A spec directory entry outside ASCII, such as Übersicht/, makes plan exit 1 with a UnicodeEncodeError traceback and write no plan. So does a code entry outside ASCII, such as pkg_Übersicht/, beside a spec directory such as docs/.
• check, and execute before any leg: _resolved hands git config a submodule's relative url, read from .gitmodules, as text. A submodule kept in a leg with url = ../Ünterlagen.git makes check exit 1 with a traceback, and execute too, before any leg exists.
• Against GitHub: gh repo create --description holds an em dash (Northwind — spec leg, …). So under this locale every GitHub-mode execute stops at the first gh repo create with a UnicodeEncodeError traceback, exit 1, before any leg exists. This was measured with a fake gh, which was never reached.
  • Behind it, gh pr create --body carries the split message, which names the moved paths. Read from the code, not run (the em dash stops the run first): it would raise UnicodeEncodeError after everything is pushed. _open_the_pull_request catches that as one of LATE_FAILURES, so the split stays pushed and the command is printed for the person to run.

Known limits

• macOS and Windows are covered only by the existing adoption tests, which do not run under a locale that is not UTF-8. Python's documented filesystem encoding on macOS is always UTF-8, so _path_argument's bytes are what os.fsencode gave there before. Windows takes the unchanged str branch.
• A Latin-1 locale was not measured. This machine has only C, C.utf8 and POSIX. Under such a locale os.fsencode would have given é as the one byte 0xE9, a path the tree does not hold. _path_argument now gives UTF-8 there too. This is reasoned from the codecs, not run.
• NFD names on macOS were not measured. Git there may precompose a command line's NFD names to NFC (core.precomposeunicode). That is git's own behaviour, the same before and after this PR.

Closes #527

🤖 Generated with Claude Code

Summary by Sourcery

Make project adoption reliably remove moved and dropped source paths across POSIX locales without Unicode encoding failures.

Bug Fixes:

• Ensure project adoption succeeds for non-ASCII and non-UTF-8 source paths under non-UTF-8 POSIX locales by passing Git the exact bytes recorded in the source tree.
• Preserve readable command errors and refusal reporting when Git rejects a removal path.

Enhancements:

• Refactor source-tree path removal during leg mounting to use locale-independent path handling while leaving Windows behavior unchanged.

Tests:

• Add coverage for Unicode spellings, non-UTF-8 filenames, directories, dropped paths, and Git removal failures under a C locale.

[sourcery-ai]

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 4 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[sourcery-ai]

Reviewer's Guide

Fix execute failures under non-UTF-8 POSIX locales by passing Git the exact bytes recorded in the source tree for each moved or dropped path, while retaining readable errors and Windows behavior. New tests cover Unicode spellings, invalid UTF-8 bytes, directories, drops, end-to-end adoption, and refusal reporting.

Sequence diagram for byte-preserving git rm during adoption

sequenceDiagram
    participant Execute
    participant Mount as _mount_the_legs
    participant GitRm as _git_rm
    participant Subprocess
    participant Git

    Execute->>Mount: _mount_the_legs(assembly, ...)
    loop moved or dropped paths
        Mount->>GitRm: _git_rm(assembly, path)
        GitRm->>GitRm: check_program(command)
        GitRm->>Subprocess: subprocess.run(GIT_RM + _path_argument(path))
        alt POSIX
            GitRm->>Git: git rm receives UTF-8 bytes with surrogateescape
        else Windows
            GitRm->>Git: git rm receives unchanged str
        end
        Git-->>Subprocess: exit status and output
        Subprocess-->>GitRm: result
    end
    GitRm-->>Mount: success or CommandFailed
    Mount-->>Execute: adoption continues or reports refusal

Loading

File-Level Changes

Change	Details	Files
Preserve the source tree’s exact path bytes when removing moved or dropped entries during leg mounting.	Add _path_argument to encode POSIX paths as UTF-8 with surrogateescape and retain text handling on Windows. Introduce _git_rm to separate the human-readable command/error representation from the byte-valued subprocess argument while preserving refusal handling. Route _mount_the_legs removals through _git_rm without changing removal order or command semantics.	adopt-project.py
Add locale-sensitive regression coverage for Unicode, decomposed Unicode, and non-UTF-8 filenames.	Test direct mounting under LC_ALL=C PYTHONUTF8=0, comparing index entries byte-for-byte and ensuring kept twins remain intact. Test complete plan/check/execute adoption flows, including non-ASCII directories and dropped paths. Test readable CommandFailed reporting when git rm refuses a missing non-ASCII path, while skipping unsupported platforms/locales.	tests/test_adopt_git_rm_bytes.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#527	Pass each moved or dropped source-tree path to git rm as the exact bytes held by the tree, so non-ASCII paths do not trigger UnicodeEncodeError under a non-UTF-8 locale.	✅
#527	Preserve the existing execute behavior for paths Git refuses and for platforms such as Windows while applying the byte-path fix.	✅

Possibly linked issues

• adopt-project.py execute: under a non-UTF-8 locale, a moved or dropped path that is not ASCII kills git rm after both legs are pushed #527: The PR implements issue adopt-project.py execute: under a non-UTF-8 locale, a moved or dropped path that is not ASCII kills git rm after both legs are pushed #527 by passing git rm the exact tree-path bytes, avoiding the reported post-push UnicodeEncodeError.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud