Skip to content

scripts/shape_materialize.py run() decodes every git and gh message with the locale codec, so a push rejection or any other failure that names a non-ASCII path raises UnicodeDecodeError in place of REFUSED and the ruleset hint under a non-UTF-8 locale, after both leg remotes exist #217

Closed

Description

@brettheap

Lane: openreposhape-3 (openRepoShape-3)

Found by this lane's work on #196 (branch fix/adopt-check-unsafe-path), which gives env_commit in the same file encoding="utf-8", errors="replace"; pre-existing on main 20e13f0, and e4decfb behaves the same. This is the DECODE side: how the tool reads what a child process wrote. #214 is the other direction, the tool's own stdout, where plan prints a name its stdout cannot encode. Fixing one leaves the other.

What is wrong. run (scripts/shape_materialize.py:478) carries most of the git and gh calls adopt-project.py execute makes: the clone, the filter, each push, the mount, the split commit's git add. It calls subprocess.run(..., capture_output=True, text=True) with no encoding, so the bytes are decoded with the locale's codec and strict errors: the ANSI code page on a Windows without PYTHONUTF8=1, ASCII under a POSIX C locale with UTF-8 mode off. Git writes a path as UTF-8. A message that names one with a character the codec cannot decode raises UnicodeDecodeError inside subprocess.run, so CommandFailed is never built, and main() has nothing to catch but Refusal and CommandFailed. The person gets a traceback and exit 1 in place of REFUSED <step>, and after a refused push no RULESET_HINT, with both leg remotes already created.

Measured on main 20e13f0. The source holds specs/001-feature/Übersicht.md (nested, because a top-level non-ASCII plan path is refused as unsafe-value, #196); plan and check are clean (exit 0, plan ok). The new leg remotes carry a pre-receive hook, installed through GIT_TEMPLATE_DIR, that rejects the push naming that file, as a hosted push rule does. execute --yes --local-remote-dir twice:

this shell (Python in UTF-8 mode):   exit 2
    REFUSED pushing the spec leg. THE EXACT COMMAND WAS: ... exit 1
    remote: error: push rule: the file 'specs/001-feature/Übersicht.md' is blocked
    ruleset hint printed: True    remotes: Northwind-code.git Northwind-spec.git

LC_ALL=C PYTHONUTF8=0:               exit 1
    Traceback ... _extract_leg: run(["git", "push", "-q", url, ...])
      shape_materialize.py run: subprocess.run(...) ... subprocess.py communicate
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 54: ordinal not in range(128)
    ruleset hint printed: False   remotes: Northwind-code.git Northwind-spec.git

I cannot run Windows. Its code page (cp1252 on a Western-European install) decodes most of git's UTF-8 as other characters, which is wrong text and no exception ('Ü'.encode().decode('cp1252') is 'Ü'), and raises on the five bytes it leaves undefined ('Ý'.encode().decode('cp1252') raises can't decode byte 0x9d).

git_out (scripts/repo_shape.py:657) decodes UTF-8 explicitly and is not affected. The same shape is in 30 subprocess.run calls with text=True and no encoding across 9 files (adopt-project.py twice, scaffold-project.py, update-shape.py, scripts/bump-leg.py, scripts/family.py, scripts/repo_shape.py, scripts/shape_materialize.py twice (run, and env_commit, which #196's branch fixes), and the family template's bootstrap.py and siblings.py); only setup-project.py:285 and shape-doctor.py:831 pass one. I reproduced run's, and none of the others.

What would fix it. encoding="utf-8", errors="replace" on every subprocess.run that decodes git or gh output, as setup-project.py, shape-doctor.py and #196's env_commit already do. run is one line and the one execute depends on; the other 29 are the same line each, or one shared helper they all call. Tried on a scratch copy of main with run changed, not committed: the LC_ALL=C PYTHONUTF8=0 run exits 2 with REFUSED pushing the spec leg, the hook's line (the character escaped by stderr's backslashreplace, \xdcbersicht.md) and the ruleset hint. A test runs execute over that hook under LC_ALL=C PYTHONUTF8=0; the Windows job sets PYTHONUTF8: '1' (.github/workflows/tests.yml:145), so CI does not run the cp1252 case.

Not claimed.

Activity

  1. brettheap commented on Oct 11, 2026

    @brettheap
    ContributorAuthor

    CLAIMED — lane openRepoShape-3, session 566fec94-6fde-4579-947f-ec4fda5ce14f@Eagle, 2026-10-11T16:26:11Z, for #217

    Logged in lanes/log/openRepoShape-3.md at 008852d9817be00120be3276a805aa5ab2848cb4.

    The three reads (lane-collision-protocol Rule 1):

    1. existing `CLAIMED —` comments on opensoft/openRepoShape#217:
       none
    2. `gh pr list --repo opensoft/openRepoShape --state all --search "#217"`, kept where the row or its body names `#217`:
       none
    3. `git ls-remote --heads git@github.com:opensoft/openRepoShape` branches naming `217` as a whole token:
       none
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions