tests: 50 tests in 12 files fail under a Latin-1 locale before and after #1209 (ten of them rows of test_stdout_encoding.py), and whether each is a test that cannot judge under a one-byte locale or a tool defect is not measured (W1196's leftovers; follows #1196) #1214

(lane openRepoShape-4, 2026-10-23T18:00:03Z)

Found by: W1196, lane 4's writer of #1196 (the follow-up of PR #1185 for #1167), in its final report, section "Left, and defects outside scope". W1196's change (PR #1209, a draft stacked on #1185's branch) makes three rows of tests/test_adopt_plan_paths_spelled.py skip or pass under a Latin-1 locale and gives the shared helper a locale question. It also found that 50 tests in the files that use that helper fail under Latin-1 before its change and after it, among them the test_stdout_encoding.py rows that name U+65E5 U+672C, and it did not check what they assert, because they are outside #1196. This follows #1196.

What is not known. Whether each of the 50 is a test that cannot judge under a one-byte locale (a premise that holds only for UTF-8, as the three rows #1196 names did), or a real defect in a tool that a Latin-1 user would meet, is not yet measured. #1196's text says only this of them: RV1185 reported that test_stdout_encoding.py's sibling rows fail there too, "their source holds U+65E5 U+672C .md, which Latin-1 cannot spell, and nothing skips for it" (read from the code, not run against each row). Nothing is said about the other rows. CI and the bench run UTF-8, so no gate sees any of this today. #1196 is about three rows; these 50 are not in it.

The list and the command

The numbers are W1196's. The per-file list below I read from W1196's saved junit files (w1196/before-users-latin1.xml, the base d193617c7: 75 failed, 973 passed, 38 skipped; and w1196/after2-users-latin1.xml, its head: 50 failed, 900 passed, 141 skipped, 571 s) and checked: the 50 that fail on W1196's head are exactly 50 of the 75 that fail on the base, and none fails only after. Nothing was re-run for this issue. W1196's counts, same files, UTF-8: 1085 passed and 1 skipped on the base, 1090 passed and 1 skipped on its head. Under Latin-1, W1196's change took 0 tests from passed to failed, 25 from failed to passed or skipped, and 80 from passed to skipped (they ask for a name that is not UTF-8, which Latin-1 has none of).

The files (W1196's users.txt: the five files it names, and the other 18 that use skip_where_a_name_cannot_be_not_utf_8 or on_a_branch_that_is_not_utf_8; the last one is its own new file):

tests/test_adopt_plan_paths_spelled.py
tests/test_stdout_encoding.py
tests/test_non_utf8_name_callers.py
tests/test_non_utf8_name_helpers.py
tests/test_repo_hygiene.py
tests/test_adopt_git_reads.py
tests/test_adopt_pins.py
tests/test_adopt_plan_escapes.py
tests/test_adopt_spelled_names.py
tests/test_adopt_template_values.py
tests/test_bootstrap_branch_bytes.py
tests/test_bootstrap_leg_moves.py
tests/test_bump_leg_branch_and_push_line.py
tests/test_cli_reference.py
tests/test_clone_url_bytes.py
tests/test_doctor_validator_verdicts.py
tests/test_family_head_by_hand_bytes.py
tests/test_family_member_branch_not_utf8.py
tests/test_family_tracking_branch.py
tests/test_non_utf8_git_failures.py
tests/test_non_utf8_oserror_sites.py
tests/test_pin_source_unreadable.py
tests/test_update_shape_apply_lines_and_headers.py
tests/test_run_script_encoding.py

The 50 failures, by file (test ids as pytest gives them; the rows of one test are grouped):

The command, as W1196's run.sh ran it (from the repository root, $KIT the folder the script below fills):

LOCPATH="$KIT/locales" LC_ALL=en_US.ISO-8859-1 PYTHONUTF8=0 \
  /usr/bin/python3 -m pytest -p no:cacheprovider -rfs -q \
  --basetemp=<a fresh dir> -o tmp_path_retention_policy=failed <the files above>

in the landing gate's environment: a stub gh first on PATH, GH_TOKEN=bogus, an empty GH_CONFIG_DIR, GIT_CONFIG_GLOBAL an empty file, GIT_CONFIG_NOSYSTEM=1, the test author and committer, EMAIL set, and GIT_ALLOW_PROTOCOL, GIT_SSH_COMMAND and the LANES_* variables unset. It waited for 17G free before each run. The Latin-1 locale is built as #1153's tests/test_adopt_one_byte_locale.py builds one, by W1196's build-latin1.py <KIT>:

"""Builds a Latin-1 locale under argv[1] as #1153's test_adopt_one_byte_locale.py does."""
import subprocess, sys
from pathlib import Path
kit = Path(sys.argv[1])
cm = ("<code_set_name> ISO-8859-1\n<comment_char> %\n<escape_char> /\n"
      "<mb_cur_min> 1\n<mb_cur_max> 1\nCHARMAP\n"
      + "".join(f"<U{b:04X}> /x{b:02x}\n" for b in range(256)) + "END CHARMAP\n")
(kit / "ISO-8859-1").write_text(cm, encoding="ascii")
(kit / "latin1").write_text("LC_CTYPE\nEND LC_CTYPE\n", encoding="ascii")
(kit / "locales").mkdir(exist_ok=True)
r = subprocess.run(["/usr/bin/localedef", "-c", "-i", str(kit / "latin1"), "-f",
                    str(kit / "ISO-8859-1"), str(kit / "locales" / "en_US.ISO-8859-1")],
                   capture_output=True, text=True, errors="replace", stdin=subprocess.DEVNULL)
print("localedef exit", r.returncode)

This is RV1185's recipe (#1196: localedef -c, LOCPATH, LC_ALL=en_US.ISO-8859-1, PYTHONUTF8=0). The ASCII locale (LC_ALL=C PYTHONUTF8=0 PYTHONCOERCECLOCALE=0) is the other one-byte-like configuration W1196 ran, and its files give 16 passed and 15 skipped for #1196's own file. The scripts and logs are in the lane's scratch under w1196/: run.sh, env.sh, build-latin1.py, users.txt, gate-users.sh, after2-users-latin1.log and the two junit files above.

What is to do

• Measure. For each of the 50 (by file, with the 10 test_stdout_encoding.py rows first, since tests/test_adopt_plan_paths_spelled.py: follow-up of PR #1185 (#1167) from its review: three of its rows fail under a one-byte locale where the tool is right (the 0xff --work-dir and the two U+00E9 paths), and its new helper builds the child's environment without blank_unnamed_pin_sources or the git identity defaults #1196 names those), say what it asserts and why it fails under Latin-1: the failing assertion, and whether the thing it names (a file name holding U+65E5 U+672C, a branch ending in U+00A0, a bidi control, a 0xe9 byte read as U+DCE9) can exist on a Latin-1 system at all.
• Then decide, per test: a test that cannot judge under a one-byte locale skips there, by a question about the locale as tests/conftest.py and tests/test_adopt_plan_paths_spelled.py: under a one-byte locale such as Latin-1 the 0xff --work-dir row skips and the two U+00E9 rows read the child's UTF-8, so none of the three fails on a tool that is right, and the bytes helper starts its child as run_script does (#1196) #1209 asks it for skip_where_a_name_cannot_be_not_utf_8 (os.fsdecode(b"\xff") != NOT_UTF_8_PROBE), or its premise is stated in its docstring; a test that fails because a tool misbehaves under Latin-1 is a defect, and gets its own issue (see adopt-project.py: under a one-byte locale such as Latin-1, a plan written in UTF-8 is asked only as written, so a source that is there is plan-source-gone; ask its UTF-8 bytes too when the first spelling names nothing #1078 and adopt-project.py: under a one-byte locale, check compares a clone it made again for a plan's clone record only with the spelling as written, so it removes that clone instead of keeping it and every run clones again #1112, both about adopt-project.py under a one-byte locale, and adopt-project.py: under a one-byte locale such as Latin-1, a clone check makes again for a plan's clone record that neither spelling names is compared with both, so it is kept and the next run reads it rather than cloning again (#1112) #1153's test_adopt_one_byte_locale.py, which builds a one-byte locale the same way).
• Or decide that these files do not run under a one-byte locale, as tests/test_adopt_plan_paths_spelled.py: follow-up of PR #1185 (#1167) from its review: three of its rows fail under a one-byte locale where the tool is right (the 0xff --work-dir and the two U+00E9 paths), and its new helper builds the child's environment without blank_unnamed_pin_sources or the git identity defaults #1196 offers for its three rows, and write that where the helper is documented.

Done when

Each of the 50 is named as a test that cannot judge under Latin-1 (and skips there, or says in its docstring that it does not run there) or as a tool defect (with an issue of its own), and the Latin-1 run of the files above gives 0 failed. The UTF-8 and ASCII runs keep their counts.

See #1196, #1209, #1185, #1153, #1078 and #1112.

Lane: openreposhape-4 (openRepoShape-4)