IO.args turns non-UTF-8 arguments into U+FFFD, so a program can't open a file it was given #1798

What you did

import Base

def codes(s: String) -> String:
  match s:
    case SNil{}: ""
    case SCon{head, tail}:
      String.concat([" ", U32.show(Char.to_u32(head)), codes(tail)])

def show_open(name: String, r: Result<&1, &1, U32 & String, File>) -> IO(Unit):
  match r:
    case Done{file}:
      do IO<Unit>:
        IO.print(("open: ok" : String))
        File.close(file)
    case Fail{(code, msg)}:
      IO.print(("open: " ++ msg : String))

def report(args: List<String>) -> IO(Unit):
  match args:
    case Nil{}: IO.print("no argument")
    case prog <> rest:
      match rest:
        case Nil{}: IO.print("no argument")
        case +name <> more:
          do IO<Unit>:
            IO.print(("code points:" ++ codes(name) : String))
            r : Result<&1, &1, U32 & String, File> <- File.open(name, "r")
            show_open(name, r)

def main() -> IO(Unit):
  do IO<Unit>:
    args : List<String> <- IO.args()
    report(args)

On Linux (Debian bookworm, aarch64), with the C that Bend 2.0.37 emits:

bend args.bend -o args.c
clang -O2 args.c -o args -lm -lpthread
touch "$(printf 'caf\xe9.txt')"     # a Latin-1 name: bytes 63 61 66 E9 …
./args "$(printf 'caf\xe9.txt')"
./args "$(printf 'caf\xe8.txt')"

What happened

code points: 99 97 102 65533 46 116 120 116
open: No such file or directory
code points: 99 97 102 65533 46 116 120 116
open: No such file or directory

The file exists, but the byte E9 reaches the program as U+FFFD, so it
can't open the file it was given, and two different names arrive as the
same String. A UTF-8 name works (code points: 99 97 102 233 …,
open: ok). A native build on macOS and the JavaScript build under Bun
print the same code points.

Why it matters

On Linux and other POSIX systems, arguments and file names are byte strings:
an argument may hold any byte but NUL, and a file name any byte but NUL and
/. Programs that take paths from a shell glob, find or xargs (copy,
archive, rename and search tools) can be handed names that aren't UTF-8,
from old Latin-1 media or archives made elsewhere, and today they can
neither open nor tell apart those files.

Possible shapes

String is text (#1248) and files and sockets decode with the WHATWG
decoder (#809), so this doesn't ask to change either. Two options:

1. Byte twins, as TCP got: IO.args_bytes() answering List<List<U32>>,
   plus path effects that take bytes (File.open_bytes and so on). Exact,
   but it doubles every effect that takes a path.
2. Lossless decoding only where the OS hands over names (argv, environment
   variables, paths), as Python's surrogateescape does (PEP 383): a byte
   that isn't part of valid UTF-8 decodes to U+DC00 plus the byte
   (U+DC80–U+DCFF), and encoding a String for the OS turns those code
   points back into the byte. Valid UTF-8 decodes exactly as today, and
   every effect that takes a path keeps its signature.

On the JavaScript lane, Node and Bun decode process.argv before a program
sees it, so that lane may be unable to recover the bytes either way.