C and JS lanes decode invalid UTF-8 differently: io_str accepts overlong, surrogate and out-of-range sequences (File.read, get_env, TCP.recv, UDP.recv) #809

What you did

bend u.bend          # JS lane
bend u.bend -o u && ./u   # C lane

What happened

Every effect that turns OS bytes into a String uses a different UTF-8 decoder on
each lane, so the same program on the same input produces two different Strings.

The C kit decodes with io_str (bend2/comp.ts:5357), a hand-rolled backward
scanner with no validation. The JS kit decodes with io_text (bend2/comp.ts:6114),
i.e. TextDecoder, which is WHATWG-strict and substitutes U+FFFD.

io_str is shared by File.read, IO.get_env, TCP.recv, UDP.recv_from and
UDP.poll — every untrusted-input seam, including the network ones.

Reading these 23 bytes

41 7c 80 7c c0 80 7c ed a0 80 7c f7 bf bf bf 7c ff 7c e6 97 7c c3 a9

and printing Char.to_u32 of each char:

JS: 65 124 65533 124 65533 65533 124 65533 65533 65533 124 65533 65533 65533 65533 124 65533 124 65533 124 233
C:  65 124 128   124 0           124 55296            124 2097151              124 255   124 230 151   124 233

String.length of the same file is 16 in C and 21 in JS. Neither lane
reports an error; both exit 0.

IO.get_env shows it too, so it is not specific to File.read. With
BADUTF set to 41 ed a0 80 c0 80:

JS: 65 65533 65533 65533 65533 65533
C:  65 55296 0

Three consequences beyond the divergence itself:

1. The C lane breaks its own UTF-8 law on output. tests/io/text_utf8.bend and
tests/io/print_utf8_law.bend pin that "a String reaches the OS as UTF-8, one
sequence per Char, in every lane". But io_utf8 (bend2/comp.ts:5308) does not
validate either, so reading the file above and writing it straight back gives:

in:      41 7c 80 7c c0 80 7c ed a0 80 7c f7 bf bf bf 7c ff 7c e6 97 7c c3 a9
C out:   41 7c c2 80 7c 00 7c ed a0 80 7c f7 bf bf bf 7c c3 bf 7c c3 a6 c2 97 7c c3 a9
JS out:  41 7c ef bf bd 7c ef bf bd ef bf bd 7c ef bf bd ef bf bd ef bf bd 7c ...

ed a0 80 and f7 bf bf bf come back out verbatim, so the C lane's own output is
not valid UTF-8. The overlong c0 80 also becomes a raw 00 byte in the output
stream. The bytes that are decoded as Latin-1 come back re-encoded
(80 → c2 80, ff → c3 bf, e6 97 → c3 a6 c2 97), which is exactly the
mojibake the text_utf8 comment says was fixed in the other direction.

2. The C lane can build a Char the JS lane refuses to construct. char_new
(bend2/comp.ts:567) throws on surrogates and on anything above 0x10FFFF:

if (code > 0x10FFFF || (code >= 0xD800 && code <= 0xDFFF)) {
  throw "bend: " + code + " is not a Unicode scalar value";
}

Those are precisely the values io_str hands back. So a String a C-lane program
can hold is not representable in the JS lane at all.

3. Overlong encodings are accepted. c0 80 → Chr{0} means a NUL char can be
introduced into a String from file or socket input. tests/io/nul_bytes.bend
takes care that "a NUL byte never crosses a char* OS seam"; the io_nul guard
still catches it at the path/mode/host seams, but the char itself is created
silently and the two lanes disagree on whether it exists.

The file

import Base

def codes(s: String) -> String:
  match s:
    case SNil{}:
      ""
    case SCon{c, t}:
      U32.show(Char.to_u32(c)) ++ " " ++ codes(t)

def read_go(m: File & Result<&1, &1, (U32 & String), String>) -> IO(Unit):
  (f, r) = m
  do IO<Unit>:
    data : String <- IO.pass(String, r)
    p : Unit <- IO.print(codes(data))
    File.close(f)

def main() -> IO(Unit):
  do IO<Unit>:
    f : File <- IO.try(File, File.open("./bad.bin", "r"))
    m : File & Result<&1, &1, (U32 & String), String> <- File.read(f, 64)
    read_go(m)

Make the input with:

printf 'A|\200|\300\200|\355\240\200|\367\277\277\277|\377|\346\227|\303\251' > bad.bin

bend --version

bend 2.0.5 (at 0b7e2b1)

uname -sm

Linux x86_64

clang --version (the first line)

Debian clang version 14.0.6