Char.from_u32 of a surrogate fails in JS and the interpreter but prints invalid UTF-8 natively #1080

Activity

1. VictorTaelin

   added a commit that references this issue on Sep 27, 2026

   Bend 2.0.32: the issue triage, root fixes, one verdict everywhere

   ...

   573002f
2. VictorTaelin

   added

   laterAccepted; parked until we return to itAccepted; parked until we return to it

   on Sep 27, 2026
3. jeromelaurens commented on Oct 5, 2026

   

   jeromelaurens

   on Oct 5, 2026

   More actions

   The same question seen from the JS lane's exports (2.0.35): such a character cannot be built there, but an export receives it from its host, and two of them stored one after the other in a String read back as one character.

   What you did

   Bend 2.0.35 (79df8d9c), Bun 1.3.14, macOS arm64.

   import Base
   
   def reverseOnto(s: String, acc: String) -> String:
     match s:
       case SNil{}: acc
       case SCon{c, rest}: reverseOnto(rest, SCon{c, acc})
   
   def reverseTwice(s: String) -> String:
     reverseOnto(reverseOnto(s, SNil{}), SNil{})

   bend fuse.bend -o fuse.mjs
   bun -e 'import m from "./fuse.mjs"; const s = "\udc00\ud800";
     const r = m.reverseTwice(s);
     console.log([...s].length, [...r].length, r === s)'

   What happened

   reverseTwice returns "𐀀" (one character, U+10000) for the
   input "\udc00\ud800" (two characters, lone surrogates): the output prints
   2 1 false. In Bend's model reverseTwice(s) == s for every string.

   The JS lane represents a String as a host string: SCon{h, t} is
   h + t, and a match reads the first code point, two code units when it is
   above 0xFFFF (OPTIMIZED.String.SCon in bend2/comp.ts). A lone U+D800
   put in front of a string that starts with a lone U+DC00 therefore becomes a
   surrogate pair, and the next match reads one character where two were
   stored. A program run by bend cannot build these characters
   (Char.from_u32 traps, #1080), but an export receives them: JavaScript
   strings may hold lone surrogates, and the export passes them as String
   arguments or as Char fields of data.

   Consequence in practice: a function that collects characters into a
   String and reads it again (an accumulator reversed at the end) changes
   its input; composed with a strict encoder, a text rejected for its lone
   surrogates becomes the four bytes of U+10000.

   What I expected

   The same characters back, as in Bend's model; or, if the JS lane
   does not support characters outside the scalar range, a rejection at the
   export boundary (as char_new does for Char.from_u32), so that the
   model's equations hold for every value a JS function can receive.

   Suggested fix

   Decide, with #1080, what a Char outside the scalar range means on the JS
   lane, and apply it at the export boundary: reject lone surrogates in
   String and Char arguments, or represent such a character so that
   concatenation cannot pair it.

   bend --version

   Bend 2.0.35 (79df8d9c).

   uname -sm

   Darwin arm64
4. Giulio2002 commented on Oct 8, 2026

   

   Giulio2002

   on Oct 8, 2026

   Contributor

   More actions

   Update on 2.0.36 (commit 1d895ca, Linux x86_64, bun 1.4.2), re-verified in isolation — and a wider set of entry points that hit the same JS-side abort:

   • The interpreter half of this issue has been fixed: Char.from_u32(55296) now checks, runs and prints "\u{d800}" on lane 0 (bend t.bend). The C lane prints it too. Only the JS lane still aborts: bend: 55296 is not a Unicode scalar value, exit 1, through chr_new's scalar check.
   • The abort is broader than Char.from_u32, and reaches string literals:

   import Base
   
   def main() -> String:
     "a\u{d800}b"

   • interpreter: "a\u{d800}b" (exit 0); C: same (exit 0);
     bend t.bend -o t.js && bun t.js: bend: 55296 is not a Unicode scalar value, exit 1.
   • the same file with u{110000} or u{ffffffff}: interpreter/C print '\u{110000}' / '\u{ffffffff}'; JS aborts with the same message. Char.from_u32(1114112) and Char.from_u32(4294967295) likewise.
   • it fires even when the bad code is never shown — String.length/String.eq/String.is_empty over such a value abort the JS program the moment the value crosses chr_new.
   • Front-end intent (for whoever fixes it): lit_of (bend.ts ~1077-1083) deliberately keeps a literal spelling a surrogate or a codepoint past U+10FFFF as an explicit SCon chain, and chr_show (~1288-1297) deliberately renders it — so the abort contradicts both the checker's own representable-literal model and the C lane, rather than guarding an invariant the language has.
   • C-side note, same family: through IO.print the C backend writes a lone surrogate as raw invalid UTF-8 (ed a0 80) since io_utf8 (comp.ts ~5502) does not clamp — the C lane is fine on the pure/show path, only the byte-output side is unclamped.

   Found by an automated AI-model differential-testing campaign against bend 2.0.36; ~2,200 three-lane executions over string/char/list/map cases found no other lane disagreement.
5. Giulio2002 commented on Oct 8, 2026

   

   Giulio2002

   on Oct 8, 2026

   Contributor

   More actions

   Correction to my earlier comment above — the interpreter half of this is only fixed for pure evaluation, not for the IO-run machinery every real program uses (2.0.36, commit 1d895ca, re-verified in isolation):

   import Base
   
   def main() -> IO(Unit):
     do IO<Unit>:
       IO.print(U32.show(Char.to_u32(Char.from_u32(4294967295))))
       IO.print(U32.show(Char.to_u32(Char.from_u32(65536))))
       IO.print(U32.show(Char.to_u32(Char.from_u32(55296))))

   • interpreter (bend t.bend): bend: 4294967295 is not a Unicode scalar value, exit 1;
   • the same computation with a pure main -> U32 / main -> String normalizes and prints correctly (4294967295, 5, …), so the plain-value path was indeed fixed;
   • JS lane: throws in every shape (pure and IO);
   • C lane: accepts the full U32 range everywhere.

   So the current three-lane state is: C fine everywhere; interpreter fine for pure programs, throwing under the IO run; JS throwing everywhere. Bonus oddity found while minimizing: inside one interpreter IO-run, the generic route building a List<&2, Char> of such chars folded by specialized readers succeeds with the correct value, while the direct Char.from_u32 beside it throws (61-char-code-range-divergence-generic.bend in the linked campaign directory) — the check is per-routing rather than per-language.

   Found by an automated AI-model differential-testing campaign against bend 2.0.36 (boxing/word-representation pass).
6. nicolas-abril

   added 3 commits that reference this issue on Oct 13, 2026

   A Char holds any u32 code on every lane; U+FFFD only where text leave…

   ...

   88fb722

   Process.run argv/stdin and get_env name write non-scalar Chars as U+F…

   ...

   4957361

   Simplify non-scalar Char checks and the JS foreign wrapper (bendlang#…

   ...

   e468b1b
7. nicolas-abril

   closed this as completedin #1411on Oct 13, 2026