Skip to content

Host TOON rendering throws an untyped TypeError on unpaired surrogates in extension output, while JSON passes them through #1624

Closed

Description

@unbraind

Version: @unbrained/pm-cli 2026.10.28 SDK (bundles @toon-format/toon 4.4.0), Node v24.19.0; also seen with Bun.

Observed: pm's TOON rendering throws an untyped TypeError when a value contains an unpaired UTF-16 surrogate, while JSON rendering of the same value succeeds:

import { encodeToon } from "@unbrained/pm-cli/sdk/testing";
encodeToon({ assumption: "\ud800" });
// TypeError: Cannot encode string value containing an unpaired surrogate U+D800 at index 0
//     at assertNoLoneSurrogate (…/@toon-format/toon/…)

Native item writes already refuse lone surrogates with a typed error (text_unpaired_surrogate, with the field path), so tracker data is protected. The gap is output from extensions: a package command's result rendered by the host in the default TOON format crashes the whole command with an untyped error, losing every valid sibling value. The same result renders under --json, where the lone surrogate is passed through. Data an extension reads from external sources (Slack, Jira, CSV or file contents, model output) can legitimately contain lone surrogates.

Why it matters for agents: TOON is the default output. One stray code unit in one field turns a successful read into an untyped crash, with no field path and no remedy. JSON and TOON also disagree on the same data.

Expected (any consistent policy):

  1. The host renderer refuses with a typed error naming the field path (like text_unpaired_surrogate on writes), or
  2. replaces lone surrogates with U+FFFD in both JSON and TOON output, documented, so both formats agree.

Either way, the SDK should export the policy as a public helper (for example toWellFormedText) so extensions can normalize at their own boundaries without importing @toon-format/toon directly.

Related: #1478 (compaction splitting emoji, which produced lone surrogates in output).

Activity

  1. unbraind commented on Oct 29, 2026

    @unbraind
    OwnerAuthor

    Tracked in pm-gh1624. This intake covers already-malformed extension output and its SDK contract; the completed persisted-input and valid-pair compaction work remains linked as prior lineage. It is open and unclaimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions