Skip to content

fix: carry a share link in a compact wire format instead of compressed JSON - #75

Merged
xevrion merged 1 commit into
workersio:mainfrom
kaihere14:fix/share-link-compact-wire-format
Sep 14, 2026
Merged

xevrion merged 1 commit into
workersio:mainfrom
kaihere14:fix/share-link-compact-wire-format

Conversation

@kaihere14

@kaihere14 kaihere14 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

A share link has to fit where links get pasted: Discord, Slack, Teams and most mail clients stop carrying a URL somewhere around 2000 characters. The d2 format (readable Topology JSON with default config fields stripped, then deflated) ran the largest worked examples past 2400 characters of fragment, so the designs people would most want to share were the ones a link could not carry.

The wrong fix was to compress harder. Deflate was already the last step and it cannot recover bytes the layout wastes: every node repeated its property names, its string id, and values both ends already knew. The fix is to stop sending that information at all, and only then compress.

What the developer experience keeps

  • SimNode, SimEdge, NodeConfig, Annotation and Topology are untouched. The app, the engine, the inspector, the design file and the saved session all keep reading and writing the readable model. Nothing outside share.ts knows the wire format exists.
  • encodeTopology / decodeTopology / buildShareUrl / hasShareHash keep their signatures. decodeTopology still returns the same ShareResult union, still never throws, and still hands the result through the SAME isTopology and sanitizeAnnotations gates every other untrusted input passes. There is no second validator.
  • Old links keep opening. d1. and d2. readers are kept, unchanged, behind the prefix that names them. The version prefix is what makes a format bump detectable rather than misparsed.
  • Adding a component is unchanged in shape: append the kind to the wire's kind table, and a test fails until you do. Adding a config field is a compile error in the wire tests until the field table names it, and the encoder can still carry a field its kind's schema does not list, so nothing is ever silently trimmed.

Schema over compression: one node and its wires

The wire is a separate module, src/share/wire.ts, with frozen protocol tables (KINDS, FIELDS, per-kind SCHEMA, default LABELS) that are literals, not imports of the app's own lists, so the palette can reorder freely without moving a wire value. Append only; an incompatible change is a new prefix.

One node is:

varint  kind         index into KINDS, never the string
sint    x, sint y    rounded to whole pixels (as d2 already did)
varint  label        0 = the kind's default label; else utf8 len + 1
varint  mask<<1|x    one bit per field in SCHEMA[kind], set only when
                     the value differs from defaultConfig(kind)
value*               only the values whose bit is set, in schema order
[extras]             x=1: count, then (FIELDS index, value) pairs for
                     a field outside this kind's schema

Node ids are not sent. They are kind-N counters the editor mints (or short handles an example author typed), never shown to the reader and never hashed by the engine, so they carry no design information. The decoder mints kind-N per kind, the same shape makeNode uses and the same shape the shell dedupes against, so a later add cannot collide.

One wire (edge) is therefore:

varint  from         node INDEX, not id
varint  to<<1|f      node index; low bit says whether flags follow
[byte   flags]       control, weight != 1, latencyMs, bandwidthRps,
                     lossRate
[values]             only the ones whose bit is set

A plain request edge, which is nearly all of them, is two bytes. Edge ORDER is preserved exactly, because a region node's regions are its outgoing edges by index; the id comes back as from->to.

Annotations follow the same rule: a header varint of presence bits (size, font, colour, tone, bold, italic, underline, scale for a note; tone and colour for a section), then only what is present. Ids come back as note-N / section-N.

Numbers are one varint, zigzag(mantissa) * 8 + tag, where the tag is a power of ten (x1, x10^-1..10^-4, x100, x1000) chosen from what a design actually holds; anything that does not fit takes a float64 escape. Every value round-trips bit for bit, including 1/3, 1e-7 and Number.MAX_VALUE. The only quantisation anywhere is node x/y to whole pixels, which d2 already did.

Then, and only then, deflate, and only when it helps: a three-node design is shorter raw, so the flag byte says which of the two follows.

Measured, not estimated

bun run tools/share-bench.ts prints the table this was tuned against. Full URL characters on the production origin:

example               n   e   a    d2    bin  bin+z    d3    url
single-server         3   2   1   495    254    203   275    306
discord              21  17  10  2146   1479    959  1283   1314
netflix              20  18   9  2307   1465   1026  1373   1404
spotify              21  18   8  2406   1683   1082  1447   1478

Largest link is now 1478 characters (was 2406 of fragment alone). All 23 worked examples fit. The two layouts the spec asked to be measured: per-kind bitmask 250 bytes vs count-plus-field-index 771 across every example (mask kept); 0 of 230 labels repeat inside a design (no string table). Where the bytes still go on the largest design: 1100 of 1683 raw bytes are note text, which deflate handles.

Hard limit, no silent trimming

buildShareUrl throws ShareLinkTooLargeError when the whole URL would exceed MAX_SHARE_URL_CHARS (2000). Nothing is truncated, dropped or rounded to fit. The copy-link toast says how big the design is and points at the file export, which has no limit. There is no server fallback: the app is a static site and promises a design never leaves the browser.

Tests

  • src/share/wire.test.ts: table completeness (runtime and compile time), round trips for every kind, every field, every edge option, every note style, unicode and empty labels, large and fractional coordinates, exact numbers, id minting, parallel edges, determinism (same bytes regardless of ids or key order; decode then encode is a fixed point), and hostile bytes: every truncation of a real payload, trailing bytes, unknown kind/bit/field, item-count bomb, NaN through the float escape, 500 random byte strings. None throw.
  • src/share.test.ts: both compression paths, every preset under the limit, the limit counting the base URL, the too-large error, d1 and d2 links opening as their generation wrote them, and the existing hostile-input suite.
screenshot_20260914_020452
  • bun run test passes. Note the run: plain bun test bypasses the jsdom setup and
    reports failures that are not real
  • bun run typecheck and bun run lint pass
  • bun run build passes
  • Screenshot attached, if anything visual changed

Closes #55

…d JSON

A share link has to fit where links get pasted: Discord, Slack, Teams
and most mail clients stop carrying a URL somewhere around 2000
characters. The d2 format (readable Topology JSON with default config
fields stripped, then deflated) ran the largest worked examples past
2400 characters of fragment, so the designs people would most want to
share were the ones a link could not carry.

The wrong fix was to compress harder. Deflate was already the last step
and it cannot recover bytes the layout wastes: every node repeated its
property names, its string id, and values both ends already knew. The
fix is to stop sending that information at all, and only then compress.

What the developer experience keeps
-----------------------------------

- `SimNode`, `SimEdge`, `NodeConfig`, `Annotation` and `Topology` are
  untouched. The app, the engine, the inspector, the design file and the
  saved session all keep reading and writing the readable model. Nothing
  outside `share.ts` knows the wire format exists.
- `encodeTopology` / `decodeTopology` / `buildShareUrl` / `hasShareHash`
  keep their signatures. `decodeTopology` still returns the same
  `ShareResult` union, still never throws, and still hands the result
  through the SAME `isTopology` and `sanitizeAnnotations` gates every
  other untrusted input passes. There is no second validator.
- Old links keep opening. `d1.` and `d2.` readers are kept, unchanged,
  behind the prefix that names them. The version prefix is what makes
  a format bump detectable rather than misparsed.
- Adding a component is unchanged in shape: append the kind to the
  wire's kind table, and a test fails until you do. Adding a config
  field is a compile error in the wire tests until the field table
  names it, and the encoder can still carry a field its kind's schema
  does not list, so nothing is ever silently trimmed.

Schema over compression: one node and its wires
------------------------------------------------

The wire is a separate module, `src/share/wire.ts`, with frozen
protocol tables (`KINDS`, `FIELDS`, per-kind `SCHEMA`, default
`LABELS`) that are literals, not imports of the app's own lists, so the
palette can reorder freely without moving a wire value. Append only;
an incompatible change is a new prefix.

One node is:

    varint  kind         index into KINDS, never the string
    sint    x, sint y    rounded to whole pixels (as d2 already did)
    varint  label        0 = the kind's default label; else utf8 len + 1
    varint  mask<<1|x    one bit per field in SCHEMA[kind], set only when
                         the value differs from defaultConfig(kind)
    value*               only the values whose bit is set, in schema order
    [extras]             x=1: count, then (FIELDS index, value) pairs for
                         a field outside this kind's schema

Node ids are not sent. They are `kind-N` counters the editor mints (or
short handles an example author typed), never shown to the reader and
never hashed by the engine, so they carry no design information. The
decoder mints `kind-N` per kind, the same shape `makeNode` uses and the
same shape the shell dedupes against, so a later add cannot collide.

One wire (edge) is therefore:

    varint  from         node INDEX, not id
    varint  to<<1|f      node index; low bit says whether flags follow
    [byte   flags]       control, weight != 1, latencyMs, bandwidthRps,
                         lossRate
    [values]             only the ones whose bit is set

A plain request edge, which is nearly all of them, is two bytes. Edge
ORDER is preserved exactly, because a region node's regions are its
outgoing edges by index; the id comes back as `from->to`.

Annotations follow the same rule: a header varint of presence bits
(size, font, colour, tone, bold, italic, underline, scale for a note;
tone and colour for a section), then only what is present. Ids come
back as `note-N` / `section-N`.

Numbers are one varint, `zigzag(mantissa) * 8 + tag`, where the tag is
a power of ten (x1, x10^-1..10^-4, x100, x1000) chosen from what a
design actually holds; anything that does not fit takes a float64
escape. Every value round-trips bit for bit, including 1/3, 1e-7 and
Number.MAX_VALUE. The only quantisation anywhere is node x/y to whole
pixels, which d2 already did.

Then, and only then, deflate, and only when it helps: a three-node
design is shorter raw, so the flag byte says which of the two follows.

Measured, not estimated
-----------------------

`bun run tools/share-bench.ts` prints the table this was tuned against.
Full URL characters on the production origin:

    example               n   e   a    d2    bin  bin+z    d3    url
    single-server         3   2   1   495    254    203   275    306
    discord              21  17  10  2146   1479    959  1283   1314
    netflix              20  18   9  2307   1465   1026  1373   1404
    spotify              21  18   8  2406   1683   1082  1447   1478

Largest link is now 1478 characters (was 2406 of fragment alone). All
23 worked examples fit. The two layouts the spec asked to be measured:
per-kind bitmask 250 bytes vs count-plus-field-index 771 across every
example (mask kept); 0 of 230 labels repeat inside a design (no string
table). Where the bytes still go on the largest design: 1100 of 1683
raw bytes are note text, which deflate handles.

Hard limit, no silent trimming
------------------------------

`buildShareUrl` throws `ShareLinkTooLargeError` when the whole URL
would exceed `MAX_SHARE_URL_CHARS` (2000). Nothing is truncated,
dropped or rounded to fit. The copy-link toast says how big the design
is and points at the file export, which has no limit. There is no
server fallback: the app is a static site and promises a design never
leaves the browser.

Tests
-----

- `src/share/wire.test.ts`: table completeness (runtime and compile
  time), round trips for every kind, every field, every edge option,
  every note style, unicode and empty labels, large and fractional
  coordinates, exact numbers, id minting, parallel edges, determinism
  (same bytes regardless of ids or key order; decode then encode is a
  fixed point), and hostile bytes: every truncation of a real payload,
  trailing bytes, unknown kind/bit/field, item-count bomb, NaN through
  the float escape, 500 random byte strings. None throw.
- `src/share.test.ts`: both compression paths, every preset under the
  limit, the limit counting the base URL, the too-large error, d1 and
  d2 links opening as their generation wrote them, and the existing
  hostile-input suite.
@kaihere14
kaihere14 requested a review from xevrion as a code owner September 13, 2026 20:35
@vercel

vercel Bot commented Sep 13, 2026

Copy link
Copy Markdown

@kaihere14 is attempting to deploy a commit to the whoarrryou's projects Team on Vercel.

A member of the Team first needs to authorize it.

@xevrion

xevrion commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Ran this properly: all 23 examples fit, old d1 and d2 links still open, and every decoded design simulates bit for bit identically. Threw 430 hostile inputs at the decoder too, no throws or hangs.

Merging. Really nice work, thanks!

@xevrion
xevrion merged commit b959043 into workersio:main Sep 14, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A design too big to link produces one anyway, and it breaks silently where it is pasted

2 participants