Repository navigation
fix: carry a share link in a compact wire format instead of compressed JSON - #75
Merged
xevrion merged 1 commit intoSep 14, 2026
Merged
Conversation
…d JSON
A share link has to fit where links get pasted: Discord, Slack, Teams
and most mail clients stop carrying a URL somewhere around 2000
characters. The d2 format (readable Topology JSON with default config
fields stripped, then deflated) ran the largest worked examples past
2400 characters of fragment, so the designs people would most want to
share were the ones a link could not carry.
The wrong fix was to compress harder. Deflate was already the last step
and it cannot recover bytes the layout wastes: every node repeated its
property names, its string id, and values both ends already knew. The
fix is to stop sending that information at all, and only then compress.
What the developer experience keeps
-----------------------------------
- `SimNode`, `SimEdge`, `NodeConfig`, `Annotation` and `Topology` are
untouched. The app, the engine, the inspector, the design file and the
saved session all keep reading and writing the readable model. Nothing
outside `share.ts` knows the wire format exists.
- `encodeTopology` / `decodeTopology` / `buildShareUrl` / `hasShareHash`
keep their signatures. `decodeTopology` still returns the same
`ShareResult` union, still never throws, and still hands the result
through the SAME `isTopology` and `sanitizeAnnotations` gates every
other untrusted input passes. There is no second validator.
- Old links keep opening. `d1.` and `d2.` readers are kept, unchanged,
behind the prefix that names them. The version prefix is what makes
a format bump detectable rather than misparsed.
- Adding a component is unchanged in shape: append the kind to the
wire's kind table, and a test fails until you do. Adding a config
field is a compile error in the wire tests until the field table
names it, and the encoder can still carry a field its kind's schema
does not list, so nothing is ever silently trimmed.
Schema over compression: one node and its wires
------------------------------------------------
The wire is a separate module, `src/share/wire.ts`, with frozen
protocol tables (`KINDS`, `FIELDS`, per-kind `SCHEMA`, default
`LABELS`) that are literals, not imports of the app's own lists, so the
palette can reorder freely without moving a wire value. Append only;
an incompatible change is a new prefix.
One node is:
varint kind index into KINDS, never the string
sint x, sint y rounded to whole pixels (as d2 already did)
varint label 0 = the kind's default label; else utf8 len + 1
varint mask<<1|x one bit per field in SCHEMA[kind], set only when
the value differs from defaultConfig(kind)
value* only the values whose bit is set, in schema order
[extras] x=1: count, then (FIELDS index, value) pairs for
a field outside this kind's schema
Node ids are not sent. They are `kind-N` counters the editor mints (or
short handles an example author typed), never shown to the reader and
never hashed by the engine, so they carry no design information. The
decoder mints `kind-N` per kind, the same shape `makeNode` uses and the
same shape the shell dedupes against, so a later add cannot collide.
One wire (edge) is therefore:
varint from node INDEX, not id
varint to<<1|f node index; low bit says whether flags follow
[byte flags] control, weight != 1, latencyMs, bandwidthRps,
lossRate
[values] only the ones whose bit is set
A plain request edge, which is nearly all of them, is two bytes. Edge
ORDER is preserved exactly, because a region node's regions are its
outgoing edges by index; the id comes back as `from->to`.
Annotations follow the same rule: a header varint of presence bits
(size, font, colour, tone, bold, italic, underline, scale for a note;
tone and colour for a section), then only what is present. Ids come
back as `note-N` / `section-N`.
Numbers are one varint, `zigzag(mantissa) * 8 + tag`, where the tag is
a power of ten (x1, x10^-1..10^-4, x100, x1000) chosen from what a
design actually holds; anything that does not fit takes a float64
escape. Every value round-trips bit for bit, including 1/3, 1e-7 and
Number.MAX_VALUE. The only quantisation anywhere is node x/y to whole
pixels, which d2 already did.
Then, and only then, deflate, and only when it helps: a three-node
design is shorter raw, so the flag byte says which of the two follows.
Measured, not estimated
-----------------------
`bun run tools/share-bench.ts` prints the table this was tuned against.
Full URL characters on the production origin:
example n e a d2 bin bin+z d3 url
single-server 3 2 1 495 254 203 275 306
discord 21 17 10 2146 1479 959 1283 1314
netflix 20 18 9 2307 1465 1026 1373 1404
spotify 21 18 8 2406 1683 1082 1447 1478
Largest link is now 1478 characters (was 2406 of fragment alone). All
23 worked examples fit. The two layouts the spec asked to be measured:
per-kind bitmask 250 bytes vs count-plus-field-index 771 across every
example (mask kept); 0 of 230 labels repeat inside a design (no string
table). Where the bytes still go on the largest design: 1100 of 1683
raw bytes are note text, which deflate handles.
Hard limit, no silent trimming
------------------------------
`buildShareUrl` throws `ShareLinkTooLargeError` when the whole URL
would exceed `MAX_SHARE_URL_CHARS` (2000). Nothing is truncated,
dropped or rounded to fit. The copy-link toast says how big the design
is and points at the file export, which has no limit. There is no
server fallback: the app is a static site and promises a design never
leaves the browser.
Tests
-----
- `src/share/wire.test.ts`: table completeness (runtime and compile
time), round trips for every kind, every field, every edge option,
every note style, unicode and empty labels, large and fractional
coordinates, exact numbers, id minting, parallel edges, determinism
(same bytes regardless of ids or key order; decode then encode is a
fixed point), and hostile bytes: every truncation of a real payload,
trailing bytes, unknown kind/bit/field, item-count bomb, NaN through
the float escape, 500 random byte strings. None throw.
- `src/share.test.ts`: both compression paths, every preset under the
limit, the limit counting the base URL, the too-large error, d1 and
d2 links opening as their generation wrote them, and the existing
hostile-input suite.
|
@kaihere14 is attempting to deploy a commit to the whoarrryou's projects Team on Vercel. A member of the Team first needs to authorize it. |
Collaborator
|
Ran this properly: all 23 examples fit, old d1 and d2 links still open, and every decoded design simulates bit for bit identically. Threw 430 hostile inputs at the decoder too, no throws or hangs. Merging. Really nice work, thanks! |
3 of 4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A share link has to fit where links get pasted: Discord, Slack, Teams and most mail clients stop carrying a URL somewhere around 2000 characters. The d2 format (readable Topology JSON with default config fields stripped, then deflated) ran the largest worked examples past 2400 characters of fragment, so the designs people would most want to share were the ones a link could not carry.
The wrong fix was to compress harder. Deflate was already the last step and it cannot recover bytes the layout wastes: every node repeated its property names, its string id, and values both ends already knew. The fix is to stop sending that information at all, and only then compress.
What the developer experience keeps
SimNode,SimEdge,NodeConfig,AnnotationandTopologyare untouched. The app, the engine, the inspector, the design file and the saved session all keep reading and writing the readable model. Nothing outsideshare.tsknows the wire format exists.encodeTopology/decodeTopology/buildShareUrl/hasShareHashkeep their signatures.decodeTopologystill returns the sameShareResultunion, still never throws, and still hands the result through the SAMEisTopologyandsanitizeAnnotationsgates every other untrusted input passes. There is no second validator.d1.andd2.readers are kept, unchanged, behind the prefix that names them. The version prefix is what makes a format bump detectable rather than misparsed.Schema over compression: one node and its wires
The wire is a separate module,
src/share/wire.ts, with frozen protocol tables (KINDS,FIELDS, per-kindSCHEMA, defaultLABELS) that are literals, not imports of the app's own lists, so the palette can reorder freely without moving a wire value. Append only; an incompatible change is a new prefix.One node is:
Node ids are not sent. They are
kind-Ncounters the editor mints (or short handles an example author typed), never shown to the reader and never hashed by the engine, so they carry no design information. The decoder mintskind-Nper kind, the same shapemakeNodeuses and the same shape the shell dedupes against, so a later add cannot collide.One wire (edge) is therefore:
A plain request edge, which is nearly all of them, is two bytes. Edge ORDER is preserved exactly, because a region node's regions are its outgoing edges by index; the id comes back as
from->to.Annotations follow the same rule: a header varint of presence bits (size, font, colour, tone, bold, italic, underline, scale for a note; tone and colour for a section), then only what is present. Ids come back as
note-N/section-N.Numbers are one varint,
zigzag(mantissa) * 8 + tag, where the tag is a power of ten (x1, x10^-1..10^-4, x100, x1000) chosen from what a design actually holds; anything that does not fit takes a float64 escape. Every value round-trips bit for bit, including 1/3, 1e-7 and Number.MAX_VALUE. The only quantisation anywhere is node x/y to whole pixels, which d2 already did.Then, and only then, deflate, and only when it helps: a three-node design is shorter raw, so the flag byte says which of the two follows.
Measured, not estimated
bun run tools/share-bench.tsprints the table this was tuned against. Full URL characters on the production origin:Largest link is now 1478 characters (was 2406 of fragment alone). All 23 worked examples fit. The two layouts the spec asked to be measured: per-kind bitmask 250 bytes vs count-plus-field-index 771 across every example (mask kept); 0 of 230 labels repeat inside a design (no string table). Where the bytes still go on the largest design: 1100 of 1683 raw bytes are note text, which deflate handles.
Hard limit, no silent trimming
buildShareUrlthrowsShareLinkTooLargeErrorwhen the whole URL would exceedMAX_SHARE_URL_CHARS(2000). Nothing is truncated, dropped or rounded to fit. The copy-link toast says how big the design is and points at the file export, which has no limit. There is no server fallback: the app is a static site and promises a design never leaves the browser.Tests
src/share/wire.test.ts: table completeness (runtime and compile time), round trips for every kind, every field, every edge option, every note style, unicode and empty labels, large and fractional coordinates, exact numbers, id minting, parallel edges, determinism (same bytes regardless of ids or key order; decode then encode is a fixed point), and hostile bytes: every truncation of a real payload, trailing bytes, unknown kind/bit/field, item-count bomb, NaN through the float escape, 500 random byte strings. None throw.src/share.test.ts: both compression paths, every preset under the limit, the limit counting the base URL, the too-large error, d1 and d2 links opening as their generation wrote them, and the existing hostile-input suite.bun run testpasses. Note therun: plainbun testbypasses the jsdom setup andreports failures that are not real
bun run typecheckandbun run lintpassbun run buildpassesCloses #55