Repository navigation
Conversation
A chat message took about 10.7 KB on the server for a few bytes of text. Three trees held a third of that as base64 and msgpack that repeats itself. Values in them are now deflated, under a marker byte that no earlier row can start with, so rows an older store wrote are read as they are. A 50-message chat drops from 10.7 KB to 8.3 KB per message. Adds an ignored test that prints the bytes per tree for a chat.
A resource's genesis commit already has a row of its own in the resource tree, so its envelope held the same signed bytes again: about 2.4 KB for every chat message. The envelope is no longer written for a genesis commit. Reading a resource's envelopes rebuilds it from that row while nothing newer has replaced it. Under the all-envelopes retention the first later commit writes the genesis envelope out before it, so history is unchanged. A chat message now takes 6.8 KB instead of 10.7 KB.
A chat message wrote ten rows into each atom index. Five were its creation commit's atoms, which nothing queries (the commit is found by its id), and the value index also held descriptions and timestamps. The commit of a creation is no longer indexed, and the value index only gets values that can be references. Rights, parent and destroy commits are still indexed by subject. A query on a plain text value without a property no longer finds new resources; references still do. Old rows go when their resource changes. A chat message now takes 4.2 KB instead of 10.7 KB. Adds planning/chat-message-storage.md with the measurement and what is left.
The genesis commit row already keeps the signed loroUpdate, and the snapshot of a resource nobody has edited is that same data plus a few server-written changes. The snapshot now keeps only what comes after the genesis (about 250 bytes for a chat message instead of 1 KB) and a read puts the two back together. It falls back to the whole snapshot when the commit row is missing, as for a resource that arrived by sync, or when the rebuild does not match. A chat message now takes 3.5 KB instead of 10.7 KB.
joepio
marked this pull request as ready for review
October 5, 2026 18:19
Member
Author
|
The failed Generated by Claude Code |
joepio
added a commit
that referenced
this pull request
Oct 6, 2026
One tested commit for the open PRs of 5 October: link device to account (#1948), one profile form (#1953), faster branch runs (#1956), end-to-end encrypted conversations (#1971), passkey ceremony header (#1972), Flutter account package (#2035), cheap canvas undo (#2053), atomic.place tab title (#2056), table resource cell (#2057), small table fixes (#2058), storage map (#2061), Sync page cloud ladder (#2072), several served-domain suffixes (#2073), one sign-in (#2074), smaller chat messages (#2077), bring-along e2e fix (#2078), inline row data loss (#2079), Retry-After (#2080), #2070, #2071, #2066, #2067, #2068, #1969, #1887, #1897 and #1920. Compressed stored values from #2077 stay readable, but a downgrade cannot read compressed rows. Co-authored-by: Joep Meindertsma <joep@ontola.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Joep · project thread
Before: a chat message with the text "hallo" took about 10.7 KB in the server store (measured on 50 messages, after #1967 (removed duplicate index rows)). The same signed data sat in four places (stored resource, genesis commit row, envelope, Loro snapshot) and the two atom indexes wrote ten rows each.
After: 3.5 KB per message (-67%), measured by the new ignored test
measure_chat_message_bytes.PropValSubValPropSubHow, in four commits:
CompressedKvwraps the store'sKvStore; resources, Loro snapshots and envelopes are deflated. A compressed value starts with0x00plus a codec byte, which no earlier value in these trees can start with, so old rows are read as they are. Pure Rust (miniz_oxide), so it builds for WASM.envelopes()rebuilds it from the commit row (same JSON, asserted in a test). Under all-envelopes retention the first later commit writes it out.ValPropSub) only gets values that can be references. Joep agreed to the two behavior changes: a query on a plain text value without a property no longer finds new resources (references still do), and a creation's commit is no longer found through thesubjectindex. The two tests that pinned the old behavior are adjusted.loroUpdate; the snapshot keeps only what came after it, and a read puts the two back together. It stores the whole snapshot when the commit row is missing (a resource that arrived by sync) or when the rebuild does not match the original version vector.Old stores keep working: nothing is rewritten on upgrade, old rows are read as they are, and old index rows go when their resource changes. A downgrade to a build without this change cannot read compressed rows. Reads of a delta snapshot cost two extra Loro imports; if that shows up in collection queries, a small cache is the next step. The compaction test fixture used repeated text and now uses noise.
planning/chat-message-storage.mdhas the measurement and what is left (shorter index keys, search rows, and short property keys inside the signed Loro document, which is a protocol change).