Repository navigation
Per-table change list with tombstones (#1850) - #1885
Closed
michielbdejong wants to merge 1 commit into
Closed
michielbdejong wants to merge 1 commit into
michielbdejong wants to merge 1 commit into
Conversation
Contributor
Author
|
Answers to the questions above, as engineering calls (Claude, atomic-server worker):
|
This was referenced Sep 29, 2026
Contributor
Author
|
Michiel's decision: no "recently deleted rows" or history view in the app for now. The change list stays API-only. |
Rows of a table changed since an opaque cursor, including rows that left
it, each with its Loro version, for plugin two-way sync ("hook for speed,
change list for correctness").
- atomic_lib::change_log: a durable per-table log in Tree::TableChanges,
written in the same transaction as the state by apply_commit, replicated
resource persistence and sync removals. One entry per row, seq allocated
under a lock held until apply, tombstones pruned after a configurable
retention (30 days), typed CURSOR_EXPIRED below the floor, backfill of
pre-existing rows on first read, rebuild when a table's classtype changes.
- GET /changes?table=&since=&limit= with read rights on the table and rows.
- @tomic/lib Store.getTableChanges and fetchTableChanges.
- --table-change-retention-days, docs page, TESTING_COVERAGE entries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joepio
force-pushed
the
claude/table-change-list
branch
from
October 5, 2026 09:35
6b9243b to
2bdec6b
Compare
This was referenced Oct 5, 2026
Contributor
Author
|
Closing as part of the plugin-host stack cleanup (Q-101): not needed by atomic-plugins now. See the inventory at #1726 (comment). The branch is kept, and pin candidates are unaffected. Closed by Claude (VPS coordinator) on behalf of Michiel. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1850.
A per-table change list: "which rows of table T changed since cursor C?", with tombstones and each row's Loro version, for plugin two-way sync ("hook for speed, change list for correctness", ontola/atomic-plugins#177 Q4–7). It also covers the "durable resource-change cursors/tombstones" item of ontola/atomic-plugins#167 §5. The durable
afterCommithook (#1851) is expected to read from it.Design
What counts as a table. I used the data browser's own rule (
useTableData). A table is a resource whoseisAincludesTableand that has aclasstype. A row is a resource whoseparentis the table and whoseisAincludes that classtype. The table's views and other children are not rows.created.updated.deletedfor that table.Storage. A new tree,
Tree::TableChanges(table_changes_v1), is keyed by the table's canonical pure id. It holds four key families:m\0T: metadata (epoch, last seq, floor, backfilled flag).e\0T\0seq: one entry per change ({subject, kind, version, at}).r\0T\0row: the seq of that row's latest entry.d\0T\0seq: a tombstone time index, so pruning never scans live entries.The log is written on the three paths that change stored state, in the same transaction as the state:
apply_commit, which covers local and remote commits, including the WS/IrohCOMMITframe;persist_resource_projection, which covers replicated resources from sync and import;remove_applied, which covers sync-applied removals.A destroy logs each removed row. Destroying a table deletes its whole log.
Ordering. Sequence numbers are allocated under a per-store mutex. The mutex is held from allocation until the transaction is applied and never across an
.await. So entries become visible in seq order, and a reader can't skip one that commits late with a lower number.Compaction. Only the latest entry per row per table is kept. A new entry deletes the row's previous one in the same transaction. The log therefore never needs a separate compaction pass. A row edited while a client is paging moves to a higher seq, so it is still delivered.
Cursor. Opaque to clients: base64url of
{v:1, t:table, e:epoch, s:seq, f:full}.classtypechanges, because that changes which children are rows. The log is rebuilt and older cursors expire.fmarks the pages of a first listing (a request withoutsince). Pruning doesn't expire those, because a client starting from nothing never needed the pruned tombstones.Retention. Tombstones are kept for 30 days by default. This is configurable with
--table-change-retention-days/ATOMIC_TABLE_CHANGE_RETENTION_DAYS, orDb::set_table_change_retention.flooris raised to the highest pruned seq.CURSOR_EXPIREDanswer, and the client resyncs.db::compaction, which reclaims redb file pages. Pruning only frees pages that a later file compaction can give back.Backfill. The first read of a table with no log, or one not yet backfilled, lists its current children of the classtype. It writes a
createdentry for each row that has none yet, then marks the table backfilled.Version. Each entry carries the row's oplog version vector as
{peerId: counter}. This is whatloro-crdtgives fordoc.version().toJSON(), and what anexport({mode:'update', from})needs. For a tombstone, it is the last version this node held.API
atomic_lib::change_log::table_changes(&db, &table, since, limit, &for_agent) -> Result<ChangePage, ChangeListError>.GET /changes?table=<subject>&since=<cursor>&limit=<n>returns{changes: [{subject, kind, version, at}], cursor, hasMore}.410 {error:"CURSOR_EXPIRED"},400 INVALID_CURSOR,400 NOT_A_TABLE, and 401/404 when the caller can't read the table. A 401/404 names no row.@tomic/lib:store.getTableChanges(table, {since, limit}), plus a standalonefetchTableChanges. A 410 is thrown asTableChangesCursorExpiredError(code: 'CURSOR_EXPIRED').ctx): left out on purpose, because Durable JS afterCommit class hook for user-installed plugins, scoped to tables where their view is installed #1851 defines the plugin side.Docs:
docs/src/changes.md(under Endpoints).TESTING_COVERAGE.mdis updated.Tests
lib/src/change_log_test.rs(15 tests):classtypechange rebuilds the log and expires cursors;persist_replicated_resource) and sync-removed rows are logged;server/src/tests.rstable_changes_endpoint: the real HTTP handler returns 200 with the row's version, refuses a stranger, and returns typed 400 and 410 answers.browser/lib/src/table-changes.test.ts: parsing, the request shape, and 410 mapped to the typed error.browser/libintegration test against a real server. Creating a table from Node needs the NodeClientDb/WASM harness; the Rust HTTP test covers the server side.Run:
cargo test -p atomic_lib --features db-redb --lib: all pass exceptdb::compaction::tests::startup_compaction_shrinks_a_bloated_store_and_keeps_every_resource. That test also fails on a cleanorigin/developon this machine (checked by stashing this change; macOS redb file sizes vary run to run), so it is unrelated to this PR.cargo test -p atomic-server --lib-D warnings@tomic/libvitest, lint and typecheckNo UI changes, so there are no screenshots.
Carrying it to the plugin line (
claude/atomic-plugins-pin-candidate14)The change is self-contained:
lib/src/change_log.rsand its tests;Treevariant, with the sled wiring;change_logfield onDb;db.rsthat swapapply_transactionforapply_with_change_log;/changesroute and handler;@tomic/libmethod.It should cherry-pick onto the candidate branch with at most small conflicts in
db.rsandroutes.rs. There is no migration: the tree is created on open, and existing tables are backfilled on first read.On the plugin line, #1851 would:
table_changesdirectly from Rust in the trigger queue worker, keeping a durable cursor per installation and table inPluginTrigger/PluginMeta;afterCommitevent only as a wake-up;CURSOR_EXPIREDas "run the plugin's full compare".A
ctx.changes(table, since)host call would be a thin wrapper around the same function.Questions for Michiel
afterCommit? That would stop other readers from seeing when rows changed and which rows were deleted.🤖 Generated with Claude Code