Skip to content

Per-table change list with tombstones (#1850) - #1885

Closed
michielbdejong wants to merge 1 commit into
developfrom
claude/table-change-list
Closed

michielbdejong wants to merge 1 commit into
developfrom
claude/table-change-list

Conversation

@michielbdejong

Copy link
Copy Markdown
Contributor

Closes #1850.

A per-table change list: "which rows of table T changed since cursor C?", with tombstones and each row's Loro version, for plugin two-way sync ("hook for speed, change list for correctness", ontola/atomic-plugins#177 Q4–7). It also covers the "durable resource-change cursors/tombstones" item of ontola/atomic-plugins#167 §5. The durable afterCommit hook (#1851) is expected to read from it.

Design

What counts as a table. I used the data browser's own rule (useTableData). A table is a resource whose isA includes Table and that has a classtype. A row is a resource whose parent is the table and whose isA includes that classtype. The table's views and other children are not rows.

  • A row enters the table when it is created there, moved in, or given the class. That is logged as created.
  • A row that is edited and stays in the table is logged as updated.
  • A row leaves the table when it is destroyed, moved out, or loses the class. That is logged as deleted for that table.

Storage. A new tree, Tree::TableChanges (table_changes_v1), is keyed by the table's canonical pure id. It holds four key families:

  • m\0T: metadata (epoch, last seq, floor, backfilled flag).
  • e\0T\0seq: one entry per change ({subject, kind, version, at}).
  • r\0T\0row: the seq of that row's latest entry.
  • d\0T\0seq: a tombstone time index, so pruning never scans live entries.

The log is written on the three paths that change stored state, in the same transaction as the state:

  • apply_commit, which covers local and remote commits, including the WS/Iroh COMMIT frame;
  • persist_resource_projection, which covers replicated resources from sync and import;
  • remove_applied, which covers sync-applied removals.

A destroy logs each removed row. Destroying a table deletes its whole log.

Ordering. Sequence numbers are allocated under a per-store mutex. The mutex is held from allocation until the transaction is applied and never across an .await. So entries become visible in seq order, and a reader can't skip one that commits late with a lower number.

Compaction. Only the latest entry per row per table is kept. A new entry deletes the row's previous one in the same transaction. The log therefore never needs a separate compaction pass. A row edited while a client is paging moves to a higher seq, so it is still delivered.

Cursor. Opaque to clients: base64url of {v:1, t:table, e:epoch, s:seq, f:full}.

  • The epoch changes when the table's classtype changes, because that changes which children are rows. The log is rebuilt and older cursors expire.
  • f marks the pages of a first listing (a request without since). Pruning doesn't expire those, because a client starting from nothing never needed the pruned tombstones.

Retention. Tombstones are kept for 30 days by default. This is configurable with --table-change-retention-days / ATOMIC_TABLE_CHANGE_RETENTION_DAYS, or Db::set_table_change_retention.

  • Expired tombstones are pruned when the table is read, and the table's floor is raised to the highest pruned seq.
  • A cursor below the floor gets a typed CURSOR_EXPIRED answer, and the client resyncs.
  • Live entries are never pruned.
  • This does not interact with db::compaction, which reclaims redb file pages. Pruning only frees pages that a later file compaction can give back.

Backfill. The first read of a table with no log, or one not yet backfilled, lists its current children of the classtype. It writes a created entry for each row that has none yet, then marks the table backfilled.

  • This is idempotent, and race-safe: any row changed in the meantime already has an entry and is skipped.
  • It also covers rows that arrived by sync before their table did.

Version. Each entry carries the row's oplog version vector as {peerId: counter}. This is what loro-crdt gives for doc.version().toJSON(), and what an export({mode:'update', from}) needs. For a tombstone, it is the last version this node held.

API

  • Rust: atomic_lib::change_log::table_changes(&db, &table, since, limit, &for_agent) -> Result<ChangePage, ChangeListError>.
  • HTTP: GET /changes?table=<subject>&since=<cursor>&limit=<n> returns {changes: [{subject, kind, version, at}], cursor, hasMore}.
    • The default page size is 100 and the maximum is 500.
    • Refusals: 410 {error:"CURSOR_EXPIRED"}, 400 INVALID_CURSOR, 400 NOT_A_TABLE, and 401/404 when the caller can't read the table. A 401/404 names no row.
  • Rights:
    • The caller must be able to read the table, or it gets nothing.
    • Each live row is also read-checked, with a per-request rights cache. A row the caller can't read is skipped, and the cursor moves past it.
    • Tombstones go to the table's readers. While the row was in the table, rights were inherited from it, so those readers could already read it.
    • Note: under today's additive rights, every row of a readable table is readable. The per-row check is defense in depth.
  • @tomic/lib: store.getTableChanges(table, {since, limit}), plus a standalone fetchTableChanges. A 410 is thrown as TableChangesCursorExpiredError (code: 'CURSOR_EXPIRED').
  • Plugin host call (ctx): left out on purpose, because Durable JS afterCommit class hook for user-installed plugins, scoped to tables where their view is installed #1851 defines the plugin side.

Docs: docs/src/changes.md (under Endpoints). TESTING_COVERAGE.md is updated.

Tests

  • lib/src/change_log_test.rs (15 tests):
    • the kinds for create, update, move out, destroy and move back in, checked across two tables;
    • one stored entry per row;
    • the per-row version is present, moves with each edit, and a tombstone keeps the last version;
    • bounded pages that together return every row;
    • a row edited between pages is still delivered;
    • cursor expiry after a tombstone is pruned, followed by a working resync;
    • tombstones are kept within the retention period;
    • invalid, foreign-table and not-a-table requests are refused;
    • a classtype change rebuilds the log and expires cursors;
    • backfill of rows that predate the log, skipping non-row children;
    • replicated (persist_replicated_resource) and sync-removed rows are logged;
    • destroying a table drops its log;
    • a reader without read rights on the table gets nothing, even with a grant on a single row;
    • a missing row is skipped and the cursor passes it;
    • restart durability (reopening the redb file).
  • server/src/tests.rs table_changes_endpoint: the real HTTP handler returns 200 with the row's version, refuses a stranger, and returns typed 400 and 410 answers.
  • browser/lib/src/table-changes.test.ts: parsing, the request shape, and 410 mapped to the typed error.
  • Not added: a browser/lib integration test against a real server. Creating a table from Node needs the NodeClientDb/WASM harness; the Rust HTTP test covers the server side.

Run:

  • cargo test -p atomic_lib --features db-redb --lib: all pass except db::compaction::tests::startup_compaction_shrinks_a_bloated_store_and_keeps_every_resource. That test also fails on a clean origin/develop on this machine (checked by stashing this change; macOS redb file sizes vary run to run), so it is unrelated to this PR.
  • cargo test -p atomic-server --lib
  • clippy with -D warnings
  • @tomic/lib vitest, lint and typecheck

No UI changes, so there are no screenshots.

Carrying it to the plugin line (claude/atomic-plugins-pin-candidate14)

The change is self-contained:

  • a new lib/src/change_log.rs and its tests;
  • one new Tree variant, with the sled wiring;
  • a change_log field on Db;
  • three call sites in db.rs that swap apply_transaction for apply_with_change_log;
  • the /changes route and handler;
  • the config flag;
  • the @tomic/lib method.

It should cherry-pick onto the candidate branch with at most small conflicts in db.rs and routes.rs. There is no migration: the tree is created on open, and existing tables are backfilled on first read.

On the plugin line, #1851 would:

  1. read table_changes directly from Rust in the trigger queue worker, keeping a durable cursor per installation and table in PluginTrigger/PluginMeta;
  2. use the live afterCommit event only as a wake-up;
  3. treat CURSOR_EXPIRED as "run the plugin's full compare".

A ctx.changes(table, since) host call would be a thin wrapper around the same function.

Questions for Michiel

  1. Who may read a table's change list? Anyone who can read the table: the conservative default, the same rule as reading its rows. Should it instead require a plugin grant, such as a view installed on the table, the way Durable JS afterCommit class hook for user-installed plugins, scoped to tables where their view is installed #1851 scopes afterCommit? That would stop other readers from seeing when rows changed and which rows were deleted.
  2. Should people see this history in the app (for example, "recently deleted rows" in a table)? It is not exposed in the UI; it is an API for plugins only.
  3. Is 30 days of tombstone retention right? A plugin that stays off longer than that does a full resync.

🤖 Generated with Claude Code

@michielbdejong

Copy link
Copy Markdown
Contributor Author

Answers to the questions above, as engineering calls (Claude, atomic-server worker):

  1. Rights: read rights on the table are enough. Anyone who can read the table can already see its rows. The change list only adds timing and deletions, which aren't more sensitive than the rows themselves. Plugin-specific scoping belongs to Durable JS afterCommit class hook for user-installed plugins, scoped to tables where their view is installed #1851's hook, not to this API.
  2. Showing history in the app: API only for now. A "recently deleted rows" UI would be a separate product decision, not part of this PR.
  3. Retention: 30 days is kept as the default, and it's configurable.

@michielbdejong

Copy link
Copy Markdown
Contributor Author

Michiel's decision: no "recently deleted rows" or history view in the app for now. The change list stays API-only.

Rows of a table changed since an opaque cursor, including rows that left
it, each with its Loro version, for plugin two-way sync ("hook for speed,
change list for correctness").

- atomic_lib::change_log: a durable per-table log in Tree::TableChanges,
  written in the same transaction as the state by apply_commit, replicated
  resource persistence and sync removals. One entry per row, seq allocated
  under a lock held until apply, tombstones pruned after a configurable
  retention (30 days), typed CURSOR_EXPIRED below the floor, backfill of
  pre-existing rows on first read, rebuild when a table's classtype changes.
- GET /changes?table=&since=&limit= with read rights on the table and rows.
- @tomic/lib Store.getTableChanges and fetchTableChanges.
- --table-change-retention-days, docs page, TESTING_COVERAGE entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@michielbdejong

Copy link
Copy Markdown
Contributor Author

Closing as part of the plugin-host stack cleanup (Q-101): not needed by atomic-plugins now. See the inventory at #1726 (comment). The branch is kept, and pin candidates are unaffected. Closed by Claude (VPS coordinator) on behalf of Michiel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request lib rust atomic-lib (rust) plugin Should probably be an Atomic Plugin server atomic-server @tomic/lib @tomic/lib

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-table change list: rows changed since a cursor, including deletions (tombstones)

1 participant