From 5e407160d8c1e2c20427a6343250289767d116e0 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 18 Sep 2026 20:54:54 +0000 Subject: [PATCH 001/110] docs: tell the local-first story (URLs, Atomic Sync, Flutter) The docs still read as if AtomicServer were an HTTP origin you post commits to. This refresh sells what the code now does, without rewriting the spec pages. New pages: - urls.md: one map of every identifier shape (did:ad resource / agent / commit / blob / node, https, internal:, atomic://pair, ?drive= hints, HTTP aliases of a DID), placed early in the book. - local-first.md: data ownership as "the bytes are on your device", what the server is for now, and the honest trade-offs. - sync.md: the transport-agnostic sync model (signed commits, Loro, version vectors, outbox, rights on every transport) with a transport table. websockets.md and browser-peer-sync.md now sit under it as the wire-format reference and the WebRTC path. - flutter.md: the Dart SDK on top of atomic_lib via flutter_rust_bridge, its API groups, a minimal flow, and how it compares to the other clients. Edits: overview, motivation, when-to-use, Extended index and table, DID status line, API page (fetching did:ad over HTTP, /commit not /commits), tooling, rust-lib feature list, personal-data-store use case, Subject field in core concepts, README realtime bullet. Commits are described as the signed envelope around a write rather than the versioning mechanism. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_018ZmptEWmdz92GXTuP77vrb --- README.md | 2 +- docs/src/SUMMARY.md | 12 ++- docs/src/atomic-data-overview.md | 17 ++-- docs/src/atomic-server.md | 3 +- docs/src/atomicserver/API.md | 15 +++- docs/src/browser-peer-sync.md | 4 +- docs/src/commits/intro.md | 2 +- docs/src/core/concepts.md | 4 +- docs/src/did.md | 7 +- docs/src/extended-table.md | 11 +-- docs/src/extended.md | 2 + docs/src/flutter.md | 109 +++++++++++++++++++++++ docs/src/local-first.md | 59 ++++++++++++ docs/src/motivation.md | 6 +- docs/src/roadmap.md | 2 +- docs/src/rust-lib.md | 10 ++- docs/src/sync.md | 71 +++++++++++++++ docs/src/tooling.md | 5 +- docs/src/urls.md | 104 +++++++++++++++++++++ docs/src/usecases/personal-data-store.md | 13 +-- docs/src/websockets.md | 3 +- docs/src/when-to-use.md | 3 +- 22 files changed, 426 insertions(+), 38 deletions(-) create mode 100644 docs/src/flutter.md create mode 100644 docs/src/local-first.md create mode 100644 docs/src/sync.md create mode 100644 docs/src/urls.md diff --git a/README.md b/README.md index de86ffaa5..580ed1fac 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ _Status: alpha. [Breaking changes](CHANGELOG.md) are expected until 1.0._ - πŸ’¬ **Group chat**, performant and flexible message channels with attachments, search and replies. - πŸ“‚ **File management**: Upload, download and preview attachments. - πŸ’Ύ **Versioning** / history from the Loro oplog, with writes authorized by [Atomic Commits](https://docs.atomicdata.dev/commits/intro.html) -- πŸ”„ **Real-time synchronization**: instantly communicates state changes with a client. Build dynamic, collaborative apps using [websockets](https://docs.atomicdata.dev/websockets) (using a [single one-liner in react](https://docs.atomicdata.dev/usecases/react) or [svelte](https://docs.atomicdata.dev/svelte)). +- πŸ”„ **Real-time synchronization**: instantly communicates state changes with a client. Build dynamic, collaborative apps using [Atomic Sync](https://docs.atomicdata.dev/sync) over WebSocket, Iroh or WebRTC (using a [single one-liner in react](https://docs.atomicdata.dev/usecases/react) or [svelte](https://docs.atomicdata.dev/svelte)). - 🧰 **Many serialization options**: to JSON, [JSON-AD](https://docs.atomicdata.dev/core/json-ad.html), and various Linked Data / RDF formats (RDF/XML, N-Triples / Turtle / JSON-LD). - πŸ“– **Pagination, sorting and filtering** queries using [Atomic Collections](https://docs.atomicdata.dev/schema/collections.html). - πŸ” **Authorization** (read / write permissions) and Hierarchical structures powered by [Atomic Hierarchy](https://docs.atomicdata.dev/hierarchy.html) diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index b7d1ab86e..5f14b9fda 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -4,6 +4,8 @@ - [Atomic Data Overview](atomic-data-overview.md) - [Motivation](motivation.md) + - [Local-first and data ownership](local-first.md) + - [URLs and identifiers](urls.md) - [Strategy, history and roadmap](roadmap.md) - [When (not) to use it](when-to-use.md) @@ -12,7 +14,6 @@ - [AtomicServer](atomic-server.md) - [When (not) to use it](atomicserver/when-to-use.md) - [Installation](atomicserver/installation.md) - - [Browser peer sync](browser-peer-sync.md) - [Using the GUI](atomicserver/gui.md) - [Tables](atomicserver/gui/tables.md) - [AI and Atomic Assistant](atomicserver/gui/ai-and-atomic-assistant.md) @@ -47,6 +48,7 @@ - [Rust](rust.md) - [CLI](rust-cli.md) - [Lib](rust-lib.md) + - [Flutter / Dart (native apps)](flutter.md) # Guides @@ -76,15 +78,17 @@ - [FAQ](schema/faq.md) - [Atomic Data Extended](extended.md) - - [Agents](agents.md) - [Decentralized Identifiers (DIDs)](did.md) + - [Agents](agents.md) - [Hierarchy and authorization](hierarchy.md) - [Authentication](authentication.md) - [Invitations and sharing](invitations.md) - - [Commits (writing data)](commits/intro.md) + - [Atomic Sync](sync.md) + - [Wire protocol: WebSocket and Iroh](websockets.md) + - [Browser peer sync (WebRTC)](browser-peer-sync.md) + - [Commits (signed writes)](commits/intro.md) - [Concepts](commits/concepts.md) - [Compared to](commits/compare.md) - - [WebSockets](websockets.md) - [Endpoints](endpoints.md) - [Collections, filtering, sorting](schema/collections.md) - [Uploading and downloading files](files.md) diff --git a/docs/src/atomic-data-overview.md b/docs/src/atomic-data-overview.md index e899da5b5..91f2003da 100644 --- a/docs/src/atomic-data-overview.md +++ b/docs/src/atomic-data-overview.md @@ -7,10 +7,13 @@ Atomic Data uses links to connect pieces of data, and therefore makes it easier to connect datasets to each other - even when these datasets exist on separate machines. +It is also [local-first](local-first.md): your identity is a key you hold, your data lives on your own devices, every edit is signed by you, and devices [sync](sync.md) with each other whenever they can. A server is optional, useful as an always-on replica rather than as the place your data has to live. + ## AtomicServer [AtomicServer](atomic-server.md) is an open source, powerful graph database + headless CMS. It's the reference implementation for the Atomic Data specification, written in Rust. +The same library, `atomic_lib`, runs in the browser (as WASM), in [Flutter apps](flutter.md) and in the CLI, so every client keeps a full local copy of its data and does its own signing and syncing. ## Atomic Data Core @@ -31,17 +34,19 @@ These Properties are responsible for setting the `datatype` (to ensure type-safe ## Atomic Data Extended -Atomic Data Extended is a set of extra modules (on top of Atomic Data Core) that deal with data that changes over time, authentication, and authorization. +Atomic Data Extended is a set of extra modules (on top of Atomic Data Core) that deal with identity, data that changes over time, authentication, authorization and synchronization between devices. +If you are new here, read [URLs and identifiers](urls.md) first: it explains the `did:ad:` identifiers that show up everywhere in Extended. {{#include extended-table.md}} ## Tools & libraries -- Browser app [atomic-data-browser](https://github.com/atomicdata-dev/atomic-data-browser) ([demo on atomicdata.dev](https://atomicdata.dev)) -- Build a react app using [typescript & react libraries](https://github.com/atomicdata-dev/atomic-data-browser). Start with the [react template on codesandbox](https://codesandbox.io/s/atomic-data-react-template-4y9qu?file=/src/MyResource.tsx) -- Host your own [atomic-server](https://github.com/atomicdata-dev/atomic-server) (powers [atomicdata.dev](https://atomicdata.dev), run with `docker run -p 80:80 -v atomic-storage:/atomic-storage ghcr.io/ontola/atomic-server`) -- Discover the command line tool: [atomic-cli](https://github.com/atomicdata-dev/atomic-server) (`cargo install atomic-cli`) -- Use the Rust library: [atomic-lib](https://github.com/atomicdata-dev/atomic-server) +- The web app ([demo on atomicdata.dev](https://atomicdata.dev)): documents, tables, chat, files and an ontology editor, working offline in the browser +- Build a web app with [@tomic/lib](js.md), [@tomic/react](usecases/react.md) or [@tomic/svelte](svelte.md) +- Build a native app with [Flutter / Dart](flutter.md), on top of the same Rust core +- Host your own always-on [atomic-server](atomicserver/installation.md) (powers [atomicdata.dev](https://atomicdata.dev), run with `docker run -p 80:80 -v atomic-storage:/atomic-storage ghcr.io/ontola/atomic-server`), or use [Atomic Cloud](https://atomicserver.eu) +- Discover the command line tool: [atomic-cli](rust-cli.md) (`cargo install atomic-cli`) +- Use the Rust library: [atomic_lib](rust-lib.md) ## Get involved diff --git a/docs/src/atomic-server.md b/docs/src/atomic-server.md index 954f39976..8f07e0be2 100644 --- a/docs/src/atomic-server.md +++ b/docs/src/atomic-server.md @@ -5,6 +5,7 @@ It was developed parallel to this specification, and it served as a testing ground for various ideas (some of which didn't work, and some of which ended up in the spec). AtomicServer is a real-time headless CMS, graph database server for storing and sharing typed linked data. +In a [local-first](local-first.md) setup it is the always-on device: the replica that a browser tab can reach, that stays online while your phone and laptop sleep, and that serves the same data over HTTP. It's free, open source (MIT license), and has a ton of features: @@ -23,7 +24,7 @@ It's free, open source (MIT license), and has a ton of features: - πŸ’¬ **Group chat**, performant and flexible message channels with attachments, search and replies. - πŸ“‚ **File management**: Upload, download and preview attachments. - πŸ’Ύ **Versioning** / history from the Loro oplog, with writes authorized by [Atomic Commits](https://docs.atomicdata.dev/commits/intro.html) -- πŸ”„ **Real-time synchronization**: instantly communicates state changes with a client. Build dynamic, collaborative apps using [websockets](https://docs.atomicdata.dev/websockets) (using a [single one-liner in react](https://docs.atomicdata.dev/usecases/react) or [svelte](https://docs.atomicdata.dev/svelte)). +- πŸ”„ **Real-time synchronization**: instantly communicates state changes with a client. Build dynamic, collaborative apps using [Atomic Sync](https://docs.atomicdata.dev/sync) over WebSocket, Iroh or WebRTC (using a [single one-liner in react](https://docs.atomicdata.dev/usecases/react) or [svelte](https://docs.atomicdata.dev/svelte)). - 🧰 **Many serialization options**: to JSON, [JSON-AD](https://docs.atomicdata.dev/core/json-ad.html), and various Linked Data / RDF formats (RDF/XML, N-Triples / Turtle / JSON-LD). - πŸ“– **Pagination, sorting and filtering** queries using [Atomic Collections](https://docs.atomicdata.dev/schema/collections.html). - πŸ” **Authorization** (read / write permissions) and Hierarchical structures powered by [Atomic Hierarchy](https://docs.atomicdata.dev/hierarchy.html) diff --git a/docs/src/atomicserver/API.md b/docs/src/atomicserver/API.md index 36d2bd41d..2745796c7 100644 --- a/docs/src/atomicserver/API.md +++ b/docs/src/atomicserver/API.md @@ -24,11 +24,24 @@ curl -i -H "Accept: text/turtle" https://atomicdata.dev/properties/shortname ## Endpoints The various [Endpoints](../endpoints.md) in AtomicServer can be seen at `/endpoints` of your local instance. -These include functionality to create changes using `/commits`, query data using `/query`, get `/versions`, or do full-text search queries using `/search`. +These include functionality to post signed changes to `/commit`, query data using `/query`, get `/versions`, or do full-text search queries using `/search`. Typically, you pass query parameters to these endpoints to specify what you want to do. +## Fetching `did:ad:` resources over HTTP + +Most resources created since the move to [local-first](../local-first.md) are named by a [`did:ad:` identifier](../urls.md) rather than by an HTTP URL. +An AtomicServer that holds such a resource serves it at a `/did` endpoint, and also accepts the identifier as a path: + +```sh +curl -H "Accept: application/ad+json" "https://atomicdata.dev/did?subject=did%3Aad%3A{genesis}" +curl -H "Accept: application/ad+json" "https://atomicdata.dev/did:ad:{genesis}" +``` + +The response's `@id` is the DID, and a `Link: ; rel="canonical"` header names it too, so a client that arrived through this server's address can remember the location-independent identity. +Writes never go through a resource URL: they are signed commits, sent over the [sync](../sync.md) connection or to `/commit`. + ## Libraries or API? You can use the REST API if you want, but it's recommended to use one of our [libraries](../tooling.md). diff --git a/docs/src/browser-peer-sync.md b/docs/src/browser-peer-sync.md index c2bed18d2..8781cb7b2 100644 --- a/docs/src/browser-peer-sync.md +++ b/docs/src/browser-peer-sync.md @@ -1,4 +1,6 @@ -# Browser peer sync +# Browser peer sync (WebRTC) + +This is the browser-to-browser transport of [Atomic Sync](sync.md): the same signed commits and the same rights checks as over WebSocket or Iroh, carried over a WebRTC data channel because a browser tab cannot be dialed by a peer. The Sync page can connect up to eight browsers without an Atomic Cloud Server subscription. Create a peer link in the source drive's Sync page, share it with the other person, diff --git a/docs/src/commits/intro.md b/docs/src/commits/intro.md index 08abe4f3d..0c34e5eaf 100644 --- a/docs/src/commits/intro.md +++ b/docs/src/commits/intro.md @@ -33,7 +33,7 @@ See [concepts](concepts.md) for the field list. 3. It signs a commit containing `subject`, `signer`, `createdAt`, `loroUpdate`, and a signature. 4. The server verifies the signature and the signer's rights, imports the Loro bytes, and materializes properties. Ordinary content commits are not stored as resources; genesis and rights/parent/destroy are. -HTTP `POST /commit` remains the fallback. The WebSocket `COMMIT` frame is the live path. +HTTP `POST /commit` remains the fallback. The WebSocket `COMMIT` frame is the live path, and a paired peer accepts the same frame over Iroh. How commits travel between devices, and what happens to edits made offline, is the subject of [Atomic Sync](../sync.md). ## Motivation diff --git a/docs/src/core/concepts.md b/docs/src/core/concepts.md index faf251945..5da351b2b 100644 --- a/docs/src/core/concepts.md +++ b/docs/src/core/concepts.md @@ -112,7 +112,9 @@ It is the identifier that the rest of the Atom is providing information about. The Subject field is a URL that points to the Resource. The creator of the Subject MUST make sure that it resolves. In other words: following / downloading the Subject link will provide you with all the Atoms about the Subject (see [Querying Atomic Data](querying.md). -This also means that the creator of a Resource must make sure that it is available at its URL - probably by hosting the data, or by using some service that hosts it. +For an `https://` Subject this means hosting the data at that URL, or using some service that hosts it. +For a `did:ad:` Subject it means at least one device that holds the Resource is reachable, directly or through an always-on replica; the identifier itself does not depend on any host. +See [URLs and identifiers](../urls.md) for the different Subject shapes and how each resolves. In JSON-AD, the Subject is denoted by `@id`. ## Property field diff --git a/docs/src/did.md b/docs/src/did.md index 5a0680161..b59115114 100644 --- a/docs/src/did.md +++ b/docs/src/did.md @@ -1,10 +1,11 @@ {{#title Decentralized Identifiers (DIDs) in Atomic Data }} # Decentralized Identifiers -_status: work in progress_ +_status: implemented for Resources, Agents, Commits, blobs and nodes; mesh resolution is planned_ -Atomic Data is moving from HTTP URLs to Decentralized Identifiers (DIDs) as the primary way to address resources. -This makes resources portable, self-authenticating, and resolvable over both the internet and local mesh networks. +Atomic Data uses Decentralized Identifiers (DIDs) as the primary way to address resources, with HTTP URLs kept for vocabularies, external linked data and older servers. +This makes resources portable, self-authenticating, and resolvable over both the internet and, in the future, local mesh networks. +For an overview of every identifier shape and where you meet it, start with [URLs and identifiers](urls.md); this page is the specification. ## Design goals diff --git a/docs/src/extended-table.md b/docs/src/extended-table.md index 4a57309dc..b1734ad34 100644 --- a/docs/src/extended-table.md +++ b/docs/src/extended-table.md @@ -1,9 +1,10 @@ -- [Commits](commits/intro.md) communicate state changes. These Commits are signed using cryptographic keys, which ensures that every change can be audited. Commits are also used to construct a history of versions. -- [Agents](agents.md) are Users that enable [authentication](authentication.md). They are Resources with their own Public and Private keys, which they use to identify themselves. +- [Decentralized Identifiers](did.md): `did:ad:` identifiers name Resources, Agents, Commits, blobs and nodes independently of where they are hosted. See also [URLs and identifiers](urls.md). +- [Agents](agents.md) are Users that enable [authentication](authentication.md). An Agent is an Ed25519 keypair; its identifier is derived from the public key, so nobody has to issue it. +- [Hierarchies](hierarchy.md) used for authorization and keeping data organized. Similar to folder structures on file-systems. Rights are checked on every device and every transport. +- [Invites](invitations.md): create new users and provide them with rights. +- [Atomic Sync](sync.md): how devices reconcile their copies of a Drive, offline edits included, over WebSocket, Iroh or WebRTC. +- [Commits](commits/intro.md): the signed envelope around every write. They prove who changed what; current state and history live in each Resource's CRDT document. - [Collections](schema/collections.md): querying, filtering, sorting and pagination. - [Paths](core/paths.md): traverse graphs. -- [Hierarchies](hierarchy.md) used for authorization and keeping data organized. Similar to folder structures on file-systems. -- [Invites](invitations.md): create new users and provide them with rights. -- [WebSockets](websockets.md): real-time updates. - [Endpoints](endpoints.md): provide machine-readable descriptions of web services. - [Files](files.md): upload, download and metadata for files. diff --git a/docs/src/extended.md b/docs/src/extended.md index d77903883..625e212e2 100644 --- a/docs/src/extended.md +++ b/docs/src/extended.md @@ -7,4 +7,6 @@ The _Core_ of the specification (described in the previous chapter) is required However, many of the parts of Extended do depend on _eachother_. +Together they are what turns a data model into a [local-first](local-first.md) system: identity that lives in a key, writes that carry their own proof, and devices that reconcile whenever they meet. + {{#include extended-table.md}} diff --git a/docs/src/flutter.md b/docs/src/flutter.md new file mode 100644 index 000000000..46614151c --- /dev/null +++ b/docs/src/flutter.md @@ -0,0 +1,109 @@ +{{#title Flutter and Dart: native Atomic Data apps}} +# Flutter / Dart: native apps with `atomic_lib` + +The Rust library that powers AtomicServer, [`atomic_lib`](rust-lib.md), also compiles into Flutter apps. +A phone or tablet running your app gets the same local store, the same signing, the same CRDT merging and the same [peer sync](sync.md) as the desktop and the browser. +It is not a REST wrapper around a server: the app works with no server at all, and syncs to one, or to another phone, when you ask it to. + +The bridge is [`flutter_rust_bridge`](https://github.com/fzyzcjy/flutter_rust_bridge). +On Android and iOS it goes through `dart:ffi`; on the web the same Rust is compiled to WASM. + +## Status + +The Dart SDK lives in the repository under [`flutter/lib/atomic`](https://github.com/atomicdata-dev/atomic-server/tree/develop/flutter/lib/atomic), inside **Atomic Canvas**, a collaborative infinite drawing canvas that ships as the reference app. +It is not yet published on pub.dev. +The API is still moving, so expect to vendor it or depend on the git path for now. +The canvas-specific calls (strokes, folders, thumbnails) sit next to the general ones; only the general ones are described here. + +## What the SDK gives you + +The client is grouped by concern, and everything up to the last group is purely local: nothing touches the network until you call a sync method. + +| Group | Calls | What it does | +| --- | --- | --- | +| Database | `openDb(path)` | Open (or create) the local store on disk | +| Agent | `setup(name)`, `loadAgent(secret)`, `getActiveAgent()` | Create a keypair and a personal Drive, or restore one from a secret. Pure local. | +| Drive | `createDrive(name)`, `listDrives()`, `setActiveDrive(subject)` | Workspaces, addressed by `did:ad:` identifiers | +| Resource | `getProperty(subject, property)`, `setProperty(subject, property, value)` | Read and write; a write is signed and applied locally at once | +| History | `getResourceHistory(subject)`, `getResourceAtVersion(subject, version)` | Time travel over the Resource's Loro oplog, no network needed | +| Peer sync | `startPeer()`, `getPeerId()`, `peerSync(nodeId)`, `peerAnnounce(drive)`, `peerDiscoverSync(drive)` | Device-to-device over Iroh: pair by scanning a code, or discover a Drive on the DHT | +| Server sync | `openWsSync(serverUrl)`, `syncDriveToServer(serverUrl)` | Talk to an always-on device over WebSocket: subscribe to live updates, or push a Drive that was created on this phone so a browser can reach it | + +The distinction in the last two rows matters. +Connecting to a server *fetches* a Drive this device lacks and pushes commits made from then on. +A Drive created on the phone before it ever connected anywhere has to be offered explicitly with `syncDriveToServer`; that is the only way it reaches a browser tab, which cannot be paired with. + +## A minimal flow + +```dart +import 'package:your_app/atomic/atomic_client.dart'; + +// 1. Local store. Nothing here needs a network. +await AtomicClient.openDb('$appDir/atomic.redb'); + +// 2. Identity. Keep the secret; it is the account. +final setup = await AtomicClient.setup('Alice'); +// setup.agentSubject -> did:ad:agent:... +// setup.driveSubject -> did:ad:... (Alice's personal Drive) +// setup.agentSecret -> store this in secure storage + +// 3. Read and write. The write is signed and lands locally immediately. +await AtomicClient.setProperty( + setup.driveSubject, + 'https://atomicdata.dev/properties/description', + 'Written on a train', +); +final desc = await AtomicClient.getProperty( + setup.driveSubject, + 'https://atomicdata.dev/properties/description', +); + +// 4. Later, with a network: sync. +await AtomicClient.startPeer(); // this device becomes dialable +final myCode = await AtomicClient.getPeerId(); // show as a QR code +// ...or, on the other device, after scanning: +await AtomicClient.peerSync(scannedNodeId); +``` + +On the next launch, `loadAgent(secret)` restores the identity and the local store already holds the data. + +## Reacting to changes + +Sync applies remote commits to the local store in the background. +The app learns about them by polling the store's event stream: + +```dart +while (mounted) { + final event = await AtomicClient.pollDbEvent(timeoutMs: 60000); + if (event == null) continue; // timeout, nothing changed + // event['subject'] changed, was destroyed, or entered / left a watched query + refresh(event['subject'] as String); +} +``` + +`AtomicStore`, a `ChangeNotifier` in the same folder, wraps this into `watch(subject)` streams for widgets. + +## Signing HTTP requests from Dart + +Some endpoints on an always-on device, such as file download, check who is asking. +`atomic_auth.dart` signs those requests in pure Dart using the same scheme as `@tomic/lib` and `atomic_lib`: sign `" "` with the Agent key and send the signature, public key, timestamp and Agent identifier as headers. +See [Authentication](authentication.md#per-request-signing). + +## Running the reference app + +Atomic Canvas targets Android, iOS and web from one codebase. +The [`flutter/README.md`](https://github.com/atomicdata-dev/atomic-server/blob/develop/flutter/README.md) has the dev loop (`make phone`, `make tablet`, `make web`, hot reload from any terminal) and the pairing walkthrough: open Settings, show a code on one device, scan it on the other, and both devices exchange data regardless of who started. + +## Compared to the other clients + +| | Flutter (`atomic_lib`) | Browser (`@tomic/lib` + WASM) | Rust (`atomic_lib`) | +| --- | --- | --- | --- | +| Local store | redb on disk | OPFS, encrypted per Agent | redb | +| Can be dialed by a peer | Yes (Iroh) | No; reaches peers through a server or a WebRTC room | Yes (Iroh) | +| Talks to a server | WebSocket, HTTP fallback | WebSocket, HTTP fallback | WebSocket, HTTP | +| Same sync engine | Yes | Yes (WASM build) | Yes | + +## Where the vocabulary comes from + +The words on screen (workspace, device, pairing code, sync) are shared across the Flutter app, the web app and the desktop app on purpose, so a person meeting more than one of them does not have to learn each. +If you build your own UI on the SDK, borrowing them will make your app feel like part of the same system. diff --git a/docs/src/local-first.md b/docs/src/local-first.md new file mode 100644 index 000000000..eab6d4974 --- /dev/null +++ b/docs/src/local-first.md @@ -0,0 +1,59 @@ +{{#title Local-first: data ownership you can hold}} +# Local-first and data ownership + +Atomic Data was started to give people [control over their data](motivation.md#give-people-more-control-over-their-data). +For years the answer to "where does my data live?" was "on an AtomicServer you host". +That is real ownership, but it asks a lot: a domain, a machine that stays on, and an app that stops working the moment it cannot reach that machine. + +Since 2026 the answer is simpler. Your data lives **on your device**, and a server is one more device that happens to never sleep. + +## What local-first means here + +- **Your identity is a key, not an account.** An [Agent](agents.md) is an Ed25519 keypair. Its identifier, `did:ad:agent:{publicKey}`, is derived from the key, so nobody issues it and nobody can take it away. There is no sign-up form and no server that has to exist first. +- **Your data is a local database.** The web app keeps an encrypted store in the browser (WASM plus OPFS). The desktop and mobile apps keep one on disk (redb). Reading and writing never wait on a network. +- **Every edit is signed by you.** A change is a [Commit](commits/intro.md): a Loro CRDT delta plus your signature. Anyone can verify who made it without asking a server, and nobody can forge one in your name. +- **Resources have location-independent names.** A `did:ad:` identifier is derived from a signature, so the same Resource keeps the same name on your phone, your laptop and a replica in a data center. See [URLs and identifiers](urls.md). +- **Sync is optional and additive.** Devices that have been introduced to each other [reconcile](sync.md) whenever they can reach each other. Edits made apart merge without conflicts. Being offline is a state, not an error. + +## Where the server fits + +A server has not gone away; its role has changed. +An always-on device is useful for exactly the things a phone in a pocket cannot do: + +- Be reachable by a browser tab, which cannot accept connections itself. +- Stay online so collaborators can fetch a Drive while your devices sleep. +- Hold a backup that survives a lost phone. +- Serve HTTP, so the same data is fetchable by `curl`, a static site generator, or a search engine. + +None of that requires the server to be trusted with authorship. +Any node can replicate a Drive without holding the Drive owner's key, because every commit it forwards carries its own signature. +You can run that node yourself, use [Atomic Cloud](https://atomicserver.eu), or do both. + +## What this means for a person + +- A new device gets your identity from a secret or a passkey, and your data from any device that already has it. A secret restores *who you are*; a paired device or a server restores *what you have*. +- Losing a device does not mean losing an account: a WebAuthn passkey can wrap a backup of the agent secret. +- Signing out on a shared machine leaves the local cache encrypted under your key, unreadable to the next person. +- Two people editing the same document at once see each other's cursors and never a merge conflict. + +## What this means for a developer + +- **No backend to build.** Persistence, identity, authorization, history and sync come with the library. `@tomic/lib` in the browser and `atomic_lib` in Rust, Flutter and the CLI run the same store and the same sync engine. +- **Same code, every platform.** A [Flutter app](flutter.md) calls `atomic_lib` through `flutter_rust_bridge` and gets a local store, signing and peer sync, not a REST wrapper. +- **Schema still applies.** Local-first does not mean schemaless. Properties, Classes and datatypes are validated on every device, so data created offline is as well-formed as data created against a server. +- **History for free.** Every Resource's Loro oplog is its version history, so undo, time travel and audit views need no extra tables. + +## Trade-offs + +Local-first is not free, and these docs try not to pretend otherwise. + +- **Key management is on the client.** The private key has to be on the device that signs. Passkeys and encrypted caches make this safer; they do not make it trivial. +- **Availability follows the devices.** If every device holding a Drive is off, nobody else can read it until one returns. An always-on replica is the fix. +- **Two browser tabs on two machines cannot reach each other on their own.** A tab is not a peer. They need a server in common or a WebRTC room. +- **Rights are per Resource, checked everywhere.** That is a feature, but it means "I paired my phone" is not the same as "my phone can see everything". What crosses a link is what the connecting Agent may read. + +## Read on + +- [URLs and identifiers](urls.md), for the naming scheme that makes location-independence work. +- [Atomic Sync](sync.md), for how devices reconcile. +- [Decentralized Identifiers](did.md), for the derivation and resolution details. diff --git a/docs/src/motivation.md b/docs/src/motivation.md index 01805a0f7..f71479557 100644 --- a/docs/src/motivation.md +++ b/docs/src/motivation.md @@ -37,6 +37,10 @@ If we want to have that, we need to standardize the _read-write web_, which incl I want all people to have a (virtual) private server that contains their own data, that they control. This [Personal Data Store](usecases/personal-data-store.md) could very well be an old smartphone with a broken screen that is always on, running next to your router. +Since then the bar has moved: you should not even need that always-on box to own your data. +Today an Atomic identity is a keypair on your device, your data is a database on that same device, and a server is one more device that syncs with the others. +That is what we mean by [local-first](local-first.md), and it is the most direct form of ownership there is: the bytes are in your hand. + Atomic Data is designed to be a standard that achieves this. But we need more than a standard to get adoption - we need implementations. That's why I've been working on a server, various libraries, a GUI and [more](tooling.md) - all MIT licensed. @@ -86,6 +90,6 @@ A lot of time is essentially wasted on solving these issues time and time again. By having a more complete, strict standard, Atomic Data aims to decrease this burden. [Atomic Schema](schema/intro.md) enables developers to easily share their datamodels, and re-use those from others. -[Atomic Commits](commits/intro.md) helps developers to deal with versioning, history, undo and audit logs. +[Atomic Sync](sync.md) gives them a local store, offline editing and conflict-free collaboration, with versioning, history, undo and audit logs coming from the CRDT document underneath, and every write proven by a signed [Commit](commits/intro.md). [Atomic Hierarchies](hierarchy.md) provides an intuitive model for authorization and access control. And finally, the [existing open source Atomic Data software](tooling.md) (such as a server + database, a browser GUI, various libraries and React templates) help developers to have these features without having to do the heavy lifting themselves. diff --git a/docs/src/roadmap.md b/docs/src/roadmap.md index ba1323a84..6c76ac68a 100644 --- a/docs/src/roadmap.md +++ b/docs/src/roadmap.md @@ -28,7 +28,7 @@ This should help you understand how and where you may be able to contribute. - **[Endpoints](endpoints.md)** (2021-03). Machine readable API endpoints (think Swagger / OpenAPI spec) for things like versioning, path traversal and more. - **Classes and Properties editable from the browser** (2021-04). The data-browser is now powerful enough to use for managing the core ontological data of the project. - **[Hierarchies](hierarchy.md) & [Invitations](invitations.md)** (2021-06). Users can set rights, structure Resources and invite new people to collaborate. -- **[Websockets](websockets.md)** (2021-08). Live synchronization between client and server. +- **[Websockets](websockets.md)** (2021-08). Live synchronization between client and server; the start of what is now [Atomic Sync](sync.md). - **Use case: Document Editor** (2021-09). Notion-like editor with real-time synchronization. - **Full-text search** (2021-11). Powered by Tantivy. - **Authentication for read access** (2021-11). Allows for private data. diff --git a/docs/src/rust-lib.md b/docs/src/rust-lib.md index d1649a1ef..77a8f335c 100644 --- a/docs/src/rust-lib.md +++ b/docs/src/rust-lib.md @@ -1,14 +1,18 @@ {{#title atomic-lib: Rust library for Atomic Data}} # atomic-lib: Rust library for Atomic Data -Library that powers `atomic-server` and `atomic-cli`. Features: +Library that powers `atomic-server`, `atomic-cli`, the WASM build inside the web app and the [Flutter SDK](flutter.md). Features: -- An in-memory store +- A persistent local store (redb on native, OPFS in the browser through WASM) and an in-memory store - Parsing (JSON-AD) / Serialization (JSON-AD, JSON-LD, TTL, N-Triples) +- Agents, `did:ad:` identifiers and signing +- Loro CRDT documents per Resource, with history - Commit validation and processing +- The transport-agnostic [sync engine](sync.md), with WebSocket and Iroh peer transports behind feature flags +- Full-text search index - Constructing Collections - Path traversal -- Basic validation +- Schema validation [docs.rs](https://docs.rs/atomic_lib/latest/atomic_lib/) diff --git a/docs/src/sync.md b/docs/src/sync.md new file mode 100644 index 000000000..5d256c6ce --- /dev/null +++ b/docs/src/sync.md @@ -0,0 +1,71 @@ +{{#title Atomic Sync: keeping devices in agreement without a central server}} +# Atomic Sync + +Atomic Sync is how two copies of a Drive end up holding the same data. +It is one protocol with one set of rules, spoken over whichever transport connects the two devices: a WebSocket to an always-on device, an Iroh QUIC stream between two phones, or a WebRTC data channel between two browser tabs. +The transport changes; what crosses it, and who is allowed to send it, does not. + +This page explains the model. The [wire protocol reference](websockets.md) has every frame and byte; [browser peer sync](browser-peer-sync.md) covers the WebRTC path and its operator settings. + +## Why sync instead of request and response + +In the HTTP era a client asked a server for the current state, and the server was the only place that state lived. +Every edit was a round trip, and no connection meant no work. + +In the [local-first](local-first.md) model each device keeps a full copy of the Drives it uses. +Edits are applied to that local copy first, immediately, and are signed by the Agent making them. +Sync is what happens afterwards: the device tells its peers what it changed and asks what it missed. +Because every Resource is a [Loro CRDT](https://loro.dev) document, two devices that edited the same Resource while apart merge without conflicts and without a coordinator. + +The pieces: + +- **A signed [Commit](commits/intro.md)** is the unit of authority. It carries a Loro delta and an Ed25519 signature. Any device that receives one verifies the signature and the signer's rights before applying it. That check is the same code on a server, a phone and a browser tab. +- **A version vector** per Resource lets two devices find out what the other lacks by comparing summaries, not by re-sending everything. +- **An outbox** holds edits made while offline. It drains to the first reachable peer, in order, and survives a restart. +- **Rights decide, on every transport.** A device serves a peer only the Resources that peer's Agent may read, and accepts only writes that Agent may make. Pairing two devices introduces them; it grants nothing. + +## Transports + +| Transport | Between | When it is used | Live updates | +| --- | --- | --- | --- | +| WebSocket | A browser, phone or CLI and an always-on device (an AtomicServer) | Whenever the client knows the server's address. The default for the web app. | Yes: subscriptions push each applied commit | +| Iroh (QUIC, peer to peer) | Two devices running `atomic_lib`: phones, desktops, home servers | After scanning a pairing code. Works through NAT via a relay, direct when possible. | Yes, once both sides are in live mode | +| WebRTC | Up to eight browser tabs, with no server holding the data | From the Sync page in the web app. A signaling service introduces the tabs; data flows tab to tab. | Yes, plus presence and cursors | +| HTTP `POST /commit` | Any client and an AtomicServer | Fallback when no socket is open, and for scripts | No | + +A browser tab is the odd one out: it cannot accept incoming connections, so it is never a peer another device can dial. +It reaches other devices through an always-on device, or through the WebRTC room. +A phone or desktop running `atomic_lib` *is* a peer: it can be dialed, and it can serve. + +Discovery, the step of finding an address for a Drive you only know by its `did:ad:` identifier, is separate from transport. +A device can announce that it holds a Drive on the Mainline DHT, and a client can look that up; see [resolution](did.md#resolution). +A mesh transport over [Reticulum](https://reticulum.network/) is planned but not built. + +## What crosses the link + +Not everything a device holds travels, and not everything that travels is kept. + +- **Persisted state**: signed commits, and on trusted links the Loro snapshots they produce. This is what a peer stores, indexes and serves onward. +- **Ephemeral state**: edits in progress on a rich-text document, cursor positions, who is present in a Drive. Broadcast to whoever is connected right now, never written to disk, gone when the tab closes. +- **Blobs**: the bytes behind a [File](files.md), fetched by hash on demand rather than pushed with the Drive. +- **Deletes**: a signed destroy commit, so a peer can check that whoever deleted was allowed to. A bare "delete this" frame from a peer is ignored. + +## A day in the life of an edit + +1. You change a cell in a table on your laptop, on a train, with no connection. +2. The change is written to the laptop's local store and shows up in the UI at once. The laptop signs a commit for it and puts it in the outbox. +3. Back online, the laptop opens a WebSocket to your always-on device, proves who it is, and drains the outbox. The server verifies the signature and your write right, applies the Loro delta, and pushes the update to every other subscribed client. +4. Your phone, paired to the laptop directly, dials it over Iroh the next time both are awake. They compare version vectors, exchange only what differs, and end up identical. +5. Meanwhile a colleague edited the same table from a browser tab. Their commit and yours merge in the CRDT. Nobody sees a conflict dialog. + +## Where the code lives + +The sync engine is transport-agnostic and lives in `atomic_lib` (`lib/src/sync/`). +The server, the Flutter app and the WASM build in the browser all run that same engine, which is why a phone can act as a hub for another phone. +The TypeScript client in `@tomic/lib` mirrors the frame encoding for the WebSocket and WebRTC paths. + +## Up next + +- The full frame-by-frame [wire protocol](websockets.md), if you are implementing a client or a peer. +- [Browser peer sync](browser-peer-sync.md), for the WebRTC room and how to run the signaling endpoint yourself. +- [Decentralized Identifiers](did.md), for how a device finds a Drive it has never seen. diff --git a/docs/src/tooling.md b/docs/src/tooling.md index a628cde39..30c5449c2 100644 --- a/docs/src/tooling.md +++ b/docs/src/tooling.md @@ -9,6 +9,7 @@ Libraries and clients (all MIT licenced) that work great with [atomic-server](at - Svelte library: [@tomic/svelte](svelte.md) - Client CLI (rust): [atomic-cli](rust-cli.md) for fetching & editing data - Rust library: [atomic-lib](rust-lib.md) powers `atomic-server` and `atomic-cli`, and can be used in other Rust projects ([docs.rs](https://docs.rs/atomic_lib/latest/atomic_lib/)) +- Flutter / Dart: [native mobile and desktop apps](flutter.md) on top of `atomic_lib`, with a local store and peer sync - [Raycast Extension](https://www.raycast.com/atomicdata-dev/atomic-data-browser): full-text search ## Want to add to this list? Some ideas for tooling @@ -35,6 +36,6 @@ A mobile app for granting permissions to your data and signing things. See [gith - Would be useful for documentation, and as a default view for Atomic Data. - Use `@tomic/react` and `@tomic/lib` to get started -### Atomic-Dart + Flutter +### Atomic-Dart as a pub.dev package -Library + front-end app for browsing / manipulating Atomic Data on mobile devices. +The [Flutter SDK](flutter.md) exists but lives inside the reference app. Extracting it into a published Dart package would let other apps depend on it directly. diff --git a/docs/src/urls.md b/docs/src/urls.md new file mode 100644 index 000000000..e3c0c650c --- /dev/null +++ b/docs/src/urls.md @@ -0,0 +1,104 @@ +{{#title URLs and identifiers in Atomic Data}} +# URLs and identifiers + +Everything in Atomic Data is named by a URL. +A Resource has a Subject URL, every Property is a URL, and Values can be URLs that point to other Resources. +That is what makes the data a graph, and what makes datasets on different machines link to each other. + +You will meet a handful of URL shapes in these docs and in the software. +This page is the map. Each shape names a different kind of thing, and each resolves in a different way. + +## Two jobs a URL can do + +A URL can tell you **what** something is (its identity), or **where** to get it (its location), or both. + +- An `https://` URL bundles the two: the domain is the location, and the path is the name. If the server moves, the name breaks. +- A `did:ad:` identifier only carries identity. It is derived from a signature or a key, so it stays the same no matter which device or server holds the data. Location is added separately, as a routing hint, or discovered over the network. + +Atomic Data started out HTTP-only. Since the move to [local-first](local-first.md), `did:ad:` identifiers are the primary way to name Resources, Agents, Commits, blobs and nodes, and HTTP URLs remain for vocabularies, external linked data and older servers. +The [DID specification](did.md) has the full derivation rules and the resolution strategies; this page stays at the level of "which URL is which". + +## The identifier families + +| Shape | Names | Minted by | Resolves through | +| --- | --- | --- | --- | +| `did:ad:{genesis}` | A Resource inside a Drive (documents, tables, folders, the Drive itself) | The signature over its genesis certificate | Local store, a paired device, an always-on device, or peer discovery | +| `did:ad:agent:{publicKey}` | An [Agent](agents.md): a person, device or program that can sign | Generating an Ed25519 keypair | No network needed: the key is inside the identifier | +| `did:ad:commit:{signature}` | A signed write envelope, see [Commits](commits/intro.md) | The signature over the commit | Usually kept only as a receipt; not something you fetch | +| `did:ad:blob:{blake3}` | The raw bytes behind a [File](files.md) | The BLAKE3 hash of the bytes | Any device that holds the bytes; knowing the hash is the capability | +| `did:ad:node:{nodeId}` | A device or server as a network endpoint | The device's transport keypair (Iroh) | Not a Resource; used for pairing and routing only | +| `https://example.com/…` | Properties, Classes, Ontologies, external linked data, Resources on HTTP-era servers | The domain owner | An HTTP `GET` with an `application/ad+json` accept header | +| `internal:/path` | A Resource on *this* server, in config files and server logs | The server | Rewritten to the server's own origin before it leaves the server | + +Two things to notice in the table. + +First, only the `https://` family and `internal:` rows have a path. +`did:ad:` identifiers are flat: there is no `did:ad:abc/child`. +Structure comes from the [`parent` property](hierarchy.md), not from the URL. + +Second, the Agent row needs no lookup. +The public key is part of the identifier, so any device can verify a signature from that Agent offline. +That is what lets a signed edit made on a train be checked on a laptop that has never seen the network. + +## Routing hints + +A pure identity says nothing about where to find the data. +For `did:ad:` Resources, Commits and blobs, a client can append the Drive that is expected to hold it: + +```text +did:ad:{genesis}?drive=did:ad:{drive_genesis} +``` + +The hint is not part of the identity. +Two URLs that differ only in the `?drive=` hint name the same Resource, and stores strip the hint before using the identifier as a key. +It is a hint in the literal sense: it tells a client which Drive to look up on the network, and a client that already has the Resource locally ignores it. + +## HTTP forms of a DID + +A browser tab cannot dial a peer directly, so an always-on device (an AtomicServer) exposes DID Resources over HTTP as well. +Any of these fetch the same Resource: + +```text +GET https://example.com/did?subject=did:ad:{genesis} +GET https://example.com/did:ad:{genesis} +``` + +The response carries the canonical identity in a header, so a client that arrived through a location can switch to the identity: + +```http +Link: ; rel="canonical" +``` + +The same applies to blobs: `https://example.com/download/files/{blake3}` is the HTTP alias for `did:ad:blob:{blake3}`. +The alias is a convenience of one deployment. The DID is the name. + +## URLs you see in the app + +The web app puts the Subject in the query string, so a `did:ad:` identifier survives copy and paste without escaping: + +```text +https://example.com/app/show?subject=did:ad:{genesis} +``` + +Pairing codes use their own scheme. A QR code or deep link such as + +```text +atomic://pair?v=1&node=did:ad:node:{nodeId}&drives=* +``` + +carries the *route* to a device. It never contains an Agent secret, and scanning it grants nothing by itself: what crosses the link is decided by the rights on each Resource, on every transport. See [Atomic Sync](sync.md). + +## Encoding + +Keys and signatures inside `did:ad:` identifiers use URL-safe, unpadded base64 (`A–Z a–z 0–9 - _`), so an identifier can be dropped into a query string verbatim. +Blob hashes are hex. +Decoders also accept the older standard alphabet, so data written before this convention still resolves. +The [DID page](did.md#encoding) explains why. + +## Which one should I use? + +- **Naming a Property, Class or Ontology**: an `https://` URL. Vocabularies are meant to be public, stable and fetchable by anyone, and `atomicdata.dev` hosts the core ones. +- **Naming a Resource you create in a Drive**: let the library mint a `did:ad:` identifier. You never choose it, and it never changes. +- **Linking to a Resource from code or a document**: use the `did:ad:` identifier, optionally with a `?drive=` hint if the reader may not have it yet. +- **Sharing a link with a person**: the app's `/app/show?subject=` form, from a device they can reach. +- **Configuring a server**: `internal:/` for "this server's root", or a `did:ad:` for a specific Drive. Both work; see [installation](atomicserver/installation.md). diff --git a/docs/src/usecases/personal-data-store.md b/docs/src/usecases/personal-data-store.md index 1e8ddb91d..e1e921366 100644 --- a/docs/src/usecases/personal-data-store.md +++ b/docs/src/usecases/personal-data-store.md @@ -9,10 +9,13 @@ More and more of our personal information is stored by large corporations who us And as cloud consumers, we often don't have the luxury of moving our personal data to a place to where we want it to be. Many services don't even provide export functionality, and even if they do, the exports often lack information or are not interoperable with other apps. -Atomic Data could help to re-introduce data ownership. +Atomic Data re-introduces data ownership. Because the specification helps to standardize information, it becomes easier to make data interoperable. -And even more important: Apps don't need their own back-end - they can use the same personal data store: an Atomic Server (such as [this one](https://github.com/atomicdata-dev/atomic-serverob/master/server/README.md)). +And even more important: apps don't need their own back-end. They can use the same personal data store. -Realizing this goal requires quite a bit of work, though. -This specification needs to mature, and we need reliable implementations. -We also need proper tutorials, libraries and tools that convince developers to use atomic data to power their applications. +That store is no longer a server you have to run. +In the [local-first](../local-first.md) model your personal data store is the device in your hand: an encrypted database owned by your key, with every edit signed by you. +An always-on [AtomicServer](../atomic-server.md), self-hosted or on Atomic Cloud, is a replica of it that stays reachable while your devices sleep and that a browser can talk to. +Any number of apps, on any of your devices, read and write the same Drives and [sync](../sync.md) with each other. + +What still needs work is the developer side: tutorials, published SDKs for every platform, and the ecosystem of apps that makes a shared personal store worth having. diff --git a/docs/src/websockets.md b/docs/src/websockets.md index 0b8bccf51..cb529a522 100644 --- a/docs/src/websockets.md +++ b/docs/src/websockets.md @@ -4,7 +4,8 @@ One binary frame format, spoken over two transports: a browser-facing WebSocket and a node-to-node Iroh QUIC stream. This page is the canonical -wire-format reference. It describes what the code on this branch actually +wire-format reference; for the model behind it (what syncs, why, and which +transport is used when) read [Atomic Sync](sync.md) first. It describes what the code on this branch actually sends and accepts, including which side sends each frame and which responders have an arm for it. diff --git a/docs/src/when-to-use.md b/docs/src/when-to-use.md index 59e841042..7df3ec109 100644 --- a/docs/src/when-to-use.md +++ b/docs/src/when-to-use.md @@ -7,7 +7,8 @@ - **Open data**. Atomic Data is a bit harder to create than plain JSON, for example, but it is easier to re-use and understand. It's use of URLs for properties makes data self-documenting. - **High interoperability requirements**. When multiple groups of people have to use the same schema, Atomic Data provides easy ways to constrain and validate the data and ensure type safety. - **Connected / decentralized data**. With Atomic Data, you use URLs to point to things on other computers. This makes it possible to connect datasets very explicitly, without creating copies. Very useful for decentralized social networks, for example. -- **Auditability & Versioning**. Using Atomic Commits, we can store all changes to data as transactions that can be replayed. This creates a complete audit log and history. +- **Auditability & Versioning**. Every Resource is a CRDT document with a full oplog, and every write is a signed Commit. That gives you history, time travel and an attributable audit trail without extra tables. +- **Offline and multi-device**. Data lives on each device and [syncs](sync.md) when devices meet, so an app keeps working on a train and merges cleanly afterwards. - **JSON or RDF as Output**. Atomic Data serializes to idiomatic, clean JSON as well as various RDF formats (Turtle / JSON-LD / n-triples / RDF/XML). ## When not to use Atomic Data From 803aeafc6ae97a00a899b57bddfeae2d690e8393 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 18 Sep 2026 21:01:18 +0000 Subject: [PATCH 002/110] docs: local-first guide, server-optional Store docs, always-on framing Second pass on the docs refresh. - New end-to-end guide "Build a local-first app with @tomic/lib": identity from a keypair, Store and Drive with no server, OPFS persistence via the WASM client database, promoting the Drive to an always-on device, and pairing a phone through the Flutter SDK. Every call is one the library exposes today; where a step still needs a checkout (the WASM build) the guide says so. - AtomicServer chapter opens on "the always-on device" instead of on the feature list. - js.md and js-lib/store.md: serverUrl is optional, createDrive with localOnly, promoteLocalDrive, sync status, and did:ad subjects in getResource; save() no longer described as "commit to the server". - Solid comparison: local-first device store vs POD, DIDs vs WebID, CRDT sync vs document writes. - Astro guide: note that it is the HTTP path and where the local-first guide fits. - planning/dart-sdk-package.md: the plan for extracting the Dart SDK into a pub.dev package, referenced from flutter.md, tooling.md and the roadmap. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_018ZmptEWmdz92GXTuP77vrb --- docs/src/SUMMARY.md | 6 ++ docs/src/astro-guide/1-index.md | 4 + docs/src/astro-guide/2-setup.md | 1 + docs/src/atomic-server.md | 26 ++++-- docs/src/flutter.md | 2 +- docs/src/interoperability/solid.md | 4 +- docs/src/js-lib/store.md | 27 ++++-- docs/src/js.md | 16 ++-- docs/src/local-first-guide/1-index.md | 38 +++++++++ docs/src/local-first-guide/2-identity.md | 44 ++++++++++ .../local-first-guide/3-store-and-drive.md | 84 +++++++++++++++++++ docs/src/local-first-guide/4-persist.md | 62 ++++++++++++++ docs/src/local-first-guide/5-sync-server.md | 62 ++++++++++++++ docs/src/local-first-guide/6-phone.md | 64 ++++++++++++++ docs/src/roadmap.md | 1 + docs/src/tooling.md | 2 +- planning/dart-sdk-package.md | 77 +++++++++++++++++ 17 files changed, 501 insertions(+), 19 deletions(-) create mode 100644 docs/src/local-first-guide/1-index.md create mode 100644 docs/src/local-first-guide/2-identity.md create mode 100644 docs/src/local-first-guide/3-store-and-drive.md create mode 100644 docs/src/local-first-guide/4-persist.md create mode 100644 docs/src/local-first-guide/5-sync-server.md create mode 100644 docs/src/local-first-guide/6-phone.md create mode 100644 planning/dart-sdk-package.md diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 5f14b9fda..e43977638 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -52,6 +52,12 @@ # Guides +- [Build a local-first app with @tomic/lib](local-first-guide/1-index.md) + - [Step 1: an identity that is a key](local-first-guide/2-identity.md) + - [Step 2: a Store and a Drive, no server](local-first-guide/3-store-and-drive.md) + - [Step 3: persist across reloads](local-first-guide/4-persist.md) + - [Step 4: sync to an always-on device](local-first-guide/5-sync-server.md) + - [Step 5: the same Drive on a phone](local-first-guide/6-phone.md) - [Build a portfolio using Astro and Atomic Server](astro-guide/1-index.md) - [Setup](astro-guide/2-setup.md) - [Frontend setup](astro-guide/3-frontend-setup.md) diff --git a/docs/src/astro-guide/1-index.md b/docs/src/astro-guide/1-index.md index d31b94e28..9ee9243da 100644 --- a/docs/src/astro-guide/1-index.md +++ b/docs/src/astro-guide/1-index.md @@ -13,4 +13,8 @@ Feel free to spice it up and add some styling while following along though. I will also not cover every little detail about Astro, only what is necessary to follow along with this guide. If you're completely new to Astro consider skimming the [documentation](https://docs.astro.build/en/getting-started/) to see what it has to offer. +> [!NOTE] +> This guide uses AtomicServer as an HTTP data source: the site is built from data on a server you host, and the resources are addressed by that server's URLs. That is the right shape for a public website rendered at build time, and it still works as described. +> If you are building an app that people use on their own devices, start with the [local-first guide](../local-first-guide/1-index.md) instead, where a server is optional and added last. + With all that out of the way let's start by setting up your atomic data server. If you already have a server running skip to [Creating the frontend](3-frontend-setup.md) diff --git a/docs/src/astro-guide/2-setup.md b/docs/src/astro-guide/2-setup.md index 2ccc02e4b..bf9be7d6d 100644 --- a/docs/src/astro-guide/2-setup.md +++ b/docs/src/astro-guide/2-setup.md @@ -1,6 +1,7 @@ # Setup We recommend you set up an AtomicServer on a networked machine like a VPS and couple it to a domain name you intend to use for the final product as this can be difficult to change later. +The server is the always-on device your site reads from at build time; the data itself can be edited from the web app on any device and [synced](../sync.md) to it. For example, if you intend to build a portfolio website with the domain `my-portfolio.com` you could couple your atomic server to `atomic.my-portfolio.com`. diff --git a/docs/src/atomic-server.md b/docs/src/atomic-server.md index 8f07e0be2..e8435958e 100644 --- a/docs/src/atomic-server.md +++ b/docs/src/atomic-server.md @@ -1,11 +1,27 @@ -{{#title AtomicServer: An open-source, realtime, headless CMS}} -# AtomicServer and its features +{{#title AtomicServer: the always-on device for your Atomic Data}} +# AtomicServer -[`AtomicServer`](https://github.com/atomicdata-dev/atomic-server/blob/master/server/README.md) is the _reference implementation_ of the Atomic Data Core + Extended specification. +[`AtomicServer`](https://github.com/atomicdata-dev/atomic-server) is the _reference implementation_ of the Atomic Data Core + Extended specification: a single binary that is a graph database, a real-time headless CMS and a web app. It was developed parallel to this specification, and it served as a testing ground for various ideas (some of which didn't work, and some of which ended up in the spec). -AtomicServer is a real-time headless CMS, graph database server for storing and sharing typed linked data. -In a [local-first](local-first.md) setup it is the always-on device: the replica that a browser tab can reach, that stays online while your phone and laptop sleep, and that serves the same data over HTTP. +## The always-on device + +In a [local-first](local-first.md) system every device holds its own copy of the data and does its own signing. +What a phone or a browser tab cannot do is stay reachable. +That is AtomicServer's job: it is the device that never sleeps. + +- A **browser tab** cannot accept connections, so it needs a server to reach any other device. AtomicServer is what the web app talks to. +- **Collaborators** fetch a Drive from it while your own devices are off. +- It holds a **backup** that survives a lost phone, without holding your key: every commit it stores is signed by the Agent that made it, so the server can replicate but never author. +- It serves the same data over **HTTP**, for `curl`, static site generators and search engines, next to the sync connection the apps use. + +You can run one on a laptop, a Raspberry Pi, a VPS or a home NAS. [Atomic Cloud](https://atomicserver.eu) runs one for you. +The same `atomic_lib` also runs inside the browser and in [Flutter apps](flutter.md), which is why a phone can act as an always-on device for another phone, but a server is the shape most people want for the role. + +If you are here to build an app, the [local-first guide](local-first-guide/1-index.md) starts without a server and adds one in step 4; the rest of this chapter is about running and using the server itself. + +## Features + It's free, open source (MIT license), and has a ton of features: diff --git a/docs/src/flutter.md b/docs/src/flutter.md index 46614151c..bc1c6d6d5 100644 --- a/docs/src/flutter.md +++ b/docs/src/flutter.md @@ -12,7 +12,7 @@ On Android and iOS it goes through `dart:ffi`; on the web the same Rust is compi The Dart SDK lives in the repository under [`flutter/lib/atomic`](https://github.com/atomicdata-dev/atomic-server/tree/develop/flutter/lib/atomic), inside **Atomic Canvas**, a collaborative infinite drawing canvas that ships as the reference app. It is not yet published on pub.dev. -The API is still moving, so expect to vendor it or depend on the git path for now. +The API is still moving, so expect to vendor it or depend on the git path for now; the extraction into a package is planned in `planning/dart-sdk-package.md`. The canvas-specific calls (strokes, folders, thumbnails) sit next to the general ones; only the general ones are described here. ## What the SDK gives you diff --git a/docs/src/interoperability/solid.md b/docs/src/interoperability/solid.md index f9bb55a9c..b056b47e2 100644 --- a/docs/src/interoperability/solid.md +++ b/docs/src/interoperability/solid.md @@ -10,7 +10,7 @@ In many ways, it has **similar goals** to Atomic Data: Technically, both are also similar: -- Usage of **personal servers**, or PODs (Personal Online Datastores). Both Atomic Data and Solid aim to provide users with a highly personal server where all sorts of data can be stored. +- Usage of **personal data stores**. Solid gives every user a POD (Personal Online Datastore), a server they control. Atomic Data started from the same idea and has since moved the store onto the device itself: data lives [local-first](../local-first.md) on your phone or laptop, and a personal server is one replica among your devices rather than the place the data has to be. - Usage of **linked data**. All Atomic Data is valid RDF, which means that **all Atomic Data is compatible with Solid**. However, the other way around is more difficult. In other words, if you choose to use Atomic Data, you can always put it in your Solid Pod. But there are some important **differences**, too, which will be explained in more detail below. @@ -19,6 +19,8 @@ But there are some important **differences**, too, which will be explained in mo - Atomic Data standardizes state changes (which also provides version control / history, audit trails) - Atomic Data is more easily serializable to other formats (like JSON) - Atomic Data has different models for authentication, authorization and hierarchies +- Atomic Data identifies users and resources by `did:ad:` identifiers derived from keys and signatures, where Solid uses WebIDs (HTTP URLs) that depend on a host +- Atomic Data syncs signed CRDT updates between devices; Solid reads and writes documents on a server - Atomic Data does not depend on existing semantic web specifications - Atomic Data is a smaller and younger project, and as of now a one-man show diff --git a/docs/src/js-lib/store.md b/docs/src/js-lib/store.md index 50a1840ce..0893737b0 100644 --- a/docs/src/js-lib/store.md +++ b/docs/src/js-lib/store.md @@ -14,8 +14,9 @@ It takes an object with the following options | Name | Type | Description | |-----------|---------------------|------------------------------------------------------------------------------------------------------------------------------------------| -| serverUrl | string | URL of your atomic server | +| serverUrl | string | **(optional)** URL of the always-on device to sync with. Without it the store works locally; set one later with `setServerUrl()` | | agent | [Agent](./agent.md) | **(optional)** The agent the store should use to fetch resources and to sign commits when editing resources, defaults to a public agent | +| connect | boolean | **(optional)** Open a WebSocket to `serverUrl` right away. Defaults to `true` | ```typescript const store = new Store({ @@ -27,6 +28,21 @@ const store = new Store({ > [!NOTE] > You can always change or set both the serverUrl and agent at a later time using `store.setServerUrl()` and `store.setAgent()` respectively. +### A store without a server + +A store with an Agent and no `serverUrl` is a complete [local-first](../local-first.md) client: it mints `did:ad:` identifiers, creates Drives, signs edits and keeps them in an outbox. + +```typescript +const store = new Store({ agent }); + +const drive = await store.createDrive('Notes', { localOnly: true }); +store.setDrive(drive.subject); +``` + +`localOnly` keeps the sync engine from offering the Drive to a server you connect later; lift it with `store.promoteLocalDrive(drive.subject)` once you want it replicated. +To keep data across reloads attach the WASM client database with `store.setClientDb()`; the [local-first guide](../local-first-guide/4-persist.md) shows the setup. +Sync status (connected, pending edits, last drive sync) is on `store.getSyncStatus()` and the `StoreEvents.SyncStatusChanged` event. + ### One vs Many Stores Generally in a client application with one authenticated user, you'll want to have a single instance of a `Store` that is shared throughout the app. @@ -45,12 +61,13 @@ const resource = await store.getResource('https://my-resource-subject'); ``` `getResource` takes the [subject](../core/concepts.md#subject-field) of the resource as a parameter and returns a promise that resolves to the requested resource. -The store will cache the resource in memory and subscribe to the server for changes to the resource, subsequent requests for the resource will not fetch over the network but return the cached version. +The subject can be an `https://` URL or a `did:ad:` identifier; see [URLs and identifiers](../urls.md). +The store answers from memory, then from the local database if one is attached, and only then fetches from the server. It subscribes to changes, so subsequent requests return the cached version and updates arrive through the sync connection. ## Subscribe to changes Atomic makes it easy to build real-time applications. -When you subscribe to a subject you get notified every time the resource changes on the server. +When you subscribe to a subject you get notified every time the resource changes: by your own code, or by another device whose edit arrived through sync. ```typescript store.subscribe('https://my-resource-subject', myResource => { @@ -92,8 +109,8 @@ It takes an options object with the following properties: | Name | Type | Description | |----------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------| -| subject | string | **(optional)** The subject the new resource should have, by default a random subject is generated | -| parent | string | **(optional)** The parent of the new resource, defaults to the store's `serverUrl` | +| subject | string | **(optional)** The subject the new resource should have. By default a `did:ad:` identifier is minted from a genesis certificate signed by the store's Agent | +| parent | string | **(optional)** The parent of the new resource, defaults to the store's `serverUrl`. Pass the Drive or folder it belongs in | | isA | string \| string[] | **(optional)** The 'type' of the resource. determines what class it is. Supports multiple classes. | | propVals | Record | **(optional)** Any additional properties you want to set on the resource. Should be an object with subjects of properties as keys | diff --git a/docs/src/js.md b/docs/src/js.md index ada2c980a..807cf549a 100644 --- a/docs/src/js.md +++ b/docs/src/js.md @@ -2,9 +2,11 @@ # @tomic/lib: The Atomic Data library for typescript/javascript -Core typescript library for fetching data, handling JSON-AD parsing, storing data, signing Commits, setting up WebSockets and full-text search and more. +Core typescript library for creating and editing data locally, signing Commits, [syncing](sync.md) with servers and peers, handling JSON-AD parsing, full-text search and more. Runs in most common JS contexts like the browser, node, deno, bun etc. +A server is optional: a `Store` with only an Agent can mint identifiers, create Drives and save signed edits on its own, and connect to a server later. +The [local-first guide](local-first-guide/1-index.md) walks through that path end to end; the snippets below show the shortest route when you already have a server. ## Installation @@ -24,7 +26,7 @@ import { Store, Agent, core } from '@tomic/lib'; const store = new Store({ // You can create a secret from the `User settings` page using the AtomicServer UI agent: Agent.fromSecret('my-secret-key'), - // Set a default server URL + // Optional. Leave it out to work locally and call store.setServerUrl() later. serverUrl: 'https://my-atomic-server.dev', }); ``` @@ -46,7 +48,8 @@ const job = resource.get(myOntology.properties.job); ```ts resource.set(core.properties.description, 'Hello World'); -// Commit the changes to the server. +// Sign the change and apply it locally. If a server is connected it is sent +// right away; otherwise it waits in the outbox until one is. await resource.save(); ``` @@ -60,16 +63,17 @@ const newResource = await store.newResource({ }, }); -// Commit the new resource to the server. +// Sign the genesis and save. The subject is a did:ad: identifier minted here. await newResource.save(); ``` ### Subscribing to changes ```ts -// --------- Subscribe to changes (using websockets) --------- +// --------- Subscribe to changes --------- const unsub = store.subscribe('https://my-atomic-server.dev/some-resource', resource => { - // This callback is called each time a change is made to the resource on the server. + // Called for every change: your own local edits, and edits arriving from + // other devices over the sync connection. // Do something with the changed resource... }); ``` diff --git a/docs/src/local-first-guide/1-index.md b/docs/src/local-first-guide/1-index.md new file mode 100644 index 000000000..d51ce5c9f --- /dev/null +++ b/docs/src/local-first-guide/1-index.md @@ -0,0 +1,38 @@ +{{#title Build a local-first app with @tomic/lib}} +# Build a local-first app with `@tomic/lib` + +This guide walks through an app that works with no server at all, and then grows: first it persists across reloads, then it syncs to an always-on device, then it reaches a phone. +Every step is a small change to the same code. +It uses `@tomic/lib` directly, so what you learn applies to React, Svelte, Astro or plain TypeScript. + +The app is a reading list: a Drive that holds Book resources with a title and a note. +It is deliberately tiny; the interesting part is where the data lives. + +## What you will end up with + +1. An identity that is a keypair, minted in the browser, with nothing registered anywhere. +2. A Drive and some Books, created and edited offline and signed by that identity. +3. Persistence across reloads, using the encrypted local database. +4. Sync to an always-on AtomicServer, so a second browser and a colleague can see the list. +5. The same Drive on a phone, paired directly. + +## Prerequisites + +- Node 20 or later and a package manager. +- A bundler that serves static files, such as Vite. The examples assume it. +- For step 4, an AtomicServer you can reach: `atomic-server` [installed locally](../atomicserver/installation.md) is enough. +- For step 5, the [Flutter reference app](../flutter.md) on a phone. + +Install the library: + +```sh +npm install @tomic/lib +``` + +## How this differs from the Astro guide + +The [Astro guide](../astro-guide/1-index.md) starts from a server: you set one up, model data in its web app, and the site reads from it over HTTP. +That is still the right shape for a public website built at deploy time. +This guide starts from the device and treats the server as something you add later, which is the right shape for an app a person uses. + +Next: [minting an identity](2-identity.md). diff --git a/docs/src/local-first-guide/2-identity.md b/docs/src/local-first-guide/2-identity.md new file mode 100644 index 000000000..1be734cc2 --- /dev/null +++ b/docs/src/local-first-guide/2-identity.md @@ -0,0 +1,44 @@ +# Step 1: an identity that is a key + +In the HTTP era the first thing an app did was register a user on a server. +Here the first thing it does is generate a keypair. +That keypair *is* the [Agent](../agents.md): its identifier is derived from the public key, so it exists the moment it is generated and no server has to know about it. + +```ts +import { Agent, JSCryptoProvider } from '@tomic/lib'; + +const SECRET_KEY = 'reading-list.agent-secret'; + +async function loadOrCreateAgent(): Promise { + const stored = localStorage.getItem(SECRET_KEY); + + if (stored) { + // 'js' picks the pure-JS Ed25519 implementation. It signs deterministically, + // which the personal-drive derivation in the next step relies on. + return Agent.fromSecret(stored, 'js'); + } + + const keys = await Agent.generateKeyPair(); + const subject = `did:ad:agent:${keys.publicKey}`; + + // One string that encodes the private key and the subject. This is the + // account: whoever holds it, is this Agent. + const secret = Agent.buildSecret(keys.privateKey, subject); + localStorage.setItem(SECRET_KEY, secret); + + return new Agent(new JSCryptoProvider(keys.privateKey), subject); +} + +const agent = await loadOrCreateAgent(); +console.log(agent.subject); // did:ad:agent:… +``` + +Three things to notice. + +**Nothing left the device.** No request was made. The identifier is valid everywhere already, because anyone who later receives a signature from this Agent can verify it with the public key that is inside the identifier. See [URLs and identifiers](../urls.md). + +**The secret is the account.** `localStorage` is fine for a tutorial. A real app keeps it somewhere better: the web app wraps it with a passkey, and native apps use the platform keystore. Losing the secret without a backup means losing the identity, which is the trade-off [local-first](../local-first.md) makes explicit. + +**There is no password.** Signing in on a new device means pasting the secret, scanning it, or restoring it from a passkey. It restores *who you are*. Getting *what you have* onto that device is the job of sync, in step 4. + +Next: [a Store and a Drive, with no server](3-store-and-drive.md). diff --git a/docs/src/local-first-guide/3-store-and-drive.md b/docs/src/local-first-guide/3-store-and-drive.md new file mode 100644 index 000000000..5df8abe2f --- /dev/null +++ b/docs/src/local-first-guide/3-store-and-drive.md @@ -0,0 +1,84 @@ +# Step 2: a Store and a Drive, with no server + +A `Store` is the in-memory graph your app reads and writes. +It used to need a `serverUrl`; now that is optional. +Give it the Agent from step 1 and it can create, sign and edit data on its own. + +```ts +import { Store, core } from '@tomic/lib'; + +const store = new Store({ agent }); +``` + +## Create a Drive + +A [Drive](../hierarchy.md) is the top-level container: the unit that gets shared, synced and backed up. +Every Agent has a *personal* Drive whose identifier is derived from the Agent's key, so the same identity gets the same Drive on every device without anything to look up. + +```ts +const drive = await store.createDrive('Reading list', { localOnly: true }); +store.setDrive(drive.subject); + +console.log(drive.subject); // did:ad:… +``` + +`localOnly: true` tells the sync engine to leave this Drive alone even if a server is connected later. +You will lift that in step 4. +Without it, the Store would try to offer the Drive to whatever server it connects to, which is the right default for an app that always has one. + +## Create and edit Resources + +A Book is a Resource with a name and a description. +This guide reuses two Properties from the core vocabulary so it needs no schema of its own; a real app defines its own Classes and Properties in the [ontology editor](../atomicserver/gui.md) and generates types with [`@tomic/cli`](../js-cli.md). + +```ts +async function addBook(title: string, note: string) { + const book = await store.newResource({ + parent: drive.subject, + propVals: { + [core.properties.name]: title, + [core.properties.description]: note, + }, + }); + + await book.save(); + + return book; +} + +const book = await addBook('The Dispossessed', 'Recommended by Anna'); +console.log(book.subject); // did:ad:… (a fresh identifier, minted here) +``` + +`newResource` mints a `did:ad:` identifier from a genesis certificate signed by your Agent. +`save()` writes to the Resource's Loro document, marks it dirty in the outbox, and applies it locally at once. +There is no server to wait for, so the Promise resolves as soon as the local write is done. + +Editing is the same call: + +```ts +book.set(core.properties.description, 'Recommended by Anna. Finished it in a weekend.'); +await book.save(); +``` + +## Read and subscribe + +```ts +const same = await store.getResource(book.subject); +console.log(same.get(core.properties.name)); // The Dispossessed + +const unsubscribe = store.subscribe(book.subject, updated => { + render(updated.get(core.properties.description)); +}); +``` + +`subscribe` fires for local edits now, and for edits arriving from other devices once you sync. +The callback does not care which. + +## What you have + +An identity, a Drive and a Book, all signed, all created without a network request. +Reload the page, though, and the Store starts empty again: this step keeps everything in memory. +Fixing that is the next step. + +Next: [persisting across reloads](4-persist.md). diff --git a/docs/src/local-first-guide/4-persist.md b/docs/src/local-first-guide/4-persist.md new file mode 100644 index 000000000..ecd6ca0cd --- /dev/null +++ b/docs/src/local-first-guide/4-persist.md @@ -0,0 +1,62 @@ +# Step 3: persist across reloads + +The Store keeps resources in memory. +Durable storage in the browser comes from the **client database**: the same Rust store that powers AtomicServer, compiled to WASM, writing an encrypted file to the Origin Private File System (OPFS). +It runs in a dedicated worker; one tab per origin owns the file and the others talk to it over a `BroadcastChannel`, so several tabs share one database. + +## Where the WASM comes from + +The worker script ships inside `@tomic/lib`. +The WASM module does not, yet: it is built from the repository's `wasm/` crate and served by your app as static files. +Build it once and copy the two output files into your public folder: + +```sh +# in a checkout of atomic-server +cd wasm +wasm-pack build --target web --out-dir pkg +cp pkg/atomic_wasm.js pkg/atomic_wasm_bg.wasm /path/to/your-app/public/wasm/ +``` + +Publishing this as a package so the step becomes an `npm install` is on the [roadmap](../roadmap.md). + +## Attach the database to the Store + +```ts +import { ClientDbWorker, Store } from '@tomic/lib'; +import workerUrl from '@tomic/lib/client-db.worker.js?url'; // Vite syntax + +const clientDb = new ClientDbWorker( + `${location.origin}/wasm/atomic_wasm.js`, + workerUrl, + { + // One file per Agent, so two identities on the same origin never share a + // cache. Derive the name from the subject however you like. + dbName: `reading-list.${hash(agent.subject)}.redb`, + // 32 random bytes. Keep them with the secret; without the key the file is + // unreadable, which is the point. + dbKey: loadOrCreateDbKey(agent.subject), + }, +); + +const store = new Store({ agent }); +store.setClientDb(clientDb); +await clientDb.init(); +``` + +The order matters: attach first, then initialize. +Reads that happen while the worker is still opening the file wait for it instead of failing. + +## What changes + +- **Reads hit the database first.** `getResource` returns the stored copy without a network request, and only fetches when the resource is not present locally. +- **Writes are persisted with their history.** Each Resource's Loro snapshot is stored next to its properties, so version history survives a reload and offline edits made before a reload are still in the outbox afterwards. +- **Queries run locally.** `store.queryLocalDb` and `store.search` work against the local database, with the same index the server uses. + +Reload the page: the Drive and the Book are still there, and the outbox still knows the Book has never been sent anywhere. + +## What you have + +A single-device app that survives reloads, with an encrypted local database and version history, and still no server. +The next step adds one, so the reading list can reach a second browser and a colleague. + +Next: [syncing to an always-on device](5-sync-server.md). diff --git a/docs/src/local-first-guide/5-sync-server.md b/docs/src/local-first-guide/5-sync-server.md new file mode 100644 index 000000000..3e53b7cc2 --- /dev/null +++ b/docs/src/local-first-guide/5-sync-server.md @@ -0,0 +1,62 @@ +# Step 4: sync to an always-on device + +So far the reading list exists on one device. +To reach a second browser, a colleague, or a backup, it needs a copy on something that stays online: an [AtomicServer](../atomic-server.md). +The server does not become the owner of the data. It becomes one more replica, one that can be dialed by a browser tab. + +## Connect + +Point the Store at the server. It opens a WebSocket, authenticates with your Agent, and starts the [sync](../sync.md) engine. + +```ts +import { StoreEvents } from '@tomic/lib'; + +store.setServerUrl('http://localhost:9883'); + +store.on(StoreEvents.SyncStatusChanged, status => { + console.log(status.serverConnected, status.pendingDirtyCount); +}); +``` + +Nothing is pushed yet. +The Drive was created with `localOnly: true`, so the engine skips it. +Connecting to a server *fetches* Drives this device lacks; it never offers the ones it has. +That is deliberate: a workspace should not leave a device because that device happened to reach a server. + +## Offer the Drive + +Lifting the local-only flag runs a normal reconcile against the server, which pushes every Resource in the Drive as signed commits: + +```ts +await store.promoteLocalDrive(drive.subject); +``` + +The server verifies each signature and each write right. +A self-hosted server accepts a new Drive from any Agent. +A managed node accepts it only if the Drive is enrolled, which is handled by the hosting layer, not by this call. +If the server refuses, the Drive stays local and the error is on the sync status. + +From this point on, `save()` still applies locally first, and the outbox drains to the server whenever the socket is open. +Close the laptop, edit on the train, open it again: the outbox catches up. + +## Read it from somewhere else + +Open the server's web app in another browser, sign in with the same secret, and the Drive is there. +Or fetch it with `curl`, since the server also serves DIDs over HTTP: + +```sh +curl -H "Accept: application/ad+json" \ + "http://localhost:9883/did?subject=$(printf %s "$DRIVE_DID" | jq -sRr @uri)" +``` + +## Share it + +Give a colleague read or write access by adding their Agent to the Drive's `read` or `write` list, or send them an [invite](../invitations.md). +Their edits arrive through the same WebSocket subscription your `store.subscribe` callback already listens to, and merge in the CRDT with yours. + +## What you have + +A Drive that lives on your device and on a server, editable offline, visible to a second browser, shareable. +Everything the server holds is signed by you, so any other node, self-hosted or not, could take its place. + +Next: [the same Drive on a phone](6-phone.md). diff --git a/docs/src/local-first-guide/6-phone.md b/docs/src/local-first-guide/6-phone.md new file mode 100644 index 000000000..6aa78376f --- /dev/null +++ b/docs/src/local-first-guide/6-phone.md @@ -0,0 +1,64 @@ +# Step 5: the same Drive on a phone + +A browser tab cannot be dialed by another device, so in step 4 the server was the meeting point. +A phone running `atomic_lib` is different: it is a full peer. +It holds its own copy, can be dialed directly, and can serve the Drive to yet another device. + +This step uses the [Flutter reference app](../flutter.md). +Building your own Flutter UI on the same SDK is the same sequence of calls. + +## Restore the identity + +On the phone, sign in with the secret from step 1. +In the SDK that is one call: + +```dart +await AtomicClient.openDb('$appDir/atomic.redb'); +await AtomicClient.loadAgent(secret); // the string from Agent.buildSecret +``` + +This restores *who you are*. +The phone's store is still empty: a secret carries no data. + +## Get the data across + +Two routes, and they can both be on: + +**Through the server.** Give the phone the server's address. It opens a WebSocket as your Agent and fetches the Drives it lacks, including the reading list, then subscribes to live updates. + +```dart +await AtomicClient.openWsSync('http://your-server:9883'); +``` + +**Directly from the laptop**, if the laptop runs the desktop app or a local AtomicServer rather than only a browser tab. +Show the pairing code on one side and scan it on the other. +Both devices prove their Agent over the link, compare version vectors, and exchange only what differs. + +```dart +await AtomicClient.startPeer(); +final code = await AtomicClient.getPeerId(); // render as a QR code +// on the scanning side: +await AtomicClient.peerSync(scannedNodeId); +``` + +The code carries the *route* to the device and nothing else. +What crosses the link is decided by the rights on each Resource, the same check the server ran in step 4. + +## Edit anywhere + +Add a Book on the phone in a tunnel. +It is signed, saved locally, and queued. +When the phone next sees the server or the laptop, the commit travels, the browser's `store.subscribe` callback fires, and the list updates. +Edit the same Book on both devices while apart and the two edits merge; nobody is asked to pick a winner. + +## What you have + +The reading list on a laptop, a server and a phone. +Each copy is complete, each edit is signed, and any one of the three can go away without the other two losing anything. +That is the whole local-first story, in one small app. + +## Where to go from here + +- Model your own data with the [ontology editor](../atomicserver/gui.md) and generate types with [`@tomic/cli`](../js-cli.md). +- Use [`@tomic/react`](../usecases/react.md) or [`@tomic/svelte`](../svelte.md) instead of raw `store.subscribe` calls. +- Read [Atomic Sync](../sync.md) for what the engine does under the hood, and [URLs and identifiers](../urls.md) for every identifier you met along the way. diff --git a/docs/src/roadmap.md b/docs/src/roadmap.md index 6c76ac68a..28fde7a6e 100644 --- a/docs/src/roadmap.md +++ b/docs/src/roadmap.md @@ -58,4 +58,5 @@ We are working on polishing the developer experience (SDKs/APIs), expanding offl - **Passkey + PRF integration** (2026). Derive the encryption key directly from WebAuthn, making "sign in with passkey" a true one-step account restore. - **Cross-Agent Suggestions / Distributor Mode** (2026). Enabling users to suggest edits to other agents' drives via direct mesh channels. - **Multi-user P2P Presence** (2026). Ephemeral cursors and awareness syncing device-to-device directly over Iroh without a central hub. +- **Published SDK packages** (2026). Ship the WASM client database as an npm package and the Dart SDK on pub.dev, so a local-first app is an install away instead of a checkout. See `planning/dart-sdk-package.md`. - **1.0 release** (tbd). Mark the specification, the server, and the browser as *stable*. diff --git a/docs/src/tooling.md b/docs/src/tooling.md index 30c5449c2..1dbb3cc6d 100644 --- a/docs/src/tooling.md +++ b/docs/src/tooling.md @@ -38,4 +38,4 @@ A mobile app for granting permissions to your data and signing things. See [gith ### Atomic-Dart as a pub.dev package -The [Flutter SDK](flutter.md) exists but lives inside the reference app. Extracting it into a published Dart package would let other apps depend on it directly. +The [Flutter SDK](flutter.md) exists but lives inside the reference app. Extracting it into a published Dart package would let other apps depend on it directly; the plan is in `planning/dart-sdk-package.md` in the repository. diff --git a/planning/dart-sdk-package.md b/planning/dart-sdk-package.md new file mode 100644 index 000000000..7379f17d8 --- /dev/null +++ b/planning/dart-sdk-package.md @@ -0,0 +1,77 @@ +# Dart SDK as a published package + +> **Status:** Proposal (2026-09-18). Nothing moved yet. Written alongside the +> docs refresh that added `docs/src/flutter.md`; that page currently points at +> a folder inside the reference app because there is nothing else to point at. + +## Goal + +`flutter/lib/atomic/` is a general-purpose Atomic Data SDK (`AtomicClient`, +`AtomicStore`, `AtomicSession`, `atomic_auth.dart`, the FRB bridge) that +happens to live inside Atomic Canvas. Extract it into a package other Flutter +apps can depend on, so the docs can say `flutter pub add atomic_data` instead +of "vendor this folder". + +## What is general and what is canvas + +| General (moves) | Canvas-only (stays) | +| --- | --- | +| `openDb`, agent setup / load, drives, `getProperty` / `setProperty` | `createCanvas*`, `loadCanvasStrokes`, `listCanvases*`, folders, thumbnails | +| history (`warmResourceHistory`, `getResourceHistory`, `getResourceAtVersion`) | `wsSubscribeCanvas` | +| peer sync (`startPeer`, `peerSync`, `peerAnnounce`, `peerDiscoverSync`, known peers) | stroke models, gallery, canvas painter | +| WS sync (`openWsSync`, `syncDriveToServer`, `resumeSession`, `pollDbEvent`) | | +| `atomic_auth.dart` (pure-Dart request signing) | | +| `server_url.dart`, `server_info.dart`, `session.dart` | | +| `widgets/` sync + settings screens (twins of `SyncRoute` in the browser) | | + +The Rust side (`flutter/rust/src/api/simple/`) has the same split: `state.rs`, +`ws_sync.rs` and the peer functions are general; the `CANVAS_*` constants and +stroke functions are not. + +## Shape + +``` +flutter/ + packages/ + atomic_data/ # the SDK: Dart + rust_builder + FRB config + lib/ + rust/ # today's flutter/rust, minus canvas api + rust_builder/ + pubspec.yaml # name: atomic_data + apps/ + atomic_canvas/ # today's flutter/, depends on ../packages/atomic_data +``` + +`flutter_rust_bridge` generates bindings per crate, so the canvas keeps a +small FRB crate of its own for stroke functions, depending on `atomic_data`'s +crate for the store. Alternatively expose a generic `callAction(json)` from the +SDK and keep canvas logic in Dart; decide after measuring the stroke path. + +## Steps + +- [ ] Split `flutter/rust/src/api/simple.rs` into `atomic` (general) and + `canvas` modules; make `canvas` depend on `atomic`'s `db()` / `node()`. +- [ ] Create `flutter/packages/atomic_data` with its own `pubspec.yaml`, + `flutter_rust_bridge.yaml` and `rust_builder`; move the general Dart + files and the generated `src/rust/` bindings. +- [ ] Point Atomic Canvas at the package with a path dependency; keep + `make phone` / `make tablet` / `make web` working. +- [ ] Move the Dart unit tests in `flutter/test/atomic/` (URL rules, pairing + parsing, signing parity with Rust) with the package. +- [ ] CI: `cargo test --manifest-path flutter/packages/atomic_data/rust/Cargo.toml` + replaces the current `flutter/rust` invocation in `AGENTS.md` and the + dagger job. +- [ ] Publish `atomic_data` to pub.dev from the `v*` tag, the same trigger that + publishes `@tomic/*` to npm (`.github/workflows/release.yml`). +- [ ] Update `docs/src/flutter.md` (install line, import path) and + `docs/src/tooling.md`. + +## Open questions + +- Package name: `atomic_data` vs `tomic` (matching the npm scope). Prefer + `atomic_data`; the npm scope exists only because `atomic` was taken. +- The web target: FRB compiles to WASM, but the browser already has the + `wasm/` crate through `@tomic/lib`. One WASM build or two? Not blocking for + mobile. +- Whether `AtomicStore` (the `ChangeNotifier`) belongs in the SDK or in a + separate `atomic_data_flutter` package, keeping the core pure Dart. From b0c604c18aa467ddde5d705e2f5996861efcc655 Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Fri, 18 Sep 2026 23:32:54 +0200 Subject: [PATCH 003/110] Pin Rust builds and enforce Server/SaaS dependency alignment (#1499) * Pin Rust builds and enforce Server/SaaS dependency alignment * Run trusted CI orchestration directly on Mancave --- .dagger/src/index.ts | 30 ++++++--- .github/workflows/main.yml | 91 +++++++++------------------ .github/workflows/release.yml | 15 ++--- .github/workflows/rust-alignment.yml | 17 +++++ .github/workflows/tauri-release.yml | 6 +- CONTRIBUTING.md | 37 +++++++++++ TESTING_COVERAGE.md | 9 +++ rust-toolchain.toml | 4 +- scripts/check-rust-alignment.py | 93 ++++++++++++++++++++++++++++ scripts/test_rust_alignment.py | 67 ++++++++++++++++++++ 10 files changed, 286 insertions(+), 83 deletions(-) create mode 100644 .github/workflows/rust-alignment.yml create mode 100644 scripts/check-rust-alignment.py create mode 100644 scripts/test_rust_alignment.py diff --git a/.dagger/src/index.ts b/.dagger/src/index.ts index 2a2d40e1c..3554e1df8 100644 --- a/.dagger/src/index.ts +++ b/.dagger/src/index.ts @@ -50,7 +50,9 @@ function touchWorkspaceSources(container: Container): Container { } const NODE_IMAGE = 'node:22'; -const RUST_IMAGE = 'rust:bookworm'; +// Checked against rust-toolchain.toml by scripts/check-rust-alignment.py. +const RUST_VERSION = '1.98.1'; +const RUST_IMAGE = `rust:${RUST_VERSION}-bookworm`; // Pin the tools and their published dependency locks. Unlocked linkcheck // installation picked up jiff 0.2.36, whose packaged doc includes are broken. @@ -327,6 +329,11 @@ export class AtomicServer { ): Container { return ( container + .withExec([ + 'rustup', 'toolchain', 'install', RUST_VERSION, + '--profile', 'minimal', '--component', 'rustfmt,clippy', + ]) + .withEnvVariable('RUSTUP_TOOLCHAIN', RUST_VERSION) .withMountedCache( `${cargoHome}/registry`, dag.cacheVolume('cargo-shared-locks-v1'), @@ -607,6 +614,7 @@ export class AtomicServer { dag .container() .from(FLUTTER_IMAGE) + .withEnvVariable('RUSTUP_TOOLCHAIN', RUST_VERSION) .withEnvVariable('CI', 'true') // Same pin as withCargoHomeCache β€” flutter's cargokit build would // otherwise see all host CPUs. @@ -638,6 +646,10 @@ export class AtomicServer { '-c', 'if [ ! -x "$HOME/.cargo/bin/rustc" ]; then curl --proto "=https" --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal; fi', ]) + .withExec([ + 'sh', '-c', + `${pathPrefix}; rustup toolchain install ${RUST_VERSION} --profile minimal`, + ]) .withDirectory('/workspace/lib', this.source.directory('lib')) .withDirectory('/workspace/flutter', this.source.directory('flutter')) .withMountedCache('/workspace/flutter/rust/target', flutterRustTarget, { @@ -810,6 +822,7 @@ export class AtomicServer { .withExec([ 'cargo', 'build', + '--locked', '-p', 'atomic-server', '--no-default-features', @@ -1203,6 +1216,7 @@ export class AtomicServer { .withExec(['apt', 'install', '-y', 'nasm', 'protobuf-compiler']), CARGO_HOME_MUSL, ) + .withExec(['rustup', 'target', 'add', target]) .withExec(['rustup', 'component', 'add', 'clippy']) .withExec(['rustup', 'component', 'add', 'rustfmt']); // cargo-nextest used to be installed here, but recent versions need @@ -1245,7 +1259,7 @@ export class AtomicServer { .withMountedCache('/code/target', dag.cacheVolume('rust-target-v3')) .with(touchWorkspaceSources) .withWorkdir('/code') - .withExec(['cargo', 'fetch']); + .withExec(['cargo', 'fetch', '--locked']); const browserDir = this.jsBuild(e2e).directory('/app/data-browser/dist'); const containerWithAssets = sourceContainer.withDirectory( @@ -1261,10 +1275,10 @@ export class AtomicServer { // at an optimisation level where commit round-trips stop dominating, and // cheap enough to compile that Playwright is not left waiting on LTO. const buildArgs = e2e - ? ['cargo', 'build', '--profile', 'e2e', '-p', 'atomic-server'] + ? ['cargo', 'build', '--locked', '--profile', 'e2e', '-p', 'atomic-server'] : release - ? ['cargo', 'build', '--release', '-p', 'atomic-server'] - : ['cargo', 'build', '-p', 'atomic-server']; + ? ['cargo', 'build', '--locked', '--release', '-p', 'atomic-server'] + : ['cargo', 'build', '--locked', '-p', 'atomic-server']; // ⚠️ PRODUCTION IMPACT, not just CI plumbing (2026-07-02): this // function backs BOTH the e2e test server (atomicService) AND the real @@ -1398,6 +1412,7 @@ export class AtomicServer { .withExec(['apt', 'install', '-y', 'nasm', 'protobuf-compiler']), CARGO_HOME_MUSL, ) + .withExec(['rustup', 'target', 'add', 'x86_64-unknown-linux-musl']) .withExec(['rustup', 'component', 'add', 'clippy']) .withExec(['rustup', 'component', 'add', 'rustfmt']) .withFile('/code/Cargo.toml', source.file('Cargo.toml')) @@ -1450,7 +1465,7 @@ export class AtomicServer { '-c', 'echo "checks stub" > /code/server/assets_tmp/index.html', ]) - .withExec(['cargo', 'fetch']) + .withExec(['cargo', 'fetch', '--locked']) ); } @@ -1499,7 +1514,7 @@ export class AtomicServer { 'BIN_DIR=/opt/cargo-bin/bin && mkdir -p "$BIN_DIR" && ' + 'if [ ! -x "$BIN_DIR/cargo-nextest" ]; then ' + 'curl -LsSf https://get.nexte.st/latest/linux-musl | tar zxf - -C "$BIN_DIR"; fi && ' + - 'cargo nextest run --workspace --exclude atomic-server-tauri ' + + 'cargo nextest run --locked --workspace --exclude atomic-server-tauri ' + '--no-default-features --features light,wasm-plugins ' + `--build-jobs ${this.hostKnobs.nextestBuildJobs} ` + `--test-threads ${this.hostKnobs.nextestTestThreads} ` + @@ -1531,6 +1546,7 @@ export class AtomicServer { .withExec([ 'cargo', 'clippy', + '--locked', '--workspace', '--exclude', 'atomic-server-tauri', diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index aabd7b787..0f5f362ea 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -19,25 +19,19 @@ concurrency: cancel-in-progress: ${{ github.ref_type == 'branch' && github.ref_name != 'develop' }} # Main prefers Mancave (self-hosted) so Dagger cache volumes stay warm. -# GitHub-hosted runs ONLY when Mancave is unavailable β€” never as a retry +# GitHub-hosted runs only when explicitly selected or required for PRs, never as a retry # after a red test suite. # # Flow: -# 1. Pick runner β€” online self-hosted? (optional CI_RUNNER_STATUS_TOKEN) +# 1. Pick runner β€” trusted self-hosted or explicitly hosted # 2. Main (Mancave) XOR Main (GitHub) β€” one path, not failure-failover # 3. Main β€” aggregator for the required status check # -# Availability check needs a PAT that can list repo runners (GITHUB_TOKEN -# cannot). Store it as Actions secret CI_RUNNER_STATUS_TOKEN. Without that -# secret we assume Mancave is up (same as before) and will not fall back -# when tests fail. -# -# Escape hatch (force hosted, skip Mancave): -# Settings -> Secrets and variables -> Actions -> Variables -# CI_RUNNER = ["ubuntu-latest"] -# -# Only this workflow uses the self-hosted runner. Releasing/deploying stay -# on ubuntu-latest β€” a release must not depend on one PC being awake. +# Trusted runs start directly on Mancave, including selector and result jobs. +# There is no GitHub-hosted prerequisite, so exhausted Actions budgets do not +# block self-hosted CI. If Mancave is offline, jobs queue; explicitly set the +# repository variable CI_RUNNER = ["ubuntu-latest"] to use hosted runners. +# No runner-status PAT is needed. Deployments retain their existing runners. # # SECURITY: pull_request / pull_request_target are forced onto hosted # runners (never Mancave). @@ -47,7 +41,9 @@ name: "Main pipeline: build, lint, test" jobs: pick: name: Pick runner - runs-on: ubuntu-latest + # Trusted runs can start without a GitHub-hosted billing prerequisite. + # Mancave must be online; use CI_RUNNER to explicitly select hosted runners. + runs-on: ${{ fromJSON((github.event_name == 'pull_request_target' || github.event_name == 'pull_request' || vars.CI_RUNNER == '["ubuntu-latest"]') && '["ubuntu-latest"]' || (vars.CI_RUNNER || '["self-hosted", "Linux", "X64"]')) }} permissions: contents: read pull-requests: read @@ -57,7 +53,6 @@ jobs: steps: - id: pick env: - GH_TOKEN: ${{ secrets.CI_RUNNER_STATUS_TOKEN }} FORCE_RUNNER: ${{ vars.CI_RUNNER }} EVENT_NAME: ${{ github.event_name }} REPO: ${{ github.repository }} @@ -76,37 +71,8 @@ jobs: exit 0 fi - if [ -z "${GH_TOKEN:-}" ]; then - echo "No CI_RUNNER_STATUS_TOKEN β€” assuming Mancave online" - echo "host=mancave" >> "$GITHUB_OUTPUT" - exit 0 - fi - - # Any online runner carrying the self-hosted label counts as - # available (busy is fine β€” the job will wait its turn). - # On API errors (bad/insufficient PAT), prefer Mancave β€” do NOT - # treat a 403 as "offline" or we skip the warm box forever. - set +e - body=$(gh api "repos/$REPO/actions/runners" 2>/tmp/runner-api.err) - api_status=$? - set -e - if [ "$api_status" -ne 0 ]; then - echo "Runner API failed (is CI_RUNNER_STATUS_TOKEN a fine-grained PAT with Repository Administration: Read?):" - cat /tmp/runner-api.err || true - echo "Assuming Mancave online" - echo "host=mancave" >> "$GITHUB_OUTPUT" - exit 0 - fi - - if printf '%s' "$body" | jq -e \ - 'any(.runners[]; .status == "online" and any(.labels[]; .name == "self-hosted"))' \ - >/dev/null; then - echo "Self-hosted runner online β€” using Mancave" - echo "host=mancave" >> "$GITHUB_OUTPUT" - else - echo "No online self-hosted runner β€” falling back to GitHub-hosted" - echo "host=hosted" >> "$GITHUB_OUTPUT" - fi + echo "Selector is running on the self-hosted runner β€” using Mancave" + echo "host=mancave" >> "$GITHUB_OUTPUT" - id: e2e env: @@ -206,7 +172,7 @@ jobs: id: saas env: GH_TOKEN: ${{ secrets.ATOMIC_SAAS_DISPATCH_TOKEN }} - BRANCH: ${{ github.ref_name }} + BRANCH: ${{ github.head_ref || github.ref_name }} run: | set -euo pipefail if [ -z "${GH_TOKEN:-}" ]; then @@ -238,6 +204,13 @@ jobs: # (where `clean: false` keeps the checkout around between runs). persist-credentials: false + - name: Set up Python for Rust alignment checks + uses: actions/setup-python@v5 + with: + python-version: '3.12' + - name: Verify paired Rust toolchains and dependencies + run: python3 atomic-server/scripts/check-rust-alignment.py --server atomic-server --saas atomic-saas + - name: Install Rust uses: dtolnay/rust-toolchain@1.98.1 with: @@ -267,28 +240,22 @@ jobs: npm ci npm run build - # Not `--locked`: this commit may have changed atomic_lib's own - # dependencies, which legitimately moves atomic-saas's lockfile. What is - # being checked is that the code still builds and passes; a lock update - # is reported so the saas side commits it on its next touch. + # Both lockfiles must describe the tested pair; update them in paired PRs. - name: cargo check, clippy and test atomic-saas against this commit working-directory: atomic-saas run: | set -euo pipefail - cargo check --all-targets - cargo clippy --all-targets -- -D warnings - cargo test --all-targets + cargo check --locked --all-targets + cargo clippy --locked --all-targets -- -D warnings + cargo test --locked --all-targets # The managed node embeds this repo's server crate and is not a # default workspace member, so the root build does not touch it. # Its own CI runs check, clippy and test; the same here. - cargo clippy -p atomic-managed-node --all-targets -- -D warnings - cargo test -p atomic-managed-node - if ! git diff --quiet -- Cargo.lock; then - git diff --stat -- Cargo.lock - echo "::warning::atomic-saas's Cargo.lock drifted against this commit; run cargo check there and commit Cargo.lock." - fi + cargo clippy --locked -p atomic-managed-node --all-targets -- -D warnings + cargo test --locked -p atomic-managed-node + git diff --exit-code -- Cargo.lock - # Hosted only when pick said Mancave is unavailable (or escape/PR). + # Hosted only when explicitly selected or required for PR events. # Deliberately does NOT run when Mancave finished with test failures. github: name: Main (GitHub) @@ -312,7 +279,7 @@ jobs: name: Main needs: [pick, mancave, github, downstream-saas] if: always() && needs.pick.result == 'success' - runs-on: ubuntu-latest + runs-on: ${{ fromJSON(needs.pick.outputs.host == 'hosted' && '["ubuntu-latest"]' || (vars.CI_RUNNER || '["self-hosted", "Linux", "X64"]')) }} steps: - name: Resolve runner outcome run: | diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 9ee329cb0..1246bdad0 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -33,11 +33,9 @@ jobs: - uses: actions/checkout@v4 with: ref: ${{ inputs.tag || github.ref }} - - uses: actions-rs/toolchain@v1 + - uses: dtolnay/rust-toolchain@1.98.1 with: - toolchain: stable - override: true - target: wasm32-unknown-unknown + targets: wasm32-unknown-unknown # `version: 8` used to be pinned here while browser/package.json declares # `packageManager: pnpm@10.15.1`, and there was no Node at all β€” so this @@ -281,7 +279,7 @@ jobs: permissions: contents: write steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - uses: pnpm/action-setup@v2.0.1 name: Install pnpm id: pnpm-install @@ -392,12 +390,9 @@ jobs: version: 10 run_install: false - - uses: actions-rs/toolchain@v1 + - uses: dtolnay/rust-toolchain@1.98.1 with: - profile: minimal - toolchain: stable - override: true - target: wasm32-unknown-unknown + targets: wasm32-unknown-unknown - uses: taiki-e/upload-rust-binary-action@v1 with: # (required) diff --git a/.github/workflows/rust-alignment.yml b/.github/workflows/rust-alignment.yml new file mode 100644 index 000000000..4a2a3346d --- /dev/null +++ b/.github/workflows/rust-alignment.yml @@ -0,0 +1,17 @@ +name: Rust build policy +on: + pull_request: + push: + branches: [develop] +permissions: + contents: read +jobs: + check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + - run: python3 -m unittest discover -s scripts -p 'test_rust_alignment.py' -v + - run: python3 scripts/check-rust-alignment.py diff --git a/.github/workflows/tauri-release.yml b/.github/workflows/tauri-release.yml index 0a017c3f5..ecaf27ad7 100644 --- a/.github/workflows/tauri-release.yml +++ b/.github/workflows/tauri-release.yml @@ -187,7 +187,7 @@ jobs: version: 10 run_install: false - - uses: dtolnay/rust-toolchain@stable + - uses: dtolnay/rust-toolchain@1.98.1 with: targets: ${{ matrix.rust-targets }} @@ -400,7 +400,7 @@ jobs: sdkmanager --install "ndk;27.2.12479018" >/dev/null echo "NDK_HOME=${ANDROID_SDK_ROOT}/ndk/27.2.12479018" >> "$GITHUB_ENV" - - uses: dtolnay/rust-toolchain@stable + - uses: dtolnay/rust-toolchain@1.98.1 with: # arm64 only: phones/tablets we ship to. The other three Android # targets were Tauri's generated default and made the universal APK @@ -547,7 +547,7 @@ jobs: version: 10 run_install: false - - uses: dtolnay/rust-toolchain@stable + - uses: dtolnay/rust-toolchain@1.98.1 with: targets: aarch64-apple-ios,aarch64-apple-ios-sim,x86_64-apple-ios,wasm32-unknown-unknown diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 361f2d20b..4c3ae6d63 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -481,3 +481,40 @@ journalctl -u atomic.service --since "1 hour ago" -f 1. `cd cli` 1. run `cargo wasi build --release --no-default-features` (note: this fails, as ring does not compile to WASI [at this moment](https://github.com/briansmith/ring/issues/1043)) 1. `wapm publish` + +## Coordinated Rust updates + +Server and SaaS use the same exact Rust release, selected by each repository's +`rust-toolchain.toml`. CI pins and Dagger's `RUST_VERSION` must match it. The +Server-owned checker validates those copies; do not switch a workflow or Docker +image back to floating `stable`/`rust:bookworm`. + +For an update, create branches with the same name in both repositories. Existing +paired-branch CI resolves both heads to commit SHAs and tests that exact pair. +Update both toolchain files, the workflow pins, and Dagger's version together. +Keep separate committed `Cargo.lock` files. Update selected shared versions with +`cargo update -p NAME@OLD --precise VERSION`, then run from the Server checkout: + +```sh +python3 scripts/check-rust-alignment.py --saas ../atomic-saas +cargo metadata --locked --format-version 1 > /dev/null +(cd ../atomic-saas && cargo metadata --locked --format-version 1 > /dev/null) +``` + +The check compares Loro (including common transitive Loro crates), Iroh, Serde, +JSON, Ed25519/Curve25519, Tokio, Reqwest, and Object Store. It fails on different +resolved version sets, including additional prereleases. Other dependencies may +differ: SaaS's own redb database intentionally still uses major 2 while Server +uses major 4. Add a crate to the policy when its alignment matters; do not copy +whole lockfiles or force unrelated dependencies to match. + +Development profiles also match (`line-tables-only` for local crates, no debug +info for dependencies). Features, targets, native toolchains, and intentionally +different CI profiles can still produce separate cached artifacts. A shared +build cache does not synchronize dependencies. + +Paired Rust checks and SaaS build/release commands use `--locked` so source changes cannot silently +rewrite the tested dependency graph. Commit any needed lockfile updates in the +paired PR, run the affected Rust suites, and merge the pair together after CI. +The SaaS alignment gate requires the Server checker, so land the Server PR +before the SaaS PR when first introducing this policy. diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index 6cda0d8c7..f3942bb83 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -2151,3 +2151,12 @@ and upload hook, then delivers multiple files through the drop callback. It verifies the upload targets the displayed drive even when the current drive setting differs. Native drag events, overlay geometry and the refreshed child list are not covered by this component test. + +## Rust build alignment + +`scripts/test_rust_alignment.py` tests matching pairs, compiler/workflow pin drift, +development profile drift, transitive Loro versions, missing shared crates, extra +cryptography prereleases, and allowed unrelated dependency differences. Run +`python3 -m unittest discover -s scripts -p test_rust_alignment.py -v`. +The Rust build policy workflow runs these checks; downstream CI checks both +repositories and rejects dependency lockfile drift before builds. diff --git a/rust-toolchain.toml b/rust-toolchain.toml index 292fe499e..c3f67b677 100644 --- a/rust-toolchain.toml +++ b/rust-toolchain.toml @@ -1,2 +1,4 @@ [toolchain] -channel = "stable" +channel = "1.98.1" +profile = "minimal" +components = ["rustfmt", "clippy"] diff --git a/scripts/check-rust-alignment.py b/scripts/check-rust-alignment.py new file mode 100644 index 000000000..8d9a9c288 --- /dev/null +++ b/scripts/check-rust-alignment.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +"""Check local Rust pins, or a Server/SaaS pair. Requires Python 3.11+.""" +import argparse +from pathlib import Path +import re +import sys +import tomllib + +# These libraries cross persistence, transport, serialization or signing boundaries. +# redb is intentionally excluded: SaaS's own database still uses a different major. +SHARED = { + "loro", "iroh", "serde", "serde_json", "ed25519-dalek", + "curve25519-dalek", "tokio", "reqwest", "object_store", +} + + +def read_toml(path): + return tomllib.loads(path.read_text()) + + +def versions(root): + result = {} + for package in read_toml(root / "Cargo.lock")["package"]: + if package.get("source", "").startswith("registry+"): + result.setdefault(package["name"], set()).add(package["version"]) + return result + + +def check_local(root): + errors = [] + channel = read_toml(root / "rust-toolchain.toml")["toolchain"]["channel"] + if not re.fullmatch(r"\d+\.\d+\.\d+", channel): + errors.append(f"{root.name}: Rust channel must be an exact release, got {channel}") + for workflow in sorted((root / ".github/workflows").glob("*.yml")): + text = workflow.read_text() + pins = re.findall(r"uses:\s*dtolnay/rust-toolchain@([^\s#]+)", text) + # Legacy actions-rs/toolchain uses a `toolchain:` input. + if "actions-rs/toolchain@" in text: + pins += re.findall(r"^\s+toolchain:\s*([^\s#]+)", text, re.MULTILINE) + for pin in pins: + if pin.strip("\"'") != channel: + errors.append(f"{workflow}: Rust {pin} differs from {channel}") + dagger = root / ".dagger/src/index.ts" + if dagger.exists(): + text = dagger.read_text() + if f"const RUST_VERSION = '{channel}';" not in text: + errors.append(f"{dagger}: RUST_VERSION must match {channel}") + if 'const RUST_IMAGE = `rust:${RUST_VERSION}-bookworm`;' not in text: + errors.append(f"{dagger}: Rust image must use RUST_VERSION") + return errors + + +def check_pair(server, saas): + errors = check_local(server) + check_local(saas) + server_toolchain = read_toml(server / "rust-toolchain.toml")["toolchain"] + saas_toolchain = read_toml(saas / "rust-toolchain.toml")["toolchain"] + if server_toolchain != saas_toolchain: + errors.append("rust-toolchain.toml differs between Server and SaaS") + for section in ["dev"]: + a = read_toml(server / "Cargo.toml").get("profile", {}).get(section, {}) + b = read_toml(saas / "Cargo.toml").get("profile", {}).get(section, {}) + if a != b: + errors.append(f"profile.{section} differs between Server and SaaS") + a, b = versions(server), versions(saas) + names = SHARED | {name for name in a.keys() & b.keys() if name.startswith("loro-")} + for name in sorted(names): + if not a.get(name) or a.get(name) != b.get(name): + errors.append( + f"{name}: Server={sorted(a.get(name, []))}, SaaS={sorted(b.get(name, []))}; " + "align with cargo update -p NAME@OLD --precise VERSION and commit Cargo.lock" + ) + return errors + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--server", type=Path, default=Path(__file__).resolve().parents[1]) + parser.add_argument("--saas", type=Path, help="also check this SaaS checkout") + args = parser.parse_args() + try: + errors = check_pair(args.server, args.saas) if args.saas else check_local(args.server) + except (OSError, KeyError, tomllib.TOMLDecodeError) as error: + print(f"Rust alignment configuration error: {error}", file=sys.stderr) + return 1 + if errors: + print("Rust alignment failed:\n- " + "\n- ".join(errors), file=sys.stderr) + return 1 + print("Rust toolchain, workflow pins" + (", profiles and shared lockfile versions" if args.saas else "") + " are aligned.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/test_rust_alignment.py b/scripts/test_rust_alignment.py new file mode 100644 index 000000000..d2ef25955 --- /dev/null +++ b/scripts/test_rust_alignment.py @@ -0,0 +1,67 @@ +import importlib.util +from pathlib import Path +import tempfile +import unittest + +spec = importlib.util.spec_from_file_location("alignment", Path(__file__).with_name("check-rust-alignment.py")) +alignment = importlib.util.module_from_spec(spec) +spec.loader.exec_module(alignment) + + +class AlignmentTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.server = Path(self.temp.name) / "server" + self.saas = Path(self.temp.name) / "saas" + for root in [self.server, self.saas]: + (root / ".github/workflows").mkdir(parents=True) + (root / "rust-toolchain.toml").write_text('[toolchain]\nchannel="1.98.1"\n') + (root / "Cargo.toml").write_text('[profile.dev]\ndebug="line-tables-only"\n') + packages = alignment.SHARED | {"loro-internal"} + (root / "Cargo.lock").write_text("\n".join( + f'[[package]]\nname="{name}"\nversion="1.0.0"\nsource="registry+https://example.org"\n' + for name in sorted(packages) + )) + + def check(self): + return alignment.check_pair(self.server, self.saas) + + def test_matching_pair(self): + self.assertEqual(self.check(), []) + + def test_toolchain_drift(self): + (self.saas / "rust-toolchain.toml").write_text('[toolchain]\nchannel="1.97.0"\n') + self.assertIn("rust-toolchain.toml differs", "\n".join(self.check())) + + def test_floating_workflow_pin(self): + (self.server / ".github/workflows/ci.yml").write_text('steps:\n- uses: dtolnay/rust-toolchain@stable\n') + self.assertIn("Rust stable differs", "\n".join(self.check())) + + def test_profile_drift(self): + (self.saas / "Cargo.toml").write_text('[profile.dev]\ndebug=true\n') + self.assertIn("profile.dev differs", "\n".join(self.check())) + + def test_transitive_loro_drift(self): + p = self.saas / "Cargo.lock" + p.write_text(p.read_text().replace('name="loro-internal"\nversion="1.0.0"', 'name="loro-internal"\nversion="1.1.0"')) + self.assertIn("loro-internal:", "\n".join(self.check())) + + def test_missing_critical_dependency(self): + p = self.saas / "Cargo.lock" + p.write_text(p.read_text().replace('name="iroh"', 'name="something-else"')) + self.assertIn("iroh:", "\n".join(self.check())) + + def test_additional_crypto_version_is_drift(self): + p = self.saas / "Cargo.lock" + p.write_text(p.read_text() + '\n[[package]]\nname="ed25519-dalek"\nversion="3.0.0-rc.1"\nsource="registry+https://example.org"\n') + self.assertIn("ed25519-dalek:", "\n".join(self.check())) + + def test_unrelated_dependency_can_differ(self): + p = self.saas / "Cargo.lock" + p.write_text(p.read_text() + '\n[[package]]\nname="redb"\nversion="2.6.3"\nsource="registry+https://example.org"\n') + self.assertEqual(self.check(), []) + + +if __name__ == "__main__": + unittest.main() From 5403ff45897d1ca8f1242cf97fd7e8bd625c0e6c Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 05:35:03 +0200 Subject: [PATCH 004/110] fix(server,lib): security hygiene from the September 2026 audit (#1558) Six small items from planning/security-audit-2026-09.md (D, C16, F), each verified against current code and covered by a test, plus the C20 verification recorded in the audit doc. - CORS: any origin may still read, but Access-Control-Allow-Credentials is sent only to origins this server answers for (configured domain, base domain tenants, loopback, the desktop webview origins), by the same rule RequestContext uses for Host. server/src/cors.rs, gate outside the permissive Cors layer. - Client errors as client errors: ParseError maps to 400, malformed commit bodies are parse errors, signature and auth-header failures are typed Unauthorized (AtomicError::into_unauthorized) and the server keeps that type instead of stringifying it. 400/401 instead of 500, so they no longer reach Sentry as crashes. - C16: the importing flag and import source are a tokio task-local (ws_apply::import_scope) instead of process-wide statics; the live push loop skips the one peer a change came from via the DbEvent source_id, which now also covers COMMIT frames applied for a live peer. CommitIngestOpts::suppress_live_echo is removed. The Flutter listener reads the event's source_id instead of the flag. - collections::sort_resources is a total order (missing last, numbers numeric with NaN placed, numbers before strings); the old comparator panicked sort_by from 21 elements up on Rust 1.81+. - sync::peer: LIVE_PEERS recovers from a poisoned lock instead of unwrapping it. - /upload answers 400 on a multipart error instead of 200 with a partial list. - Drive-by for the clippy gate on stable 1.94: an unnecessary to_string in a lib/src/db/website.rs test, pre-existing on develop. Claude-Session: https://claude.ai/code/session_01H1U7TNdjwqXKv8MQYproiq Co-authored-by: Claude --- CHANGELOG.md | 39 +++++ flutter/rust/src/api/simple.rs | 22 ++- lib/src/authentication.rs | 154 +++++++++-------- lib/src/collections.rs | 126 ++++++++++++-- lib/src/commit.rs | 10 ++ lib/src/db/website.rs | 2 +- lib/src/errors.rs | 16 ++ lib/src/parse.rs | 18 +- lib/src/sync/engine.rs | 43 ++--- lib/src/sync/peer.rs | 121 ++++++------- lib/src/sync/tests.rs | 4 - lib/src/sync/ws_apply.rs | 176 +++++++++++++++---- planning/production-readiness.md | 5 +- planning/security-audit-2026-09.md | 24 ++- server/src/bin.rs | 1 + server/src/context.rs | 2 +- server/src/cors.rs | 268 +++++++++++++++++++++++++++++ server/src/errors.rs | 5 +- server/src/handlers/commit.rs | 2 +- server/src/handlers/upload.rs | 19 +- server/src/helpers.rs | 17 +- server/src/lib.rs | 1 + server/src/serve.rs | 10 +- server/src/tests.rs | 203 ++++++++++++++++++++++ 24 files changed, 1032 insertions(+), 256 deletions(-) create mode 100644 server/src/cors.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 29a26cf56..ec6fed867 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,45 @@ See [STATUS.md](server/STATUS.md) to learn more about which features will remain ## UNRELEASED +- Security hygiene, from the September 2026 audit (`planning/security-audit-2026-09.md` + D, C16, F): + - CORS: any origin may still read (Atomic is a headless CMS), but + `Access-Control-Allow-Credentials` is now sent only to origins this + server answers for β€” the configured domain, a subdomain of the base + domain, loopback, and the desktop webview origins (`tauri://localhost`, + `http://tauri.localhost`) β€” by the same rule `RequestContext` uses for + trusting `Host`. Before, every origin was told it could attach the + session cookie; `SameSite=Lax` was the only thing stopping it + (`server/src/cors.rs`). + - Client errors are answered as client errors instead of 500: a request + body that does not parse is 400 (`AtomicErrorType::ParseError` maps to + `BadRequest`; malformed commits are parse errors), a signature that does + not verify, an unknown signer, or authentication headers that do not + parse are 401 (`Commit::validate_signature` and + `get_agent_from_auth_values_and_check` type every failure as + unauthorized, and `get_client_agent` no longer flattens the type into a + string). These no longer reach Sentry as crashes. + - `/upload` returns 400 when the multipart body cannot be read (no + boundary, truncated part) instead of ending its loop quietly and + answering 200 for the files that made it through. + - `atomic_lib`: the "importing from a peer" flag and the import source are + a tokio task-local scope (`sync::ws_apply::import_scope`) instead of + process-wide statics, so two peer connections importing at once no + longer clear each other's flag or stamp each other's peer id on their + writes (audit C16). The live push loop no longer consults a global mute; + it skips the one peer a change came from, via the `source_id` the scope + stamps on the `DbEvent`, which now also covers `COMMIT` frames applied + for a live peer. `CommitIngestOpts::suppress_live_echo` is gone: a peer + commit is attributed to the connection's import scope instead. + `sync::peer::is_importing()` is now per task; a listener on another + task should read the event's `source_id` (the Flutter binding does). + - `collections::sort_resources` is a total order (missing values last in + both directions, numbers compared numerically with `NaN` in a fixed + place, numbers before strings), so a collection sorted through the + `sort_by` query parameter can no longer panic `sort_by` on Rust 1.81+. + - `sync::peer`: the live peer map recovers from a poisoned lock instead of + unwrapping it, so one panic no longer stops live broadcast for the rest + of the process. - CI: the `:develop` docker image is published by its own job instead of a step tacked onto the end of the CI job. As a step it inherited whatever the CI step had already spent (a wedged Dagger engine on the runner burned the diff --git a/flutter/rust/src/api/simple.rs b/flutter/rust/src/api/simple.rs index 41b24ba81..49a459c1b 100644 --- a/flutter/rust/src/api/simple.rs +++ b/flutter/rust/src/api/simple.rs @@ -466,24 +466,22 @@ fn ensure_cache_listener() { // our own commit β€” a stroke merged in from a peer β€” is NOT in the // cached doc, so we must drop it: otherwise the next local // `push_stroke` appends to a stale doc and its save overwrites the - // just-merged remote stroke, silently reverting it. `is_importing` - // alone missed this: it is only set around the agent-resource apply, - // never around a remote canvas UPDATE, so canvas edits raced and - // lost. `from_commit == false` is the reliable signal (remote sync + // just-merged remote stroke, silently reverting it. + // `from_commit == false` is the reliable signal (remote sync // applies via `add_resource_opts`; local strokes via a signed - // commit), and over-invalidating only ever costs a reload. + // commit), and a commit a peer sent carries its `source_id`. The + // import flag is per task now (`ws_apply::import_scope`), so it is + // never visible from this listener; the event itself says where + // the change came from. Over-invalidating only ever costs a + // reload, so a destroyed subject always drops its cached session. let (subject, from_remote) = match rx.recv().await { Ok(atomic_lib::DbEvent::Changed { subject, from_commit, + source_id, .. - }) => ( - subject, - !from_commit || atomic_lib::sync::ws_apply::is_importing(), - ), - Ok(atomic_lib::DbEvent::Destroyed { subject, .. }) => { - (subject, atomic_lib::sync::ws_apply::is_importing()) - } + }) => (subject, !from_commit || source_id.is_some()), + Ok(atomic_lib::DbEvent::Destroyed { subject, .. }) => (subject, true), Ok(atomic_lib::DbEvent::QueryMembershipChanged { .. }) => continue, Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => continue, Err(_) => { diff --git a/lib/src/authentication.rs b/lib/src/authentication.rs index 09cfcab95..b5ee5e56c 100644 --- a/lib/src/authentication.rs +++ b/lib/src/authentication.rs @@ -2,7 +2,7 @@ use crate::{ agents::{decode_base64, ForAgent}, - errors::AtomicResult, + errors::{AtomicError, AtomicResult}, urls, utils::check_timestamp_fresh, Storelike, @@ -103,89 +103,99 @@ pub async fn get_agent_from_auth_values_and_check( auth_header_values: Option, store: &impl Storelike, ) -> AtomicResult { - if let Some(auth_vals) = auth_header_values { - // If there are auth headers, check 'em, make sure they are valid. - check_auth_signature(&auth_vals.requested_subject, &auth_vals) - .map_err(|e| format!("Error checking authentication headers. {}", e))?; - // check if the timestamp is valid: not in the future, and not so old - // that a captured proof could be replayed indefinitely. - check_timestamp_fresh( - auth_vals.timestamp, - ACCEPTABLE_TIME_DIFFERENCE, - AUTH_MAX_AGE_MS, - ) - .map_err(|e| format!("Authentication timestamp rejected. {}", e))?; - // check if the public key belongs to the agent - // For DID subjects, we need to fetch the agent resource locally - // unless it's a DID based on the public key, in which case we can verify it directly. - let agent_subject = crate::Subject::from_raw(auth_vals.agent_subject.trim(), None); - let public_key_trimmed = auth_vals.public_key.trim(); - - if agent_subject.is_did() { - // The DID subject embeds the agent's public key - // (`did:ad:agent:{pubkey}`) and the auth header carries the same - // key. The two may use different base64 alphabets β€” the url-safe - // alphabet (the new default) vs the legacy standard alphabet - // (`+` `/` `=`) β€” so a raw-string `ends_with` wrongly rejects a key - // whose decoded BYTES are identical. Compare the decoded bytes. - let did_pubkey = agent_subject - .as_str() - .strip_prefix(crate::subject::DID_AD_AGENT_PREFIX) - .unwrap_or_else(|| agent_subject.as_str()); - if public_keys_match(did_pubkey, public_key_trimmed) { - return Ok(ForAgent::AgentSubject(agent_subject)); - } else { - return Err(format!( - "The public key in the auth headers '{}' does not match the DID subject '{}'", - public_key_trimmed, auth_vals.agent_subject - ) - .into()); - } - } + match auth_header_values { + // Every failure below means the caller is not who the headers say + // (a server answers 401), whichever check tripped. + Some(auth_vals) => check_auth_values(auth_vals, store) + .await + .map_err(AtomicError::into_unauthorized), + None => Ok(ForAgent::Public), + } +} - // Legacy `https://host/agents/{pubkey}` subjects are treated as - // `did:ad:agent:{pubkey}` by every rights check, so the key in the - // path is the identity being claimed: bind it to the signing key. - // Looking the key up in a resource at that URL instead would let - // anyone host `https://theirs/agents/` carrying their own - // key and be authorized as the victim (including as the server's own - // root agent, whose key is public). - if let Some(path_key) = crate::agents::legacy_agent_pubkey(agent_subject.as_str()) { - if public_keys_match(&path_key, public_key_trimmed) { - return Ok(ForAgent::AgentSubject(agent_subject)); - } +async fn check_auth_values( + auth_vals: AuthValues, + store: &impl Storelike, +) -> AtomicResult { + // If there are auth headers, check 'em, make sure they are valid. + check_auth_signature(&auth_vals.requested_subject, &auth_vals) + .map_err(|e| format!("Error checking authentication headers. {}", e))?; + // check if the timestamp is valid: not in the future, and not so old + // that a captured proof could be replayed indefinitely. + check_timestamp_fresh( + auth_vals.timestamp, + ACCEPTABLE_TIME_DIFFERENCE, + AUTH_MAX_AGE_MS, + ) + .map_err(|e| format!("Authentication timestamp rejected. {}", e))?; + // check if the public key belongs to the agent + // For DID subjects, we need to fetch the agent resource locally + // unless it's a DID based on the public key, in which case we can verify it directly. + let agent_subject = crate::Subject::from_raw(auth_vals.agent_subject.trim(), None); + let public_key_trimmed = auth_vals.public_key.trim(); + + if agent_subject.is_did() { + // The DID subject embeds the agent's public key + // (`did:ad:agent:{pubkey}`) and the auth header carries the same + // key. The two may use different base64 alphabets β€” the url-safe + // alphabet (the new default) vs the legacy standard alphabet + // (`+` `/` `=`) β€” so a raw-string `ends_with` wrongly rejects a key + // whose decoded BYTES are identical. Compare the decoded bytes. + let did_pubkey = agent_subject + .as_str() + .strip_prefix(crate::subject::DID_AD_AGENT_PREFIX) + .unwrap_or_else(|| agent_subject.as_str()); + if public_keys_match(did_pubkey, public_key_trimmed) { + return Ok(ForAgent::AgentSubject(agent_subject)); + } else { return Err(format!( - "The public key in the auth headers '{}' does not match the agent subject '{}'", + "The public key in the auth headers '{}' does not match the DID subject '{}'", public_key_trimmed, auth_vals.agent_subject ) .into()); } + } - // Any other agent subject must be a resource this store already - // holds. Never fetch it over the network during authentication: that - // is an unauthenticated SSRF, and the fetched body would be written - // into the store as a trusted resource. - let normalized_agent = store.normalize_subject(&agent_subject); - if !normalized_agent.is_local() { - return Err(format!( + // Legacy `https://host/agents/{pubkey}` subjects are treated as + // `did:ad:agent:{pubkey}` by every rights check, so the key in the + // path is the identity being claimed: bind it to the signing key. + // Looking the key up in a resource at that URL instead would let + // anyone host `https://theirs/agents/` carrying their own + // key and be authorized as the victim (including as the server's own + // root agent, whose key is public). + if let Some(path_key) = crate::agents::legacy_agent_pubkey(agent_subject.as_str()) { + if public_keys_match(&path_key, public_key_trimmed) { + return Ok(ForAgent::AgentSubject(agent_subject)); + } + return Err(format!( + "The public key in the auth headers '{}' does not match the agent subject '{}'", + public_key_trimmed, auth_vals.agent_subject + ) + .into()); + } + + // Any other agent subject must be a resource this store already + // holds. Never fetch it over the network during authentication: that + // is an unauthenticated SSRF, and the fetched body would be written + // into the store as a trusted resource. + let normalized_agent = store.normalize_subject(&agent_subject); + if !normalized_agent.is_local() { + return Err(format!( "Agent subject '{}' is hosted elsewhere and cannot be used to authenticate here; sign in with a did:ad:agent identity", auth_vals.agent_subject ) .into()); - } - let agent_resource = store.get_resource(&normalized_agent).await?; - let found_public_key = agent_resource.get(urls::PUBLIC_KEY)?; - if !public_keys_match(found_public_key.to_string().trim(), public_key_trimmed) { - Err( - "The public key in the auth headers does not match the public key in the agent" - .to_string() - .into(), - ) - } else { - Ok(ForAgent::AgentSubject(agent_subject)) - } + } + let agent_resource = store.get_resource(&normalized_agent).await?; + let found_public_key = agent_resource.get(urls::PUBLIC_KEY)?; + if !public_keys_match(found_public_key.to_string().trim(), public_key_trimmed) { + Err( + "The public key in the auth headers does not match the public key in the agent" + .to_string() + .into(), + ) } else { - Ok(ForAgent::Public) + Ok(ForAgent::AgentSubject(agent_subject)) } } diff --git a/lib/src/collections.rs b/lib/src/collections.rs index f8e0d423f..8d631a6b3 100644 --- a/lib/src/collections.rs +++ b/lib/src/collections.rs @@ -253,33 +253,57 @@ pub struct Collection { } /// Sorts a vector or resources by some property. +/// +/// Resources without the property sort last in both directions. The +/// comparator is a total order: `sort_by` requires one, and since Rust 1.81 +/// it may panic on one that is not (the previous comparator answered +/// `Greater` for a missing value on either side and for equal values, so a +/// collection with two resources missing the property, or two equal ones, +/// could bring the request down through the `sort_by` query parameter). #[tracing::instrument(skip_all)] pub fn sort_resources( mut resources: ResourceCollection, sort_by: &str, sort_desc: bool, ) -> ResourceCollection { - resources.sort_by(|a, b| { - let val_a = a.get(sort_by); - let val_b = b.get(sort_by); - if val_a.is_err() || val_b.is_err() { - return std::cmp::Ordering::Greater; - }; - if val_b.unwrap().to_string() > val_a.unwrap().to_string() { + resources.sort_by(|a, b| match (a.get(sort_by).ok(), b.get(sort_by).ok()) { + (None, None) => std::cmp::Ordering::Equal, + (None, Some(_)) => std::cmp::Ordering::Greater, + (Some(_), None) => std::cmp::Ordering::Less, + (Some(val_a), Some(val_b)) => { + let ordering = compare_sort_values(val_a, val_b); if sort_desc { - std::cmp::Ordering::Greater + ordering.reverse() } else { - std::cmp::Ordering::Less + ordering } - } else if sort_desc { - std::cmp::Ordering::Less - } else { - std::cmp::Ordering::Greater } }); resources } +/// Total order over two present values. Numbers sort before everything else +/// and compare numerically (`f64::total_cmp`, so `NaN` has a fixed place +/// instead of comparing as neither smaller nor larger); everything else +/// compares as its string form, which is what the collection sorted by +/// before. The kind is compared first because mixing the two rules within +/// one ordering is not transitive (`10 > 9`, `"9" > "5x"`, `"10" < "5x"`). +fn compare_sort_values(a: &Value, b: &Value) -> std::cmp::Ordering { + fn as_number(value: &Value) -> Option { + match value { + Value::Integer(i) | Value::Timestamp(i) => Some(*i as f64), + Value::Float(f) => Some(*f), + _ => None, + } + } + match (as_number(a), as_number(b)) { + (Some(a), Some(b)) => a.total_cmp(&b), + (Some(_), None) => std::cmp::Ordering::Less, + (None, Some(_)) => std::cmp::Ordering::Greater, + (None, None) => a.to_string().cmp(&b.to_string()), + } +} + impl Collection { /// Constructs a Collection, which is a paginated list of items with some sorting applied. /// Gets the required data from the store. @@ -1434,6 +1458,82 @@ mod test { ); } + /// The comparator must be a total order: `sort_by` may panic otherwise + /// (Rust 1.81+), and the `sort_by` query parameter reaches it with any + /// collection a client asks for. Mixed, equal, missing and NaN values are + /// the cases the old comparator got wrong. Thirty resources, not ten: + /// slices up to twenty elements are insertion-sorted, which never + /// notices a broken order, so a small collection hid the panic. + #[test] + fn sorting_mixed_missing_and_nan_values_is_a_total_order() { + let prop = urls::DESCRIPTION.to_string(); + let pattern = [ + Some(Value::Float(2.5)), + None, + Some(Value::Integer(1)), + Some(Value::Float(f64::NAN)), + Some(Value::String("b".into())), + None, + Some(Value::Integer(1)), + Some(Value::String("a".into())), + Some(Value::Float(f64::NAN)), + Some(Value::Integer(-3)), + ]; + let mut resources = Vec::new(); + for i in 0..30 { + let mut r = Resource::new(format!("r{i}")); + if let Some(v) = &pattern[i % pattern.len()] { + r.set_unsafe(prop.clone(), v.clone()).unwrap(); + } + resources.push(r); + } + + // Panicked with the old comparator ("user-provided comparison + // function does not correctly implement a total order"). + let asc = sort_resources(resources.clone(), &prop, false); + let desc = sort_resources(resources, &prop, true); + + // 24 resources carry the property, 6 do not. Missing values sit at + // the end, whichever direction was asked for. + let present = 24; + for sorted in [&asc, &desc] { + assert_eq!(sorted.len(), 30); + assert!(sorted[..present].iter().all(|r| r.get(&prop).is_ok())); + assert!(sorted[present..].iter().all(|r| r.get(&prop).is_err())); + } + + let shown = |sorted: &Vec| -> Vec { + sorted[..present] + .iter() + .map(|r| match r.get(&prop).unwrap() { + Value::Integer(i) => i.to_string(), + Value::Float(f) if f.is_nan() => "NaN".into(), + Value::Float(f) => f.to_string(), + other => other.to_string(), + }) + .collect() + }; + // Numbers compare numerically (-3 < 1 == 1 < 2.5 whatever their + // string forms say), NaN has a fixed place after them, and numbers + // sort before strings so the two kinds never interleave (which would + // make the order depend on which pair happened to be compared). + let mut expected_asc: Vec = Vec::new(); + for (value, count) in [ + ("-3", 3), + ("1", 6), + ("2.5", 3), + ("NaN", 6), + ("a", 3), + ("b", 3), + ] { + expected_asc.extend(std::iter::repeat_n(value.to_string(), count)); + } + assert_eq!(shown(&asc), expected_asc); + let mut expected_desc = expected_asc; + expected_desc.reverse(); + assert_eq!(shown(&desc), expected_desc); + } + /// Verifies that resources with DID subjects (`did:ad:...`) are correctly indexed and /// returned by sorted queries. This simulates the chatroom refresh scenario where messages /// have DID subjects but must appear when the chatroom queries by parent + sort by createdAt. diff --git a/lib/src/commit.rs b/lib/src/commit.rs index 8bd988fae..de013d1d1 100644 --- a/lib/src/commit.rs +++ b/lib/src/commit.rs @@ -362,7 +362,17 @@ impl Commit { } /// Check if the Commit's signature matches the signer's public key. + /// + /// Every failure is an authentication failure (a server answers 401): + /// there is no signature, the signer is unknown here, its key is not a + /// key, or the signature does not verify. pub async fn validate_signature(&self, store: &impl Storelike) -> AtomicResult<()> { + self.check_signature(store) + .await + .map_err(crate::errors::AtomicError::into_unauthorized) + } + + async fn check_signature(&self, store: &impl Storelike) -> AtomicResult<()> { let commit = self; let signature = match commit.signature.as_ref() { Some(sig) => sig, diff --git a/lib/src/db/website.rs b/lib/src/db/website.rs index 4fa0108ec..e6202f357 100644 --- a/lib/src/db/website.rs +++ b/lib/src/db/website.rs @@ -399,7 +399,7 @@ mod tests { let manifest = db.website_verified_blob(&deployment).await.unwrap(); assert_eq!(manifest, serde_json::to_vec(&package.manifest()).unwrap()); assert_eq!( - db.website_verified_blob(&blake3::hash(b"Hello").to_hex().to_string()) + db.website_verified_blob(blake3::hash(b"Hello").to_hex().as_str()) .await .unwrap(), b"Hello" diff --git a/lib/src/errors.rs b/lib/src/errors.rs index 9c01f377d..23faa04e0 100644 --- a/lib/src/errors.rs +++ b/lib/src/errors.rs @@ -67,6 +67,22 @@ impl AtomicError { } } + /// This error, typed as an authentication or authorization failure (a + /// server answers 401), unless it already is one. For the paths that + /// decide who a caller is, where every failure means "not you": a + /// signature that does not verify, a key that does not match the + /// subject, an agent this store does not know. The message is kept. + pub fn into_unauthorized(self) -> AtomicError { + match self.error_type { + AtomicErrorType::UnauthorizedError => self, + _ => AtomicError { + message: format!("Unauthorized. {}", self.message), + error_type: AtomicErrorType::UnauthorizedError, + subject: self.subject, + }, + } + } + /// A server will probably return a 500. pub fn other_error(message: String) -> AtomicError { AtomicError { diff --git a/lib/src/parse.rs b/lib/src/parse.rs index 1823020ff..d26099888 100644 --- a/lib/src/parse.rs +++ b/lib/src/parse.rs @@ -102,7 +102,8 @@ pub async fn parse_json_ad_resource( store: &impl crate::Storelike, parse_opts: &ParseOpts, ) -> AtomicResult { - let json: Map = serde_json::from_str(string)?; + let json: Map = serde_json::from_str(string) + .map_err(|e| AtomicError::parse_error(&format!("Invalid JSON: {}", e), None, None))?; parse_json_ad_map_to_resource(json, store, None, parse_opts).await } @@ -334,21 +335,28 @@ pub async fn parse_json_ad_commit_resource( string: &str, store: &impl crate::Storelike, ) -> AtomicResult { - let mut json: Map = serde_json::from_str(string)?; + // A body that is not a commit is the client's mistake, so these are parse + // errors (a server answers 400), not internal ones. + let mut json: Map = serde_json::from_str(string) + .map_err(|e| AtomicError::parse_error(&format!("Invalid JSON: {}", e), None, None))?; + let malformed = |message: &str| AtomicError::parse_error(message, None, None); // Get the signature - this is required for all commits let signature = json .get(urls::SIGNATURE) - .ok_or("No signature field in Commit.")? + .ok_or_else(|| malformed("No signature field in Commit."))? .as_str() - .ok_or("Signature must be a string")? + .ok_or_else(|| malformed("Signature must be a string"))? .to_string(); // Get or derive the subject. // For genesis commits the client omits the subject; it is always derived // from the signature as `did:ad:`. let _target_subject = match json.get(urls::SUBJECT) { - Some(subj) => subj.as_str().ok_or("Subject must be a string")?.to_string(), + Some(subj) => subj + .as_str() + .ok_or_else(|| malformed("Subject must be a string"))? + .to_string(), None => { let derived_subject = format!("did:ad:{}", signature); diff --git a/lib/src/sync/engine.rs b/lib/src/sync/engine.rs index 8e56d9c7a..72239dbcb 100644 --- a/lib/src/sync/engine.rs +++ b/lib/src/sync/engine.rs @@ -605,12 +605,6 @@ pub struct CommitIngestOpts { /// for peer replicas β€” hosting subjects the node does not own is what /// replication is. pub enforce_subject_ownership: bool, - /// Peer-transport semantics: hold the importing flag while applying so the - /// live push loop doesn't rebroadcast the commit back to live peers (the - /// sender included). Off on the hub, where WS fanout is suppressed - /// per-source via `source_id` and Iroh live peers SHOULD receive the - /// update. - pub suppress_live_echo: bool, /// Origin used to resolve `internal:/` subjects in the response JSON-AD. /// `None` falls back to the store's base domain. pub response_origin: Option, @@ -624,7 +618,6 @@ impl CommitIngestOpts { source_id, validate_loro_causality: true, enforce_subject_ownership: true, - suppress_live_echo: false, response_origin, } } @@ -632,15 +625,17 @@ impl CommitIngestOpts { /// Peer semantics: a signed commit from another full node over a peer /// transport. Signature, schema and rights still run; ownership and Loro /// causality do not (a replica hosts subjects it does not own, and - /// concurrent writes are expected), and the live push loop is muted so - /// the commit is not echoed back to the peers it came from. No - /// `source_id`: peers do not fan out through the commit monitor. + /// concurrent writes are expected). No `source_id` of its own: the + /// change is attributed to the peer whose [`ws_apply::import_scope`] + /// the caller runs this in, so the live push loop skips exactly that + /// peer and nothing else. + /// + /// [`ws_apply::import_scope`]: super::ws_apply::import_scope pub fn peer() -> Self { Self { source_id: None, validate_loro_causality: false, enforce_subject_ownership: false, - suppress_live_echo: true, response_origin: None, } } @@ -674,8 +669,9 @@ pub async fn verify_commit_signer( /// Signature, schema, and signer-rights validation always run β€” the commit is /// a self-authorizing certificate, so those checks (not a connection's AUTH /// identity) are the authority. What varies is domain-ownership enforcement, -/// Loro-causality enforcement, live-echo suppression, and source-id-based -/// echo suppression, all controlled by `opts`. +/// Loro-causality enforcement and source-id-based echo suppression, all +/// controlled by `opts` (a peer transport runs this inside +/// [`super::ws_apply::import_scope`] so the change is attributed to the peer). pub async fn ingest_commit_json( store: &Db, commit_json: &str, @@ -795,20 +791,17 @@ pub async fn ingest_commit( validate_loro_causality: opts.validate_loro_causality, validate_for_agent: Some(signer.to_string()), update_index: true, - source_id: opts.source_id.clone(), + // A commit applied inside a peer's import scope is attributed to that + // peer, so the live push loop does not send it straight back. Not a + // process-wide mute: a concurrent import on another connection, or a + // local edit, is broadcast as usual (security audit C16). + source_id: opts + .source_id + .clone() + .or_else(super::ws_apply::current_import_source), }; - let response = if opts.suppress_live_echo { - // Applying a remote peer's commit must not rebroadcast to live peers - // (the sender included) β€” the same mute the peer read loop holds - // around `persist_update`. - super::ws_apply::set_importing(true); - let result = store.apply_commit(incoming_commit, &commit_opts).await; - super::ws_apply::set_importing(false); - result - } else { - store.apply_commit(incoming_commit, &commit_opts).await - }?; + let response = store.apply_commit(incoming_commit, &commit_opts).await?; if needs_agent_resource { let mut new_agent = crate::Resource::new_instance(crate::urls::AGENT, store).await?; diff --git a/lib/src/sync/peer.rs b/lib/src/sync/peer.rs index 391bc32e8..33641a91a 100644 --- a/lib/src/sync/peer.rs +++ b/lib/src/sync/peer.rs @@ -342,6 +342,17 @@ use std::sync::{LazyLock, Mutex}; /// still belongs to the connection asking to remove it. static LIVE_PEERS: LazyLock>> = LazyLock::new(Default::default); +/// The live peer map, whether or not a previous holder panicked. The map only +/// holds channel senders and connection handles, which a panic cannot leave +/// half-updated in a way that matters, so a poisoned lock is recovered rather +/// than propagated: unwrapping it here made one panic (in whichever task +/// happened to hold the lock) kill live broadcast for the rest of the process. +fn live_peers() -> std::sync::MutexGuard<'static, HashMap> { + LIVE_PEERS + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) +} + /// One live peer: the write loop's channel, the generation that installed /// it, and (dial side only) the QUIC connection kept alive for it. Dropping /// the entry drops the connection, so a re-dial replaces the link it @@ -387,15 +398,12 @@ pub fn live_peer_name(peer_id: &str) -> Option { /// Returns the number of currently connected live peers. pub fn live_peer_count() -> usize { - LIVE_PEERS.lock().map(|m| m.len()).unwrap_or(0) + live_peers().len() } /// Returns the node IDs of currently connected live peers. pub fn live_peer_ids() -> Vec { - LIVE_PEERS - .lock() - .map(|m| m.keys().cloned().collect()) - .unwrap_or_default() + live_peers().keys().cloned().collect() } type CommitWaiter = @@ -447,10 +455,7 @@ impl LivePeerCommitTransport { /// Whether the peer currently has a live link. pub fn is_live(&self) -> bool { - LIVE_PEERS - .lock() - .map(|m| m.contains_key(&self.peer)) - .unwrap_or(false) + live_peers().contains_key(&self.peer) } } @@ -480,11 +485,7 @@ impl crate::sync::outbox::CommitTransport for LivePeerCommitTransport { } }; - let Some(tx) = LIVE_PEERS - .lock() - .ok() - .and_then(|m| m.get(&self.peer).map(|p| p.tx.clone())) - else { + let Some(tx) = live_peers().get(&self.peer).map(|p| p.tx.clone()) else { forget(); return Err(transport_error(format!( "peer {} is not live", @@ -573,7 +574,8 @@ fn remove_live_peer_inner(peer_id: &str, generation: Option, notify: bool) // Take the entry out under the lock and drop it (connection included) // after; the superseding-connection case is excluded, so this never // closes a link a newer registration is using. - let removed = LIVE_PEERS.lock().ok().and_then(|mut map| { + let removed = { + let mut map = live_peers(); let is_current = match generation { Some(generation) => map.get(&key).is_some_and(|p| p.generation == generation), None => true, @@ -589,7 +591,7 @@ fn remove_live_peer_inner(peer_id: &str, generation: Option, notify: bool) } None } - }); + }; if let Some(peer) = removed { drop(peer); tracing::info!("[live] removed peer {}", &key[..key.len().min(12)]); @@ -608,7 +610,8 @@ fn last_event_ms() -> &'static std::sync::Mutex bool { super::ws_apply::is_importing() } @@ -655,7 +658,7 @@ fn own_agent_update_frame(store: &Db) -> Option> { /// snapshot indefinitely. fn send_live_update_wire_msg_except(msg: Vec, skip_peer: Option<&str>) { let mut dead_peers = Vec::new(); - let peers = LIVE_PEERS.lock().unwrap(); + let peers = live_peers(); { for (peer_id, LivePeer { generation, tx, .. }) in peers.iter() { if skip_peer.is_some_and(|skip| normalize_node_id(skip) == *peer_id) { @@ -747,10 +750,6 @@ fn start_live_sync(store: Db) { } }; - if super::ws_apply::is_importing() { - continue; - } - let subject_key = match &event { crate::DbEvent::Changed { subject, .. } | crate::DbEvent::Destroyed { subject, .. } => subject.pure_id(), @@ -974,7 +973,7 @@ fn register_live_peer( // Add to peer map β€” replace if already connected (incoming may supersede outgoing) let generation = LIVE_PEER_GENERATION.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + 1; let is_new_peer = { - let mut map = LIVE_PEERS.lock().unwrap(); + let mut map = live_peers(); let replacing = map.contains_key(&key); if replacing { tracing::info!( @@ -1362,48 +1361,29 @@ fn register_live_peer( ) .await { - // Hold the importing flag across EVERY live - // import so the push loop doesn't re-broadcast - // what we just received β€” an unconditional - // re-send of an identical snapshot ping-pongs - // between the two nodes. - // - // This used to apply only when the subject was - // this node's own agent, which suppressed the - // echo on exactly one side: the device whose - // agent it is stays quiet, the peer for whom it - // is a stranger's agent re-sends it, and a drive - // β€” nobody's own agent β€” echoes on both. Two - // idle nodes then traded ~8.6KB frames - // indefinitely (measured: 355 frames in 58s, - // ~50KB/s, for one agent resource and one - // drive). - // - // The WS announcer ignores this flag, so the - // local browser still sees the merged state. - // - // A global mute is the blunt version of this: - // `DbEvent` already carries `source_id`, so the - // push loop could instead skip only the peer the - // update came from, and never mute a concurrent - // local edit. That needs `persist_update` to - // take a source and the push loop to read it. // Attribute this write to the peer it came - // from. `add_resource_opts` reads it while the - // write is still on the stack and stamps it on - // the `DbEvent`, so the push loop can skip that - // one peer rather than muting every broadcast - // for the duration of an import. - super::ws_apply::set_import_source(Some(read_peer_id.clone())); - super::ws_apply::set_importing(true); - let _ = super::ws_apply::persist_update( - &store, - &decoded.subject, - resolved, + // from: `add_resource_opts` reads the import + // scope while the write is still on the stack + // and stamps it on the `DbEvent`, so the push + // loop skips exactly that peer. Without it an + // unconditional re-send of an identical + // snapshot ping-pongs between the two nodes + // (measured: 355 frames in 58s, ~50KB/s, for + // one agent resource and one drive). The scope + // is per task, so another connection's import + // or a concurrent local edit is never muted by + // this one (security audit C16). The WS + // announcer ignores it, so the local browser + // still sees the merged state. + let _ = super::ws_apply::import_scope( + Some(read_peer_id.clone()), + super::ws_apply::persist_update( + &store, + &decoded.subject, + resolved, + ), ) .await; - super::ws_apply::set_importing(false); - super::ws_apply::set_import_source(None); tracing::trace!( "[live] imported update for {} from {}", &decoded.subject[..decoded.subject.len().min(20)], @@ -1435,8 +1415,16 @@ fn register_live_peer( // Public here would silently downgrade every fallback-routed // frame regardless of the AUTH this connection already completed // (or a later AUTH mid-session, which this call can apply). + // + // Inside this connection's import scope, so a `COMMIT` the engine + // applies is attributed to this peer and the push loop does not + // send it straight back (see `ws_apply::import_scope`). let agent_before_frame = agent.clone(); - let responses = super::engine::handle_frame(&buf, &store, &mut agent).await; + let responses = super::ws_apply::import_scope( + Some(read_peer_id.clone()), + super::engine::handle_frame(&buf, &store, &mut agent), + ) + .await; invalidate_drive_cache_on_identity_change( &agent, &agent_before_frame, @@ -3952,7 +3940,7 @@ mod live_peer_registry_tests { let generation = LIVE_PEER_GENERATION.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + 1; let (tx, _rx) = tokio::sync::mpsc::channel(4); - LIVE_PEERS.lock().unwrap().insert( + live_peers().insert( normalize_node_id(key), LivePeer { generation, @@ -3965,10 +3953,7 @@ mod live_peer_registry_tests { } fn is_registered(key: &str) -> bool { - LIVE_PEERS - .lock() - .unwrap() - .contains_key(&normalize_node_id(key)) + live_peers().contains_key(&normalize_node_id(key)) } /// A reconnect installs a new connection under the same node id, and the diff --git a/lib/src/sync/tests.rs b/lib/src/sync/tests.rs index c2945cb45..a27af8446 100644 --- a/lib/src/sync/tests.rs +++ b/lib/src/sync/tests.rs @@ -1980,14 +1980,12 @@ mod peer_sync_tests { source_id: None, validate_loro_causality: true, enforce_subject_ownership: true, - suppress_live_echo: false, response_origin: None, }, CommitIngestOpts { source_id: None, validate_loro_causality: false, enforce_subject_ownership: false, - suppress_live_echo: true, response_origin: None, }, ] @@ -2133,7 +2131,6 @@ mod peer_sync_tests { source_id: None, validate_loro_causality: true, enforce_subject_ownership: true, - suppress_live_echo: false, response_origin: None, }; let hub_err = ingest_commit_json(&db, &commit_json, &hub_opts) @@ -2149,7 +2146,6 @@ mod peer_sync_tests { source_id: None, validate_loro_causality: false, enforce_subject_ownership: false, - suppress_live_echo: true, response_origin: None, }; // With the gate off, the outcome must differ from the hub case above: diff --git a/lib/src/sync/ws_apply.rs b/lib/src/sync/ws_apply.rs index 4bf8b0f30..561ba58fb 100644 --- a/lib/src/sync/ws_apply.rs +++ b/lib/src/sync/ws_apply.rs @@ -3,41 +3,54 @@ //! //! Shared by Iroh live sync (`peer.rs`) and native WS sync sessions. -use std::sync::atomic::{AtomicBool, Ordering}; +use std::future::Future; use crate::{db::Db, errors::AtomicResult, Storelike}; -static IMPORTING: AtomicBool = AtomicBool::new(false); - -/// True while applying remote data (suppresses live-sync echo). -pub fn is_importing() -> bool { - IMPORTING.load(Ordering::Relaxed) +/// What an import in progress knows about where its data came from. +#[derive(Clone, Debug, Default)] +struct ImportScope { + /// The peer the data arrived from, when the transport can name one. + source: Option, } -pub(crate) fn set_importing(v: bool) { - IMPORTING.store(v, Ordering::Relaxed); +tokio::task_local! { + /// The import the current task is applying, if any. + /// + /// A task-local rather than a process-wide flag (security audit C16): + /// every peer connection runs its own read loop, and with one global + /// `AtomicBool` the first connection to finish cleared the flag while a + /// second was still mid-import, and one connection's peer id was stamped + /// on writes made for another. The scope now travels with the task that + /// does the import, so concurrent connections cannot observe each other. + static IMPORT_SCOPE: ImportScope; } -/// The peer an import is currently being applied from, if any. +/// Run `f` as an import from `source` (the peer id, when the transport knows +/// one). Inside it, [`is_importing`] is true and [`current_import_source`] +/// returns `source`, for this task only. /// -/// Read synchronously by the write that emits `DbEvent::Changed`, so the event -/// carries the peer it came from. That is what makes echo suppression -/// deterministic: the alternative β€” a bool the live push loop checks when it -/// eventually processes the event β€” cannot work, because the push loop is a -/// separate task consuming a broadcast channel and may not be scheduled until -/// after the flag is cleared. Two idle nodes then trade the same snapshot -/// forever (see `peer.rs`'s live read loop). -static IMPORT_SOURCE: std::sync::Mutex> = std::sync::Mutex::new(None); - -pub(crate) fn set_import_source(source: Option) { - if let Ok(mut guard) = IMPORT_SOURCE.lock() { - *guard = source; - } +/// The write that emits `DbEvent::Changed` reads the source synchronously, +/// while the import is still on the stack, so the event carries the peer it +/// came from. That is what makes echo suppression deterministic: a flag the +/// live push loop checks when it eventually processes the event cannot work, +/// because the push loop is a separate task consuming a broadcast channel and +/// may not be scheduled until after the import has finished. Two idle nodes +/// then trade the same snapshot forever (see `peer.rs`'s live read loop). +pub async fn import_scope(source: Option, f: F) -> F::Output { + IMPORT_SCOPE.scope(ImportScope { source }, f).await +} + +/// True while the current task is applying remote data (see [`import_scope`]). +pub fn is_importing() -> bool { + IMPORT_SCOPE.try_with(|_| ()).is_ok() } -/// The peer id to attribute a write to, for echo suppression. +/// The peer id to attribute a write made by the current task to, for echo +/// suppression. `None` outside an import, or inside one from a transport +/// that names no peer. pub fn current_import_source() -> Option { - IMPORT_SOURCE.lock().ok().and_then(|g| g.clone()) + IMPORT_SCOPE.try_with(|s| s.source.clone()).ok().flatten() } /// Import a remote UPDATE frame into the local store. Trusted callers only β€” @@ -46,16 +59,13 @@ pub fn current_import_source() -> Option { /// established (i.e. Iroh's live loop) must use [`resolve_update`] + /// [`persist_update`] instead, so a check can run before anything is written. pub async fn apply_state_update(store: &Db, subject: &str, state_bytes: &[u8]) -> AtomicResult<()> { - set_importing(true); - let result = async { + import_scope(None, async { if let Some(resolved) = resolve_update(store, subject, state_bytes).await { persist_update(store, subject, resolved).await?; } Ok(()) - } - .await; - set_importing(false); - result + }) + .await } /// A merged-in-memory UPDATE, not yet persisted. Lets the caller resolve the @@ -227,10 +237,7 @@ pub async fn apply_destroy(store: &Db, subject: &str) -> AtomicResult<()> { return Ok(()); } - set_importing(true); - let result = apply_destroy_unchecked(store, subject).await; - set_importing(false); - result + import_scope(None, apply_destroy_unchecked(store, subject)).await } async fn apply_destroy_unchecked(store: &Db, subject: &str) -> AtomicResult<()> { @@ -469,3 +476,104 @@ mod destroy_phantom_tombstone_tests { assert!(tombstones::is_tombstoned(&db, subject)); } } + +#[cfg(test)] +mod import_scope_tests { + use super::*; + use crate::loro::AtomicLoroDoc; + use crate::values::Value; + + /// Security audit C16: the import flag and the import source were + /// process-global, so two peer connections importing at the same time + /// observed each other: the first to finish cleared the flag for the + /// second, and a write made for peer A could be stamped with peer B's id. + /// The scope is per task now; two imports that interleave at every await + /// each see only their own source, and the task outside sees none. + #[tokio::test] + async fn concurrent_imports_do_not_observe_each_other() { + async fn import_from(peer: &str) -> Vec> { + import_scope(Some(peer.to_string()), async move { + let mut seen = Vec::new(); + for _ in 0..8 { + // Hand control to the other import between reads, so the + // reads below are taken while the other task is mid-import. + tokio::task::yield_now().await; + assert!(is_importing(), "inside the scope of {peer}"); + seen.push(current_import_source()); + } + seen + }) + .await + } + + assert!(!is_importing(), "no import is running in this task"); + assert_eq!(current_import_source(), None); + + let a = tokio::spawn(import_from("peer-a")); + let b = tokio::spawn(import_from("peer-b")); + let (a, b) = (a.await.unwrap(), b.await.unwrap()); + + assert!(a.iter().all(|s| s.as_deref() == Some("peer-a")), "{a:?}"); + assert!(b.iter().all(|s| s.as_deref() == Some("peer-b")), "{b:?}"); + assert!( + !is_importing(), + "finishing an import leaves this task alone" + ); + assert_eq!(current_import_source(), None); + } + + /// The scope reaches the write: two interleaved imports of two subjects + /// each emit a `DbEvent::Changed` attributed to their own peer, which is + /// what the live push loop uses to skip exactly the peer an update came + /// from. + #[tokio::test] + async fn concurrent_imports_stamp_their_own_source_on_the_event() { + let db = Db::init_temp("import_scope_two_peers").await.unwrap(); + let (_alice, drive) = db.setup("Alice").await.unwrap(); + let mut events = db.subscribe_events(); + + let mut imports = Vec::new(); + for (peer, subject) in [ + ("peer-a", "did:ad:import-scope-a"), + ("peer-b", "did:ad:import-scope-b"), + ] { + let doc = AtomicLoroDoc::new(); + doc.set_property(crate::urls::PARENT, &Value::AtomicUrl(drive.clone().into())) + .unwrap(); + doc.set_property(crate::urls::NAME, &Value::String(format!("from {peer}"))) + .unwrap(); + let bytes = doc.export_snapshot(); + let db = db.clone(); + imports.push(tokio::spawn(import_scope( + Some(peer.to_string()), + async move { + tokio::task::yield_now().await; + let resolved = resolve_update(&db, subject, &bytes).await.unwrap(); + tokio::task::yield_now().await; + persist_update(&db, subject, resolved).await.unwrap(); + }, + ))); + } + for import in imports { + import.await.unwrap(); + } + + let mut attributed = std::collections::HashMap::new(); + while let Ok(event) = events.try_recv() { + if let crate::DbEvent::Changed { + subject, source_id, .. + } = event + { + attributed.insert(subject.pure_id(), source_id); + } + } + assert_eq!( + attributed.get("did:ad:import-scope-a"), + Some(&Some("peer-a".to_string())) + ); + assert_eq!( + attributed.get("did:ad:import-scope-b"), + Some(&Some("peer-b".to_string())) + ); + } +} diff --git a/planning/production-readiness.md b/planning/production-readiness.md index 3dfcf0f10..86a220b46 100644 --- a/planning/production-readiness.md +++ b/planning/production-readiness.md @@ -20,8 +20,9 @@ owning plan where one exists. Verification logs stay out of this file. - [x] Rate limiting on write endpoints (`/commit`, `/upload`, `/blob`, `/iroh-sync`, WS `COMMIT`): `server/src/rate_limit.rs`, 2026-09-15. - Still open from the same audit line: permissive CORS, client errors - answered as 500 ([`security-audit-2026-09.md`](./security-audit-2026-09.md) D). +- [x] Credentialed CORS only for the server's own origins (`server/src/cors.rs`) + and client errors answered as 400/401 instead of 500, 2026-09-18 + ([`security-audit-2026-09.md`](./security-audit-2026-09.md) D, H). - [x] Library-owned durable flush so the Flutter binding stops losing writes on app kill ([`atomic-lib-runtime.md`](./atomic-lib-runtime.md), 2026-09-15). - [x] Desktop CSP (audit B7) set 2026-09-15; needs one packaged-build smoke test before a release. diff --git a/planning/security-audit-2026-09.md b/planning/security-audit-2026-09.md index a7cc70855..5648413eb 100644 --- a/planning/security-audit-2026-09.md +++ b/planning/security-audit-2026-09.md @@ -92,11 +92,11 @@ Fix: for `/agents/{key}` subjects, require the key in the path to equal the auth - **C13. Panic-on-`unwrap` in the auth path with attacker-controlled input.** Verified. `helpers.rs:60-71` `origin()` unwraps `Uri` parse, scheme and authority; reachable via `/ws` Bearer tokens and via `X-Forwarded-Host: a b`. - **C14. Auth signature fallback accepts a path-only message.** Verified. `authentication.rs:51-87`; a signature over `"/ "` is valid on every host and tenant for 5 minutes and bypasses the WS challenge binding for clients that sign that way. - **C15. TLS certificates only checked at startup; ACME code panics (`todo!()`, `unwrap`).** Confirmed. `serve.rs:474-479`, `https.rs:64-86,249,269`. -- **C16. Process-global import flags race across connections.** Confirmed. `ws_apply.rs:10,30`, `peer.rs:1186-1195`. +- **C16. Process-global import flags race across connections.** Confirmed. `ws_apply.rs:10,30`, `peer.rs:1186-1195`. **Fixed 2026-09-18:** see H. - **C17. DID resources leak into other drives' watched queries.** Likely. `query_index.rs:479-482,382-445`. Read rights still apply at query time. - **C18. Every commit is written with `Durability::None`.** Confirmed. `redb_store.rs:477,567` (`// EXPERIMENT`). Acknowledged writes are lost on crash until the next flush. **Amended 2026-09-15:** the server and desktop run a 100ms durable-flush tick (`serve.rs`) and the WASM worker flushes on a tick, so the loss window there is 100ms. The Flutter binding opened the same redb store directly and never flushed except in `set_active_drive`, so on Android every edit since the last drive switch was lost on app kill. **Fixed 2026-09-15:** `Db::init_redb_file` owns the flush tick, so every file-backed binding gets the same 100ms bound; idle ticks no longer write. See [`atomic-lib-runtime.md`](./atomic-lib-runtime.md). - **C19. Unbounded response body on the bookmark fetch.** Confirmed. `client/helpers.rs:355-366`, `bookmark.rs:47`; SSRF guard is present, size cap is not. -- **C20. Client-stamped `drive` trusted when the parent is not materialized.** Needs-verification. `commit.rs:242-256,876-893`; with B2 this may allow cross-drive fan-out injection. +- **C20. Client-stamped `drive` trusted when the parent is not materialized.** Verified 2026-09-18, not exploitable as described. The stamp is only ever a rights *lookup target*, never a grant: with the parent materialized, `check_append` checks `Append` on the parent and `commit.rs` re-derives the stamp from the parent, discarding the client's; with the parent absent, `hierarchy.rs` (`check_append`, `did:` branch) accepts the stamp only if the signer has `Append` on the stamped drive *and* that drive exists here, so stamping a victim's drive is refused and stamping your own drive gains nothing (fan-out routes to your own subscribers). Residual, hygiene rather than rights: a child created while its parent was absent keeps the stamp it was created with when the parent arrives later (`parent_changed` does not fire), so the drive stamp can disagree with the parent's drive until the resource is next moved. Not fixed here. - **C21. Agent private key written with default file mode.** Confirmed. `lib/src/config.rs:62-74` (`config.toml` with `agent_secret` lands 0644). - **C22. CI: `permissions: write-all` on `release.yml` with unpinned third-party actions; PAT persisted in `.git/config` on the self-hosted runner (`main.yml:230-237`, no `persist-credentials: false`, `clean: false`); deploy SSH uses `ssh-keyscan` TOFU every run (`.dagger/src/index.ts:1816-1821`, `deployment.yml:114-116`); secrets expanded inline into `run:` (`tauri-release.yml:305-309`).** Confirmed. - **C23. Flutter bridge deletes the database on any open error.** Confirmed. `flutter/rust/src/api/simple.rs:143-151`. @@ -104,10 +104,10 @@ Fix: for `/agents/{key}` subjects, require the key in the path to equal the auth ## D. Low and Info (selected) -- `Cors::permissive()` with credentials (`serve.rs:390`); currently saved by `SameSite=Lax` on the client cookie. +- ~~`Cors::permissive()` with credentials (`serve.rs:390`); currently saved by `SameSite=Lax` on the client cookie.~~ Fixed 2026-09-18 (`server/src/cors.rs`): credentials only for the server's own origins, see H. - `atomic_session` cookie set without `Secure` (`browser/lib/src/authentication.ts:118-121`). - Unauthenticated WS `SUBSCRIBE_INDEX_STATUS` inserts per-connection state for arbitrary drive strings; `RBSR_FP` ranges unbounded (`web_sockets.rs:566-579,638-682`). ~~No rate limiting on `/commit`, `/upload`, `/iroh-sync`.~~ Fixed 2026-09-15 (`server/src/rate_limit.rs`, per-agent and per-peer budgets on every write endpoint and the WS `COMMIT` frame). **Amended 2026-09-18:** the HTTP `/commit` limiter keyed on the signer the body *named* before the signature was verified, so a forged commit naming a victim spent the victim's budget (targeted write lock-out, GHSA-f9rw-g3gr-3jw5, reported by hackchang). Fixed: `verify_commit_signer` runs first and the budget goes to the signer it proves; an unverifiable body spends the peer-address budget. The WS `COMMIT` frame was never affected: it is keyed on the connection's AUTH-verified agent. -- Auth failures and most client errors return HTTP 500 (`helpers.rs:183`, `errors.rs:119-137`), filling Sentry; `default_service` logs `error!` on every unmatched request (`serve.rs:405-408`). +- ~~Auth failures and most client errors return HTTP 500 (`helpers.rs:183`, `errors.rs:119-137`), filling Sentry;~~ Fixed 2026-09-18, see H. `default_service` logs `error!` on every unmatched request (`serve.rs:405-408`) β€” fixed in the second round (logs at debug). - `/plugin-ui` and `/plugin-list` unauthenticated, read with `ForAgent::Sudo`, echo filesystem paths (`plugin_ui.rs:443-521`). - `Agent` and `Config` derive `Debug` including the private key (`agents.rs:57-69`, `config.rs:129-142`). - Agent identity strings not base64-canonicalized: one key, several identities (`authentication.rs:139-171`, `hierarchy.rs:310-313`). @@ -150,7 +150,7 @@ JavaScript (`pnpm audit --prod` in `browser/`): 21 advisories, 11 high. Runtime- - `e2e/tests/drafts.spec.ts:22`, `resource-context-menu.spec.ts:150` `test.fixme`. **Panics on untrusted or user input** -- `lib/src/collections.rs:257-281` `sort_resources` comparator is not a total order; reachable via `sort_by` query param, may panic on Rust β‰₯1.81. +- ~~`lib/src/collections.rs:257-281` `sort_resources` comparator is not a total order; reachable via `sort_by` query param, may panic on Rust β‰₯1.81.~~ Fixed 2026-09-18. - `lib/src/agents.rs:212-215` `unwrap`/`expect` in `generate_public_key` on user-supplied secrets. - `lib/src/parse.rs:880,871,925` `unwrap` on `importer`/`resource_new`. - `lib/src/storelike.rs:513-521` `to_single()` panics on `Redirect`; reachable from `/path` for `did:ad:blob:` subjects (needs-verification). @@ -163,9 +163,9 @@ JavaScript (`pnpm audit --prod` in `browser/`): 21 advisories, 11 high. Runtime- - `lib/src/db.rs:1997,3711`, `db/migrations.rs:105-200` deliberate crash-on-corruption during full scans; `db.rs:3196-3201` production asserts. **Ignored results and swallowed errors** -- `server/src/handlers/upload.rs:47` `while let Ok(Some(field))` ends the loop silently on a multipart error and returns 200. +- ~~`server/src/handlers/upload.rs:47` `while let Ok(Some(field))` ends the loop silently on a multipart error and returns 200.~~ Fixed 2026-09-18. - `lib/src/envelopes.rs:181-189`, `lib/src/sync/engine.rs:1529` `let _ =` on store writes. -- `lib/src/sync/peer.rs:501,820,2079` `LIVE_PEERS.lock().unwrap()`: one panic kills live broadcast permanently. +- ~~`lib/src/sync/peer.rs:501,820,2079` `LIVE_PEERS.lock().unwrap()`: one panic kills live broadcast permanently.~~ Fixed 2026-09-18. **Dead, duplicated or misleading code** - `lib/src/commit.rs:1165-1167,1191-1194` `SIGNER` set twice. @@ -246,5 +246,13 @@ Not fixed here: - **C17** (DID resources in other drives' watched queries): a first version compared a resource's `drive` stamp with the query's `drive`, but for a `did:` query `QueryFilter.drive` is `drive_prefix_from_subject` of the queried subject, which is that subject itself rather than the drive root, so legitimate rows were excluded and the `query_aggregates` tests failed. Reverted. A correct fix has to carry the drive root in the filter (resolve the queried subject's `drive` stamp when the filter is registered) and only then compare stamps, or keep the constraint check as the scope and rely on read rights at query time, which still apply. - **B7** (desktop CSP disabled): a CSP is set since 2026-09-15 (`desktop/tauri.conf.json`: `script-src 'self' 'wasm-unsafe-eval'`, `style-src 'self' 'unsafe-inline'` plus Google Fonts for styled-components, `connect-src` to any http/https/ws/wss server plus the Tauri IPC origins, workers from the bundle or blobs, plugin iframes over http/https, `object-src 'none'`), with Tauri's own hashing of the bundle's inline scripts left on. It mirrors what the web build already runs under (`single_page_app.rs` nonce CSP) and was validated against the Tauri schema, not yet by a packaged desktop run; smoke-test a release build (sign-in, document editing, image upload, plugin view) before shipping. The `devtools` feature is opt-in. -- **C16** (process-global import flags), **C18** (`Durability::None`), **C24** (loopback NFS): structural changes with performance or design trade-offs, listed above with the fix direction. +- **C18** (`Durability::None`), **C24** (loopback NFS): structural changes with performance or design trade-offs, listed above with the fix direction. - The `stringToSlug` duplicate stays: the data-browser copy fixes a case (`Meat & fish`) the lib copy gets wrong; port the fix into lib first. + +Fixed in the security-hygiene round (2026-09-18, branch `claude/technical-debt-analysis-b4ifz2-security-hygiene`): + +- **D (CORS with credentials)**: `Cors::permissive()` stays β€” any origin may read, Atomic is a headless CMS β€” but a middleware outside it (`server/src/cors.rs`, `credentials_gate`) removes `Access-Control-Allow-Credentials` unless the `Origin` is one this server answers for: the configured domain, a subdomain of the base domain, loopback (which covers the Vite dev server on another port and `http://tauri.localhost`), or `tauri://localhost`. Same rule as B3's `host_is_served_here`. Tested against the real stack: a foreign origin gets `Allow-Origin` and no `Allow-Credentials`, an own origin gets both, preflight works for both. +- **D (client errors as 500)**: `AtomicErrorType::ParseError` maps to 400; `parse_json_ad_commit_resource` types a body that is not JSON or not a commit as a parse error; `Commit::validate_signature` and `get_agent_from_auth_values_and_check` type every failure as `Unauthorized`; the server's `get_client_agent` keeps that type instead of stringifying it. Tests post a non-JSON body (400), an unsigned commit (400), a tampered signature (401), malformed auth headers (401) and a bad auth signature (401) at real handlers. +- **C16**: `ws_apply::import_scope` is a tokio task-local carrying the peer id; `is_importing()` and `current_import_source()` read the current task's scope only. The live UPDATE import, the live `COMMIT` fallback dispatch and `apply_state_update`/`apply_destroy` run inside it, and `ingest_commit` attributes a commit to the enclosing scope's peer, so the `DbEvent` carries the peer and the push loop skips exactly that one. The push loop's global `is_importing()` check and `CommitIngestOpts::suppress_live_echo` (whose only effect was that global mute) are gone. Trade-off accepted: a peer's commit is now relayed to *other* live peers (as bulk `SYNC_PUSH` imports always were) instead of being muted for everyone during the import; the sender is skipped deterministically. Test: two interleaved imports each see only their own scope, and each stamps its own source on its `DbEvent`. +- **F**: `sort_resources` is a total order; `/upload` returns 400 on a multipart error; every `LIVE_PEERS` access goes through a poison-recovering accessor. +- **C20**: verified, see C; not exploitable via the stamp, residual noted there. diff --git a/server/src/bin.rs b/server/src/bin.rs index 0369dfeef..06da6c1ee 100644 --- a/server/src/bin.rs +++ b/server/src/bin.rs @@ -9,6 +9,7 @@ mod commit_monitor; pub mod config; mod content_types; mod context; +mod cors; mod errors; mod handlers; mod helpers; diff --git a/server/src/context.rs b/server/src/context.rs index c784f563f..57baf82da 100644 --- a/server/src/context.rs +++ b/server/src/context.rs @@ -50,7 +50,7 @@ impl RequestContext { /// default domain (`localhost`) has not told us its public name, so it trusts /// the header as before; that is the unconfigured dev/desktop case, which is /// reached over the network only when the operator chose to. -fn host_is_served_here(host: &str, opts: &crate::config::Opts) -> bool { +pub(crate) fn host_is_served_here(host: &str, opts: &crate::config::Opts) -> bool { let hostname = strip_port(host).to_ascii_lowercase(); if hostname.is_empty() || hostname diff --git a/server/src/cors.rs b/server/src/cors.rs new file mode 100644 index 000000000..2bbaa5b61 --- /dev/null +++ b/server/src/cors.rs @@ -0,0 +1,268 @@ +//! Cross-origin access in two tiers (security audit D, "permissive CORS with +//! credentials"). +//! +//! Atomic is a headless CMS: a page on any web origin may read public data +//! from it, so every origin gets `Access-Control-Allow-Origin`, any method and +//! any header. What used to come with that was `Access-Control-Allow-Credentials` +//! for every origin too, which tells the browser it may attach the +//! `atomic_session` cookie to a request made by any site. The cookie's own +//! `SameSite=Lax` was the only thing preventing that. +//! +//! The clients that need credentials are all on origins this server answers +//! for: the data-browser served by the server itself, a tenant on a subdomain +//! of the base domain, the Vite dev server on another loopback port, and the +//! desktop and Android webviews (`tauri://localhost` on macOS and iOS, +//! `http://tauri.localhost` on Windows and Android), which talk to a server +//! bound to loopback. So credentials are allowed for exactly those, by the +//! same rule `RequestContext` uses to decide which `Host` to trust, and every +//! other origin keeps its any-origin access without them. + +use actix_cors::Cors; +use actix_web::{ + body::MessageBody, + dev::{ServiceRequest, ServiceResponse}, + http::header, + middleware::Next, + web, Error, +}; + +use crate::{appstate::AppState, config::Opts}; + +/// The CORS layer: any origin, any method, any header, credentials. It is +/// wrapped by [`credentials_gate`], which takes the credentials back for +/// origins that are not this server's own. +pub(crate) fn any_origin() -> Cors { + Cors::permissive().expose_headers([crate::serve::SERVER_VERSION_HEADER]) +} + +/// Whether a browser on `origin` (the `Origin` header, `scheme://host[:port]`) +/// may send the session cookie along: the configured domain, a subdomain of +/// the multi-tenant base domain, loopback (which covers `tauri.localhost`), or +/// the desktop webview's own `tauri://localhost` scheme. +pub(crate) fn origin_may_send_credentials(origin: &str, opts: &Opts) -> bool { + let Some((scheme, host)) = origin.split_once("://") else { + // `null` (a sandboxed frame, a `file:` page) and anything else that + // is not an origin with a host. + return false; + }; + if host.is_empty() || host.contains('/') { + return false; + } + match scheme.to_ascii_lowercase().as_str() { + "http" | "https" => crate::context::host_is_served_here(host, opts), + "tauri" => host.eq_ignore_ascii_case("localhost"), + _ => false, + } +} + +/// Removes `Access-Control-Allow-Credentials` from the response unless the +/// request's `Origin` is one this server answers for. Register it outside +/// the CORS layer (`.wrap(cors).wrap(from_fn(credentials_gate))`) so it sees +/// the headers that layer added, on preflight answers and on real responses +/// alike. A request without an `Origin` is not a cross-origin request, so +/// the header (which `actix-cors` adds regardless) grants nothing there and +/// is removed too. +pub(crate) async fn credentials_gate( + req: ServiceRequest, + next: Next, +) -> Result, Error> { + let trusted = match ( + req.headers().get(header::ORIGIN).map(|o| o.to_str()), + req.app_data::>(), + ) { + (Some(Ok(origin)), Some(appstate)) => { + origin_may_send_credentials(origin, &appstate.config.opts) + } + // Fail closed: no origin, an origin that is not even a string, or no + // policy to consult, gets no credentials. + _ => false, + }; + let mut res = next.call(req).await?; + if !trusted { + res.headers_mut() + .remove(header::ACCESS_CONTROL_ALLOW_CREDENTIALS); + } + Ok(res) +} + +#[cfg(test)] +mod tests { + use super::*; + // `actix_web::test` under its own name would shadow the `#[test]` + // attribute for the unit tests below. + use actix_web::{ + http::{Method, StatusCode}, + middleware, test as actix_test, App, HttpResponse, + }; + use clap::Parser; + + fn opts(domain: &str, base: Option<&str>) -> Opts { + let mut args = vec!["atomic-server", "--domain", domain]; + if let Some(b) = base { + args.push("--base-domain"); + args.push(b); + } + Opts::parse_from(args) + } + + #[test] + fn own_origins_may_send_credentials() { + let o = opts("atomicdata.dev", Some("atomicserver.eu")); + for origin in [ + "https://atomicdata.dev", + "https://AtomicData.dev:443", + "https://joep.atomicserver.eu", + "http://localhost:6747", + "http://127.0.0.1:9883", + "http://[::1]:9883", + "http://atomic.localhost:9883", + "tauri://localhost", + "http://tauri.localhost", + ] { + assert!(origin_may_send_credentials(origin, &o), "{origin}"); + } + } + + #[test] + fn foreign_origins_may_not() { + let o = opts("atomicdata.dev", Some("atomicserver.eu")); + for origin in [ + "https://evil.example", + "https://atomicdata.dev.evil.example", + "https://notatomicserver.eu", + "null", + "", + "atomicdata.dev", + "https://atomicdata.dev/path", + "tauri://evil.example", + "chrome-extension://abcdefghijklmnop", + "ftp://atomicdata.dev", + ] { + assert!(!origin_may_send_credentials(origin, &o), "{origin:?}"); + } + } + + /// The full stack as `serve.rs` registers it: permissive CORS with the + /// credentials gate outside it, against a server configured for + /// `atomicdata.dev`. A foreign origin gets any-origin access without + /// credentials, an own origin gets both, on preflight and on a real + /// request. + #[actix_rt::test] + async fn foreign_origin_reads_without_credentials_own_origin_with() { + let appstate = crate::tests::init_test_appstate(&[ + "--domain", + "atomicdata.dev", + "--base-domain", + "atomicdata.dev", + ]) + .await; + let app = actix_test::init_service( + App::new() + .app_data(web::Data::new(appstate)) + .wrap(any_origin()) + .wrap(middleware::from_fn(credentials_gate)) + .route( + "/ping", + web::get().to(|| async { HttpResponse::Ok().body("pong") }), + ), + ) + .await; + + let foreign = "https://evil.example"; + let own = "https://atomicdata.dev"; + let tenant = "https://joep.atomicdata.dev"; + let desktop = "tauri://localhost"; + + // Real requests. + for (origin, credentials) in [ + (foreign, false), + (own, true), + (tenant, true), + (desktop, true), + ] { + let req = actix_test::TestRequest::get() + .uri("/ping") + .insert_header((header::ORIGIN, origin)) + .to_request(); + let resp = actix_test::call_service(&app, req).await; + assert_eq!(resp.status(), StatusCode::OK, "{origin}"); + let headers = resp.headers(); + assert_eq!( + headers + .get(header::ACCESS_CONTROL_ALLOW_ORIGIN) + .and_then(|v| v.to_str().ok()), + Some(origin), + "{origin}: any origin may read" + ); + assert_eq!( + headers + .get(header::ACCESS_CONTROL_ALLOW_CREDENTIALS) + .and_then(|v| v.to_str().ok()), + credentials.then_some("true"), + "{origin}: credentials only for our own origins" + ); + assert!( + headers + .get(header::ACCESS_CONTROL_EXPOSE_HEADERS) + .and_then(|v| v.to_str().ok()) + .is_some_and(|v| v.eq_ignore_ascii_case(crate::serve::SERVER_VERSION_HEADER)), + "{origin}: the version header stays exposed" + ); + } + + // Preflights. + for (origin, credentials) in [(foreign, false), (own, true)] { + let req = actix_test::TestRequest::default() + .method(Method::OPTIONS) + .uri("/ping") + .insert_header((header::ORIGIN, origin)) + .insert_header((header::ACCESS_CONTROL_REQUEST_METHOD, "POST")) + .insert_header(( + header::ACCESS_CONTROL_REQUEST_HEADERS, + "content-type, x-atomic-signature", + )) + .to_request(); + let resp = actix_test::call_service(&app, req).await; + assert_eq!(resp.status(), StatusCode::OK, "{origin}: preflight passes"); + let headers = resp.headers(); + assert_eq!( + headers + .get(header::ACCESS_CONTROL_ALLOW_ORIGIN) + .and_then(|v| v.to_str().ok()), + Some(origin), + "{origin}" + ); + assert!( + headers + .get(header::ACCESS_CONTROL_ALLOW_METHODS) + .is_some_and(|v| v.to_str().unwrap_or_default().contains("POST")), + "{origin}: POST is allowed" + ); + assert!( + headers + .get(header::ACCESS_CONTROL_ALLOW_HEADERS) + .is_some_and(|v| v + .to_str() + .unwrap_or_default() + .contains("x-atomic-signature")), + "{origin}: the signing headers are allowed" + ); + assert_eq!( + headers + .get(header::ACCESS_CONTROL_ALLOW_CREDENTIALS) + .and_then(|v| v.to_str().ok()), + credentials.then_some("true"), + "{origin}: credentials only for our own origins" + ); + } + + // Not a cross-origin request: nothing to grant. + let req = actix_test::TestRequest::get().uri("/ping").to_request(); + let resp = actix_test::call_service(&app, req).await; + assert_eq!(resp.status(), StatusCode::OK); + assert!(resp + .headers() + .get(header::ACCESS_CONTROL_ALLOW_CREDENTIALS) + .is_none()); + } +} diff --git a/server/src/errors.rs b/server/src/errors.rs index b5e2ad85d..a63515632 100644 --- a/server/src/errors.rs +++ b/server/src/errors.rs @@ -126,7 +126,10 @@ impl From for AtomicServerError { atomic_lib::AtomicErrorType::NotFoundError => AppErrorType::NotFound, atomic_lib::AtomicErrorType::UnauthorizedError => AppErrorType::Unauthorized, atomic_lib::AtomicErrorType::MethodNotAllowed => AppErrorType::MethodNotAllowed, - atomic_lib::AtomicErrorType::ParseError => AppErrorType::Other, + // A body the lib could not parse is the client's mistake, not a + // fault here: 400, so it neither reads as a crash to the caller + // nor reaches Sentry as one. + atomic_lib::AtomicErrorType::ParseError => AppErrorType::BadRequest, atomic_lib::AtomicErrorType::OtherError => AppErrorType::Other, }; let subject = error diff --git a/server/src/handlers/commit.rs b/server/src/handlers/commit.rs index a8a8d59da..c5332ec6f 100644 --- a/server/src/handlers/commit.rs +++ b/server/src/handlers/commit.rs @@ -57,7 +57,7 @@ pub async fn post_commit( /// Loro-causality are enforced, the WS commit monitor's `source_id` is /// threaded through for echo suppression, and live-peer fanout is left /// unsuppressed (that suppression is per-`source_id`, handled by the commit -/// monitor, not by the engine's importing flag). +/// monitor). pub async fn apply_commit_json( store: &Db, origin: &str, diff --git a/server/src/handlers/upload.rs b/server/src/handlers/upload.rs index 71b88afda..cf4203077 100644 --- a/server/src/handlers/upload.rs +++ b/server/src/handlers/upload.rs @@ -45,7 +45,17 @@ pub async fn upload_handler( let mut created_resources: Vec = Vec::new(); - while let Ok(Some(field)) = body.try_next().await { + // A multipart body that stops parsing (a truncated body, a missing + // boundary, a bad part header) is an error, not the end of the list: + // `while let Ok(Some(..))` used to end the loop quietly and answer 200 + // for the files that made it through, so a client could not tell a + // failed upload from a finished one. + while let Some(field) = body.try_next().await.map_err(|e| { + crate::errors::AtomicServerError::bad_request(format!( + "Error while reading multipart data. {}", + e + )) + })? { let mut resource = save_file_and_create_resource(field, &appstate, &query.parent, store, &origin).await?; resource.save(store).await?; @@ -82,7 +92,12 @@ async fn save_file_and_create_resource( // Field in turn is stream of *Bytes* object while let Some(chunk) = field.next().await { - let data = chunk.map_err(|e| format!("Error while reading multipart data. {}", e))?; + let data = chunk.map_err(|e| { + crate::errors::AtomicServerError::bad_request(format!( + "Error while reading multipart data. {}", + e + )) + })?; // `PayloadConfig` only bounds the `Bytes`/`String` extractors, not a // multipart stream read by hand, so bound it here. if buffer.len() + data.len() > crate::serve::PAYLOAD_MAX { diff --git a/server/src/helpers.rs b/server/src/helpers.rs index 8fb7c4bc2..21801747e 100644 --- a/server/src/helpers.rs +++ b/server/src/helpers.rs @@ -178,16 +178,29 @@ pub async fn get_client_agent( return Ok(ForAgent::Public); } // Authentication check. If the user has no headers, continue with the Public Agent. - let auth_header_values = get_auth(headers, requested_subject)?; + // Whatever goes wrong from here β€” headers that do not parse, a signature + // that does not verify, an unknown agent β€” the caller is not who they + // claim to be, which is a 401, not a 500. Converting the error to a + // string here used to lose the lib's `Unauthorized` type, so every + // failed sign-in reported itself as a server crash (security audit D). + let auth_header_values = get_auth(headers, requested_subject).map_err(unauthorized)?; let for_agent = atomic_lib::authentication::get_agent_from_auth_values_and_check( auth_header_values, &appstate.store, ) .await - .map_err(|e| format!("Authentication failed: {}", e))?; + .map_err(|e| unauthorized(format!("Authentication failed: {}", e).into()))?; Ok(for_agent) } +/// `error`, answered as 401 whatever it was typed as. +fn unauthorized(error: AtomicServerError) -> AtomicServerError { + AtomicServerError { + error_type: AppErrorType::Unauthorized, + ..error + } +} + /// Rate-limit key for a request without a signed agent: the socket peer /// address. Deliberately not `X-Forwarded-For`: a direct client could spoof /// that header to dodge the limit. Behind a reverse proxy every anonymous diff --git a/server/src/lib.rs b/server/src/lib.rs index 181ae1ac4..2a33d1fb6 100644 --- a/server/src/lib.rs +++ b/server/src/lib.rs @@ -10,6 +10,7 @@ mod commit_monitor; pub mod config; mod content_types; pub mod context; +mod cors; mod errors; mod handlers; mod helpers; diff --git a/server/src/serve.rs b/server/src/serve.rs index 0b9f7270b..ef481a8bf 100644 --- a/server/src/serve.rs +++ b/server/src/serve.rs @@ -1,4 +1,3 @@ -use actix_cors::Cors; use actix_web::{ body::MessageBody, dev::{ServiceRequest, ServiceResponse}, @@ -182,7 +181,7 @@ async fn announce_drives_pkarr( // Increase the maximum payload size (for POSTing a body, for example) to 50MB pub(crate) const PAYLOAD_MAX: usize = 50_242_880; -const SERVER_VERSION_HEADER: &str = "X-Atomic-Server-Version"; +pub(crate) const SERVER_VERSION_HEADER: &str = "X-Atomic-Server-Version"; const SERVER_VERSION: &str = env!("CARGO_PKG_VERSION"); /// Start the server @@ -382,12 +381,13 @@ where on_ready(&appstate); let server = HttpServer::new(move || { - let cors = Cors::permissive().expose_headers([SERVER_VERSION_HEADER]); - actix_web::App::new() .app_data(web::PayloadConfig::new(PAYLOAD_MAX)) .app_data(web::Data::new(appstate.clone())) - .wrap(cors) + .wrap(crate::cors::any_origin()) + // Outside the CORS layer, so it sees the headers CORS added: + // credentials only for origins this server answers for. + .wrap(middleware::from_fn(crate::cors::credentials_gate)) // Attaches the request (method, url, headers) to any Sentry event // raised while handling it, and reports handler panics and 5xx // errors. No-op without a bound Sentry client. diff --git a/server/src/tests.rs b/server/src/tests.rs index 113495c2e..832211947 100644 --- a/server/src/tests.rs +++ b/server/src/tests.rs @@ -579,6 +579,209 @@ async fn self_signed_agent_commit_keeps_name() { ); } +/// A fresh, initialized `AppState` on its own temporary directories, with +/// `extra_args` appended to the command line (`["--domain", "x"]`). +pub(crate) async fn init_test_appstate(extra_args: &[&str]) -> AppState { + let unique_string = atomic_lib::utils::random_string(10); + use clap::Parser; + let data_dir = format!("./.temp/{}/db", unique_string); + let config_dir = format!("./.temp/{}/config", unique_string); + let mut args = vec![ + "atomic-server", + "--initialize", + "--data-dir", + &data_dir, + "--config-dir", + &config_dir, + ]; + args.extend_from_slice(extra_args); + let opts = Opts::parse_from(args); + + let mut config = config::build_config(opts).expect("failed init config"); + // Every test gets its own index directories: parallel runs sharing the + // default ones trip Tantivy's `LockBusy`. + config.search_index_path = format!("./.temp/{}/search_index", unique_string).into(); + config.vector_search_index_path = + format!("./.temp/{}/vector_search_index", unique_string).into(); + + crate::appstate::AppState::init(config) + .await + .expect("failed init appstate") +} + +/// A client's mistake is answered as a client error, not as 500 (security +/// audit D: auth failures and malformed bodies used to report themselves as +/// crashes and fill Sentry). Real handlers, real bodies. +#[actix_rt::test] +async fn client_errors_are_not_server_errors() { + let appstate = init_test_appstate(&[]).await; + atomic_lib::test_utils::setup_test_env(&appstate.store) + .await + .unwrap(); + let app = test::init_service( + App::new() + .app_data(Data::new(appstate.clone())) + .configure(crate::routes::config_routes), + ) + .await; + + let post_commit = |body: String| { + TestRequest::post() + .uri("/commit") + .insert_header(("Content-Type", "application/ad+json")) + .set_payload(body) + .to_request() + }; + + // A body that is not JSON. + let resp = test::call_service(&app, post_commit("{\"not json".into())).await; + assert_eq!( + resp.status(), + actix_web::http::StatusCode::BAD_REQUEST, + "{}", + get_body(resp) + ); + + // JSON, but not a commit: no signature. + let signer = atomic_lib::agents::Agent::new(None).unwrap(); + let unsigned = serde_json::json!({ + urls::SUBJECT: "did:ad:something", + urls::SIGNER: signer.subject.to_string(), + urls::CREATED_AT: atomic_lib::utils::now(), + }) + .to_string(); + let resp = test::call_service(&app, post_commit(unsigned)).await; + assert_eq!( + resp.status(), + actix_web::http::StatusCode::BAD_REQUEST, + "a commit without a signature is malformed" + ); + let body = get_body(resp); + assert!(body.contains("No signature field in Commit"), "{body}"); + + // A real commit whose signature was tampered with. + let agent = atomic_lib::agents::Agent::new(None).unwrap(); + let agent_did = agent.subject.pure_id(); + let empty = atomic_lib::Resource::new(agent_did.clone()); + let mut builder = atomic_lib::commit::CommitBuilder::new(agent_did.clone().into()); + builder.is_genesis = true; + builder.set( + urls::IS_A.into(), + atomic_lib::Value::ResourceArray(vec![urls::AGENT.to_string().into()]), + ); + let commit = builder.sign(&agent, &appstate.store, &empty).await.unwrap(); + let mut json: serde_json::Value = serde_json::from_str( + &commit + .into_resource(&appstate.store) + .await + .unwrap() + .to_json_ad(Some(&appstate.config.get_origin())) + .unwrap(), + ) + .unwrap(); + json[urls::SIGNATURE] = serde_json::Value::String("AAAA".into()); + let resp = test::call_service(&app, post_commit(json.to_string())).await; + assert_eq!( + resp.status(), + actix_web::http::StatusCode::UNAUTHORIZED, + "a signature that does not verify is an authentication failure" + ); + + // A read with authentication headers that do not parse, and one whose + // signature is garbage. + let drive_did = appstate + .store + .get_drive_did("localhost") + .await + .unwrap() + .unwrap(); + let path = format!("/did?subject={}", urlencoding::encode(drive_did.as_str())); + let resp = test::call_service( + &app, + TestRequest::get() + .uri(&path) + .insert_header(("Accept", "application/ad+json")) + .insert_header(("x-atomic-public-key", "AAAA")) + .insert_header(("x-atomic-signature", "AAAA")) + .insert_header(("x-atomic-agent", "did:ad:agent:AAAA")) + .insert_header(("x-atomic-timestamp", "not-a-number")) + .to_request(), + ) + .await; + assert_eq!( + resp.status(), + actix_web::http::StatusCode::UNAUTHORIZED, + "malformed auth headers" + ); + + let req = build_request_authenticated(&path, &appstate) + .insert_header(("x-atomic-signature", "AAAA")) + .to_request(); + let resp = test::call_service(&app, req).await; + assert_eq!( + resp.status(), + actix_web::http::StatusCode::UNAUTHORIZED, + "an auth signature that does not verify" + ); + let body = get_body(resp); + assert!(body.contains("Authentication failed"), "{body}"); +} + +/// A multipart body that cannot be read is an error, not a shorter list of +/// files: `/upload` used to end its loop quietly on a multipart error and +/// answer 200 with whatever had been stored so far. +#[actix_rt::test] +async fn upload_reports_a_broken_multipart_body() { + let appstate = init_test_appstate(&[]).await; + let app = test::init_service( + App::new() + .app_data(Data::new(appstate.clone())) + .configure(crate::routes::config_routes), + ) + .await; + let drive_did = atomic_lib::test_utils::create_test_drive(&appstate.store) + .await + .unwrap(); + let path = format!("/upload?parent={}", urlencoding::encode(drive_did.as_str())); + + // `multipart/form-data` without a boundary: nothing can be parsed. + let req = build_request_authenticated(&path, &appstate) + .method(actix_web::http::Method::POST) + .insert_header(("Content-Type", "multipart/form-data")) + .set_payload("--boundary\r\nContent-Disposition: form-data; name=\"file\"; filename=\"a.txt\"\r\n\r\nhello\r\n--boundary--\r\n") + .to_request(); + let resp = test::call_service(&app, req).await; + assert_eq!( + resp.status(), + actix_web::http::StatusCode::BAD_REQUEST, + "no boundary: {}", + get_body(resp) + ); + + // A body that ends in the middle of a part. + let req = build_request_authenticated(&path, &appstate) + .method(actix_web::http::Method::POST) + .insert_header(("Content-Type", "multipart/form-data; boundary=boundary")) + .set_payload("--boundary\r\nContent-Disposition: form-data; name=\"file\"; filename=\"a.txt\"\r\n\r\nhel") + .to_request(); + let resp = test::call_service(&app, req).await; + assert_eq!( + resp.status(), + actix_web::http::StatusCode::BAD_REQUEST, + "truncated: {}", + get_body(resp) + ); + assert_eq!( + appstate + .store + .kv + .len(atomic_lib::db::trees::Tree::Blobs) + .unwrap(), + 0, + "nothing from a broken body is stored" + ); +} + /// Gets the body from the response as a String. Why doen't actix provide this? fn get_body(resp: ServiceResponse) -> String { let boxbody = resp.into_body(); From da88d8481206d56f107c2c63868214ec2dfa8d45 Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Fri, 18 Sep 2026 20:53:32 +0000 Subject: [PATCH 005/110] planning: retire the #1500 session artifacts and reconcile the index against develop #1500 merged on 2026-09-17, so its two e2e handoffs, the website handoff and the start-server script leave planning/ per the folder's own rules. Their open items and durable lessons move to the owning plans: the website follow-ups and the inline-RTE flake diagnosis to website-publishing.md, the CI-versus-local worker topology to e2e-concurrency.md, the remaining red specs and the useDriveClass gap to e2e-diagnostic-hygiene.md, the harness traps (redb lock outliving the socket, desynced compiled i18n catalogs, a borrowed node_modules, the server-root fetch) to silent-failures.md, and the two rules every agent needs (.wuchale regeneration, HTTP-200 gating) to AGENTS.md. The README's "Biggest gaps" list keeps only open gaps, each checked against code: the query index dropping rows (ignored test in lib/src/db/test.rs) and store growth (full snapshot per commit, CLI-only compaction) are added, both with a PR in flight; the plugin model is on develop since #1500, so only code-first schemas (#1262) remain PR-bound. Status rows corrected: #1355 (npm job) and #1347 (MCP plan) merged 2026-09-18, #1500 merged, #1498 still a draft, and the addResource / CommitBuilder counts recounted (0 non-test call sites in data-browser, 31 non-test references all in browser/lib/src). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01H1U7TNdjwqXKv8MQYproiq --- AGENTS.md | 17 + planning/E2E_HANDOFF_1500.md | 365 ------------------ planning/E2E_HANDOFF_FABLE.md | 248 ------------ planning/README.md | 74 ++-- planning/WEBSITE_HANDOFF.md | 332 ---------------- planning/assistant-websites.md | 8 +- ...sk-storage-and-persistence-optimization.md | 18 +- planning/e2e-concurrency.md | 15 + planning/e2e-diagnostic-hygiene.md | 33 ++ planning/extension-architecture.md | 3 +- planning/plugins.md | 2 +- planning/production-readiness.md | 6 +- planning/silent-failures.md | 59 +++ planning/start-server-9897.sh | 36 -- planning/unified-data-layer.md | 9 +- planning/website-publishing.md | 81 +++- 16 files changed, 276 insertions(+), 1030 deletions(-) delete mode 100644 planning/E2E_HANDOFF_1500.md delete mode 100644 planning/E2E_HANDOFF_FABLE.md delete mode 100644 planning/WEBSITE_HANDOFF.md delete mode 100755 planning/start-server-9897.sh diff --git a/AGENTS.md b/AGENTS.md index 9b331c520..17a6b89e4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -44,6 +44,16 @@ second writer on tracked files. Four ways that has cost real time: vite sees a file change, and the page reloads. That is not cosmetic: a reload abandons in-flight work, and this silently cancelled a destroy mid-test. If `git status` shows `.po` churn on a clean checkout, settle them and commit. +- **The compiled catalogs are a second artifact.** Wuchale compiles the `.po` + files into `src/locales/.wuchale/`, which is gitignored. `git checkout -- + src/locales/` resets the `.po` files and leaves the compiled output from + another state; the two then disagree and *every indexed string shifts*. + Missing entries show as `[i18n-404:NNN]`; shifted entries show a different + real string, so nothing looks wrong and tests fail on the wrong element. + Whenever you reset the `.po` files, also `rm -rf + browser/data-browser/src/locales/.wuchale` and restart vite. Do not make + the `.po` files read-only to stop the rewrite; the compiled output goes + stale instead. - **`pnpm clean-translations` is not the same writer.** It extracts from test files too, which the vite plugin does not, so its output is a *different* fixed point. Settling by running the app is what matches the dev server. @@ -283,6 +293,13 @@ cd browser && pnpm run test-e2e:light # Playwright @smoke (feature-br cd browser && pnpm run test-e2e # Full Playwright suite (develop / tags) ``` +When you restart an `atomic-server` for tests, gate on `curl` returning HTTP +200, never on the port: redb's lock outlives the listening socket by several +seconds, so an immediate restart exits with "Database already open" and the +suite runs against nothing, failing in ways that look like product bugs. Retry +the spawn when the log says so, and give a fresh server about ten seconds +before the first spec. + `atomic_lib`'s unit tests need the `db` feature β€” `hierarchy.rs`'s test module calls `Db::init_temp` and `test_utils::setup_test_env`, both gated behind it. So `cargo test -p atomic_lib --no-default-features` does **not** compile; don't diff --git a/planning/E2E_HANDOFF_1500.md b/planning/E2E_HANDOFF_1500.md deleted file mode 100644 index 37c7fb6ae..000000000 --- a/planning/E2E_HANDOFF_1500.md +++ /dev/null @@ -1,365 +0,0 @@ -# Handoff: e2e suite for PR #1500 - -Updated 2026-09-16 (second session). Goal unchanged: get the Playwright suite -green for `codex/website-self-hosted-publishing`, then **Rebase and merge**. - -## The most important correction to the previous handoff - -Most of the "unexplained" website failures (group 4) and a good share of the -rest were **not product bugs β€” there was no server running**. - -`redb`'s lock outlives the listening socket. Killing the server frees port 9897 -within a second, but `atomic.redb` stays locked for several more, so a restart -that only waits for the port exits immediately with -`Error: Failed to create redb ...: Database already open. Cannot acquire lock.` -The suite then runs against nothing and every spec fails in ways that look like -real defects: missing menu items, disabled buttons, empty panels. - -**Never gate on the port. Gate on `curl` returning HTTP 200, and retry the -spawn β€” not just the wait β€” when the log says "Database already open".** -`scripts/start-server.sh` (below) does this. Also give a freshly started server -~10s before running specs; within that window `menu-item-website-prepare` -renders but stays `disabled`. - -## Stack setup (delta from the previous handoff) - -Everything in the previous handoff still applies, plus: - -1. **Patched loro-prosemirror.** `data-browser/node_modules` resolves - loro-prosemirror out of `~/dev/atomic-server-plugin-model`, where - `browser/patches/loro-prosemirror@0.4.3.patch` is **not** applied (that - checkout's `.modules.yaml` has no `patchedDependencies`). CI applies it. - Without it the inline editors drop characters and - `website-inline-content` / `website-inline-rte` fail. Fix, touching neither - of Joep's checkouts: - - copy the resolved package to `/private/tmp/lp-patched/node_modules/loro-prosemirror` - - `/opt/homebrew/bin/git apply` the patch there - - symlink its five sibling deps (`lib0`, `loro-crdt`, `prosemirror-model`, - `-state`, `-view`) beside it so its own imports resolve - - alias `loro-prosemirror` to that `dist/index.js` in - `/private/tmp/hosting-vite.config.mts` and add `/private/tmp/lp-patched` - to `server.fs.allow` - - `rm -rf /private/tmp/hosting-vite-cache` and restart Vite, or the stale - pre-bundle is still served - -2. **Mock integration proxy**, matching `atomicService` in `.dagger/src/index.ts`: - ``` - cd integrations/localthought && MOCK_FRONTEND_ORIGIN=http://localhost:6763 \ - MOCK_PROXY_HOST=127.0.0.1 MOCK_PROXY_PORT=19090 \ - TENANT_SECRET='bW9jay10ZW5hbnQ.mock-signature' node mock-proxy.mjs - ``` - The server wants `ATOMIC_INTEGRATION_PROXY_URL`, `TENANT_SECRET` and - `ATOMIC_INTEGRATION_FRONTEND_ORIGIN`; Vite wants `VITE_E2E=true` and - `VITE_INTEGRATION_PROXY_URL=http://127.0.0.1:19090`. - `ATOMIC_MOCK_INTEGRATION_PROXY=1` on the Playwright side only gates one - `test.skip` in `plugins.spec.ts` β€” it cannot affect any other spec. - -3. **create-template** needs building and its own `node_modules`: - `../node_modules/.bin/tsc` in `browser/create-template` emits - `bin/src/index.js` (type errors are expected without deps), and - `browser/create-template/node_modules` can symlink to the one in - `/private/tmp/atomic-assistant-website`, matching the other packages. - -4. **Worker count matters.** At `PLAYWRIGHT_WORKERS=4`, `documents.spec.ts`, - `username-live.spec.ts` and two `apps.spec.ts` tests fail and then pass at 2 - workers. Treat 4-worker-only failures as contention until reproduced lower. - -## Fixed this session (commit `6e79f6a4e`) - -- **Test bug class, the right panel is transient (#1475).** - `RightPanelProvider` keeps open/closed in `useState` and deletes - `atomic.rightPanel.active`; `atomic.sidebar-panels` only controls which - panels are *available*. Specs must click the navbar toggle β€” - `openAISidebar(page)` / `sendChatMessage(page, ...)` in `tests/ai-mock.ts`. - `website.spec.ts`, `assistant-file-drop.spec.ts` and the shared - `tests/legacy-github-setup.ts` (which broke both `app-setup.spec.ts` tests) - hand-rolled `page.locator('[data-open]')` instead. A bare `[data-open]` - locator is the smell; correct uses scope it to a testid. -- **`integration-workspace.spec.ts`**: `ConnectGitHub.tsx` became a thin - wrapper around the schema-driven `AppSetupForm` and no longer names the - installer path the spec regexes out of the served module β€” read - `githubInstaller.ts` instead. The button is labelled "Edit with AI". -- **`browser/lib/src/websockets.ts`**: guard `importEnvelopes` the way - `client-db.node.ts` does. `getClientDb()` can return an implementation - without it, and the unguarded call threw a TypeError out of the SYNC_PUSH - handler, taking the rest of that frame's handling with it. This was the one - real product defect the previous session identified; `onboarding-storage` - passes now. **Rebuild `browser/lib/dist` (`pnpm exec tsup`) after changing - lib, and restart Vite.** - -## Where the numbers stand - -Two full runs on a correct stack, both **236 passed / 42 failed / 7 skipped** -(the previous session's baseline was 231/46/8). The headline is the same at 4 -and 2 workers but the *composition* differs β€” `apps.spec.ts` contributed 4 -failures in one run and 6 in the other, `documents.spec.ts` and -`username-live.spec.ts` failed only at 4 workers. A meaningful share of the 42 -is flake, so treat any single run's list as provisional and reproduce before -chasing. - -Latest run, by spec: plugins 11, apps 6, table-create-perf 3, -vault-backup-restore 2, template 2, recovery-option 2, meetings 2, -google-calendar-import 2, file-upload-offline 2, drive-template-onboarding 2, -and one each in website-inline-rte, second-device-load, opfs-init-perf, -offline-tables, kanban, integration-workspace, integration-visibility, -devonian-issue-sync. - -`@smoke` β€” the only gate CI runs β€” is 17/18, the exception being the -order-dependent `second-device-load` noted below. - -## Still failing β€” triage - -- **`plugins.spec.ts` (9), `google-calendar-import` (2), - `integration-visibility` (2), `devonian-issue-sync` (1).** Not yet diagnosed - individually. These now run against a real mock proxy, so what remains is - worth reading as real. -- **Diagnostics group: `offline-tables`, `file-upload-offline` (2), - `opfs-init-perf`, `recovery-option` (2), `vault-backup-restore` (2).** These - fail the fixture's zero-console-error gate, not their own assertions. The - error is always the same: the app fetches `http://localhost:9897/` and the - server 404s it. On this branch drives are `did:ad:drive:...` subjects and - **nothing lives at the server's HTTP root** β€” a fresh data dir with - `ATOMIC_INITIALIZE=true` 404s there too, so this is architectural, not - misconfiguration. Either the client should not fetch the root, or these - tests should declare the diagnostic. `fixtures.ts` has no global allowlist by - design ("expected failures belong to the test that causes them"). -- **`integration-workspace.spec.ts:159`** β€” genuine product suspicion. After - "New automation", `WorkspaceControls` calls `onStart={() => close()}` and - `askAI(...)`, but the `` stays open with its - content unmounted, and it is the only thing in the a11y tree. If - `handleClosed` never runs, `useDialog` also leaves `inert` on ``, - which would make the app unclickable. Worth chasing in - `components/Dialog/index.tsx` (`finishClose` / the `data-closing` effect). -- **`apps.spec.ts`** is the worst flake in the suite: 4 failures in one full - run and 6 in the next, a different test at 1 vs 2 workers, always waiting on - `iframe[title="App"]`. Look at what gates that iframe before reading any - individual failure as real. -- **`second-device-load.spec.ts` (@smoke)** fails standalone but passed inside - the OPFS group run β€” order-dependent. Green on CI. -- **`template.spec.ts` (2)** should be unblocked by the build above, but each - test scaffolds and builds a real Next/SvelteKit site and is very slow. -- **Not investigated:** `table-create-perf` (3, perf thresholds), `kanban:177`, - `meetings` (2), `drive-template-onboarding` (2). - -## Merge posture - -CI's e2e gate is `@smoke` only (~18 tests) and is green. The full suite has -never been green on this branch, and several of the failures above are stale -tests or missing declarations in code the PR itself introduces β€” they are not -regressions against `develop`. Deciding how many of them must be green before -merging is Joep's call. - ---- - -# Second pass: where the failures actually come from - -Prompted by "develop is 0 failed" β€” that is **not reproducible in this -environment**, and the comparison needs care. - -## The local harness was never the CI harness - -**The checkout's dependencies came from a different branch.** Every -`node_modules` under `browser/` was a symlink into -`/private/tmp/atomic-assistant-website` (which itself resolves into -`~/dev/atomic-server-plugin-model`). This branch's own `pnpm-lock.yaml` had -never been installed. Consequences, all of which looked like product bugs: - -- `browser/patches/loro-prosemirror@0.4.3.patch` was not applied (the manual - copy + Vite alias described above is no longer needed β€” a real install - applies it, and the result is byte-identical). -- A **hard WASM crash** in the production build: `[ClientDb Worker Error] - Uncaught Error: null pointer passed to rust`, which broke every `@smoke` - test (18/18 failing). - -Fix: remove the symlinks (never `rm -rf` β€” they point at other checkouts) and -`pnpm install --frozen-lockfile` in `browser/`. It takes ~8s. Several workspace -packages must then be built before the data-browser will bundle: -`@tomic/service-ui`, `@tomic/plugin`, `@tomic/react`, `@tomic/lib` -(`pnpm --filter build`). After that the prod `@smoke` run went 18 failed -β†’ **2 failed, 16 passed**. - -**CI serves a production build, not Vite.** `.dagger/src/index.ts` builds -`data-browser/dist`, copies it to `server/assets_tmp`, and the server embeds -and serves it; Playwright's `FRONTEND_URL` is the server itself. Reproduce with: - -``` -cd browser/data-browser && SKIP_WASM_BUILD=1 VITE_E2E=true \ - VITE_INTEGRATION_PROXY_URL=http://127.0.0.1:19090 pnpm build -cd ../.. && rm -rf server/assets_tmp && mkdir -p server/assets_tmp \ - && cp -R browser/data-browser/dist/. server/assets_tmp/ -cargo build -p atomic-server # with the CLT env; build.rs embeds assets_tmp -# then FRONTEND_URL=http://localhost:9897 -``` - -## Measured, on a correct install - -| Stack | Result | -|---|---| -| Vite dev, correct deps | 230 passed / **47 failed** / 7 skipped | -| Production build (CI topology) | 229 passed / **49 failed** / 7 skipped | -| `@smoke` on the production build | 16 passed / 2 failed | - -The *number* barely moved; the *composition* did (prod fixes meetings, -table-create-perf, opfs-init-perf, template, recovery-option and breaks the -website specs β€” the website preview iframe wants an origin separate from the -app, which the same-origin prod topology does not give it). Failures stable -across all three topologies are the real ones. - -## Attribution β€” the answer to "develop is 0 failed" - -Of the failing specs, **these do not exist on develop at all**: `plugins` -(11), `apps` (6), `google-calendar-import` (2), `integration-workspace` (2), -`integration-visibility` (2), `drive-template-onboarding` (2), -`devonian-issue-sync` (1), `website-inline-rte` (1). That is **~27 of 47** β€” -new tests for new features, which develop cannot fail because it does not have -them. `develop` being green says nothing about them. - -Four failing specs are **byte-identical** to develop: `meetings`, -`second-device-load`, `opfs-init-perf`, `offline-tables`. Run against a -develop stack built in this same environment (worktree at -`/private/tmp/atomic-develop-base`, server on 9898, Vite on 6764), **all of -them fail there too** β€” 7 failures on develop versus 4 on the branch. So they -are not branch regressions. (That develop stack has its own setup gap β€” -`/app/dev-drive` times out for most specs β€” so treat it as evidence about -these four specs, not as a full baseline.) - -Conclusion: the branch is not ~47 regressions deep. The bulk is new-feature -tests that have never passed, plus environment and flake. What genuinely needs -work before "green" is the new-feature specs, led by `plugins` and `apps`. - -## `apps.spec.ts` β€” two real bugs found (6 failures) - -1. **A freshly created app renders its generic resource page, not its own - view; a reload fixes it.** `ResourcePage` gates the app view on - `appClass !== undefined && resource.hasClasses(appClass)`, and `appClass` - comes from `useDriveClass` in `chunks/PluginRuns/runScript.ts`. That hook - reads the drive's `defaultOntology` once and subscribes to it β€” but a drive - with no plugin schema yet **has no `defaultOntology`**, so it subscribes to - nothing. Creating the first app mints the schema and sets the property, and - the hook never hears about it; `appClass` stays `undefined` until a - remount. Fix direction: also subscribe to the drive resource and - re-subscribe when the ontology appears. (Written and reverted here β€” it is - correct on its own terms but bug 2 masks any test-visible benefit, so it - should land with a test that proves it.) -2. **App creation itself races.** In some runs `createApp` completes and - navigation lands on the app (verified by instrumenting the action: - `run:start β†’ createApp:done β†’ handOver:ok β†’ navigate:`). In others the - URL stays on the drive and `findSchema(store, drive, pluginSchema())` - returns a schema with an **empty `classes` map**. Same code, same stack β€” - so `ensureSchema` has a race. This is the one to chase first. - ---- - -# Third session, 2026-09-17 β€” 41 failures down to 14 - -Full suite on the local stack (Vite dev, 2 workers): **265 passed / 14 failed / -7 skipped**, from 237/41/7 at the start of the session. - -## The operational finding that mattered most - -**The server degrades as its data dir grows.** After a day of runs -`/private/tmp/atomic-hosting-node/data` had reached **1.1 GB**, and on that -server all six `apps` tests and most of `plugins` failed on a 10s -`iframe[title="App"]` wait β€” which reads exactly like a product bug and is not -one. Moving the data dir aside and restarting took `apps` + `plugins` at two -workers from **16 failures to 1**. Do this before any full run; -`scripts/start-server.sh` in the session scratchpad starts the server and gates -on HTTP 200 (never on the port β€” redb's lock outlives the socket). - -Everything else about the stack is as the second session described. Firefox was -missing from the Playwright cache (`pnpm exec playwright install firefox`); -`client-db-locks` passes with it. - -## Fixed β€” product - -- `027c09094` **The empty string is not a subject.** `normalizeSubject('')` - resolves against the server URL, so `useResource(x ?? '')` and every render - before the drive setting hydrated fetched `http://localhost:9897/`. Nothing - lives at the root on a DID-drive branch, so each of those 404s and logged two - console errors β€” failing the zero-diagnostics gate in `offline-tables`, - `sync`, `file-upload-offline` and `drive-deeplink @smoke`. Guarded in - `getResourceLoading` and the async `getResource`. `CustomViewProvider` had - the same shape one level up (`/plugin-list?drive=`). -- `d1e91ddba` **The app-setup dialog could not close.** It derived `show` from - `useDialog`'s post-animation `isOpen`, which only goes false once the Dialog - has seen `show` go false: a deadlock, with the modal swallowing every click - behind it. That was `plugins.spec.ts:893`. -- `e4141c008` **A regression from #1510.** Reusing a same-shortname select - property threw when it lacked a requested option, and the Student and - Personal templates each pair a task "Status" with a reading-list "Status". - The whole gallery died with `has no option "Want to read"`. Now it - disambiguates to `status-2`, as the incompatible-datatype branch already did. - -## Fixed β€” tests - -- `2bf4e927f` / `3e86c4be2` **Kanban drags.** The board FLIP-animates after a - drop, so the second drag pressed where the card used to be and dnd-kit never - activated. `await source.hover()` first; measure the drop target after - activation, not before. Same fix in the Devonian spec. -- `00414440d` `integration-visibility` used `check()` on a toggle that - disappears once checked; `integration-workspace` asserted "no dialog at all" - when the app legitimately raises the AI model-setup dialog. - -## What is still red, and why - -| Spec | Verdict | -|---|---| -| `recovery-option` (2), `opfs-init-perf`, `table-create-perf` (3) | Dev-topology only. The SW warning `recovery-option` declares happens only in a production build (VitePWA is off in dev); the `clientdb.*` perf marks are likewise absent from the dev module graph. All pass in CI's topology. | -| `vault-backup-restore` (2) | Points at a portal on :49237, which is **another checkout's** atomic-saas dev server. Needs this branch's portal to mean anything. | -| `google-calendar-import` (2) | Real gap. The new integration-visibility preference is only readable with the server reachable, and this spec runs the whole scenario with the server cut off. Enabling discovery before the interception starts lists the Calendar card; the reload that follows loses it again. Fix = make the preference and its schema readable from the local DB. | -| `second-device-load` @smoke | Real, pre-existing flake β€” ~1 in 3 passes locally, green on CI. Evidence: after unlocking, `document.body` has no `SecondDeviceChild`; **a reload makes it appear**, so the data is on the server and it is the post-unlock render that misses it. A `StoreEvents.AgentChanged` β†’ `invalidateCollection` hook in `useCollection` was tried and did **not** fix it (reverted). Next place to look is what the sidebar's `useChildren` collection does between "locked, anonymous query" and "signed in". | -| `offline-chatroom`, `website-inline-rte` | Flaky; each passes when run alone. `website-inline-rte` is the known dropped-character flake in the preview editor. | - -Nothing in the remaining list is a regression this branch introduces, and the -`@smoke` gate CI runs is green apart from the `second-device-load` flake above. - -## Later the same night β€” 41 β†’ 11, and three more real fixes - -Full suite, dev topology, 2 workers: **268 passed / 11 failed / 7 skipped**. - -- `f0de176c6` **A children page older than its own count.** - `getMemberWithIndex` returns undefined when the page it reads does not hold - the index the count implies; `useChildren` dropped that slot and never looked - again, so a child could be missing for the life of the view. - `second-device-load @smoke` went from ~1 pass in 3 to 6 in 7. The retry only - runs while the hook has never produced a list β€” after a delete the count runs - ahead of the page too, and refreshing there would pull the destroyed resource - back into the store. -- `993ad653b` **A child that did not exist when the query was answered.** - The collection has no reason to ask again, so a folder created moments - earlier stayed missing from the drive's list until a reload β€” - `server-only-fallback.spec.ts`, failing about half the time. `useChildren` - already listens for `ResourceUpdated` to re-sort; it now also re-reads the - query when the changed resource names this parent and is absent from the - list. That spec: 4/4. `e2e.spec.ts` "delete resource" @smoke, which had been - ~50/50 in this environment, also went 4/4. -- `8183a3bc6` **`google-calendar-import` (both tests) now pass.** They enabled - API-plugin discovery *after* cutting the server off, and that preference is a - write whose `/commit` the spec aborts. Set it up first, read the card back so - the preference and schema are in the local DB, flush, then go offline. The - offline phase must also open the drive by subject β€” `/app/dev-drive` - bootstraps the drive and, with no server, rebuilds the ontology empty. - -### What is left, honestly - -`vault-backup-restore` (2) needs its own control plane: `atomic-saas` -hardcodes port 3030, where Joep already runs one, and `vault-stack.sh` points -enrolled drives at the node in `.env.development` (9885 β€” the real dev server). -Not something to start from a session that must not disturb that stack. - -`table-create-perf` (3), `opfs-init-perf` and `recovery-option` (2) were -called "dev-topology only" above. That is right for `recovery-option` (the -service-worker warning it declares only happens in a production build), but a -production build made in this environment fails the other two *differently* -(`/app/dev-drive` never reaches a drive), so the prod topology here is not -clean enough to prove them either way. Treat them as unexplained rather than -explained. - -The rest is flake that passes when run alone: `offline-chatroom`, -`tables.spec.ts` Shift+Enter, `website-inline-rte` (the known dropped-character -flake in the preview editor). - -**CI is red for an unrelated reason.** Every Mancave run in the repo since -10:34 on 2026-09-16 fails in `Log in to Docker Hub` with `Error: spawn EIO`, -before any build step β€” `develop`, `feat/plugin-catalog` and three other -branches alike. The runner host needs attention; re-running does not help. diff --git a/planning/E2E_HANDOFF_FABLE.md b/planning/E2E_HANDOFF_FABLE.md deleted file mode 100644 index c5b40219b..000000000 --- a/planning/E2E_HANDOFF_FABLE.md +++ /dev/null @@ -1,248 +0,0 @@ -# e2e handoff β€” PR #1500 (`codex/website-self-hosted-publishing`) - -Written 2026-09-17. HEAD at handoff: `077555c32`. - -Goal: get the Playwright suite green so #1500 can merge (**Rebase and merge** β€” -merge commits are disabled and squash would collapse 167 commits including -Michiel's). - -Last measured full run: **269 passed / 5 failed / 4 flaky / 7 skipped** (19 min, -2 workers, retries=1). It was 41 failed at the start of the day. - ---- - -## 1. Read this first: the harness lies - -More of my wasted time came from the environment than from the code. Three traps, -in order of how much they cost: - -### 1.1 The i18n catalogs silently corrupt results - -The `wuchale` Vite plugin rewrites `browser/data-browser/src/locales/*.po` while -the dev server runs. The **compiled** catalogs live in `src/locales/.wuchale/`, -which is **gitignored**. So `git checkout -- src/locales/` restores the `.po` -files and leaves the compiled artifacts from some other state β€” the two disagree -and every indexed string lookup shifts. - -Two failure shapes; the second is the dangerous one: - -- missing entries render as `[i18n-404:2275]` β€” visibly broken -- shifted entries render as **a different real string** β€” nothing looks wrong - -That second one is how `iframe[title="Website preview"]` became -`"Preserves recurring series and previews supported edits to send back to -Google."` and every website spec failed. I bisected code for an hour before -finding it. - -**Before trusting any run**, regenerate both together and restart Vite: - -```bash -git checkout -- browser/data-browser/src/locales/ -rm -rf browser/data-browser/src/locales/.wuchale -# then restart Vite -``` - -Do **not** `chmod 444` the `.po` files to stop the rewrite β€” I tried; it freezes -them while `.wuchale` goes stale and `integration-visibility` then fails with -`[i18n-404]` toggles. The `.po` files going dirty *during* a run is fine as long -as both were generated in step. - -### 1.2 redb's lock outlives the listening socket - -After killing the server, port 9897 frees within a second but `atomic.redb` stays -locked for several more. Starting immediately gives `Database already open` and -the process exits β€” leaving **no server** while the suite runs, and every spec -fails in ways that look like product bugs. Gate on `curl` returning HTTP 200, not -on the port, and retry the spawn. `planning/start-server-9897.sh` does this. - -Also: give a freshly started server ~10s before running specs. - -### 1.3 Worker count and retries - -CI runs **2 workers per shard, 4 shards, each with its own atomic-server**, plus -retries (`.dagger/src/index.ts`, `HOST_PROFILES.mancave`). A local 4-worker run -against one server is much harsher and invents failures. Use -`PLAYWRIGHT_WORKERS=2 PLAYWRIGHT_RETRIES=1`. - -A rotating pair of specs fails per run purely from load (`plugins:909`, -`tables:482`, `drive-catalog`, `offline-tables`, … different each time). **Always -re-run a red in isolation before believing it.** - ---- - -## 2. Running the stack - -Checkout: `/private/tmp/atomic-pr-website`. Use `/opt/homebrew/bin/git` β€” the -Xcode git/python on PATH exit 69 (unaccepted licence). Rust builds need the -Command Line Tools env; see `project_rust_build_env.md` in the memory dir. - -**Dependencies are already installed correctly.** They were previously symlinked -from an unrelated branch's checkout, which is why the loro patch was missing and -why a production build died with `null pointer passed to rust`. If you ever need -to redo it: remove the symlinks (never `rm -rf` β€” they point at live checkouts), -`pnpm install --frozen-lockfile` in `browser/`, then build `@tomic/service-ui`, -`@tomic/plugin`, `@tomic/react`, `@tomic/lib` before bundling the data-browser. - -Three processes: - -```bash -# 1. server (9897) -bash planning/start-server-9897.sh - -# 2. mock integration proxy (19090) -cd integrations/localthought && MOCK_FRONTEND_ORIGIN=http://localhost:6763 \ - MOCK_PROXY_HOST=127.0.0.1 MOCK_PROXY_PORT=19090 \ - TENANT_SECRET='bW9jay10ZW5hbnQ.mock-signature' node mock-proxy.mjs & - -# 3. Vite (6763) β€” MUST have VITE_ATOMIC_SERVER_URL or it points at :9885, -# which is Joep's real dev server -cd browser/data-browser && VITE_ATOMIC_SERVER_URL=http://localhost:9897 \ - VITE_E2E=true VITE_INTEGRATION_PROXY_URL=http://127.0.0.1:19090 \ - node node_modules/.bin/../vite/bin/vite.js \ - --config /private/tmp/hosting-vite.config.mts --host localhost --port 6763 & -``` - -Verify after every Vite restart: - -```bash -curl -s http://localhost:6763/src/config.ts | grep -oE '"VITE_ATOMIC_SERVER_URL": "[^"]*"' -``` - -Run the suite: - -```bash -FRONTEND_URL=http://localhost:6763 SERVER_URL=http://localhost:9897 \ -WEBSITE_HOSTING_E2E=1 ATOMIC_MOCK_INTEGRATION_PROXY=1 \ -PLAYWRIGHT_WORKERS=2 PLAYWRIGHT_RETRIES=1 \ -browser/e2e/node_modules/.bin/playwright test \ - --config browser/e2e/playwright.config.ts --project chromium --trace off --reporter line -``` - -**Note on topology:** CI builds the data-browser and embeds it in the server -(`data-browser/dist` β†’ `server/assets_tmp`), so Playwright hits the *server*, not -Vite. I've verified both work; the dev-server topology is what the numbers above -were measured on. Prod fixes some specs and breaks the website ones (the preview -iframe wants an origin separate from the app), so don't switch topology -mid-investigation and compare. - ---- - -## 3. What is actually failing - -### 3.1 `vault-backup-restore` Γ—2 β€” blocked on a missing service - -"Generate my recovery code" is clicked, the code never renders, and -`http://localhost:6763/api/recovery-secret` returns **404**. That is a -managed-node/portal API; a plain `atomic-server` does not serve it. The other -session's summary agrees: the vault panel needs the app to know a portal, and -only `atomic-managed-node` advertises one. - -Nothing to fix in the spec. Either run a managed node beside it or the spec needs -to declare the dependency. - -### 3.2 `website-inline-rte` β€” real, and the obvious fix is a dead end - -Symptom: ~50% failure, always a popup inside the preview iframe (mention list at -`:58`, link field at `:92`). The `@` trigger itself works 6/6 in isolation, so it -is state from earlier steps in the long sequence. - -**Diagnosis is confirmed by instrumentation.** During inline typing: - -| event kind | count | path | target | -|---|---|---|---| -| non-local (`import`) | 17/17 | `["properties"]` | `cid:root-properties:Map` | -| local | 18/18 | `["doc", …]` | β€” | - -The server echoes the author's own commit with a `lastCommit` stamp on the -resource's `properties` map. `LoroSyncPlugin` subscribes to the **doc**, not the -store (the store *does* suppress notify for an own-save echo β€” see -`commit-echo-and-save-cursor` in memory), so it rebuilds the whole ProseMirror -document for a change that never touches the document, and the rebuild destroys -any open popup. - -**Do not re-attempt the `updateNodeOnLoroEvent` filter.** The older notes call it -a "verified fix candidate". I implemented and measured it (skip non-local events -with no `path[0] === ROOT_DOC_KEY`, only when `containerId` is unset). Result: -`website-inline-rte` + `website-inline-content` went from a steady -1-failed-1-passed to **6 failed / 2 passed** over four runs, with two separate -failures β€” the mention list stops populating (the Suggestion popup really was -relying on those rebuild transactions) and the character drop at -`inline-content:107` persists anyway. I reverted it; the dependency is -byte-identical to its patched state and `browser/patches/` is untouched. - -The tractable framing is the **popup, not the patch**: make the tiptap Suggestion -plugin re-establish itself after a document rebuild, then the filter becomes -viable. That is product surgery in the editor β€” Joep has not green-lit it. - -Related and already committed: `a2503df27` clears the tiptap renderer reference -on destroy (`CommandsExtension` destroyed it on Escape and left the reference -set, so `onExit` destroyed it again). Genuine latent bug; does **not** fix this -flake, and the commit message says so. - -Iterating tip: edit the installed `dist/index.js` directly, then clear -`/private/tmp/hosting-vite-cache` and restart Vite. Vite pre-bundles the -dependency, so an edit is invisible until that cache is cleared β€” this cost me a -measurement that showed zero events. - -### 3.3 `plugins:909`, `tables:482` β€” not real - -Both pass in isolation (2/2 each). Load rotation, see Β§1.3. - ---- - -## 4. Fixed today (for context on what the causes look like) - -- `cc06e53ff` β€” `ensureAll` recovered each schema term with its own - `findByLocalId` **inside a loop**: 19 serialised `/query` round-trips, 13 of the - 14 seconds creating an app. Now issued together. `apps` 6β†’0, `plugins` 11β†’2. -- `077555c32` β€” `signIn()` decided "already signed in" from the settings link - being visible, which renders before the agent is in the store. Sessions with no - agent navigated on, read their drive unauthenticated, never adopted one, and so - never got a presence manager (it is keyed on the session drive). Agent state - correlated exactly with outcome. `presence-follow` + `meetings` 15/15. -- `82847e965` β€” spec edited `subjects[0]` of a `parent=` query assuming it was the - imported row; tables also hold draft placeholder rows with the same parent. -- `366156f53` β€” `AppSetupProvider` derived the Dialog's `show` from `isOpen`, - which only goes false *after* a close that needs `show` false. Deadlock: the - modal swallowed every click including its own close button. -- `6e79f6a4e` β€” specs hand-rolled `page.locator('[data-open]')` for the AI panel, - which is transient state and never restored (#1475). Use `openAISidebar` / - `sendChatMessage` from `tests/ai-mock.ts`. -- `a2f8f0125`, `45f721587` β€” action-menu snapshot race; Calendar spec read - `VITE_INTEGRATION_PROXY_URL` from the test process, but that is a build input - CI only passes to the bundler. - -## 5. Things I got wrong β€” don't redo them - -- `a5406e512` reverted `366156f53` on a measurement taken while the catalogs were - desynced. Reinstated. **Any measurement taken with `src/locales/` dirty is - suspect.** -- `9a1934038` retried the deep-link drive fetch on a 401, on a pending-handshake - theory. Reverted in `077555c32`: the session simply had no agent. -- The loro filter, above. - -All three shared a cause: acting on a 3–5 run sample in an environment I had not -yet established was trustworthy. Take more samples than feels necessary, and -re-check Β§1.1 before each. - -## 6. Coordination - -Another agent has been working **in this same checkout** on the same assignment -(`useChildren`, vault/portal, tombstones, perf-trace, optional diagnostics β€” -commits `8183a3bc6`, `993ad653b`, `0cd15fe9a`, `4117e9bd8`, `68af82756`, -`b9d03fcf7`, `fb56cfdba`, `f0de176c6`, …). Changing `node_modules` (a patch + -lockfile hash + reinstall) would swap dependencies under their running Vite, which -is why I left the loro patch alone. `366156f53` touches `AppSetupProvider`, close -to their integration work β€” likely conflict point. - -There is also a separate manual-testing stack for Joep on **6765** (server 9899, -data `/private/tmp/atomic-manual`) β€” leave it running. - -## 7. Suggested order - -1. Confirm Β§1.1 is clean, run the full suite once, and see which reds reproduce in - isolation. Expect ~2 of them not to. -2. `vault-backup-restore`: decide with Joep whether to run a managed node locally - or have the spec declare the dependency. Cheapest real win. -3. `website-inline-rte`: only worth starting if Joep green-lights the Suggestion - popup work. Do not touch the loro patch first. diff --git a/planning/README.md b/planning/README.md index efc0ad27c..3b643e874 100644 --- a/planning/README.md +++ b/planning/README.md @@ -31,26 +31,52 @@ now live in [`completed/`](./completed/): - [`trust-model-decision.md`](./completed/trust-model-decision.md) β€” the node that owns the URL is trusted with plaintext; anything that only stores is blind. - [`schema-routes-decision.md`](./completed/schema-routes-decision.md) β€” `did:ad:frozen` is the on-ramp, optional schema is the write-path policy. -## Biggest gaps (reconciled against code 2026-09-15) - -Ranked by impact. Each links to the plan that owns the work. - -1. ~~**Flutter writes are not durable.**~~ Fixed 2026-09-15: `Db::init_redb_file` owns the durable-flush tick, so every file-backed binding has the same 100ms loss bound. [`atomic-lib-runtime.md`](./atomic-lib-runtime.md). -2. ~~**No rate limiting on any write endpoint**~~ (fixed 2026-09-15, `server/src/rate_limit.rs`); the managed-node bootstrap grace still has no reaper. [`foss-public-host-mode.md`](./foss-public-host-mode.md) Phase 3, [`security-audit-2026-09.md`](./security-audit-2026-09.md) D, [`cloud-sync-managed-node.md`](./cloud-sync-managed-node.md). -3. ~~**Signed envelopes do not replicate**~~ Fixed 2026-09-15: envelopes ride in `SYNC_PUSH` and vault packs, verified on receipt. [`auditability-loro-history.md`](./auditability-loro-history.md). -4. **Flutter and desktop bypass the runtime boundary**: both hold a raw `Db` instead of `AtomicNode`, which is how gap 1 happened. [`atomic-lib-runtime.md`](./atomic-lib-runtime.md). -5. **`SYNC_PUSH` is an unsigned cross-trust import** gated by one drive-level ACL verdict; no grant-chain proof exists. [`authorization-sync.md`](./authorization-sync.md). -6. **npm is seven betas behind**: `@tomic/*` still serves `0.41.0-beta.0`. The `npm` release job exists since 2026-09-15; the next tag publishes once `plugin` and `edit-mode` have trusted-publisher entries. [`production-readiness.md`](./production-readiness.md). -7. ~~**No Rust outbox**~~ Ported 2026-09-16 (`lib/src/sync/outbox.rs`), drained over WS and over live Iroh links. [`unified-sync.md`](./unified-sync.md), [`serverless-p2p.md`](./serverless-p2p.md). -8. ~~**Desktop CSP is disabled**~~ Set 2026-09-15, pending a packaged-build smoke test. [`security-audit-2026-09.md`](./security-audit-2026-09.md). -9. **Plugins and code-first schemas live only in unmergeable PRs** (#1307 at 532 files with conflicts, #1262 a stale draft). [`plugins.md`](./plugins.md), [`json-schema-code-first.md`](./json-schema-code-first.md). -10. ~~**Dashboards have no entry point and fork review shows only a count.**~~ Both closed 2026-09-16: a table's dashboard is a view tab, and the fork bar shows the per-property diff. Still open there: suggest-for-non-writers, per-property revert, Canvas forks. [`dashboards.md`](./dashboards.md), [`drafts-and-suggestions.md`](./drafts-and-suggestions.md). +## Biggest gaps (reconciled against code 2026-09-18) + +Ranked by impact. Each links to the plan that owns the work. Gaps closed since +the 2026-09-15 pass (durable Flutter writes, write-endpoint rate limits, +envelope replication, the Rust outbox, desktop CSP, dashboard entry point and +fork review diff) are recorded in their owning plans, not here. + +1. **The query index silently drops rows.** A drive-scoped, sorted, + class-filtered query returns fewer members than exist: the ignored test + `is_a_encodings_all_match_the_class_constraint` in `lib/src/db/test.rs` + reproduces `query_sorted_indexed` returning 2 of 3 matching rows, and + [`silent-failures.md`](./silent-failures.md) carries two open entries for + the same shape (5 of 22 Houseplants rows). PR in flight. +2. **Flutter and desktop bypass the runtime boundary.** Flutter keeps a + `static DB: OnceLock>` and wraps it in `AtomicNode::from_db` per + call (`flutter/rust/src/api/simple/state.rs`); the desktop app holds a raw + `atomic_lib::Db` (`desktop/src/lib.rs`). [`atomic-lib-runtime.md`](./atomic-lib-runtime.md). +3. **`SYNC_PUSH` is an unsigned cross-trust import** gated by one drive-level + verdict (`admit_drive_write` / write rights in `lib/src/sync/engine.rs`); + no `AuthorizationProof` type exists in `lib/src`. [`authorization-sync.md`](./authorization-sync.md). +4. **The store grows with every write and is never compacted.** `sign_at` + (`lib/src/commit.rs`) exports a full Loro snapshot per commit although its + docstring promises an incremental update, and compaction exists only as the + `atomic-server compact` subcommand (`server/src/bin.rs`). PR in flight. + [`disk-storage-and-persistence-optimization.md`](./disk-storage-and-persistence-optimization.md). +5. **npm is seven betas behind**: the workspace is at `0.41.0-beta.7`, the + registry serves `0.41.0-beta.0`. The `npm` job in `release.yml` exists + since 2026-09-15 and #1355 (merged 2026-09-18) fixed its build order and + publish step; the next `v*` tag publishes once the remaining + trusted-publisher entries are in place. [`production-readiness.md`](./production-readiness.md). +6. **The managed-node bootstrap grace has no reaper.** `SyncPolicy` admits + any drive for ten minutes after first sight (`lib/src/commit.rs`, + `lib/src/sync/policy.rs`) and nothing reaps drives that never enroll. + [`cloud-sync-managed-node.md`](./cloud-sync-managed-node.md) item 4, + [`foss-public-host-mode.md`](./foss-public-host-mode.md) Phase 3. +7. **Code-first schemas live only in an unmergeable PR.** `defineSchema` and + frozen `did:ad:` schemas are in #1262, a draft with conflicts last touched + 2026-09-01; nothing of it is on `develop`. The plugin model itself is no + longer in this position: #1500 (merged 2026-09-17) carried it onto + `develop`, and #1307 is now behind `develop`. [`json-schema-code-first.md`](./json-schema-code-first.md), + [`plugins.md`](./plugins.md). ## Active - [Drive sharing and hosting state](drive-sharing-state.md) β€” verified transition for unenrolled drives; authoritative seat counts and staging acceptance remain. -- [Website publishing](./website-publishing.md) β€” FOSS publication adapter and shared contract for managed SaaS hosting. - -- [Assistant-authored websites](./assistant-websites.md) β€” first local prototype validated; plugin abstraction audit and remaining SaaS deployment work. +- [Website publishing](./website-publishing.md) β€” FOSS publication on `develop` since #1500 (2026-09-17); managed SaaS adapter and the open follow-ups carried from that PR's handoff. +- [Assistant-authored websites](./assistant-websites.md) β€” prototype merged with #1500; plugin abstraction audit and remaining SaaS deployment work. Remaining work, not "this file exists." @@ -63,7 +89,7 @@ browser flow; standalone recovery remains self-managed. | Document | Status | | --- | --- | -| [`production-readiness.md`](./production-readiness.md) | **Gate list.** What stands between `develop` and production: npm publishing, rate limiting, library-owned durability, desktop CSP, managed-node abuse gate, source maps, SaaS billing checks. | +| [`production-readiness.md`](./production-readiness.md) | **Gate list.** Still open between `develop` and production: the first npm publish on a `v*` tag (job landed, #1355 merged 2026-09-18), green paired server/SaaS CI, the managed-node abuse gate, private source maps, backend synthetic error reporting, SaaS billing checks. Rate limiting, library-owned durability and desktop CSP shipped 2026-09-15. | | [`security-audit-2026-09.md`](./security-audit-2026-09.md) | **Mostly fixed** (beta.6, plus B7 CSP, C18 and rate limits on 2026-09-15). Open: C16 process-global import flags, C17 DID watched-query leak, C20, C24 loopback NFS, permissive CORS, client errors as 500, section F, transitive advisories via actix-http and iroh 0.35. | | [`drive-sharing-state.md`](./drive-sharing-state.md) | **In progress.** Verified transition for unenrolled drives shipped (#1466). Remaining: authoritative per-drive editor usage from the backend, root cause of the retained remote routing, staging acceptance. | | [`cloud-subscription-panel.md`](./cloud-subscription-panel.md) | **Partial.** Profile and link-invite steps shipped. Remaining: SaaS email invitation to drive authorization in one journey, invitation usage limits, paid-seat approval, real seat counts, 50 GB pool. | @@ -86,7 +112,7 @@ browser flow; standalone recovery remains self-managed. | [`serverless-p2p.md`](./serverless-p2p.md) | **Planned.** Device sync without a hub (written same-agent-first; admission is rights-based since 2026-07-17). AUTH-before-SYNC and the `AUTH.requestedSubject`↔drive binding landed 2026-09-01 (Iroh). Live-link destroys travel as signed `COMMIT` frames since 2026-09-03. P0 remaining: require envelopes on every `remove[]` (`Tree::Envelopes` exists; nothing blocks it). `AtomicTransport` / `SyncSession::serve` first slice landed 2026-09-05 with only `ChannelTransport`; outbox port and the four `sync_drive_with_peer*` variants are open. | | [`foss-public-host-mode.md`](./foss-public-host-mode.md) | **Partial.** Phase 1–2 built; OQ5 library path closed 2026-09-05 (`admit_unknown_drive`: Public never creates, Owner enrolls only the owner). Phase 3 (rate limits, Iroh stream refusal) is untouched. | | [`authorization-sync.md`](./authorization-sync.md) | **Draft.** P1 done, P2 partial (`classify_auth_impact` exists). P3 open: no `AuthorizationProof`, signer still auto-inserted into `write`, `genesis_signer()` has no non-test caller. P4 open: `SYNC_PUSH` imports raw Loro on a drive-level verdict. | -| [`unified-data-layer.md`](./unified-data-layer.md) | **Partial.** Browser/JS: one ingress, one outbox, one subscription model. Atomic writes, outbox, ingress entry points and immutable read/save subscriptions shipped. Remaining: consumer migration (38 `addResource` call sites, 42 `CommitBuilder` refs) and boundary extraction. | +| [`unified-data-layer.md`](./unified-data-layer.md) | **Partial.** Browser/JS: one ingress, one outbox, one subscription model. Atomic writes, outbox, ingress entry points and immutable read/save subscriptions shipped. Remaining: boundary extraction and the `CommitBuilder` review (31 non-test references, all in `browser/lib/src`; `addResource(` has no non-test call site left in `browser/data-browser/src`, counted 2026-09-18). | | [`loro-source-of-truth.md`](./loro-source-of-truth.md) | **Partial.** Sparse `datatypes` map + Phase 2a–2c shipped (`Tree::Resources` is a derived cache). Remaining: drop the untagged heuristic and the 3-way `build_state_doc` fallback, snapshot backfill (Phase 4), Phase 1.6 `Value` reshape (~966 sites), Flutter undo. | | [`atomic-lib-runtime.md`](./atomic-lib-runtime.md) | **Partial.** `AtomicNode` is the binding runtime; the WASM `ClientDb` is its only adapter. Flutter and desktop still hold a raw `Db`; the node has no blob API; `NodeConfig` was cut. Library-owned durable flush is the first slice. Local KV FTS landed in [`local-search.md`](./local-search.md). | | [`genesis-self-verifying.md`](./genesis-self-verifying.md) | **Partial.** Server and browser mint and verify inline genesis certs. Remaining: DataRoute verify UI, `genesis` propval immutability, cert-signed `drive` in `check_rights`. | @@ -102,16 +128,16 @@ browser flow; standalone recovery remains self-managed. | [`unified-templates.md`](./unified-templates.md) | **Initial slice shipped** (#1428: catalogue, editable previews, template chat). Remaining: portable format, website adapter, demo-lifecycle extraction, provenance and resume, live-AI acceptance. | | [`dashboards.md`](./dashboards.md) | **First slice shipped**, reachable from its table as a `dashboard` view tab since 2026-09-16. Open: the set-level action verb, parameters, templates shipping a dashboard. | | [`content-i18n.md`](./content-i18n.md) | **LocalizedText + template locales shipped.** Nothing in the app resolves translation siblings. Remaining: TranslationsBar, `useTranslation`, `/query` `lang`, search language filter. | -| [`website-templates.md`](./website-templates.md) | Template repair complete (DID), two-locale E2E exists. Its CMS list is the website view of `drafts-and-suggestions.md` and `content-i18n.md`. Open: publication visibility, CMS origin, in-page edit affordance, canonical paths. PRs #1498 and #1500 (assistant-designed and self-hosted sites) are in flight. | +| [`website-templates.md`](./website-templates.md) | Template repair complete (DID), two-locale E2E exists. Its CMS list is the website view of `drafts-and-suggestions.md` and `content-i18n.md`. Open: publication visibility, CMS origin, in-page edit affordance, canonical paths. Self-hosted publishing (#1500) merged 2026-09-17; the assistant-designed-sites prototype PR #1498 is still an open draft against `feat/plugin-model`. | | [`structural-problems-index.md`](./structural-problems-index.md) | **Live index.** React subscription audit is partial; save-state APIs shipped with two consumers. Browser metadata cleanup and subject-brand consumers remain; server subscription work is complete. | | [`react-compiler-resource-proxy.md`](./react-compiler-resource-proxy.md) | **Partial.** Immutable read/save status hooks and the data-inspector subscription shipped; about 13% of property reads are still render-time getters, migrating one regression at a time. | | [`canvas-undo-consolidation.md`](./canvas-undo-consolidation.md) | Phase A + C landed (browser). Phase B (Flutter action-stack removal, `replace_list_items`) open and unstarted. | | [`index-performance.md`](./index-performance.md) | First tranche shipped. Remaining: exact counts, typed sort keys, batched reads, watched-filter LRU, re-profiling. Structural permission-check fix is `zones.md`, not built. | -| [`disk-storage-and-persistence-optimization.md`](./disk-storage-and-persistence-optimization.md) | **Proposal.** Full-snapshot writes per commit, no auto-compaction, O(file) open fsync, SIGTERM path. Fresh e2e data dirs are done. | +| [`disk-storage-and-persistence-optimization.md`](./disk-storage-and-persistence-optimization.md) | **Proposal, PR in flight** (gap 4). Full-snapshot writes per commit, compaction only via the CLI, O(file) open fsync, SIGTERM path. Fresh e2e data dirs are done. | | [`virtual-drive.md`](./virtual-drive.md) | **Shipped** as a local NFS mount in the Tauri desktop app (`desktop/src/vfs.rs`, unauthenticated loopback, audit C24). Still proposal: headless-server mount, FUSE/WinFSP, native cloud-sync APIs, mobile providers. | | [`commit-retention-and-state-certificates.md`](./commit-retention-and-state-certificates.md) | **Superseded.** Envelope-on-resource replaced the retention design; Phase 1 and 2.5 shipped. Only the `stateHash` certificate and per-resource `retention` propval remain from this document. | | [`s3-blob-storage.md`](./s3-blob-storage.md) | **Partial.** Server-wide S3, verified migration and SaaS enforcement implemented. The backend trait is `get/put/size` only: streaming, delete/GC, per-tenant configuration and encrypted Vault attachments remain. | -| [`plugins.md`](./plugins.md) | **Partial, off `develop`** β€” one plugin model (`run` end to end, per-app agents, unattended runs). The code lives on `feat/plugin-model` (PR #1307, 532 files, conflicts with `develop`) and #1482. On `develop` the plugin RPC still answers "not implemented". | +| [`plugins.md`](./plugins.md) | **Partial, on `develop` since #1500** (2026-09-17) β€” one plugin model (`run` end to end, per-app agents, unattended runs): `browser/plugin`, `browser/lib/src/plugin-run.ts`, `integrations/`, the `wasm-plugins` server routes. PR #1307 is still open but behind `develop` (one file only on that branch). Remaining work is the list in #1307 and `extension-architecture.md`. | | [`json-schema-code-first.md`](./json-schema-code-first.md) | **Proposal**; nothing on `develop`. `defineSchema` + frozen `did:ad:` schemas are in PR #1262, a draft last touched 2026-09-01. | | [`android-data-reuse.md`](./android-data-reuse.md) | **Draft.** One store/agent/Iroh node per Android device. Nothing built. Supersedes `on-device-atomic-daemon.md`. | | [`SDK-API-design.md`](./SDK-API-design.md) | SDK / agent DX direction. | @@ -144,8 +170,8 @@ Not top-level plans. Indexed so they do not go missing. | [`arc-actor-message-payloads.md`](./arc-actor-message-payloads.md) | **Stretch shipped.** `SendFrame { Arc<[u8]> }` encode-once is in; only the `atomic_lib` `loro_update: Vec` question remains and is low value now. | | [`sync-onboarding-ux.md`](./sync-onboarding-ux.md) | Reference. Cross-client copy for what can reach what. Companion to [`device-pairing.md`](./device-pairing.md). | | [`main-drive-and-paths.md`](./main-drive-and-paths.md) | Strategy. DID-branch deployment: root drive, legacy URLs, human-readable paths. Phases 1–3 unstarted. | -| [`actions.md`](./actions.md) | **Steps 1–4 shipped.** Registry drives ⌘M, ⌘K (capped prefix match), hotkeys, the shortcuts overlay/page, and simple AI tools. Remaining: MCP projection when a server exists (PR #1347 is the plan). | -| [`silent-failures.md`](./silent-failures.md) | Living log of error-handling failures that reported success (2026-08-21). Carries M8 from the pairing field test and the unreproduced Safari fork query. | +| [`actions.md`](./actions.md) | **Steps 1–4 shipped.** Registry drives ⌘M, ⌘K (capped prefix match), hotkeys, the shortcuts overlay/page, and simple AI tools. Remaining: MCP projection when a server exists ([`mcp-endpoint.md`](./mcp-endpoint.md), #1347 merged 2026-09-18). | +| [`silent-failures.md`](./silent-failures.md) | Living log of error-handling failures that reported success (2026-08-21). Carries the two open query-index entries (gap 1), M8 from the pairing field test, the unreproduced Safari fork query and the e2e harness traps from the #1500 pass. | Closed decisions, as-built records, closed explorations and fixed notes live in [`completed/`](./completed/): the five decisions above, the 2026-07 sync diff --git a/planning/WEBSITE_HANDOFF.md b/planning/WEBSITE_HANDOFF.md deleted file mode 100644 index 7b458af6b..000000000 --- a/planning/WEBSITE_HANDOFF.md +++ /dev/null @@ -1,332 +0,0 @@ -# Atomic websites: implementation handoff - -Updated 15 September 2026. Read this before continuing work. This is a working -prototype with focused tests, not a production-ready hosting product. - -## Start here - -- Repository: `ontola/atomic-server`, checkout `/private/tmp/atomic-pr-website` - (`/tmp/atomic-pr-website` is the same location). -- Branch: `codex/website-self-hosted-publishing`. -- Draft PR: https://github.com/ontola/atomic-server/pull/1500 -- PR base: `codex/assistant-website-publication`, **not develop**. This is stacked - on the Assistant website work originating from the plugin work. Inspect the - current dependency/merge state before changing its base. -- Last implementation commit: `cb3e6ae00` (pushed). The handoff itself follows it. -- SaaS checkout: `/Users/joep/dev/atomic-saas`. The implemented hosting backend in - this PR is in atomic-server; a managed SaaS publishing adapter is still needed. -- Read repository AGENTS.md files and `browser/data-browser/UI_COMPONENTS.md` - before editing. Check git status and preserve existing work. -- Companion architecture document: [website-publishing.md](website-publishing.md). - This handoff supersedes its stale references to a custom β€œPublishing options” - menu: website actions now use the standard resource More menu. - -Do not reset the user's database or identity, overwrite another checkout, or -restart unrelated servers. Create fresh test resources for reproduction. Never -copy invitation tokens, signing secrets or credentials into docs/logs. - -## Product intent and decisions - -The user opens Atomic and asks, β€œI want a website for my bakery.” The Assistant -finds existing Products, prices, documents and photos, then creates/designs the -website using those resources. Atomic remains the source of truth; the website -must not become a disconnected copy of editable content. - -The user edits through normal tables/documents or inline in the preview. Those -edits save to the original Atomic resources. **Production changes only after -Publish site / Update site.** This explicit publication boundary is intentional, -including after inline editing. Rollback changes delivered website content, not -source records, operational app data or form responses. - -UX requirements from repeated user feedback: - -- One blue primary button at a time: Publish site or Update site. One action - performs snapshot preparation, upload and activation; no mandatory manual - release/upload/review sequence. -- Hosting status refreshes automatically. No β€œRefresh hosting status” button. -- β€œEdit with AI” is now back next to β€œEdit on page” in the preview toolbar as a - subtle button. Both that shortcut and the More action share the same callback. - The More action currently retains the label β€œDesign with AI”. -- View site, versions, unpublish and export operations use standard context - actions. Page-specific actions precede generic actions with one divider. -- Normal View/Data View are ordinary icon actions; hide the active view action. -- Use existing ResourceRow/components for content, meaningful labels and types. - Add content is a button. Avoid a custom CMS UI full of technical export terms. -- Do not reload the preview merely because AI/sidebar state changes. -- Do not add version resources to the sidebar or version-specific show-more UI. -- Show failures through `store.notifyError`, with a useful visible failure state; - don't leave a spinner forever or silently disable the only useful action. - -Longer-term intent: strong interactive apps as well as beautiful websites, reuse -the plugin abstractions and CMS editing controls, FOSS hosting plus managed SaaS. -**The current implementation is static-v1; that wider ambition is not delivered.** -CLI uploads are not the main authoring flow. Registrar/DNS purchasing is deferred. -Account entitlements should eventually be visible in Atomic, with actual payment -in the trusted SaaS account flow. No approved public pricing or automatic overages. - -## What is implemented - -### Authoring and preview - -`browser/data-browser/src/chunks/Website/` contains the implementation: - -| File | Responsibility | -| --- | --- | -| `websiteModel.ts`, `websiteTools.ts` | Website resource/config and Assistant authoring tools | -| `websiteExport.ts`, `renderWebsite.ts` | Selected source snapshot and static rendering | -| `WebsitePage.tsx` | Content sidebar, draft preview, AI shortcut, export context actions | -| `WebsitePreview.tsx`, `useWebsitePreviewHtml.ts` | Preview rendering and media URLs | -| `WebsiteInlinePreview.tsx`, `websiteInlineEditing.ts` | Authorized inline source editing | -| `WebsiteHosting.tsx`, `hostingClient.ts` | Status, publish/update, versions, rollback and HTTP client | -| `WebsiteExportPage.tsx` | Frozen version preview, including legacy exports | -| `websiteMedia.ts`, `websiteAssets.ts`, `optimizeWebsiteImage.ts` | Media selection, blobs and optimization | - -Preview subscriptions filter changes to website design rather than rebuilding on -any resource notification. Source changes still trigger debounced rebuilds. The -previous preview remains visible while refreshing. Unchanged output preserves -iframe identity. This is a correctness improvement, not a measured performance SLA. - -Inline editing supports selected string, integer and float table fields. Numbers -are parsed as numbers, reject invalid/nonfinite values, and integers require safe -whole values. Commits check current permissions, binding membership and conflicts. -Document sections mount the existing `CollaborativeEditor` in a React portal in -the preview iframe, reusing structured rich text and normal autosave. - -The latest fix passes the iframe body as `menuContainer` to the editor. Resource -and slash suggestion menus are adopted into that document before positioning; -keyboard scrolling uses its ownerDocument. Embedded editor backgrounds are -transparent. Styled-components styles target the iframe head. The preview remains -sandboxed without script permission; trusted Atomic code owns the editor/writes. - -### Versions and delivery - -- Browser uploads a version-1 WebsitePackage: code files, asset references and - optional **private** metadata for reconstructing previews. -- Server stores code and a version-2 manifest as immutable BLAKE3 blobs through - BlobBackend (local or configured internal S3). Images are separate HTTP blobs, - never base64/bytes embedded in HTML. S3 stays private behind serving endpoints. -- KV/redb stores membership, deployment IDs/timestamps, active pointer and history. - New exports create no Atomic child resources and no website-release pointer. - Legacy resource/KV exports remain readable; they are not deleted. -- Identical exports deduplicate. The current pilot caps distinct deployments at - 20 and activation history at 100. There is no blob garbage collector. -- Publish uses expectedRevision; stale activation gets HTTP 409. Failed upload or - build preserves the existing public site. Upload alone does not publish. -- Public files are served on a separate customer origin. Delivery does not read - the authoring drive. HTML pins runtime/assets to its version. -- Previously activated versions remain accessible while the site is published; - unpublishing hides them all. Downloaded copies cannot be recalled. -- Source edit authority is narrower than publication: current publishing requires - write access to the website and containing drive root. - -Server entry points: `lib/src/website.rs`, `lib/src/db/website.rs`, -`server/src/handlers/website.rs`, `server/src/blob_storage.rs`. - -Control API (signed requests): - -- GET `/website-hosting?project=...&drive=...`: status/history/URL. -- POST `/website-hosting/deployments?project=...&drive=...`: private package upload. -- GET `/website-hosting/preview/{deployment}?project=...&drive=...`: private JSON - package, no-store; never serve customer HTML on the editor origin. -- POST `/website-hosting/activate?project=...&drive=...`: - `{ expectedRevision, deployment }`; null unpublishes; an old ID rolls back. - -### Media - -Browser optimization preserves originals and generates derivatives with longest -edge at most 1920px, targeting 600 KB WebP. Small images and GIF animation are -preserved; large GIFs have an explicit rejection. Input caps are 50 MB/80MP. -Errors identify the source/field/page. Private unpublished asset availability is -currently tied to local cached blobs; see improvements below. - -## Improvements needed, in suggested order - -### 1. Finish validating inline RTE behavior (immediate) - -The user reported @/# appearing frozen and a white editor background. The last -commit fixes the known menu-document mismatch and background, and focused tests -pass. **Do not interpret this as all rich-text features verified.** - -Next reproduce in Firefox (the user's reported browser) and a real bakery page. -Test selecting a mention, arrow navigation, escaping then reopening, formatting -bubble menus, link/color popovers, image insertion/upload, tables, drag handles, -undo/redo, selection across blocks, and long/scrolled documents. Some shared RTE -components still refer to global document/window or use portal defaults. Audit -these against iframe ownership rather than loosening iframe sandbox permissions. -Check CSS/font inheritance, dark backgrounds and popup readability. `# ` with a -space is the heading input rule; a bare # is not established as a suggestion menu. - -Acceptance: each supported control works visibly in the iframe, saves to the -original resource, survives reload and does not leak keys/credentials to site code. -Add focused regressions for actual failures; don't build a separate rich-text editor. - -### 2. Preview speed, invalidation and actionable failures - -The user repeatedly reported slow β€œPreparing preview”, unnecessary reloads and -missing content errors. Some fixes exist; investigate remaining bottlenecks with -measurements (resource fetching, image decode/optimization, render, blob URL setup). -Avoid rebuilding/re-fetching every asset on unrelated changes. Cover remote source -updates, additions/removals in tables, replacing files, concurrent inline edits and -navigation. Preserve cursor/selection during relevant background updates. - -There is an Up to date / Unpublished changes state. Audit that it compares actual -published output to current draft across related rows/documents/assets; never -infer β€œclean” from an unavailable source or failed preview. Explain disabled -publish states. Missing resource errors should offer practical recovery (repair or -remove the selected binding) rather than leave a long DID and a dead end. Do not -silently publish with omitted private/unreadable content. - -### 3. Managed atomic-saas publishing (major missing deliverable) - -Implement a hosting destination adapter with the same status/upload/private-preview/ -activate semantics. Reuse the builder and package contract, not a second CMS. -SaaS must validate account-to-drive ownership, hosting entitlement and publishing -rights; use the trusted account session flow without forwarding signing secrets. -Use durable transactional compare-and-swap for activation, private object storage -for immutable blobs, and a customer-content serving origin independent of a -managed drive process. FOSS redb locking is not a multi-process SaaS transaction. - -Prove staging publication, private previews, unauthorized access denial, activation -conflict, failed-build preservation, rollback, unpublish, source drive offline, -and image delivery. Live S3/managed delivery has NOT been demonstrated here. -Coordinate atomic-server dependency pin updates only with compatibility checks. -Keep shared drive subscriptions from being charged again for every website. - -### 4. Storage/delivery production work - -Add retention and reference-aware GC (including failed uploads and concurrent -publication), quota enforcement/reporting, meaningful cap-reached UX, backups and -restore tests covering BOTH metadata and blobs. Measure cache behavior and serving -cost/latency. Never delete a blob still referenced by an active or retained version. -Verify internal S3 end to end, not merely the local BlobBackend path. Make unpublished -exports/assets portable across devices; currently local cache alone is insufficient. -Document manifest compatibility/migration and limits before treating this as stable. - -### 5. Reassess renderer/plugin abstractions for the actual product - -Current static rendering does not prove arbitrary AI designs or powerful interactive -apps. Review the plugin package, grants and bridge abstractions against design -freedom, structured bindings, reusable CMS controls and a standalone app host. -Prefer extending shared abstractions; don't create parallel form schemas or auth. -Current static CSP blocks arbitrary scripts, live network interactions/workers and -form actions. Changing that requires an explicit runtime/security design, not just -turning off restrictions to make a demo work. - -Forms should reuse the Forms implementation associated with PR #1281 once a -compatible revision is verified; its current merge status was not rechecked here. -Keep site deployment, form availability and private response rows independent. -Website rollback must not reopen forms or delete responses. Preview submissions -need an explicit test path. Domain registration/DNS and arbitrary runtimes remain -out of this immediate milestone. - -### 6. Remaining authoring/product polish - -Review document File embeds, a folder-to-gallery flow, labels/icons, responsive -layout and accessibility. Audit screenshot capture and Assistant input drag/drop -requests against current code before assuming completion; they were discussed but -are not established as delivered by this handoff. Likewise, prior chat persistence, -rate-limit/tool-result recovery and sidebar chat requests are separate work: inspect -their PRs before conflating them with website-hosting completion. - -## Local development and verification - -These are the previous session's local service paths; check listeners/process -ownership before using/restarting them. Do not replace user state. - -- Vite: `http://localhost:6763`, config `/private/tmp/hosting-vite.config.mts`, - cache `/private/tmp/hosting-vite-cache`, log `/private/tmp/hosting-vite.log`. - Aliases use this checkout's `browser/lib/dist/index.js` and `browser/react/src/index.ts`. -- Backend: `http://localhost:9897`, binary - `/private/tmp/atomic-website-rust-build/debug/atomic-server`. - Data/config/cache under `/private/tmp/atomic-hosting-node/{data,config,cache}`. -- Website origin: `ATOMIC_WEBSITE_ORIGIN=http://sites.localhost:9897`. - Public URLs are `http://.sites.localhost:9897/`. -- Do not restart the backend for UI-only changes. If Rust changes, ensure the - running binary actually includes them before claiming live validation. -- Vite/Wuchale updates locale .po files. Inspect those diffs; avoid extraction and - resets racing each other. HMR can disturb tests if edits continue during a run. - -From the atomic-server checkout: - -```sh -browser/node_modules/.bin/oxfmt -c browser/.oxfmtrc.json -browser/node_modules/.bin/oxlint -c browser/.oxlintrc.json -``` - -From `browser/data-browser`: - -```sh -../node_modules/.bin/tsc --noEmit -p tsconfig.json -``` - -Focused live browser checks (existing services required): - -```sh -FRONTEND_URL=http://localhost:6763 SERVER_URL=http://localhost:9897 WEBSITE_HOSTING_E2E=1 \ - browser/e2e/node_modules/.bin/playwright test \ - --config browser/e2e/playwright.config.ts \ - browser/e2e/tests/website-inline-content.spec.ts \ - browser/e2e/tests/website-preview-stability.spec.ts \ - --project chromium --workers 1 --trace off --reporter line -``` - -Other suites in `browser/e2e/tests`: `website.spec.ts`, `website-publishing.spec.ts`, -`website-media.spec.ts`, `website-errors.spec.ts`, `website-versions.spec.ts`, -`website-export.spec.ts`. Use the normal isolated E2E setup if existing services -aren't available; don't point destructive tests at the user's drive. - -### Evidence and limits - -At cb3e6ae00: the two focused Chromium tests above passed together (18.3s), -TypeScript passed, targeted lint had no errors (existing set-state-in-effect -warning in CollaborativeEditor). Tests cover visible slash/@ menus, heading -input, transparent editor background, numeric/document save and reload, and -AI opening without iframe replacement. They do not cover every RTE feature. - -Earlier commits passed focused publication/authoring/media/error/version browser -checks and unit/Rust tests. They were not all rerun after the final commit. -No claim of green full CI, Firefox parity, production deployment, load testing or -live S3 verification. Read current PR checks before merge. - -#### Session of 15 September 2026 (after f63987493) - -- `website-inline-rte.spec.ts` (new, fixture in `website-inline-fixture.ts`) - exercises bold, node select, the link popover, undo/redo, `@` mention - navigation/escape/reopen, slash escape/reopen, image picker, drag handle and - reload persistence inside the preview iframe. It passed in Firefox (2/2, via a - scratch config adding a Firefox project) and in Chromium, but see the flake below. -- Fixed: the link popover rendered in the app document, unstyled and at - iframe-relative coordinates (the "frozen" symptom). It now portals into the - bubble menu, which stays shown while the form is open (`BubbleMenu.tsx`, - `PopoverContainer` in `WebsiteInlinePreview.tsx`). The `Set link` button has a - title. Draft rebuilds are paused while inline editing (`WebsitePage.tsx`). -- Not fixed, pre-existing: characters intermittently vanish or land in a new - paragraph while typing in the iframe editor at human speed (40ms per key). - Reproduced with all session changes stashed (`website-inline-content.spec.ts` - failed 2/3). The main document editor also commits roughly once per keystroke - at that speed (measured 35 commits for 43 characters), so the save scheduler - is a suspect; `loro-prosemirror` logs "Cannot find the loroNode" around the - failures. Treat both inline specs as flaky in Chromium until this is found. -- Firefox at 0ms synthetic key delay mis-sequences keys (not a product bug); - adding `white-space: pre-wrap` to the editor made Chromium worse and was reverted. -- Website page: AI edit / Page edit buttons moved into the header row with - icons, "Content" and "Live draft preview" labels removed, preview fills the page. -- AI chats: titles are generated on the first user message (fallback on the - reply) and include an emoji shown in the sidebar and the chat panel heading. -- Chat loading: messages load in two rounds instead of one per message, and - `store.getResource` / the new `store.getResources` are local-first (embedded - database before the server). A `GET_MANY` (0x15) / `GET_MANY_RESULT` (0x16) - WebSocket frame pair, capability `get-many`, batches server fetches; on - localhost it measured within noise of pipelined single GETs (about 200ms - for 120 subjects either way), so the transport was not the bottleneck. - -## Suggested next session - -1. Confirm branch/status, read applicable repo instructions, and inspect PR checks. -2. Reproduce the user's inline RTE interactions in Firefox and address any remaining - ownerDocument/portal issues. Keep the working publication and source-save paths. -3. Measure preview work and fix one concrete bottleneck or stale-content case. -4. Implement the managed SaaS adapter as a separately reviewable milestone, with - coordinated compatibility tests and a real staging demo. -5. Report exactly what is verified, what remains a prototype, and which user-visible - behavior to test next. Do not substitute a toast/build for the requested outcome. diff --git a/planning/assistant-websites.md b/planning/assistant-websites.md index 89b31a4c6..78535dd7b 100644 --- a/planning/assistant-websites.md +++ b/planning/assistant-websites.md @@ -2,9 +2,11 @@ # Assistant-authored websites: first implementation and abstraction review -Status: first local prototype validated on `codex/assistant-website-publication`, based -on `feat/plugin-model` at `d0e49b8b9`. The source worktree's staged and unstaged -changes were preserved. No production deployment or domain integration. +Status: prototype built on `codex/assistant-website-publication` (PR #1498, still +an open draft against `feat/plugin-model`) and merged into `develop` with the +self-hosted publishing PR #1500 that was stacked on it (2026-09-17). The +validation notes below describe the prototype as it was validated on that +branch. No production deployment or domain integration. ## Product decisions from the discussion diff --git a/planning/disk-storage-and-persistence-optimization.md b/planning/disk-storage-and-persistence-optimization.md index aa5ea75ea..7d85c3a60 100644 --- a/planning/disk-storage-and-persistence-optimization.md +++ b/planning/disk-storage-and-persistence-optimization.md @@ -1,6 +1,7 @@ # Disk Storage & Persistence Optimization -> **Status:** Proposal (2026-06-04). Diagnoses why server boot time and overall +> **Status:** Proposal (2026-06-04); fix 1 and fix 2 have a PR in flight +> (2026-09-18). Diagnoses why server boot time and overall > performance degrade as the store grows, and proposes fixes. Builds on > [`commit-retention-and-state-certificates.md`](./commit-retention-and-state-certificates.md) > (history retention is node policy) and @@ -64,10 +65,11 @@ scales with the file, for two reasons the code already documents: Three compounding behaviors, all in code: 1. **Full Loro snapshots on every commit β€” not deltas.** `sign_at` - (`lib/src/commit.rs:1121`) calls `doc.export_snapshot()`, serializing the + (`lib/src/commit.rs`, `fn sign_at` near line 1493 on 2026-09-18) calls + `doc.export_snapshot()`, serializing the resource's **entire** CRDT document after applying the change. A 1-character title edit re-stores the whole resource's Loro state. **NB:** the docstring - at `commit.rs:1088` says *"an incremental update is exported"*, but the call + just above it says *"an incremental update is exported"*, but the call is `export_snapshot()` (full). This mismatch looks **unintended** and is the single biggest growth lever. 2. **History retention.** Each commit is retained (the `previousCommit` audit @@ -106,7 +108,7 @@ what the `commit.rs:1088` docstring already intended. - The materialized "current state" snapshot per resource (`TABLE_LORO_SNAPSHOTS`) can stay full for fast reads; it's the **per-commit `loroUpdate`** that should be a delta β€” that's the part multiplied by history. -- **First step:** confirm `export_snapshot()` at `commit.rs:1121` is an +- **First step:** confirm `export_snapshot()` in `sign_at` is an unintended full-write (vs. a deliberate choice for import-merge correctness), and that `from` (the prior version vector) is available at sign time. - **Downstream wins:** the per-commit `loroUpdate` is also the payload that gets @@ -161,7 +163,7 @@ change, but it's the fastest path to a non-flaky suite and isolates real bugs ## Open questions -- Is `export_snapshot()` at `commit.rs:1121` load-bearing for import-merge +- Is `export_snapshot()` in `sign_at` load-bearing for import-merge correctness, or safe to switch to an incremental export? (Determines fix #1.) - What's the dead-page ratio of a real aged store? Run `atomic-server compact` on a production-like store and compare before/after size to size the win. @@ -174,7 +176,7 @@ change, but it's the fastest path to a non-flaky suite and isolates real bugs - `lib/src/db/redb_store.rs:93-160` β€” open path, fsync/repair cost notes, `compact_file`, `set_quick_repair`. -- `lib/src/commit.rs:1085-1124` β€” `sign_at`; `export_snapshot()` at line 1121 - (full snapshot despite "incremental" docstring at 1088). -- `server/src/bin.rs:119` β€” manual `compact` subcommand wiring. +- `lib/src/commit.rs` β€” `sign_at` (about line 1490 on 2026-09-18); + `export_snapshot()` at its end, full snapshot despite the "incremental" docstring. +- `server/src/bin.rs` β€” manual `compact` subcommand wiring (`Command::Compact`). - `TABLE_LORO_SNAPSHOTS`, `TABLE_BLOBS` β€” per-resource snapshot + blob storage. diff --git a/planning/e2e-concurrency.md b/planning/e2e-concurrency.md index 0df1baefd..675ca02b3 100644 --- a/planning/e2e-concurrency.md +++ b/planning/e2e-concurrency.md @@ -388,3 +388,18 @@ at the selected position. Use the existing waitForGridMounted after reload befor selecting the row. Keyboard assertions and timeouts are unchanged. E2E typecheck passes; focused browser validation and the complete comparison remain pending. Trace: Mancave /tmp/e2e-1465-shift-enter-trace.zip. + +## Local worker count versus CI (from the #1500 pass, 2026-09-16/17) + +CI runs **two workers per shard, four shards on Mancave, each shard with its +own `atomic-server`**, with retries (`.dagger/src/index.ts`, `HOST_PROFILES`). +A local four-worker run against one server is much harsher than that and +invents failures: on the #1500 branch `documents`, `username-live` and two +`apps` tests failed at four workers and passed at two, and a rotating pair of +specs (`plugins:909`, `tables:482`, `drive-catalog`, `offline-tables`, ...) +failed purely from load, different each run. Measure locally with +`PLAYWRIGHT_WORKERS=2 PLAYWRIGHT_RETRIES=1` and re-run any red in isolation +before treating it as real. The full-suite numbers on that branch moved from +41 failed to 5 failed / 4 flaky in a day, and roughly half of the original +list was environment (no server running, wrong dependencies, desynced i18n +catalogs, a 1.1 GB data dir) rather than product. diff --git a/planning/e2e-diagnostic-hygiene.md b/planning/e2e-diagnostic-hygiene.md index d19ad7e68..dc49a7226 100644 --- a/planning/e2e-diagnostic-hygiene.md +++ b/planning/e2e-diagnostic-hygiene.md @@ -152,3 +152,36 @@ Fixed statement spacing, shadowed test variables and the database-handoff mock's Validation after cleanup: 381 library tests, nine diagnostic-fixture self-tests, the database-handoff regression and the app typecheck pass. Generated local `dist-hosted` output was moved outside the source tree before linting. + +## Open after the #1500 pass (2026-09-17) + +Carried from that PR's e2e handoff when it was deleted. Fixed on the way and +worth knowing about: a spec can declare a diagnostic that only some topologies +produce (`expect(..., { optional: true })` in `diagnostic-collector.ts`, +because `recovery-option`'s service-worker warning exists only in a production +build); the app no longer fetches the server root for an empty subject +(`normalizeSubject('')` resolved to `http://host/`, which 404s on a DID-drive +server and logged two console errors per render β€” that was the whole +"diagnostics group" of failures); `useChildren` re-reads a page older than its +own count and a query whose answer predates a new child. + +- [ ] `website-inline-rte` / `website-inline-content` flake: see + [`website-publishing.md`](./website-publishing.md) Remaining work 1. +- [ ] `table-create-perf` (3) and `opfs-init-perf` fail against a Vite dev + server because the `clientdb.*` perf marks are absent from the dev module + graph; a production build made locally failed them differently + (`/app/dev-drive` never reached a drive), so they are unexplained rather + than explained. `recovery-option` is dev-topology only and now declares + its diagnostic as optional. +- [ ] `vault-backup-restore` skips without the atomic-saas control plane on + :3030 (`vault-stack.sh`); it has not been run green against this + branch's own portal. +- [ ] `useDriveClass` (`chunks/PluginRuns/runScript.ts`) subscribes to the + drive's `defaultOntology` only if it exists when the hook mounts; a drive + that gains its first plugin schema in this tab does not re-resolve, so a + freshly created app renders the generic resource page until a reload. A + fix (also subscribe to the drive resource) was written and reverted + because it needs a test that proves it; the `apps` specs pass since the + schema-term lookup was batched, so the gap is no longer test-visible. +- [ ] `second-device-load @smoke` went from about 1 pass in 3 to 6 in 7 after + the `useChildren` fix; the remaining miss is not diagnosed. diff --git a/planning/extension-architecture.md b/planning/extension-architecture.md index 554406b47..6dc43e50e 100644 --- a/planning/extension-architecture.md +++ b/planning/extension-architecture.md @@ -2,7 +2,8 @@ Status: product direction agreed with the user, 2026-09-08; technical migration in progress. This describes the target, not guarantees already implemented. Current -extension work is on PR #1307; Reflector integration is proposed in PR #1383. +extension code reached `develop` with #1500 (2026-09-17); PR #1307 remains open +for its remaining-work list. Reflector integration is proposed in PR #1383. Workspace navigation, shared view protocol and installation identity have implementation checkpoints below; the complete package and permission migration remains open. diff --git a/planning/plugins.md b/planning/plugins.md index d982062d1..14adafa3f 100644 --- a/planning/plugins.md +++ b/planning/plugins.md @@ -5,7 +5,7 @@ defines the target boundaries and phased retirement of overlapping models. This document retains earlier design and implementation history; its β€œone model” title does not mean the existing UI bridges and lifecycles have already converged. -> **Status:** Partial, off `develop`. Track A built 2026-08-21 (`run` end to end: sandbox, planner, applier, secrets, server placement, scheduled and query-triggered runs, auto-apply); Track B revised 2026-08-22 onto that substrate. The code lives on `feat/plugin-model` (PR #1307). Absorbed `llm-wasm-gui-plugins.md`, `importers.md` and `habits-app.md` on 2026-09-01. +> **Status:** Partial, on `develop` since #1500 (2026-09-17), which was stacked on `feat/plugin-model` and carried it in; PR #1307 is still open but now behind `develop`. Track A built 2026-08-21 (`run` end to end: sandbox, planner, applier, secrets, server placement, scheduled and query-triggered runs, auto-apply); Track B revised 2026-08-22 onto that substrate. Absorbed `llm-wasm-gui-plugins.md`, `importers.md` and `habits-app.md` on 2026-09-01. ## Status diff --git a/planning/production-readiness.md b/planning/production-readiness.md index 86a220b46..f06bf6adf 100644 --- a/planning/production-readiness.md +++ b/planning/production-readiness.md @@ -9,7 +9,9 @@ owning plan where one exists. Verification logs stay out of this file. - [ ] **Publish `@tomic/*` to npm on a `v*` tag.** The registry still serves `0.41.0-beta.0` (2025-06) for `lib`, `react`, `svelte` and `cli`; `plugin` and `edit-mode` were never published; seven betas have shipped - since. The `npm` job now exists in `release.yml` (2026-09-15). Remaining: + since (the workspace is at `0.41.0-beta.7`). The `npm` job exists in + `release.yml` since 2026-09-15; #1355 (merged 2026-09-18) fixed its + build order (`@tomic/lib` first) and publishes with `--ignore-scripts`. Remaining: trusted-publisher entries on npmjs.com for `@tomic/plugin` and `@tomic/edit-mode` (the other five are configured), then the next tag. - [ ] Green full CI on the release commit including the atomic-saas @@ -26,7 +28,7 @@ owning plan where one exists. Verification logs stay out of this file. - [x] Library-owned durable flush so the Flutter binding stops losing writes on app kill ([`atomic-lib-runtime.md`](./atomic-lib-runtime.md), 2026-09-15). - [x] Desktop CSP (audit B7) set 2026-09-15; needs one packaged-build smoke test before a release. -- [x] npm publish job in `release.yml` (2026-09-15, ported from #1355). First tag after merge publishes seven `@tomic/*` packages through Trusted Publishing; `plugin` and `edit-mode` need their npmjs.com trusted-publisher entries first. +- [x] npm publish job in `release.yml` (2026-09-15, ported from #1355; #1355 itself merged 2026-09-18). First tag after merge publishes seven `@tomic/*` packages through Trusted Publishing; `plugin` and `edit-mode` need their npmjs.com trusted-publisher entries first. - [ ] Managed-node paid-abuse gate: the bootstrap grace admits any drive for ten minutes with no reaper ([`cloud-sync-managed-node.md`](./cloud-sync-managed-node.md) item 4). diff --git a/planning/silent-failures.md b/planning/silent-failures.md index 15ea998ac..d5e01e945 100644 --- a/planning/silent-failures.md +++ b/planning/silent-failures.md @@ -81,6 +81,10 @@ away. *Should have:* wiped it. A warning is the wrong shape when the consequence is a failure list that changes every run. *Fixed 2026-08-18 (87e12c9a) β€” wipes by default, `--keep-store` to override.* +The same shape returned on the #1500 branch with a custom data dir that +`e2e-server.sh` does not manage: at 1.1 GB, all six `apps` tests and most of +`plugins` failed a 10s `iframe[title="App"]` wait; a fresh data dir took the +two specs from 16 failures to 1. Wipe before any full run. **`waitForSearchIndex(page)` with no query is a 1.5s sleep wearing a readiness-helper's name.** @@ -94,6 +98,61 @@ a name that promises a readiness check. Passing two spec files with two `-g` filters ran one test and reported `1 passed`. Nothing indicated the other filter had been discarded. +**redb's lock outlives the listening socket, so a restart that waits for the +port runs the suite against nothing.** *(open)* +Killing the server frees the port within a second, but `atomic.redb` stays +locked for several more. A restart that only waits for the port exits at once +with `Failed to create redb ...: Database already open. Cannot acquire lock.`, +and every spec then fails in ways that look like product defects: missing menu +items, disabled buttons, empty panels. On the #1500 branch this accounted for +most of one session's "unexplained" failures. `e2e-server.sh` refuses a busy +port before starting, but nothing in the repo waits for HTTP 200 or retries the +spawn. +*Should have:* gate readiness on `curl` returning 200, never on the port, and +retry the spawn (not just the wait) when the log says "Database already open". +Give a fresh server about ten seconds before the first spec; within that +window `menu-item-website-prepare` renders but stays disabled. + +**Compiled i18n catalogs silently shift every string.** +The `wuchale` plugin rewrites `src/locales/*.po` while Vite runs; the compiled +catalogs live in `src/locales/.wuchale/`, which is gitignored. So +`git checkout -- src/locales/` restores the `.po` files and leaves compiled +artifacts from some other state, and every indexed string lookup shifts. +Missing entries render as `[i18n-404:NNN]`, which is visible; *shifted* entries +render as a different real string, which is not. That is how +`iframe[title="Website preview"]` became a Google Calendar sentence and every +website spec failed, an hour of bisecting code later. A measurement taken with +`src/locales/` dirty is suspect; one revert on the #1500 branch was made on +exactly such a measurement. Making the `.po` files read-only does not help: the +`.wuchale` output goes stale instead. +*Should have:* the compiled output regenerated whenever the `.po` files change +under it, or a refusal to serve when the two disagree. Until then: delete +`src/locales/.wuchale` together with any `.po` reset and restart Vite (rule in +`AGENTS.md`). + +**Vite serves a stale pre-bundle after a dependency's `dist` is edited.** +Editing an installed package's `dist/index.js` to instrument it is invisible +until the Vite cache directory is cleared; one measurement showed zero events +for that reason alone. +*Should have:* said which pre-bundle it was serving and when it was built. + +**A dependency install borrowed from another checkout.** +Every `node_modules` under `browser/` was a symlink into a different branch's +checkout, so this branch's `pnpm-lock.yaml` had never been installed: +`browser/patches/loro-prosemirror@0.4.3.patch` was not applied (the inline +editors dropped characters) and the production build died with +`null pointer passed to rust`, failing every `@smoke` test. Both read as product +bugs. A real `pnpm install --frozen-lockfile` fixed both. +*Should have:* the runner refusing to start when `node_modules` does not match +the lockfile, as `test-server` already refuses a stale binary. + +**`normalizeSubject('')` resolved to the server root.** +`useResource(x ?? '')` and every render before the drive setting hydrated +fetched `http://host/`, which 404s on a DID-drive server and logged two console +errors each time. It failed the zero-diagnostics gate in four specs and looked +like an architectural "nothing lives at the root" problem. +*Fixed 2026-09-16 (027c090) β€” the empty string is not a subject.* + **No `webServer` block, so every spec fails on a dead port.** With no dev server on 6747 the whole suite fails at `page.goto` with `ERR_CONNECTION_REFUSED`, which reads as a suite-wide code regression. diff --git a/planning/start-server-9897.sh b/planning/start-server-9897.sh deleted file mode 100755 index bd4bda8ea..000000000 --- a/planning/start-server-9897.sh +++ /dev/null @@ -1,36 +0,0 @@ -#!/bin/bash -# Start the e2e atomic-server on 9897 and do not return until it answers HTTP 200. -# redb's lock outlives the listening socket, so retry the spawn, not just the wait. -set -u -DATA=/private/tmp/atomic-hosting-node -BIN=/private/tmp/atomic-website-rust-build/debug/atomic-server -LOG=/tmp/server-9897.log - -if curl -s --max-time 2 -o /dev/null -w '' http://localhost:9897/ 2>/dev/null; then - code=$(curl -s --max-time 2 -o /dev/null -w "%{http_code}" http://localhost:9897/) - [ "$code" = "200" ] && { echo "already up"; exit 0; } -fi - -for attempt in 1 2 3 4 5 6; do - cd "$DATA" || exit 1 - ATOMIC_WEBSITE_ORIGIN=http://sites.localhost:9897 \ - ATOMIC_INITIALIZE=true \ - ATOMIC_INTEGRATION_PROXY_URL=http://127.0.0.1:19090 \ - TENANT_SECRET='bW9jay10ZW5hbnQ.mock-signature' \ - ATOMIC_INTEGRATION_FRONTEND_ORIGIN=http://localhost:6763 \ - nohup "$BIN" --port 9897 --ip 127.0.0.1 \ - --data-dir "$DATA/data" --config-dir "$DATA/config" --cache-dir "$DATA/cache" \ - > "$LOG" 2>&1 & - for i in $(seq 1 25); do - code=$(curl -s --max-time 2 -o /dev/null -w "%{http_code}" http://localhost:9897/ 2>/dev/null) - [ "$code" = "200" ] && { echo "server up (attempt $attempt, ${i}s)"; exit 0; } - sleep 1 - done - if grep -q "Database already open" "$LOG"; then - echo "attempt $attempt: redb lock still held, retrying" - sleep 5 - continue - fi - echo "attempt $attempt: did not become healthy"; tail -5 "$LOG"; sleep 3 -done -echo "FAILED to start server"; tail -20 "$LOG"; exit 1 diff --git a/planning/unified-data-layer.md b/planning/unified-data-layer.md index 9986cb9bf..937ff38e6 100644 --- a/planning/unified-data-layer.md +++ b/planning/unified-data-layer.md @@ -73,9 +73,14 @@ See [unify-resource-dirty-signals.md](./unify-resource-dirty-signals.md). regression before each flow change. Keep read and persistence state separate. - [ ] Audit direct `addResource` producers before further ingress consolidation. Preserve normalization, alias handling, echo imports/dedup, unsaved local state, - atomic persistence and notification ordering. + atomic persistence and notification ordering. Counted 2026-09-18: no non-test + `addResource(` call site remains in `browser/data-browser/src` (one in a + `.test.ts`); the remaining producers are inside `browser/lib/src`. - [ ] Review remaining Resource signing/retry responsibilities and `CommitBuilder` - consumers as a separate change. Preserve immutable signed envelopes and export + consumers as a separate change. Counted 2026-09-18: 31 `CommitBuilder` + references outside `*.test.ts`, all in `browser/lib/src` (`commit.ts` 18, + `resource.ts` 10, `store.ts` 2, `test-store.ts` 1), none in `browser/react/src` + or `browser/data-browser/src`. Preserve immutable signed envelopes and export baselines; do not conflate queued writes with persisted local durability. - [ ] Evaluate whether a persistence facade removes meaningful duplication. Existing atomic writes must remain intact; index-only seeding needs an explicit path. diff --git a/planning/website-publishing.md b/planning/website-publishing.md index db20a295b..af545d86b 100644 --- a/planning/website-publishing.md +++ b/planning/website-publishing.md @@ -1,9 +1,9 @@ # Website publishing: self-hosted first, managed hosting alongside it -Status: first FOSS implementation, built on the Assistant website prototype. - -For current implementation status, verification limits and prioritized follow-up -work, read [the implementation handoff](WEBSITE_HANDOFF.md). +Status: FOSS implementation on `develop` since #1500 (merged 2026-09-17), built +on the Assistant website prototype (#1498, still an open draft). The managed SaaS +adapter is not started. The open follow-ups from the #1500 handoff are in +[Remaining work](#remaining-work) below. ## Contract and boundaries @@ -79,10 +79,12 @@ Create a website and click **Publish site**. Later edits go live with **Update site**. That single explicit action saves the draft snapshot, uploads it and conditionally activates it. Conflicting publication changes are refused; the current site stays available if saving/uploading fails. One blue primary button -is visible at a time. **Publishing options** contains version restore and -unpublish. The same **Publishing options** menu contains export controls; there is no separate versions panel. Status loads on -mount, after actions and when the browser window regains focus; no manual -refresh control is exposed. Action errors use `store.notifyError` for the +is visible at a time. Version restore, unpublish, export, Design with AI and +View site are entries in the standard resource More menu (registered through +`useCustomContextItems` in `WebsitePage.tsx` / `WebsiteHosting.tsx`); versions +open a shared dialog and there is no separate "Publishing options" menu or +versions panel. Status loads on mount, after actions and when the browser +window regains focus; no manual refresh control is exposed. Action errors use `store.notifyError` for the standard toast and logging pipeline. API routes (all control requests signed with the existing Atomic request proof): @@ -147,3 +149,66 @@ structured rich text and its normal autosave path. Source permissions and privat authoring checks apply before enabling either editor. The iframe has no scripts or credentials; React and writes are owned by the Atomic host. Changes update the draft; publication still requires Update site. + +## Remaining work + +Carried from the #1500 handoff (2026-09-15/17) when it was deleted. In +suggested order; none of these is a regression against `develop`. + +1. **Inline rich text in the preview iframe is not fully verified.** Focused + Chromium and Firefox runs pass, but not every control is covered: mention + selection and arrow navigation, bubble menus, link and colour popovers, image + upload, tables, drag handles, undo/redo, cross-block selection, long + documents. Some shared RTE components still reference the global + `document`/`window` or portal defaults; audit them against iframe ownership + rather than loosening the sandbox. Two known flakes: + - `website-inline-content` / `website-inline-rte`: characters intermittently + vanish or land in a new paragraph when typing at human speed. The main + document editor also commits about once per keystroke at that speed, so + the save scheduler is a suspect; `loro-prosemirror` logs "Cannot find the + loroNode" around the failures. + - `website-inline-rte`: a popup (mention list, link field) open inside the + iframe is destroyed when the server echoes the author's own commit. The + echo stamps `lastCommit` on the resource's `properties` map; `LoroSyncPlugin` + subscribes to the whole doc, so it rebuilds the ProseMirror document for a + change that never touched `doc`. **Do not re-attempt filtering non-local + events in `updateNodeOnLoroEvent`**: measured, it broke the mention list + (the Suggestion plugin relies on those rebuild transactions) and did not + stop the character drop. The tractable fix is to make the tiptap + Suggestion plugin re-establish itself after a rebuild; then the filter + becomes viable. Product decision pending. +2. **Preview speed, invalidation and actionable failures.** Measure the + "Preparing preview" path (resource fetch, image decode/optimisation, render, + blob URL setup) before optimising; avoid re-fetching every asset on unrelated + changes; preserve cursor/selection during background updates. Audit the + Up to date / Unpublished changes state against actual published output, + never infer "clean" from an unavailable source. Missing-resource errors + should offer repair or removal of the binding, not a dead end with a DID. +3. **Managed atomic-saas publishing** (the major missing deliverable): a + destination adapter with the same status/upload/preview/activate semantics, + account-to-drive ownership, hosting entitlement, durable compare-and-swap + activation, private object storage and a customer-content origin + independent of the managed drive process. Prove staging publication, + private preview, unauthorised denial, activation conflict, failed-build + preservation, rollback, unpublish, source drive offline and image delivery. + Live S3/managed delivery has not been demonstrated. +4. **Storage and delivery production work**: retention and reference-aware + GC (failed uploads, concurrent publication), quota enforcement and + cap-reached UX, backup and restore covering metadata and blobs, cache and + serving cost measurements, portable unpublished exports across devices, + documented manifest compatibility and limits. +5. **Reassess renderer/plugin abstractions** against arbitrary AI designs and + interactive apps; the static CSP blocks scripts, network, workers and + forms, and changing that needs an explicit runtime/security design. Forms + should reuse the implementation from PR #1281 once a compatible revision + is verified (its merge status was not rechecked). Site deployment, form + availability and private response rows stay independent; rollback must + not reopen forms or delete responses. +6. **Authoring polish**: document File embeds, folder-to-gallery, labels and + icons, responsive layout, accessibility. Screenshot capture and Assistant + drag/drop were discussed but are not established as delivered. + +Harness notes that only matter for these specs: the website preview iframe +wants an origin separate from the app, so the production (embedded-in-server) +topology CI uses breaks the website specs while the Vite topology passes them; +do not compare results across topologies mid-investigation. From 58bc532cdced4ad9f62ef75964d228ea6307587f Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 05:41:13 +0200 Subject: [PATCH 006/110] Reconcilable host-to-Drive mappings, for hosted vanity subdomains (#1539) Adds Db::sync_drive_mappings, list_drive_mappings and managed_alias_hosts so a control plane that holds no user signing keys can install and reconcile host to Drive mappings without /bind-drive. Removal is scoped to the hosts a previous run installed, so the localhost entries and hand-made bindings survive. Mapping keys are normalized, since the Host header arrives in whatever case the client sent. A host bound to a missing Drive now 404s instead of falling through to the store root, which on a multi-tenant node answered one tenant's hostname with another namespace's content. --served-domain-suffix makes a server answer for *. without taking on the rest of --base-domain, which would also change how subjects are normalized on a box that already holds customer data. --- CHANGELOG.md | 19 +++ TESTING_COVERAGE.md | 32 ++++ lib/src/db.rs | 385 +++++++++++++++++++++++++++++++++++++++++- server/src/config.rs | 26 +++ server/src/context.rs | 42 ++++- 5 files changed, 495 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ec6fed867..edd5df417 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -69,6 +69,25 @@ See [STATUS.md](server/STATUS.md) to learn more about which features will remain `.map`). `AppState::init` drops its own core-models bootstrap branch: its "store did not exist" check ran after the store directory had been created, and `Db::init_redb_file` already seeds a fresh store on open. + +- `atomic_lib`: host-to-Drive mappings are reconcilable, so a control plane can + install hosted vanity subdomains without an agent that can sign on the + Drive's behalf. `Db::sync_drive_mappings` adds, repoints and removes a + desired set idempotently and scopes removal to the hosts it installed + itself (so `/bind-drive` bindings and the `localhost` entries from setup + survive); `Db::list_drive_mappings` and `Db::managed_alias_hosts` read the + table, which had no read path beyond a single-host lookup. Mapping keys are + now normalized, so a mixed-case `Host` header resolves the Drive it was + bound to instead of falling through to the store root. +- A host bound to a Drive now serves that Drive or answers 404. It used to + fall through to the store root when the bound Drive was missing β€” not synced + yet, or moved to another node β€” which on a multi-tenant server answers one + tenant's hostname with another namespace's content. +- New `--served-domain-suffix` / `ATOMIC_SERVED_DOMAIN_SUFFIX`: answer for + `*.` so the request origin follows the hostname the visitor used, + without `--base-domain`'s other effect of also becoming the store's base + domain (which changes how subjects are normalized and migrated, and is not + something to switch on for a server that already holds data). - The outbox drains over a live Iroh link too (`sync::peer::LivePeerCommitTransport`): a device with no hub in reach delivers its queued writes to a paired peer as signed `COMMIT` frames, which the peer validates and applies like a hub diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index f3942bb83..dfa6473f6 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -845,6 +845,38 @@ Cloud Vault display metadata: `vaultAutoBackup.test.ts` verifies name/emoji enro authenticated read from the real managed node. Plan purchase alone creates no enrollment. Real Stripe-hosted test-card checkout remains a deployment check. +## Host-to-Drive routing and hosted vanity subdomains + +`Tree::DriveMapping` is what makes one server answer for many hostnames. It +backs `/bind-drive` for self-hosters and hosted vanity subdomains for +`atomic-saas`, whose control plane reconciles it through +`Db::sync_drive_mappings`. + +- `db::drive_mapping_tests`: the reconcile a managed node runs on every policy + poll β€” add, repoint, remove; idempotent on an unchanged list; scoped so a + binding it did not install (including the `localhost` / `127.0.0.1` entries + from `setup_test_env`, and anything bound by hand through `/bind-drive`) is + never removed; keys normalized so a mixed-case `Host` still resolves; empty + hosts and empty drives skipped. +- `db::resolver_tests::a_bound_host_whose_drive_is_missing_does_not_serve_the_store_root`: + the multi-tenant leak. A host bound to a Drive this node does not hold (not + synced yet, or migrated away) must 404 rather than fall through to the store + root, which would answer one tenant's hostname with another namespace's + content. This is the property that lets the control plane authorize a + certificate on reservation instead of only after a node confirms. +- `context::tests::a_served_domain_suffix_accepts_tenants_without_a_base_domain`: + `--served-domain-suffix` makes the request origin follow the hostname the + visitor used, without turning on `--base-domain` and with it the store's + subject normalization. +- Paired `atomic-saas` coverage (registry, plan gating, `/caddy-ask`, the + heartbeat report) is listed in that repo's + `planning/TEST_COVERAGE_AND_CI.md`. + +**Not covered:** no test drives a real HTTP request against a vanity host +end to end β€” the reconcile and the resolver are tested separately, and joining +them needs the representative two-service environment. The multi-node gateway +routing that a second node would require does not exist yet. + ## Error reporting and feedback - `browser/data-browser/src/helpers/feedback.test.ts`: unavailable reporting, failed delivery, blank input and successful submission. diff --git a/lib/src/db.rs b/lib/src/db.rs index d13c9f389..185ef97b4 100644 --- a/lib/src/db.rs +++ b/lib/src/db.rs @@ -229,6 +229,48 @@ pub struct ReplicationTarget { pub authorized_by: String, } +/// What [`Db::sync_drive_mappings`] changed in the host β†’ Drive mapping table. +#[derive(Debug, Default, Clone, PartialEq, Eq)] +pub struct DriveMappingSync { + /// Hosts that had no mapping before. + pub added: Vec, + /// Hosts that pointed at a different Drive before. + pub updated: Vec, + /// Hosts an earlier reconcile installed and that the desired set no longer + /// names. + pub removed: Vec, +} + +impl DriveMappingSync { + /// Whether anything actually changed, so a caller polling on a timer can + /// log real edits instead of one line per poll. + pub fn is_empty(&self) -> bool { + self.added.is_empty() && self.updated.is_empty() && self.removed.is_empty() + } +} + +/// `Tree::PluginMeta` key holding the hosts whose Drive mapping was installed +/// by [`Db::sync_drive_mappings`]. +/// +/// Removal is scoped to this set rather than to everything in +/// `Tree::DriveMapping`, because that table also holds bindings made by hand +/// through `/bind-drive` plus the `localhost` / `127.0.0.1` entries every store +/// gets at setup. A reconcile that treated its input as authoritative for the +/// whole table would delete those the first time it ran. +const MANAGED_ALIAS_HOSTS_KEY: &[u8] = b"managed_drive_alias_hosts"; + +/// Normalizes a host into a `Tree::DriveMapping` key. +/// +/// Hostnames are case-insensitive (RFC 4343), but the `Host` header comes back +/// in whatever case the client sent, and `/bind-drive` derives its key from a +/// parsed URL (which the `url` crate has already lowercased). Storing and +/// looking up lowercased is what makes `ACME.example.com` resolve the Drive +/// that `acme.example.com` was bound to, instead of silently missing the +/// mapping and falling through to the store root. +fn drive_mapping_key(host: &str) -> String { + host.trim().trim_end_matches('.').to_ascii_lowercase() +} + const REPLICATION_PREFIX: &str = "replication:"; /// `Tree::PluginMeta` key holding the fingerprint of the built-in defaults @@ -1593,8 +1635,16 @@ impl Db { return Ok(subject.clone()); }; + // A host bound to a Drive serves that Drive or nothing. Falling through + // to the store root here is what let a vanity host whose Drive has not + // synced yet β€” or was migrated to another node β€” quietly serve the + // node's *own* root Drive under a customer's hostname. On a + // multi-tenant node that is one tenant's domain answering with another + // namespace's content, so it is a 404 instead. if self.get_resource(&drive_did).await.is_err() { - return Ok(subject.clone()); + return Err(AtomicError::not_found(format!( + "Host '{host}' is bound to drive {drive_did}, which this server does not have." + ))); } if subject_string == "/" { @@ -1763,19 +1813,131 @@ impl Db { _ => drive_did.to_string(), }; + let key = drive_mapping_key(host); self.kv - .insert(Tree::DriveMapping, host.as_bytes(), did_str.as_bytes())?; - tracing::info!("Added drive mapping: {} -> {}", host, did_str); + .insert(Tree::DriveMapping, key.as_bytes(), did_str.as_bytes())?; + tracing::info!("Added drive mapping: {} -> {}", key, did_str); Ok(()) } /// Removes the drive mapping for a given host. pub fn remove_drive_mapping(&self, host: &str) -> AtomicResult<()> { - self.kv.remove(Tree::DriveMapping, host.as_bytes())?; - tracing::info!("Removed drive mapping for host: {}", host); + let key = drive_mapping_key(host); + self.kv.remove(Tree::DriveMapping, key.as_bytes())?; + tracing::info!("Removed drive mapping for host: {}", key); Ok(()) } + /// Every host β†’ Drive mapping in this store, sorted by host. + pub fn list_drive_mappings(&self) -> AtomicResult> { + let mut out = Vec::new(); + + for (key, value) in self.kv.scan_prefix(Tree::DriveMapping, b"").flatten() { + let (Ok(host), Ok(did)) = (std::str::from_utf8(&key), std::str::from_utf8(&value)) + else { + continue; + }; + out.push((host.to_string(), did.to_string())); + } + + out.sort(); + Ok(out) + } + + /// The hosts [`Self::sync_drive_mappings`] installed on its last run. + pub fn managed_alias_hosts(&self) -> AtomicResult> { + let Some(bytes) = self.kv.get(Tree::PluginMeta, MANAGED_ALIAS_HOSTS_KEY)? else { + return Ok(Vec::new()); + }; + + serde_json::from_slice(&bytes) + .map_err(|e| format!("Corrupt managed alias host list: {e}").into()) + } + + /// Reconciles the host β†’ Drive mappings this server installs on behalf of a + /// control plane against `desired`, reporting what changed. + /// + /// Idempotent, so a managed node can call it on every policy poll: a host + /// already pointing at the right Drive is left untouched. Hosts a previous + /// run installed that `desired` no longer names are removed, which is what + /// makes a released vanity name stop resolving without anyone touching the + /// box. Ownership is tracked in [`MANAGED_ALIAS_HOSTS_KEY`], so a mapping + /// made by hand through `/bind-drive` is never removed by a reconcile that + /// does not name it. + /// + /// A host the control plane *does* name is taken over even if something + /// else bound it first: the control plane owns the namespace it hands out, + /// and a stale local binding shadowing an issued name is the failure this + /// is meant to correct. + pub fn sync_drive_mappings( + &self, + desired: &[(String, String)], + ) -> AtomicResult { + let mut report = DriveMappingSync::default(); + let mut managed: Vec = Vec::with_capacity(desired.len()); + + for (host, drive_did) in desired { + let key = drive_mapping_key(host); + // An empty host would claim the zero-length key and match no + // request; an empty DID would point a live hostname at nothing. + // Either is a bug upstream, and neither is worth writing. + if key.is_empty() || drive_did.trim().is_empty() { + tracing::warn!("Skipping drive mapping with an empty host or drive: {key:?}"); + continue; + } + + let current = self + .kv + .get(Tree::DriveMapping, key.as_bytes())? + .and_then(|bytes| String::from_utf8(bytes).ok()); + + match current { + Some(current) if current == *drive_did => {} + Some(_) => { + self.kv + .insert(Tree::DriveMapping, key.as_bytes(), drive_did.as_bytes())?; + report.updated.push(key.clone()); + } + None => { + self.kv + .insert(Tree::DriveMapping, key.as_bytes(), drive_did.as_bytes())?; + report.added.push(key.clone()); + } + } + + if !managed.contains(&key) { + managed.push(key); + } + } + + for host in self.managed_alias_hosts()? { + if managed.contains(&host) { + continue; + } + if self.kv.get(Tree::DriveMapping, host.as_bytes())?.is_some() { + self.kv.remove(Tree::DriveMapping, host.as_bytes())?; + report.removed.push(host); + } + } + + managed.sort(); + let bytes = serde_json::to_vec(&managed) + .map_err(|e| format!("Could not encode managed alias hosts: {e}"))?; + self.kv + .insert(Tree::PluginMeta, MANAGED_ALIAS_HOSTS_KEY, &bytes)?; + + if !report.is_empty() { + tracing::info!( + "Drive mappings reconciled: {} added, {} updated, {} removed", + report.added.len(), + report.updated.len(), + report.removed.len() + ); + } + + Ok(report) + } + /// Where a drive is replicated to, and who authorized it. /// /// This is **server-local config, and must stay out of the drive**. A @@ -1842,7 +2004,8 @@ impl Db { /// Returns the full Drive DID for a given host (domain/subdomain). pub async fn get_drive_did(&self, host: &str) -> AtomicResult> { - if let Some(did_bin) = self.kv.get(Tree::DriveMapping, host.as_bytes())? { + let key = drive_mapping_key(host); + if let Some(did_bin) = self.kv.get(Tree::DriveMapping, key.as_bytes())? { let did_str = std::str::from_utf8(&did_bin) .map_err(|e| format!("Failed to parse DID from database: {}", e))?; return Ok(Some(Subject::from_raw(did_str, None))); @@ -4562,6 +4725,216 @@ mod resolver_tests { "did:ad:test-child?drive=".to_string() + drive_did.as_str() ); } + + /// The vanity-subdomain hazard: a host is bound to a Drive this node does + /// not hold (not synced yet, or migrated away). Serving the store root + /// instead would answer a customer's hostname with another namespace's + /// content, so the request must fail. + #[tokio::test] + async fn a_bound_host_whose_drive_is_missing_does_not_serve_the_store_root() { + let store = Db::init_temp("resolver_missing_drive").await.unwrap(); + setup_test_env(&store).await.unwrap(); + let root_drive = store.get_drive_did("localhost").await.unwrap().unwrap(); + + store + .add_drive_mapping( + "acme.example.com", + &Value::AtomicUrl("did:ad:not-on-this-node".into()), + ) + .unwrap(); + + for path in ["/", "/about"] { + let resolved = store + .resolve_request_target( + &Subject::from_raw(path, None), + "acme.example.com", + path, + "https://acme.example.com", + ) + .await; + + let Err(err) = resolved else { + panic!("{path} on a bound host with a missing drive must not resolve"); + }; + assert!( + !format!("{err}").contains(root_drive.as_str()), + "must not leak the root drive: {err}" + ); + } + + // The node's own hostname is unaffected: it has no mapping, so it + // keeps serving its root as before. + assert!(store + .resolve_request_target( + &Subject::from_raw("/", None), + "node1.example.com", + "/", + "https://node1.example.com", + ) + .await + .is_ok()); + } +} + +#[cfg(test)] +mod drive_mapping_tests { + use super::*; + use crate::{test_utils::setup_test_env, Value}; + + fn desired(pairs: &[(&str, &str)]) -> Vec<(String, String)> { + pairs + .iter() + .map(|(h, d)| (h.to_string(), d.to_string())) + .collect() + } + + /// The reconcile a managed node runs on every policy poll: new names are + /// installed, a repointed name follows its Drive, and a released name + /// stops resolving. All three without an operator touching the box. + #[tokio::test] + async fn reconcile_adds_updates_and_removes() { + let store = Db::init_temp("drive_mapping_reconcile").await.unwrap(); + + let report = store + .sync_drive_mappings(&desired(&[ + ("acme.example.com", "did:ad:acme"), + ("blog.example.com", "did:ad:blog"), + ])) + .unwrap(); + assert_eq!(report.added, vec!["acme.example.com", "blog.example.com"]); + assert!(report.updated.is_empty()); + assert!(report.removed.is_empty()); + + // `acme` is repointed at another Drive (a migration), `blog` is + // released, and `docs` is new. + let report = store + .sync_drive_mappings(&desired(&[ + ("acme.example.com", "did:ad:acme-moved"), + ("docs.example.com", "did:ad:docs"), + ])) + .unwrap(); + assert_eq!(report.added, vec!["docs.example.com"]); + assert_eq!(report.updated, vec!["acme.example.com"]); + assert_eq!(report.removed, vec!["blog.example.com"]); + + assert_eq!( + store + .get_drive_did("acme.example.com") + .await + .unwrap() + .unwrap() + .as_str(), + "did:ad:acme-moved" + ); + assert!(store + .get_drive_did("blog.example.com") + .await + .unwrap() + .is_none()); + } + + /// Polled on a timer, so an unchanged policy must be a no-op rather than a + /// write per tick. + #[tokio::test] + async fn reconcile_is_idempotent() { + let store = Db::init_temp("drive_mapping_idempotent").await.unwrap(); + let want = desired(&[("acme.example.com", "did:ad:acme")]); + + assert_eq!( + store.sync_drive_mappings(&want).unwrap().added, + vec!["acme.example.com"] + ); + for _ in 0..3 { + assert!(store.sync_drive_mappings(&want).unwrap().is_empty()); + } + } + + /// The reconcile owns only what it installed. `setup_test_env` binds + /// `localhost` and `127.0.0.1`, and an operator may have bound a host by + /// hand through `/bind-drive`; treating the control plane's list as + /// authoritative for the whole table would delete all of those on the + /// first poll. + #[tokio::test] + async fn reconcile_leaves_bindings_it_did_not_install_alone() { + let store = Db::init_temp("drive_mapping_scope").await.unwrap(); + setup_test_env(&store).await.unwrap(); + let local_drive = store.get_drive_did("localhost").await.unwrap().unwrap(); + + store + .add_drive_mapping( + "hand-bound.example.com", + &Value::AtomicUrl("did:ad:hand-bound".into()), + ) + .unwrap(); + + let report = store + .sync_drive_mappings(&desired(&[("acme.example.com", "did:ad:acme")])) + .unwrap(); + assert!(report.removed.is_empty()); + + // Both survive a reconcile that names neither. + assert_eq!( + store.get_drive_did("localhost").await.unwrap().unwrap(), + local_drive + ); + assert!(store + .get_drive_did("hand-bound.example.com") + .await + .unwrap() + .is_some()); + } + + /// Hostnames are case-insensitive, but the `Host` header is echoed in + /// whatever case the client sent. Without normalization a request for + /// `ACME.example.com` misses the mapping and falls through to the store + /// root. + #[tokio::test] + async fn mapping_keys_are_case_insensitive() { + let store = Db::init_temp("drive_mapping_case").await.unwrap(); + + store + .sync_drive_mappings(&desired(&[("ACME.Example.COM", "did:ad:acme")])) + .unwrap(); + + for spelling in ["acme.example.com", "ACME.EXAMPLE.COM", "Acme.Example.Com"] { + assert_eq!( + store + .get_drive_did(spelling) + .await + .unwrap() + .unwrap_or_else(|| panic!("{spelling} should resolve")) + .as_str(), + "did:ad:acme", + ); + } + + assert_eq!( + store.list_drive_mappings().unwrap(), + vec![("acme.example.com".to_string(), "did:ad:acme".to_string())] + ); + } + + /// A host with no Drive, or a Drive with no host, is a bug upstream. Both + /// are skipped rather than written: the empty key would match no request, + /// and an empty Drive would point a live hostname at nothing. + #[tokio::test] + async fn reconcile_skips_empty_hosts_and_drives() { + let store = Db::init_temp("drive_mapping_empty").await.unwrap(); + + let report = store + .sync_drive_mappings(&desired(&[ + ("", "did:ad:nohost"), + ("nodrive.example.com", " "), + ("good.example.com", "did:ad:good"), + ])) + .unwrap(); + + assert_eq!(report.added, vec!["good.example.com"]); + assert_eq!( + store.list_drive_mappings().unwrap(), + vec![("good.example.com".to_string(), "did:ad:good".to_string())] + ); + } } #[cfg(test)] diff --git a/server/src/config.rs b/server/src/config.rs index 57bb555ec..fb89aec65 100644 --- a/server/src/config.rs +++ b/server/src/config.rs @@ -143,9 +143,35 @@ pub struct Opts { /// The base domain for multi-tenant hosting. /// If set, the server will allow serving subdomains of this domain (e.g. *.atomicserver.eu). + /// + /// Note that this does two things at once: it accepts subdomain `Host` + /// headers *and* it becomes the store's base domain, which is what + /// `Subject::from_raw` normalizes absolute URLs against. If you only need + /// the former β€” answering for names a control plane routes to you, without + /// changing how subjects are stored β€” use `--served-domain-suffix`. #[clap(long, env = "ATOMIC_BASE_DOMAIN")] pub base_domain: Option, + /// An extra domain whose subdomains this server answers for, e.g. + /// `atomicserver.eu` so that `acme.atomicserver.eu` is treated as one of + /// this server's own names. + /// + /// Deliberately separate from `--base-domain`. Only the set of `Host` + /// headers the server trusts is affected: the request origin (what signed + /// auth proofs bind to, and what served subjects are localized to) follows + /// the hostname the request actually arrived on, instead of falling back to + /// the configured origin and handing a visitor links to a different + /// hostname than the one they typed. The store's own base domain is + /// untouched, so nothing about how subjects are normalized or migrated + /// changes β€” which is what makes this safe to switch on for a node that + /// already holds data. + /// + /// This is what a managed node is given for hosted vanity subdomains; the + /// mapping from such a hostname to a Drive is separate, and lives in + /// `Tree::DriveMapping`. + #[clap(long, env = "ATOMIC_SERVED_DOMAIN_SUFFIX")] + pub served_domain_suffix: Option, + /// Serve one Drive as this server's front page. /// /// Set this to a Drive's subject (e.g. `internal:/` for the Drive at the diff --git a/server/src/context.rs b/server/src/context.rs index 57baf82da..084a1d09a 100644 --- a/server/src/context.rs +++ b/server/src/context.rs @@ -72,9 +72,15 @@ pub(crate) fn host_is_served_here(host: &str, opts: &crate::config::Opts) -> boo { return true; } - if let Some(base) = opts.base_domain.as_deref() { - let base = base.trim().trim_start_matches('.').to_ascii_lowercase(); - if !base.is_empty() && (hostname == base || hostname.ends_with(&format!(".{base}"))) { + for suffix in [ + opts.base_domain.as_deref(), + opts.served_domain_suffix.as_deref(), + ] + .into_iter() + .flatten() + { + let suffix = suffix.trim().trim_start_matches('.').to_ascii_lowercase(); + if !suffix.is_empty() && (hostname == suffix || hostname.ends_with(&format!(".{suffix}"))) { return true; } } @@ -121,6 +127,36 @@ mod tests { crate::config::Opts::parse_from(args) } + fn opts_with_served_suffix(domain: &str, suffix: &str) -> crate::config::Opts { + crate::config::Opts::parse_from([ + "atomic-server", + "--domain", + domain, + "--served-domain-suffix", + suffix, + ]) + } + + /// A managed node serving hosted vanity subdomains gets the suffix without + /// `--base-domain`, so the request origin follows the hostname the visitor + /// used while the store's subject normalization stays as it was. + #[test] + fn a_served_domain_suffix_accepts_tenants_without_a_base_domain() { + let o = opts_with_served_suffix("node1.atomicserver.eu", "atomicserver.eu"); + assert!(o.base_domain.is_none()); + + assert!(host_is_served_here("acme.atomicserver.eu", &o)); + assert!(host_is_served_here("ACME.AtomicServer.eu:443", &o)); + // The node's own name and the apex are both names it answers for. + assert!(host_is_served_here("node1.atomicserver.eu", &o)); + assert!(host_is_served_here("atomicserver.eu", &o)); + + // The same near-misses the base-domain path rejects. + assert!(!host_is_served_here("notatomicserver.eu", &o)); + assert!(!host_is_served_here("atomicserver.eu.evil.example", &o)); + assert!(!host_is_served_here("evil.example", &o)); + } + #[test] fn configured_domain_and_tenants_are_served_here() { let o = opts("atomicdata.dev", Some("atomicserver.eu")); From b79754ae3d5a799f796d071ab1680c90e7ca25de Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 05:43:53 +0200 Subject: [PATCH 007/110] fix(lib): write datatype tags in the shared export path, after sealing the edit (#1553) Datatype tags are now written inside exportLoroDeltaInternal, after the user's ops are sealed, so a drained commit carries them and undo no longer steps back into the tag write. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KxExJcGw13DnHknGTMAhXY --- TESTING_COVERAGE.md | 7 +- browser/CHANGELOG.md | 1 + browser/lib/src/drain-datatype-tags.test.ts | 88 ++++++++++++++++++++- browser/lib/src/resource.ts | 66 ++++++++++------ planning/completed/sign-at-drain.md | 16 +++- 5 files changed, 150 insertions(+), 28 deletions(-) diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index dfa6473f6..0e6eeaf1f 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -1382,7 +1382,12 @@ network transport, OS-process isolation or reviewed alias/reference repair. - Connection-state test verifies alias provenance, preserved baseline, incremented revision, idempotent reads and rejection of the earlier checkpoint revision. - `drain-datatype-tags.test.ts` reproduces and fixes newly added JSON values becoming - strings on incremental saves. The full client suite has 564 passing tests. + strings on incremental saves. It also drives the public `newResource β†’ set β†’ save` + flow through the real outbox drain and replays the posted `loroUpdate`s: the signed + incremental commit carries `json`/`resourceArray` tags for properties first set + after genesis, and the tag write runs after the user's ops are sealed, so the edit + keeps its own commit origin and stays on the undo stack. The full client suite has + 564 passing tests. - Five Chromium flows pass against rebuilt native/WASM code. Duplicate review uses real authenticated SYNC_PUSH plus a signed primary decision and fresh lookup; the other flows cover setup recovery, MT940 and Clockify. It does not yet test diff --git a/browser/CHANGELOG.md b/browser/CHANGELOG.md index 5e2c3de2f..d16eb0b48 100644 --- a/browser/CHANGELOG.md +++ b/browser/CHANGELOG.md @@ -4,6 +4,7 @@ This changelog covers all five packages, as they are (for now) updated as a whol ## UNRELEASED +- Fix: `@tomic/lib` writes the Loro `datatypes` tags in the export path shared by genesis and drain-time signing, after the user's pending ops are sealed into their own commit. The drain used to tag first, which folded a `set()` edit into the tag write's `atomic:system` commit β€” the canvas undo stack skips that origin, so the first edit of a `json`/reference property after a save could not be undone. - Local Thought connections install a folder after one access check and import in the browser without a preview dialog. Opening the folder or a table refreshes automatically, with five-minute refreshes while open, visible sync status, and local-edit preservation. - Fix: form inputs validate before writing. `InputString` and `InputURI` no longer put an invalid value into the resource while showing an error; all inputs share one validate-then-set hook (`useValidatedInput`) and surface the datatype's actual error message (e.g. "Not an integer") instead of a generic "Invalid value". `ResourceField` now accepts and forwards `commit` / `commitDebounceInterval`, and every input honours `commitDebounceInterval` (Markdown, Number, Boolean, Date, Timestamp, Resource, ResourceArray and FilePicker previously fell back to the 100 ms default). A required field whose value loads after the first render is now flagged correctly. - Fork bar: "Review changes" opens the per-property diff (the original's current value against the fork's) and names the properties the original also changed since the fork, so a reviewer sees what a merge writes over instead of a count. diff --git a/browser/lib/src/drain-datatype-tags.test.ts b/browser/lib/src/drain-datatype-tags.test.ts index 76011e4d2..92a6a76e7 100644 --- a/browser/lib/src/drain-datatype-tags.test.ts +++ b/browser/lib/src/drain-datatype-tags.test.ts @@ -1,10 +1,13 @@ import { it, expect } from 'vitest'; import { Store } from './store.js'; -import { Resource } from './resource.js'; +import { Resource, SYSTEM_COMMIT_ORIGIN } from './resource.js'; import { enableLoro } from './loro-loader.js'; import { core } from './ontologies/core.js'; import { Datatype } from './datatypes.js'; import { IMPORT_RESOLUTION } from './import-resolution.js'; +import { testStore } from './test-store.js'; +import { LoroLoader } from './loro-loader.js'; +import type { Commit } from './commit.js'; it('incremental drain tags a newly added JSON field so it survives materialization', async () => { await enableLoro(); const store = new Store(); @@ -24,3 +27,86 @@ it('incremental drain tags a newly added JSON field so it survives materializati received.importLoroUpdate(update.bytes); expect(received.get(IMPORT_RESOLUTION)).toEqual(value); }); + +it('the drained incremental commit tags properties first set after genesis', async () => { + const { store, postCommitSpy } = await testStore(); + const JSON_PROP = 'https://example.com/properties/settings'; + const ARRAY_PROP = 'https://example.com/properties/members'; + + for (const [subject, datatype] of [ + [JSON_PROP, Datatype.JSON], + [ARRAY_PROP, Datatype.RESOURCEARRAY], + ] as const) { + const property = new Resource(subject); + store.addResource(property); + await property.set(core.properties.datatype, datatype, false); + } + + const doc = await store.newResource({ + isA: 'https://atomicdata.dev/classes/Drive', + propVals: { [core.properties.name]: 'Tagged later' }, + noParent: true, + }); + expect(await doc.save()).toBe('persisted'); + // Neither property existed at genesis: the incremental commit the drain + // signs is the only place their tags can travel. + await doc.set(JSON_PROP, { theme: 'dark' }, false); + await doc.set(ARRAY_PROP, ['did:ad:member'], false); + expect(await doc.save()).toBe('persisted'); + expect(postCommitSpy.mock.calls.length).toBe(2); + const [genesis, update] = postCommitSpy.mock.calls.map( + ([commit]) => (commit as Commit).loroUpdate!, + ); + const { LoroDoc } = LoroLoader.Loro; + const afterGenesis = new LoroDoc(); + afterGenesis.import(genesis); + const genesisTags = afterGenesis.getMap('datatypes').toJSON(); + expect(genesisTags[JSON_PROP]).toBeUndefined(); + expect(genesisTags[ARRAY_PROP]).toBeUndefined(); + // What the server sees: the delta applied on top of the genesis snapshot. + const server = new LoroDoc(); + server.import(genesis); + server.import(update); + const tags = server.getMap('datatypes').toJSON(); + expect(tags[JSON_PROP]).toBe('json'); + expect(tags[ARRAY_PROP]).toBe('resourceArray'); + expect(server.getMap('properties').get(JSON_PROP)).toBe( + JSON.stringify({ theme: 'dark' }), + ); +}); + +it('the drain seals the edit before tagging, so the edit stays undoable', async () => { + const { store, postCommitSpy } = await testStore(); + const JSON_PROP = 'https://example.com/properties/settings'; + const property = new Resource(JSON_PROP); + store.addResource(property); + await property.set(core.properties.datatype, Datatype.JSON, false); + const doc = await store.newResource({ + isA: 'https://atomicdata.dev/classes/Drive', + propVals: { [core.properties.name]: 'Undo me' }, + noParent: true, + }); + expect(await doc.save()).toBe('persisted'); + // The canvas page's undo stack: it skips `atomic:system` commits. + doc.ensureUndoManager(); + const origins: string[] = []; + doc.getLoroDoc()!.subscribe(batch => { + if (batch.by === 'local') origins.push(batch.origin ?? ''); + }); + // `set()` leaves the op in an open transaction. The drain's first + // `commit()` seals it β€” that must be the edit's own commit, not the + // datatype-tag housekeeping write, or the user's undo skips the edit. + await doc.set(JSON_PROP, { theme: 'dark' }, false); + expect(await doc.save()).toBe('persisted'); + expect(origins[0]).not.toBe(SYSTEM_COMMIT_ORIGIN); + expect(doc.canUndo()).toBe(true); + // The tags still travel in the same signed commit. + const { LoroDoc } = LoroLoader.Loro; + const server = new LoroDoc(); + + for (const [commit] of postCommitSpy.mock.calls) { + server.import((commit as Commit).loroUpdate!); + } + + expect(server.getMap('datatypes').get(JSON_PROP)).toBe('json'); +}); diff --git a/browser/lib/src/resource.ts b/browser/lib/src/resource.ts index 15573fece..001040875 100644 --- a/browser/lib/src/resource.ts +++ b/browser/lib/src/resource.ts @@ -828,6 +828,10 @@ export class Resource { * of guessing. The map is sparse β€” only load-bearing datatypes get a tag; * see {@link datatypeTag}. Idempotent: re-signing rewrites nothing. * + * Called from `exportLoroDeltaInternal` after the user's ops are sealed β€” + * the shared path of genesis and drain-time signing β€” and, for a genesis + * sign, once more up front by `signChanges` (see the commit note below). + * * Cache-only β€” never triggers a fetch. A property whose definition is not * already cached is left untagged; the server then falls back to its * materialization heuristic, exactly as before this map existed. Properties @@ -1137,9 +1141,9 @@ export class Resource { isFirstCommit: boolean, commitMessage?: string, ): { bytes: Uint8Array; versionAfterExport: VersionVector } | undefined { - // Incremental saves bypass signChanges; they still need datatype tags for - // newly added JSON/reference fields before capturing the signed delta. - this.writeDatatypeTags(); + // Incremental saves bypass `signChanges`; the datatype tags for newly + // added JSON/reference fields are written by `exportLoroDeltaInternal`, + // the path both signers share. const bytes = this.exportLoroDeltaInternal(isFirstCommit, commitMessage); if (!bytes) return undefined; if (!this._loroDoc) return undefined; @@ -1379,6 +1383,26 @@ export class Resource { }); this._stagedCommitToken = undefined; + // Stamp the sibling `datatypes` map for whatever the doc now holds, so + // the server materializes references/arrays/JSON exactly (see + // `writeDatatypeTags`). This is the shared path for BOTH signers: + // `signChanges` (genesis, local-only drives) and the store-level drain. + // It runs AFTER the user's ops were sealed above, on purpose. `set()` + // leaves its ops in an open transaction, and the first `commit()` seals + // everything pending β€” so tagging first would fold the user's edit into + // the tag write's `atomic:system` commit, where the UndoManager + // (`excludeOriginPrefixes`) never records it and the user's undo skips + // straight past their own change. It reuses this commit's message so + // history buckets the tags with the edit they belong to, and it is a + // no-op once every tag is present, so a clean resource stays clean. + // Genesis signs already tagged before calling in (the agent message has + // to ride on the doc's first change), and land here as that no-op. + this.writeDatatypeTags({ + origin: SYSTEM_COMMIT_ORIGIN, + timestamp: Date.now(), + ...(message ? { message } : {}), + }); + // If it's the first commit, we must export a full snapshot. if (isFirstCommit || !this._loroVersionAtLastSave) { const snapshot = this._loroDoc.export({ mode: 'snapshot' }); @@ -3003,25 +3027,23 @@ export class Resource { } // Stamp the sibling `datatypes` map so the server materializes - // references/arrays exactly. Runs here β€” after - // every property is in the doc, before the snapshot export below β€” so it - // covers props set via `set()` and via cache hydration alike. - // - // On a genesis sign this is the FIRST commit on the doc, so it creates the - // genesis change β€” tag it with the signing agent's subject (β†’ `createdBy`) - // and a millisecond timestamp (β†’ `createdAt`). The oplog records only a - // random peer id, never the agent, so this commit message is what carries - // authorship inside the doc, readable without fetching the commit (which is - // no longer refetchable under sign-at-drain). - this.writeDatatypeTags( - isFirstCommit - ? { - origin: SYSTEM_COMMIT_ORIGIN, - timestamp: Date.now(), - message: agent.subject, - } - : undefined, - ); + // references/arrays exactly. The export below does this for every sign + // (see `exportLoroDeltaInternal`), after the user's ops are sealed. A + // genesis sign tags up front as well: this is the FIRST commit on the + // doc, so it creates the genesis change β€” tag it with the signing agent's + // subject (β†’ `createdBy`) and a millisecond timestamp (β†’ `createdAt`). The + // oplog records only a random peer id, never the agent, so this commit + // message is what carries authorship inside the doc, readable without + // fetching the commit (which is no longer refetchable under + // sign-at-drain). Runs after every property is in the doc, so it covers + // props set via `set()` and via cache hydration alike. + if (isFirstCommit) { + this.writeDatatypeTags({ + origin: SYSTEM_COMMIT_ORIGIN, + timestamp: Date.now(), + message: agent.subject, + }); + } // Export Loro delta β€” the sole carrier of property changes. Pass the agent // again as a fallback: if `writeDatatypeTags` had nothing to commit, this diff --git a/planning/completed/sign-at-drain.md b/planning/completed/sign-at-drain.md index c807ec55c..e7e98625f 100644 --- a/planning/completed/sign-at-drain.md +++ b/planning/completed/sign-at-drain.md @@ -761,10 +761,18 @@ The Commit wire format does not change between profiles. Only Questions 4 and 5 in the original draft (`writeDatatypeTags` placement under drain-time signing; `logPendingCommit` having no pending state under drain-time signing) were artefacts of the option 1 (sign-at-drain) -direction. Under option 2 (eager sign, async POST), both are non-issues: -`writeDatatypeTags` still runs inside the eager `signChanges`, and -`logPendingCommit` still fires per commit with "pending β†’ sent β†’ failed" -transitions intact. +direction, and question 4 did become real once the store-level drain +signed incremental commits without going through `signChanges`. The +resolved state: `writeDatatypeTags` runs inside +`Resource.exportLoroDeltaInternal`, the export path both signers share +(`signChanges` for genesis and local-only drives, `drainOutboxSubject` via +`exportLoroDeltaForDrain`), right after the user's pending ops are sealed +into their own commit and before the bytes are exported. Ordering matters: +tagging first would seal the user's `set()` ops inside the tag write's +`atomic:system` commit, which the UndoManager excludes. Genesis signs still +tag up front so the agent message rides on the doc's first change; the +in-export call is then a no-op. `logPendingCommit` still fires per commit +with "pending β†’ sent β†’ failed" transitions intact. ## Related plans From 6b9f431a494f5cdf690de4ad7129e8f2f7ce0cc3 Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 05:46:16 +0200 Subject: [PATCH 008/110] perf(lib): coalesce cold-load OPFS reads into one worker round trip per tick (#1554) Store batches the render-phase local-db cache misses of a tick into a single getResourcesWithSnapshots call, chunked at 200 subjects, and shares an in-flight read between duplicate requests for one subject. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KxExJcGw13DnHknGTMAhXY --- TESTING_COVERAGE.md | 1 + browser/CHANGELOG.md | 1 + browser/lib/src/store.read-policy.test.ts | 149 +++++++++++++++++++++- browser/lib/src/store.ts | 136 ++++++++++++++++++-- 4 files changed, 272 insertions(+), 15 deletions(-) diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index 0e6eeaf1f..1f1275607 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -552,6 +552,7 @@ Not covered: table `contains`; Playwright search overlay on the KV path and asse | Offline edits persist and sync on reconnect | `sync.spec.ts` | | Second device cold-loads a drive from the server | `second-device-load.spec.ts` | | Property reads stay pending through loading-placeholder notifications until hydration completes | `browser/lib/src/store.test.ts` | +| Cold-load local hydration: all `useResource` misses of one tick share one worker round trip, a duplicate subject is asked once, a miss during a flush lands in the next batch, a failed bulk read is a per-subject miss, batches chunk at 200 | `browser/lib/src/store.read-policy.test.ts` | --- diff --git a/browser/CHANGELOG.md b/browser/CHANGELOG.md index d16eb0b48..12c00c630 100644 --- a/browser/CHANGELOG.md +++ b/browser/CHANGELOG.md @@ -5,6 +5,7 @@ This changelog covers all five packages, as they are (for now) updated as a whol ## UNRELEASED - Fix: `@tomic/lib` writes the Loro `datatypes` tags in the export path shared by genesis and drain-time signing, after the user's pending ops are sealed into their own commit. The drain used to tag first, which folded a `set()` edit into the tag write's `atomic:system` commit β€” the canvas undo stack skips that origin, so the first edit of a `json`/reference property after a save could not be undone. +- Perf: a cold page load asks the local database (OPFS worker) for all the resources a render pass mounts in one round trip instead of one per `useResource`. `Store` coalesces the render-phase cache misses of a tick into a single `getResourcesWithSnapshots` call (chunked at 200 subjects), and a subject asked for twice while a read is pending or in flight shares that read. 50 mounted resources went from 50 worker messages to 1; the tri-state answer (`true` / `false` / no database to ask) and the "ask again once the database attaches" behaviour are unchanged. - Local Thought connections install a folder after one access check and import in the browser without a preview dialog. Opening the folder or a table refreshes automatically, with five-minute refreshes while open, visible sync status, and local-edit preservation. - Fix: form inputs validate before writing. `InputString` and `InputURI` no longer put an invalid value into the resource while showing an error; all inputs share one validate-then-set hook (`useValidatedInput`) and surface the datatype's actual error message (e.g. "Not an integer") instead of a generic "Invalid value". `ResourceField` now accepts and forwards `commit` / `commitDebounceInterval`, and every input honours `commitDebounceInterval` (Markdown, Number, Boolean, Date, Timestamp, Resource, ResourceArray and FilePicker previously fell back to the 100 ms default). A required field whose value loads after the first render is now flagged correctly. - Fork bar: "Review changes" opens the per-property diff (the original's current value against the fork's) and names the properties the original also changed since the fork, so a reviewer sees what a merge writes over instead of a count. diff --git a/browser/lib/src/store.read-policy.test.ts b/browser/lib/src/store.read-policy.test.ts index f70297c87..f1b75e3c7 100644 --- a/browser/lib/src/store.read-policy.test.ts +++ b/browser/lib/src/store.read-policy.test.ts @@ -26,8 +26,15 @@ function row(subject: string, name: string) { }; } -/** A database that answers from `rows` and counts its round trips. */ -function fakeDb(rows: Record>) { +/** + * A database that answers from `rows` and counts its round trips. `onRead` + * runs before a bulk read answers (to hold the worker busy); `fail` makes + * every bulk read reject with it. + */ +function fakeDb( + rows: Record>, + opts: { onRead?: (subjects: string[]) => Promise; fail?: Error } = {}, +) { const calls: string[][] = []; const db = { isReady: true, @@ -41,6 +48,8 @@ function fakeDb(rows: Record>) { }, getResourcesWithSnapshots: async (subjects: string[]) => { calls.push(subjects); + await opts.onRead?.(subjects); + if (opts.fail) throw opts.fail; return subjects.map(s => rows[s] ?? { jsonAd: null, snapshot: null }); }, @@ -168,3 +177,139 @@ describe('store read policy', () => { expect(socket.fetchMany).toHaveBeenCalledWith(['did:ad:c']); }); }); + +/** + * A cold render pass issues one `getResourceLoading` miss per mounted + * `useResource`, all in the same tick. They must share one worker round trip + * instead of each asking the database on their own. + */ +describe('local hydration batching', () => { + afterEach(() => vi.restoreAllMocks()); + + const subjects = (n: number) => + Array.from({ length: n }, (_, i) => `did:ad:s${i}`); + const rowsFor = (list: string[]) => + Object.fromEntries(list.map(s => [s, row(s, `Name ${s}`)])); + + it('reads every miss of one tick in a single round trip', async ({ + expect, + }) => { + await enableLoro(); + const { store } = await testStore(); + const wanted = subjects(30); + const { db, calls } = fakeDb(rowsFor(wanted)); + store.setClientDb(db); + const socket = fakeSocket(store); + + for (const subject of wanted) store.getResourceLoading(subject); + const loaded = await Promise.all(wanted.map(s => store.getResource(s))); + + expect(calls).toEqual([wanted]); + expect(loaded.map(r => r.get(NAME))).toEqual(wanted.map(s => `Name ${s}`)); + expect(socket.fetch).not.toHaveBeenCalled(); + expect(socket.fetchMany).not.toHaveBeenCalled(); + }); + + it('asks once for a subject two callers miss in the same tick', async ({ + expect, + }) => { + await enableLoro(); + const { store } = await testStore(); + const { db, calls } = fakeDb(rowsFor(['did:ad:twice'])); + store.setClientDb(db); + fakeSocket(store); + + // A render-phase miss and a direct local read of the same subject. + const fromRender = store.getResourceLoading('did:ad:twice'); + const fromLocal = store.getLocalResource('did:ad:twice'); + const [local, awaited] = await Promise.all([ + fromLocal, + store.getResource('did:ad:twice'), + ]); + + expect(calls).toEqual([['did:ad:twice']]); + expect(local).toBe(fromRender); + expect(awaited).toBe(fromRender); + expect(local.get(NAME)).toBe('Name did:ad:twice'); + }); + + it('puts a miss issued while a batch is at the worker into the next batch', async ({ + expect, + }) => { + await enableLoro(); + const { store } = await testStore(); + let release!: () => void; + let reached!: () => void; + const busy = new Promise(resolve => (release = resolve)); + const atWorker = new Promise(resolve => (reached = resolve)); + const { db, calls } = fakeDb(rowsFor(['did:ad:first', 'did:ad:second']), { + onRead: async () => { + reached(); + await busy; + }, + }); + store.setClientDb(db); + fakeSocket(store); + + store.getResourceLoading('did:ad:first'); + await atWorker; + // The first batch has been handed over; this one must not be lost. + store.getResourceLoading('did:ad:second'); + release(); + const [first, second] = await Promise.all([ + store.getResource('did:ad:first'), + store.getResource('did:ad:second'), + ]); + + expect(calls).toEqual([['did:ad:first'], ['did:ad:second']]); + expect(first.get(NAME)).toBe('Name did:ad:first'); + expect(second.get(NAME)).toBe('Name did:ad:second'); + }); + + it('treats a failed bulk read as a miss for every subject in it', async ({ + expect, + }) => { + await enableLoro(); + const { store } = await testStore(); + const { db, calls } = fakeDb(rowsFor(['did:ad:x', 'did:ad:y']), { + fail: new Error('worker gone'), + }); + store.setClientDb(db); + const socket = fakeSocket(store); + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined); + + store.getResourceLoading('did:ad:x'); + store.getResourceLoading('did:ad:y'); + await Promise.all([ + store.getResource('did:ad:x'), + store.getResource('did:ad:y'), + ]); + + // One round trip failed; each subject went on to the server as a miss. + expect(calls).toEqual([['did:ad:x', 'did:ad:y']]); + expect( + warn.mock.calls.filter(([message]) => + String(message).includes('OPFS lookup failed'), + ), + ).toHaveLength(1); + expect(socket.fetch).toHaveBeenCalledWith('did:ad:x'); + expect(socket.fetch).toHaveBeenCalledWith('did:ad:y'); + }); + + it('splits a very large batch into worker-sized chunks', async ({ + expect, + }) => { + await enableLoro(); + const { store } = await testStore(); + const wanted = subjects(250); + const { db, calls } = fakeDb(rowsFor(wanted)); + store.setClientDb(db); + fakeSocket(store); + + for (const subject of wanted) store.getResourceLoading(subject); + await Promise.all(wanted.map(s => store.getResource(s))); + + expect(calls.map(c => c.length)).toEqual([200, 50]); + expect(calls.flat()).toEqual(wanted); + }); +}); diff --git a/browser/lib/src/store.ts b/browser/lib/src/store.ts index 82fc41d0d..bb352a010 100644 --- a/browser/lib/src/store.ts +++ b/browser/lib/src/store.ts @@ -394,6 +394,17 @@ export interface IncomingChange { const supportsWebSockets = () => typeof WebSocket !== 'undefined'; /** Subjects per GET_MANY frame; keeps one answer well under a single large frame. */ const GET_MANY_CHUNK = 200; +/** Subjects per local-database read. The worker walks the list serially and a + * follower tab's read also crosses a BroadcastChannel hop with a fixed + * timeout, so one very large batch is split instead of sent whole. */ +const LOCAL_HYDRATION_CHUNK = GET_MANY_CHUNK; + +/** One caller's pending local-database read; see `Store.hydrateFromLocalDb`. */ +interface LocalHydrationRequest { + promise: Promise; + resolve: (hydrated: boolean | undefined) => void; + reject: (error: unknown) => void; +} /** * How long resource fallback and collection reads wait for the app's @@ -3031,15 +3042,6 @@ export class Store { }); } - /** - * Try the local WASM DB (OPFS) for a persisted copy of `subject` and hydrate - * the store from it. - * - * @returns `true` when a local copy hydrated into something renderable, - * `false` when the database was asked and does not have it, and - * `undefined` when there was no database to ask β€” callers must not read - * that silence as "not stored locally". - */ /** * True when the resource carries enough state to stand on its own: a * class, or any property beyond the server-managed skeleton @@ -3057,10 +3059,118 @@ export class Store { .some(([prop]) => !SERVER_MANAGED_SKELETON_PROPS.has(prop)); } - private async hydrateFromLocalDb( - subject: string, - ): Promise { - return (await this.hydrateManyFromLocalDb([subject]))?.[0]; + /** Local-database reads asked for since the last flush, one entry per + * subject. A subject asked for twice before the flush shares the entry. */ + private _pendingLocalHydration = new Map(); + + /** Subject β†’ promise of a flushed read that still awaits the worker. */ + private _inFlightLocalHydration = new Map< + string, + Promise + >(); + + /** Set from the first request of a batch until that batch is handed to the + * worker, so one flush is queued per batch and no more. */ + private _localHydrationFlushQueued = false; + + /** + * Try the local WASM DB (OPFS) for a persisted copy of `subject` and hydrate + * the store from it. + * + * Every `useResource` that misses the in-memory cache lands here, from + * inside React's render. One worker round trip per subject made a cold page + * with many resources slow: each `postMessage` costs milliseconds, queues + * behind sync traffic, and in a follower tab crosses a BroadcastChannel + * hop as well. So a call does not read at once. It joins the pending batch + * and the batch is read in one {@link hydrateManyFromLocalDb} call from a + * microtask β€” after the synchronous render pass that issued it. React + * renders a default-lane update in one task, so a whole cold mount is one + * round trip; a time-sliced (transition) render becomes one round trip per + * slice rather than per resource. A microtask beats `setTimeout(0)` here: + * that would put a timer clamp on every cold read, and `notify` already + * relies on a microtask being enough to leave the render phase. + * + * A subject that is pending or in flight reuses the existing read. + * + * @returns `true` when a local copy hydrated into something renderable, + * `false` when the database was asked and does not have it, and + * `undefined` when there was no database to ask β€” callers must not read + * that silence as "not stored locally". + */ + private hydrateFromLocalDb(subject: string): Promise { + const inFlight = this._inFlightLocalHydration.get(subject); + if (inFlight) return inFlight; + + const pending = this._pendingLocalHydration.get(subject); + if (pending) return pending.promise; + + let resolve!: LocalHydrationRequest['resolve']; + let reject!: LocalHydrationRequest['reject']; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + this._pendingLocalHydration.set(subject, { promise, resolve, reject }); + + if (!this._localHydrationFlushQueued) { + this._localHydrationFlushQueued = true; + queueMicrotask(() => void this.flushLocalHydration()); + } + + return promise; + } + + /** + * Hand the pending batch to the worker and settle each caller's promise + * with its own answer. A worker failure resolves every subject of the + * chunk to `false` inside `hydrateManyFromLocalDb`, exactly as the + * single-subject read did; only an unexpected throw rejects the callers. + */ + private async flushLocalHydration(): Promise { + // Hold the batch open while the database initializes. A reload mounts its + // resources before the worker is ready, and everything asked for in that + // window should ride the one round trip that follows. + if (this.clientDb) await this.clientDb.waitForInit(); + + const batch = this._pendingLocalHydration; + this._pendingLocalHydration = new Map(); + this._localHydrationFlushQueued = false; + + for (const [subject, request] of batch) { + this._inFlightLocalHydration.set(subject, request.promise); + } + + const subjects = [...batch.keys()]; + const chunks: string[][] = []; + + for (let i = 0; i < subjects.length; i += LOCAL_HYDRATION_CHUNK) { + chunks.push(subjects.slice(i, i + LOCAL_HYDRATION_CHUNK)); + } + + await Promise.all( + chunks.map(async chunk => { + try { + const hydrated = await this.hydrateManyFromLocalDb(chunk); + + chunk.forEach((subject, index) => + batch.get(subject)!.resolve(hydrated?.[index]), + ); + } catch (e) { + for (const subject of chunk) batch.get(subject)!.reject(e); + } finally { + // Before any caller resumes (their continuations are microtasks), + // so a re-ask after this answer is a fresh read, not this one. + for (const subject of chunk) { + if ( + this._inFlightLocalHydration.get(subject) === + batch.get(subject)!.promise + ) { + this._inFlightLocalHydration.delete(subject); + } + } + } + }), + ); } /** From fa1891a47565c33b2ab269ef5dac1c853fbd8ce3 Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 05:49:42 +0200 Subject: [PATCH 009/110] fix(lib,react): emit LocalChange from Resource.remove/push; route useArray.push through the save scheduler (#1541) Resource.push and Resource.remove now emit ResourceEvents.LocalChange like set does, so subscribed hooks re-render; useArray.push goes through the same save scheduler as the other writes; StoreContext no longer hands out a phantom Store when a provider is missing. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KxExJcGw13DnHknGTMAhXY --- browser/CHANGELOG.md | 1 + browser/lib/src/resource.test.ts | 57 +++++++++++++++++++++++++++++++ browser/lib/src/resource.ts | 4 ++- browser/react/src/hooks.ts | 58 ++++++++++++++++++++++---------- 4 files changed, 101 insertions(+), 19 deletions(-) diff --git a/browser/CHANGELOG.md b/browser/CHANGELOG.md index 12c00c630..85ba83ae6 100644 --- a/browser/CHANGELOG.md +++ b/browser/CHANGELOG.md @@ -8,6 +8,7 @@ This changelog covers all five packages, as they are (for now) updated as a whol - Perf: a cold page load asks the local database (OPFS worker) for all the resources a render pass mounts in one round trip instead of one per `useResource`. `Store` coalesces the render-phase cache misses of a tick into a single `getResourcesWithSnapshots` call (chunked at 200 subjects), and a subject asked for twice while a read is pending or in flight shares that read. 50 mounted resources went from 50 worker messages to 1; the tri-state answer (`true` / `false` / no database to ask) and the "ask again once the database attaches" behaviour are unchanged. - Local Thought connections install a folder after one access check and import in the browser without a preview dialog. Opening the folder or a table refreshes automatically, with five-minute refreshes while open, visible sync status, and local-edit preservation. - Fix: form inputs validate before writing. `InputString` and `InputURI` no longer put an invalid value into the resource while showing an error; all inputs share one validate-then-set hook (`useValidatedInput`) and surface the datatype's actual error message (e.g. "Not an integer") instead of a generic "Invalid value". `ResourceField` now accepts and forwards `commit` / `commitDebounceInterval`, and every input honours `commitDebounceInterval` (Markdown, Number, Boolean, Date, Timestamp, Resource, ResourceArray and FilePicker previously fell back to the 100 ms default). A required field whose value loads after the first render is now flagged correctly. +- Fix: `Resource.remove()` and `Resource.push()` emit `LocalChange` like `set()` does, so `useValue` / `useArray` re-render when a field is cleared or an item is appended. `useValue`'s setter clears a previously reported validation error when called with `undefined`. `useArray`'s `push` saves through the same debounced scheduler and error handling (`handleValidationError` / `store.notifyError`) as `set`, instead of an immediate `resource.save()` that only logged failures. `StoreContext` no longer defaults to a phantom `Store`: `useStore()` throws when no `` is mounted. - Fork bar: "Review changes" opens the per-property diff (the original's current value against the fork's) and names the properties the original also changed since the fork, so a reviewer sees what a merge writes over instead of a count. - A dashboard is reachable from its table: "Add view" offers **Dashboard**, which creates an empty Dashboard as a child of the table and shows it as a tab (`view-kind: dashboard`, `view-dashboard`). Switching an existing tab to Dashboard does the same. The Dashboard stays a resource of its own, so a Drive page or a document can still embed it. - Fix: creating a second table column with a name that already exists in the drive's ontology (e.g. two "Status" columns) no longer mints a colliding property shortname that silently corrupts the ontology. A compatible existing property is reused instead; an incompatible one gets a disambiguated shortname (`status-2`) ([#1504](https://github.com/ontola/atomic-server/issues/1504)). diff --git a/browser/lib/src/resource.test.ts b/browser/lib/src/resource.test.ts index f1097f7ac..0891fb5de 100644 --- a/browser/lib/src/resource.test.ts +++ b/browser/lib/src/resource.test.ts @@ -514,6 +514,63 @@ describe('resource.ts', () => { off2(); }); + /** + * Regression: `remove()` and `push()` wrote to the Loro doc and marked the + * resource dirty but never fired `LocalChange`. `useValue` only listens to + * that per-property event, so clearing a field or appending to an array + * left the bound hooks rendering the old value. + */ + it('remove and push emit a LocalChange for the property', async ({ + expect, + }) => { + const { Resource: ResourceClass, ResourceEvents } = + await import('./resource.js'); + const r = new ResourceClass('https://example.com/remove-push-event'); + const name = 'https://atomicdata.dev/properties/name'; + const tags = 'https://example.com/properties/tags'; + await r.set(name, 'to be removed', false); + r.push(tags, ['a']); + + const events: { prop: string; value: unknown }[] = []; + const off = r.on(ResourceEvents.LocalChange, (prop, value) => + events.push({ prop, value }), + ); + + r.remove(name); + expect(events).toEqual([{ prop: name, value: undefined }]); + expect(r.get(name)).toBeUndefined(); + + r.push(tags, ['b', 'c']); + expect(events).toHaveLength(2); + expect(events[1]).toEqual({ prop: tags, value: ['a', 'b', 'c'] }); + expect(r.get(tags)).toEqual(['a', 'b', 'c']); + + off(); + }); + + it('set(prop, undefined) emits exactly one LocalChange', async ({ + expect, + }) => { + const { Resource: ResourceClass, ResourceEvents } = + await import('./resource.js'); + const r = new ResourceClass('https://example.com/set-undefined-event'); + // A non-ontology property: `set()`'s value type for a known property + // (e.g. `core.properties.name`) excludes `undefined` at compile time. + const note = 'https://example.com/properties/note'; + await r.set(note, 'value', false); + + const events: { prop: string; value: unknown }[] = []; + const off = r.on(ResourceEvents.LocalChange, (prop, value) => + events.push({ prop, value }), + ); + + await r.set(note, undefined, false); + expect(events).toEqual([{ prop: note, value: undefined }]); + expect(r.get(note)).toBeUndefined(); + + off(); + }); + /** * Regression: the resource history page used to read only `getMap('properties')`, * so a Document's body content (which loro-prosemirror writes into a separate diff --git a/browser/lib/src/resource.ts b/browser/lib/src/resource.ts index 001040875..13e053fa5 100644 --- a/browser/lib/src/resource.ts +++ b/browser/lib/src/resource.ts @@ -2588,6 +2588,7 @@ export class Resource { this.loroSetProperty(propUrl, newArray); this.#cacheDirty = true; this._dirty = true; + this.eventManager.emit(ResourceEvents.LocalChange, propUrl, newArray); } /** @@ -2892,6 +2893,7 @@ export class Resource { public remove(propertyUrl: string): void { this.removeUnsafe(propertyUrl); this._dirty = true; + this.eventManager.emit(ResourceEvents.LocalChange, propertyUrl, undefined); } /** @@ -3687,8 +3689,8 @@ export class Resource { } if (value === undefined) { + // `remove` emits the `LocalChange` for this property. this.remove(prop); - this.eventManager.emit(ResourceEvents.LocalChange, prop, value); return; } diff --git a/browser/react/src/hooks.ts b/browser/react/src/hooks.ts index b571f455e..40689c9e5 100644 --- a/browser/react/src/hooks.ts +++ b/browser/react/src/hooks.ts @@ -243,6 +243,21 @@ export function useValue( propertyURL: string, opts: useValueOptions = {}, ): [JSONValue | undefined, SetValue] { + const [val, set] = useValueWithSave(resource, propertyURL, opts); + + return [val, set]; +} + +/** + * `useValue` plus the debounced `saveResource` its setter schedules with, so + * sibling hooks (`useArray`'s `push`) write through the same scheduler and + * error path instead of calling `resource.save()` themselves. + */ +function useValueWithSave( + resource: Resource, + propertyURL: string, + opts: useValueOptions = {}, +): [JSONValue | undefined, SetValue, () => void] { const { commit = false, validate = true, @@ -294,15 +309,13 @@ export function useValue( */ const validateAndSet = useCallback( async (newVal: JSONValue): Promise => { - if (newVal === undefined) { - resource.__internalObject.remove(propertyURL); - saveResource(); - - return; - } - try { - await resource.__internalObject.set(propertyURL, newVal, validate); + if (newVal === undefined) { + resource.__internalObject.remove(propertyURL); + } else { + await resource.__internalObject.set(propertyURL, newVal, validate); + } + saveResource(); handleValidationError?.(undefined); } catch (e) { @@ -325,7 +338,7 @@ export function useValue( // `resource.get(prop)` is typed AtomicValue (JSONValue | Uint8Array). // useValue's contract is JSONValue-only (binary props live in auxValues // and are not surfaced through this hook), so narrow at the boundary. - return [val as JSONValue | undefined, validateAndSet]; + return [val as JSONValue | undefined, validateAndSet, saveResource]; } /** @@ -444,7 +457,13 @@ export function useArray( propertyURL: string, opts?: useValueOptions, ): [string[], SetValue, (vals: string[]) => void] { - const [value, set] = useValue(resource, propertyURL, opts); + const [value, set, saveResource] = useValueWithSave( + resource, + propertyURL, + opts, + ); + const store = useStore(); + const handleValidationError = opts?.handleValidationError; const [stableEmptyResourceArray] = useState([]); const values = useMemo(() => { @@ -467,16 +486,19 @@ export function useArray( const push = useCallback( (val: string[]) => { - resource.push(propertyURL, val); - - if (opts?.commit) { - resource.save().catch(err => { - console.error('Failed to save resource after push', err); - }); + try { + resource.push(propertyURL, val); + // Same debounced scheduler as `set`; honours `commit` and + // `commitDebounce`, and save failures reach `store.notifyError`. + saveResource(); + handleValidationError?.(undefined); + } catch (e) { + if (handleValidationError) handleValidationError(asError(e)); + else store.notifyError(asError(e)); } }, - [resource, propertyURL, opts?.commit], + [resource, propertyURL, saveResource, handleValidationError, store], ); return [values as string[], set, push]; @@ -745,7 +767,7 @@ export function useCanWrite(resource: Resource): boolean { * The context must be provided by wrapping a high level React element in * `My App` */ -export const StoreContext = createContext(new Store()); +export const StoreContext = createContext(undefined); function useMemoizedOpts( opts: FetchOpts | undefined = { From b4036c350e74a311295f48bdd7788b6f70f44c61 Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 05:51:04 +0200 Subject: [PATCH 010/110] refactor(lib): one server-managed property list and one client-db serializer (#1542) Server-managed properties are declared once (server-managed-props.ts) and derived from that list everywhere, and the OPFS row is written through a single Resource.toClientDbJsonAd/Store.recordPersistedState path instead of two hand-rolled serializers that had drifted. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KxExJcGw13DnHknGTMAhXY --- browser/CHANGELOG.md | 1 + .../lib/src/opfs-incomplete-snapshot.test.ts | 31 +++++ browser/lib/src/resource.ts | 115 +++++++----------- browser/lib/src/server-managed-props.ts | 67 ++++++++++ browser/lib/src/store.test.ts | 93 ++++++++++++++ browser/lib/src/store.ts | 50 ++++---- 6 files changed, 259 insertions(+), 98 deletions(-) create mode 100644 browser/lib/src/server-managed-props.ts diff --git a/browser/CHANGELOG.md b/browser/CHANGELOG.md index 85ba83ae6..6674e77da 100644 --- a/browser/CHANGELOG.md +++ b/browser/CHANGELOG.md @@ -9,6 +9,7 @@ This changelog covers all five packages, as they are (for now) updated as a whol - Local Thought connections install a folder after one access check and import in the browser without a preview dialog. Opening the folder or a table refreshes automatically, with five-minute refreshes while open, visible sync status, and local-edit preservation. - Fix: form inputs validate before writing. `InputString` and `InputURI` no longer put an invalid value into the resource while showing an error; all inputs share one validate-then-set hook (`useValidatedInput`) and surface the datatype's actual error message (e.g. "Not an integer") instead of a generic "Invalid value". `ResourceField` now accepts and forwards `commit` / `commitDebounceInterval`, and every input honours `commitDebounceInterval` (Markdown, Number, Boolean, Date, Timestamp, Resource, ResourceArray and FilePicker previously fell back to the 100 ms default). A required field whose value loads after the first render is now flagged correctly. - Fix: `Resource.remove()` and `Resource.push()` emit `LocalChange` like `set()` does, so `useValue` / `useArray` re-render when a field is cleared or an item is appended. `useValue`'s setter clears a previously reported validation error when called with `undefined`. `useArray`'s `push` saves through the same debounced scheduler and error handling (`handleValidationError` / `store.notifyError`) as `set`, instead of an immediate `resource.save()` that only logged failures. `StoreContext` no longer defaults to a phantom `Store`: `useStore()` throws when no `` is mounted. +- `@tomic/lib`: one list of server-managed properties (`server-managed-props.ts`) replaces the three copies in `resource.ts` and `store.ts` that had drifted apart; the OPFS cold-load guard now also treats an inline `genesis` certificate as skeleton, not content. One serializer (`Resource.toClientDbJsonAd()`) writes the local-database row from both `Store.addResource` and `Resource.persistToClientDb`, and a durable save now records its write in the store's dedup stamp so the next ingress does not rewrite the same row. - Fork bar: "Review changes" opens the per-property diff (the original's current value against the fork's) and names the properties the original also changed since the fork, so a reviewer sees what a merge writes over instead of a count. - A dashboard is reachable from its table: "Add view" offers **Dashboard**, which creates an empty Dashboard as a child of the table and shows it as a tab (`view-kind: dashboard`, `view-dashboard`). Switching an existing tab to Dashboard does the same. The Dashboard stays a resource of its own, so a Drive page or a document can still embed it. - Fix: creating a second table column with a name that already exists in the drive's ontology (e.g. two "Status" columns) no longer mints a colliding property shortname that silently corrupts the ontology. A compatible existing property is reused instead; an incompatible one gets a disambiguated shortname (`status-2`) ([#1504](https://github.com/ontola/atomic-server/issues/1504)). diff --git a/browser/lib/src/opfs-incomplete-snapshot.test.ts b/browser/lib/src/opfs-incomplete-snapshot.test.ts index 194bf8685..ba98f8026 100644 --- a/browser/lib/src/opfs-incomplete-snapshot.test.ts +++ b/browser/lib/src/opfs-incomplete-snapshot.test.ts @@ -25,6 +25,7 @@ const PARENT = 'https://atomicdata.dev/properties/parent'; const DRIVE = 'https://atomicdata.dev/properties/drive'; const CREATED_AT = 'https://atomicdata.dev/properties/createdAt'; const LAST_COMMIT = 'https://atomicdata.dev/properties/lastCommit'; +const GENESIS = 'https://atomicdata.dev/properties/genesis'; const NAME = 'https://atomicdata.dev/properties/name'; const FOLDER = 'https://atomicdata.dev/classes/Folder'; @@ -146,4 +147,34 @@ describe('OPFS cold-load β€” incomplete snapshot must not yield a silent-faulty expect(r.loading).toBe(true); expect(r.get(core.properties.isA)).toBeUndefined(); }); + + it('a skeleton that also carries its genesis certificate is still a skeleton', async ({ + expect, + }) => { + // Every DID resource this client creates carries an inline genesis + // certificate, and `rebuildCacheFromLoro` preserves it like the other + // server-managed props. It is not content: a skeleton row that has it + // must be judged by the same list the rebuild preserves, or the guard + // waves through exactly the state the two cases above reject. + const store = await makeStore(); + store.setClientDb( + fakeClientDb( + subject, + JSON.stringify({ + ...JSON.parse(skeletonJsonAd), + [GENESIS]: 'AQID', + }), + incompleteDelta(), + ) as unknown as Parameters[0], + ); + + const r = store.getResourceLoading(subject); + + for (let i = 0; i < 50 && r.loading; i++) { + await new Promise(res => setTimeout(res, 10)); + } + + expect(r.loading).toBe(true); + expect(r.get(core.properties.isA)).toBeUndefined(); + }); }); diff --git a/browser/lib/src/resource.ts b/browser/lib/src/resource.ts index 13e053fa5..21bfa9407 100644 --- a/browser/lib/src/resource.ts +++ b/browser/lib/src/resource.ts @@ -38,6 +38,11 @@ import { type MergeForkOptions, } from './forks.js'; import { GENESIS, properties, instances } from './urls.js'; +import { + DERIVED_BY_SERVER, + SERVER_MANAGED_PROPS, + isDerivedByServer, +} from './server-managed-props.js'; import { valToArray, type JSONValue, @@ -49,33 +54,6 @@ import { /** Contains the PropertyURL / Value combinations */ export type PropVals = Map; -/** - * Propvals the server derives and ships alongside a resource, which must never - * be written into its CRDT. - * - * They all have another source of truth: `lastCommit` is commit metadata, - * `createdAt` and `createdBy` come from the genesis certificate (see - * {@link Resource.getCreatedBy}). Writing one into Loro produces a LOCAL - * operation, and a local operation means a dirty subject and a commit β€” for a - * value the client never authored. On a resource you may write that is a - * redundant commit per hydration; on one you may only READ, the server refuses - * it forever, which is how an invitee who had written nothing ended up with a - * blocked outbox entry and a permanent "changes pending". - * - * They stay in the read cache, so `resource.get()` and JSON-AD round-trips are - * unaffected β€” which is why every `serverManaged` preservation list below has - * to include them. - */ -const DERIVED_BY_SERVER: ReadonlySet = new Set([ - properties.commit.lastCommit, - commits.properties.createdAt, - properties.createdBy, -]); - -/** True for a propval the server derives β€” see {@link DERIVED_BY_SERVER}. */ -const isDerivedByServer = (prop: string): boolean => - DERIVED_BY_SERVER.has(prop); - export interface MergeOptions { replaceLoroDocs?: boolean; /** @@ -797,23 +775,9 @@ export class Resource { // Preserve server-managed / genesis-immutable properties in the cache. // These are set once (at genesis or by the server) and are NOT necessarily // re-encoded into a later Loro delta β€” so a rebuild from a delta-only doc - // would otherwise drop them. `drive` and `parent` matter especially for a - // GUEST who loaded a shared resource: losing the parent's `drive` here - // leaves a reply unstamped, so the drive-scoped commit fan-out never - // delivers it to the owner. See planning/commit-fanout-drive-isolation.md. - const serverManaged = [ - properties.commit.lastCommit, - commits.properties.createdAt, - properties.createdBy, - 'https://atomicdata.dev/properties/drive', - // The inline genesis certificate: set once at creation, immutable, and - // must not be dropped when the cache is rebuilt from a delta-only doc β€” - // it's what verifies the resource's DID. - 'https://atomicdata.dev/properties/genesis', - core.properties.parent, - ]; - - for (const key of serverManaged) { + // would otherwise drop them. See `SERVER_MANAGED_PROPS` for why each is + // on the list. + for (const key of SERVER_MANAGED_PROPS) { if (this.#cache[key] !== undefined && nextCache[key] === undefined) { nextCache[key] = this.#cache[key]; } @@ -1689,15 +1653,13 @@ export class Resource { } } - // Copy housekeeping properties from resourceB.#cache to this.#cache - // before rebuilding cache so they are preserved - const serverManaged = [ - properties.commit.lastCommit, - commits.properties.createdAt, - properties.createdBy, - ]; - - for (const key of serverManaged) { + // Copy the server-derived properties from resourceB.#cache to + // this.#cache before rebuilding the cache so they are preserved. + // Deliberately the narrower `DERIVED_BY_SERVER`, not + // `SERVER_MANAGED_PROPS`: drive/genesis/parent live in the Loro doc + // and arrive through the CRDT import above, whereas these three are + // in no doc at all and only the incoming (server-fresh) cache has them. + for (const key of DERIVED_BY_SERVER) { if (resourceB.#cache[key] !== undefined) { this.#cache[key] = resourceB.#cache[key]; } @@ -1721,14 +1683,8 @@ export class Resource { } } else { // No incoming Loro snapshot (e.g. metadata-only update or non-crdt resource) - // Copy housekeeping properties first - const serverManaged = [ - properties.commit.lastCommit, - commits.properties.createdAt, - properties.createdBy, - ]; - - for (const key of serverManaged) { + // Copy the server-derived properties first (same reasoning as above). + for (const key of DERIVED_BY_SERVER) { if (resourceB.#cache[key] !== undefined) { this.#cache[key] = resourceB.#cache[key]; } @@ -2472,6 +2428,25 @@ export class Resource { return count === 0 ? null : obj; } + /** + * The JSON-AD row the client database (OPFS) stores for this resource: + * `@id` plus every non-binary propval. Binary aux values are left out β€” a + * blob lives in the blob store, keyed by hash, and the snapshot travels next + * to this row (see `ClientDb.putResourceWithSnapshot`). + * + * The one serializer for that row: `Store.addResource` and + * {@link persistToClientDb} both write through it, so the store's + * "already persisted" stamp (`Store.recordPersistedState`) is computed over + * exactly the bytes either path writes. + * + * Returns `null` when the resource has no non-binary propvals at all. + */ + public toClientDbJsonAd(): string | null { + const obj = this.toObject({ includeBinary: false }); + + return obj ? JSON.stringify(obj) : null; + } + /** Compact debug representation for error messages. */ public debugValueSummary(): string { const objectView = this.toObject({ includeBinary: false }); @@ -3599,11 +3574,10 @@ export class Resource { const clientDb = this.store.getClientDb(); if (!clientDb || clientDb.unsupportedEnvironment) return; - const obj: Record = { '@id': this.subject }; - - for (const [k, v] of this.getEntries()) { - if (!(v instanceof Uint8Array)) obj[k] = v; - } + // A resource without propvals still gets its `@id` row: this path is the + // durable one, and a caller that awaited it expects a row to exist. + const jsonAd = + this.toClientDbJsonAd() ?? JSON.stringify({ '@id': this.subject }); // The export seals pending ops; keep them under their history token. this.sealPendingEdits(); @@ -3615,14 +3589,13 @@ export class Resource { const closePersist = perfSpan('resource.persistToClientDb'); try { - await clientDb.putResourceWithSnapshot( - this.subject, - JSON.stringify(obj), - snapshot, - ); + await clientDb.putResourceWithSnapshot(this.subject, jsonAd, snapshot); // This RPC includes the durable flush. A second RPC could race the // identity handoff closing this worker after the write has completed. closePersist(); + // Tell the store's dedup cache what is now on disk, so the next + // `addResource` for this subject does not rewrite the same row. + this.store.recordPersistedState(this.subject, jsonAd, snapshot); } catch (e) { closePersist({ err: e instanceof Error ? e.message : String(e) }); diff --git a/browser/lib/src/server-managed-props.ts b/browser/lib/src/server-managed-props.ts new file mode 100644 index 000000000..27297e37b --- /dev/null +++ b/browser/lib/src/server-managed-props.ts @@ -0,0 +1,67 @@ +import { commits } from './ontologies/commits.js'; +import { core } from './ontologies/core.js'; +import { server } from './ontologies/server.js'; +import { GENESIS } from './urls.js'; + +/** + * The one place that says which properties the client never authors. + * + * Three call sites used to carry their own copy of this list and had drifted + * apart (`Resource.rebuildCacheFromLoro`, `Resource.merge`, the store's OPFS + * cold-load guard). They are now the two sets below; one is derived from the + * other, so a property added here reaches every consumer. + */ + +/** + * Properties the SERVER derives for a resource and that the client must never + * write into its CRDT. + * + * They all have another source of truth: `lastCommit` is commit metadata, + * `createdAt` and `createdBy` come from the genesis certificate (see + * `Resource.getCreatedBy`). Writing one into Loro produces a LOCAL operation, + * and a local operation means a dirty subject and a commit β€” for a value the + * client never authored. On a resource you may write that is a redundant + * commit per hydration; on one you may only READ, the server refuses it + * forever, which is how an invitee who had written nothing ended up with a + * blocked outbox entry and a permanent "changes pending". + * + * They stay in the read cache, so `resource.get()` and JSON-AD round-trips are + * unaffected β€” which is why {@link SERVER_MANAGED_PROPS} includes them, and why + * `Resource.merge` copies exactly this set from the incoming resource's cache: + * a CRDT merge cannot carry a value that is in no Loro doc, so the incoming + * (server-fresh) copy has to be taken by hand. + */ +export const DERIVED_BY_SERVER: ReadonlySet = new Set([ + commits.properties.lastCommit, + commits.properties.createdAt, + server.properties.createdBy, +]); + +/** True for a propval the server derives β€” see {@link DERIVED_BY_SERVER}. */ +export const isDerivedByServer = (prop: string): boolean => + DERIVED_BY_SERVER.has(prop); + +/** + * Everything in {@link DERIVED_BY_SERVER} plus the genesis-immutable + * properties: set once at creation (by the client, into Loro) and never + * re-encoded into a later delta, so a cache rebuilt from a delta-only doc would + * otherwise drop them. `Resource.rebuildCacheFromLoro` preserves these from the + * previous cache. + * + * A resource that has ONLY these β€” no class, no user content β€” is a skeleton, + * not a renderable resource. The store's OPFS cold-load guard + * (`Store.hasRenderableContent`) uses the same set to decide whether a local + * hit is authoritative, so a preserved-but-contentless hydration cannot pass + * for content. + */ +export const SERVER_MANAGED_PROPS: ReadonlySet = new Set([ + ...DERIVED_BY_SERVER, + 'https://atomicdata.dev/properties/drive', + // The inline genesis certificate: it verifies the resource's DID. `drive` + // and `parent` matter especially for a GUEST who loaded a shared resource: + // losing the parent's `drive` leaves a reply unstamped, so the drive-scoped + // commit fan-out never delivers it to the owner. See + // planning/commit-fanout-drive-isolation.md. + GENESIS, + core.properties.parent, +]); diff --git a/browser/lib/src/store.test.ts b/browser/lib/src/store.test.ts index c7ba9d7ba..a667978e7 100644 --- a/browser/lib/src/store.test.ts +++ b/browser/lib/src/store.test.ts @@ -569,6 +569,99 @@ describe('Store', () => { ); }); + it('writes the same client-DB row from addResource and persistToClientDb', async ({ + expect, + }) => { + await enableLoro(); + const store = new Store({ serverUrl: 'https://example.com' }); + const putResourceWithSnapshot = vi.fn().mockResolvedValue(undefined); + store.setClientDb({ + isReady: true, + flush: async () => undefined, + putResourceWithSnapshot, + } as unknown as Parameters[0]); + + const binaryProp = 'https://example.com/properties/bytes'; + const loroUpdate = 'https://atomicdata.dev/properties/loroUpdate'; + const resource = new Resource('did:ad:same-row-both-paths'); + resource.setStore(store); + await resource.set( + core.properties.isA, + ['https://atomicdata.dev/classes/Folder'], + false, + ); + await resource.set(core.properties.name, 'Same row', false); + // A binary propval and a Loro snapshot are the two entry kinds a row + // must leave out: the first is a Uint8Array aux value, the second is the + // doc itself and travels next to the row. + resource.applyHydratedValues([ + [binaryProp, new Uint8Array([1, 2, 3])], + [loroUpdate, resource.getLoroDoc()!.export({ mode: 'snapshot' })], + ]); + resource.loading = false; + resource.new = false; + expect(resource.getEntries().map(([k]) => k)).toContain(binaryProp); + + store.addResource(resource); + expect(putResourceWithSnapshot).toHaveBeenCalledTimes(1); + await resource.persistToClientDb(); + expect(putResourceWithSnapshot).toHaveBeenCalledTimes(2); + + const [, rowFromAddResource, snapshotFromAddResource] = + putResourceWithSnapshot.mock.calls[0]; + const [, rowFromPersist, snapshotFromPersist] = + putResourceWithSnapshot.mock.calls[1]; + expect(rowFromPersist).toBe(rowFromAddResource); + expect(rowFromPersist).toBe(resource.toClientDbJsonAd()); + expect(snapshotFromAddResource).toBeInstanceOf(Uint8Array); + expect(snapshotFromPersist).toBeInstanceOf(Uint8Array); + + const row = JSON.parse(rowFromPersist); + expect(row['@id']).toBe(resource.subject); + expect(row[core.properties.name]).toBe('Same row'); + expect(row[core.properties.isA]).toEqual([ + 'https://atomicdata.dev/classes/Folder', + ]); + expect(row).not.toHaveProperty(binaryProp); + expect(row).not.toHaveProperty(loroUpdate); + }); + + it('does not rewrite a row that persistToClientDb already wrote', async ({ + expect, + }) => { + await enableLoro(); + + const build = async () => { + const store = new Store({ serverUrl: 'https://example.com' }); + const putResourceWithSnapshot = vi.fn().mockResolvedValue(undefined); + store.setClientDb({ + isReady: true, + flush: async () => undefined, + putResourceWithSnapshot, + } as unknown as Parameters[0]); + const resource = new Resource('did:ad:persisted-then-added'); + resource.setStore(store); + await resource.set(core.properties.name, 'Persisted first', false); + resource.loading = false; + resource.new = false; + + return { store, resource, putResourceWithSnapshot }; + }; + + // Control: the ingress path writes a resource the store has not seen. + const control = await build(); + control.store.addResource(control.resource); + expect(control.putResourceWithSnapshot).toHaveBeenCalledTimes(1); + + // The durable save path wrote the row; the ingress that follows (a WS + // echo, a notify) finds the same state on disk and skips its write. + const { store, resource, putResourceWithSnapshot } = await build(); + await resource.persistToClientDb(); + expect(putResourceWithSnapshot).toHaveBeenCalledTimes(1); + store.addResource(resource); + expect(putResourceWithSnapshot).toHaveBeenCalledTimes(1); + }); + it('clears the previous identity drive before sign-out authentication changes', async ({ expect, }) => { diff --git a/browser/lib/src/store.ts b/browser/lib/src/store.ts index bb352a010..4a6a39a63 100644 --- a/browser/lib/src/store.ts +++ b/browser/lib/src/store.ts @@ -62,6 +62,7 @@ import { bytesToHex, hexToBytes, type JSONValue } from './value.js'; import { WSClient } from './websockets.js'; import { withDeadline } from './withDeadline.js'; import { BLOB, endpoints, INTERNAL_ID } from './urls.js'; +import { SERVER_MANAGED_PROPS } from './server-managed-props.js'; import { initOntologies } from './ontologies/index.js'; import { decodeB64, encodeB64Url } from './base64.js'; import { @@ -416,22 +417,6 @@ interface LocalHydrationRequest { */ const CLIENT_DB_ATTACH_GRACE = 5000; -/** - * The server-managed props that `Resource.rebuildCacheFromLoro` preserves even - * when a Loro doc carries no delta for them (drive/parent/lastCommit/createdAt). - * A resource that has ONLY these β€” no class, no user content β€” is a skeleton, - * not a renderable resource. Used by the OPFS cold-load guard to decide whether - * a local hit is authoritative. Keep in sync with the `serverManaged` list in - * resource.ts. - */ -const SERVER_MANAGED_SKELETON_PROPS: ReadonlySet = new Set([ - commits.properties.lastCommit, - commits.properties.createdAt, - 'https://atomicdata.dev/properties/createdBy', - 'https://atomicdata.dev/properties/drive', - core.properties.parent, -]); - /** * Cheap equality for commit-log property values. Strict `===` would always * report arrays/objects as different even when their contents match, so the @@ -2191,7 +2176,7 @@ export class Store { !emitResource.get(core.properties.incomplete) ) { try { - const jsonAd = resourceToJsonAd(emitResource); + const jsonAd = emitResource.toClientDbJsonAd(); if (jsonAd) { const doc = emitResource.getLoroDoc?.(); @@ -3045,8 +3030,8 @@ export class Store { /** * True when the resource carries enough state to stand on its own: a * class, or any property beyond the server-managed skeleton - * (drive/parent/lastCommit/createdAt) that `rebuildCacheFromLoro` - * preserves. A resource that passes this is worth more than a failed + * (`SERVER_MANAGED_PROPS`) that `rebuildCacheFromLoro` preserves. A + * resource that passes this is worth more than a failed * fetch β€” it renders, and the alternative is showing the user nothing. */ private hasRenderableContent(resource: Resource | undefined): boolean { @@ -3056,7 +3041,7 @@ export class Store { return resource .getEntries() - .some(([prop]) => !SERVER_MANAGED_SKELETON_PROPS.has(prop)); + .some(([prop]) => !SERVER_MANAGED_PROPS.has(prop)); } /** Local-database reads asked for since the last flush, one entry per @@ -6342,6 +6327,24 @@ export class Store { * dropped by `removeResource` and by a failed write. */ private lastPersistedStamp = new Map(); + /** + * Record that `jsonAd` + `snapshot` is what the local DB now holds for + * `subject`, so `addResource` skips re-writing that same state. Called by + * `Resource.persistToClientDb` after its durable write lands β€” without it + * the dedup cache only knew about writes `addResource` itself made, and + * rewrote the row on the next ingress. + * + * @internal + */ + public recordPersistedState( + subject: string, + jsonAd: string, + snapshot?: Uint8Array, + ): void { + // Keyed like `addResource` keys it: by the resource's own subject. + this.lastPersistedStamp.set(subject, hashPersistedState(jsonAd, snapshot)); + } + private snapshotReadDepth = 0; public getResourceSnapshot( @@ -6488,13 +6491,6 @@ export interface FetchOpts { newResource?: boolean; } -/** Convert a Resource to a JSON-AD string for storage in the WASM DB. */ -function resourceToJsonAd(resource: Resource): string | null { - const obj = resource.toObject({ includeBinary: false }); - - return obj ? JSON.stringify(obj) : null; -} - /** * Cheap 53-bit content stamp over exactly what a local-DB write persists: the * JSON-AD and the Loro snapshot. Used only to skip re-writing state identical From e9ec2f35e9f7cf9ef1065fc8f045d1879116b8bf Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 05:52:23 +0200 Subject: [PATCH 011/110] fix(lib): route destroy through the durable outbox so offline deletes survive (#1556) A destroy is signed into the durable outbox and drained like every other commit, instead of being POSTed straight to the server, so a delete made offline (or during a reload) is retried rather than lost. Incoming updates and local-db hydration skip a subject with a pending destroy. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KxExJcGw13DnHknGTMAhXY --- TESTING_COVERAGE.md | 13 + browser/CHANGELOG.md | 1 + .../src/actions/resourceActions.tsx | 9 +- browser/lib/src/destroy-via-outbox.test.ts | 356 ++++++++++++++++++ browser/lib/src/local-outbox.ts | 114 +++++- browser/lib/src/resource.ts | 97 +++-- browser/lib/src/store.ts | 95 +++++ 7 files changed, 641 insertions(+), 44 deletions(-) create mode 100644 browser/lib/src/destroy-via-outbox.test.ts diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index 1f1275607..4b06283e0 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -921,6 +921,19 @@ mounts without resetting or re-registering the global parser. drops), backoff, blocked entries and cancellation cannot report persistence. It also covers offline transport failures, successful retries, unrelated subjects and edits arriving during an acknowledged save (#1388). +- `destroy-via-outbox.test.ts` exercises `Resource.destroy()` through the same + outbox: an online delete POSTs one destroy commit and removes the resource; a + delete while disconnected queues the pre-signed envelope, survives a simulated + reload (fresh `LocalOutbox` hydrating the same agent namespace) and is POSTed + exactly once on reconnect; create + delete while offline POSTs neither + envelope; a never-saved `newResource` is dropped without a POST; a server + refusal rejects `destroy()` and keeps the entry queued; a transport failure + resolves as queued and flips the store offline; "already gone" server answers + (`already applied here`, `predates the resource's genesis`, `does not exist + yet`) count as acknowledged; a pending destroy blocks resurrection through + `applyIncoming` / `hydrateResourceFromJsonAd` and is excluded from + `computeDriveSyncState`. Not covered: a real server round trip for the + reconnect drain (no `*.integration.test.ts` or Playwright variant yet). - `scripts/owned-process.node.mjs` exercises the template runner process lifecycle, including independent ephemeral ports and descendant cleanup. The superseded diff --git a/browser/CHANGELOG.md b/browser/CHANGELOG.md index 6674e77da..e5ba7d0c5 100644 --- a/browser/CHANGELOG.md +++ b/browser/CHANGELOG.md @@ -6,6 +6,7 @@ This changelog covers all five packages, as they are (for now) updated as a whol - Fix: `@tomic/lib` writes the Loro `datatypes` tags in the export path shared by genesis and drain-time signing, after the user's pending ops are sealed into their own commit. The drain used to tag first, which folded a `set()` edit into the tag write's `atomic:system` commit β€” the canvas undo stack skips that origin, so the first edit of a `json`/reference property after a save could not be undone. - Perf: a cold page load asks the local database (OPFS worker) for all the resources a render pass mounts in one round trip instead of one per `useResource`. `Store` coalesces the render-phase cache misses of a tick into a single `getResourcesWithSnapshots` call (chunked at 200 subjects), and a subject asked for twice while a read is pending or in flight shares that read. 50 mounted resources went from 50 worker messages to 1; the tri-state answer (`true` / `false` / no database to ask) and the "ask again once the database attaches" behaviour are unchanged. +- Fix: deleting a resource while offline no longer loses the delete. `Resource.destroy()` now goes through the durable `LocalOutbox` like every other write: the signed destroy commit is queued (`OutboxEntry.signedDestroy`, persisted in localStorage), the resource is removed locally at once, and the drain POSTs the envelope with the usual backoff and reconnect replay. Online, `await destroy()` still resolves after the server acknowledgement and rejects on a real refusal; offline it resolves with the delete queued. A resource created and deleted before either write reached the server is dropped without a POST, and a server answer that the resource is already gone counts as acknowledged instead of retrying forever. While a delete is queued, incoming state for that subject (a subscription push, a fetch that raced the delete, a stale OPFS row) no longer resurrects it. `Store.hasPendingDestroy(subject)` is new; `LocalOutbox` gains `setDestroyCommit`, `clearDestroy` and `discard`. - Local Thought connections install a folder after one access check and import in the browser without a preview dialog. Opening the folder or a table refreshes automatically, with five-minute refreshes while open, visible sync status, and local-edit preservation. - Fix: form inputs validate before writing. `InputString` and `InputURI` no longer put an invalid value into the resource while showing an error; all inputs share one validate-then-set hook (`useValidatedInput`) and surface the datatype's actual error message (e.g. "Not an integer") instead of a generic "Invalid value". `ResourceField` now accepts and forwards `commit` / `commitDebounceInterval`, and every input honours `commitDebounceInterval` (Markdown, Number, Boolean, Date, Timestamp, Resource, ResourceArray and FilePicker previously fell back to the 100 ms default). A required field whose value loads after the first render is now flagged correctly. - Fix: `Resource.remove()` and `Resource.push()` emit `LocalChange` like `set()` does, so `useValue` / `useArray` re-render when a field is cleared or an item is appended. `useValue`'s setter clears a previously reported validation error when called with `undefined`. `useArray`'s `push` saves through the same debounced scheduler and error handling (`handleValidationError` / `store.notifyError`) as `set`, instead of an immediate `resource.save()` that only logged failures. `StoreContext` no longer defaults to a phantom `Store`: `useStore()` throws when no `` is mounted. diff --git a/browser/data-browser/src/actions/resourceActions.tsx b/browser/data-browser/src/actions/resourceActions.tsx index d42b35239..569d1e079 100644 --- a/browser/data-browser/src/actions/resourceActions.tsx +++ b/browser/data-browser/src/actions/resourceActions.tsx @@ -472,9 +472,12 @@ export const resourceActions: ActionDefinition[] = [ } } catch (error) { // A failed delete is the one that most needs saying so: `destroy()` - // throws before it removes the resource locally, so the row stays in - // the tree and the only thing distinguishing "deleted" from "refused" - // is this message. It used to read `(error as Error).message`, which is + // removes the resource locally at once and queues the signed destroy + // in the outbox, so when the server refuses it the row is already gone + // from the tree while the server still has it β€” the only thing + // distinguishing "deleted" from "refused" is this message (the entry + // stays queued and is retried / parked by the outbox). It used to read + // `(error as Error).message`, which is // `undefined` for anything thrown that isn't an Error β€” an empty toast, // i.e. a delete that silently did nothing. const detail = diff --git a/browser/lib/src/destroy-via-outbox.test.ts b/browser/lib/src/destroy-via-outbox.test.ts new file mode 100644 index 000000000..ad92d79a4 --- /dev/null +++ b/browser/lib/src/destroy-via-outbox.test.ts @@ -0,0 +1,356 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { Agent } from './agent.js'; +import { CommitBuilder } from './commit.js'; +import { JSCryptoProvider } from './CryptoProvider.js'; +import { core } from './ontologies/core.js'; +import { server } from './ontologies/server.js'; +import { LocalOutbox, isSettledDestroyErrorMessage } from './local-outbox.js'; +import { testStore } from './test-store.js'; +import type { ClientDbWorker } from './client-db.js'; + +afterEach(() => { + vi.restoreAllMocks(); + localStorage.clear(); + // `test-setup.ts` relies on this flag to keep the Store off a real WS. + localStorage.setItem('ws-disconnected', '1'); +}); + +/** A saved (server-acknowledged) drive: `new` is false, the outbox is empty. */ +async function savedDrive(name = 'Doomed') { + const harness = await testStore(); + const doc = await harness.store.newResource({ + isA: server.classes.drive, + noParent: true, + propVals: { [core.properties.name]: name }, + }); + await doc.save(); + expect(harness.store.outbox.hasPending(doc.subject)).toBe(false); + harness.postCommitSpy.mockClear(); + + return { ...harness, doc }; +} + +describe('Resource.destroy() through the outbox', () => { + it('online: POSTs one destroy commit and removes the resource', async () => { + const { store, doc, postCommitSpy } = await savedDrive(); + + await expect(doc.destroy()).resolves.toBeUndefined(); + + expect(postCommitSpy).toHaveBeenCalledTimes(1); + const commit = postCommitSpy.mock.calls[0][0]; + expect(commit.destroy).toBe(true); + expect(commit.subject).toBe(doc.subject); + expect(commit.loroUpdate).toBeUndefined(); + expect(store.resources.has(doc.subject)).toBe(false); + expect(store.outbox.hasPending(doc.subject)).toBe(false); + expect(store.getSyncStatus().pendingDirtyCount).toBe(0); + store.setServerConnected(false); + }); + + it('offline: queues the signed destroy, survives a reload, drains once on reconnect', async () => { + const { store, doc, agentDID, postCommitSpy } = await savedDrive(); + store.setServerConnected(false); + + await expect(doc.destroy()).resolves.toBeUndefined(); + + // Removed locally at once, nothing sent, the envelope is queued. + expect(postCommitSpy).not.toHaveBeenCalled(); + expect(store.resources.has(doc.subject)).toBe(false); + const queued = store.outbox.getEntry(doc.subject)?.signedDestroy; + expect(queued?.destroy).toBe(true); + expect(queued?.signature).toBeTruthy(); + expect(store.hasPendingDestroy(doc.subject)).toBe(true); + + // Simulated reload: a fresh outbox hydrating the same agent's storage + // must hold the SAME signed envelope (a destroy cannot be re-signed β€” + // the resource it names is already gone locally). + store.outbox.flush(); + const reloaded = new LocalOutbox(); + reloaded.rebind(agentDID); + const restored = reloaded.getEntry(doc.subject)?.signedDestroy; + expect(restored?.destroy).toBe(true); + expect(restored?.subject).toBe(doc.subject); + expect(restored?.signature).toBe(queued!.signature); + expect(restored?.signer).toBe(agentDID); + + // Reconnect: the drain POSTs it exactly once and drops the entry. + store.setServerConnected(true); + await store.syncDirtyResources(); + + expect(postCommitSpy).toHaveBeenCalledTimes(1); + expect(postCommitSpy.mock.calls[0][0].signature).toBe(queued!.signature); + expect(store.outbox.hasPending(doc.subject)).toBe(false); + expect(store.hasPendingDestroy(doc.subject)).toBe(false); + expect(store.resources.has(doc.subject)).toBe(false); + store.setServerConnected(false); + }); + + it('offline create + destroy: POSTs neither the genesis nor the destroy', async () => { + const { store, postCommitSpy } = await testStore(); + store.setServerConnected(false); + + const doc = await store.newResource({ + isA: server.classes.drive, + noParent: true, + propVals: { [core.properties.name]: 'Never synced' }, + }); + await expect(doc.save()).resolves.toBe('offline'); + expect(store.outbox.getEntry(doc.subject)?.signedGenesis).toBeTruthy(); + + await expect(doc.destroy()).resolves.toBeUndefined(); + expect(store.resources.has(doc.subject)).toBe(false); + // Both envelopes sit on the entry until the drain resolves them. + const entry = store.outbox.getEntry(doc.subject); + expect(entry?.signedGenesis).toBeTruthy(); + expect(entry?.signedDestroy).toBeTruthy(); + + store.setServerConnected(true); + await store.syncDirtyResources(); + + expect(postCommitSpy).not.toHaveBeenCalled(); + expect(store.outbox.hasPending(doc.subject)).toBe(false); + expect(store.resources.has(doc.subject)).toBe(false); + expect(store.getSyncStatus().pendingDirtyCount).toBe(0); + store.setServerConnected(false); + }); + + it('a newResource that was never saved is dropped without a POST', async () => { + const { store, postCommitSpy } = await testStore(); + const doc = await store.newResource({ + isA: server.classes.drive, + noParent: true, + propVals: { [core.properties.name]: 'Placeholder' }, + }); + // Genesis is parked on the resource until the first `save()`. + expect(store.outbox.hasPending(doc.subject)).toBe(false); + + await expect(doc.destroy()).resolves.toBeUndefined(); + await store.syncDirtyResources(); + + expect(postCommitSpy).not.toHaveBeenCalled(); + expect(store.resources.has(doc.subject)).toBe(false); + expect(store.outbox.hasPending(doc.subject)).toBe(false); + store.setServerConnected(false); + }); + + it('online: a server refusal rejects destroy() and keeps the delete queued', async () => { + const { store, doc, postCommitSpy } = await savedDrive(); + const error = new Error( + 'No https://atomicdata.dev/properties/write right has been found for did:ad:agent:x in its parents', + ); + postCommitSpy.mockRejectedValue(error); + + await expect(doc.destroy()).rejects.toBe(error); + + expect(postCommitSpy).toHaveBeenCalledTimes(1); + // Not acknowledged: the entry stays for the outbox's backoff / blocking + // handling, and the local removal is not undone by the refusal. + expect(store.outbox.getEntry(doc.subject)?.signedDestroy).toBeTruthy(); + expect(store.outbox.getEntry(doc.subject)?.failures).toBe(1); + expect(store.resources.has(doc.subject)).toBe(false); + store.setServerConnected(false); + }); + + it('online: a transport failure resolves as queued and flips the store offline', async () => { + const { store, doc, postCommitSpy } = await savedDrive(); + postCommitSpy.mockRejectedValue(new TypeError('Failed to fetch')); + + await expect(doc.destroy()).resolves.toBeUndefined(); + + expect(store.serverConnected).toBe(false); + expect(store.outbox.getEntry(doc.subject)?.signedDestroy).toBeTruthy(); + expect(store.resources.has(doc.subject)).toBe(false); + }); + + it.each([ + 'Destroy commit for did:ad:x was already applied here; refusing replay', + "Destroy commit for did:ad:x (created 2) predates the resource's genesis (3); refusing replay", + 'Commit for did:ad:x has is_genesis: false, but the resource does not exist yet.', + ])('"%s" counts as acknowledged: entry dropped, no retry', async message => { + const { store, doc, postCommitSpy } = await savedDrive(); + postCommitSpy.mockRejectedValue(new Error(message)); + const notifyError = vi.spyOn(store, 'notifyError'); + + await expect(doc.destroy()).resolves.toBeUndefined(); + + expect(postCommitSpy).toHaveBeenCalledTimes(1); + expect(store.outbox.hasPending(doc.subject)).toBe(false); + expect(store.resources.has(doc.subject)).toBe(false); + expect(notifyError).not.toHaveBeenCalled(); + store.setServerConnected(false); + }); + + it('does not resurrect a destroyed-but-unacked resource from a push or a fetch', async () => { + const { store, doc } = await savedDrive(); + store.setServerConnected(false); + await doc.destroy(); + expect(store.resources.has(doc.subject)).toBe(false); + + // A late WS push / SYNC frame for the subject. + const pushed = store.applyIncoming({ + subject: doc.subject, + source: 'ws-sub-push', + loroBytes: doc.getLoroDoc()!.export({ mode: 'snapshot' }), + commitId: 'did:ad:commit:someoneElse', + }); + expect(pushed).toBe('deduped'); + expect(store.resources.has(doc.subject)).toBe(false); + + // A fetch result that raced the delete. + const fetched = store.applyIncoming({ + subject: doc.subject, + source: 'http-fetch', + resource: doc, + }); + expect(fetched).toBe('deduped'); + expect(store.resources.has(doc.subject)).toBe(false); + + // A stale OPFS row the worker had not yet tombstoned. + expect( + store.hydrateResourceFromJsonAd( + doc.subject, + JSON.stringify({ + '@id': doc.subject, + [core.properties.name]: 'Back from the dead', + }), + ), + ).toBe(true); + expect(store.resources.has(doc.subject)).toBe(false); + }); + + it('excludes a subject with a pending destroy from the drive VV', async () => { + const { store, doc } = await savedDrive('Drive'); + const child = await store.newResource({ + parent: doc.subject, + propVals: { [core.properties.name]: 'Child' }, + }); + await child.save(); + store.setServerConnected(false); + + // Pretend OPFS still lists the child (the tombstone write is async). + (store as unknown as { clientDb: ClientDbWorker }).clientDb = { + isReady: true, + getVersionVectorsForDrive: async () => ({ + [child.subject]: { '1': 1 }, + }), + removeResource: async () => undefined, + } as unknown as ClientDbWorker; + + await child.destroy(); + const state = await store.computeDriveSyncState(doc.subject); + + expect(Object.keys(state.resources)).not.toContain(child.subject); + expect(Object.keys(state.resources)).toContain(doc.subject); + }); +}); + +describe('LocalOutbox destroy envelopes', () => { + async function signedDestroy(subject: string) { + const keys = await Agent.generateKeyPair(); + const agentDID = `did:ad:agent:${keys.publicKey}`; + const agent = new Agent(new JSCryptoProvider(keys.privateKey), agentDID); + const builder = new CommitBuilder(subject); + builder.setDestroy(true); + + return { commit: await builder.sign(agent), agentDID }; + } + + it('setDestroyCommit keeps the entry across clearDirty; clearDestroy drops it whole', async () => { + const outbox = new LocalOutbox(); + const subject = 'did:ad:doomed'; + const { commit } = await signedDestroy(subject); + + outbox.markDirty(subject); + outbox.setBaseVersion(subject, 'AAAA'); + outbox.setDestroyCommit(subject, commit); + + const entry = outbox.getEntry(subject)!; + expect(entry.signedDestroy).toBe(commit); + // No Loro delta is left to replay for a subject that is going away. + expect(entry.baseVersion).toBeUndefined(); + + outbox.clearDirty(subject); + expect(outbox.getEntry(subject)?.signedDestroy).toBe(commit); + + outbox.clearDestroy(subject); + expect(outbox.hasPending(subject)).toBe(false); + }); + + it('setDestroyCommit re-arms a blocked entry', async () => { + const outbox = new LocalOutbox(); + const subject = 'did:ad:blocked'; + const { commit } = await signedDestroy(subject); + outbox.markDirty(subject); + const entry = outbox.getEntry(subject)!; + entry.blocked = true; + entry.failures = 8; + + outbox.setDestroyCommit(subject, commit); + + expect(entry.blocked).toBe(false); + expect(entry.failures).toBe(0); + expect(outbox.nextDueAt()).toBe(0); + }); + + it('persists and rehydrates the signed destroy under the agent namespace', async () => { + const subject = 'did:ad:persisted'; + const { commit, agentDID } = await signedDestroy(subject); + + const outbox = new LocalOutbox(); + outbox.rebind(agentDID); + outbox.setDestroyCommit(subject, commit); + outbox.flush(); + + const reloaded = new LocalOutbox(); + reloaded.rebind(agentDID); + const restored = reloaded.getEntry(subject)?.signedDestroy; + expect(restored).toBeDefined(); + expect(restored?.destroy).toBe(true); + expect(restored?.subject).toBe(subject); + expect(restored?.signature).toBe(commit.signature); + expect(restored?.signer).toBe(agentDID); + expect(restored?.createdAt).toBe(commit.createdAt); + + // Another identity never sees it. + const other = new LocalOutbox(); + other.rebind('did:ad:agent:someoneElse'); + expect(other.hasPending(subject)).toBe(false); + }); + + it('discard forgets every queued write for the subject', async () => { + const outbox = new LocalOutbox(); + const subject = 'did:ad:discarded'; + const { commit } = await signedDestroy(subject); + outbox.setGenesisCommit(subject, { ...commit, destroy: undefined }); + outbox.markDirty(subject); + + outbox.discard(subject); + + expect(outbox.hasPending(subject)).toBe(false); + expect(outbox.size).toBe(0); + }); + + it('classifies only the "already gone" server answers as settled', () => { + expect( + isSettledDestroyErrorMessage( + 'Destroy commit for did:ad:x was already applied here; refusing replay', + ), + ).toBe(true); + expect( + isSettledDestroyErrorMessage( + "Destroy commit for did:ad:x (created 1) predates the resource's genesis (2); refusing replay", + ), + ).toBe(true); + expect( + isSettledDestroyErrorMessage( + 'Commit for did:ad:x has is_genesis: false, but the resource does not exist yet.', + ), + ).toBe(true); + expect( + isSettledDestroyErrorMessage( + 'No https://atomicdata.dev/properties/write right has been found', + ), + ).toBe(false); + expect(isSettledDestroyErrorMessage('Failed to fetch')).toBe(false); + }); +}); diff --git a/browser/lib/src/local-outbox.ts b/browser/lib/src/local-outbox.ts index 6554a7839..bc2c2ddf0 100644 --- a/browser/lib/src/local-outbox.ts +++ b/browser/lib/src/local-outbox.ts @@ -12,6 +12,11 @@ import { RequestCancelledError } from './error.js'; * a synchronous sign of the genesis commit (the signature *is* the * subject). That envelope is stored here and POSTed verbatim before * any incremental delta sign for the same subject. + * - `signedDestroy`: a delete has no Loro delta to export at drain time β€” + * the signed destroy envelope IS the whole write. `Resource.destroy()` + * signs it eagerly, stores it here, and removes the resource locally; + * the drain POSTs it verbatim (with the same backoff / offline replay + * as any other entry) and drops the entry on ack. * * Drain is idempotent β€” a call that arrives while a pass is in flight * chains ONE follow-up pass (with a fresh entry snapshot) instead of @@ -32,6 +37,12 @@ export interface OutboxEntry { * the subject from a sync sign. Drain POSTs this verbatim before * attempting any incremental Loro-delta sign. Cleared on ack. */ signedGenesis?: Commit; + /** Pre-signed destroy commit. Set by `Resource.destroy()`; the resource + * is removed locally at once and the drain POSTs this verbatim. Cleared + * (with the whole entry) on ack. When `signedGenesis` is ALSO still + * pending the resource never reached the server, so the drain POSTs + * neither and just drops the entry. */ + signedDestroy?: Commit; /** Base64-encoded Loro `VersionVector` of the last version that was * successfully synced to the server, captured when this subject went * dirty WHILE OFFLINE. On reload the Loro doc rehydrates from clientDb @@ -349,6 +360,7 @@ interface PersistedEntry { subject: string; enqueuedAt: number; signedGenesis?: unknown; + signedDestroy?: unknown; baseVersion?: string; } @@ -428,16 +440,16 @@ export class LocalOutbox { /** Clear the dirty bit for a subject. Called by the drain after the * Loro delta has been signed + POSTed + acked. If a `signedGenesis` - * is still pending, the entry stays (use `clearGenesis` to also - * remove that). */ + * or `signedDestroy` is still pending, the entry stays (use + * `clearGenesis` / `clearDestroy` to also remove that). */ clearDirty(subject: string): void { const entry = this.entries.get(subject); if (!entry) return; - if (entry.signedGenesis) { - // Still holding a genesis envelope β€” keep the entry but treat + if (entry.signedGenesis || entry.signedDestroy) { + // Still holding a signed envelope β€” keep the entry but treat // it as "no incremental delta dirty"; the next drain will POST - // the genesis envelope and recheck. + // the envelope and recheck. this.schedulePersist(); this.onChange(); @@ -480,6 +492,53 @@ export class LocalOutbox { this.onChange(); } + /** Stash a pre-signed destroy commit for `subject`. Called by + * `Resource.destroy()` once the envelope is signed; the drain POSTs it + * verbatim. Any offline `baseVersion` is dropped β€” there is no Loro + * delta left to replay for a subject that is going away. Re-arms a + * blocked entry: the delete is a fresh user intent worth attempting. */ + setDestroyCommit(subject: string, commit: Commit): void { + const entry: OutboxEntry = this.entries.get(subject) ?? { + subject, + enqueuedAt: Date.now(), + }; + entry.signedDestroy = commit; + entry.baseVersion = undefined; + + if (entry.blocked) { + entry.blocked = false; + entry.failures = 0; + } + + this.entries.set(subject, entry); + this.schedulePersist(); + this.onChange(); + } + + /** Drop the entry once the destroy has been acked (or found to be moot). + * Unlike `clearGenesis` this removes the WHOLE entry: a destroyed + * subject has no dirty Loro ops or unposted genesis left worth + * syncing. No-op when the entry holds no `signedDestroy`. */ + clearDestroy(subject: string): void { + const entry = this.entries.get(subject); + if (!entry || !entry.signedDestroy) return; + + this.entries.delete(subject); + this.schedulePersist(); + this.onChange(); + } + + /** Forget every queued write for `subject` without POSTing anything. + * For a resource that never reached the server (still `new`) and is + * being discarded: an unposted genesis or stray dirty bit would + * otherwise recreate it on the next drain. */ + discard(subject: string): void { + if (!this.entries.delete(subject)) return; + + this.schedulePersist(); + this.onChange(); + } + /** Record the last-synced Loro version for an offline edit, so a reload * can restore the save cursor before draining (see `baseVersion` on * `OutboxEntry`). Captures only the FIRST offline baseline per dirty @@ -765,6 +824,9 @@ export class LocalOutbox { signedGenesis: e.signedGenesis ? commitToJsonADObject(e.signedGenesis) : undefined, + signedDestroy: e.signedDestroy + ? commitToJsonADObject(e.signedDestroy) + : undefined, baseVersion: e.baseVersion, })); localStorage.setItem(this.activeKey, JSON.stringify(out)); @@ -882,6 +944,18 @@ export class LocalOutbox { } } + let signedDestroy: Commit | undefined; + + if (obj.signedDestroy) { + try { + signedDestroy = parseCommitJSON(JSON.stringify(obj.signedDestroy)); + } catch { + // skip β€” an unparseable destroy envelope can't be re-signed (the + // resource is already gone locally), so the delete is lost; the + // entry stays dirty-only and the drain clears it as a no-op. + } + } + // Backcompat: old persisted entries had `commits: Commit[]`. We // no longer store signed envelopes here (the Loro state is the // source of truth), so the array is discarded β€” the next drain @@ -894,12 +968,42 @@ export class LocalOutbox { subject: obj.subject, enqueuedAt, signedGenesis, + signedDestroy, baseVersion: typeof obj.baseVersion === 'string' ? obj.baseVersion : undefined, }); } } +/** + * Server refusals of a destroy commit that mean "the resource you are + * deleting is not there to delete (any more)". The delete's goal is met, so + * the drain treats a match as an acknowledgement: it drops the entry and + * keeps the local removal, instead of retrying forever or surfacing a + * "dropped stuck commit" toast for a resource the user deliberately removed. + * + * - `Db::apply_commit` (`lib/src/db.rs`): "Destroy commit for {} was already + * applied here; refusing replay" β€” this exact envelope already landed + * (a reconnect re-POST after the ack was lost). + * - `Commit::reject_destroy_older_than_genesis` (`lib/src/commit.rs`): + * "Destroy commit for {} (created {}) predates the resource's genesis + * ({}); refusing replay" β€” the subject was re-created after our delete; + * this destroy is about a resource that no longer exists. + * - `validate_and_build_response` (`lib/src/commit.rs`): "Commit for {} has + * is_genesis: false, but the resource does not exist yet." β€” someone else + * deleted it first. + * + * Anything else (401, transport, unknown) goes through the ordinary + * terminal / blocking / backoff classification like every other entry. + */ +export function isSettledDestroyErrorMessage(message: string): boolean { + return ( + message.includes('was already applied here; refusing replay') || + message.includes("predates the resource's genesis") || + message.includes('is_genesis: false, but the resource does not exist yet') + ); +} + /** Split sorted entries into runs of equal `tierOf` key. Without `tierOf` * every entry is its own run, which keeps the drain strictly sequential. */ export function groupTiers( diff --git a/browser/lib/src/resource.ts b/browser/lib/src/resource.ts index 21bfa9407..fadd56006 100644 --- a/browser/lib/src/resource.ts +++ b/browser/lib/src/resource.ts @@ -2509,9 +2509,34 @@ export class Resource { return !!this.error && isUnauthorized(this.error); } - /** Removes the resource form both the server and locally */ + /** + * Removes the resource both locally and from the server. + * + * The delete is durable the moment this is called: the signed destroy + * commit goes into the store's `LocalOutbox` (localStorage-backed, survives a + * reload) and the resource is removed from the store and tombstoned in + * OPFS right away. The outbox drain POSTs the envelope with the same + * backoff and reconnect replay as any other write. + * + * Resolution semantics mirror {@link save}: + * - **online**: resolves once the server has acknowledged the destroy; + * rejects with the server's error on a real refusal (e.g. no write + * right). The entry then stays queued and is retried / parked by the + * outbox like any other refused write. + * - **offline** (or the POST fails with a transport error): resolves + * with the delete queued; the reconnect drain sends it. + * - a resource that never reached the server (`new`) is dropped locally + * along with any unposted genesis for it β€” nothing is POSTed. + * - a local-only drive materializes the commit locally, as before. + */ public async destroy(agent?: Agent): Promise { - if (this.new) { + if (this.new || this._pendingGenesis) { + // Never synced (a `_new:` placeholder, or a `store.newResource` whose + // genesis is still parked on the resource because `save()` never + // ran): no server-side tombstone needed. Also forget any queued + // genesis, or the next drain would create what we just deleted. + this._pendingGenesis = undefined; + this.store.outbox.discard(this.subject); this.store.removeResource(this.subject); return; @@ -2544,8 +2569,41 @@ export class Resource { return; } - await this.store.postCommit(commit, this.getCommitEndpoint()); + // Queue first, then remove: `hasPendingDestroy` must already answer + // true when the `ResourceRemoved` listeners run, so nothing they + // trigger (a refetch, a SUB push) can bring the resource back. + this.store.outbox.setDestroyCommit(this.subject, commit); this.store.removeResource(this.subject); + + if (!this.store.serverConnected) { + // Queued; the reconnect drain POSTs it. + return; + } + + // Await the drain so a real refusal surfaces to the caller. Read the + // entry BEFORE draining: a terminal refusal removes it from the queue, + // and that must never be mistaken for an acknowledgement. + const entry = this.store.outbox.getEntry(this.subject); + await this.store.syncDirtyResources(); + + if (entry?.lastAttemptFailure) { + const cause = entry.lastAttemptFailure.cause; + + if (isNetworkError(cause)) { + // Transport, not refusal: the delete stays queued for reconnect. + this.store.setServerConnected(false); + + return; + } + + throw cause; + } + + if (this.store.hasPendingDestroy(this.subject)) { + throw new Error( + 'Delete is still queued; the server has not acknowledged it.', + ); + } } /** Appends a Resource to a ResourceArray */ @@ -3853,39 +3911,6 @@ export class Resource { }); } - /** Resolves the `/commit` endpoint for this resource. */ - private getCommitEndpoint(): string { - const serverUrl = this.store.getServerUrl(); - - if (!serverUrl || serverUrl === 'null') { - console.warn( - `Resource ${this.subject} has an invalid server URL: ${serverUrl}. Falling back to origin.`, - ); - } - - const base = !serverUrl || serverUrl === 'null' ? '' : serverUrl; - const fallbackBase = base || window.location.origin; - - if ( - this.subject.startsWith('did:') || - this.subject.startsWith('internal:') - ) { - return new URL('/commit', fallbackBase).toString(); - } - - try { - const url = new URL(this.subject); - - if (url.origin && url.origin !== 'null') { - return url.origin + `/commit`; - } - } catch { - // ignore - } - - return new URL('/commit', fallbackBase).toString(); - } - private isParentNew() { const parentSubject = this.get(core.properties.parent) as string; diff --git a/browser/lib/src/store.ts b/browser/lib/src/store.ts index 4a6a39a63..8de613a53 100644 --- a/browser/lib/src/store.ts +++ b/browser/lib/src/store.ts @@ -80,6 +80,7 @@ import { DrivePresenceManager } from './presence.js'; import { perfMark, perfSpan } from './perf-trace.js'; import { LocalOutbox, + isSettledDestroyErrorMessage, isTerminalCommitError, isUnrecoverableCommitError, type OutboxEntry, @@ -1197,6 +1198,12 @@ export class Store { /** * Drain one subject's outbox entry under sign-at-drain. Steps: * + * 0. If a `signedDestroy` envelope is queued (`Resource.destroy()` + * while the POST could not be made or acked), POST it verbatim and + * drop the entry β€” there is no Loro delta to sign for a subject + * that is going away, so steps 1-2 are skipped. If the entry ALSO + * still holds an unposted `signedGenesis`, the resource never + * existed on the server: nothing is POSTed and the entry is dropped. * 1. If a `signedGenesis` envelope is queued (DID-derived subject * whose POST hadn't acked yet), POST it. Server idempotently * applies. On success: clear genesis, run the wasNew pipeline @@ -1238,6 +1245,17 @@ export class Store { const endpoint = new URL('/commit', this.serverUrl).toString(); + // Step 0: a queued destroy supersedes everything else for this subject. + // Handled BEFORE the cold-drain `getResource` below, which would refetch + // the (server-side still existing) resource and resurrect it locally. + if (entry.signedDestroy) { + await this.drainDestroy(subject, entry.signedDestroy, endpoint, { + neverSynced: !!entry.signedGenesis, + }); + + return; + } + // Step 1: POST the pre-signed genesis if present. The genesis // commit's `loroUpdate` was captured in `signChanges` at sign // time; `_loroVersionAtLastSave` was advanced THERE to the same @@ -1520,6 +1538,58 @@ export class Store { this.emitSyncStatus(); }; + /** + * Drain a queued destroy (step 0 of {@link drainOutboxSubject}). + * + * `neverSynced`: the entry still carries an unposted `signedGenesis`, so + * the resource was created AND deleted before either write reached the + * server. A destroy of a subject the server has never seen would only be + * refused, and a never-synced resource needs no tombstone β€” POST neither, + * drop the entry, make sure the local copy is gone. + * + * A server answer that says the resource is already gone (see + * {@link isSettledDestroyErrorMessage}) counts as an ack: the goal of the + * delete is met. Every other failure is rethrown so the outbox applies + * its usual backoff / terminal / blocking classification and + * `Resource.destroy()` can surface it to the caller. + */ + private async drainDestroy( + subject: string, + destroy: Commit, + endpoint: string, + opts: { neverSynced: boolean }, + ): Promise { + if (!opts.neverSynced) { + // Registered before the POST like every own commit: should the server + // echo anything for this subject before the ack, it is ours. + if (destroy.signature) { + this.resources + .get(subject) + ?.appliedCommitSignatures.add(destroy.signature); + } + + try { + await this.postCommit(destroy, endpoint); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + + if (!isSettledDestroyErrorMessage(msg)) throw e; + + console.debug( + `[Outbox] destroy of ${subject} already settled on the server: ${msg}`, + ); + } + } + + // Drops the whole entry: nothing else about this subject is worth + // syncing. Then make sure the resource is gone locally too β€” `destroy()` + // already removed it optimistically, but a cold drain after a reload + // (or a fetch that raced the delete) may have brought it back. + this.outbox.clearDestroy(subject); + this.removeResource(subject); + this.emitSyncStatus(); + } + /** * Compute the drive sync state: a hash summarizing all resources' Loro * version vectors, plus the individual VV data for diff computation. @@ -1915,6 +1985,13 @@ export class Store { public applyIncoming( change: IncomingChange, ): 'applied' | 'deduped' | 'invalid' { + // A locally-destroyed resource whose destroy commit has not been acked + // yet must not come back: the server still holds it until the drain + // POSTs the envelope, so a SUB push, a reconnect SYNC or a fetch that + // raced the delete can all still deliver its state. Treat that state as + // already superseded β€” the pending destroy wins. + if (this.hasPendingDestroy(change.subject)) return 'deduped'; + // Resource-direct path: caller is the authoritative producer. if (change.resource) { if ( @@ -3436,6 +3513,13 @@ export class Store { parsed: Record, snapshot?: Uint8Array, ): boolean { + // `removeResource` tombstoned this subject in OPFS, but the worker may + // not have processed that yet when a read races the delete. Whatever is + // still on disk is the pre-delete state; do not bring it back into + // memory. Report "handled" so the caller does not fall back to a server + // fetch that would resurrect it either. + if (this.hasPendingDestroy(subject)) return true; + const existing = this.getResolved(subject); // Don't overwrite a resource that has a Loro snapshot with one that doesn't. @@ -4607,6 +4691,17 @@ export class Store { return url.toString(); } + /** True while `Resource.destroy()` has removed `subject` locally but the + * signed destroy commit is still queued in the outbox (offline, or the + * POST has not acked yet). Incoming state for such a subject is stale by + * definition and must not resurrect it. */ + public hasPendingDestroy(subjectRaw: string): boolean { + const subject = this.normalizeSubject(subjectRaw); + + return !!this.outbox.getEntry(this.aliases.get(subject) ?? subject) + ?.signedDestroy; + } + /** Removes resource from this store, does not delete it from the server, use `resource.destroy()` to delete it from the server. */ public removeResource(subjectRaw: string, shouldNotify = true): void { const resolved = this.resolveSubject(subjectRaw); From a8b1f315cf2805bcaa2787014ddccab21a81675a Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 05:56:25 +0200 Subject: [PATCH 012/110] fix(lib): apply destroy removal in the same transaction as the commit envelope (#1544) A signed destroy now removes the resource, its Loro snapshot, index and search rows and its cascade-deleted children in the same redb write that stores the envelope and commit row, with tombstones recorded only after that write lands. Malformed destroy commits return an error instead of panicking the request. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KxExJcGw13DnHknGTMAhXY --- CHANGELOG.md | 8 +++++ TESTING_COVERAGE.md | 1 + lib/src/db.rs | 72 +++++++++++++++++++++++++++++----------- lib/src/db/test.rs | 80 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 142 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index edd5df417..b56ea4df8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,14 @@ See [STATUS.md](server/STATUS.md) to learn more about which features will remain ## UNRELEASED +- `atomic_lib`: a signed destroy commit now removes the resource (and its + cascade-deleted children, Loro snapshot, index and search rows) in the same + redb transaction that stores its envelope and commit row. `Db::apply_commit` + used to call `remove_resource`, which applied a transaction of its own, so a + crash between the two left a deleted resource with no signed destroy for + `SYNC_DIFF.removeCommits` to carry; tombstones are now recorded once that + single transaction has landed. The destroy branch also returns an error on a + malformed commit instead of panicking the request. - Security hygiene, from the September 2026 audit (`planning/security-audit-2026-09.md` D, C16, F): - CORS: any origin may still read (Atomic is a headless CMS), but diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index 4b06283e0..a72879e6b 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -785,6 +785,7 @@ No automated end-to-end coverage: uploaded-file conversion through the full UI a |---|---|---| | `LoroDoc` values are not KV-index keys | protocol | `lib/src/values.rs::loro_doc_is_not_indexed` | | Content commits are not stored; genesis/ACL/destroy are | protocol | `lib/src/db/test.rs::content_commits_are_not_stored` | +| Signed destroy removes the resource, keeps its envelope and tombstones the subject in one apply | protocol | `lib/src/db/test.rs::destroy_commit_removes_resource_and_keeps_envelope_atomically` | | Sequential saves do not chain `previousCommit`; commit DIDs are not store resources | glue | `browser/lib/src/commit.test.ts` | ## Personal drive identity diff --git a/lib/src/db.rs b/lib/src/db.rs index 185ef97b4..ba32e4780 100644 --- a/lib/src/db.rs +++ b/lib/src/db.rs @@ -3973,27 +3973,43 @@ impl Storelike for Db { // signed it, independent of the commit rows above. crate::envelopes::record_ops(store, &commit_response, &mut transaction)?; + // `pure_id()` of every subject a destroy removes below (the resource + // and its cascade-deleted children). Tombstoned once the transaction + // has landed; empty for anything but a destroy. + let mut removed: Vec = Vec::new(); + match (&commit_response.resource_old, &commit_response.resource_new) { (None, None) if !commit_response.commit.destroy.unwrap_or(false) => { return Err("Neither an old nor a new resource is returned from the commit - something went wrong.".into()); } (None, None) => {} - (Some(_old), None) => { + (Some(old), None) => { let normalized_commit_subject = - self.normalize_subject(&commit_response.commit.subject.clone()); - assert_eq!( - _old.get_subject().to_string(), - normalized_commit_subject.to_string() - ); - assert!(&commit_response - .commit - .destroy - .expect("Resource was removed but `commit.destroy` was not set!")); + self.normalize_subject(&commit_response.commit.subject); + if old.get_subject().as_str() != normalized_commit_subject.as_str() { + return Err(format!( + "Commit for {} removed a different resource ({}) - refusing to apply.", + normalized_commit_subject, + old.get_subject() + ) + .into()); + } + if !commit_response.commit.destroy.unwrap_or(false) { + return Err(format!( + "Resource {} was removed but `commit.destroy` was not set.", + normalized_commit_subject + ) + .into()); + } let subject: Subject = commit_response.commit.subject.clone(); - // `remove_resource` records the tombstone; the signed destroy - // is the envelope row `record_ops` queued above, which is what - // `SYNC_DIFF.removeCommits` carries. - self.remove_resource(&subject).await?; + // Queue the removal (resource, Loro snapshot, index and search + // rows, cascaded children) into the same transaction as the + // envelope row `record_ops` queued above, so the signed + // destroy β€” what `SYNC_DIFF.removeCommits` carries β€” lands + // with the deletion it signs, or not at all. Tombstones are + // recorded after the apply below. + self.recursive_remove(&subject, &mut transaction, &mut removed, None) + .await?; } _ => {} }; @@ -4013,12 +4029,22 @@ impl Storelike for Db { } } + // A destroy's `recursive_remove` above already queued the index and + // search removals for every stored atom of the subject (and its + // children); queuing them again here would only duplicate the same + // deletes and re-run the watched-query checks. + let removal_queued = !removed.is_empty(); + if opts.update_index { if let Some(old) = &commit_response.resource_old { - for atom in &commit_response.remove_atoms { - store - .remove_atom_from_index(atom, old, &mut transaction) - .map_err(|e| format!("Error removing atom from index: {e} Atom: {e}"))? + if !removal_queued { + for atom in &commit_response.remove_atoms { + store + .remove_atom_from_index(atom, old, &mut transaction) + .map_err(|e| { + format!("Error removing atom from index: {e} Atom: {e}") + })? + } } } if let Some(new) = &commit_response.resource_new { @@ -4035,7 +4061,7 @@ impl Storelike for Db { )?; crate::search::index_resource(store, new, &mut transaction)?; } - if commit_response.resource_new.is_none() { + if commit_response.resource_new.is_none() && !removal_queued { if let Some(old) = &commit_response.resource_old { crate::search::unindex_subject( store, @@ -4064,6 +4090,14 @@ impl Storelike for Db { store.flush()?; } + // Tombstone every subject the destroy removed, so bulk sync (Iroh / + // WS `SYNC`) does not resurrect them from a peer that still holds a + // stale copy. Only after the transaction above succeeded: a tombstone + // for a resource that is still present would suppress it forever. + for s in &removed { + crate::sync::tombstones::record_tombstone(store, s); + } + // Notify subscribers let subject = commit_response.commit.subject.without_params(); let is_destroy = commit_response.commit.destroy.unwrap_or(false); diff --git a/lib/src/db/test.rs b/lib/src/db/test.rs index d30220747..18c223a79 100644 --- a/lib/src/db/test.rs +++ b/lib/src/db/test.rs @@ -1514,6 +1514,86 @@ async fn loro_non_property_container_survives_commit_roundtrip() { ); } +/// A signed destroy commit removes the resource and stores its envelope in +/// one transaction, then tombstones the subject. Before, `apply_commit` +/// called `remove_resource`, which applied its own transaction: the resource +/// was gone before the envelope row landed, and a crash in between left a +/// deletion with no signed destroy for `SYNC_DIFF.removeCommits` to carry. +#[tokio::test] +#[timeout(120000)] +async fn destroy_commit_removes_resource_and_keeps_envelope_atomically() { + let store = Db::init_temp("destroy_commit_envelope").await.unwrap(); + + let mut resource = crate::Resource::new("did:ad:placeholder".into()); + resource + .set(urls::NAME.into(), Value::String("doomed".into()), &store) + .await + .unwrap(); + let genesis = resource.save_as_genesis(&store).await.unwrap(); + let subject = genesis.resource_new.unwrap().get_subject().clone(); + let pure_id = subject.pure_id(); + assert!( + store + .kv + .get(Tree::LoroSnapshots, pure_id.as_bytes()) + .unwrap() + .is_some(), + "test premise: a saved resource has a Loro snapshot" + ); + assert!( + !crate::sync::tombstones::is_tombstoned(&store, &pure_id), + "test premise: a live resource is not tombstoned" + ); + + let mut resource = store.get_resource(&subject).await.unwrap(); + let destroy = resource.destroy(&store).await.unwrap(); + let signature = destroy + .commit + .signature + .clone() + .expect("destroy via the default agent is a signed commit"); + assert!(destroy.resource_new.is_none()); + + // (a) the resource, its Loro snapshot and its stored propvals are gone. + assert!( + store.get_resource(&subject).await.is_err(), + "destroyed resource must not resolve" + ); + assert!( + store + .kv + .get(Tree::Resources, pure_id.as_bytes()) + .unwrap() + .is_none(), + "destroyed resource row must be deleted" + ); + assert!( + store + .kv + .get(Tree::LoroSnapshots, pure_id.as_bytes()) + .unwrap() + .is_none(), + "Loro snapshot must be deleted with the resource" + ); + + // (b) the signed destroy envelope is the subject's latest envelope. + let envelope = crate::envelopes::latest_envelope(&store, &pure_id) + .expect("the destroy envelope must be stored for the destroyed subject"); + assert_eq!(envelope.signature, signature); + assert!(envelope.is_destroy(), "latest envelope must be the destroy"); + + // (c) the subject is tombstoned, and the tombstone resolves to the + // signed destroy `SYNC_DIFF.removeCommits` forwards. + assert!( + crate::sync::tombstones::is_tombstoned(&store, &pure_id), + "destroyed subject must be tombstoned" + ); + assert!( + crate::sync::tombstones::destroy_envelope(&store, &pure_id).is_some(), + "tombstone must resolve to the signed destroy envelope" + ); +} + /// A deleted resource must not leave its Loro snapshot orphaned in /// `Tree::LoroSnapshots`, and the subject must be tombstoned so bulk sync /// does not resurrect it. From 7f2c8428f3c7d9786f2972dbb7f4b6a30c642f70 Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 06:11:05 +0200 Subject: [PATCH 013/110] fix(lib): emit DbEvent::Destroyed for cascaded children only after the removal is applied (#1561) recursive_remove no longer announces anything; it collects each removed subject with its drive. remove_resource and the destroy branch of apply_commit announce them through Db::emit_destroyed once the removal has actually landed, so a subscriber never learns of a deletion that could still roll back. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KxExJcGw13DnHknGTMAhXY --- CHANGELOG.md | 12 +++++ TESTING_COVERAGE.md | 1 + lib/src/db.rs | 112 ++++++++++++++++++++++++++++++-------------- lib/src/db/test.rs | 90 +++++++++++++++++++++++++++++++++++ 4 files changed, 181 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b56ea4df8..611da7796 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -54,6 +54,18 @@ See [STATUS.md](server/STATUS.md) to learn more about which features will remain - `sync::peer`: the live peer map recovers from a poisoned lock instead of unwrapping it, so one panic no longer stops live broadcast for the rest of the process. +- `atomic_lib`: `DbEvent::Destroyed` for a cascade-deleted child is sent only + after the removal has been applied. `Db::recursive_remove` announced each + child while it was still queueing the deletes into a transaction the caller + had yet to apply, so a listener (`atomic-server`'s `CommitMonitor`, which + fans removals out to WebSocket subscribers; a peer transport) heard of a + deletion that could still fail or roll back, and heard of the children + before their parent. The callers (`remove_resource`, the destroy branch of + `apply_commit`) now announce every removed subject once the transaction has + landed; the destroyed subject itself is still announced exactly once, wrapped + in its signed destroy commit. `remove_resource` now also announces the + subject it was called for, which it never did. Follow-up to #1544. + - CI: the `:develop` docker image is published by its own job instead of a step tacked onto the end of the CI job. As a step it inherited whatever the CI step had already spent (a wedged Dagger engine on the runner burned the diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index a72879e6b..51927706e 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -492,6 +492,7 @@ Two things worth knowing about the runners: | RBSR reconciliation, drive hashing | `lib/src/sync/rbsr.rs`, `tests.rs` | | RBSR finds a remote-only subject sorting below every local one | `lib/src/sync/rbsr.rs` **and** `browser/lib/src/rbsr.test.ts` (regression, see below) | | Remote update merge, drive-spoof rejection, tombstones | `lib/src/sync/ws_apply.rs`, `tombstones.rs` | +| `DbEvent::Destroyed` for a removed resource and its cascade-deleted children arrives only after the removal is applied (the store no longer holds them when a listener hears), each subject announced exactly once | `lib/src/db/test.rs` (`destroyed_events_follow_the_applied_removal`) | | Pairing envelope encode/decode | `browser/lib/src/pairing.test.ts` | ### Cross-process β€” covered since 2026-07 diff --git a/lib/src/db.rs b/lib/src/db.rs index ba32e4780..c977bf4a7 100644 --- a/lib/src/db.rs +++ b/lib/src/db.rs @@ -112,6 +112,16 @@ pub struct EphemeralEvent { pub from_peer: String, } +/// A subject `Db::recursive_remove` queued for deletion, kept until its +/// transaction has been applied so the caller can tombstone and announce it. +#[derive(Debug, Clone)] +struct RemovedSubject { + /// The subject without query params, as [`DbEvent::Destroyed`] names it. + subject: Subject, + /// See [`DbEvent::Destroyed::drive`]: read before the resource was gone. + drive: Option, +} + /// Event emitted when a resource is created, updated, or deleted. #[derive(Debug, Clone)] pub enum DbEvent { @@ -2259,6 +2269,27 @@ impl Db { self.on_commit = Some(Arc::new(on_commit)); } + /// Removes `subject` and its children in one transaction and tombstones + /// them, returning what was removed so the caller can announce it once it + /// has decided how: `remove_resource` sends a plain `DbEvent::Destroyed` + /// for each, `apply_commit` wraps the destroyed subject in its signed + /// destroy commit and announces only the children this way. + async fn remove_applied(&self, subject: &Subject) -> AtomicResult> { + let mut transaction = Transaction::new(); + let mut removed = Vec::new(); + self.recursive_remove(subject, &mut transaction, &mut removed, None) + .await?; + self.apply_transaction(&mut transaction)?; + // Tombstone every removed subject so bulk sync (Iroh / WS `SYNC`) + // does not resurrect them from a peer that still holds a stale copy. + // Only after the apply succeeded: a tombstone for a resource that is + // still present would suppress it forever. + for r in &removed { + crate::sync::tombstones::record_tombstone(self, &r.subject.pure_id()); + } + Ok(removed) + } + /// Subscribe to all DB events (changes, deletions). pub fn subscribe_events(&self) -> tokio::sync::broadcast::Receiver { self.db_events.subscribe() @@ -3538,8 +3569,9 @@ impl Db { } /// Recursively removes a resource and its children from the database. - /// `removed` collects the `pure_id()` of every deleted subject so the - /// caller can tombstone them after the transaction is applied. + /// `removed` collects every deleted subject so the caller can tombstone + /// and announce them after the transaction is applied β€” nothing is sent on + /// `db_events` from here, since the removal has not happened yet. /// `inherited_drive` is the drive of the resource whose cascade brought us /// here. A child that carries no `drive` of its own β€” anything created /// before the server stamped it β€” is still in its parent's drive, and the @@ -3548,7 +3580,7 @@ impl Db { &self, subject: &Subject, transaction: &mut Transaction, - removed: &mut Vec, + removed: &mut Vec, inherited_drive: Option, ) -> AtomicResult<()> { // Key by `pure_id()` β€” that is how resources and Loro snapshots are @@ -3563,22 +3595,15 @@ impl Db { // the snapshot is orphaned in `Tree::LoroSnapshots` and leaks // forever β€” only the WS/Iroh DESTROY path cleaned it before. transaction.push(Operation::remove_loro_snapshot(&subject_str)); - removed.push(subject_str.clone()); + // Read the drive now, while the resource still exists: a listener + // reacting to the removal cannot look it up any more. let drive = resource.get_drive().or(inherited_drive); + removed.push(RemovedSubject { + subject: subject.without_params(), + drive: drive.clone(), + }); let mut children = resource.get_children(self).await?; for child in children.iter_mut() { - // Notify subscribers so clients evict the cascade-deleted - // child from their cache. The signed destroy commit only - // fires DbEvent::Destroyed for the top-level subject; without - // this, children remain in WASM-DB / store and the UI keeps - // rendering them. - let _ = self.db_events.send(DbEvent::Destroyed { - subject: child.get_subject().without_params(), - drive: child.get_drive().or_else(|| drive.clone()), - source_id: None, - from_commit: false, - commit_json: None, - }); // Because the function is async we need to box it to use recursion. Box::pin(self.recursive_remove( child.get_subject(), @@ -3603,6 +3628,25 @@ impl Db { Ok(()) } + /// Announces subjects `recursive_remove` collected, once the transaction + /// holding their removal has been applied. Any earlier and a listener + /// (the WS fan-out in `atomic-server`'s `CommitMonitor`, a peer + /// transport) would tell clients about a deletion that can still fail or + /// roll back, and hear of a child before the parent whose destroy caused + /// it. Removals no commit authorises carry no `commit_json`, so they are + /// not propagated live to peers; see [`DbEvent::Destroyed`]. + fn emit_destroyed<'a>(&self, removed: impl IntoIterator) { + for r in removed { + let _ = self.db_events.send(DbEvent::Destroyed { + subject: r.subject.clone(), + drive: r.drive.clone(), + source_id: None, + from_commit: false, + commit_json: None, + }); + } + } + fn is_endpoint(&self, url: &url::Url) -> bool { self.endpoints.iter().any(|e| e.path == url.path()) } @@ -3973,10 +4017,10 @@ impl Storelike for Db { // signed it, independent of the commit rows above. crate::envelopes::record_ops(store, &commit_response, &mut transaction)?; - // `pure_id()` of every subject a destroy removes below (the resource - // and its cascade-deleted children). Tombstoned once the transaction - // has landed; empty for anything but a destroy. - let mut removed: Vec = Vec::new(); + // Every subject a destroy removes below (the resource and its + // cascade-deleted children). Tombstoned and announced once the + // transaction has landed; empty for anything but a destroy. + let mut removed: Vec = Vec::new(); match (&commit_response.resource_old, &commit_response.resource_new) { (None, None) if !commit_response.commit.destroy.unwrap_or(false) => { @@ -4006,8 +4050,8 @@ impl Storelike for Db { // rows, cascaded children) into the same transaction as the // envelope row `record_ops` queued above, so the signed // destroy β€” what `SYNC_DIFF.removeCommits` carries β€” lands - // with the deletion it signs, or not at all. Tombstones are - // recorded after the apply below. + // with the deletion it signs, or not at all. Tombstones and + // the children's `Destroyed` events follow the apply below. self.recursive_remove(&subject, &mut transaction, &mut removed, None) .await?; } @@ -4094,10 +4138,16 @@ impl Storelike for Db { // WS `SYNC`) does not resurrect them from a peer that still holds a // stale copy. Only after the transaction above succeeded: a tombstone // for a resource that is still present would suppress it forever. - for s in &removed { - crate::sync::tombstones::record_tombstone(store, s); + for r in &removed { + crate::sync::tombstones::record_tombstone(store, &r.subject.pure_id()); } + // Announce the cascade-deleted children now that the removal has + // landed. The destroyed subject itself is announced below, wrapped in + // the signed destroy commit, so it is left out here. + let top_level = commit_response.commit.subject.pure_id(); + store.emit_destroyed(removed.iter().filter(|r| r.subject.pure_id() != top_level)); + // Notify subscribers let subject = commit_response.commit.subject.without_params(); let is_destroy = commit_response.commit.destroy.unwrap_or(false); @@ -4647,16 +4697,10 @@ impl Storelike for Db { #[instrument(skip_all)] async fn remove_resource(&self, subject: &Subject) -> AtomicResult<()> { - let mut transaction = Transaction::new(); - let mut removed = Vec::new(); - self.recursive_remove(subject, &mut transaction, &mut removed, None) - .await?; - self.apply_transaction(&mut transaction)?; - // Tombstone every removed subject so bulk sync (Iroh / WS `SYNC`) - // does not resurrect them from a peer that still holds a stale copy. - for s in &removed { - crate::sync::tombstones::record_tombstone(self, s); - } + let removed = self.remove_applied(subject).await?; + // Only now, with the removal on disk: a listener that hears of it + // earlier evicts a resource the store may still hold. + self.emit_destroyed(&removed); // TODO: deletion sync β€” should create a signed destroy commit // and push it through the normal commit pipeline, not a raw DESTROY frame. Ok(()) diff --git a/lib/src/db/test.rs b/lib/src/db/test.rs index 18c223a79..7a19a86da 100644 --- a/lib/src/db/test.rs +++ b/lib/src/db/test.rs @@ -1967,6 +1967,96 @@ async fn a_cascade_deleted_child_names_its_drive() { ); } +/// `DbEvent::Destroyed` is sent only once the removal has been applied. +/// +/// `recursive_remove` used to announce each cascade-deleted child while it +/// was still queueing operations into a transaction the caller had yet to +/// apply. A listener β€” the WS fan-out in `atomic-server`'s `CommitMonitor`, a +/// peer transport β€” heard of a deletion that could still fail or roll back, +/// and heard of the children before the parent whose removal caused it. Now +/// the caller announces every removed subject, the parent included, after the +/// transaction has landed. +/// +/// The listener runs on its own worker thread and checks the store the moment +/// each event arrives. With the old ordering there is a wide window in which +/// that read still finds the resource: the children have yet to be queried, +/// unindexed and applied. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[timeout(120000)] +async fn destroyed_events_follow_the_applied_removal() { + use crate::DbEvent; + + let store = Db::init_temp("destroyed_after_apply").await.unwrap(); + let drive = store.create_drive("destroyed-after-apply").await.unwrap(); + let parent = store + .create_resource(urls::FOLDER, &drive, "parent", None) + .await + .unwrap(); + let mut expected: Vec = vec![parent.clone()]; + for name in ["child-1", "child-2"] { + let child = store + .create_resource(urls::FOLDER, &parent, name, None) + .await + .unwrap(); + expected.push(child); + } + let base = store.get_base_domain(); + let to_subject = |s: &str| Subject::from_raw(s, base.as_deref()).without_params(); + expected.sort(); + + // Subscribe before the removal, and observe concurrently: what matters is + // the state of the store at the moment each event can be seen. + let mut events = store.subscribe_events(); + let observer_store = store.clone(); + let observer = tokio::spawn(async move { + let mut seen: Vec<(Subject, bool)> = Vec::new(); + while seen.len() < 3 { + match events.recv().await.unwrap() { + DbEvent::Destroyed { subject, .. } => { + let still_present = observer_store.get_resource(&subject).await.is_ok(); + seen.push((subject, still_present)); + } + _ => continue, + } + } + (seen, events) + }); + + store.remove_resource(&to_subject(&parent)).await.unwrap(); + + let (seen, mut events) = tokio::time::timeout(std::time::Duration::from_secs(30), observer) + .await + .expect("the removal never announced parent and both children") + .unwrap(); + + let mut announced: Vec = seen.iter().map(|(s, _)| s.pure_id()).collect(); + announced.sort(); + assert_eq!( + announced, expected, + "parent and both children must each be announced exactly once" + ); + for (subject, still_present) in &seen { + assert!( + !still_present, + "{subject} was announced as destroyed while the store still held it" + ); + assert!( + store.get_resource(subject).await.is_err(), + "{subject} must be gone once its removal is announced" + ); + } + + // `remove_resource` has returned, so anything it sent is already in the + // channel: there must be no fourth removal β€” the parent is not announced + // twice. + while let Ok(event) = events.try_recv() { + assert!( + !matches!(event, DbEvent::Destroyed { .. }), + "unexpected extra removal event: {event:?}" + ); + } +} + /// A resource created under a drive must be findable by that drive. /// /// `drive` is stamped onto the resource by the server (`commit.rs`, from the From 33baaa810988403200e2f06f83d34590d1cf815f Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 06:11:39 +0200 Subject: [PATCH 014/110] fix(lib): classify commit refusals by error code, not message text (#1557) "Commits cannot be edited" now carries error_code::IMMUTABLE_COMMIT (10) on both transports, and the client decides whether a terminal drop is benign from the code, falling back to the legacy message match only when a response carries no recognised code. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KxExJcGw13DnHknGTMAhXY --- CHANGELOG.md | 5 ++ TESTING_COVERAGE.md | 9 ++ browser/CHANGELOG.md | 1 + browser/lib/src/index.ts | 1 + browser/lib/src/local-outbox.test.ts | 82 ++++++++++++++++++ browser/lib/src/local-outbox.ts | 38 +++++++- browser/lib/src/save-acknowledgement.test.ts | 91 +++++++++++++++++++- browser/lib/src/store.ts | 23 ++--- browser/lib/src/ws-v2.ts | 4 + docs/src/websockets.md | 3 +- lib/src/sync/outbox.rs | 4 +- lib/src/sync/protocol.rs | 17 ++++ 12 files changed, 259 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 611da7796..fb0149677 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,11 @@ See [STATUS.md](server/STATUS.md) to learn more about which features will remain in its signed destroy commit. `remove_resource` now also announces the subject it was called for, which it never did. Follow-up to #1544. +- `classify_commit_error` classifies "Commits cannot be edited." (a write whose + subject is itself a Commit) as the new `IMMUTABLE_COMMIT` (10) error code, on + the WS `ERROR` frame and the HTTP `/commit` error body's `errorCode` alike. + It was going out as `UNKNOWN`, leaving clients to match the message text to + know the refusal is terminal. The Rust outbox drops such entries as terminal. - CI: the `:develop` docker image is published by its own job instead of a step tacked onto the end of the CI job. As a step it inherited whatever the CI step had already spent (a wedged Dagger engine on the runner burned the diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index 51927706e..f409ec8ad 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -1562,6 +1562,15 @@ setup currently fails opening OPFS before it can create its dev drive. structured `SYNC_REJECTED` classification are covered. - `store-commit-fallback.test.ts`: a WebSocket enrollment refusal is not duplicated over HTTP; a transport failure still falls back. +- `local-outbox.test.ts` ("classification is code-first") and + `save-acknowledgement.test.ts` ("terminal drops are classified by error + code"): a recognized `AtomicError.code` (`GENESIS_COLLISION`, + `IMMUTABLE_COMMIT`, ...) decides terminal/benign/blocking regardless of + message wording, including one parsed off an HTTP `/commit` JSON-AD error + body's `errorCode`; a code-less legacy message still classifies; another + recognized code wins over a legacy phrase in the message. Rust: + `protocol::classify_commit_error_matches_known_patterns` covers + `IMMUTABLE_COMMIT`. - Server `errors::admission_error_tests`: enrollment/quota refusals carry a blocking code and HTTP 403 rather than an internal-error response. - Server `tests::content_addressed_image_download`: raw, WebP and AVIF downloads diff --git a/browser/CHANGELOG.md b/browser/CHANGELOG.md index e5ba7d0c5..9bf785042 100644 --- a/browser/CHANGELOG.md +++ b/browser/CHANGELOG.md @@ -7,6 +7,7 @@ This changelog covers all five packages, as they are (for now) updated as a whol - Fix: `@tomic/lib` writes the Loro `datatypes` tags in the export path shared by genesis and drain-time signing, after the user's pending ops are sealed into their own commit. The drain used to tag first, which folded a `set()` edit into the tag write's `atomic:system` commit β€” the canvas undo stack skips that origin, so the first edit of a `json`/reference property after a save could not be undone. - Perf: a cold page load asks the local database (OPFS worker) for all the resources a render pass mounts in one round trip instead of one per `useResource`. `Store` coalesces the render-phase cache misses of a tick into a single `getResourcesWithSnapshots` call (chunked at 200 subjects), and a subject asked for twice while a read is pending or in flight shares that read. 50 mounted resources went from 50 worker messages to 1; the tri-state answer (`true` / `false` / no database to ask) and the "ask again once the database attaches" behaviour are unchanged. - Fix: deleting a resource while offline no longer loses the delete. `Resource.destroy()` now goes through the durable `LocalOutbox` like every other write: the signed destroy commit is queued (`OutboxEntry.signedDestroy`, persisted in localStorage), the resource is removed locally at once, and the drain POSTs the envelope with the usual backoff and reconnect replay. Online, `await destroy()` still resolves after the server acknowledgement and rejects on a real refusal; offline it resolves with the delete queued. A resource created and deleted before either write reached the server is dropped without a POST, and a server answer that the resource is already gone counts as acknowledged instead of retrying forever. While a delete is queued, incoming state for that subject (a subscription push, a fetch that raced the delete, a stale OPFS row) no longer resurrects it. `Store.hasPendingDestroy(subject)` is new; `LocalOutbox` gains `setDestroyCommit`, `clearDestroy` and `discard`. +- Fix: the outbox decides whether a refused commit is terminal (drop it), benign (drop it quietly) or blocking (park it) from the server's structured error code first β€” the `ERROR` frame's `code` over WebSocket, or the `errorCode` property on the HTTP `/commit` error body β€” and only falls back to matching message text when the response carries no recognized code (older servers). Previously the store's "benign drop" check (`is_genesis: true, but the resource already exists`, `Commits cannot be edited`) matched message text only, so a wording change on the server would have turned a silent bookkeeping drop into an error toast per commit, and "Commits cannot be edited" had no code at all. New `ErrorCode.IMMUTABLE_COMMIT` (10) and `isBenignTerminalCommitError(message, code)` in `@tomic/lib`. - Local Thought connections install a folder after one access check and import in the browser without a preview dialog. Opening the folder or a table refreshes automatically, with five-minute refreshes while open, visible sync status, and local-edit preservation. - Fix: form inputs validate before writing. `InputString` and `InputURI` no longer put an invalid value into the resource while showing an error; all inputs share one validate-then-set hook (`useValidatedInput`) and surface the datatype's actual error message (e.g. "Not an integer") instead of a generic "Invalid value". `ResourceField` now accepts and forwards `commit` / `commitDebounceInterval`, and every input honours `commitDebounceInterval` (Markdown, Number, Boolean, Date, Timestamp, Resource, ResourceArray and FilePicker previously fell back to the 100 ms default). A required field whose value loads after the first render is now flagged correctly. - Fix: `Resource.remove()` and `Resource.push()` emit `LocalChange` like `set()` does, so `useValue` / `useArray` re-render when a field is cleared or an item is appended. `useValue`'s setter clears a previously reported validation error when called with `undefined`. `useArray`'s `push` saves through the same debounced scheduler and error handling (`handleValidationError` / `store.notifyError`) as `set`, instead of an immediate `resource.save()` that only logged failures. `StoreContext` no longer defaults to a phantom `Store`: `useStore()` throws when no `` is mounted. diff --git a/browser/lib/src/index.ts b/browser/lib/src/index.ts index d72a4b590..cb4c00f02 100644 --- a/browser/lib/src/index.ts +++ b/browser/lib/src/index.ts @@ -85,6 +85,7 @@ export { isUnrecoverableCommitErrorMessage, isTerminalCommitError, isUnrecoverableCommitError, + isBenignTerminalCommitError, type OutboxEntry, type OutboxDrainContext, } from './local-outbox.js'; diff --git a/browser/lib/src/local-outbox.test.ts b/browser/lib/src/local-outbox.test.ts index 6f8c4562f..ea7e92f3e 100644 --- a/browser/lib/src/local-outbox.test.ts +++ b/browser/lib/src/local-outbox.test.ts @@ -6,6 +6,7 @@ import { isUnrecoverableCommitErrorMessage, isTerminalCommitError, isUnrecoverableCommitError, + isBenignTerminalCommitError, isCommitSubject, drainBackoffMs, groupTiers, @@ -1031,3 +1032,84 @@ describe('LocalOutbox.drain tiers (pipelined COMMITs)', () => { expect(outbox.getEntry('did:ad:s1')?.failures).toBe(1); }); }); + +describe('classification is code-first, message text is only a fallback', () => { + // A server wording change must not turn a terminal refusal into infinite + // retries (or a benign drop into an error toast). When the server sends a + // recognized code, that code decides; the legacy message patterns only + // apply to code-less responses from older servers. + const UNRELATED = 'some unrelated wording the client has never seen'; + const LEGACY_GENESIS = + 'Commit for did:ad:abc has is_genesis: true, but the resource already exists.'; + const LEGACY_IMMUTABLE = 'Commits cannot be edited.'; + + it('GENESIS_COLLISION with an unrelated message is terminal and benign', ({ + expect, + }) => { + expect(isTerminalCommitError(UNRELATED, ErrorCode.GENESIS_COLLISION)).toBe( + true, + ); + expect( + isBenignTerminalCommitError(UNRELATED, ErrorCode.GENESIS_COLLISION), + ).toBe(true); + }); + + it('IMMUTABLE_COMMIT with an unrelated message is terminal and benign', ({ + expect, + }) => { + expect(isTerminalCommitError(UNRELATED, ErrorCode.IMMUTABLE_COMMIT)).toBe( + true, + ); + expect( + isBenignTerminalCommitError(UNRELATED, ErrorCode.IMMUTABLE_COMMIT), + ).toBe(true); + expect( + isUnrecoverableCommitError(UNRELATED, ErrorCode.IMMUTABLE_COMMIT), + ).toBe(false); + }); + + it('a code-less error with the legacy message still classifies', ({ + expect, + }) => { + for (const message of [LEGACY_GENESIS, LEGACY_IMMUTABLE]) { + expect(isTerminalCommitError(message, undefined)).toBe(true); + expect(isBenignTerminalCommitError(message, undefined)).toBe(true); + expect(isTerminalCommitError(message, ErrorCode.UNKNOWN)).toBe(true); + expect(isBenignTerminalCommitError(message, ErrorCode.UNKNOWN)).toBe( + true, + ); + } + }); + + it('another recognized code wins over a legacy phrase in the message', ({ + expect, + }) => { + for (const message of [LEGACY_GENESIS, LEGACY_IMMUTABLE]) { + expect(isTerminalCommitError(message, ErrorCode.UNAUTHORIZED_WRITE)).toBe( + false, + ); + expect( + isBenignTerminalCommitError(message, ErrorCode.UNAUTHORIZED_WRITE), + ).toBe(false); + } + }); + + it('a lost-write terminal code is terminal but not benign', ({ expect }) => { + expect( + isTerminalCommitError(UNRELATED, ErrorCode.MISSING_REQUIRED_PROPERTY), + ).toBe(true); + expect( + isBenignTerminalCommitError( + UNRELATED, + ErrorCode.MISSING_REQUIRED_PROPERTY, + ), + ).toBe(false); + }); + + it('an unrecognized nonzero code (garbled pre-F5 frame) falls back to text', ({ + expect, + }) => { + expect(isBenignTerminalCommitError(LEGACY_GENESIS, 0x4322)).toBe(true); + expect(isBenignTerminalCommitError(UNRELATED, 0x4322)).toBe(false); + }); +}); diff --git a/browser/lib/src/local-outbox.ts b/browser/lib/src/local-outbox.ts index bc2c2ddf0..9a295bbc2 100644 --- a/browser/lib/src/local-outbox.ts +++ b/browser/lib/src/local-outbox.ts @@ -270,6 +270,7 @@ const KNOWN_ERROR_CODES: ReadonlySet = new Set([ ErrorCode.UNAUTHORIZED_WRITE, ErrorCode.MISSING_CLASS, ErrorCode.SYNC_REJECTED, + ErrorCode.IMMUTABLE_COMMIT, ]); /** @@ -284,13 +285,48 @@ export function isTerminalCommitError(message: string, code?: number): boolean { if (code !== undefined && KNOWN_ERROR_CODES.has(code)) { return ( code === ErrorCode.GENESIS_COLLISION || - code === ErrorCode.MISSING_REQUIRED_PROPERTY + code === ErrorCode.MISSING_REQUIRED_PROPERTY || + code === ErrorCode.IMMUTABLE_COMMIT ); } return isTerminalCommitErrorMessage(message); } +/** + * Whether a terminal refusal is bookkeeping rather than a lost write: the + * user's data is already where it should be, so the drop should be logged, + * not surfaced as an error toast. + * + * - A redundant genesis (`GENESIS_COLLISION`): the resource already exists on + * the server; only the never-applied diff in this one commit is gone. These + * arrive in bulk when local state lost its `lastCommit` chain (e.g. after + * switching servers), so a toast per commit is pure noise. + * - A write aimed at a Commit (`IMMUTABLE_COMMIT`): a Commit is whatever was + * signed; no local edit to it could ever have applied. + * + * Same code-first / string-fallback shape as {@link isTerminalCommitError}: + * a recognized `code` is authoritative (so a server wording change cannot + * turn a benign drop into a scary one, or vice versa), and only an absent or + * unrecognized code falls back to the legacy message text. + */ +export function isBenignTerminalCommitError( + message: string, + code?: number, +): boolean { + if (code !== undefined && KNOWN_ERROR_CODES.has(code)) { + return ( + code === ErrorCode.GENESIS_COLLISION || + code === ErrorCode.IMMUTABLE_COMMIT + ); + } + + return ( + message.includes('is_genesis: true, but the resource already exists') || + message.includes('Commits cannot be edited') + ); +} + /** Blocking-error check the outbox should actually call β€” see * {@link isTerminalCommitError} for the code-first/string-fallback shape. */ export function isUnrecoverableCommitError( diff --git a/browser/lib/src/save-acknowledgement.test.ts b/browser/lib/src/save-acknowledgement.test.ts index c96093892..d1b0c8add 100644 --- a/browser/lib/src/save-acknowledgement.test.ts +++ b/browser/lib/src/save-acknowledgement.test.ts @@ -1,8 +1,9 @@ import { afterEach, describe, expect, it, vi } from 'vitest'; import { core } from './ontologies/core.js'; import { testStore } from './test-store.js'; -import { RequestCancelledError } from './error.js'; +import { AtomicError, ErrorType, RequestCancelledError } from './error.js'; import { BLOCK_AFTER_FAILURES } from './local-outbox.js'; +import { ErrorCode } from './ws-v2.js'; afterEach(() => vi.restoreAllMocks()); @@ -162,3 +163,91 @@ describe('explicit save acknowledgement', () => { store.setServerConnected(false); }); }); + +describe('terminal drops are classified by error code, not message text', () => { + // The server's structured `code` (WS `ERROR` frame or the `errorCode` + // property on the HTTP `/commit` error body) decides whether a refusal is + // terminal and whether the drop is benign. Message matching only covers + // code-less responses from older servers, so a wording change on the server + // cannot turn a terminal refusal into infinite retries. + const LEGACY_GENESIS = + 'Commit for did:ad:abc has is_genesis: true, but the resource already exists.'; + + async function saveRejectedWith(error: Error) { + const { store, postCommitSpy } = await testStore(); + const notify = vi + .spyOn(store, 'notifyError') + .mockImplementation(() => undefined); + const doc = await store.newResource({ + isA: 'https://atomicdata.dev/classes/Drive', + noParent: true, + }); + postCommitSpy.mockRejectedValue(error); + await expect(doc.save()).rejects.toBe(error); + const pending = store.outbox.hasPending(doc.subject); + store.setServerConnected(false); + + return { pending, notified: notify.mock.calls.length }; + } + + it.each([ErrorCode.GENESIS_COLLISION, ErrorCode.IMMUTABLE_COMMIT])( + 'code %s with an unrelated message drops the entry silently', + async code => { + const { pending, notified } = await saveRejectedWith( + new AtomicError( + 'refused, in words this client has never seen', + ErrorType.Server, + code, + ), + ); + expect(pending).toBe(false); + expect(notified).toBe(0); + }, + ); + + it('a code-less legacy genesis-collision message still drops silently', async () => { + const { pending, notified } = await saveRejectedWith( + new Error(LEGACY_GENESIS), + ); + expect(pending).toBe(false); + expect(notified).toBe(0); + }); + + it("the HTTP error body's errorCode is honoured over its description", async () => { + // `Client.postCommit` wraps a non-200 body in `AtomicError(body)`, whose + // constructor reads `errorCode` off the JSON-AD error resource. + const body = JSON.stringify({ + 'https://atomicdata.dev/properties/description': 'unrelated wording', + 'https://atomicdata.dev/properties/errorCode': + ErrorCode.GENESIS_COLLISION, + }); + const { pending, notified } = await saveRejectedWith( + new AtomicError(body, ErrorType.Server), + ); + expect(pending).toBe(false); + expect(notified).toBe(0); + }); + + it('another code keeps the entry even when the message has the legacy phrase', async () => { + const { pending } = await saveRejectedWith( + new AtomicError( + LEGACY_GENESIS, + ErrorType.Server, + ErrorCode.UNAUTHORIZED_WRITE, + ), + ); + expect(pending).toBe(true); + }); + + it('a lost-write terminal code drops the entry and tells the user', async () => { + const { pending, notified } = await saveRejectedWith( + new AtomicError( + 'unrelated wording', + ErrorType.Server, + ErrorCode.MISSING_REQUIRED_PROPERTY, + ), + ); + expect(pending).toBe(false); + expect(notified).toBe(1); + }); +}); diff --git a/browser/lib/src/store.ts b/browser/lib/src/store.ts index 8de613a53..04d478908 100644 --- a/browser/lib/src/store.ts +++ b/browser/lib/src/store.ts @@ -83,6 +83,7 @@ import { isSettledDestroyErrorMessage, isTerminalCommitError, isUnrecoverableCommitError, + isBenignTerminalCommitError, type OutboxEntry, } from './local-outbox.js'; @@ -1057,22 +1058,14 @@ export class Store { }, onTerminalDrop: (entry, e) => { const msg = e instanceof Error ? e.message : String(e); - // A redundant genesis commit (the resource already exists on the - // server) is a benign reconciliation drop, not a lost write β€” the - // data is already there. These can arrive in bulk when local state - // lost its `lastCommit` chain (e.g. after switching servers), so - // alarming the user with an error toast per commit is pure noise. - // Other terminal drops may mean a genuinely lost edit, so those still - // surface β€” silent discards are worse than visible recoveries there. - const isRedundantGenesis = msg.includes( - 'is_genesis: true, but the resource already exists', - ); - // Same reasoning for a Commit that was queued as an editable - // resource: no user write is lost (a Commit is immutable), so the - // drop is bookkeeping, not a recovery the user needs to know about. - const isCommitWrite = msg.includes('Commits cannot be edited'); + const code = e instanceof AtomicError ? e.code : undefined; - if (isRedundantGenesis || isCommitWrite) { + // A redundant genesis (the resource already exists on the server) + // or a write aimed at an immutable Commit is a benign bookkeeping + // drop, not a lost write β€” see `isBenignTerminalCommitError`. Other + // terminal drops may mean a genuinely lost edit, so those still + // surface β€” silent discards are worse than visible recoveries there. + if (isBenignTerminalCommitError(msg, code)) { console.debug( `Dropped unsyncable commit for ${entry.subject}: ${msg}`, ); diff --git a/browser/lib/src/ws-v2.ts b/browser/lib/src/ws-v2.ts index f47167fca..9fa06d1e5 100644 --- a/browser/lib/src/ws-v2.ts +++ b/browser/lib/src/ws-v2.ts @@ -131,6 +131,10 @@ export const ErrorCode = { * that carries none). Terminal for that envelope: the client must sign * again; re-sending the same bytes changes nothing. */ INVALID_SIGNATURE: 9, + /** The commit's subject is itself a Commit (`did:ad:commit:` or the + * legacy `/commits/`), which can never be edited. Terminal: + * drop the entry; nothing is lost, a Commit is whatever was signed. */ + IMMUTABLE_COMMIT: 10, } as const; /** Capability names a server may advertise in its AUTH_OK payload (mirrors diff --git a/docs/src/websockets.md b/docs/src/websockets.md index 0b8bccf51..79b8baf4d 100644 --- a/docs/src/websockets.md +++ b/docs/src/websockets.md @@ -438,8 +438,9 @@ Everything else is open to an anonymous socket and gated per subject by | `7` | `UNAUTHORIZED_READ` | A subscription or a read-side reconcile frame was refused. `request_id = 0`. Message: ` refused for : `. | | `8` | `AUTH_FAILED` | An `AUTH` frame was refused. `request_id = 0`. | | `9` | `INVALID_SIGNATURE` | A `COMMIT` whose signature does not verify against its signer's key, or that has none. Terminal for that envelope: sign again. | +| `10` | `IMMUTABLE_COMMIT` | The commit's subject is itself a Commit, which can never be edited. Terminal: drop the entry; nothing is lost. | -Codes `1` to `4` and `9` come from `classify_commit_error`, which pattern-matches the +Codes `1` to `4`, `9` and `10` come from `classify_commit_error`, which pattern-matches the underlying error text where the frame is built. Many other engine failures are **not** classified and go out as `UNKNOWN` with a descriptive message: an invalid frame of any kind, `No state`, a failed `GET` lookup, diff --git a/lib/src/sync/outbox.rs b/lib/src/sync/outbox.rs index a622bcf08..fa378933f 100644 --- a/lib/src/sync/outbox.rs +++ b/lib/src/sync/outbox.rs @@ -113,7 +113,9 @@ impl CommitRefused { pub fn is_terminal(&self) -> bool { matches!( self.effective_code(), - error_code::GENESIS_COLLISION | error_code::MISSING_REQUIRED_PROPERTY + error_code::GENESIS_COLLISION + | error_code::MISSING_REQUIRED_PROPERTY + | error_code::IMMUTABLE_COMMIT ) } diff --git a/lib/src/sync/protocol.rs b/lib/src/sync/protocol.rs index dbb18434a..7a22faaa1 100644 --- a/lib/src/sync/protocol.rs +++ b/lib/src/sync/protocol.rs @@ -258,6 +258,13 @@ pub mod error_code { /// (or that carries none). Terminal for that envelope: re-sending the /// same bytes changes nothing; the client has to sign again. pub const INVALID_SIGNATURE: u16 = 9; + /// The commit's subject is itself a Commit (`did:ad:commit:` or the + /// legacy `/commits/`). Commits are immutable, so + /// `hierarchy.rs` refuses every write to one ("Commits cannot be + /// edited."). Terminal β€” drop the entry. Nothing is lost: a Commit's + /// content is whatever was signed, and no local edit to it could ever + /// have applied. + pub const IMMUTABLE_COMMIT: u16 = 10; } /// Decode the payload of an `ERROR` frame (slice *after* the tag byte): @@ -313,6 +320,12 @@ pub fn classify_commit_error(message: &str) -> u16 { return error_code::MISSING_REQUIRED_PROPERTY; } + // `hierarchy.rs` β€” the commit's subject is a Commit, which can never be + // edited, whoever signs. + if message.contains("Commits cannot be edited") { + return error_code::IMMUTABLE_COMMIT; + } + // `hierarchy.rs` β€” no write right on the resource, or (for a create) no // append right on the parent. Both are the client trying to write where // it may not. @@ -1860,6 +1873,10 @@ mod tests { ), error_code::INVALID_SIGNATURE ); + assert_eq!( + classify_commit_error("Commits cannot be edited."), + error_code::IMMUTABLE_COMMIT + ); assert_eq!( classify_commit_error("some other error"), error_code::UNKNOWN From 859af202119e8a48abe6e7d0ca7b16e426281eae Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 18 Sep 2026 21:16:19 +0000 Subject: [PATCH 015/110] fix(query-index): list every isA encoding, check the index against the store, scope DID rows by drive stamp MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A drive-scoped, sorted, class-filtered collection query returned fewer rows than the store held, with nothing to show for it (planning/silent-failures.md, "A query index silently disagreed with the data it indexes"). The ignored reproduction `is_a_encodings_all_match_the_class_constraint` found 2 of 4 rows. Two causes, both in the read that only this query shape takes: - Rows whose `isA` read back as a plain `String` were in the index and were read, then hidden: the built-in collection class extender did `is_a.to_subjects()?` on the value, and `Db::resolve_query_member` turned any extender error into "drop the row" β€” from the page and from the count. Class extenders now read `isA` through the same reference strings the index uses, and an extender that cannot decide is skipped with a `warn!` instead of hiding a row the reader may see. - Rows whose `isA` was a `String` holding the JSON array (the legacy Loro encoding of a `ResourceArray`) were never candidates: the index keyed them by the literal `["…"]`, and the matcher rejected them for the same reason. `Value::to_reference_index_strings` and `Value::contains_value` now read the array's elements from one helper, so index and matcher cannot disagree. What should have shouted now does: - `Db::check_query_index(&Query)` compares a query's member index with a scan of the store and reports the missing and stale subjects (`QueryIndexReport`). Used by the tests; a full scan, so not on the query path. - A first build of a filter cross-checks the equality constraints the planner did not scan, bounded by the planner's 512-entry scan cap, warns with the filter and the subjects it missed, and files them (`Db::cross_check_first_build`). The hot path is untouched. Security audit C17: a DID resource cannot be routed to a drive by its subject, so every drive's watched filters saw it on commit. The `Db` now resolves each watched filter's `drive` subject to its drive root (`filter_drive_roots`, filled in `register_watched_query` and `populate_watched_queries_cache`) and compares a DID row's `drive` stamp with that root on the commit path and in the index build. The filter's identity and encoding are unchanged, which is what broke the first attempt. A row without a stamp, or a filter whose subject is not stored here, is not excluded; read rights still apply at query time. Also: `lib/src/db/website.rs` test β€” `to_hex().to_string()` is `clippy::unnecessary_to_owned` on the current toolchain, which fails the workspace clippy gate on develop; one-line fix so the gate is green. Tests: the reproduction is un-ignored; `replicated_rows_reach_a_watched_scoped_sorted_query` (5 rows watched, 17 replicated through the sync import path, 22 from every shape), `first_build_cross_checks_the_unscanned_constraint`, `check_query_index_names_missing_and_stale_members`, `did_rows_stamped_into_another_drive_stay_out_of_a_watched_query`. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01H1U7TNdjwqXKv8MQYproiq --- CHANGELOG.md | 20 ++ TESTING_COVERAGE.md | 21 ++ lib/src/class_extender.rs | 22 +- lib/src/db.rs | 287 +++++++++++++++++++- lib/src/db/query_index.rs | 20 +- lib/src/db/test.rs | 408 ++++++++++++++++++++++++++++- lib/src/values.rs | 32 ++- planning/index-performance.md | 29 ++ planning/security-audit-2026-09.md | 7 +- planning/silent-failures.md | 32 ++- 10 files changed, 840 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fb0149677..4d4215af2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,26 @@ See [STATUS.md](server/STATUS.md) to learn more about which features will remain the WS `ERROR` frame and the HTTP `/commit` error body's `errorCode` alike. It was going out as `UNKNOWN`, leaving clients to match the message text to know the refusal is terminal. The Rust outbox drops such entries as terminal. +- Query index: a drive-scoped, sorted, class-filtered collection could list + and count fewer rows than the store held, with nothing to show for it + (`planning/silent-failures.md`). Rows whose `isA` read back as a plain + `String` were in the index and then hidden, because the built-in + collection class extender could not parse the value and + `Db::resolve_query_member` dropped any row an extender check errored on; + an extender that cannot decide is now skipped with a warning and the row + is listed. Rows whose `isA` was a `String` holding the JSON array were + never indexed under their class and never matched it; the index keys and + the matcher now read the array's elements from one helper + (`Value::to_reference_index_strings`, `Value::contains_value`). New + `Db::check_query_index(&Query)` compares a query's member index with a + scan of the store and reports the missing and stale subjects; the first + build of a filter cross-checks the equality constraints the planner did + not scan (bounded by its 512-entry scan cap), warns with the filter and + the subjects, and files them. DID resources stamped into another drive no + longer enter a drive's watched query on commit or build: the `Db` resolves + each watched filter's `drive` subject to its drive root and compares + stamps (security audit C17; the filter's identity is unchanged). + - CI: the `:develop` docker image is published by its own job instead of a step tacked onto the end of the CI job. As a step it inherited whatever the CI step had already spent (a wedged Dagger engine on the runner burned the diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index f409ec8ad..3b8406c60 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -2069,6 +2069,27 @@ on Linux x86_64, including a cached install followed by changed downstream source input and execution of the retained binary. Its aarch64 archive digest is pinned to the upstream release; native aarch64 execution is not covered by that check. Full CI wall-time savings require a completed hosted run. +## Query index consistency (2026-09-18) + +`db::test::is_a_encodings_all_match_the_class_constraint` (formerly +`#[ignore]`d as an open bug) writes four rows whose `isA` names one class in +four encodings and asserts a drive-scoped, sorted, class-filtered query lists +all of them and that `Db::check_query_index` finds index and store in +agreement. `replicated_rows_reach_a_watched_scoped_sorted_query` watches that +query shape with 5 rows and then replicates 17 more through +`persist_replicated_resource` (the sync import path, propvals materialized +from a Loro doc), asserting the sorted, unsorted and differently scoped shapes +all answer 22. `first_build_cross_checks_the_unscanned_constraint` removes one +row's `isA` entry from `PropValSub` and asserts the first build still files +the row through the `parent` constraint. +`check_query_index_names_missing_and_stale_members` corrupts a member index in +both directions and asserts the report names each subject. +`did_rows_stamped_into_another_drive_stay_out_of_a_watched_query` covers the +audit's C17 on both the build and the commit path, including the unstamped +row that is deliberately not excluded. Not covered: the runtime `warn!` text +itself, and a UI-level comparison of a client's local answer with the +server's (see `planning/silent-failures.md`). + ## External cache access and authentication origins (#170) Paired SaaS `portal/e2e/recovery-passkey.spec.ts` uses Chromium virtual PRF authenticators with the real control plane to verify app enrollment followed by portal login using one credential, reuse of a portal-created credential, and account-settings migration without replacing ciphertext or old wrappers. Physical Safari/iCloud, Android/password-manager and native-shell behavior remain device acceptance checks. diff --git a/lib/src/class_extender.rs b/lib/src/class_extender.rs index 5582d0279..7dbdc307b 100644 --- a/lib/src/class_extender.rs +++ b/lib/src/class_extender.rs @@ -195,7 +195,7 @@ impl ClassExtender { return Ok(false); }; - let resource_classes = is_a.to_subjects(None)?; + let resource_classes = class_subjects(is_a); let matched = resource_classes .iter() .any(|c| self.classes.contains(&normalize_class(c))); @@ -317,15 +317,25 @@ impl ClassExtender { return true; }; - let Ok(is_a_subjects) = is_a.to_subjects(None) else { - return true; - }; - // Check if the resource is a plugin, if so return false. - !is_a_subjects.contains(&urls::PLUGIN.to_string()) + !class_subjects(is_a).contains(&urls::PLUGIN.to_string()) } } +/// Every class an `isA` value names, whatever its encoding. +/// +/// A row created by a local commit carries `isA` as a `ResourceArray`; one +/// rebuilt from a Loro doc, or written through a client that pins no datatype, +/// can read back as an `AtomicUrl`, a plain `String`, or a `String` holding +/// the JSON array. They all name the same class. `Value::to_subjects` errors on +/// the scalar shapes, and that error used to be returned from +/// [`ClassExtender::resource_has_extender`] β€” where `Db::resolve_query_member` +/// took it as a reason to drop the row from a collection query, without a +/// word. Membership in a class is not something an encoding gets to veto. +fn class_subjects(is_a: &crate::Value) -> Vec { + is_a.to_reference_index_strings().unwrap_or_default() +} + #[cfg(test)] mod tests { use super::*; diff --git a/lib/src/db.rs b/lib/src/db.rs index c977bf4a7..30fde0cf4 100644 --- a/lib/src/db.rs +++ b/lib/src/db.rs @@ -291,6 +291,34 @@ fn replication_key(drive: &str) -> String { format!("{REPLICATION_PREFIX}{drive}") } +/// Upper bound on the `PropValSub` entries the planner counts per constraint +/// when ranking them, and on what the first-build cross-check will walk. +const PLANNER_SCAN_CAP: usize = 512; + +/// What [`Db::check_query_index`] found when it compared a query's member +/// index with a scan of the store. +#[derive(Debug, Clone)] +pub struct QueryIndexReport { + /// The filter whose index was checked. + pub filter: query_index::QueryFilter, + /// Whether the filter is watched (its index is maintained on commit). + pub watched: bool, + /// Members the index holds. + pub indexed: usize, + /// Resources in the store that match the filter. + pub expected: usize, + /// Matching subjects the index does not list. + pub missing: Vec, + /// Indexed subjects that no longer match (or no longer exist). + pub stale: Vec, +} + +impl QueryIndexReport { + pub fn is_consistent(&self) -> bool { + self.missing.is_empty() && self.stale.is_empty() + } +} + /// One drive's watched query filters, routed by the properties they touch so /// a changed atom only evaluates filters that could care about it. #[derive(Default, Debug)] @@ -395,6 +423,15 @@ pub struct Db { /// `check_if_atom_matches_watched_query_filters` reads from here and /// never touches msgpack on a commit. watched_queries_by_drive: Arc>>, + /// The drive root each watched filter's `drive` resolves to, keyed by the + /// filter's `drive` string. For a `did:` query, `QueryFilter::drive` is + /// the queried subject (a table, a folder), not the drive it lives in; + /// its stored `drive` stamp names the root. Resolved when a filter is + /// registered and looked up per DID atom on the commit path, so that a + /// DID resource stamped into another drive never enters this drive's + /// members (security audit C17). `None` records that the subject could + /// not be resolved here, in which case the filter is not drive-checked. + filter_drive_roots: Arc>>>, /// Serialises writers that read-modify-write the same subject's state, so /// a commit and a sync apply cannot replace each other's snapshot. Per /// store, not global β€” see [`crate::subject_lock`]. @@ -670,6 +707,7 @@ impl Db { db_events: tokio::sync::broadcast::channel(64).0, ephemeral_events: tokio::sync::broadcast::channel(32).0, watched_queries_by_drive: Arc::new(RwLock::new(HashMap::new())), + filter_drive_roots: Arc::new(RwLock::new(HashMap::new())), subject_locks: Default::default(), plugin_locks: Default::default(), base_domain, @@ -712,6 +750,7 @@ impl Db { db_events: tokio::sync::broadcast::channel(64).0, ephemeral_events: tokio::sync::broadcast::channel(32).0, watched_queries_by_drive: Arc::new(RwLock::new(HashMap::new())), + filter_drive_roots: Arc::new(RwLock::new(HashMap::new())), subject_locks: Default::default(), plugin_locks: Default::default(), base_domain, @@ -750,6 +789,7 @@ impl Db { db_events: tokio::sync::broadcast::channel(64).0, ephemeral_events: tokio::sync::broadcast::channel(32).0, watched_queries_by_drive: Arc::new(RwLock::new(HashMap::new())), + filter_drive_roots: Arc::new(RwLock::new(HashMap::new())), subject_locks: Default::default(), plugin_locks: Default::default(), base_domain, @@ -849,6 +889,7 @@ impl Db { db_events: tokio::sync::broadcast::channel(64).0, ephemeral_events: tokio::sync::broadcast::channel(32).0, watched_queries_by_drive: Arc::new(RwLock::new(HashMap::new())), + filter_drive_roots: Arc::new(RwLock::new(HashMap::new())), subject_locks: Default::default(), plugin_locks: Default::default(), base_domain, @@ -1031,6 +1072,7 @@ impl Db { db_events: tokio::sync::broadcast::channel(64).0, ephemeral_events: tokio::sync::broadcast::channel(32).0, watched_queries_by_drive: Arc::new(RwLock::new(HashMap::new())), + filter_drive_roots: Arc::new(RwLock::new(HashMap::new())), subject_locks: Default::default(), plugin_locks: Default::default(), base_domain, @@ -2863,11 +2905,17 @@ impl Db { /// starting set β€” the estimate only decides how few resources get row- /// checked. Constraints with non-equality operators can't be point- /// scanned; they contribute a whole-property scan as their candidate set. - fn plan_candidate_iterator(&self, q: &Query, q_filter: &QueryFilter) -> IndexIterator { - const PLANNER_SCAN_CAP: usize = 512; - - let mut best: Option<(usize, &crate::storelike::PropVal)> = None; - for constraint in &q_filter.filters { + /// + /// Also returns the position of the chosen constraint in + /// `q_filter.filters`, so the build can cross-check its result against + /// the constraints it did not scan (see `cross_check_first_build`). + fn plan_candidate_iterator( + &self, + q: &Query, + q_filter: &QueryFilter, + ) -> (IndexIterator, Option) { + let mut best: Option<(usize, usize, &crate::storelike::PropVal)> = None; + for (position, constraint) in q_filter.filters.iter().enumerate() { let Some(prop) = &constraint.property else { continue; }; @@ -2876,13 +2924,13 @@ impl Db { _ => None, }; let estimate = self.estimate_prop_val_count(prop, scan_val, PLANNER_SCAN_CAP); - if best.is_none_or(|(current, _)| estimate < current) { - best = Some((estimate, constraint)); + if best.is_none_or(|(current, _, _)| estimate < current) { + best = Some((estimate, position, constraint)); } } match best { - Some((_, constraint)) => { + Some((_, position, constraint)) => { let prop = constraint .property .as_ref() @@ -2891,12 +2939,113 @@ impl Db { crate::storelike::FilterOperator::Equal => constraint.value.as_ref(), _ => None, }; - find_in_prop_val_sub_index(self, prop, val) + (find_in_prop_val_sub_index(self, prop, val), Some(position)) } // No property-bearing constraint (value-only filters): fall back // to the query's own iterator (value index or full scan). - None => self.get_index_iterator_for_query(q), + None => (self.get_index_iterator_for_query(q), None), + } + } + + /// The members a first build missed, found by walking the small + /// equality constraints the planner did *not* scan. + /// + /// The planner trusts that any one constraint's `PropValSub` entries are a + /// superset of the members. That held only as long as every encoding of a + /// value produced the same index key; when one did not, the cheapest + /// constraint had fewer candidates than rows, the build filed a partial + /// member list, and the filter was watched β€” so the partial list was + /// trusted for good and every count it produced looked plausible + /// (planning/silent-failures.md). This walks each other equality + /// constraint whose entries fit under the planner's scan cap (so at most + /// `PLANNER_SCAN_CAP` extra key reads per constraint, once per first + /// build), and returns every matching resource that is not in `built`. + /// Constraints beyond the cap are not walked here; the full comparison is + /// [`Db::check_query_index`]. + fn cross_check_first_build( + &self, + q_filter: &QueryFilter, + chosen: Option, + built: &HashSet, + ) -> Vec { + let mut missing = Vec::new(); + let mut seen: HashSet = HashSet::new(); + for (position, constraint) in q_filter.filters.iter().enumerate() { + if Some(position) == chosen { + continue; + } + let (Some(prop), Some(val)) = (&constraint.property, &constraint.value) else { + continue; + }; + if constraint.operator != crate::storelike::FilterOperator::Equal { + continue; + } + if self.estimate_prop_val_count(prop, Some(val), PLANNER_SCAN_CAP) >= PLANNER_SCAN_CAP { + continue; + } + for atom in find_in_prop_val_sub_index(self, prop, Some(val)).flatten() { + let subject = atom.subject.as_str().to_string(); + if built.contains(&subject) || !seen.insert(subject) { + continue; + } + let Ok(resource) = self.get_resource_shallow(&atom.subject) else { + continue; + }; + if query_index::resource_matches_filter(&resource, q_filter) + && self.filter_accepts_resource_drive(q_filter, &resource) + { + missing.push(resource); + } + } + } + missing + } + + /// Compare a query's member index with the resources it claims to index. + /// + /// Walks every stored resource, applies the query's filter to it the way + /// the index maintenance does (`resource_matches_filter` plus the drive + /// check), and reports which matching subjects the index lacks and which + /// indexed subjects no longer match. Reading a row set out of the index + /// is fast because it trusts the index; this is the check that trust is + /// deserved. It is a full scan, so it is for tests, diagnostics and + /// one-off repairs, not for the query path. + pub fn check_query_index(&self, q: &Query) -> AtomicResult { + let q_filter = QueryFilter::try_from_query(q)?; + let id = query_index::query_id(&q_filter)?; + // Index keys hold the subject as the writer spelled it; rows come + // back under the store's normalized spelling. Compare one form. + let base_domain = self.get_base_domain(); + let canonical = |s: &str| Subject::from_raw(s, base_domain.as_deref()).pure_id(); + + let mut indexed: HashSet = HashSet::new(); + for kv in self.kv.scan_prefix(Tree::QueryMembers, &id) { + let (key, _) = kv?; + let (_id, _sort, subject) = query_index::parse_members_key(&key)?; + indexed.insert(canonical(subject)); + } + + let mut expected: HashSet = HashSet::new(); + for resource in self.all_resources(true) { + if query_index::resource_matches_filter(&resource, &q_filter) + && self.filter_accepts_resource_drive(&q_filter, &resource) + { + expected.insert(canonical(resource.get_subject().as_str())); + } } + + let mut missing: Vec = expected.difference(&indexed).cloned().collect(); + let mut stale: Vec = indexed.difference(&expected).cloned().collect(); + missing.sort(); + stale.sort(); + Ok(QueryIndexReport { + watched: q_filter.is_watched(self), + filter: q_filter, + indexed: indexed.len(), + expected: expected.len(), + missing, + stale, + }) } /// Register a filter to be watched. Persists to `Tree::WatchedQueries` @@ -2909,6 +3058,9 @@ impl Db { &self, filter: query_index::QueryFilter, ) -> AtomicResult<()> { + // Before the persisted short-circuit: a client re-watching after the + // queried subject arrived here gets its drive root resolved now. + self.cache_filter_drive_root(&filter.drive); let filter_bytes = filter.encode()?; // Skip if already persisted β€” avoids growing the in-memory Vec on // re-watches. The KV is authoritative for "what filters exist"; the @@ -2953,6 +3105,7 @@ impl Db { continue; } }; + self.cache_filter_drive_root(&qf.drive); let drive_key = qf.drive.as_str().to_string(); new_map.entry(drive_key).or_default().insert(Arc::new(qf)); } @@ -2962,6 +3115,57 @@ impl Db { Ok(()) } + /// Resolve the drive root a filter's `drive` subject lives in and remember + /// it. A subject that is not stored here (an `https://host` prefix, a DID + /// that has not arrived yet) records `None`; the next `watch()` retries. + fn cache_filter_drive_root(&self, drive: &Subject) { + let key = drive.as_str().to_string(); + let known = self + .filter_drive_roots + .read() + .ok() + .and_then(|m| m.get(&key).cloned()); + if matches!(known, Some(Some(_))) { + return; + } + let root = self + .get_resource_shallow(drive) + .ok() + .and_then(|resource| resource.get_drive()); + if let Ok(mut map) = self.filter_drive_roots.write() { + map.insert(key, root); + } + } + + /// The drive root a watched filter was resolved to, if known. + pub(crate) fn filter_drive_root(&self, filter: &query_index::QueryFilter) -> Option { + self.filter_drive_roots + .read() + .ok() + .and_then(|m| m.get(filter.drive.as_str()).cloned().flatten()) + } + + /// Whether a resource may be a member of `filter` as far as drive scope + /// goes. HTTP and `internal:` subjects are already routed to their drive + /// by prefix. A DID subject has no prefix, so the only evidence is its + /// `drive` stamp: when both it and the filter's root are known, they + /// must agree. A row without a stamp, or a filter whose root could not be + /// resolved, is not excluded β€” a missing stamp is not proof of a leak, + /// and read rights still apply at query time. + pub(crate) fn filter_accepts_resource_drive( + &self, + filter: &query_index::QueryFilter, + resource: &Resource, + ) -> bool { + if !resource.get_subject().as_str().starts_with("did:") { + return true; + } + match (resource.get_drive(), self.filter_drive_root(filter)) { + (Some(stamp), Some(root)) => stamp.pure_id() == root.pure_id(), + _ => true, + } + } + /// The watched filters a changed atom in `drive_key` with `property` must /// be checked against: the drive's filters that reference that property /// (constraint or `sort_by`) plus its value-only filters. Returns @@ -3157,6 +3361,13 @@ impl Db { Ok(guard) => guard.clone(), Err(_) => return None, }; + // An extender that cannot decide whether it applies is skipped, not + // treated as a reason to hide the row. The row exists and the reader + // may see it; the only thing at stake here is the `incomplete` + // marker. Returning `None` from here used to drop every row whose + // `isA` the built-in collection extender could not parse, silently, + // from every collection listing and count (planning/silent-failures.md, + // "A query index silently disagreed with the data it indexes"). for extender in extenders.iter() { if !extender.can_extend(&resource) { continue; @@ -3164,7 +3375,14 @@ impl Db { match extender.resource_has_extender(&resource) { Ok(true) => {} Ok(false) => continue, - Err(_) => return None, + Err(e) => { + tracing::warn!( + subject = %resource.get_subject(), + extender = extender.id.as_deref().unwrap_or(""), + "query member listed without its extender check: {e}" + ); + continue; + } } match extender.check_scope(&resource, self, None).await { Ok((true, _)) => { @@ -3173,7 +3391,14 @@ impl Db { break; } Ok((false, _)) => continue, - Err(_) => return None, + Err(e) => { + tracing::warn!( + subject = %resource.get_subject(), + extender = extender.id.as_deref().unwrap_or(""), + "query member listed without its extender scope check: {e}" + ); + continue; + } } } @@ -3488,7 +3713,7 @@ impl Db { if !q_filter.is_watched(self) { info!(filter = ?q_filter, "Building query index"); crate::metrics::query_indexed(); - let atoms = self.plan_candidate_iterator(q, &q_filter); + let (atoms, chosen) = self.plan_candidate_iterator(q, &q_filter); q_filter.watch(self)?; let mut transaction = Transaction::new(); @@ -3511,13 +3736,16 @@ impl Db { // constraints on its materialized row (no Loro decode), so the // member index only holds true AND-matches β€” including for // non-equality operators, whose candidate sets are supersets. + let mut built: HashSet = HashSet::new(); for atom in atoms.flatten() { let Ok(resource) = self.get_resource_shallow(&atom.subject) else { // No row to verify against (external/never-stored) β€” // don't index what we can't confirm. continue; }; - if !query_index::resource_matches_filter(&resource, &q_filter) { + if !query_index::resource_matches_filter(&resource, &q_filter) + || !self.filter_accepts_resource_drive(&q_filter, &resource) + { continue; } let prop = query_index::index_key_property(&q_filter, &atom); @@ -3529,6 +3757,37 @@ impl Db { false, &mut transaction, )?; + built.insert(atom.subject.as_str().to_string()); + } + // A member the scanned constraint's index does not know about is + // a member the filter would have listed forever without. Say so, + // and file it. + let missed = self.cross_check_first_build(&q_filter, chosen, &built); + if !missed.is_empty() { + let subjects: Vec = missed + .iter() + .map(|r| r.get_subject().as_str().to_string()) + .collect(); + tracing::warn!( + filter = ?q_filter, + missing = ?subjects, + "query index build missed members its candidate index did not list; added" + ); + for resource in &missed { + // The cross-check only walks property-bearing + // constraints, so the filter always has a key property. + let prop = query_index::filter_key_property(&q_filter) + .cloned() + .unwrap_or_default(); + let sort_key = query_index::sort_key_for(resource, &prop); + update_indexed_member( + &q_filter, + resource.get_subject().as_str(), + &sort_key, + false, + &mut transaction, + )?; + } } self.apply_transaction(&mut transaction)?; diff --git a/lib/src/db/query_index.rs b/lib/src/db/query_index.rs index 2dcb8099d..c4c91a7ea 100644 --- a/lib/src/db/query_index.rs +++ b/lib/src/db/query_index.rs @@ -407,15 +407,16 @@ pub(crate) fn index_key_property<'a>( q_filter: &'a QueryFilter, index_atom: &'a IndexAtom, ) -> &'a String { + filter_key_property(q_filter).unwrap_or(&index_atom.property) +} + +/// [`index_key_property`] without an atom: `sort_by`, else the first +/// constraint property; `None` only for an all-value-only filter. +pub(crate) fn filter_key_property(q_filter: &QueryFilter) -> Option<&String> { if let Some(sort_by) = &q_filter.sort_by { - return sort_by; + return Some(sort_by); } - for c in &q_filter.filters { - if let Some(property) = &c.property { - return property; - } - } - &index_atom.property + q_filter.filters.iter().find_map(|c| c.property.as_ref()) } /// Checks if a new IndexAtom should be updated for a specific [QueryFilter] @@ -492,6 +493,11 @@ pub fn check_if_atom_matches_watched_query_filters( ); for q_filter in &filters { + // A DID atom reaches every drive's filters (no prefix to route by); + // the resource's `drive` stamp is what keeps it out of the others. + if !store.filter_accepts_resource_drive(q_filter, resource) { + continue; + } if let Some(prop) = should_update_property(q_filter, index_atom, resource) { let sort_key = sort_key_for(resource, prop); update_indexed_member( diff --git a/lib/src/db/test.rs b/lib/src/db/test.rs index 7a19a86da..d3d327c91 100644 --- a/lib/src/db/test.rs +++ b/lib/src/db/test.rs @@ -2398,11 +2398,16 @@ async fn sorted_query_index_keeps_up_with_two_constraints() { /// as the same subject fails the constraint, the row is silently absent from /// the index, and therefore from the table, while being present in the store /// and in every unfiltered query. +/// +/// This reproduced two silent failures at once. The `String` row was in the +/// index and was read, and then hidden: the built-in collection class +/// extender did `is_a.to_subjects()?` on it, and `resolve_query_member` took +/// any extender error as a reason to drop the row from the page and the +/// count. The JSON-array `String` row was never a candidate: its `isA` was +/// indexed under the literal `["…"]` string, so the class constraint could +/// not find it and the matcher did not accept it either. #[tokio::test] #[timeout(120000)] -#[ignore = "reproduces an OPEN bug: query_sorted_indexed returns 2 of 3 members \ - that are written, parseable, local and matching β€” the read stops \ - after the second. Run with `--ignored` while working on it."] async fn is_a_encodings_all_match_the_class_constraint() { use crate::storelike::Query; @@ -2470,6 +2475,403 @@ async fn is_a_encodings_all_match_the_class_constraint() { encodings.len(), res.subjects ); + assert_eq!(res.subjects.len(), encodings.len()); + assert_eq!(res.resources.len(), encodings.len()); + + let report = store.check_query_index(&query).unwrap(); + assert!( + report.is_consistent(), + "index and store disagree after the read: {report:?}" + ); + assert_eq!(report.expected, encodings.len()); +} + +/// A row built the way sync builds it β€” a fresh `Resource` given a Loro doc +/// through `apply_state_doc`, persisted by `persist_replicated_resource` β€” +/// must land in a watched, drive-scoped, sorted, class-filtered index, and +/// the sorted read must then agree with the unsorted and the unscoped +/// shapes of the same question. +/// +/// This is the shape from planning/silent-failures.md: a table rendered 5 +/// rows through `query_complex` while 22 existed and every other query shape +/// returned 22. The first 5 rows are written before the query is watched +/// (so the build files them), the other 17 arrive as replicas afterwards (so +/// only commit-time maintenance can file them). +#[tokio::test] +#[timeout(120000)] +async fn replicated_rows_reach_a_watched_scoped_sorted_query() { + use crate::storelike::Query; + + let store = Db::init_temp("query_index_replicated_rows").await.unwrap(); + + let drive = + "did:ad:driveREPLaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + let table = + "did:ad:tableREPLaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + let cls = + "did:ad:classREPLaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + + let mut t = crate::Resource::new(table.into()); + t.set_unsafe(urls::PARENT.into(), Value::AtomicUrl(drive.into())) + .unwrap(); + t.set_unsafe(urls::DRIVE_PROP.into(), Value::AtomicUrl(drive.into())) + .unwrap(); + store + .add_resource_opts(&t, false, true, true) + .await + .unwrap(); + + // What a peer's SYNC_PUSH entry becomes here: propvals materialized from + // the Loro doc, not the ones a local caller set. + let replicate = |i: usize| { + let subj = format!("did:ad:repl{:0>69}==", format!("{i}")); + let mut authored = crate::Resource::new(subj.clone()); + authored + .set_unsafe(urls::PARENT.into(), Value::AtomicUrl(table.into())) + .unwrap(); + authored + .set_unsafe(urls::DRIVE_PROP.into(), Value::AtomicUrl(drive.into())) + .unwrap(); + authored + .set_unsafe( + urls::IS_A.into(), + Value::ResourceArray(vec![crate::values::SubResource::Subject(cls.into())]), + ) + .unwrap(); + authored + .set_unsafe(urls::NAME.into(), Value::String(format!("row {i:02}"))) + .unwrap(); + let doc = authored.build_state_doc().unwrap(); + let mut replica = crate::Resource::new(subj); + replica.apply_state_doc(doc).unwrap(); + replica + }; + + for i in 0..5 { + store + .persist_replicated_resource(&replicate(i)) + .await + .unwrap(); + } + + let table_query = |sort: bool, scope: &str| { + let mut q = Query::new_prop_val(urls::PARENT, table); + q.filters = vec![crate::storelike::PropVal { + property: Some(urls::IS_A.to_string()), + value: Some(Value::AtomicUrl(cls.into())), + ..Default::default() + }]; + q.sort_by = sort.then(|| urls::NAME.to_string()); + q.drive = Some(scope.into()); + q.limit = Some(100); + q + }; + let scoped_sorted = table_query(true, drive); + + assert_eq!(store.query(&scoped_sorted).await.unwrap().count, 5); + assert!( + crate::db::query_index::QueryFilter::try_from_query(&scoped_sorted) + .unwrap() + .is_watched(&store) + ); + + for i in 5..22 { + store + .persist_replicated_resource(&replicate(i)) + .await + .unwrap(); + } + + let unsorted = table_query(false, drive); + let other_scope = table_query(true, table); + + let sorted_res = store.query(&scoped_sorted).await.unwrap(); + assert_eq!( + sorted_res.count, 22, + "the watched sorted index lists {} of 22 replicated rows", + sorted_res.count + ); + assert_eq!(store.query(&unsorted).await.unwrap().count, 22); + assert_eq!(store.query(&other_scope).await.unwrap().count, 22); + + for q in [&scoped_sorted, &unsorted, &other_scope] { + let report = store.check_query_index(q).unwrap(); + assert!(report.is_consistent(), "{report:?}"); + assert_eq!(report.indexed, 22); + } +} + +/// The planner scans one constraint's index and trusts it to hold every +/// member. When it does not β€” here one row's `isA` entry is simply gone β€” +/// the first build must notice through the other constraint, say so, and +/// file the row anyway, instead of watching a partial list for good. +#[tokio::test] +#[timeout(120000)] +async fn first_build_cross_checks_the_unscanned_constraint() { + use crate::storelike::Query; + + let store = Db::init_temp("query_index_cross_check").await.unwrap(); + let drive = + "did:ad:driveXCHKaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + let table = + "did:ad:tableXCHKaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + let cls = + "did:ad:classXCHKaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + + // More rows in the table than in the class, so the planner scans `isA`. + for i in 0..6 { + let subj = format!("did:ad:xchk{:0>69}==", format!("{i}")); + let mut r = crate::Resource::new(subj); + r.set_unsafe(urls::PARENT.into(), Value::AtomicUrl(table.into())) + .unwrap(); + if i < 4 { + r.set_unsafe( + urls::IS_A.into(), + Value::ResourceArray(vec![crate::values::SubResource::Subject(cls.into())]), + ) + .unwrap(); + } + r.set_unsafe(urls::NAME.into(), Value::String(format!("row{i}"))) + .unwrap(); + store + .add_resource_opts(&r, false, true, true) + .await + .unwrap(); + } + + // Knock one row out of the class index only. Its row still names the + // class, and its `parent` entry is intact. + let gone = "did:ad:xchk000000000000000000000000000000000000000000000000000000000000000000002=="; + let mut transaction = crate::db::trees::Transaction::new(); + transaction.push( + crate::db::trees::Operation::remove_atom_from_prop_val_sub_index( + &crate::atoms::IndexAtom { + subject: Subject::from(gone), + property: urls::IS_A.to_string(), + ref_value: cls.to_string(), + sort_value: cls.to_string(), + }, + ), + ); + store.apply_transaction(&mut transaction).unwrap(); + + let mut query = Query::new_prop_val(urls::PARENT, table); + query.filters = vec![crate::storelike::PropVal { + property: Some(urls::IS_A.to_string()), + value: Some(Value::AtomicUrl(cls.into())), + ..Default::default() + }]; + query.sort_by = Some(urls::NAME.to_string()); + query.drive = Some(drive.into()); + query.limit = Some(100); + + let res = store.query(&query).await.unwrap(); + assert_eq!(res.count, 4, "got {:?}", res.subjects); + assert!(res.subjects.iter().any(|s| s.as_str() == gone)); + let report = store.check_query_index(&query).unwrap(); + assert!(report.is_consistent(), "{report:?}"); +} + +/// The consistency check names what is wrong, in both directions. +#[tokio::test] +#[timeout(120000)] +async fn check_query_index_names_missing_and_stale_members() { + use crate::storelike::Query; + + let store = Db::init_temp("query_index_report").await.unwrap(); + let drive = + "did:ad:driveRPRTaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + let table = + "did:ad:tableRPRTaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + + let mut subjects = vec![]; + for i in 0..3 { + let subj = format!("did:ad:rprt{:0>69}==", format!("{i}")); + let mut r = crate::Resource::new(subj.clone()); + r.set_unsafe(urls::PARENT.into(), Value::AtomicUrl(table.into())) + .unwrap(); + r.set_unsafe(urls::NAME.into(), Value::String(format!("row{i}"))) + .unwrap(); + store + .add_resource_opts(&r, false, true, true) + .await + .unwrap(); + subjects.push(subj); + } + + let mut query = Query::new_prop_val(urls::PARENT, table); + query.sort_by = Some(urls::NAME.to_string()); + query.drive = Some(drive.into()); + assert_eq!(store.query(&query).await.unwrap().count, 3); + + let report = store.check_query_index(&query).unwrap(); + assert!(report.is_consistent(), "{report:?}"); + assert!(report.watched); + assert_eq!((report.indexed, report.expected), (3, 3)); + + // Drop one member from the index, and file one that is not a row. + let q_filter = crate::db::query_index::QueryFilter::try_from_query(&query).unwrap(); + let ghost = + "did:ad:rprtGHOSTaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + let mut transaction = crate::db::trees::Transaction::new(); + let dropped = store + .get_resource_shallow(&Subject::from(subjects[1].clone())) + .unwrap(); + let sort_key = crate::db::query_index::sort_key_for(&dropped, urls::NAME); + crate::db::query_index::update_indexed_member( + &q_filter, + &subjects[1], + &sort_key, + true, + &mut transaction, + ) + .unwrap(); + crate::db::query_index::update_indexed_member( + &q_filter, + ghost, + &crate::db::query_index::encode_sort_value(Some(&Value::String("zzz".into()))), + false, + &mut transaction, + ) + .unwrap(); + store.apply_transaction(&mut transaction).unwrap(); + + let report = store.check_query_index(&query).unwrap(); + assert!(!report.is_consistent()); + assert_eq!(report.missing, vec![subjects[1].clone()]); + assert_eq!(report.stale, vec![ghost.to_string()]); + assert_eq!((report.indexed, report.expected), (3, 3)); +} + +/// A DID row cannot be routed to a drive by its subject, so every drive's +/// watched filters see it on commit. Its `drive` stamp is what keeps it out +/// of the drives it does not belong to β€” compared against the *root* the +/// filter's `drive` resolves to, since for a `did:` query that field is the +/// queried subject (a table), not the drive itself. Security audit C17. +#[tokio::test] +#[timeout(120000)] +async fn did_rows_stamped_into_another_drive_stay_out_of_a_watched_query() { + use crate::storelike::Query; + + let store = Db::init_temp("query_index_drive_stamp").await.unwrap(); + let drive_a = + "did:ad:driveSTMPAaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + let drive_b = + "did:ad:driveSTMPBaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + let table_a = + "did:ad:tableSTMPAaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + let cls = + "did:ad:classSTMPaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa=="; + + for drive in [drive_a, drive_b] { + let mut d = crate::Resource::new(drive.into()); + d.set_unsafe( + urls::IS_A.into(), + Value::ResourceArray(vec![crate::values::SubResource::Subject( + urls::DRIVE.into(), + )]), + ) + .unwrap(); + store + .add_resource_opts(&d, false, true, true) + .await + .unwrap(); + } + let mut t = crate::Resource::new(table_a.into()); + t.set_unsafe(urls::PARENT.into(), Value::AtomicUrl(drive_a.into())) + .unwrap(); + t.set_unsafe(urls::DRIVE_PROP.into(), Value::AtomicUrl(drive_a.into())) + .unwrap(); + store + .add_resource_opts(&t, false, true, true) + .await + .unwrap(); + + let row = |name: &str, drive: Option<&str>| { + let subj = format!("did:ad:stmp{name}{}==", "a".repeat(69 - name.len())); + let mut r = crate::Resource::new(subj.clone()); + r.set_unsafe( + urls::IS_A.into(), + Value::ResourceArray(vec![crate::values::SubResource::Subject(cls.into())]), + ) + .unwrap(); + r.set_unsafe(urls::NAME.into(), Value::String(name.into())) + .unwrap(); + if let Some(drive) = drive { + r.set_unsafe(urls::DRIVE_PROP.into(), Value::AtomicUrl(drive.into())) + .unwrap(); + } + (subj, r) + }; + + // Written before the query exists: the build path. + let (in_a, r) = row("inA", Some(drive_a)); + store + .add_resource_opts(&r, false, true, true) + .await + .unwrap(); + let (in_b, r) = row("inB", Some(drive_b)); + store + .add_resource_opts(&r, false, true, true) + .await + .unwrap(); + let (unstamped, r) = row("noStamp", None); + store + .add_resource_opts(&r, false, true, true) + .await + .unwrap(); + + // Every instance of the class, as seen from table A. + let mut query = Query::new_class(cls); + query.sort_by = Some(urls::NAME.to_string()); + query.drive = Some(table_a.into()); + query.limit = Some(100); + + let found: Vec = store + .query(&query) + .await + .unwrap() + .subjects + .iter() + .map(|s| s.as_str().to_string()) + .collect(); + assert!(found.contains(&in_a), "{found:?}"); + assert!( + !found.contains(&in_b), + "a row stamped into drive B is listed by drive A's query: {found:?}" + ); + // No stamp is no evidence; read rights decide at query time. + assert!(found.contains(&unstamped), "{found:?}"); + + // Written while the query is watched: the commit path. + let (in_a2, r) = row("inA2", Some(drive_a)); + store + .add_resource_opts(&r, false, true, true) + .await + .unwrap(); + let (in_b2, r) = row("inB2", Some(drive_b)); + store + .add_resource_opts(&r, false, true, true) + .await + .unwrap(); + + let found: Vec = store + .query(&query) + .await + .unwrap() + .subjects + .iter() + .map(|s| s.as_str().to_string()) + .collect(); + assert!(found.contains(&in_a2), "{found:?}"); + assert!( + !found.contains(&in_b2), + "a row committed into drive B reached drive A's watched query: {found:?}" + ); + assert_eq!(found.len(), 3, "{found:?}"); + + let report = store.check_query_index(&query).unwrap(); + assert!(report.is_consistent(), "{report:?}"); } /// Narrows the previous failure: is the row rejected by the MATCHER diff --git a/lib/src/values.rs b/lib/src/values.rs index bf843683f..50b68ccae 100644 --- a/lib/src/values.rs +++ b/lib/src/values.rs @@ -99,13 +99,20 @@ impl Value { } } /// Check if the value `q_val` is present in `val` + /// + /// Uses the same reference strings the index is keyed by + /// ([`Self::to_reference_index_strings`]), so a query's matcher and its + /// candidate index can never disagree about which values name `q_val`. + /// A `String` holding a JSON array of subjects (`["did:…"]`, the legacy + /// Loro encoding of a `ResourceArray`) names each of its elements. pub fn contains_value(&self, q_val: &Value) -> bool { let query_value = q_val.to_string(); match self { - Value::ResourceArray(_vec) => { - let subs = self.to_subjects(None).unwrap_or_default(); - subs.iter().any(|v| v == &query_value) - } + Value::ResourceArray(_) | Value::String(_) => self + .to_reference_index_strings() + .unwrap_or_default() + .iter() + .any(|v| v == &query_value), other => other.to_string() == query_value, } } @@ -355,11 +362,18 @@ impl Value { /// Converts one Value to a bunch of indexable items. /// Returns None for unsupported types. + /// + /// A `String` that holds a JSON array of strings is the legacy Loro + /// encoding of a `ResourceArray` (see `loro_value_to_atomic_value`); it + /// is indexed under every element, exactly as the array it stands for, + /// so a row whose `isA` reads back in that shape is still found by its + /// class. [`Self::contains_value`] uses the same strings. pub fn to_reference_index_strings(&self) -> Option> { let vals = match self { // TODO: This results in wrong indexing, as some subjects will be numbers. Value::ResourceArray(_v) => self.to_subjects(None).unwrap_or_else(|_| vec![]), Value::AtomicUrl(v) => vec![v.to_string()], + Value::String(s) => json_string_array(s).unwrap_or_else(|| vec![s.clone()]), Value::NestedResource(_) | Value::LoroDoc(_) => return None, // This might result in unnecessarily long strings, sometimes. We may want to shorten them later. val => vec![val.to_string()], @@ -368,6 +382,16 @@ impl Value { } } +/// The elements of `s` when it is a JSON array of strings (`["a","b"]`), +/// which is how older Loro docs serialized `ResourceArray`s. Anything else, +/// including an array holding non-strings, is `None`. +fn json_string_array(s: &str) -> Option> { + if !s.starts_with('[') { + return None; + } + serde_json::from_str::>(s).ok() +} + /// A value that is meant for checking reference indexes. /// short. Vectors of subjects are turned into individual ReferenceStrings. pub type ReferenceString = String; diff --git a/planning/index-performance.md b/planning/index-performance.md index 250917d39..877a04668 100644 --- a/planning/index-performance.md +++ b/planning/index-performance.md @@ -70,6 +70,8 @@ a derived, rebuildable projection: | Compact `query_id` keys in `QueryMembers` + id-keyed live `QUERY_UPDATE` routing | **built (this pass)** | | `(drive, property)`-routed watched-filter matching | **built (this pass)** | | Most-selective-constraint planner for AND filters (bounded cardinality estimates) | **built (this pass)** | +| First-build cross-check of the constraints the planner did not scan (bounded by the scan cap) + `Db::check_query_index` full-scan diagnostic | **built (2026-09-18, finding 7)** | +| Drive-stamp scoping of DID resources for watched filters (`filter_drive_roots`, audit C17) | **built (2026-09-18)** | | Typed sort keys in `PropValSub`/`ValPropSub` sort segment | not built (their sort segment is currently unused by ordering-sensitive paths) | | Cursor pagination / `hasMore` instead of exact counts | not built (wire + client change) | | Batched KV reads (one read txn per query) | not built (`KvStore` trait change; per-`get` redb txns remain) | @@ -157,6 +159,33 @@ filters for every indexable atom of every commit. The registry now routes by `(drive, property)`: a filter is registered under each constraint property and its `sort_by`; only value-only filters stay in a per-drive catch-all bucket. +### Finding 7 β€” an index was trusted against evidence it could not have (fixed 2026-09-18) + +A drive-scoped, sorted, class-filtered table query returned 5 of 22 rows +while the store, the unscoped shape and the unsorted shape all had 22 +(`planning/silent-failures.md`). Two things were wrong, neither of them +visible: `resolve_query_member` hid any member whose class extender check +errored (a `String`-encoded `isA` did that, for every row so encoded), and +the planner's premise β€” that any one constraint's `PropValSub` entries are a +superset of the members β€” failed for values whose encoding produced a +different index key (a `String` holding the JSON array of a `ResourceArray`). +A build from that constraint filed a partial list; being watched, the list +was trusted from then on, and every count it produced looked plausible. + +Now: the reference strings a value is indexed under and the strings the +matcher compares come from one helper (`Value::to_reference_index_strings`, +`Value::contains_value`), so index and matcher cannot disagree; an extender +that cannot decide is skipped with a warning, never a reason to drop a row; +a first build walks each other equality constraint whose entries fit under +`PLANNER_SCAN_CAP` (at most 512 extra key reads per constraint, once per +filter registration β€” the hot path is untouched), warns with the filter and +the missing subjects, and files them (`Db::cross_check_first_build`); and +`Db::check_query_index(query)` compares a member index with a full scan of +the store and reports the missing and stale subjects, for tests and +diagnostics. The build and the commit path also compare a DID resource's +`drive` stamp with the root its filter resolves to (`filter_drive_roots`), +so a DID row in another drive no longer enters a filter it merely matches. + ## Empirical numbers (pre-rework reference) Isolated measurements against a live server (fixed overhead vs. per-member diff --git a/planning/security-audit-2026-09.md b/planning/security-audit-2026-09.md index 5648413eb..28e4d6a71 100644 --- a/planning/security-audit-2026-09.md +++ b/planning/security-audit-2026-09.md @@ -93,7 +93,7 @@ Fix: for `/agents/{key}` subjects, require the key in the path to equal the auth - **C14. Auth signature fallback accepts a path-only message.** Verified. `authentication.rs:51-87`; a signature over `"/ "` is valid on every host and tenant for 5 minutes and bypasses the WS challenge binding for clients that sign that way. - **C15. TLS certificates only checked at startup; ACME code panics (`todo!()`, `unwrap`).** Confirmed. `serve.rs:474-479`, `https.rs:64-86,249,269`. - **C16. Process-global import flags race across connections.** Confirmed. `ws_apply.rs:10,30`, `peer.rs:1186-1195`. **Fixed 2026-09-18:** see H. -- **C17. DID resources leak into other drives' watched queries.** Likely. `query_index.rs:479-482,382-445`. Read rights still apply at query time. +- **C17. DID resources leak into other drives' watched queries.** Likely. `query_index.rs:479-482,382-445`. Read rights still apply at query time. **Fixed 2026-09-18:** see section H, third round. - **C18. Every commit is written with `Durability::None`.** Confirmed. `redb_store.rs:477,567` (`// EXPERIMENT`). Acknowledged writes are lost on crash until the next flush. **Amended 2026-09-15:** the server and desktop run a 100ms durable-flush tick (`serve.rs`) and the WASM worker flushes on a tick, so the loss window there is 100ms. The Flutter binding opened the same redb store directly and never flushed except in `set_active_drive`, so on Android every edit since the last drive switch was lost on app kill. **Fixed 2026-09-15:** `Db::init_redb_file` owns the flush tick, so every file-backed binding gets the same 100ms bound; idle ticks no longer write. See [`atomic-lib-runtime.md`](./atomic-lib-runtime.md). - **C19. Unbounded response body on the bookmark fetch.** Confirmed. `client/helpers.rs:355-366`, `bookmark.rs:47`; SSRF guard is present, size cap is not. - **C20. Client-stamped `drive` trusted when the parent is not materialized.** Verified 2026-09-18, not exploitable as described. The stamp is only ever a rights *lookup target*, never a grant: with the parent materialized, `check_append` checks `Append` on the parent and `commit.rs` re-derives the stamp from the parent, discarding the client's; with the parent absent, `hierarchy.rs` (`check_append`, `did:` branch) accepts the stamp only if the signer has `Append` on the stamped drive *and* that drive exists here, so stamping a victim's drive is refused and stamping your own drive gains nothing (fan-out routes to your own subscribers). Residual, hygiene rather than rights: a child created while its parent was absent keeps the stamp it was created with when the parent arrives later (`parent_changed` does not fire), so the drive stamp can disagree with the parent's drive until the resource is next moved. Not fixed here. @@ -242,9 +242,12 @@ Fixed in the second round on the same branch: - **D**: `Agent` and `SharedConfig` redact secrets in `Debug`; genesis certs with a `parent`/`drive` must match the document; `/plugin-list` and the plugin UI files are read as the calling agent (the data-browser signs that request) and the `plugin` parameter is validated; `default_service` logs at debug; the desktop `devtools` feature is opt-in. - **Dependencies**: `wasmtime` 47.0.4 (sandbox escape), `h2` 0.4.19, `quinn-proto`, `rustls-webpki` 0.103.13; `@tiptap/*` 3.30, `@modelcontextprotocol/sdk` 1.30 and the transitive build tools, after which `pnpm audit --prod` reports nothing. Still open: `h2 0.3` (via actix-http 3, no patched 0.3 line) and `rustls-webpki 0.102` / `hickory-proto 0.25` (via iroh 0.35; an iroh upgrade is the only route). +Fixed in the third round (`claude/technical-debt-analysis-b4ifz2-query-index`, 2026-09-18): + +- **C17** (DID resources in other drives' watched queries): a first version compared a resource's `drive` stamp with the query's `drive`, but for a `did:` query `QueryFilter.drive` is `drive_prefix_from_subject` of the queried subject, which is that subject itself rather than the drive root, so legitimate rows were excluded and the `query_aggregates` tests failed. That version was reverted. The fix now carries the drive root beside the filter: when a filter is registered (`Db::register_watched_query`, and `populate_watched_queries_cache` at open) the `Db` resolves its `drive` subject's stored `drive` stamp (or the subject itself when it is a Drive) into `filter_drive_roots`, keyed by the filter's `drive` string, so the filter's identity and encoding are unchanged. `Db::filter_accepts_resource_drive` then compares a DID resource's stamp with that root on the commit path (`check_if_atom_matches_watched_query_filters`) and in the index build (`query_complex`). A resource without a stamp, or a filter whose subject is not stored here, is not excluded β€” a missing stamp is no evidence, and read rights still apply at query time. Regression test `did_rows_stamped_into_another_drive_stay_out_of_a_watched_query`; the `query_aggregates` tests pass. + Not fixed here: -- **C17** (DID resources in other drives' watched queries): a first version compared a resource's `drive` stamp with the query's `drive`, but for a `did:` query `QueryFilter.drive` is `drive_prefix_from_subject` of the queried subject, which is that subject itself rather than the drive root, so legitimate rows were excluded and the `query_aggregates` tests failed. Reverted. A correct fix has to carry the drive root in the filter (resolve the queried subject's `drive` stamp when the filter is registered) and only then compare stamps, or keep the constraint check as the scope and rely on read rights at query time, which still apply. - **B7** (desktop CSP disabled): a CSP is set since 2026-09-15 (`desktop/tauri.conf.json`: `script-src 'self' 'wasm-unsafe-eval'`, `style-src 'self' 'unsafe-inline'` plus Google Fonts for styled-components, `connect-src` to any http/https/ws/wss server plus the Tauri IPC origins, workers from the bundle or blobs, plugin iframes over http/https, `object-src 'none'`), with Tauri's own hashing of the bundle's inline scripts left on. It mirrors what the web build already runs under (`single_page_app.rs` nonce CSP) and was validated against the Tauri schema, not yet by a packaged desktop run; smoke-test a release build (sign-in, document editing, image upload, plugin view) before shipping. The `devtools` feature is opt-in. - **C18** (`Durability::None`), **C24** (loopback NFS): structural changes with performance or design trade-offs, listed above with the fix direction. - The `stringToSlug` duplicate stays: the data-browser copy fixes a case (`Meat & fish`) the lib copy gets wrong; port the fix into lib first. diff --git a/planning/silent-failures.md b/planning/silent-failures.md index d5e01e945..bbe8b003b 100644 --- a/planning/silent-failures.md +++ b/planning/silent-failures.md @@ -171,7 +171,7 @@ errored. that, a rejected write should surface where the person who caused it is looking. *Fixed 2026-08-18 (ecaa4a63).* -**A client renders a stale row set and never reconciles with its server.** *(open)* +**A client renders a stale row set and never reconciles with its server.** *(fixed, see below)* The Houseplants table showed 22+ rows on desktop and 15 in the browser against Home Assistant. It looked exactly like a sync failure, and the peer log supported that reading: `SYNC_DIFF: server pushes 0, server pulls 1` β€” "I have @@ -195,13 +195,22 @@ with `lastCommit` the peer "did not have". All three were consistent with the wrong conclusion. What settled it was querying both servers directly and getting 24 = 24. Measure the thing itself before believing a story that explains the symptoms. +*Fixed 2026-09-18 (`claude/technical-debt-analysis-b4ifz2-query-index`, see +the next entry for the mechanism).* The browser answers collections from the +same `atomic_lib` `Db` the server runs, so this was the same index: rows the +index held but the read hid. `Db::check_query_index` now compares a query's +member index with a scan of the store and names the missing and stale +subjects; the index build cross-checks the constraints it did not scan and +warns when its candidate index came up short. The UI does not yet compare its +local answer with the server's β€” a disagreement between two nodes still needs +the check run on each. **`fetchResourceHTTP(url, {agent})` silently ignores the option.** Returns `Unauthorized` rather than either signing the request or rejecting an unknown option. Reads as a permissions problem, is actually a typo-shaped API gap. -**A query index silently disagreed with the data it indexes.** *(open)* +**A query index silently disagreed with the data it indexes.** *(fixed)* The Houseplants table rendered 5 rows on the desktop node and 22 on Home Assistant. Every resource was present on both, and every probe said they were converged β€” because the probes were reconstructions of the table's query rather @@ -226,6 +235,25 @@ the real query in exactly the parameters that break. Capture what the application actually sends (patch the fetch, read the server's access log) before comparing anything. "Both servers agree" is worthless if the question is not the one the product asks. +*Fixed 2026-09-18 (`claude/technical-debt-analysis-b4ifz2-query-index`).* Two +causes, both in the read that only this query shape takes. (1) Rows whose +`isA` read back as a plain `String` (one of four encodings that name the same +class) were *in* the index and *were* read, then hidden: the built-in +collection class extender did `to_subjects()?` on the value, and +`resolve_query_member` treated any extender error as "drop the row" β€” from the +page and from `totalMembers`. An extender that cannot decide now logs a +warning and is skipped; the row is listed. (2) Rows whose `isA` was a `String` +holding the JSON array were never candidates, because the index keyed them by +the literal `["…"]`, and the matcher rejected them for the same reason; both +now read the array's elements, from one helper, so they cannot disagree. What +should have shouted now does: `Db::check_query_index(query)` reports the +difference between an index and the store, and a first build that finds +members through a constraint the planner did not scan says so in the log and +files them (`lib/src/db.rs`, `cross_check_first_build`). Regression tests: +`is_a_encodings_all_match_the_class_constraint`, +`replicated_rows_reach_a_watched_scoped_sorted_query`, +`first_build_cross_checks_the_unscanned_constraint`, +`check_query_index_names_missing_and_stale_members` in `lib/src/db/test.rs`. ## Environment and deployment From 51a009fb8b60264d2e88f89dd3c837b1f6556c2d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 18 Sep 2026 22:14:11 +0000 Subject: [PATCH 016/110] feat(server,lib): startup store-size diagnostics and automatic redb compaction Opening the file store now logs its size (on disk and length) and open duration, a warning above 1 GiB naming `atomic-server compact`, and redb's `DatabaseStats` breakdown: allocated/leaf/branch pages, stored, metadata and fragmented bytes, and the reclaimable bytes (on disk minus pages in use, which is what `compact()` gives back; redb's sparse growth headroom does not count). When the file is at least 256 MiB on disk and at least 30% of it is dead, `Db::init_redb_file` compacts it on the handle it already holds, before any table is read and before the server listens, logs before/after sizes and duration, and keeps the outcome in `Tree::PluginMeta` (`Db::last_compaction`). Any failure is a warning; startup never fails on it. Not applied to the OPFS or sled backends. Server flags `--auto-compact` (`ATOMIC_AUTO_COMPACT`, default true), `--auto-compact-min-mb` (256) and `--auto-compact-min-reclaimable-percent` (30) build the `CompactionPolicy`; library callers pass one to `Db::init_redb_file_with_policy`. Documented in the installation guide. Test: a real redb file churned through the same open path (doubling overwrites plus throwaway resources deleted mid-file) is compacted on reopen, gives back most of the measured free space, reads every kept resource back, keeps the record across the next open, and is left alone with the policy disabled. Config tests cover the flags. Also fixes a pre-existing `unnecessary_to_owned` lint in `db::website` tests so `clippy -D warnings` passes. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01H1U7TNdjwqXKv8MQYproiq --- CHANGELOG.md | 16 + TESTING_COVERAGE.md | 12 + docs/src/atomicserver/faq.md | 2 + docs/src/atomicserver/installation.md | 71 ++ lib/src/db.rs | 64 +- lib/src/db/compaction.rs | 637 ++++++++++++++++++ lib/src/db/redb_store.rs | 50 +- planning/README.md | 2 +- ...sk-storage-and-persistence-optimization.md | 83 ++- server/src/appstate.rs | 5 +- server/src/config.rs | 92 +++ 11 files changed, 1002 insertions(+), 32 deletions(-) create mode 100644 lib/src/db/compaction.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d4215af2..31e94594f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -90,6 +90,22 @@ See [STATUS.md](server/STATUS.md) to learn more about which features will remain longer enter a drive's watched query on commit or build: the `Db` resolves each watched filter's `drive` subject to its drive root and compares stamps (security audit C17; the filter's identity is unchanged). +- Startup store-size diagnostics and automatic redb compaction. Opening the + store now logs the file's size and open duration (a warning above 1 GiB, + naming `atomic-server compact`) and, from redb's `DatabaseStats`, how much + of the file is live, fragmented and reclaimable (bytes on disk minus pages + in use; redb's sparse growth headroom does not count). When the file is at + least 256 MiB on disk and at least 30% of that is dead, `Db::init_redb_file` + compacts it before the server listens and logs the before/after sizes and + duration; the outcome is kept in the store and readable through + `Db::last_compaction`. A compaction failure is a warning, never a refused + start. New options `--auto-compact` (`ATOMIC_AUTO_COMPACT`, default + `true`), `--auto-compact-min-mb` (`ATOMIC_AUTO_COMPACT_MIN_MB`, `256`) and + `--auto-compact-min-reclaimable-percent` + (`ATOMIC_AUTO_COMPACT_MIN_RECLAIMABLE_PERCENT`, `30`); `atomic_lib` callers + pass a `db::compaction::CompactionPolicy` to + `Db::init_redb_file_with_policy`. Not applied to the OPFS (browser) or sled + backends. See `planning/disk-storage-and-persistence-optimization.md`. - CI: the `:develop` docker image is published by its own job instead of a step tacked onto the end of the CI job. As a step it inherited whatever the diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index 3b8406c60..9b5f64a33 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -960,6 +960,18 @@ mounts without resetting or re-registering the global parser. - `client-db.worker.test.ts` requires vault cursor commits to flush before the worker acknowledges backup completion, and propagates flush failures. The SaaS `vault-refresh.spec.ts` checks stored objects and bytes across reloads. +- `db::compaction::tests::startup_compaction_shrinks_a_bloated_store_and_keeps_every_resource` + (`cargo test -p atomic_lib --features db-redb --lib`) churns a real redb + file through `Db::init_redb_file_with_policy` β€” overwrites that double in + size plus throwaway resources deleted mid-file, so the buddy allocator + cannot reuse the holes β€” and reopens it: the policy compacts, the file + gives back most of the measured free space, every kept resource reads its + last value, the record survives the next open, and a disabled policy leaves + the file byte-for-byte alone. Overwrites *alone* leave only ~20% dead + (freed blocks coalesce and get reused), which is why the test deletes. + `server::config::tests` cover the `--auto-compact*` flags. Not covered: + compaction of a store another process holds open (the open itself fails + first, as before), and the cost of `DatabaseStats` on a multi-GB file. - `synthetic_agent_reads_have_stable_history_without_persisting` checks that fallback agent lookups neither invent creation timestamps nor generate new CRDT history or persist a resource merely by reading it. diff --git a/docs/src/atomicserver/faq.md b/docs/src/atomicserver/faq.md index 7d31183aa..04d87a181 100644 --- a/docs/src/atomicserver/faq.md +++ b/docs/src/atomicserver/faq.md @@ -72,4 +72,6 @@ Make sure AtomicServer is not publicly accessible, because this will allow anyon It depends on your operating system, because some data is _temporary_, others are _configuration files_, and so forth. Run `atomic-server show-config` to see the used paths. You can overwrite these if you want, see `--help`. +The store itself is a single file that only ever grows unless it is compacted; see [Store size and automatic compaction](installation.md#store-size-and-automatic-compaction) if it is larger than your data. + diff --git a/docs/src/atomicserver/installation.md b/docs/src/atomicserver/installation.md index 66879a1b1..5fd075955 100644 --- a/docs/src/atomicserver/installation.md +++ b/docs/src/atomicserver/installation.md @@ -340,6 +340,56 @@ systemctl restart atomic journalctl -u atomic.service --since "1 hour ago" -f ``` +## Store size and automatic compaction + +The store is one [redb](https://github.com/cberner/redb) file, +`/store/atomic.redb`. redb never gives pages back to the filesystem +on its own: an overwrite or delete frees pages that later writes may reuse, +but the file only shrinks when its *tail* happens to be free. A store that has +churned through many resources (long editing sessions, imports that were +deleted again, test data) stays as large as its high-water mark, and opening +a large file is slow: the open-time `fsync` scales with file size, so a +multi-GB store can take a minute to open where a fresh one takes +milliseconds. + +Every boot logs the file's size and open time, and how much of it is dead: + +``` +Store /var/lib/atomic/store/atomic.redb is 1843.2 MiB on disk (2048.0 MiB long), opened in 21.4s +Store pages: 224511 allocated (...) = 877.0 MiB live; ... 966.2 MiB (52%) reclaimable by compaction +``` + +"On disk" is what the file costs; the length also includes headroom redb +reserves ahead of writes, which is sparse and costs nothing until used. +Reclaimable space is what is on disk minus the pages still in use. Above +1 GiB on disk the first line is a warning. When both of these hold, the +server compacts the file **before** it starts listening: + +- the file is at least `--auto-compact-min-mb` on disk + (`ATOMIC_AUTO_COMPACT_MIN_MB`, default `256`): a small file opens fast + whatever its layout, so a startup pause is not worth it; +- at least `--auto-compact-min-reclaimable-percent` + (`ATOMIC_AUTO_COMPACT_MIN_RECLAIMABLE_PERCENT`, default `30`) of what is on + disk is reclaimable: compaction reads and rewrites the whole file, and a + store with steady churn always carries some free pages that the next writes + reuse. + +Compaction logs the before/after sizes and how long it took. It is +`O(file size)`, seconds per GB on an SSD, and it runs while the server holds +the store's exclusive lock, so the server is not reachable until it is done. +If it fails for any reason the store opens as it was and the failure is a +warning, never a refused start. Set `--auto-compact false` +(`ATOMIC_AUTO_COMPACT=false`) to turn it off, for example on a node where +every second of downtime matters and you would rather schedule it. That also +skips the reclaimable-space measurement, which walks every page of the file. +The size and open-time line is logged either way. + +`atomic-server compact` does the same by hand, with the server stopped. Use +it after a large clean-up, or when the boot log says a store below the size +floor has a lot to reclaim. + +Neither applies to the browser (OPFS) store or to a legacy sled store. + ## AtomicServer CLI options / ENV vars (run `atomic-server --help` to see the latest options) @@ -360,6 +410,8 @@ Commands: Returns the currently selected options, based on the passed flags and parsed environment variables reset Danger! Removes all data from the store + compact + Compact the on-disk redb file (rebuilds page layout, truncates dead-page tail). Slow β€” typically minutes on a multi-GB store β€” but makes future boots dramatically faster because the open-time `fsync` cost scales with file size. Server MUST be stopped: redb takes an exclusive file lock and will fail otherwise help Print this message or the help of the given subcommand(s) @@ -386,6 +438,25 @@ Options: [env: ATOMIC_ENVELOPE_RETENTION=] [default: latest] + --auto-compact + Compact the store file at startup when it is at least `--auto-compact-min-mb` and at least `--auto-compact-min-reclaimable-percent` of it is dead space (pages freed by overwrites and deletes that redb never returns to the filesystem). The server listens only when the compaction is done: expect seconds per GB. `false` skips the check; `atomic-server compact` does the same by hand while the server is stopped + + [env: ATOMIC_AUTO_COMPACT=] + [default: true] + [possible values: true, false] + + --auto-compact-min-mb + Store files smaller than this many MiB are never compacted at startup: they open fast whatever their layout + + [env: ATOMIC_AUTO_COMPACT_MIN_MB=] + [default: 256] + + --auto-compact-min-reclaimable-percent + Startup compaction runs only when at least this percentage of the store file is reclaimable (0-100) + + [env: ATOMIC_AUTO_COMPACT_MIN_RECLAIMABLE_PERCENT=] + [default: 30] + --write-rate-limit How many write requests per minute one signed agent may make (commits over HTTP or WebSocket, uploads, blob puts, peer sync pushes, resource posts) before the server answers `429 Too Many Requests`. The budget refills continuously, so a burst up to this size is fine. `0` disables the limit diff --git a/lib/src/db.rs b/lib/src/db.rs index 30fde0cf4..d9b8af99a 100644 --- a/lib/src/db.rs +++ b/lib/src/db.rs @@ -4,6 +4,8 @@ pub mod app_agent; pub mod blob_backend; pub mod btreemap_store; +#[cfg(all(feature = "db-redb", not(target_arch = "wasm32")))] +pub mod compaction; mod encoding; #[cfg(feature = "db-redb")] pub mod encrypted_backend; @@ -810,11 +812,36 @@ impl Db { /// Creates a Db backed by redb with file-based persistent storage. /// Works on all native targets (not WASM β€” use init_redb_opfs for that). + /// + /// Runs the default [`compaction::CompactionPolicy`] on the file before + /// serving it; `init_redb_file_with_policy` takes another. #[cfg(all(feature = "db-redb", not(target_arch = "wasm32")))] pub async fn init_redb_file( path: &std::path::Path, base_domain: Option, uploads_path: &std::path::Path, + ) -> AtomicResult { + Self::init_redb_file_with_policy( + path, + base_domain, + uploads_path, + &compaction::CompactionPolicy::default(), + ) + .await + } + + /// `init_redb_file` with an explicit startup compaction policy. The + /// file's size and open duration are logged either way; with the policy + /// enabled the reclaimable space is measured and, past both thresholds, + /// compacted before any table is read. A compaction failure (including + /// redb refusing because of a live transaction) is logged and the store + /// opens as it was; it never fails startup. + #[cfg(all(feature = "db-redb", not(target_arch = "wasm32")))] + pub async fn init_redb_file_with_policy( + path: &std::path::Path, + base_domain: Option, + uploads_path: &std::path::Path, + policy: &compaction::CompactionPolicy, ) -> AtomicResult { tracing::info!("Opening ReDB database at {:?}", path); @@ -874,7 +901,8 @@ impl Db { #[cfg(not(feature = "db-sled"))] let _ = uploads_path; - let redb_store = redb_store::RedbStore::new_file(&redb_path)?; + let (redb_store, compaction) = + redb_store::RedbStore::new_file_with_policy(&redb_path, policy)?; let store = Db { path: path.to_path_buf(), @@ -900,6 +928,14 @@ impl Db { store.add_class_extender(crate::collections::get_collection_class_extender())?; + // Bookkeeping only: the compaction itself already happened, and + // failing to note it must not refuse the start. + if let Ok(record) = compaction { + if let Err(e) = store.record_compaction(&record) { + tracing::warn!("Could not record the startup compaction: {e}"); + } + } + store.populate_watched_queries_cache()?; crate::populate::bootstrap(&store) .await @@ -909,6 +945,32 @@ impl Db { Ok(store) } + /// Persist the outcome of a startup compaction so `last_compaction` + /// can report it after the log line is gone. Durable at once: the + /// compaction it describes already cost seconds, a 100 ms flush + /// window is nothing next to it. + #[cfg(all(feature = "db-redb", not(target_arch = "wasm32")))] + fn record_compaction(&self, record: &compaction::CompactionRecord) -> AtomicResult<()> { + let bytes = serde_json::to_vec(record) + .map_err(|e| format!("Could not serialize the compaction record: {e}"))?; + self.kv + .insert(trees::Tree::PluginMeta, compaction::RECORD_KEY, &bytes)?; + self.kv.flush() + } + + /// The most recent automatic startup compaction of this store's file, + /// if one ever ran. `None` on a store that was never compacted at + /// startup, and on the in-memory, OPFS and sled backends. + #[cfg(all(feature = "db-redb", not(target_arch = "wasm32")))] + pub fn last_compaction(&self) -> Option { + let bytes = self + .kv + .get(trees::Tree::PluginMeta, compaction::RECORD_KEY) + .ok() + .flatten()?; + serde_json::from_slice(&bytes).ok() + } + /// Make `Durability::None` commits durable on a fixed cadence. /// /// Every redb write skips the fsync for throughput and only becomes diff --git a/lib/src/db/compaction.rs b/lib/src/db/compaction.rs new file mode 100644 index 000000000..174647673 --- /dev/null +++ b/lib/src/db/compaction.rs @@ -0,0 +1,637 @@ +//! Startup store-size diagnostics and automatic compaction for the +//! file-backed redb store. +//! +//! redb never gives dead pages back to the filesystem on its own: every +//! overwrite and delete frees pages that later writes may reuse, but the +//! file only shrinks when the *tail* of it happens to be free, so a store +//! that has churned through many resources stays as large as its high-water +//! mark. `Database::create` then pays for that size on every boot (see +//! `planning/disk-storage-and-persistence-optimization.md`). This module +//! measures the file on open, says how much of it is dead, and reclaims it +//! when the policy says it is worth the one-off pause. +//! +//! Only the native file backend uses it: the WASM/OPFS store never reaches +//! `RedbStore::new_file`, and the sled backend has its own lifecycle. + +use std::time::{Duration, Instant}; + +use redb::Database; +use serde::{Deserialize, Serialize}; + +use crate::errors::AtomicResult; + +pub const MIB: u64 = 1024 * 1024; + +/// Above this file size the open-time log line is a warning that names +/// `atomic-server compact`, whatever the reclaimable fraction: at 1 GiB the +/// open alone is measured in seconds on macOS. +pub const LARGE_STORE_WARN_BYTES: u64 = 1024 * MIB; + +/// When `Db::init_redb_file` compacts the store before serving it. +/// +/// Both thresholds must hold: a file smaller than `min_file_bytes` is cheap +/// to open however wasteful it is, and a large file whose free space is +/// below `min_reclaimable_fraction` would cost an O(file) compaction for +/// little gain (a store with steady churn always carries some free pages +/// that the next writes reuse). +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct CompactionPolicy { + /// `false` skips the reclaimable-space measurement and the compaction. + /// The file size and open duration are still logged. + pub enabled: bool, + /// Files below this size are never compacted automatically. + pub min_file_bytes: u64, + /// Compact when `free_bytes / file_bytes` reaches this (0.0 – 1.0). + pub min_reclaimable_fraction: f64, +} + +impl Default for CompactionPolicy { + fn default() -> Self { + Self { + enabled: true, + min_file_bytes: 256 * MIB, + min_reclaimable_fraction: 0.30, + } + } +} + +impl CompactionPolicy { + pub const fn disabled() -> Self { + Self { + enabled: false, + min_file_bytes: u64::MAX, + min_reclaimable_fraction: 1.0, + } + } +} + +/// Bytes the file occupies on disk, as opposed to its length. redb grows the +/// file with `set_len`, by doubling below 4 GiB and by whole regions above, +/// which leaves a sparse, never-written tail on every common filesystem. +/// That tail is headroom the next writes fill, not dead data, and the length +/// alone would call a freshly grown store half reclaimable. Dead pages were +/// written, so they do occupy blocks. Falls back to the length where the +/// platform reports no block count. +pub fn disk_usage(meta: &std::fs::Metadata) -> u64 { + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + meta.blocks().saturating_mul(512).min(meta.len()) + } + #[cfg(not(unix))] + { + meta.len() + } +} + +/// `(length, bytes on disk)` of `path`, both 0 when it cannot be read. +pub fn file_sizes(path: &std::path::Path) -> (u64, u64) { + match std::fs::metadata(path) { + Ok(meta) => (meta.len(), disk_usage(&meta)), + Err(_) => (0, 0), + } +} + +/// What a redb file holds, measured right after open. +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct StoreFileStats { + /// `metadata(path).len()`, including redb's sparse growth headroom. + pub file_bytes: u64, + /// See [`disk_usage`]; the size the thresholds are applied to. + pub disk_bytes: u64, + pub page_size: u64, + /// Pages the allocator considers in use, including redb's own tables. + pub allocated_pages: u64, + pub leaf_pages: u64, + pub branch_pages: u64, + /// Bytes of keys and values (`DatabaseStats::stored_bytes`). + pub stored_bytes: u64, + /// Branch keys and other index overhead (`DatabaseStats::metadata_bytes`). + pub metadata_bytes: u64, + /// Slack inside allocated pages (`DatabaseStats::fragmented_bytes`). + /// Compaction moves whole pages, so this is *not* reclaimable by it. + pub fragmented_bytes: u64, + /// `DatabaseStats` walks every B-tree page, so this is O(file) and + /// worth knowing. + pub measured_in: Duration, +} + +impl StoreFileStats { + /// Walks the whole store; see `measured_in`. redb exposes stats on a + /// write transaction only; it is aborted, nothing is written. + pub fn collect(db: &Database, path: &std::path::Path) -> AtomicResult { + let (file_bytes, disk_bytes) = file_sizes(path); + let t = Instant::now(); + let tx = db + .begin_write() + .map_err(|e| format!("redb stats tx: {e}"))?; + let stats = tx.stats().map_err(|e| format!("redb stats: {e}"))?; + tx.abort().map_err(|e| format!("redb stats abort: {e}"))?; + Ok(Self { + file_bytes, + disk_bytes, + page_size: stats.page_size() as u64, + allocated_pages: stats.allocated_pages(), + leaf_pages: stats.leaf_pages(), + branch_pages: stats.branch_pages(), + stored_bytes: stats.stored_bytes(), + metadata_bytes: stats.metadata_bytes(), + fragmented_bytes: stats.fragmented_bytes(), + measured_in: t.elapsed(), + }) + } + + pub fn allocated_bytes(&self) -> u64 { + self.allocated_pages.saturating_mul(self.page_size) + } + + /// Bytes on disk that no live page occupies: dead pages `compact()` can + /// move out of the way and truncate. + pub fn free_bytes(&self) -> u64 { + self.disk_bytes.saturating_sub(self.allocated_bytes()) + } + + /// `free_bytes / disk_bytes`, 0.0 for an empty file. + pub fn reclaimable_fraction(&self) -> f64 { + if self.disk_bytes == 0 { + 0.0 + } else { + self.free_bytes() as f64 / self.disk_bytes as f64 + } + } +} + +/// Why the startup policy did not compact. +#[derive(Debug, Clone, PartialEq)] +pub enum Skip { + Disabled, + /// The file is below `CompactionPolicy::min_file_bytes`. + TooSmall, + /// The reclaimable fraction is below the policy's threshold. + NotWorthIt, + /// Measuring or compacting failed; the store is still usable. + Failed(String), +} + +impl CompactionPolicy { + /// Whether these stats meet both thresholds. + pub fn decide(&self, stats: &StoreFileStats) -> Result<(), Skip> { + if !self.enabled { + return Err(Skip::Disabled); + } + if stats.disk_bytes < self.min_file_bytes { + return Err(Skip::TooSmall); + } + if stats.reclaimable_fraction() < self.min_reclaimable_fraction { + return Err(Skip::NotWorthIt); + } + Ok(()) + } +} + +/// Outcome of one compaction. `Db::init_redb_file` stores the latest one +/// under `Tree::PluginMeta` (`RECORD_KEY`) so it can be read back after the +/// log line has scrolled away; `Db::last_compaction` returns it. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct CompactionRecord { + /// Unix time in milliseconds at which the compaction finished. + pub compacted_at_ms: u64, + /// File length before and after. + pub bytes_before: u64, + pub bytes_after: u64, + /// Bytes on disk before and after (see [`disk_usage`]). + pub disk_bytes_before: u64, + pub disk_bytes_after: u64, + /// `free_bytes` measured before compacting. + pub free_bytes_before: u64, + pub duration_ms: u64, + /// `false` when redb found nothing to move (the file was already dense). + pub did_compact: bool, +} + +pub const RECORD_KEY: &[u8] = b"store_compaction_last"; + +/// `Database::compact` with an `AtomicResult`. The caller owns the only +/// handle, so no read transaction can be live. +pub fn compact_database(db: &mut Database) -> AtomicResult { + db.compact() + .map_err(|e| format!("Compaction failed: {e}").into()) +} + +fn mib(bytes: u64) -> f64 { + bytes as f64 / MIB as f64 +} + +/// Log what the just-opened file looks like and compact it if the policy +/// says so. Never fails: a measurement or compaction error is a warning and +/// the store opens as it was. `open_duration` is how long `Database::create` +/// took, logged next to the size it scales with. +pub fn run_startup_policy( + db: &mut Database, + path: &std::path::Path, + open_duration: Duration, + policy: &CompactionPolicy, +) -> Result { + let (file_bytes, disk_bytes) = file_sizes(path); + if disk_bytes >= LARGE_STORE_WARN_BYTES { + tracing::warn!( + "Store {} is {:.1} MiB on disk ({:.1} MiB long) and took {:?} to open; open time \ + grows with file size. `atomic-server compact` reclaims dead pages while the \ + server is stopped.", + path.display(), + mib(disk_bytes), + mib(file_bytes), + open_duration + ); + } else { + tracing::info!( + "Store {} is {:.1} MiB on disk ({:.1} MiB long), opened in {:?}", + path.display(), + mib(disk_bytes), + mib(file_bytes), + open_duration + ); + } + + if !policy.enabled { + tracing::info!( + "Automatic store compaction is disabled; skipping the reclaimable-space measurement" + ); + return Err(Skip::Disabled); + } + + let stats = match StoreFileStats::collect(db, path) { + Ok(stats) => stats, + Err(e) => { + tracing::warn!("Could not measure the store file: {e}"); + return Err(Skip::Failed(e.to_string())); + } + }; + tracing::info!( + "Store pages: {} allocated ({} leaf, {} branch) Γ— {} B = {:.1} MiB live; \ + {:.1} MiB stored + {:.1} MiB metadata + {:.1} MiB fragmented; \ + {:.1} MiB ({:.0}%) reclaimable by compaction (measured in {:?})", + stats.allocated_pages, + stats.leaf_pages, + stats.branch_pages, + stats.page_size, + mib(stats.allocated_bytes()), + mib(stats.stored_bytes), + mib(stats.metadata_bytes), + mib(stats.fragmented_bytes), + mib(stats.free_bytes()), + 100.0 * stats.reclaimable_fraction(), + stats.measured_in + ); + + match policy.decide(&stats) { + Ok(()) => {} + Err(skip) => { + if stats.free_bytes() >= policy.min_file_bytes / 4 + && stats.reclaimable_fraction() >= policy.min_reclaimable_fraction + { + // Big enough to matter and mostly dead, but under the size + // floor: say so, once, in case the operator wants it now. + tracing::info!( + "{:.1} MiB reclaimable but the store is below the {:.0} MiB automatic \ + compaction floor; run `atomic-server compact` to reclaim it now", + mib(stats.free_bytes()), + mib(policy.min_file_bytes) + ); + } + return Err(skip); + } + } + + tracing::warn!( + "Compacting store {} before serving: {:.1} MiB on disk of which {:.1} MiB ({:.0}%) is \ + reclaimable. This is O(file size); the server listens when it finishes. \ + Set ATOMIC_AUTO_COMPACT=false to skip.", + path.display(), + mib(disk_bytes), + mib(stats.free_bytes()), + 100.0 * stats.reclaimable_fraction() + ); + let t = Instant::now(); + let did_compact = match compact_database(db) { + Ok(did) => did, + Err(e) => { + tracing::warn!( + "Automatic store compaction failed; continuing with the store as it is: {e}" + ); + return Err(Skip::Failed(e.to_string())); + } + }; + let duration = t.elapsed(); + let (bytes_after, disk_bytes_after) = file_sizes(path); + let saved = disk_bytes.saturating_sub(disk_bytes_after); + tracing::info!( + "{} in {:?}: {:.1} MiB β†’ {:.1} MiB on disk (saved {:.1} MiB, {:.0}%), \ + file length {:.1} MiB β†’ {:.1} MiB", + if did_compact { + "Compacted store" + } else { + "Store already dense, nothing moved" + }, + duration, + mib(disk_bytes), + mib(disk_bytes_after), + mib(saved), + if disk_bytes > 0 { + 100.0 * saved as f64 / disk_bytes as f64 + } else { + 0.0 + }, + mib(file_bytes), + mib(bytes_after) + ); + Ok(CompactionRecord { + compacted_at_ms: std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_millis() as u64) + .unwrap_or(0), + bytes_before: file_bytes, + bytes_after, + disk_bytes_before: disk_bytes, + disk_bytes_after, + free_bytes_before: stats.free_bytes(), + duration_ms: duration.as_millis() as u64, + did_compact, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn stats(file_bytes: u64, allocated_pages: u64) -> StoreFileStats { + StoreFileStats { + file_bytes, + disk_bytes: file_bytes, + page_size: 4096, + allocated_pages, + leaf_pages: allocated_pages, + branch_pages: 0, + stored_bytes: 0, + metadata_bytes: 0, + fragmented_bytes: 0, + measured_in: Duration::ZERO, + } + } + + #[test] + fn free_bytes_is_file_minus_allocated_pages() { + let s = stats(100 * 4096, 40); + assert_eq!(s.allocated_bytes(), 40 * 4096); + assert_eq!(s.free_bytes(), 60 * 4096); + assert!((s.reclaimable_fraction() - 0.6).abs() < 1e-9); + // A file shorter than its allocation (mid-write) never underflows. + assert_eq!(stats(10, 40).free_bytes(), 0); + assert_eq!(stats(0, 0).reclaimable_fraction(), 0.0); + } + + #[test] + fn sparse_growth_headroom_is_not_reclaimable() { + // A file redb just doubled: 100 pages long, 40 live, the rest a + // sparse tail. On disk it is only the written pages. + let s = StoreFileStats { + file_bytes: 100 * 4096, + disk_bytes: 42 * 4096, + ..stats(100 * 4096, 40) + }; + assert_eq!(s.free_bytes(), 2 * 4096); + assert!(s.reclaimable_fraction() < 0.05); + assert_eq!( + CompactionPolicy { + enabled: true, + min_file_bytes: 0, + min_reclaimable_fraction: 0.30, + } + .decide(&s), + Err(Skip::NotWorthIt) + ); + } + + #[test] + fn disk_usage_ignores_a_sparse_tail() { + let dir = temp_store_dir("sparse"); + let path = dir.join("sparse.bin"); + let file = std::fs::File::create(&path).unwrap(); + file.set_len(64 * MIB).unwrap(); + drop(file); + let meta = std::fs::metadata(&path).unwrap(); + assert_eq!(meta.len(), 64 * MIB); + // Tolerate a filesystem that cannot do sparse files, but on one + // that can (every CI and dev box) the tail costs nothing. + assert!(disk_usage(&meta) <= meta.len()); + if cfg!(unix) && disk_usage(&meta) < meta.len() { + assert!(disk_usage(&meta) < MIB, "{}", disk_usage(&meta)); + } + std::fs::remove_dir_all(&dir).ok(); + } + + #[test] + fn policy_needs_both_thresholds() { + let policy = CompactionPolicy { + enabled: true, + min_file_bytes: 100 * 4096, + min_reclaimable_fraction: 0.5, + }; + assert_eq!(policy.decide(&stats(100 * 4096, 40)), Ok(())); + assert_eq!( + policy.decide(&stats(99 * 4096, 1)), + Err(Skip::TooSmall), + "a small file is never compacted however wasteful" + ); + assert_eq!( + policy.decide(&stats(1000 * 4096, 600)), + Err(Skip::NotWorthIt) + ); + assert_eq!( + CompactionPolicy::disabled().decide(&stats(1000 * 4096, 0)), + Err(Skip::Disabled) + ); + } + + fn temp_store_dir(name: &str) -> std::path::PathBuf { + let dir = std::env::temp_dir().join(format!( + "atomic-compaction-{name}-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + std::fs::create_dir_all(dir.join("uploads")).unwrap(); + dir + } + + async fn open(dir: &std::path::Path, policy: &CompactionPolicy) -> crate::Db { + crate::Db::init_redb_file_with_policy(dir, None, &dir.join("uploads"), policy) + .await + .unwrap() + } + + const RESOURCES: usize = 32; + const ROUNDS: usize = 3; + /// Throwaway resources per round; deleted at the end of the fill. + const THROWAWAY: usize = 24; + + fn subject(i: usize) -> String { + format!("did:ad:compaction-test-{i}") + } + + /// The description written in `round`; the last round's value is what + /// must survive compaction. + fn payload(i: usize, round: usize) -> String { + // Doubles per round so an overwrite never fits the block it frees, + // the way a resource whose Loro snapshot grows with every edit + // leaves the file. + format!("r{i}-{round}-").repeat(512 << round) + } + + /// Churn the store the way a long test session does: the kept + /// resources are overwritten every round with a larger value, and a + /// batch of throwaway resources is written between the rounds and + /// deleted at the end. Their pages are freed mid-file, where redb's own + /// tail truncation cannot reach them, so the file keeps its high-water + /// size. Returns the file's `(length, bytes on disk)` after a clean close. + async fn fill(dir: &std::path::Path) -> (u64, u64) { + use crate::{urls, Resource, Storelike, Value}; + let store = open(dir, &CompactionPolicy::disabled()).await; + let mut throwaways = Vec::new(); + for round in 0..ROUNDS { + for j in 0..THROWAWAY { + let subject = format!("did:ad:compaction-throwaway-{round}-{j}"); + let mut r = Resource::new(subject.clone()); + r.set_unsafe( + urls::DESCRIPTION.into(), + Value::String(format!("t{round}-{j}-").repeat(3_000)), + ) + .unwrap(); + // No index update: the test is about the file, and indexing + // kilobytes of repeated text is most of the cost. + store + .add_resource_opts(&r, false, false, true) + .await + .unwrap(); + throwaways.push(subject); + } + for i in 0..RESOURCES { + let mut r = Resource::new(subject(i)); + r.set_unsafe(urls::DESCRIPTION.into(), Value::String(payload(i, round))) + .unwrap(); + store + .add_resource_opts(&r, false, false, true) + .await + .unwrap(); + } + } + for subject in &throwaways { + store + .remove_resource(&crate::Subject::from_raw(subject, None)) + .await + .unwrap(); + } + store.flush().unwrap(); + drop(store); + file_sizes(&dir.join("atomic.redb")) + } + + async fn assert_intact(store: &crate::Db) { + use crate::{urls, Storelike}; + for i in 0..RESOURCES { + let r = store + .get_resource(&crate::Subject::from_raw(&subject(i), None)) + .await + .unwrap(); + assert_eq!( + r.get(urls::DESCRIPTION).unwrap().to_string(), + payload(i, ROUNDS - 1), + "resource {i} must read back its last value" + ); + } + } + + /// The whole startup path: a store bloated by overwrites is compacted + /// when reopened through `Db::init_redb_file_with_policy`, shrinks, + /// reads back every resource, and records what it did; the same file + /// opened with the policy disabled is left alone. + #[tokio::test(flavor = "multi_thread")] + async fn startup_compaction_shrinks_a_bloated_store_and_keeps_every_resource() { + let dir = temp_store_dir("startup"); + let (bloated, bloated_disk) = fill(&dir).await; + { + // A copy for the disabled case, so both branches see the same + // bloat rather than one fill each. + let untouched = temp_store_dir("disabled"); + std::fs::copy(dir.join("atomic.redb"), untouched.join("atomic.redb")).unwrap(); + let store = open(&untouched, &CompactionPolicy::disabled()).await; + assert_eq!(store.last_compaction(), None); + assert_intact(&store).await; + drop(store); + // The open still writes (bootstrap fingerprint, flush sentinel) + // and redb may trim a few trailing free pages on those commits, + // so the size can move by a page or two either way; what must + // not have happened is the reclaim: the dead space is still there. + let path = untouched.join("atomic.redb"); + let db = Database::create(&path).unwrap(); + let still = StoreFileStats::collect(&db, &path).unwrap(); + drop(db); + assert!( + still.reclaimable_fraction() >= 0.30, + "a disabled policy must leave the dead space alone: {still:?}" + ); + assert!( + still.disk_bytes * 10 >= bloated_disk * 9, + "{still:?} vs {bloated_disk}" + ); + std::fs::remove_dir_all(&untouched).ok(); + } + + let policy = CompactionPolicy { + enabled: true, + // Far below the production floor: the test store is a few MiB. + min_file_bytes: MIB, + min_reclaimable_fraction: 0.30, + }; + let store = open(&dir, &policy).await; + let record = store + .last_compaction() + .expect("the startup policy must have compacted this store"); + assert!(record.did_compact, "{record:?}"); + assert_eq!(record.bytes_before, bloated); + assert!( + record.free_bytes_before * 10 >= record.disk_bytes_before * 3, + "the fill must leave at least the 30% the policy asks for: {record:?}" + ); + // Compaction moves live pages down and truncates the free tail; a + // few pages of allocator metadata may not move, so demand most of + // the free space back, not all of it. + assert!( + record.disk_bytes_before - record.disk_bytes_after >= record.free_bytes_before * 3 / 4, + "compaction should reclaim most of the free space: {record:?}" + ); + assert!(record.bytes_after < record.bytes_before, "{record:?}"); + assert_intact(&store).await; + + // The open that followed the compaction wrote to the store, and redb + // grew the file's length again for that (a sparse tail). That is + // headroom, not waste: the next open must not compact again, and + // must still report the compaction that did run. + drop(store); + let store = open(&dir, &policy).await; + assert_eq!(store.last_compaction(), Some(record)); + assert_intact(&store).await; + drop(store); + std::fs::remove_dir_all(&dir).ok(); + } + + #[test] + fn defaults_are_the_documented_ones() { + let d = CompactionPolicy::default(); + assert!(d.enabled); + assert_eq!(d.min_file_bytes, 256 * MIB); + assert!((d.min_reclaimable_fraction - 0.30).abs() < 1e-9); + } +} diff --git a/lib/src/db/redb_store.rs b/lib/src/db/redb_store.rs index a20b5f101..d994d6d6d 100644 --- a/lib/src/db/redb_store.rs +++ b/lib/src/db/redb_store.rs @@ -156,18 +156,34 @@ pub fn compact_file(path: &std::path::Path) -> AtomicResult<(u64, u64, bool)> { let size_before = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0); let mut db = redb::Database::create(path) .map_err(|e| format!("Failed to open redb at {}: {e}", path.display()))?; - let did_compact = db - .compact() - .map_err(|e| format!("Compaction failed: {e}"))?; + let did_compact = super::compaction::compact_database(&mut db)?; drop(db); let size_after = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0); Ok((size_before, size_after, did_compact)) } impl RedbStore { - /// Create a RedbStore backed by a file on disk. + /// Create a RedbStore backed by a file on disk. No startup compaction; + /// see `new_file_with_policy` for the path `Db::init_redb_file` takes. #[cfg(not(target_arch = "wasm32"))] pub fn new_file(path: &std::path::Path) -> AtomicResult { + Self::new_file_with_policy(path, &super::compaction::CompactionPolicy::disabled()) + .map(|(store, _)| store) + } + + /// Open (or create) the file, log its size and open duration, and run + /// the startup compaction `policy` on it before any table is touched. + /// The second value says whether a compaction ran; a failure there is a + /// `Skip::Failed`, never an error, so a store that cannot be compacted + /// still opens. + #[cfg(not(target_arch = "wasm32"))] + pub fn new_file_with_policy( + path: &std::path::Path, + policy: &super::compaction::CompactionPolicy, + ) -> AtomicResult<( + Self, + Result, + )> { // `Database::create` with defaults uses a 1 GiB cache and the // slow full-scan repair path on any unclean shutdown. On a // multi-GB store that's 40+ seconds added to every boot @@ -177,9 +193,18 @@ impl RedbStore { // next open is "almost instant" (redb transactions.rs:1246-1258 // describes the mechanism). let t = std::time::Instant::now(); - let db = Database::create(path) + let mut db = Database::create(path) .map_err(|e| format!("Failed to create redb at {}: {e}", path.display()))?; - tracing::info!("RedbStore::new_file: Database::create in {:?}", t.elapsed()); + let open_duration = t.elapsed(); + tracing::info!( + "RedbStore::new_file: Database::create in {:?}", + open_duration + ); + + // On the handle we already hold: the file lock is taken once, and + // a bloated store is not opened a second time just to measure it. + let compaction = + super::compaction::run_startup_policy(&mut db, path, open_duration, policy); // Create all tables upfront let t = std::time::Instant::now(); @@ -203,11 +228,14 @@ impl RedbStore { } tracing::info!("RedbStore::new_file: table-create tx in {:?}", t.elapsed()); - Ok(RedbStore { - db: Arc::new(db), - batch_buffer: std::sync::Mutex::new(None), - dirty: AtomicBool::new(false), - }) + Ok(( + RedbStore { + db: Arc::new(db), + batch_buffer: std::sync::Mutex::new(None), + dirty: AtomicBool::new(false), + }, + compaction, + )) } /// Create a new in-memory RedbStore. diff --git a/planning/README.md b/planning/README.md index 3b643e874..4d05dbada 100644 --- a/planning/README.md +++ b/planning/README.md @@ -133,7 +133,7 @@ browser flow; standalone recovery remains self-managed. | [`react-compiler-resource-proxy.md`](./react-compiler-resource-proxy.md) | **Partial.** Immutable read/save status hooks and the data-inspector subscription shipped; about 13% of property reads are still render-time getters, migrating one regression at a time. | | [`canvas-undo-consolidation.md`](./canvas-undo-consolidation.md) | Phase A + C landed (browser). Phase B (Flutter action-stack removal, `replace_list_items`) open and unstarted. | | [`index-performance.md`](./index-performance.md) | First tranche shipped. Remaining: exact counts, typed sort keys, batched reads, watched-filter LRU, re-profiling. Structural permission-check fix is `zones.md`, not built. | -| [`disk-storage-and-persistence-optimization.md`](./disk-storage-and-persistence-optimization.md) | **Proposal, PR in flight** (gap 4). Full-snapshot writes per commit, compaction only via the CLI, O(file) open fsync, SIGTERM path. Fresh e2e data dirs are done. | +| [`disk-storage-and-persistence-optimization.md`](./disk-storage-and-persistence-optimization.md) | **Partial.** Startup size/reclaimable-space diagnostics and automatic compaction shipped (fix 2); fresh e2e data dirs done. Open: incremental Loro updates per commit (fix 1), history pruning (fix 3), SIGTERM path (fix 4). | | [`virtual-drive.md`](./virtual-drive.md) | **Shipped** as a local NFS mount in the Tauri desktop app (`desktop/src/vfs.rs`, unauthenticated loopback, audit C24). Still proposal: headless-server mount, FUSE/WinFSP, native cloud-sync APIs, mobile providers. | | [`commit-retention-and-state-certificates.md`](./commit-retention-and-state-certificates.md) | **Superseded.** Envelope-on-resource replaced the retention design; Phase 1 and 2.5 shipped. Only the `stateHash` certificate and per-resource `retention` propval remain from this document. | | [`s3-blob-storage.md`](./s3-blob-storage.md) | **Partial.** Server-wide S3, verified migration and SaaS enforcement implemented. The backend trait is `get/put/size` only: streaming, delete/GC, per-tenant configuration and encrypted Vault attachments remain. | diff --git a/planning/disk-storage-and-persistence-optimization.md b/planning/disk-storage-and-persistence-optimization.md index 7d85c3a60..7f84ed5bc 100644 --- a/planning/disk-storage-and-persistence-optimization.md +++ b/planning/disk-storage-and-persistence-optimization.md @@ -1,8 +1,10 @@ # Disk Storage & Persistence Optimization -> **Status:** Proposal (2026-06-04); fix 1 and fix 2 have a PR in flight -> (2026-09-18). Diagnoses why server boot time and overall -> performance degrade as the store grows, and proposes fixes. Builds on +> **Status:** Partial (2026-09-18). Fix 2 (startup diagnostics + automatic +> compaction) shipped; fix 5 (fresh e2e data dirs) was done earlier. Fixes 1 +> (incremental Loro updates per commit), 3 (history pruning) and 4 (SIGTERM +> path) remain. See "Progress" at the end. Diagnoses why server boot time and +> overall performance degrade as the store grows, and proposes fixes. Builds on > [`commit-retention-and-state-certificates.md`](./commit-retention-and-state-certificates.md) > (history retention is node policy) and > [`loro-source-of-truth.md`](./loro-source-of-truth.md) (Loro is the history @@ -118,16 +120,44 @@ what the `commit.rs:1088` docstring already intended. state profiles"). Making it a true delta shrinks encrypted envelope size and per-update sync, not just on-disk history. -### 2. Automatic compaction - -Don't rely on the manual CLI. Options (not exclusive): - -- **Boot-time conditional compact:** if `file_size > k Γ— estimated_live_size`, - run `compact()` during startup (it already exists). Trade boot time once for - fast boots after. -- **Background/scheduled compact:** during idle windows, behind the exclusive - lock, with progress logging. -- Surface store-size + dead-page ratio as a metric so the policy is observable. +### 2. Automatic compaction β€” shipped (2026-09-18) + +`lib/src/db/compaction.rs`, run by `RedbStore::new_file_with_policy` on the +handle `Db::init_redb_file` opens anyway (one open, one lock; a bloated file +is never opened a second time to measure it): + +- Every boot logs file size (on disk and length) + `Database::create` + duration (`warn` above 1 GiB, naming `atomic-server compact`), then redb's + `DatabaseStats`: allocated/leaf/branch pages, stored + metadata + + fragmented bytes, and the **reclaimable** bytes = + `bytes_on_disk βˆ’ allocated_pages Γ— page_size`. That is what `compact()` + gives back: it moves live pages down and truncates the free tail. + `fragmented_bytes` (slack inside allocated pages) is logged but is *not* + recoverable by compaction, so the policy ignores it. +- Why on disk (`st_blocks`) and not the length: redb grows the file with + `set_len` β€” doubling below 4 GiB, a whole 4 GiB region above β€” so the + first write after a compaction leaves a sparse tail as long as the live + data. Measured by length that is "50% free" and a 256 MiB–4 GiB store + would compact on every boot. The test + (`startup_compaction_…`) reopens a compacted store and asserts it is left + alone. On a filesystem without sparse files the two coincide and the + policy degrades to length; harmless, since compaction is then a cheap + truncate. +- Boot-time conditional compact when `file_size β‰₯ 256 MiB` **and** + `reclaimable / file_size β‰₯ 30%` (`CompactionPolicy`; server flags + `--auto-compact`, `--auto-compact-min-mb`, + `--auto-compact-min-reclaimable-percent`, env `ATOMIC_AUTO_COMPACT*`). + Before/after sizes and duration are logged and kept in `Tree::PluginMeta` + (`Db::last_compaction`). A failure is a warning; startup never fails on it. + Not applied to OPFS or sled. +- Caveat worth knowing: `DatabaseStats` walks every B-tree page, so the + measurement itself is O(file) β€” a full read, not a full fsync. On the test + store it is ~1.5 ms for 5 MiB; on a multi-GB cold file expect seconds. + `ATOMIC_AUTO_COMPACT=false` skips it; the size + open-time line stays. +- **Not done:** background/scheduled compaction during idle windows (redb + needs `&mut Database` and no live read transaction, so it means a stop-the- + world window anyway), and exporting the numbers as a metric endpoint. + `Db::last_compaction` is the hook for the latter. ### 3. History pruning (ties into commit-retention) @@ -164,14 +194,31 @@ change, but it's the fastest path to a non-flaky suite and isolates real bugs ## Open questions - Is `export_snapshot()` in `sign_at` load-bearing for import-merge - correctness, or safe to switch to an incremental export? (Determines fix #1.) -- What's the dead-page ratio of a real aged store? Run `atomic-server compact` - on a production-like store and compare before/after size to size the win. -- Should compaction be opportunistic (boot threshold) or scheduled? What's the - acceptable one-time boot cost vs. ongoing fast boots trade-off? + correctness, or safe to switch to an incremental export? (Determines fix #1. + PR #1250, Loro fork + incremental sign, is the in-flight answer.) +- What's the dead-page ratio of a real aged store? The boot log now prints it + (`… MiB (NN%) reclaimable by compaction`); collect a few from long-lived + nodes before moving the 30% default either way. +- ~~Should compaction be opportunistic (boot threshold) or scheduled?~~ Boot + threshold, shipped. Scheduled compaction stays open (needs a quiesce window). - How does this interact with `s3-blob-storage.md` (blobs may move out of redb entirely, removing one growth source)? +## Progress + +- [x] Fix 2: startup size/open-time diagnostics, `DatabaseStats` breakdown, + conditional compaction with config + env, docs + (`docs/src/atomicserver/installation.md`), test + (`db::compaction::tests::startup_compaction_shrinks_a_bloated_store_and_keeps_every_resource`). +- [x] Fix 5: fresh e2e data dirs (done earlier, see `e2e-light-heavy.md`). +- [ ] Fix 1: incremental Loro updates per commit β€” PR #1250 (do not touch the + per-commit snapshot write in `db.rs` / `commit.rs` elsewhere; it + conflicts). PR #1531 owns the save-durability path for the same reason. +- [ ] Fix 3: history pruning / envelope retention bounding disk. +- [ ] Fix 4: graceful `SIGTERM` β†’ flush + clean close; verify quick-repair + under `SIGKILL` at arbitrary points. +- [ ] Metric endpoint for store size / reclaimable / last compaction. + ## Code references - `lib/src/db/redb_store.rs:93-160` β€” open path, fsync/repair cost notes, diff --git a/server/src/appstate.rs b/server/src/appstate.rs index a106ac6df..8da60a60e 100644 --- a/server/src/appstate.rs +++ b/server/src/appstate.rs @@ -60,10 +60,13 @@ impl AppState { tracing::warn!("Development mode is enabled. This will use staging environments for services like LetsEncrypt."); } - let mut store = atomic_lib::Db::init_redb_file( + // Opens the file, logs its size and open time, and compacts it + // first when `config.compaction` says the dead space is worth it. + let mut store = atomic_lib::Db::init_redb_file_with_policy( &config.store_path, Some(config.get_origin()), &config.uploads_path, + &config.compaction, ) .await?; diff --git a/server/src/config.rs b/server/src/config.rs index fb89aec65..5224fd929 100644 --- a/server/src/config.rs +++ b/server/src/config.rs @@ -37,6 +37,34 @@ pub struct Opts { #[clap(long, default_value = "latest", env = "ATOMIC_ENVELOPE_RETENTION")] pub envelope_retention: String, + /// Compact the store file at startup when it is at least + /// `--auto-compact-min-mb` and at least `--auto-compact-min-reclaimable-percent` + /// of it is dead space (pages freed by overwrites and deletes that redb + /// never returns to the filesystem). The server listens only when the + /// compaction is done: expect seconds per GB. `false` skips the check; + /// `atomic-server compact` does the same by hand while the server is stopped. + #[clap( + long, + default_value_t = true, + action = clap::ArgAction::Set, + env = "ATOMIC_AUTO_COMPACT" + )] + pub auto_compact: bool, + + /// Store files smaller than this many MiB are never compacted at + /// startup: they open fast whatever their layout. + #[clap(long, default_value_t = 256, env = "ATOMIC_AUTO_COMPACT_MIN_MB")] + pub auto_compact_min_mb: u64, + + /// Startup compaction runs only when at least this percentage of the + /// store file is reclaimable (0-100). + #[clap( + long, + default_value_t = 30, + env = "ATOMIC_AUTO_COMPACT_MIN_RECLAIMABLE_PERCENT" + )] + pub auto_compact_min_reclaimable_percent: u8, + /// How many write requests per minute one signed agent may make (commits /// over HTTP or WebSocket, uploads, blob puts, peer sync pushes, resource /// posts) before the server answers `429 Too Many Requests`. The budget @@ -377,6 +405,10 @@ pub struct Config { /// Who may enroll a new Drive here. Resolved once at boot from explicit /// configuration; see [`crate::host_mode`] for why it is never guessed. pub host_mode: crate::host_mode::HostModeConfig, + /// When the store file is compacted before the server opens it for + /// serving. Built from `--auto-compact*`; see + /// [`atomic_lib::db::compaction::CompactionPolicy`]. + pub compaction: atomic_lib::db::compaction::CompactionPolicy, } impl Config { @@ -567,8 +599,24 @@ pub fn build_config(opts: Opts) -> AtomicServerResult { || cfg!(test) || store_path_looks_like_test_harness(&store_path); + if opts.auto_compact_min_reclaimable_percent > 100 { + return Err(format!( + "ATOMIC_AUTO_COMPACT_MIN_RECLAIMABLE_PERCENT is a percentage (0-100), got {}", + opts.auto_compact_min_reclaimable_percent + ) + .into()); + } + let compaction = atomic_lib::db::compaction::CompactionPolicy { + enabled: opts.auto_compact, + min_file_bytes: opts + .auto_compact_min_mb + .saturating_mul(atomic_lib::db::compaction::MIB), + min_reclaimable_fraction: f64::from(opts.auto_compact_min_reclaimable_percent) / 100.0, + }; + Ok(Config { host_mode, + compaction, initialize, repopulate_defaults, gpu_indexing, @@ -592,3 +640,47 @@ pub fn build_config(opts: Opts) -> AtomicServerResult { base_domain, }) } + +#[cfg(test)] +mod tests { + use super::*; + use atomic_lib::db::compaction::MIB; + + fn config_from(args: &[&str]) -> AtomicServerResult { + let mut argv = vec!["atomic-server", "--data-dir", "./.temp/config-test/db"]; + argv.extend_from_slice(args); + build_config(Opts::parse_from(argv)) + } + + #[test] + fn auto_compact_defaults_match_the_library_policy() { + let config = config_from(&[]).unwrap(); + assert_eq!( + config.compaction, + atomic_lib::db::compaction::CompactionPolicy::default() + ); + } + + #[test] + fn auto_compact_flags_build_the_policy() { + let config = config_from(&[ + "--auto-compact", + "false", + "--auto-compact-min-mb", + "64", + "--auto-compact-min-reclaimable-percent", + "50", + ]) + .unwrap(); + assert!(!config.compaction.enabled); + assert_eq!(config.compaction.min_file_bytes, 64 * MIB); + assert!((config.compaction.min_reclaimable_fraction - 0.5).abs() < 1e-9); + } + + #[test] + fn auto_compact_percent_over_100_is_refused() { + let err = config_from(&["--auto-compact-min-reclaimable-percent", "101"]) + .expect_err("101% must not build a config"); + assert!(err.to_string().contains("0-100"), "{err}"); + } +} From 652c8a643faebd443cc2e02ace252fada1cf2f01 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 19 Sep 2026 09:10:12 +0000 Subject: [PATCH 017/110] docs: address review on the overview and URLs pages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - URLs and identifiers is Core, not Extended: the pointer now sits in the Core paragraph of the overview. - Tools list leads with Atomic Cloud (atomicserver.eu); self-hosting is one line without the docker incantation; the Tauri desktop and mobile apps are listed with where to download them. - urls.md: add localId (a name within a parent, used by imports and plugins with the atomic:… namespace convention), the atomic://open deep link next to atomic://pair, and a short list of strings that look like identifiers but are not (Loro commit origins, old placeholders). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_018ZmptEWmdz92GXTuP77vrb --- docs/src/atomic-data-overview.md | 15 ++++++++------- docs/src/urls.md | 19 ++++++++++++++++--- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/docs/src/atomic-data-overview.md b/docs/src/atomic-data-overview.md index 91f2003da..7173cbe2e 100644 --- a/docs/src/atomic-data-overview.md +++ b/docs/src/atomic-data-overview.md @@ -30,23 +30,24 @@ It is type-safe (you know if something is a `string`, `number`, `date`, `URL`, e The default serialization format for Atomic Data is [JSON-AD](core/json-ad.md), which is simply JSON where each key is a URL of an Atomic Property. These Properties are responsible for setting the `datatype` (to ensure type-safety) and setting `shortnames` (which help to keep names short, for example in JSON serialization) and `descriptions` (which provide semantic explanations of what a property should be used for). +Every Resource, Property and Class is named by a URL. Which URL shapes exist, from `https://` vocabularies to location-independent `did:ad:` identifiers, is explained in [URLs and identifiers](urls.md); read that first if you are new here. + [Read more about Atomic Data Core](core/concepts.md) ## Atomic Data Extended Atomic Data Extended is a set of extra modules (on top of Atomic Data Core) that deal with identity, data that changes over time, authentication, authorization and synchronization between devices. -If you are new here, read [URLs and identifiers](urls.md) first: it explains the `did:ad:` identifiers that show up everywhere in Extended. {{#include extended-table.md}} ## Tools & libraries -- The web app ([demo on atomicdata.dev](https://atomicdata.dev)): documents, tables, chat, files and an ontology editor, working offline in the browser -- Build a web app with [@tomic/lib](js.md), [@tomic/react](usecases/react.md) or [@tomic/svelte](svelte.md) -- Build a native app with [Flutter / Dart](flutter.md), on top of the same Rust core -- Host your own always-on [atomic-server](atomicserver/installation.md) (powers [atomicdata.dev](https://atomicdata.dev), run with `docker run -p 80:80 -v atomic-storage:/atomic-storage ghcr.io/ontola/atomic-server`), or use [Atomic Cloud](https://atomicserver.eu) -- Discover the command line tool: [atomic-cli](rust-cli.md) (`cargo install atomic-cli`) -- Use the Rust library: [atomic_lib](rust-lib.md) +- [Atomic Cloud](https://atomicserver.eu) is the place to start: the web app, an always-on device for your data, and the app downloads, all in one place +- The **web app**: documents, tables, chat, files and an ontology editor, working offline in the browser ([demo on atomicdata.dev](https://atomicdata.dev)) +- The **desktop and mobile apps** (macOS, Windows, Linux, Android and iOS), built with Tauri: the same app with a full node inside, so it can be paired with directly and act as an always-on device. Download them from [atomicserver.eu](https://atomicserver.eu) or the [GitHub releases](https://github.com/atomicdata-dev/atomic-server/releases) +- Build a web app with [@tomic/lib](js.md), [@tomic/react](usecases/react.md) or [@tomic/svelte](svelte.md), or a native app with [Flutter / Dart](flutter.md), on top of the same Rust core +- Prefer to run your own always-on device? [Self-host atomic-server](atomicserver/installation.md), a single binary +- The command line tool [atomic-cli](rust-cli.md) and the Rust library [atomic_lib](rust-lib.md) ## Get involved diff --git a/docs/src/urls.md b/docs/src/urls.md index e3c0c650c..fc96d66c9 100644 --- a/docs/src/urls.md +++ b/docs/src/urls.md @@ -29,8 +29,9 @@ The [DID specification](did.md) has the full derivation rules and the resolution | `did:ad:node:{nodeId}` | A device or server as a network endpoint | The device's transport keypair (Iroh) | Not a Resource; used for pairing and routing only | | `https://example.com/…` | Properties, Classes, Ontologies, external linked data, Resources on HTTP-era servers | The domain owner | An HTTP `GET` with an `application/ad+json` accept header | | `internal:/path` | A Resource on *this* server, in config files and server logs | The server | Rewritten to the server's own origin before it leaves the server | +| `localId` (a string, not a URL) | A stable name for a Resource *within its parent*, used by imports and plugins | The data producer | Looked up as `parent` + `localId`; the Resource keeps its normal Subject | -Two things to notice in the table. +Three things to notice in the table. First, only the `https://` family and `internal:` rows have a path. `did:ad:` identifiers are flat: there is no `did:ad:abc/child`. @@ -40,6 +41,10 @@ Second, the Agent row needs no lookup. The public key is part of the identifier, so any device can verify a signature from that Agent offline. That is what lets a signed edit made on a train be checked on a laptop that has never seen the network. +Third, a `localId` is not a Subject. It is a property (`https://atomicdata.dev/properties/localId`) whose value is a string chosen by whoever produces the data, unique among the children of one parent. +When you [publish JSON-AD for others to import](create-json-ad.md), resources carry a `localId` instead of an `@id`: the importer mints a `did:ad:` Subject on first import, finds the same Resource again by `parent` + `localId` on the next one, and rewrites references between `localId`s into real links. +Plugins use the same mechanism to find the resources they created, with a namespaced convention such as `atomic:pets:table`. + ## Routing hints A pure identity says nothing about where to find the data. @@ -80,13 +85,21 @@ The web app puts the Subject in the query string, so a `did:ad:` identifier surv https://example.com/app/show?subject=did:ad:{genesis} ``` -Pairing codes use their own scheme. A QR code or deep link such as +The desktop and mobile apps register the `atomic://` scheme for deep links, which is a transport for other identifiers rather than an identifier itself: ```text atomic://pair?v=1&node=did:ad:node:{nodeId}&drives=* +atomic://open?subject=did:ad:{genesis} ``` -carries the *route* to a device. It never contains an Agent secret, and scanning it grants nothing by itself: what crosses the link is decided by the rights on each Resource, on every transport. See [Atomic Sync](sync.md). +A pairing code, as a QR code or a link, carries the *route* to a device. It never contains an Agent secret, and scanning it grants nothing by itself: what crosses the link is decided by the rights on each Resource, on every transport. See [Atomic Sync](sync.md). +An open link hands the app a Subject to navigate to. + +## Strings that look like identifiers but are not + +- `atomic:system` and `sys:init` are Loro commit *origins*, tags on an edit inside a Resource's CRDT document that the undo manager uses to skip internal writes. They never name anything. +- `_new:` and similar placeholders appeared in older clients before a Resource had signed its genesis. Current clients mint the `did:ad:` Subject up front, so a placeholder should not reach the wire. +- A `localId` value (see above) is a name inside a parent, not a global identifier, even when it is written in a URL-like namespace. ## Encoding From 753a8a63a766fbbcf3834fc20b5b7ba4453298af Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 12:28:51 +0200 Subject: [PATCH 018/110] fix(data-browser): make page transitions opt-in until every browser works (#1563) (#1566) View transitions were on by default with only a user opt-out. They are still broken outside Chromium on desktop: the card-to-page morph blows up into an overlay on Firefox, and #1563 reports the same on Android. Flip the setting to opt-in. The stored key is new (`viewTransitionsEnabled`, default false) rather than a flipped default on `viewTransitionsDisabled`: that key already holds `false` for anyone who toggled the old checkbox back off, so reusing it would have left transitions on for exactly the people who had noticed them misbehaving. The transition machinery and CSS are untouched, so re-enabling by default is a one-line change once each browser is verified. - Settings > Accessibility now reads "Enable page transition animations". - Playwright config and fixtures seed the new key. - New test pins both sides of the default in useNavigateWithTransition. Claude-Session: https://claude.ai/code/session_01Pxhq3hXh8AgAk4NZzNdnkR Co-authored-by: Claude --- TESTING_COVERAGE.md | 3 +- browser/CHANGELOG.md | 1 + .../data-browser/src/helpers/AppSettings.tsx | 35 +++++--- .../hooks/useNavigateWithTransition.test.tsx | 85 +++++++++++++++++++ .../src/hooks/useNavigateWithTransition.ts | 10 +-- .../data-browser/src/routes/AppSettings.tsx | 15 ++-- browser/e2e/playwright.config.ts | 16 ++-- browser/e2e/tests/session-fixtures.ts | 2 +- browser/e2e/tests/signout-signin-data.spec.ts | 2 +- 9 files changed, 138 insertions(+), 31 deletions(-) create mode 100644 browser/data-browser/src/hooks/useNavigateWithTransition.test.tsx diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index 9b5f64a33..8faad7014 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -767,8 +767,9 @@ Not covered: derived AI tools invoked through a real model; MCP protocol project |---|---|---| | Hashed `view-transition-name` plus `view-transition-class` per tag | glue | `browser/data-browser/src/helpers/viewTransition.test.ts` | | `startViewTransition` throw / hung `finished` / rejected `ready` still navigates and skips the overlay | glue | `browser/data-browser/src/helpers/viewTransition.test.ts` | +| Navigation skips `startViewTransition` unless the user opts in, and uses it once they do | glue | `browser/data-browser/src/hooks/useNavigateWithTransition.test.tsx` | -Not covered: visual morph of a grid card into the resource page in Firefox (needs a headed Firefox run; Playwright's firefox project is locks-only and automation bypasses view transitions unless `forceViewTransitions` is set). +Not covered: visual morph of a grid card into the resource page in Firefox (needs a headed Firefox run; Playwright's firefox project is locks-only and automation bypasses view transitions unless `forceViewTransitions` is set). Android Chrome is not covered at all, which is why transitions are off by default ([#1563](https://github.com/ontola/atomic-server/issues/1563)): re-enabling by default needs a per-browser check first. ## Documents diff --git a/browser/CHANGELOG.md b/browser/CHANGELOG.md index 9bf785042..c55722872 100644 --- a/browser/CHANGELOG.md +++ b/browser/CHANGELOG.md @@ -22,6 +22,7 @@ This changelog covers all five packages, as they are (for now) updated as a whol - Fix: file previews and avatars wait for the local blob lookup before requesting the server, avoiding transient 404s for offline uploads. `useFileObjectUrl` accepts an optional network fallback URL and returns `undefined` while the local lookup is pending. - Fix: `@tomic/cli` `ad-generate ontologies` works with `did:ad:` ontology subjects. A DID is not an HTTP URL: the CLI needs `serverUrl` (the Atomic Server origin) and fetches `GET {serverUrl}/did?subject=…`. Pasting the address-bar form `https://host/did:ad:…` no longer fails with `Resource has wrong subject in @id` β€” that URL is an alias, and the resource's `@id` is the DID. `ad-generate init` writes `serverUrl`. +- Page transition animations are now **opt-in**. They still break outside Chromium on desktop (a card-to-page morph blows up into an overlay on Firefox and on Android Chrome), so they are off until every browser we support is verified ([#1563](https://github.com/ontola/atomic-server/issues/1563)). Settings > Accessibility gains "Enable page transition animations" in place of the old "Disable" checkbox, backed by a new `viewTransitionsEnabled` localStorage key (default `false`). The old `viewTransitionsDisabled` key is no longer read: reusing it would have left transitions on for anyone who had ever toggled the old checkbox back off. The transition machinery and CSS are untouched, so turning them back on is a default flip. - Fix: page transitions on Firefox no longer smear into a giant overlay or get skipped entirely. Snapshot sizing is scoped by `view-transition-class` (titles keep the height-based aspect-ratio rule; cardβ†’page morphs fill the group), title links that wrap a heading are `inline-block` so Firefox does not treat them as duplicate names (IB splits), and a failed or hung `startViewTransition` still navigates and skips the overlay instead of covering the page. ## [v0.41.0-beta.7] - 2026-09-12 diff --git a/browser/data-browser/src/helpers/AppSettings.tsx b/browser/data-browser/src/helpers/AppSettings.tsx index 66bff3c79..1b0000a4f 100644 --- a/browser/data-browser/src/helpers/AppSettings.tsx +++ b/browser/data-browser/src/helpers/AppSettings.tsx @@ -78,8 +78,18 @@ export const AppSettingsContextProvider = ( }, [drive, store]); // == ACCESSIBILITY == - const [viewTransitionsDisabled, setViewTransitionsDisabled] = useLocalStorage( - 'viewTransitionsDisabled', + // Opt-in, not opt-out. View transitions are still broken outside Chromium + // on desktop: Firefox and Android both show the card-to-page morph blowing + // up into an overlay ([#1563](https://github.com/ontola/atomic-server/issues/1563)). + // Until every browser we support is verified, they stay off unless the user + // asks for them. + // + // The key is deliberately *not* the old `viewTransitionsDisabled`: that one + // has `false` stored in plenty of browsers already (anyone who ever toggled + // the old checkbox twice), and reusing it would leave transitions on for + // exactly the people who noticed they were broken. + const [viewTransitionsEnabled, setViewTransitionsEnabled] = useLocalStorage( + 'viewTransitionsEnabled', false, ); const [sidebarKeyboardDndEnabled, setSidebarKeyboardDndEnabled] = @@ -229,8 +239,8 @@ export const AppSettingsContextProvider = ( setSideBarLocked, agent, setAgent: setAgentAndShowToast, - viewTransitionsDisabled, - setViewTransitionsDisabled, + viewTransitionsEnabled, + setViewTransitionsEnabled, sidebarKeyboardDndEnabled, setSidebarKeyboardDndEnabled, hideTemplates, @@ -257,8 +267,8 @@ export const AppSettingsContextProvider = ( setSideBarLocked, agent, setAgentAndShowToast, - viewTransitionsDisabled, - setViewTransitionsDisabled, + viewTransitionsEnabled, + setViewTransitionsEnabled, sidebarKeyboardDndEnabled, setSidebarKeyboardDndEnabled, hideTemplates, @@ -304,9 +314,12 @@ export interface AppSettings { /** The currently signed in Agent */ agent: Agent | undefined; setAgent: (a: Agent | undefined) => void; - /** If the app should use view transitions */ - viewTransitionsDisabled: boolean; - setViewTransitionsDisabled: (b: boolean) => void; + /** + * If the app should use view transitions. Off unless the user opts in, see + * the hook in `AppSettingsContextProvider`. + */ + viewTransitionsEnabled: boolean; + setViewTransitionsEnabled: (b: boolean) => void; sidebarKeyboardDndEnabled: boolean; setSidebarKeyboardDndEnabled: (b: boolean) => void; hideTemplates: boolean; @@ -339,8 +352,8 @@ const initialState: AppSettings = { setSideBarLocked: () => undefined, agent: undefined, setAgent: () => undefined, - viewTransitionsDisabled: true, - setViewTransitionsDisabled: () => undefined, + viewTransitionsEnabled: false, + setViewTransitionsEnabled: () => undefined, sidebarKeyboardDndEnabled: false, setSidebarKeyboardDndEnabled: () => undefined, hideTemplates: false, diff --git a/browser/data-browser/src/hooks/useNavigateWithTransition.test.tsx b/browser/data-browser/src/hooks/useNavigateWithTransition.test.tsx new file mode 100644 index 000000000..fcaac1912 --- /dev/null +++ b/browser/data-browser/src/hooks/useNavigateWithTransition.test.tsx @@ -0,0 +1,85 @@ +// @vitest-environment jsdom +// @wc-ignore-file +import { renderHook } from '@testing-library/react'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +const { settings, navigate } = vi.hoisted(() => ({ + settings: { viewTransitionsEnabled: false }, + navigate: vi.fn(async () => undefined), +})); + +vi.mock('react-dom', () => ({ + flushSync: (fn: () => void) => fn(), +})); + +vi.mock('../helpers/AppSettings', () => ({ + useSettings: () => settings, +})); + +vi.mock('@tanstack/react-router', () => ({ + useNavigate: () => navigate, + useRouter: () => ({ history: { back: vi.fn(), forward: vi.fn() } }), +})); + +import { resetViewTransitionQueue } from '../helpers/viewTransition'; +import { useNavigateWithTransition } from './useNavigateWithTransition'; + +/** Minimal stand-in for the real `ViewTransition` the UA hands back. */ +function fakeTransition( + callbackOptions?: ViewTransitionUpdateCallback | StartViewTransitionOptions, +): ViewTransition { + const update = + typeof callbackOptions === 'function' + ? callbackOptions + : callbackOptions?.update; + const done = Promise.resolve(update?.()).then(() => undefined); + + return { + ready: Promise.resolve(), + finished: done, + updateCallbackDone: done, + skipTransition: vi.fn(), + types: new Set() as ViewTransition['types'], + }; +} + +const startViewTransition = vi.fn(fakeTransition); + +describe('useNavigateWithTransition', () => { + beforeEach(() => { + resetViewTransitionQueue(); + navigate.mockClear(); + startViewTransition.mockClear(); + // jsdom has no view transitions, so stand the API in: the only thing + // deciding whether we take that path should be the setting. + document.startViewTransition = startViewTransition; + }); + + afterEach(() => { + Reflect.deleteProperty(document, 'startViewTransition'); + resetViewTransitionQueue(); + }); + + // Transitions are opt-in since #1563 β€” they still break on Firefox and + // Android. A default that silently flips back would be invisible in review, + // so pin both sides of it here. + it('navigates without a view transition when the user has not opted in', async () => { + settings.viewTransitionsEnabled = false; + + const { result } = renderHook(() => useNavigateWithTransition()); + await result.current('/app/drive'); + + expect(navigate).toHaveBeenCalledWith({ to: '/app/drive' }); + expect(startViewTransition).not.toHaveBeenCalled(); + }); + + it('wraps the navigation in a view transition once the user opts in', async () => { + settings.viewTransitionsEnabled = true; + + const { result } = renderHook(() => useNavigateWithTransition()); + await result.current('/app/drive'); + + expect(navigate).toHaveBeenCalledWith({ to: '/app/drive' }); + expect(startViewTransition).toHaveBeenCalledOnce(); + }); +}); diff --git a/browser/data-browser/src/hooks/useNavigateWithTransition.ts b/browser/data-browser/src/hooks/useNavigateWithTransition.ts index 412a62e0f..34327c5f3 100644 --- a/browser/data-browser/src/hooks/useNavigateWithTransition.ts +++ b/browser/data-browser/src/hooks/useNavigateWithTransition.ts @@ -7,10 +7,10 @@ import { useNavigate, useRouter } from '@tanstack/react-router'; */ export function useNavigateWithTransition() { const navigate = useNavigate(); - const { viewTransitionsDisabled } = useSettings(); + const { viewTransitionsEnabled } = useSettings(); const navigateWithTransition = wrapWithViewTransition( - viewTransitionsDisabled, + !viewTransitionsEnabled, (options: Parameters[0] | string) => { const newOptions = typeof options === 'string' @@ -28,10 +28,10 @@ export function useNavigateWithTransition() { export function useBackForward() { const router = useRouter(); - const { viewTransitionsDisabled } = useSettings(); + const { viewTransitionsEnabled } = useSettings(); const back = wrapWithViewTransition( - viewTransitionsDisabled, + !viewTransitionsEnabled, () => new Promise(resolve => { router.history.back(); @@ -40,7 +40,7 @@ export function useBackForward() { ); const forward = wrapWithViewTransition( - viewTransitionsDisabled, + !viewTransitionsEnabled, () => new Promise(resolve => { router.history.forward(); diff --git a/browser/data-browser/src/routes/AppSettings.tsx b/browser/data-browser/src/routes/AppSettings.tsx index 9bb486241..299f433eb 100644 --- a/browser/data-browser/src/routes/AppSettings.tsx +++ b/browser/data-browser/src/routes/AppSettings.tsx @@ -46,8 +46,8 @@ const AppSettings: React.FunctionComponent = () => { setDarkMode, colorfulMode, setColorfulMode, - viewTransitionsDisabled, - setViewTransitionsDisabled, + viewTransitionsEnabled, + setViewTransitionsEnabled, sidebarKeyboardDndEnabled, setSidebarKeyboardDndEnabled, hideTemplates, @@ -190,15 +190,18 @@ const AppSettings: React.FunctionComponent = () => { setViewTransitionsDisabled(checked)} + checked={viewTransitionsEnabled} + onChange={checked => setViewTransitionsEnabled(checked)} />{' '} - Disable page transition animations + + Enable page transition animations (experimental, known to + misbehave outside Chromium on desktop) + !!url) .map(url => new URL(url).origin) .map(origin => ({ origin, - localStorage: [{ name: 'viewTransitionsDisabled', value: 'true' }], + localStorage: [{ name: 'viewTransitionsEnabled', value: 'false' }], })), { origin: 'http://localhost:6747', - localStorage: [{ name: 'viewTransitionsDisabled', value: 'true' }], + localStorage: [{ name: 'viewTransitionsEnabled', value: 'false' }], }, { origin: 'http://localhost:9883', - localStorage: [{ name: 'viewTransitionsDisabled', value: 'true' }], + localStorage: [{ name: 'viewTransitionsEnabled', value: 'false' }], }, { origin: 'http://atomic:9883', - localStorage: [{ name: 'viewTransitionsDisabled', value: 'true' }], + localStorage: [{ name: 'viewTransitionsEnabled', value: 'false' }], }, { // Dagger e2e FRONTEND_URL β€” chromium treats `*.localhost` as a // secure context (needed for WASM ClientDb / crypto.subtle). origin: 'http://atomic.localhost:9883', - localStorage: [{ name: 'viewTransitionsDisabled', value: 'true' }], + localStorage: [{ name: 'viewTransitionsEnabled', value: 'false' }], }, ], }, diff --git a/browser/e2e/tests/session-fixtures.ts b/browser/e2e/tests/session-fixtures.ts index 8cae73116..88c9afab8 100644 --- a/browser/e2e/tests/session-fixtures.ts +++ b/browser/e2e/tests/session-fixtures.ts @@ -69,7 +69,7 @@ export const test = base.extend<{}, { sessionSeed: Seed | undefined }>({ await installEmptyDiscoveryRoom(context); await context.addInitScript(origin => { if (location.origin !== origin) return; - localStorage.setItem('viewTransitionsDisabled', 'true'); + localStorage.setItem('viewTransitionsEnabled', 'false'); }, new URL(FRONTEND_URL).origin); const page = context.pages()[0] ?? (await context.newPage()); await devDrive(page); diff --git a/browser/e2e/tests/signout-signin-data.spec.ts b/browser/e2e/tests/signout-signin-data.spec.ts index 602ad7079..f9882f891 100644 --- a/browser/e2e/tests/signout-signin-data.spec.ts +++ b/browser/e2e/tests/signout-signin-data.spec.ts @@ -53,7 +53,7 @@ const test = base.extend({ timezoneId, }); await persistent.addInitScript(() => { - localStorage.setItem('viewTransitionsDisabled', 'true'); + localStorage.setItem('viewTransitionsEnabled', 'false'); }); try { From 086f4f151e4996e78f94fa36fa518e600b775355 Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 12:35:07 +0200 Subject: [PATCH 019/110] Default the navbar to the bottom on touch devices (#1567) The navbar started at the top everywhere. On a phone or tablet that puts the sidebar toggle, the search button and the context menu out of thumb reach, which matters more than a navbar that sits in the same place on every device. The default now keys on a touch-primary pointer rather than viewport width: a landscape tablet is as wide as a laptop, and a desktop window dragged narrow is still mouse-operated. New `isTouchPrimary()` helper, which `useHover` now shares instead of inlining the same media query. Only the default changes. `useLocalStorage` falls back to it solely when the key is absent, so a position picked in Settings is preserved. Claude-Session: https://claude.ai/code/session_01VT2jpYQ73mPfiueW25ty3y Co-authored-by: Claude --- browser/CHANGELOG.md | 1 + .../data-browser/src/helpers/AppSettings.tsx | 10 ++++++- .../data-browser/src/helpers/pointer.test.ts | 28 +++++++++++++++++++ browser/data-browser/src/helpers/pointer.ts | 15 ++++++++++ browser/data-browser/src/helpers/useHover.tsx | 3 +- 5 files changed, 55 insertions(+), 2 deletions(-) create mode 100644 browser/data-browser/src/helpers/pointer.test.ts create mode 100644 browser/data-browser/src/helpers/pointer.ts diff --git a/browser/CHANGELOG.md b/browser/CHANGELOG.md index c55722872..84386e8f4 100644 --- a/browser/CHANGELOG.md +++ b/browser/CHANGELOG.md @@ -4,6 +4,7 @@ This changelog covers all five packages, as they are (for now) updated as a whol ## UNRELEASED +- The navbar sits at the bottom by default on phones and tablets, so the sidebar toggle, the search button and the context menu stay within thumb reach. The signal is a touch-primary pointer rather than viewport width, so a landscape tablet also gets the bottom bar and a narrow desktop window does not. Desktop keeps the top bar, and a position picked in Settings is untouched ([#1565](https://github.com/ontola/atomic-server/issues/1565)). - Fix: `@tomic/lib` writes the Loro `datatypes` tags in the export path shared by genesis and drain-time signing, after the user's pending ops are sealed into their own commit. The drain used to tag first, which folded a `set()` edit into the tag write's `atomic:system` commit β€” the canvas undo stack skips that origin, so the first edit of a `json`/reference property after a save could not be undone. - Perf: a cold page load asks the local database (OPFS worker) for all the resources a render pass mounts in one round trip instead of one per `useResource`. `Store` coalesces the render-phase cache misses of a tick into a single `getResourcesWithSnapshots` call (chunked at 200 subjects), and a subject asked for twice while a read is pending or in flight shares that read. 50 mounted resources went from 50 worker messages to 1; the tri-state answer (`true` / `false` / no database to ask) and the "ask again once the database attaches" behaviour are unchanged. - Fix: deleting a resource while offline no longer loses the delete. `Resource.destroy()` now goes through the durable `LocalOutbox` like every other write: the signed destroy commit is queued (`OutboxEntry.signedDestroy`, persisted in localStorage), the resource is removed locally at once, and the drain POSTs the envelope with the usual backoff and reconnect replay. Online, `await destroy()` still resolves after the server acknowledgement and rejects on a real refusal; offline it resolves with the delete queued. A resource created and deleted before either write reached the server is dropped without a POST, and a server answer that the resource is already gone counts as acknowledged instead of retrying forever. While a delete is queued, incoming state for that subject (a subscription push, a fetch that raced the delete, a stale OPFS row) no longer resurrects it. `Store.hasPendingDestroy(subject)` is new; `LocalOutbox` gains `setDestroyCommit`, `clearDestroy` and `discard`. diff --git a/browser/data-browser/src/helpers/AppSettings.tsx b/browser/data-browser/src/helpers/AppSettings.tsx index 1b0000a4f..df5f286e4 100644 --- a/browser/data-browser/src/helpers/AppSettings.tsx +++ b/browser/data-browser/src/helpers/AppSettings.tsx @@ -24,6 +24,7 @@ import { errorHandler } from '../handlers/errorHandler'; import { isDev } from '../config'; import { getLocalServerOrigin, isRunningInTauri } from './tauri'; import { fetchManagedInfo, isAtomicServer } from './managedServer'; +import { isTouchPrimary } from './pointer'; interface ProviderProps { children: ReactNode; @@ -53,7 +54,14 @@ export const AppSettingsContextProvider = ( 'sideBarOpen', window.innerWidth > SIDEBAR_TOGGLE_WIDTH, ); - const [navbarTop, setNavbarTop] = useLocalStorage('navbarTop', true); + // Phones and tablets default to a bottom navbar: the sidebar toggle, the + // search button and the context menu matter more within thumb reach than a + // navbar that sits in the same place on every device. Only the default β€” + // an explicit choice in Settings is stored and wins. + const [navbarTop, setNavbarTop] = useLocalStorage( + 'navbarTop', + !isTouchPrimary(), + ); // == CONTENT LANGUAGE == // The language localized *content* (LocalizedText values, translated diff --git a/browser/data-browser/src/helpers/pointer.test.ts b/browser/data-browser/src/helpers/pointer.test.ts new file mode 100644 index 000000000..33cbccb38 --- /dev/null +++ b/browser/data-browser/src/helpers/pointer.test.ts @@ -0,0 +1,28 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { isTouchPrimary } from './pointer'; + +afterEach(() => vi.unstubAllGlobals()); + +const stubPointer = (coarse: boolean) => + vi.stubGlobal('window', { + matchMedia: (query: string) => ({ + matches: query === '(pointer: coarse)' ? coarse : false, + }), + }); + +describe('isTouchPrimary', () => { + it('is true when the primary pointer is coarse', () => { + stubPointer(true); + expect(isTouchPrimary()).toBe(true); + }); + + it('is false on a mouse or trackpad', () => { + stubPointer(false); + expect(isTouchPrimary()).toBe(false); + }); + + it('is false where matchMedia is missing', () => { + vi.stubGlobal('window', {}); + expect(isTouchPrimary()).toBe(false); + }); +}); diff --git a/browser/data-browser/src/helpers/pointer.ts b/browser/data-browser/src/helpers/pointer.ts new file mode 100644 index 000000000..9bf61aab3 --- /dev/null +++ b/browser/data-browser/src/helpers/pointer.ts @@ -0,0 +1,15 @@ +/** + * Whether the primary pointing device is touch, i.e. a phone or a tablet. + * + * Use this rather than a viewport width breakpoint when the question is "is + * this thumb-operated": a landscape tablet is as wide as a laptop, and a + * desktop window dragged narrow is still mouse-operated. A touchscreen laptop + * reports a fine primary pointer, so it counts as desktop here. + */ +export function isTouchPrimary(): boolean { + if (!window.matchMedia) { + return false; + } + + return window.matchMedia('(pointer: coarse)').matches; +} diff --git a/browser/data-browser/src/helpers/useHover.tsx b/browser/data-browser/src/helpers/useHover.tsx index 3991d620f..3a42cecf3 100644 --- a/browser/data-browser/src/helpers/useHover.tsx +++ b/browser/data-browser/src/helpers/useHover.tsx @@ -1,4 +1,5 @@ import { RefObject, useCallback, useMemo, useRef, useState } from 'react'; +import { isTouchPrimary } from './pointer'; type Listeners = { onMouseOver?: React.MouseEventHandler; @@ -42,7 +43,7 @@ export function useHover(): [ ); // don't hover on touch screen devices - if (window.matchMedia('(pointer: coarse)').matches) { + if (isTouchPrimary()) { return [ref, false, {}]; } From 6d558268987437e81a3c67a82f6b0ea7cdcea60b Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 12:55:19 +0200 Subject: [PATCH 020/110] Keep toasts clear of the bottom navbar, drop the old sidebar spacer (#1570) Toasts stack from the bottom right, which is exactly where the navbar sits when it is at the bottom, so a toast covered the search button and the context menu. The toast container is now lifted by the navbar's height in that case; with the navbar at the top nothing changes. The sidebar's OverlapSpacer goes with it. It added 3.8rem beneath the App panel below 950px with a bottom navbar, dating from the floating navbar mode. The overlap it guarded cannot happen: SideBarWrapper's height is the viewport minus the navbar, so the bar never covers the sidebar. --- browser/CHANGELOG.md | 1 + .../src/components/SideBar/OverlapSpacer.tsx | 22 ------------------- .../src/components/SideBar/index.tsx | 2 -- .../data-browser/src/components/Toaster.tsx | 11 ++++++++++ 4 files changed, 12 insertions(+), 24 deletions(-) delete mode 100644 browser/data-browser/src/components/SideBar/OverlapSpacer.tsx diff --git a/browser/CHANGELOG.md b/browser/CHANGELOG.md index 84386e8f4..7aba03e86 100644 --- a/browser/CHANGELOG.md +++ b/browser/CHANGELOG.md @@ -4,6 +4,7 @@ This changelog covers all five packages, as they are (for now) updated as a whol ## UNRELEASED +- Toasts no longer cover the navbar's buttons when the navbar is at the bottom: the toast container is lifted by the navbar's height, so the sidebar toggle, search and context menu stay tappable while a toast is up. The sidebar's 3.8rem `OverlapSpacer`, beneath the App panel, is gone: it dated from the floating-navbar mode and guarded an overlap that cannot happen, since the sidebar's height already excludes the navbar ([#1565](https://github.com/ontola/atomic-server/issues/1565)). - The navbar sits at the bottom by default on phones and tablets, so the sidebar toggle, the search button and the context menu stay within thumb reach. The signal is a touch-primary pointer rather than viewport width, so a landscape tablet also gets the bottom bar and a narrow desktop window does not. Desktop keeps the top bar, and a position picked in Settings is untouched ([#1565](https://github.com/ontola/atomic-server/issues/1565)). - Fix: `@tomic/lib` writes the Loro `datatypes` tags in the export path shared by genesis and drain-time signing, after the user's pending ops are sealed into their own commit. The drain used to tag first, which folded a `set()` edit into the tag write's `atomic:system` commit β€” the canvas undo stack skips that origin, so the first edit of a `json`/reference property after a save could not be undone. - Perf: a cold page load asks the local database (OPFS worker) for all the resources a render pass mounts in one round trip instead of one per `useResource`. `Store` coalesces the render-phase cache misses of a tick into a single `getResourcesWithSnapshots` call (chunked at 200 subjects), and a subject asked for twice while a read is pending or in flight shares that read. 50 mounted resources went from 50 worker messages to 1; the tri-state answer (`true` / `false` / no database to ask) and the "ask again once the database attaches" behaviour are unchanged. diff --git a/browser/data-browser/src/components/SideBar/OverlapSpacer.tsx b/browser/data-browser/src/components/SideBar/OverlapSpacer.tsx deleted file mode 100644 index a60fb7046..000000000 --- a/browser/data-browser/src/components/SideBar/OverlapSpacer.tsx +++ /dev/null @@ -1,22 +0,0 @@ -import { useMediaQuery } from '../../hooks/useMediaQuery'; -import { useSettings } from '../../helpers/AppSettings'; -import { styled } from 'styled-components'; -import { transition } from '../../helpers/transition'; - -import type { JSX } from 'react'; - -export function OverlapSpacer(): JSX.Element { - const narrow = useMediaQuery('(max-width: 950px)'); - const { navbarTop } = useSettings(); - // The floating-navbar setting was folded into `navbarTop`. When the - // navbar sits at the bottom on narrow screens, the sidebar content - // gets covered, so we lift it with a spacer. - const elevate = narrow && !navbarTop; - - return ; -} - -const Elevator = styled.div<{ elevate: boolean }>` - height: ${p => (p.elevate ? '3.8rem' : '0rem')}; - ${transition('height')} -`; diff --git a/browser/data-browser/src/components/SideBar/index.tsx b/browser/data-browser/src/components/SideBar/index.tsx index afffdd443..d70a26154 100644 --- a/browser/data-browser/src/components/SideBar/index.tsx +++ b/browser/data-browser/src/components/SideBar/index.tsx @@ -9,7 +9,6 @@ import { useResizable, } from '../../hooks/useResizable'; import { useCombineRefs } from '../../hooks/useCombineRefs'; -import { OverlapSpacer } from './OverlapSpacer'; import { AppMenu } from './AppMenu'; import { SideBarHomePanels } from './SideBarHomePanels'; import { useMediaQuery } from '../../hooks/useMediaQuery'; @@ -118,7 +117,6 @@ export function SideBar(): JSX.Element { - {!isRearanging && ( Date: Sat, 19 Sep 2026 14:09:31 +0200 Subject: [PATCH 021/110] docs: rewrite the feature list (#1572) Every line is one short clause. Ordered core-first: local-first, then real-time collaboration and presence, then the things you make with it, then the platform underneath. Adds the features that were missing: websites, the kanban / calendar / dashboard / timer views, table templates, the virtual drive, apps, plugins and integrations, meetings and canvas. Drops the passkey recovery line, folds invites into authorization and the serialization formats into the API line. docs/src/atomic-server.md carries the same table and is updated with it. --- README.md | 49 +++++++++++++++++++++------------------ docs/src/atomic-server.md | 49 +++++++++++++++++++++------------------ 2 files changed, 52 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index de86ffaa5..4425f3a30 100644 --- a/README.md +++ b/README.md @@ -26,29 +26,32 @@ _Status: alpha. [Breaking changes](CHANGELOG.md) are expected until 1.0._ ## AtomicServer -- 🏠 **Local-first**: create and edit data with no server at all. Resources are addressed by [`did:ad` identifiers](https://docs.atomicdata.dev/did) and resolve peer-to-peer over the Mainline DHT, so an identity is a keypair you hold rather than an account on someone else's machine. Edits are signed CRDT commits that merge when you reconnect. -- πŸ”’ **Encrypted at rest, per agent**: each agent's in-browser database is encrypted with XChaCha20-Poly1305, under a key wrapped by that agent's own private key. Signing out leaves the cache in place but unreadable to the next session, so no wipe is required. -- πŸ”‘ **Passkey-backed recovery**: a WebAuthn passkey wraps the backup of your agent secret (Argon2id + AES-GCM), so onboarding hands you nothing to write down, and a lost device doesn't have to mean a lost account. -- πŸš€ **Fast** (less than 1ms median response time on my laptop), powered by [actix-web](https://github.com/actix/actix-web) and [redb](https://github.com/cberner/redb) -- πŸͺΆ **One self-contained binary** (~70MB): server, web app, full-text search and database in a single file, with no runtime dependencies and nothing to install alongside it. -- πŸ’» **Runs everywhere** (linux, windows, mac, arm) -- πŸ”§ **Custom data models**: create your own classes, properties and schemas using the built-in Ontology Editor. All data is verified and the models are sharable using [Atomic Schema](https://docs.atomicdata.dev/schema/intro.html) -- βš™οΈ **Restful API**, with [JSON-AD](https://docs.atomicdata.dev/core/json-ad.html) responses. -- πŸ”Ž **Full-text search** with prefix typeahead and 1-edit fuzzy matching, often <3ms responses. Same KV index on the server and in the browser. -- ✨ **AI** with [MCP](https://modelcontextprotocol.io/) support, use any model via OpenRouter or host your own with Ollama. -- πŸ—„οΈ **Tables**, with strict schema validation, keyboard support, copy / paste support. Similar to Airtable. -- πŸ“„ **Documents**, collaborative, rich text, similar to Google Docs / Notion. -- πŸ’¬ **Group chat**, performant and flexible message channels with attachments, search and replies. -- πŸ“‚ **File management**: Upload, download and preview attachments. -- πŸ’Ύ **Versioning** / history from the Loro oplog, with writes authorized by [Atomic Commits](https://docs.atomicdata.dev/commits/intro.html) -- πŸ”„ **Real-time synchronization**: instantly communicates state changes with a client. Build dynamic, collaborative apps using [websockets](https://docs.atomicdata.dev/websockets) (using a [single one-liner in react](https://docs.atomicdata.dev/usecases/react) or [svelte](https://docs.atomicdata.dev/svelte)). -- 🧰 **Many serialization options**: to JSON, [JSON-AD](https://docs.atomicdata.dev/core/json-ad.html), and various Linked Data / RDF formats (RDF/XML, N-Triples / Turtle / JSON-LD). -- πŸ“– **Pagination, sorting and filtering** queries using [Atomic Collections](https://docs.atomicdata.dev/schema/collections.html). -- πŸ” **Authorization** (read / write permissions) and Hierarchical structures powered by [Atomic Hierarchy](https://docs.atomicdata.dev/hierarchy.html) -- πŸ“² **Invite and sharing system** with [Atomic Invites](https://docs.atomicdata.dev/invitations.html) -- 🌐 **Embedded server** with support for HTTP / HTTPS / HTTP2.0 (TLS) and Built-in LetsEncrypt handshake. -- πŸ“± **Runs on mobile**: `atomic_lib` compiles into Flutter apps through [flutter_rust_bridge](https://github.com/fzyzcjy/flutter_rust_bridge), so phones get the same local-first store, signing and peer sync as the browser, not a thin REST wrapper. See [`/flutter`](/flutter). -- πŸ“š **Libraries**: [Javascript / Typescript](https://www.npmjs.com/package/@tomic/lib), [React](https://www.npmjs.com/package/@tomic/react), [Svelte](https://www.npmjs.com/package/@tomic/svelte), [Rust](https://crates.io/crates/atomic-lib), and a [Dart / Flutter client](/flutter/lib/atomic) +- 🏠 **Local-first**: works offline in the browser, syncs when you reconnect. +- πŸ”„ **Real-time collaboration**: live cursors, typing indicators, and following what a teammate is doing. +- πŸ“„ **Documents**: collaborative rich text, like Google Docs or Notion. +- πŸ—„οΈ **Tables**: strict schema, keyboard navigation, copy / paste. Like Airtable. +- πŸ“‹ **Kanban, calendar, dashboard and timer views** on any table. +- 🧰 **Templates**: issue tracker, CRM, project tasks, time tracker and more. +- 🌐 **Websites**: design pages in the browser, publish in one click, roll back any version. +- πŸ—‚οΈ **Virtual drive**: mount your drive as a folder in Finder or Explorer (desktop app). +- ✨ **AI** with [MCP](https://modelcontextprotocol.io/) support, any model via OpenRouter or local Ollama. +- 🧩 **Apps**: custom screens in plain JavaScript, backed by your own data. +- πŸ”Œ **Plugins and integrations**: Wasm plugins, and syncing from Notion, GitHub and more. +- πŸ’¬ **Group chat**: channels with attachments, search and replies. +- πŸŽ₯ **Meetings**: video calls with shared notes and presence. +- 🎨 **Canvas**: an infinite drawing surface, shared live. +- πŸ“‚ **Files**: upload, download and preview attachments. +- πŸ”§ **Custom data models**: your own classes and properties in the Ontology Editor, shared as [Atomic Schema](https://docs.atomicdata.dev/schema/intro.html). +- πŸ”’ **Encrypted at rest**: each agent's local database is encrypted under their own key. +- πŸ’Ύ **Versioning**: full history, every write a signed [Atomic Commit](https://docs.atomicdata.dev/commits/intro.html). +- πŸ”Ž **Full-text search**: typeahead and fuzzy, often under 3ms, same index on server and client. +- πŸ” **Authorization**: read / write rights, [hierarchies](https://docs.atomicdata.dev/hierarchy.html) and [invite links](https://docs.atomicdata.dev/invitations.html). +- βš™οΈ **RESTful API** with [JSON-AD](https://docs.atomicdata.dev/core/json-ad.html), plus RDF, Turtle, N-Triples and JSON-LD. +- πŸ“– **Pagination, sorting and filtering** with [Atomic Collections](https://docs.atomicdata.dev/schema/collections.html). +- πŸš€ **Fast**: under 1ms median response time, powered by actix-web and redb. +- πŸͺΆ **One binary** (~70MB): server, web app, search, database and automatic HTTPS. +- πŸ’» **Runs everywhere**: linux, windows, mac, arm, plus desktop and mobile apps. +- πŸ“š **Libraries** for JavaScript, React, Svelte, Rust and Dart / Flutter. https://user-images.githubusercontent.com/2183313/139728539-d69b899f-6f9b-44cb-a1b7-bbab68beac0c.mp4 diff --git a/docs/src/atomic-server.md b/docs/src/atomic-server.md index 954f39976..1c4f14cc0 100644 --- a/docs/src/atomic-server.md +++ b/docs/src/atomic-server.md @@ -8,29 +8,32 @@ AtomicServer is a real-time headless CMS, graph database server for storing and It's free, open source (MIT license), and has a ton of features: -- 🏠 **Local-first**: create and edit data with no server at all. Resources are addressed by [`did:ad` identifiers](https://docs.atomicdata.dev/did) and resolve peer-to-peer over the Mainline DHT, so an identity is a keypair you hold rather than an account on someone else's machine. Edits are signed CRDT commits that merge when you reconnect. -- πŸ”’ **Encrypted at rest, per agent**: each agent's in-browser database is encrypted with XChaCha20-Poly1305, under a key wrapped by that agent's own private key. Signing out leaves the cache in place but unreadable to the next session β€” no wipe required. -- πŸ”‘ **Passkey-backed recovery**: a WebAuthn passkey wraps the backup of your agent secret (Argon2id + AES-GCM), so onboarding hands you nothing to write down, and a lost device doesn't have to mean a lost account. -- πŸš€ **Fast** (less than 1ms median response time on my laptop), powered by [actix-web](https://github.com/actix/actix-web) and [redb](https://github.com/cberner/redb) -- πŸͺΆ **One self-contained binary** (~70MB): server, web app, full-text search and database in a single file, with no runtime dependencies and nothing to install alongside it. -- πŸ’» **Runs everywhere** (linux, windows, mac, arm) -- πŸ”§ **Custom data models**: create your own classes, properties and schemas using the built-in Ontology Editor. All data is verified and the models are sharable using [Atomic Schema](https://docs.atomicdata.dev/schema/intro.html) -- βš™οΈ **Restful API**, with [JSON-AD](https://docs.atomicdata.dev/core/json-ad.html) responses. -- πŸ”Ž **Full-text search** with prefix typeahead and 1-edit fuzzy matching, often <3ms responses. Same KV index on the server and in the browser. -- ✨ **AI** with [MCP](https://modelcontextprotocol.io/) support, use any model via OpenRouter or host your own with Ollama. -- πŸ—„οΈ **Tables**, with strict schema validation, keyboard support, copy / paste support. Similar to Airtable. -- πŸ“„ **Documents**, collaborative, rich text, similar to Google Docs / Notion. -- πŸ’¬ **Group chat**, performant and flexible message channels with attachments, search and replies. -- πŸ“‚ **File management**: Upload, download and preview attachments. -- πŸ’Ύ **Versioning** / history from the Loro oplog, with writes authorized by [Atomic Commits](https://docs.atomicdata.dev/commits/intro.html) -- πŸ”„ **Real-time synchronization**: instantly communicates state changes with a client. Build dynamic, collaborative apps using [websockets](https://docs.atomicdata.dev/websockets) (using a [single one-liner in react](https://docs.atomicdata.dev/usecases/react) or [svelte](https://docs.atomicdata.dev/svelte)). -- 🧰 **Many serialization options**: to JSON, [JSON-AD](https://docs.atomicdata.dev/core/json-ad.html), and various Linked Data / RDF formats (RDF/XML, N-Triples / Turtle / JSON-LD). -- πŸ“– **Pagination, sorting and filtering** queries using [Atomic Collections](https://docs.atomicdata.dev/schema/collections.html). -- πŸ” **Authorization** (read / write permissions) and Hierarchical structures powered by [Atomic Hierarchy](https://docs.atomicdata.dev/hierarchy.html) -- πŸ“² **Invite and sharing system** with [Atomic Invites](https://docs.atomicdata.dev/invitations.html) -- 🌐 **Embedded server** with support for HTTP / HTTPS / HTTP2.0 (TLS) and Built-in LetsEncrypt handshake. -- πŸ“± **Runs on mobile**: `atomic_lib` compiles into Flutter apps through [flutter_rust_bridge](https://github.com/fzyzcjy/flutter_rust_bridge), so phones get the same local-first store, signing and peer sync as the browser β€” not a thin REST wrapper. -- πŸ“š **Libraries**: [Javascript / Typescript](https://www.npmjs.com/package/@tomic/lib), [React](https://www.npmjs.com/package/@tomic/react), [Svelte](https://www.npmjs.com/package/@tomic/svelte), [Rust](https://crates.io/crates/atomic-lib), and a Dart / Flutter client +- 🏠 **Local-first**: works offline in the browser, syncs when you reconnect. +- πŸ”„ **Real-time collaboration**: live cursors, typing indicators, and following what a teammate is doing. +- πŸ“„ **Documents**: collaborative rich text, like Google Docs or Notion. +- πŸ—„οΈ **Tables**: strict schema, keyboard navigation, copy / paste. Like Airtable. +- πŸ“‹ **Kanban, calendar, dashboard and timer views** on any table. +- 🧰 **Templates**: issue tracker, CRM, project tasks, time tracker and more. +- 🌐 **Websites**: design pages in the browser, publish in one click, roll back any version. +- πŸ—‚οΈ **Virtual drive**: mount your drive as a folder in Finder or Explorer (desktop app). +- ✨ **AI** with [MCP](https://modelcontextprotocol.io/) support, any model via OpenRouter or local Ollama. +- 🧩 **Apps**: custom screens in plain JavaScript, backed by your own data. +- πŸ”Œ **Plugins and integrations**: Wasm plugins, and syncing from Notion, GitHub and more. +- πŸ’¬ **Group chat**: channels with attachments, search and replies. +- πŸŽ₯ **Meetings**: video calls with shared notes and presence. +- 🎨 **Canvas**: an infinite drawing surface, shared live. +- πŸ“‚ **Files**: upload, download and preview attachments. +- πŸ”§ **Custom data models**: your own classes and properties in the Ontology Editor, shared as [Atomic Schema](https://docs.atomicdata.dev/schema/intro.html). +- πŸ”’ **Encrypted at rest**: each agent's local database is encrypted under their own key. +- πŸ’Ύ **Versioning**: full history, every write a signed [Atomic Commit](https://docs.atomicdata.dev/commits/intro.html). +- πŸ”Ž **Full-text search**: typeahead and fuzzy, often under 3ms, same index on server and client. +- πŸ” **Authorization**: read / write rights, [hierarchies](https://docs.atomicdata.dev/hierarchy.html) and [invite links](https://docs.atomicdata.dev/invitations.html). +- βš™οΈ **RESTful API** with [JSON-AD](https://docs.atomicdata.dev/core/json-ad.html), plus RDF, Turtle, N-Triples and JSON-LD. +- πŸ“– **Pagination, sorting and filtering** with [Atomic Collections](https://docs.atomicdata.dev/schema/collections.html). +- πŸš€ **Fast**: under 1ms median response time, powered by actix-web and redb. +- πŸͺΆ **One binary** (~70MB): server, web app, search, database and automatic HTTPS. +- πŸ’» **Runs everywhere**: linux, windows, mac, arm, plus desktop and mobile apps. +- πŸ“š **Libraries** for JavaScript, React, Svelte, Rust and Dart / Flutter. ## Document undo and redo From 03c1a38fc239101f942ee75a1ec8fc012037aafd Mon Sep 17 00:00:00 2001 From: Joep Meindertsma Date: Sat, 19 Sep 2026 17:35:04 +0200 Subject: [PATCH 022/110] Converge JS and WASM plugins: one host, one manifest, one install path (#1546) (#1571) * Generalize PluginRelease to both runtimes and add Release, Installation and Listing classes * Expose validate_plugin_zip so publish can check a package before storing it * Add one host core for sandboxed plugins One HostCore carries the effective grant (caller, never sudo, bounded by the installation identity), the fetch policy, secret substitution, the pinned egress check and a streaming response cap, plus get_config, get_plugin_agent and commit. commit is refused for proposal-only (extension world) installations. One shared wasmtime Engine and one fuel/memory policy keyed by runtime and capability live here too. * Make the JS StoreHost a thin delegate to HostCore Reads and queries now go through the shared grant, fetch through the shared policy, and the runtime uses the shared engine and limits. The PluginHost trait moves to host_core and is re-exported here. * Make the wasm class-extender host a thin delegate to HostCore The host impl only translates WIT types now. The server WIT copy is synced with atomic-plugin/wit so get-plugin-agent, which the guest SDK already exposes, is served. * Install both runtimes through an Installation commit and publish wasip2 releases from zip bytes * Add the plugin runtime convergence plan for #1546 * Add manifest schemaVersion 2 shared by JS and WASM plugins One versioned manifest carries runtime, world, entrypoints, capabilities and network origins. v1 upgrades with defaults and serializes unchanged, so existing release ids are stable. plugin.json translates at the boundary. Fixtures under testdata/plugin-manifest are shared with the TS mirror. * Mirror manifest v2 parsing in @tomic/lib against the shared fixtures * Document manifest v2 and mark plugin.json as the translated legacy form * Add Release, Installation and Listing to the browser server ontology * Add installRelease, publishZipRelease and the install review reader to @tomic/lib * Publish uploaded zips as releases and install them through the Installation review * Install Store listings through the Installation review, keeping drafts as a secondary action * Add the Installation view with pause, revoke and uninstall, and list installations on the drive * Cover the Installation review screen in the plugin and Store e2e specs * Fix padding-line lint in the manifest v2 mirror * Derive plugin resource limits from v2 capabilities for both hosts * Publish wasip2 releases with a v2 manifest and enforce the Installation grant set * Serve wasip2 releases and their zip from /plugin-package * Record every published release as a Release resource and serve private releases to its readers * Record the plugin convergence implementation checkpoint * Treat schemaVersion 2 as valid in the app package malformed-release test * Box the manifest in FetchPolicy to keep the enum small * browser: remove the legacy Plugin view and zip reader Legacy Plugin resources are migrated into Installations by the server at startup, so the browser no longer renders or manages class Plugin: PluginPage, PluginCard, UpdatePluginButton, useCreatePlugin (uninstall/update) and the client-side zip reader go. The server validates the uploaded zip and translates plugin.json, so NewPluginButton no longer inspects it. AssignRights, PermissionRow and ConfigReference move under views/Installation, their only user. PluginPermissions and pluginUtils fold into a shared CapabilityList (extracted from InstallationReviewDialog) that also renders the pluginPermissions the server writes on an Installation. InstallationPage shows the manifest author the server fills in. * browser: drive plugin list reads Installations only The drive's plugins property was the legacy Plugin list; every plugin is an Installation found by class under the drive now. * lib: drop unused manifest and install exports resolveManifest/ResolvedManifest had no caller outside its test; the server resolves defaults, the browser validates and reviews. Also removes the unused RUNTIME_WASIP2, WORLD_SERVER_EXTENSION and INSTALLATION_STATUSES constants and un-exports the two helper types nothing imported. The Store compares against RUNTIME_JS instead of a literal. * e2e: align plugin spec with the Installation page * Replace the legacy Plugin install hook with a startup migration and one manifest per plugin PluginMeta now holds a single unified manifest (JSON-encoded records; legacy MessagePack records upgrade on read and are translated from plugin.json the first time the plugin loads). Legacy Plugin + pluginFile resources are rewritten at startup into active Installations pinned to a Release published from their zip, keeping subject and on-disk files. Re-activating an already materialized release no longer re-extracts. * Replace the KV plugin catalog with Listing resources and collapse the publish paths A public publish now records a publicly readable Listing at /listings/ next to the Release; /plugin-catalog is a class query over Listings and answers with the Listing fields plus release URL, id, runtime and world. The three publish handlers share release::publish_release for recording, listing and world checks. CatalogEntry and the plugin-catalog/v1 keys are gone; the KV release store stays as the cache it is. * Document the Installation-only plugin flow and record the cleanup checkpoint Docs describe Release, Listing and Installation, the Store and zip-upload paths that both end in the Installation review, and how legacy Plugin resources are migrated. The planning checkpoint lists what this cleanup removed and what is still open. * Read the flat Listing shape from /plugin-catalog in the Store * Pin a materialized install to its release id, and make pausing stop a plugin Two things an Installation got wrong once it was the single install path. An activation decided it had nothing to materialize by comparing the installed manifest with the release's. Two releases of one plugin agree on every manifest field and differ in their package bytes, so repointing an Installation at a new release skipped the extraction and left the previous code serving under the new release's id. `PluginMeta` now records the release whose code is on disk and that is what the check compares; a record from before the field existed says nothing about the bytes, so it materializes again. The legacy migration records it too, so it still re-extracts nothing. `paused` and `draft` did nothing at all. The hook handled `active` and `revoked` and fell through for the rest, and no runtime path read the status, so Pause wrote a property while the plugin kept serving its hooks. Pausing now unregisters the class extender, which is not an uninstall: the files and `PluginMeta` stay, so resuming re-registers the same agent rather than minting a new identity. `installation::resolve` refuses anything but `active`, which stops JS runs, and the startup loader skips extenders whose Installation is not active, so a pause survives a restart. The wasip2 install test asserted the old behaviour, that pausing and resuming leaves the extender registered. It now asserts what pausing is for, and the 'must not re-extract' case it was really about moved to a config change, which is where it happens in practice. * Update an installed plugin in place, rather than uninstalling it first The Installation page showed its pinned release read-only and the only way to get newer code onto a drive was Uninstall then install again. That retires the plugin's agent, so everything it owned stopped being its own, and a second install of the same namespace and name is refused while the first one is there. Uploading a zip on the Installation page now publishes it as a private release, reviews it through the dialog the Store and the zip upload already use, and repoints this same resource at it. The subject, the config and the agent stay; only the release, the grants and the version change, and they change in one commit, because the server checks the grants against the new release's manifest and would refuse a release that arrived on its own. The review dialog takes the wording as a prop and starts its config editor from the config in use rather than the release default, since for an update that is what the installer is deciding about. A zip for a different plugin is refused in the page with the names in the message, ahead of the server refusing it on the identifier comparison. * Record the materialized release from the install hook, and cover the encoding Which release's code is on disk is the Installation's business, not the zip extractor's, so `activate` writes it after `install_or_update_plugin` returns rather than passing it down. A failed extraction then leaves no id, and no id means materialize again, which is the safe direction. It also keeps that function at seven arguments, so clippy gains nothing new. `PluginMeta`'s encoding test constructed the struct literally and so did not compile. It now round-trips the id, and a second test pins the compatibility that matters: a record written before the field existed reads back as `None` rather than as some release, and is written out without the key. * Format the touched browser files with oxfmt The browser workspace formats with oxfmt, not prettier, and `pnpm lint` runs `oxfmt --check` in each package. Four files had been written by prettier, which reformatted them whole, so `lib lint` and `data-browser lint` both failed on the format check. Running the repo's own formatter over them puts them back, and the net change against the previous head is once again only the lines this branch actually adds. `pnpm run -r lint` now exits 0 across the workspace. --- .../src/chunks/Plugins/CapabilityList.tsx | 121 ++ .../Plugins/InstallationReviewDialog.tsx | 231 +++ .../src/chunks/Plugins/NewPluginButton.tsx | 237 +-- .../src/chunks/Plugins/UpdatePluginButton.tsx | 203 -- .../src/chunks/Plugins/plugins.ts | 97 - browser/data-browser/src/helpers/iconMap.ts | 1 + .../src/routes/IntegrationStore.tsx | 220 ++- .../src/views/Card/ResourceCard.tsx | 3 - .../src/views/Drive/PluginList.tsx | 60 +- .../{Plugin => Installation}/AssignRights.tsx | 11 +- .../ConfigReference.tsx | 0 .../views/Installation/InstallationPage.tsx | 445 +++++ .../PermissionRow.tsx | 0 .../src/views/Plugin/PluginCard.tsx | 29 - .../src/views/Plugin/PluginPage.tsx | 177 -- .../src/views/Plugin/PluginPermissions.tsx | 89 - .../src/views/Plugin/createPlugin.ts | 118 -- .../src/views/Plugin/pluginUtils.ts | 28 - .../data-browser/src/views/ResourcePage.tsx | 6 +- .../tests/integration-store-install.spec.ts | 82 + browser/e2e/tests/plugin.spec.ts | 15 +- browser/lib/src/app-package.test.ts | 3 +- browser/lib/src/index.ts | 25 + browser/lib/src/ontologies/server.ts | 141 ++ browser/lib/src/plugin-install.test.ts | 357 ++++ browser/lib/src/plugin-install.ts | 355 ++++ browser/lib/src/plugin-manifest.test.ts | 39 +- browser/lib/src/plugin-manifest.ts | 386 +++- docs/src/plugins.md | 30 +- docs/src/plugins/creating-plugins.md | 84 +- lib/defaults/plugins.json | 243 ++- lib/src/class_extender.rs | 7 +- lib/src/db.rs | 8 + lib/src/db/encoding.rs | 110 +- lib/src/db/plugin_meta.rs | 97 +- lib/src/db/plugin_release.rs | 359 +++- lib/src/urls.rs | 22 +- planning/plugin-runtime-convergence.md | 357 ++++ server/src/appstate.rs | 25 +- server/src/handlers/mod.rs | 2 + server/src/handlers/plugin_release.rs | 310 +++- server/src/handlers/plugin_release_test.rs | 263 +++ server/src/handlers/plugin_schedule.rs | 3 +- server/src/handlers/plugin_secret.rs | 3 +- server/src/handlers/plugin_trigger.rs | 3 +- server/src/plugins/actions.rs | 9 +- server/src/plugins/host_core.rs | 1249 +++++++++++++ server/src/plugins/installation.rs | 41 +- server/src/plugins/js_runtime.rs | 243 +-- server/src/plugins/manifest.rs | 572 +++++- server/src/plugins/mod.rs | 4 + server/src/plugins/notion_sync_tests.rs | 3 +- server/src/plugins/plugin.rs | 1634 +++++++++++++---- server/src/plugins/release.rs | 684 +++++++ server/src/plugins/scheduler.rs | 33 +- server/src/plugins/sync_session.rs | 18 +- server/src/plugins/sync_session_tests.rs | 7 +- server/src/plugins/sync_worker.rs | 6 +- server/src/plugins/triggers.rs | 16 +- server/src/plugins/wasm.rs | 803 ++++---- server/src/routes.rs | 8 + server/wit/class-extender.wit | 20 + testdata/plugin-manifest/index.json | 372 ++++ .../plugin-json/network-without-origins.json | 11 + .../plugin-json/no-classes.json | 5 + .../plugin-json/notion-sync.json | 41 + .../translated-no-classes.json | 9 + .../translated-notion-sync.json | 45 + testdata/plugin-manifest/translations.json | 24 + testdata/plugin-manifest/unknown-version.json | 3 + .../plugin-manifest/v1-bad-secret-origin.json | 9 + .../plugin-manifest/v1-credentials-url.json | 11 + .../v1-duplicate-operation.json | 17 + .../plugin-manifest/v1-github-issues.json | 92 + testdata/plugin-manifest/v1-post-query.json | 11 + testdata/plugin-manifest/v1-public-read.json | 11 + .../v1-secret-description.json | 10 + .../plugin-manifest/v1-unknown-effect.json | 11 + .../plugin-manifest/v1-unknown-field.json | 4 + .../plugin-manifest/v1-with-v2-field.json | 4 + .../v2-action-undeclared-operation.json | 16 + testdata/plugin-manifest/v2-bad-name.json | 5 + .../plugin-manifest/v2-bad-namespace.json | 5 + .../v2-capability-unknown-field.json | 9 + .../plugin-manifest/v2-classes-not-url.json | 9 + .../v2-config-schema-not-object.json | 4 + .../v2-default-config-not-object.json | 6 + .../v2-duplicate-capability.json | 10 + .../plugin-manifest/v2-duplicate-classes.json | 10 + .../plugin-manifest/v2-empty-classes.json | 7 + .../v2-entrypoints-unknown-field.json | 7 + .../v2-extension-with-classes.json | 10 + testdata/plugin-manifest/v2-js-extension.json | 53 + .../v2-js-server-extension-no-classes.json | 5 + .../v2-js-server-extension.json | 10 + testdata/plugin-manifest/v2-minimal.json | 3 + .../v2-network-origin-duplicate.json | 9 + .../v2-network-origin-path.json | 8 + .../v2-network-origin-slash.json | 8 + .../v2-network-unknown-field.json | 9 + .../v2-secret-origin-path.json | 9 + .../v2-unknown-capability.json | 6 + .../plugin-manifest/v2-unknown-field.json | 4 + .../plugin-manifest/v2-unknown-runtime.json | 4 + .../plugin-manifest/v2-unknown-world.json | 4 + .../plugin-manifest/v2-view-traversal.json | 10 + .../v2-view-without-capability.json | 7 + .../v2-wasm-extension-run.json | 16 + .../v2-wasm-server-extension.json | 26 + testdata/plugin-manifests.json | 111 -- 110 files changed, 9454 insertions(+), 2391 deletions(-) create mode 100644 browser/data-browser/src/chunks/Plugins/CapabilityList.tsx create mode 100644 browser/data-browser/src/chunks/Plugins/InstallationReviewDialog.tsx delete mode 100644 browser/data-browser/src/chunks/Plugins/UpdatePluginButton.tsx delete mode 100644 browser/data-browser/src/chunks/Plugins/plugins.ts rename browser/data-browser/src/views/{Plugin => Installation}/AssignRights.tsx (93%) rename browser/data-browser/src/views/{Plugin => Installation}/ConfigReference.tsx (100%) create mode 100644 browser/data-browser/src/views/Installation/InstallationPage.tsx rename browser/data-browser/src/views/{Plugin => Installation}/PermissionRow.tsx (100%) delete mode 100644 browser/data-browser/src/views/Plugin/PluginCard.tsx delete mode 100644 browser/data-browser/src/views/Plugin/PluginPage.tsx delete mode 100644 browser/data-browser/src/views/Plugin/PluginPermissions.tsx delete mode 100644 browser/data-browser/src/views/Plugin/createPlugin.ts delete mode 100644 browser/data-browser/src/views/Plugin/pluginUtils.ts create mode 100644 browser/e2e/tests/integration-store-install.spec.ts create mode 100644 browser/lib/src/plugin-install.test.ts create mode 100644 browser/lib/src/plugin-install.ts create mode 100644 planning/plugin-runtime-convergence.md create mode 100644 server/src/handlers/plugin_release_test.rs create mode 100644 server/src/plugins/host_core.rs create mode 100644 server/src/plugins/release.rs create mode 100644 testdata/plugin-manifest/index.json create mode 100644 testdata/plugin-manifest/plugin-json/network-without-origins.json create mode 100644 testdata/plugin-manifest/plugin-json/no-classes.json create mode 100644 testdata/plugin-manifest/plugin-json/notion-sync.json create mode 100644 testdata/plugin-manifest/translated-no-classes.json create mode 100644 testdata/plugin-manifest/translated-notion-sync.json create mode 100644 testdata/plugin-manifest/translations.json create mode 100644 testdata/plugin-manifest/unknown-version.json create mode 100644 testdata/plugin-manifest/v1-bad-secret-origin.json create mode 100644 testdata/plugin-manifest/v1-credentials-url.json create mode 100644 testdata/plugin-manifest/v1-duplicate-operation.json create mode 100644 testdata/plugin-manifest/v1-github-issues.json create mode 100644 testdata/plugin-manifest/v1-post-query.json create mode 100644 testdata/plugin-manifest/v1-public-read.json create mode 100644 testdata/plugin-manifest/v1-secret-description.json create mode 100644 testdata/plugin-manifest/v1-unknown-effect.json create mode 100644 testdata/plugin-manifest/v1-unknown-field.json create mode 100644 testdata/plugin-manifest/v1-with-v2-field.json create mode 100644 testdata/plugin-manifest/v2-action-undeclared-operation.json create mode 100644 testdata/plugin-manifest/v2-bad-name.json create mode 100644 testdata/plugin-manifest/v2-bad-namespace.json create mode 100644 testdata/plugin-manifest/v2-capability-unknown-field.json create mode 100644 testdata/plugin-manifest/v2-classes-not-url.json create mode 100644 testdata/plugin-manifest/v2-config-schema-not-object.json create mode 100644 testdata/plugin-manifest/v2-default-config-not-object.json create mode 100644 testdata/plugin-manifest/v2-duplicate-capability.json create mode 100644 testdata/plugin-manifest/v2-duplicate-classes.json create mode 100644 testdata/plugin-manifest/v2-empty-classes.json create mode 100644 testdata/plugin-manifest/v2-entrypoints-unknown-field.json create mode 100644 testdata/plugin-manifest/v2-extension-with-classes.json create mode 100644 testdata/plugin-manifest/v2-js-extension.json create mode 100644 testdata/plugin-manifest/v2-js-server-extension-no-classes.json create mode 100644 testdata/plugin-manifest/v2-js-server-extension.json create mode 100644 testdata/plugin-manifest/v2-minimal.json create mode 100644 testdata/plugin-manifest/v2-network-origin-duplicate.json create mode 100644 testdata/plugin-manifest/v2-network-origin-path.json create mode 100644 testdata/plugin-manifest/v2-network-origin-slash.json create mode 100644 testdata/plugin-manifest/v2-network-unknown-field.json create mode 100644 testdata/plugin-manifest/v2-secret-origin-path.json create mode 100644 testdata/plugin-manifest/v2-unknown-capability.json create mode 100644 testdata/plugin-manifest/v2-unknown-field.json create mode 100644 testdata/plugin-manifest/v2-unknown-runtime.json create mode 100644 testdata/plugin-manifest/v2-unknown-world.json create mode 100644 testdata/plugin-manifest/v2-view-traversal.json create mode 100644 testdata/plugin-manifest/v2-view-without-capability.json create mode 100644 testdata/plugin-manifest/v2-wasm-extension-run.json create mode 100644 testdata/plugin-manifest/v2-wasm-server-extension.json delete mode 100644 testdata/plugin-manifests.json diff --git a/browser/data-browser/src/chunks/Plugins/CapabilityList.tsx b/browser/data-browser/src/chunks/Plugins/CapabilityList.tsx new file mode 100644 index 000000000..77fb4f6cb --- /dev/null +++ b/browser/data-browser/src/chunks/Plugins/CapabilityList.tsx @@ -0,0 +1,121 @@ +import { Column, Row } from '@components/Row'; +import type { ReviewCapability } from '@tomic/react'; +import { + FaDesktop, + FaFire, + FaGlobe, + FaHardDrive, + FaKey, + FaMemory, + FaShield, +} from 'react-icons/fa6'; +import { styled } from 'styled-components'; + +const LABELS: Record = { + network: 'Network', + storage: 'Storage', + 'full-drive-access': 'Full Drive Access', + 'extended-fuel': 'Extended Fuel', + 'extended-memory': 'Extended Memory', + 'custom-view': 'Custom View', +}; + +const ICONS: Record = { + network: , + storage: , + 'full-drive-access': , + 'extended-fuel': , + 'extended-memory': , + 'custom-view': , +}; + +function iconFor(capability: ReviewCapability): React.ReactNode { + if (ICONS[capability.title]) return ICONS[capability.title]; + + switch (capability.kind) { + case 'secret': + return ; + case 'operation': + case 'network': + return ; + default: + return ; + } +} + +/** + * The `pluginPermissions` the server writes onto an Installation + * (`[{ permission, reason }]`), as review lines. Anything malformed is left + * out rather than trusted. + */ +export function capabilitiesFromPermissions( + permissions: unknown, +): ReviewCapability[] { + if (!Array.isArray(permissions)) return []; + + return permissions.flatMap(entry => { + if (!entry || typeof entry !== 'object') return []; + const { permission, reason } = entry as Record; + if (typeof permission !== 'string') return []; + + return [ + { + kind: 'permission' as const, + title: permission, + reason: typeof reason === 'string' ? reason : undefined, + grant: permission, + }, + ]; + }); +} + +/** What a plugin asks for and why, on the review dialog and the Installation page. */ +export const CapabilityList: React.FC<{ + capabilities: ReviewCapability[]; + title?: string; +}> = ({ capabilities, title = 'What it can do' }) => ( + +

{title}

+ + {capabilities.length === 0 && ( +
  • +

    No permissions required

    +
  • + )} + {capabilities.map(capability => ( +
  • + + {iconFor(capability)} {LABELS[capability.title] ?? capability.title} + +

    {capability.reason || 'No reason provided'}

    +
  • + ))} +
    +
    +); + +const List = styled.ul` + display: flex; + flex-direction: column; + gap: ${p => p.theme.size()}; + padding: 0; + margin: 0; + + li { + background-color: ${p => p.theme.colors.bg1}; + border-radius: ${p => p.theme.radius}; + list-style: none; + padding: ${p => p.theme.size()}; + margin: 0; + + p { + margin: 0; + } + } +`; + +const CapabilityTitle = styled(Row)` + font-weight: bold; + font-size: 0.9rem; + color: ${p => p.theme.colors.textLight}; +`; diff --git a/browser/data-browser/src/chunks/Plugins/InstallationReviewDialog.tsx b/browser/data-browser/src/chunks/Plugins/InstallationReviewDialog.tsx new file mode 100644 index 000000000..418932133 --- /dev/null +++ b/browser/data-browser/src/chunks/Plugins/InstallationReviewDialog.tsx @@ -0,0 +1,231 @@ +import { Button } from '@components/Button'; +import { Dialog, useDialog } from '@components/Dialog'; +import { JSONEditor } from '@components/JSONEditor'; +import Markdown from '@components/datatypes/Markdown'; +import { Column, Row } from '@components/Row'; +import { ConfigReference } from '@views/Installation/ConfigReference'; +import { CapabilityList } from './CapabilityList'; +import { + grantsFor, + type InstallationReview, + type JSONValue, + type ReleaseReference, +} from '@tomic/react'; +import type { JSONSchema7 } from 'ai'; +import { useEffect, useId, useState } from 'react'; +import toast from 'react-hot-toast'; +import { styled } from 'styled-components'; + +/** What the review screen needs, whichever path produced the release. */ +export interface PendingInstallation { + review: InstallationReview; + release: ReleaseReference; + /** Shown when the manifest carries no name (catalog JS releases). */ + title?: string; + description?: string; + emoji?: string; + /** The config already in use, for an update. Overrides the release default. */ + currentConfig?: JSONValue; +} + +interface InstallationReviewDialogProps { + pending: PendingInstallation | undefined; + /** Called after the dialog closed, whether or not anything was installed. */ + onClose: () => void; + onInstall: ( + pending: PendingInstallation, + config: JSONValue | undefined, + grants: string[], + ) => Promise; + /** Alternative to installing, e.g. creating an editable draft. */ + secondary?: { + label: string; + onClick: (pending: PendingInstallation) => Promise | void; + }; + /** + * What this review is for. Updating an installed plugin shows the same + * capabilities and config, so only the wording differs. + */ + verb?: { + title: string; + confirm: string; + busy: string; + }; +} + +const INSTALL_VERB = { + title: 'Install plugin', + confirm: 'Install', + busy: 'Installing…', +}; + +/** + * The one Installation review screen. The Store, the zip upload and the + * paste-source path all end here: what the plugin asks for and why, its + * config, and the release id that will be pinned. + */ +export const InstallationReviewDialog: React.FC< + InstallationReviewDialogProps +> = ({ pending, onClose, onInstall, secondary, verb = INSTALL_VERB }) => { + const configLabelId = useId(); + const [config, setConfig] = useState(); + const [configValid, setConfigValid] = useState(true); + const [configSyntaxValid, setConfigSyntaxValid] = useState(true); + const [busy, setBusy] = useState(false); + const [dialogProps, show, hide] = useDialog({ + onCancel: onClose, + onSuccess: onClose, + }); + + useEffect(() => { + if (!pending) return; + setConfig(pending.currentConfig ?? pending.review.defaultConfig); + setConfigValid(true); + setConfigSyntaxValid(true); + show(); + }, [pending, show]); + + if (!pending) return null; + + const { review } = pending; + const title = + review.name && review.namespace + ? `${review.namespace}/${review.name}` + : (review.name ?? pending.title ?? 'plugin'); + const description = review.description ?? pending.description; + const hasConfig = + review.configSchema !== undefined || + review.defaultConfig !== undefined || + pending.currentConfig !== undefined; + + const run = async (action: () => Promise | void) => { + setBusy(true); + + try { + await action(); + hide(true); + } catch (err) { + toast.error(err instanceof Error ? err.message : String(err)); + } finally { + setBusy(false); + } + }; + + return ( + + +

    {verb.title}

    +
    + + +
    + + + {pending.emoji && {pending.emoji}} + {title} + + {review.version && v{review.version}} + + + {review.author && by {review.author} Β· } + {review.runtime} + {review.world !== 'extension' && Β· {review.world}} + +
    + {description && ( + + + + )} + + {hasConfig && ( + <> + + { + try { + setConfig(JSON.parse(value)); + setConfigSyntaxValid(true); + } catch { + setConfigSyntaxValid(false); + } + }} + schema={review.configSchema as JSONSchema7 | undefined} + showErrorStyling={!configValid} + onValidationChange={setConfigValid} + /> + {review.configSchema && ( + + )} + + )} +
    + Release + + Pinned to {pending.release.id} + {pending.release.url !== pending.release.id && ( + <> +
    + from {pending.release.url} + + )} +
    +
    +
    +
    + + + {secondary && ( + + )} + + +
    + ); +}; + +const PluginName = styled.span` + font-weight: bold; +`; + +const Meta = styled.p` + color: ${p => p.theme.colors.textLight}; + margin: 0; +`; + +const DescriptionWrapper = styled.div` + background-color: ${p => p.theme.colors.bg1}; + padding: ${p => p.theme.size()}; + border-radius: ${p => p.theme.radius}; +`; + +const Label = styled.label` + font-weight: bold; +`; + +const Identity = styled.p` + overflow-wrap: anywhere; + font-size: 0.8rem; + color: ${p => p.theme.colors.textLight}; +`; diff --git a/browser/data-browser/src/chunks/Plugins/NewPluginButton.tsx b/browser/data-browser/src/chunks/Plugins/NewPluginButton.tsx index 0f9744366..de98affc0 100644 --- a/browser/data-browser/src/chunks/Plugins/NewPluginButton.tsx +++ b/browser/data-browser/src/chunks/Plugins/NewPluginButton.tsx @@ -1,191 +1,118 @@ import { Button } from '@components/Button'; -import { Dialog, useDialog } from '@components/Dialog'; -import type { JSONValue, Resource, Server } from '@tomic/react'; -import { useId, useRef, useState } from 'react'; +import { + installRelease, + publishZipRelease, + readInstallationReview, + useStore, + type JSONValue, + type Resource, + type Server, +} from '@tomic/react'; +import { useRef, useState } from 'react'; import { FaPlus } from 'react-icons/fa6'; -import { styled } from 'styled-components'; -import { Column, Row } from '@components/Row'; -import { JSONEditor } from '@components/JSONEditor'; -import Markdown from '@components/datatypes/Markdown'; -import { useCreatePlugin } from '@views/Plugin/createPlugin'; -import { readZip, type PluginMetadata } from './plugins'; -import { ConfigReference } from '@views/Plugin/ConfigReference'; -import { PluginPermissions } from '@views/Plugin/PluginPermissions'; -import toast from 'react-hot-toast'; import { useCustomViews } from '@components/CustomViewProvider'; +import { useNavigateWithTransition } from '@hooks/useNavigateWithTransition'; +import { constructOpenURL } from '@helpers/navigation'; +import { + InstallationReviewDialog, + type PendingInstallation, +} from './InstallationReviewDialog'; interface NewPluginButtonProps { drive: Resource; } +/** + * Uploading a zip publishes it as a private Release on this server and then + * installs that release through the same review screen the Store uses. The + * server materializes the wasm files when the Installation commits. + */ const NewPluginButton: React.FC = ({ drive }) => { - const configLabelId = useId(); + const store = useStore(); + const navigate = useNavigateWithTransition(); + const { refresh: refreshCustomViews } = useCustomViews(); const [error, setError] = useState(); - const [file, setFile] = useState(null); + const [publishing, setPublishing] = useState(false); + const [pending, setPending] = useState(); const fileInputRef = useRef(null); - const [metadata, setMetadata] = useState(); - const [configValid, setConfigValid] = useState(true); - const [config, setConfig] = useState(); - const { refresh: refreshCustomViews } = useCustomViews(); - const { createPluginResource, addPluginToDrive } = useCreatePlugin(); - - const reset = () => { - setError(undefined); - setFile(null); - setMetadata(undefined); - setConfig(undefined); - setConfigValid(true); - if (fileInputRef.current) fileInputRef.current.value = ''; - }; - - const [dialogProps, show, hide] = useDialog({ - onCancel: reset, - onSuccess: async () => { - if (!metadata || !file) { - return setError('Please fill in all fields'); - } - - try { - const plugin = await createPluginResource({ - metadata, - file, - drive, - config, - }); - await addPluginToDrive(plugin, drive); - await refreshCustomViews(); - reset(); - } catch (err) { - toast.error('Failed to install plugin'); - console.error(err); - reset(); - } - }, - }); const handleFileInputChange = async ( e: React.ChangeEvent, ) => { - const targetFile = e.target.files?.[0]; + const file = e.target.files?.[0]; + if (!file) return; + setPublishing(true); + setError(undefined); + + try { + // The server validates the zip and translates its plugin.json into the + // manifest the review shows; nothing is inspected client-side. + const { id, release } = await publishZipRelease( + store, + drive.subject, + file, + ); + setPending({ + review: readInstallationReview({ ...release, id }), + release: { url: id, id }, + }); + } catch (err) { + setError(err instanceof Error ? err.message : String(err)); + } finally { + setPublishing(false); + if (fileInputRef.current) fileInputRef.current.value = ''; + } + }; - if (targetFile) { - try { - const readMetadata = await readZip(targetFile); - setMetadata(readMetadata); - setConfig(readMetadata.defaultConfig); - setFile(targetFile); - setError(undefined); - show(); - } catch (err) { - setError(err.message); - } + const install = async ( + p: PendingInstallation, + config: JSONValue | undefined, + grants: string[], + ) => { + // The server compares these with the zip's manifest, so they must be + // the manifest's own identifiers. + if (!p.review.name || !p.review.namespace) { + throw new Error('The plugin manifest has no name or namespace'); } + + const subject = await installRelease(store, { + drive: drive.subject, + release: p.release, + name: p.review.name, + namespace: p.review.namespace, + description: p.review.description, + version: p.review.version, + config, + grants, + }); + await refreshCustomViews(); + navigate(constructOpenURL(subject)); }; return ( <> - {error &&

    {error}

    } - - -

    Add Plugin

    -
    - - {metadata && ( - -
    - - - {metadata.namespace}/{metadata.name} - - v{metadata.version} - - by {metadata.author} -
    - {metadata.description && ( - - - - )} - - - - { - try { - setConfig(JSON.parse(val)); - } catch (e) { - // Do nothing - } - }} - schema={metadata.configSchema} - showErrorStyling={!configValid} - onValidationChange={setConfigValid} - /> - {metadata.configSchema && ( - - )} -
    - )} - {!metadata && ( - - )} -
    - - - - -
    + {error &&

    {error}

    } + setPending(undefined)} + onInstall={install} + /> ); }; export default NewPluginButton; - -const PluginName = styled.span` - font-weight: bold; -`; - -const DescriptionWrapper = styled.div` - background-color: ${p => p.theme.colors.bg1}; - padding: ${p => p.theme.size()}; - border-radius: ${p => p.theme.radius}; -`; - -const PluginAuthor = styled.span` - color: ${p => p.theme.colors.textLight}; -`; - -const Label = styled.label` - font-weight: bold; -`; diff --git a/browser/data-browser/src/chunks/Plugins/UpdatePluginButton.tsx b/browser/data-browser/src/chunks/Plugins/UpdatePluginButton.tsx deleted file mode 100644 index a27dc692f..000000000 --- a/browser/data-browser/src/chunks/Plugins/UpdatePluginButton.tsx +++ /dev/null @@ -1,203 +0,0 @@ -import { Button } from '@components/Button'; -import { Dialog } from '@components/Dialog'; -import { useDialog } from '@components/Dialog/useDialog'; -import { - server, - useValue, - type JSONValue, - type Resource, - type Server, -} from '@tomic/react'; -import { useRef, useState } from 'react'; -import { FaUpload } from 'react-icons/fa6'; -import { - readZip, - validateConfig, - type PluginMetadata, - type PluginPermission, -} from './plugins'; -import toast from 'react-hot-toast'; -import { Column, Row } from '@components/Row'; -import { useCreatePlugin } from '@views/Plugin/createPlugin'; -import { styled } from 'styled-components'; -import type { JSONSchema7 } from 'ai'; -import { JSONEditor } from '@components/JSONEditor'; -import { WarningBlock } from '@components/WarningBlock'; -import { ConfigReference } from '@views/Plugin/ConfigReference'; -import { isPluginPermissions } from '@views/Plugin/pluginUtils'; -import { PluginPermissions } from '@views/Plugin/PluginPermissions'; -import { useCustomViews } from '@components/CustomViewProvider'; - -interface UpdatePluginButtonProps { - plugin: Resource; -} - -const UpdatePluginButton: React.FC = ({ plugin }) => { - const fileInputRef = useRef(null); - const [file, setFile] = useState(); - const [metadata, setMetadata] = useState(); - const [configValid, setConfigValid] = useState(true); - const [jsonEditorValid, setJsonEditorValid] = useState(true); - const { refresh: refreshCustomViews } = useCustomViews(); - const [updatedConfig, setUpdatedConfig] = useState(); - const [oldPermissions] = useValue( - plugin, - server.properties.pluginPermissions, - ); - - const { updatePlugin } = useCreatePlugin(); - - const newPermissions: PluginPermission[] = []; - - if (isPluginPermissions(oldPermissions) && metadata?.permissions) { - for (const perm of metadata.permissions) { - if (!oldPermissions.some(p => p.permission === perm.permission)) { - newPermissions.push(perm); - } - } - } - - const reset = () => { - fileInputRef.current!.value = ''; - setFile(undefined); - setMetadata(undefined); - setConfigValid(true); - setJsonEditorValid(true); - setUpdatedConfig(undefined); - }; - - const [dialogProps, show, hide] = useDialog({ - onCancel: () => { - reset(); - }, - onSuccess: async () => { - if (!metadata || !file) { - return; - } - - try { - await updatePlugin(plugin, metadata, file, updatedConfig); - await refreshCustomViews(); - } catch (err) { - toast.error(err.message); - } finally { - reset(); - } - }, - }); - - const handleInputChange = async (e: React.ChangeEvent) => { - const targetFile = e.target.files?.[0]; - - if (targetFile) { - try { - const meta = await readZip(targetFile); - - if ( - meta.name !== plugin.props.name || - meta.namespace !== plugin.props.namespace - ) { - toast.error( - "The update's identifier does not match the existing plugin.", - ); - reset(); - - return; - } - - setMetadata(meta); - setFile(targetFile); - const valid = validateConfig( - meta.defaultConfig, - meta.configSchema as JSONSchema7, - ); - setConfigValid(valid); - show(); - } catch (err) { - toast.error(err.message); - reset(); - } - } - }; - - return ( - <> - - - -

    Change Version

    -
    - - {metadata && ( - - - - {plugin.props.version} β†’ {metadata.version} - - - {newPermissions.length > 0 && ( - - )} - {!configValid && ( - <> - - Your config is not fully compatible with the new version. - - { - try { - return setUpdatedConfig(JSON.parse(val)); - } catch (e) { - // Do nothing - } - }} - showErrorStyling={!jsonEditorValid} - onValidationChange={setJsonEditorValid} - /> - - - )} - - )} - - - - - -
    - - ); -}; - -export default UpdatePluginButton; - -const VersionChange = styled.span` - font-weight: bold; - font-size: 1.5rem; -`; diff --git a/browser/data-browser/src/chunks/Plugins/plugins.ts b/browser/data-browser/src/chunks/Plugins/plugins.ts deleted file mode 100644 index 5307c125f..000000000 --- a/browser/data-browser/src/chunks/Plugins/plugins.ts +++ /dev/null @@ -1,97 +0,0 @@ -import type { JSONValue } from '@tomic/react'; -import { - ZipReader, - Uint8ArrayReader, - TextWriter, - configure, - type Entry, -} from '@zip.js/zip.js'; -import type { JSONSchema7 } from 'ai'; -import { Ajv } from 'ajv'; - -// zip.js defaults to decompressing in a Web Worker spawned from a `blob:` -// URL. The production server's CSP is `worker-src 'self'` -// (server/src/handlers/single_page_app.rs), which blocks blob workers β€” so -// reading an uploaded plugin zip throws on a real (CSP-enforced) server and -// the "Add Plugin" dialog never opens. (Dev/Vite has no CSP, so it only -// reproduced in the production bundle.) Plugin zips are tiny, so main-thread -// inflate is fine; this keeps the CSP strict instead of allowing `blob:`. -configure({ useWebWorkers: false }); - -export type PluginPermissionType = - | 'network' - | 'storage' - | 'full-drive-access' - | 'extended-fuel' - | 'extended-memory' - | 'custom-view'; -export interface PluginPermission { - permission: PluginPermissionType; - reason: string; -} - -export interface PluginMetadata { - name: string; - namespace: string; - author?: string; - description?: string; - version: string; - permissions?: PluginPermission[]; - defaultConfig?: JSONValue; - configSchema?: JSONSchema7; -} - -export async function readZip(file: File): Promise { - const zip = new ZipReader(new Uint8ArrayReader(await file.bytes())); - const entries = await zip.getEntries(); - - if (!validateZip(entries)) { - throw new Error('Invalid plugin zip file.'); - } - - for (const entry of entries) { - if (!entry.directory && entry.filename === 'plugin.json') { - const metadata = await entry.getData(new TextWriter()); - - return JSON.parse(metadata) as PluginMetadata; - } - } - - throw new Error('Plugin metadata not found in zip file.'); -} - -function validateZip(entries: Entry[]): boolean { - const allowedRootFiles = ['plugin.json', 'plugin.wasm', 'ui.js', 'ui.css']; - let foundWasm = false; - let foundJson = false; - - for (const entry of entries) { - if (entry.filename.startsWith('assets/')) { - continue; - } - - if (!allowedRootFiles.includes(entry.filename)) { - return false; - } - - if (entry.filename === 'plugin.wasm') { - foundWasm = true; - } - - if (entry.filename === 'plugin.json') { - foundJson = true; - } - } - - return foundWasm && foundJson; -} - -export const validateConfig = ( - config: JSONValue, - schema: JSONSchema7, -): boolean => { - const ajv = new Ajv(); - const validate = ajv.compile(schema); - - return validate(config); -}; diff --git a/browser/data-browser/src/helpers/iconMap.ts b/browser/data-browser/src/helpers/iconMap.ts index a16aeb462..20227535e 100644 --- a/browser/data-browser/src/helpers/iconMap.ts +++ b/browser/data-browser/src/helpers/iconMap.ts @@ -54,6 +54,7 @@ const iconMap = new Map([ [server.classes.file, FaFile], [server.classes.drive, FaHardDrive], [server.classes.plugin, FaPuzzlePiece], + [server.classes.installation, FaPuzzlePiece], [commits.classes.commit, FaClock], [dataBrowser.classes.importer, FaFileImport], [server.classes.invite, FaShareFromSquare], diff --git a/browser/data-browser/src/routes/IntegrationStore.tsx b/browser/data-browser/src/routes/IntegrationStore.tsx index f070865d7..ad0ba15a9 100644 --- a/browser/data-browser/src/routes/IntegrationStore.tsx +++ b/browser/data-browser/src/routes/IntegrationStore.tsx @@ -15,12 +15,23 @@ import { FaPlug } from 'react-icons/fa6'; import { useStore, core, + server, findSchema, pluginSchema, readConnectionSubjects, - type PluginRelease, + installRelease, + installationIdentifier, + readInstallationReview, + DEFAULT_INSTALLATION_NAMESPACE, + RUNTIME_JS, + type JSONValue, + type PublishedRelease, } from '@tomic/react'; import { ResourceInline } from '../views/ResourceInline/ResourceInline'; +import { + InstallationReviewDialog, + type PendingInstallation, +} from '../chunks/Plugins/InstallationReviewDialog'; import toast from 'react-hot-toast'; import { appRoute } from './RootRoutes'; import { pathNames } from './paths'; @@ -36,17 +47,21 @@ import { useNavigateWithTransition } from '@hooks/useNavigateWithTransition'; import { constructOpenURL } from '@helpers/navigation'; import type { IntegrationVisibilityKey } from '@helpers/integrationVisibility'; +/** One entry of `/plugin-catalog`: a public Listing resource on this server. */ interface Listing { - metadata: { - release: string; - emoji?: string; - name: string; - description: string; - publisher: string; - domains: string[]; - standards: string[]; - }; - verification: 'unverified'; + subject: string; + name: string; + emoji: string | null; + description: string; + publisher: string | null; + domains: string[]; + standards: string[]; + /** The Release resource URL an Installation pins. */ + release: string; + /** The `blake3:` id, which `/plugin-package/{id}` takes. */ + releaseId: string; + runtime: string | null; + world: string | null; } export const IntegrationStoreRoute = createRoute({ @@ -75,6 +90,7 @@ function IntegrationStore(): React.JSX.Element { const navigate = useNavigateWithTransition(); const [listings, setListings] = useState(); const [installed, setInstalled] = useState([]); + const [installations, setInstallations] = useState([]); const [automations, setAutomations] = useState([]); const [error, setError] = useState(); const [catalogError, setCatalogError] = useState(); @@ -123,9 +139,40 @@ function IntegrationStore(): React.JSX.Element { active = false; }; }, [store, drive, pluginClass]); + useEffect(() => { + let active = true; + + if (!drive) { + setInstallations([]); + + return; + } + + // Installations are the installed form for both runtimes; connections + // above are the drafts and the legacy plugin-script installs. + void readConnectionSubjects( + store, + drive, + core.properties.isA, + server.classes.installation, + ) + .then(subjects => { + if (active) setInstallations(subjects); + }) + .catch(reason => { + if (active) setError(String(reason)); + }); + + return () => { + active = false; + }; + }, [store, drive]); const [search, setSearch] = useState(''); const [creating, setCreating] = useState(); - const server = store.getServerUrl(); + const [pending, setPending] = useState< + PendingInstallation & { entry: Listing } + >(); + const serverUrl = store.getServerUrl(); useEffect(() => { setCatalogError(undefined); @@ -136,7 +183,7 @@ function IntegrationStore(): React.JSX.Element { } const controller = new AbortController(); - void fetch(`${server}/plugin-catalog`, { signal: controller.signal }) + void fetch(`${serverUrl}/plugin-catalog`, { signal: controller.signal }) .then(async response => { if (!response.ok) throw new Error(await response.text()); const entries = await response.json(); @@ -147,37 +194,32 @@ function IntegrationStore(): React.JSX.Element { }); return () => controller.abort(); - }, [server, showExperimentalPlugins]); + }, [serverUrl, showExperimentalPlugins]); - const createDraft = async (entry: Listing['metadata']) => { - if (!drive) return; - setCreating(entry.release); + const fetchRelease = async (id: string): Promise => { + const response = await fetch( + `${serverUrl}/plugin-package/${encodeURIComponent(id)}`, + ); + if (!response.ok) throw new Error(await response.text()); - try { - const response = await fetch( - `${server}/plugin-package/${encodeURIComponent(entry.release)}`, - ); - if (!response.ok) throw new Error(await response.text()); - const release = (await response.json()) as PluginRelease; - const { createPlugin } = await import('../chunks/PluginRuns/runScript'); - const subject = await createPlugin( - store, - { drive, parent: drive }, - entry.name, - release.source, - release.schemas, - ); + return (await response.json()) as PublishedRelease; + }; - if (entry.emoji) { - const resource = await store.getResource(subject); - await resource.set( - 'https://atomicdata.dev/properties/emoji', - entry.emoji, - ); - await resource.save(); - } + /** Opening a Listing: fetch its release and review it before installing. */ + const openReview = async (entry: Listing) => { + if (!drive) return; + setCreating(entry.releaseId); - navigate(constructOpenURL(subject)); + try { + const release = await fetchRelease(entry.releaseId); + setPending({ + entry, + review: readInstallationReview({ ...release, id: entry.releaseId }), + release: { url: entry.release, id: entry.releaseId }, + title: entry.name, + description: entry.description, + emoji: entry.emoji ?? undefined, + }); } catch (reason) { toast.error(String(reason)); } finally { @@ -185,6 +227,55 @@ function IntegrationStore(): React.JSX.Element { } }; + const install = async ( + p: PendingInstallation, + config: JSONValue | undefined, + grants: string[], + ) => { + if (!drive) return; + const subject = await installRelease(store, { + drive, + release: p.release, + name: p.review.name ?? installationIdentifier(p.title ?? 'plugin'), + namespace: p.review.namespace ?? DEFAULT_INSTALLATION_NAMESPACE, + description: p.review.description ?? p.description, + version: p.review.version, + config, + grants, + }); + navigate(constructOpenURL(subject)); + }; + + /** Drafts remain the authoring form: a copy of the source you can edit. */ + const createDraft = async (entry: Listing) => { + if (!drive) return; + const release = await fetchRelease(entry.releaseId); + + if (!release.source) { + throw new Error('Only JS releases can be opened as an editable draft'); + } + + const { createPlugin } = await import('../chunks/PluginRuns/runScript'); + const subject = await createPlugin( + store, + { drive, parent: drive }, + entry.name, + release.source, + release.schemas ?? {}, + ); + + if (entry.emoji) { + const resource = await store.getResource(subject); + await resource.set( + 'https://atomicdata.dev/properties/emoji', + entry.emoji, + ); + await resource.save(); + } + + navigate(constructOpenURL(subject)); + }; + const query = search.trim().toLocaleLowerCase(); const bundled = visibleBundledIntegrations( showExperimentalPlugins, @@ -194,12 +285,11 @@ function IntegrationStore(): React.JSX.Element { .toLocaleLowerCase() .includes(query), ); - const visible = (showExperimentalPlugins ? listings : [])?.filter( - ({ metadata: entry }) => - [entry.name, entry.description, ...entry.domains, ...entry.standards] - .join(' ') - .toLocaleLowerCase() - .includes(query), + const visible = (showExperimentalPlugins ? listings : [])?.filter(entry => + [entry.name, entry.description, ...entry.domains, ...entry.standards] + .join(' ') + .toLocaleLowerCase() + .includes(query), ); return ( @@ -216,7 +306,7 @@ function IntegrationStore(): React.JSX.Element { you need them.

    - {installed.length > 0 && ( + {(installed.length > 0 || installations.length > 0) && (

    Your connections

    @@ -227,6 +317,11 @@ function IntegrationStore(): React.JSX.Element { drive={drive!} /> ))} + {installations.map(subject => ( + + + + ))}
    )} @@ -297,18 +392,16 @@ function IntegrationStore(): React.JSX.Element { <>

    Community plugins

    - Published code you can adapt. Creating a draft does not - connect an app or enable sync. + Published releases. Open one to review what it can do before + installing it into this drive, or create a draft to adapt its + code.

    )} - {visible?.map(({ metadata: entry }) => ( - + {visible?.map(entry => ( + {entry.emoji || } @@ -353,11 +446,9 @@ function IntegrationStore(): React.JSX.Element { @@ -365,6 +456,19 @@ function IntegrationStore(): React.JSX.Element { + setPending(undefined)} + onInstall={install} + secondary={ + pending && pending.review.runtime === RUNTIME_JS + ? { + label: 'Create draft', + onClick: () => createDraft(pending.entry), + } + : undefined + } + /> ); } diff --git a/browser/data-browser/src/views/Card/ResourceCard.tsx b/browser/data-browser/src/views/Card/ResourceCard.tsx index 3ed506c46..073f2ff79 100644 --- a/browser/data-browser/src/views/Card/ResourceCard.tsx +++ b/browser/data-browser/src/views/Card/ResourceCard.tsx @@ -35,7 +35,6 @@ import { MeetingCard } from './MeetingCard'; import { HideInPrint } from '@components/HideInPrint'; import { useOnValueChange } from '@helpers/useOnValueChange'; import { FolderCard } from './FolderCard'; -import { PluginCard } from '@views/Plugin/PluginCard'; interface ResourceCardProps extends CardViewPropsBase { /** The subject URL - the identifier of the resource. */ @@ -130,8 +129,6 @@ function ResourceCardInner(props: ResourceCardProps): JSX.Element { return ; case dataBrowser.classes.folder: return ; - case server.classes.plugin: - return ; default: return ; } diff --git a/browser/data-browser/src/views/Drive/PluginList.tsx b/browser/data-browser/src/views/Drive/PluginList.tsx index 7a5f04b7a..2a6188644 100644 --- a/browser/data-browser/src/views/Drive/PluginList.tsx +++ b/browser/data-browser/src/views/Drive/PluginList.tsx @@ -1,12 +1,14 @@ import { core, + readConnectionSubjects, server, - useArray, useResource, + useStore, useString, type Resource, type Server, } from '@tomic/react'; +import { useEffect, useState } from 'react'; import type React from 'react'; import { AtomicLink } from '@components/AtomicLink'; import styled from 'styled-components'; @@ -16,37 +18,66 @@ interface PluginListProps { drive: Resource; } +/** + * Every plugin on a drive is an `Installation` (the server migrates legacy + * `Plugin` resources at startup), found by class under the drive like the + * Store does for connections. + */ export const PluginList: React.FC = ({ drive }) => { - // Read via `useArray` (not `drive.props.plugins`) so the component - // re-renders when the array changes. Reading `.props.X` directly inside - // render is memoized by the React Compiler on the stable `drive` proxy - // ref β€” an internal `push()` mutation doesn't change that ref, so a - // direct read would never invalidate and the list would stay stuck on - // "No plugins installed" after a fresh install. - const [plugins] = useArray(drive, server.properties.plugins); + const installations = useInstallations(drive.subject); - if (plugins.length === 0) { + if (installations.length === 0) { return No plugins installed; } return ( - {plugins.map(plugin => ( - + {installations.map(subject => ( + ))} ); }; +function useInstallations(drive: string): string[] { + const store = useStore(); + const [subjects, setSubjects] = useState([]); + + useEffect(() => { + let active = true; + readConnectionSubjects( + store, + drive, + core.properties.isA, + server.classes.installation, + ) + .then(found => { + if (active) setSubjects(found); + }) + .catch(() => { + // Shown as empty; the drive may be on a server without the + // Installation class. + }); + + return () => { + active = false; + }; + }, [store, drive]); + + return subjects; +} + const PluginItem: React.FC<{ subject: string }> = ({ subject }) => { - // Subscribe to each field so the row re-renders when the plugin resource - // finishes loading β€” same React Compiler reasoning as above. - const resource = useResource(subject); + // Subscribe to each field so the row re-renders when the resource finishes + // loading; the React Compiler memoizes direct `.props` reads on the stable + // proxy ref. + const resource = useResource(subject); const [namespace] = useString(resource, server.properties.namespace); const [name] = useString(resource, core.properties.name); const [version] = useString(resource, server.properties.version); + const [status] = useString(resource, server.properties.installationStatus); const title = `${namespace ?? ''}/${name ?? ''}`; @@ -56,6 +87,7 @@ const PluginItem: React.FC<{ subject: string }> = ({ subject }) => { {title} {version} + {status} ); }; diff --git a/browser/data-browser/src/views/Plugin/AssignRights.tsx b/browser/data-browser/src/views/Installation/AssignRights.tsx similarity index 93% rename from browser/data-browser/src/views/Plugin/AssignRights.tsx rename to browser/data-browser/src/views/Installation/AssignRights.tsx index 0f5df0935..5ba99f93a 100644 --- a/browser/data-browser/src/views/Plugin/AssignRights.tsx +++ b/browser/data-browser/src/views/Installation/AssignRights.tsx @@ -20,7 +20,8 @@ import { DashedButton } from '@views/OntologyPage/DashedButton'; import { styled } from 'styled-components'; interface AssignRightsProps { - plugin: Resource; + /** Carries the server-set plugin agent the rights are assigned to. */ + installation: Resource; disabled?: boolean; } @@ -28,18 +29,18 @@ const shouldRender = (resource: Resource) => { return [ commits.classes.commit, dataBrowser.classes.tag, - server.classes.plugin, + server.classes.installation, ].every(c => !resource.hasClasses(c)); }; export const AssignRights: React.FC = ({ - plugin, + installation, disabled, }) => { const store = useStore(); const [selectedResource, setSelectedResource] = useState(); - const pluginAgent = plugin.props.pluginAgent; + const pluginAgent = installation.props.pluginAgent; const { invalidateCollection, mapAll } = useCollection({ property: core.properties.read, @@ -89,7 +90,7 @@ export const AssignRights: React.FC = ({ key={index} collection={collection} index={index} - pluginAgent={plugin.props.pluginAgent ?? ''} + pluginAgent={pluginAgent} onReadUpdate={invalidateCollection} /> ))} diff --git a/browser/data-browser/src/views/Plugin/ConfigReference.tsx b/browser/data-browser/src/views/Installation/ConfigReference.tsx similarity index 100% rename from browser/data-browser/src/views/Plugin/ConfigReference.tsx rename to browser/data-browser/src/views/Installation/ConfigReference.tsx diff --git a/browser/data-browser/src/views/Installation/InstallationPage.tsx b/browser/data-browser/src/views/Installation/InstallationPage.tsx new file mode 100644 index 000000000..b66f4d114 --- /dev/null +++ b/browser/data-browser/src/views/Installation/InstallationPage.tsx @@ -0,0 +1,445 @@ +import { Button } from '@components/Button'; +import { + ConfirmationDialog, + ConfirmationDialogTheme, +} from '@components/ConfirmationDialog'; +import { ContainerNarrow } from '@components/Containers'; +import Markdown from '@components/datatypes/Markdown'; +import { JSONEditor } from '@components/JSONEditor'; +import { Column, Row } from '@components/Row'; +import { useNavigateWithTransition } from '@hooks/useNavigateWithTransition'; +import { + core, + publishZipRelease, + readInstallationReview, + server, + updateInstallationRelease, + useCanWrite, + useSaveState, + useStore, + useString, + useValue, + type InstallationStatus, + type JSONValue, + type Server, +} from '@tomic/react'; +import type { ResourcePageProps } from '@views/ResourcePage'; +import type { JSONSchema7 } from 'ai'; +import { constructOpenURL } from '@helpers/navigation'; +import { useId, useRef, useState } from 'react'; +import { + FaFloppyDisk, + FaGear, + FaPause, + FaPlay, + FaBan, + FaTrash, + FaArrowUp, +} from 'react-icons/fa6'; +import { styled } from 'styled-components'; +import toast from 'react-hot-toast'; +import { + CapabilityList, + capabilitiesFromPermissions, +} from '@chunks/Plugins/CapabilityList'; +import { ConfigReference } from './ConfigReference'; +import { AssignRights } from './AssignRights'; +import { ResourceInline } from '@views/ResourceInline/ResourceInline'; +import { useCustomViews } from '@components/CustomViewProvider'; +import { + InstallationReviewDialog, + type PendingInstallation, +} from '@chunks/Plugins/InstallationReviewDialog'; + +const UPDATE_VERB = { + title: 'Update plugin', + confirm: 'Update', + busy: 'Updating…', +}; + +/** + * One installed release on a drive, for either runtime. Status changes are + * commits the server's Installation hook acts on: `active` installs, + * `paused` unregisters it and keeps everything else, `revoked` or destroying + * uninstalls. Uploading a newer zip repoints this same resource at the new + * release, so the plugin keeps its config and the agent it signs as. + */ +export const InstallationPage: React.FC< + ResourcePageProps +> = ({ resource }) => { + const configLabelId = useId(); + const store = useStore(); + const canWrite = useCanWrite(resource); + const navigate = useNavigateWithTransition(); + const { refresh: refreshCustomViews } = useCustomViews(); + const [confirm, setConfirm] = useState<'revoke' | 'uninstall'>(); + const [name] = useString(resource, core.properties.name); + const [namespace] = useString(resource, server.properties.namespace); + const [version] = useString(resource, server.properties.version); + const [description] = useString(resource, core.properties.description); + const [author] = useString(resource, server.properties.pluginAuthor); + const [release] = useString(resource, server.properties.release); + const [releaseId] = useString(resource, server.properties.releaseId); + const [status, setStatus] = useValue( + resource, + server.properties.installationStatus, + ); + const [grants] = useValue(resource, server.properties.grants); + const [config, setConfig] = useValue(resource, server.properties.config); + const [schema] = useValue(resource, server.properties.jsonSchema); + const [permissions] = useValue(resource, server.properties.pluginPermissions); + const [pluginAgent] = useString(resource, server.properties.pluginAgent); + const [configValid, setConfigValid] = useState(true); + const [configSyntaxValid, setConfigSyntaxValid] = useState(true); + const [configEdited, setConfigEdited] = useState(false); + const saveState = useSaveState(resource); + const [changing, setChanging] = useState(false); + const [pending, setPending] = useState(); + const [publishing, setPublishing] = useState(false); + const zipInputRef = useRef(null); + + const title = `${namespace ? `${namespace}/` : ''}${name ?? ''}`; + const currentStatus = (status as InstallationStatus | undefined) ?? 'draft'; + const declared = capabilitiesFromPermissions(permissions); + const hasFullDriveAccess = declared.some( + c => c.title === 'full-drive-access', + ); + const grantNames = Array.isArray(grants) + ? grants.map(String) + : grants && typeof grants === 'object' + ? Object.keys(grants) + : []; + + const changeStatus = async (next: InstallationStatus) => { + setChanging(true); + + try { + await setStatus(next); + await resource.save(); + await refreshCustomViews(); + toast.success(`Installation ${next}`); + } catch (err) { + toast.error(err instanceof Error ? err.message : String(err)); + } finally { + setChanging(false); + } + }; + + /** + * Publishes the uploaded zip as a release and reviews it. Nothing is + * installed until the review is confirmed; the release itself is private + * to this drive either way. + */ + const handleZipChosen = async (e: React.ChangeEvent) => { + const file = e.target.files?.[0]; + + if (!file) return; + setPublishing(true); + + try { + const drive = resource.get(core.properties.parent) as string; + const { id, release: published } = await publishZipRelease( + store, + drive, + file, + ); + setPending({ + review: readInstallationReview({ ...published, id }), + release: { url: id, id }, + currentConfig: config as JSONValue | undefined, + }); + } catch (err) { + toast.error(err instanceof Error ? err.message : String(err)); + } finally { + setPublishing(false); + if (zipInputRef.current) zipInputRef.current.value = ''; + } + }; + + const applyUpdate = async ( + p: PendingInstallation, + nextConfig: JSONValue | undefined, + nextGrants: string[], + ) => { + // The server compares these with the package, so a zip for a different + // plugin is refused there. Saying so here is the clearer error. + if (p.review.name !== name || p.review.namespace !== namespace) { + throw new Error( + `That package is ${p.review.namespace}/${p.review.name}, not ${title}. A plugin's name and namespace cannot change.`, + ); + } + + await updateInstallationRelease(store, resource.subject, { + release: p.release, + grants: nextGrants, + config: nextConfig, + version: p.review.version, + }); + await refreshCustomViews(); + toast.success('Plugin updated'); + }; + + return ( + + +
    + + + {title} + + {currentStatus} + + + {version && v{version}} + + {author && by {author}} +
    + + {canWrite && ( + + {currentStatus === 'active' && ( + + )} + {(currentStatus === 'paused' || currentStatus === 'draft') && ( + + )} + {currentStatus !== 'revoked' && ( + + )} + {currentStatus !== 'revoked' && ( + + )} + + + )} + {description && ( + + + + )} + + +

    Release

    + + Pinned to {releaseId} + {release && release !== releaseId && ( + <> +
    + from{' '} + {/^https?:\/\//.test(release) ? ( + + {release} + + ) : ( + release + )} + + )} +
    +
    + +

    Grants

    + {grantNames.length === 0 ? ( + No capabilities were granted. + ) : ( + + {grantNames.map(grant => ( + {grant} + ))} + + )} +
    + {pluginAgent && ( + +

    Plugin agent

    + +
    + )} + {canWrite && pluginAgent && ( + + )} + + +

    + + + Config + +

    + {canWrite && ( + + )} +
    + { + try { + setConfig(JSON.parse(v)); + setConfigEdited(true); + setConfigSyntaxValid(true); + } catch { + setConfigSyntaxValid(false); + } + }} + schema={schema as JSONSchema7 | undefined} + showErrorStyling={!configValid} + onValidationChange={setConfigValid} + /> +
    + {schema && } + {declared.length > 0 && ( + + )} +
    + setPending(undefined)} + onInstall={applyUpdate} + verb={UPDATE_VERB} + /> + !open && setConfirm(undefined)} + onConfirm={() => changeStatus('revoked')} + onCancel={() => setConfirm(undefined)} + > + Revoking uninstalls the plugin and retires its agent. The Installation + record stays, so you can see what was installed; installing again needs + a new Installation. + + !open && setConfirm(undefined)} + onConfirm={async () => { + const parent = resource.props.parent; + await resource.destroy(); + await refreshCustomViews(); + navigate(constructOpenURL(parent)); + toast.success('Plugin uninstalled'); + }} + onCancel={() => setConfirm(undefined)} + > + Are you sure you want to uninstall this plugin? This removes the + Installation record. + +
    + ); +}; + +const PluginName = styled.span` + font-weight: bold; + font-size: 1.2rem; +`; + +const PluginAuthor = styled.span` + color: ${p => p.theme.colors.textLight}; +`; + +const StatusBadge = styled.span` + font-size: 0.8rem; + padding: 0.1rem 0.5rem; + border-radius: ${p => p.theme.radius}; + border: 1px solid ${p => p.theme.colors.bg2}; + color: ${p => p.theme.colors.textLight}; + text-transform: capitalize; + + &[data-status='active'] { + color: ${p => p.theme.colors.main}; + border-color: ${p => p.theme.colors.main}; + } +`; + +const DescriptionWrapper = styled.section` + background-color: ${p => p.theme.colors.bg1}; + padding: ${p => p.theme.size()}; + border-radius: ${p => p.theme.radius}; + max-height: 33rem; + overflow-y: auto; +`; + +const Identity = styled.p` + overflow-wrap: anywhere; + font-size: 0.9rem; + color: ${p => p.theme.colors.textLight}; + margin: 0; +`; + +const Muted = styled.p` + color: ${p => p.theme.colors.textLight}; + margin: 0; +`; + +const Tag = styled.span` + border: 1px solid ${p => p.theme.colors.bg2}; + padding: 0.2rem 0.5rem; + border-radius: ${p => p.theme.radius}; + font-size: 0.85rem; +`; diff --git a/browser/data-browser/src/views/Plugin/PermissionRow.tsx b/browser/data-browser/src/views/Installation/PermissionRow.tsx similarity index 100% rename from browser/data-browser/src/views/Plugin/PermissionRow.tsx rename to browser/data-browser/src/views/Installation/PermissionRow.tsx diff --git a/browser/data-browser/src/views/Plugin/PluginCard.tsx b/browser/data-browser/src/views/Plugin/PluginCard.tsx deleted file mode 100644 index a4bbf7b1d..000000000 --- a/browser/data-browser/src/views/Plugin/PluginCard.tsx +++ /dev/null @@ -1,29 +0,0 @@ -import Markdown from '@components/datatypes/Markdown'; -import { HideInPrint } from '@components/HideInPrint'; -import { ResourceContextMenu } from '@components/ResourceContextMenu'; -import { Column, Row } from '@components/Row'; -import { core, server, useResource, useString } from '@tomic/react'; -import type { CardViewProps } from '@views/Card/CardViewProps'; -import { ResourceCardTitle } from '@views/Card/ResourceCardTitle'; - -export const PluginCard: React.FC = ({ resource }) => { - const [name] = useString(resource, core.properties.name); - const [namespace] = useString(resource, server.properties.namespace); - const isAResource = useResource(resource.props.isA[0]); - - const title = `${namespace ? `${namespace}/` : ''}${name}`; - - return ( - - - - {isAResource.title} - - - - - - - - ); -}; diff --git a/browser/data-browser/src/views/Plugin/PluginPage.tsx b/browser/data-browser/src/views/Plugin/PluginPage.tsx deleted file mode 100644 index 20c9c9fe9..000000000 --- a/browser/data-browser/src/views/Plugin/PluginPage.tsx +++ /dev/null @@ -1,177 +0,0 @@ -import { Button } from '@components/Button'; -import { - ConfirmationDialog, - ConfirmationDialogTheme, -} from '@components/ConfirmationDialog'; -import { ContainerNarrow } from '@components/Containers'; -import Markdown from '@components/datatypes/Markdown'; -import { JSONEditor } from '@components/JSONEditor'; -import { Column, Row } from '@components/Row'; -import { useNavigateWithTransition } from '@hooks/useNavigateWithTransition'; -import { - core, - server, - useCanWrite, - useString, - useSaveState, - useValue, - type Server, -} from '@tomic/react'; -import { useCreatePlugin } from '@views/Plugin/createPlugin'; -import type { ResourcePageProps } from '@views/ResourcePage'; -import type { JSONSchema7 } from 'ai'; -import { constructOpenURL } from '@helpers/navigation'; -import { lazy, useId, useState } from 'react'; -import { FaFloppyDisk, FaGear, FaTrash } from 'react-icons/fa6'; -import { styled } from 'styled-components'; -import toast from 'react-hot-toast'; -import { ConfigReference } from './ConfigReference'; -import { AssignRights } from './AssignRights'; -import { PluginPermissions } from './PluginPermissions'; -import { hasPermission, isPluginPermissions } from './pluginUtils'; - -const UpdatePluginButton = lazy( - () => import('@chunks/Plugins/UpdatePluginButton'), -); - -export const PluginPage: React.FC> = ({ - resource, -}) => { - const configLabelId = useId(); - const canWrite = useCanWrite(resource); - const navigate = useNavigateWithTransition(); - const [showUninstallDialog, setShowUninstallDialog] = useState(false); - const [name] = useString(resource, core.properties.name); - const [namespace] = useString(resource, server.properties.namespace); - const [config, setConfig] = useValue(resource, server.properties.config); - const [permissions] = useValue(resource, server.properties.pluginPermissions); - const [configValid, setConfigValid] = useState(true); - const [configSyntaxValid, setConfigSyntaxValid] = useState(true); - const [configEdited, setConfigEdited] = useState(false); - const title = `${namespace ? `${namespace}/` : ''}${name}`; - const [version] = useString(resource, server.properties.version); - const [author] = useString(resource, server.properties.pluginAuthor); - const [description] = useString(resource, core.properties.description); - const [schema] = useValue(resource, server.properties.jsonSchema); - const saveState = useSaveState(resource); - - const { uninstallPlugin } = useCreatePlugin(); - - const hasFullDriveAccess = hasPermission(permissions, 'full-drive-access'); - - return ( - - -
    - - {title} - v{version} - - by {author} -
    - - {canWrite && ( - - - - - )} - {description && ( - - - - )} - - {canWrite && ( - - )} - {/* Secrets are declared by name in a plugin's source; a WASM plugin's - manifest carries origins but no names, so it has nothing to render - slots from yet. The endpoint still serves it. */} - - -

    - - - Config - -

    - -
    - { - try { - setConfig(JSON.parse(v)); - setConfigEdited(true); - setConfigSyntaxValid(true); - } catch (e) { - setConfigSyntaxValid(false); - } - }} - schema={schema as JSONSchema7} - showErrorStyling={!configValid} - onValidationChange={setConfigValid} - /> -
    - {schema && } - {isPluginPermissions(permissions) && ( - - )} -
    - { - const parent = resource.props.parent; - await uninstallPlugin(resource); - navigate(constructOpenURL(parent)); - toast.success('Plugin uninstalled'); - }} - onCancel={() => setShowUninstallDialog(false)} - > - Are you sure you want to uninstall this plugin? - -
    - ); -}; - -const PluginName = styled.span` - font-weight: bold; - font-size: 1.2rem; -`; - -const DescriptionWrapper = styled.section` - background-color: ${p => p.theme.colors.bg1}; - padding: ${p => p.theme.size()}; - border-radius: ${p => p.theme.radius}; - max-height: 33rem; - overflow-y: auto; -`; - -const PluginAuthor = styled.span` - color: ${p => p.theme.colors.textLight}; -`; diff --git a/browser/data-browser/src/views/Plugin/PluginPermissions.tsx b/browser/data-browser/src/views/Plugin/PluginPermissions.tsx deleted file mode 100644 index da39bd28b..000000000 --- a/browser/data-browser/src/views/Plugin/PluginPermissions.tsx +++ /dev/null @@ -1,89 +0,0 @@ -import type { - PluginPermission, - PluginPermissionType, -} from '@chunks/Plugins/plugins'; -import { Column, Row } from '@components/Row'; -import { - FaDesktop, - FaFire, - FaGlobe, - FaHardDrive, - FaMemory, - FaShield, -} from 'react-icons/fa6'; -import styled from 'styled-components'; - -interface PluginPermissionsProps { - permissions?: PluginPermission[]; - title?: string; -} - -const TITLES = { - network: 'Network', - storage: 'Storage', - 'full-drive-access': 'Full Drive Access', - 'extended-fuel': 'Extended Fuel', - 'extended-memory': 'Extended Memory', - 'custom-view': 'Custom View', -} satisfies Record; - -const ICONS = { - network: , - storage: , - 'full-drive-access': , - 'extended-fuel': , - 'extended-memory': , - 'custom-view': , -} satisfies Record; - -export const PluginPermissions: React.FC = ({ - permissions = [], - title = 'Permissions', -}) => { - return ( - -

    {title}

    - - {permissions.length === 0 && ( -
  • -

    No permissions required

    -
  • - )} - {permissions.map(permission => ( -
  • - - {ICONS[permission.permission]} {TITLES[permission.permission]} - -

    {permission.reason || 'No reason provided'}

    -
  • - ))} -
    -
    - ); -}; - -const PermissionList = styled.ul` - display: flex; - flex-direction: column; - gap: ${p => p.theme.size()}; - padding: 0; - margin: 0; - - li { - background-color: ${p => p.theme.colors.bg1}; - border-radius: ${p => p.theme.radius}; - list-style: none; - padding: ${p => p.theme.size()}; - margin: 0; - - p { - margin: 0; - } - } -`; - -const PermissionTitle = styled(Row)` - font-weight: bold; - font-size: 0.9rem; - color: ${p => p.theme.colors.textLight}; -`; diff --git a/browser/data-browser/src/views/Plugin/createPlugin.ts b/browser/data-browser/src/views/Plugin/createPlugin.ts deleted file mode 100644 index 031dcc01f..000000000 --- a/browser/data-browser/src/views/Plugin/createPlugin.ts +++ /dev/null @@ -1,118 +0,0 @@ -import type { PluginMetadata } from '@chunks/Plugins/plugins'; -import { - core, - server, - useStore, - type JSONValue, - type Resource, - type Server, -} from '@tomic/react'; -import toast from 'react-hot-toast'; -import { useCustomViews } from '@components/CustomViewProvider'; - -interface CreatePluginProps { - metadata: PluginMetadata; - config: JSONValue; - file: File; - drive: Resource; -} - -export function useCreatePlugin() { - const store = useStore(); - const { refresh: refreshCustomViews } = useCustomViews(); - - const createPluginResource = async ({ - metadata, - file, - drive, - config, - }: CreatePluginProps): Promise> => { - const plugin = await store.newResource({ - isA: server.classes.plugin, - parent: drive.subject, - propVals: { - [core.properties.name]: metadata.name, - [server.properties.namespace]: metadata.namespace, - [server.properties.version]: metadata.version, - [server.properties.config]: config, - }, - }); - - await plugin.save(); - - const [fileSubject] = await store.uploadFiles([file], plugin.subject); - - // Setting the file triggers the installation on the server. - await plugin.set(server.properties.pluginFile, fileSubject); - - await plugin.save(); - await plugin.refresh(); - - // We refresh the resource so we can see the dynamic plugin-agent property that was added by the server. - - return plugin; - }; - - const addPluginToDrive = async ( - plugin: Resource, - drive: Resource, - ): Promise => { - drive.push(server.properties.plugins, [plugin.subject], true); - await drive.save(); - }; - - const uninstallPlugin = async ( - plugin: Resource, - ): Promise => { - const driveSubject = plugin.props.parent; - await plugin.destroy(); - - const drive = await store.getResource(driveSubject); - await drive.set( - server.properties.plugins, - drive.props.plugins?.filter(p => p !== plugin.subject), - ); - await drive.save(); - await refreshCustomViews(); - }; - - const updatePlugin = async ( - plugin: Resource, - metadata: PluginMetadata, - file: File, - updatedConfig?: JSONValue, - ): Promise => { - if ( - metadata.name !== plugin.props.name || - metadata.namespace !== plugin.props.namespace - ) { - throw new Error( - "The update's identifier does not match the existing plugin.", - ); - } - - const [fileSubject] = await store.uploadFiles([file], plugin.subject); - - await plugin.set(server.properties.pluginFile, fileSubject); - - if (updatedConfig) { - await plugin.set(server.properties.config, updatedConfig); - } - - try { - await plugin.save(); - } catch (err) { - toast.error(err.message); - } - - // Refresh so we see any new dynamic properties if those were added. - await plugin.refresh(); - }; - - return { - createPluginResource, - addPluginToDrive, - uninstallPlugin, - updatePlugin, - }; -} diff --git a/browser/data-browser/src/views/Plugin/pluginUtils.ts b/browser/data-browser/src/views/Plugin/pluginUtils.ts deleted file mode 100644 index 69ef0840f..000000000 --- a/browser/data-browser/src/views/Plugin/pluginUtils.ts +++ /dev/null @@ -1,28 +0,0 @@ -import type { - PluginPermission, - PluginPermissionType, -} from '@chunks/Plugins/plugins'; - -export function isPluginPermissions( - permissions: unknown, -): permissions is PluginPermission[] { - if (!Array.isArray(permissions)) return false; - - return permissions.every( - permission => - typeof permission === 'object' && - 'permission' in permission && - typeof permission.permission === 'string' && - 'reason' in permission && - typeof permission.reason === 'string', - ); -} - -export function hasPermission( - permissions: unknown, - permission: PluginPermissionType, -): boolean { - if (!isPluginPermissions(permissions)) return false; - - return permissions.some(p => p.permission === permission); -} diff --git a/browser/data-browser/src/views/ResourcePage.tsx b/browser/data-browser/src/views/ResourcePage.tsx index 2348a9948..10bfb4df4 100644 --- a/browser/data-browser/src/views/ResourcePage.tsx +++ b/browser/data-browser/src/views/ResourcePage.tsx @@ -41,7 +41,7 @@ import { AIChatPage } from '@views/AIChat/AIChatPage'; import { DocumentV2FullPage } from './Document/DocumentV2FullPage'; import { CanvasPage } from './Canvas/CanvasPage'; import { canvas } from '@tomic/lib'; -import { PluginPage } from '@views/Plugin/PluginPage'; +import { InstallationPage } from '@views/Installation/InstallationPage'; import { useCustomViews } from '@components/CustomViewProvider'; import { PluginView } from './PluginView/PluginView'; import { MeetingPage } from './Meeting/MeetingPage'; @@ -318,8 +318,8 @@ function selectComponent(klass: string | undefined) { return MeetingPage; case canvas.classes.canvas: return CanvasPage; - case server.classes.plugin: - return PluginPage; + case server.classes.installation: + return InstallationPage; default: return ResourcePageDefault; } diff --git a/browser/e2e/tests/integration-store-install.spec.ts b/browser/e2e/tests/integration-store-install.spec.ts new file mode 100644 index 000000000..bca830123 --- /dev/null +++ b/browser/e2e/tests/integration-store-install.spec.ts @@ -0,0 +1,82 @@ +import { test, expect } from '@playwright/test'; +import { before } from './test-utils'; +import { enableIntegrationDiscovery } from './integration-settings-utils'; + +test.beforeEach(before); + +/** + * Opening a marketplace Listing shows the Installation review: what the + * release asks for and why, its pinned id, and the draft alternative. The + * catalog is mocked; installing a fixture release would fail on the server, + * so the actual install commit is covered by `plugin.spec.ts` (zip upload) + * and the unit tests around `installRelease`. + */ +test('a Listing opens the Installation review before anything is installed', async ({ + page, +}) => { + await page.route('**/plugin-catalog', route => + route.fulfill({ + json: [ + { + subject: 'https://example.com/listings/fixture', + name: 'Calendar sync', + emoji: 'πŸ“…', + description: 'Keeps a calendar in sync', + publisher: 'https://example.com/agents/test', + domains: ['calendar'], + standards: [], + release: 'https://example.com/releases/fixture', + releaseId: 'blake3:fixture', + runtime: 'atomic-js/1', + world: 'extension', + }, + ], + }), + ); + await page.route('**/plugin-package/*', route => + route.fulfill({ + json: { + runtime: 'atomic-js/1', + source: 'export function run() { return {}; }', + schemas: {}, + manifest: { + schemaVersion: 2, + capabilities: [{ name: 'storage', reason: 'Remembers the cursor' }], + secrets: [ + { + name: 'google', + origin: 'https://www.googleapis.com', + description: 'Calendar API key', + }, + ], + }, + }, + }), + ); + await enableIntegrationDiscovery(page); + await page.goto(new URL('/app/integrations', page.url()).href); + + const card = page.locator('[data-release="blake3:fixture"]'); + await expect( + card.getByRole('heading', { name: 'Calendar sync' }), + ).toBeVisible(); + await card.getByRole('button', { name: 'Open' }).click(); + + const dialog = page.locator('dialog[open]'); + await expect( + dialog.getByRole('heading', { name: 'Install plugin' }), + ).toBeVisible(); + await expect(dialog.getByText('Storage', { exact: true })).toBeVisible(); + await expect(dialog.getByText('Remembers the cursor')).toBeVisible(); + await expect(dialog.getByText('Secret "google"')).toBeVisible(); + await dialog.getByText('Release', { exact: true }).click(); + await expect(dialog.getByText('blake3:fixture')).toBeVisible(); + await expect( + dialog.getByRole('button', { name: 'Create draft' }), + ).toBeVisible(); + await expect(dialog.getByRole('button', { name: 'Install' })).toBeEnabled(); + + await dialog.getByRole('button', { name: 'Cancel' }).click(); + await expect(dialog).toBeHidden(); + await expect(page.locator('[data-installation]')).toHaveCount(0); +}); diff --git a/browser/e2e/tests/plugin.spec.ts b/browser/e2e/tests/plugin.spec.ts index e2fc94349..793cc6963 100644 --- a/browser/e2e/tests/plugin.spec.ts +++ b/browser/e2e/tests/plugin.spec.ts @@ -49,9 +49,11 @@ test.describe('Plugins', () => { await fileChooser.setFiles(testFilePath('test-plugin.zip')); + // The upload publishes a private Release, then the same Installation + // review screen the Store uses opens for it. await inDialog(page, async (dialog, closeWith) => { await expect( - dialog.getByRole('heading', { name: 'Add Plugin' }), + dialog.getByRole('heading', { name: 'Install plugin' }), ).toBeVisible(); await expect( dialog.getByText('ontola/test-plugin', { exact: true }), @@ -75,9 +77,12 @@ test.describe('Plugins', () => { await closeWith('Install'); }); + // Installing navigates to the new Installation, which the server has + // activated by the time the commit round-trips. await expect( - page.getByRole('link', { name: 'ontola/test-plugin' }), + page.getByText('ontola/test-plugin', { exact: true }), ).toBeVisible(); + await expect(page.getByLabel('Status: active')).toBeVisible(); // Now create the folder. The plugin's `after_commit` fires on the // folder's first commit and emits a follow-up commit setting the @@ -202,7 +207,9 @@ test.describe('Plugins', () => { await page.getByRole('main').getByText('Plugins', { exact: true }).click(); await page.getByRole('link', { name: 'ontola/test-plugin' }).click(); - // Keep the mounted resource identity while changing metadata and saving config. + // Keep the mounted resource identity while changing metadata and saving + // config. The server's Installation extender fills version, author and + // description from the manifest, so these read as plain properties. const saveConfig = page.getByRole('button', { name: 'Save', exact: true }); await expect(saveConfig).toBeDisabled(); await page @@ -293,7 +300,7 @@ test.describe('Plugins', () => { await inDialog(page, async (dialog, closeWith) => { await expect( - dialog.getByRole('heading', { name: 'Uninstall Plugin' }), + dialog.getByRole('heading', { name: 'Uninstall plugin' }), ).toBeVisible(); await closeWith('Uninstall'); }); diff --git a/browser/lib/src/app-package.test.ts b/browser/lib/src/app-package.test.ts index 7fda3d233..db8d7c70d 100644 --- a/browser/lib/src/app-package.test.ts +++ b/browser/lib/src/app-package.test.ts @@ -150,7 +150,8 @@ describe('portable app package resources', () => { { runtime: 'other/1' }, { source: '' }, { token: 'secret' }, - { manifest: { schemaVersion: 2 } }, + { manifest: { schemaVersion: 3 } }, + { manifest: { schemaVersion: 2, capabilities: ['teleport'] } }, { schemas: { item: 'local:item' } }, ]) { const pkg = JSON.parse(json); diff --git a/browser/lib/src/index.ts b/browser/lib/src/index.ts index cb4c00f02..01adee1e7 100644 --- a/browser/lib/src/index.ts +++ b/browser/lib/src/index.ts @@ -194,11 +194,19 @@ export { export { parseManifest, validateManifest, + type CapabilityName, + type DeclaredAction, + type DeclaredCapability, + type DeclaredEntrypoints, + type DeclaredNetwork, type DeclaredOperation, secretsMentionedIn, originsMentionedIn, type DeclaredSecret, + type ManifestRuntime, + type ManifestWorld, type PluginManifest, + type PluginManifestV2, } from './plugin-manifest.js'; export { describePlugin } from './plugin-runner.js'; @@ -239,6 +247,23 @@ export { pluginSyncSchedule, type PluginSyncSchedule, } from './plugin-connection.js'; +export { + installRelease, + updateInstallationRelease, + publishZipRelease, + readInstallationReview, + grantsFor, + installationIdentifier, + DEFAULT_INSTALLATION_NAMESPACE, + RUNTIME_JS, + type InstallationStatus, + type InstallationReview, + type InstallReleaseOptions, + type UpdateReleaseOptions, + type PublishedRelease, + type ReleaseReference, + type ReviewCapability, +} from './plugin-install.js'; export * from './integration-actions.js'; export { taskSchema } from './task-schema.js'; diff --git a/browser/lib/src/ontologies/server.ts b/browser/lib/src/ontologies/server.ts index c2f2bf88f..192868c24 100644 --- a/browser/lib/src/ontologies/server.ts +++ b/browser/lib/src/ontologies/server.ts @@ -16,6 +16,9 @@ export const server = { invite: 'https://atomicdata.dev/classes/Invite', redirect: 'https://atomicdata.dev/classes/Redirect', plugin: 'https://atomicdata.dev/classes/Plugin', + release: 'https://atomicdata.dev/classes/Release', + installation: 'https://atomicdata.dev/classes/Installation', + listing: 'https://atomicdata.dev/classes/Listing', }, properties: { agent: 'https://atomicdata.dev/properties/invite/agent', @@ -59,6 +62,22 @@ export const server = { pluginPermissions: 'https://atomicdata.dev/properties/pluginPermissions', searchChunks: 'https://atomicdata.dev/properties/search/chunks', llmTxt: 'https://atomicdata.dev/properties/llm-txt', + runtime: 'https://atomicdata.dev/properties/runtime', + world: 'https://atomicdata.dev/properties/world', + manifest: 'https://atomicdata.dev/properties/manifest', + source: 'https://atomicdata.dev/properties/source', + package: 'https://atomicdata.dev/properties/package', + schemas: 'https://atomicdata.dev/properties/schemas', + releaseId: 'https://atomicdata.dev/properties/releaseId', + previousRelease: 'https://atomicdata.dev/properties/previousRelease', + publisher: 'https://atomicdata.dev/properties/publisher', + release: 'https://atomicdata.dev/properties/release', + grants: 'https://atomicdata.dev/properties/grants', + installationStatus: 'https://atomicdata.dev/properties/installationStatus', + domains: 'https://atomicdata.dev/properties/domains', + standards: 'https://atomicdata.dev/properties/standards', + evidence: 'https://atomicdata.dev/properties/evidence', + supportTier: 'https://atomicdata.dev/properties/supportTier', }, __classDefs: { ['https://atomicdata.dev/classes/Drive']: [ @@ -117,6 +136,45 @@ export const server = { 'https://atomicdata.dev/properties/pluginAgent', 'https://atomicdata.dev/properties/pluginPermissions', ], + ['https://atomicdata.dev/classes/Release']: [ + 'https://atomicdata.dev/properties/runtime', + 'https://atomicdata.dev/properties/manifest', + 'https://atomicdata.dev/properties/releaseId', + 'https://atomicdata.dev/properties/world', + 'https://atomicdata.dev/properties/source', + 'https://atomicdata.dev/properties/package', + 'https://atomicdata.dev/properties/schemas', + 'https://atomicdata.dev/properties/version', + 'https://atomicdata.dev/properties/previousRelease', + 'https://atomicdata.dev/properties/publisher', + 'https://atomicdata.dev/properties/name', + 'https://atomicdata.dev/properties/description', + ], + ['https://atomicdata.dev/classes/Installation']: [ + 'https://atomicdata.dev/properties/release', + 'https://atomicdata.dev/properties/releaseId', + 'https://atomicdata.dev/properties/installationStatus', + 'https://atomicdata.dev/properties/config', + 'https://atomicdata.dev/properties/grants', + 'https://atomicdata.dev/properties/name', + 'https://atomicdata.dev/properties/namespace', + 'https://atomicdata.dev/properties/description', + 'https://atomicdata.dev/properties/pluginAgent', + 'https://atomicdata.dev/properties/pluginPermissions', + 'https://atomicdata.dev/properties/jsonSchema', + 'https://atomicdata.dev/properties/version', + ], + ['https://atomicdata.dev/classes/Listing']: [ + 'https://atomicdata.dev/properties/name', + 'https://atomicdata.dev/properties/release', + 'https://atomicdata.dev/properties/emoji', + 'https://atomicdata.dev/properties/description', + 'https://atomicdata.dev/properties/publisher', + 'https://atomicdata.dev/properties/domains', + 'https://atomicdata.dev/properties/standards', + 'https://atomicdata.dev/properties/evidence', + 'https://atomicdata.dev/properties/supportTier', + ], }, } as const satisfies OntologyBaseObject; @@ -130,6 +188,9 @@ export namespace Server { export type Invite = typeof server.classes.invite; export type Redirect = typeof server.classes.redirect; export type Plugin = typeof server.classes.plugin; + export type Release = typeof server.classes.release; + export type Installation = typeof server.classes.installation; + export type Listing = typeof server.classes.listing; } declare module '../index.js' { @@ -204,6 +265,54 @@ declare module '../index.js' { | typeof server.properties.pluginAgent | typeof server.properties.pluginPermissions; }; + [server.classes.release]: { + requires: + | BaseProps + | typeof server.properties.runtime + | typeof server.properties.manifest + | typeof server.properties.releaseId; + recommends: + | typeof server.properties.world + | typeof server.properties.source + | typeof server.properties.package + | typeof server.properties.schemas + | typeof server.properties.version + | typeof server.properties.previousRelease + | typeof server.properties.publisher + | 'https://atomicdata.dev/properties/name' + | 'https://atomicdata.dev/properties/description'; + }; + [server.classes.installation]: { + requires: + | BaseProps + | typeof server.properties.release + | typeof server.properties.releaseId + | typeof server.properties.installationStatus; + recommends: + | typeof server.properties.config + | typeof server.properties.grants + | 'https://atomicdata.dev/properties/name' + | typeof server.properties.namespace + | 'https://atomicdata.dev/properties/description' + | typeof server.properties.pluginAgent + | typeof server.properties.pluginPermissions + | typeof server.properties.jsonSchema + | typeof server.properties.version; + }; + [server.classes.listing]: { + requires: + | BaseProps + | 'https://atomicdata.dev/properties/name' + | typeof server.properties.release; + recommends: + | 'https://atomicdata.dev/properties/emoji' + | 'https://atomicdata.dev/properties/description' + | typeof server.properties.publisher + | typeof server.properties.domains + | typeof server.properties.standards + | typeof server.properties.evidence + | typeof server.properties.supportTier; + }; } interface PropTypeMapping { @@ -245,6 +354,22 @@ declare module '../index.js' { [server.properties.pluginPermissions]: JSONValue; [server.properties.searchChunks]: JSONValue; [server.properties.llmTxt]: string; + [server.properties.runtime]: string; + [server.properties.world]: string; + [server.properties.manifest]: JSONValue; + [server.properties.source]: string; + [server.properties.package]: string; + [server.properties.schemas]: JSONValue; + [server.properties.releaseId]: string; + [server.properties.previousRelease]: string; + [server.properties.publisher]: string; + [server.properties.release]: string; + [server.properties.grants]: JSONValue; + [server.properties.installationStatus]: string; + [server.properties.domains]: JSONValue; + [server.properties.standards]: string[]; + [server.properties.evidence]: JSONValue; + [server.properties.supportTier]: string; } interface PropSubjectToNameMapping { @@ -286,5 +411,21 @@ declare module '../index.js' { [server.properties.pluginPermissions]: 'pluginPermissions'; [server.properties.searchChunks]: 'searchChunks'; [server.properties.llmTxt]: 'llmTxt'; + [server.properties.runtime]: 'runtime'; + [server.properties.world]: 'world'; + [server.properties.manifest]: 'manifest'; + [server.properties.source]: 'source'; + [server.properties.package]: 'package'; + [server.properties.schemas]: 'schemas'; + [server.properties.releaseId]: 'releaseId'; + [server.properties.previousRelease]: 'previousRelease'; + [server.properties.publisher]: 'publisher'; + [server.properties.release]: 'release'; + [server.properties.grants]: 'grants'; + [server.properties.installationStatus]: 'installationStatus'; + [server.properties.domains]: 'domains'; + [server.properties.standards]: 'standards'; + [server.properties.evidence]: 'evidence'; + [server.properties.supportTier]: 'supportTier'; } } diff --git a/browser/lib/src/plugin-install.test.ts b/browser/lib/src/plugin-install.test.ts new file mode 100644 index 000000000..4ffe126c5 --- /dev/null +++ b/browser/lib/src/plugin-install.test.ts @@ -0,0 +1,357 @@ +import { describe, expect, it, vi } from 'vitest'; +import { core } from './ontologies/core.js'; +import { server } from './ontologies/server.js'; +import { + grantsFor, + installRelease, + installationIdentifier, + publishZipRelease, + readInstallationReview, + updateInstallationRelease, +} from './plugin-install.js'; +import { testStore } from './test-store.js'; + +describe('readInstallationReview', () => { + it('reads a plugin.json style manifest with permission reasons and config', () => { + const review = readInstallationReview({ + id: 'blake3:abc', + runtime: 'wasip2/1', + version: '1.0.0', + manifest: { + name: 'test-plugin', + namespace: 'ontola', + author: 'Ontola', + description: 'Renames folders', + permissions: [ + { permission: 'storage', reason: 'Keeps a counter' }, + 'network', + { reason: 'dropped: no permission name' }, + ], + defaultConfig: { folderPrefix: 'My' }, + configSchema: { + type: 'object', + properties: { folderPrefix: { type: 'string' } }, + }, + }, + }); + + expect(review).toMatchObject({ + name: 'test-plugin', + namespace: 'ontola', + author: 'Ontola', + description: 'Renames folders', + version: '1.0.0', + runtime: 'wasip2/1', + world: 'extension', + releaseId: 'blake3:abc', + defaultConfig: { folderPrefix: 'My' }, + }); + expect(review.configSchema).toMatchObject({ type: 'object' }); + expect(review.capabilities).toEqual([ + { + kind: 'permission', + title: 'storage', + reason: 'Keeps a counter', + grant: 'storage', + }, + { + kind: 'permission', + title: 'network', + reason: undefined, + grant: 'network', + }, + ]); + expect(grantsFor(review)).toEqual(['storage', 'network']); + }); + + it('reads a versioned JS manifest: secrets and operations are shown but not granted', () => { + const review = readInstallationReview({ + runtime: 'atomic-js/1', + manifest: { + schemaVersion: 1, + version: '2.1.0', + secrets: [ + { + name: 'google', + origin: 'https://www.googleapis.com', + description: 'Calendar API key', + }, + { origin: 'https://dropped.example' }, + ], + operations: [ + { + id: 'list-events', + method: 'GET', + url: 'https://www.googleapis.com/calendar/v3/events', + effect: 'read', + }, + ], + }, + }); + + expect(review.runtime).toBe('atomic-js/1'); + expect(review.version).toBe('2.1.0'); + expect(review.name).toBeUndefined(); + expect(review.capabilities).toEqual([ + { + kind: 'secret', + title: 'Secret "google"', + reason: 'Calendar API key. Sent only to https://www.googleapis.com', + }, + { + kind: 'operation', + title: 'GET https://www.googleapis.com/calendar/v3/events', + reason: 'list-events: reads external data', + }, + ]); + expect(grantsFor(review)).toEqual([]); + }); + + it('shows and grants v2 capabilities in either spelling', () => { + const review = readInstallationReview({ + runtime: 'atomic-js/1', + world: 'server-extension', + manifest: { + schemaVersion: 2, + capabilities: [ + 'custom-view', + { name: 'storage', reason: 'Caches pages' }, + ], + network: { + origins: ['https://api.example'], + reason: 'Fetches feeds', + }, + }, + }); + + expect(review.world).toBe('server-extension'); + expect(review.capabilities).toEqual([ + { + kind: 'capability', + title: 'custom-view', + reason: undefined, + grant: 'custom-view', + }, + { + kind: 'capability', + title: 'storage', + reason: 'Caches pages', + grant: 'storage', + }, + { + kind: 'network', + title: 'Network access to https://api.example', + reason: 'Fetches feeds', + }, + ]); + expect(grantsFor(review)).toEqual(['custom-view', 'storage']); + }); + + it('tolerates a manifest that is not an object', () => { + const review = readInstallationReview({ + runtime: 'atomic-js/1', + manifest: null, + }); + expect(review.capabilities).toEqual([]); + expect(review.name).toBeUndefined(); + }); +}); + +describe('installationIdentifier', () => { + it('produces identifiers the server accepts', () => { + expect(installationIdentifier('Community fixture')).toBe( + 'Community-fixture', + ); + expect(installationIdentifier(' --ÜnΓ―code plugin! ')).toBe( + 'n-code-plugin', + ); + expect(installationIdentifier('')).toBe('plugin'); + expect(installationIdentifier('x'.repeat(200))).toHaveLength(128); + }); +}); + +describe('installRelease', () => { + it('commits an active Installation under the drive with the pinned release', async () => { + const { store, posted } = await testStore(); + const drive = await store.newResource({ + isA: server.classes.drive, + noParent: true, + propVals: { [core.properties.name]: 'Team' }, + }); + await drive.save(); + + const subject = await installRelease(store, { + drive: drive.subject, + release: { url: 'blake3:abc', id: 'blake3:abc' }, + name: 'test-plugin', + namespace: 'ontola', + description: 'Renames folders', + version: '1.0.0', + config: { folderPrefix: 'My' }, + grants: ['storage'], + }); + + const installation = store.getResourceLoading(subject); + expect(installation.hasClasses(server.classes.installation)).toBe(true); + expect(installation.get(core.properties.parent)).toBe(drive.subject); + expect(installation.get(core.properties.name)).toBe('test-plugin'); + expect(installation.get(server.properties.namespace)).toBe('ontola'); + expect(installation.get(core.properties.description)).toBe( + 'Renames folders', + ); + expect(installation.get(server.properties.version)).toBe('1.0.0'); + expect(installation.get(server.properties.release)).toBe('blake3:abc'); + expect(installation.get(server.properties.releaseId)).toBe('blake3:abc'); + expect(installation.get(server.properties.installationStatus)).toBe( + 'active', + ); + expect(installation.get(server.properties.grants)).toEqual(['storage']); + // The test store skips the datatype fetch, so an object value is kept + // serialized; against a server the JSON datatype keeps it an object. + const config = installation.get(server.properties.config); + expect(typeof config === 'string' ? JSON.parse(config) : config).toEqual({ + folderPrefix: 'My', + }); + expect(posted.map(c => c.subject)).toContain(subject); + }); + + it('leaves optional fields off and honours a draft status', async () => { + const { store } = await testStore(); + const subject = await installRelease(store, { + drive: 'https://example.com/drive', + release: { + url: 'https://marketplace.example/release/1', + id: 'blake3:def', + }, + name: 'importer', + grants: [], + status: 'draft', + }); + const installation = store.getResourceLoading(subject); + expect(installation.get(server.properties.installationStatus)).toBe( + 'draft', + ); + expect(installation.get(server.properties.release)).toBe( + 'https://marketplace.example/release/1', + ); + expect(installation.get(server.properties.namespace)).toBeUndefined(); + expect(installation.get(server.properties.config)).toBeUndefined(); + expect(installation.get(server.properties.grants)).toEqual([]); + }); +}); + +describe('updateInstallationRelease', () => { + it('repoints the same Installation and keeps its identity and name', async () => { + const { store, posted } = await testStore(); + const drive = await store.newResource({ + isA: server.classes.drive, + noParent: true, + propVals: { [core.properties.name]: 'Team' }, + }); + await drive.save(); + + const subject = await installRelease(store, { + drive: drive.subject, + release: { url: 'blake3:one', id: 'blake3:one' }, + name: 'test-plugin', + namespace: 'ontola', + version: '1.0.0', + config: { folderPrefix: 'My' }, + grants: ['storage'], + }); + + const beforeUpdate = posted.length; + await updateInstallationRelease(store, subject, { + release: { url: 'blake3:two', id: 'blake3:two' }, + grants: ['storage', 'custom-view'], + version: '1.1.0', + }); + + const installation = store.getResourceLoading(subject); + expect(installation.get(server.properties.release)).toBe('blake3:two'); + expect(installation.get(server.properties.releaseId)).toBe('blake3:two'); + expect(installation.get(server.properties.grants)).toEqual([ + 'storage', + 'custom-view', + ]); + expect(installation.get(server.properties.version)).toBe('1.1.0'); + // An update is not a new install: same resource, same identifiers, and the + // config it was running is untouched when the caller passes none. + expect(installation.get(core.properties.name)).toBe('test-plugin'); + expect(installation.get(server.properties.namespace)).toBe('ontola'); + expect(installation.get(server.properties.installationStatus)).toBe( + 'active', + ); + const config = installation.get(server.properties.config); + expect(typeof config === 'string' ? JSON.parse(config) : config).toEqual({ + folderPrefix: 'My', + }); + // One commit, not one per property: the server checks the grants against + // the new release's manifest, so a release that arrived on its own could be + // refused for capabilities the next commit was about to approve. + expect(posted.length - beforeUpdate).toBe(1); + }); +}); + +describe('publishZipRelease', () => { + it('posts the signed zip bytes to /plugin-release-package for the drive', async () => { + const { store } = await testStore(); + const bytes = new Uint8Array([0x50, 0x4b, 0x03, 0x04]); + const transport = vi.fn( + async () => + new Response( + JSON.stringify({ + id: 'blake3:zip', + release: { + runtime: 'wasip2/1', + package: 'ab'.repeat(32), + manifest: { name: 'test-plugin', namespace: 'ontola' }, + }, + }), + { status: 200 }, + ), + ); + + const result = await publishZipRelease( + store, + 'did:ad:drive', + new Blob([bytes]), + {}, + transport as unknown as typeof fetch, + ); + + expect(result.id).toBe('blake3:zip'); + expect(result.release.manifest).toEqual({ + name: 'test-plugin', + namespace: 'ontola', + }); + const [url, init] = transport.mock.calls[0] as unknown as [ + string, + RequestInit, + ]; + const parsed = new URL(url); + expect(parsed.pathname).toBe('/plugin-release-package'); + expect(parsed.searchParams.get('drive')).toBe('did:ad:drive'); + expect(init.method).toBe('POST'); + const headers = init.headers as Record; + expect(headers['Content-Type']).toBe('application/zip'); + expect(headers['x-atomic-signature']).toBeTruthy(); + expect(new Uint8Array(init.body as ArrayBuffer)).toEqual(bytes); + }); + + it('surfaces the server error body', async () => { + const { store } = await testStore(); + const transport = vi.fn( + async () => new Response('Body is not a zip archive', { status: 400 }), + ); + await expect( + publishZipRelease( + store, + 'did:ad:drive', + new Blob([]), + {}, + transport as unknown as typeof fetch, + ), + ).rejects.toThrow('Body is not a zip archive'); + }); +}); diff --git a/browser/lib/src/plugin-install.ts b/browser/lib/src/plugin-install.ts new file mode 100644 index 000000000..dfc8290fd --- /dev/null +++ b/browser/lib/src/plugin-install.ts @@ -0,0 +1,355 @@ +/** + * One install path for both plugin runtimes. + * + * A Release is an immutable, content-addressed package (JS source or a wasip2 + * zip). Installing it into a drive means committing an `Installation` + * resource with `installationStatus: active`; the server resolves the pinned + * release, verifies its id, checks the grants and materializes by runtime. + * Pausing, revoking or destroying that resource undoes it. See + * `planning/plugin-runtime-convergence.md`, "One install path and a + * marketplace". + */ +import { signRequest } from './authentication.js'; +import { core } from './ontologies/core.js'; +import { server, type Server } from './ontologies/server.js'; +import type { Store } from './store.js'; +import type { JSONValue } from './value.js'; + +export const RUNTIME_JS = 'atomic-js/1'; +const WORLD_EXTENSION = 'extension'; + +export type InstallationStatus = 'draft' | 'active' | 'paused' | 'revoked'; + +/** The release record the server publishes and serves (`lib/src/db/plugin_release.rs`). */ +export interface PublishedRelease { + runtime: string; + /** Defaults to `extension` when absent. */ + world?: string; + manifest: JSONValue; + source?: string; + /** Blake3 hex of the zip bytes for `wasip2/1`. */ + package?: string; + schemas?: Record; + version?: string; + previousRelease?: string; +} + +/** + * Where an Installation finds its Release and what it pins. + * + * `url` is what the server resolves: a `Release` resource URL, or a bare + * release id (`blake3:…`) for releases that only exist in this server's + * release cache (the catalog and zip-publish paths today). `id` is always the + * content hash the installer reviewed; the server refuses to install when + * `url` resolves to anything else. + */ +export interface ReleaseReference { + url: string; + id: string; +} + +type CapabilityKind = + | 'permission' + | 'secret' + | 'operation' + | 'capability' + | 'network'; + +/** One line of the install review: what the plugin asks for and why. */ +export interface ReviewCapability { + kind: CapabilityKind; + title: string; + reason?: string; + /** + * The grant name written to `Installation.grants` when the installer + * approves; `check_grants` on the server requires exactly the declared + * set. Absent for secrets, operations and network, which are declared, + * not granted. + */ + grant?: string; +} + +export interface InstallationReview { + name?: string; + namespace?: string; + description?: string; + author?: string; + version?: string; + runtime: string; + world: string; + releaseId?: string; + capabilities: ReviewCapability[]; + configSchema?: JSONValue; + defaultConfig?: JSONValue; +} + +function asString(value: unknown): string | undefined { + return typeof value === 'string' && value.length > 0 ? value : undefined; +} + +function asObject(value: unknown): Record | undefined { + return value && typeof value === 'object' && !Array.isArray(value) + ? (value as Record) + : undefined; +} + +function asArray(value: unknown): unknown[] { + return Array.isArray(value) ? value : []; +} + +/** + * Turns a release's manifest into what the review screen shows before the + * installer approves. Drops anything malformed instead of trusting it. + */ +export function readInstallationReview( + release: Pick & + Partial & { id?: string }, +): InstallationReview { + const manifest = asObject(release.manifest) ?? {}; + const capabilities: ReviewCapability[] = []; + + // A release published before plugin.json was translated at the boundary + // still stores `permissions: [{ permission, reason }]`; the server's + // `declared_capabilities` reads those the same way. + for (const entry of asArray(manifest.permissions)) { + const name = asString(entry) ?? asString(asObject(entry)?.permission); + if (!name) continue; + capabilities.push({ + kind: 'permission', + title: name, + reason: asString(asObject(entry)?.reason), + grant: name, + }); + } + + // Secrets and operations (`plugin-manifest.ts`). + for (const entry of asArray(manifest.secrets)) { + const secret = asObject(entry); + const name = asString(secret?.name); + if (!name) continue; + const origin = asString(secret?.origin); + const description = asString(secret?.description); + capabilities.push({ + kind: 'secret', + title: `Secret "${name}"`, + reason: [description, origin && `Sent only to ${origin}`] + .filter(Boolean) + .join('. '), + }); + } + + for (const entry of asArray(manifest.operations)) { + const operation = asObject(entry); + const id = asString(operation?.id); + if (!id) continue; + const method = asString(operation?.method) ?? 'GET'; + const url = asString(operation?.url) ?? ''; + const effect = asString(operation?.effect); + capabilities.push({ + kind: 'operation', + title: `${method} ${url}`.trim(), + reason: effect ? `${id}: ${effect}s external data` : id, + }); + } + + // `DeclaredCapability`: a bare name or `{ name, reason }`. These are what + // `Installation.grants` records. + for (const entry of asArray(manifest.capabilities)) { + const object = asObject(entry); + const name = asString(entry) ?? asString(object?.name); + if (!name) continue; + capabilities.push({ + kind: 'capability', + title: name, + reason: asString(object?.reason), + grant: name, + }); + } + + // `network`: coarse egress for host `fetch` without an operation id. + const network = asObject(manifest.network); + const origins = asArray(network?.origins).flatMap(o => asString(o) ?? []); + + if (origins.length > 0) { + capabilities.push({ + kind: 'network', + title: `Network access to ${origins.join(', ')}`, + reason: asString(network?.reason), + }); + } + + return { + name: asString(manifest.name), + namespace: asString(manifest.namespace), + description: asString(manifest.description), + author: asString(manifest.author), + version: release.version ?? asString(manifest.version), + runtime: release.runtime ?? RUNTIME_JS, + world: release.world ?? WORLD_EXTENSION, + releaseId: release.id, + capabilities, + configSchema: manifest.configSchema as JSONValue | undefined, + defaultConfig: manifest.defaultConfig as JSONValue | undefined, + }; +} + +/** The grant names an installer approves when accepting the whole review. */ +export function grantsFor(review: InstallationReview): string[] { + return review.capabilities.flatMap(c => (c.grant ? [c.grant] : [])); +} + +/** Namespace for releases whose manifest declares none (catalog JS releases). */ +export const DEFAULT_INSTALLATION_NAMESPACE = 'community'; + +/** + * A plugin identifier the server accepts (`validate_plugin_identifier`): + * ASCII letters, digits, `-` and `_`, at most 128 characters. + */ +export function installationIdentifier(name: string): string { + const cleaned = name + .trim() + .replace(/[^A-Za-z0-9_-]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 128); + + return cleaned || 'plugin'; +} + +export interface InstallReleaseOptions { + drive: string; + release: ReleaseReference; + /** Plugin identifier on the drive; for wasip2 it must equal the manifest's name. */ + name: string; + /** For wasip2 it must equal the manifest's namespace. */ + namespace?: string; + description?: string; + version?: string; + config?: JSONValue; + /** Capability names the installer approved. */ + grants: string[]; + /** Committing `active` (the default) is what installs. */ + status?: InstallationStatus; +} + +/** + * Creates and saves an `Installation` under the drive. The server's commit + * hook does the actual install; the returned subject is the resource to + * open, pause, revoke or destroy. + */ +export async function installRelease( + store: Store, + options: InstallReleaseOptions, +): Promise { + const { + drive, + release, + name, + namespace, + description, + version, + config, + grants, + status = 'active', + } = options; + const propVals: Record = { + [core.properties.name]: name, + [server.properties.releaseId]: release.id, + [server.properties.installationStatus]: status, + [server.properties.grants]: grants, + }; + if (namespace) propVals[server.properties.namespace] = namespace; + if (description) propVals[core.properties.description] = description; + if (version) propVals[server.properties.version] = version; + if (config !== undefined) propVals[server.properties.config] = config; + + const installation = await store.newResource({ + isA: server.classes.installation, + parent: drive, + propVals, + }); + // `release` is an atomicURL, but a release published to this server's + // cache has no Release resource yet, only an id. The server resolves + // either, so skip the client-side URL check for it. + await installation.set(server.properties.release, release.url, false); + await installation.save(); + + return installation.subject; +} + +export interface UpdateReleaseOptions { + release: ReleaseReference; + /** Capability names the installer approved for the new release. */ + grants: string[]; + config?: JSONValue; + version?: string; +} + +/** + * Repoints an existing `Installation` at another release, in place. + * + * This is how a plugin is updated: the resource keeps its subject, its config + * and the agent it signs as, so everything it already owns stays its own. The + * server re-runs the activation, which verifies the new id, checks the grants + * and puts the new code on disk. + * + * All of it is one commit on purpose. `check_grants` requires the grants to + * equal what the new release declares, so a release that asks for a capability + * the old grants do not cover would be refused if the release landed first. + */ +export async function updateInstallationRelease( + store: Store, + installation: string, + options: UpdateReleaseOptions, +): Promise { + const { release, grants, config, version } = options; + const resource = await store.getResource(installation); + + await resource.set(server.properties.releaseId, release.id); + // A release that only exists in this server's cache is an id, not a URL, + // so the atomicURL check is skipped here as it is on install. + await resource.set(server.properties.release, release.url, false); + await resource.set(server.properties.grants, grants); + + if (version !== undefined) { + await resource.set(server.properties.version, version); + } + + if (config !== undefined) { + await resource.set(server.properties.config, config); + } + + await resource.save(); +} + +type InstallStore = Pick; + +/** + * Publishes a wasip2 zip as a private release on the store's server and + * returns its id with the record the server built, so the caller can review + * and install it without a second request. + */ +export async function publishZipRelease( + store: InstallStore, + drive: string, + file: Blob, + options: { world?: string; public?: boolean } = {}, + transport: typeof fetch = fetch, +): Promise<{ id: string; release: PublishedRelease }> { + const agent = store.getAgent(); + if (!agent) throw new Error('sign in before publishing a plugin release'); + const url = new URL('/plugin-release-package', store.getServerUrl()); + url.searchParams.set('drive', drive); + if (options.world) url.searchParams.set('world', options.world); + if (options.public) url.searchParams.set('public', 'true'); + const response = await transport(url.toString(), { + method: 'POST', + headers: { + ...(await signRequest(url.toString(), agent, {})), + 'Content-Type': 'application/zip', + }, + body: await file.arrayBuffer(), + }); + if (!response.ok) throw new Error(await response.text()); + + return response.json() as Promise<{ id: string; release: PublishedRelease }>; +} diff --git a/browser/lib/src/plugin-manifest.test.ts b/browser/lib/src/plugin-manifest.test.ts index 5477fabab..77ce09287 100644 --- a/browser/lib/src/plugin-manifest.test.ts +++ b/browser/lib/src/plugin-manifest.test.ts @@ -131,18 +131,33 @@ describe('originsMentionedIn', () => { }); describe('versioned manifest conformance', () => { - const cases = JSON.parse( - readFileSync( - new URL('../../../testdata/plugin-manifests.json', import.meta.url), - 'utf8', - ), - ); - - for (const fixture of cases) { - it(fixture.name, () => { - if (fixture.valid) - expect(() => validateManifest(fixture.manifest)).not.toThrow(); - else expect(() => validateManifest(fixture.manifest)).toThrow(); + // Shared with server/src/plugins/manifest.rs; both sides must agree on + // acceptance, the error and the canonical serialized form. + const fixture = (name: string) => + JSON.parse( + readFileSync( + new URL(`../../../testdata/plugin-manifest/${name}`, import.meta.url), + 'utf8', + ), + ); + const cases: { + name: string; + file: string; + error?: string; + serialized?: unknown; + }[] = fixture('index.json'); + + for (const entry of cases) { + it(entry.name, () => { + const raw = fixture(entry.file); + + if (entry.error !== undefined) { + expect(() => validateManifest(raw)).toThrow(entry.error); + } else { + const manifest = validateManifest(raw); + if (entry.serialized !== undefined) + expect(manifest).toEqual(entry.serialized); + } }); } }); diff --git a/browser/lib/src/plugin-manifest.ts b/browser/lib/src/plugin-manifest.ts index 928028600..9caae980c 100644 --- a/browser/lib/src/plugin-manifest.ts +++ b/browser/lib/src/plugin-manifest.ts @@ -16,6 +16,10 @@ import type { JSONValue } from './value.js'; * The declaration cannot drift from the code that spends it β€” the same file * says `secret:google` and asks for `google`. And an author, human or model, * writes one artifact rather than remembering to fill in a form elsewhere. + * + * Version two of the schema is the single manifest for both runtimes; the + * server's `server/src/plugins/manifest.rs` is the reference and both sides + * are checked against the fixtures in `testdata/plugin-manifest/`. */ export interface DeclaredSecret { /** Referred to in the source as `secret:`. */ @@ -34,11 +38,67 @@ export interface DeclaredOperation { effect: 'read' | 'write'; } -export interface PluginManifest { - actions?: DeclaredAction[]; - schemaVersion?: 1; - operations?: DeclaredOperation[]; +export type ManifestRuntime = 'atomic-js/1' | 'wasip2/1'; + +/** The trust boundary, independent of the language. */ +export type ManifestWorld = 'extension' | 'server-extension'; + +export type CapabilityName = + | 'storage' + | 'full-drive-access' + | 'extended-fuel' + | 'extended-memory' + | 'custom-view'; + +/** Either the bare name or the name with the reason shown at review. */ +export type DeclaredCapability = + | CapabilityName + | { name: CapabilityName; reason?: string }; + +export interface DeclaredEntrypoints { + /** Exports `run`. Defaults to true only when `entrypoints` is absent. */ + run?: boolean; + /** Package-relative path of the custom view module. Needs `custom-view`. */ + view?: string; + /** Class URLs whose hooks this package exports. Only in `server-extension`. */ + classExtender?: string[]; +} + +/** + * Coarse egress allowance for packages that call the host `fetch` without an + * operation id. Exact origins, no wildcards. Never widens `operations`. + */ +export interface DeclaredNetwork { + origins?: string[]; + reason?: string; +} + +export interface PluginManifestV2 { + schemaVersion: 2; + /** Defaults to `atomic-js/1`. */ + runtime?: ManifestRuntime; + /** Defaults to `extension`. */ + world?: ManifestWorld; + entrypoints?: DeclaredEntrypoints; + capabilities?: DeclaredCapability[]; secrets: DeclaredSecret[]; + operations?: DeclaredOperation[]; + actions?: DeclaredAction[]; + network?: DeclaredNetwork; + configSchema?: Record; + defaultConfig?: Record; + name?: string; + namespace?: string; + version?: string; + description?: string; + author?: string; +} + +export interface PluginManifest extends Omit< + PluginManifestV2, + 'schemaVersion' +> { + schemaVersion?: 1 | 2; } export interface DeclaredAction { @@ -145,22 +205,49 @@ export function originsMentionedIn(source: string): string[] { return [...found].sort(); } -/** Strict validation for released code. Draft source scans never grant access. */ +const RUNTIMES: ManifestRuntime[] = ['atomic-js/1', 'wasip2/1']; +const WORLDS: ManifestWorld[] = ['extension', 'server-extension']; +const CAPABILITIES: CapabilityName[] = [ + 'storage', + 'full-drive-access', + 'extended-fuel', + 'extended-memory', + 'custom-view', +]; +const IDENTIFIER = /^[A-Za-z0-9_-]{1,128}$/; + +/** + * Strict validation for released code. Draft source scans never grant access. + * + * Returns the manifest in its canonical serialized form: the one the server + * stores in a release and addresses it by. A version-one manifest stays + * version one; version-two fields at their default are left out. + */ export function validateManifest(raw: unknown): PluginManifest { - const object = (value: unknown): Record => { + const object = (value: unknown, what = 'manifest entry') => { if (!value || typeof value !== 'object' || Array.isArray(value)) - throw new Error('manifest entry must be an object'); + throw new Error(`${what}: invalid type, expected a map`); return value as Record; }; const known = (entry: Record, keys: string[]) => { - if (Object.keys(entry).some(key => !keys.includes(key))) - throw new Error('unknown manifest field'); + for (const key of Object.keys(entry)) + if (!keys.includes(key)) throw new Error(`unknown field \`${key}\``); + }; + + const text = (value: unknown, what: string): string => { + if (typeof value !== 'string') + throw new Error(`${what}: invalid type, expected a string`); + + return value; }; const endpoint = (value: unknown) => { - if (typeof value !== 'string') throw new Error('endpoint must be a URL'); + if (typeof value !== 'string') + throw new Error( + 'operation URLs must be HTTP endpoints without credentials, query or fragment', + ); const url = new URL(value); if ( !['http:', 'https:'].includes(url.protocol) || @@ -169,26 +256,58 @@ export function validateManifest(raw: unknown): PluginManifest { url.search || url.hash ) - throw new Error('invalid endpoint'); + throw new Error( + 'operation URLs must be HTTP endpoints without credentials, query or fragment', + ); return url; }; - const entry = object(raw); - known(entry, ['schemaVersion', 'secrets', 'operations', 'actions']); - if (entry.schemaVersion !== 1) - throw new Error('unsupported manifest schemaVersion'); + const exactOrigin = (value: unknown, what: string) => { + if (endpoint(value).origin !== value) + throw new Error(`${what} must be an exact HTTP origin`); - const list = (value: unknown): unknown[] => { + return value as string; + }; + + const list = (value: unknown, what = 'manifest field'): unknown[] => { if (value === undefined) return []; if (!Array.isArray(value)) - throw new Error('manifest field must be an array'); + throw new Error(`${what}: invalid type, expected a sequence`); return value; }; + const entry = object(raw, 'manifest'); + const version = entry.schemaVersion; + if (version !== 1 && version !== 2) + throw new Error('unsupported manifest schemaVersion'); + known( + entry, + version === 1 + ? ['schemaVersion', 'secrets', 'operations', 'actions'] + : [ + 'schemaVersion', + 'runtime', + 'world', + 'entrypoints', + 'capabilities', + 'secrets', + 'operations', + 'actions', + 'network', + 'configSchema', + 'defaultConfig', + 'name', + 'namespace', + 'version', + 'description', + 'author', + ], + ); + const names = new Set(); - const secrets = list(entry.secrets).map(value => { + const secrets = list(entry.secrets, 'secrets').map(value => { const secret = object(value); known(secret, ['name', 'origin', 'description']); if ( @@ -196,21 +315,19 @@ export function validateManifest(raw: unknown): PluginManifest { !secret.name || names.has(secret.name) ) - throw new Error('secret names must be unique'); + throw new Error('secret names must be nonempty and unique'); names.add(secret.name); - const url = endpoint(secret.origin); - if (url.origin !== secret.origin) - throw new Error('secret origin must be exact'); if ( secret.description !== undefined && typeof secret.description !== 'string' ) - throw new Error('description must be text'); + throw new Error('secret description must be text'); + exactOrigin(secret.origin, 'secret origin'); return secret as unknown as DeclaredSecret; }); names.clear(); - const operations = list(entry.operations).map(value => { + const operations = list(entry.operations, 'operations').map(value => { const operation = object(value); known(operation, ['id', 'method', 'url', 'effect']); if ( @@ -218,7 +335,7 @@ export function validateManifest(raw: unknown): PluginManifest { !operation.id || names.has(operation.id) ) - throw new Error('operation IDs must be unique'); + throw new Error('operation IDs must be nonempty and unique'); names.add(operation.id); endpoint(operation.url); if ( @@ -227,17 +344,17 @@ export function validateManifest(raw: unknown): PluginManifest { operation.method, ) ) - throw new Error('unsupported method'); + throw new Error('operation has an unsupported HTTP method'); if ( typeof operation.effect !== 'string' || !['read', 'write'].includes(operation.effect) ) - throw new Error('unsupported effect'); + throw new Error('operation effect must be read or write'); return operation as unknown as DeclaredOperation; }); names.clear(); - const actions = list(entry.actions).map(value => { + const actions = list(entry.actions, 'actions').map(value => { const action = object(value); known(action, ['name', 'title', 'description', 'operation', 'inputSchema']); if ( @@ -247,10 +364,11 @@ export function validateManifest(raw: unknown): PluginManifest { typeof action.title !== 'string' || !action.title || typeof action.description !== 'string' || - action.description.length > 8192 || - !operations.some(o => o.id === action.operation) + action.description.length > 8192 ) - throw new Error('invalid action declaration'); + throw new Error('invalid or duplicate action name/description'); + if (!operations.some(o => o.id === action.operation)) + throw new Error('action references an undeclared operation'); names.add(action.name); const schema = object(action.inputSchema); known(schema, ['type', 'properties', 'required', 'additionalProperties']); @@ -272,17 +390,219 @@ export function validateManifest(raw: unknown): PluginManifest { !['string', 'integer', 'boolean'].includes(String(field.type)) || typeof field.description !== 'string' ) - throw new Error('unsupported action field'); + throw new Error('unsupported action input schema'); } return action as unknown as DeclaredAction; }); if (actions.length > 64) throw new Error('at most 64 actions per release'); + if (version === 1) { + return { + schemaVersion: 1, + secrets, + operations, + ...(actions.length ? { actions } : {}), + }; + } + + const variant = ( + value: unknown, + allowed: T[], + fallback: T, + ): T => { + if (value === undefined) return fallback; + if (typeof value !== 'string' || !allowed.includes(value as T)) + throw new Error(`unknown variant \`${String(value)}\``); + + return value as T; + }; + + const runtime = variant(entry.runtime, RUNTIMES, 'atomic-js/1'); + const world = variant(entry.world, WORLDS, 'extension'); + + const network = { origins: [] as string[], reason: undefined as unknown }; + + if (entry.network !== undefined) { + const declared = object(entry.network, 'network'); + known(declared, ['origins', 'reason']); + network.origins = list(declared.origins, 'network.origins').map(origin => + exactOrigin(origin, 'network origin'), + ); + if (new Set(network.origins).size !== network.origins.length) + throw new Error('network origins must be unique'); + if (declared.reason !== undefined) + network.reason = text(declared.reason, 'network.reason'); + } + + const seenCapabilities = new Set(); + const capabilities = list(entry.capabilities, 'capabilities').map(value => { + let name: unknown; + let reason: unknown; + + if (typeof value === 'string') name = value; + else if (value && typeof value === 'object' && !Array.isArray(value)) { + const declared = value as Record; + if (Object.keys(declared).some(key => !['name', 'reason'].includes(key))) + throw new Error('capability must be a known name or {name, reason}'); + name = declared.name; + reason = declared.reason; + } + + if ( + typeof name !== 'string' || + !CAPABILITIES.includes(name as CapabilityName) || + (reason !== undefined && typeof reason !== 'string') + ) + throw new Error('capability must be a known name or {name, reason}'); + if (seenCapabilities.has(name)) + throw new Error('capabilities must be unique'); + seenCapabilities.add(name); + + return reason === undefined + ? (name as CapabilityName) + : { name: name as CapabilityName, reason: reason as string }; + }); + + const entrypoints: Required = { + run: entry.entrypoints === undefined, + view: undefined as unknown as string, + classExtender: undefined as unknown as string[], + }; + + if (entry.entrypoints !== undefined) { + const declared = object(entry.entrypoints, 'entrypoints'); + known(declared, ['run', 'view', 'classExtender']); + + if (declared.run !== undefined) { + if (typeof declared.run !== 'boolean') + throw new Error('entrypoints.run: invalid type, expected a boolean'); + entrypoints.run = declared.run; + } + + if (declared.view !== undefined) + entrypoints.view = text(declared.view, 'entrypoints.view'); + if (declared.classExtender !== undefined) + entrypoints.classExtender = list( + declared.classExtender, + 'entrypoints.classExtender', + ).map(url => text(url, 'entrypoints.classExtender')); + } + + const classUrls = entrypoints.classExtender ?? []; + if (entrypoints.classExtender !== undefined && classUrls.length === 0) + throw new Error('classExtender must list at least one class URL'); + names.clear(); + + for (const cls of classUrls) { + const url = new URL(cls); + if (!['http:', 'https:'].includes(url.protocol) || !url.hostname) + throw new Error('classExtender entries must be class URLs'); + if (names.has(cls)) throw new Error('classExtender entries must be unique'); + names.add(cls); + } + + if (world === 'extension' && classUrls.length > 0) + throw new Error('world extension may not declare classExtender'); + if ( + world === 'server-extension' && + runtime !== 'wasip2/1' && + classUrls.length === 0 + ) + throw new Error( + 'world server-extension requires runtime wasip2/1 or entrypoints.classExtender', + ); + + if (entrypoints.view !== undefined) { + const view = entrypoints.view; + if ( + !view || + view.startsWith('/') || + view.includes('\\') || + view.split('/').some(segment => !segment || segment === '..') + ) + throw new Error('view entrypoint must be a package-relative path'); + if (!seenCapabilities.has('custom-view')) + throw new Error('view entrypoint requires the custom-view capability'); + } + + const metadata: Partial< + Pick< + PluginManifestV2, + 'name' | 'namespace' | 'version' | 'description' | 'author' + > + > = {}; + + for (const key of [ + 'name', + 'namespace', + 'version', + 'description', + 'author', + ] as const) { + if (entry[key] !== undefined) metadata[key] = text(entry[key], key); + } + + for (const key of ['namespace', 'name'] as const) { + const value = metadata[key]; + if (value !== undefined && !IDENTIFIER.test(value)) + throw new Error( + `plugin ${key} '${value}' is invalid: only ASCII letters, digits, '-' and '_' are allowed`, + ); + } + + const entrypointsDefault = + entrypoints.run === true && + entrypoints.view === undefined && + entrypoints.classExtender === undefined; + return { - schemaVersion: 1, + schemaVersion: 2, + ...(runtime !== 'atomic-js/1' ? { runtime } : {}), + ...(world !== 'extension' ? { world } : {}), + ...(entrypointsDefault + ? {} + : { + entrypoints: { + ...(entrypoints.run ? { run: true } : {}), + ...(entrypoints.view !== undefined + ? { view: entrypoints.view } + : {}), + ...(entrypoints.classExtender !== undefined + ? { classExtender: entrypoints.classExtender } + : {}), + }, + }), + ...(capabilities.length ? { capabilities } : {}), secrets, operations, ...(actions.length ? { actions } : {}), + ...(network.origins.length || network.reason !== undefined + ? { + network: { + ...(network.origins.length ? { origins: network.origins } : {}), + ...(network.reason !== undefined + ? { reason: network.reason as string } + : {}), + }, + } + : {}), + ...(entry.configSchema !== undefined + ? { + configSchema: object(entry.configSchema, 'configSchema') as Record< + string, + JSONValue + >, + } + : {}), + ...(entry.defaultConfig !== undefined + ? { + defaultConfig: object(entry.defaultConfig, 'defaultConfig') as Record< + string, + JSONValue + >, + } + : {}), + ...metadata, }; } diff --git a/docs/src/plugins.md b/docs/src/plugins.md index 61dc5138e..458125cd4 100644 --- a/docs/src/plugins.md +++ b/docs/src/plugins.md @@ -6,22 +6,34 @@ They enhance the functionality of an Atomic Server by extending one or more clas For example they can be used to create more restrictive requirements for classes, like requiring names to start with an uppercase letter. They can also add dynamic properties to classes that get populated each time the resource is fetched. -Plugins can be created in any programming language that compiles to Wasm. +Plugins can be created in any programming language that compiles to Wasm, or written in JavaScript in the browser. For more information on how to create a plugin, see [Creating Plugins](plugins/creating-plugins.md). +## Releases, Listings and Installations + +Three classes describe a plugin's life on a server: + +- A **Release** is an immutable, content-addressed package: JS source (`atomic-js/1`) or a WASM zip (`wasip2/1`), its manifest, and a `releaseId` (`blake3:…` over every field). It lives at `/releases/` on the server it was published to, and can be fetched from any other server. +- A **Listing** is a marketplace entry pointing at a Release: name, emoji, description, domains, standards. Every Listing the public can read on a server is part of that server's Store; `GET /plugin-catalog` returns them. Any drive of Listings is a marketplace. +- An **Installation** is one installed Release on one Drive. It pins the `release` URL and its `releaseId`, holds the `grants` you approved, your `config`, and an `installationStatus` (`draft`, `active`, `paused`, `revoked`). Committing it as `active` is the single install trigger for both runtimes; `paused` stops the plugin without uninstalling it, keeping its files, its config and the agent it signs as; `revoked` or destroying the resource uninstalls it and retires that agent. Updating a plugin means pointing the same Installation at a newer Release, which keeps everything the plugin already owns. + ## Installing a plugin -To install a plugin you need to have write access to the drive that the plugin will be installed on. -Navigate to the drive and click on the 'Upload Plugin' button. -Select your plugin zip file. -You will see a description of the plugin, any permissions it requires and a config field. -Edit the config if needed and click 'Install'. -You can change the config at any time once the plugin is installed. +You need write access to the drive the plugin will be installed on. Two ways in, one review: + +- **From the Store**: open the Store, pick a Listing and click 'Install'. +- **From a zip**: on the drive, choose 'Upload plugin' and select the zip. The server publishes it as a private Release on this server first. + +Both end in the Installation review: the release's manifest, every capability it declares with the reason its author gave, the origins it may reach, and a config field. The server only activates an Installation whose `grants` equal exactly the set of capabilities the manifest declares, so nothing is approved by omission. You can change the config at any time once the plugin is installed; changing it does not reinstall anything. + +Pausing an Installation keeps its files and identity; resuming the same release does not extract or compile it again. Revoking or deleting it removes the plugin's files, its class extender and its agent. + +Servers that still have plugins installed the old way (a `Plugin` resource with a `pluginFile`) migrate them on startup: the zip becomes a Release, and the same resource becomes an active Installation pinned to it, without touching what is on disk. ## Giving your plugin access to resources -By default plugins do not have access to any resources unless they have the `full-drive-access` permission. -To add access to specific resources (and their children) navigate to the plugin page and add the resource in the 'Assign Rights' section. +By default plugins do not have access to any resources unless they have the `full-drive-access` capability. +To add access to specific resources (and their children) navigate to the Installation page and add the resource in the 'Assign Rights' section. + + + + + + diff --git a/brand/src/place-mark.svg b/brand/src/place-mark.svg new file mode 100644 index 000000000..1a2ca9c44 --- /dev/null +++ b/brand/src/place-mark.svg @@ -0,0 +1,14 @@ + + + + + + + + + + + + + + diff --git a/browser/data-browser/index.html b/browser/data-browser/index.html index c1f811481..783d29e12 100644 --- a/browser/data-browser/index.html +++ b/browser/data-browser/index.html @@ -10,27 +10,27 @@ content="The easiest way to create, share and model Linked Atomic Data." name="description" /> - + @@ -70,7 +70,7 @@ href="https://fonts.googleapis.com/css2?family=Montserrat:wght@700&family=Open+Sans:ital,wght@0,400;0,700;1,400;1,700&display=swap" rel="stylesheet" /> - Atomic Data + Atomic Place