docs(pnpr): SQL metadata adapters, shared deployments on Turso by zkochan · Pull Request #66 · pnpm/rfcs

zkochan · 2026-11-02T07:48:03Z

Summary

Adds Backend adapters and Shared deployments sections to the SQL metadata storage RFC.

Metadata is implemented once, over SQL, with a small adapter per database. The schema, queries, removed-version refusal, generations, outcomes and caches are shared, and the queries stay in the subset SQLite, libSQL and PostgreSQL have in common. An adapter supplies only what differs: connecting, starting a write transaction (BEGIN IMMEDIATE or SERIALIZABLE), recognizing a retryable conflict, placeholders and types, and replica freshness. The operator picks the database with metadata.url (sqlite:, libsql:, postgres:), and every adapter must pass the conformance suite. SQLite and libSQL ship first, PostgreSQL when needed. Document storage can't be an adapter because it has no transactions.
The shared store holds only what needs a transaction:
- the index of accepted versions and their small accepted manifests;
- tags, revision selections, deprecations and policy bindings;
- generations, publication outcomes and sequenced log entries.
Its size and cost then grow with the number of packages and mutations, not with the size of what was published.
Large immutable data goes to object storage, keyed by digest: archives, evidence bundles, rendered release records, and the transparency log's tiles and checkpoints. Each is written before the transaction that references it, as archives already are. A CDN can serve the public ones directly.
Turso is the first shared backend. pnpr already keeps users and tokens in libSQL, with a Turso primary and an embedded replica. libSQL runs the embedded backend's schema unchanged, so metadata gets one implementation with two connection modes: a local file, or a libSQL primary with its replica.
- Writes run as BEGIN IMMEDIATE on the primary.
- Reads that must be current (a just-published version, a client's minimum generation, an authorization check) go to the primary or wait for the replica to sync past the needed generation.
Scaling: a primary serializes writes. Publications and tag moves are rare next to reads, and reads scale with replicas and caches. A deployment that outgrows one primary gives each hosted catalog its own database, since a catalog is already the transaction boundary.
One metadata model: once Turso passes the conformance suite, the per-package document metadata is removed. Atomic publication, generations and the rest then work in every deployment.
Other shared stores stay possible, each subject to the same conformance suite: PostgreSQL, Cloudflare D1 (one batch per transaction, writes conditional on the generation read), and Durable Objects (one object per catalog).

Single-instance deployments keep embedded SQLite. The tiles-in-blob-storage rule this relies on is in #62.

Written by an agent (Claude Code, claude-opus-5-5).

🤖 Generated with Claude Code

Summary by CodeRabbit

Documentation
- Updated backend guidance to describe SQLite as the initial single-instance option and Turso as a proposed shared option, subject to conformance and workload testing.
- Clarified that shared deployments keep transactional metadata in the database and store large, immutable artifacts in object storage.
- Documented backend selection by URL, read-freshness behavior, and catalog-specific database selection.
- Added guidance on Cloudflare D1 and Durable Objects, including their transaction constraints.

coderabbitai · 2026-11-02T07:48:12Z

No actionable comments were generated in the recent review.

Recent review info

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 1a71af13-6782-4535-a377-df15e20a97a2

Commits

Reviewing files that changed from the base of the PR and between 9a3aada and dca0b39.

Files selected for processing: 1

pnpr/text/0000-sql-metadata-storage.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Recent review details

Context from checks skipped due to timeout: 1

GitHub Check: CodeRabbit

Additional comments: 1

pnpr/text/0000-sql-metadata-storage.md: 1

168-168: Specify that eviction skips in-flight connections.

The cache rule does not require the least-recently-used connection to be idle. Transactions use a pinned connection for every statement. If catalog churn evicts that connection during an active request, the operation can fail. State that eviction skips leased or pinned connections and waits for active work to finish. (raw.githubusercontent.com)

Walkthrough

The RFC defines shared SQL domain operations and database adapters for backend-specific differences. It describes SQLite and Turso deployments, data placement, read freshness, catalog-specific databases, and possible alternative backends.

Changes: SQL Metadata Storage

Layer / File(s)	Summary
Shared SQL backend model `pnpr/text/0000-sql-metadata-storage.md`	The design assigns domain operations to a shared SQL implementation. Adapters handle connection setup, transactions, retryable conflicts, SQL placeholders and types, and read freshness. SQLite and libSQL ship first; PostgreSQL may follow.
Deployment and read freshness `pnpr/text/0000-sql-metadata-storage.md`	The design specifies Turso as the first proposed shared backend, data placement, read consistency rules, catalog-specific database selection, and possible alternative backends. It also adds prior-art references for D1 batches, SQLite-backed Durable Objects, and tiled logs served as static files.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

pnpm/rfcs#64: Defines the SQL metadata contract that this RFC extends with backend adapters and shared deployment behavior.
pnpm/rfcs#46: Defines publication transactions and backend conformance that relate to this RFC’s adapter and deployment rules.
pnpm/rfcs#49: Covers immutable release records and transparency-log data stored across SQL metadata and object storage.

Merge Risk: Minimal · up to dca0b

This documentation-only proposal has no runtime impact and no actionable merge-blocking issues remain.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely identifies the main changes: SQL metadata adapters and shared deployments using Turso.
Docstring Coverage		No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

Finishing Touches

Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

_{A rabbit checks the SQL flow,
While Turso’s fresh reads come and go.
The catalogs find their place,
Large artifacts keep their space.
Adapters guide each query through,
And tiled logs join the view.}

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

pnpr already keeps users and tokens in libSQL with a Turso primary and an embedded replica, and libSQL runs the embedded backend's schema unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…r database Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

qodo-code-review · 2026-11-02T14:46:54Z

PR Summary by Qodo

Document SQL adapters and Turso shared metadata deployments

📝 Documentation 🕐 20-40 Minutes

AI Description

• Define shared SQL domain operations and database-specific adapters governed by a conformance
 suite.
• Keep transactional metadata in SQL and large immutable artifacts in digest-keyed object storage.
• Select Turso as the first shared backend, with freshness rules and per-catalog scaling.

Diagram

graph TD
  H["Registry handlers"] --> D["SQL domain operations"] --> A["Database adapter"] --> T[("Turso primary")]
  A --> S[("Embedded SQLite")]
  T --> R[("Embedded replica")]
  D --> O[("Object storage")]
  D --> C["Generated views"]

High-Level Assessment

Reusing SQLite-compatible SQL and pnpr's existing libSQL connection is a coherent first shared-backend strategy. The RFC already considers PostgreSQL and Cloudflare stores; those would add deployment or transaction-model differences without an identified need to replace Turso as the first candidate. Keep the conformance suite as the gate before support or removal of document metadata.

Files changed (1) +100 / -6

Documentation (1) +100 / -6

0000-sql-metadata-storage.mdSpecify SQL adapters and shared metadata storage +100/-6

Specify SQL adapters and shared metadata storage

• Defines shared SQL domain operations, adapter responsibilities, URL-based backend selection, and conformance requirements. Specifies which data belongs in transactional metadata versus object storage, selects Turso for shared deployments, and documents replica freshness, per-catalog scaling, and possible later backends.

pnpr/text/0000-sql-metadata-storage.md

coderabbitai

Fix comments: Fix CodeRabbit comments on this PR

…the primary, route catalogs by URL Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai

Fix comments: Fix CodeRabbit comments on this PR

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The libSQL metadata adapter needs the same connection code the auth backend has: the TLS connector, the connection pool that discards a stream whose baton was lost, and the embedded replica of a Turso primary. Move it out of `pnpr-auth` into a new `pnpr-libsql` crate so both use one copy, as pnpm/rfcs#66 asks ("Metadata reuses that connection code"). `pnpr-libsql` holds: - `LibsqlConnector` and `connector(settings)`, which refuses an auth token over plain HTTP to a host other than loopback; - `open(settings, connector)`, an embedded replica when `replicaPath` is set and the remote primary otherwise, and `open_primary`; - `ConnectionPool` and `first_row`; - `is_transaction_conflict` and `is_unique_violation`; - behind a `testing` feature, the local primary that serves libsql's sync protocol, and a plain-HTTP canned server, for tests in other crates. `LibsqlAuth` keeps its schema, retries, and records, and opens its database through `pnpr-libsql`. The auth tests that need the TLS test server from `pnpm-testing-utils` stay in `pnpr-auth`. The other connector and pool tests move with the code. A unified test build links two SQLite copies, libsql's and rusqlite's, and fails on duplicate symbols unless libsql's comes first on the link line. `pnpr-libsql` loads `pnpr-error` (and with it rusqlite) ahead of libsql, the order `pnpr-auth` and `pnpr` already link in. No behavior changes, except that the auth-token refusal no longer names the `backend.libsql` key, since metadata will reuse it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The SQL metadata tests now run once per adapter, as pnpm/rfcs#66 requires before an adapter is supported: SQLite, a libSQL primary, and a libSQL primary read through an embedded replica on each instance. The libSQL variants run against the local primary from `pnpr-libsql::testing`, which serves libsql's sync protocol, so the replica variant exercises real frame replication and replica lag. `conformance!` instantiates each `async fn(Db)` test in `on_sqlite`, `on_libsql`, and `on_libsql_replica` modules. `Db` is the database under test; every `storage(&db)` is another pnpr instance on it with a handle of its own, and in the replica variant a replica of its own that syncs every hour, so it lags unless a read requires otherwise. The suite covers concurrent publication from two instances, a lost commit response retried against a restarted instance, recorded outcomes, removed versions, generations, fragment reuse, and release notes. Two tests changed to state what holds on every adapter: - The test that held the write connection during a read holds it from a blocking task, which libSQL's adapter needs to reach the runtime. - A cache that missed an invalidation is now checked with a read that names the writer's generation, as a client that saw the commit would: another instance's replica may lag otherwise, within its sync interval. The crash test (a raw transaction left uncommitted in the file), the schema refusal, and the busy-code classification stay SQLite-only. The mock primary now decodes Hrana's unpadded base64 blobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(pnpr): put SQL metadata behind a database adapter Split the SQLite metadata backend into a shared SQL layer and an adapter per database, as pnpm/rfcs#66 describes under "Backend adapters". The shared layer keeps the schema, the queries, removed-version refusal, generations, publication outcomes, release notes, and the render caches. The `Adapter` trait supplies what differs between databases: - connections: one writer, readers that never wait for it; - the statements that begin a write or a read transaction; - which errors are conflicts a rerun of the whole transaction clears; - placeholders and types, translated by its `Connection`; - where a read runs, the primary or a replica. The shared layer runs every statement through `Connection` with `?N` placeholders and owned `Param`/`Value` types, so a libSQL adapter can run the same code. It stays synchronous on the blocking pool. Each operation names the freshness it needs (`Current` for reads that decide a write or answer a retry, `Bounded` for views), which an adapter with a replica maps to the primary or the replica; SQLite reads its one database either way. Queries now stay in the subset SQLite, libSQL, and PostgreSQL share: `INSERT OR IGNORE` became `ON CONFLICT DO NOTHING`, removing a version or package deletes its rows explicitly instead of relying on a connection-level foreign-key cascade, and a taken publication name is found by a lookup inside the write transaction instead of by decoding a driver-specific constraint error. A write that the adapter reports as a conflict (SQLITE_BUSY or SQLITE_LOCKED for SQLite) runs again from its start, up to four times. Every read now runs in one read transaction. `SqliteMetadata` is `SqlMetadata`, opened with `SqlMetadata::open_sqlite`, and `Storage::sqlite_metadata` is `Storage::sql_metadata`. The existing tests pass with renames only; the test that held the write connection now holds it through the adapter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(pnpr): order SQL metadata items as the workspace lints require Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnpr): roll back a metadata transaction whose COMMIT fails A failed `COMMIT` can leave its transaction open, but the write transaction marked itself closed before running it, so its drop skipped the rollback and the writer's next `BEGIN IMMEDIATE` failed inside the open transaction. It is now marked closed only once the commit succeeds. The SQLite adapter also ends a transaction its writer was left in before the next write, and closes a reader still in its transaction instead of returning it to the idle readers, where every read on it would fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(pnpr): drop an unused import from the concurrency tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…7578) * feat(pnpr): configure metadata by URL, with SQLite by default The `metadata` block now names its database by URL (`url: sqlite:<path>`), replacing `backend` and `path`, as pnpm/rfcs#66 describes. A filesystem hosted store keeps its metadata in SQLite at `<storage>/.metadata/metadata.sqlite` when no URL is given. The documents backend stays only where nothing else serves yet: an `s3:` hosted store without a metadata URL. Removing it is a later step. `sqlite:` with an `s3:` block is refused, as `backend: sqlite` was: the database is single-instance. An unknown scheme or an empty path is refused at config load. pnpr is pre-1.0, so the old keys are not accepted and an existing filesystem store's documents are not imported. The auth and tooling tests that seed or read package documents on disk now pin the documents backend explicitly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnpr): keep `backend: documents` for stores seeded with documents The registry mock, the TypeScript test harness, and the integrated benchmark seed and edit `document.json` files in a filesystem store, which SQLite metadata does not read. Keep `metadata: { backend: documents }` as the explicit choice of the documents backend, and set it in the fixture config. Main already accepts that key, so the benchmark's mock of the main revision keeps working with the harness's config. `backend: sqlite`, `path`, and a `url` together with `backend` are refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pnpr): describe document metadata on a filesystem store too A filesystem store can choose `backend: documents`, so the release-notes error and the configuration guide no longer tie package documents to object storage, and the backup guide asks for the SQLite database only when the registry keeps one. The backup guide also says to stop pnpr or use SQLite's backup API to copy the database, and that `pnpr export` leaves out the history only SQLite metadata holds. The changeset states one change per sentence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pnpr): keep the backup instruction next to the database it names Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pnpr): name the whole dist-tag history an export leaves out Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pnpr): say the hosted store holds documents only without SQLite metadata Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pnpr): give the live metadata backup the database's full path Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pnpr): restore a metadata database the storage archive does not hold Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pnpr): clear a stale write-ahead log before restoring the metadata database Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(pnpr): keep documents for the current-version registry mock The pnpm v11 test setup starts a second pnpr on the storage pnpr-prepare seeds with package documents, from a config it writes itself. That config named no metadata backend, so the server now opened an empty SQLite database and answered 404 for the seeded `pnpm` package. It now sets `backend: documents`, as the fixture config does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

) * refactor(pnpr): move libSQL connection code into pnpr-libsql The libSQL metadata adapter needs the same connection code the auth backend has: the TLS connector, the connection pool that discards a stream whose baton was lost, and the embedded replica of a Turso primary. Move it out of `pnpr-auth` into a new `pnpr-libsql` crate so both use one copy, as pnpm/rfcs#66 asks ("Metadata reuses that connection code"). `pnpr-libsql` holds: - `LibsqlConnector` and `connector(settings)`, which refuses an auth token over plain HTTP to a host other than loopback; - `open(settings, connector)`, an embedded replica when `replicaPath` is set and the remote primary otherwise, and `open_primary`; - `ConnectionPool` and `first_row`; - `is_transaction_conflict` and `is_unique_violation`; - behind a `testing` feature, the local primary that serves libsql's sync protocol, and a plain-HTTP canned server, for tests in other crates. `LibsqlAuth` keeps its schema, retries, and records, and opens its database through `pnpr-libsql`. The auth tests that need the TLS test server from `pnpm-testing-utils` stay in `pnpr-auth`. The other connector and pool tests move with the code. A unified test build links two SQLite copies, libsql's and rusqlite's, and fails on duplicate symbols unless libsql's comes first on the link line. `pnpr-libsql` loads `pnpr-error` (and with it rusqlite) ahead of libsql, the order `pnpr-auth` and `pnpr` already link in. No behavior changes, except that the auth-token refusal no longer names the `backend.libsql` key, since metadata will reuse it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pnpr): drop a link to a private item from the pool's docs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnpr): keep the libSQL token check and timeout error with their owner `open` and `open_primary` now set the connector's token flag from the settings they open, so a connector built for no token still refuses to send the settings' token over plain HTTP to a host other than loopback. `ConnectionPool::new` takes the error a timed-out operation fails with, so a pool other than the auth backend's does not report its timeouts as `AuthDatabaseTimeout`. The auth backend passes `AuthDatabaseTimeout`. `connector` takes the config section its settings come from and names `<section>.authToken` when it refuses a token, as the auth backend's error did before the move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(pnpr): end the authToken refusal's format! with a comma Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…pnpm#17583) * feat(pnpr): keep metadata in libSQL, read through an embedded replica Add the libSQL metadata adapter that pnpm/rfcs#66 describes under "Shared deployments", so several pnpr instances share one metadata database such as Turso. `metadata.url` takes `libsql://` (or `http(s)://` for a self-hosted sqld), with `authToken`, `replicaPath`, `syncInterval`, and `maxConnections` beside it. A libSQL URL works with an `s3:` hosted store, so archives stay in object storage. The adapter runs the shared SQL layer unchanged, through `pnpr-libsql`'s connector and pool: - Writes run on the primary as `BEGIN IMMEDIATE` transactions, one at a time per process. A busy primary is a retryable conflict. - `ReadSource::Primary` reads the primary. `ReadSource::Replica` reads the embedded replica in a `BEGIN READONLY` transaction (a plain `BEGIN` on a replica connection would send the transaction to the primary), while the replica's last sync is within `syncInterval`. A replica past it syncs first; one that cannot sync leaves the read to the primary. The adapter syncs the replica itself every `syncInterval`, since libsql's background sync does not report when it last finished. - The database opens on first use, under a startup deadline, because the shared layer is synchronous on the blocking pool and opening reaches the network. Every `Storage` of a process that names the same settings shares one handle, so one replica file has one owner. The shared layer decides freshness. `Current` reads (read-for-update, recorded outcomes, catalog generation) go to the primary. `Bounded` and the new `AtLeast(generation)` read the replica when there is one, in one read transaction that first checks the catalog's generation: it must reach what this process last committed there, and the caller's minimum. A replica found behind is synced once and otherwise left for the primary. `Storage::read_hosted_package_view_at_least` exposes the minimum for a client-sent generation; no request header carries one yet. An authorization check (`is_package_owner`) now reads the current document, so a replica's copy cannot hide a maintainer's removal. Other changes: - `catalog_generation` and `tag_history` are async: on libSQL they reach the network. - The schema's statements run one at a time: libsql's remote `execute_batch` re-renders statements from its parser, which drops the comma in `STRICT, WITHOUT ROWID`. - The SQL layer's runners live in `sql/transactions.rs` and the package view in `sql/view.rs`. - `ConnectionPool::checkout` hands out a connection for statements run from a blocking thread. The mock primary in `pnpr-libsql::testing` gives each Hrana stream its own connection, serves `/v3/cursor`, carries blobs, and reports lock conflicts as `SQLITE_BUSY`. - The `Sqlite` and `Libsql` errors no longer say "Token database" or "Auth database", since metadata uses them too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(pnpr): format Cargo.toml Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnpr): read this process's own writes, notes included, from a replica A release-note edit commits without advancing the catalog generation, so checking the replica against the generation this process last committed missed it, and the replica served the old note. The libSQL conformance run in the next change found it. A write now marks the replica stale instead: the next replica read syncs first, and reads the primary when it cannot. That covers every write, whether or not it moves a generation, so the shared layer no longer tracks committed generations. A failed write marks it stale too, since its commit may have landed. `AtLeast` reads keep checking the catalog's generation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(pnpr): state what marking the replica stale records Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnpr): keep the replica stale when a write overtakes its sync A sync pulls a snapshot of the primary, and a write of this process can commit after that snapshot but before the sync finishes. The finished sync then overwrote the write's stale mark with a fresh time, and views read the replica without the write until the next sync. The replica now counts this process's writes. A sync records itself as fresh only if no write came after it started, and a read whose sync left the replica stale goes to the primary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnpr): fall back to the primary when the replica fails A replica read that fails in the database, rather than in the operation, now runs again on the primary instead of failing the read. A replica that cannot sync when the schema is first checked no longer fails every operation: the schema is marked ready, the error is logged, and the replica is left stale, so its reads sync first or go to the primary. A read's statements now stop waiting at a deadline, so a stalled connection fails the read and is closed instead of holding a blocking thread and a pool slot. Writes keep no deadline on their statements, because a write cut off after it reached the primary leaves its commit state unknown. The first operations of a process check the schema once between them, and a read of an at-least generation does not sync the replica again when its first attempt already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnpr): date a replica sync by its start, and refuse stray libSQL keys A replica's freshness now counts from when its last sync started, since its pull holds at least what the primary held then. A read waiting for a sync no longer takes one that started before it asked. `authToken`, `replicaPath`, `syncInterval`, and `maxConnections` without a metadata URL are refused, instead of being ignored by the SQLite or documents default. The test primary's stalled `/sync` answers every request with the frames it had when the stall began, so a test can commit while a pull misses it. A new test commits while a view's own sync is stalled and checks the next view sees the write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnpr): read the primary at once when the replica fails A replica read that failed in the database went down the path of a replica behind the required generation: it synced the replica and read it again before reading the primary, two waits of up to the operation deadline each. A failed replica read now goes to the primary at once; only a replica that is behind syncs and reads again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(pnpr): place the request after the sync start without reading the clock Two readings of `Instant::now` can be equal on macOS, so a request read just after a sync started could tie with it and count as answered. The test now builds the request time from the sync's start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ter (pnpm#17585) * test(pnpr): run the SQL metadata conformance tests against every adapter The SQL metadata tests now run once per adapter, as pnpm/rfcs#66 requires before an adapter is supported: SQLite, a libSQL primary, and a libSQL primary read through an embedded replica on each instance. The libSQL variants run against the local primary from `pnpr-libsql::testing`, which serves libsql's sync protocol, so the replica variant exercises real frame replication and replica lag. `conformance!` instantiates each `async fn(Db)` test in `on_sqlite`, `on_libsql`, and `on_libsql_replica` modules. `Db` is the database under test; every `storage(&db)` is another pnpr instance on it with a handle of its own, and in the replica variant a replica of its own that syncs every hour, so it lags unless a read requires otherwise. The suite covers concurrent publication from two instances, a lost commit response retried against a restarted instance, recorded outcomes, removed versions, generations, fragment reuse, and release notes. Two tests changed to state what holds on every adapter: - The test that held the write connection during a read holds it from a blocking task, which libSQL's adapter needs to reach the runtime. - A cache that missed an invalidation is now checked with a read that names the writer's generation, as a client that saw the commit would: another instance's replica may lag otherwise, within its sync interval. The crash test (a raw transaction left uncommitted in the file), the schema refusal, and the busy-code classification stay SQLite-only. The mock primary now decodes Hrana's unpadded base64 blobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(pnpr): build the per-instance stores outside the storage module Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pnpr): retry a libSQL write that the primary reports locked by message CI ran the libSQL conformance variants on three platforms and two concurrent-instance tests failed: - A busy primary can reach the client as `Hrana(Api(..))`, which carries the step's message but not its result code, so `database is locked` was not classified as a conflict and the write failed instead of running again. `is_transaction_conflict` now also recognizes SQLite's messages for SQLITE_BUSY and SQLITE_LOCKED. - The mock primary's `/export` checkpointed its WAL while another stream held a transaction, and panicked on SQLITE_LOCKED. It now retries the checkpoint until the transaction ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(pnpr): retry the test primary's export while its checkpoint is busy `PRAGMA wal_checkpoint(TRUNCATE)` reports a checkpoint that a reader or writer blocked in its `busy` column, not as an error, and leaves frames in the WAL. The test primary's export now retries such a checkpoint too, so it never serves a database file that misses committed frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(pnpr): move the plain-HTTP test server out of the test primary The busy-checkpoint retry took `testing.rs` past the workspace's 400-line limit. The plain-HTTP server that answers every request alike has nothing to do with the primary, so it moves into `testing/plain_http.rs`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(pnpr): put the test primary's re-exports before its imports Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

zkochan force-pushed the rfc/shared-deployment-storage branch from 056f423 to c3e8c3e Compare November 2, 2026 08:00

docs(pnpr): keep only the mutable index in a shared deployment's store

02f6855

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zkochan force-pushed the rfc/shared-deployment-storage branch from c3e8c3e to 02f6855 Compare November 2, 2026 12:42

zkochan and others added 2 commits November 2, 2026 15:30

docs(pnpr): make Turso the first shared metadata backend

2d75074

pnpr already keeps users and tokens in libSQL with a Turso primary and an embedded replica, and libSQL runs the embedded backend's schema unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(pnpr): scale a shared backend with one database per catalog

ca1f23d

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zkochan changed the title ~~docs(pnpr): keep only the mutable index in a shared deployment's store~~ docs(pnpr): shared deployments on Turso, bulk data in object storage Nov 2, 2026

zkochan mentioned this pull request Nov 2, 2026

Implement the pnpr RFCs merged October 31 – November 2, 2026 pnpm/tasks#244

Open

14 of 39 tasks

docs(pnpr): implement metadata once over SQL, with a small adapter pe…

e2555a4

…r database Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zkochan changed the title ~~docs(pnpr): shared deployments on Turso, bulk data in object storage~~ docs(pnpr): SQL metadata adapters, shared deployments on Turso Nov 2, 2026

zkochan marked this pull request as ready for review November 2, 2026 14:46

coderabbitai Bot requested changes Nov 2, 2026

View reviewed changes

Comment thread pnpr/text/0000-sql-metadata-storage.md Resolved

Comment thread pnpr/text/0000-sql-metadata-storage.md Resolved

Comment thread pnpr/text/0000-sql-metadata-storage.md Resolved

Comment thread pnpr/text/0000-sql-metadata-storage.md Resolved

docs(pnpr): fix PostgreSQL transaction form, read authorization from …

9a3aada

…the primary, route catalogs by URL Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai Bot requested changes Nov 2, 2026

View reviewed changes

Comment thread pnpr/text/0000-sql-metadata-storage.md Resolved

docs(pnpr): bound the per-catalog connection cache

dca0b39

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai Bot approved these changes Nov 2, 2026

View reviewed changes

zkochan merged commit 1366c48 into main Nov 2, 2026
1 check passed

This was referenced Nov 2, 2026

refactor(pnpr): put SQL metadata behind a database adapter pnpm/pnpm#17577

Merged

feat(pnpr): configure metadata by URL, with SQLite by default pnpm/pnpm#17578

Merged

refactor(pnpr): move libSQL connection code into pnpr-libsql pnpm/pnpm#17580

Merged

This was referenced Nov 2, 2026

feat(pnpr): keep metadata in libSQL, read through an embedded replica pnpm/pnpm#17583

Merged

test(pnpr): run the SQL metadata conformance tests against every adapter pnpm/pnpm#17585

Merged

This was referenced Nov 3, 2026

test(pnpr): run the SQL metadata conformance tests on an external libSQL database pnpm/pnpm#17600

Draft

pnpr: make remaining local state pluggable for stateless / serverless deployment pnpm/tasks#21

Closed

Navigation Menu

Repository navigation

docs(pnpr): SQL metadata adapters, shared deployments on Turso - #66

docs(pnpr): SQL metadata adapters, shared deployments on Turso#66
zkochan merged 6 commits into
mainfrom
rfc/shared-deployment-storage

zkochan commented Nov 2, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

qodo-code-review Bot commented Nov 2, 2026

Uh oh!

coderabbitai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zkochan commented Nov 2, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qodo-code-review Bot commented Nov 2, 2026

PR Summary by Qodo

Uh oh!

coderabbitai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

zkochan commented Nov 2, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Nov 2, 2026 •

edited

Loading

coderabbitai Bot left a comment •

edited

Loading

coderabbitai Bot left a comment •

edited

Loading