[zkochan]

Summary

Split from #44. Stores pnpr's authoritative metadata in a transactional SQL database and keeps archives in the filesystem or object storage. npm package documents become generated views, cached as per-version fragments. Embedded SQLite comes first; a shared backend (PostgreSQL or Turso) is added only after it passes a shared conformance suite.

This is the storage contract that #45 and #46 build on, in place of the SQL section of #44.

Related drafts

• rfc(pnpr): publish-time manifest verification #47
• rfc(pnpr): verifiable release records for every ecosystem, on transparency logs, in-toto, and TUF #49
• rfc(pnpr): resolution snapshots and retained dependency sets #45, rfc(pnpr): atomic workspace publication #46

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 55 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 75fd11f2-76eb-4755-a5c8-19ee4600cee1

Commits

Reviewing files that changed from the base of the PR and between ea127a2 and 81ed74d.

Files selected for processing: 1

• pnpr/text/0000-sql-metadata-storage.md

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[zkochan]

Updated in e27176e.

• Added evidence from the npm registry: a 17.9 MB typescript document, a 7-minute deprecate, and ETARGET after publish caused by separately cached full and abbreviated views.
• Added insert-only tag_history and a separate readmes relation.
• Added Read-after-write: responses carry the committed package generation, so a client can read its own publication without polling.

---

Written by an agent (Claude Code, claude-opus-5-5).

[zkochan]

A proposed answer to the open question "Should the per-version fragments themselves become a client-facing read path?", with measurements. It came up in the platform variants RFC (#51), where variant integrities would make packuments several times larger, but it applies to every package.

Why

Most of a gzipped packument is hashes. Per-version digests and npm registry signatures do not compress, while everything else repeats from version to version. I measured npm's abbreviated documents of 44 popular packages, in the form pnpr serves (without npm-signature, and without shasum when integrity is present). Moving each version's dist out of the document cuts the gzipped total from 7.9 MB to 1.3 MB (83%). Some examples:

Package	gzip today	gzip without per-version dist
next	1040 KB	391 KB
typescript	647 KB	67 KB
react	473 KB	33 KB
@types/node	366 KB	16 KB
esbuild	90 KB (361 KB with variant integrities)	21 KB

Proposal

Every response stays a plain GET addressed by its URL, so a CDN caches it for everyone. There are no range queries, ?since= cursors or POST lookups, because each of those makes its own cache key, or none.

• A media type that clients opt into (application/vnd.pnpr.install-v2+json in the prototype), with Vary: Accept. npm, Yarn and Bun keep today's abbreviated document.
• Split only large documents. Above a threshold, each version's dist (integrity, sizes, bins, signatures, variant archives) moves to the immutable version document <name>/<version>, served with Cache-Control: immutable. Small packages stay whole and cost no extra request. Once a package is split it stays split, so it does not flip between shapes near the threshold.
• An explicit marker per version: dist.pnpr.details: "external". A missing integrity cannot be the signal, because it already means the client computes the integrity while fetching the tarball. A per-version marker also lets one document split only some of its versions.
• The likely picks stay inline: every dist-tag target and the highest release of each major (of each minor below 1.0). A fresh resolution almost always picks one of these, so it rarely needs a second request. In the benchmark below, this cut version-document requests from 180 to 14.
• Provenance (dist.attestations) stays inline, because the trust policy reads it before a version is picked.
• A cache of its own on the client. A client must keep split documents apart from the abbreviated documents it cached before, because an older release sharing the cache directory would read versions without their integrity. The prototype hit this: an older pnpm on PATH read a split document that the new one had cached.

The fragment cache this RFC already describes maps directly onto it: a version document is one rendered fragment, and the split document is the package view with the large fragments cut down.

Prototype and benchmark

Implemented in pnpm/pnpm#17522: pnpr's split_packuments.rs, and pnpm's external_dist.rs, which fetches the version document of a picked version once per install. The benchmark resolves an app of 787 packages proxied from npmjs. It runs through a proxy that gzips and caches like a CDN, adds a round trip per request and caps bandwidth, and every run starts with an empty pnpm cache and store. Medians of 3 runs:

Network	Whole	Split, likely picks inline
10 ms, 200 Mbit/s, resolve	1.71 s	1.21 s
40 ms, 50 Mbit/s, resolve	6.67 s	3.98 s
100 ms, 10 Mbit/s, resolve	31.9 s	19.0 s
40 ms, 50 Mbit/s, full install	29.9 s	27.4 s
Packument bytes, gzip	39.2 MB	23.0 MB

Only 180 of the 711 documents crossed the 64 KiB threshold, but they hold most of the bytes.

Open points

• Version documents are not yet cached on disk by pnpm. They never change, so caching them is simple, and without it a warm-cache resolution refetches them.
• The threshold should be chosen from the size distribution of the registry, measured on gzipped size.
• Incremental updates of the index itself (only what was added since the client's copy) can stay cacheable: Cargo-style conditional GETs of the whole document, or append-only segments where every closed segment is immutable.

---

🤖 Generated with Claude Code (Claude Opus 5.5)

[zkochan]

A suggested review order for the pnpr metadata drafts, since they reference each other:

1. rfc(pnpr): SQL metadata storage #48 (this one), SQL metadata storage: the storage contract. Accepted manifests are insert-only; tags, revision selections, and deprecations are mutable state beside them.
2. rfc(pnpr): verifiable release records for every ecosystem, on transparency logs, in-toto, and TUF #49, release records and upstream verification: immutable records with endorsements and evidence kept apart.
3. rfc(pnpr): split packuments #54, split packuments: what clients download. A large package document lists only its likely picks in full, with every other version reduced to its publish time and mutable flags, and each release record served at an immutable <name>/<version>+r<revision> URL. This answers this RFC's open question about a client-facing per-version read path.
4. rfc(pnpr): platform variants, one version with many artifacts #51, platform variants: uses rfc(pnpr): split packuments #54 to keep variant archives out of large documents.

These can be reviewed at any point: #47 (publish-time manifest verification) doesn't depend on the others, and #45 and #46 build only on #48. #44 is closed, superseded by #47, #48, and #49.

The proposal in my earlier comment is now #54, and the design changed since. Its benchmark numbers came from a setup where pnpm fetched full documents for optional dependencies, so they understated the gain. With that fixed, cold resolution of a 787-package app is 72–81% faster and downloads 5.3 MB of metadata instead of 30.3 MB; the numbers are in #54.

---

Written by an agent (Claude Code, claude-opus-5-5).

🤖 Generated with Claude Code

[qodo-code-review]

PR Summary by Qodo

RFC: Store pnpr metadata in SQL and generate cached npm views

📝 Documentation 🕐 20-40 Minutes

AI Description

• Proposes transactional SQL as pnpr’s metadata authority, with archives retained in file or object
 storage.
• Defines immutable version records, mutable state, and generation-based caching to avoid
 whole-document rewrites and stale reads.
• Starts with SQLite; requires conformance tests before adding a shared backend.

Diagram

graph TD
  H["Registry handlers"] --> D["Domain operations"] --> S[("SQL metadata")] --> F["Version fragments"] --> V["npm views"]
  D --> B[("Artifact blobs")]
  S --> V

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Keep package documents authoritative

• ➕ Requires less initial storage redesign.
• ➖ Retains whole-document mutation and cache invalidation for tag or deprecation changes.

2. Keep metadata in object storage

• ➕ Avoids operating a SQL database.
• ➖ Requires custom coordination for uniqueness, concurrent policy changes, and durable publication outcomes.

3. Begin with a shared SQL backend

• ➕ Supports multi-instance deployments sooner.
• ➖ Commits to a backend before its isolation, freshness, and workload behavior pass conformance tests.

Recommendation: The proposed SQL/blob split is the strongest fit for transactional metadata and immutable artifacts. Starting with SQLite keeps the first implementation bounded; the shared conformance suite is an appropriate gate before choosing a multi-instance backend.

Files changed (1) +195 / -0

Documentation (1) +195 / -0

0000-sql-metadata-storage.mdDefine pnpr’s SQL metadata storage contract +195/-0

Define pnpr’s SQL metadata storage contract

• Adds a draft RFC separating authoritative SQL metadata from artifact storage and specifying relational identities, transactional backend requirements, and durable publication outcomes. It proposes generated npm views with per-version fragment caching and package-generation read-after-write guarantees, then outlines implementation, conformance testing, and unresolved decisions. No storage implementation is added.

pnpr/text/0000-sql-metadata-storage.md