Skip to content

Latest commit

 

History

History
831 lines (633 loc) · 37 KB

README.md

File metadata and controls

831 lines (633 loc) · 37 KB

PDPP Reference Implementation

This package is the forkable PDPP reference substrate in this repository.

It contains the current:

  • authorization server and resource server
  • Collection Profile runtime
  • CLI
  • reference manifests and sample connector
  • executable black-box test suite

It is not the public site or the operator console. The public site in apps/site/ explains and showcases the reference implementation; the operator console in apps/console/ fronts a running instance. The runnable implementation lives here.

What it proves today

The current provider-connect story is intentionally thin and honest. It proves:

  • standards-based discovery via RFC 9728 protected-resource metadata and RFC 8414 AS metadata
  • PAR-backed request staging through POST /oauth/par
  • public-client self-registration through POST /oauth/register
  • a consent shell and approval surface for issued grants
  • owner self-export through the device flow
  • pre-registered and dynamically registered client paths for the current third-party connect flow

It does not yet prove:

  • a full generic third-party authorization-code redirect flow
  • a broader ecosystem profile beyond the currently advertised metadata and authorization_details type

What this package is proving

The current reference is centered on one architectural claim:

  • one engine substrate can support both a native provider realization and a polyfill/connector realization
  • public source identity stays honest:
    • provider_id for native providers such as Northstar HR
    • connector_id for collected/polyfill sources such as Spotify
  • owner self-export, client grants, and reference-only traces can all be exercised against the same running system

Package layout

  • server/
    • authorization server, resource server, metadata, consent, grant issuance, introspection
  • runtime/
    • Collection Profile runner and related runtime helpers
  • cli/
    • pdpp CLI for owner login, export, provider inspection, grant staging, and trace/timeline inspection
  • manifests/
    • sample connector manifests plus the native northstar-hr manifest
  • test/
    • black-box integration, metadata, CLI, event-spine, and Collection Profile conformance tests
  • lib/
    • shared implementation helpers such as the durable event spine

Primary surfaces

Discovery

  • GET /.well-known/oauth-protected-resource
  • GET /.well-known/oauth-authorization-server

The bare AS/RS root (GET /) is content-negotiated:

  • Accept: application/json (or ?format=json, or no Accept header) returns the existing discovery-index envelope byte-for-byte. Existing JSON-shaped clients are unaffected.
  • Accept: text/html returns a small operator/admin landing page that names this server, links to the well-known discovery endpoint above, and points at the operator console origin (taken from PDPP_REFERENCE_ORIGIN, defaulting to http://localhost:3002). The landing page is reachable from reference-implementation alone and does not require apps/console to be running. See openspec/changes/split-public-site-and-operator-console.

Protected-resource metadata includes advisory agent discovery. The generated CLI command is:

npx -y @pdpp/cli connect <provider-url>

The no-owner-token completion path is prelaunch 0.x software but complete in the reference flow. Metadata sets pdpp_agent_discovery.cli.no_owner_token true when the AS can complete owner-approved scoped token handoff without asking for an owner bearer token.

Client request start

  • POST /oauth/par

Client registration

  • POST /oauth/register

Dynamic client registration is enabled by default. Public clients can self-register supported public-client metadata without an initial access token. Registration creates only a public client_id; it does not grant data access or mint bearer tokens.

Overrides:

  • PDPP_DCR_INITIAL_ACCESS_TOKENS=token1,token2 - comma-separated initial access tokens for optional operator/bootstrap registration. If a caller sends a bearer token, it must be one of these tokens; callers can omit the bearer token for public self-registration.
  • PDPP_ENABLE_DYNAMIC_CLIENT_REGISTRATION=0 - explicitly disables DCR. The AS metadata then omits registration_endpoint and advertises only pdpp_registration_modes_supported: ["pre_registered_public"].

Default pre-registered public clients seeded at startup are the product clients pdpp_cli (the published CLI), pdpp-web-dashboard (the console's operator bootstrap), dataconnect-console (the console's owner bearer) and dataconnect-connector-runtime (the connector runtime's owner bearer). The console and the connector runtime hold one owner bearer per subject: an owner device approval for either client returns the live bearer that client already has while it has at least half its lifetime left, and the runtime keeps that bearer in memory between runs. The runtime keeps the bearer in its own process: connector child processes never receive it. Each admitted run gets a random run-scoped credential (PDPP_BLOB_UPLOAD_TOKEN) that is valid only for POST /v1/blobs, only for that run's connector and connection, and only until the run ends. Connectors built before that name read the same credential as PDPP_OWNER_TOKEN. The pre-registered set is reference-local convenience; production deployments should supply their own preRegisteredPublicClients option.

Consent and grant issuance

  • GET /consent?request_uri=...
  • POST /consent/review
  • POST /consent/approve
  • POST /consent/deny

POST /consent/review returns the exact approval_review artifact and approval_review_revision. Final POST /consent/approve sends the request_uri and that revision only. Batch approval also sends confirm_reviewed_decision.

The reference AS also exposes a stable owner-entry page at GET /owner/login. It behaves as a small reference-only owner access hub:

  • when PDPP_OWNER_PASSWORD is unset, it explains that placeholder auth is disabled and points operators at the hosted device approval UI
  • when PDPP_OWNER_PASSWORD is set, it renders the owner sign-in form
  • when already signed in, it becomes a signed-in landing page with device approval and sign-out actions

The sign-in page itself names only the product and what the password protects. It does not print operator configuration. On a server, the password comes from PDPP_OWNER_PASSWORD (see "Environment variables" below). The desktop app keeps its owner password in the OS keychain and shows it under Settings > Owner password. POST /owner/logout is listed under "Stable owner-entry routes" below.

Owner self-export

  • POST /oauth/device_authorization
  • GET /device
  • POST /device/approve
  • POST /oauth/token

Resource access

  • /v1/streams/...
  • owner and client queries under the current reference contract

Expandable records

GET /v1/streams/:stream/records and GET /v1/streams/:stream/records/:id support expand=<relation> only when the stream manifest declares the relation under both relationships[] and query.expand[]. Relationship metadata is descriptive; query.expand[] is the public allowlist.

Expansion is one hop, grant-safe, and bounded. Expanded child records are authorized through the child stream grant and projected to the child fields the caller can read. has_many children use expand_limit[relation], returning a list object with data[] and has_more. Missing has_one children return null; missing has_many children return an empty list.

The first-party Gmail manifest enables messages -> message_bodies and messages -> attachments. Attachment expansion is grant-safe and field-scoped: when the child attachments grant includes a hydrated blob_ref, the resource server decorates it with blob_ref.fetch_url, and bytes are fetched only through GET /v1/blobs/{blob_id} under the same grant. Metadata-only attachment records continue to expose no bytes and no fabricated blob reference. Reverse/belongs-to expansion such as messages -> thread remains deferred; clients can still query directly by the relevant foreign key.

Filtered retrieval

GET /v1/search and GET /v1/search/semantic accept record-list compatible filter[...] parameters only when the request names exactly one streams value. Exact filters apply to authorized top-level scalar fields. Range filters must be declared by the named stream under query.range_filters, so clients should inspect stream metadata before sending filter[field][gte|gt|lte|lt].

Cross-stream filtered search, public relevance scores/reranking, and caller-controlled hybrid ranking are intentionally deferred.

Semantic retrieval

GET /v1/search/semantic is an experimental optional extension. In normal local operation (pnpm run dev or this package's dev/server scripts), the reference uses a local Transformers.js embedding backend by default:

  • backend mode: PDPP_SEMANTIC_EMBEDDING_BACKEND=local
  • default profile: PDPP_EMBEDDING_PROFILE_ID=minilm
  • default model: Xenova/all-MiniLM-L6-v2
  • dimensions / metric: 384 / cosine
  • default dtype: PDPP_EMBEDDING_DTYPE=q4
  • default cache: reference-implementation/.cache/transformers

The first semantic backfill downloads model files unless PDPP_EMBEDDING_DOWNLOAD_ALLOWED=0 is set. Tests and programmatic startServer calls keep the deterministic stub backend unless they opt into operational defaults with PDPP_REFERENCE_OPERATIONAL_DEFAULTS=1; the stub is only for deterministic exact-match assertions and does not claim paraphrase or multilingual behavior.

Operators can switch profiles without changing the public API:

  • PDPP_EMBEDDING_PROFILE_ID=minilm - compact English-biased default.
  • PDPP_EMBEDDING_PROFILE_ID=multilingual-minilm - multilingual MiniLM profile suitable for Italian-language data.
  • PDPP_EMBEDDING_MODEL_ID=... - override the Hugging Face model ID.
  • PDPP_EMBEDDING_CACHE_DIR=... - override the local model cache.
  • PDPP_SEMANTIC_EMBEDDING_BACKEND=stub|local|disabled - force a backend mode.

Changing the profile/model/dtype/dimensions/metric invalidates existing semantic vectors. The reference reports index_state: "stale" or "building" and rebuilds from stored records; it does not require connector re-ingest. Use /deployment or GET /_ref/deployment to inspect backend availability, cache state, active language bias, participating semantic fields, and warnings such as zero participation or a rebuilding index.

For reference inspection, successful and route-level rejected /v1/streams, /v1/streams/:stream, /v1/streams/:stream/records, and /v1/streams/:stream/records/:id responses also expose:

  • Request-Id
  • PDPP-Reference-Trace-Id

That pair lets a caller correlate a live query response to GET /_ref/traces/:traceId without widening the _ref surface itself.

Reference-only debugging surfaces

  • GET /_ref/traces/:traceId
  • GET /_ref/grants/:grantId/timeline
  • GET /_ref/runs/:runId/timeline
  • GET /_ref/dataset/summary

These _ref endpoints are intentionally reference-only artifacts, not core PDPP protocol requirements.

Use pdpp ref ... for CLI inspection, for example:

pdpp ref run timeline <run-id>
pdpp ref grant timeline <grant-id>
pdpp ref trace show <trace-id>

From a local repo shell, prefix those examples with pnpm exec unless you have linked or installed a pdpp binary into PATH:

pnpm exec pdpp ref run timeline <run-id>

GET /_ref/dataset/summary returns a live aggregate description of what the substrate is holding: connector count, stream count, live record count, and three separately-labeled byte totals (record_json_bytes for live payloads, record_changes_json_bytes for retained change history, blob_bytes for blobs), summed into total_retained_bytes. Counts exclude soft-deleted records. The response also carries two pairs of temporal bounds: earliest_record_time / latest_record_time are real-world timestamps mined from record payloads via each stream's manifest-declared consent_time_field, and earliest_ingested_at / latest_ingested_at are the substrate's own emitted_at bounds (when the runtime wrote each row). Plus a top-3 top_connectors list. Used by the operator-console hero band.

Time-bucketed aggregate reads are manifest-declared: a stream supports group_by_time only for fields listed in its connector manifest's query.aggregations.group_by_time array, and registration rejects entries that are not string date or date-time fields.

Reference-only owner-auth placeholder

The reference ships a minimal local-only owner-auth placeholder for the current owner/operator browser surfaces. It is not part of the PDPP protocol and is not a finished owner-auth product.

Environment variables:

  • PDPP_OWNER_PASSWORD - if set, the current owner/operator browser surfaces below require a valid owner session. If unset, the server keeps its current open local-dev behavior.
  • PDPP_OWNER_SUBJECT_ID - optional. Defaults to owner_local. When placeholder auth is enabled, this value is the owner subject id used for every approved grant and device authorization; any subject_id submitted from a form or JSON body is ignored.

Routes gated by the placeholder (when enabled):

  • GET /consent, POST /consent/review, POST /consent/approve, POST /consent/deny
  • GET /device, POST /device/approve, POST /device/deny
  • clean owner-console routes (via the composed console origin)
  • every reference-only _ref read (GET /_ref/*) and mutation (POST/PUT /_ref/*). When PDPP_OWNER_PASSWORD is unset, _ref routes preserve the open local-dev behavior. When set, callers must present an owner session - the dashboard already forwards the pdpp_owner_session cookie, and CLI callers can pass the same value via PDPP_OWNER_SESSION_COOKIE.

Stable owner-entry routes:

  • GET /owner/login - owner access page (supports a safe same-origin return_to query parameter). When placeholder auth is disabled it renders an honest disabled-state landing page; when enabled it renders either the sign-in form or a signed-in landing page.
  • POST /owner/login - when placeholder auth is enabled, submits the owner password; on success sets a signed HTTP-only session cookie (pdpp_owner_session, 7 day lifetime by default, configurable with PDPP_OWNER_SESSION_TTL_SECONDS, SameSite=Lax, Secure when served over HTTPS) and redirects to return_to
  • POST /owner/logout - clears the session cookie when present
  • GET /owner/session - body-less admission check: 204 when the request would pass the owner-session gate (a valid session, or the open local-dev posture), otherwise the same 401 / login redirect as a protected route. A server that forwards the browser's cookie but cannot validate it itself (the console in a split deployment) asks this before it uses owner authority.

Unauthenticated HTML requests to the protected routes redirect to /owner/login?return_to=...; non-HTML callers receive an honest 401 with error code owner_session_required.

The placeholder is intentionally narrow:

  • no user table, no external IdP, no multi-user auth
  • owner sessions are stored in the database and are not bound to the password. Changing PDPP_OWNER_PASSWORD and restarting leaves existing sessions and owner bearers valid; revoke them under Settings > Owner sessions. A password change in the DataConnect desktop app restarts the server and revokes every owner session and bearer. A change through /owner/password/change (the app-managed password) keeps the session that made it and revokes the others
  • public protocol surfaces (/oauth/par, /oauth/register, /oauth/token, /v1/*, /.well-known/*) are not gated
  • the placeholder is still not a durable owner-auth story; it is only the current reference-local browser/session gate

Config precedence: stored values vs. environment variables

Most config keys have exactly one precedence rule, implemented once in server/stores/config-precedence-resolver.ts (createConfigPrecedenceResolver):

An env var is read only when the config store has no value for that key and the key is not on the platform-owned list; everywhere else, once the owner has set a value through the UI, that stored value wins even if the env var is still present, and the env var is consulted again only if the stored value is cleared.

Two exceptions:

  • Platform-owned keys (PORT, AS_PORT, RS_PORT today) never consult the store at all — env always wins, so the UI can never fight the platform that injects them (Railway/PaaS PORT injection, or the core image's internal AS/RS supervisor ports).
  • Bootstrap values — PDPP_DATABASE_URL (or PDPP_STORAGE_BACKEND=sqlite
    • PDPP_DB_PATH), PDPP_OWNER_PASSWORD, PDPP_CREDENTIAL_ENCRYPTION_KEY / _FILE, and PDPP_REFERENCE_ORIGIN — can never come from the store, because each is needed before the store can be read (it names the database, decrypts it, or classifies hosted/local deployment before any store lookup) or because storing it would let store-write access grant itself store-read access. These are read directly from process.env, never through this resolver.

createDeploymentConfigResolver (in server/stores/provider-app-config-store.ts) is the shipped, tested instance of this rule for provider OAuth app config (identityGroup/logicalKey-scoped); it is implemented in terms of the generic resolver above. No provider-app-config key is platform-owned.

This PR lands the rule, its generic form, and the platform-owned exception — it does not migrate additional env vars into the store. PDPP_INSTANCE_NAME, PDPP_TRUSTED_HOSTS/PDPP_TRUSTED_PROXIES, the GOOGLE_DATAPORTABILITY_* vars, VAPID web-push keys, operational limits (PDPP_MANUAL_UPLOAD_MAX_BYTES, PDPP_RECORD_REJECTION_*_QUOTA_BYTES, PDPP_CHANGE_HISTORY_LIMIT), feature toggles (PDPP_ENABLE_DYNAMIC_CLIENT_REGISTRATION, PDPP_DCR_INITIAL_ACCESS_TOKENS, PDPP_ENABLE_STREAM_PLAYGROUND), and owner-session policy (PDPP_OWNER_SESSION_TTL_SECONDS, PDPP_OWNER_SAMESITE, PDPP_OWNER_FORCE_SECURE_COOKIES) are candidates for the same treatment, but that migration is a separate, larger, not-yet-scoped change. There is no PDPP_BIND_HOST env var in this server today — bindHost is an in-process start option, not a config-store or bootstrap candidate.

Reference-only hosted-UI layer

Server-rendered HTML pages (GET /consent, GET /device and its result pages, POST /consent/approve/deny result pages, and the stable owner-entry page at GET /owner/login) all go through a small shared hosted-UI module, server/hosted-ui.ts. That module renders the DataConnect product mark and wordmark (plus the PDPP_INSTANCE_NAME label when one is set) with the PDPP typography, reuses the data-surface="human" / data-surface="protocol" language from packages/pdpp-brand/styles/base.css, and serves a single shared stylesheet at GET /__pdpp/hosted-ui.css.

This hosted-UI layer is reference-only implementation support. It is not a PDPP protocol surface; clients and providers never need to fetch /__pdpp/hosted-ui.css or consume any of the hosted-ui-* class names. The React/Next public site in apps/site/ and operator console in apps/console/ remain the canonical app-layer design-system surfaces.

How to use it

Reference hosting modes

The reference now supports two deliberate local hosting modes:

  • direct - AS on :7662, RS on :7663; best for protocol debugging, conformance-style testing, CLI, and agents
  • composed - one browser-facing origin (default http://localhost:3002) proxying the internal AS/RS; best for the dashboard, owner flows, and demos

The shared topology inputs are:

  • PDPP_REFERENCE_MODE=direct|composed
  • PDPP_REFERENCE_ORIGIN=http://... for the browser-facing origin in composed mode

Legacy AS_PUBLIC_URL / RS_PUBLIC_URL overrides still work, but the preferred local/product path is the explicit composed-mode pair above.

Same-origin local reference composition

The preferred local reference-product entrypoint is now the composed browser origin at http://localhost:3002.

Run the full local stack from the repo root:

pnpm dev

In that mode:

  • the Next app serves the browser-facing origin on http://localhost:3002
  • the internal AS/RS still listen on :7662 / :7663
  • the browser-facing origin proxies the reference namespaces:
    • /.well-known/*
    • /oauth/*
    • /v1/*
    • /_ref/*
    • /owner/*
    • /device
    • /consent
    • /__pdpp/hosted-ui.css
  • when PDPP_OWNER_PASSWORD is set, /owner/* and the clean owner-console routes share the same pdpp_owner_session cookie on the browser-facing origin

If you only need the backing AS/RS side of that composed setup while the web app is already running, start this package in composition mode:

pnpm --dir reference-implementation dev

That mode sets PDPP_REFERENCE_MODE=composed and defaults PDPP_REFERENCE_ORIGIN to http://localhost:3002, so the internal AS/RS still listen on :7662/:7663 while advertising the browser-facing origin in metadata, device verification URLs, and PAR authorization URLs.

Standalone reference server

The reference server imports built output from the @pdpp/connector-protocol and @pdpp/collector-runtime workspace packages (packages/connector-protocol, packages/collector-runtime). A root-level npm install builds both automatically via their prepare scripts — if you ever see ERR_MODULE_NOT_FOUND for one of those packages' dist/ files (for example after npm install --workspace=<something> instead of a plain npm install at the repo root), rebuild them directly:

npm run build --workspace=packages/connector-protocol --workspace=packages/collector-runtime

Run the server:

pnpm --dir reference-implementation run server

That starts the AS/RS directly on their own listen ports (:7662 / :7663) without the composed browser-facing console origin.

If you need to force direct mode while other composition-oriented env is set in your shell, run:

PDPP_REFERENCE_MODE=direct pnpm --dir reference-implementation run server

Inspect the CLI:

pnpm --dir reference-implementation cli --help

Run the test suite:

pnpm --dir reference-implementation test

Verify the generated contract artifacts are current:

pnpm reference-contract:check-generated

Docker Compose reference stack

For an end-to-end operator runbook (Docker host or RunPod CPU Pod, env vars, dashboard verification, MCP wiring), see deploy/docker/README.md.

The notes below are the topology reference; the quickstart is the procedure.

The supported Docker path is a root-level Compose assembly for the live reference stack. The quickest self-hosted path uses the public GHCR images:

cp .env.docker.example .env.docker
# Set PDPP_OWNER_PASSWORD and PDPP_CREDENTIAL_ENCRYPTION_KEY in .env.docker
# set PDPP_REFERENCE_ORIGIN in .env.docker to your deployment URL
docker compose --env-file .env.docker pull
pnpm docker:reference:quick

Open http://localhost:${PDPP_WEB_PORT:-3002} for the browser-facing reference origin. The Compose stack runs:

  • reference - one AS/RS process, AS on :7662, RS on :7663
  • web - the Next app on container :3000, mapped to host ${PDPP_WEB_PORT:-3002} by default, proxying the AS/RS in composed mode

To test the owner-present n.eko interaction-streaming backend, use the reference-stack wrapper. It always includes the n.eko Compose overlay and the neko-dynamic profile, then verifies that the running reference container received the managed browser-surface configuration:

pnpm docker:neko

or, for the same stack without rebuilding images:

pnpm docker:reference:quick

Then open http://localhost:${PDPP_WEB_PORT:-3002}/stream-playground?backend=neko. The overlay runs n.eko in the reference container's network namespace, so the reference process uses the loopback http://127.0.0.1:8080/neko target. The browser still uses the same public origin through the reference /neko/* proxy, including WebSocket upgrade traffic.

The overlay builds a thin local image on top of the pinned upstream n.eko Chromium image. That layer only overrides the Chromium launcher/supervisor entry so Chromium runs with the container-required --no-sandbox flag and the same /neko path-prefix health shape used by the proxy.

The playground is disabled in production-mode builds unless PDPP_ENABLE_STREAM_PLAYGROUND=1 is set. The n.eko Compose overlay sets that flag for both web and reference; the default Compose stack does not. The overlay also sets PDPP_NEKO_PROXY_AUTOLOGIN=1, which lets the token-scoped entry route pass dummy noauth usr/pwd query params to n.eko so the owner lands directly in the WebRTC control surface instead of a sidecar login form.

For phone/LAN testing, set NEKO_WEBRTC_NAT1TO1 in .env.docker to the host IP the device can reach and allow NEKO_WEBRTC_PORT over both TCP and UDP. The HTTPS route and /neko/* proxy cover page/signaling traffic; WebRTC media still needs reachable ICE candidates. For public-device or LTE testing where direct candidates may fail, start the authenticated coturn fallback:

pnpm docker:neko:turn

Set TURN_PUBLIC_IP and NEKO_WEBRTC_ICESERVERS in .env.docker to the public address and TURN credentials for that host. TURN is a relay fallback and adds bandwidth/latency only when WebRTC selects it; direct LAN/public candidates remain the preferred path when they work.

Default public images:

  • ghcr.io/pdp-connect/pdpp/reference:main
  • ghcr.io/pdp-connect/pdpp/web:main

The main tag is a moving default-branch build. Stable semantic-release images are published as exact version tags such as 1.2.3, moving minor-series tags such as 1.2, and latest. For durable self-hosting, pin an exact version, sha-* tag, or digest in .env.docker:

PDPP_REFERENCE_IMAGE=ghcr.io/pdp-connect/pdpp/reference:1.2.3
PDPP_WEB_IMAGE=ghcr.io/pdp-connect/pdpp/web:1.2.3

The important topology rule is that PDPP_REFERENCE_ORIGIN is what the browser uses, while PDPP_AS_URL and PDPP_RS_URL are container-internal service URLs. The default Compose values are:

PDPP_REFERENCE_ORIGIN=http://localhost:3002
PDPP_AS_URL=http://reference:7662
PDPP_RS_URL=http://reference:7663

Do not use localhost for cross-container AS/RS calls; inside Docker, localhost means the current container.

Persistent mounts are defined for:

  • packages/polyfill-connectors/.pdpp-data/ bind-mounted to /var/lib/pdpp, with PDPP_DB_PATH at /var/lib/pdpp/pdpp.sqlite
  • PDPP_EMBEDDING_CACHE_DIR at /var/cache/pdpp/transformers

Connector catalog and air-gapped preload

The reference Docker image does not bundle @pdpp/polyfill-connectors or a default connector source list. A new instance starts with an empty installed connector state. The owner console reads the verified catalog at GET /v1/owner/connector-install/catalog; an owner can install a selected digest, after which the server activates the verified immutable root and adds its manifest to the source catalog.

For an air-gapped deployment, set PDPP_CONNECTOR_PRELOAD_DIR to a mounted directory containing connector-install state prepared by the normal verified installer. The directory uses the install module's layout:

$PDPP_CONNECTOR_PRELOAD_DIR/
├── connector-install-state.json
└── connectors/<connector-id>/<sha256:digest>/
    ├── profile/collection-profile.json
    ├── dist/collection-profile.mjs
    └── provenance.json

The state file must contain the installer-produced hashes, provenance, digest, and entrypoint metadata. Do not hand-edit it or copy an unverified connector tree. The preload directory is used as the file-backed install store, so it must be writable when the service updates catalog high-water state; mount it read-only only when no catalog refresh or install operation is needed.

Durable connector artifacts — the Slack workspace archive, downloaded statement PDFs — resolve under PDPP_CONNECTOR_ARTIFACT_ROOT, which Core pins to /var/lib/pdpp/connector-artifacts. They are on the volume above; no separate mount is needed. Outside a container, with neither that variable nor PDPP_DB_PATH set, they fall back to ~/.pdpp/connector-artifacts and each run says so in its log. See packages/polyfill-connectors/src/connector-artifact-root.ts.

The first boot may download the default MiniLM model into the embedding cache. Set PDPP_EMBEDDING_DOWNLOAD_ALLOWED=0 if you want to avoid that and accept the corresponding deployment diagnostic warning until the cache is prewarmed.

Secrets must be supplied at runtime through environment variables, .env.docker, or Docker secrets. Do not bake PDPP_OWNER_PASSWORD, connector passwords, tokens, cookies, or DCR initial access tokens into images. The repo-root .env.local remains a local development convenience, not a Docker or production posture.

SQLite encryption boundary

The managed DataConnect desktop stack always supplies a database key from the OS keychain as PDPP_DATABASE_ENCRYPTION_KEY to the RI child at spawn time. Desktop SQLite storage is encrypted at rest with no user setting or opt-out. Existing plaintext desktop vaults are migrated on startup with a verified backup and rollback path.

Self-hosted and Docker deployments intentionally remain plaintext when PDPP_DATABASE_ENCRYPTION_KEY is absent. They have no desktop OS keychain, and their database-key lifecycle is not defined by this release. Do not treat the absence of the variable as permission for a desktop install to mint a replacement key: an encrypted vault fails closed until its original key is restored.

Browser-based polyfill connectors are not clean-room portable demos. They need persistent browser profiles and remain subject to upstream anti-bot behavior. Mount any optional local connector inputs, such as Slack archives, explicitly when testing those connectors.

Slack imports require slackdump v4.4.2 (AGPL-3.0). The core, core-browser, railway-core, and platform-core Docker images ship slackdump bundled by default. Source tree and license: bundled at /usr/local/share/slackdump/ in the image (LICENSE.agpl-3.0.txt, SOURCE_URL, SLACKDUMP_BIN=/usr/local/bin/slackdump).

To override with a different slackdump build or version, set SLACKDUMP_BIN to point at an alternative executable on PATH or an in-container path. For Docker runs, you can mount a host directory via PDPP_DOCKER_SLACKDUMP_DIR in .env.docker:

PDPP_DOCKER_SLACKDUMP_DIR=/home/user/go/bin
SLACKDUMP_BIN=/opt/pdpp-tools/slackdump/slackdump

Alternatively, place or symlink an alternative executable at packages/polyfill-connectors/.pdpp-tools/slackdump/slackdump; that local runtime directory is gitignored and is the default compose mount source.

Update public-image deployments by pulling newer images and restarting the stack. This keeps the persisted SQLite DB, embedding cache, and browser profile volumes in place:

docker compose --env-file .env.docker pull
pnpm docker:reference:quick

To build from the current local checkout instead of pulling public images:

docker build -f deploy/docker/Dockerfile .

pnpm docker:reference:up

The .git directory is excluded from the Docker build context, so the reference image cannot derive a real revision at startup and falls back to pdpp-reference@<package-version>+unknown in the PDPP-Reference-Revision response header and the GET / discovery index. Pass the running commit through the PDPP_REFERENCE_REVISION build arg so production images publish a real revision. Use the full SHA, not an abbreviated form — the core/core-browser targets' identity gate (deploy/docker/check-image-identity.sh) requires a full 40- or 64-character hex value and rejects abbreviated SHAs as not shaped like a real git object id; using the full SHA here too keeps every PDPP_REFERENCE_REVISION invocation in this repo consistent, even though this reference target itself carries no OCI revision label and is not gated:

docker build \
  --build-arg PDPP_REFERENCE_REVISION="$(git rev-parse HEAD)" \
  --target reference \
  -t pdpp-reference:local .

The runtime continues to honor the PDPP_REFERENCE_REVISION env var, so the revision can also be set or overridden at container start time.

Run the smoke validation:

pnpm docker:smoke

The smoke check builds the stack, verifies AS/RS metadata through the composed browser origin, checks that public metadata does not leak Docker service names, and confirms / redirects to /owner/login when owner auth is configured.

Scheduler startup uses the same reference-implementation/server/index.ts entrypoint in Docker as local long-lived startup. To smoke enabled schedule execution in Compose, seed a connector and enabled connector_schedules row in the mounted reference database, restart only the reference service, then confirm a new scheduler_run_history row appears for that connector without pressing Run now. Disabled or deleted rows should not add history after restart.

For Docker-based development with hot reload, run:

pnpm docker:dev

That overlays docker-compose.dev.yml, bind-mounts the repo into the containers, runs the reference server with Node watch mode, and runs the web app with Next dev. Dependency folders stay in Docker named volumes so host installs and container installs do not trample each other.

Docker dev loads repo-root .env.local in the reference container so local connector credentials such as GITHUB_PERSONAL_ACCESS_TOKEN are available to controller-managed connector runs. Public-image and default Compose operation do not load .env.local; use .env.docker, environment variables, or Docker secrets there.

When accessing Docker dev through a LAN IP or reverse proxy, add the browser hostnames to PDPP_WEB_ALLOWED_DEV_ORIGINS in .env.docker, for example:

PDPP_WEB_ALLOWED_DEV_ORIGINS=pdpp-dev.example.com,192.0.2.10

Reverse proxies must also forward WebSocket upgrade traffic for /_next/webpack-hmr; otherwise the page loads but Next HMR cannot connect.

CI builds the Docker targets on pull requests without pushing images. On main, semantic-release creates GitHub releases from Conventional Commits and the same release workflow publishes the npm packages (@pdpp/cli, @pdpp/local-collector, and @pdpp/mcp-server) plus stable GHCR tags for both Docker targets: ${version}, ${major}.${minor}, latest, and sha-*. Maintainers should make the first published GHCR packages public in GitHub's package settings if the registry creates them private.

Maintainers can preview the next release calculation locally:

GITHUB_TOKEN=$(gh auth token) pnpm release:dry-run

The release path uses GitHub Actions credentials for GitHub releases and GHCR, and npm trusted publishing for public packages. It must not bundle .env.local, owner passwords, connector credentials, SQLite data, model cache files, or browser profiles into images.

Example third-party client app

A minimal example app illustrates the current thin reference provider-connect flow end to end (register → PAR → owner approval → token → RS query). It lives at examples/third-party-app/ and runs on its own port (default 7674) separate from the AS/RS:

pnpm --dir reference-implementation example-client

Defaults: PORT=7674, AS_URL=http://localhost:7662, RS_URL=http://localhost:7663.

The example uses the two-phase approval flow:

  • the example calls POST /consent/review, inspects the exact approval artifact, and then calls POST /consent/approve with its revision
  • when PDPP_OWNER_PASSWORD is set, the inline shortcut is refused by the reference server. The example links out to the hosted /consent page.

The example is a third-party client illustration - it is not a full generic OAuth authorization-code redirect client. It has no PKCE, no /callback, and no code exchange. It only exercises the endpoints the reference currently advertises.

Published generated artifacts

The reference publishes generated machine-readable and human-readable contract artifacts derived from @pdpp/reference-contract. These are treated as versioned reference outputs, not throwaway build noise.

  • openapi/reference-public.openapi.json
    • public PDPP JSON APIs only
  • openapi/reference-full.openapi.json
    • public APIs plus reference-only /_ref operator surfaces
  • docs/generated/reference-routes.md
    • generated route index for public APIs
  • docs/generated/reference-ref-routes.md
    • generated route index for reference-only APIs
  • docs/generated/query-cookbook.md
    • generated query and flow cookbook

Regenerate them with:

pnpm reference-contract:generate

Relationship to the root PDPP specs

The root spec-*.md files remain normative for PDPP protocol semantics.

This package is the executable reference implementation:

  • use the root specs to understand what PDPP means
  • use this package to see what the current reference actually does
  • use OpenSpec to understand project-level architecture and active implementation changes