This package is the forkable PDPP reference substrate in this repository.
It contains the current:
- authorization server and resource server
- Collection Profile runtime
- CLI
- reference manifests and sample connector
- executable black-box test suite
It is not the public site or the operator console. The public site in
apps/site/ explains and showcases the reference implementation; the operator
console in apps/console/ fronts a running instance. The runnable
implementation lives here.
The current provider-connect story is intentionally thin and honest. It proves:
- standards-based discovery via RFC 9728 protected-resource metadata and RFC 8414 AS metadata
- PAR-backed request staging through
POST /oauth/par - public-client self-registration through
POST /oauth/register - a consent shell and approval surface for issued grants
- owner self-export through the device flow
- pre-registered and dynamically registered client paths for the current third-party connect flow
It does not yet prove:
- a full generic third-party authorization-code redirect flow
- a broader ecosystem profile beyond the currently advertised metadata and
authorization_detailstype
The current reference is centered on one architectural claim:
- one engine substrate can support both a native provider realization and a polyfill/connector realization
- public source identity stays honest:
provider_idfor native providers such asNorthstar HRconnector_idfor collected/polyfill sources such as Spotify
- owner self-export, client grants, and reference-only traces can all be exercised against the same running system
server/- authorization server, resource server, metadata, consent, grant issuance, introspection
runtime/- Collection Profile runner and related runtime helpers
cli/pdppCLI for owner login, export, provider inspection, grant staging, and trace/timeline inspection
manifests/- sample connector manifests plus the native
northstar-hrmanifest
- sample connector manifests plus the native
test/- black-box integration, metadata, CLI, event-spine, and Collection Profile conformance tests
lib/- shared implementation helpers such as the durable event spine
GET /.well-known/oauth-protected-resourceGET /.well-known/oauth-authorization-server
The bare AS/RS root (GET /) is content-negotiated:
Accept: application/json(or?format=json, or noAcceptheader) returns the existing discovery-index envelope byte-for-byte. Existing JSON-shaped clients are unaffected.Accept: text/htmlreturns a small operator/admin landing page that names this server, links to the well-known discovery endpoint above, and points at the operator console origin (taken fromPDPP_REFERENCE_ORIGIN, defaulting tohttp://localhost:3002). The landing page is reachable fromreference-implementationalone and does not requireapps/consoleto be running. Seeopenspec/changes/split-public-site-and-operator-console.
Protected-resource metadata includes advisory agent discovery. The generated CLI command is:
npx -y @pdpp/cli connect <provider-url>The no-owner-token completion path is prelaunch 0.x software but complete in
the reference flow.
Metadata sets pdpp_agent_discovery.cli.no_owner_token true when the AS can
complete owner-approved scoped token handoff without asking for an owner bearer
token.
POST /oauth/par
POST /oauth/register
Dynamic client registration is enabled by default. Public clients can
self-register supported public-client metadata without an initial access token.
Registration creates only a public client_id; it does not grant data access or
mint bearer tokens.
Overrides:
PDPP_DCR_INITIAL_ACCESS_TOKENS=token1,token2- comma-separated initial access tokens for optional operator/bootstrap registration. If a caller sends a bearer token, it must be one of these tokens; callers can omit the bearer token for public self-registration.PDPP_ENABLE_DYNAMIC_CLIENT_REGISTRATION=0- explicitly disables DCR. The AS metadata then omitsregistration_endpointand advertises onlypdpp_registration_modes_supported: ["pre_registered_public"].
Default pre-registered public clients seeded at startup are the product
clients pdpp_cli (the published CLI), pdpp-web-dashboard (the console's
operator bootstrap), dataconnect-console (the console's owner bearer) and
dataconnect-connector-runtime (the connector runtime's owner bearer). The
console and the connector runtime hold one owner bearer per subject: an owner
device approval for either client returns the live bearer that client already
has while it has at least half its lifetime left, and the runtime keeps that
bearer in memory between runs. The
runtime keeps the bearer in its own process: connector child processes never
receive it. Each admitted run gets a random run-scoped credential
(PDPP_BLOB_UPLOAD_TOKEN) that is valid only for POST /v1/blobs, only for
that run's connector and connection, and only until the run ends. Connectors
built before that name read the same credential as PDPP_OWNER_TOKEN. The pre-registered set
is reference-local convenience; production deployments should supply their
own preRegisteredPublicClients option.
GET /consent?request_uri=...POST /consent/reviewPOST /consent/approvePOST /consent/deny
POST /consent/review returns the exact approval_review artifact and
approval_review_revision. Final POST /consent/approve sends the
request_uri and that revision only. Batch approval also sends
confirm_reviewed_decision.
The reference AS also exposes a stable owner-entry page at GET /owner/login.
It behaves as a small reference-only owner access hub:
- when
PDPP_OWNER_PASSWORDis unset, it explains that placeholder auth is disabled and points operators at the hosted device approval UI - when
PDPP_OWNER_PASSWORDis set, it renders the owner sign-in form - when already signed in, it becomes a signed-in landing page with device approval and sign-out actions
The sign-in page itself names only the product and what the password
protects. It does not print operator configuration. On a server, the
password comes from PDPP_OWNER_PASSWORD (see "Environment variables"
below). The desktop app keeps its owner password in the OS keychain and shows
it under Settings > Owner password. POST /owner/logout is listed under
"Stable owner-entry routes" below.
POST /oauth/device_authorizationGET /devicePOST /device/approvePOST /oauth/token
/v1/streams/...- owner and client queries under the current reference contract
GET /v1/streams/:stream/records and
GET /v1/streams/:stream/records/:id support expand=<relation> only when
the stream manifest declares the relation under both relationships[] and
query.expand[]. Relationship metadata is descriptive; query.expand[] is
the public allowlist.
Expansion is one hop, grant-safe, and bounded. Expanded child records are
authorized through the child stream grant and projected to the child fields the
caller can read. has_many children use expand_limit[relation], returning a
list object with data[] and has_more. Missing has_one children return
null; missing has_many children return an empty list.
The first-party Gmail manifest enables messages -> message_bodies and
messages -> attachments. Attachment expansion is grant-safe and field-scoped:
when the child attachments grant includes a hydrated blob_ref, the resource
server decorates it with blob_ref.fetch_url, and bytes are fetched only through
GET /v1/blobs/{blob_id} under the same grant. Metadata-only attachment records
continue to expose no bytes and no fabricated blob reference. Reverse/belongs-to
expansion such as messages -> thread remains deferred; clients can still query
directly by the relevant foreign key.
GET /v1/search and GET /v1/search/semantic accept record-list compatible
filter[...] parameters only when the request names exactly one streams
value. Exact filters apply to authorized top-level scalar fields. Range filters
must be declared by the named stream under query.range_filters, so clients
should inspect stream metadata before sending filter[field][gte|gt|lte|lt].
Cross-stream filtered search, public relevance scores/reranking, and caller-controlled hybrid ranking are intentionally deferred.
GET /v1/search/semantic is an experimental optional extension. In normal
local operation (pnpm run dev or this package's dev/server scripts), the
reference uses a local Transformers.js embedding backend by default:
- backend mode:
PDPP_SEMANTIC_EMBEDDING_BACKEND=local - default profile:
PDPP_EMBEDDING_PROFILE_ID=minilm - default model:
Xenova/all-MiniLM-L6-v2 - dimensions / metric:
384/cosine - default dtype:
PDPP_EMBEDDING_DTYPE=q4 - default cache:
reference-implementation/.cache/transformers
The first semantic backfill downloads model files unless
PDPP_EMBEDDING_DOWNLOAD_ALLOWED=0 is set. Tests and programmatic startServer
calls keep the deterministic stub backend unless they opt into operational
defaults with PDPP_REFERENCE_OPERATIONAL_DEFAULTS=1; the stub is only for
deterministic exact-match assertions and does not claim paraphrase or
multilingual behavior.
Operators can switch profiles without changing the public API:
PDPP_EMBEDDING_PROFILE_ID=minilm- compact English-biased default.PDPP_EMBEDDING_PROFILE_ID=multilingual-minilm- multilingual MiniLM profile suitable for Italian-language data.PDPP_EMBEDDING_MODEL_ID=...- override the Hugging Face model ID.PDPP_EMBEDDING_CACHE_DIR=...- override the local model cache.PDPP_SEMANTIC_EMBEDDING_BACKEND=stub|local|disabled- force a backend mode.
Changing the profile/model/dtype/dimensions/metric invalidates existing
semantic vectors. The reference reports index_state: "stale" or "building"
and rebuilds from stored records; it does not require connector re-ingest.
Use /deployment or GET /_ref/deployment to inspect backend
availability, cache state, active language bias, participating semantic fields,
and warnings such as zero participation or a rebuilding index.
For reference inspection, successful and route-level rejected /v1/streams, /v1/streams/:stream, /v1/streams/:stream/records, and /v1/streams/:stream/records/:id responses also expose:
Request-IdPDPP-Reference-Trace-Id
That pair lets a caller correlate a live query response to GET /_ref/traces/:traceId without widening the _ref surface itself.
GET /_ref/traces/:traceIdGET /_ref/grants/:grantId/timelineGET /_ref/runs/:runId/timelineGET /_ref/dataset/summary
These _ref endpoints are intentionally reference-only artifacts, not core PDPP protocol requirements.
Use pdpp ref ... for CLI inspection, for example:
pdpp ref run timeline <run-id>
pdpp ref grant timeline <grant-id>
pdpp ref trace show <trace-id>From a local repo shell, prefix those examples with pnpm exec unless you have
linked or installed a pdpp binary into PATH:
pnpm exec pdpp ref run timeline <run-id>GET /_ref/dataset/summary returns a live aggregate description of what the
substrate is holding: connector count, stream count, live record count, and
three separately-labeled byte totals (record_json_bytes for live payloads,
record_changes_json_bytes for retained change history, blob_bytes for
blobs), summed into total_retained_bytes. Counts exclude soft-deleted
records. The response also carries two pairs of temporal bounds:
earliest_record_time / latest_record_time are real-world timestamps
mined from record payloads via each stream's manifest-declared
consent_time_field, and earliest_ingested_at / latest_ingested_at are
the substrate's own emitted_at bounds (when the runtime wrote each row).
Plus a top-3 top_connectors list. Used by the operator-console hero band.
Time-bucketed aggregate reads are manifest-declared: a stream supports
group_by_time only for fields listed in its connector manifest's
query.aggregations.group_by_time array, and registration rejects entries that
are not string date or date-time fields.
The reference ships a minimal local-only owner-auth placeholder for the current owner/operator browser surfaces. It is not part of the PDPP protocol and is not a finished owner-auth product.
Environment variables:
PDPP_OWNER_PASSWORD- if set, the current owner/operator browser surfaces below require a valid owner session. If unset, the server keeps its current open local-dev behavior.PDPP_OWNER_SUBJECT_ID- optional. Defaults toowner_local. When placeholder auth is enabled, this value is the owner subject id used for every approved grant and device authorization; anysubject_idsubmitted from a form or JSON body is ignored.
Routes gated by the placeholder (when enabled):
GET /consent,POST /consent/review,POST /consent/approve,POST /consent/denyGET /device,POST /device/approve,POST /device/deny- clean owner-console routes (via the composed console origin)
- every reference-only
_refread (GET /_ref/*) and mutation (POST/PUT /_ref/*). WhenPDPP_OWNER_PASSWORDis unset,_refroutes preserve the open local-dev behavior. When set, callers must present an owner session - the dashboard already forwards thepdpp_owner_sessioncookie, and CLI callers can pass the same value viaPDPP_OWNER_SESSION_COOKIE.
Stable owner-entry routes:
GET /owner/login- owner access page (supports a safe same-originreturn_toquery parameter). When placeholder auth is disabled it renders an honest disabled-state landing page; when enabled it renders either the sign-in form or a signed-in landing page.POST /owner/login- when placeholder auth is enabled, submits the owner password; on success sets a signed HTTP-only session cookie (pdpp_owner_session, 7 day lifetime by default, configurable withPDPP_OWNER_SESSION_TTL_SECONDS,SameSite=Lax,Securewhen served over HTTPS) and redirects toreturn_toPOST /owner/logout- clears the session cookie when presentGET /owner/session- body-less admission check:204when the request would pass the owner-session gate (a valid session, or the open local-dev posture), otherwise the same401/ login redirect as a protected route. A server that forwards the browser's cookie but cannot validate it itself (the console in a split deployment) asks this before it uses owner authority.
Unauthenticated HTML requests to the protected routes redirect to /owner/login?return_to=...; non-HTML callers receive an honest 401 with error code owner_session_required.
The placeholder is intentionally narrow:
- no user table, no external IdP, no multi-user auth
- owner sessions are stored in the database and are not bound to the password. Changing
PDPP_OWNER_PASSWORDand restarting leaves existing sessions and owner bearers valid; revoke them under Settings > Owner sessions. A password change in the DataConnect desktop app restarts the server and revokes every owner session and bearer. A change through/owner/password/change(the app-managed password) keeps the session that made it and revokes the others - public protocol surfaces (
/oauth/par,/oauth/register,/oauth/token,/v1/*,/.well-known/*) are not gated - the placeholder is still not a durable owner-auth story; it is only the current reference-local browser/session gate
Most config keys have exactly one precedence rule, implemented once in
server/stores/config-precedence-resolver.ts
(createConfigPrecedenceResolver):
An env var is read only when the config store has no value for that key and the key is not on the platform-owned list; everywhere else, once the owner has set a value through the UI, that stored value wins even if the env var is still present, and the env var is consulted again only if the stored value is cleared.
Two exceptions:
- Platform-owned keys (
PORT,AS_PORT,RS_PORTtoday) never consult the store at all — env always wins, so the UI can never fight the platform that injects them (Railway/PaaSPORTinjection, or thecoreimage's internal AS/RS supervisor ports). - Bootstrap values —
PDPP_DATABASE_URL(orPDPP_STORAGE_BACKEND=sqlitePDPP_DB_PATH),PDPP_OWNER_PASSWORD,PDPP_CREDENTIAL_ENCRYPTION_KEY/_FILE, andPDPP_REFERENCE_ORIGIN— can never come from the store, because each is needed before the store can be read (it names the database, decrypts it, or classifies hosted/local deployment before any store lookup) or because storing it would let store-write access grant itself store-read access. These are read directly fromprocess.env, never through this resolver.
createDeploymentConfigResolver (in
server/stores/provider-app-config-store.ts)
is the shipped, tested instance of this rule for provider OAuth app config
(identityGroup/logicalKey-scoped); it is implemented in terms of the
generic resolver above. No provider-app-config key is platform-owned.
This PR lands the rule, its generic form, and the platform-owned exception —
it does not migrate additional env vars into the store. PDPP_INSTANCE_NAME,
PDPP_TRUSTED_HOSTS/PDPP_TRUSTED_PROXIES, the GOOGLE_DATAPORTABILITY_*
vars, VAPID web-push keys, operational limits
(PDPP_MANUAL_UPLOAD_MAX_BYTES, PDPP_RECORD_REJECTION_*_QUOTA_BYTES,
PDPP_CHANGE_HISTORY_LIMIT), feature toggles
(PDPP_ENABLE_DYNAMIC_CLIENT_REGISTRATION, PDPP_DCR_INITIAL_ACCESS_TOKENS,
PDPP_ENABLE_STREAM_PLAYGROUND), and owner-session policy
(PDPP_OWNER_SESSION_TTL_SECONDS, PDPP_OWNER_SAMESITE,
PDPP_OWNER_FORCE_SECURE_COOKIES) are candidates for the same treatment, but
that migration is a separate, larger, not-yet-scoped change. There is no
PDPP_BIND_HOST env var in this server today — bindHost is an in-process
start option, not a config-store or bootstrap candidate.
Server-rendered HTML pages (GET /consent, GET /device and its result pages, POST /consent/approve/deny result pages, and the stable owner-entry page at GET /owner/login) all go through a small shared hosted-UI module, server/hosted-ui.ts. That module renders the DataConnect product mark and wordmark (plus the PDPP_INSTANCE_NAME label when one is set) with the PDPP typography, reuses the data-surface="human" / data-surface="protocol" language from packages/pdpp-brand/styles/base.css, and serves a single shared stylesheet at GET /__pdpp/hosted-ui.css.
This hosted-UI layer is reference-only implementation support. It is not a PDPP protocol surface; clients and providers never need to fetch /__pdpp/hosted-ui.css or consume any of the hosted-ui-* class names. The React/Next public site in apps/site/ and operator console in apps/console/ remain the canonical app-layer design-system surfaces.
The reference now supports two deliberate local hosting modes:
direct- AS on:7662, RS on:7663; best for protocol debugging, conformance-style testing, CLI, and agentscomposed- one browser-facing origin (defaulthttp://localhost:3002) proxying the internal AS/RS; best for the dashboard, owner flows, and demos
The shared topology inputs are:
PDPP_REFERENCE_MODE=direct|composedPDPP_REFERENCE_ORIGIN=http://...for the browser-facing origin in composed mode
Legacy AS_PUBLIC_URL / RS_PUBLIC_URL overrides still work, but the preferred local/product path is the explicit composed-mode pair above.
The preferred local reference-product entrypoint is now the composed browser
origin at http://localhost:3002.
Run the full local stack from the repo root:
pnpm devIn that mode:
- the Next app serves the browser-facing origin on
http://localhost:3002 - the internal AS/RS still listen on
:7662/:7663 - the browser-facing origin proxies the reference namespaces:
/.well-known/*/oauth/*/v1/*/_ref/*/owner/*/device/consent/__pdpp/hosted-ui.css
- when
PDPP_OWNER_PASSWORDis set,/owner/*and the clean owner-console routes share the samepdpp_owner_sessioncookie on the browser-facing origin
If you only need the backing AS/RS side of that composed setup while the web app is already running, start this package in composition mode:
pnpm --dir reference-implementation devThat mode sets PDPP_REFERENCE_MODE=composed and defaults
PDPP_REFERENCE_ORIGIN to http://localhost:3002, so the internal AS/RS
still listen on :7662/:7663 while advertising the browser-facing origin in
metadata, device verification URLs, and PAR authorization URLs.
The reference server imports built output from the @pdpp/connector-protocol
and @pdpp/collector-runtime workspace packages
(packages/connector-protocol, packages/collector-runtime). A root-level
npm install builds both automatically via their prepare scripts — if you
ever see ERR_MODULE_NOT_FOUND for one of those packages' dist/ files (for
example after npm install --workspace=<something> instead of a plain
npm install at the repo root), rebuild them directly:
npm run build --workspace=packages/connector-protocol --workspace=packages/collector-runtimeRun the server:
pnpm --dir reference-implementation run serverThat starts the AS/RS directly on their own listen ports (:7662 / :7663)
without the composed browser-facing console origin.
If you need to force direct mode while other composition-oriented env is set in your shell, run:
PDPP_REFERENCE_MODE=direct pnpm --dir reference-implementation run serverInspect the CLI:
pnpm --dir reference-implementation cli --helpRun the test suite:
pnpm --dir reference-implementation testVerify the generated contract artifacts are current:
pnpm reference-contract:check-generatedFor an end-to-end operator runbook (Docker host or RunPod CPU Pod, env vars,
dashboard verification, MCP wiring), see
deploy/docker/README.md.
The notes below are the topology reference; the quickstart is the procedure.
The supported Docker path is a root-level Compose assembly for the live reference stack. The quickest self-hosted path uses the public GHCR images:
cp .env.docker.example .env.docker
# Set PDPP_OWNER_PASSWORD and PDPP_CREDENTIAL_ENCRYPTION_KEY in .env.docker
# set PDPP_REFERENCE_ORIGIN in .env.docker to your deployment URL
docker compose --env-file .env.docker pull
pnpm docker:reference:quickOpen http://localhost:${PDPP_WEB_PORT:-3002} for the browser-facing reference origin. The
Compose stack runs:
reference- one AS/RS process, AS on:7662, RS on:7663web- the Next app on container:3000, mapped to host${PDPP_WEB_PORT:-3002}by default, proxying the AS/RS in composed mode
To test the owner-present n.eko interaction-streaming backend, use the
reference-stack wrapper. It always includes the n.eko Compose overlay and the
neko-dynamic profile, then verifies that the running reference container
received the managed browser-surface configuration:
pnpm docker:nekoor, for the same stack without rebuilding images:
pnpm docker:reference:quickThen open http://localhost:${PDPP_WEB_PORT:-3002}/stream-playground?backend=neko.
The overlay runs n.eko in the reference container's network namespace, so the
reference process uses the loopback http://127.0.0.1:8080/neko target. The
browser still uses the same public origin through the reference /neko/*
proxy, including WebSocket upgrade traffic.
The overlay builds a thin local image on top of the pinned upstream n.eko
Chromium image. That layer only overrides the Chromium launcher/supervisor
entry so Chromium runs with the container-required --no-sandbox flag and the
same /neko path-prefix health shape used by the proxy.
The playground is disabled in production-mode builds unless
PDPP_ENABLE_STREAM_PLAYGROUND=1 is set. The n.eko Compose overlay sets that
flag for both web and reference; the default Compose stack does not. The
overlay also sets PDPP_NEKO_PROXY_AUTOLOGIN=1, which lets the token-scoped
entry route pass dummy noauth usr/pwd query params to n.eko so the owner
lands directly in the WebRTC control surface instead of a sidecar login form.
For phone/LAN testing, set NEKO_WEBRTC_NAT1TO1 in .env.docker to the host
IP the device can reach and allow NEKO_WEBRTC_PORT over both TCP and UDP.
The HTTPS route and /neko/* proxy cover page/signaling traffic; WebRTC media
still needs reachable ICE candidates. For public-device or LTE testing where
direct candidates may fail, start the authenticated coturn fallback:
pnpm docker:neko:turnSet TURN_PUBLIC_IP and NEKO_WEBRTC_ICESERVERS in .env.docker to the
public address and TURN credentials for that host. TURN is a relay fallback and
adds bandwidth/latency only when WebRTC selects it; direct LAN/public candidates
remain the preferred path when they work.
Default public images:
ghcr.io/pdp-connect/pdpp/reference:mainghcr.io/pdp-connect/pdpp/web:main
The main tag is a moving default-branch build. Stable semantic-release images
are published as exact version tags such as 1.2.3, moving minor-series tags
such as 1.2, and latest. For durable self-hosting, pin an exact version,
sha-* tag, or digest in .env.docker:
PDPP_REFERENCE_IMAGE=ghcr.io/pdp-connect/pdpp/reference:1.2.3
PDPP_WEB_IMAGE=ghcr.io/pdp-connect/pdpp/web:1.2.3The important topology rule is that PDPP_REFERENCE_ORIGIN is what the
browser uses, while PDPP_AS_URL and PDPP_RS_URL are container-internal
service URLs. The default Compose values are:
PDPP_REFERENCE_ORIGIN=http://localhost:3002
PDPP_AS_URL=http://reference:7662
PDPP_RS_URL=http://reference:7663Do not use localhost for cross-container AS/RS calls; inside Docker,
localhost means the current container.
Persistent mounts are defined for:
packages/polyfill-connectors/.pdpp-data/bind-mounted to/var/lib/pdpp, withPDPP_DB_PATHat/var/lib/pdpp/pdpp.sqlitePDPP_EMBEDDING_CACHE_DIRat/var/cache/pdpp/transformers
The reference Docker image does not bundle @pdpp/polyfill-connectors or a
default connector source list. A new instance starts with an empty installed
connector state. The owner console reads the verified catalog at
GET /v1/owner/connector-install/catalog; an owner can install a selected
digest, after which the server activates the verified immutable root and adds
its manifest to the source catalog.
For an air-gapped deployment, set PDPP_CONNECTOR_PRELOAD_DIR to a mounted
directory containing connector-install state prepared by the normal verified
installer. The directory uses the install module's layout:
$PDPP_CONNECTOR_PRELOAD_DIR/
├── connector-install-state.json
└── connectors/<connector-id>/<sha256:digest>/
├── profile/collection-profile.json
├── dist/collection-profile.mjs
└── provenance.json
The state file must contain the installer-produced hashes, provenance, digest, and entrypoint metadata. Do not hand-edit it or copy an unverified connector tree. The preload directory is used as the file-backed install store, so it must be writable when the service updates catalog high-water state; mount it read-only only when no catalog refresh or install operation is needed.
Durable connector artifacts — the Slack workspace archive, downloaded
statement PDFs — resolve under PDPP_CONNECTOR_ARTIFACT_ROOT, which Core
pins to /var/lib/pdpp/connector-artifacts. They are on the volume above; no
separate mount is needed. Outside a container, with neither that variable nor
PDPP_DB_PATH set, they fall back to ~/.pdpp/connector-artifacts and each
run says so in its log. See
packages/polyfill-connectors/src/connector-artifact-root.ts.
The first boot may download the default MiniLM model into the embedding cache.
Set PDPP_EMBEDDING_DOWNLOAD_ALLOWED=0 if you want to avoid that and accept
the corresponding deployment diagnostic warning until the cache is prewarmed.
Secrets must be supplied at runtime through environment variables,
.env.docker, or Docker secrets. Do not bake PDPP_OWNER_PASSWORD, connector
passwords, tokens, cookies, or DCR initial access tokens into images. The
repo-root .env.local remains a local development convenience, not a Docker
or production posture.
The managed DataConnect desktop stack always supplies a database key from the OS keychain as PDPP_DATABASE_ENCRYPTION_KEY to the RI child at spawn time. Desktop SQLite storage is encrypted at rest with no user setting or opt-out. Existing plaintext desktop vaults are migrated on startup with a verified backup and rollback path.
Self-hosted and Docker deployments intentionally remain plaintext when PDPP_DATABASE_ENCRYPTION_KEY is absent. They have no desktop OS keychain, and their database-key lifecycle is not defined by this release. Do not treat the absence of the variable as permission for a desktop install to mint a replacement key: an encrypted vault fails closed until its original key is restored.
Browser-based polyfill connectors are not clean-room portable demos. They need persistent browser profiles and remain subject to upstream anti-bot behavior. Mount any optional local connector inputs, such as Slack archives, explicitly when testing those connectors.
Slack imports require slackdump v4.4.2 (AGPL-3.0). The core, core-browser,
railway-core, and platform-core Docker images ship slackdump bundled by default.
Source tree and license: bundled at /usr/local/share/slackdump/ in the image
(LICENSE.agpl-3.0.txt, SOURCE_URL, SLACKDUMP_BIN=/usr/local/bin/slackdump).
To override with a different slackdump build or version, set SLACKDUMP_BIN
to point at an alternative executable on PATH or an in-container path. For
Docker runs, you can mount a host directory via PDPP_DOCKER_SLACKDUMP_DIR
in .env.docker:
PDPP_DOCKER_SLACKDUMP_DIR=/home/user/go/bin
SLACKDUMP_BIN=/opt/pdpp-tools/slackdump/slackdumpAlternatively, place or symlink an alternative executable at
packages/polyfill-connectors/.pdpp-tools/slackdump/slackdump; that local
runtime directory is gitignored and is the default compose mount source.
Update public-image deployments by pulling newer images and restarting the stack. This keeps the persisted SQLite DB, embedding cache, and browser profile volumes in place:
docker compose --env-file .env.docker pull
pnpm docker:reference:quickTo build from the current local checkout instead of pulling public images:
docker build -f deploy/docker/Dockerfile .
pnpm docker:reference:upThe .git directory is excluded from the Docker build context, so the
reference image cannot derive a real revision at startup and falls back to
pdpp-reference@<package-version>+unknown in the
PDPP-Reference-Revision response header and the GET / discovery index.
Pass the running commit through the PDPP_REFERENCE_REVISION build arg so
production images publish a real revision. Use the full SHA, not an
abbreviated form — the core/core-browser targets' identity gate
(deploy/docker/check-image-identity.sh) requires a full 40- or
64-character hex value and rejects abbreviated SHAs as not shaped like a
real git object id; using the full SHA here too keeps every
PDPP_REFERENCE_REVISION invocation in this repo consistent, even though
this reference target itself carries no OCI revision label and is not
gated:
docker build \
--build-arg PDPP_REFERENCE_REVISION="$(git rev-parse HEAD)" \
--target reference \
-t pdpp-reference:local .The runtime continues to honor the PDPP_REFERENCE_REVISION env var, so the
revision can also be set or overridden at container start time.
Run the smoke validation:
pnpm docker:smokeThe smoke check builds the stack, verifies AS/RS metadata through the composed
browser origin, checks that public metadata does not leak Docker service names,
and confirms / redirects to /owner/login when owner auth is
configured.
Scheduler startup uses the same reference-implementation/server/index.ts
entrypoint in Docker as local long-lived startup. To smoke enabled schedule
execution in Compose, seed a connector and enabled connector_schedules row in
the mounted reference database, restart only the reference service, then
confirm a new scheduler_run_history row appears for that connector without
pressing Run now. Disabled or deleted rows should not add history after restart.
For Docker-based development with hot reload, run:
pnpm docker:devThat overlays docker-compose.dev.yml, bind-mounts the repo into the
containers, runs the reference server with Node watch mode, and runs the web
app with Next dev. Dependency folders stay in Docker named volumes so host
installs and container installs do not trample each other.
Docker dev loads repo-root .env.local in the reference container so local
connector credentials such as GITHUB_PERSONAL_ACCESS_TOKEN are available to
controller-managed connector runs. Public-image and default Compose operation
do not load .env.local; use .env.docker, environment variables, or Docker
secrets there.
When accessing Docker dev through a LAN IP or reverse proxy, add the browser
hostnames to PDPP_WEB_ALLOWED_DEV_ORIGINS in .env.docker, for example:
PDPP_WEB_ALLOWED_DEV_ORIGINS=pdpp-dev.example.com,192.0.2.10Reverse proxies must also forward WebSocket upgrade traffic for
/_next/webpack-hmr; otherwise the page loads but Next HMR cannot connect.
CI builds the Docker targets on pull requests without pushing images. On
main, semantic-release creates GitHub releases from Conventional Commits and
the same release workflow publishes the npm packages (@pdpp/cli,
@pdpp/local-collector, and @pdpp/mcp-server) plus stable GHCR tags for both Docker targets:
${version}, ${major}.${minor}, latest, and sha-*. Maintainers should
make the first published GHCR packages public in GitHub's package settings if
the registry creates them private.
Maintainers can preview the next release calculation locally:
GITHUB_TOKEN=$(gh auth token) pnpm release:dry-runThe release path uses GitHub Actions credentials for GitHub releases and GHCR,
and npm trusted publishing for public packages. It must not bundle .env.local,
owner passwords, connector credentials, SQLite data, model cache files, or
browser profiles into images.
A minimal example app illustrates the current thin reference
provider-connect flow end to end (register → PAR → owner approval
→ token → RS query). It lives at
examples/third-party-app/ and runs on its own
port (default 7674) separate from the AS/RS:
pnpm --dir reference-implementation example-clientDefaults: PORT=7674, AS_URL=http://localhost:7662, RS_URL=http://localhost:7663.
The example uses the two-phase approval flow:
- the example calls
POST /consent/review, inspects the exact approval artifact, and then callsPOST /consent/approvewith its revision - when
PDPP_OWNER_PASSWORDis set, the inline shortcut is refused by the reference server. The example links out to the hosted/consentpage.
The example is a third-party client illustration - it is not a full
generic OAuth authorization-code redirect client. It has no PKCE, no
/callback, and no code exchange. It only exercises the endpoints the
reference currently advertises.
The reference publishes generated machine-readable and human-readable contract
artifacts derived from @pdpp/reference-contract. These are treated as
versioned reference outputs, not throwaway build noise.
openapi/reference-public.openapi.json- public PDPP JSON APIs only
openapi/reference-full.openapi.json- public APIs plus reference-only
/_refoperator surfaces
- public APIs plus reference-only
docs/generated/reference-routes.md- generated route index for public APIs
docs/generated/reference-ref-routes.md- generated route index for reference-only APIs
docs/generated/query-cookbook.md- generated query and flow cookbook
Regenerate them with:
pnpm reference-contract:generateThe root spec-*.md files remain normative for PDPP protocol semantics.
This package is the executable reference implementation:
- use the root specs to understand what PDPP means
- use this package to see what the current reference actually does
- use OpenSpec to understand project-level architecture and active implementation changes