From a99832aeba51870490e2ff68e734a89321b60ba8 Mon Sep 17 00:00:00 2001 From: Michiel de Jong Date: Thu, 24 Sep 2026 16:22:34 +0200 Subject: [PATCH 1/2] integration-proxy: add a minimal Dockerfile for the published crate Builds atomic-integration-proxy from crates.io (PROXY_VERSION, --locked) into a debian:bookworm-slim image with libssl3 and ca-certificates, running as a non-root user. The build context is unused (.dockerignore excludes everything). Not packaged in the crate (Cargo.toml include list) and not built in CI; tested locally: build, start against PostgreSQL 16, /healthz, /catalog. Refs #79. Co-Authored-By: Claude Opus 5.5 --- integration-proxy/.dockerignore | 2 ++ integration-proxy/Dockerfile | 28 ++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 integration-proxy/.dockerignore create mode 100644 integration-proxy/Dockerfile diff --git a/integration-proxy/.dockerignore b/integration-proxy/.dockerignore new file mode 100644 index 000000000..14c4463e4 --- /dev/null +++ b/integration-proxy/.dockerignore @@ -0,0 +1,2 @@ +# The Dockerfile builds from crates.io; it needs nothing from this directory. +* diff --git a/integration-proxy/Dockerfile b/integration-proxy/Dockerfile new file mode 100644 index 000000000..536c18d91 --- /dev/null +++ b/integration-proxy/Dockerfile @@ -0,0 +1,28 @@ +# A minimal image of the published `atomic-integration-proxy` crate; see +# SELF_HOSTING.md. It builds the version named by PROXY_VERSION from +# crates.io with the crate's own Cargo.lock (--locked), not this checkout's +# source, so the build context is irrelevant: +# +# docker build -t atomic-integration-proxy \ +# --build-arg PROXY_VERSION=0.2.0 integration-proxy/ +# +# Configuration is environment variables only; nothing is read from disk +# unless CATALOG_PATH names a local file (mount it). +ARG PROXY_VERSION=0.2.0 + +FROM rust:1-bookworm AS build +ARG PROXY_VERSION +RUN cargo install atomic-integration-proxy --version "=${PROXY_VERSION}" --locked --root /out + +FROM debian:bookworm-slim +# libssl3: the PostgreSQL client uses native-tls (OpenSSL on Linux). +# ca-certificates: provider APIs, the catalog, and a TLS PostgreSQL. +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates libssl3 \ + && rm -rf /var/lib/apt/lists/* \ + && useradd --system --no-create-home --shell /usr/sbin/nologin integration-proxy +COPY --from=build /out/bin/integration-proxy /usr/local/bin/integration-proxy +USER integration-proxy +ENV PORT=8080 +EXPOSE 8080 +ENTRYPOINT ["/usr/local/bin/integration-proxy"] From 71de89621519531f3a4302d900a589e62a338088 Mon Sep 17 00:00:00 2001 From: Michiel de Jong Date: Thu, 24 Sep 2026 16:22:34 +0200 Subject: [PATCH 2/2] integration-proxy: public self-hosting guide; host-agnostic README (#79) SELF_HOSTING.md (fixed path, linked from the connect UI): what the proxy does and doesn't, security model, cargo install / Docker / wrapper crate, every environment variable, PostgreSQL sslmode behaviour (measured: prefer falls back to plaintext against a non-TLS server, require refuses it), ENCRYPTION_KEY generation and the lack of rotation, pinning the catalog to a commit, ALLOWED_AGENTS, Caddy/nginx, a systemd unit (systemd-analyze verify only), per-platform OAuth registration with exact callback and OAUTH_* names, pointing atomic-server (#1702, unmerged) and the data browser at it, upgrades, backups, /healthz behaviour (measured: hangs without a timeout while PostgreSQL is paused; recovers after a restart), limits and what is not verified. README: link the guide; drop the Heroku-only framing of TLS, DATABASE_URL and callbacks; label the Heroku section as the localthought.io deployment. Root README: link the guide. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 + integration-proxy/README.md | 60 ++- integration-proxy/SELF_HOSTING.md | 636 ++++++++++++++++++++++++++++++ 3 files changed, 679 insertions(+), 19 deletions(-) create mode 100644 integration-proxy/SELF_HOSTING.md diff --git a/README.md b/README.md index fd4395c1e..2c75b5d4c 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,8 @@ deprecated. - `integration-proxy/` — the Rust OAuth/API proxy (LocalThought) that drive apps reach through atomic-server's host relay. See its own [README](integration-proxy/README.md) and [AGENTS.md](integration-proxy/AGENTS.md). + To run your own instance instead of atomic.place's, see + [integration-proxy/SELF_HOSTING.md](integration-proxy/SELF_HOSTING.md). See [AGENTS.md](AGENTS.md) for how to work in each. diff --git a/integration-proxy/README.md b/integration-proxy/README.md index c97e14603..0f3816ba0 100644 --- a/integration-proxy/README.md +++ b/integration-proxy/README.md @@ -21,6 +21,12 @@ the single-use record of signed requests. Built with [axum](https://github.com/tokio-rs/axum), following the OAuth 2.0 Authorization Code flow with PKCE towards providers. +**To run your own instance**, see [SELF_HOSTING.md](SELF_HOSTING.md): +installing (`cargo install`, Docker, systemd), configuration, PostgreSQL TLS, +a reverse proxy, registering OAuth apps, pointing Atomic Server and the data +browser at it, upgrades, backups and health checks. This README describes +the protocol and the code. + ## How it works ### Identity @@ -62,8 +68,8 @@ minutes). **Full URL** is `BASE_URL` followed by the request's path and query exactly as received, e.g. `https://localthought.io/proxy//github-issues/user/repos?page=2`. -It is never rebuilt from the `Host` header: behind TLS termination (Heroku) -the process sees plain HTTP. Clients sign the URL they fetch (`new URL(u).href`). +It is never rebuilt from the `Host` header: behind a TLS-terminating reverse +proxy or platform router the process sees plain HTTP. Clients sign the URL they fetch (`new URL(u).href`). `BASE_URL` must therefore be exactly the public origin clients use. ### Routes @@ -188,9 +194,9 @@ Nothing in the process reads `.env` files; export the variables, or load a | --- | --- | --- | | `BASE_URL` | no | Public URL of the proxy, e.g. `https://localthought.io`. Defaults to `http://localhost:8080`. Used for OAuth callback URLs, as the prefix of every signed URL, and (its origin) as a capability's `aud`. Must be exactly what clients use. | | `PORT` | no | Port to listen on. Defaults to `8080`. | -| `SESSION_SECRET` | no | Secret for the short-lived consent and OAuth-binding cookies. If unset, a random key is generated at startup, and a consent screen open during a restart must be started again. | +| `SESSION_SECRET` | no; set it in production | Secret for the short-lived consent and OAuth-binding cookies. If unset, a random key is generated at startup (with a warning in the log), and a consent screen open during a restart must be started again. Instances behind one name must share it. | | `CATALOG_PATH` | no | Local path or HTTPS URL for the catalog JSON. Defaults to `https://ontola.github.io/atomic-plugins/overlays/catalog.json`, this repository's `overlays/catalog.json` as GitHub Pages publishes it from `main`. | -| `DATABASE_URL` | yes | PostgreSQL connection URL. | +| `DATABASE_URL` | yes | PostgreSQL connection URL. See the TLS note below. | | `ENCRYPTION_KEY` | yes | Base64url-encoded, random 32-byte key for sealed provider credentials. Changing it makes every stored connection unreadable. | | `REVOKED_SUBJECTS` | no | Comma-separated agent ids (any accepted spelling) the default access policy refuses. | | `ALLOWED_AGENTS` | no | When set, comma-separated agent ids; the default access policy admits only these owners. | @@ -207,15 +213,25 @@ The server owns the OAuth endpoints and scopes for every platform in the catalog, reading them from that platform's composed OpenAPI document; a request cannot supply a provider URL, token URL, or scope. -The PostgreSQL client validates the database TLS certificate. Heroku assigns -`DATABASE_URL` automatically when its Postgres add-on is attached. Schema -setup runs in a transaction under an advisory lock, so several instances can -start against an empty database at once. +The PostgreSQL client validates the database's TLS certificate against the +system roots when TLS is used; `sslmode` in `DATABASE_URL` decides whether it +is (`prefer` by default, which falls back to plaintext against a server +without TLS). Use `?sslmode=disable` for a PostgreSQL on the same host and +`?sslmode=require` across any network; see +[SELF_HOSTING.md](SELF_HOSTING.md#database_url-and-postgresql-tls). A managed +PostgreSQL may set `DATABASE_URL` for you (Heroku's Postgres add-on does). +Schema setup runs in a transaction under an advisory lock, so several +instances can start against an empty database at once. ## Catalog +Per-platform OAuth app registration for any deployment is in +[SELF_HOSTING.md](SELF_HOSTING.md#registering-oauth-apps). Every callback is +`/oauth//callback`; the examples below use the +localthought.io deployment's `BASE_URL`. + Discord uses `OAUTH_DISCORD_CLIENT_ID` and `OAUTH_DISCORD_CLIENT_SECRET`, -with production callback `https://localthought.io/oauth/discord/callback`. +with callback `https://localthought.io/oauth/discord/callback` on localthought.io. Register an OAuth application in the Discord Developer Portal. The initial read-only integration uses `identify` and `guilds` to read your profile and import server memberships; it does not import messages or require a bot token. @@ -225,11 +241,12 @@ operation, not an imported collection. Discord access tokens expire and use the existing refresh-token flow. Spotify uses `OAUTH_SPOTIFY_CLIENT_ID` and the callback -`https://localthought.io/oauth/spotify/callback` in production. Register a +`/oauth/spotify/callback` (on localthought.io, +`https://localthought.io/oauth/spotify/callback`). Register a Spotify Web API app with that exact redirect URI. The integration uses Authorization Code with PKCE, so no client secret is required or transmitted; -set `OAUTH_SPOTIFY_CLIENT_AUTH_METHOD=none` so the proxy does not require or -send one. It imports playlists with `playlist-read-private` and +the catalog declares only `none`, so the proxy neither requires nor sends +one (`OAUTH_SPOTIFY_CLIENT_AUTH_METHOD=none` is equivalent). It imports playlists with `playlist-read-private` and `playlist-read-collaborative`; no write scopes are requested. No account ID parameter is needed. Development-mode access is subject to Spotify's Premium and app-user allowlist requirements. Access tokens refresh automatically; @@ -238,7 +255,7 @@ expired or revoked refresh tokens require reconnecting through OAuth. Moneybird uses `OAUTH_MONEYBIRD_CLIENT_ID` and `OAUTH_MONEYBIRD_CLIENT_SECRET`, with callback -`https://localthought.io/oauth/moneybird/callback` in production. Register an +`/oauth/moneybird/callback`. Register an external OAuth application, rather than a personal API token. The `sales_invoices` scope grants access to contacts (Moneybird has no contacts-only scope). The initial integration imports contacts; supply the administration ID @@ -284,7 +301,9 @@ its overlays are whatever `main` serves when the proxy starts. cargo run ``` -Then open `http://localhost:8080` (or your configured `BASE_URL`). +Then open `http://localhost:8080` (or your configured `BASE_URL`). For a +deployment, see [SELF_HOSTING.md](SELF_HOSTING.md); `GET /healthz` is the +health check. ## Development @@ -355,9 +374,12 @@ Only `src/`, `static/index.html`, `static/logo.png`, `Cargo.toml`, (`cargo package --list` shows the exact list). `tests/` fixtures are not, so `cargo test` only works from a checkout of this repository. -### Production deployment (localthought.io) +### The localthought.io deployment (Heroku) -Production runs on Heroku from the separate repository +This section is about one deployment, not a requirement: the crate runs on +any host that provides the environment variables, one HTTP port and +PostgreSQL ([SELF_HOSTING.md](SELF_HOSTING.md)). localthought.io runs on +Heroku from the separate repository `localthought/integration-proxy`, which today still carries its own full copy of this source. The target state is that it contains only the files in [`examples/heroku-wrapper/`](examples/heroku-wrapper/) and picks up proxy @@ -391,12 +413,12 @@ The synced `Procfile` runs `target/release/integration-proxy` (the binary was 0.2 changes the client protocol; deploy it together with the atomic-server release that signs requests (Atomic v2) and uses `/proxy/{connection_id}/…`, -never before it. Heroku config vars: +never before it. Environment (on Heroku, config vars): | Variable | Change | | --- | --- | | `BASE_URL` | **Check**: must be exactly the public origin clients use (for production `https://localthought.io`, no trailing path). Every signature covers it; a mismatch makes every signed request fail with `bad_signature`. | -| `APP_AUTH_CLIENT_ID`, `APP_AUTH_CLIENT_SECRET`, `APP_AUTH_AUTHORIZATION_URL`, `APP_AUTH_TOKEN_URL`, `APP_AUTH_USERINFO_URL`, `APP_AUTH_LABEL`, `APP_AUTH_IDENTITY_NAMESPACE` | No longer read; unset them (`heroku config:unset …`). 0.1 refused to start without the first five; 0.2 ignores them. | +| `APP_AUTH_CLIENT_ID`, `APP_AUTH_CLIENT_SECRET`, `APP_AUTH_AUTHORIZATION_URL`, `APP_AUTH_TOKEN_URL`, `APP_AUTH_USERINFO_URL`, `APP_AUTH_LABEL`, `APP_AUTH_IDENTITY_NAMESPACE` | No longer read; unset them (on Heroku, `heroku config:unset …`). 0.1 refused to start without the first five; 0.2 ignores them. | | `SERVER_SECRET` | No longer read; unset it. | | `REVOKED_SUBJECTS` | Now means agent ids; tenant ids and OIDC subjects listed there no longer match anything. Rewrite or unset it. | | `ALLOWED_AGENTS` | New, optional: restrict the proxy to these owners. | @@ -438,7 +460,7 @@ top-level browser navigation. The `todoist` platform imports projects and active tasks through Todoist API v1 with the read-only `data:read` scope. Configure `OAUTH_TODOIST_CLIENT_ID` and `OAUTH_TODOIST_CLIENT_SECRET`, and register -`https://localthought.io/oauth/todoist/callback` as the OAuth redirect URL. +`/oauth/todoist/callback` as the OAuth redirect URL. New Todoist applications issue expiring access tokens and rotating refresh tokens; the proxy stores and refreshes these through its existing credential flow. Legacy non-expiring access tokens are also supported. No provider writes are exposed. diff --git a/integration-proxy/SELF_HOSTING.md b/integration-proxy/SELF_HOSTING.md new file mode 100644 index 000000000..81cc2a15f --- /dev/null +++ b/integration-proxy/SELF_HOSTING.md @@ -0,0 +1,636 @@ +# Running your own integration proxy + +When you connect a third-party service (GitHub, Google Calendar, Notion, …) +from Atomic Server or the Atomic data browser, the OAuth grant or API key for +that service is held by an **integration proxy**, and requests to the service +go through it. By default that proxy is the one atomic.place runs. This page +is for anyone who would rather run their own: it covers what the proxy does, +how to run and configure it, how to register your own OAuth apps, and how to +point Atomic Server and the data browser at it. + +It describes `atomic-integration-proxy` +[0.2.0 on crates.io](https://crates.io/crates/atomic-integration-proxy). +Everything here was checked against that version's source in this directory; +[What is not verified](#what-is-not-verified) lists what was not tried. + +- [What the proxy does, and what it doesn't](#what-the-proxy-does-and-what-it-doesnt) +- [Security model](#security-model) +- [Requirements](#requirements) +- [Installing](#installing) +- [Configuration](#configuration) +- [TLS and a reverse proxy](#tls-and-a-reverse-proxy) +- [Running it as a service](#running-it-as-a-service) +- [Registering OAuth apps](#registering-oauth-apps) +- [Pointing Atomic at your proxy](#pointing-atomic-at-your-proxy) +- [Upgrading](#upgrading) +- [Backups](#backups) +- [Health checks and monitoring](#health-checks-and-monitoring) +- [Limits](#limits) +- [What is not verified](#what-is-not-verified) + +## What the proxy does, and what it doesn't + +The proxy: + +- runs the OAuth 2.0 authorization-code flow (with PKCE where the provider + supports it) against a provider, or shows a form where the user pastes an + API key for API-key providers such as Clockify; +- stores the resulting token or key encrypted in PostgreSQL, as a + **connection** owned by the user's Atomic agent; +- forwards requests from that agent, and from app agents the owner has + **delegated** the connection to, to the provider, attaching the credential + and refreshing an expiring OAuth token; +- forwards only the HTTP methods and paths that the provider's entry in the + **catalog** allows. + +The proxy does not: + +- have accounts, logins or passwords. The caller's Atomic agent + (`atomic:agent:`) is the account, and every request is + signed with that agent's key; +- return a provider token or API key to any client, ever; +- store any provider data. It passes responses through without keeping them; + syncing and storing records is the job of Atomic Server and its plugins; +- let a caller choose the provider's host, token endpoint or scopes. These + come from the catalog, which you control. + +## Security model + +In short (the full list is in [SECURITY.md](SECURITY.md)): + +- **Sealed credentials.** Each provider credential is encrypted with + XChaCha20-Poly1305 under `ENCRYPTION_KEY`, with the connection id as + associated data, so a sealed value copied into another row does not + decrypt. Anyone with the database but not the key cannot read credentials. + Anyone with both can. +- **Catalog allowlist.** The upstream host is the `servers` URL of the + platform's composed OpenAPI document. A request is forwarded only if its + method and path match an operation in that document. Paths with `.` or + `..` segments are refused before matching. A bounded subset of each + request is validated: required query parameters, enum values of query + parameters, and the presence and content type of the body. Full JSON Schema + validation of bodies is **not** done. +- **No redirect-following.** Upstream redirects are never followed, so a + provider cannot redirect a credentialed request somewhere the catalog did + not allow. The caller's `Authorization` and `x-atomic-*` headers are never + forwarded. +- **Signed requests.** Every request that uses or manages a connection + carries an Atomic v2 signature: Ed25519 over the method, the full URL + (`BASE_URL` plus path and query), a timestamp, and a SHA-256 of the body. + Timestamps must be within ±5 minutes of the proxy's clock, and each signed + message is accepted once. The proxy stores the agent id only; it looks no + key up, because the key is part of the id. +- **Delegations.** A connection's owner can delegate it to an app agent (an + installed plugin) and register runtimes (a node's agent for that app). + Delegations and runtimes are checked on every request, so removing one + takes effect on the next request. A plugin frame that cannot keep a key + uses a capability, signed by the owner, valid for at most 15 minutes, and + bound to a key the frame holds in memory. +- **Idle expiry.** A connection with no authenticated request for 90 days is + deleted. + +Things this model does **not** protect against are listed under +[Limits](#limits). + +## Requirements + +- **PostgreSQL.** CI runs the tests against PostgreSQL 16. The schema uses + nothing recent (`CREATE TABLE IF NOT EXISTS`, `make_interval`), but older + versions have not been tested. The proxy creates its own tables at startup, + so its role needs `CREATE` on the `public` schema. On PostgreSQL 15 and + newer, `public` no longer grants that to everyone, so make the role the + database's owner: + + ```sh + sudo -u postgres createuser --pwprompt integration_proxy + sudo -u postgres createdb --owner integration_proxy integration_proxy + ``` + +- **A public HTTPS origin**, such as `https://proxy.example.org`. OAuth + providers redirect browsers to it, the consent cookies are `Secure`, and + every signature covers it. `http://localhost` works for development only. +- **Outbound HTTPS** to the providers you enable, and, at every start, to + wherever the catalog and its OpenAPI and overlay documents are hosted + (by default `ontola.github.io` and `raw.githubusercontent.com`; see + [`CATALOG_PATH`](#catalog_path)). +- **One TCP port** for HTTP (`PORT`, default 8080). +- To build from source: a stable Rust toolchain. The published binary is not + distributed pre-built. + +## Installing + +The crate ships a binary called `integration-proxy`. Choose one of these +three ways to run it. + +### cargo install + +```sh +cargo install atomic-integration-proxy --version 0.2.0 --locked +# installs ~/.cargo/bin/integration-proxy +``` + +`--locked` builds with the exact dependency versions the crate was published +with. On Linux the PostgreSQL client uses OpenSSL (`native-tls`), so the build +needs `pkg-config` and the OpenSSL headers (`libssl-dev` on Debian and +Ubuntu), and the running binary needs `libssl3` and `ca-certificates`. + +### Docker + +[`Dockerfile`](Dockerfile) builds that same crates.io release (not the source +of the checkout it sits in) into a `debian:bookworm-slim` image that runs as +a non-root user: + +```sh +docker build -t atomic-integration-proxy \ + --build-arg PROXY_VERSION=0.2.0 \ + https://github.com/ontola/atomic-plugins.git#main:integration-proxy + +docker run -d --name integration-proxy --restart unless-stopped \ + -p 127.0.0.1:8080:8080 \ + --env-file /etc/integration-proxy/env \ + atomic-integration-proxy +``` + +Publish the port on `127.0.0.1` and put a TLS-terminating reverse proxy in +front (see [below](#tls-and-a-reverse-proxy)); inside the container the +proxy listens on `0.0.0.0`. If PostgreSQL runs on the Docker host, a +container reaches it at the host's bridge address (or +`host.docker.internal` with Docker Desktop), not at `127.0.0.1`. + +### A wrapper crate + +The library exposes the whole server as `atomic_integration_proxy::run()`, so +a deployment can also be its own small crate that depends on the published +version. That is what [`examples/heroku-wrapper/`](examples/heroku-wrapper/) +is. Its entire `main.rs` is: + +```rust +#[tokio::main] +async fn main() -> std::process::ExitCode { + atomic_integration_proxy::run().await +} +``` + +A wrapper is useful if you want to commit a `Cargo.lock`, or to plug in your +own admission check with `build_app_with_access` (see the +[library API](README.md#library-crate)). + +## Configuration + +All configuration comes from environment variables. The process does not +read `.env` files itself; systemd's `EnvironmentFile=` or Docker's +`--env-file` can load one. An example +`/etc/integration-proxy/env` (readable only by root and the service user). +Neither systemd nor Docker strips a `#` comment that follows a value, so keep +comments on their own lines: + +```sh +BASE_URL=https://proxy.example.org +PORT=8080 +DATABASE_URL=postgres://integration_proxy:PASSWORD@127.0.0.1:5432/integration_proxy?sslmode=disable +# Generate both secrets as described below. +ENCRYPTION_KEY=... +SESSION_SECRET=... +# CATALOG_PATH=/etc/integration-proxy/catalog.json +# ALLOWED_AGENTS=atomic:agent:... +OAUTH_GITHUB_ISSUES_CLIENT_ID=... +OAUTH_GITHUB_ISSUES_CLIENT_SECRET=... +RUST_LOG=info +``` + +| Variable | Required | Meaning | +| --- | --- | --- | +| `DATABASE_URL` | yes | PostgreSQL URL. See [PostgreSQL and TLS](#database_url-and-postgresql-tls). | +| `ENCRYPTION_KEY` | yes | Unpadded base64url encoding of 32 random bytes (43 characters). | +| `BASE_URL` | yes in production | The exact public origin, e.g. `https://proxy.example.org`. Defaults to `http://localhost:8080`. | +| `SESSION_SECRET` | strongly recommended | Any long random string; keys the consent and OAuth cookies. | +| `PORT` | no | Listening port, default `8080`. The proxy always binds `0.0.0.0`. | +| `CATALOG_PATH` | no | Catalog location. Defaults to `https://ontola.github.io/atomic-plugins/overlays/catalog.json`. | +| `ALLOWED_AGENTS` | no | Comma-separated agent ids. When set, only these agents may own connections. | +| `REVOKED_SUBJECTS` | no | Comma-separated agent ids that may not own connections. | +| `OAUTH__CLIENT_ID`, `_CLIENT_SECRET`, `_CLIENT_AUTH_METHOD` | per OAuth platform | See [Registering OAuth apps](#registering-oauth-apps). | +| `RUST_LOG` | no | Log filter, default `info`. | + +Variables from before 0.2 (`APP_AUTH_*`, `SERVER_SECRET`) are ignored. + +### `DATABASE_URL` and PostgreSQL TLS + +The PostgreSQL client uses a certificate-verifying TLS connector with the +system's trusted roots. The URL's `sslmode` decides whether TLS is used at +all. The client supports `disable`, `prefer` (the default) and `require`; +the `verify-ca`, `verify-full` and `sslrootcert` options of `libpq` are not +supported. Tested against a PostgreSQL 16 container with TLS off: + +| `sslmode` | Server without TLS | Server with TLS | +| --- | --- | --- | +| omitted or `prefer` | connects **without** TLS | TLS, certificate must verify | +| `disable` | connects without TLS | connects without TLS | +| `require` | refuses to start (`error performing TLS handshake`) | TLS, certificate must verify | + +So: + +- **PostgreSQL on the same host, over loopback or a Unix socket:** use + `?sslmode=disable`. It says what happens, and it avoids a failed handshake + if the server has TLS on with a self-signed certificate. +- **PostgreSQL across any network:** use `?sslmode=require`, with a + certificate that chains to a public root. With the default `prefer`, an + attacker on the path can make the server look like it has no TLS, and the + client then continues in plaintext. A private CA would have to be added to + the system trust store; this has not been tried. +- Never use `sslmode=disable` across a network. + +### `ENCRYPTION_KEY` + +Generate one once and keep it: + +```sh +openssl rand -base64 32 | tr '+/' '-_' | tr -d '=' +``` + +The value must be unpadded base64url (a trailing `=` is refused) that decodes +to exactly 32 bytes. Otherwise the proxy refuses to start. + +**There is no key rotation.** Sealed values carry a format version but no key +id, and the proxy holds one key. If `ENCRYPTION_KEY` changes, the proxy can +no longer open any stored connection. Each one then answers +`404 unknown_connection`, as if it had been deleted, and every user has to +connect again. Those rows stay until the 90-day idle sweep removes them. If +the key leaks, the only remedy is to set a new key and have everyone +reconnect (and delete the old rows: +`DELETE FROM agent_connections;` also removes their delegations). + +### `SESSION_SECRET` + +This secret keys the two short-lived (10-minute) encrypted cookies that carry +a user from the consent page to the provider and back. If it is unset, the +proxy logs a warning and generates a random key at every start. A user who is +halfway through connecting when the proxy restarts then has to start again. +Stored connections are not affected. Set it to a long random value, such as +`openssl rand -base64 48`, and keep it stable across restarts. If you run +several instances behind one name, they must share it. + +### `BASE_URL` + +`BASE_URL` must be exactly the origin that browsers and Atomic Server use to +reach the proxy: scheme, host, and port if it is not the default. Use no +trailing path. It is used for three things: + +- the OAuth callback URL `/oauth//callback` you register + with each provider; +- the first part of the URL covered by every request signature. The proxy + rebuilds the signed URL from `BASE_URL` and the path and query it received, + never from the `Host` header. If `BASE_URL` is wrong, every signed request + fails with `401 bad_signature`; +- its origin is the required `aud` of frame capabilities. + +At startup the proxy refuses a `BASE_URL` that is not http(s) or that has +credentials, a query or a fragment. A trailing `/` is removed. + +### `CATALOG_PATH` + +The catalog lists the platforms the proxy supports. For each one it names a +pinned OpenAPI document and the overlays applied to it. At every start the +proxy downloads the catalog and each of those documents over HTTPS, composes +them, and keeps the result in memory. If any download fails, the proxy does +not start. Non-HTTPS sources are refused. `CATALOG_PATH` may be an HTTPS URL +or a local file; the documents it lists must be HTTPS URLs. + +The default is `catalog.json` as GitHub Pages serves it from this +repository's `main` branch. The OpenAPI documents it lists are pinned to +commits, but **the overlays are not**: a restart picks up whatever `main` +serves at that moment. To make restarts reproducible, pin the whole catalog +to a commit and serve it locally: + +```sh +SHA= +curl -fsSL "https://raw.githubusercontent.com/ontola/atomic-plugins/$SHA/overlays/catalog.json" \ + | sed "s#https://ontola.github.io/atomic-plugins/overlays/#https://raw.githubusercontent.com/ontola/atomic-plugins/$SHA/overlays/#g" \ + > /etc/integration-proxy/catalog.json +# then: CATALOG_PATH=/etc/integration-proxy/catalog.json +``` + +This procedure was run against this repository's `main` and composed all +eight platforms. The same file is also where you narrow the allowlist: +delete the platform entries you don't want to offer. Keep the remaining +entries' `name` values unchanged, because the `OAUTH_*` variable names and +callback URLs are derived from them. + +### `ALLOWED_AGENTS` and `REVOKED_SUBJECTS` + +By default, any Atomic agent that can reach the proxy can create +connections. The proxy has no signup step, so anyone can use your OAuth apps +and your server. To restrict it to your own people, list their agents: + +```sh +ALLOWED_AGENTS=atomic:agent:AbC...,atomic:agent:XyZ... +``` + +The check is on the connection's **owner**, the agent that signed +`/connect/redeem`. It runs at redeem time and on every signed request, and +delegated app agents and plugin frames are judged as their owner, so you +only list the users' own agents, not each installed app. `REVOKED_SUBJECTS` +does the opposite: it refuses the agents it lists, and it takes precedence. +Both accept `atomic:agent:` or legacy `did:ad:agent:` ids in either base64 +alphabet. An entry that does not parse as an agent id is ignored, with a +warning in the log at startup. If every `ALLOWED_AGENTS` entry is invalid, +nobody is admitted. + +A refused agent gets `403 access_denied`. The unauthenticated routes (`/`, +`/catalog`, the consent page and the OAuth callback) stay reachable by +anyone; the list only stops anyone else from redeeming or using a +connection. + +## TLS and a reverse proxy + +The proxy speaks plain HTTP. Put a TLS-terminating reverse proxy in front of +it, and make sure the proxy's own port is not reachable from outside. It +binds `0.0.0.0` and has no setting to bind only loopback, so use a firewall, +or publish only on `127.0.0.1` as in the Docker example. + +**Caddy** (obtains and renews the certificate itself): + +```caddyfile +proxy.example.org { + reverse_proxy 127.0.0.1:8080 +} +``` + +**nginx** (certificate paths from certbot, adjust to yours): + +```nginx +server { + listen 443 ssl; + http2 on; + server_name proxy.example.org; + ssl_certificate /etc/letsencrypt/live/proxy.example.org/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/proxy.example.org/privkey.pem; + + # The proxy accepts request bodies up to 2 MiB; nginx's default is 1 MiB. + client_max_body_size 2m; + + location / { + proxy_pass http://127.0.0.1:8080; + proxy_set_header Host $host; + proxy_read_timeout 60s; + } +} +``` + +The reverse proxy must pass the path and query through unchanged: they are +part of what the client signed. Neither snippet rewrites them. Do not mount +the proxy under a sub-path. Atomic Server expects a bare origin (see +[below](#pointing-atomic-at-your-proxy)). + +## Running it as a service + +A systemd unit for a `cargo install`ed binary copied to +`/usr/local/bin/integration-proxy`, with the environment file above. This +unit passes `systemd-analyze verify` (systemd 252, Debian bookworm) but has +not been run as a service; verify it on your own system too. + +```ini +# /etc/systemd/system/integration-proxy.service +[Unit] +Description=Atomic integration proxy +Wants=network-online.target +After=network-online.target postgresql.service + +[Service] +Type=exec +ExecStart=/usr/local/bin/integration-proxy +EnvironmentFile=/etc/integration-proxy/env +DynamicUser=yes +# The proxy exits if PostgreSQL or the catalog is unreachable at startup. +Restart=on-failure +RestartSec=5s +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +PrivateDevices=yes +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectControlGroups=yes +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +LockPersonality=yes +MemoryDenyWriteExecute=yes + +[Install] +WantedBy=multi-user.target +``` + +```sh +sudo install -m 0755 ~/.cargo/bin/integration-proxy /usr/local/bin/ +sudo install -d -m 0750 /etc/integration-proxy +sudo install -m 0600 env /etc/integration-proxy/env # read by systemd as root +sudo systemctl daemon-reload +sudo systemctl enable --now integration-proxy +curl -fsS https://proxy.example.org/healthz # prints: ok +``` + +With `DynamicUser=yes` the process can write nothing on disk. It needs to +read only a local `CATALOG_PATH` file, if you use one, so make that file +world-readable or add `ReadOnlyPaths=`. On stop, the process exits at once +and does not drain in-flight requests. + +The same binary runs on any host that provides these environment variables +and a port. For Heroku, see [`examples/heroku-wrapper/`](examples/heroku-wrapper/) +and the [`Procfile`](Procfile). + +## Registering OAuth apps + +For each OAuth platform you want to offer, register an OAuth app (client) +with the provider under your own account, and give the proxy its +credentials. Platforms without credentials still appear in `/catalog`, but +their consent page answers "This platform is not available for connection". + +Two things are the same for every platform: + +- **Callback (redirect) URL:** `/oauth//callback`. For + example, `https://proxy.example.org/oauth/github-issues/callback`. Register + it exactly: same scheme, host, port and path, no trailing slash. +- **Environment variables:** `OAUTH_` + the platform name in upper case with + `-` replaced by `_`, followed by `_CLIENT_ID`, `_CLIENT_SECRET` and, + optionally, `_CLIENT_AUTH_METHOD` (`none`, `client_secret_post` or + `client_secret_basic`). When `_CLIENT_AUTH_METHOD` is omitted, the proxy + uses the first method the catalog declares that it implements. With no + declaration it uses `client_secret_post`, or `client_secret_basic` when the + token operation requires it. A method of `none` needs no secret. + +Scopes are fixed by the catalog; you cannot add or remove them through +configuration. The table lists the platforms in the catalog on `main` at the +time of writing. `GET /catalog` shows what your proxy actually +loaded. + +| Platform | Variables | Where to register | Scopes requested | Notes | +| --- | --- | --- | --- | --- | +| `github-issues` | `OAUTH_GITHUB_ISSUES_CLIENT_ID`, `_CLIENT_SECRET` | [GitHub → Settings → Developer settings → OAuth Apps](https://github.com/settings/developers) | `repo` | An OAuth App, not a GitHub App. `repo` grants read and write access to all of the user's repositories, including private ones. | +| `google-calendar` | `OAUTH_GOOGLE_CALENDAR_CLIENT_ID`, `_CLIENT_SECRET` | [Google Cloud console → Credentials](https://console.cloud.google.com/apis/credentials): an OAuth client ID of type *Web application*; also enable the Google Calendar API and configure the consent screen | `calendar.events`, `calendar.calendarlist.readonly` | The catalog selects offline access, so Google issues a refresh token. Google treats these scopes as sensitive. An unverified app shows a warning, is limited to test users while in *Testing*, and, per Google's documentation, its refresh tokens then expire after 7 days. | +| `notion` | `OAUTH_NOTION_CLIENT_ID`, `_CLIENT_SECRET` | [Notion → Integrations](https://www.notion.so/profile/integrations): a *public* integration, with your callback as a redirect URI | none (Notion shares pages the user picks) | Uses `client_secret_basic`, selected automatically; no PKCE (Notion does not support it). | +| `todoist` | `OAUTH_TODOIST_CLIENT_ID`, `_CLIENT_SECRET` | [Todoist App Management Console](https://developer.todoist.com/appconsole.html) | `data:read` | Read-only. | +| `moneybird` | `OAUTH_MONEYBIRD_CLIENT_ID`, `_CLIENT_SECRET` | [Moneybird → register an application](https://moneybird.com/user/applications/new) (an external application, not a personal API token) | `sales_invoices documents estimates bank time_entries settings` | The user supplies their administration id when connecting. | +| `spotify` | `OAUTH_SPOTIFY_CLIENT_ID` | [Spotify for Developers → Dashboard](https://developer.spotify.com/dashboard) | `playlist-read-private`, `playlist-read-collaborative` | A public client: the catalog declares only `none`, so no secret is needed or sent (setting `OAUTH_SPOTIFY_CLIENT_AUTH_METHOD=none` is equivalent). Apps in development mode only admit users you allowlist. | +| `discord` | `OAUTH_DISCORD_CLIENT_ID`, `_CLIENT_SECRET` | [Discord Developer Portal → Applications](https://discord.com/developers/applications) → OAuth2 → Redirects | `identify`, `guilds` | No bot token needed. | +| `clockify` | none | nothing to register | n/a | An API-key platform. The consent page asks the user for their own key (Clockify → Profile settings → API), which is sealed like an OAuth token. | + +The consoles' names and layouts change over time; the links were current +when this was written, and no registration was redone for this guide. A +provider may also restrict what it accepts as a redirect URL. Spotify, for +example, documents that it no longer accepts `http://localhost`, only HTTPS +or a loopback IP such as `http://127.0.0.1:`. + +After changing any `OAUTH_*` variable, restart the proxy. The variables are +read when a connect starts or a token is refreshed, but restarting is the +supported way to apply them. + +## Pointing Atomic at your proxy + +Atomic Server and the data browser are configured **separately**. Both must +point at the same proxy. Nothing keeps them in sync yet; that is tracked in +[atomic-server#1700](https://github.com/ontola/atomic-server/issues/1700). + +**Atomic Server** (for plugins that call a provider through the server): +start it with `--integration-proxy-url https://proxy.example.org`, or set +`ATOMIC_INTEGRATION_PROXY_URL=https://proxy.example.org`. The value is the +proxy's bare origin, exactly its `BASE_URL`, and the server checks at boot +that it has no path, query, fragment or credentials. The server signs +requests to exactly that origin as the installed plugin's app agent. This +option comes from +[atomic-server#1702](https://github.com/ontola/atomic-server/pull/1702), +which was **not yet merged** when this was written. Check your Atomic +Server's `--help` for it. What the server does when the option is unset, and +whether that will default to atomic.place's proxy, is not settled by that +PR. + +**The data browser** (where users click *Connect*): open **Settings → +Integrations** and enter the proxy's origin (HTTPS, or loopback HTTP for +development). The setting is stored per browser, in `localStorage` under +`integration-proxy-url`. Other browsers and other users still use the +default until they change it too. An empty value restores the default, and +an invalid value is ignored in favour of the default. The default is fixed +when the data browser is built, from `VITE_INTEGRATION_PROXY_URL`. It is +meant to be atomic.place's proxy (`https://integrations.atomic.place`, once +that is live); builds at the time of writing fall back to +`https://localthought.io`. If you build the data browser yourself, set +`VITE_INTEGRATION_PROXY_URL=https://proxy.example.org` to make your proxy the +default for everyone who uses that build. + +Connections are stored in the proxy that created them. Pointing Atomic at a +different proxy does not move them: users connect again on the new one. + +## Upgrading + +- **Patch releases** (`0.2.x`): install the new version (`cargo install + atomic-integration-proxy --version 0.2.x --locked`, or rebuild the image + with a new `PROXY_VERSION`) and restart. +- **Minor releases before 1.0** (`0.2` to `0.3`) may be breaking, as 0.1 to + 0.2 was. Read the [README](README.md) of the new version, in particular + anything like [What was removed](README.md#what-was-removed-flag-day), and + the commits to `integration-proxy/` in between, before upgrading. There is + no separate changelog. +- **Schema:** the proxy has no migration tool. At startup it runs + `CREATE TABLE/INDEX IF NOT EXISTS` for its tables, in a transaction under + an advisory lock, so several instances can start at once. Tables that an + older version used are left in place. After upgrading from 0.1, + `DROP TABLE IF EXISTS connection_codes, oauth_states;` removes the old + ones. That is optional and irreversible. +- **Rolling back** to an older binary against a database a newer one has + used has not been tested. Take a backup before upgrading. +- **Clients:** 0.2 accepts only Atomic v2 signatures. Atomic Server and data + browser builds from before the v2 work cannot use it. + +## Backups + +Back up the database and `ENCRYPTION_KEY` together, but store them +**separately**. The dump alone does not reveal credentials, the key alone is +useless, and a dump restored without its key is a set of connections that +all answer `404 unknown_connection`. + +```sh +pg_dump --format=custom --file=integration-proxy-$(date +%F).dump \ + "postgres://integration_proxy@127.0.0.1/integration_proxy" +``` + +`SESSION_SECRET` and the `OAUTH_*` values do not need to match on restore: +the former only affects connects in progress, and the latter can be read +from the providers' consoles again, or regenerated there. + +A restore puts back each connection's credential as it was at backup time. +Providers that rotate refresh tokens (Todoist, for one, documents this) +will have rotated them since, so the restored ones may be refused. The +affected users get `401 credential_refresh_failed` and have to connect +again. Delegations made after the backup are lost. + +## Health checks and monitoring + +- `GET /healthz` answers `200 ok` when a `SELECT 1` on the database + succeeds, and `503 database unavailable` when it fails. It checks nothing + else: not the catalog (that is loaded once, at startup) and not any + provider. **Always use a timeout.** The check itself has none: with the + database container paused, `/healthz` did not answer within 5 seconds. +- A lost database connection is re-established automatically, with backoff + from 1 to 30 seconds. After a PostgreSQL restart, `/healthz` answered + `503` for about a second and then `200` again, without restarting the + proxy. +- The process exits with status 1 and a one-line message on stderr if the + configuration is invalid, or if the catalog or the database is unreachable + at startup. +- Logs go to stderr via `tracing`, filtered by `RUST_LOG`. The default is + `info`. `RUST_LOG=info,tower_http=debug` adds a line per HTTP request. The + proxy does not log tokens, keys or request bodies. Request paths, which + contain connection ids, can appear at debug level. Connection ids are not + secrets. +- There is no metrics endpoint. + +## Limits + +These are properties of 0.2.0 that affect whether self-hosting fits your +needs: + +- **No rate limiting.** One agent can send as many requests as the provider + allows. `ALLOWED_AGENTS` is the only way to limit who can use the proxy at + all. +- **Anyone can start a connect flow.** The consent page and OAuth callback + need no login. `ALLOWED_AGENTS` stops strangers from redeeming or using a + connection, not from seeing your consent page. +- **The catalog is trusted.** Whoever controls the catalog's URLs decides + which hosts receive users' credentials. Requests to private or internal + network ranges are not blocked if a catalog entry points there. +- **Sizes and timeouts:** request bodies up to 2 MiB; upstream responses up + to 10 MiB; 10 s to connect upstream and 30 s per upstream request. +- **One `ENCRYPTION_KEY`**, with no rotation (see above). +- **Branding:** the landing and consent pages say "LocalThought" and use its + logo, whoever runs the proxy. They are compiled into the binary. +- **Binds `0.0.0.0` only**, with no graceful drain on shutdown. +- **Startup depends on the network:** the catalog and every document it + lists are downloaded at each start. + +## What is not verified + +- Tested for this guide, against PostgreSQL 16 in Docker on macOS: `cargo + install`-equivalent release build, the Docker image (build, start, + `/healthz`, `/catalog`, non-root user), the three `sslmode` behaviours, the + pinned-catalog procedure, and `/healthz` during a database pause and + restart. That is all. +- **Not tried:** running the systemd unit (only `systemd-analyze verify` + was run on it), the Caddy and nginx configurations, a + PostgreSQL other than 16, a PostgreSQL behind TLS, and a private CA. +- **No provider connection was made** against a self-hosted proxy for this + guide. The per-platform registration steps are *declared* from the catalog + and the providers' documentation, not verified by a live connect. Which + platforms currently have live evidence is tracked in this repository's + [integrations/README.md](../integrations/README.md). +- [SECURITY.md](SECURITY.md#not-verified) lists what is unverified about the + protocol itself. For example, it states that no real client had signed + against the proxy yet. +- The Atomic Server option (atomic-server#1702) was unmerged when this was + written. The data-browser setting was described from the atomic-server + source on its `feat/plugin-debug` branch + (`browser/data-browser/src/components/Settings/IntegrationSettings.tsx`, + `browser/data-browser/src/helpers/integrationProxy.ts`, and this + repository's + [`integrations/localthought/browser.ts`](../integrations/localthought/browser.ts) + for the fallback default), not from a release. + +Questions and corrections: +[ontola/atomic-plugins issues](https://github.com/ontola/atomic-plugins/issues).