Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -432,6 +432,7 @@ jobs:
node --test \
integrations/tooling/apps.test.mjs \
integrations/tooling/certify.test.mjs \
integrations/tooling/deps.test.mjs \
integrations/tooling/dev-server.test.mjs \
integrations/tooling/lanes.test.mjs \
integrations/tooling/link-atomic-server.test.mjs \
Expand Down
9 changes: 9 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,15 @@ writes them. A file that is already on `main` is never changed or deleted;
CI's `apps.mjs check --published origin/main` enforces that. See
[Publishing a drive app](integrations/README.md#publishing-a-drive-app).

Drive app bundles are minified with esbuild, JS and CSS: `minify: true` in
`app/build.mjs`, and embedded CSS goes through
`esbuild.transform(css, { loader: 'css', minify: true })`. Each app's
build-test size limit (`app/build.test.ts`) is the measured minified size
plus about 10%, with a comment stating the measured size and the date. No
silent headroom: when a change needs a higher limit, re-measure and raise the
limit and its comment in the same commit. See
[Bundle size](integrations/README.md#bundle-size).

## Style notes for docs and code in `integrations/`

- Prose here is precise and hedged, not marketing copy: state exact limits
Expand Down
40 changes: 34 additions & 6 deletions integration-proxy/SELF_HOSTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -497,10 +497,35 @@ that it has no path, query, fragment or credentials. The server signs
requests to exactly that origin as the installed plugin's app agent. This
option comes from
[atomic-server#1702](https://github.com/ontola/atomic-server/pull/1702),
which was **not yet merged** when this was written. Check your Atomic
Server's `--help` for it. What the server does when the option is unset, and
whether that will default to atomic.place's proxy, is not settled by that
PR.
which is in the atomic-server commit this repository pins
(`.atomic-server-ref`) but was not yet merged upstream when this was
written. Check your Atomic Server's `--help` for it.

- **Unset, today:** with #1702, if neither `--integration-proxy-url` nor
`ATOMIC_INTEGRATION_PROXY_URL` is set, no proxy is configured, and plugins
cannot reach one. Set it to use any proxy at all.
- **Unset, later (planned, not yet released):** the default is planned to
become atomic.place's proxy, `https://integrations.atomic.place`, in the
same change that switches the data browser's build-time default (below)
away from `https://localthought.io`. From then on: if unset, the server
uses atomic.place's proxy (from version **TODO: fill in the atomic-server
release that ships this default**); set this flag to use your own.

Which addresses the proxy may be on: the server refuses outgoing requests to
private, loopback and link-local addresses, and before
[atomic-server#1731](https://github.com/ontola/atomic-server/pull/1731) it
exempted the proxy only when `--integration-proxy-url` was a literal
loopback address or `localhost`. With #1731, a URL whose origin (scheme,
host and port) equals exactly the configured proxy origin may resolve to a
loopback or private address (RFC 1918, CGNAT `100.64.0.0/10`, IPv6 ULA
`fc00::/7`, and their IPv4-mapped forms). That allows a proxy on your LAN
(`http://proxy.lan:8787`) or on the Docker host
(`http://host.docker.internal:8787`). Link-local addresses (including cloud
metadata endpoints such as `169.254.169.254`), unspecified and multicast
addresses are still refused, even for the proxy. Every other destination
keeps the full checks. #1731 was **open, and not in this repository's
pinned atomic-server commit**, when this was written; without it, use a
public address or a literal loopback address for the proxy.

**The data browser** (where users click *Connect*): open **Settings →
Integrations** and enter the proxy's origin (HTTPS, or loopback HTTP for
Expand Down Expand Up @@ -639,8 +664,11 @@ needs:
- [SECURITY.md](SECURITY.md#not-verified) lists what is unverified about the
protocol itself. For example, it states that no real client had signed
against the proxy yet.
- The Atomic Server option (atomic-server#1702) was unmerged when this was
written. The data-browser setting was described from the atomic-server
- The Atomic Server option (atomic-server#1702) was unmerged upstream when
this was written, though in this repository's pinned atomic-server commit;
the private-address rule (atomic-server#1731) was open and not in the pin.
The planned default (`https://integrations.atomic.place`) is not
released. The data-browser setting was described from the atomic-server
source on its `feat/plugin-debug` branch
(`browser/data-browser/src/components/Settings/IntegrationSettings.tsx`
and `browser/data-browser/src/helpers/integrationProxy.ts`), not from a
Expand Down
33 changes: 33 additions & 0 deletions integrations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,39 @@ node integrations/tooling/apps.mjs check --published origin/main
Builds pin esbuild's `absWorkingDir` to the repository root, so the bytes do
not depend on the directory the build ran from.

### Bundle size

The module is stored as a string property on a resource, so its size is
checked, not guessed at. The rule for every drive app:

- **Minify, JS and CSS.** `app/build.mjs` passes `minify: true` to
`esbuild.build`. CSS that the module embeds as a string (a `<style>` it
injects) is minified too, with
`(await esbuild.transform(css, { loader: 'css', minify: true })).code`,
using the same esbuild as the JS (`browser/`'s, from the pinned
atomic-server).
- **The size limit is the measured size plus about 10%.** The
`expect(bytes).toBeLessThan(...)` assertion in `app/build.test.ts` is the
minified `bytes` that `build()` returns, rounded up by about 10%, and the
line carries a comment stating the measured size and the date it was
measured, for example:

```ts
// Measured 41,212 bytes minified on 2026-09-24; limit is that plus ~10%.
expect(bytes).toBeLessThan(45_400);
```

- **No silent headroom.** A round-number ceiling far above the real size
(`160 * 1024` for a 40 KB module) hides growth until it is large. When a
change pushes the module past its limit, re-measure, and raise the limit
and the comment together in the same commit, so the growth is visible in
review. Lowering it after a size win follows the same rule.

Not every app follows this yet: at the time of writing, the app builds on
`main` do not pass `minify` and their tests use round ceilings (64 KiB to
160 KiB). An app moves to this rule the next time its build or its limit
changes; a change of the build output needs a new version (see above).

Pages itself is mutable: anyone who can push to `main` can change a file
there. The host's integrity check is what makes that safe. A module that no
longer matches the catalog's pin is refused, and nothing is installed or
Expand Down
8 changes: 8 additions & 0 deletions integrations/tooling/certify.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { resolve, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { createHash } from 'node:crypto';
import { spawnSync } from 'node:child_process';
import { installMissing, pluginDependencyDirs } from './deps.mjs';

export const root = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
export function bundleArguments(entry) {
Expand Down Expand Up @@ -181,6 +182,13 @@ export function certify({
packages = discover();
if (only && !packages.some(p => p.id === only))
throw new Error(`Unknown integration: ${only}`);
// Locally, a plugin that pins its own npm packages may not have them
// installed yet; CI already has, so this is a no-op there (deps.mjs).
installMissing(
packages
.filter(pkg => !only || pkg.id === only)
.flatMap(pkg => pluginDependencyDirs(pkg.id)),
);
} catch (e) {
writeFileSync(
resolve(output, 'report.json'),
Expand Down
90 changes: 90 additions & 0 deletions integrations/tooling/deps.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
/**
* Per-folder npm dependencies, installed on demand for local runs.
*
* A plugin that bundles published npm packages pins them in its own
* package.json and pnpm-lock.yaml (integrations/notion/, integrations/pets/,
* integrations/issue-tracker/app/, …), and a lane that declares shared
* `paths` in lanes.json imports that package from source (devonian/). CI
* installs every one of those lockfiles before certify.mjs and run-lane.mjs
* run ("Install plugin npm dependencies" in .github/workflows/ci.yml). A
* local checkout usually has not, and the resulting failure — a vitest or
* tsc "Cannot find module" — does not say what is missing.
*
* installMissing() runs `pnpm install --frozen-lockfile` in each given folder
* that has a pnpm-lock.yaml and no node_modules, printing one line per
* install. A folder that already has node_modules is left alone, so on CI,
* where the install step ran first, this does nothing: CI behaviour is
* unchanged. It never updates a lockfile (`--frozen-lockfile`), and it does
* not notice a node_modules that is present but stale — run
* `pnpm install --frozen-lockfile` in that folder by hand.
*/
import { spawnSync } from 'node:child_process';
import { existsSync } from 'node:fs';
import { resolve } from 'node:path';
import { root } from './lanes.mjs';

/**
* The folders, relative to `base`, that hold one plugin's own lockfiles: the
* plugin folder itself and its drive app, the same two globs CI installs
* (`integrations/*\/pnpm-lock.yaml`, `integrations/*\/app/pnpm-lock.yaml`).
*/
export const pluginDependencyDirs = id => [
`integrations/${id}`,
`integrations/${id}/app`,
];

/**
* The shared packages (devonian/, syncables/, reflector/) a lane's `paths`
* point into, e.g. `devonian/src/**` → `devonian`.
*/
export const sharedDependencyDirs = (paths = []) => [
...new Set(paths.map(path => path.split('/')[0])),
];

/** Those of `dirs` that have a pnpm-lock.yaml but no node_modules yet. */
export function missingInstalls(dirs, base = root) {
return [...new Set(dirs)].filter(
dir =>
existsSync(resolve(base, dir, 'pnpm-lock.yaml')) &&
!existsSync(resolve(base, dir, 'node_modules')),
);
}

function pnpmInstall(cwd) {
return spawnSync('pnpm', ['install', '--frozen-lockfile'], {
cwd,
encoding: 'utf8',
maxBuffer: 32 * 1024 * 1024,
});
}

/**
* Install each of `dirs` that needs it. Returns the folders it installed.
* Throws on the first failed install, with pnpm's output in the message.
*/
export function installMissing(
dirs,
{ base = root, install = pnpmInstall, log = console.log } = {},
) {
const installed = [];

for (const dir of missingInstalls(dirs, base)) {
log(`Installing ${dir} dependencies (pnpm install --frozen-lockfile)`);
const result = install(resolve(base, dir));

if (result.error || result.status !== 0) {
const output = [result.error?.message, result.stdout, result.stderr]
.filter(Boolean)
.join('\n')
.trim();

throw new Error(
`pnpm install --frozen-lockfile failed in ${dir}${output ? `:\n${output}` : ''}`,
);
}

installed.push(dir);
}

return installed;
}
Loading
Loading