Skip to content

Publish integration catalog and make private-home recovery optional - #1527

Merged
joepio merged 28 commits into
developfrom
feat/plugin-catalog
Sep 22, 2026
Merged

joepio merged 28 commits into
developfrom
feat/plugin-catalog

Conversation

@michielbdejong

@michielbdejong michielbdejong commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Adds a published integration catalog and serves integration bundles as static server assets. Catalog visibility and plugin toggles control discovery, with preferences saved in the background.

Release and E2E builds now include the catalog and bundles. Bundle loaders use the connected Atomic server, isolate their caches by server URL, and retry after failed network, HTTP or response-body reads.

Also resolves the inherited sign-in regressions exposed by the current smoke suite: an account without recoverable data receives a writable derived private home, including when restoring a persisted session through a direct link. Connecting another device or restoring a backup is an optional nudge. Existing data is read/recovered first; unrelated workspace links are never synthesized. Sync no longer claims that an unreadable workspace is cached locally or known to live on another device. Explicit sign-out and device lock take precedence over the account-settings unlock guard.

Validation:

  • 1,192 frontend unit tests; frontend typecheck.
  • Four Dagger build-input regression tests.
  • Five Chromium regressions pass: new and legacy secrets, persisted-session direct links, document creation/reload persistence, truthful missing-workspace status, and sign-out returning to welcome after reload.
  • Translation catalogs settled through Vite and reviewed.

The local browser check uses isolated services and existing local server/WASM binaries. CI builds and tests this PR's full source. Flutter's separate sign-in flow is unchanged.

@michielbdejong

Copy link
Copy Markdown
Contributor Author

`integrations/catalog.json` (added on this branch, forked from #1307's `feat/plugin-model` at plugin model improvements (#1482)) is a temporary gate: it just toggles `experimental` visibility for the hardcoded `BundledIntegration[]` list in `IntegrationDiscovery.tsx` (GitHub, Google Calendar, MT940, Clockify, Notion).

#1307 already introduces the real long-term mechanism for this: the package/release model behind `/plugin-catalog` (`server/src/handlers/plugin_release.rs`, `CatalogEntry` in `lib/src/db/plugin_release.rs`, rendered as "Community plugins" in `IntegrationStore.tsx`). Its own remaining-work list calls the bundled provider-specific setup adapters "a migration gap, not the intended extension mechanism" and has "removal of bundled provider installer adapters" as unfinished work.

So the plan is: these bundled integrations get published through `/plugin-catalog` like any other package/release, and `catalog.json` + the hardcoded `BundledIntegration[]` gating go away once that migration lands.

@michielbdejong

Copy link
Copy Markdown
Contributor Author

Yeah and for now I want to see a clean shippable atomic-server UI that has just the Pets plugin as an experimental bundled plugin available. I will then do per-plugin feature branches to get each of them to a quality level where we think they are shippable as bundled plugins. And bundled here means they are in this monorepo but not built in the server build or in the data-browser build; their are loaded into the running server over https. For now, they are shipped in atomic-server, but as assets that are hosted as same-host statics, and they are still tied to the server's release cycle. In the future we can keep them in this monorepo or not, but stop shipping them as static assets, and instead host them somewhere. That way we can make them available to all running instances, untied from the server's release cycle. And then the catalog.json also becomes the planned database version.

@michielbdejong
michielbdejong changed the base branch from feat/plugin-model to develop September 18, 2026 06:03
michielbdejong and others added 12 commits September 18, 2026 11:46
…1526)

* Serve integration plugins as static assets instead of bundling them

Plugin sources under integrations/*/plugin.js were pulled into the
data-browser's own JS bundle at build time via Vite `?raw` imports. Embed
them into the server binary separately instead (mirroring how the SPA/logo
assets are already embedded) and serve them at /integrations, so the browser
fetches a plugin's source at install time rather than shipping it inline.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Document fresh-worktree build setup in AGENTS.md

A new worktree has no installed browser/node_modules and no built dist/
for the internal JS packages, since neither is git-tracked. Skipping a
full build there surfaces as a confusing pre-commit failure instead: the
Rust step's cargo clippy runs against an isolated snapshot of just staged,
tracked files, so server/build.rs has to run its own pnpm install/build
from scratch to embed the frontend, and any gaps in this worktree's own
browser/ setup fail that inner build with errors that look unrelated to
the actual change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Michiel de Jong <michielbdejong@ontola.io>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Add integrations/catalog.json, a JSON-AD file listing each bundled
integration id with an experimental flag. visibleBundledIntegrations now
shows only non-experimental entries by default, and shows every entry in
the folder once a visitor enables "Show experimental plugins".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The label text was passed through a `label` prop that got resolved via
a build-time localization index which was never populated for this
call site, so it rendered empty. The checkboxes were also wrapped in
`!showApiPlugins`/`!showExperimentalPlugins` conditions, hiding them
entirely as soon as they were checked and the plugin cards loaded,
leaving no way to turn them back off.

Inline the checkbox markup with literal label text, matching the
working pattern in Settings, and bind `checked` to the real state so
the controls stay visible and reflect their actual value.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ndled list (#1536)

LocalThoughtCatalog rendered a card for every platform the proxy's
.catalog() call returned, with no relation to integrations/catalog.json's
experimental gate — that's how uncertified platforms like Moneybird and
Todoist were reachable as soon as a visitor checked "Show API plugins",
bypassing "Show experimental plugins" entirely. It now filters through the
same isExperimental() check (extracted into a shared pluginCatalog.ts),
threaded down via a new showExperimentalPlugins prop.

catalog.json's entries are emptied out, so every bundled and raw-proxy
integration defaults to experimental until it's explicitly re-certified.
Moneybird, Todoist and Pets get explicit (experimental) entries so the
catalog documents every platform the proxy can surface, not just the
curated five.

Also adds an empty-state message to the Integrations page for when
nothing is available to discover, and updates integration-visibility.spec
for the now-fixed gating.

Co-authored-by: Michiel de Jong <michielbdejong@ontola.io>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Gates the devonian-google-calendar integration (IntegrationDiscovery.tsx)
behind the experimental plugin toggle, alongside pets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ar entry

did:ad:devonian-google-calendar is not a cryptographically-derived genesis
DID (catalog.json is hand-authored, not signed) but it's syntactically
valid and, unlike the https://atomicdata.dev/... form used elsewhere in
this file, doesn't imply a fetchable HTTP location that 404s.

pluginUrl points at the actual source (the GitHub-pinned devonian
dependency in browser/data-browser/package.json) for entries, like this
one, that aren't a local package.json in this folder.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…alendar entry

did:ad: implied a cryptographic identity this hand-authored file can't
back (see lib/src/genesis.rs verify()), and this codebase doesn't
implement did:uuid at all. localId is the actual Atomic Data mechanism
for a resource that has no URL yet: a caller-chosen slug, unique within
its future parent, that the importer turns into a real subject on
import (see lib/src/parse.rs ParseOpts.importer).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Keeps both PluginCatalogEntry instances on the same addressing scheme
now that devonian-google-calendar uses localId instead of a fabricated
@id.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
isExperimental() defaulted a missing catalog.json id to experimental,
so every bundled integration was reachable via 'Show experimental
plugins' regardless of whether the catalog documented it. Replace it
with isCatalogVisible(): an id with no catalog.json entry is now dark
matter, shipped in the bundle but never offered by either toggle. A
cataloged id keeps the existing experimental/toggle behavior.

catalog.json stays at its current single 'pets' entry, so the five
curated bundled integrations (devonian-github-issues,
devonian-google-calendar, mt940, clockify, notion) are dark matter
until they're explicitly added back to the catalog.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…fetch it at runtime

Moves the bundled integrations' card copy (name, icon, description,
capabilities, events, limitation, keywords, requiresApiPlugins,
platform, callbackPlatform) out of the hardcoded bundledIntegrations()
array in IntegrationDiscovery.tsx and into integrations/catalog.json as
proper Atomic Data properties. The one field that can't be data — the
live LocalThought extension module reference for Google Calendar —
stays in code as a small id-keyed lookup.

Adds an explicit `enabled` property: false for all five curated
bundled integrations, true for `pets`. isCatalogVisible() now checks
`enabled` before the experimental/toggle rule, so disabling an entry
is explicit rather than relying only on its absence from the file.

catalog.json is no longer statically imported into the SPA bundle.
server/build.rs now embeds it alongside the plugin.js bundles at
/integrations/catalog.json, and the data-browser fetches it at runtime
from store.getServerUrl() — the connected server, not window.location
— since a Tauri desktop/mobile build ships a separate frontend that
can pair with any server and doesn't run at the server's own origin.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ConnectPets card copy and its dispatch branch in IntegrationDiscovery
were dropped in an earlier squash-merge, leaving the catalog.json
"pets" entry without the name/icon/description fields
toBundledIntegration() requires. The Pets card never rendered, even
with "Show experimental plugins" enabled.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…background

The two discovery checkboxes on /app/integrations only decide what the page
lists, but they were disabled until the private drive resolved and again for
the whole round trip of every save. On a server that is still starting that
means the checkboxes are dead for seconds, and a click that does land takes
until the write returns to show anything.

Make them local-first instead: a toggle updates the displayed value straight
away, is cached per agent in localStorage so the last choice shows on the next
cold start, and is queued for a write that flushes once the drive is readable.
The drive stays the source of truth -- its value takes over again as soon as a
key is confirmed saved -- and a failed write keeps the choice applied while
surfacing the error, since nothing but the listing depends on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017tcPgGBeX66DMYfhApUkgS
@michielbdejong

Copy link
Copy Markdown
Contributor Author

I reduced this to the smaller scope so it becomes reviewable. This PR puts the bundled plugins into the server as static assets that the data browser fetches from its own (local)host over http. There is a catalog.json where we can disable plugins that are not ready for prime time yet. Only the Pets demo plugin is enabled by default.

Michiel de Jong and others added 2 commits September 18, 2026 12:15
A reconciling re-run (e.g. Pets) legitimately yields 0 changes; showing a
disabled "Apply 0 changes" button read as a bug. Render only "Close" in
that case and update the e2e specs that asserted the disabled button.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only show 'Show API plugins' when the catalog enables an entry that
requires API plugins, and 'Show experimental plugins' when it enables
an experimental one. Reword the empty discovery state to
'No plugins to show here.'

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An import whose config never reached the run died with "run() threw:
TypeError: Cannot destructure property 'table' of 'ctx.config' as it is
undefined", and the import paused on a message that named an internal
field rather than anything a user could act on.

The cause was on the host side: prepareRun() — the one path behind
preview, a manual run and the assistant's runs alike — assembled the
sandbox input as `{ trigger, schemas }` and never passed `config` at
all. The install wizards (ConnectPets, ImportMT940) pass it when they
preview right after writing it, so an import worked once and then
crashed on every later run from the plugin page.

Host:

- `pluginConfigFor()` reads the config from wherever the installation
  stored it — `plugin-schemas` under the key the plugin declares, or
  flat, or the `config` a `plugin-connection` carries — parsing it when
  it was stored as a JSON string, and always returning an object.
- `pluginConfigProblems()` checks it against the plugin's declaration
  before the sandbox starts, so a missing field pauses the import on
  "This import is missing required config field `table`" instead of on
  whatever run() throws.
- prepareRun() uses both, and passes the same `config` to the Worker and
  to the server-side runtime, so preview and run cannot disagree.

Plugins: `manifest.config` is a new optional declaration (key,
properties, required), validated by validateManifest and carried
through the server's manifest parser. Pets and MT940 declare theirs;
Pets, MT940 and LocalThought also guard `ctx.config` themselves and
name the fields they are missing, so a plugin run by an older host
still fails readably.

Regression tests cover the missing-config case end to end: the config
builder, the validation, a manifest declaration, the pets plugin run
with no config at all, and a run() that throws rather than TypeErrors.

integrations/evidence.json is regenerated from a full certification run
for the rebuilt bundles; it also picks up a stale github-issues hash and
the pets entry that was never recorded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XE56taqaP5JQb3y3ubWF8U
Michiel de Jong and others added 4 commits September 22, 2026 09:53
The staged-snapshot hook rebuilt the entire workspace on every commit, and
worse, ran the full JS build first: the snapshot has no `dist` and no
`assets_tmp`, so server/build.rs ran pnpm install, vite and wasm-pack plus
the wasm32-wasip2 plugin runtime — none of which Clippy needs. Skip both
the way CI's rustChecksContainer does, with the same assets_tmp stub.

Cargo fingerprints path packages by their path and detects staleness by
mtime, so a fresh mkdtemp snapshot per commit could never reuse anything.
The snapshot now lives at a fixed target/pre-commit/snapshot and is synced
incrementally: the hook records the `git write-tree` OID it last checked
out and only re-checks-out / deletes the paths `git diff-tree` reports
against the current index. Partial staging works as before.

Warm runs go from a full workspace build to ~6-10s for a one-crate edit
and ~2s when nothing changed.

Also fixes a pre-existing clippy::unnecessary_to_owned in lib's website
tests, which the -D warnings hook blocked on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Worktrees commonly share one CARGO_TARGET_DIR, so a single
target/pre-commit/snapshot would be raced by two of them committing at
once. Put each checkout's snapshot under a hash of its root path.

Git does not hold index.lock while pre-commit runs, so two commits in the
same checkout can also overlap; Cargo's target-dir lock covers the build
but not the snapshot sync. A pid lock file makes the hook wait for the
other run, and a lock left by a dead process is reclaimed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s stop there (#1607)

The persistent snapshot this PR introduces lives under target/, which the
repo's own .gitignore excludes. With no .git of its own, oxlint's file
walker searches upward for a repository boundary, finds the real repo's
.git, and applies its /target ignore rule to the whole snapshot -- every
file under it reads as gitignored, so oxlint reports "No files found to
lint" for every package, even ones the commit never touched.

Verified with git check-ignore (matches .gitignore:1:/target) and by
adding this same empty .git marker by hand: lint immediately finds and
checks all files again (lib: 212 files, data-browser: 1225 files, etc.),
same as before this PR moved the snapshot inside target/.


Claude-Session: https://claude.ai/code/session_01AHZRWHyQNxcw5fbTUSm3Hm

Co-authored-by: Claude <noreply@anthropic.com>
Brings in the CI job split (#1550) and the rest of develop, so this
branch's checks run against a current base.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XE56taqaP5JQb3y3ubWF8U
@michielbdejong
michielbdejong marked this pull request as ready for review September 22, 2026 09:13
claude and others added 2 commits September 22, 2026 10:31
`lib lint` fails on the current head with three eslint(no-shadow)
errors in plugin-manifest.ts. The config declaration I added binds
`declared` at function scope in validateManifest; the manifest v2 work
that landed since then binds `declared` again inside the network,
capabilities and entrypoints blocks, and each of those now shadows
mine.

`declared` is the established local idiom in those blocks, so the
function-scope binding is the one that should move: it becomes
`declaredConfig`, in its own block and in both the v1 and v2 return
spreads. No behaviour changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XE56taqaP5JQb3y3ubWF8U
`embed_integrations` removes `server/integrations_assets_tmp` before
re-populating it, but the fallback that creates an empty one sat in an
`else if` reached only when `../integrations` is absent entirely.

Several Dagger containers (`rustBuild`, `rustBuildSlim`) mount just the
`integrations/localthought/syncables` subtree, because that is the only
part of `integrations/` a Rust crate depends on and mounting the whole
tree would invalidate the Rust layer on every front-end edit. There
`../integrations` *does* exist, so the copy loop runs, matches no
`plugin.js` and no top-level `catalog.json`, and leaves the destination
it just deleted missing — `static_files::resource_dir` then panics with
"failed to open integrations assets from ./integrations_assets_tmp".

Make the empty-set fallback unconditional. Verified by reproducing the
CI mount layout locally: panics before, builds after, and a full tree
still embeds all six plugin bundles plus `catalog.json`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@michielbdejong

Copy link
Copy Markdown
Contributor Author

Ready for review
Screenshot 2026-09-22 at 14 08 54

@michielbdejong

Copy link
Copy Markdown
Contributor Author

Not sure why the main pipeline was not triggered by that last commit, triggered it manually

@gitguardian

gitguardian Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@joepio
joepio force-pushed the feat/plugin-catalog branch from 894c655 to e65dbe5 Compare September 22, 2026 15:18
…w-fixes

# Conflicts:
#	TESTING_COVERAGE.md
#	browser/data-browser/src/helpers/syncPresentation.ts
#	browser/data-browser/src/locales/de.po
#	browser/data-browser/src/locales/en.po
#	browser/data-browser/src/locales/es.po
#	browser/data-browser/src/locales/fr.po
@joepio joepio changed the title plugin catalog Publish integration catalog and make private-home recovery optional Sep 22, 2026
@joepio
joepio merged commit a7a6075 into develop Sep 22, 2026
9 checks passed
@joepio
joepio deleted the feat/plugin-catalog branch September 22, 2026 18:47
michielbdejong pushed a commit that referenced this pull request Sep 23, 2026
Brings in the four commits #1527 gained after this branch forked from it.
f2dd42f served plugin bundles from the connected server's /integrations;
this branch removes that route, so its helper, Dagger mount and regression
test are dropped here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michielbdejong pushed a commit that referenced this pull request Sep 23, 2026
Its tree is exactly d165424 merged with 756741d, both already merged
above. Recording it lets the develop merge that follows use it as base
instead of re-adding what this branch removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michielbdejong pushed a commit that referenced this pull request Sep 23, 2026
Merged with #1527's squash recorded as the base, so develop's copy of
the plugin-specific code this branch removes is not re-added. Plugin
bundles and evidence changed by #1585 are deleted here: they live in
atomic-plugins. Catalogs settled with pnpm clean-translations; no
translated string that still has a source was dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants