Repository navigation
Publish integration catalog and make private-home recovery optional - #1527
Conversation
|
`integrations/catalog.json` (added on this branch, forked from #1307's `feat/plugin-model` at plugin model improvements (#1482)) is a temporary gate: it just toggles `experimental` visibility for the hardcoded `BundledIntegration[]` list in `IntegrationDiscovery.tsx` (GitHub, Google Calendar, MT940, Clockify, Notion). #1307 already introduces the real long-term mechanism for this: the package/release model behind `/plugin-catalog` (`server/src/handlers/plugin_release.rs`, `CatalogEntry` in `lib/src/db/plugin_release.rs`, rendered as "Community plugins" in `IntegrationStore.tsx`). Its own remaining-work list calls the bundled provider-specific setup adapters "a migration gap, not the intended extension mechanism" and has "removal of bundled provider installer adapters" as unfinished work. So the plan is: these bundled integrations get published through `/plugin-catalog` like any other package/release, and `catalog.json` + the hardcoded `BundledIntegration[]` gating go away once that migration lands. |
|
Yeah and for now I want to see a clean shippable atomic-server UI that has just the Pets plugin as an experimental bundled plugin available. I will then do per-plugin feature branches to get each of them to a quality level where we think they are shippable as bundled plugins. And bundled here means they are in this monorepo but not built in the server build or in the data-browser build; their are loaded into the running server over https. For now, they are shipped in atomic-server, but as assets that are hosted as same-host statics, and they are still tied to the server's release cycle. In the future we can keep them in this monorepo or not, but stop shipping them as static assets, and instead host them somewhere. That way we can make them available to all running instances, untied from the server's release cycle. And then the catalog.json also becomes the planned database version. |
d1ff226 to
cba53ae
Compare
…1526) * Serve integration plugins as static assets instead of bundling them Plugin sources under integrations/*/plugin.js were pulled into the data-browser's own JS bundle at build time via Vite `?raw` imports. Embed them into the server binary separately instead (mirroring how the SPA/logo assets are already embedded) and serve them at /integrations, so the browser fetches a plugin's source at install time rather than shipping it inline. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Document fresh-worktree build setup in AGENTS.md A new worktree has no installed browser/node_modules and no built dist/ for the internal JS packages, since neither is git-tracked. Skipping a full build there surfaces as a confusing pre-commit failure instead: the Rust step's cargo clippy runs against an isolated snapshot of just staged, tracked files, so server/build.rs has to run its own pnpm install/build from scratch to embed the frontend, and any gaps in this worktree's own browser/ setup fail that inner build with errors that look unrelated to the actual change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Michiel de Jong <michielbdejong@ontola.io> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Add integrations/catalog.json, a JSON-AD file listing each bundled integration id with an experimental flag. visibleBundledIntegrations now shows only non-experimental entries by default, and shows every entry in the folder once a visitor enables "Show experimental plugins". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The label text was passed through a `label` prop that got resolved via a build-time localization index which was never populated for this call site, so it rendered empty. The checkboxes were also wrapped in `!showApiPlugins`/`!showExperimentalPlugins` conditions, hiding them entirely as soon as they were checked and the plugin cards loaded, leaving no way to turn them back off. Inline the checkbox markup with literal label text, matching the working pattern in Settings, and bind `checked` to the real state so the controls stay visible and reflect their actual value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ndled list (#1536) LocalThoughtCatalog rendered a card for every platform the proxy's .catalog() call returned, with no relation to integrations/catalog.json's experimental gate — that's how uncertified platforms like Moneybird and Todoist were reachable as soon as a visitor checked "Show API plugins", bypassing "Show experimental plugins" entirely. It now filters through the same isExperimental() check (extracted into a shared pluginCatalog.ts), threaded down via a new showExperimentalPlugins prop. catalog.json's entries are emptied out, so every bundled and raw-proxy integration defaults to experimental until it's explicitly re-certified. Moneybird, Todoist and Pets get explicit (experimental) entries so the catalog documents every platform the proxy can surface, not just the curated five. Also adds an empty-state message to the Integrations page for when nothing is available to discover, and updates integration-visibility.spec for the now-fixed gating. Co-authored-by: Michiel de Jong <michielbdejong@ontola.io> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Gates the devonian-google-calendar integration (IntegrationDiscovery.tsx) behind the experimental plugin toggle, alongside pets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ar entry did:ad:devonian-google-calendar is not a cryptographically-derived genesis DID (catalog.json is hand-authored, not signed) but it's syntactically valid and, unlike the https://atomicdata.dev/... form used elsewhere in this file, doesn't imply a fetchable HTTP location that 404s. pluginUrl points at the actual source (the GitHub-pinned devonian dependency in browser/data-browser/package.json) for entries, like this one, that aren't a local package.json in this folder. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…alendar entry did:ad: implied a cryptographic identity this hand-authored file can't back (see lib/src/genesis.rs verify()), and this codebase doesn't implement did:uuid at all. localId is the actual Atomic Data mechanism for a resource that has no URL yet: a caller-chosen slug, unique within its future parent, that the importer turns into a real subject on import (see lib/src/parse.rs ParseOpts.importer). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Keeps both PluginCatalogEntry instances on the same addressing scheme now that devonian-google-calendar uses localId instead of a fabricated @id. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
isExperimental() defaulted a missing catalog.json id to experimental, so every bundled integration was reachable via 'Show experimental plugins' regardless of whether the catalog documented it. Replace it with isCatalogVisible(): an id with no catalog.json entry is now dark matter, shipped in the bundle but never offered by either toggle. A cataloged id keeps the existing experimental/toggle behavior. catalog.json stays at its current single 'pets' entry, so the five curated bundled integrations (devonian-github-issues, devonian-google-calendar, mt940, clockify, notion) are dark matter until they're explicitly added back to the catalog. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…fetch it at runtime Moves the bundled integrations' card copy (name, icon, description, capabilities, events, limitation, keywords, requiresApiPlugins, platform, callbackPlatform) out of the hardcoded bundledIntegrations() array in IntegrationDiscovery.tsx and into integrations/catalog.json as proper Atomic Data properties. The one field that can't be data — the live LocalThought extension module reference for Google Calendar — stays in code as a small id-keyed lookup. Adds an explicit `enabled` property: false for all five curated bundled integrations, true for `pets`. isCatalogVisible() now checks `enabled` before the experimental/toggle rule, so disabling an entry is explicit rather than relying only on its absence from the file. catalog.json is no longer statically imported into the SPA bundle. server/build.rs now embeds it alongside the plugin.js bundles at /integrations/catalog.json, and the data-browser fetches it at runtime from store.getServerUrl() — the connected server, not window.location — since a Tauri desktop/mobile build ships a separate frontend that can pair with any server and doesn't run at the server's own origin. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ConnectPets card copy and its dispatch branch in IntegrationDiscovery were dropped in an earlier squash-merge, leaving the catalog.json "pets" entry without the name/icon/description fields toBundledIntegration() requires. The Pets card never rendered, even with "Show experimental plugins" enabled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…background The two discovery checkboxes on /app/integrations only decide what the page lists, but they were disabled until the private drive resolved and again for the whole round trip of every save. On a server that is still starting that means the checkboxes are dead for seconds, and a click that does land takes until the write returns to show anything. Make them local-first instead: a toggle updates the displayed value straight away, is cached per agent in localStorage so the last choice shows on the next cold start, and is queued for a write that flushes once the drive is readable. The drive stays the source of truth -- its value takes over again as soon as a key is confirmed saved -- and a failed write keeps the choice applied while surfacing the error, since nothing but the listing depends on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017tcPgGBeX66DMYfhApUkgS
bd53e82 to
c9a2349
Compare
|
I reduced this to the smaller scope so it becomes reviewable. This PR puts the bundled plugins into the server as static assets that the data browser fetches from its own (local)host over http. There is a catalog.json where we can disable plugins that are not ready for prime time yet. Only the Pets demo plugin is enabled by default. |
A reconciling re-run (e.g. Pets) legitimately yields 0 changes; showing a disabled "Apply 0 changes" button read as a bug. Render only "Close" in that case and update the e2e specs that asserted the disabled button. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only show 'Show API plugins' when the catalog enables an entry that requires API plugins, and 'Show experimental plugins' when it enables an experimental one. Reword the empty discovery state to 'No plugins to show here.' Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An import whose config never reached the run died with "run() threw:
TypeError: Cannot destructure property 'table' of 'ctx.config' as it is
undefined", and the import paused on a message that named an internal
field rather than anything a user could act on.
The cause was on the host side: prepareRun() — the one path behind
preview, a manual run and the assistant's runs alike — assembled the
sandbox input as `{ trigger, schemas }` and never passed `config` at
all. The install wizards (ConnectPets, ImportMT940) pass it when they
preview right after writing it, so an import worked once and then
crashed on every later run from the plugin page.
Host:
- `pluginConfigFor()` reads the config from wherever the installation
stored it — `plugin-schemas` under the key the plugin declares, or
flat, or the `config` a `plugin-connection` carries — parsing it when
it was stored as a JSON string, and always returning an object.
- `pluginConfigProblems()` checks it against the plugin's declaration
before the sandbox starts, so a missing field pauses the import on
"This import is missing required config field `table`" instead of on
whatever run() throws.
- prepareRun() uses both, and passes the same `config` to the Worker and
to the server-side runtime, so preview and run cannot disagree.
Plugins: `manifest.config` is a new optional declaration (key,
properties, required), validated by validateManifest and carried
through the server's manifest parser. Pets and MT940 declare theirs;
Pets, MT940 and LocalThought also guard `ctx.config` themselves and
name the fields they are missing, so a plugin run by an older host
still fails readably.
Regression tests cover the missing-config case end to end: the config
builder, the validation, a manifest declaration, the pets plugin run
with no config at all, and a run() that throws rather than TypeErrors.
integrations/evidence.json is regenerated from a full certification run
for the rebuilt bundles; it also picks up a stale github-issues hash and
the pets entry that was never recorded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XE56taqaP5JQb3y3ubWF8U
The staged-snapshot hook rebuilt the entire workspace on every commit, and worse, ran the full JS build first: the snapshot has no `dist` and no `assets_tmp`, so server/build.rs ran pnpm install, vite and wasm-pack plus the wasm32-wasip2 plugin runtime — none of which Clippy needs. Skip both the way CI's rustChecksContainer does, with the same assets_tmp stub. Cargo fingerprints path packages by their path and detects staleness by mtime, so a fresh mkdtemp snapshot per commit could never reuse anything. The snapshot now lives at a fixed target/pre-commit/snapshot and is synced incrementally: the hook records the `git write-tree` OID it last checked out and only re-checks-out / deletes the paths `git diff-tree` reports against the current index. Partial staging works as before. Warm runs go from a full workspace build to ~6-10s for a one-crate edit and ~2s when nothing changed. Also fixes a pre-existing clippy::unnecessary_to_owned in lib's website tests, which the -D warnings hook blocked on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Worktrees commonly share one CARGO_TARGET_DIR, so a single target/pre-commit/snapshot would be raced by two of them committing at once. Put each checkout's snapshot under a hash of its root path. Git does not hold index.lock while pre-commit runs, so two commits in the same checkout can also overlap; Cargo's target-dir lock covers the build but not the snapshot sync. A pid lock file makes the hook wait for the other run, and a lock left by a dead process is reclaimed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s stop there (#1607) The persistent snapshot this PR introduces lives under target/, which the repo's own .gitignore excludes. With no .git of its own, oxlint's file walker searches upward for a repository boundary, finds the real repo's .git, and applies its /target ignore rule to the whole snapshot -- every file under it reads as gitignored, so oxlint reports "No files found to lint" for every package, even ones the commit never touched. Verified with git check-ignore (matches .gitignore:1:/target) and by adding this same empty .git marker by hand: lint immediately finds and checks all files again (lib: 212 files, data-browser: 1225 files, etc.), same as before this PR moved the snapshot inside target/. Claude-Session: https://claude.ai/code/session_01AHZRWHyQNxcw5fbTUSm3Hm Co-authored-by: Claude <noreply@anthropic.com>
Brings in the CI job split (#1550) and the rest of develop, so this branch's checks run against a current base. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XE56taqaP5JQb3y3ubWF8U
0b5d7ca to
51cd69b
Compare
`lib lint` fails on the current head with three eslint(no-shadow) errors in plugin-manifest.ts. The config declaration I added binds `declared` at function scope in validateManifest; the manifest v2 work that landed since then binds `declared` again inside the network, capabilities and entrypoints blocks, and each of those now shadows mine. `declared` is the established local idiom in those blocks, so the function-scope binding is the one that should move: it becomes `declaredConfig`, in its own block and in both the v1 and v2 return spreads. No behaviour changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XE56taqaP5JQb3y3ubWF8U
`embed_integrations` removes `server/integrations_assets_tmp` before re-populating it, but the fallback that creates an empty one sat in an `else if` reached only when `../integrations` is absent entirely. Several Dagger containers (`rustBuild`, `rustBuildSlim`) mount just the `integrations/localthought/syncables` subtree, because that is the only part of `integrations/` a Rust crate depends on and mounting the whole tree would invalidate the Rust layer on every front-end edit. There `../integrations` *does* exist, so the copy loop runs, matches no `plugin.js` and no top-level `catalog.json`, and leaves the destination it just deleted missing — `static_files::resource_dir` then panics with "failed to open integrations assets from ./integrations_assets_tmp". Make the empty-set fallback unconditional. Verified by reproducing the CI mount layout locally: panics before, builds after, and a full tree still embeds all six plugin bundles plus `catalog.json`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Not sure why the main pipeline was not triggered by that last commit, triggered it manually |
️✅ There are no secrets present in this pull request anymore.If these secrets were true positive and are still valid, we highly recommend you to revoke them. 🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request. |
…w-fixes # Conflicts: # TESTING_COVERAGE.md
894c655 to
e65dbe5
Compare
…w-fixes # Conflicts: # TESTING_COVERAGE.md # browser/data-browser/src/helpers/syncPresentation.ts # browser/data-browser/src/locales/de.po # browser/data-browser/src/locales/en.po # browser/data-browser/src/locales/es.po # browser/data-browser/src/locales/fr.po
Brings in the four commits #1527 gained after this branch forked from it. f2dd42f served plugin bundles from the connected server's /integrations; this branch removes that route, so its helper, Dagger mount and regression test are dropped here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged with #1527's squash recorded as the base, so develop's copy of the plugin-specific code this branch removes is not re-added. Plugin bundles and evidence changed by #1585 are deleted here: they live in atomic-plugins. Catalogs settled with pnpm clean-translations; no translated string that still has a source was dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Adds a published integration catalog and serves integration bundles as static server assets. Catalog visibility and plugin toggles control discovery, with preferences saved in the background.
Release and E2E builds now include the catalog and bundles. Bundle loaders use the connected Atomic server, isolate their caches by server URL, and retry after failed network, HTTP or response-body reads.
Also resolves the inherited sign-in regressions exposed by the current smoke suite: an account without recoverable data receives a writable derived private home, including when restoring a persisted session through a direct link. Connecting another device or restoring a backup is an optional nudge. Existing data is read/recovered first; unrelated workspace links are never synthesized. Sync no longer claims that an unreadable workspace is cached locally or known to live on another device. Explicit sign-out and device lock take precedence over the account-settings unlock guard.
Validation:
The local browser check uses isolated services and existing local server/WASM binaries. CI builds and tests this PR's full source. Flutter's separate sign-in flow is unchanged.