Skip to content

Handoff: Reflector/devonian architecture for retiring reflector-rs (supersedes ontola/atomic-server#1599 discussion) #6

Closed
localthought/reflector
#58

Description

@michielbdejong

Context

ontola/atomic-server#1599
asked to move plugin sync (syncables + devonian) into a TS reflector package
and retire the Rust syncables/reflector-rs crate + wasm/src/integrations.rs
so atomic-server has no code specific to any individual plugin or sync platform.

That issue's own "open questions for implementation" needed deciding before any
code moves. This issue is the handoff record of those decisions — for whoever
picks up the actual migration work across localthought/reflector,
localthought/syncables, the new localthought/devonian, and this repo
(atomic-plugins).

A companion docs-only PR recording the atomic-server-side implications is at
ontola/atomic-server#1604
(updates planning/extension-architecture.md and planning/api-plugins.md).

Key finding: reflector already exists and is further along than the issue assumed

localthought/reflector is not a blank slate. It's a working, deployed TS app
(Express server, OAuth/PKCE handshake, Postgres/file/remoteStorage persistence,
Heroku/DO/Docker deploy configs) that already does most of what #1599 proposed:

  • Generic OpenAPI+Overlays-driven sync via localthought/syncables, live today
    against Google Calendar.
  • A working bidirectional reflection engine between two endpoints (id-map,
    hidden origin markers, reflect loop) — see its README's "Reflecting between
    two systems" section. This does the same job the issue assigns to devonian.

Also, atomic-server's planning/api-plugins.md documents a third,
independently-built mechanism already live for LocalThought platforms: a
browser/WASM + integration-proxy path (codex/browser-integrations,
codex/localthought-api-plugins) where the WASM bundle parses the OpenAPI
catalog and does pagination client-side, with the browser owning a signed
TENANT_SECRET handoff and rotating credentials against an integration-proxy
service. This is live-verified against real GitHub OAuth (29 records) and
Google Calendar OAuth (54 records) already. It's built on the same Rust
syncables crate #1599 wants retired — but it isn't legacy, it's the current
active path for those platforms.

Decisions (walked through with @michielbdejong, 2026-09-21)

  1. Reflector's repo: localthought/reflector (the existing app above), not
    a new repo and not folded into atomic-plugins.

  2. How a plugin actually uses reflector: confirmed against
    plugin-runtime/wit/plugin-runtime.wit — a plugin has no sockets, only a
    host-mediated fetch (origin-checked against the manifest, secret:<name>
    substitution for credentials). It cannot receive an OAuth redirect or run a
    background process. So reflector stays a separate deployed service; a
    plugin calls a self-hosted reflector instance's REST API
    (/api/reflect, /api/reflect/status, etc.) through the sandboxed fetch,
    exactly like any other third-party API (MT940/Notion/GitHub-Issues already
    do this). No new host capability needed — this matches what #1599 proposed,
    it just means reflector is called, not embedded.

  3. Reflector supersedes the browser/WASM/integration-proxy path for
    LocalThought platforms going forward — it isn't a parallel mechanism, it's
    the replacement. Migration is per platform, not a flag-day cutover:

    • Google Calendar first — reflector already supports it live.
    • Todoist and Clockify keep using the browser/WASM/integration-proxy path
      until each is ported.
    • Only once every platform has migrated: delete wasm/src/integrations.rs,
      the integrations/localthought/syncables Rust crate, and the syncables
      path dependency in wasm/Cargo.toml (all in ontola/atomic-server).
  4. Devonian: promote it to its own top-level repo, localthought/devonian
    (matching the syncables/reflector pattern), rather than a folder inside
    atomic-plugins.

    Given the overlap finding above: generalize reflector's own existing
    id-map/origin-marker/reflect-loop code and extract that as the new
    localthought/devonian package
    , rather than building devonian from the
    narrower, GitHub-issues-specific copy currently nested in
    atomic-plugins/integrations/github-issues/devonian/. Reflector then
    depends on devonian instead of keeping its own copy. One implementation,
    proven out by reflector's own live use.

  5. syncables: already sandbox-safe as-is (pure library, no Node-specific
    assumptions beyond an injectable fetch and a pluggable StorageAdapter —
    confirmed by reading its source). Nothing needs to move into it. It stays
    auth-agnostic; ApiClientOptions.fetch remains its only auth extension
    point.

  6. Reflector's OAuth/PKCE handshake (src/oauth/, deriveAuthProfile):
    stays in reflector, exposed as a fetch-wrapper factory that feeds
    syncables' ApiClientOptions.fetch. It is not moved into syncables —
    syncables has no persistence or HTTP-callback story today and shouldn't
    grow one just for this.

  7. PR Add OpenAPI imports and live OAuth integrations atomic-server#1383 (reflector-rs, the Rust OpenAPI-import
    approach): closed, unmerged, nothing worth salvaging. Already effectively
    noted as superseded in planning/api-plugins.md; not being revived.

What's NOT done yet (this issue is a decision record, not a completed migration)

  • Generalize reflector's reflection engine and extract it as
    localthought/devonian; update reflector to depend on it.
  • Port Google Calendar's plugin usage to call reflector via the sandboxed
    fetch capability instead of the browser/WASM/integration-proxy path.
  • Port Todoist, then Clockify, the same way.
  • Once all three are migrated: delete wasm/src/integrations.rs, the
    integrations/localthought/syncables Rust crate, and its wasm/Cargo.toml
    path dependency, in ontola/atomic-server.
  • Retire (or explicitly re-scope) the browser/WASM/integration-proxy path
    and the codex/browser-integrations / codex/localthought-api-plugins
    branches once superseded per platform.

🤖 Generated with Claude Code

https://claude.ai/code/session_012ye7mjmWCpsZnJNsKXSZbw

Activity

  1. michielbdejong commented on Sep 21, 2026

    @michielbdejong
    ContributorAuthor

    It is ok if the feat/plugin-debug branch of atomic-server still has code, docs and tests for the plugin install mechanism and the plugin runtime, and the one 'Pets' plugin which is purely a 'hello world' demo template. but the actual plugins themselves should live in localthought/atomic-plugins. Do not merge feat/plugin-debug because in the future my proposal (if the rest of the team agrees) is for the atomic-plugins to move to the ontola org, and until that happens we should keep the plugins available inside atomic-server and not merge feat/plugin-debug. but in the feat/plugin-debug branch you can already do the full deletion of anything that has been copied to localthought/atomic-plugins. Make sure the existing e2e tests and docs for the plugins in this repo are copied over and added to CI. my hope is that in this repo we can iterate faster developing just these plugins without having to run Rust compilation etc of the bigger atomic-server project.

  2. michielbdejong commented on Sep 21, 2026

    @michielbdejong
    ContributorAuthor

    Status check on the follow-up ask (copy tests/docs, delete duplicates from feat/plugin-debug)

    Already done, verified today: atomic-plugins's integrations/ tree is byte-identical (same git tree SHA) to atomic-server's feat/plugin-debug integrations/ — including clockify, github-issues, mt940, notion, localthought, pets, tooling, and the shared docs (ACTIONS.md, AUTHORIZATION.md, LIVE_TESTING.md, README.md, catalog.json, evidence.json). atomic-plugins/.github/workflows/ci.yml already checks out the pinned atomic-server commit, overlays this repo's integrations/, builds atomic-server from source, and runs offline certification plus live tests (github-issues, notion) against it. So the "copy the existing e2e tests and docs over, add to CI" ask is already satisfied.

    Deletion from feat/plugin-debug — not safe yet, for any of the 6 folders:

    I checked each candidate folder for references outside integrations/ in ontola/atomic-server (feat/plugin-debug, commit 2febc11b) before deleting anything, per "full deletion of anything that has been copied." None qualify as unreferenced:

    • github-issues, notion: imported directly by browser/data-browser/src/components/AppSetup/registry.ts (import ... from '../../../../../integrations/github-issues/setup', same for notion), plus githubInstaller.ts/notionInstaller.ts, browser/lib/src/plugin-manifest.test.ts, browser/e2e/tests/legacy-github-setup.ts, integration-workspace.spec.ts.
    • notion, mt940, pets: embedded at Rust compile time via include_str! in server/src/plugins/{notion_sync,mt940,pets}_tests.rs (plugin.js and fixture files).
    • mt940, clockify: referenced by browser/data-browser/src/chunks/PluginRuns/{integrationEvidenceLoader,IntegrationDiscovery.test,plugin-workspace,localThoughtExtension,parameterOptions}.ts, browser/lib/src/import-resolution.test.ts, and dedicated Playwright specs (mt940.spec.ts, clockify-import.spec.ts, integration-visibility.spec.ts).
    • localthought: the most heavily referenced — integrationProxy.ts, localThoughtSync.ts, localThought.ts, browserPluginSync.ts, localImportVerdict.ts, localThoughtTables.ts, DevonianDemo/demo.mjs, two Playwright specs, and .dagger/src/index.ts's jsTest() step hardcodes vitest run --config /integrations/localthought/vitest.config.ts. This is also the still-active browser/WASM/integration-proxy path decision Issue Tracker #3 says must stay live per-platform until ported — deleting it now would both break CI and contradict that decision.
    • tooling: certify.mjs/certify.test.mjs are invoked directly by browser/package.json's certify:integrations script and by .dagger/src/index.ts (jsBuild/jsTest/integrationCertificationReport) — this is the shared certification harness the "plugin install mechanism" docs/tests explicitly get to keep.
    • catalog.json: embedded by server/build.rs and served at /integrations/catalog.json; read by browser/data-browser/src/chunks/PluginRuns/pluginCatalog.ts and integration-visibility.spec.ts.
    • evidence.json: read by integrationEvidenceLoader.ts.

    Deleting any of these now would break atomic-server's build/CI. Doing it properly means first: rewriting registry.ts's GitHub/Notion adapters to stop importing from integrations/, moving/duplicating the Rust test fixtures, fixing the .dagger localthought vitest step, and updating the Playwright specs — a real refactor, not a straight delete, and out of scope for this pass. Left feat/plugin-debug untouched; atomic-plugins is unaffected since its CI already brings its own copy of integrations/ rather than depending on atomic-server's.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01AHZRWHyQNxcw5fbTUSm3Hm


    Generated by Claude Code

  3. michielbdejong commented on Sep 21, 2026

    @michielbdejong
    ContributorAuthor

    Progress update: retired the github-issues and notion AppSetupForm/assistant install adapters

    Pushed to ontola/atomic-server's feat/plugin-debug (fast-forwarded, 2febc11 → 72f2992):

    • Emptied registry.ts's adapters list. Both github-issues and notion were only reachable through the AI assistant's list_app_setups/setup_app tools (github-issues) or that plus the "Advanced setup with a token" fallback inside ConnectNotion.tsx (notion) — catalog-driven discovery already routes GitHub Issues through the newer devonian flow, and Notion's primary path is the OAuth/proxy flow in ConnectNotion.tsx, which I left untouched.
    • Deleted the now-orphaned ConnectGitHub.tsx, ConnectNotionManual.tsx, and integrations/{github-issues,notion}/setup.ts (zero remaining importers once the adapters were gone).
    • Deleted their dedicated e2e coverage: app-setup.spec.ts in full (100% GitHub/Notion AppSetupForm flows), legacy-github-setup.ts (its only caller), and three tests inside plugins.spec.ts (Notion manual setup, two GitHub-through-the-assistant tests).
    • Left integrations/github-issues/atomic.ts and githubInstaller.ts alone — integration-workspace.spec.ts imports them directly, bypassing AppSetupForm entirely, to test legacy-installation compatibility.

    A real mistake along the way, for the record: my first pass at the github-issues removal broke two live e2e tests I hadn't checked against (browser/e2e/tests/plugins.spec.ts's assistant-driven GitHub setup tests). Caught it, reverted, rechecked every consumer, then redid it properly once the test dependency was actually gone.

    Explicitly not done, follow-up work:

    • Pets has no SetupAdapter/AppSetupForm wiring today, so atomic-server has no generic "install via assistant" coverage left. Per @michielbdejong's steer, that coverage should be rebuilt against Pets rather than a real third-party plugin — needs a Pets SetupAdapter + schema declaration first.
    • atomic-plugins has no Playwright/browser e2e infrastructure yet, so genuinely plugin-specific install-flow testing (real GitHub/Notion behavior) that "belongs" here per the same steer can't just be moved as a file copy — it needs that infra stood up first.
    • Also opened ontola/atomic-server#1607 against the still-unmerged #1548, fixing an unrelated pre-commit hook bug I hit along the way (oxlint's ignore-aware walker treating the whole target/-nested snapshot as gitignored).

    Next: continuing bit-by-bit into the plugin-specific Rust code cleanup, per the "not done yet" checklist above.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01AHZRWHyQNxcw5fbTUSm3Hm


    Generated by Claude Code

  4. michielbdejong commented on Sep 21, 2026

    @michielbdejong
    ContributorAuthor

    Progress update: extracted reflector's reflect loop into devonian, wired reflector to depend on it

    Picked up the first unchecked item on the "not done yet" list: "Generalize reflector's reflection engine and extract it as localthought/devonian; update reflector to depend on it." Verified it genuinely hadn't happened yet — devonian's existing src//platform-lenses/ had no origin-marker/echo-suppression concept anywhere, and reflector's src/reflect/{marker,id-map,kv-store,engine,service}.ts (the live engine behind POST /api/reflect) was still reflector-only. Did the extraction and the reflector-side switch-over in this session, since it's the concrete prerequisite everything else in the "port the plugins" ask depends on.

    localthought/devonian (branch claude/youthful-mendel-3ujtxo, 2 commits):

    • New devonian/reflect export: ReflectionEngine + ReflectionRunner (the background-loop/reflectNow/status wrapper) + the marker/IdMap/KvStore support types, generalized so the engine depends on nothing but a syncables ApiClient per side — no OpenAPI document, resource model, or auth of its own (that stays with the host).
    • The origin-marker codec is now namespaced (<!-- <namespace>:origin ... -->, default devonian) so a host with markers already in production — like reflector's reflector:origin — can keep its own wire format.
    • Full test coverage under __tests__/unit/reflect/ (32 new tests, built against a real syncables client and a minimal synthetic document, no dependency on reflector's own document/overlay pipeline).
    • devonian/reflect resolves to compiled JS (build/src/reflect/*.js), unlike this package's other exports which ship raw .ts for bundler-consumed callers — reflect's actual first consumer (reflector) is a plain tsc-built Node app with no TS loader at runtime.
    • Version bumped to 0.6.0. Not yet published to npm — that's a maintainer action.

    localthought/reflector (branch claude/youthful-mendel-3ujtxo, 1 commit):

    • Added devonian@^0.6.0 as a dependency; src/reflect/service.ts now builds two ReflectionSides via a new src/reflect/sides.ts adapter (the reflector-specific document/overlay/OAuth wiring devonian's engine doesn't know about) and delegates to devonian's ReflectionEngine/ReflectionRunner, pinned to the reflector marker namespace so already-embedded markers in production keep parsing.
    • ReflectionService's public API (fromConfig/start/stop/reflectNow/status) is unchanged, so src/server/app.ts's /api/reflect* routes and src/main.ts needed no changes.
    • Verified against a locally-linked build of the devonian branch: tsc --noEmit, eslint, pnpm test (88 passing, including a rewritten engine.test.ts that now integration-tests through reflector's real document/overlay pipeline into devonian's real engine), and pnpm test:harness (9 passing). The local link was reverted before committing — pnpm-lock.yaml is intentionally left as-is rather than faking a resolution against a version that doesn't exist on the registry yet.

    Sequencing note: reflector's branch won't pnpm install cleanly until devonian@0.6.0 is actually published — that's expected, not a bug. Merge/publish order: devonian first, then reflector.

    Still open (per the original checklist, now unblocked by this): porting Google Calendar's plugin usage to call reflector via the sandboxed fetch capability instead of the browser/WASM/integration-proxy path, then Todoist and Clockify the same way, then the eventual atomic-server Rust-crate deletion. Each of those is its own substantial piece of work — this session focused on the shared foundation rather than spreading thin across all three.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01B2mbssSNDiu8GpKjaCFRn1


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions