Skip to content

Follow-ups after the per-plugin CI lane split (#37, #38) #43

Closed

Description

@michielbdejong

Outstanding work after #37 (per-plugin lanes) and #38 (the fixes that followed). Grouped by whether it blocks a green CI, weakens the gate, is planned-but-unstarted, or needs a decision.

Everything below was checked against main at d84f046 and the pinned atomic-server 057e2b67, unless marked otherwise.


1. Blocking a green CI

1a. pets e2e: the Integrations card detaches mid-click — fix belongs in atomic-server

integrations/pets/e2e/pets.spec.ts now gets through sign-in (it used to time out at devDrive() → page.waitForURL), then fails later:

TimeoutError: locator.click: Timeout 10000ms exceeded.
  waiting for locator('[data-integration="proxy:pets"]').getByRole('button', { name: 'Set up connection' })
  - locator resolved to <button …>Set up connection</button>
  - element is visible, enabled and stable
  - element was detached from the DOM, retrying

The card renders, so the cross-origin catalog fetch and CORS are fine. The Playwright trace shows /integrations/catalog.json fetched 7 times in the ~10s retry window — the Integrations page refetches on every render and replaces the list, detaching the button before a click can land.

Adding ETag + revalidation to the dev-server did not change this (still exactly 7 fetches), so it is not a caching problem. The churn is in browser/data-browser/src/chunks/PluginRuns/ (IntegrationDiscovery / LocalThoughtCatalog / pluginCatalog.ts), i.e. an atomic-server change, not one in this repo.

The same failure hits E2E: generic plugin system, so CI cannot go green until it is fixed or the affected tests are quarantined.

1b. main merged without CI ever completing

#38 merged while its run sat pending with zero jobs (the merge queue was mid-reconfiguration). So the proxy deletion, the direct-to-atomic-server topology, the teardown fix and the ci gate are all on main having never run green. The next PR's run is their first real exercise — expect red, and read failures as "not yet verified" rather than "newly broken".


2. The gate can report a false green

ci treats skipped as a pass — correct for path-gated lanes, and merge_group is handled explicitly (forced to ["shared"], full matrix).

But on workflow_dispatch the changes job still runs dorny/paths-filter, which has no pull-request context there. If any comes back empty, every downstream job skips and ci reports success on a run that tested nothing.

Not urgent (the queue path is safe), but a manual dispatch is exactly when someone is trying to prove main is healthy. Suggested fix: treat workflow_dispatch like merge_group and force the full matrix.


3. Planned but not started

3a. Per-platform mock fixtures — §4 of integrations/PARALLEL_LANES.md

integrations/localthought/mock-proxy.mjs still dispatches on platform through an if chain, with pets inline. run-lane.mjs already passes MOCK_PROXY_PLATFORMS, which the mock ignores — every lane's proxy serves the full built-in set instead of its own.

Three platforms in catalog.json have no fixture at all: todoist, moneybird, notion.

The design (fixture registry per platform, record.mjs against live creds, fixture.test.mjs replaying recordings through that lane's adapter, plus a nightly drift guard against the real Rust integration-proxy) is written up in §4.

3b. Move the two Clockify e2e tests down — integrations/HANDOFF-e2e-split.md

browser/e2e/tests/plugins.spec.ts upstream still holds:

  • Clockify discovers named workspaces and surfaces preview transport errors
  • Clockify applies linked entries through the real sandbox and skips repeats

So timesheets has no e2e tier, and its Clockify coverage runs ungated in E2E: generic plugin system on every PR touching anything under integrations/. Delete them upstream and add them here in the same change as the next .atomic-server-ref bump, or they run twice.

3c. money has a lane entry with no tiers

Covered only by the unsharded certification step, so a money-only change gets no lane feedback. Fine today; worth revisiting if it grows a server-dependent test.


4. Decisions needed

4a. reflector/ — product or scaffold?

Nothing in the Atomic stack consumes it, integration-proxy covers the browser path, and its best parts have already migrated out (reflect/ → devonian, with real re-export shims). But it is a deployable service (Dockerfile, Procfile, express, pg) whose 19 test files pass.

Correcting an earlier estimate made in discussion: a claim that ~45% of it is duplicate code was wrong.

  • remotestorage/ (537 lines) is a live feature — wired through main.ts, four routes in server/app.ts, persisted in user-store.ts. Removing it is a product decision, not dead-code cleanup.
  • sync/overlay.ts is not a duplicate of syncables' version. syncables' parseTarget rejects bracket paths (/[[\]*]/), and reflector's shipped overlays use exactly that syntax (spec/overlays/github/pagination-overlay.yaml: target: $.paths['/repos/{owner}/{repo}/issues'].get). Swapping would break them. If anything the dedup runs the other way.
  • oauth/ + session + user-store overlaps integration-proxy, but with a different trust model (reflector holds tokens and uses them server-side; the proxy hands a browser rotating one-time codes). A consolidation option, not a deletion.

Blocked on one question: is there a live deployed instance?

4b. Repo boundary: atomic-server should host plugins, not contain them

  • atomic-server still carries a full integrations/ tree (96 files) that has diverged — still clockify/, github-issues/, mt940/, no calendar/ — and server/build.rs embeds it.
  • Six data-browser files import plugin source directly: ConnectNotion.tsx, ConnectPets.tsx, ImportMT940.tsx, ClockifyLocalThought.tsx, githubInstaller.ts, notionInstaller.ts.
  • Three plugin-specific Rust test files live in the host: server/src/plugins/{mt940,notion_sync,pets}_tests.rs.

The generic mechanism already half-exists (localThoughtExtension.ts + the catalog's "live reference to a LocalThought extension module"). Completing it is the real work; deleting atomic-server/integrations/ and dropping the build.rs embed are cheap once it is done.

Also proposed: rename integrations/localthought/ to a top-level proxy-client/ (it is the client for the integration-proxy protocol), and move the mock-*.mjs fixtures next to integration-proxy/. Not atomic-reflector — reflector is a server, this is browser-only code.

4c. Connection-code rotation vs. an iframe

If connector execution moves into the iframe, single-use rotation becomes a correctness hazard rather than a safety property. The client consumes the code before dispatch, so: no concurrency (a second overlapping call throws), no retries, and a dropped response permanently breaks the connection. Fine for one serialized sync loop in one bundle; not fine behind postMessage, which gives no ordering guarantee.

Suggested direction: do not give the iframe the credential at all — keep it in the host page and let the iframe request operations over the existing FrameBridge.

For reference, TTLs in integration-proxy/src/security.rs: handoff 5 min, connection code 10 min, oauth states 10 min, nonces 10 min — and the connection-code TTL resets on every rotation, so an active client never expires.


Context: what the lane split surfaced

Five distinct bugs, four pre-existing and mutually hidden by the old single-job workflow's fail-fast behaviour:

  1. integrations/timesheets/ could never typecheck or unit-test (devonian and @tomic/lib unresolvable) — fixed in ci: one lane per plugin, so every plugin can be worked on in parallel #37
  2. dev-server.mjs rewrote the Host header, breaking signed auth for the live tier — fixed, then superseded
  3. wasm32-wasip2 was added to stable while atomic-server pins 1.98.1, so build.rs degraded silently into a binary with no plugin runtime — fixed in ci: fix the three e2e lane failures from #37 #38
  4. A teardown race between two server-backed tiers in one lane — fixed in ci: fix the three e2e lane failures from #37 #38
  5. The dev-server proxy was unworkable in principle: RequestContext::new takes the origin from Host, while get_origin() comes from ATOMIC_PORT (the bind port, no override). Forward Host and auth verifies while lookups 401; rewrite it and lookups work while auth fails. Deleted in ci: fix the three e2e lane failures from #37 #38.

🤖 Generated with Claude Code

Activity

  1. michielbdejong commented on Sep 23, 2026

    @michielbdejong
    ContributorAuthor

    Status update, from the work in #48:

    Section Where it stands
    §1a (card detaches mid-click) ontola/atomic-server#1637 against feat/plugin-debug. A mitigation, not a confirmed fix: the 7× refetch didn't reproduce locally. The actual remount comes from LocaleProvider keying the app on localeLoaded, and the PR makes the pets card survive it.
    New: e2e sign-in hang Every e2e job hung on "Setting up dev drive…" because CI set SKIP_WASM_BUILD=1 without building atomic_wasm.js separately (ClientDb can't load). Fixed in #48 by building it the way atomic-server's dagger pipeline does. Not yet verified in CI.
    New: pets setup dialog Blocked on #52. The data-browser still calls describe/fetchRecords/validateConnection, which the WASM removal took away on both sides. Decision: move this to reflector.
    §2 Fixed in #48, for all four workflows.
    §3a Registry and MOCK_PROXY_PLATFORMS are done in #48. Fixtures live in each plugin's folder (ontola/atomic-server#1639 tracks the dagger mount). Missing fixtures: #45 (moneybird), #46 (todoist), #47 (notion). Recordings, record.mjs, fixture.test.mjs and the drift guard are not started.
    §3b The upstream half is already done (4bab16ee6 on feat/plugin-debug). The remaining half is #44.
    §3c #45
    §4a, §4c Out of scope here. §4c's points were added to #40.
    §4b Not started. #52 is a prerequisite for atomic-server taking this repo's integrations/.

    Also found in #48: serve.mjs pointed the mock proxy's allowed frontend origin at the dev-server instead of atomic-server, so the pets consent step would have failed. And wasm-smoke.mjs imported removed WASM exports. Both are fixed.

    🤖 Generated with Claude Code

  2. added 2 commits that reference this issue on Sep 23, 2026
  3. added a commit that references this issue on Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions