Skip to content

syncables: add CLI and macOS daemon with local PostgreSQL cache, coverage index and durable outbox #566

Open

Description

@michielbdejong

Problem and intended behavior

Build a TypeScript/Node CLI and background service for dogfooding Syncables against existing APIs, initially GitHub issues through localthought.io. Keep the compressed replayable request/response archive, collection metadata and outgoing outbox in a separate PostgreSQL database on the user's laptop. Hydrated records may remain entirely in memory.

Device authentication is owned by #565. That agent must publish the endpoint/scope/identity contract; develop against its mock contract while server deployment is pending.

Placement and npm compatibility (required)

  • All implementation lives inside syncables/, in the existing npm package named syncables; do not create a separate CLI package.
  • Add an executable npm bin entry, proposed command name syncables, backed by a compiled Node entry point with the appropriate shebang.
  • Preserve current root and syncables/browser exports, types, import behavior and existing library usage. Importing the library must not start a daemon, read CLI configuration/credentials, connect to PostgreSQL, install services or register CLI handlers.
  • Isolate Node-only PostgreSQL, compression, Keychain and launchd dependencies from browser imports/bundles. Additional exported adapters/subpaths may be additive.
  • Ensure the npm tarball contains the executable and required runtime files. Use a Zstandard implementation compatible with the supported Node version; do not silently assume newer Node built-ins or change engine support merely to obtain them.

CLI and daemon

Proposed commands (engineering may refine names while preserving capabilities): login, logout, connections, collection add/list/show, sync, status, replay, outbox list/retry/resolve, daemon, and service install/start/stop/status/uninstall.

  • Device login to Local Thought; credentials in macOS Keychain, never arguments, archive rows or logs. Provider grants remain at Local Thought. Bind all local namespaces to the verified canonical owner and provider connection.
  • One per-user daemon owns scheduling, token refresh and pending operations. CLI commands communicate through a permission-restricted local IPC endpoint, preferably a Unix socket.
  • Bounded parallel work across independent collections/connections, serialization of conflicting record writes, defined concurrent token-refresh behavior, rate-limit backoff and graceful shutdown.
  • Durable recovery after process interruption, PostgreSQL outage, network loss and macOS sleep/wake. Authentication failures become actionable status rather than unattended browser prompts.
  • Run as an unprivileged macOS launchd LaunchAgent with absolute runtime paths, login startup, restart throttling and bounded/rotated logs. The daemon itself runs in the foreground under launchd.
  • Inspect the existing pilot io.atomic.syncables-daemon before installing anything; explicitly migrate/reuse or report it rather than creating competing workers. Its historical runtime is ~/Library/Application Support/Atomic Syncables Daemon.
  • Document provisioning and migrations for the separate local PostgreSQL database. Service installation must be explicit and idempotent; ordinary npm installation/import must not install or start PostgreSQL or a LaunchAgent.

Local compressed archive and caching transport

  • Implement an injectable caching Transport (or fetch-compatible wrapper) for Syncables. The current client already accepts transport or fetch; all reads and writes use it.
  • Separate live/revalidation, permitted cache reads, and offline replay of a specific recorded run. Replay is deterministic, fails visibly on a missing response and never dispatches mutations, including restored outbox writes.
  • Store request reason/trigger, run and attempt identity, method, full URL/query, relevant representation headers and request body reference; store response status, headers, timing, outcome and body reference. Record cache hits separately from actual upstream attempts.
  • Index request identity by canonical owner, connection/auth context, method, URL, relevant headers and request-body hash. Do not conflate different users, provider accounts, query scopes or permission generations; omit authentication secrets from stored data.
  • Keep immutable, owner-scoped deduplicated payloads, Zstandard-compressed where beneficial, with checksum, codec version and uncompressed size. Preserve response text before parsing/mapping rather than reserializing JSON. Bound decompression and memory use.
  • Commit archived responses before applying observations in memory. Distinguish failed/uncertain requests from completed responses. Preserve conditional/pagination headers and correct 304-to-stored-body relationships.
  • Use transactions, uniqueness and expiring refresh leases for parallel access and duplicate-fetch control; an older fetch finishing late cannot overwrite newer cache state. Start with logical owner partitioning and suitable indexes; physical hash partitioning can follow measurements.
  • Never satisfy normal upstream writes from cache; only explicitly declared POST read operations may be cached. Reads for write preconditions/conflict checks must be fresh.
  • The existing transport lacks request-purpose and cache-result metadata and current pacing/budgets count cache hits. Add the smallest backward-compatible extension needed for freshness, provenance and correct upstream accounting.
  • Archive retention/GC must keep payloads referenced by active collection coverage, replay pins or unresolved outbox operations.

Explicit collections and compact coverage index

Persist versioned collection definitions: source/connection, parent bindings, filter expression, order/comparator, stable tie-breakers, null/collation/time-zone semantics where relevant, schema/mapping/API-description versions, and which predicates/orderings are executed by the provider versus locally.

Do not require one durable row or copied hydrated payload per item. Persist page/range coverage segments pointing to immutable responses and the item array/JSON pointer:

  • Collection definition, parent/auth/query scope, generation or observation/run identity, ordering key, interval bounds and inclusivity, response/payload reference, count and observation time.
  • A page can establish complete coverage of its bounded interval at query time when the provider contract guarantees it. Model that guarantee explicitly; [min,max] of observed IDs alone is not proof. Ordered compound keys such as (updatedAt,id) may be required.
  • Later observations replace older coverage only over the interval they actually cover; preserve older observations for uncovered intervals. Record freshness per segment; do not present observations from several times as one atomic provider snapshot.
  • Handle empty bounded results, partial/error pages, overlapping ranges, deletions versus filter-membership changes, and explicitly incomplete coverage.
  • Persist segment publication atomically. Hydrate lazily from referenced payloads, extract the segment's active interval, apply mappings/filters/order, and use a bounded decoded-response cache. Reuse valid source coverage across derived collection definitions.
  • Capture the coverage metadata from Syncables/provider declarations as needed; current ReadCoverage is limited and must not be mistaken for general range-completeness proof.

Durable outgoing changes

Persist outgoing local intent in the local PostgreSQL database before acknowledging an edit. Hydrated views overlay that intent on archived observations.

Store stable operation identity, owner/connection/destination, intended fields/body format, conflict baseline/preconditions, ordering/dependencies, idempotency keys, state/attempts, retry time, leases and confirmed/rejected/uncertain outcomes. Journal response and outcome transactionally; reconcile ambiguous outcomes before retrying non-idempotent operations.

Current Syncables uses one serialized syncables:outbox storage record with one client owner. A PostgreSQL-backed record can be an explicitly bounded first milestone. The final parallel design needs additive per-operation persistence/claim interfaces; simply running multiple clients against that record is unsafe. Parallel workers use expiring claims and preserve record/dependency order. Losing leases must not let a stale worker commit over a newer claim, and provider duplicates still require idempotency or uncertainty reconciliation.

Keep library in-memory defaults and the existing StorageAdapter API usable. Do not silently lose confirmed/local-only edits when outbox entries settle or observations are replayed; document what durable source reconstructs each state.

Replication backend configuration (additional user requirement)

Support three selectable replication backends: Atomic native sync, NextGraph native sync, and remoteStorage. Provide CLI commands to set up, list, inspect, enable/disable and remove named replication connections, separate from Local Thought provider connections. Definitions must name local collection scope, destination, backend identity/authentication reference and observed status.

First scaffold may save/list non-secret backend definitions and explicitly report that authentication/runtime replication is not implemented; do not claim configuration alone establishes sync. After Michiel tests device/API access, implement backends in separate stages.

  • Atomic: reuse the compatible Atomic SDK/engine for signed commits, Loro state/version vectors, subscriptions, permissions and reconnect/outbox recovery; a plain signed HTTP import is not native replication.
  • NextGraph: use its SDK/native wallet-verifier sidecar, with wallet/document permissions, encrypted commit synchronization and persistent protocol state. Existing snapshot interchange is not evidence of full native peer sync.
  • remoteStorage: discover the server from the user address using WebFinger, authorize the selected storage module, and implement ETag-aware document/listing synchronization and conflicts; document storage is not a CRDT protocol. Reuse the existing Reflector adapter where appropriate.
  • Keep native protocol durable state distinct from the provider API outbox, preserve cross-format stable IDs/provenance, and prevent replication echoes from creating repeated provider writes.
  • Native peer protocol state may need durable SDK-owned storage in addition to PostgreSQL; hydrated API projections can remain disposable.
  • Initial configuration must not start background jobs, silently grant access, or overwrite another backend connection.

Delivery and validation

Implement in reviewable stages: package/CLI isolation and auth mock; local PostgreSQL archive and deterministic replay; coverage/hydration; durable mutation delivery; launchd integration and live pilot.

  • Packed-package smoke tests prove existing Node and browser library imports still work without CLI dependencies/side effects, and the CLI executable works from the same npm tarball.
  • PostgreSQL-backed tests cover owner isolation, identical-payload deduplication, parallel cache misses, late responses, transaction failures and restart recovery.
  • Replay reproduces a paginated run without upstream access, including errors/304s; missing or partial coverage is visible and replay emits no writes.
  • Coverage tests exercise bounded completeness, sparse IDs, compound order, empty/overlapping ranges, mixed observation times, filters and partial scans.
  • Hydrated collections can be discarded/rebuilt; decoded-cache memory is bounded.
  • Outbox tests cover interruption around dispatch/acknowledgement, lease contention, ordering, rejected writes, conflicts and uncertain creates without blind replay.
  • launchd install/uninstall/status and sleep/reconnect behavior are validated on macOS, with old-daemon coexistence handled explicitly.
  • Demonstrate the read-only GitHub pilot with integration-proxy: add OAuth device flow for CLI authentication to Local Thought #565's deployed contract; add a supported update workflow against a disposable test repository with explicit write permission.
  • Record import, rebuild and memory measurements using representative invented data; clearly distinguish fixtures from live provider evidence.

References and boundaries

The Local Thought proxy forwards authenticated requests; this issue does not place the archive/cache at the proxy. This ticket does not implement platform retirement/leader promotion or require a new UI. Existing release/publication workflows still apply.

Activity

  1. michielbdejong commented on Oct 13, 2026

    @michielbdejong
    ContributorAuthor

    Pickup: Codex chat 01a13516 is implementing only the first scaffold requested by Michiel: npm CLI entry point, device-flow login, connection discovery and a read-only provider API probe, with a local mock for testing before #565 is deployed. Cache/PostgreSQL/outbox/launchd and native Atomic/NextGraph replication remain later stages after Michiel tests the scaffold. Library exports and browser use will remain intact.

  2. michielbdejong commented on Oct 13, 2026

    @michielbdejong
    ContributorAuthor

    First scaffold is ready in draft PR #567 for Michiel's testing: npm CLI auth-test (device login → connection discovery → read-only API probe), a local mock, and replication add/list for Atomic, NextGraph and remoteStorage. All three backends report configuration-only, not active replication. Node 22: 1,307 unit tests, lint, formatting and TypeScript checks passed; packed-package library imports/bin and mock end-to-end API access passed. Real localthought.io/GitHub testing waits for #565's deployed contract. After user feedback, remaining work is persistent credentials, local PostgreSQL archive/coverage/outbox, launchd and actual replication backends. This issue remains open.

  3. michielbdejong commented on Oct 13, 2026

    @michielbdejong
    ContributorAuthor

    Terminal-testing follow-up in PR #567: removed the inherited Node 22-only upper limit; minimum remains >=22.11. All 1,307 tests pass on Node 26.11.1 locally, and GitHub's new Node 22/24/26 lint/test/build matrix passes. The guide now gives an explicit scaffold-branch clone and syncables directory; cli:mock is a source-branch script, not present in main/current npm release.

  4. michielbdejong commented on Oct 13, 2026

    @michielbdejong
    ContributorAuthor

    Continuing after Michiel confirmed the mock flow works in his terminal. Next stage: a separate local PostgreSQL compressed request/response archive, injectable caching transport and recorded-run read-only replay, with real PostgreSQL concurrency/recovery tests. No proxy-side archive. #565 is still open without an implementation contract, so live device auth remains a dependency; the mock remains available for this storage stage. Actual replication, persistent credentials and launchd follow in subsequent stages.

  5. michielbdejong commented on Oct 13, 2026

    @michielbdejong
    ContributorAuthor

    Codex CLI session 01a13516 published the next stage in draft PR #567, commit 5355ef97: a separate local PostgreSQL archive with owner/connection/authorization-context isolation, compressed deduplicated payloads, URL/request indexes, concurrent refresh leases and offline recorded-run replay. All 1,320 unit tests passed locally with PostgreSQL; a Node 22 CLI test replayed successfully after stopping the mock and restarting the database. CI now tests PostgreSQL on Node 22/24/26.

    The user tested mock device login successfully and will test real Local Thought after #565 is deployed. Updated live instructions explain the deployed public client/read scope, approved GitHub connection and expected HTTP 200 output: https://github.com/ontola/atomic-plugins/blob/codex/syncables-cli-auth-scaffold/syncables/docs/cli.md

    #566 remains open: collection coverage metadata, durable outgoing outbox, persistent auth, daemon installation and native Atomic/NextGraph/remoteStorage replication remain later stages. No live auth/deployment or macOS runtime claim is made.

  6. michielbdejong commented on Oct 16, 2026

    @michielbdejong
    ContributorAuthor

    Pickup: Codex chat 01a1438a is resuming draft PR #567 after Michiel's go-ahead. Updating its CLI/cache stage against current main, checking review feedback and the merged device-auth contract, then running library/CLI/PostgreSQL validation. Worktree branch: codex/syncables-cli-resume; updates will go to the existing PR branch. Native replication, durable outbox and daemon installation remain subsequent stages.

  7. michielbdejong commented on Oct 16, 2026

    @michielbdejong
    ContributorAuthor

    Resume complete in draft PR #567, head 18857ae2, updated against main 18e42516. Fixed package/changelog conflicts, review simplifications, concrete device-auth instructions and the missing pnpm archive dependencies. Full Syncables CI passes on Node 22/24/26 with PostgreSQL (1,638 tests per version); required repository CI is also green, including issue-tracker typecheck/unit/e2e and generic plugin-system browser tests: https://github.com/ontola/atomic-plugins/actions/runs/38872176591

    An installed npm tarball passed mock login, upstream/cache reads and offline replay after stopping the mock and restarting PostgreSQL. A direct, unauthenticated GitHub labels smoke test passed twice (9 records across 5 pages each, matching independent IDs); this used a minimal labels document and does not verify the proxy catalog/device flow. No provider writes or real data fixtures were committed.

    The live localthought.io/GitHub device flow still needs a successful user approval/read outside this cloud environment, whose outbound policy excludes localthought.io. Its current deployment state is unknown here. Published client/scopes and instructions: https://github.com/ontola/atomic-plugins/blob/codex/syncables-cli-auth-scaffold/syncables/docs/cli.md

    Keep #566 open and #567 in draft for staged testing. Remaining implementation: collection coverage/hydration metadata, durable outgoing changes, persistent auth, daemon/launchd lifecycle and actual Atomic/NextGraph/remoteStorage replication. Replication definitions remain configuration-only.

  8. michielbdejong commented on Oct 16, 2026

    @michielbdejong
    ContributorAuthor

    Pickup: Codex session 01a13516 is documenting Michiel's CLI/daemon, local PostgreSQL archive, compact collection coverage, durable outbox, replication and device-approval ideas in a standalone Syncables design document on codex/syncables-daemon-design. This is documentation only, separate from the resumed #567 implementation branch, with links to #565/#566 and explicit implemented/planned status.

  9. michielbdejong commented on Oct 16, 2026

    @michielbdejong
    ContributorAuthor

    Documentation milestone from Codex 01a13516: PR #644 adds the consolidated Syncables CLI/daemon design, linked from the package README. It covers laptop PostgreSQL archive/cache/replay, compact collection coverage with disposable hydration, durable outgoing intent, launchd, the three replication backends and a future source-retirement cutover. It also explains the owner-hub/browser approval integration and records the published pilot client/scopes without claiming a live deployment.

    Design: https://github.com/ontola/atomic-plugins/blob/codex/syncables-daemon-design/syncables/docs/daemon-design.md

    Documentation only; local link targets, changelog validation and diff whitespace pass. #567 remains untouched, and #565/#566 remain open for their implementation and live-validation work.

  10. michielbdejong commented on Oct 16, 2026

    @michielbdejong
    ContributorAuthor

    Handoff to “Plan stand-alone Syncable”

    Michiel explicitly requested handing this work from Assess Syncables leader promotion (01a13516) to Plan stand-alone Syncable (01a14512-47d4-728a-9c9c-98c92f64906d). Direct chat-to-chat messaging is unavailable in the source session, so this comment is the durable handoff; it has not been delivered to or resumed in that chat automatically.

    Start from the existing work

    Prior user decisions to preserve or explicitly reconcile

    • All CLI implementation lives inside syncables/ in the existing npm package, preserving root/browser library imports and isolating Node-only side effects.
    • Michiel explicitly requested a separate PostgreSQL database on the laptop for the archive and durable outgoing outbox, with no request/response archive at the integration-proxy. The receiving chat's SQLite recommendation is a new proposal, not a recorded replacement of that choice.
    • Hydrated collections can be disposable in-memory projections. Persist versioned definitions with explicit filters/order/tie-breakers and compact coverage segments pointing to immutable responses and item-array locations. A segment may cover a guaranteed bounded interval without one durable row per item; observed min/max alone do not prove completeness.
    • Persist outgoing intent before acknowledging edits; replay must not dispatch mutations. Parallel delivery needs per-operation claims, ordering and uncertain-outcome reconciliation.
    • Atomic native sync, NextGraph native sync and remoteStorage are planned backends; current CLI replication add/list saves non-secret configuration only. The receiving chat adds Jazz and atproto as delayed plans; none is implemented by this scaffold.
    • The product direction includes adding a UI before potentially retiring the original platform. Promotion needs a separate authority/cutover plan, not just copying API responses.

    Receiving chat's additional scope

    Michiel requested architecture and plan first, and confirmed content-addressed nested objects. This extends the current owner-scoped whole-response payload deduplication: define versioned typed node encoding, lossless numeric handling, stable remote identities versus content hashes, revision history (including A → B → A), raw response retention, hash-graph reachability and GC before implementing it.

    Also carry forward its requested configuration-driven selected collection fetches, provider-supported incremental checkpoints, polling/webhook daemon, queued webhook relay, and offline hierarchy for platform/account/collection, filtering/order/paging, item history and revision retrieval. Ordinary full reconciliation is not incremental sync-token support. Preserve commit-before-ack webhook ingestion and checkpoint recovery. Avoid creating a second competing CLI/storage design without reconciling #567.

    Device-flow continuation

    The receiving chat already reports that API keys (local) configured Heroku app integration-proxy, release v97, with discovery/health HTTP 200 and DEVICE_APPROVAL_URL=http://127.0.0.1:11873. This is a report from that chat, not a fresh live check by this handoff.

    The local owner bridge must run on the laptop with the existing Atomic owner; owner seeds stay local. Browser approval and an authenticated GitHub GET are still unverified in the reported result. The proxy /device page only links to an owner-signer UI; setting an arbitrary localthought.io approval URL does not implement that UI. A production approval page remains separate from the loopback dogfooding bridge.

    Keep #565/#566 open for remaining implementation/live evidence. #644 records the design; #567 stays draft for staged user testing. No further implementation, deployment, service installation or provider writes were performed as part of this handoff.

  11. michielbdejong commented on Oct 16, 2026

    @michielbdejong
    ContributorAuthor

    Pickup: “Plan stand-alone Syncable” (01a14512) has received the handoff in comment 6217793510. Continuing the architecture-and-plan stage by extending existing design PR #644 on codex/syncables-daemon-design; preserving laptop PostgreSQL, compact response-backed coverage, and the existing draft #567 implementation. Adding nested content-addressed JSON storage, revision browsing, selected incremental runs, polling/webhook recovery, and delayed Jazz/atproto plans. No competing CLI/storage implementation or service installation. Laptop owner approval and live authenticated GitHub evidence remain outstanding; owner keys stay local.

  12. michielbdejong commented on Oct 16, 2026

    @michielbdejong
    ContributorAuthor

    Handoff picked up by “Plan stand-alone Syncable” (01a14512). The original design PR #644 merged during this work, so its extension is in follow-up PR #645, head 3e7e16d8, on codex/syncables-standalone-handoff:

    • Preserves syncables: add device-auth CLI and local PostgreSQL cache/replay #567's same-package executable, separate laptop PostgreSQL archive and compact response-backed coverage.
    • Adds nested content-addressed JSON and lossless encoding requirements; stable remote identities, revision occurrences (A → B → A), raw response retention, indexing/backfill and graph GC.
    • Specifies selected one-shot incremental fetches, checkpoint publication/recovery, offline hierarchical filtering/order/paging/history, polling and durable webhook/relay ingestion.
    • Carries delayed Jazz/atproto adapters alongside the three existing planned destinations, with independent delivery state.
    • Defines implementation stages and acceptance cases. All new commands/storage features remain explicitly planned.

    Documentation checks passed: changelog fragment, relative file links, balanced code fences and diff whitespace. #567 implementation is unchanged. Michiel is testing device flow in API keys (local); this change neither duplicates that test nor claims live authenticated GitHub success. Owner keys remain local. #565/#566 stay open; #567 remains draft.

  13. michielbdejong commented on Oct 16, 2026

    @michielbdejong
    ContributorAuthor

    Device-flow handoff incorporated in documentation PR #646 (fad8c056). API keys (local) reports completed live testing on 2026-10-16 against the Heroku deployment at https://localthought.io: existing-owner browser approval for syncables-cli, bearer discovery of exactly the approved GitHub connection, proxied GitHub issues GET 200, session revocation and subsequent bearer GET /connections 401. The unauthorized local signing request returned 403. This is attributed laptop evidence, not a fresh cloud test or an identified packed #567 executable test.

    The plan now records the reusable bridge/UI/example client under integration-proxy and the laptop-only atomic-drive/Keychain signer adaptation. No owner seed was exported, printed or stored; device tokens stayed in process memory. Private local evidence/adapter files remain on the Mac. Production owner-hub integration, durable bridge lifecycle and live refresh rotation remain follow-ups. Branding is atomic.place; the service issuer stays https://localthought.io.

    Documentation checks passed. No implementation, deployment, service installation, new grant or provider write. #565/#566 remain open; #567 remains draft.

  14. michielbdejong commented on Oct 20, 2026

    @michielbdejong
    ContributorAuthor

    Implementation pickup: “Plan stand-alone Syncable” (01a14512) is now building the standalone read-only daemon on codex/syncables-daemon, reusing draft #567's CLI/PostgreSQL archive and integrating current main. Scope: nested JSON content storage, durable configured collection reads/history/browsing, and foreground polling/webhook scheduling with restart recovery. No provider writes or replication backends. Persistent local credential/lifecycle integration will remain explicit and must be tested on macOS; owner signing stays separate. Work is code, PostgreSQL tests and a runnable CLI, beyond the completed design.

  15. michielbdejong commented on Oct 20, 2026

    @michielbdejong
    ContributorAuthor

    Implementation milestone from “Plan stand-alone Syncable” (01a14512): PR #675 adds the read-only standalone daemon, nested content-addressed JSON response logs, configured full/query-token incremental fetches, durable polling/relay hints and offline hierarchy/filter/order/page/history. Michiel’s new Versioned Atomic Data experiment is documented in docs/design/versioned-atomic-data.md: resources and imported schemas share private Git snapshots and default to commit-pinned resolution, with commit/tag lookup. PostgreSQL remains the response archive and scheduler/inbox store.

    All 1,795 tests pass against current main with PostgreSQL 17; the installed tarball passes mock login/sync/daemon/webhook/shutdown and offline browsing/schema/history. PR CI runs Node 22/24/26. Runnable instructions: syncables/docs/daemon.md on the PR branch. #567 and its draft branch are untouched.

    Remaining: laptop validation of the Keychain adapter and new daemon against localthought.io, full schema closure/typed lenses, broader incremental/coverage strategies, launchd/IPC, outgoing outbox and the five remote destinations. No new owner grant, owner keys, provider writes, deployment or service installation. Keep #566 open.

  16. michielbdejong commented on Oct 20, 2026

    @michielbdejong
    ContributorAuthor

    Final implementation checkpoint: PR #675 (3dc62185) is ready for the staged laptop trial. All 1,798 tests pass locally with PostgreSQL 17. Final-head CI passes Node 22/24/26 (including the installed-package smoke), the repository-wide CI gate and the security scan. The preceding issue-tracker host connection-setup failure did not recur on the final head; its tests were neither changed nor skipped.

    The Versioned Atomic Data Git experiment and default commit-pinned resource/schema reads are documented in docs/design/versioned-atomic-data.md. Exact runnable commands and limits are in syncables/docs/daemon.md; fixed response-graph hashes are in syncables/docs/json-graph-v1.md. Persistent device refresh has a durable non-secret guard for crash/uncertain-response recovery, including Keychain deletion refusal.

    PR #675 remains open for laptop validation. Existing #567 remains OPEN/DRAFT at its unchanged 18857ae2 head. Remaining work: live macOS Keychain/deployed-daemon evidence, full schema closure and typed lenses, broader coverage/incremental strategies, launchd/IPC, outgoing outbox and the delayed remote destinations. No service was installed, no deployment or provider write occurred, and owner keys stayed local. Keep #566 open.

  17. michielbdejong commented on Oct 20, 2026

    @michielbdejong
    ContributorAuthor

    Continuing standalone Syncables work in session 01a14512 on codex/syncables-daemon (PR #675), at Michiel’s request: add explicit snapshot intent messages and readable commit history, including schema history by IRI. Existing Mac trial remains pending; no merge or live grant is part of this update.

  18. michielbdejong commented on Oct 20, 2026

    @michielbdejong
    ContributorAuthor

    PR #675 now includes snapshot intent messages (sync/daemon --message), generated archive-run provenance, readable history entries, resource/schema history by --subject, and pinned --at history. Head 41f208b passes all 1,801 local tests, installed-package smoke and Syncables CI on Node 22/24/26. Full repository CI is still running. Added Homebrew PostgreSQL setup for the laptop trial. PR remains open; live macOS/Keychain validation and the broader roadmap remain follow-ups.

  19. michielbdejong commented on Oct 22, 2026

    @michielbdejong
    ContributorAuthor

    PR #675 merged to main as 14819ce after Michiel reported a successful local trial and explicitly approved merging. Exact-head CI and all checks passed. The report establishes the exercised local trial; additional live GitHub/Keep and Keychain evidence remains separate. Continuing in session 01a14512 to prepare real-repository and Google Keep daemon trials, as requested. No provider writes, deployment or package publication.

  20. michielbdejong commented on Oct 22, 2026

    @michielbdejong
    ContributorAuthor

    Preparing source-run live-trial profiles on codex/syncables-live-trial: a private config helper using approved connection discovery, the existing read-only GitHub issue profile, and a GET-only Google Keep note profile. Google Keep OAuth/catalog readiness still needs to be established in the authenticated local worker; cloud work will use invented provider data only.

  21. michielbdejong commented on Oct 22, 2026

    @michielbdejong
    ContributorAuthor

    PR #676 merged the clarified private-file copy/config instructions and a read-only Google Keep daemon profile (981cb76). All 1,802 tests and exact-head checks passed. Live setup exposed an authentication UX issue: CLI prints verification_uri instead of the server-supplied verification_uri_complete, so /device shows no code/input until continuing to the local owner approval page. Investigating a CLI-only fix; owner signing remains local, no live grant is approved from the cloud.

  22. michielbdejong commented on Oct 22, 2026

    @michielbdejong
    ContributorAuthor

    Handoff to the existing “API keys (local)” worker for continuing Michiel’s stand-alone Syncables live tests (22 October 2026; cloud session 01a14512).

    Current status:

    • PR Add standalone Syncables daemon with versioned Atomic Git snapshots #675 is MERGED on main as 14819ce: read-only configured sync/daemon, local PostgreSQL response archive with content-addressed nested JSON, Git JSON-AD resources/schemas, commit/tag/history browsing and --message intent provenance.
    • PR Clarify daemon live setup and add a read-only Google Keep profile #676 is MERGED as 981cb76: clearer exact-file/private-directory setup, real GitHub instructions and syncables/examples/daemon/google-keep.json. All 1,802 tests and exact-head checks passed.
    • Michiel’s invented-data laptop test worked. He copied config.demo.json and github-issues.json into his chosen private directory; the recommended location was $HOME/.local/share/syncables-live. Confirm his actual path rather than overwriting anything. The copied github-issues.json needs no edits. It describes GET issues, state=all and Link pagination; real repo bindings belong in config.
    • During real auth-test, the public /device page had no paste field; Continue led to an unreachable loopback bridge. Michiel asked this local worker to restart the bridge and has now explicitly reported that approval worked. That report establishes approval success, not yet a successful new provider GET.
    • PR Print complete device approval links and explain owner approval #677 is now MERGED as bef8e0a. It prints/validates verification_uri_complete and explains the owner-approval step. All 1,808 local tests, installed-package smoke, Node 22/24/26 checks, repository CI gate and security check passed on exact head fad8b78. Fetch main and rebuild to get the complete approval URL with its code. This change does not start the local bridge or approve a grant.

    Live test scope and next steps:

    1. Keep the existing Mac bridge listening at http://127.0.0.1:11873 and targeting https://localthought.io. Reuse its existing Keychain-based Atomic owner adapter under $HOME/Documents/Codex/2026-10-08/check-which-x20/work/device-login-live/. Do not replace it with the upstream seed-based example or export any owner key. This remains a manually started bridge, not a launchd installation.
    2. Update/build the source on main (preserve private config/data). From syncables/: npm ci then npm run build:release. Real issuer https://localthought.io, registered public client syncables-cli, device scopes connections:read proxy:read.
    3. Use the owner and github-issues connection_id returned by the approved auth-test in the private config. Top-level owner is the Atomic owner; bindings.owner is the GitHub namespace. Earlier verified repo/example: ontola/usertest-findings, unless Michiel chooses another. Set account=ontola, name=usertest-findings, collection=issues, document=./github-issues.json, bindings={owner:ontola,repo:usertest-findings}, query={per_page:"100"}, pollSeconds=0 initially. The GitHub installation must cover the repo.
    4. Use the separate local PostgreSQL database already created for the trial: SYNCABLES_DATABASE_URL=postgresql://localhost/syncables_local (respect any actual local auth settings). One-shot: node build/src/cli/main.js sync --config PRIVATE_CONFIG --collections github-issues/ontola/usertest-findings --interactive --message "First snapshot of the real repository". This performs a full collection fetch within documented caps. A one-page check before that is auth-test --path '/repos/ontola/usertest-findings/issues?state=all&per_page=1' with the real issuer/client; choose the connection explicitly if multiple are approved.
    5. Browse the same private config, check collection/item history, pin a returned commit or tag and retrieve its resources/schemas. Optionally make an intentional provider-side edit through its UI, rerun sync and compare revisions. The daemon itself makes GET requests only. Persistent login/Keychain refresh can be tested separately with fresh explicit browser approval. Approve exactly the intended connections.
    6. Google Keep is the same standalone daemon path, not a requirement to test the Drive app’s two-way mapping. First establish the existing Google Keep provider connection and Google Workspace/API/OAuth readiness. Previous proxy-setup handoff described provider client credentials and catalog setup; cloud work has not verified that deployment config or a live Keep GET. Keep API access requires an appropriate Workspace account/authorization. Keep server-side secrets server-side. Preserve all existing catalog platforms.
    7. Once available, select google-keep during device approval; it must belong to the same Atomic owner for this config. Copy examples/daemon/google-keep.json beside config.json. Add {platform:"google-keep",account:"workspace",name:"notes",connection:APPROVED_ID,document:"./google-keep.json",collection:"notes",bindings:{},query:{pageSize:"100"},pollSeconds:0}. Read-only one-page probe: auth-test --path '/v1/notes?pageSize=1' approving/selecting Keep. Then sync only google-keep/workspace/notes with an intent message. For both collections in one persistent session, select both during login; adding config cannot extend the old session’s grant.
    8. Keep sync fetches/archives the account’s full default non-trashed note collection. A browse title filter is LOCAL, not a fetch restriction. Keep item IDs are notes/ID; percent-encode the slash in a browse path, but pass the full raw ID to history. Original nested list items are in the note’s source JSON; attachment files are not downloaded. This profile uses full reconciliations, not provider-token deltas. Leave sending to Keep off in any Drive app used to establish OAuth.

    Authoritative runnable guide and limits:
    https://github.com/ontola/atomic-plugins/blob/main/syncables/docs/daemon.md
    Protocol: integration-proxy/DEVICE_AUTH.md. Versioned data: docs/design/versioned-atomic-data.md.

    Record sanitized evidence (approval outcome, HTTP status/counts, collection scope, local commit ID, and whether persistent login/refresh was actually exercised). Keep response bodies, note contents, user/device codes, device tokens and owner keys out of public issue logs. No cloud grant, provider write, new owner identity, service installation, npm publish or deployment was performed. #566 remains open for the broader roadmap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions