Generic Composio sync orchestrator so every toolkit enforces max_items and sync_depth_days #3333

Summary

Move the per-toolkit memory-sync loop (paginate → filter → dedup → ingest → advance cursor) into one shared sync orchestrator, and shrink each provider to small primitives (fetch_page, ingest_item, item_timestamp/item_id). The orchestrator owns cap enforcement (max_items), the sync_depth_days window, pagination, and dedup — so every current and future toolkit gets these for free, with zero per-provider cap code.

Problem

Each Composio provider implements its entire sync() loop itself — ComposioProvider::sync (src/openhuman/memory_sync/composio/providers/traits.rs:55) — and these are 600–1000 lines each, all differently shaped:

• gmail/provider.rs batch-ingests a whole page; github/notion/linear ingest per-item; clickup/provider.rs nests workspaces → pages → tasks; slack/provider.rs (1043 lines) nests channels → history → messages plus a separate user-list backfill.

Because the cap/window logic must live inside each loop, it is duplicated 6×. This directly caused two bugs (fixed in #3304): max_items was page-granular in every provider (a "max 10" pulled a whole page, ~25–50 items), and Slack had no precise cap at all — it was simply missed. ItemCap (providers/helpers.rs) centralized the math, but each provider still wires it by hand, so:

• A new toolkit (7th provider) must re-implement the cap/window wiring or it silently ignores the user's limits.
• Toolkits without a registered provider — only 6 are registered (registry.rs:81-86); e.g. Excel / Google Calendar / Outlook connected via Composio — don't sync at all (run_connection_sync → "no native memory sync provider registered for toolkit", composio/mod.rs:149), so the per-source limit UI is inert for them.

This is fragile by construction: correctness depends on every author remembering to wire caps into a bespoke 600–1000-line loop.

Solution (optional)

Invert control. Introduce a generic run_sync(provider, ctx, reason) orchestrator plus a slim primitive trait (e.g. IncrementalSource):

• Provider supplies: a preamble hook (resolve identity / list scopes like workspaces/channels), fetch_page(scope, cursor) -> (items, next), item_dedup_key, item_sort_ts, an optional server-side depth_query_fragment, and ingest(items).
• Orchestrator owns: budget check → preamble → scope/page loop bounded by ItemCap::max_pages → sync_depth_days filter → dedup → ItemCap::clamp_batch → ingest → record → cursor advance → is_reached() stop → save state → build SyncOutcome.
• A scopes abstraction handles flat (gmail/github/notion/linear) and nested (clickup workspaces, slack channels) uniformly; Slack's user-list backfill becomes a preamble hook.
• Migrate incrementally: land the orchestrator alongside the existing sync(), convert one clean per-item provider first (notion or github) and keep its e2e green, convert the rest one commit at a time, then delete the per-provider cap code.

Alternative (rejected as primary): keep ItemCap per-provider and add a lint/test gate that fails if a provider does not route caps through it. Cheaper, but still leaves 6 bespoke loops and does nothing for unregistered toolkits.

Acceptance criteria

• Generic orchestrator — one run_sync owns pagination + max_items + sync_depth_days + dedup + cursor for all toolkits.
• Slim providers — each provider implements only fetch/ingest primitives; no per-provider cap/window/pagination code remains.
• All 6 converted — gmail/github/notion/linear/clickup/slack run through the orchestrator with behavioral parity (cursor, dedup, budget, thread/account handling).
• Caps proven generically — e2e: max_items=N caps to exactly N and sync_depth_days filters, asserted via the shared orchestrator path so a new toolkit inherits both.
• New-toolkit ergonomics — adding a toolkit needs only the primitive methods to get caps + window.
• Diff coverage ≥ 80% — the implementing PR meets the changed-lines coverage gate (Vitest + cargo-llvm-cov).

Related

• PR feat(memory-sources): default sync sources on with conservative caps, per-source settings & All In #3304 — default caps + per-source settings UI + ItemCap centralization (the stopgap this supersedes).
• Sources: providers/traits.rs, providers/helpers.rs (ItemCap), providers/registry.rs, composio/mod.rs (run_connection_sync), and the six providers/{gmail,github,notion,linear,clickup,slack}/provider.rs.