Skip to content

feat(desktop): implement 30178 team catalog backend - #5112

Open
wpfleger96 wants to merge 1 commit into
mainfrom
duncan/team-catalog-backend
Open

feat(desktop): implement 30178 team catalog backend#5112
wpfleger96 wants to merge 1 commit into
mainfrom
duncan/team-catalog-backend

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Aug 6, 2026

Copy link
Copy Markdown
Member

Implements the Rust/backend half of 30178 team catalog sharing on the community catalog. No TS callers yet — this is PR 1 of 2; #3995 (stacked here) adds the parse layer, hooks, CommunityCatalogDialog, and e2e tests.

What this adds

Projection builder (team_catalog.rs): build_team_catalog_event/content produces a 30178 event from a team + member definitions. Size contracts: 192 KiB total ceiling, per-field bounds (name 256 B, text 4 KiB, system-prompt 16 KiB, avatar URL 32 KiB for projection fields; https URLs additionally validated at 2,048 bytes). Avatar handling: oversized raster data URLs downscaled to fit; oversized built-in avatars silently omitted; oversized https URLs rejected with a named error.

Share/unshare/tombstone (commands/teams/pending.rs): prepare_team_publication_at retains a signed 30178 head for the flush loop; share state is relay-scoped (one community can share while another does not). refresh_or_retract_shared_head_at rebuilds or tombstones the head immediately on team/member edit — no stale publication until the next boot. Both the 30178 catalog and 30176 team tombstones are signed with a created_at that strictly dominates the retained head's (read inside the delete transaction), so a future-dated head cannot survive its own deletion under the relay's created_at <= soft-delete gate.

Domination-aware flush (managed_agents/persona_events.rs): a strictly-dominating tombstone can be signed past the relay's ingest acceptance window (MAX_TIMESTAMP_DRIFT_SECS, ±900s from server time). Republishing such a byte-frozen event verbatim from the pending queue lets it age out of the window and be rejected forever, stranding the head live. The flush loop is now domination-aware for every retained kind:5 (covering both 30176 and 30178). For a pending tombstone with floor f (= the retained row's own created_at):

  • f <= now → re-date and re-sign at now (mirrors the existing archive-request re-sign branch; mark_synced stays keyed to the untouched retained row, so a re-date can't mask a concurrent edit),
  • now < f <= now + 900 → publish verbatim at f, inside the window,
  • f > now + 900 → skip the sweep; the event stays pending and its replacement keeps deferring (via failed_tombstones) so out-of-order retraction remains impossible, converging as the wall clock advances.

No path emits an event the gate rejects, and a boundary reject self-heals through the submit-error requeue. Durable across offline gaps of any length.

Atomic adopt (commands/teams/adopt/): add_team_from_catalog re-fetches and signature-verifies the head from the relay, then plans and commits a multi-entity add across two store writes with byte-exact rollback on any failure (crash window between writes explicitly retained). plan_add resolves full member provenance (owner, d-tag, member-key, projection-hash), reuses exact-match built-in personas, and reactivates deactivated copies on re-add. commit_stores snapshots both stores before writing and restores them on failure. The commit and the retention enqueue are sequenced inside commit_and_enqueue, the sole route to a durable adoption commit: once the store write succeeds it enqueues a pending 30175 for every member copy the add wrote or reactivated and a pending 30176 for the team, so a crash before the next boot reconcile cannot lose the only adopted copy. A byte-identical reused built-in and an idempotent replay write nothing and enqueue nothing; a failed commit propagates and enqueues nothing. Enqueue is best-effort per row (the boot reconcile is the backstop). The frontend refreshes via the useAddTeamFromCatalogMutation query invalidation in #3995, so no agents-data-changed emit is needed here.

Startup reconcile (event_sync.rs): reconcile_team_catalog_heads_at walks all retained 30178 heads at boot: republishes heads whose content changed, tombstones heads whose team or member was deleted, skips unshared heads and unchanged content. Multi-team continuation — all shared teams processed in one pass.

Types: TeamRecord and AgentDefinition extended with shared, catalog_source, team_catalog_source fields. All commands registered in lib.rs.

Tests

  • team_catalog/tests.rs: projection, size contracts, member-key stability, tombstone rollback, fixture matrix
  • adopt/tests.rs: head verification, store planning, provenance, rollback
  • adopt/tests/retention.rs: adoption drives commit_and_enqueue through a spy commit + a real temp-dir retention scope and asserts persisted pending rows — commits-then-enqueues (30175 per minted member + 30176 team), a failed commit enqueues nothing, an idempotent replay skips both the commit and the enqueue, a reused built-in retains only the team, and a reactivated copy is re-retained. Deleting the enqueue inside the seam turns these red — the wiring, not just the helper, is protected
  • pending/tests.rs: share/unshare/tombstone lifecycle, edit refresh/retract, tombstone timestamp domination, typed outcomes
  • pending/tests/gate.rs: flush driven through a stub relay that logs every POST /events with its accept/reject status and enforces the real ±900s ingest gate, for both 30176 and 30178 — within-window publish+dominate; beyond-window stays-pending with zero POSTs (the gate never receives a rejectable event); and the delayed/offline-retry case where a tombstone signed strictly past a then-future head has aged more than 900s into the past, so flush must re-date to now to publish. The reversal check — restore the byte-frozen replay in persona_events.rs and the delayed-retry and zero-POST assertions go red — is what proves the suite discriminates the fix from the rejected implementation
  • teams/tests.rs: 30176 tombstone timestamp domination and no-head fallback
  • event_sync_team_catalog_tests.rs: reconcile scenarios including multi-head continuation
  • 27 shared JSON parity fixtures (tests/fixtures/team_catalog_content/) consumed by both this PR's Rust tests and feat(desktop): add team sharing to community catalog #3995's TS tests

Stack: this PR → #3995

@wpfleger96
wpfleger96 requested a review from a team as a code owner August 6, 2026 21:08
@wpfleger96 wpfleger96 changed the title feat(desktop): add 30178 team catalog backend feat(desktop): implement 30178 team catalog backend Aug 6, 2026
@wpfleger96
wpfleger96 force-pushed the duncan/team-catalog-backend branch 2 times, most recently from 16a1ff3 to e5e5c2a Compare August 12, 2026 15:48

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 — the tombstone can be older than the retained catalog head, so deletion silently leaves the team public.

prepare_team_publication_at and reconcile intentionally assign replacement heads max(now, prior.created_at + 1), which permits a retained 30178 head to be future-dated (commands/teams/pending.rs:159-183; persona_events.rs:112-127). But tombstone_team_catalog_coordinate signs the kind:5 at plain wall-clock now before it opens or reads the retention database (managed_agents/team_catalog.rs:728-756). The relay only deletes coordinate versions whose created_at is at or before the tombstone (crates/buzz-relay/src/handlers/side_effects.rs:2170-2181).

Reproduction shape: retain a shared 30178 head with created_at = now + 60, then delete the team or trigger either unrebuildable-head retraction path. The tombstone is accepted and flushed but applies to zero relay rows. Locally, this helper has already purged the retained 30178 row and queued only that ineffective kind:5 (team_catalog.rs:756-772), so successful flush removes the sole retry witness. The supposedly deleted/retracted team remains publicly discoverable indefinitely. All direct-delete, edit-retraction, and boot-reconcile paths converge on this helper.

Please load the retained head timestamp inside the same BEGIN IMMEDIATE transaction, sign the tombstone with monotonic_created_at(Some(head.created_at)) (falling back to None only when no head exists), then purge and retain atomically. Add a future-dated retained-head regression asserting the tombstone timestamp strictly dominates the head; call-path coverage for direct delete and both retraction paths would prevent wiring regressions.

@wpfleger96
wpfleger96 force-pushed the duncan/team-catalog-backend branch 3 times, most recently from 728ae20 to 9a086df Compare August 18, 2026 16:05

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carl, an automated reviewer, commenting via Wes’s GitHub account.

Reviewed exact head 9a086df0167a34482d427a35d2d524315275e0c3 with the Royal Court. Requesting changes for one merge-blocking lifecycle defect.

P1: future-dated tombstones can be rejected, then become permanently unpublishable

The new deletion protocol signs the 30178 kind:5 tombstone strictly after the retained head and atomically purges that head (desktop/src-tauri/src/managed_agents/team_catalog.rs:724-786). That fixes replacement ordering locally, but it conflicts with the relay contract: ingest rejects every event outside ±900 seconds (crates/buzz-relay/src/handlers/ingest.rs:2005-2011). Pending flush republishes ordinary events, including kind:5, byte-for-byte; only identity archive requests receive a fresh signature/timestamp (desktop/src-tauri/src/managed_agents/persona_events.rs:308-337).

The added regression demonstrates the failure shape while accidentally blessing it: it seeds a head at now + 86,400, requires an even later tombstone, and never sends that tombstone through relay ingest (desktop/src-tauri/src/commands/teams/pending/tests.rs:720-788). That tombstone is rejected until it enters the relay's 15-minute future window. If Desktop does not flush successfully during the resulting narrow acceptance interval, the unchanged event becomes stale and is rejected forever. Because the 30178 head was already purged atomically, the durable witness needed to regenerate a usable deletion is gone while the public catalog coordinate can remain live.

The same incompatible pattern exists for the 30176 team tombstone (desktop/src-tauri/src/commands/teams/mod.rs:297-339), so fixing only the catalog helper would leave sibling deletion broken.

Please implement a relay-compatible durable recovery protocol for both coordinates that preserves domination of the target head without relying on publishing an arbitrarily future-dated event. Validate it through the actual relay timestamp gate, including delayed/offline retry past the original acceptance window. A local assertion that tombstone.created_at > head.created_at is not sufficient.

Non-blocking completeness note: the auto-retraction paths emit team-catalog-auto-retracted, but a scoped search under desktop/src found no listener. If user notification is part of the intended behavior, wire the frontend consumer or adjust the claim.

No additional actionable defects survived consolidation. Existing CI was used for broad validation; I did not duplicate CI-equivalent suites locally.

@wpfleger96
wpfleger96 force-pushed the duncan/team-catalog-backend branch 2 times, most recently from a66f7fa to 6a57da7 Compare August 19, 2026 00:44

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carl, an automated reviewer, commenting via Wes’s GitHub account.

Re-reviewed exact head 6a57da79961be71055724ca328a1018f2ad8e5b9 with the Royal Court. The prior P1 is resolved, and no new blocking findings survived consolidation.

flush_pending_events_at now treats a retained kind:5 timestamp as the domination floor: it re-signs at max(now, floor) when that timestamp is relay-acceptable, leaves a farther-future tombstone pending without submitting it, and defers a same-coordinate replacement for the sweep (desktop/src-tauri/src/managed_agents/persona_events.rs:311-360). Compare-and-clear still uses the untouched retained row (persona_events.rs:364-372), preserving the concurrent-update fence.

Regression coverage drives both kind:30176 and kind:30178 tombstones through the submission path against the relay’s ±900-second timestamp contract, including in-window domination, zero-POST future deferral, and delayed/offline stale-floor re-signing (desktop/src-tauri/src/commands/teams/pending/tests/gate.rs:157-402). The production 30-second flush loop provides the durable retry path (desktop/src-tauri/src/lib.rs:570-594).

All current PR checks pass. Existing CI provided broad validation; reviewers did not duplicate CI-equivalent suites locally.

Verdict: no blocking findings at this head. Wes retains final approval authority.

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carl, an automated reviewer, commenting via Wes’s GitHub account.

Re-reviewed exact head 6a57da79961be71055724ca328a1018f2ad8e5b9 against base bbd20fae75ecc3bd7a83cc12a65379fac22a2b79.

[P1] Catalog adoption does not enqueue owner-device sync heads

add_verified_team commits the adopted personas and team, regenerates Nest, and returns (desktop/src-tauri/src/commands/teams/adopt/apply.rs:109-136), but the adoption path never calls retain_persona_pending or retain_team_pending. The analogous team-snapshot import enqueues each member and the team after its writes (desktop/src-tauri/src/commands/team_snapshot.rs:751-761).

Consequently, a successful catalog adoption has no pending kind:30175/30176 heads for another device owned by the same user. The only apparent repair is the later startup migration (desktop/src-tauri/src/event_sync.rs:16-31,57-77,202-229), leaving a window where a crash or device loss before restart can lose the only adopted copy despite the command reporting success.

Please enqueue every newly copied or reactivated non-built-in persona and the adopted team after the atomic store commit. Add command-path regression coverage asserting pending 30175 and 30176 rows, including idempotent replay.

The independent schema/protocol and relay/authorization reviews found no additional blocker. Exact-head required checks are green, but they do not cover this successful-adoption retention contract.

@wpfleger96
wpfleger96 force-pushed the duncan/team-catalog-backend branch from 6a57da7 to da22f55 Compare August 20, 2026 20:44
Implements the backend for 30178 team catalog sharing on the community
catalog. No UI callers yet — this PR is the backend half of a two-PR
split; desktop PR #3995 (stacked here) adds the TS parse layer, hooks,
CommunityCatalogDialog, and e2e.

Changes:
- team_catalog.rs: 30178 projection builder + size contracts (32 KiB
  ceiling, per-field bounds, avatar downscaling for raster data URLs);
  build_team_catalog_event/content, team_catalog_content_from_event,
  tombstone_team_catalog_coordinate
- commands/teams/pending.rs: publish/unshare/tombstone commands with
  relay-scoped share state; refresh_or_retract_shared_head_at for
  immediate retraction on member edits that exceed the size contract;
  persona-edit refresh guard so unrelated personas are never embedded
- commands/teams/adopt/: add_team_from_catalog with backend head
  verification + byte-level rollback across both stores; plan_add with
  full member provenance (owner, d-tag, member-key, projection-hash);
  builtin reuse via hint matching; commit_stores atomic writer. The
  commit and the retention enqueue are sequenced inside commit_and_enqueue,
  the sole route to a durable adoption commit: after the store write
  succeeds it enqueues retention heads for every member copy the add
  wrote or reactivated plus the adopted team, so a crash before the next
  boot reconcile cannot lose the only adopted copy; reused built-ins and
  replays write nothing and enqueue nothing, and a failed commit enqueues
  nothing.
- event_sync.rs: reconcile_team_catalog_heads_at startup reconcile;
  republish changed heads, tombstone unrebuildable ones, skip unshared;
  multi-head continuation so a single pass handles all shared teams
- persona_events.rs: domination-aware flush for kind:5 tombstones. A
  tombstone is signed strictly past the future-dated head it retracts,
  so its retained created_at is the domination floor. The relay ingest
  gate rejects events beyond ±900s of server time, so a byte-frozen
  future-dated replay can age out of the acceptance window and strand
  the head live forever. Flush now re-dates to now when the floor has
  passed, publishes at the floor when it is within the window, and
  leaves the tombstone pending to converge when the floor is further
  ahead than the window — never emitting an event the relay rejects.
- Rust tests: team_catalog/tests.rs, adopt/tests.rs + adopt/tests/
  retention.rs (adoption drives commit_and_enqueue through a spy commit +
  temp-dir scope: commits-then-enqueues, commit-failure enqueues nothing,
  idempotent replay skips both, reused-builtin/reactivation provenance),
  pending/tests.rs
  (incl. gate tests driving both the 30176 and 30178 tombstones through
  a stub relay that enforces the real ±900s ingest gate, covering the
  delayed/offline-retry case), event_sync_team_catalog_tests.rs
- 27 shared JSON parity fixtures (team_catalog_content/)
- Existing TeamRecord + AgentDefinition types extended with sharing
  fields (team_catalog_source, catalog_source, shared flag, etc.)
- All commands registered in lib.rs; dormant until UI PR merges

Stack: this PR -> #3995
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 force-pushed the duncan/team-catalog-backend branch from da22f55 to 43026a7 Compare August 20, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants