Skip to content

feat(archive): add observer-frame retention schema and gated DB adapter - #5719

Merged
wpfleger96 merged 14 commits into
mainfrom
wpfleger/archive-retention-phase1
Aug 21, 2026
Merged

feat(archive): add observer-frame retention schema and gated DB adapter#5719
wpfleger96 merged 14 commits into
mainfrom
wpfleger/archive-retention-phase1

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

The local-save archive (~/.buzz/archive/archive.db) grows without bound — observer frames (kind 24200) are 99.95% of it by bytes (~1.3M rows, 75–375 MB/day). This lays the schema and process-wide foundations for bounding that growth: a single global retention window for observer frames, kept in archive_meta, plus the gated DB adapter every later phase builds on. The prune worker that actually deletes lands in a follow-up; nothing here removes data.

What this adds

  • Migration M4 (add_archive_meta, crash-safe): creates the archive_meta k/v table and the archived_at-covering scope-age index used by the future prune scan, and seeds observer_retention_days=30. Runs under one BEGIN IMMEDIATE with an in-lock marker recheck and the marker written last, so a crash before COMMIT rolls back every object and the next open re-runs from scratch. Fails closed on an externally-created archive_meta present without the marker.
  • Process init barrier + gated ArchiveDb adapter owning every production open. All subscription-mutation commands and the archive sync task route their DB work through ArchiveDb::with_conn, which awaits the init barrier once. This also fixes a pre-existing bug where create_save_subscription ran blocking DB work on the async runtime.
  • Commands: get_observer_retention_days / set_observer_retention_days (fail-closed — rejects days < 1 or > 36500 before writing) and archive_size_stats (physical file bytes for the main DB + -wal sidecar, plus page_size / page_count / freelist_count — PRAGMAs and file metadata only, no payload scans).

Scope

Observer frames (24200) are the only kind with a retention setting. NIP-AM metrics (44200) and every other archived kind are kept indefinitely with no retention state — no policy table, no per-kind or per-subscription configurability. This is the simplified design ruled in over the earlier per-subscription × per-kind approach: bounding observer frames alone captures essentially all the value.

@wpfleger96
wpfleger96 requested a review from a team as a code owner August 13, 2026 00:26
Duncan and others added 5 commits August 14, 2026 10:38
Duncan @wpfleger96
feat(archive): add local retention policy storage and lifecycle
34773f0 
Introduce per-subscription/per-kind retention policies for the local
archive, governing how long archived events of each kind are kept before
a future prune pass expires them. Observer frames (kind 24200) default to
a 30-day rolling window; every other kind defaults to Forever.

Policies live in a normalized `retention_policies` table created by a new
crash- and race-safe migration (M4). Because the archive DB is opened by
two connections on first use, M4 cannot use the M1-M3 DEFERRED pattern; it
takes the write lock up front with BEGIN IMMEDIATE, rechecks the marker and
schema shape inside the lock, seeds defaults from existing subscriptions,
and writes its marker last so a crash before COMMIT rolls back cleanly.
Malformed subscription `kinds` JSON fails the migration closed rather than
leaving already-archived data ungoverned.

All production archive DB access now routes through a gated `ArchiveDb`
adapter: a process-wide init barrier runs every migration exactly once
before any command opens a connection, and a maintenance RwLock reserves a
home for the Phase-4 VACUUM path. The policy lifecycle is deliberately
independent of `save_subscriptions` — disabling a kind or deleting a
subscription orphans its policy so historical data keeps expiring; only the
explicit delete command removes one.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Duncan @wpfleger96
test(archive): harden M4 shape validation and pin the init barrier
85438f6 
Resolve the three review findings on the retention Phase-1 branch.

M4 now validates the COMPLETE expected shape of its objects inside the
BEGIN IMMEDIATE transaction — every column's name, type, nullability and
primary-key position via PRAGMA table_info, and the scope-age index's key
order via pragma_index_info — instead of counting object names. CREATE ...
IF NOT EXISTS preserves a wrong-shaped object that already carries the
expected name, so name presence alone could mark a wrong-shaped named
table valid and hand Phase 2 an unusable table or missing access path. A
wrong-shaped named index is dropped and rebuilt (an index carries no data);
an incompatible named table is rejected and the whole transaction rolls
back with no marker.

The init-barrier and guard-lifetime contracts are now pinned directly
against the ArchiveDb OnceCell/RwLock orchestration through a cfg(test)
path/hook seam, rather than via raw SQLite contention: production-shaped
with_conn callers race a held init and prove none opens a connection until
initialization completes and exactly one initialization runs; a separate
test proves a write-lock contender cannot enter until a with_conn closure
returns and its connection drops.

Adds the full policy lifecycle on one evolving state (active -> kind
disabled -> subscription deleted -> orphan edited -> orphan deleted) and a
deterministic concurrent merge/remove/set interleaving asserting the
subscription stays valid, the explicit choice survives, and no policy row
is deleted by a kinds mutation.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Duncan @wpfleger96
fix(archive): reject a partial scope-age index in M4 shape validation
f462156 
`pragma_index_info` reports an index's key columns but not whether it
carries a `WHERE` predicate, so a partial index with the exact expected
columns passed `scope_age_index_is_correct` and M4 wrote its marker over
it. That index cannot serve the unrestricted scope-age range scan the
Phase-2 prune query needs — SQLite falls back to the primary-key
autoindex — so the marker would certify a missing access path.

Probe `pragma_index_list`'s `partial` flag inside the same
`BEGIN IMMEDIATE` transaction and treat a partial named age index like
any other wrong shape: drop and rebuild it non-partial (an index carries
no data, so a rebuild is safe). A mutation-sensitive test precreates a
partial index with correct table and ordered columns and asserts M4
replaces it with a non-partial index before committing the marker.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Duncan @wpfleger96
fix(archive): validate scope-age index collation and direction in M4
0da7b4e 
pragma_index_info reports key column names but not per-key collation or
sort direction, so a named non-partial index with the exact ordered
columns but COLLATE NOCASE on a key was certified by M4's shape check
even though the binary-equality prune-age query cannot use it (the plan
falls back to the PK autoindex plus a temp B-tree sort). Switch key-shape
validation to pragma_index_xinfo as the single source of key semantics,
asserting name + BINARY collation + ascending direction; index_list stays
only for ownership and partiality. A wrong-collation index is dropped and
rebuilt inside M4's BEGIN IMMEDIATE like any other wrong shape.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Duncan @wpfleger96
refactor(archive): drop M4 schema-shape validator for fail-closed guard
2c555c4 
M4 runs its whole body under one BEGIN IMMEDIATE with the marker written
last, and SQLite DDL is transactional, so no shipped code path can leave
a wrong-shaped object without the marker — a crash rolls everything back.
The shape validator (table_shape_matches, scope_age_index_is_correct,
validate_retention_schema_shape, and their shape constants) only defended
against hand-mutated or foreign-tool DBs, an out-of-scope threat model.

Replace it with the reachability-honest minimum: refuse to certify a
pre-existing retention_policies/archive_meta table (fail closed, no
marker), plain CREATE TABLE otherwise, and an unconditional DROP INDEX IF
EXISTS + CREATE INDEX for the scope-age index (carries no data, so a fresh
build is always correct). The BEGIN IMMEDIATE + pre-lock guard + in-lock
recheck race machinery, seeding, and marker-last ordering are unchanged.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 force-pushed the wpfleger/archive-retention-phase1 branch from 8e06fd1 to 2c555c4 Compare August 14, 2026 14:52
Duncan and others added 4 commits August 19, 2026 11:59
Duncan @wpfleger96
refactor(archive): scope retention to one global observer-frame window
e000280 
Will ruled the per-subscription/per-kind retention machinery down to a
single global setting: observer frames (kind 24200) are kept N days
(default 30, user-customizable); NIP-AM metrics and every other archived
kind are kept indefinitely with no retention state.

Drops the normalized `retention_policies` table, its per-(subscription,
kind) seeding, the policy CRUD/lifecycle helpers, and the
`set_save_subscription_retention` / `list_retention_policies` /
`delete_retention_policy` commands. M4 now creates only `archive_meta`
(k/v) plus the scope-age index and seeds `observer_retention_days=30`,
keeping the same crash-safe BEGIN IMMEDIATE + in-lock recheck +
marker-last discipline. `create_save_subscription` and
`merge_owner_p_kinds` return to plain subscription upserts.

Adds `get_observer_retention_days` / `set_observer_retention_days`
(fail-closed on days < 1 or > 36500) and `archive_size_stats` (physical
file bytes + page/freelist counts, PRAGMAs only). The process init
barrier and gated DB adapter are unchanged.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Duncan
Merge remote-tracking branch 'origin/main' into wpfleger/archive-rete…
84e61d4 
…ntion-phase1

* origin/main: (71 commits)
  Revert "fix(acp): gate relay-signed workflow messages on their attributed author" (#6311)
  fix(desktop): morph the drawer panel icon instead of sliding it (#6306)
  feat(desktop): refine repository-aware project workspaces (#6003)
  Fix mobile Activity thread navigation (#5850)
  perf(desktop): parallelize relay agent directory rebuild (#6258)
  Refine the mobile emoji picker (#5853)
  fix(desktop): exclude archived agents from nest, order regeneration (#5905)
  Add font size and conversation density preferences (#5644)
  fix(desktop): emit camelCase config-write payload fields (#6062)
  fix(desktop): downscale large avatars for agent-share PNG body (#6260)
  fix(desktop): preserve early relay auth challenges (#3320)
  Polish mobile message actions (#5873)
  Refine mobile pairing confirmation (#6018)
  chore(scripts): add buzz-adopt-prod-agents.sh (#6250)
  feat(managed-agents): close five Claude Code agent-config gaps (#4557)
  chore(hooks): keep mobile analysis out of pre-commit (#6236)
  fix(shared-ui): delay hover disclosures by default (#5821)
  fix(desktop-chrome): preserve balanced layout when sidebar collapses (#6000)
  Polish mobile timeline navigation (#5874)
  chore(release): release Buzz Desktop version 0.5.17 (#6234)
  ...

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Duncan @wpfleger96
chore(desktop): keep lib.rs at the file-size ceiling after archive wi…
ab8c1b2 
…ring

Wiring the three archive retention commands and the init-barrier warm
call adds four lines to lib.rs, which origin/main had already grown to
the 1000-line ratchet ceiling. Reclaim the four lines by inlining three
single-use bindings (recovery_mode, custom_harness_dir block scope, and
the huddle-window guard formatting) with no behavior change.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Duncan @wpfleger96
refactor(archive): name the M4 migration for archive_meta
0629266 
The v4 rescope repurposed M4 to create archive_meta + the scope-age index instead of the dropped retention_policies table, but the migration function, marker name, and guard still read add_retention_policies. Rename to add_archive_meta so the marker matches what M4 builds; #5719 is unmerged so no shipped DB carries the old marker.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96 wpfleger96 changed the title feat(archive): add local retention policy storage and lifecycle feat(archive): add observer-frame retention schema and gated DB adapter Aug 19, 2026
Duncan @wpfleger96
Merge remote-tracking branch 'origin/main' into wpfleger/archive-rete…
5b95964 
…ntion-phase1

Integrates #6024 (Rust archive sync task): subscription-mutation commands gain the sync_state param + notify_subscriptions_changed(); DB access stays on the gated ArchiveDb adapter (run_archive_db_task was removed in the v4 rescope), routing sync's list_subscriptions through it too. Command registrations from both branches merged in lib.rs.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 force-pushed the wpfleger/archive-retention-phase1 branch from 900f887 to 5b95964 Compare August 19, 2026 19:52
Duncan and others added 4 commits August 19, 2026 16:18
Duncan @wpfleger96
fix(archive): order the scope-age index for the v4 global age scan
9aa6435 
The scope-age index kept v3's key order (identity, relay, scope_type, scope_value, archived_at, id), which matched the removed per-scope policy query. V4's only consumer is the global observer-frame prune scan — it filters identity + relay + archived_at and does not constrain scope_type/scope_value — so SQLite could not range-seek archived_at past the two unconstrained scope keys and fell back to a bare identity/relay seek plus a temp b-tree.

Move archived_at immediately after (identity, relay) and keep id, scope_type, scope_value trailing so the index stays covering and still supplies the scope-row PK the Phase-2 bounded DELETE needs. Add an EXPLAIN QUERY PLAN test pinning the real prune-candidate SQL to a direct archived_at<? seek with no ORDER-BY temp b-tree.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Duncan @wpfleger96
docs(archive): drop dangling v3 doc-comment opener on the scope-age i…
ed837c2 
…ndex

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Duncan @wpfleger96
fix(archive): serialize ArchiveSizeStats as camelCase
462dae3 
Match the archive module's other frontend-facing response types
(ArchiveBatchResult, agent_usage responses) so the archive_size_stats
wire shape is camelCase before any frontend consumer hard-codes the
snake_case field names.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Duncan
Merge remote-tracking branch 'origin/main' into wpfleger/archive-rete…
4106128 
…ntion-phase1

* origin/main: (37 commits)
  fix(desktop): sender names in notifications + macOS click-through routing (#6427)
  docs: clarify two-layer moderation ownership (#6481)
  Fix mobile thread tail and iOS channel header (#6399)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)
  feat(desktop-messages): show compact Buzz link metadata (#6252)
  feat(workflows): reply in-thread from send_message action (#6178)
  perf(desktop): split discover_acp_providers into cheap and forced paths (#6330)
  fix(desktop): restore recent channel sorting (#6402)
  fix(desktop): isolate main timeline stacking context from focus drawer (#6398)
  fix(desktop): make reconnect repair lossless (#6415)
  fix(hooks): scope pre-push lanes to branch merge-base diff (#6423)
  Enforce a three-day dependency cooldown (#6426)
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  ...

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
@wpfleger96
wpfleger96 merged commit fc2ce67 into main Aug 21, 2026
24 checks passed
@wpfleger96
wpfleger96 deleted the wpfleger/archive-retention-phase1 branch August 21, 2026 17:35
This was referenced Aug 21, 2026
Open
Open
salman1993 added a commit that referenced this pull request Aug 21, 2026
@salman1993
Merge remote-tracking branch 'origin/main' into smohammed/agent-instr…
40ab9ab 
…uctions-prompt

* origin/main:
  fix(benchmarks): wait for scripted event delivery (#6487)
  Polish mobile channel navigation and message sends (#6488)
  Clarify huddle message destination (#6496)
  feat(archive): add observer-frame retention schema and gated DB adapter (#5719)
  fix(desktop): restore human barge-in over agent TTS in huddles (#6431)
  chore(release): release Buzz Desktop version 0.5.18 (#6489)
  fix(desktop): simplify duplicate agent provenance (#6401)
  test(benchmarks): expand Buzz-native dataset (#6448)
  fix(desktop): sender names in notifications + macOS click-through routing (#6427)
  docs: clarify two-layer moderation ownership (#6481)
  Fix mobile thread tail and iOS channel header (#6399)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
baxen pushed a commit that referenced this pull request Aug 21, 2026
bestie00
Merge remote-tracking branch 'origin/main' into agent-crud-cli
2bb12d1 
* origin/main: (88 commits)
  docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946)
  fix(benchmarks): wait for scripted event delivery (#6487)
  Polish mobile channel navigation and message sends (#6488)
  Clarify huddle message destination (#6496)
  feat(archive): add observer-frame retention schema and gated DB adapter (#5719)
  fix(desktop): restore human barge-in over agent TTS in huddles (#6431)
  chore(release): release Buzz Desktop version 0.5.18 (#6489)
  fix(desktop): simplify duplicate agent provenance (#6401)
  test(benchmarks): expand Buzz-native dataset (#6448)
  fix(desktop): sender names in notifications + macOS click-through routing (#6427)
  docs: clarify two-layer moderation ownership (#6481)
  Fix mobile thread tail and iOS channel header (#6399)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)
  feat(desktop-messages): show compact Buzz link metadata (#6252)
  feat(workflows): reply in-thread from send_message action (#6178)
  perf(desktop): split discover_acp_providers into cheap and forced paths (#6330)
  fix(desktop): restore recent channel sorting (#6402)
  ...

Co-authored-by: Bradley Axen <baxen@squareup.com>
Signed-off-by: Bradley Axen <baxen@squareup.com>

# Conflicts:
#	crates/buzz-acp/src/base_prompt.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wesbillman wesbillman wesbillman approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wpfleger96 @wesbillman