Streaming daemon: Slice 1 · Layer 2 — Storage primitives

[chowbao]

Part of #808 (epic) · Builds on: #809 · Reference implementation: #806

Slice 1 · Layer 2 of 4 — Storage primitives. Produces and freezes a single chunk's cold artifacts from a ledger source. Still no long-running daemon — these are the building blocks Layer 3 drives.

Context

A chunk is the atomic unit of ingestion/freezing (10k ledgers). This layer adds the per-chunk hot RocksDB and the processChunk primitive that materializes a chunk's cold .pack from whatever source is cheapest. The merged ledger stores compose into one multi-CF hot DB so a ledger commits as one atomic synced WriteBatch.

Scope

• Per-chunk hot DB lifecycle (pkg/stores/hotchunk)
  • one RocksDB per chunk holding the ledgers column family (events/txhash CFs are added in later slices)
  • openHotDBForChunk:
    • ready → open existing
    • transient | absent → wipe + recreate (dirent + grandparent fsync)
    • fatal on a ready key whose dir is missing (hot-volume loss — no auto-heal)
  • discardHotDBForChunk: transient bracket → rmdir → delete key
  • a read-only view for freezing; the transient / ready state machine
• processChunk
  • single-pass materialization of a chunk's requested cold artifacts (here: ledgers → .pack) through the one-write protocol
  • per-kind idempotency (skip if already "frozen")
• backfillSource — picks a chunk's ledger source in a fixed preference order:
  • ready + complete hot DB
  • frozen local .pack (when ledgers not requested)
  • bulk backend — with the loss-vs-staleness rule + a bounded waitForBackendCoverage (fatal on timeout) for backend-only chunks above a lagging tip
• Ledger cold-ingest wiring (ingest/driver, ingest/service) — drive the ledger ingester into the hot DB / cold .pack

Acceptance

• Atomicity: a ledger is fully present or fully absent; create/discard idempotent across mid-op crashes.
• Hot-volume loss: ready-but-missing-dir fatals with a curated recovery instruction; the read handle closes before any same-tick discard.
• Determinism: re-materialization overwrites at the canonical path and is byte-identical; widening re-derives a covered chunk from local .pack with no download; the backend-lag wait fires only for genuinely backend-only chunks.
• Green stacked on Layer 1 (build / vet / -short).

Design references

• Per-chunk hot DB lifecycle → The chunk hot DB · Hot DB helpers
• processChunk + backfillSource → Backfill → The primitives
• The .pack cold artifact → Filesystem artifacts
• Hot-tier semantics → gettx §5 Hot tier

Dependencies

• Depends on: Layer 1 (Foundations)
• Composes (already merged): pkg/stores/ledger, internal/packfile, pkg/rocksdb, pkg/chunk; ledger ingest primitives from Ingestion: all data types for both cold and hot stores #765

Out of scope

• the resolver/executor, ingestion loop, lifecycle tick → Layer 3
• events CFs / tx-hash CF + .bin → Slices 2 & 3

[chowbao]

Closing — the epic #808 has been re-sliced by phase instead of by data type (see the breakdown comment). The "events" and "tx-hash" slices dissolve (each data type is just a Kind/CF/cold-writer parameter to processChunk, and the window index is a backfill primitive), and some scope is dropped for the MVP. The work now lives under two issues:

• Phase 1 · Backfill — Streaming daemon: Phase 1 — Backfill (catch-up + all primitives) #815
• Phase 2 · Live ingestion + lifecycle — Streaming daemon: Phase 2 — Live ingestion + lifecycle #816This issue's scope (Layer 2 · Storage primitives): split by tier — processChunk + backfillSource (cold materialization) → Phase 1 (Streaming daemon: Phase 1 — Backfill (catch-up + all primitives) #815); the per-chunk hot DB lifecycle (hotchunk) → Phase 2 (Streaming daemon: Phase 2 — Live ingestion + lifecycle #816).