Streaming epic — orchestration + live ingestion in the unified RPC v2 service

[karthikiyer56]

Streaming epic — orchestration + live ingestion in the unified RPC v2 service

TL;DR

• The streaming epic owns every part of the unified stellar-rpc service that isn't backfill, events, or packfile.
• It's the orchestration layer that ties everything together end-to-end.
• Sibling to the backfill (PRD: Full History RPC Backfill Pipeline #678), events (Full-History Event Store #665), and packfile (Packfile library — shared infrastructure for ledger + events frozen artifacts #714) epics under the unified-service umbrella PRD (PRD: Stellar RPC v2 — Unified Service #723).

What this epic owns

The orchestration of the unified service's lifecycle:

• Run backfill on first start, if needed (delegates the actual chunk work to the backfill epic).
• Any cleanup / pruning stages, if needed.
• Start live ingestion of the network tip via captive Stellar Core.
• Serve live queries alongside historical ones.
• Freeze data from active storage to immutable files, on the cadence the design doc settles on.

Plus the supporting infrastructure:

• Active storage layer — RocksDB-backed mutable stores for in-flight live data.
• Captive Stellar Core wiring.
• JRPC server wiring — getStatus, getHealth, and the rest of the v2 method set.
• Service entry point — func main, TOML parsing.
• Service-startup config validation — config-immutability checks (e.g., LEDGERS_PER_TX_INDEX).
• Logging plumbing — initializes a single, global, concurrent-safe, structured logger at service startup that writes to the log file specified in the unified TOML's [LOGGING] section. The same logger reference is handed down to every subsystem at construction time; subsystems do NOT construct their own loggers and do NOT open their own log files. Mutation-safe writes ensure logs from different subsystems don't trim or overwrite each other. Implementation lands as a thin slice under this epic when the streaming design doc breaks down. Until then, individual backfill / streaming slices that need to log just take a *logrus.Logger (or narrow interface) via constructor arg; tests pass a discard or capture logger.

The exact stages and their boundaries may shift as the design doc lands. This issue intentionally avoids enumerating fixed phases until then.

Where it depends on the other epics

• Calls into backfill (PRD: Full History RPC Backfill Pipeline #678) for the chunk-fill subroutine on first start.
• Calls into events (Full-History Event Store #665) for active events handling and live event ingestion (hot side).
• Reads + writes via packfile primitives (Packfile library — shared infrastructure for ledger + events frozen artifacts #714) for freezing data from active stores to immutable files.

Status

• Design doc in active draft on the design-doc branch.
• One open sub-issue today: the design doc tracker.
• Implementation breakdown into thin slices happens once the design doc lands on feature/full-history.