Packfile library — shared infrastructure for ledger + events frozen artifacts

[karthikiyer56]

Packfile library — shared infrastructure for ledger + events frozen artifacts

TL;DR

Packfile is a shared on-disk format and Go library — owned inside this repository, not external — that stores immutable, query-ready artifacts. It is the single substrate underneath two parallel application paths:

• Ledger pack files — consumed by the backfill epic (PRD: Full History RPC Backfill Pipeline #678).
• Events cold segments — consumed by the Full-History Event Store epic #665.

This epic tracks the packfile-side work end-to-end: design, CI infrastructure for compression, low-level encoding primitives, and generic writer + reader. Items 6 and 7 (originally planned zstd-enabled wrappers) were superseded by the caller-supplied-codec design in #702 + #712 — each consumer plumbs the zstd codec at its own call site.

Where this epic sits

• Standalone, dependency-wise. Both consumer epics (PRD: Full History RPC Backfill Pipeline #678 backfill, Full-History Event Store #665 events) depend on packfile primitives. Packfile work depends on neither consumer.
• All packfile-side work is complete (items 1-5 merged or in-review). Items 6 and 7 were superseded by the caller-supplied-codec design — there is no separate "zstd-enabled" primitive; *zstd.Compressor / *zstd.Decompressor satisfy the packfile codec interfaces directly via compile-time assertions in the packfile tests.
• Backfill (PRD: Full History RPC Backfill Pipeline #678) and events (Full-History Event Store #665) can proceed in parallel.

Contents (linked work)

#	Item	State	PR / issue
1	Packfile design doc	merged	#633
2	Zstd CGo wrapper for packfile compression (CI integration)	merged	#650
3	Intpack FOR encoding + packfile offset index codec	merged	#649
4	Generic packfile writer	merged	#702
5	Generic packfile reader	merged	#712
6	Zstd-enabled packfile writer primitive — superseded; #702 ships caller-supplied codec, consumers plumb zstd at the call site	superseded	#720 (closed)
7	Packfile decoder primitive — superseded; #712 ships caller-supplied codec, consumers plumb zstd at the call site	superseded	never filed

Consumers (cross-epic)

• Backfill epic PRD: Full History RPC Backfill Pipeline #678 — "Backfill 13: Cold ledger store (packfile-backed reader + writer)" (Backfill 13: Cold ledger store (packfile-backed reader + writer) #695) consumes the generic writer + reader directly, plumbing *zstd.Compressor / *zstd.Decompressor at the call site.
• Full-History Event Store epic Full-History Event Store #665 — "[events] Cold segment writer" ([events] Cold segment writer #656) and "[events] Cold segment reader" ([events] Cold segment reader #657) similarly consume the generic writer + reader directly.

References

• Packfile design doc: design-docs/packfile-library.md on the feature/full-history branch (landed via PR Add packfile library design doc #633; drift fix in flight at Update packfile design doc to match merged writer and reader implementations #737).
• All packfile work lands on feature/full-history.

[tamirms]

All packfile work is complete — items 1–5 merged, items 6–7 superseded by the caller-supplied-codec design (#702/#712). Closing this epic as done.