POC: Full-history RPC benchmarking — May 2026 session reference

[chowbao]

POC / reference PR. Not for merge — captures the end-state of the May 2026 full-history RPC benchmarking session so the team can see exactly what was built and measured. Numbers, methodology, raw CSVs, and tooling all in one tree. No tests; some files remain after merging upstream rpc-hack (which has since landed PR #740 and #743).

What's in this PR

1. Merge of upstream/rpc-hack (d5bb683) — upstream landed PR Add eventstore: per-Chunk hot + cold event storage for full-history #740 (eventstore) and PR Add exploratory full-history-backfill script: BSB -> ColdStoreWriter #743 (full-history-backfill), superseding our original cherry-pick. Conflicts resolved in favor of upstream for the eventstore + events files; go.mod kept both indirect deps (streamhash + xxh3); go.sum reconciled via go mod tidy. Minor post-merge fixups: scripts updated for the new error-returning APIs (Reader.Offsets, HotStore.EventCount) and the removal of pkg/geometry (now pkg/chunk.LedgersPerChunk).
2. Phase 5 — cold-cache + concurrency bench extensions (7757289):
   • cache.go — posix_fadvise(FADV_DONTNEED) eviction + mincore residency check (Linux only). No sudo required; sidecar-fd pattern.
   • bench_concurrent_runner.go — shared per-worker scaffold (evict → open ColdStoreReader → caller workload → close → record); used by both point and range concurrent benches.
   • bench_ledger_cold.go — ledger-point-cold-open (1 worker, fresh open per iter) and ledger-point-concurrent-cold (N workers).
   • bench_ledger_range_cold.go — ledger-range-concurrency-sweep: grid of (workers × page-size) cold-open range reads, per-cell p50/p99 + saturation summary.
3. Phase 1–4 bench harness (d552a95) — 8-file harness under cmd/stellar-rpc/scripts/bench-fullhistory/ covering ledger-point/range, tx-page, tx-by-hash, events (4 filter scenarios × 2 tiers). Also: cold-read/, migrate-cold/, spot-check/, verify-pack/, upload-cold/ supporting tools.
4. Results, learnings, raw CSVs (b526fa8, 7757289, 4908d62):
   • BENCHMARK_GOAL.md — full writeup of Phase 0–4 (plan, methodology, results, interpretation).
   • SESSION_LEARNINGS.md — state-of-the-world, decisions, surprises, gotchas, open follow-ups. Includes Phase 5 addendum with cold-cache methodology and getLedgers projections.
   • bench-out/ — 26 per-scenario CSVs + summary.csv from Phase 1–4; Phase 5 concurrency sweep (ledger-range-concurrency-sweep.csv) plus rep2/, rep3/ for run-to-run variance and post-merge/ confirming the harness behaves identically after the merge.

Headline results

Phase 1–4 (warm-cache, single-thread, chunk 5000)

Workload	Hot vs Cold-NVMe
Ledger / tx-page / tx-by-hash	~tie; zstd decompression (~1.3 ms/ledger) dominates
Events sequential (no-filter)	Cold 2.3× faster
Events high-cardinality filter (contract)	p50 tie; cold tail much worse (p99 44 ms vs 12 ms)
Events low-cardinality filter (topic)	Hot 1.8× faster

Full table (26 rows) in BENCHMARK_GOAL.md and bench-out/summary.csv.

Phase 5 — Cold-cache getLedgers concurrency sweep (chunks 4999–5050)

Each iteration: evict the chunk's packfile, open a fresh ColdStoreReader, read N consecutive ledgers, close. 100 iters/worker.

Page size	Peak workers	Peak ops/sec	p50 @ peak	p99 @ peak
1	48	5,721	8.0 ms	11.7 ms
10	24	1,262	17.9 ms	27.2 ms
20	24	687	33.2 ms	47.4 ms

Storage-layer getLedgers model: T(N) = cold-open (~1.3 ms) + N × per-ledger-decode (~1.3 ms). Default page size 50 → ~75 ms p50, ~150 req/s at the box's saturation knee. Theoretical aggregate ceiling ~7,000–10,000 cold ledgers/sec.

Below the knee, p50/ops are stable to ~2% across runs. Past the knee, p99 inflates 3–5× and peak ops/sec drifts 10–30% run-to-run. Variance + reproducibility analysis in SESSION_LEARNINGS.md Phase 5 addendum.

How to reproduce

export PATH="/tmp/go126/go/bin:$PATH"   # Go 1.26 needed for streamhash
go build -o /tmp/bench-fullhistory ./cmd/stellar-rpc/scripts/bench-fullhistory/...

# Seed hot store (only for hot-tier benches)
/tmp/bench-fullhistory seed-hot --chunk 5000 \
    --cold-dir /mnt/nvme/disk2/ledgers/cold \
    --hot-dir /mnt/nvme/disk2/ledgers/hot-5000

# Phase 1 warm-cache point lookup
/tmp/bench-fullhistory ledger-point --tier=cold --chunk=5000 --iters=2000 \
    --hot-dir=/mnt/nvme/disk2/ledgers/hot-5000 --out=bench-out

# Phase 5 cold-cache concurrency sweep
/tmp/bench-fullhistory ledger-range-concurrency-sweep \
    --workers 1,2,4,8,16,24,32,48,64 --page-sizes 1,10,20 --iters 100 \
    --chunk-lo 4999 --chunk-hi 5050 --out bench-out

Full reproduction recipes in SESSION_LEARNINGS.md.

Caveats / known not-production

• Cold txhash uses a sorted-.bin shim, not the production RecSplit index. Sub-millisecond lookup either way; dwarfed by ~12 ms ledger-decode in the tx-hash path.
• Warm-cache, single-threaded only. → Cold-cache + multi-worker now covered by Phase 5.
• Single dataset (pubnet 50M–60M) for everything. Other chunks may differ in tx/event density.
• EBS tier dropped from scope at user request.
• No multi-chunk iterator — every bench stays within a single 10K-ledger chunk.
• Nothing wired into methods.getLedger / getEvents — stand-alone bench drivers only. Production routing layer is separate work.
• Concurrency curve untested past 64 workers. Page=1 still scaling at 48, collapses by 64 — real saturation likely 50–60.
• All numbers are storage-layer only — no HTTP/TCP/TLS/JSON-RPC framing, no SQLite-first lookup, no XDR→JSON serialization. End-to-end RPC latency adds overhead on top.

Files NOT to merge

Most of this PR is reference material:

• migrate-cold/, spot-check/, verify-pack/, upload-cold/ — session artifacts.
• bench-out/, BENCHMARK_GOAL.md, SESSION_LEARNINGS.md — reference docs and raw data.

The bench harness in cmd/stellar-rpc/scripts/bench-fullhistory/ is the most reusable piece; productizing would require tests, configurable paths, and integration with the actual JSON-RPC handler.

After the merge, the cherry-pick of PR #740 (bfa3860) is content-redundant — the upstream version of those files is what's on this branch now.