bench(fullhistory): synthetic-ledger datasets via stellar-core apply-load as an ingest/query bench source

[tamirms]

Motivation

We can only benchmark ingestion/query against real captured ledgers today. We want a controllable dataset whose transaction profile we set deliberately (tx mix, Soroban load, etc.), so we can characterize how ingestion and queries behave under specific load shapes rather than only whatever pubnet happened to produce.

Background

• go-stellar-sdk PR ingest/loadtest: add stellar-core apply-load tooling go-stellar-sdk#5940 (merged) adds ingest/loadtest.ApplyLoad, which drives stellar-core's apply-load to emit synthetic ledgers from a .cfg profile (default-apply-load.cfg).
• The bench (cmd/stellar-rpc/scripts/bench-fullhistory) already abstracts ingest input behind --source (sources.go: pack, bsb) as a ledgerbackend.LedgerStream. cold-ingest/hot-ingest consume it; the resulting cold packfiles + hot RocksDB feed the cold-*/hot-* query benches.
• Chunk size is fixed at LedgersPerChunk = 10_000 (pkg/chunk/chunk.go).

Proposed work

1. Synthetic generator — a command/tool that runs loadtest.ApplyLoad with a given apply-load config and writes a chunk-aligned dump of K × 10_000 ledgers. Reproducible (config + seed) so runs are comparable.
2. New bench source — --source=synthetic (synthStream) yielding the dump's raw ledgers per chunk, so cold-ingest/hot-ingest run unchanged on it. (alt: reuse --source=bsb against a local datastore)
3. Query benches — confirm the cold packs + hot store produced by ingesting the synthetic source feed cold-*/hot-* with no further changes (point --cold-dir/hot store at the synthetic-derived outputs).

Acceptance criteria

• Generate an N-chunk synthetic dataset from an apply-load config.
• cold-ingest/hot-ingest --source=synthetic --types=... ingest it and report per-stage timings.
• Resulting stores drive cold-*/hot-* query benches.
• README documents the end-to-end flow.

Dependencies

• go-stellar-sdk dep bumped to a revision including ingest/loadtest (ingest/loadtest: add stellar-core apply-load tooling go-stellar-sdk#5940).
• A stellar-core binary with apply-load support available to the generator.

Open question

Source wiring — dedicated --source=synthetic (A) vs reuse --source=bsb over a local datastore (B).

[chowbao]

FYI

Reproducible (config + seed) so runs are comparable

It doesn't seem like apply-load supports passing a seed so the actual contents of the synthetic ledgers will always be a little different

Can the ledgers be byte-for-byte identical given the same seed?
Not with stellar-core's apply-load as it exists today

But as long as we save the config and the config is consistent we should be fine. I'll also upload the synthetic ledgers to gcs so we can always redownload them from there like a fake LCM GCS data lake

[chowbao]

PR with benchmark results