Cold txhash: getTransaction(hash) end-to-end + demo + tx-hash bench --xdr-views - #747
Merged
Merged
Conversation
5 tasks
txquery.ColdLookup composes the cold readers from stellar#746 into the same db.TransactionReader interface methods.GetTransaction already consumes from the hot DB: txhash.ColdReader.Lookup hash -> ledgerSeq (MPHF + fingerprint) ledger.ColdStoreReader ledgerSeq -> raw LCM bytes xdr.LedgerCloseMeta UnmarshalBinary ingest.NewLedgerTxReader... walk the LCM, match by hash db.ParseTransaction materialize db.Transaction stores.ErrNotFound from either reader maps to db.ErrNoTransaction so the type drops directly into the RPC handler. Walking past the end of the LCM (residual MPHF false positive on an unseen hash) also maps to ErrNoTransaction. bench-fullhistory additions: - demo-tx-hash subcommand: opens a cold MPHF + cold pack pair and prints the parsed db.Transaction for an arbitrary hash, for quick smoke-testing. - tx-hash --tier=cold-mphf-txquery: routes per-op through ColdLookup, so the reported latency reflects the full pipeline (pack read + LCM unmarshal + ingest parse + db.Transaction materialization). - tx-hash --xdr-views: when set, after the per-iter ledger fetch the bench finds the matching transaction by walking the raw LCM as XDR views (LedgerCloseMetaView -> V0/V1/V2 -> TxProcessing.Iter -> Result.TransactionHash) and comparing hash bytes, instead of lcm.UnmarshalBinary + struct scan. Same pattern as extractTxHashesView in bench_ingest_raw_txhash.go, reusing its txResultMeta interface for the V2-vs-V0/V1 result element type split. Ignored (with warning) for --tier=cold-mphf-txquery (decode happens inside ColdLookup). Bench label and CSV filename get an "-xdrviews" suffix so paired runs don't overwrite each other. Known gap: the cold MPHF seeder indexes outer hashes only via lcm.TransactionHash(i), so fee-bump inner-hash lookups miss in the cold tier even though the hot DB resolves them. Fix is a seeder change, not a lookup change (noted in cold.go). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tamirms
force-pushed
the
txhash-reads
branch
2 times, most recently
from
May 21, 2026 15:24
ac808a6 to
fa3347a
Compare
…timing; full view materializer; drop ColdLookup
Query benches now split per tier (cold-tx-page/hot-tx-page,
cold-tx-hash/hot-tx-hash, cold-events/hot-events) because cold and
hot have genuinely different methodologies that can't share one
loop body without lying about what each tier costs in production:
cold-* per-iter: evict packfile from OS page cache, open a fresh
reader, do the call, close. No warmup.
hot-* shared HotStore handle for the whole run, N-iter
block-cache warmup before timed iters.
Previously tx-page/tx-hash/events all used --tier=hot|cold but
both branches ran the hot methodology (shared handle, warmup) even
on cold — so the cold numbers were measuring steady-state cold-
reader-after-warmup, not a realistic cold-cache request.
tx-hash CSVs now decompose per-iter latency into open_ns (cold
only), lookup_ns, fetch_ns, scan_ns, materialize_ns, total_ns. Same
shape for tx-page (cursor_resolve + fetch + decode + scan) and
events (filter + fetch). --xdr-views (default true) toggles both
scan and materialize between an XDR-view path and the
production-shape lcm.UnmarshalBinary + db.ParseTransaction
round-trip. CSV filename gets a -xdrviews / -roundtrip suffix so
paired runs don't overwrite each other.
The view materializer (tx_hash_helpers.go) is end-to-end view-based:
no lcm.UnmarshalBinary anywhere on the timed path, no MarshalBinary
on any field. Result/Meta/Envelope come from sub-view .Raw() slices
into the original raw buffer (per the SDK's xdr_views_generated.go
.Raw() exposed on every generated view type). Events extracted via
SorobanMeta (V3) or TransactionMetaV4 (V4) sub-views' .Raw() — one
per DiagnosticEvent / TransactionEvent / per-op ContractEvent.
Envelope navigation walks GeneralizedTransactionSet phases →
components → txs in apply order (or ParallelTxsComponent stages →
clusters for V1-shape phases), mirroring
xdr.LedgerCloseMeta.TransactionEnvelopes()'s walk so the i'th view
matches what production resolves for the matching hash.
Bookkeeping fields (TransactionHash, ApplicationOrder, FeeBump,
Successful, Ledger.{Sequence,CloseTime}) all derived from view-level
primitives — envelope Type discriminator, result Code, header
LedgerSeq + ScpValue.CloseTime.
TestMaterializeViewsMatchesRoundtrip cross-checks the view path
against materializeRoundtripFromLCM (lcm.UnmarshalBinary + ingest
reader + db.ParseTransaction) for both V3-meta-no-soroban and
V4-meta-with-events shapes; all 12 db.Transaction fields must match
byte-for-byte. Without this, a quietly-wrong byte slice from the
view path could make the latency comparison the bench produces
meaningless.
txquery.ColdLookup deleted (no production caller — was only
constructed by the bench/demo). demo-tx-hash deleted (cold-tx-hash
now covers the same ground). bench_tx_hash.go, bench_tx_page.go,
bench_events.go deleted (replaced per tier).
Flag normalization:
--dir on cold-ledgers / hot-ledgers → --cold-dir / --hot-dir
--chunk on hot-ledgers / *-ingest → uint (was int64 + -1 sentinel)
build-txhash-index: --out (the .idx output) → --idx-out,
--csv-out → --out
ingest-raw-txhash: --out-dir → --bin-out, --csv-out → --out
--out now uniformly means CSV output directory everywhere.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes chowbao#3, rebased on top of #746 (now in
rpc-hack).Summary
txquery.ColdLookupcomposes the cold readers from Cold txhash MPHF store: streamhash-backed point lookup + bench wiring #746 into the samedb.TransactionReaderinterfacemethods.GetTransactionalready consumes from the hot DB:stores.ErrNotFoundfrom either reader maps todb.ErrNoTransaction, so the type drops directly into the RPC handler. Walking past the end of the LCM (residual MPHF false positive on an unseen hash) also maps toErrNoTransaction.bench-fullhistory demo-tx-hash— a one-shot subcommand that opens a cold MPHF + cold pack pair and prints the parseddb.Transactionfor an arbitrary hash, for quick smoke-testing.bench-fullhistory tx-hash --tier=cold-mphf-txquery— new tier that routes per-op throughColdLookup, so the reported latency reflects the full pipeline (pack read + LCM unmarshal + ingest parse +db.Transactionmaterialization). Existinghot/cold/cold-mphftiers unchanged.bench-fullhistory tx-hash --xdr-views— optional flag. When set, after the per-iter ledger fetch the bench finds the matching transaction by walking the raw LCM as XDR views (LedgerCloseMetaView → V0/V1/V2 → TxProcessing → Iter → Result.TransactionHash) and comparing hash bytes, instead oflcm.UnmarshalBinary+ struct scan. Same pattern asextractTxHashesViewinbench_ingest_raw_txhash.go, reusing itstxResultMetainterface for the V2-vs-V0/V1 result element type split. Ignored (with warning) for--tier=cold-mphf-txquerysince that tier's decode happens insideColdLookup. Bench label and CSV filename get an-xdrviewssuffix so paired runs don't overwrite each other.Known gap (from #3)
The cold MPHF seeder indexes outer hashes only via
lcm.TransactionHash(i), so fee-bump inner-hash lookups miss in the cold tier even though the hot DB resolves them. Fix is a seeder change, not a lookup change (noted incold.go).Test plan
go test ./cmd/stellar-rpc/internal/fullhistory/pkg/txquery/...— unit tests build a real cold pack + matching MPHF and exercise the round-trip + unknown-hash pathgo test ./cmd/stellar-rpc/internal/fullhistory/...— no regressions in sibling packagesgo build ./cmd/stellar-rpc/scripts/...andgo vet ./cmd/stellar-rpc/scripts/... ./cmd/stellar-rpc/internal/fullhistory/...ingest-raw-txhash --all --xdr-views(1001 chunks, 2.92 B tx hashes, 12 min 1.5 s) →build-txhash-index(56 s, 12.5 GB merged MPHF) on chunks 5000–5999 to confirm the rebuilt pipeline produces a usable cold MPHF at scaleColdLookupintomethods.NewGetTransactionHandleras thedb.TransactionReaderand verify the RPC response shape matches the hot path against a known tx hash🤖 Generated with Claude Code