ingest: walk TxProcessing once and record where each product's fields sit

[tamirms]

The per-ledger products re-walked every transaction's meta. ExtractLedgerTxParts sized each TxProcessing element whole, then EventsFromTxParts located a V4 meta's fields again, iterated its operations, located each operation's events and sized every event twice, and FeesFromTxParts sized everything in front of SorobanMeta once more. The operation state changes, most of a ledger's bytes, were sized about five times per transaction.

This PR makes the walk sizing every byte once also record where each product's fields sit:

• One walk over TxProcessing sizes every field of every element exactly once, through the generated views. For each element it reads the result pair and asks a predicate whether to keep the element and whether to stop there.
• For a kept element it walks the TransactionMeta V3 or V4 and appends an entry to a pointer-free []uint32 record: the meta's version and size, the SorobanMeta body offset, the DiagnosticEvents array offset, and the element boundaries of every contract-event and transaction-event array, all relative to the meta's first byte. LedgerTxParts points at its entry. Other meta versions are sized whole.
• EventsFromTxParts and FeesFromTxParts decode the entry and slice Meta; they size nothing. The events product carves all transactions' slices from one backing array per element type.
• LedgerTransactionViewByHash and LedgerTransactionViewRange use the same walk: by-hash keeps only the match and stops there, range keeps its page and stops after it.
• Parts built by hand have no entry, so the products locate their meta with the same walk. A walked part whose Meta no longer matches its entry's version and size is an error, never a panic.

Outputs are unchanged: every event is the same bytes at the same address, the empty-but-not-nil shapes and the fee classification are the same.

The hand-written part. The walk reaches the first recorded array of each meta and operation with the generated accessors and sizes every field with its generated view, so no size, count or optional flag is computed by hand. What it writes out is the order of the fields that follow those arrays (in TransactionMetaV3's SorobanMeta, TransactionMetaV4 and OperationMetaV2) and the element layout of TransactionResultMeta and TransactionResultMetaV1. Three tests pin that against the generated code: every recorded offset equals what the generated accessors report, the generated Fields structs of those types keep the field names, types and order the walk assumes, and each V4 operation's events are the ones the generated random-access accessor finds. No generated code changes.

API changes (the extractors are experimental):

TxEvents.TransactionEvents               []xdr.TransactionEventView    (was [][]byte)
TxEvents.OperationEvents                 [][]xdr.ContractEventView     (was [][][]byte)
LedgerTransactionView.DiagnosticEvents   []xdr.DiagnosticEventView     (was [][]byte)
LedgerTransactionView.TransactionEvents  []xdr.TransactionEventView    (was [][]byte)
LedgerTransactionView.ContractEvents     [][]xdr.ContractEventView     (was [][][]byte)
LedgerTxParts.ElemStart, ElemEnd         int                           (new)

A view is its wire bytes, so []byte(ev) gives them without a copy. ElemStart/ElemEnd delimit each transaction's TxProcessing element in the buffer passed to ExtractLedgerTxParts.

Measurements

Pubnet ledgers, Xeon 8375C. Each time is one ledger's ExtractLedgerTxParts, EventsFromTxParts and FeesFromTxParts: the median over the set's ledgers, each ledger the median of 11 runs. The two dense sets each merge 8 or 16 consecutive pubnet ledgers into one.

ledgers	main	this PR	allocations	bytes
LCM V2 over 4 MiB (32 ledgers, median 724 txs)	3.68 ms	1.32 ms	2,036 to 29	418 to 337 KB
8 merged, median 13.9 MiB (16 ledgers, median 2,129 txs)	16.1 ms	5.52 ms	5,392 to 33	1,524 to 990 KB
16 merged, median 24.3 MiB (16 ledgers, median 4,267 txs)	27.8 ms	9.77 ms	10,438 to 37	3,084 to 1,946 KB
recent ordinary ledgers (200 ledgers, median 250 txs)	1.84 ms	0.645 ms	707 to 26	143 to 133 KB

On every ledger the products were checked equal to main's, run from main's own code.

The package's existing benchmarks, main against this PR, benchstat over 10 runs each on the same machine. A row naming a product includes the ExtractLedgerTxParts call it reads from.

Pubnet ledger 58752000 (1.28 MB, 249 transactions):

calls	main	this PR	allocations	bytes
ExtractLedgerTxParts	447 µs	442 µs, no significant change	17 to 3	64.1 to 54.0 KiB
with EventsFromTxParts	700 µs	453 µs (-35%)	133 to 6	79.1 to 69.1 KiB
with FeesFromTxParts	700 µs	452 µs (-35%)	34 to 20	70.8 to 60.5 KiB
with both products	947 µs	464 µs (-51%)	150 to 23	86.1 to 75.6 KiB
LedgerTransactionViewRange, whole ledger	1.58 ms	1.02 ms (-35%)	671 to 251	381 to 233 KiB
LedgerTransactionViewRange, first 10	75.7 µs	68.1 µs (-10%)	37 to 32	6.81 to 6.93 KiB
LedgerTransactionViewByHash, last transaction	729 µs	721 µs (-1%)	44 to 35	3.95 to 2.50 KiB

Tests

• A differential test built only from generated accessors (Fields(), All(), SorobanMeta().Unwrap()) checks every exported extractor on every fixture ledger (LCM V0, V1 and V2, TransactionMeta V0 to V4 under every envelope shape, the fee matrix, V4 operations with ledger-entry changes, parallel and multi-phase TxSets, empty ledgers, the pubnet ledger): same elements at the same addresses, same empty-versus-nil shapes, same fee buckets, same read-path diagnostics and per-operation arity. It is the first commit and passes on main unchanged.
• The walk's record against the generated accessors, the schema tripwire on the mirrored types, and each V4 operation at its true offset.
• Element spans tile TxProcessing, start with the transaction hash and enclose Result and Meta.
• Hand-built parts (trimmed or untrimmed Meta) and parts with a copied Meta give the same products; a shortened or replaced Meta is an error.
• Optional flag 2, hostile array counts, an unknown meta version, bad TxProcessing counts and every truncation of a small ledger are errors, never panics, on every entry point.
• Every returned slice is allocated at its final length. ExtractLedgerTxParts allocates three objects per ledger whatever its transaction count, as long as the offset record fits its presize of 14 words per transaction; that is pinned at 4 and 256 transactions and on the pubnet ledger.
• A fuzz target checks the walk and the events product against the generated-accessor reference on any input the walk accepts.

Supersedes #5972, #5996 and #6011, and takes ElemStart/ElemEnd from #6010.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

Three moderate issues remain unresolved in result shape preservation, metadata validation, and diagnostic-event sizing.

Review effort: Lite
Findings: 1

Open (1)

• Validate metadata content identity before using cached offsets · New

What changed in this PR

This PR refactors ingest extraction to walk TxProcessing once, cache metadata offsets, and provide zero-copy typed event views.

Changes:

• Adds cached metadata offsets and transaction spans.
• Reworks event, fee, range, and hash extraction.
• Updates APIs, tests, benchmarks, and documentation.
• Review issues remain around empty-result shape, metadata validation, and diagnostic-event sizing.

File	Summary
ingest/​tx_processing_walk.go	Shared walk and offset records.
ingest/​tx_processing_walk_test.go	Walk, malformed-input, span, allocation, and fuzz tests.
ingest/​transaction_view.go	Cached range and hash views.
ingest/​transaction_view_test.go	Updated typed event assertions.
ingest/​transaction_events_view.go	Superseded event-walking logic removed.
ingest/​transaction_events_view_test.go	Migrated event extractor tests.
ingest/​ledger_close_meta_view_nav.go	TxProcessing navigation state.
ingest/​extract.go	Cached parts, spans, fees, and typed events.
ingest/​extract_real_ledger_test.go	Updated real-ledger comparisons.
ingest/​extract_equivalence_test.go	Generated-accessor equivalence coverage.
ingest/​CHANGELOG.md	API and performance documentation.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[tamirms]

caller can replace a walked part's Meta with different bytes having the same version and length
That contradicts the LedgerTxParts.Meta contract that only an identical byte copy may replace the walked

this finding is invalid because your scenario requires you to replace Meta with a corrupted value