ingest: one ledger walk — ExtractLedgerTxParts + EventsFromTxParts/FeesFromTxParts (supersedes the extractor bundles)

[karthikiyer56]

TL;DR

Reshape of the experimental view-extractor family per tamirms' review, spec'd in stellar/stellar-rpc#912: one function walks the ledger; every product is a plain function of the walk's output.

Before (bundles)	After (parts + products)
• ExtractTxHashes — hashes, one walk • ExtractLedgerEvents — hashes + events, one walk • ExtractFees(lcm, passphrase) — fees, ANOTHER walk + whole-TxSet SHA-256 pairing pass	• ExtractLedgerTxParts(lcmView) — the ONLY TxProcessing walk, returns per-tx handles • EventsFromTxParts(txParts) — events product, no walking • FeesFromTxParts(txParts) — fees product, no walking, no passphrase
• "one walk per ledger" is a doc-comment promise each bundle must make • rpc v2's hot path walks twice to get all three products	• "one walk per ledger" is a structural property of the API • each consumer composes exactly the subset it needs

Why the old functions are DELETED, not kept alongside

Deleted	Replaced by
ExtractTxHashes	txParts[i].Hash off the walk
ExtractLedgerEvents + LedgerTransactionEvents	EventsFromTxParts + TxEvents (hash lives on txParts)
ExtractFees(lcm, passphrase) + LedgerFees.LedgerSequence/CloseTime	FeesFromTxParts (no passphrase; seq/close-time are the caller's)

• None of these ever shipped — all Experimental:, still under CHANGELOG Pending. Deleting now is free; one release later it's a deprecation cycle + major-version break.
• Keeping ExtractFees would keep a SECOND fee definition (envelope gate, uint64 wrap) — same ledger, different buckets, permanent "which one is right?" trap.
• "One walk per ledger" is only a structural property if the API offers no second way to walk — a surviving bundle is exactly that second way.

The API

txParts, err := ingest.ExtractLedgerTxParts(lcmView) // the one walk
// txParts[i]: Hash, InnerHash, FeeBump, Result view, Meta view (views alias the buffer)

txEvents, err := ingest.EventsFromTxParts(txParts)   // []TxEvents, index-aligned with txParts
fees, err := ingest.FeesFromTxParts(txParts)         // LedgerFees{ClassicFeesPerOp, SorobanInclusionFees}

Consumer (stellar-rpc v2)	Composition
Hot ingestion (live streaming)	• parts → events + fees • one walk, all three products
Cold backfill (bulk path)	• parts → events • fees never computed
Fee warm-up replay on restart (stellar/stellar-rpc#888)	• parts → fees • events never computed, no passphrase to plumb

Fee classification — TxProcessing only

The envelope leaves fee ingestion entirely: soroban-ness and op count are read from the result + meta, so the TxSet pairing pass, the per-envelope hashing, and the passphrase parameter all delete.

Transaction shape (all read from TxProcessing)	Contribution
Meta V3/V4 carries SorobanMeta, Ext.V == 1	• FeeCharged − (TotalNonRefundable + TotalRefundable resource fee charged) → Soroban bucket
SorobanMeta present, ext absent	• skipped (always present since P21 — safety net)
SorobanMeta absent	• FeeCharged / opCount → classic bucket • opCount = number of per-op results (outer result for txSUCCESS/txFAILED, INNER result pair's for fee-bumps)
No per-op result list (txINTERNAL_ERROR) or empty	• skipped
Fee-bump	• FeeCharged from the OUTER result, opCount from the INNER result
Negative FeeCharged / negative resource-fee component (or int64-overflowing sum) / resource fee > FeeCharged	• error

Delta from v1's IngestFees	Why it's deliberate
Transactions whose operations never ran are skipped — their results carry no per-op list (v1 counted them via envelope ops)	• two ways in: core malfunction (txINTERNAL_ERROR), or an account invalidating its own pending tx with an operation it signed inside another account's tx (merge / signer removal / sequence bump — the godoc has the worked example); pre-P20 tx sets also produced these organically • the dropped fee says nothing about what a fee bidder should pay • a stellar-rpc#883 getFeeStats divergence therefore means "go look at that ledger", not automatically "core had an incident"
Resource fee > FeeCharged is an error (v1 wrapped the uint64)	• with mechanism parity gone, there's no reason to let corrupt input drop a ~2^64 sample into the fee window

Parity contract: identical output to v1 on organic current-protocol traffic — deltas confined to never-ran transactions — verified empirically on the recorded pubnet ledger (envelope-derived and results/meta-derived definitions agree per-tx on op count, soroban gate, and ext presence across all 249 txs, incl. 41 failed and 113 fee-bumps).

How it's verified

Layer	What it checks
Fixture matrix (extract_fees_test.go)	• one cell per classification rule row: LCM V0/V1/V2 × meta V0–V4 × envelope V0/V1/fee-bump × result shapes • results-derived op counts: failed multi-op, fee-bump inner counts, results_count_is_authoritative (result count ≠ envelope count) • the no-op-list family skipped: txINTERNAL_ERROR, self-invalidated txNO_ACCOUNT, fee-bump-inner-internal-error • soroban gate flips pinned: SorobanMeta present w/ classic op → soroban; soroban op w/o SorobanMeta → classic • error cells: negative fees, int64 overflow, resource fee > FeeCharged
TestFeesFromTxParts_TxSetNotConsulted	• gutting every envelope from the TxSet changes nothing (was a missing-envelope ERROR under the old extractor)
Real-ledger equivalence (extract_real_ledger_test.go)	• pubnet ledger 58752000 (249 txs) • parts: hash/inner-hash/fee-bump/result-bytes/meta-bytes vs the parsed reader • events: wire-identical vs GetTransactionEvents • fees: re-derived with the OLD envelope-based oracle — deliberately kept as the standing cross-definition check
V4 meta walker (metaEventRaws)	• one Fields() locate pass instead of per-field prefix walks (DiagnosticEvents is the LAST field — its accessor re-walked the whole meta per tx on the read path) • ~10% on a synthetic V4 walker (10 ops × 3 events), growing with op count; the pubnet fixture is protocol 22 / all-V3, where measurements show parity • V3 arm uses the locate pass only when both event sets are wanted
Benchmarks (same ledger)	• walk alone: 0.29 ms / 17 allocs • walk + events: 0.42 ms / 137 allocs • walk + fees: 0.42 ms / 34 allocs (old ExtractFees bundle: 0.58 ms / 67 allocs) • walk + all products: 0.58 ms / 154 allocs — the hot composition now costs what the fee bundle alone used to • parsed path: ~5.7 ms / ~95k allocs

[Copilot]

Pull request overview

Adds a zero-copy, view-based fee extractor matching stellar-rpc v1 fee classification.

Changes:

• Introduces LedgerFees and ExtractFees.
• Adds comprehensive synthetic, real-ledger, multiphase, and benchmark coverage.
• Extends transaction fixtures with fee-processing data.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
ingest/extract.go	Implements fee extraction and classification.
ingest/extract_fees_test.go	Adds fee matrix and oracle tests.
ingest/extract_fees_multiphase_test.go	Tests mixed transaction-set phases.
ingest/extract_real_ledger_test.go	Tests real-ledger equivalence.
ingest/extract_bench_test.go	Benchmarks parsed and view paths.
ingest/transaction_view_test.go	Enhances shared transaction fixtures.
ingest/CHANGELOG.md	Documents the experimental API.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tamirms]

Wiring this into rpc v2 surfaces an API-shape problem worth fixing while the extractor family is still experimental.

rpc v2 has three consumers of per-ledger extraction, and each wants a different subset of the same three products. Hot ingestion wants events, txhashes, and fees. Cold backfill wants events and txhashes, and since it is the bulk path, it should never pay for fees it will not serve. The fee-window warm-up replay (stellar/stellar-rpc#888) wants fees alone. But the API sells fixed bundles rather than products: ExtractLedgerEvents is hashes plus events in one walk, ExtractFees is fees in another. So the hot path has to walk TxProcessing twice per ledger to get all three, and the only fix the current shape allows is a third combined function, then a fourth for whatever subset comes next.

The intuition for the fix: walking the ledger and reading products off it are different jobs, and only the walk is worth sharing. Let one function own the walk and return per-transaction handles, and make each product a plain function of those handles:

// The only function in the package that walks TxProcessing.
func ExtractLedgerTxParts(lcm xdr.LedgerCloseMetaView) ([]LedgerTxParts, error)

type LedgerTxParts struct {
    Hash, InnerHash [32]byte // filled during the walk; everything pairs by hash
    FeeBump         bool
    Result xdr.TransactionResultPairView // lazy handle into the buffer
    Meta   xdr.TransactionMetaView       // lazy handle into the buffer
}

// Products. No walking, just reads off the handles.
func EventsFromTxParts(parts []LedgerTxParts) ([]TxEvents, error)
func FeesFromTxParts(parts []LedgerTxParts) (LedgerFees, error)

Each consumer composes exactly what it needs, and "one walk per ledger" becomes a property of the API instead of a promise each bundle's doc comment has to make:

// hot: one walk, both products
parts, err := ingest.ExtractLedgerTxParts(lcm)
events, err := ingest.EventsFromTxParts(parts)
fees, err := ingest.FeesFromTxParts(parts)

// cold: one walk, fees never computed
parts, err := ingest.ExtractLedgerTxParts(lcm)
events, err := ingest.EventsFromTxParts(parts)

// fee-only replay (stellar/stellar-rpc#888): same two calls, fees instead of events
parts, err := ingest.ExtractLedgerTxParts(lcm)
fees, err := ingest.FeesFromTxParts(parts)

There is a catch: that signature is only possible if we stop copying v1 exactly. v1 answers two questions by reading the envelope: is this transaction soroban, and how many operations does it have. Envelopes live in the TxSet, and the only way to match a TxSet envelope to its result is to hash every envelope with the network passphrase. That matching is the entire reason ExtractFees needs a passphrase and a second pass over the ledger.

But on any ledger the current protocol can produce, both answers are already sitting in TxProcessing. A transaction is soroban exactly when its meta carries SorobanMeta (soroban transactions always have exactly one operation, and the fee-breakdown ext is always present since protocol 21). The operation count is just the number of per-operation results in the result. So on real ledgers the envelope adds nothing.

The one real difference is txINTERNAL_ERROR, a result core emits only when it malfunctions. It has no per-operation list, so v1 counts that transaction and this definition skips it. That is one sample in a distribution over dozens of ledgers, from a transaction that says nothing about what a fee bidder should pay. For the stellar/stellar-rpc#883 harness this means getFeeStats compares identical on all real traffic, and if the soak ever shows a difference, core had an incident, not rpc a bug.

In exchange, the pairing pass, the per-envelope hashing, and the passphrase leave ingestion entirely. Fees become a plain fold over the walk output like every other product, and hot and cold are both exactly one walk per ledger with nothing extra.

For this PR that means the classification arithmetic and the meta reading carry over (sorobanFeesFromMeta, the negative-fee errors; the deliberate uint64 underflow wrap should become an error, since with the parity requirement gone there is no reason to preserve it), while the pairing machinery (feeShapesByHash, the envelope shape reading) deletes, and the tests re-target the TxProcessing-only rules: the TxSet-shape and passphrase cases go away, and cells for results-derived op counts (failed multi-op, fee-bump inner results, txINTERNAL_ERROR skipped) come in. The fallout for the rest of the family is the bundles themselves, all three of which delete: ExtractTxHashes (that consumer reads parts[i].Hash), ExtractLedgerEvents (with LedgerTransactionEvents slimming to an events-only type aligned with parts), and ExtractFees (its classification moves to FeesFromTxParts). LedgerFees slims to the two buckets, since the ledger sequence and close time are the caller's. After the reshape the package exports one walk and one function per product, nothing else.

[karthikiyer56]

@tamirms - responding here to segments of your comment here

---

Re:

walking the ledger and reading products off it are different jobs, and only the walk is worth sharing (...) "one walk per ledger" becomes a property of the API instead of a promise each bundle's doc comment has to make

this wasnt lost on me fwiw.
i had it penciled in as a fast followup once 772 landed — unify the ingestion facets so hot/cold/replay all ride a single walk.
That being said, I see no reason to punt it anymore.
everything here is still Experimental: so reshaping now is cheap, and it only gets more expensive once rpc v2 starts consuming it or changing it after the fact

the bigger consequence is what this does to the tickets.
this body of work stops being "fee stuff" and becomes "walk the ledger exactly once during ingestion — backfill or streaming — and derive everything you need off that one walk".
fees then falls out naturally as one more product of the walk, same as events and txhashes.

• opened SDK: one TxProcessing walk per ledger — ExtractLedgerTxParts + per-product functions (supersedes the extractor bundles) stellar-rpc#912 for the SDK reshape — ExtractLedgerTxParts + EventsFromTxParts/FeesFromTxParts, all three bundles deleted. this PR is re-targeted at it (same PR, so the review context stays put).
• rewrote v2 ingestion: one walk per ledger — migrate backfill + streaming to parts/products, and feed the getFeeStats windows stellar-rpc#881 — now purely the rpc side: migrate the hot/cold walk sites to parts+products, share the window out of rpcv1, feed it from the live loop. no passphrase anywhere in fee ingestion anymore.
• rewrote getFeeStats: rebuild fee windows on restart stellar-rpc#888 — restart warm-up becomes raw bytes → parts → fees over the query router's read view. no decode

---

Re:

The one real difference is txINTERNAL_ERROR, a result core emits only when it malfunctions. (...) if the soak ever shows a difference, core had an incident, not rpc a bug.

ngl this is a refreshing take, and one i shouldve arrived at myself. i was treating v1's mechanism as the contract and porting its misgivings over bug-for-bug just so i could say "parity".
but parity for paritys sake isnt the goal — identical answers on real traffic is.
I will make the changes for it to error out

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (1)

ingest/extract.go:234

• The signed addition can underflow back to a non-negative value: for example, two math.MinInt64 components produce 0, bypass this check, and yield an inclusion fee even though the resource-fee sum is negative. Reject negative components before adding them so corrupt values cannot wrap through the validation; a regression case for this boundary would also prevent recurrence.

		// int64 addition first: two huge fees wrap negative and hit the error
		// below rather than summing wide.
		resourceFee := nonRefundable + refundable
		if resourceFee < 0 {
			return 0, feeBucketNone, fmt.Errorf("ingest: tx %x: resource fee charged cannot be negative", txParts.Hash)

[tamirms]

The reshape lands exactly as proposed: the classification, the error edges, and the test matrix all read right to me, and TestFeesFromTxParts_TxSetNotConsulted pins the property this was all for. One correction, to a claim that originated in my comment and that the PR now states in several places: that txINTERNAL_ERROR is the only transaction core can put in a ledger without a per-operation result list, so the only skip-divergence from v1 short of a core malfunction.

That is not quite true. Core re-runs transaction validation at apply time (TransactionFrame::commonPreApply runs commonValid with applying=true), and a transaction that fails there still lands in TxProcessing with its fee charged, a tx-level error code, and no per-operation results. On current protocol the consensus rules close every accidental path to this: generalized tx sets enforce one transaction per source account, and both the queue and the set validity check key fee-bumps by the inner source, so ordinary sequence races cannot produce it. One path stays open: an account can sign an operation inside another account's transaction that invalidates its own pending transaction in the same ledger. Account merge produces txNO_ACCOUNT, removing its own signer produces txBAD_AUTH, bump-sequence produces txBAD_SEQ. All of these need the account's own signature, so they are self-inflicted and absent from organic traffic, but a correctly functioning network will happily include them. Separately, pre-protocol-20 tx sets allowed several transactions per account, so old ledgers contain such failures organically; fee stats only ever ingests tip, but the doc comment describes a public function someone can point at history.

I think the behavior should stay exactly as is. Skipping these is the same principled rule as the rest of the classification (count a transaction when its record carries what the observation needs), the dropped samples say nothing a fee bidder should act on, and the soroban bucket is untouched by all of this: core writes the fee ext before validation runs (setNonRefundableResourceFee is called from commonPreApply), so even a never-executed soroban transaction carries its charged fees and v1 and this code count it identically. So this is a wording fix, plus one framing fix for the stellar/stellar-rpc#883 soak: a getFeeStats divergence means "go look at that ledger" (core malfunction, or someone constructed a self-invalidating pair), not "core had an incident" full stop.

Suggested wording for the comparison paragraph on FeesFromTxParts:

// The output matches stellar-rpc v1's FeeWindows.IngestFees on organic
// current-protocol traffic, but the definition differs: v1 classifies from
// the ENVELOPE (a single operation of a soroban type; opCount = envelope
// operations), this classifies from TxProcessing alone as described above.
// The behavioral deltas are confined to classic transactions whose
// operations never ran, so their results carry no per-operation list: v1
// counts them from the envelope, this skips them. Only two things put such
// a transaction in a ledger: a core malfunction (txINTERNAL_ERROR), or an
// account invalidating its own pending transaction with an operation it
// signed inside another account's transaction in the same ledger (account
// merge, signer removal, sequence bump). Pre-protocol-20 tx sets also
// allowed several transactions per account, so old ledgers contain such
// failures organically; fee stats only ever ingests tip ledgers. v1 also
// lets a resource fee above FeeCharged wrap around uint64 where this
// errors.

and for the LedgerFees skip bullet:

//   - A transaction with no per-operation result list is skipped. That
//     covers txINTERNAL_ERROR (core malfunction) and transactions
//     invalidated by an earlier transaction in the same ledger before their
//     operations ran (possible when an account signs an operation against
//     itself inside another account's transaction).

The same sentence recurs in the classifyTxFee inline comment, the feeInternalErrorResult helper comment, the changelog entry, and the description's parity-contract line (and the stellar/stellar-rpc#912 copy); they all take the same one-line widening. Two cheap additions while in there, both optional: a no_account_skipped cell in the matrix (same code path as txINTERNAL_ERROR, but it pins that the skip is intentional for the whole void-code family, not a txINTERNAL_ERROR special case), and one sentence on LedgerTxParts noting that FeeBump is derived from the result code, so a fee-bump that never executed reports FeeBump=false with no inner hash. That last one is inherent to reading TxProcessing only (a void outer result carries no inner pair) and only reachable through the same constructed cases, but consumers building hash indexes off the walk should get to read it rather than discover it.

One item from auditing the view usage, separate from the parity note. v4EventRaws is the multi-field case Fields() exists for: it reads the V4 meta through single-field accessors, and each accessor re-sizes the struct prefix from the top. On the product path (wantEvents only) the overlap is just ext plus txChangesBefore, but on the read path (metaEventRaws with wantDiag) DiagnosticEvents is the last field, so MustDiagnosticEvents walks the entire meta interior, operations included, a second time per transaction:

// today: three prefix walks over the same V4 meta
tev.TransactionEvents = collectRaws(metaV4View.MustEvents().MustIter())
opsView := metaV4View.MustOperations()
// ...
*diag = collectRaws(metaV4View.MustDiagnosticEvents().MustIter())

// one locate pass
f := mustView(metaV4View.Fields())
tev.TransactionEvents = collectRaws(f.Events.MustIter())
// ... f.Operations ...
*diag = collectRaws(f.DiagnosticEvents.MustIter())

The V3 arm has the same shape one level down when both event sets are wanted: MustEvents plus MustDiagnosticEvents on the SorobanMeta view re-sizes the events vector, and SorobanTransactionMetaView.Fields() covers it. Fields() is error-returning with no generated Must twin, so the Must-style walkers need either a small local shim that panics with the returned *xdr.ViewError (TryVoid already recovers those) or a generated MustFields(), which may be worth adding to the generator regardless. The per-op double sizing inside the loop (Iter() advances by re-sizing elements the body also reads) is a generator limitation rather than a call-site problem, fine to leave as is.

[karthikiyer56]

That is not quite true. (...) One path stays open: an account can sign an operation inside another account's transaction that invalidates its own pending transaction in the same ledger.

Taken in full — thanks for the catch, and for the pre-protocol-20 history note. All the doc sites now describe the whole never-ran family instead of "txINTERNAL_ERROR only": the FeesFromTxParts comparison paragraph, the LedgerFees skip bullet, the classifyTxFee / txOperationCount inline comments, the test-helper comment, and the changelog. I kept your content but wrote it out in plain English with the self-invalidation example inline (Alice's payment + Bob's transaction carrying an Alice-signed merge/signer-removal/sequence-bump), so the next reader doesn't have to re-derive how such a transaction gets into a ledger. Your soroban note landed too — the docs now say why the soroban bucket is untouched (the fee ext is written before validation can kill the tx).

Behavior unchanged everywhere. The PR description, stellar/stellar-rpc#912, and stellar/stellar-rpc#881 got the same widening, and the #883 framing everywhere is now "a divergence means go look at that ledger", not "core had an incident" full stop. (fa830cc)

[karthikiyer56]

Both optional additions taken as well:

• no_account_skipped now sits next to internal_error_skipped in the matrix (the builder generalized to feeNoOpListResult(fee, code)). Two different codes, same outcome — the tests now read as "any transaction whose result has no per-operation list skips", so nobody later mistakes the skip for a txINTERNAL_ERROR special case and "fixes" it.
• LedgerTxParts discloses the FeeBump derivation: FeeBump comes from the result code (the only place TxProcessing carries the inner hash), so a fee-bump whose operations never ran reports FeeBump=false with a zero InnerHash — and a hash index built from the walk will not find that transaction by its inner hash. Index builders read it in the doc now instead of discovering it in production. (fa830cc)

[karthikiyer56]

v4EventRaws is the multi-field case Fields() exists for

Done, with measurements — and one honest wrinkle: the pubnet fixture can't show this fix at all. ledger_58752000 is protocol 22, so all 249 metas are V3 and the V4 arm never runs there. So I benchmarked the walker directly on a synthetic V4 meta (10 ops × 3 events each, both event sets wanted): ~4.5µs → ~4.1µs per meta (~10%), and the win grows with op count since the saved work is exactly the DiagnosticEvents accessor re-walking every operation. Events-only measured parity on that shape.

• V4 arm: one Fields() locate pass unconditionally — Events and Operations sit deep in the struct, so even single-set reads paid overlapping prefix walks.
• V3 arm: locate pass only when BOTH sets are wanted, per your scoping. With one set wanted the single accessor sizes strictly less than a full locate, and on the real all-V3 ledger the read path measured parity both ways — V3's prefix fields (a 4-byte ext, a small events vector) are too cheap for the dedup to matter.
• The shim: a local mustFields that panics with the returned *xdr.ViewError (the existing TryVoid wrappers recover it), mirroring the generated Must accessors. I left MustFields() on the generator as a follow-up rather than smuggling a regen of xdr_views_generated.go into this PR.
• Per-op double-sizing inside the loop left as is, as you said. (fa830cc)