fullhistory: archive tip + captive-core backfill for no-lake deployments (#833)

[chowbao]

Closes #833.

A no-lake deployment ([backfill.datastore].type empty) had no tip source on first start: the only source was notConfiguredTip, which always errors, so no earliest_ledger form could pin a floor. And a tip alone would only have made the floor pinnable, not fillable — backfillSource step (3) errors on a backend-less chunk, so the sole truly runnable no-lake config would have been earliest_ledger = "now". Per the review discussion, both halves land together: the archive tip and captive-core backfill.

The tip is a property of the backend. Every backfill.Backend owns its frontier — the lake's export frontier for bsbSource (datastore.FindLatestLedgerSequence), the archives' root HAS for captiveSource — and all three consumers read it: validateConfig and backfillToTip through sampleTipWithRetry(ctx, backend.Tip, …), the freeze's waitForCoverage raw. sampleTipWithRetry is the single home for the retry semantics, moved verbatim from the old networkTip helper: cenkalti/backoff constant interval, count-bounded, sub-genesis tip is backoff.Permanent ("not ready"), ctx cancellation aborts the wait. This replaces both the round-1 (NetworkTipBackend, TipBackoff, TipMaxAttempts) trio and the interim tipSampler type — the multi-source lake→archive fallback from the #820 threads is deliberately dropped (review rationale): once each backend owns its frontier, the fallback's only user was the lake, where a failing LIST almost always means failing GETs too, and an anchor overshoot just times out coverage and re-samples on restart. With it go StartConfig.Tip and the captiveSource type-assertion the dedup needed.

captiveSource: no-lake backfill through captive core. With no datastore but [ingestion].history_archive_urls configured (which live ingestion requires anyway — no new config), buildBackfillBackend returns a captiveSource: the core opener's LedgerStream plus the archives' root HAS as Tip. Everything downstream runs unchanged — executePlan's parallel pool, backfillSource, WriteColdChunk, and the waitForCoverage poll on Backend.Tip. Parallel per-chunk replays compose because the SDK's captiveCoreStream.RawLedgers is a stateless factory: each bounded call builds a fresh core in its own ephemeral working dir, so N concurrent RawLedgers(BoundedRange(chunk)) calls run N independent cores. The real cost is one core catch-up per chunk, which makes the retention-window size the natural selector: no-lake + archives suits a small window; deep history stays on the bulk lake.

One core opener, resolved up front. Every daemon runs the live ingestion loop, so the opener resolves once before the backend (whose no-lake path is built from its stream), replacing the early/late split. The one behavior change: a bad [ingestion] config now surfaces before a bad earliest_ledger — cosmetic, both are fatal startup errors.

Archives absorb their own lag. The archives' one-checkpoint (64-ledger) lag is absorbed by the existing anchor = max(tip, lastCommitted) and the signed withinOneChunkOfTip; no lag handling is added.

Synthetic tip deleted. An unavailable tip now means the backend's frontier query kept failing — so the tip = lastCommitted degraded mode in backfillToTip is gone. A tip failure errors the pass and the supervisor restarts (a transient outage self-heals on re-sample); the daemon never serves behind an unknown frontier.

Tests: the #833 acceptance test boots the real entrypoint with no datastore, a file:// archive (real pool construction + root-HAS frontier read), and a core stream serving the bounded replay, and asserts chunk 0's cold artifacts freeze exactly as on the lake path; captiveSource construction + frontier mapping; a fail-fast startup error when neither source is configured; the retry/sub-genesis/ctx-cancel suite on sampleTipWithRetry; and TestBackfill_RestartTipUnavailableErrors replacing the deleted degraded-mode test. Full fullhistory -short suite and -race on the fullhistory package pass; repo golangci-lint clean on the diff.

Deliberate skip: the archive pool is built without a network passphrase, so the SDK's HAS passphrase check is off. A wrong-network archive URL already fails loudly at live ingestion — captive core consumes the same [ingestion].history_archive_urls — with recovery being a wipe of the still-empty first-start data dir. (With the opener now resolved before the pool this is wireable if wanted; left out to keep the change to the reviewed scope.)

[tamirms]

The archive tip is the right first half, but I think #850 should go one step further and let a no-lake deployment actually backfill, because otherwise the tip it can now resolve is only usable with earliest_ledger = "now".

With this PR a frontfill-only daemon can pin a below-now floor (the archive tip validates it in resolveEarliestFirstStart), but backfillToTip then reaches backfillSource step (3), finds cfg.Backend == nil, and errors with "chunk has no local copy and no bulk backend is configured." So the floor is pin-able but not fill-able, and the only config that truly runs no-lake is "now", where there is nothing to fill.

The captiveSource the issue originally sketched closes this, and it is a genuine drop-in rather than the half-implemented interface the description worries about:

type captiveSource struct {
	ledgerbackend.LedgerStream   // NewCaptiveCoreStream(coreConfig, log)
	archives rootHASGetter
}
func (s *captiveSource) Tip(ctx context.Context) (uint32, error) { return archiveTip(s.archives)(ctx) }

buildBackfillBackend returns it instead of nil when there is no datastore but archives are configured, and everything downstream runs unchanged: executePlan's parallel pool, backfillSource, WriteColdChunk, and the waitForCoverage poll on Backend.Tip. The reason it composes with the parallel per-chunk consumption is that the SDK's captiveCoreStream.RawLedgers is a stateless factory that builds a fresh core per call, so N concurrent RawLedgers(BoundedRange(chunk)) calls spin up N independent cores rather than contending on one cursor.

The "Not a Backend" reasoning is right for a pure-frontfill deployment that never freezes bulk history, but it inverts once we want backfill: a captive-core-backfill deployment does freeze bulk history, so it needs the LedgerStream, and captive core replaying [floor, tip] supplies it. The Backend is complete, not half-implemented.

The one real cost is that each RawLedgers call catches a fresh core up to its range start, so it is one catch-up per chunk. That is fine for a small window and infeasible for deep history, which makes the retention-window size the natural selector: no datastore plus archives builds captiveSource, a lake stays the path for deep history.

It also settles the tip wiring in one place: captiveSource.Tip is the archive, so the primary tip becomes Backend.Tip uniformly (lake for bsb, archive for captiveSource), and the standalone archiveTip stays only as the lake's fallback rather than the no-lake special case.

Net, the tip and the captive-core backfill are two halves of the same capability, and since the tip alone only unlocks "now", I think they belong in this PR together.

[chowbao]

Done in 0912052 — you're right that the tip alone only unlocked "now", and the fresh-core-per-RawLedgers point holds up (I verified the SDK stream: each bounded call builds its own core via newCaptive() inside streamRaw, and bounded ranges run catchup mode in an ephemeral working dir — captive-core-<hex8> under the storage path — so N parallel chunk replays don't collide on disk either).

What landed, matching the sketch:

• captiveSource (daemon layer) embeds the core opener's LedgerStream and answers Tip from the archives' root HAS. buildBackfillBackend returns it when there's no datastore but archives are configured; executePlan, backfillSource, WriteColdChunk, and waitForCoverage run unchanged.
• Tip wiring settled in one place as you described: the primary source is Backend.Tip uniformly (lake for bsbSource, archive for captiveSource), and the standalone archiveTip survives only as the lake's fallback — buildTipSampler skips the pool when the backend's Tip already is the archive, so no source is consulted twice per attempt.
• The one-catch-up-per-chunk cost is documented on the type with the retention-window-as-selector guidance; no new config knob (the selector is whether you configure a datastore).

One ordering consequence worth flagging: on the no-lake path the core opener now resolves before validateConfig (its stream is the backend), so [ingestion] config errors surface first there; the lake and injected-backend paths keep the opener after validateConfig as before.

The new acceptance test boots the real entrypoint with no datastore, a file:// archive (real pool construction + root-HAS frontier read), and a core stream serving the bounded replay, and asserts chunk 0's cold artifacts freeze identically to the lake path. PR title/description updated — the old "Not a Backend" paragraph is gone since, as you said, it inverts once backfill is in scope.

[tamirms]

Since every fullhistory daemon runs the live ingestion loop, core is needed regardless of the backfill source, so it can be resolved once up front instead of split between this early conditional and the late fallback at line 185:

core := opts.Core
if core == nil {
    core, err = newCaptiveCoreOpener(cfg.Ingestion, cfg.Service.DefaultDataDir, logger)
    if err != nil {
        return err
    }
}

backend := opts.Backend
if backend == nil {
    built, cleanup, berr := buildBackfillBackend(ctx, cfg, core, pool, logger)
    // ...
}

That drops this compound predicate and the duplicated newCaptiveCoreOpener call, removes the late-resolution block, and lets buildBackfillBackend's guard shrink to if pool == nil since core is always present. The only behavior change is that a bad [ingestion].captive_core_config would surface before a bad earliest_ledger on a lake deployment, which is cosmetic since both are fatal startup errors. Non-blocking.

[chowbao]

Done in 49a7a81 — core now resolves once, right after the catalog opens, and the late fallback block is gone. buildBackfillBackend's guard shrank to if pool == nil with the non-nil-core contract noted in its doc.

The ordering change you called cosmetic did surface in one test: TestRunDaemon_NowFloorRequiresTip relied on validateConfig's "now" error firing before the stub captive_core_config was ever parsed; it now injects a fake Core so it keeps pinning the validateConfig failure it was written for.

[tamirms]

Now that captiveSource lands, the tip is effectively a property of the backend: bsbSource.Tip is the lake's export frontier, captiveSource.Tip is the archive's live frontier, and waitForCoverage already reads Backend.Tip directly rather than through the sampler. That leaves tipSampler doing two separable jobs: a retry-and-sub-genesis wrapper around a tip source (real, but that is a function), and a multi-source fallback list, which after the dedup is exercised only by the lake-primary-archive-fallback case.

That fallback no longer earns the abstraction, though to be clear it is not a correctness problem. If the anchor over-shoots what the lake holds, waitForCoverage just times out on the first uncovered chunk and the pass restarts and re-samples, the same graceful failure as any range that runs past the lake, and the retention floor follows lastCommitted so nothing over-prunes. It is marginal rather than wrong: if the lake's FindLatestLedgerSequence (a LIST) is failing, its ledger GETs are almost certainly failing too, so the archive anchor rarely buys real progress, and the retry already covers transient blips.

So the sampler should collapse into sampleWithRetry(ctx, backend.Tip): validateConfig and backfillToTip wrap backend.Tip with the retry, and coverage keeps reading it raw as it does today. That removes the tipSampler type, buildTipSampler's source-list assembly, StartConfig.Tip (callers reach the backend directly), and the backend.(*captiveSource) type-assertion, which only exists to avoid double-adding the archive the fallback introduces.

This composes with the inline note on the core resolution rather than replacing it: both shrink the same runDaemonWith block, but the core early/late split is independent, since captiveSource still needs the opener regardless of how the tip is sampled. It does reverse the lake-to-archive fallback from the #820 review threads, but captiveSource is exactly what changes that calculus: with every backend now owning its own frontier, the fallback stops paying for the sampler machinery. This should land in this PR rather than carry the extra type and the type-assertion forward.

[chowbao]

Done in 49a7a81 — tipSampler is collapsed into sampleTipWithRetry(ctx, backend.Tip, interval, maxAttempts), exactly the split you described: the retry-and-sub-genesis wrapper survives as a function (semantics unchanged — constant backoff, count-bounded, sub-genesis permanent, ctx aborts), and the multi-source fallback list is gone.

Removed with it, as listed: the tipSampler type, buildTipSampler's source-list assembly, StartConfig.Tip (validateConfig takes the raw tipSource and backfillToTip reaches cfg.Exec.Process.Backend.Tip directly; coverage keeps reading Backend.Tip raw), and the backend.(*captiveSource) type-assertion. The archive pool is now built only on the no-lake path, since captiveSource is its sole consumer. Net −141 lines on top of the previous head.

One consequence worth noting: with the fallback gone, "no tip source configured" can no longer be detected at sampler build, so the equivalent fail-fast moved to runDaemonWith — a nil backend after buildBackfillBackend errors at startup with the same config-shaped message (in production newCaptiveCoreOpener already rejects missing archive URLs first; the guard covers the injected-Core seam, and TestRunDaemon_NoBulkSourceFailsFast pins it).

Test fallout: the three multi-source Sample tests (prefer-first / fall-over / join-errors) are deleted with the machinery; the retry suite now drives sampleTipWithRetry directly. Fakes that drove a failing tip through validateConfig/backfillToTip switched from erroring tips to sub-genesis tips — permanent "not ready" fails in one poll, so those tests don't sleep through the production backoff the call sites now apply. The PR description's tip section is rewritten to match, including your rationale for reversing the #820-thread fallback.

[tamirms]

Two tests slipped past the sub-genesis switch and now sleep ~4s each. The collapse hardcodes the production backoff (defaultTipBackoff = 1s, defaultTipMaxAttempts = 5) at the sampleTipWithRetry call sites, so an erroring tip driven through backfillToTip/run burns 4×1s of real sleep. TestSupervise_FirstStartNoTipRetries was switched to a sub-genesis fake (tips: []uint32{0}, one permanent poll, no sleep) to avoid exactly this, but these two still use err: ..., errFirst: 99:

• TestBackfill_RestartTipUnreachableErrors
• TestRun_FirstStartNoTipErrors

Each adds ~4s to the suite, and they re-exercise retry exhaustion that TestSampleTip_ExhaustedRetriesErrors already covers at a 1ms interval. Giving them sub-genesis fakes like the supervise test makes them fast, and the assertions (errors out, reads never served, core never starts) hold identically, since sub-genesis and an errored tip both resolve to "tip unavailable." The one nuance is that RestartTipUnreachableErrors names the "unreachable" case specifically, so it would either soften to "tip unavailable" or keep the erroring fake and accept the sleep.

[tamirms]

Small leftover from removing the fallback: archiveTip now has a single production caller, this method, and archiveTip(s.archives)(ctx) builds a tipSource closure only to invoke it immediately. That closure shape existed so the multi-source sampler could hold it in a source list; with the sampler gone it can fold into the method:

func (s *captiveSource) Tip(ctx context.Context) (uint32, error) {
	has, err := s.archives.GetRootHAS()
	if err != nil {
		return 0, fmt.Errorf("history archive root HAS: %w", err)
	}
	return has.CurrentLedger, nil
}

That drops the archiveTip function and the build-a-closure-then-call-it indirection, and TestArchiveTip_RootHASErrorSurfaces becomes a test of captiveSource.Tip with the same coverage. tipSource the type stays as the param abstraction for sampleTipWithRetry and validateConfig; only its "returned by archiveTip" use goes away.

[chowbao]

Done in 425208e — archiveTip is folded into captiveSource.Tip exactly as sketched, and the build-a-closure-then-call-it indirection is gone. The 64-ledger checkpoint-lag note (anchor max(tip, lastCommitted) + signed withinOneChunkOfTip absorb it) moved onto the method so it stays with the code it describes. rootHASGetter moved next to captiveSource too, since Tip is now its only consumer — tip.go is left holding just tipSource and sampleTipWithRetry. TestArchiveTip_RootHASErrorSurfaces became TestCaptiveSourceTip_RootHASErrorSurfaces with the same coverage.

[chowbao]

@codex[agent] review

[chowbao]

Done in 425208e — both switched to sub-genesis fakes (tips: []uint32{0}), matching the supervise test: one permanent poll, no sleeps. Confirmed the timing: each dropped from ~4.1s to ~0.1s, and the assertions are unchanged (errors out with no pass recorded / reads never served and core never started).

Took the softening option on the name: TestBackfill_RestartTipUnavailableErrors, with its doc comment noting that sub-genesis and an erroring tip both resolve to "tip unavailable" and pointing retry exhaustion at TestSampleTip_ExhaustedRetriesErrors (1ms interval). Also swept the rest of the tree for the same pattern: the only remaining err/errFirst fakes are the three direct sampleTipWithRetry tests (millisecond intervals, intentional), and the backfill package's tipErr fakes drive waitForCoverage, which takes explicit test intervals — no production backoff anywhere else.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 49a7a81930

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Verify archive tip against the configured network

In the no-lake path this archive pool is the Tip source used by validateConfig, but it is created without the captive-core NETWORK_PASSPHRASE. If history_archive_urls accidentally points at another Stellar network and earliest_ledger = "now", GetRootHAS can still return that archive's height, PinEarliestLedger persists a floor derived from the wrong network, and only the later captive-core replay fails the passphrase check; after fixing the URL the immutable catalog pin remains wrong and requires wiping. Please pass the configured network passphrase into this pool or otherwise verify the archive before using its root HAS to pin config.

Useful? React with 👍 / 👎.