Skip to content

Release eval: job to measure data-lake backfill ingestion perf #709

Description

@Shaptic

Parent: #706
Depends on: #707

Summary

Add a release-evaluation job that measures RPC database load while backfilling from public pubnet ledger data without relying on live Core nodes.

Scope

  • Start stellar-rpc with --backfill, --serve-ledgers-from-datastore, pubnet network settings, and a public data-lake datastore configuration.
  • Use an isolated SQLite DB path on the runner and collect DB size, runtime, ledgers/sec, and final health/latest-ledger state.
  • Backfill with a hardcoded history-retention-window.
  • Validate that no Stellar Core node is required for the backfill ingestion path.

Acceptance criteria

  • The job can ingest a configured pubnet ledger window from public data-lake files and exit deterministically.
  • The job fails when metrics exceed TBD thresholds.
  • Metrics and logs are uploaded for final reporting.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions