Skip to content

events: batch fetched payload copies into arenas, and fan cold reads out - #1047

Draft
tamirms wants to merge 1 commit into
feature/full-historyfrom
tamirms/events-fetch-arenas-fanout
Draft

tamirms wants to merge 1 commit into
feature/full-historyfrom
tamirms/events-fetch-arenas-fanout

Conversation

@tamirms

@tamirms tamirms commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

What

  • Hot fetch: the batched read's options are built once at open, the key list is cut from one buffer, each pinned value is read once, and the values are copied into one arena per batch.
  • Cold fetch: payloads are copied into one arena per call, locked against packfile's fan-out.
  • Cold events reads fan out over the packfiles with a concurrency of 8 (defaultColdEventReadConcurrency).

Why

  • Allocations: a hot fetch now makes two heap allocations per event id, both inside grocksdb, which TestHotStore_FetchEventsAllocationBudget pins.
  • Latency: a page's cold payload fetch is hundreds of scattered records, and serial reads add their latencies together. Eight concurrent reads overlap them.
  • The lock: once reads fan out, ReadItems calls back from several goroutines. Unlocked, concurrent copies into the arena's one buffer crashed or silently dropped payloads in testing. The lock covers only the copy, and the fan-out test forces an 8-way read and compares whole payloads.

Split out of #968.

Known limitations

  • The concurrency of 8 was measured on NVMe and is compiled in. A config knob can follow if a deployment needs another value.
  • Not benchmarked on its own. rpcv2: getEvents read-path performance across both tiers #968's cold numbers (p50 18.3 → 4.3 ms at sac density) include this together with the match engine and packfile changes.

🤖 Generated with Claude Code

Hot fetch: the batched read's options are built once, at open; the key
list is cut from one buffer; each pinned value is read once; and the
values are copied into one arena per batch instead of one allocation
each. A fetch now makes two heap allocations per event id, both inside
grocksdb, which TestHotStore_FetchEventsAllocationBudget pins.

Cold fetch: payload copies go into one arena per call instead of one
allocation each, and cold events reads now fan out over the packfiles
with a concurrency of 8. packfile's ReadItems then calls back from
several goroutines, so the arena is locked around each copy: unlocked,
concurrent copies into its one buffer crashed or silently dropped
payloads in testing. The lock covers only the copy, so reads, decodes
and unmarshals still overlap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant