exec module: decouple startup, execution, prune, retire, and publish flows

[mh0lt]

Problem

The exec module's flows (ProcessFrozenBlocks, ValidateChain, UpdateForkChoice, prune, retire, publish) are heavily intertwined because they share a single RW transaction — an MDBX limitation where only one goroutine can hold an RwTx at a time.

This creates a serialization chain:

execute → flush → commit → prune → retire → publish
    all serialized because they share one RwTx

Three resources compete:

1. ExecModule semaphore(1) — serializes Start/Validate/FCU; background prune holds it
2. snBuildAllowed semaphore — serializes block retirement vs state aggregation
3. MDBX write lock — only one RwTx at a time

Current Flow Issues

Startup (ProcessFrozenBlocks)

• ExecModule.Start() holds the semaphore for the entire duration of ProcessFrozenBlocks (potentially hours during initial sync)
• During this time, Ready() returns false, ValidateChain()/UpdateForkChoice() return Busy
• Snapshot download runs inside ProcessFrozenBlocks — downloading blocks the entire exec module
• No error recovery between startup and steady state: if ProcessFrozenBlocks fails partway, the exec module releases the semaphore in an inconsistent state

Steady State (forkchoice/newpayload)

• ValidateChain does NOT persist state — validates in a throwaway tx, stores in forkValidator.sharedDom
• UpdateForkChoice merges extending fork state, then runs execution pipeline
• Two separate stage pipelines: ProcessFrozenBlocks uses full pipeline, ValidateChain uses in-memory pipeline with different badBlockHalt settings
• No distinction between "catching up" (big jump) and "synced" — CL-driven catch-up takes a different code path than startup catch-up

Background Prune/Merge

• Triggered per-forkchoice via runPostForkchoice()
• Semaphore is transferred to background goroutine — while pruning, all CL requests return Busy
• Domain pruning bounded by slot time / 6

Block Retirement

• Triggered per-cycle in SnapshotsPrune stage
• Non-blocking (background goroutine), but snBuildAllowed semaphore serializes it with state aggregation
• FrozenBlocks boundary advances when indices become visible (eventually consistent)

Snapshot Publishing

• Only happens after merge (not initial dump)
• Publishing via seeder → BitTorrent after merge step completes

Architectural Direction

The driver for the interconnection is the single-writer RwTx constraint. This is changing:

1. Execution no longer needs RwTx — it writes to memory (SharedDomains). Pending: a final refactor to move remaining rawdb RW interactions (WriteCanonicalHash, WriteHeader, SaveStageProgress, etc.) into SD writes + an additional memorydb overlay.

2. Async transactions allow cooperative sharing.

Once (1) is complete, flush/commit is the only point where execution needs a tx, and this can happen asynchronously:

BEFORE (current):
  execute ──RwTx──► flush ──RwTx──► commit ──RwTx──► prune ──RwTx──► retire
  [all serialized on one goroutine]

AFTER (target):
  execute (memory only, no RwTx)     ← can run continuously
       │
       ▼ async handoff
  flush/commit (brief RwTx window)   ← only serialization point
       │
       ▼ independent
  prune (own RwTx, own schedule)     ← no longer blocks execution
  retire (own RwTx, own schedule)    ← no longer blocks prune
  publish (no RwTx needed)           ← fully independent

Separation Opportunities

• Snapshot download could run independently of execution (doesn't need the exec semaphore)
• Block retirement could be event-driven (trigger when execution advances N blocks) rather than polled per-cycle
• Domain pruning could have its own scheduling, decoupled from forkchoice timing pressure
• The ProcessFrozenBlocks "catch up" path and the updateForkChoice "big jump" path are essentially the same work with different error handling — unify them
• Move remaining rawdb RW writes out of execution into SD/memorydb to break the RwTx dependency
• Distinguish "busy doing initial sync" from "busy processing a forkchoice" in semaphore status

Related PRs

• stagedsync, execmodule: halt on invalid block during parallel initial sync #19870: halt on invalid block during parallel initial sync (gated on Exec3Parallel)
• stagedsync: add BAD_BLOCK_HALT env var, fix parallel STOP_AFTER_BLOCK #19803: BAD_BLOCK_HALT + STOP_AFTER_BLOCK fix

Entry Points Reference

Chain type	Code path	File
PoS (mainnet)	ExecModule.Start() → ProcessFrozenBlocks	backend.go:1496
Bor (polygon)	ExecModule.Start() → ProcessFrozenBlocks	backend.go:1502
PoW (legacy)	StageLoop() → ProcessFrozenBlocks + iteration loop	backend.go:1523

[VBulikov]

yperbasis added Imp2 as Importance