scorch: a failing merge is retried immediately and forever, so a corrupt segment becomes an endless write loop #2402

When planMergeAtSnapshot returns an error, mergerLoop calls fireAsyncError and does continue OUTER straight away (index/scorch/merge.go, v2.6.0 and v2.6.1). ctrlMsg is still set and lastEpochMergePlanned isn't updated, so the next iteration re-plans the same merge on the same snapshot with no delay.

For a transient error that's fine. For a persistent one, like a segment that can't be read, the merger turns into an endless write loop: each attempt writes the whole merged output file, fails, removes it and starts over.

What I observed

This was in production, embedded in OpenCloud (bleve v2.6.0, zapx v17.1.2). One segment was corrupt:

merging failed: error reading frequency: memUvarintReader overflow

(When I merged each segment of the index alone, only that one failed.)

• The loop ran for about two weeks before anyone noticed. The host embedding bleve didn't register an async error callback, which is being fixed on their side.
• It wrote roughly 50 MB/s to disk non-stop (each failed attempt wrote a ~35 MB .zap) and kept a core busy.
• The introducer and persister stayed parked the whole time, so nothing was ever introduced.
• On a copy of the index, 90 s gave TotFileMergeLoopErr = 249, TotFileMergePlanTasksErr = 249 and TotFileMergeWrittenBytes ≈ 9.8 GB.

Suggestion

Any of these would turn a silent disk-burner into a bounded, visible failure:

• back off (exponentially) after a failed merge plan instead of continue OUTER;
• after N consecutive failures on the same snapshot epoch, stop retrying that plan until the root changes (the way the loop already waits on lastEpochMergePlanned after a success), or leave the failing segment out of later plans;
• expose the consecutive-failure state in StatsMap, so embedders can alert on it.

I can't share the corrupt segment because it contains private file metadata. I'm happy to run a patched build against a local copy of it.

Possibly related: blevesearch/zapx#442 (merger goroutine dying on a panic in mergeStoredAndRemap). That's the opposite failure mode, a dead merger rather than one that spins forever, in the same code path.