search: bleve async errors are silently dropped, a corrupt segment turns the merger into an endless write loop #3594

Describe the bug

The search service opens its bleve index without an async error callback, so errors from bleve's background goroutines (merger, persister) are silently dropped. With one corrupt segment in the index this became a silent infinite write loop: for about two weeks the OpenCloud container sat at ~97 % CPU and wrote ~50 MB/s to the index directory around the clock, with nothing in the logs at any level.

services/search/pkg/bleve/index.go opens the index with bleve.Open(destination) in 7.5.0 and bleve.OpenUsing(destination, openRuntimeConfig) in 8.0.1, where openRuntimeConfig only sets bolt_timeout. Neither registers asyncErrorCallbackName, so scorch's fireAsyncError has no handler.

What was actually happening

Found through a goroutine dump from the debug server (/debug/pprof/goroutine?debug=2):

• scorch.(*Scorch).introducerLoop and persisterLoop had been parked in select for 687 minutes, so nothing had been introduced into the index for hours.
• scorch.(*Scorch).mergerLoop → planMergeAtSnapshot → zapx.mergeSegmentBases → mergeStoredAndRemap → FileWriter.Write kept running.
• The search event workers were idle and the search-pull consumer had nothing pending. No indexing was going on.

To see the swallowed error, I copied the index directory and opened the copy with the same library versions (bleve v2.6.0, zapx v17.1.2) plus a registered asyncErrorCallbackName. The error showed up immediately and repeated about 3 times a second:

source: merger, persist error: merging err: merging failed: error reading frequency: memUvarintReader overflow

Scorch stats after 90 s on the copy: TotFileMergeLoopErr = 249, TotFileMergePlanTasksErr = 249, TotFileMergeWrittenBytes ≈ 9.8 GB, TotIntroduceLoop = 2.

When I opened each .zap segment on its own and merged it alone, exactly one of 36 failed with the same error: a 24,605-document segment written about two weeks before the loop was noticed. Every merge plan that includes it fails. Scorch then retries at once (details below), and each attempt writes a full merged segment (~35 MB) before failing and deleting it.

Steps to reproduce

1. Have a bleve index with a segment that fails to merge. I can't share mine, because it contains private file metadata. I also don't know how it got corrupted: there was no host crash, OOM kill or disk-full around the time it was written.
2. Start OpenCloud with the default bleve engine.
3. Watch search/bleve/store: segment file IDs keep climbing (~1/s here), CPU stays high, disk writes stay high, and nothing is logged.

Expected behavior

• bleve async errors are logged by the search service, at least at error level and ideally rate-limited, so an admin can see that the index is broken.
• Ideally the search service reacts to a persistent merger error, for example by marking the index unhealthy or exposing a metric (opencloud_search_*), instead of letting it spin forever.

Workaround

Stop OpenCloud, move the whole search/bleve directory aside (don't delete single .zap files, see #3588), start it again, then run opencloud search index --all-spaces. Disk writes dropped from 27–70 MB/s to 2–5 MB/s right after the restart.

Setup

• opencloudeu/opencloud-rolling:7.5.0, single container, bleve engine, basic extractor, decomposed storage driver
• Docker on Colima (macOS, virtiofs bind mount) with the data directory on an external USB HDD
• The corresponding scorch behaviour (immediate retry of a failing merge, no backoff) is reported to bleve in scorch: a failing merge is retried immediately and forever, so a corrupt segment becomes an endless write loop blevesearch/bleve#2402.