[edalzell]

Comb\Index::insertDocuments() and delete() read the whole index file, change it in memory, and write it back, with nothing held between the read and the write. lock: true on get/put (#10695) only stops torn reads of a single call. With more than one queue worker, php please search:update dispatches one InsertMultipleJob per chunk and two workers can both read {a}, then write {a, b} and {a, c}; whichever write lands last silently drops the other chunk. An entry save racing a reindex chunk does the same.

Reproduced on a 2,001-entry collection, chunk_size 10, Redis queue, four queue:work processes started before the jobs are dispatched, counting documents in default.json afterwards (script: https://gist.github.com/edalzell/f2e053aef66515967c4c340e95cd6534):

run	before (docs / lost)	after (docs / lost)
1	1,581 / 420	2,001 / 0
2	1,521 / 480	2,001 / 0
3	1,191 / 810	2,001 / 0
4	1,450 / 551	2,001 / 0
5	1,541 / 460	2,001 / 0
6	1,081 / 920	2,001 / 0
7	1,421 / 580	2,001 / 0
8	1,590 / 411	2,001 / 0
9	1,241 / 760	2,001 / 0
10	1,301 / 700	2,001 / 0

No job failed in any run, and each worker processed roughly a quarter of the 201 jobs both before and after.

Fix wraps each read-modify-write in a blocking cache lock (Cache::lock('search-index-{name}', 60)->block(30, ...)), which waits up to 30 seconds for the lock and releases it once the read and write finish, even if one throws. The lock only spans processes when every worker shares a lock-capable cache store (Redis, database, file, etc.). With the array or null driver, it doesn't protect anything and you get today's behaviour. Tests assert the lock is held from before the read until after the write for insert and delete.