[Engine] scope proposal: opt-in reuse of a Linux cold-read worker #4657

Area

[Engine]

What do you want to do?

Agree a small contribution scope for opt-in reuse of one Linux cold-read worker, before preparing an engine PR. I run Flash-Next on an RTX 5060 Ti 16 GB, Core i7-13700 and 32 GB RAM with NVMe/NTFS3. At 128K, experts outside RAM/VRAM still need file reads. I would like to reduce avoidable host overhead in that path without changing routing, expert bytes, KV format or existing defaults.

What do you do today instead?

I use a local experimental I/O bundle on engine 0.1.52 with the 0.2.8 server. The bundle includes worker reuse, completion publication, CPU affinity and bounded polling, plus disabled experiments. Its exact sources are archived at this evidence commit; it is not the proposed merge scope. The separate results-only benchmark report is #4656.

Source observation and evidence limits

At upstream 870aedd9e1ebe1c91f94da93b59af17729f6c5c0, FileExpertSource::prefetch_begin() creates threads for cold work and prefetch_end() joins them. Linux read_direct() owns its AIO context and aligned scratch buffer in thread-local state. The proposed reuse would keep that state alive across eligible layers.

Unconfirmed static-analysis performance hypothesis: repeatedly creating the worker and its TLS AIO state may add avoidable overhead. I have not isolated or quantified that cost. Existing combined-build measurements do not establish that worker reuse alone is faster, and do not justify a default change. The report includes a slower full-engine candidate and uncontrolled-background limitations.

Proposed first scope

• Linux only, opt-in and disabled by default, for an existing direct-read batch that would otherwise use exactly one worker.
• Reuse one worker owned by the file source. Wait for its work at the existing boundary and drain/join it before releasing file descriptors or stage buffers.
• Keep current batch completion, mapped fallback, multi-worker behavior and auxiliary affinity policy. Do not add explicit CPU pinning in this first change.
• Exclude per-expert early publication, polling, registered io_uring buffers, direct destinations, speculative read-ahead, server tuning and other configuration changes.

Tradeoffs include a longer-lived idle thread and TLS scratch/AIO resources, plus lifecycle synchronization that must be checked during source close/reopen and transitions between single-worker and multi-worker jobs. The first change should reuse existing project infrastructure where suitable.

Validation before an engine PR is ready

Test worker/TLS lifetime, repeated batches, exact read bytes, failed reads, close/reopen and transitions to the existing multi-worker path. Demonstrate default-path equivalence, then perform repeated, order-swapped opt-in/off comparisons on the same source/settings, preserving all outputs and both first-use and reused-prompt cases. Measure thread/AIO setup activity to test the hypothesis rather than attributing the whole bundle's numbers to this feature.

I can validate Linux/CUDA on this machine. A future code PR will remain draft while required HIP/SYCL compilation and any other checks for touched files are missing; the archived CUDA checks do not establish those backends' compatibility.

Existing discussions checked

• file tier: unbuffered reads on Linux too (O_DIRECT + the kernel's asynchronous reads) #773 already added Linux O_DIRECT/AIO; this proposal concerns the lifetime of its worker/TLS state, not a replacement I/O implementation.
• file_cache_keeps() picks unbuffered (O_DIRECT) expert reads on a 32 GB + 12 GB GPU box with --resident-budget-gib, halving decode; STRATA_UNBUFFERED_LOAD=0 restores it #1194 concerns the page-cache heuristic. Keep that behavior; do not force O_DIRECT on machines where the useful expert set fits in page cache.
• [Engine] Linux: read the experts from the GGUF with O_DIRECT under STRATA_UNBUFFERED_LOAD=1 #3624 concerns startup GGUF loading, a different path.

Would maintainers accept a first PR limited to this worker-lifetime scope, or prefer a measurement-only investigation of the setup/teardown cost first?

Prepared with AI assistance. No engine PR has been opened pending agreement on scope.