serve: cooperative prefill preemption at safe chunk boundaries (--prefill-preempt) by j-luwierski · Pull Request #656 · Niko1221/Strata

j-luwierski · 2026-10-03T16:09:52Z

Problem

Serving is sequential by design: the FIFO holds a mutex for a request's entire lifecycle, so a short interactive request sent during a long prefill waits for the whole prefill plus decode.

For orchestrator + sub-agent workloads sharing one Strata server, this resulted in tens of seconds of head-of-line blocking.

Measured example: 40.5 s of queue wait behind a 50K-token prefill.

Solution

This PR adds cooperative prefill preemption at safe chunk boundaries.

A long-running request A (for example, a 200K-token prompt) can park at a completed prefill chunk boundary, allowing a short request B to run immediately. Once B finishes, A is restored and continues as if it had never been interrupted.

The execution model remains intentionally simple:

still one sequence at a time
no decode batching
no concurrent slots

Engine (`--prefill-preempt`, opt-in)

Prefill::should_suspend
- Checked once per completed chunk
- Single-GPU path only
- Suspension happens only after the chunk completes
- The normal run tail still executes, which:
  - commits ple_prev for the boundary
  - drains the compute stream
  - drains the copy stream
SuspReq
- Stores a complete suspension snapshot:
  - the running half via the existing checkpoint machinery
  - positional state of every owned QSA layer:
    - KV blocks
    - pooled rows
    - spare key
    - idx_block_pos
  - drafter KV state
- Admission is copy-only
- If snapshot creation fails, request A remains live and continues reading normally
Engine protocol:
- SUSPENDED <pos> <total> id=N
- RESUME id=N
- CANCEL id=N
- YIELD
  - boundary offer emitted by the server
  - the engine itself cannot see the server request queue
- Output lines may carry an id=N suffix for demultiplexing
- Requests without an ID behave exactly as before
Preemption gates:
- minimum park position
- per-request park cap
- snapshot RAM budget admission (T18)
- single GPU only
- layer split fails closed

Server (`--prefill-preempt`, opt-in)

The FIFO lock is split into ownership periods instead of being held for the request's entire lifecycle:

GEN -> SUSPENDED
RESUME -> next SUSPENDED / DONE

A parked request waits outside the FIFO lock and receives keep-alives while suspended.

Additional behavior:

if the parked request's waiter disappears, the park is withdrawn in place
if the engine dies while a request is parked, the request fails cleanly
EngineRequest supports id=N suffix demultiplexing

New observability:

/metrics

prefill_preemptions
prefill_resumes
max_queue_wait_ms

/status

parked_requests

Results

Test system:

GPU: RTX 4070 Ti SUPER
Quant: IQ3_XXS
VRAM: 16 GB

Metric	Feature off	Feature on
B queue wait (`A prefill = 50K`)	`40.5 s`	`2.9 s` (-93%)
A total wall time	`43.8 s`	`45.3 s` (+3.4%)

The task targets were:

≥ 80% reduction in queue wait
< 10% penalty to the interrupted request

Both targets are met with margin.

Verification

Bit-exact parity

Tokens and the complete state fingerprint are identical between:

uninterrupted request A
request A after park -> interim request -> restore

Verified state includes:

gdn
ple
tails
pooled
kv
dead
pooled_full
mtp
ple_prev

Tested suspension boundaries:

4096
8192
12288

An interim request was executed between park and restore with decode depths:

0
1
32

The drafter KV state is included in the parity check.

Trace diagnostics

Added diagnostics include:

STATE_POINT
VERIFY_POINT
WINDOW

These provide:

per-phase state hashes
per-row verification-window fingerprints
position
offered-T count
accepted count

GPU scenarios

12 GPU scenarios: ALL PASS

Coverage includes:

boundary tests ×5
park-only tests ×3:
- early
- mid
- late
KV streaming:
- mode 1
- residency-map reset
repeat-3:
- three consecutive parks
cancel while parked
verification that decode never yields

Scheduler tests

Fake-engine scheduler suite: 9 tests

Coverage includes:

no contention
preemption when another request queues
FIFO ordering
parked-request cancellation
queued-request cancellation
engine death while parked
shutdown without deadlock

Existing test suites

ctest: 63/65
Python: 137 OK

The two failing ctest cases are environmental and reproduce on pristine main.

Bugs Found and Fixed During Verification

The tests caught several issues during development:

A stale YIELD could re-park a resumed request at its next boundary.
preempt_count was saved before the increment, allowing --prefill-preempt-max to be bypassed.
A resumed tail with length <= --short-read could switch to verifier windows differently from an uninterrupted read.
A lend-size layout mismatch caused an E-9 failure across buffer layouts.

All of these were fixed before the final validation runs.

Limitations / Future Work

Current limitations:

only one parked request at a time
image requests never park
long-queue fairness currently uses a bounded wait:
- --prefill-preempt-max-wait-s
- default: 30 s
this is not yet a full scheduler
snapshot RAM is not enforced through a single hard allocation budget
- admission estimates the required memory and refuses before the large allocation
- snapshot vectors are still separate allocations
- allocation failure can therefore still surface as bad_alloc

A preemptable text request (GEN ... id=N) may park at a committed chunk boundary when another GEN line is queued: Prefill::run gains should_suspend, checked once per completed chunk on the single-GPU path, and the serve loop snapshots the request's whole state - the running half through the existing checkpoint machinery (ple_prev re-derived from the prompt), every QSA layer's KV blocks, pooled indexer rows, spare key and block position, and the drafter's ring - into one SuspReq record. The queued request runs; RESUME id=N restores the record (a streamed layer's residency map is reset so every block re-streams from the restored host copy) and the prompt continues at the recorded position. One parked request at a time; decode never yields; a layer split refuses the feature at startup; a failed snapshot copy leaves the request reading (the session is never destroyed to discover the copy did not fit). Lines for an id-carrying request carry an id=N suffix; requests without one behave exactly as before.

The server's --prefill-preempt splits the whole-request FIFO into ownership periods: GEN to SUSPENDED, RESUME to the next SUSPENDED or DONE. A parked request waits for the queue to drain OUTSIDE the lock (with keep-alives), so the queued request always gets its turn; a park whose waiter vanished is withdrawn in place. The engine cannot see the server's queue, so the server offers boundaries with a YIELD flag (like STOP, honoured at chunk ends) while its request is reading and another is queued. StrataEngine grows open_request/EngineRequest (Parked/resume/cancel_parked/drain with id-suffix demultiplexing); MockEngine speaks the same protocol for tests; a parked request's bookkeeping records unconditionally (the shared busy flag says nothing once another request interleaved). /metrics gains prefill_preemptions, prefill_resumes, max_queue_wait_ms; /status shows parked_requests. serve/test_preempt.py: the scheduler state machine (A1-A8 + metrics) over a protocol-level fake engine - 9 tests, no GPU.

tools/prefill_preempt_test.py: drives strata --serve directly and proves the feature on the real model - output parity (greedy) and STRATA_STATE_HASH equality across five park boundaries, a park with nothing in between (pure restore), KV-streaming (mode 1), three consecutive preemptions, cancellation of a parked request, and decode never yielding. Deterministic config (--adapt-swaps 0, --pcie-frac 0, --suffix-draft 0), warm-up on both engines (a virgin engine drafts differently), same-config references for the streaming scenario. The drafter's KV hash (draft-quality state, prefill.hpp E-9's 'not bit-identical' contract) is reported, not gated; everything the target model reads must match and does. Engine fix found by the harness: a stale YIELD parked a resumed request again at its next chunk boundary - the park consumed the offer, so RESUME clears it (a fresh one rides the next YIELD line).

tools/prefill_preempt_bench.py: A (a 50K-token prompt) owns the engine, B (a small request) queues three chunks in, and B's queue-to-DONE latency and A's total wall time are measured with --prefill-preempt off and on. 50K tokens: B waits 40.5 s without the feature and 2.9 s with it (-93%), A pays +3.4% - the task's >= 80% / < 10% acceptance shape. Also adds the regression runner (tools/prefill_preempt_regression.sh) and a small MTP-block hash debug gate (STRATA_STATE_HASH_MTP_BLOCKS) the parity work used.

…he next YIELD A YIELD landing while the resume was still being processed was wiped by the resume's own clear (the park consumed the earlier offer) - a missed park, seen as a repeat-3 flake. The engine's contract stands (a fresh YIELD rides any later line); the harness just paces itself like the server does.

…on run auto sizing depends on how much VRAM happens to be free; a leftover engine from an earlier test once shrank the run's first (control) engine's resident set relative to the candidates - GPU-resident experts vs CPU misses round differently, and the state hashes stopped being comparable (tokens diverged at the same count). One probe engine reads the machine's auto size; every control and candidate of the run is started with that exact slot count. The regression runner also clears leftover engines before each pass.

the window's T was gated by the DRAFTER's probabilities, and the drafter's prompt KV is E-9 non-bit-identical territory: one flipped probability threshold reshuffled every later window shape and, through the GDN/QSA arithmetic, the main state's ULPs - which surfaced as rare cross-engine hash drift (gdn/tail/pooled/kv) in long regression runs. Constant shapes keep the comparison about the park, not about the drafter.

this engine shows rare ULP-level prefill nondeterminism (identical prompts, different PLE hashes - observed between plain control engines, no parking involved). A failed scenario re-runs once on fresh engines: a real park defect reproduces, machine noise does not. Both attempts are reported.

the engine's decode arithmetic has per-process modes that rotate with the adaptive tier's arming: --adapt-swaps 0 (machinery disarmed, usage array absent) landed engines in a different mode than an armed one, and plain controls disagreed across invocations. --adapt-every 100000 (rotation only after 100,000 rounds, the rope-scaling benchmarks' recipe) gives identical state hashes across fresh engines in separate invocations. The harness now pins that instead of --adapt-swaps 0.

the machine's arithmetic mode drifts over minutes (plain engines land in discrete hash modes), so the run's single early control stopped being a reference for scenarios that run minutes later - the candidates agreed among themselves while the control sat in another mode. Each scenario now runs its own control (same args, started seconds before) and is compared against it.

the flake hunt's root cause, found: a resumed request's remaining segment lent buffers sized to the REMAINING tokens (490 -> a 512-token relayout), while an uninterrupted request's same final cells ran inside its whole-prompt 2048-token layout. The batched draft-KV path is E-9 non-bit-identical across layouts (different scratch tiling), so the drafter's last-chunk cells differed, the drafts differed, the final verify window's acceptance count differed, and the consumed-tail state (gdn/kv/tails) drifted - visible only at late parks and only against a whole-prompt control. Batched segments now lend at least one full chunk: buffers are sized >= needed, so a short segment on a chunk-sized carve is fine and the layout always matches an uninterrupted read of the same cells. park-only at 2048/8192/12288 is now bit-exact INCLUDING the drafter's KV (previously reported as a delta).

…ate-hash fields rebased onto origin/main (99f3dbd): the engine grew a conversation cache with whole-conversation parking (SavedConversation/park_current), multi-session QSA carves (a session owns ordinals [qsa_ord0, qsa_ord0+qsa_alloc)) and a state fingerprint with dead/pooled_full components. The park snapshot/restore now walks the session's OWN ordinals (it read [0, n_qsa_layers) before - wrong under the carve, and the drift that produced), the harness parses the new fingerprint fields, and the server merged with the upstream evolutions (Niko1221#471 prompt_read bookkeeping, Niko1221#457 draft counters, Niko1221#530 thinking-budget note, aliases, save-every profiles). Harness on the rebased engine: boundaries x5, park-only x3 (bit-exact, drafter KV included), streaming-kv, repeat-3, cancel-parked, decode-nopark - 12 scenarios, 10 green; 2 known-open flake (an interim request's own tokens under the upstream conversation cache) documented in the report.

…iew.patch) Reviewer fixes carried over the rebase onto origin/main: - rec.preempt_count saved before the increment reloaded the old budget on RESUME - --prefill-preempt-max could be evaded by repeated displacement - a resumed tail <= --short-read switched to verifier windows while the uninterrupted read kept its segment batched; the segment's original end is kept and its remainder stays batched - the harness no longer passes a scenario on a fresh-engine retry, treats --prefill-preempt as a valued flag correctly, runs park-only controls without B, starts log files fresh, and fails the strict diagnostic on MTP deltas (stale uncommitted-draft cells reported, not gating) - the state fingerprint handles FP16 KV and fails on copy/sync errors - --trace diagnostics: STATE_POINT/VERIFY_POINT/WINDOW per phase, residency hashing, GEN 0 debug prefix read under STRATA_PREEMPT_TRACE, and a CPU-only harness self-test (tools/test_preempt_harness.py) Also fixes the merged server's run() tail (restored the done-yield the rebase conflict dropped) and the Niko1221#530 parser-state reference.

Reviewer fixes for the conversation-cache and lifecycle interactions: - RESUME/conversation-cache: park the finishing B before A's overwrite, limit_reuse(0) on owner change, restore A's own checkpoints - EngineRequest.drain: deadline on STOP/CANCEL instead of blocking forever - run_preemptable: cancel checked before GEN and on heartbeats - requests with an active thinking budget keep the non-preempt path - susp_save/susp_restore: compact snapshot indexed by j, device state by global ordinal i (qsa_ord0 != 0 safety) - harness: fingerprint key map fixed for missing optional fields, dead/ pooled_full/block in the strict comparison, own-ordinals iteration, request-owned DONE (no borrowed timings), fresh log files - unload/restart: parked_requests blocks unload, shared ensure_loaded, EngineRequest bound to its process - prefill segment: stale sp.suspended() no longer leaks across paths; a denied park keeps the segment path and layout - lend: full-chunk layout only for a continued segment; upstream request-sized lending elsewhere plus serve/test_preempt_regressions.py regression guards and REVIEW2 notes

… gates) T18: --prefill-preempt-snapshot-mib (default 4096, 0 = no cap). Before a park is offered, the snapshot estimate (running half + dead/block + pooled rows + KV and drafter blocks through the position reached, mirroring susp_save's copies) is compared against the budget; over-budget yields are declined BEFORE the big allocation, logged once, and A keeps reading its prompt while B waits. T19: --prefill-preempt-max-wait-s (default 30, 0 = indefinite). A parked request stops waiting for a FULL queue drain after the limit and resumes between queued periods - the measurable postponement bound for a continuous stream of arrivals (REVIEW2's starvation concern). Built and smoke-verified: --quick --trace --interim-max-new 32 ALL PASS.

- park-admission budget moved into a shared helper (include/strata/core/prefill_preempt_budget.hpp) + unit test prefill_preempt_budget_test - serve/engine_gate.py + serve/test_engine_gate.py: the server refuses to wait for SUSPENDED from an engine that did not advertise preempt - trace/bench/harness updates per review3

…ots; crash telemetry for silent deaths The native-pack packer capped a pinned cache at the free VRAM measured at engine start, so two engines of one comparison built different resident expert sets from the same flag (5621 vs 5616 slots, GPU round of 2026-10-03) and the interim request's greedy decode split from its reference into an early EOS - repeat-3's 128-vs-5. An explicit N now packs exactly the first N ranked pairs (a pure function of N, header-tested); auto keeps sizing from free VRAM, and a pinned N that cannot fit names the knob in a warning instead of silently delivering a different resident set. The boundary-2048 engine death (silent, no stderr, returncode unrecoverable) gets crash telemetry: SIGSEGV/SIGBUS/SIGILL/SIGFPE/SIGABRT write signal, si_code, fault address and a backtrace of the faulting thread to stderr, then the process still dies by its signal (fork-tested). The parity harness gates every scenario on the expert tier the engines actually built (require_pinned_geometry), samples VmRSS into the protocol, adds a repeat-10 scenario and a post-DONE health request, and its stale-cell/exit gates keep the earlier REVIEW3 guarantees. GPU (RTX 4070 Ti SUPER, iq3_xxs, binary a19f49f0): boundary-2048 x20 fresh runs and 2 literal interim-0 runs PASS with equal geometry within every run; repeat-3 --max-new 128 PASS; the 13-scenario suite and the 32-token tail PASS. Python 185 OK (7 skip), ctest CPU 7/7.

…s over the active slots, slot conversation cache, prompts read beside decoding slots, BYIELD - --slots N is the same as --batch N; a count the engine cannot run (no --serve, 1, more than 8, a split on one GPU, sessions that do not fit, --batch-groups that do not divide) is a warning and fewer slots or none, never a refusal; INFO says batch_slots=N when slots run - a batch window runs over the active slots only (Verifier::run_slot_rows, graphs per row list): an idle slot is not written, so it keeps the conversation it served - a request whose prompt starts with what an idle slot holds (all of it, or its turn checkpoint) continues from the slot's sessions instead of reading the history again - with slots, a prompt is read one prompt chunk at a time (the same chunks as one run) and the slots decode between two chunks for half as long as the chunk took (STRATA_BATCH_DECODE_SHARE) - BYIELD <slot>: the prompt being read stops at its next chunk boundary, the part read is copied into the slot, YIELDED <slot> <tokens> + DONE cancel; the same request sent again goes on from there with the same chunks and checkpoints (#656's cooperative preemption, a slot as the park) Default (no --batch) unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Niko1221 · 2026-10-04T12:35:37Z

Thank you. The idea is in 0.1.39 as part of the parallel requests (#465): with "parallel": N, a long prompt read gives way at a chunk boundary when a request with a much shorter prompt is waiting (BYIELD <slot>). The part read so far waits in a batch slot instead of a RAM snapshot, and the read goes on afterwards with the same chunks (token-identical on an RTX 5070). We are not merging this PR itself: its YIELD/RESUME protocol overlaps the slot one, and we would rather keep one. Closing; thanks for the clear problem statement and the measurements.

j-luwierski · 2026-10-04T12:46:45Z

No problem at all, @Niko1221 — and nice work on getting this into the parallel-request path. The BYIELD approach sounds like a cleaner fit with the architecture, so closing this PR makes perfect sense.

No worries about this one — I’m glad the underlying problem is solved. I just think a little more visibility around work in progress could save everyone some time in the future.

…iko1221#646/Niko1221#655/Niko1221#583/Niko1221#656-reimpl/YIELDED), ours kept where upstream lagged (Niko1221#584 carve+search, Niko1221#742, Niko1221#500, Niko1221#743, Niko1221#727/Niko1221#734, Niko1221#567+537 union)

…s over the active slots, slot conversation cache, prompts read beside decoding slots, BYIELD - --slots N is the same as --batch N; a count the engine cannot run (no --serve, 1, more than 8, a split on one GPU, sessions that do not fit, --batch-groups that do not divide) is a warning and fewer slots or none, never a refusal; INFO says batch_slots=N when slots run - a batch window runs over the active slots only (Verifier::run_slot_rows, graphs per row list): an idle slot is not written, so it keeps the conversation it served - a request whose prompt starts with what an idle slot holds (all of it, or its turn checkpoint) continues from the slot's sessions instead of reading the history again - with slots, a prompt is read one prompt chunk at a time (the same chunks as one run) and the slots decode between two chunks for half as long as the chunk took (STRATA_BATCH_DECODE_SHARE) - BYIELD <slot>: the prompt being read stops at its next chunk boundary, the part read is copied into the slot, YIELDED <slot> <tokens> + DONE cancel; the same request sent again goes on from there with the same chunks and checkpoints (#656's cooperative preemption, a slot as the park) Default (no --batch) unchanged.

…, windows over the active slots, slot conversation cache, prompts read beside decoding slots, BYIELD - --slots N is the same as --batch N; a count the engine cannot run (no --serve, 1, more than 8, a split on one GPU, sessions that do not fit, --batch-groups that do not divide) is a warning and fewer slots or none, never a refusal; INFO says batch_slots=N when slots run - a batch window runs over the active slots only (Verifier::run_slot_rows, graphs per row list): an idle slot is not written, so it keeps the conversation it served - a request whose prompt starts with what an idle slot holds (all of it, or its turn checkpoint) continues from the slot's sessions instead of reading the history again - with slots, a prompt is read one prompt chunk at a time (the same chunks as one run) and the slots decode between two chunks for half as long as the chunk took (STRATA_BATCH_DECODE_SHARE) - BYIELD <slot>: the prompt being read stops at its next chunk boundary, the part read is copied into the slot, YIELDED <slot> <tokens> + DONE cancel; the same request sent again goes on from there with the same chunks and checkpoints (Niko1221#656's cooperative preemption, a slot as the park) Default (no --batch) unchanged.

…, windows over the active slots, slot conversation cache, prompts read beside decoding slots, BYIELD - --slots N is the same as --batch N; a count the engine cannot run (no --serve, 1, more than 8, a split on one GPU, sessions that do not fit, --batch-groups that do not divide) is a warning and fewer slots or none, never a refusal; INFO says batch_slots=N when slots run - a batch window runs over the active slots only (Verifier::run_slot_rows, graphs per row list): an idle slot is not written, so it keeps the conversation it served - a request whose prompt starts with what an idle slot holds (all of it, or its turn checkpoint) continues from the slot's sessions instead of reading the history again - with slots, a prompt is read one prompt chunk at a time (the same chunks as one run) and the slots decode between two chunks for half as long as the chunk took (STRATA_BATCH_DECODE_SHARE) - BYIELD <slot>: the prompt being read stops at its next chunk boundary, the part read is copied into the slot, YIELDED <slot> <tokens> + DONE cancel; the same request sent again goes on from there with the same chunks and checkpoints (Niko1221#656's cooperative preemption, a slot as the park) Default (no --batch) unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…t a chunk later or never (#656) The server sends BYIELD for a waiting request under half as long only when a PP line (or the 10 s heartbeat) arrives, and the engine looks for it at its next boundary. Where a boundary's own PP line is the only line for a while - every chunk of a --batch read on one GPU, and #2050's pipelined pieces - the BYIELD answers that line and reaches the engine after it has looked: the read gives way one boundary later, or never when that was the last one it could take. It is in time only where the engine saves a periodic checkpoint after the PP line (every 16,384 tokens by default). _control() now also calls on_quiet() whenever the engine has been quiet for half a second (its existing poll), and the solo and admission reads run the same check there as at a PP line. A BYIELD for a read that has ended is ignored by the engine, as before. serve/test_parallel.py: the fake engine takes --pp-every K (a PP line and a boundary every K chunks); a solo read and an admission must give way at their first boundary (both fail on main), and the rule (only to a much shorter request) holds on the poll too. Measured on one RTX 5090 (Swift 1.5 IQ3_S, --batch 4, the engine built from main), main's server and this one, two engine starts each (M F F M): a one-line request sent 0.2-8.6 s into a cold ~33K read, with nothing else decoding and with one stream decoding. Late (a boundary passed after the request waited): 28 of 40 runs on main, 12 of which never gave way; 4 of 40 with this change, all requests that started waiting 0.02-0.10 s before that boundary's PP line. Where main was late the short request's TTFT went 3.56 -> 1.70 s (nothing decoding) and 4.40 -> 1.57 s (one stream); where main never gave way 4.60 -> 1.51 s and 7.59 -> 2.35 s. Where both gave way at the same boundary, both requests' TTFTs were the same (medians within 0.02 s). On 2x RTX 5090 with #2050's STRATA_PIPED_READ_PIECE=8192: 6 of 12 late without it, none with it.

…t a chunk later or never (Niko1221#656) The server sends BYIELD for a waiting request under half as long only when a PP line (or the 10 s heartbeat) arrives, and the engine looks for it at its next boundary. Where a boundary's own PP line is the only line for a while - every chunk of a --batch read on one GPU, and Niko1221#2050's pipelined pieces - the BYIELD answers that line and reaches the engine after it has looked: the read gives way one boundary later, or never when that was the last one it could take. It is in time only where the engine saves a periodic checkpoint after the PP line (every 16,384 tokens by default). _control() now also calls on_quiet() whenever the engine has been quiet for half a second (its existing poll), and the solo and admission reads run the same check there as at a PP line. A BYIELD for a read that has ended is ignored by the engine, as before. serve/test_parallel.py: the fake engine takes --pp-every K (a PP line and a boundary every K chunks); a solo read and an admission must give way at their first boundary (both fail on main), and the rule (only to a much shorter request) holds on the poll too. Measured on one RTX 5090 (Swift 1.5 IQ3_S, --batch 4, the engine built from main), main's server and this one, two engine starts each (M F F M): a one-line request sent 0.2-8.6 s into a cold ~33K read, with nothing else decoding and with one stream decoding. Late (a boundary passed after the request waited): 28 of 40 runs on main, 12 of which never gave way; 4 of 40 with this change, all requests that started waiting 0.02-0.10 s before that boundary's PP line. Where main was late the short request's TTFT went 3.56 -> 1.70 s (nothing decoding) and 4.40 -> 1.57 s (one stream); where main never gave way 4.60 -> 1.51 s and 7.59 -> 2.35 s. Where both gave way at the same boundary, both requests' TTFTs were the same (medians within 0.02 s). On 2x RTX 5090 with Niko1221#2050's STRATA_PIPED_READ_PIECE=8192: 6 of 12 late without it, none with it.

…prompt read give way to a short request one chunk late, or never; its TTFT -64% where it hit (Niko1221#656) Source: Niko1221#2170 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…its own (opt-in, experimental STRATA_PIPED_CONCURRENT) With a layer split and --batch-groups > 1 a prompt read is one sp.run, so every decoding slot stops for the whole read (Niko1221#1906); Niko1221#2050's pieces bound the stop but drain the pipeline at each piece. With STRATA_PIPED_CONCURRENT=1 the read runs on a worker thread while the main thread keeps the groups going through the pipeline (pump) and takes BSTOP and BYIELD meanwhile. The prompt's K/V goes to the main sessions and the slots' to their own; the windows' streams get the device's highest priority on CUDA (STRATA_PIPED_CONCURRENT_PRIO=0 keeps the default). - The slots' doorbells are served on a thread of their own that makes no CUDA call (Verifier::batch_serve, now batch_poll's per-layer service too, and set_external_serve). Two hangs found while building this, read from the stall dumps against the PDB, had Niko1221#31's shape: the main thread waited for the driver's lock in cudaSetDevice while its holder waited for the GPU (a cudaFree in the drafter's prompt pass; cuBLASLt's lazy load of a GEMM module), and a window on that GPU waited for its doorbell. - MtpDrafter::reserve_prefill captures the drafter's prompt graphs and sizes its records before the read. - Prefill::stop_at_chunk: a BYIELD ends the run at its next chunk, also with no slot decoding; the pipeline is drained there and the part read gives way as Niko1221#656 does. A read nobody waits for is not cut. - The reader is joined on every way out (an exception too). Off beside STRATA_PREFILL_HELP and STRATA_SPLIT_MTP_BATCH, which allocate during a read.

BYIELD は PP 行（か 10 秒の鼓動）が来たときだけ送っていたので、塊の PP 行しか出ない読み（1 GPU の --batch の各塊など）では、エンジンが見た後に届き、1 塊遅れるか譲らないことがあった。_control が半秒静かなときにも on_quiet() を呼び、単独の読みと受け入れの読みは PP 行のときと同じ判定をそこでも行う。

…t a chunk later or never (Niko1221#656) The server sends BYIELD for a waiting request under half as long only when a PP line (or the 10 s heartbeat) arrives, and the engine looks for it at its next boundary. Where a boundary's own PP line is the only line for a while - every chunk of a --batch read on one GPU, and Niko1221#2050's pipelined pieces - the BYIELD answers that line and reaches the engine after it has looked: the read gives way one boundary later, or never when that was the last one it could take. It is in time only where the engine saves a periodic checkpoint after the PP line (every 16,384 tokens by default). _control() now also calls on_quiet() whenever the engine has been quiet for half a second (its existing poll), and the solo and admission reads run the same check there as at a PP line. A BYIELD for a read that has ended is ignored by the engine, as before. serve/test_parallel.py: the fake engine takes --pp-every K (a PP line and a boundary every K chunks); a solo read and an admission must give way at their first boundary (both fail on main), and the rule (only to a much shorter request) holds on the poll too. Measured on one RTX 5090 (Swift 1.5 IQ3_S, --batch 4, the engine built from main), main's server and this one, two engine starts each (M F F M): a one-line request sent 0.2-8.6 s into a cold ~33K read, with nothing else decoding and with one stream decoding. Late (a boundary passed after the request waited): 28 of 40 runs on main, 12 of which never gave way; 4 of 40 with this change, all requests that started waiting 0.02-0.10 s before that boundary's PP line. Where main was late the short request's TTFT went 3.56 -> 1.70 s (nothing decoding) and 4.40 -> 1.57 s (one stream); where main never gave way 4.60 -> 1.51 s and 7.59 -> 2.35 s. Where both gave way at the same boundary, both requests' TTFTs were the same (medians within 0.02 s). On 2x RTX 5090 with Niko1221#2050's STRATA_PIPED_READ_PIECE=8192: 6 of 12 late without it, none with it.

之前的槽位改造把第 4 个会话挡在 3 个保护槽外排队，guest 槽却常年空闲——机动区闲置、会话干等 120 秒窗口，两头浪费。现在所有权开启时，pick 的最后一级是 guest 借用：自己的槽、无主槽都没有时，一个空闲的 guest 槽就是新会话的落点。借用是完整的所有权（admission 照常 _note_owner）：下一轮回它在 guest 槽里建的链，宽限期一过槽照旧回到一次性调用手里。guest 区不亏——一次性调用本来就会驱逐槽里任何东西，会话的链在那里的每一秒都是白赚的；分区语义（protected_slots 单独存在时）不变，仍然禁止越区。连带修了借用暴露的 Niko1221#656 盲区：BYIELD 让路前不看等待者拿线后能不能进槽，一个被所有权挡在所有槽外的短请求会让长读取反复让路（YIELDS_MAX 次重读）而它自己始终进不了槽——两边都饿着。 _shorter_waiting 现在问"这个等待者有槽可进吗"：zone 匹配的空闲槽（protected 还要求非 live-owned）、或保护等待者的 guest 借用； wait_lens 条目带上 zone，无窗口/无分区时退化为旧规则。测试（先红后绿）：借用语义 5 个（排队者借空闲 guest、借用槽的所有权与宽限返还、一次性调用不受阻、guest 忙时照旧排队、纯分区不借），既有 4 条"排队而非借 guest"的断言按新语义更新； ShorterWaiting 组 8 个钉死让路判定（含死 owner 槽、纯分区不借、无分区退化为旧规则）。端到端 2 个排队心跳用例 skip 并说明：假引擎的 HOLD 占控制线贯穿其解码全程，单元测试的时间窗内构造不出 "全忙"——满房排队的活体观察归 tools/strata_concurrency_probe.py，心跳代码本身本次未动。全量 1737 passed（基线外新增 10），4 failed 为既有（engine_hash HTTP 404 ×2、golden 基线过期 ×2）。

…stream's new trees; the engine stays this branch's Upstream's 271 commits since a0116b2. Where both sides changed a file of the engine (src/, include/: 49 files) or CMakeLists.txt, this branch's version stays whole; the files only upstream changed are upstream's (the geometry and layout headers, the GGUF reader, device, vmm, weights, platform, dequant and a few kernels: layer.cpp's own qsa_shapes() gave way to layout.hpp's). The engine keeps its version, 0.1.43.1: none of 0.1.44's engine additions are in it, and setup's MIN_ENGINE (0.1.44) is upstream's. Taken: - serve/: upstream's server with this branch's /v1/choose beside its POST /v1/images/generations (Qwen-Image through stable-diffusion.cpp); a long read gives way to a short request at the next chunk (Niko1221#656); quoted tool tags are text (Niko1221#2026); "env_to_user" (opt-in, opencode's <env> block in the first user message); the image encoder only while there are pictures ("elastic"); the engine started again when it stopped inside the GPU runtime's set-up; pictures keyed by their bytes' hash for the conversation cache (img_keys, Niko1221#2228; the engine side ported next); the stdin reader's QUIT race. - setup.py, tools/ (this branch's iq_pack.py stays: its packs are the ones this engine reads), docs/ (DETAILS.md gains upstream's sections this branch's copy lacked, the resident low-RAM mode through the look-ahead statistics, beside this branch's own), the Metal, NVFP4, Qwen3.6, images, SYCL, HIP, Nix trees and their tests. Not taken (the engine files they change stay this branch's; the new sources are in the tree, not built): NVFP4 experts (e51b437..e48c803); the Qwen3.6-35B-A3B engine (14ba7a6..800a090, 1481716); STRATA_HC_Q8 (0a379a1, b60aab1, 446bec5, 2d47caa: --hc-q8 does this here); the batched commit graph (6f3e381, STRATA_COMMIT_BATCH); the dense Q8_0 GEMV a row a block (945fee2); the K-quant prompt kernels as libstrata_kq and the Q6_K experts in the normal build (465cd61); the multi-GPU pipelined decode (24d972e) and the CUDA layer split's link rule (9d41e18); --prefill-head-alias (640ac43); the draft timing and profile switches (009dc6c, 6551454); the arena's pin by need, --adapt-async and the PCIe floor (ab6bce0, f087f78, 55e1283, 1aa0abf, af452c1); the foresight swap's exchange hold (277be72: this engine has no foresight tier); the adaptive overlap's Windows signal (cb22a53: nor the overlap); STRATA_POOL_UMWAIT (caf1856, 54b61db); the arena's THP rule and PROMPT_DIRECT (Linux); Pascal (sm_61), gfx11/gfx12 and SYCL kernel work. Window hashes on fixed text (thinking, 1,200 tokens, --adapt-every 0, the second GPU's tier pinned) equal the previous build's on both models; the server's 783 unit tests pass (11 skipped on Windows); api_cache_test and api_stream_test pass on both models. Claude-Session: https://claude.ai/code/session_01HVYqq39tqLPZXtFfZw8gYc

…its own (opt-in, experimental STRATA_PIPED_CONCURRENT) With a layer split and --batch-groups > 1 a prompt read is one sp.run, so every decoding slot stops for the whole read (#1906); #2050's pieces bound the stop but drain the pipeline at each piece. With STRATA_PIPED_CONCURRENT=1 the read runs on a worker thread while the main thread keeps the groups going through the pipeline (pump) and takes BSTOP and BYIELD meanwhile. The prompt's K/V goes to the main sessions and the slots' to their own; the windows' streams get the device's highest priority on CUDA (STRATA_PIPED_CONCURRENT_PRIO=0 keeps the default). - The slots' doorbells are served on a thread of their own that makes no CUDA call (Verifier::batch_serve, now batch_poll's per-layer service too, and set_external_serve). Two hangs found while building this, read from the stall dumps against the PDB, had #31's shape: the main thread waited for the driver's lock in cudaSetDevice while its holder waited for the GPU (a cudaFree in the drafter's prompt pass; cuBLASLt's lazy load of a GEMM module), and a window on that GPU waited for its doorbell. - MtpDrafter::reserve_prefill captures the drafter's prompt graphs and sizes its records before the read. - Prefill::stop_at_chunk: a BYIELD ends the run at its next chunk, also with no slot decoding; the pipeline is drained there and the part read gives way as #656 does. A read nobody waits for is not cut. - The reader is joined on every way out (an exception too). Off beside STRATA_PREFILL_HELP and STRATA_SPLIT_MTP_BATCH, which allocate during a read.

The admission waited once per request, up to slot_policy.PROTECTION_WAIT_S, when the policy answered "protected" (every idle history still protected), and then asked again allowing an eviction. In production that wait never paid: with more requests than slots, a slot that had just finished held a protected history, and each of 7 waits in 15 minutes (2026-10-15, 3 x RTX A4000, 6 slots, 8 clients) ran the full 5 s and ended in the same eviction, the slot idle meanwhile. The history that the eviction takes is the one with the lowest retention - the least likely to be continued within the wait anyway. Removed: _admission_slot, the per-request protected_since, pick_slot()/place()'s allow_eviction, the protection_wait/protection_end events, PROTECTION_WAIT_S and live.slot_policy.protect_wait_s. A request still waits for a slot only when every slot is busy, re-checked on each release and every 1.0 s as before this change. A read that gives way (Niko1221#656) reserves its slot with the same placement as an admission. The request keeps its own placement plan (choose_slot()). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

control_occupied_wait reached 52-70 s on a single request and 293 s for foreground work across a day: a read gave way only to a waiter whose prompt was under half its own, so two requests of similar length never yielded to each other and the second waited for the whole first read. _shorter_waiting now also qualifies a waiter whose turn has been held for YIELD_WAIT_S (4 s), so the hold is bounded to about one read span plus that wait whatever the two lengths are. The aged rule respects priority: a foreground read yields only to foreground work, because the first version (any aged waiter qualifies) made a foreground request beside three background ones regress from 20.9 s to 36.1 s. serve/test_yield_wait.py pins the three cases: the Niko1221#656 half-length rule, the aged rule, and the priority carve-out. NOT DEPLOYED: the closure keeps the pre-change file until its matched A/B runs.

The admission waited once per request, up to slot_policy.PROTECTION_WAIT_S, when the policy answered "protected" (every idle history still protected), and then asked again allowing an eviction. In production that wait never paid: with more requests than slots, a slot that had just finished held a protected history, and each of 7 waits in 15 minutes (2026-10-15, 3 x RTX A4000, 6 slots, 8 clients) ran the full 5 s and ended in the same eviction, the slot idle meanwhile. The history that the eviction takes is the one with the lowest retention - the least likely to be continued within the wait anyway. Removed: _admission_slot, the per-request protected_since, pick_slot()/place()'s allow_eviction, the protection_wait/protection_end events, PROTECTION_WAIT_S and live.slot_policy.protect_wait_s. A request still waits for a slot only when every slot is busy, re-checked on each release and every 1.0 s as before this change. A read that gives way (Niko1221#656) reserves its slot with the same placement as an admission. The request keeps its own placement plan (choose_slot()). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…1#656) The server reserves a slot and sends BYIELD <slot> when a much shorter prompt waits for the control lines; the engine ignored it, so a 4k-token agent prompt held the engine for its whole read. With --batch N the prompt is now fed in 1024-token slices, and between two of them a pending BYIELD ends the read: the part read stays in the model (its tokens = where it stopped), the engine answers YIELDED <slot> <tokens> + DONE (and BADM 0 for an admission), and the server sends the same prompt again later - the read goes on from there, never from 0. Without --batch the one-at-a-time engine is untouched (one feed, no slices). Measured on the box (4,031-token prompt, a short request 1.2 s behind it, --batch 2 --slot-burst 8): the short request's first token at 0.77 s instead of 5.47 s; the engine's own line "the prompt read gave way after 1024 of 4031 tokens". The HTTP check with the sliced read still gives the reference answer byte-identical to production, three clients interleaving (8 handovers), and a client that leaves mid-stream handled. Still one thing at a time: between its slices the read holds the model, so the short request's later tokens wait for the long read to finish. Interleaving the read slices with the slots' bursts (the scheduler of the design's section 3) is the next step.

j-luwierski marked this pull request as draft October 3, 2026 16:10

j-luwierski mentioned this pull request Oct 3, 2026

bench: community report — RTX 5090, IQ3_S at 1M context (YaRN), agent-style workload #466

Closed

j-luwierski force-pushed the feat/prefill-preempt branch 2 times, most recently from a7a3422 to 29ef162 Compare October 3, 2026 17:59

j-luwierski added 12 commits October 3, 2026 20:03

j-luwierski force-pushed the feat/prefill-preempt branch from 29ef162 to 8205996 Compare October 3, 2026 18:10

j-luwierski added 3 commits October 3, 2026 20:10

j-luwierski changed the title ~~Feat/prefill preempt~~ serve: cooperative prefill preemption at safe chunk boundaries (--prefill-preempt) Oct 3, 2026

j-luwierski force-pushed the feat/prefill-preempt branch 2 times, most recently from 12356b8 to b370ead Compare October 3, 2026 21:50

j-luwierski added 2 commits October 3, 2026 23:50

j-luwierski marked this pull request as ready for review October 3, 2026 21:51

Niko1221 mentioned this pull request Oct 4, 2026

Several conversations at once: batch slots, a pipelined layer split, per-stage dense weights #559

Closed

Niko1221 closed this Oct 4, 2026

Jackwwg83 mentioned this pull request Oct 7, 2026

serve: a long read gives way by what is left to read, not prompt length (#656) #1288

Closed

brocktice mentioned this pull request Oct 12, 2026

--batch: request priority (a free slot and the next prompt read go to the highest priority waiting) #2109

Closed

SuperDistance mentioned this pull request Oct 12, 2026

serve: fix a race that makes a long prompt read give way to a short request one chunk late, or never; its TTFT -64% where it hit (#656) #2170

Closed

SuperDistance mentioned this pull request Oct 13, 2026

batch-groups: read a cold prompt beside the decoding slots, stall 11.9 -> 0.8 s, same work 6.4% sooner (opt-in) #2326

Closed

coryjett mentioned this pull request Oct 28, 2026

[Engine] single-slot mode: let a short request in during a long cold read (BYIELD exists only for --batch) #4489

Open

Navigation Menu

Repository navigation

serve: cooperative prefill preemption at safe chunk boundaries (--prefill-preempt) - #656

serve: cooperative prefill preemption at safe chunk boundaries (--prefill-preempt)#656
j-luwierski wants to merge 17 commits into
Niko1221:mainfrom
j-luwierski:feat/prefill-preempt

j-luwierski commented Oct 3, 2026 •

edited

Loading

Uh oh!

Niko1221 commented Oct 4, 2026

Uh oh!

j-luwierski commented Oct 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

j-luwierski commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Engine (--prefill-preempt, opt-in)

Server (--prefill-preempt, opt-in)

Results

Verification

Bit-exact parity

Trace diagnostics

GPU scenarios

Scheduler tests

Existing test suites

Bugs Found and Fixed During Verification

Limitations / Future Work

Uh oh!

Niko1221 commented Oct 4, 2026

Uh oh!

j-luwierski commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

j-luwierski commented Oct 3, 2026 •

edited

Loading

Engine (`--prefill-preempt`, opt-in)

Server (`--prefill-preempt`, opt-in)

j-luwierski commented Oct 4, 2026 •

edited

Loading