[Zcg2021]

Problem and result

A short interactive request on a single native engine waits for a long response to finish. This experimental, opt-in Python scheduler lets a waiting higher-priority request interrupt and resume that response through the existing STOP/DONE protocol and conversation RAM cache. Only one native GEN runs at a time; it does not allocate batch slots or change defaults.

Related discussion

#2821; #2109 concerns batch admission rather than interruption of an already-running single-engine response. Maintainer scope agreement is still requested in #2821. This existing PR remains draft pending that discussion and hardware validation on the current base.

Approach

The scheduler drains all accepted tokens through DONE before changing ownership. Continuation uses original token IDs and keeps the original stream parser, UTF-8 decoder and tool-call identity alive. Minimum run time and waiting-age priority prevent short requests from indefinitely starving long work. Parked cancellation does not stop another owner; engine death fails affected streams.

The current rebase is onto main 870aedd9e1ebe1c91f94da93b59af17729f6c5c0 (0.2.8). README retains upstream's new layout. The rebase also preserves the new prefill_only argument when the normal generator is extracted into _generate_single; an uncorrected automatic merge raised NameError on an ordinary request. split_serve is explicitly rejected with solo_priority, because its proxy forwards single-engine attributes despite using two engines. Regression tests cover the default prefill-only protocol, invalid calls, split-mode rejection and logprob ownership across actual simulated preemption/resumption.

No native source, model, tokenizer or cache-format changes. Production Maya/Strata services and settings were not changed for this update.

Validation

Tested revision: 1e1b2672eb104d4b4c56020ba529feeaeffa2646, 2026-10-30. Pinned runtime requirements in isolated environments; protocol/streaming tests use fake engines, not real GPUs.

Passed:

• macOS / Python 3.13.0: python -m unittest serve.test_solo_priority serve.test_logprobs serve.test_split -v: 73 tests, OK, 25.258 s. Covers interruption/resumption, queued/active/parked cancellation, engine death, UTF-8, tool arguments, usage, logprobs and split-serving compatibility.
• macOS: python tools/minimal_server_smoke.py --report <new-absolute-path>: 51 tests, OK (4 skipped), 103.346 s.
• Portable monitor CI: all four Linux/Windows × Python 3.10/3.11 jobs passed on the tested revision.
• Ubuntu / Python 3.11: python -m unittest discover -s serve -p 'test_*.py': 1278 tests, OK (17 skipped), 424.214 s. Runtime smoke: 51 tests, OK (4 skipped), 106.664 s. CI on the tested commit.
• git diff --check origin/main: passed.

Failed attempts:

• Initial rebase preflight, before the compatibility fix: python -m unittest serve.test_solo_priority.Streaming.test_terminal_state_tracks_an_admitted_solo_request -v: 1 test, 1 error (NameError: name 'prefill_only' is not defined). Corrected in the tested revision; included in the passing focused suite above.

Not run:

• Native CUDA/HIP/SYCL/Metal builds/execution, current real-model speed/quality comparisons or a new soak. No native source was modified. This update does not establish native performance or correctness on 0.2.8; active user services were left running.

Earlier hardware evidence (not rerun on this base)

• 0.2.0-beta report, configuration and raw results: original non-abliterated Qwen IQ3_S, RTX 3090, Windows 11. Three flag-off/on pairs reduced median short TTFT from 20.523 to 0.610 s, with aggregate throughput 111.29 to 107.88 tok/s (-3.1%). The one-hour soak recorded 424 requests, 410 completions, 14 intentional cancellations and 290 preemptions, with no unexpected transport errors or validation failures. These are workload-specific historical measurements.
• 0.2.2 report and raw data: same original model family and native CUDA lifecycle checks, including cancellation and tool round trips. The report retains failed attempts and upstream controls.

Limits

Cold/partial-cache-hit prefill and restoration can still delay short requests. Historical soak short TTFT reached 46.663 s while reading an unfinished long prefix; this is not a real-time latency guarantee. Restored partial prefill reaches its first generated token before yielding to avoid discarding the checkpoint. Images and checkpoint opt-outs are non-preemptible. Batch, replicas and split serving cannot be combined with this option. The feature stays off by default; no universal speedup or token-identical native sampling across separate runs is claimed.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[Zcg2021]

/reopen

[github-actions]

The check passed, but GitHub would not reopen it (Validation Failed: ). If the branch was deleted, restore it or open a new pull request and link this one.

[Zcg2021]

/reopen

[github-actions]

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.