Repository navigation
serve: a client that leaves during an image encode cannot be cancelled — prepare() runs before the cancel event, and the encode holds the single request turn #1999
Description
What happened
Summary
Service.prepare() downloads image URLs and runs the image encoder before the request's
cancel event exists, so a client that hangs up during preparation cannot be cancelled: the
encode runs to completion for a request nobody is waiting for, and — because the encode takes
self.fifo — the engine sits idle for the whole encode while other requests queue behind it.
With "vision": {"gpu": false} the FIFO is not needed for the GPU-contention reason at all.
Environment
Strata 0.1.43 (7695859a), Ubuntu 24.04.5, 2× RTX 5070 Ti 16 GB (layer split 6, one card in a
chipset slot that runs at PCIe 3.0 ×2), serve/server.py --engine strata, model Qwen3.8-Flash-Next
IQ3_S, strata-iq3_s.json: "vision": {"gpu": false, "max_tokens": 1024, "threads": 8},
--vision, layer split 6, no --batch (one request at a time).
What the code does
prepare() takes the engine's turn for the encode (serve/server.py:3484, the comment above it says
it is there so the engine and the encoder are never on the GPU at the same time), and it also
fetches image URLs (3449, 3478, urllib with a 60 s timeout each). All of this happens before
any cancel event exists:
serve/server.py:5366 ids, thinking, max_new = svc.prepare(...) # downloads + encodes here
serve/server.py:5368 cancel = threading.Event() # only now
serve/server.py:5369 self._watch_client(cancel) # #430 #431: the disconnect watcher
the same order in _responses_prepare (5547 → 5590) and _anthropic (5588 → 5590).
prepare() has no cancel parameter, and Vision.encode_all / encode never look at one.
So #1619 ("give the place up when cancelled while queued") and #430/#431 ("a request is cancelled
within a second of its client hanging up") both start where prepare() has already finished.
Measured
A 768×768 random-noise PNG (unique, so nothing is cached) sent to /v1/chat/completions
with max_tokens: 1, client aborted after 2 s; a plain text request sent ~0.6 s later;
/proc/<pid>/stat sampled every 0.5 s, /status alongside. Two runs:
| run 1 | run 2 | |
|---|---|---|
| text request alone (baseline) | 0.90 s | 0.82 s |
| the same text request, sent 0.6–1.2 s after the client left | 2.92 s | 3.50 s |
| blocked by | +2.02 s | +2.68 s |
strata-vision CPU after the client was gone |
41.6 cpu·s | 41.8 cpu·s |
| engine CPU during the same window | 0.0 cpu·s | 0.0 cpu·s |
/status during the window |
busy: false, queued: 1 |
busy: false, queued: 1 |
The encoder is still working ~3.4 s after the client disappeared (8 threads, ~5 s wall for this
test picture), the engine does nothing at all during that time, and the next request waits for it.
A real screenshot costs ~11 s of CPU encode on this machine; #1404 measured 15.3 s for a
1024×1024 picture on a 9700X — so the idle engine window is that long, per new picture.
Two runs, one model, one picture size; the absolute numbers are this machine's.
Why the FIFO part is worth separating
The comment in prepare() justifies the turn by GPU contention. With "vision": {"gpu": false}
the encoder is a separate process that never touches the GPU (the docs and #1404 recommend this
setup precisely to keep VRAM for the expert cache) — yet the turn is held for the whole encode,
and the engine goes idle. On a CPU encoder that looks like pure loss.
Suggested minimal fix
- Build
cancelbeforesvc.prepare(...)at all three call sites and pass it down:
prepare(..., cancel),_note_unreadable_tool_images,Vision.encode_all/encode—
checkingcancel.is_set()between images and between the URL downloads (each download can
wait 60 s). A cancelled preparation ends the way the queued cancellation of serve: a request waiting for the single request turn sends no bytes, so clients with a stream idle timeout abort it (and the retry re-reads the prompt) #1619 does:
finish: "cancel", its own history/totals row, the request's embeddings file removed
(drop_embeddings()already covers that). - Optionally, and separately: when the encoder is CPU-only, do not take
self.fifofor the
encode at all (or take it only around the part that uses the GPU), so a picture does not stop
text requests for its whole encode.
Not tested
Strata version, GPU, OS
Strata 0.1.43 (engine 0.1.43 per GET /v1/status) · 2× RTX 5070 Ti 16 GB · Ubuntu 24.04.5
Activity
serve: a client that leaves while its pictures are downloaded, waitin…
Niko1221 commentedon Oct 12, 2026
Your reading of the code was right, and the measurements made it easy to check. The cancel event and the disconnect watcher now exist before
prepare()on all three routes (chat, messages, responses) and are passed into it. A client that leaves now stops the work at the next point it can: before each image URL download, while waiting for the engine's turn (the wait polls instead of blocking on the lock, so a cancelled request gives its place up as #1619 does for queued generations), and between pictures of a multi-image request. The same applies to the tool-result pictures and to the re-prepare after an MCP tool round. A cancelled preparation ends quietly as a disconnect, with the turn released and the embeddings file dropped.Two limits, stated plainly. A single encode already running in the vision process still finishes (it is one blocking request to that process), so a one-picture request that is abandoned mid-encode keeps the encoder busy until that picture is done; the next picture and the generation are no longer started. And your second suggestion (not taking the engine's turn for a CPU-only encoder) is not done: it changes the ordering guarantee the comment in
prepare()documents, and it needs a measurement on a CPU-encoder setup first. I will keep this open for that part.Tarisper commentedon Oct 12, 2026
Thank you for responding. Your app is great!
The CPU-encoder measurement you asked for, on the same box and setup as the report.
Setup. Ryzen 7 5700X (8 cores / 16 threads), 2× RTX 5070 Ti 16 GB with
--layer-split 6(one card sits in a chipset slot that runs at PCIe 3.0 ×2),--ple-io direct(PLE rows read from disk),--kv int8,--spec 4,--mtp,--vision,"vision": {"gpu": false, "threads": 8, "max_tokens": 1024}, one request at a time (no--batch). Build7695859a(0.1.43), i.e. without the #1999 fix - it does not matter here: the encoder was driven out-of-process withREADY/ENC <img> <out>exactly asservedrives it, so the generation and the encode really did run at the same time, which is what dropping the turn would allow.First: with
"gpu": falsethe encoder never touches the GPU at all.strata-visiondoes not appear innvidia-smi --query-compute-apps- no CUDA context, 0 VRAM, RSS 1.18 GiB. So in this configuration the lock is not buying "the encoder and the engine are never on the GPU at the same time"; it is buying an idle engine.What the turn costs here. One new 1280×800 picture (1000 image tokens, no warm-up) is 10.9 s in the vision process (10.85–10.99 s over four runs), and the engine is idle for all of it (
/statussaysbusy: false, queued: 1) with every other request waiting behind it.What running at the same time costs. The same
ENCround against a live generation (a 700 token answer, the request started 1 s into the encode). Samples that had the agent's own traffic in their window were discarded; control generations before and after each arm moved by +-2 %, so the deltas below are the contention and not drift.The contended resource is the CPU and its memory bandwidth, not the GPU: 8 encoder threads on an 8-core box take rows away from the engine's PLE reads and its routing, and the encode itself pays +19 % back. Prefill numbers are noisier run to run (+-20 %) than decode.
So the concurrency is not free at our configured 8 threads, but the turn is still the worse trade. A 700-token answer that arrives together with a new picture: 10.9 s of waiting + 6.2 s of generation = 17.1 s with the turn; 8.1 s without it at 8 encoder threads, 6.5 s at 4 threads - and the picture itself is only 1.8-9 s later than it would have been on its own.
What I would take from this. Gating the turn on the encoder holding no CUDA context (or on
"vision": {"gpu": false}) looks like a clear win on this class of machine. The encoder's thread budget decides how much the concurrent request pays: -23 % decode at 8 threads on 8 cores against -6 % at 4. If keeping the ordering guarantee for every configuration is worth more than that, then the cheaper thing is a line in the docs saying a CPU encoder should leave a couple of cores to the engine - at 2 threads the generation was not measurably affected at all, and the picture took 36.6 s.Merge upstream v0.1.43.1 (a0116b2) into gfx906
Merge upstream main (a0116b2, engine 0.1.43.1): server and bench fixes
merge upstream/main (v0.1.43.1 line) into feature/sampling-presets-11…
Merge upstream v0.1.44.1 into windows-coding
serve: 画像のダウンロード・順番待ち・エンコードの間にクライアントが去ったら準備をやめる（prepare の前に cancel を作る、…
serve: a client that leaves while its pictures are downloaded, waitin…
Niko1221 commentedon Oct 24, 2026
This is fixed in Strata 0.2.0-beta. Please update to the latest release (in the Terminal:
strata update; in the app: download the newest installer from the releases page). If it still happens on the latest version, reply here and we will reopen it. Thank you for the report.