flm serve: /v1/completions, /api/generate and /api/chat keep decoding after the client disconnects #760

Everything below comes from reading the source on main at 39ff855. None of it has been measured.

What looks wrong

The server creates a CancellationToken for every request. The token is cancelled when the client socket closes (server.cpp:357-388) or when /api/cancel is called. /v1/chat/completions uses it: it calls cancellation_token->reset() and passes [&] { return cancellation_token->cancelled(); } to both insert() and generate() (rest_handler.cpp:1194-1236, :1257-1288).

The other generation routes accept the token and never read it:

• /v1/completions: insert() and generate() at :1445, :1459, :1476, :1490
• /api/generate: :677, :691, :710, :724
• /api/chat: streaming at :793/:807. Non-streaming uses generate_with_prompt() (:835), which has no cancellation parameter at all (automodel.hpp:434).

So if a client drops mid-request on one of these routes, the server should keep decoding to max_tokens / num_predict (default 4096). For /api/generate and /api/chat, that also holds the NPU lock and blocks the queue behind it. /v1/completions isn't queued at all (see #759). /api/cancel has no effect on these routes for the same reason.

Expected

Cancelling on disconnect behaves the same on every generation route as it does on /v1/chat/completions.

Suggested fix

In each of these handlers, call cancellation_token->reset() and pass the same [&] { return cancellation_token->cancelled(); } predicate to insert() and generate(), as handle_openai_chat_completion does. Non-streaming /api/chat would need either a predicate on generate_with_prompt() or a switch to insert() + generate(). The second option changes output for Qwen3.5, Qwen3.6-MoE and GPT-OSS, whose generate_with_prompt() differs from that pair.

A probe that closes the connection mid-stream should confirm it: eval_count in the log stops short, and NPU Lock Released! follows promptly. We haven't run it yet.

Environment

Found by reading the source; checked against main at 39ff855 and v1.0.6.

Related, but a different resource: #680, where client aborts leak connection slots.