Repository navigation
Feature request: allow client to stop reasoning in realtime #23944
Description
Currently, inference request is one-way: client send a request and receives an SSE token stream. This has a limitation that the client cannot directly "control" the inference in real time, like for example in this proposed use case, skip the reasoning.
So, the first requirement is to allow a bi-directional communication between server <--> client; 2 possible methods (to be discussed):
- Websocket API: httplib already support it out of the box, so it won't be too difficult. The main question is: do we want to use our own API schema (maybe just slap the same /chat/completions schema), or going with a full OAI-compat schema
- Keep SSE stream but introduce a new API, for example
/chat/completions/controlthat can control the current completion session via an ID
The second requirement for this feature is that chat.cpp (especially the reasoning budget) need to be extended to allow arbitrary triggering "end-of-reasoning" event, though I'm not quite sure yet how to do that. Tagging @aldehir @pwilkin for discussion
CC @ggml-org/llama-server @ggml-org/llama-ui
Note for contributors / vibe-coders: do not push PR for this yet until we agree on the specs, otherwise your PR will be closed without any questions.
Activity
allozaur commentedon May 31, 2026
We were considering the WebSocket either way to handle the server-side streaming persistency at some point anywas, or am I wrong @ServeurpersoCom?
ServeurpersoCom commentedon May 31, 2026
Yeah, you're right 😅 I started a full-WebSocket PoC to supersede the SSE replay buffer (#23226) and move llama-ui to WebSocket-only, but it turned into an over-engineered mess, mostly because I haven't put enough time into it yet.
ServeurpersoCom commentedon May 31, 2026
ngxson said it all: SSE interruption currently maps to the "Stop" action. If you want to perform an action during inference, you either send a regular HTTP request in parallel to the SSE stream, or use WebSocket, which is bidirectional by design.
aldehir commentedon May 31, 2026
Reasoning budget is a state machine, so it should be simple enough to expose a function in the sampler that you can use to force it to the
forcingstate.Some models will still continue reasoning if injecting the closing tag too early. This will impact parsing, as the continued reasoning will be part of the content. Might need to define a minimum number of reasoning tokens.
ServeurpersoCom commentedon May 31, 2026
Qwen3.6 did exactly that to me once on reasoning Low: it kept reasoning for a bit, then re-emitted a closing </think> inside the final response.
aldehir commentedon May 31, 2026
Personally I'm in favor of a control endpoint. That said, with the WebUI expanding its capabilities, I can definitely see the need for a custom API where most of the work is done on the server end (e.g. background inference).
A cool feature that would naturally extend from this is a steering capability where the user injects a message into the reasoning when it notices it's going the wrong direction. A pi extension could leverage a control endpoint to provide this functionality too. Out of scope for this, just thinking aloud for possible future enhancements.
ngxson commentedon May 31, 2026
After second thought, I think it's better to go with the SSE design with a separate control endpoint, as it will simplify the implementation.
I think websocket would be a nice-to-have, but quite low prio as I consider it more or less like a wrapper instead of something brand new. Maybe when we refactor things on HTTP layer side, we can expose a single "stream" API that take the same HTTP handler(s) from
server-context, then expose it as both websocket and SSE automatically. The idea is quite similar to the definition of "MCP transport"The more complicated part now will be to have a function to signal the
forcingreasoning budge state. I'm not very familiar with this part of code so IMO the best is that @aldehir can add this API and I will build on top of that, does this sound OK to you?ServeurpersoCom commentedon May 31, 2026
The one and only advantage of WebSocket is that inference stream isn't buffered on the very rare restrictive enterprise proxies. I've only seen one, my old job's, (but I made them a tunneling monkey patch for the ui ahaha)
ngxson commentedon May 31, 2026
ah yeah almost forgot, we should add the
X-Accel-Buffering: noto server too, wdyt @ServeurpersoCom ?ServeurpersoCom commentedon May 31, 2026
Yeah, X-Accel-Buffering: no is a good idea, free win for the nginx case.
It won't save the SSL-inspecting enterprise proxies though, since there the buffering is a side effect of the MITM (the proxy has to reassemble the decrypted stream to inspect it before re-encrypting with its own CA), not something a header can opt out of.
ServeurpersoCom commentedon May 31, 2026
(With WebSocket they're kind of stuck though: their gear often can't inspect it properly, and a ton of sites would break if they blocked it, plus it's meant for real-time data that's often not worth inspecting anyway except me, shoving everything through it lol).
aldehir commentedon May 31, 2026
Not too bad, #23949. See my caveat though:
ngxson commentedon May 31, 2026
I think for the first iteration, we can add a minimum limit on web ui level, that will grey out the "skip reasoning" button until a certain number of tokens received. Exposing a setting on UI to control this should be easy.
In general, I don't think we should impose a default limit in the backend, because this is already an advanced feature. We expect users to have custom code to utilize it, and with a custom code, they can manage the skip point themself
ngxson commentedon May 31, 2026
Also, the
/chat/completions/controlendpoint will be made to be extended in the future, maybe allow things like:So I expect it to have as little default values as possible (ideally, no default values at all)
ServeurpersoCom commentedon Jun 1, 2026
I am taking the liberty of doing a quick POC because I want to see how it can get a Qwen A3B MoE out of a loop of thinking!
ServeurpersoCom commentedon Jun 1, 2026
That's so cool :) It works better and it's simpler than I thought!
think-stop.mp4
ngxson commentedon Jun 1, 2026
nice, @ServeurpersoCom feel free to push a PR for it
IMO the skip button might be better to place right after the message, or as close to the reasoning block as possible, but maybe @allozaur thinks otherwise
ServeurpersoCom commentedon Jun 1, 2026
We'll see, Alek has plenty of UI ideas, Yes I PR this with a clean minimal Skeleton with the button to the right.
ServeurpersoCom commentedon Jun 1, 2026
git rebase upstream/master :
Thanks aldehir, you added the exact piece I was missing, I'll take your #23949 version :)