Repository navigation
Dialect scope: chat completions only, or also reasoning_content / Responses API / Anthropic? #56
Description
Activity
- added a parent issue
on Sep 1, 2026 Phone-friendly options (reply with a letter):
- A (recommended): chat completions only for v1. The gateway's translated route covers
choices[].delta.content; other dialects pass through unguarded (documented limitation). - B: + reasoning_content. Also accumulate
choices[].delta.reasoning_contentinto a second detection stream. Cheap; catches thinking-mode loops (the NVIDIA/Kimi K2.6 class of bug from the prior-art survey). - C: + Responses API + Anthropic dialects. Full coverage; each dialect needs its own event parser and rewrite path — roughly 3x the prototype's parsing code.
Grill note: vLLM/oh-my-pi bugs in the wild were mostly reasoning-stream loops, so B is the sweet spot if you want v1 to catch the common case without tripling scope.
- A (recommended): chat completions only for v1. The gateway's translated route covers
WE should built IT into the global requests handler If possible somehow so we can achive ITS simply (kiss) active under all endpoints even when there are new coming in the future ?
Chokepoint inventory (from code, not guesswork). Model SSE leaves the gateway through exactly two families:
- Translated routes — chat completions, Responses API, Anthropic messages. All three already funnel through
StreamResult.WrapDeliveryStream(translated_inference_service.go:149,:329,messages_handler.go:223). The guard is already active here via the wiring we shipped. - Passthrough routes (
/p/{provider}/...) —passthrough_support.gostreams provider SSE directly to the client (flush at :359/:371), bypassesWrapDeliveryStreamtoday. Guarding it means wrapping that response writer path too.
Non-model SSE (admin live tail, MCP) is out of scope.
Confirm: should v1 also wrap the passthrough family (small extra wrapper, same guard), or is translated-routes-only fine for v1 with passthrough documented as unguarded? Reply wrap or v1-translated-only.
- Translated routes — chat completions, Responses API, Anthropic messages. All three already funnel through
Status
Options A/B/C posted in a follow-up comment. Waiting on user reply.
Question
Question
The prototype only inspects choices[].delta.content (OpenAI chat-completions SSE). reasoning_content, the Responses API event envelope, and the Anthropic messages dialect pass through unguarded. Which dialects must the first shipped version cover?