fix(qwen): always send explicit enable_thinking in chat_template_kwargs - #109
fix(qwen): always send explicit enable_thinking in chat_template_kwargs#109dylan (LESdylan) wants to merge 1 commit into
Conversation
When enable_thinking=false the payload previously omitted chat_template_kwargs entirely, leaving thinking mode to the vLLM server / chat-template default — which varies across deployments and made runs hard to reproduce (microsoft#90). Send an explicit true/false always, matching the MiniMax backend's existing behavior. Ref microsoft#90
|
Thanks for this fix — the behavior is valuable and we would like to land it through your original PR so your contribution remains fully attributed. Current
We tested that combined resolution locally: the Qwen tests and the full suite pass. Suggested verification: pytest tests/test_qwen_backend.py -q
pytest tests/ -qThe CLA check is also still waiting for your response to the policy bot. Once rebased, CLA-complete, and green, this should be a good merge candidate. Please ping us after updating; we are happy to re-review promptly. |
|
Hi dylan (@LESdylan) — thanks again for this. Of the related open PRs, this is the closest merge candidate. The explicit false behavior is genuinely valuable, and we have already validated it locally in a combined resolution alongside #128. Two things are still needed:
Once those are in, please ping us and we will re-review promptly. |
|
Thanks dylan (@LESdylan) — you identified a real gap: after c31c50b the client had no supported way to send an explicit We can't land the unconditional emit as written, though, because omitting the field is itself a fix: The protocol really has three states, so #240 makes the setting three-state — Closing in favor of #240, which credits you in the PR. This branch is also CONFLICTING against current |
chat_template_kwargs is a vLLM/SGLang extension. OpenAI, Azure, and strict OpenAI-compatible gateways reject the unknown body field with HTTP 400, and non-Qwen vLLM models served with it can emit <think> output with no <answer> tag (acc=0.000). c31c50b fixed that by only emitting the field when thinking was enabled, which closed #28 but left no supported way to send an explicit enable_thinking: false -- the request in #90/#109. The protocol has three states, so make the setting three-state: server_default (default) -> omit chat_template_kwargs enabled -> send enable_thinking: true disabled -> send enable_thinking: false server_default keeps every existing deployment on exactly the bytes it sends today, so #28 stays fixed, while disabled gives #90 the explicit false it asks for. The legacy enable_thinking boolean keeps its historical wire meaning (true -> send true, false -> omit), so no config changes behavior; setting both keys to conflicting values raises rather than silently picking a winner. Unknown tokens raise too -- a typo must not silently flip a reproducibility control. Because server_default delegates a result-affecting choice to the server's chat template, the backend warns once per role when it is used, and the resolved per-role mode is recorded in the run's config.json under resolved_qwen_thinking_modes. Also settles the docs contradiction between "local vLLM endpoint" and "OpenAI-compatible": qwen_chat speaks the OpenAI protocol and reaches both self-hosted servers and hosted gateways, which is exactly why the wire policy cannot be inferred and must be explicit. Closes #90
Problem
skillopt/model/qwen_backend.pyonly attachedchat_template_kwargswhenenable_thinkingwas true:With
enable_thinking=falsethe key was omitted entirely, so whether Qwen thinking mode ran was decided by the vLLM server / chat-template default — which varies across deployments and makes results hard to reproduce. This is exactly point 3 of #90, where Zisu Huang (@Huangzisu) confirmed the reported results used client-sideenable_thinking=falseand that the code would be adjusted.Change
Always send an explicit boolean:
This matches the MiniMax backend, which already sends the flag unconditionally (
minimax_backend.py).Tests
Updated the two tests that pinned the omit-when-disabled behavior to assert an explicit
{"enable_thinking": False}instead. Full suite: 165 passed, 5 skipped.Ref #90