[Fino-wind]

The Responses API's function_call_output only accepts input_text, so returning an image from a tool gives a 400:

400 {"error":{"message":"Output of tool call should be 'Input text'","type":"invalid_request_error"}}

This breaks Codex when it uses the vision capability of a local model - every screenshot step fails. Input messages already accept input_image a few lines above in the same file, so it is only the tool output path that is missing it.

The fix is to convert input_image to image_url and keep it inside the tool message.

This works because oaicompat_chat_params_parse() handles image_url the same way for every message role - it extracts the media into out_files and replaces it with a media marker, so the chat template never sees the media itself. This is the same approach as #22536, which already does this for the Anthropic -> OpenAI path.

Testing

Local CUDA build, Qwen3.8-27B-Q8_0 + mmproj-model-bf16, sending a real 50 KB JPEG screenshot as input_image inside function_call_output.

case	result
text + image in the tool output	200
image only, no text part	200
text-only tool output	200, unchanged
output as a plain string	200, unchanged
unsupported output type	400, should be 'Input text' or 'Input image'

In the two image cases the model describes what is actually in the picture ("a pixel-art game scene, rocky pillars, a signpost, a Poke Ball near the water"), so the image reaches the vision encoder rather than only passing validation.

The converted request keeps 3 messages (user / assistant / tool) with the image inside the tool message content array, no extra user message:

roles: ['user', 'assistant', 'tool']
tool message content: [ {"type":"text",...}, {"type":"image_url",...} ]

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: the code change and the test runs were produced with Claude Code under my direction, and I reviewed the change before submitting.

[ggml-gh-bot]

Hi @Fino-wind, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[inf9144]

I don't think the statement that chat templates cannot render media originating from a tool message holds in general.

We have a concrete counterexample in our Qwen3.8/Codex setup: input_image is kept as part of the tool result in llama.cpp's Responses -> Chat Completions representation, and we have tested that path end-to-end with vision.

In our Responses converter, input_image is normalized directly to image_url without changing the message role:

https://github.com/inf9144/llama.cpp/blob/responses-function-call-output/tools/server/server-chat.cpp#L46-L74

The resulting function_call_output remains role: "tool":

https://github.com/inf9144/llama.cpp/blob/responses-function-call-output/tools/server/server-chat.cpp#L224-L242

That is also what the current branch diff implements.

On the template side, the companion Qwen3.8/Codex template accepts both image and image_url in its general content renderer and emits the normal vision tokens:

<|vision_start|><|image_pad|><|vision_end|>

https://github.com/inf9144/qwen-codex-local/blob/main/templates/llama-cpp-qwen3.8-codex.jinja#L176-L223

The important part is that:

render_content(message.content, true)

is executed for every message before role-specific handling:

https://github.com/inf9144/qwen-codex-local/blob/main/templates/llama-cpp-qwen3.8-codex.jinja#L359-L362

The template then handles role == "tool" and wraps the already-rendered content in <tool_response>...</tool_response>:

https://github.com/inf9144/qwen-codex-local/blob/main/templates/llama-cpp-qwen3.8-codex.jinja#L547-L567

There is an important distinction here: Qwen's template ultimately projects tool responses into its own user-side ChatML framing. So I am not claiming that the final serialized prompt literally contains a native tool role token.

The point is that this projection happens in the model-specific template, while the Responses -> Chat Completions conversion preserves the tool association of the content.

That separation seems architecturally cleaner to me.

The image can remain semantically where it originated: in the tool result. The protocol converter does not need to turn part of that result into an additional standalone user message merely because a particular model uses user-side framing for tool responses. The template already owns the model-specific decision of how role: "tool" maps onto the model's actual prompt format.

That is also the relevant difference compared with the approach in #27958:

• the converter there changes the intermediate message list by extracting the media from the tool result and creating a separate user message;
• our converter preserves the tool association;
• only afterwards does the model-specific template decide how that tool message should be serialized for Qwen.

To be clear, this does not show that all existing llama.cpp chat templates already support multimodal tool content. Some may not.

It does show that multimodal content originating from a tool result is technically workable without requiring the generic protocol conversion layer to rewrite that content into a separate user turn.

For that reason, I would prefer keeping input_image associated with the tool result during Responses -> Chat Completions conversion and keeping the resulting function_call_output as role: "tool". Templates that support multimodal tool results can then render and project that content into whatever role/framing their model requires.

In other words: the protocol converter should preserve the semantic message structure and tool association; the model-dependent projection belongs in the template.

[lee-b]

I don't think the statement that chat templates cannot render media originating from a tool message holds in general.

We have a concrete counterexample in our Qwen3.8/Codex setup: input_image is kept as part of the tool result in llama.cpp's Responses -> Chat Completions representation, and we have tested that path end-to-end with vision.

In this case I would say that the error message below is too general, and therefore misleading as to the source of the bug:

400 {"error":{"message":"Output of tool call should be 'Input text'","type":"invalid_request_error"}}

I believe it should say something like Valid tool call output with image data received, but the chat template for this model does not appear to support image output from tool calls..