[quei4r]

Overview

server_chat_convert_anthropic_to_oai() silently drops image blocks when converting Anthropic tool_result content to OpenAI format. This breaks clients that expect tools to return images (e.g. Claude Code) — the model behaves as if no image was provided at all.

When tool_result contains image blocks, convert the content to OpenAI multimodal content parts (text + image array). Plain text results remain plain strings for maximum backward compatibility.

This change is safe because oaicompat_chat_params_parse() already uniformly handles image_url across all message roles — extracting media to out_files and replacing it with media_marker placeholders for the chat template.

Key Changes

In server_chat_convert_anthropic_to_oai():

• Detect images: Scan tool_result content blocks for those with type == "image".
• Pure text path (unchanged): If no images are found, concatenate all text blocks into a normal string — this preserves existing behavior for non-multimodal tool results.
• Multimodal path (new): If there are images, build a JSON array of content parts:
  • Text blocks → {"type": "text", "text": "..."}
  • Image blocks with source.type == "base64" → Build a data URI (data:<media_type>;base64,<data>) and emit {"type": "image_url", "image_url": {"url": "..."}}
  • Image blocks with source.type == "url" → Pass the URL directly as {"type": "image_url", "image_url": {"url": "..."}}

Testing

Test script

#!/bin/bash
# Reproduce: tool_result with image is silently dropped

IMG_BASE64=$(base64 -w 0 ./a.jpeg)

cat > /tmp/test_image_tool.json << EOF
{
  "model": "qwen3.6",
  "messages": [
    {"role": "user", "content": "What is in this image?"},
    {"role": "assistant", "content": [{"type": "tool_use", "id": "tool_1", "name": "read", "input": {"file": "a.jpeg"}}]},
    {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "tool_1", "content": [{"type": "text", "text": "File: a.jpeg"}, {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": "$IMG_BASE64"}}]}]}
  ],
  "max_tokens": 4096,
  "stream": false
}
EOF

curl -s -X POST http://localhost:8848/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: no-key" \
  -d @/tmp/test_image_tool.json | jq -r '.content[] | select(.type == "text") | .text'

Before

I've received the image file `a.jpeg`. Since I don't have direct visual access to view or analyze images in this text-based interface, could you please describe what's in the image?

After

This image shows a dense field of tall, bright yellow wildflowers, which appear to be goldenrod, dominating the foreground and middle ground. Below the flowers, there is lush green foliage.

In the background, you can see:
*   Left side: Dark green trees or bushes.
*   Right side: A tall, thin, grey utility pole or mast stretching up into the sky.
*   Sky: The sky is a pale, hazy white-blue, suggesting it might be a misty morning or a smoggy day.

Tested locally with Qwen3.6-27B Q8 + Claude Code read tool.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO — The core fix was authored by a human contributor. AI was used solely for code relocation assistance due to upstream refactoring (function moved from server-common.cpp to server-chat.cpp), and for grammar suggestions on the human-written PR description.

[ggml-gh-bot]

Hi @quei4r, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[ngxson]

can you add a test case in test_compat_anthropic.py ?

[quei4r]

can you add a test case in test_compat_anthropic.py ?

Okay.I want to understand the test framework properly before writing the test, so I'll add it tomorrow.

[quei4r]

can you add a test case in test_compat_anthropic.py ?

Test added and rebased to latest master. The new test_anthropic_tool_result_with_image verifies image blocks inside tool_result are converted to OpenAI image_url format.

[quei4r]

Waiting for review. Thanks!

[ServeurpersoCom]

Cool fix, I think std::ostringstream is overkill here, simple string concatenation would do. Also, the two-pass scan (detect images, then build) could be collapsed into a single pass building content_parts directly and deciding the final shape at the end.

[quei4r]

@ServeurpersoCom Thanks for the suggestions. Updated in the latest push: replaced std::ostringstream with string concatenation and collapsed the two-pass scan into a single pass. Let me know if there's anything else.

[ServeurpersoCom]

Nice, the single-pass version reads much better. LGTM.

[quei4r]

@ngxson Could you take a look when you have a moment. Let me know if you see any issues or have suggestions for improvement. Thanks.

[lc1234567484848]

你有空能看看吗？如果你发现任何问题或有改进建议，请告诉我。谢谢。

Hello, do you have time to resolve this issue: https://github.com/ggml-org/llama.cpp/issues/23524。
The main problem is that I can't get the model in Claude Code to automatically read images from local paths and get image descriptions; it gives incorrect descriptions. I can only get it to correctly recognize the image and give the correct description by passing the image file directly into the Claude Code window. However, with my local model deployed in Ollama, I can have it directly read a specific image from a local path and give me a description. For example, it can read the image E:/PythonCode/MMLLM/bbox.png and generate an image description.

[quei4r]

@lc1234567484848 If your issue is that Claude Code's read tool returns an image but the model keeps saying it doesn't see anything, that sounds like the same bug this PR fixes — the image block in tool_result gets silently dropped during conversion.You can try applying the changes from server-chat.cpp in this PR as a temporary workaround to see if it resolves your issue.

[lc1234567484848]

如果你的问题是 Claude Code 的读取工具能返回图像，但模型一直说没看到任何东西，那听起来就像这个 PR 修复的同一个漏洞——tool_result 里的图像块在转换过程中被悄无声地丢弃了。你可以试试把server-chat.cpp的更改应用到这个PR里，作为临时变通，看看是否能解决你的问题。

Thank you for your answer, I will give it a try.
I am now performing image analysis on local paths by having Claude Code's local model execute this code.

The model will automatically execute:
python e:/PythonCode/MMLLM/scripts/mm_infer.py "E:/Dataset/Diverse-Weather/YOLO格式/Daytime_Sunny/images/train/02982dfe-1dd674e5.jpg" "Please analyze: 1. Scene description (time, location, weather, and environment); 2. Positional characteristics and interrelationships of each object; 3. Occlusion relationships and recognition difficulty. Respond in Chinese." 2>&1

Of course, the ideal situation would still be to directly tell Claude code a local path to achieve what I want. My approach is just a polite, roundabout solution.

"""Multimodal inference via local llama-server API.

Usage:
    python mm_infer.py <image_path> <prompt>
"""

import argparse
import base64
import sys

import requests


SERVER_URL = "http://127.0.0.1:8001/v1/chat/completions"
MODEL = "Qwen3.6-27B-UD-Q4_K_XL.gguf"


def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode()


def infer(image_path: str, prompt: str) -> dict:
    b64 = encode_image(image_path)

    payload = {
        "model": MODEL,
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}},
                    {"type": "text", "text": prompt},
                ],
            }
        ],
    }

    resp = requests.post(SERVER_URL, json=payload, timeout=300)
    resp.raise_for_status()
    return resp.json()


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("image_path", help="Path to image file")
    parser.add_argument("prompt", help="Text prompt")

    args = parser.parse_args()

    result = infer(args.image_path, args.prompt)
    choice = result["choices"][0]

    # Print reasoning if present
    if choice["message"].get("reasoning_content"):
        sys.stdout.buffer.write(b"[Reasoning]\n")
        sys.stdout.buffer.write(choice["message"]["reasoning_content"].encode("utf-8"))
        sys.stdout.buffer.write(b"\n\n")

    sys.stdout.buffer.write(b"[Content]\n")
    sys.stdout.buffer.write(choice["message"]["content"].encode("utf-8"))
    sys.stdout.buffer.write(b"\n")

    # Print usage to stderr for debugging
    usage = result.get("usage", {})
    print(f"\n[Usage] prompt={usage.get('prompt_tokens')} output={usage.get('completion_tokens')} total={usage.get('total_tokens')}", file=sys.stderr)


if __name__ == "__main__":
    main()

[Mimocro]

It's messed up if model sends 2 or more image reads in 1 tool call. Order is unpredictable
Damn, it's kinda silly that openai api can't accept images in tools by definition, and everything is build around that api, and you need dig up things to even know that its silently dropped

[constantinpricope2013]

Hi,

Just encounter the same issue. This will be nice to be merged.

It looks like just one more approve needed @ngxson

Thanks!

[seryogakovalyov]

Confirmed this fix independently on a different model/stack: Qwen3.6-35B-A3B-Unsloth (Unsloth quant) + mmproj, built from b9957 with this patch applied, tested via the exact Claude Code Read-tool → tool_result flow this PR targets.

Before: input_tokens: 121, model responds as if no image was provided (fabricates content based on the filename).
After: input_tokens: 761, model correctly describes the actual screenshot content in detail.

Also related to #20663, which reports the same symptom. Would be great to get this merged — @ngxson, is there anything blocking beyond a final review pass?