Repository navigation
Python Mode insted of JSON mode. #1244
Description
Is your feature request related to a problem? Please describe.
LLMs were trained with a much larger Python code corpus of text and less in JSON, therefore the quality is better with Python.
Describe the solution you'd like
Here is how it works instead of asking LLM to produce a valid JSON response, you tell LLM to make Python code with the response, and then you need to execute this Python code and extract your results.
Describe alternatives you've considered
JSON-Mode
Additional context
This request is inspired by HuggingFace SmolAgents.
For this, you'll need a safe Python execution environment such as https://github.com/e2b-dev/E2B
JSON vs. Python benchmarking
Subjectively it looks like JSON fluctuates around 88% accuracy, while Python is around 91%.
JSON Benchmarks: Tool Use (BFCL), MMLU (including tool use and function calling), MT-Bench
Python Benchmarks: Code (HumanEval-python), HumanEval+, HumanEval-Pro, LiveCodeBench (Pass®1-COT), Codeforces (Percentile/Rating), SWE Verified (Resolved), Aider-Polyglot (Acc.)
https://blog.langchain.dev/extraction-benchmarking/
https://www.promptlayer.com/blog/llm-benchmarks-a-comprehensive-guide-to-ai-model-evaluation
https://llm-stats.com/
https://paperswithcode.com/sota/code-generation-on-mbpp
https://paperswithcode.com/sota/code-generation-on-humaneval
https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard
https://anonymous.4open.science/r/PythonSaga/README.md
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/
https://huggingface.co/spaces/lmarena-ai/chatbot-arena-leaderboard
https://github.com/OpenBMB/ToolBench?tab=readme-ov-file#-model-experiments-results
https://gorilla.cs.berkeley.edu/leaderboard.html#leaderboard
https://answers111.github.io/evalpro.github.io/leaderboard.html
https://evalplus.github.io/repoqa.html
https://evalplus.github.io/repoqa.html
https://evalplus.github.io/evalperf.html
https://github.com/svilupp/Julia-LLM-Leaderboard?tab=readme-ov-file#julia-llm-leaderboard
https://huggingface.co/spaces/qiantong-xu/toolbench-leaderboard
https://scale.com/leaderboard/coding
https://huggingface.co/spaces/opencompass/opencompass-llm-leaderboard
https://aider.chat/docs/leaderboards/
https://www.swebench.com/#verified
https://codeforces.com/profile/Leaderboard?graphType=all
https://livecodebench.github.io/leaderboard.html
https://www.llmcodearena.com/top-models
Activity
qdrddr commentedon Mar 2, 2025
Example of Google Gemini generating python (instead of JSON) to manipulate a game
https://github.com/lmgame-org/GamingAgent
qdrddr commentedon Mar 26, 2025
Manus/CodeAct is using Python insteasd of JSON which enables solving more complex tasks in less steps
https://github.com/langchain-ai/langgraph-codeact
chilijung commentedon Mar 27, 2025
@qdrddr, thanks for sharing! We are also experimenting with some code agents in Wren AI.
qdrddr commentedon Mar 31, 2025
Experiments were conducted across 17 LLMs, comparing CodeAct vs JSON and text-based action mechanisms. Key findings include:
Higher success rates: CodeAct achieves up to 20% improvement in complex task completion.
Fewer required actions: CodeAct performs tasks with up to 30% fewer interactions, improving efficiency.
Enhanced tool usage: CodeAct seamlessly integrates existing Python libraries, expanding the agent's capabilities.
https://arxiv.org/pdf/2402.01030
qdrddr commentedon Oct 11, 2025
Cloudflare Turns MCP Tools into TypeScript APIs
Cloudflare proposes converting MCP tools into TypeScript APIs so that LLMs can generate code using them.
They aim to address two key issues: managing many tools and chaining multiple calls efficiently while reducing token usage.
🔧 Improves handling of many complex tools
💡 Enables multi-step automation with fewer tokens
https://blog.cloudflare.com/code-mode/
fix(core-py): avoid to generate duplicate models after extracting man…