Repository navigation
Feature Request: Support for ERNIE-4.5-VL #15512
Description
Prerequisites
- I am running the latest code. Mention the version if possible as well.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
I would like https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT multimodal support.
Motivation
I couldn't find any discussion or issue for this, but that's the best open source model I could find for OCRing hand written Japanese and Chinese text that actually kind of works.
It's worse than OpenAIs recognition, but in my 3 test images I use to evaluate OCR capabilities of open source models it performed ok (ok is better than everything else I tested)
It's better than
- gemma
- qwen
- intern
- lfm2
- kimi
(I think I tested mimo, but I can't find my setup or results... so maybe mimo is ok too?)
...
and every other open model I could find.
Possible Implementation
https://ernie.baidu.com/blog/publication/ERNIE_Technical_Report.pdf
https://github.com/bigdavidone/ERNIE4_5
vllm-project/vllm#20220
Reacted by brath, TheBestX11, Sergei Kulik, Denis Zimin and Ercan Görgülü
Activity
[-]Feature Request:[/-][+]Feature Request: Support for ERNIE-4.5-VL[/+]pwilkin commentedon Aug 23, 2025
I'll try, but I might need help from some competent people (@CISC @ngxson) because the model has some pretty atypical tensor configurations (there are double expert layers basically, a big and small expert for each layer and there's a nontrivial mixing algorithm for them. I'll share my branch where I managed to get the easy part (mmproj extraction) done.
pwilkin commentedon Aug 23, 2025
Also, just FYI: vLLM does not support the visual component yet, for exactly the same reason: the implementation is very complex. As far as I know, the support for the visual component with those pesky tensors is only available on the PaddlePaddle native Ernie implementation.
Som-anon commentedon Aug 25, 2025
Should have taken a closer look at this.
Thank you very much for taking a stab at this
m1namuci commentedon Aug 27, 2025
Ernie4.5 VL Model vLLM now supported vllm-project/vllm#22514
github-actions commentedon Oct 11, 2025
This issue was closed because it has been inactive for 14 days since being marked as stale.
04RR commentedon Nov 13, 2025
same request, this is an incredible model worth supporting
github-actions commentedon Jan 30, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.