Does not support GPT-OSS 20B A770 (Windows) ] Exception from src/inference/src/dev/plugin.cpp:111:

Model Server does not support The GPT-OSS 20B model.

Since September, I haven't been able to run gpt-oss 20b.
I always get the same error with the GPU plugin.
No matter which version I install, I always get the same result. On both Windows and Linux.

Windows 11
Driver 8331
A770 (x2)

I followed the instructions from here:
https://github.com/openvinotoolkit/model_server/tree/main/demos/code_local_assistant

(ovms) C:\openvino\model_server>docker run -d --rm -p 9000:9000 -p 8080:8080 ^
More? -v "C:\openvino\model_server\models:/models:rw" ^
More? --name ovms-server ^
More? openvino/model_server:weekly ^
More? --port 9000 ^
More? --rest_port 8080 ^
More? --config_path /models/config_all.json
714c6524842e0acdca69957dfa19fa5153de099aae1dfe915fbd60302c998df8

(ovms) C:\openvino\model_server>docker logs ovms-server
[2025-12-29 18:10:12.156][1][serving][info][server.cpp:88] OpenVINO Model Server 2026.0.0.9944c3235
[2025-12-29 18:10:12.156][1][serving][info][server.cpp:89] OpenVINO backend 2026.0.0.0.dev20251222
[2025-12-29 18:10:12.156][1][serving][info][pythoninterpretermodule.cpp:37] PythonInterpreterModule starting
[2025-12-29 18:10:12.317][1][serving][info][pythoninterpretermodule.cpp:50] PythonInterpreterModule started
[2025-12-29 18:10:12.656][1][modelmanager][info][modelmanager.cpp:156] Available devices for Open VINO: CPU
[2025-12-29 18:10:12.657][1][serving][info][capimodule.cpp:40] C-APIModule starting
[2025-12-29 18:10:12.657][1][serving][info][capimodule.cpp:42] C-APIModule started
[2025-12-29 18:10:12.658][1][serving][info][grpcservermodule.cpp:110] GRPCServerModule starting
[2025-12-29 18:10:12.658][1][serving][info][grpcservermodule.cpp:137] Binding gRPC server to address: 0.0.0.0:9000
[2025-12-29 18:10:12.663][1][serving][info][grpcservermodule.cpp:192] GRPCServerModule started
[2025-12-29 18:10:12.663][1][serving][info][grpcservermodule.cpp:193] Started gRPC server on port 9000
[2025-12-29 18:10:12.663][1][serving][info][httpservermodule.cpp:35] HTTPServerModule starting
[2025-12-29 18:10:12.663][1][serving][info][httpservermodule.cpp:39] Will start 16 REST workers
[2025-12-29 18:10:12.666][53][serving][info][drogon_http_server.cpp:137] Binding REST server to address: 0.0.0.0:8080
[2025-12-29 18:10:12.716][1][serving][info][drogon_http_server.cpp:164] REST server listening on port 8080 with 16 unary threads and 16 streaming threads
[2025-12-29 18:10:12.716][1][serving][info][http_server.cpp:248] API key not provided via --api_key_file or API_KEY environment variable. Authentication will be disabled.
[2025-12-29 18:10:12.717][1][serving][info][httpservermodule.cpp:52] HTTPServerModule started
[2025-12-29 18:10:12.717][1][serving][info][httpservermodule.cpp:53] Started REST server at 0.0.0.0:8080
[2025-12-29 18:10:12.717][1][serving][info][servablemanagermodule.cpp:51] ServableManagerModule starting
[2025-12-29 18:10:12.738][1][serving][info][mediapipegraphdefinition.cpp:423] MediapipeGraphDefinition initializing graph nodes
[2025-12-29 18:10:12.741][1][modelmanager][info][servable_initializer.cpp:420] Initializing Language Model Continuous Batching servable
[2025-12-29 18:10:14.442][1][serving][error][servable_initializer.cpp:214] Error during llm node initialization for models_path: /models/openai/gpt-oss-20b/./ exception: Exception from src/inference/src/cpp/core.cpp:116:
Exception from src/inference/src/dev/plugin.cpp:111:
Check '!m_device_map.empty()' failed at src/plugins/intel_gpu/src/plugin/plugin.cpp:516:
[GPU] Can't get PERFORMANCE_HINT property as no supported devices found or an error happened during devices query.
[GPU] Please check OpenVINO documentation for GPU drivers setup guide.




[2025-12-29 18:10:14.442][1][modelmanager][error][servable_initializer.cpp:425] Error during LLM node resources initialization: The LLM Node resource initialization failed
[2025-12-29 18:10:14.442][1][serving][error][mediapipegraphdefinition.cpp:474] Failed to process LLM node graph openai/gpt-oss-20b
[2025-12-29 18:10:14.442][1][modelmanager][info][pipelinedefinitionstatus.hpp:59] Mediapipe: openai/gpt-oss-20b state changed to: LOADING_PRECONDITION_FAILED after handling: ValidationFailedEvent:
[2025-12-29 18:10:14.442][104][modelmanager][info][modelmanager.cpp:1200] Started model manager thread
[2025-12-29 18:10:14.442][1][serving][info][servablemanagermodule.cpp:55] ServableManagerModule started
[2025-12-29 18:10:14.442][105][modelmanager][info][modelmanager.cpp:1219] Started cleaner thread

(ovms) C:\openvino\model_server>uv pip list
Using Python 3.12.12 environment at: C:\Users\uuk\miniconda3\envs\ovms
Package                   Version
------------------------- ----------------------
accelerate                1.11.0
attrs                     25.4.0
diffusers                 0.35.2
einops                    0.8.1
filelock                  3.20.1
fsspec                    2025.12.0
huggingface-hub           0.36.0
joblib                    1.5.3
jsonschema                4.25.1
jsonschema-specifications 2025.9.1
markdown-it-py            4.0.0
mdurl                     0.1.2
ml-dtypes                 0.5.4
mpmath                    1.3.0
networkx                  3.4.2
ninja                     1.13.0
nncf                      3.0.0.dev0+b8243e97
numpy                     2.2.6
onnx                      1.20.0
openvino                  2026.0.0.dev20251222
openvino-telemetry        2025.2.0
openvino-tokenizers       2026.0.0.0.dev20251222
optimum                   2.1.0.dev0
optimum-intel             1.27.0.dev0+25fcb63
optimum-onnx              0.1.0.dev0
pandas                    2.3.3
pillow                    12.0.0
pip                       25.3
pydot                     3.0.4
pyparsing                 3.3.1
pyyaml                    6.0.3
referencing               0.37.0
regex                     2025.11.3
rich                      14.2.0
rpds-py                   0.30.0
safetensors               0.7.0
scikit-learn              1.8.0
scipy                     1.17.0rc1
sentence-transformers     5.2.0
sentencepiece             0.2.1
setuptools                80.9.0
sympy                     1.14.0
tabulate                  0.9.0
threadpoolctl             3.6.0
timm                      1.0.22
tokenizers                0.21.4
torch                     2.9.1+cpu
torchvision               0.24.1+cpu
tqdm                      4.67.1
transformers              4.55.4
wheel                     0.45.1

CONDA INFO

(ovms) C:\openvino\model_server>conda info

     active environment : ovms
    active env location : C:\Users\uuk\miniconda3\envs\ovms
            shell level : 1
       user config file : C:\Users\uuk\.condarc
 populated config files : C:\Users\uuk\miniconda3\.condarc
                          C:\Users\uuk\.condarc
          conda version : 25.1.1
    conda-build version : not installed
         python version : 3.12.9.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=haswell
                          __conda=25.1.1=0
                          __win=10.0.22631=0
       base environment : C:\Users\uuk\miniconda3  (writable)
      conda av data dir : C:\Users\uuk\miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://software.repos.intel.com/python/conda/win-64
                          https://software.repos.intel.com/python/conda/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\uuk\miniconda3\pkgs
                          C:\Users\uuk\.conda\pkgs
                          C:\Users\uuk\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\uuk\miniconda3\envs
                          C:\Users\uuk\.conda\envs
                          C:\Users\uuk\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/25.1.1 requests/2.32.5 CPython/3.12.9 Windows/11 Windows/10.0.22631 solver/libmamba conda-libmamba-solver/25.1.1 libmambapy/2.0.5 aau/0.5.0 c/. s/. e/.
          administrator : False
             netrc file : None
           offline mode : False

Conversion log

python C:\openvino\model_server\demos\common\export_models\export_model.py text_generation --source_model openai/gpt-oss-20b --weight-format int4 --config_file_path models/config_all.json --model_repository_path models --target_device GPU --overwrite_models

STEPS

1

2

3

4

Thank you for reporting this issue. There is indeed a gap in the demo. It will be corrected shorty.
The root cause is in the default KV cache size 10GB which is too big. It can be fixed by:
-rerunning the model export command with --cache_size 0 (dynamic value)
python export_model.py text_generation --source_model openai/gpt-oss-20b --weight-format int4 --config_file_path models/config_all.json --model_repository_path models --target_device GPU.0 --cache_size 0

updating graph.pbtxt and setting cache_size parameter to 0 instead of 10.
running ovms with all runtime parameters instead of using config file:
ovms.exe --rest_port 8000 --source_model openai/gpt-oss-20b --model_repository_path models --tool_parser gptoss --reasoning_parser gptoss --target_device GPU.0 --task text_generation

Note. Using docker image, requires WSL and passing more parameters and using the image version with gpu drivers https://github.com/openvinotoolkit/model_server/blob/main/docs/accelerators.md#starting-model-server-with-intel-gpu

--target_device GPU.0 --cache_size 0

Thank you 🙏🔥
Does it make sense GPU or GPU.0 (or GPU.1)?
Does it support Hetero (Tensor/Pipeline_parallel) now (for Alchemist, A770)?
It is working now, but speed is not good.
~8-10 t/s vs 50-65 t/s in llama.cpp (vulkan)

@savvadesogle

@savvadesogle in your case, it make no difference which gpu you specify. It will be one device. When you skip the number, OV will choose the card.
I suppose your llama test is utilizing both cards.
Tensor parallelism is under development now in OpenVINO to have latency scalability with more cards.
My tests on a single B60 cards shows ~50tok/s.
@geunhwan are there some missing optimization for A770 cards?

dynamic cache size by default (#3889)

@dtrawins

@dtrawins Optimization is done for Xe2+(LNL/BMG/PTL) architecture only in OpenVINO 2025.4. On Xe(ARL-H, ACM(inc. A770)), model should be functional, but performance is slow.

@geunhwan

I suppose your llama test is utilizing both cards.

No, it's a single gpu.
And it was 63 t/s on the b7089 build (Vulkan).

Found 1 Vulkan device from screenshot

Optimization is done for Xe2+

@geunhwan are optimisations planned for Alchemist?
Or i should to drop all of my 4x A770 to trash?

No HuC for xe driver for dg2 (only in old i915)
Ipex-llm with llama.cpp (sycl) is dead..
vllm 0.8.3 is too old..
And now is openvino.. not for gpu from 2022..
😭
Good hardware will die with no software support 🤷

@savvadesogle

@savvadesogle Optimization for ACM is planned for the next OpenVINO release.

@savvadesogle Optimization for ACM is planned for the next OpenVINO release.

Could you please take a look at the optimizations for GLM-4.7 Flash too? (DG2, A770, Alchemist)
openvinotoolkit/openvino#33776

@dtrawins

Hello, Dariusz @dtrawins

Could you help me with the launch command? I have an A770 with 16GB of VRAM (4x).

Model: https://huggingface.co/OpenVINO/gpt-oss-20b-int4-ov

ovms.exe --source_model OpenVINO/gpt-oss-20b-int4-ov --model_repository_path . --rest_port 8000 --task text_generation --target_device GPU --cache_size 0

[2026-03-13 13:32:30.273][24560][serving][info][httpservermodule.cpp:53] Started REST server at 0.0.0.0:8000
[2026-03-13 13:32:30.274][24560][serving][info][servablemanagermodule.cpp:51] ServableManagerModule starting
[2026-03-13 13:32:30.276][24560][serving][info][mediapipegraphdefinition.cpp:423] MediapipeGraphDefinition initializing graph nodes
[2026-03-13 13:32:30.277][24560][modelmanager][info][servable_initializer.cpp:420] Initializing Language Model Continuous Batching servable

[INFO] Reading chat template from chat_template.jinja file in model directory.
[2026-03-13 13:32:52.779][24560][modelmanager][info][mediapipegraphdefinition.cpp:184] Mediapipe: OpenVINO/gpt-oss-20b-int4-ov inputs:
name: input; mapping: ; shape: (); precision: UNDEFINED; layout: ...
[2026-03-13 13:32:52.779][24560][modelmanager][info][mediapipegraphdefinition.cpp:185] Mediapipe: OpenVINO/gpt-oss-20b-int4-ov outputs:
name: output; mapping: ; shape: (); precision: UNDEFINED; layout: ...
[2026-03-13 13:32:52.779][24560][modelmanager][info][mediapipegraphdefinition.cpp:186] Mediapipe: OpenVINO/gpt-oss-20b-int4-ov kfs pass through: false
[2026-03-13 13:32:52.779][24560][modelmanager][info][pipelinedefinitionstatus.hpp:60] Mediapipe: OpenVINO/gpt-oss-20b-int4-ov state changed to: AVAILABLE after handling: ValidationPassedEvent:
[2026-03-13 13:32:52.779][20720][llm_executor][info][llm_executor.hpp:90] All requests: 0; Scheduled requests: 0;
[2026-03-13 13:32:52.779][24560][serving][info][servablemanagermodule.cpp:55] ServableManagerModule started
[2026-03-13 13:32:52.780][23896][modelmanager][info][modelmanager.cpp:1224] Started model manager thread
[2026-03-13 13:32:52.780][10000][modelmanager][info][modelmanager.cpp:1243] Started cleaner thread
[2026-03-13 13:33:08.450][20720][llm_executor][info][llm_executor.hpp:67] All requests: 1; Scheduled requests: 1; Cache usage 52.6%;
[2026-03-13 13:33:09.241][20720][llm_executor][info][llm_executor.hpp:67] All requests: 1; Scheduled requests: 1; Cache usage 57.9%;
[2026-03-13 13:33:10.216][20720][llm_executor][error][llm_executor.hpp:94] Error occurred in LLM executor: Exception from src\inference\src\cpp\infer_request.cpp:224:
Exception from src\plugins\intel_gpu\src\runtime\ocl\ocl_stream.cpp:376:
[GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES

Update

@geunhwan

based on the logs, I estimate the throughput at about 16tok/s. Assuming this is from A770, it is still low. @geunhwan I know there are in progress additional optimizations for Xe platforms. Do we know the expected results and timelines?
On PTL iGPU there is expected ~35tok/s and on B70 card about 135tok/s with that model.

@dtrawins

@dtrawins gpt-oss-20b on Xe platforms was formally enabled last week after your comment. Appreciate if you can test it with the latest openvino.

@geunhwan

@geunhwan There are still 16tok/s with OV version 0c81a247d3a and gpt-oss-20b on Arc(TM) A770 Graphics .

@dtrawins

Hello, @dtrawins @geunhwan

I believe the A770's performance should be on par with the B580.
A770 256 INT8 TOPS
B580 233 INT8 TOPS

A770 - llama.cpp (Vulkan)

A770 - llama.cpp (SYCL)

B580 - OpenArc (OpenVINO GenAI)

ggml-org/llama.cpp#22066 (comment)

Someone with an Intel B70 is also having performance issues.
openvinotoolkit/openvino#35199 (comment)

😭 I ask you and the Intel team, please optimize performance on Alchemists (not just for GPT-OSS)...

@dtrawins

Hello, @dtrawins
is there any news on performance optimization for Alchemists?

There should be observed improvements in the coming release 2026.2. It can be already validated in the latest code drop in docker image openvino/model_server:weekly
I captured the following results on A770 in gpt-oss-20b with 512 input tokens with a single client.
============ Serving Benchmark Result ============
Successful requests: 20
Failed requests: 0
Maximum request concurrency: 1
Benchmark duration (s): 102.22
Total input tokens: 10240
Total generated tokens: 4000
Request throughput (req/s): 0.20
Output token throughput (tok/s): 39.13
Peak output token throughput (tok/s): 41.00
Peak concurrent requests: 2.00
Total token throughput (tok/s): 139.31
---------------Time to First Token----------------
Mean TTFT (ms): 151.78
Median TTFT (ms): 99.83
P99 TTFT (ms): 938.07
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 24.92
Median TPOT (ms): 24.86
P99 TPOT (ms): 25.21

Can you check if you have reproduction of better results on your side?

@dtrawins

Can you check if you have reproduction of better results on your side?

@dtrawins Wooow! it's cool!

I'll definitely try it. Could you share a link to the documentation on how to run the benchmark (like in your answer)?

@dtrawins

HI, @dtrawins

Windows 11
Driver 8737

As you can see in the screenshot, copy operations are still running in Task Manager. And the GPU isn't being utilized at 100%.

current speed is about ~31 t/s

command:

./bin/ovms --rest_port 8000 --port 8001 --model_repository_path /mnt/c/ovms/models --source_model OpenVINO/gpt-oss-20b-int4-ov --tool_parser gptoss --reasoning_parser gptoss --target_device GPU.0 --task text_generation --cache_size 0 --log_level DEBUG

I tried to run a docker container on Windows, but for some reason it couldn't see the dev/dri in WSL ... So I extracted the executable files and ran them directly in Ubuntu WSL (24.04 ).

uuk@ook:~/ovms_weekly$ ./bin/ovms --version
OpenVINO Model Server 2026.2.0.5e9dcfc46
OpenVINO backend 2026.2.0-21902-90214e5be05-releases/2026/2
OpenVINO GenAI backend 2026.2.0.0-3121-adf73e80e66
Bazel build flags: --strip=always --config=mp_on_py_on --//:distro=ubuntu

FULL LOG (~5500 lines ):

--log_level DEBUG

(600kb)

I don't know, GitHub doesn't allow me to attach a text file for some reason.

https://gist.github.com/savvadesogle/89b10bf10853bdd535cdd5a2fd5df062

@dtrawins

@dtrawins Good News! 🎉 🥳

~ 46 t/s

Run	TTFT (s)	PP (tok/s)	TG (tok/s)	Total (s)	Input	Output
1	0.250	1873.3	46.1	3.026	469	128
2	0.267	1755.3	47.7	2.952	469	128
3	0.252	1861.0	47.4	2.952	469	128
4	0.236	1989.6	47.1	2.956	469	128
5	0.240	1955.8	47.1	2.960	469	128
Avg	0.249	1887.0	47.1	2.969	469	128

Ubuntu 25.10 (7.0.6-070006-generic)
Driver XE
latest Compute Runtime https://github.com/intel/compute-runtime/releases/tag/26.18.38308.1

Command

docker run -d --rm -p 8000:8000 -p 8001:8001 \
  -v ~/models:/models \
  --device /dev/dri \
  --group-add 993 \
  --privileged \
  openvino/model_server:weekly \
  --rest_port 8000 --port 8001 \
  --model_repository_path /models \
  --source_model OpenVINO/gpt-oss-20b-int4-ov \
  --tool_parser gptoss --reasoning_parser gptoss \
  --target_device GPU.0 \
  --task text_generation --cache_size 0

~94% utilization and 103W

sudo xpu-smi config -d 0 -t 0 --frequencyrange 2400,2400

--request-rate 1 --max-concurrency 1

============ Serving Benchmark Result ============
Successful requests:                     200       
Benchmark duration (s):                  664.36    
Total input tokens:                      101278    
Total generated tokens:                  25600     
Request throughput (req/s):              0.30      
Output token throughput (tok/s):         38.53     
Total Token throughput (tok/s):          190.98    
---------------Time to First Token----------------
Mean TTFT (ms):                          344.68    
Median TTFT (ms):                        347.87    
P99 TTFT (ms):                           354.94    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          23.43     
Median TPOT (ms):                        23.45     
P99 TPOT (ms):                           23.93     
---------------Inter-token Latency----------------
Mean ITL (ms):                           24.24     
Median ITL (ms):                         22.99     
P99 ITL (ms):                            28.31     
==================================================

python ~/ovms/benchmark/benchmark_serving.py \
  --backend openai-chat \
  --host localhost --port 8000 \
  --endpoint /v3/chat/completions \
  --model OpenVINO/gpt-oss-20b-int4-ov \
  --dataset-name sonnet \
  --dataset-path ~/ovms/benchmark/sonnet.txt \
  --sonnet-input-len 512 \
  --sonnet-output-len 128 \
  --num-prompts 200 \
  --request-rate 1 \
  --max-concurrency 1

--request-rate inf --max-concurrency 4

============ Serving Benchmark Result ============
Successful requests:                     200       
Benchmark duration (s):                  466.61    
Total input tokens:                      101278    
Total generated tokens:                  25600     
Request throughput (req/s):              0.43      
Output token throughput (tok/s):         54.86     
Total Token throughput (tok/s):          271.91    
---------------Time to First Token----------------
Mean TTFT (ms):                          647.89    
Median TTFT (ms):                        662.58    
P99 TTFT (ms):                           910.27    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          68.35     
Median TPOT (ms):                        68.43     
P99 TPOT (ms):                           70.94     
---------------Inter-token Latency----------------
Mean ITL (ms):                           70.59     
Median ITL (ms):                         64.31     
P99 ITL (ms):                            236.93    
==================================================

python ~/ovms/benchmark/benchmark_serving.py \
  --backend openai-chat \
  --host localhost --port 8000 \
  --endpoint /v3/chat/completions \
  --model OpenVINO/gpt-oss-20b-int4-ov \
  --dataset-name sonnet \
  --dataset-path ~/ovms/benchmark/sonnet.txt \
  --sonnet-input-len 512 \
  --sonnet-output-len 128 \
  --num-prompts 200 \
  --request-rate inf \
  --max-concurrency 4

Concurrency Results (50 prompts each)

Concurrency	Duration (s)	Output tok/s	Mean TTFT (ms)	Mean TPOT (ms)	Mean ITL (ms)
1	164.38	38.93	340.52	23.19	23.98
2	130.50	49.04	439.34	37.61	38.85
4	115.04	55.63	634.52	65.92	68.06
6	115.04	55.63	663.28	100.52	103.85
8	52.96	120.83	761.35	56.24	58.10
10	45.10	141.91	934.22	62.87	65.02
12	45.12	141.83	1085.71	69.18	71.64
16	40.89	156.50	1430.07	81.42	84.09
20	37.93	168.75	1886.52	93.50	96.70

dynamic cache size by default (#3889)

Navigation Menu

Repository navigation

Does not support GPT-OSS 20B A770 (Windows) ] Exception from src/inference/src/dev/plugin.cpp:111: #3888

CONDA INFO

STEPS

1

2

3

4

A770 - llama.cpp (Vulkan)

A770 - llama.cpp (SYCL)

B580 - OpenArc (OpenVINO GenAI)

current speed is about ~31 t/s

FULL LOG (~5500 lines ):

~ 46 t/s

Command

~94% utilization and 103W

--request-rate 1 --max-concurrency 1

--request-rate inf --max-concurrency 4

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Does not support GPT-OSS 20B A770 (Windows) ] Exception from src/inference/src/dev/plugin.cpp:111: #3888

Description

CONDA INFO

Activity

savvadesogle commented on Dec 29, 2025

savvadesogle commented on Dec 29, 2025

STEPS

1

2

3

4

dtrawins commented on Dec 29, 2025

savvadesogle commented on Dec 29, 2025

dtrawins commented on Dec 30, 2025

geunhwan commented on Dec 30, 2025

savvadesogle commented on Dec 30, 2025

geunhwan commented on Dec 31, 2025

savvadesogle commented on Jan 23, 2026

savvadesogle commented on Mar 13, 2026

savvadesogle commented on Apr 8, 2026

dtrawins commented on Apr 13, 2026

geunhwan commented on Apr 21, 2026

dtrawins commented on Apr 24, 2026

savvadesogle commented on Apr 26, 2026

A770 - llama.cpp (Vulkan)

A770 - llama.cpp (SYCL)

B580 - OpenArc (OpenVINO GenAI)

savvadesogle commented on May 25, 2026

dtrawins commented on May 26, 2026

savvadesogle commented on May 26, 2026

savvadesogle commented on May 26, 2026

current speed is about ~31 t/s

FULL LOG (~5500 lines ):

savvadesogle commented on May 27, 2026

~ 46 t/s

Command

~94% utilization and 103W

savvadesogle commented on May 27, 2026

--request-rate 1 --max-concurrency 1

--request-rate inf --max-concurrency 4

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions