Repository navigation
[DML] Olive generated adapters not working with OrtGenAi #1544
Description
I am using the method of creating adapters depicted here which I have got to work when using the CPU EP, however when using DML I get the following error when calling adapters.LoadAdapter:
Unhandled exception. System.Exception: D:\a\_work\1\s\onnxruntime\core\session\lora_adapters.cc:94 onnxruntime::lora::LoraAdapter::InitializeParamsValues Data transfer is not available for the specified device allocator, it also must not be a CPU allocator
I have tested the olive auto-opt call both with and without the --use_model_builder option but they both get the same result. I have also tried using the convert-adapters olive call instead but the resulting adapters do not work with CPU EP either (see aside).
If I run the model without the adapter on CPU EP it runs fine as well, whereas when I run the model without the adapter on DML I get the following error when calling AppendTokenSequences:
Unhandled exception. System.Exception: Non-zero status code returned while running DmlFusedNode_0_5 node. Name:'DmlFusedNode_0_5' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2839)\onnxruntime.dll!00007FFE495DF44C: (caller: 00007FFE495EEEC9) Exception(1) tid(2bb4) 80070057 The parameter is incorrect.
The same does not happen when using ORTGenAi's model_builder.py and passing in an adapter path, but then you cannot use multiple LoRA weights as it is tied into the onnx model permanently.
OS: Windows 11 x64
GPU: RTX 4090
API: C#
MODEL: Qwen/Qwen2.5-1.5B
(Aside) The adapters (when used via CPU EP) appear to have significant quality degradation. I can see that convert-adapters does lora scaling (alpha/rank) but I cannot find whether the auto-opt call is doing the same. Creating adapters via convert-adapters does not work with CPU EP either as the keys are not being renamed appropriately getting an invalid key/name/parameter error (.layers.0.self_attn. rather than .layers.0.attn.).
Activity
[-][DML] Olive generated adapters not working[/-][+][DML] Olive generated adapters not working with OrtGenAi[/+]jambayk commentedon Jan 21, 2025
Hi, thanks for opening this issue.
The lora workflow with DML has not been fully tested. From the olive side, we only verified the example with CPU and CUDA EP. I see that you opened a related issue on the onnxruntime-genai repo which I think is a good idea.
For running model without adapter on DML, only the model creating using model builder is supported (as far as I am aware).
With regards to the aside
--use_model_builderoption).Olive/olive/common/hf/peft.py
Line 65 in b0896bb
ambroser53 commentedon Feb 11, 2025
Thank you for your response. I realise now that it's not an issue with olive (I have managed to avoid the quality degradation mentioned above) nor onnxruntime genai but with onnxruntimes DML + Adapters API interoperability. It's not that it isn't tested it seems that they know it does work but just haven't said anything
busishengui commentedon Feb 26, 2025
Hi, I have the same question, I want to know how to solve the problem that
The adapters (when used via CPU EP) appear to have significant quality degradation.Thanks