[Bug] Auto EP selection rejects the combination of DML EP with other EPs like OpenVINO EP

[mingmingtasd]

On my device with the Intel CPU, and both NVIDIA and Intel GPUs, if the DML EP and OpenVINO EP are both registered, PreferGpuEpPolicy::SelectProvidersForDevices will select three EP devices: DML GPU for NVIDIA, OpenVINO CPU for Intel and default ORT CPU during the session initialization. But the InferenceSession::HasInvalidCombinationOfExecutionProviders method rejects the combination of DML EP with OpenVINO EP, and reports error: "DML EP can be used with only CPU EP." which causes session initialization to fail.

1. Is it true that DML EP can only be used with the default CPU EP?
2. If so, why not reject the combination in PreferGpuEpPolicy::SelectProvidersForDevices earlier? It doesn't make sense an invalid combination can be created and then be rejected with an error.

[mingmingtasd]

Ningxin Hu (@huningxin) Dwayne Robinson (@fdwr) Scott McKay (@skottmckay) Any ideas? Thanks!

[fdwr]

I vaguely recall this PR #22657 that Pranav added, but I forget the context of why DML was considered mutually exclusive with other EP's (one would think ORT could shuffle memory between them via system RAM). Pranav Sharma (@pranavsharma) Do you recall? (edit: oops, forgot you're not at MS anymore)

I know a few years back (before the above PR), I was able to locally build onnxruntime.dll with both the DML EP and CUDA EP (did not try OpenVINO one) and switch between them at runtime (but not run both at the same time in the same session).

[skottmckay]

I'm guessing based on the linked PR that the update was involving a build with DML and CUDA enabled in it.

I believe it's invalid to have 2 GPU EPs competing because (AFAIK) the logic to insert nodes to copy between devices has no handling of copying between 2 non-cpu devices. It only does non-CPU device to CPU and vice-versa. But that's not a DML specific limitation.

To handle this specific situation I would suggest updating the DML check to allow DML EP + any other number of EPs that are CPU based.

Should be able to do that by adding this to IExecutionProvider:

const OrtDevice& GetDevice() const { return default_device_; }

and updating to this:

common::Status InferenceSession::HasInvalidCombinationOfExecutionProviders() const {
  // DML EP is only allowed with CPU EP
  bool has_dml_ep = execution_providers_.Get(kDmlExecutionProvider) != nullptr;
  if (has_dml_ep) {
    for (const auto& ep : execution_providers_) {
      if (ep->GetDevice().Type() != OrtDevice::CPU) {
        return common::Status(common::ONNXRUNTIME, common::INVALID_ARGUMENT, "DML EP can be used with only CPU EP.");
      }
    }
  }
  return Status::OK();
}

But ideally the DML EP folks make that change if it's valid given the code in ORT is DML specific.

[mingmingtasd]

To handle this specific situation I would suggest updating the DML check to allow DML EP + any other number of EPs that are CPU based.

Thanks for the explanation and suggestion! Scott McKay (@skottmckay)

Dwayne Robinson (@fdwr) Do you know which DML EP folk can help fix this issue? Could you help assign? Thanks!

[RafaelCintron]

Scott McKay (@skottmckay) , there are no longer any "DML EP folks", or real owners. Who would be the best person to fix this?

[mingmingtasd]

Thanks the fix by Scott McKay (@skottmckay)