[Feature Request] Expose API to query supported copies for EPs

[bbernhar]

Describe the feature request

When integrating ORT via the interop API, we need to determine how to handle CPU read/write ops for tensors after session initialization. While host-accessible memory can be queried statically ahead of time (via EpDevice_MemoryInfo or SessionGetMemoryInfoForInputs checking for HOST_ACCESSIBLE), device-only tensors requires a memory transfer.

Currently, there is no public query to determine which copy directions an EP supports. The CanCopy capability seems to be internal to ORT.

Since we cannot statically query copy support, we are forced to probe the EP by attempting a CopyTensors operation at startup or fallback to a slower alternative (such as using a single identity-op graph to force outputs to the CPU EP).

We are requesting a structured, public query to check supported copy directions for a given EP. This would eliminate the need for trial-and-error probing or fallbacks.

Describe scenario use case

To route device-only tensors imported from the ORT interop API (since version 1.24), WebNN currently relies on a dummy CopyTensors call to test EP support. A public query API would eliminate this trial-and-error, allowing us to instantly configure either standard transfers or the fallback identity-op graph upfront.

[skottmckay]

The EP should support copy to/from CPU for the memory info it reports from the OrtEpDevice. Is there a specific example where that's not the case?

SessionGetMemoryInfoForInputs should return the device the input is assigned to, so I would have expected it to return the default memory info rather than a host accessible memory info. See analysis below (although that's looking at implementations in the ORT repo and I believe the latest QNN EP is external now).

Are you seeing something different and if so for which EP/s?

That said there may be use usage nuances for OV and QNN. I believe they provide a shared host accessible allocator (which can be used to allocate memory that is accessible by GPU/NPU) but register the EP with a default device of CPU.

---

For the input path, InferenceSession::GetInputOutputMemoryInfo pulls the device from node_info_vec.front().device, which was set by the allocation planner via allocation_planner.cc:

return p_provider->GetOrtDeviceByMemType(
    utils::IsInputOnCpu(...) ? OrtMemTypeCPUInput : OrtMemTypeDefault);

So an input location is only ever one of two things:

• OrtMemTypeCPUInput → the EP's CPU-input device
• OrtMemTypeDefault → the EP's default compute device

Checking every GetOrtDeviceByMemType override, HOST_ACCESSIBLE is only ever returned for OrtMemTypeCPUOutput, never for OrtMemTypeCPUInput or default:

EP	OrtMemTypeCPUInput	OrtMemTypeCPUOutput	default
Base	OrtDevice() (plain CPU)	OrtDevice()	default_device_
CUDA	OrtDevice()	GPU HOST_ACCESSIBLE	GPU DEFAULT
CANN	OrtDevice()	NPU HOST_ACCESSIBLE	NPU DEFAULT
MIGraphX	OrtDevice()	GPU HOST_ACCESSIBLE	GPU DEFAULT
NV TRT RTX	OrtDevice()	GPU HOST_ACCESSIBLE	GPU DEFAULT
QNN	default_device_	default_device_	default_device_

Since inputs resolve through OrtMemTypeCPUInput/OrtMemTypeDefault only, they land on either plain CPU or the EP's DEFAULT (GPU/NPU) device — never the pinned/HOST_ACCESSIBLE device.

Two caveats on the output side (not inputs)

HOST_ACCESSIBLE can show up when you query kOutput:

• CPU outputs of a GPU/NPU EP map to OrtMemTypeCPUOutput → HOST_ACCESSIBLE.
• The planner's output-consumer propagation (lines 742–743) can upgrade a CPU output to a consumer's HOST_ACCESSIBLE device.

[bbernhar]

To your question: OpenVINO EP.

It doesn't implement IDataTransfer, so attempting to call CopyTensors returns an ORT error.

Our current OV path happens to work because the imported tensors are HOST_ACCESSIBLE, but that seems to expose a gap: unless IDataTransfer is a required contract for EPs that produce device tensors, downstream interop APIs like WebNN can't safely assume CopyTensors is supported. That's why we currently have to probe it at startup.

[skottmckay]

AFAIK OV only provides the HOST_ACCESSIBLE allocator and registers with CPU as its default device, and it doesn't have a device memory allocator (as the default CPU allocator can be used).

Why is CopyTensors needed if the memory is host accessible?

[bbernhar]

That's true for HOST_ACCESSIBLE tensors. The issue is that we don't know whether a tensor is HOST_ACCESSIBLE until after the session is initialized, and the memory info query only solves that case; it does not address the device-only transfer case.

For device-only tensors, without a public capability query, we have to rely on EP-specific implementation details to determine the transfer path. That's not a stable contract for downstream interop APIs like WebNN.

[skottmckay]

Sorry I'm not quite understanding the gap.

The session needs to be initialized to know which device/s the inference inputs/outputs will be on as that's assigned during model partitioning based on the registered EP/s in the session and their capabilities.
If an EP has a device specific allocator, it MUST implement CopyTensors (otherwise ORT has no way to move data to/from the device).

Where does "use CopyTensors if EpDevice_MemoryInfo has a device specific allocator" not work as the condition?

[bbernhar]

If an EP has a device specific allocator, it MUST implement CopyTensors

Does that guarantee also apply to tensors created from externally imported memory via OrtInteropApi, or is it only for tensors allocated by the EP's allocator? I'm only using CopyTensors with externally imported memory.

[skottmckay]

AFAIK the interop API only deals with device memory not host-accessible, so I would expect the OrtMemoryInfo in the OrtValue that OrtInteropAPI::CreateTensorFromMemory returns to be the device memory info from the EP.

[bbernhar]

I don't think the statement that the interop API only deals with device memory is correct from the public API perspective.

We import a D3D12 heap via OrtInteropApi::ImportMemory and CreateTensorFromMemory, and the resulting OrtValue can report OrtDeviceMemoryType_HOST_ACCESSIBLE via GetTensorMemoryInfo.[^1]

Additionally, the EP implementations I've looked at appear to only validate the external handle type (e.g. a D3D12 heap), not whether the heap is CPU-visible or device-only.[^2] So a CPU-visible heap also appears to be a valid import.

[bbernhar]

Scott McKay (@skottmckay) Just a gentle ping. OrtInteropApi allows importing HOST_ACCESSIBLE heaps, so the rule "device allocator == CopyTensors" doesn't fully address externally imported memory. How should downstream consumers determine the correct transfer path? Otherwise, we'll continue probing CopyTensors at startup as a workaround.

[skottmckay]

Working through the OV example if I understand it correctly. Please correct me as needed. The ChatGPT links in your previous comment no longer seem to be valid so I don't have any additional context from them.

• you import memory using the InteropAPI and the OV EP implementation uses HOST_ACCESSIBLE for the created tensor
• the OV EP does not provide a non-CPU device allocator
  • in this case the EP must also have registered with CPU as its default device even if internally it uses CPU/GPU/NPU, so DEFAULT memory for the EP is CPU based.
• all ORT can do for that EP is provide either CPU and HOST_ACCESSIBLE memory
  • an EP specific CopyTensors is not required given the memory is host accessible and the EP registered with CPU as its default device so we're only ever using those two memory types.

ORT handles copying to/from the EP's default device for nodes assigned to the EP (via CopyTensors if that is a non-CPU device). Whilst internally OV might have more specialized memory types ORT doesn't have a way to directly use those currently afaik.

Is there a scenario where the EP implements CopyTensors for some arbitrary device that is not the EP's default device and not host accessible? If you got inferencing input on that arbitrary device the EP's CopyTensors implementation could copy to its default device (which is where ORT puts the input to the node) as part of the setup to run the inference. Doesn't hitting that copy defeat the purpose of importing memory via the interop API though as I thought that was primarily a performance optimization for zero-copy usage of existing memory?

[bbernhar]

Scott McKay (@skottmckay) Thanks for working through this.

Is there a scenario where the EP implements CopyTensors for some arbitrary device that is not the EP's default device and not host accessible?

From the perspective of WebNN, we can't distinguish whether an externally imported buffer corresponds to the EP's default device, another device, or HOST_ACCESSIBLE. The public API doesn't expose that information, so we need to determine the correct transfer path based on the API contract rather than EP-specific implementation details.

Doesn't hitting that copy defeat the purpose of importing memory via the interop API

Not necessarily. Zero-copy remains the path for inference. A copy is only needed if the application explicitly requests a CPU read or write of the tensor before or after inference.

If the intended public contract is that externally imported tensors either (1) are CPU/HOST_ACCESSIBLE or (2) are guaranteed to support CopyTensors whenever the EP exposes a non-CPU default device allocator, then that answers my question. I wasn't able to find that guarantee in the public API.

Could you please clarify whether that is the intended contract?

Corrected links:
[1] https://source.chromium.org/chromium/chromium/src/+/main:services/webnn/ort/context_impl_ort.cc;drc=beef08f2197803c2660f6be4ab2bd295f5ba04ad;l=613
[2]

onnxruntime/onnxruntime/core/providers/nv_tensorrt_rtx/nv_provider_factory.cc

Line 646 in 308f24c

static bool ORT_API_CALL CanImportMemoryImpl(

[skottmckay]

Doesn't hitting that copy defeat the purpose of importing memory via the interop API

Not necessarily. Zero-copy remains the path for inference. A copy is only needed if the application explicitly requests a CPU read or write of the tensor before or after inference.

The copy would be required to put the data on the EP's default device in that hypothetical case where an arbitrary device that was not the EP's default device was supported by the EP's CopyTensors implementation. Separate to any application copy.

If the intended public contract is that externally imported tensors either (1) are CPU/HOST_ACCESSIBLE or (2) are guaranteed to support CopyTensors whenever the EP exposes a non-CPU default device allocator, then that answers my question. I wasn't able to find that guarantee in the public API.

I believe that's correct. If the EP can import the memory it's saying it can use it, which requires the EP to provide the CopyTensors implementation for (2). that implementation is also required for ORT to use the EP in general so we can move data to/from its default device.