Repository navigation
Expose CoreML EP via OrtEpFactory - #31975
Conversation
Register an internal CoreMLEpFactory with the plugin-EP infrastructure so that, on supported Apple systems with discovered accelerator hardware, the CoreML EP appears in GetEpDevices(), can be added to a session with SessionOptionsAppendExecutionProvider_V2, and is eligible for EP selection policies, notably PREFER_NPU, which can now select the Apple Neural Engine on supported systems instead of silently falling back to the CPU EP. Fixes microsoft#29066 The factory registers with vendor ID OrtDevice::VendorIds::APPLE, matching the vendor ID reported by Apple device discovery, so device ordering during EP selection prefers CoreML for Apple hardware via the vendor-affinity rule. In builds that also register the internal WebGPU EP, including the official macOS packages, PREFER_GPU and MAX_PERFORMANCE now select CoreML instead of WebGPU when both EPs claim the same Apple GPU, because CoreML has matching vendor affinity. The factory advertises at most one NPU and one GPU from device discovery. It advertises no devices below Core ML 5, the EP's minimum supported version. The NPU additionally requires Core ML 6 (iOS 16 / macOS 13), because earlier Core ML versions have no compute-units mode that enables the Neural Engine without also enabling the GPU. MLComputeUnitsCPUAndNeuralEngine requires Core ML 6, so an NPU selection could not otherwise be honored. The CPU device is deliberately left to the CPU EP. CPU-selecting policies choose a single preferred EP for the CPU device, so advertising it would displace the ORT CPU EP and other CPU-based EPs. CoreML-on-CPU remains available through an explicit MLComputeUnits=CPUOnly option, and Core ML can use its internal CPU path for operations unsupported by the selected compute units. When MLComputeUnits is not specified, the factory derives it from the selected devices: CPUAndNeuralEngine for NPU, CPUAndGPU for GPU, and ALL for NPU+GPU. User options may narrow this set, but may not enable an accelerator that was not selected. Tests in onnxruntime/test/autoep/test_selection.cc cover factory registration, explicit GPU and NPU+GPU selection, inference, PREFER_NPU/PREFER_GPU and their policy aliases, compute-unit defaulting and conflict validation, version checks, device validation and selection limits, CPU-only narrowing, and transactional cleanup after partial device-creation failures.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
@microsoft-github-policy-service agree company="Parallels" |
|
Hi Edward Chen (@edgchen1), would you be the right person to review this PR, or could you suggest an appropriate reviewer? It exposes the CoreML EP via OrtEpFactory and enables device-based EP selection. Thanks! |
The CoreML EP is implemented as part of ONNX Runtime, not provided by Apple, so its factory now reports "Microsoft" and VendorIds::MICROSOFT, like the other internal EP factories (CPU, DML, and WebGPU). Discovered Apple hardware devices keep their "Apple" / VendorIds::APPLE identifiers. VendorIds::APPLE remains the named constant for the vendor ID reported by Apple platform device discovery. Because the factory vendor no longer matches the Apple hardware vendor, CoreML is no longer preferred based on vendor matching during device ordering. When another EP claims the same hardware device, ordering falls back to the EP name tie-breaker. The CoreMLVendorAffinityPrecedesEpNameOrdering test was deleted because it tested the vendor affinity behavior removed by this change. The remaining vendor assertions now expect Microsoft identifiers for the factory and Apple identifiers for the hardware device.
Rename kCoremlComputeUnits_* to kCoremlProviderOption_MLComputeUnits_*.
Call the native binding through the _pybind_state module.
There was a problem hiding this comment.
Pull request overview
Registers CoreML with device-based EP selection on supported Apple systems.
Changes:
- Adds an internal CoreML EP factory for NPU/GPU discovery and selection.
- Derives and validates
MLComputeUnitsfrom selected devices. - Adds Apple vendor IDs and comprehensive AutoEP tests.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
onnxruntime/test/python/onnxruntime_test_python.py |
Tests Apple vendor IDs. |
onnxruntime/test/autoep/test_selection.cc |
Tests CoreML discovery, selection, validation, and cleanup. |
onnxruntime/python/onnxruntime_inference_collection.py |
Exposes Apple vendor ID to Python. |
onnxruntime/core/session/plugin_ep/ep_library_internal.h |
Declares CoreML factory creation. |
onnxruntime/core/session/plugin_ep/ep_library_internal.cc |
Registers the CoreML factory. |
onnxruntime/core/session/plugin_ep/ep_factory_coreml.h |
Defines the internal factory interface. |
onnxruntime/core/session/plugin_ep/ep_factory_coreml.cc |
Implements discovery, validation, and provider creation. |
onnxruntime/core/providers/coreml/model/host_utils.h |
Defines the NPU-selection version requirement. |
onnxruntime/core/providers/coreml/coreml_options.cc |
Uses shared compute-unit constants. |
onnxruntime/core/platform/apple/device_discovery.cc |
Uses the shared Apple vendor ID. |
include/onnxruntime/core/providers/coreml/coreml_provider_factory.h |
Publishes compute-unit option constants. |
include/onnxruntime/core/framework/ortdevice.h |
Adds the Apple vendor ID. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
After the CoreML factory switched to the Microsoft vendor ID, OrtDevice::VendorIds::APPLE was no longer used. Apple device discovery keeps its local kApplePciVendorId constant (0x106B) as before.
The CoreML EP factory is an internal EP: there is no library boundary between it and ORT, so its entry points do not need to convert exceptions into OrtStatus.
The device claiming and option validation helpers of CoreMLEpFactory were public static members only so that unit tests could call them directly. Move them into an anonymous namespace in ep_factory_coreml.cc, fold OrtEpDevice creation and publication into GetSupportedDevices, and remove the tests that called the helpers directly.
Move the CoreML EP selection tests from test_selection.cc to a dedicated CoreML test file.
There was a problem hiding this comment.
🟡 Changes recommended
Core ML version-boundary behavior lacks deterministic test coverage.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 1
- Review effort level: Balanced
|
Edward Chen (@edgchen1), I'd like to discuss one more topic. It does not expand the scope of this PR. I tried to add a CPU device to the CoreML factory, but two existing tests then failed: 1. OrtEpLibrary.PluginEp_Custom_Op_Inference_With_Prefer_Cpu This test registers example_ep, which implements Custom_Mul, and uses PREFER_CPU. In this configuration, the CPU candidates were ordered as: CoreMLExecutionProvider -> example_ep -> CPUExecutionProvider The policy selected CoreML and the ORT CPU EP. Consequently, example_ep was not added to the session. CoreML could not execute Custom_Mul, and the ORT CPU EP could not execute it either. Session creation failed with “Could not find an implementation for Custom_Mul(1) node”. 2. ModelPackageTest.LoadModelPackageAndRunInference_PreferCpu This test creates a package containing CPU and NPU variants targeting example_ep, then uses PREFER_CPU. The policy again selected CoreML and the ORT CPU EP. GetVariantSelectionEpInfo captures only provider_list[0], and VariantSelector::SelectVariant considers only ep_infos[0]. Since the first EP was CoreML and the package had no CoreML variant, the test failed with “No suitable model variant found for the configured execution providers”. This is a package-variant selection failure and keeping more EPs in the policy result would not fix it on its own: the resolver must also consider later candidates. There is already a TODO for this in model_package_variant_selector.cc. ================================== As a possible solution, I would propose supporting a ranked chain of CPU EPs, which would give another CPU EP the opportunity to handle the nodes the first one does not support. For ordinary ONNX sessions, explicit registration already provides the relevant behavior: applications can append different EPs through separate AppendExecutionProvider_V2 calls, and graph partitioning considers them in registration order. Nodes not assigned to one EP remain available to later EPs, with the ORT CPU EP last. A policy could keep multiple eligible CPU EPs in ranked order instead of reducing them to one primary EP plus the ORT CPU EP. This would reuse existing graph-partitioning behavior. Model packages would need a corresponding extension: consider EPs in priority order and choose the first one with a compatible variant. I also measured CoreML CPU against the ORT CPU EP on an Apple M3 Pro using static, batch-1 models. These are latencies in milliseconds, each pair is two separate runs:
CoreML on the CPU is up to 4× faster than the ORT CPU EP here. I should also note that these numbers are for MLProgram, not the current NeuralNetwork default, which has known gaps in FP16 support. With NeuralNetwork and CPUOnly, both FP16 models ran entirely on the ORT CPU EP. What do you think? |
Description
This PR registers an internal CoreMLEpFactory with the plugin-EP infrastructure so that, on
supported Apple systems, the CoreML EP participates in device-based EP selection.
The CoreML EP now:
of silently falling back to the CPU EP.
The implementation follows the existing internal-factory pattern used by the CPU, DML, and
WebGPU EPs.
The factory reports "Microsoft" as its vendor name and OrtDevice::VendorIds::MICROSOFT as its
vendor ID, matching the other internal factories: the factory vendor identifies who provides the
EP implementation, not the hardware vendor. Discovered Apple hardware devices keep their existing
"Apple" vendor name and PCI vendor ID (pci_vendor_ids::kApple, 0x106B).
Fixes #29066.
The factory advertises at most one NPU and one GPU from Apple device discovery. Core ML selects
compute-unit classes rather than individual physical devices, so additional devices of the same
type would not be individually addressable and are intentionally ignored.
No devices are advertised below Core ML 5 (macOS 12 / iOS 15), the minimum version accepted by
the current CoreML EP.
The NPU additionally requires Core ML 6 (iOS 16 / macOS 13). Earlier versions do not provide
MLComputeUnitsCPUAndNeuralEngine, the only mode that enables the Neural Engine without also
enabling the GPU. Core ML 5 can still use the ANE through "ALL", but it cannot honor an NPU-only
device selection.
The CPU device is deliberately not advertised. CPU-selection policies select one primary CPU EP
and add the ORT CPU EP as a fallback when needed. If CoreML advertised the CPU, it would compete
with the ORT CPU EP and other CPU-based plugin EPs to become the primary CPU EP. Models that
depend on operators provided by an EP that was not selected could then fail. CoreML-on-CPU
remains available through MLComputeUnits=CPUOnly, and Core ML may use its internal CPU path for
operations unsupported by the selected compute units.
When the caller does not provide MLComputeUnits, the factory derives one value from the complete
device selection: CPUAndNeuralEngine for NPU, CPUAndGPU for GPU, and ALL for NPU + GPU. A
user-provided MLComputeUnits value may narrow which selected accelerators CoreML uses, or disable
them all with CPUOnly, but it may not enable an accelerator that was not selected.
The CoreML EP now implements IExecutionProvider::GetProviderOptions() so its effective options can be
checked. It calls CoreMLOptions::ToProviderOptions(), which reports these options using the public option
names defined in coreml_provider_factory.h.
PREFER_NPU, MAX_EFFICIENCY, and MIN_OVERALL_POWER can now select CoreML on systems where the
factory advertises the Neural Engine.
In builds that register both CoreML and WebGPU, including the official macOS packages, both EPs
advertise the same Apple GPU. Both factories report "Microsoft" as their vendor, which does not
match the Apple hardware vendor. Device ordering therefore falls back to the EP name tie-breaker,
where "CoreMLExecutionProvider" sorts before "WebGpuExecutionProvider". As a result, PREFER_GPU
and MAX_PERFORMANCE select CoreML instead of WebGPU. This ordering comes from the EP names rather
than an explicit preference for CoreML. CoreML uses MLComputeUnits=CPUAndGPU for that selection,
which excludes the Neural Engine. WebGPU remains available through explicit device selection.
Tests
This patch adds 12 CoreML AutoEP tests covering: