[EP ABI] Add weight pre-packing support to kernel-based plugin EPs by adrianlizarraga · Pull Request #26754 · microsoft/onnxruntime

Adrian Lizarraga (adrianlizarraga) · 2025-12-08T23:20:20Z

Description

Adds C APIs to support pre-packing of const weights for OrtKernelImpl implementations.
- APIs optionally support sharing of pre-packed weight data (for cpu-accessible memory).

Updates example kernel (Mul) to use new pre-packing API. Tested by existing unit test:

onnxruntime/onnxruntime/test/autoep/test_execution.cc

Lines 242 to 256 in 549d741

    
           TEST(OrtEpLibrary, KernelPluginEp_Inference) { 
        
             RegisteredEpDeviceUniquePtr example_kernel_ep; 
        
             ASSERT_NO_FATAL_FAILURE(Utils::RegisterAndGetExampleEp(*ort_env, Utils::example_ep_kernel_registry_info, 
        
                                                                    example_kernel_ep)); 
        
             Ort::ConstEpDevice plugin_ep_device(example_kernel_ep.get()); 
        
             // Create session with example kernel-based plugin EP 
        
             Ort::SessionOptions session_options; 
        
             session_options.AddConfigEntry(kOrtSessionOptionsDisableCPUEPFallback, "1");  // Fail if any node assigned to CPU EP. 
        
             std::unordered_map<std::string, std::string> ep_options; 
        
             session_options.AppendExecutionProvider_V2(*ort_env, {plugin_ep_device}, ep_options); 
        
             // This model has Squeeze, Mul, and Relu nodes. The example plugin EP supports all nodes using registered kernels. 
        
             Ort::Session session(*ort_env, ORT_TSTR("testdata/squeeze_mul_relu.onnx"), session_options);

Motivation and Context

The previous PR added the base APIs that support kernel-based plugin EPs. This PR adds an additional feature that was identified as necessary for the port of WebGPU EP.

…ft/onnxruntime into adrianl/plugin-ep-kernel-prepack

…ations

…) receives an unexpected call (return error instead of setting a flag to false)

Adrian Lizarraga (adrianlizarraga) added 3 commits December 8, 2025 15:15

[EP ABI] Add weight pre-packing support to kernel-based plugin EPs

4f3d004

Add comment about sharing of prepacked weights (cpu ep only)

e954021

Update Mul kernel to pre-pack input b

fb2998b

Adrian Lizarraga (adrianlizarraga) commented Dec 9, 2025

View reviewed changes

Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h Outdated

Adrian Lizarraga (adrianlizarraga) commented Dec 9, 2025

View reviewed changes

Comment thread onnxruntime/core/session/plugin_ep/ep_kernel_registration.cc Outdated

Comment thread onnxruntime/core/session/plugin_ep/ep_kernel_registration.cc

Apply suggestions from code review

5e64f79

Adrian Lizarraga (adrianlizarraga) marked this pull request as ready for review December 9, 2025 16:55

Adrian Lizarraga (adrianlizarraga) requested review from Yulong Wang (fs-eire), Scott McKay (skottmckay) and Dmitri Smirnov (yuslepukhin) December 9, 2025 16:56

Adrian Lizarraga (adrianlizarraga) commented Dec 9, 2025

View reviewed changes

Comment thread onnxruntime/core/session/plugin_ep/ep_kernel_registration.cc Outdated

Adrian Lizarraga (adrianlizarraga) added 2 commits December 9, 2025 09:53

Add comments regarding prepack allocator lifetime

9b1c6a2

Merge branch 'adrianl/plugin-ep-kernel-prepack' of github.com:microso…

c638a1a

…ft/onnxruntime into adrianl/plugin-ep-kernel-prepack

Yulong Wang (fs-eire) previously approved these changes Dec 9, 2025

View reviewed changes

Dmitri Smirnov (yuslepukhin) previously requested changes Dec 9, 2025

View reviewed changes

Adrian Lizarraga (adrianlizarraga) marked this pull request as draft December 11, 2025 00:51

Added support for sharing pre-packed weights for cpu-accessible alloc…

717ed4a

…ations

Adrian Lizarraga (adrianlizarraga) dismissed Yulong Wang (fs-eire)’s stale review via 717ed4a December 11, 2025 21:12

Adrian Lizarraga (adrianlizarraga) added 5 commits December 11, 2025 18:49

Define what should happen if OrtKernelImpl::SetSharedPrePackedWeight(…

bd8f6f0

…) receives an unexpected call (return error instead of setting a flag to false)

Merge branch 'main' into adrianl/plugin-ep-kernel-prepack

fc1fd16

Clean up some exception handling

8b3f56c

Refactor example kernel classes (no inheritance)

23503a1

Merge branch 'main' into adrianl/plugin-ep-kernel-prepack

7f37ffb

Adrian Lizarraga (adrianlizarraga) marked this pull request as ready for review December 17, 2025 18:06

Adrian Lizarraga (adrianlizarraga) added 2 commits December 17, 2025 10:20

Correct use of output param

26eca56

Add more edge-case handling for PrePack() call

7af257b

Adrian Lizarraga (adrianlizarraga) requested review from Edward Chen (edgchen1) and Dmitri Smirnov (yuslepukhin) December 17, 2025 22:39

Adrian Lizarraga (adrianlizarraga) commented Dec 17, 2025

View reviewed changes

Comment thread onnxruntime/core/session/plugin_ep/ep_kernel_registration.cc Outdated

Adrian Lizarraga (adrianlizarraga) commented Dec 17, 2025

View reviewed changes

Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h

This was referenced Mar 17, 2026

Bump Microsoft.ML.OnnxRuntime from 1.17.3 to 1.24.3 Magedeline/MaggyHelper-Code#9

Open

Bump Microsoft.ML.OnnxRuntime.Gpu from 1.23.2 to 1.24.4 yuniko-software/bge-m3-onnx#68

Closed

dependabot Bot mentioned this pull request Apr 1, 2026

deps(nuget): Bump the nuget-all group with 11 updates msbrettorg/maenifold#83

Closed

This was referenced Apr 25, 2026

Bump Microsoft.ML.OnnxRuntime from 1.22.1 to 1.25.0 IoTSharp/SonnetDB#50

Closed

Bump Microsoft.ML.OnnxRuntime.Gpu from 1.23.2 to 1.25.0 yuniko-software/bge-m3-onnx#71

Closed

deps(nuget): Bump the nuget-all group with 12 updates msbrettorg/maenifold#85

Closed

This was referenced May 11, 2026

Bump Microsoft.ML.OnnxRuntime.Gpu from 1.23.2 to 1.26.0 yuniko-software/bge-m3-onnx#74

Open

Bump Microsoft.ML.OnnxRuntime from 1.18.1 to 1.26.0 uav-simulator/uavsimulator#3

Open

dependabot Bot mentioned this pull request Jun 21, 2026

Bump Microsoft.ML.OnnxRuntime from 1.22.1 to 1.27.0 DevelApp-ai/MarkdownStructureChunker#32

Merged

This was referenced Jul 1, 2026

deps(nuget): Bump the nuget-all group with 12 updates msbrettorg/maenifold#88

Closed

Bump Microsoft.ML.OnnxRuntime from 1.22.0 to 1.27.0 elbruno/ElBruno.Speech#10

Closed

This was referenced Jul 8, 2026

Bump Microsoft.AspNetCore.SignalR.Client and 8 others one-ware/OneWare#272

Closed

Bump the microsoft group with 9 updates one-ware/OneWare#317

Closed

This was referenced Jul 22, 2026

Bump the microsoft group with 9 updates one-ware/OneWare#323

Closed

Bump Microsoft.AspNetCore.SignalR.Client and 8 others one-ware/OneWare#332

Closed

This was referenced Aug 1, 2026

deps(nuget): Bump the nuget-all group with 12 updates msbrettorg/maenifold#90

Closed

Bump Microsoft.ML.OnnxRuntime from 1.20.1 to 1.28.0 varro986/codename-aurora#16

Closed

This was referenced Aug 8, 2026

Bump Microsoft.ML.OnnxRuntime from 1.23.2 to 1.28.0 marcmy/videoduplicatefinder#68

Closed

chore(deps-nuget): Bump Microsoft.ML.OnnxRuntime from 1.17.3 to 1.29.0 spelech/model-context-gateway#90

Closed

This was referenced Aug 17, 2026

Bump Microsoft.ML.OnnxRuntime from 1.20.1 to 1.29.0 varro986/codename-aurora#17

Closed

deps(nuget): Bump Microsoft.ML.OnnxRuntime from 1.19.0 to 1.29.0 techbuzzz/ai-watermark-remover#11

Closed

dependabot Bot mentioned this pull request Aug 27, 2026

Bump Microsoft.ML.OnnxRuntime from 1.22.0 to 1.29.0 briandenicola/watch-tracker-app#105

Merged

dependabot Bot mentioned this pull request Sep 3, 2026

fix(deps): Bump the dotnet group with 16 updates lukislp/studylife#129

Closed

Navigation Menu

[EP ABI] Add weight pre-packing support to kernel-based plugin EPs - #26754

[EP ABI] Add weight pre-packing support to kernel-based plugin EPs#26754
Adrian Lizarraga (adrianlizarraga) merged 31 commits into
mainfrom
adrianl/plugin-ep-kernel-prepack

Adrian Lizarraga (adrianlizarraga) commented Dec 8, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	TEST(OrtEpLibrary, KernelPluginEp_Inference) {
	RegisteredEpDeviceUniquePtr example_kernel_ep;
	ASSERT_NO_FATAL_FAILURE(Utils::RegisterAndGetExampleEp(*ort_env, Utils::example_ep_kernel_registry_info,
	example_kernel_ep));
	Ort::ConstEpDevice plugin_ep_device(example_kernel_ep.get());

	// Create session with example kernel-based plugin EP
	Ort::SessionOptions session_options;
	session_options.AddConfigEntry(kOrtSessionOptionsDisableCPUEPFallback, "1"); // Fail if any node assigned to CPU EP.

	std::unordered_map<std::string, std::string> ep_options;
	session_options.AppendExecutionProvider_V2(*ort_env, {plugin_ep_device}, ep_options);

	// This model has Squeeze, Mul, and Relu nodes. The example plugin EP supports all nodes using registered kernels.
	Ort::Session session(*ort_env, ORT_TSTR("testdata/squeeze_mul_relu.onnx"), session_options);

Conversation

Adrian Lizarraga (adrianlizarraga) commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Adrian Lizarraga (adrianlizarraga) commented Dec 8, 2025 •

edited

Loading