add parallelism to prepacking/weight loading

[Xavier Dupré (xadupre)]

Description

Prepacking was not fully parallelized during session initialization. Memory-mapped weights were retrieved sequentially before each kernel processed them, so data retrieval did not overlap with the expensive PrePack() work.

This PR:

• dispatches eligible CPU OpKernel::PrePack() calls across the session intra-op thread pool;
• synchronizes shared bookkeeping while leaving the expensive per-kernel packing work concurrent;
• avoids nested parallelism in MatMulNBits;
• keeps cross-session shared prepacked-weight caching on its existing serialized path;
• enables outer prepack parallelism in all Android variants, including basic minimal builds.

Performance

Measured with a ModelBuilder-generated Qwen3.6-27B model quantized to INT4 on an Intel Xeon Platinum 8480C.

Before:

-- start: 2026-08-06 11:16:48.388195
---- end: 2026-08-06 11:17:32.072674
loading time: 43.68443649681285
-- start: 2026-08-06 11:17:32.072778
---- end: 2026-08-06 11:18:16.123237
loading time: 44.0504179620184
-- start: 2026-08-06 11:18:16.123356
---- end: 2026-08-06 11:19:00.205266
loading time: 44.08187505789101

After:

-- start: 2026-08-06 14:56:41.478934
---- end: 2026-08-06 14:56:48.648836
loading time: 7.169875857885927
-- start: 2026-08-06 14:56:48.648966
---- end: 2026-08-06 14:56:57.359275
loading time: 8.710286310408264
-- start: 2026-08-06 14:56:57.359358
---- end: 2026-08-06 14:57:05.959221
loading time: 8.599843401927501

Android minimal binary size

With the outer prepack path included, the GitHub Actions Android minimal baseline on the current merge commit measured 1,594,365 bytes. Its threshold is raised from 1,590,272 to 1,595,392 bytes, retaining approximately 1 KiB of headroom. The independent Azure MinimalBaseline threshold remains unchanged at 1,306,224 bytes pending an Azure-specific artifact measurement.

Validation

• Built onnxruntime_test_all.
• Ran SessionStateTestSharedInitalizersWithPrePacking.ParallelPrepackConvertsExceptionsToStatus.
• Ran lintrunner on all changed files.

[github-actions]

You can commit the suggested changes from lintrunner.

[Xavier Dupré (xadupre)]

Copilot fix lint

[Xavier Dupré (xadupre)]

Copilot fix lint

[Xavier Dupré (xadupre)]

Copilot fix lint again

[Xavier Dupré (xadupre)]

Copilot resolve the merge conflicts in this pull request

[Copilot]

Copilot resolve the merge conflicts in this pull request

Resolved in 7a47e69 — merged origin/main and fixed the conflict in onnxruntime/core/framework/session_state.cc (a ReplaceWithReferenceIfSaving → DiscardAndReplaceWithReferenceIfSaving rename from main that conflicted with the unrelated parallel-prepacking changes here).

[Copilot]

Pull request overview

Adds opt-in parallel weight prepacking during session initialization to reduce model-loading latency.

Changes:

• Introduces session.prepack.enable_parallel.
• Dispatches per-node prepacking through the intra-op thread pool.
• Adds MatMulNBits coverage for the option.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
onnxruntime/core/framework/session_state.cc	Implements parallel prepacking and synchronization.
include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h	Defines and documents the option.
onnxruntime/test/contrib_ops/matmul_4bits_test.cc	Adds option-based MatMulNBits tests.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.