[Bug] Memory Contamination due to Scratchpad Reuse

[mmoffitt]

Describe the issue

I believe there is an issue where ONNX Runtime is improperly reusing scratchpad memory between sessions without initializing values back to zeros. As a result, one observes different behavior depending on the order in which models are executed, and can even inspect the internal state of models which have previously run.

Version info:

% pip3 show onnxruntime          
Name: onnxruntime
Version: 1.26.0
Summary: ONNX Runtime is a runtime accelerator for Machine Learning models
Home-page: https://onnxruntime.ai
Author: Microsoft Corporation
Author-email: onnxruntime@microsoft.com
License: MIT License
Location: /Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/site-packages
Requires: flatbuffers, numpy, packaging, protobuf

To reproduce

Reproducing the bug
See the attached ort_bug.zip, which contains ort_bug.py + A.onnx + B.onnx

Observed behavior:
Executing model A before B:

% python3 ort_bug.py A.onnx B.onnx
A.onnx yields 1.5
B.onnx yields 1.5

Executing model B before A:

% python3 ort_bug.py B.onnx A.onnx
B.onnx yields 1.0
A.onnx yields 1.0

ort_bug.zip

Urgency

It is fairly urgent -- we are using ONNX Runtime to evaluate submissions to the 2026 NeuroGolf Championship, which concludes in roughly one month.

Platform

Mac

OS Version

Version 26.5 (25F71)

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.26.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[mmoffitt]

One More Note: As the script demonstrates, I tested running the sessions with options.enable_cpu_mem_arena = False and options.enable_mem_pattern = False, yet the contamination still occurs. This suggests the leaked state is surviving outside the standard memory pool.

[hariharans29]

Did you mean to say ARM64 instead of x64 for 'Architecture' ? IIUC - OS version 26 dropped support for Intel x64 Macs - unless you are using some emulation of some sort ?

If you meant ARM64, #27136 introduced a backend kernel selector - you can disable KleidiAI kernel selection via that session option and re-try (The PR has a sample usage of that session option) ?

[mmoffitt]

Oh, yes you're right, I am in fact using ARM64 instead of Intel x64.

you can disable KleidiAI kernel selection via that session option and re-try

I added this one line, and indeed the behavior is now correct!

options.add_session_config_entry("mlas.disable_kleidiai", "1")

Is this the preferred long-term workaround? If so, please feel free to close this bug — many thanks!

[yuslepukhin]

Is this the preferred long-term workaround? If so, please feel free to close this bug — many thanks!

Can we close the security issue reported?

This problem family was addressed before in prior KleidiAI Conv fixes; current report is most likely a regression/new variant in the same area.

Previously merged PRs:

• PR Fix Conv LHS packing padding/uninitialized ptrs V2 #27215 Fix Conv LHS packing padding/uninitialized pointers.
• PR [MLAS] Add kleidiai pad ptr invalidation test case #27465: Adds regression for pad-buffer relocation and cache invalidation.
• PR Remove data hash from hashing code, to prevent unbounded growth issue #28363: Removes data hash from LHS cache key and adds cache-behavior tests.

[mmoffitt]

Can we close the security issue reported?

For context: out of concern that this issue allowed data from one model to be leaked to another, I had logged a potential security vulnerability over the weekend. I don't understand the low-level details enough to know if this is a legitimate concern, but I trust Microsoft to make the appropriate judgement call.

[hariharans29]

Is this the preferred long-term workaround? If so, please feel free to close this bug — many thanks!

Oh no no - I wanted to narrow it down to KleidiAI before tagging the relevant folks to take a look - this was not the proposed long-term fix :)

Dmitri Smirnov (@yuslepukhin) seems to already have a fix - I ll tag the KleidiAI folks for a review.

[JonathanC-ARM]

Hi Michael D. Moffitt (@mmoffitt), I looked into this issue and was able to confirm that it no longer reproduces with the latest source code. If you’re able to build a wheel from the current latest source, that should resolve the issue.

I’ve also provided Hariharan Seshadri (@hariharans29) with a targeted fix for the 1.26 branch, but moving to the latest code would be the preferred route if that’s feasible.

[yuslepukhin]

#28678 (comment)

[mmoffitt]

Excellent, many thanks Jonathan Clohessy (@JonathanC-ARM) and Dmitri Smirnov (@yuslepukhin)! Yes, this works for me — I'll let you know if I run into any additional issues.