[ci-scan] Test failure: NumberHandlingTests_Metadata.Number_AsCollectionElement_RoundTrip on NativeAOT windows-x64

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1448568
Build error leg or test failing: windows-x64 Release NativeAOT_Checked_Libs_SpeedOpt - System.Text.Json.SourceGeneration.Roslyn4.4.Tests

Error Details

[FAIL] System.Text.Json.SourceGeneration.Tests.NumberHandlingTests_Metadata.Number_AsCollectionElement_RoundTrip
Assert.Throws() Failure: Exception type was not an exact match
Expected: typeof(System.Text.Json.JsonException)
Actual:   typeof(System.NullReferenceException)
   at System.Text.Json.Serialization.Tests.NumberHandlingTests.Number_AsCollectionElement_RoundTrip() in /_/src/libraries/System.Text.Json/tests/Common/NumberHandlingTests.cs:line 388
   at System.Text.Json.Serialization.Tests.NumberHandlingTests.Number_AsCollectionElement_RoundTrip()

Error Message

true only for clear infra flakes. ExcludeConsoleLog skips helix log scanning. -->

{
  "ErrorMessage": ["NumberHandlingTests_Metadata.Number_AsCollectionElement_RoundTrip", "Expected: typeof(System.Text.Json.JsonException)"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Impact on platforms

runtime-nativeaot-outerloop (def 265) / windows-x64 Release NativeAOT_Checked_Libs_SpeedOpt / System.Text.Json.SourceGeneration.Roslyn4.4.Tests / Exit code 1
Occurrences in window: 2 builds (1448568, 1449365)

First build it occurred

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1448568
Finished: 2026-06-04T04:19:15Z

Additional context

Under NativeAOT, the test expects JsonException when deserializing a number as a collection element, but receives NullReferenceException instead — suggesting the NativeAOT compiled code path throws an unexpected null dereference before the expected validation logic fires.

Related: #66220 (same test, different platform — iOS/Mono, filed 2022, still open)

Filed by ci-failure-scan (detection only). ci-failure-fix walks open [ci-scan] KBEs and either opens a small fix PR or comments here to loop in owners — it never disables the test.

Generated by CI Outer-Loop Failure Scanner · ● 29.6M · ◷

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

@agocke

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

closing as has not failed since opening

This one is real. The test fails in various ways:

Hang in https://dev.azure.com/dnceng-public/public/_build/results?buildId=1453415&view=logs&jobId=6a7e26fa-36e7-5a45-28af-dc6c8e6724e6&j=6a7e26fa-36e7-5a45-28af-dc6c8e6724e6&t=81df285b-8200-5c2b-7a59-0b01660bb396

AV in https://dev.azure.com/dnceng-public/public/_build/results?buildId=1453266&view=logs&j=e2a090ef-1ffe-5b5a-8213-260fbf226a4b&t=f60bbc11-44b7-557d-f3f2-48d48cb3fb2d&l=120

Nullref in https://dev.azure.com/dnceng-public/public/_build/results?buildId=1452729&view=logs&j=62c84a0c-1188-56fe-6938-e058a633beb1&t=5486a8ab-9175-5ddc-478d-3ba90b918150&l=72

Hang in https://dev.azure.com/dnceng-public/public/_build/results?buildId=1449365&view=logs&j=e2a090ef-1ffe-5b5a-8213-260fbf226a4b&t=f60bbc11-44b7-557d-f3f2-48d48cb3fb2d

Nullref in https://dev.azure.com/dnceng-public/public/_build/results?buildId=1448568&view=logs&j=6a7e26fa-36e7-5a45-28af-dc6c8e6724e6&t=81df285b-8200-5c2b-7a59-0b01660bb396

We started seeing it on 6/3. All the failures seem to be Windows x64.

Looks to be runtime-async-adjacent.

@jakobbotsch

Looks to be runtime-async-adjacent.

@jakobbotsch was #128706 x86-only or could it affect x64 too?

Looks to be runtime-async-adjacent.

@jakobbotsch was #128706 x86-only or could it affect x64 too?

It should be x86 only. I did a quick audit and I didn't immediately see anything that would affect other platforms.

If you still have the setup locally, can you see if reverting #127932 fixes it?

Still fails with a revert of that. My steps:

git revert b540c365da5e5f71a6c0389f5764f363bc59d4a2  
rmdir /s /q artifacts  
build.cmd clr.aot+libs -c Release 
dotnet.cmd build src\libraries\System.Text.Json\tests\System.Text.Json.SourceGeneration.Tests\System.Text.Json.SourceGeneration.Roslyn4.4.Tests.csproj /p:TestNativeAot=true -c Release -f net11.0
run_tests.bat

Where run_tests.bat is:

@echo off
setlocal enabledelayedexpansion

set EXE=D:\git\runtime1\artifacts\bin\System.Text.Json.SourceGeneration.Roslyn4.4.Tests\Release\net11.0\publish\System.Text.Json.SourceGeneration.Roslyn4.4.Tests.exe

for /L %%i in (1,1,100) do (
    echo === Run %%i of 100 ===
    "%EXE%" -notrait category=failing
    if !errorlevel! neq 0 (
        echo.
        echo FAILED on run %%i with exit code !errorlevel!
        exit /b !errorlevel!
    )
)

echo.
echo All 100 runs completed successfully.
exit /b 0

@MichalStrehovsky

I've looked at a couple of crashes under debugger. Both of them were with functions related to PerformAsCollectionElementSerialization. This function does a lot of GVM lookups that result in runtime async methods, I am not sure if related or a red herring.
A few notes:

In both cases the continuation is null, so we haven't suspended/resumed, i.e. we are executing this function synchronously
In one case an object reference was corrupted with what looked to be string data. The value ended with 0xc, so not even a "forgotten" object reference due to a GC hole.
For both cases I saw the register r15 seemingly being incorrect inside PerformAsCollectionElementSerialization. In the one case it had string data, in the other case it does not point to the closure allocated by the method as expected (instead it has some other value with a wrong method table)

I wonder if some miscompile ends up corrupting r15 on a rare path, or if some rare GVM lookup can result in r15 not being restored. Does the GVM lookup ever end up calling into any assembly helpers @MichalStrehovsky?

I wonder if some miscompile ends up corrupting r15 on a rare path, or if some rare GVM lookup can result in r15 not being restored. Does the GVM lookup ever end up calling into any assembly helpers @MichalStrehovsky?

I double checked in a debugger, but there should be no special assembly helpers in this path. GVM calls only have this shape:

00007FF6046E8EF0  lea         rdx,[__RuntimeMethodHandle_repro_IBot__Blah<Object> (07FF604746A20h)]  
00007FF6046E8EF7  call        S_P_CoreLib_System_Runtime_TypeLoaderExports__GVMLookupForSlot (07FF6046B8460h)  
00007FF6046E8EFC  test        al,2  
00007FF6046E8EFE  je          repro_Program___Main__+2Eh (07FF6046E8F0Eh)  
00007FF6046E8F00  mov         rdx,qword ptr [rax+6]  
00007FF6046E8F04  mov         rcx,rbx  
00007FF6046E8F07  cmp         dword ptr [rcx],ecx  
00007FF6046E8F09  call        qword ptr [rax-2]  
00007FF6046E8F0C  jmp         repro_Program___Main__+35h (07FF6046E8F15h)  
00007FF6046E8F0E  mov         rcx,rbx  
00007FF6046E8F11  cmp         dword ptr [rcx],ecx  
00007FF6046E8F13  call        rax

and GVMLookupForSlot is "normal" C# and the target of the indirect call is the resolved method (possibly an unboxing thunk in assembly if the GVM is implemented on a struct, but those are simple).

@janvorli

I managed to get a TTD trace after running a bunch of these in parallel overnight.
From the trace it looks like an issue in handling GC during EH. One thread is running the test that calls PerformAsCollectionElementSerialization, and that thread is deep inside a child frame executed via a funclet. GC happens on a different thread and the GC stack walk for the test ends up skipping over ~20 managed frames.
We are unwinding over an InThrowSiteThunk that calls into HandleExCollided when we end up skipping over the frames.
The end result is that when the scanning sees r15 is live in PerformAsCollectionElementSerialization it does not relocate the correct r15 that was saved in one of the skipped frames.

@janvorli Can you please take a look?
My analysis and the trace is available at https://1drv.ms/u/c/2cb5a68f37fcb996/IQBaHvpmXzznSoOiiwPp4E-PAT8DPfLQltorhoW1Edblu4c?e=59bafV.

Fix NativeAOT GC hole issue (#129598)

Fix NativeAOT GC hole issue (dotnet#129598)

Navigation Menu

Repository navigation

[ci-scan] Test failure: NumberHandlingTests_Metadata.Number_AsCollectionElement_RoundTrip on NativeAOT windows-x64 #129010

Build Information

Error Details

Error Message

Impact on platforms

First build it occurred

Additional context

Report

Summary

4 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[ci-scan] Test failure: NumberHandlingTests_Metadata.Number_AsCollectionElement_RoundTrip on NativeAOT windows-x64 #129010

Description

Build Information

Error Details

Error Message

Impact on platforms

First build it occurred

Additional context

Report

Summary

Activity

dotnet-policy-service commented on Jun 5, 2026

jeffschwMSFT commented on Jun 5, 2026

MichalStrehovsky commented on Jun 8, 2026

MichalStrehovsky commented on Jun 8, 2026

jakobbotsch commented on Jun 8, 2026

4 remaining items

jakobbotsch commented on Jun 11, 2026

MichalStrehovsky commented on Jun 11, 2026

jakobbotsch commented on Jun 12, 2026

MichalStrehovsky commented on Jun 15, 2026

jakobbotsch commented on Jun 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions