Skip to content

Test failure: System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset #125599

Closed

Description

@VincentBu

Failed in: runtime-coreclr libraries-jitstressregs 20260315.1

Failed tests:

net11.0-windows-Release-x86-jitstressregs3-Windows.10.Amd64.Open
- System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset
net11.0-windows-Release-x86-jitstressregs0x2000-Windows.10.Amd64.Open
- System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset

Error message:

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Half-way through waiting for remote process.
Memory load: 12

Image Name                     PID Services                                    
========================= ======== ============================================
svchost.exe                    768 BrokerInfrastructure, DcomLaunch, LSM,      
                                   PlugPlay, Power, SystemEventsBroker         
svchost.exe                    828 RpcEptMapper, RpcSs                         
svchost.exe                    936 Appinfo, CertPropSvc, DsmSvc, gpsvc,        
                                   iphlpsvc, ProfSvc, sacsvr, Schedule, SENS,  
                                   SessionEnv, ShellHWDetection, Themes,       
                                   UserManager, Winmgmt, wlidsvc, WpnService,  
                                   wuauserv                                    
svchost.exe                    932 TermService                                 
svchost.exe                   1012 Dhcp, EventLog, lmhosts, NgcCtnrSvc,        
                                   TimeBrokerSvc, vmictimesync                 
svchost.exe                    336 NcbService, PcaSvc, TrkWks, UmRdpService,   
                                   vmickvpexchange, vmicshutdown               
svchost.exe                    812 vmicheartbeat                               
svchost.exe                   1124 CDPSvc, EventSystem, FontCache, netprofm,   
                                   nsi, RemoteRegistry, W32Time,               
                                   WinHttpAutoProxySvc                         
svchost.exe                   1268 CryptSvc, Dnscache, LanmanWorkstation,      
                                   NlaSvc, WinRM                               
svchost.exe                   1316 Wcmsvc                                      
svchost.exe                   1404 BFE, CoreMessagingRegistrar, DPS, MpsSvc,   
                                   pla                                         
svchost.exe                   2036 AppHostSvc                                  
svchost.exe                   1080 W3SVC, WAS                                  
svchost.exe                   1668 StateRepository, tiledatamodelsvc           
svchost.exe                   1544 DiagTrack                                   
svchost.exe                   2160 LanmanServer                                
svchost.exe                   3700 CDPUserSvc_4e672, OneSyncSvc_4e672          
svchost.exe                   4076 AppReadiness                                
svchost.exe                   3140 AppXSvc, ClipSVC                            
svchost.exe                   6268 MapsBroker                                  
Timed out at 3/15/2026 10:13:21 AM after 60000ms waiting for remote process.
Wrote mini dump to: C:\h\w\AA83099F\w\A8960946\uploads\6480.wipwppdg.feo.dmp
	Process ID: 6480
	Handle: 1112
	Name: dotnet
	MainModule: C:\h\w\AA83099F\p\dotnet.exe
	StartTime: 3/15/2026 10:12:21 AM
	TotalProcessorTime: 00:00:01.0156250
	Threads:
		Thread #1 (OS 0x196C) [Finalizer] [Background] 
			[DebuggerU2MCatchHandlerFrame]
		Thread #2 (OS 0x1954)   [MTA]
			[InlinedCallFrame]
			[InlinedCallFrame]
			System.Runtime.Loader.AssemblyLoadContext.InternalLoad(System.ReadOnlySpan`1<Byte>, System.ReadOnlySpan`1<Byte>)
			System.Reflection.Assembly.Load(Byte[], Byte[])
			System.Diagnostics.Tests.StackTraceTests+<>c.<ToString_ShowILOffset>b__27_1(System.String, System.String, System.String)
			[InlinedCallFrame]
			[InlinedCallFrame]
			System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(System.Object, IntPtr*)
			System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(System.Object, System.Span`1<System.Object>, System.Reflection.BindingFlags)
			System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, 

Stack trace:

   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing) in /_/src/arcade/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs:line 225
   at System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset() in /_/src/libraries/System.Diagnostics.StackTrace/tests/StackTraceTests.cs:line 524
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) in /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.CoreCLR.cs:line 36
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset [FAIL]",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1336130
Error message validated: [System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset [FAIL]]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 3/16/2026 3:09:39 PM UTC

Report

Build Definition Test Pull Request
1343693 dotnet/runtime System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset #125507
1342303 dotnet/runtime System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset #124716
1339391 dotnet/runtime System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset #125604
1338825 dotnet/runtime System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset #125525
1338690 dotnet/runtime System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset #123252
1334996 dotnet/runtime System.Diagnostics.Tests.StackTraceTests.ToString_ShowILOffset #125497

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
2 6 6

Activity

  1. added
    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
    on Mar 16, 2026
  2. VincentBu commented on Mar 16, 2026

    @VincentBu
    ContributorAuthor

    Is this a Windows OS issue like #122690?

  3. dotnet-policy-service commented on Mar 16, 2026

    @dotnet-policy-service
    Contributor

    Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
    See info in area-owners.md if you want to be subscribed.

  4. added
    Known Build ErrorUse this to report build issues in the .NET Helix tab
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    and removed
    JitStressCLR JIT issues involving JIT internal stress modes
    on Mar 16, 2026
  5. jakobbotsch commented on Mar 16, 2026

    @jakobbotsch
    Member

    Failed in innerloop in #125497 too.

  6. mangod9 commented on Mar 19, 2026

    @mangod9
    Member

    Here is some copilot analysis:

    Summary
    
      ToString_ShowILOffset hangs indefinitely while loading an in-memory assembly via Assembly.Load(byte[]). The hang
      is in AMSI (Antimalware Scan Interface) — Windows Defender scans the in-memory assembly blob via an RPC call to
      MpClient, which blocks on NtAlpcConnectPort. This is a Windows OS/antivirus issue, not a .NET or JIT bug.
    
      -----------------------------------------------------------------------------------------------------------------
    
      Crash Context
    
      ┌───────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
      │ Field     │ Value                                                                                              │
      ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
      │ Build     │ runtime-coreclr libraries-jitstressregs 20260315.1                                                 │
      │           │ (https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build                   │
      │           │ /results?buildId=1336130)                                                                          │
      ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
      │ Helix     │ 5470f2b2 (jitstressregs3), 1d15360d (jitstressregs0x2000)                                          │
      │ Jobs      │                                                                                                    │
      ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
      │ Work Item │ System.Diagnostics.StackTrace.Tests                                                                │
      ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
      │ Test      │ StackTraceTests.ToString_ShowILOffset                                                              │
      ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
      │ Platform  │ Windows x86 on Windows.10.Amd64.Open Helix machines                                                │
      ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
      │ Dumps     │ Downloaded but DAC unavailable (x86 Release dev build, no public symbol server)                    │
      └───────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘
    
      -----------------------------------------------------------------------------------------------------------------
    
      Root Cause — Antivirus AMSI Hang
    
      This is the same bug as #122690 (diagnosed by @jkotas on 2025-12-22). The native stack from that issue:
    
       00  ntdll!NtAlpcConnectPort+0xc
       01  rpcrt4!I_RpcReceive+0x4ae1
       02  rpcrt4!RpcBindingFree+0x1f91
       03  rpcrt4!I_RpcReceive+0x882d
       04  rpcrt4!NdrClientCall2+0x8eb
       05  rpcrt4!NdrClientCall4+0x14
       06  MpClient!MpAmsiScan+0x416          ← Windows Defender AMSI scan
       07  MpOAV!DllRegisterServer+0x907
       08  amsi!AmsiUacScan+0x9e7
       09  amsi!AmsiScanBuffer+0xa1
       0a  coreclr!Amsi::IsBlockedByAmsiScan  ← coreclr calls AMSI on byte[] load
       0b  coreclr!FlatImageLayout::FlatImageLayout
       0c  coreclr!PEImageLayout::CreateFromByteArray
       0d  coreclr!PEImage::CreateFromByteArray
       0e  coreclr!AssemblyNative_LoadFromStream  ← Assembly.Load(byte[]) entry
    
      When Assembly.Load(byte[]) is called, CoreCLR calls Amsi::IsBlockedByAmsiScan which invokes AmsiScanBuffer →
      Windows Defender RPC (MpClient!MpAmsiScan). On these Helix Windows machines, the Windows Defender RPC endpoint
      hangs indefinitely, causing the 60-second RemoteExecutor timeout.
    
      The JitStress flags (JitStressRegs=3, JitStressRegs=0x2000) are incidental — they slow execution down and likely
      make the AMSI timeout window more likely to be hit, but they are not the cause.
    
      -----------------------------------------------------------------------------------------------------------------
    
      Stack Trace (from managed dump header in console log)
    
       Thread #2 [MTA] — hung for 60s at 1s CPU total
         AssemblyLoadContext.InternalLoad(ReadOnlySpan<byte>, ReadOnlySpan<byte>)
         Assembly.Load(byte[], byte[])                       ← Assembly.Load(byte[]) case in test
         StackTraceTests.<>c.<ToString_ShowILOffset>b__27_1(...)
         MethodBaseInvoker.InterpretedInvoke_Method(...)
         RemoteExecutor.Program.Main(...)
    
      -----------------------------------------------------------------------------------------------------------------
    
      Anomalous Findings
    
       - CPU time:
        1.0s over 60s wall-clock — pure block on external I/O (RPC), no spin
       - Hit count is 5× in 7 days, 5 distinct PRs — indicates ongoing flakiness on Windows 10 Helix machines, not
      regression-specific
       - Only on Windows x86 — x64 not affected (AMSI behavior may differ by process bitness)
    
      -----------------------------------------------------------------------------------------------------------------
    
      Confidence Level
    
      98/100 — Identical symptom (same managed stack, same 60s timeout, same Assembly.Load(byte[]) hang, same Windows
      x86 Helix machines) as #122690 which was definitively diagnosed from a native stack by @jkotas. The dump DAC was
      unavailable for independent verification but is not needed given the exact match.
    
  7. added a commit that references this issue on Mar 20, 2026
    6a815e8
  8. locked and limited conversation to collaborators on Apr 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions