b425314 test failing due to StackOverflow in nativeaot outerloop runs (CET+CFG leg)

[eduardo-vp]

Process is terminating due to StackOverflowException.

Return code:      1
Raw output file:      C:hw958A08B4wA958091AuploadsCLR-x86-JITV2.0-Beta2�425314�425314output.txt
Raw output:
BEGIN EXECUTION
call C:hw958A08B4p
ativeaottest.cmd C:hw958A08B4wA958091AeJITRegressionCLR-x86-JITV2.0-Beta2�425314�425314 b425314.dll 
Expected: 100
Actual: -1073741571
END EXECUTION - FAILED
FAILED

This test consistently fails in the CET+CFG leg being added in #105288.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

[MichalStrehovsky]

In the test infra this is failing with:

BEGIN EXECUTION
call C:\h\w\B0250941\p\nativeaottest.cmd C:\h\w\B0250941\w\A565094A\e\JIT\Regression\CLR-x86-JIT\V2.0-Beta2\b425314\b425314\ b425314.dll 
--------------------------------------------------
Debug Assertion Violation

Message: Hardware exception raised inside the runtime.

Expression: 'ASSERT_UNCONDITIONALLY'

File: D:\a\_work\1\s\src\coreclr\nativeaot\Runtime\EHHelpers.cpp, Line: 576
--------------------------------------------------
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED

What I'm seeing instead on my AMD machine is Process is terminating due to StackOverflowException. like the title says.

Of course this doesn't repro under a debugger. But eventually I found out we seem to be overflowing the shadow stack. I captured a dump and in that dump:

0:005> dps @ssp L30000
000000b3`4c082000  00007ff7`27154c91 b425314!S_P_CoreLib_System_Text_UTF8Encoding__GetCharCount_0+0x31 [/_/src/libraries/System.Private.CoreLib/src/System/Text/UTF8Encoding.cs @ 480]
000000b3`4c082008  00007ff7`270fcc81 b425314!String__CreateStringFromEncoding+0x61 [/_/src/libraries/System.Private.CoreLib/src/System/String.cs @ 540]
000000b3`4c082010  00007ff7`27186b84 b425314!S_P_CoreLib_Internal_NativeFormat_NativeReader__DecodeString+0x64 [/_/src/coreclr/tools/Common/Internal/NativeFormat/NativeFormatReader.String.cs @ 76]
000000b3`4c082018  00007ff7`2717e174 b425314!S_P_CoreLib_System_Reflection_Runtime_TypeInfos_NativeFormat_NativeFormatRuntimeNamedTypeInfo__get_Name+0x44 [/_/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/TypeInfos/NativeFormat/NativeFormatRuntimeNamedTypeInfo.cs @ 193]
000000b3`4c082020  00007ff7`2717da23 b425314!S_P_CoreLib_System_Reflection_Runtime_TypeInfos_RuntimeNamedTypeInfo__get_FullName+0x13 [/_/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/TypeInfos/RuntimeNamedTypeInfo.cs @ 96]
000000b3`4c082028  00007ff7`2717e0a1 b425314!S_P_CoreLib_System_Reflection_Runtime_TypeInfos_NativeFormat_NativeFormatRuntimeNamedTypeInfo__ToString+0x201 [/_/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/TypeInfos/NativeFormat/NativeFormatRuntimeNamedTypeInfo.cs @ 154]
000000b3`4c082030  00007ff7`271032ee b425314!S_P_CoreLib_System_Exception__AppendExceptionStackFrame+0x13e [/_/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Exception.NativeAot.cs @ 141]
000000b3`4c082038  00007ff7`2715f369 b425314!S_P_CoreLib_System_Runtime_EH__AppendExceptionStackFrameViaClasslib+0x79 [/_/src/coreclr/nativeaot/Runtime.Base/src/System/Runtime/ExceptionHandling.cs @ 266]
000000b3`4c082040  00007ff7`2715f7c0 b425314!S_P_CoreLib_System_Runtime_EH__DispatchEx+0x140 [/_/src/coreclr/nativeaot/Runtime.Base/src/System/Runtime/ExceptionHandling.cs @ 818]
000000b3`4c082048  00007ff7`2715f608 b425314!S_P_CoreLib_System_Runtime_EH__RhThrowEx+0x48 [/_/src/coreclr/nativeaot/Runtime.Base/src/System/Runtime/ExceptionHandling.cs @ 673]
000000b3`4c082050  00007ff7`2725c1e9 b425314!RhpThrowEx2 [D:\a\_work\1\s\src\coreclr\nativeaot\Runtime\amd64\ExceptionHandling.asm @ 198]
000000b3`4c082058  00007ff7`27161d16 b425314!S_P_CoreLib_System_Runtime_TypeCast__ThrowInvalidCastException+0x16 [/_/src/coreclr/nativeaot/Runtime.Base/src/System/Runtime/TypeCast.cs @ 946]
...
...
...
[BELOW IS REPEATED THOUSANDS OF TIMES]
000000b3`4c1fefb0  00007ff7`271c28b9 b425314!b425314_CastClass__CheckObjects+0x99 [D:\a\_work\1\s\src\tests\JIT\Regression\CLR-x86-JIT\V2.0-Beta2\b425314\b425314.cs @ 502]
000000b3`4c1fefb8  00007ff7`271c29a1 b425314!b425314_CastClass__CheckObjects+0x181 [D:\a\_work\1\s\src\tests\JIT\Regression\CLR-x86-JIT\V2.0-Beta2\b425314\b425314.cs @ 494]
000000b3`4c1fefc0  00007ff7`271c294e b425314!b425314_CastClass__CheckObjects+0x12e [D:\a\_work\1\s\src\tests\JIT\Regression\CLR-x86-JIT\V2.0-Beta2\b425314\b425314.cs @ 486]
000000b3`4c1fefc8  00007ff7`271c28b9 b425314!b425314_CastClass__CheckObjects+0x99 [D:\a\_work\1\s\src\tests\JIT\Regression\CLR-x86-JIT\V2.0-Beta2\b425314\b425314.cs @ 502]
000000b3`4c1fefd0  00007ff7`271c29a1 b425314!b425314_CastClass__CheckObjects+0x181 [D:\a\_work\1\s\src\tests\JIT\Regression\CLR-x86-JIT\V2.0-Beta2\b425314\b425314.cs @ 494]
000000b3`4c1fefd8  00007ff7`271c294e b425314!b425314_CastClass__CheckObjects+0x12e [D:\a\_work\1\s\src\tests\JIT\Regression\CLR-x86-JIT\V2.0-Beta2\b425314\b425314.cs @ 486]
[END REPEATED THOUSANDS OF TIMES]
...
...
...
000000b3`4c1fefe0  00007ff7`2713d7f5 b425314!S_P_CoreLib_System_Threading_Thread__StartThread+0x155 [/_/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.cs @ 457]
000000b3`4c1fefe8  00007ff7`2713df9a b425314!S_P_CoreLib_System_Threading_Thread__ThreadEntryPoint+0x1a [/_/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.Windows.cs @ 226]
000000b3`4c1feff0  00007ffa`cacee8d7 kernel32!BaseThreadInitThunk+0x17
000000b3`4c1feff8  00007ffa`cca7c34c ntdll!RtlUserThreadStart+0x2c

So it seems like we don't pop something off the shadow stack that we should be popping off. Unclear why this is the only test that runs into this.

Cc @VSadov for CET