Repository navigation
AV in WKS::gc_heap::relocate_address #34279
Description
Hit in Common.Tests on Windows x86 Checked in #34261:
ntdll!KiUserExceptionDispatcher+0xf [d:\rs1\minkernel\ntos\rtl\i386\userdisp.asm @ 597]
coreclr!WKS::tree_search+0x6 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 24978]
coreclr!WKS::gc_heap::relocate_address+0x5e [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 25068]
coreclr!WKS::GCHeap::Relocate+0x45 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 36864]
coreclr!PromoteCarefully+0x83 [F:\workspace\_work\1\s\src\coreclr\src\vm\siginfo.cpp @ 4875]
coreclr!GcEnumObject+0x4b [F:\workspace\_work\1\s\src\coreclr\src\vm\gcenv.ee.common.cpp @ 167]
coreclr!EECodeManager::EnumGcRefs+0xc48 [F:\workspace\_work\1\s\src\coreclr\src\vm\eetwain.cpp @ 4702]
coreclr!GcStackCrawlCallBack+0x152 [F:\workspace\_work\1\s\src\coreclr\src\vm\gcenv.ee.common.cpp @ 290]
coreclr!Thread::MakeStackwalkerCallback+0x48 [F:\workspace\_work\1\s\src\coreclr\src\vm\stackwalk.cpp @ 841]
coreclr!Thread::StackWalkFramesEx+0x193 [F:\workspace\_work\1\s\src\coreclr\src\vm\stackwalk.cpp @ 917]
coreclr!Thread::StackWalkFrames+0x159 [F:\workspace\_work\1\s\src\coreclr\src\vm\stackwalk.cpp @ 1001]
coreclr!ScanStackRoots+0x1e9 [F:\workspace\_work\1\s\src\coreclr\src\vm\gcenv.ee.cpp @ 146]
coreclr!GCToEEInterface::GcScanRoots+0xe4 [F:\workspace\_work\1\s\src\coreclr\src\vm\gcenv.ee.cpp @ 175]
coreclr!WKS::gc_heap::relocate_phase+0x35 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 25922]
coreclr!WKS::gc_heap::plan_phase+0x1aa8 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 24246]
coreclr!WKS::gc_heap::gc1+0x173 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 16782]
coreclr!WKS::gc_heap::garbage_collect+0x3bd [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 18414]
coreclr!WKS::GCHeap::GarbageCollectGeneration+0x239 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 38169]
coreclr!WKS::gc_heap::trigger_gc_for_alloc+0x40 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14039]
coreclr!WKS::gc_heap::try_allocate_more_space+0x1ba [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14157]
coreclr!WKS::gc_heap::allocate_more_space+0x14 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14587]
coreclr!WKS::gc_heap::allocate+0x31 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14618]
coreclr!WKS::GCHeap::Alloc+0x199 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 37242]
coreclr!Alloc+0x13e [F:\workspace\_work\1\s\src\coreclr\src\vm\gchelpers.cpp @ 231]
coreclr!AllocateSzArray+0x2ea [F:\workspace\_work\1\s\src\coreclr\src\vm\gchelpers.cpp @ 486]
coreclr!JIT_NewArr1+0x185 [F:\workspace\_work\1\s\src\coreclr\src\vm\jithelpers.cpp @ 2718]
System_Private_CoreLib!System.Span`1[[System.Byte, System.Private.CoreLib]].ToArray()+0x9a6b2fa1 [/_/src/libraries/System.Private.CoreLib/src/System/Span.cs @ 477]
Common_Tests!System.Net.Test.Common.VirtualNetworkStream.Write(Byte[], Int32, Int32)+0x67
Common_Tests!System.Net.Test.Common.VirtualNetworkStreamTest.VirtualNetworkStream_SingleThreadIntegrityTest_Ok()+0x137
Activity
jkotas commentedon Mar 30, 2020
Crash while stackwalking
Interop+Kernel32.GetFullPathNameW(Char ByRef, UInt32, Char ByRef, IntPtr)jkotas commentedon Mar 30, 2020
@fadimounir This looks like a duplicate of #31809
fadimounir commentedon Mar 30, 2020
Thanks @jkotas, I will take a look. Are there crash dumps in the failures since I wasn't able to repro this locally?
jkotas commentedon Mar 30, 2020
I have saved the dump and binaries at \jkotas9\drops\issue-34279 so that it won't get lost.
fadimounir commentedon Apr 1, 2020
I took a look at the dump, and it looks like we're enumerating the gs cookie of the InlinedCallFrame as a gcref object. Still not sure why
fadimounir commentedon Apr 2, 2020
@jkotas I've been able to trace the root cause of this issue. The problem is here: https://github.com/dotnet/runtime/blob/master/src/coreclr/src/vm/i386/PInvokeStubs.asm#L58
We're using the return address from the call to the
JIT_PInvokeBeginhelper, and not the return address of the actual PInvoke. The effect of this is that we report GC roots using incorrect GC data. I've been able to manually get a similar state under debugger, and saw that we were enumerating the gs cookie of the InlinedCallFrame. When I manually got this state under debugger however, thePromote()function was able to ignore this bad value, since it detected it wasn't a valid address in the GC heaps. In the crash dump though, things were not as lucky (maybe the gs cookie value we use in checked builds (0x12345678) fell between the low/high range of the gc heap?)Anyways, I did test under debugger what the correct behavior would have been by manually patching the return address in the
InlinedCallFrameto the correct one, and the GC refs enumeration seemed to be doing the correct thing (nothing was being enumerated from the pinvoke frame at that point).I think the fix is to pass the correct return address value as a parameter to the
JIT_PInvokeBeginhelper here: https://github.com/dotnet/runtime/blob/master/src/coreclr/src/jit/lower.cpp#L3635@dotnet/jit-contrib I need some help figuring out how to do that.
fadimounir commentedon Apr 2, 2020
BTW, the bug exists on all architectures, all platforms, and not just on Windows-x86. It's just extremely rare that it would ever fail
30 remaining items
jkotas commentedon Apr 14, 2020
@jkoritzinsky @AaronRobinsonMSFT The interop stub generators are generating invalid IL (passing byref values as pointer arguments) that is hitting subtle issues in the JIT.
Can you please take a look at fixing the stub generators and adding the assert that Andy suggested?
AndyAyersMS commentedon Apr 14, 2020
I see 93 cases when crossgenning SPC for x86 (changing the assert to a printf) -- some of these stubs have multiple byref args, so maybe 50 stubs in all?
AaronRobinsonMSFT commentedon Apr 14, 2020
@jkotas and @AndyAyersMS Will take over issue.
JIT: retype byrefs passed to unmanaged callers as native int
JIT: retype byrefs passed to unmanaged callers as native int (#39105)
mjsabby commentedon Jul 13, 2020
A couple of questions:
(1) Does this not impact x64 for some reason? I don't understand why not if that is the case.
(2) Should we add this to IL Verification also?
I'm trying to determine if I'm hitting this or not, and I don't have a good way to audit all my generated IL at runtime in a system where this is happening.
AndyAyersMS commentedon Jul 15, 2020
I would guess it is x86 specific. x86 uses a different gc encoding and may rely more heavily on the pinvoke signature to make sense of the stack.
I can get you a version of the jit that will (optionally) fail with "invalid program" errors instead of asserts or silent casts, if you want to look for this sort of IL in your apps by running tests.
mjsabby commentedon Jul 15, 2020
@AndyAyersMS I see you've made a change that ignores this assert. I suppose if I use that as a baseline build for my testing I can assume that it is not the problem. And you've also said it doesn't seem likely on x64.
AndyAyersMS commentedon Jul 15, 2020
Yes, more or less, at least for byrefs, motivated by IL from C++/CLI that passes byrefs to unmanaged callers.
We're hoping that the IL producer has somehow guaranteed that the byref won't change values across a GC, and want to make sure we don't cause the program to fail for other reasons.
We could just reject the IL, but since we've tolerated this for a long time it would likely cause more trouble than it's worth. But you might want to do that if you are able/willing to fix up your IL producers.
I might be wrong.... @jkotas ?
jkotas commentedon Jul 15, 2020
I agree with your statement: "It doesn't seem likely on x64". We have not seen an actual problem caused by it on x64.