Netcore3.0 app aborts in release configuration on a Linux machine

Possibly related to this previous issue: https://github.com/dotnet/coreclr/issues/27734

Running the attached app (which targets netcoreapp3.0) in release configuration on our Linux machines aborts with one of the following errors:

System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.'
Segmentation fault

Although unlike the previous bug, the occurrence is not 100% deterministic. It does however, usually occur on the first run of the program at some point and has been observed to occur more frequently when the following environment variables were set for debugging purposes.

COMPlus_DbgEnableMiniDump=1
COMPlus_StressLog=1
COMPlus_DbgMiniDumpType=4 (we found that our symbols weren't loading properly without this)

The program is reasonably simple, contains no unmanaged code, no third party dependencies, and has some fairly simple logic intended to use some memory (but not anywhere close to all of it) and stress the cpu in a manner similar to the way our full program does.

Operating system: CentOS Linux 7 (Core)
Kernel: Linux 3.10.0-862.11.6.el7.x86_64

dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100
 Commit:    04339c3a26

Runtime Environment:
 OS Name:     centos
 OS Version:  7
 OS Platform: Linux
 RID:         centos.7-x64
 Base Path:   /usr/share/dotnet/sdk/3.0.100/

Host (useful for support):
  Version: 3.0.0
  Commit:  95a0a61858

.NET Core SDKs installed:
  2.2.402 [/usr/share/dotnet/sdk]
  3.0.100 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Program.zip

category:correctness
theme:tail-call
skill-level:intermediate
cost:medium

I can confirm that it repros on my Linux box as well. I will look into it.

I've found the culprit. It is a bug in fast tailcall argument shuffling order where we end up overwriting lower 32 bits of an incoming argument stack slot before we actually extract that stack slot to a new parameter register. The problem is in the generated code for the FailingLinuxTestApp.Thing1.DoWork1 method. See the disass below with my comment on what happens.

clru $rip
Normal JIT generated code
FailingLinuxTestApp.Thing1.DoWork1(FailingLinuxTestApp.IThing2, Double[][], Double[], UInt16[], UInt16[], Double[], Int32, Int32, Int32, Double)
Begin 00007FFF7CAB1990, size 45

/home/janvorli/issues/27904/Program/Thing1.cs @ 17:
>>> 00007fff7cab1990 55                   push    rbp
00007fff7cab1991 488bec               mov     rbp, rsp
00007fff7cab1994 90                   nop
00007fff7cab1995 448b5d18             mov     r11d, dword ptr [rbp + 0x18]
00007fff7cab1999 8b7d20               mov     edi, dword ptr [rbp + 0x20]
00007fff7cab199c 897d10               mov     dword ptr [rbp + 0x10], edi <<< Here we store 32 bits from [rbp + 0x20] to [rbp + 0x10]
00007fff7cab199f 8b7d28               mov     edi, dword ptr [rbp + 0x28]
00007fff7cab19a2 897d18               mov     dword ptr [rbp + 0x18], edi
00007fff7cab19a5 488bfe               mov     rdi, rsi
00007fff7cab19a8 488bf2               mov     rsi, rdx
00007fff7cab19ab 498bd0               mov     rdx, r8
00007fff7cab19ae 498bc9               mov     rcx, r9
00007fff7cab19b1 4c8b4510             mov     r8, qword ptr [rbp + 0x10] <<< Here we load R8 from [rbp + 0x10], expecting the original incoming parameter, but the comment above shows that we have already overwritten it.
00007fff7cab19b5 458bcb               mov     r9d, r11d
00007fff7cab19b8 49bb6000bd7bff7f0000 movabs  r11, 0x7fff7bbd0060
00007fff7cab19c2 48b86000bd7bff7f0000 movabs  rax, 0x7fff7bbd0060
00007fff7cab19cc 488b00               mov     rax, qword ptr [rax]
00007fff7cab19cf 393f                 cmp     dword ptr [rdi], edi
00007fff7cab19d1 5d                   pop     rbp
00007fff7cab19d2 48ffe0               jmp     rax

cc: @dotnet/jit-contrib - this issue is in the .NET Core 3.0 release.

I've forgotten to mention - setting COMPlus_TieredCompilation=0 results in reproing the issue almost instantly.

@janvorli

I will take this, thank you @janvorli for doing the triage.

There have been a bunch of fixes in this area that are in 5.0.

dotnet/coreclr#20643
dotnet/coreclr#26255
dotnet/coreclr#26158

The test passes on master.

@jakobbotsch

/cc @jakobbotsch

Seems likely this was fixed in dotnet/coreclr#26255.

@jashook

@jashook
Is it possible / likely for these fixes to make it into the 3.1 release? It's currently preventing us from running an important chunk of our software on Linux machines and waiting for 5.0 in November doesn't seem like an option.
Alternatively, if this won't make it into 3.1, is there any guaranteed way of working around it and if there is, is it likely to have any negative effect on performance?

@jakobbotsch

@jakobbotsch is correct, this is an instance of https://github.com/dotnet/coreclr/issues/12468, dotnet/coreclr#26255 fixes this. I have a pr out into 3.1, the change is to turn of fast tail calls when the callee has stack space. Note I am not porting dotnet/coreclr#26255 back into 3.1, that will go out with 5.0.

@jashook

@jashook I note that 3.1 was released yesterday and your pull request temporarily fixing this issue didn't appear to be merged into the release. Is there any plan for when this will be fixed moving forward?

@rossgurbutt

@rossgurbutt the change is targeting a servicing release that will happen potentially a few months from now. That assumes that the change is approved, which currently does not look like it meets the bar to make it into release/3.1.

That being said there are workarounds to this issue.

The last method on the call stack can be marked as Optimization Sensitive. You can use [MethodImplAttribute(MethodImplOptions.NoOptimization)] to compile the method without optimizations.
This specific optimization is tail call related. The JIT does not require are recursive method to do a tail call optimization. It is sometimes pretty easy to avoid doing an opportunistic tail call. Off the top of my head the following can avoid this:
- MethodImplOptions.NoInlining disables Fast Tail Calls.
- The call site has to return the result of a call, without using the result in the calling method. For example:
```
 return foo()
```
Changing to use the result will invalidate the ability to tail call.
```
var temp = foo()
return temp;
```

I understand that these workarounds may not work for everyone. Please let me know if I can help out further.

Also please note that another possibility is to set the following environment variable: COMPlus_TailCallOpt=0

@jashook

Thanks for all of your help on this @jashook and @janvorli and apologies for the delay in replying.
We have tried several of the options and the one that worked easiest / best for us was MethodImplOptions.NoInlining.

Navigation Menu

Repository navigation

Netcore3.0 app aborts in release configuration on a Linux machine #2409

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Netcore3.0 app aborts in release configuration on a Linux machine #2409

Description

Activity

janvorli commented on Nov 20, 2019

janvorli commented on Nov 21, 2019

janvorli commented on Nov 21, 2019

jashook commented on Nov 21, 2019

jashook commented on Nov 21, 2019

jashook commented on Nov 21, 2019

jashook commented on Nov 21, 2019

jakobbotsch commented on Nov 21, 2019

rossgurbutt commented on Nov 22, 2019

jashook commented on Nov 22, 2019

rossgurbutt commented on Dec 4, 2019

jashook commented on Dec 6, 2019

jashook commented on Dec 6, 2019

rossgurbutt commented on Jan 7, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions