Repository navigation
Netcore2.2 app aborts in release configuration on a Linux machine #13749
Description
We have an issue with the attached (heavily cut down) project which targets netcoreapp2.2.
We build and publish the app in both Release and Debug configurations.
When running on Windows, Release and Debug both complete successfully.
When running on Linux, Debug completes successfully, however Release aborts with one of the following errors:
- System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.'
- Segmentation fault
The program is very simple, contains no unmanaged code, no third party dependencies, and almost no logic at all.
Linux machine details (however this happens on all the different Linux machines / OSs available to us):
Operating system: CentOS Linux 7 (Core)
Kernel: Linux 3.10.0-862.11.6.el7.x86_64
dotnet --info:
.NET Core SDK (reflecting any global.json):
Version: 2.2.402
Commit: c7f2f96116
Runtime Environment:
OS Name: centos
OS Version: 7
OS Platform: Linux
RID: centos.7-x64
Base Path: /usr/share/dotnet/sdk/2.2.402/
Host (useful for support):
Version: 2.2.7
Commit: b1e29ae826
.NET Core SDKs installed:
2.2.402 [/usr/share/dotnet/sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.2.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.2.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.2.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
category:correctness
theme:calling-convention
skill-level:beginner
cost:small
Activity
janvorli commentedon Nov 8, 2019
@rossgurbutt than you for reporting it. I can confirm that I can repro it locally on my Ubuntu 16.04 box with .NET Core 2.2. However, it works fine with .NET Core 3.0.
Since .NET 2.2 in now in maintenance mode (only security updates may be added until it reaches its end of life), I would suggest to move your app to 3.0.
rossgurbutt commentedon Nov 8, 2019
@janvorli Thanks for the quick response.
With our full solution that we can't share here (for obvious proprietary information reasons) we still get the issue in Net Core 3.0, albeit less frequently and less deterministically. Our hope was that the issue may be the same and that identifying the easier to reproduce cause in Net Core 2.2 may also provide the information necessary to fix Net Core 3.0.
We're going to look into raising a premium support ticket so that we can share more of what is causing it in 3.0, meanwhile also trying to produce a cut-down app that we would be able to log here.
RobSwDev commentedon Nov 8, 2019
@janvorli Was this a known issue in 2.2? Do you have any information about what fails in 2.2 in case that helps us isolate the issue in 3.0?
For example, the crash we see is very near where we make a call to a native assembly, and I thought for sure that the native assembly would be the culprit, until Ross managed to repro (on 2.2 only) the issue without any native assembly references.
janvorli commentedon Nov 8, 2019
@RobSwDev I debugged the build with 2.2 and seeing what happens was somehow familiar, so I have a feeling that I've debugged the original issue in the past myself. But I wasn't able to find a related PR yet.
I could see that lowest 32 bits of an address that the code was using as a base address to access memory with an added offset of 8 when it crashed were all zeros. That's the pattern that made it feel familiar.
I'll try to find the PR that has fixed it.
janvorli commentedon Nov 8, 2019
The issue was caused by a JIT bug. When Thing1::DoWork1 calls thing2.DoWork2 and passes the same arguments to it (except for the first one that was the
thing2), it incorrectly shuffles the arguments from the incoming argument registers and stack slots to the DoWork2's argument registers and slots. Based on the Unix AMD64 calling convention, the parameters to DoWork1 are passed as follows:When it calls the thing2.DoWork2(array1, double1, array3, array4, array5, integer1, integer2, integer3);, the arguments should passed as follows:
So as you can see, the JIT needs to do these argument moves:
But in .NET 2.2, there was a bug in JIT that did the moves in an incorrect order. It moved 32 bit from stack slot 2 to stack slot 0 before moving stack slot 0 to R8. So the R8 (which should contain reference to array5) ended up having broken value and accessing it resulted in the crash in your testing app.
Unfortunately, I was unable to find the PR that fixed this problem.
jkotas commentedon Mar 1, 2020
.NET Core 2.2 is out of support.