Capture AccessViolationException

[BaronKiko]

Hi
I would like to be able to capture an AccessViolationException to provide useful information about why the application crashed. I don't intend to continue running the application, I just want some of the information that would be available when run inside visual studio.
Currently the program silently closes without reporting anything when run outside visual studio. At the very least I would like a dialog box saying the application crashed due to an AccessViolationException.
Taking this example program: https://gist.github.com/gdkchan/bd18dea1679283ff6e2c152b5f836927
In .Net framework I can simply use a try/catch around line 115. (In 4.0 onward there is some tags needed)
I would like to be able to do that in Core however the tags aren't honored so it's not possible
Thank you for your time.

[filipnavara]

I'd say it's generally a bad idea to execute any code when this happens. Windows Error Reporting (aka Watson) can provide memory dumps in these occasions. This can be used both for local diagnosis and for tracking the errors from customers through the Partner Portal (as long as the application is signed with valid code signing certificate).

[BaronKiko]

I don't want to use Window error reporting because:

• To enable it each user need to mess with the registry which most people just aren't comfortable doing.
• We would have to sign our application which we don't currently do and see no reason to do so.
• It only works on windows, defeating the whole point of using core (saying that for our case if windows allowed 4kb pages in the first place non of this would be needed)
• It would require the user to actively work with us to debug the issue when most users, if you are lucky. report it and never follow up.
• Many of the issues we will likely face will be hard to replicate so after the first crash it may never come up again, especially if the error only comes up after extended use.

In short it's a terrible solution for us.

I'm well aware that it's not smart to continue running but ultimately I think that decision should be left to the developer, not the .net runtime. Just like it is with framework. And by default there needs to be something to tell the user a crash has occurred, not just closing the application.

As a bare minimum the documentation needs to make it clear that this feature doesn't work in .net core, really the tag should just be removed if it's not going to be supported (it should be supported)
https://docs.microsoft.com/en-us/dotnet/api/system.runtime.exceptionservices.handleprocesscorruptedstateexceptionsattribute?redirectedfrom=MSDN&view=netcore-2.2

[RussKeldorph]

@dotnet/dotnet-diag for suggestions

[filipnavara]

saying that for our case if windows allowed 4kb pages in the first place non of this would be needed

I kinda started writing a long follow-up before I noticed this sentence... Can you elaborate on why 64Kb pages are a problem for you? How would 4Kb pages help you? Are you trying to create some guard page scheme?

I will post my original follow-up anyway in case someone else stumbles upon it:

We are shipping a large .NET application and we spend quite some time with trying many error reporting tools (in-app, systems for collecting, external tools, etc.) and 10+ years on debugging various errors with a desktop application deployed to more than million of users. Maybe my perspective will help you understand why I suggest to rely on WER (or any other operating system mechanism) for these kind of errors.

I understand that it's nearly inevitable that AccessViolationException will happen at some point if you deploy application at scale to unknown environments. However, in .NET programs the major cause of these exceptions are unmanaged libraries and improper interfaces to unmanaged libraries. It's very rare that the access violation is caused by a managed unsafe code because you have to go out of your way to allow it. For properly written code it should never happen (this is not kernel code where you may do it intentionally). The unmanaged libraries could be something you distribute with your application (such as SQLite) or a 3rd-party library loaded through some extension method (injected hooks through registry, codec loaded by sound player, shell extension loaded into process through open file dialog, network library loaded through the WinSock layer, etc.).

For normal managed exceptions the exception itself usually contains enough information to reasonably identify the cause or at least the location of the error. That's not the case for AccessViolationException. Only very rarely the actual memory corruption happens at the time it is discovered. And even if it was the case then you will likely be interested in the native stack trace, not the managed one. No matter how hard you try to recover from it or capture enough information [in process] it's likely not going to be enough to solve it.

We eventually ended up with multiple ways to debug these errors:

1. Unhandled exceptions that can be caught are processed in managed code. We use custom handler and serialize the exception to file and then run another process to submit it. Open source tools like HockeyApp or Sentry can do this for you and deliver reasonably good job at it.
2. We check for existence of our "ClrDump.exe" tool in the exception handler filter (exposed as when keyword in C#) and run it to produce memory dumps. The actual memory dump is created out-of-process, but it's triggered from in-process code which is still unusable for access violations.
3. If the error is in any way reproducible we use the ProcDump tool to launch our application. It can be configured to produce memory dumps on first chance exceptions or many other conditions and it can be deployed to customers easily.
4. If everything else fails we resort to WER, either through registry or through the developer portal

Given this introduction to where I come from let me try to answer your questions:

We would have to sign our application which we don't currently do and see no reason to do so.

Two reasons: 1) Windows SmartScreen is less annoying for signed applications 2) You get access to the error reports with the built-in system mechanism, no special effort necessary on your part; the certification you sign with serves as a proof of ownership that gives you access to the the collected error reports

It only works on windows, defeating the whole point of using core

WER works on Windows. Other systems have similar mechanisms. On Linux you often get core dumps automatically, on macOS they can be enabled for local debugging or there's WER-like system with online submissions.

It would require the user to actively work with us to debug the issue when most users, if you are lucky. report it and never follow up.

Nope. The nice thing about WER is that is in every machine and it's just one click of a button (often not even that) for the user to submit the memory dump which is created automatically. For errors like access violation that's the best information you can get, much better than anything you can read from the managed exception object.

[filipnavara]

As a bare minimum the documentation needs to make it clear that this feature doesn't work in .net core, really the tag should just be removed if it's not going to be supported (it should be supported)

Agreed. If the documentation is wrong it should be fixed. Note that the documentation is hosted on GitHub, so issues/PRs can be submitted for it.

[BaronKiko]

To start the reason all this is needed is because we are writing a switch emulator called Ryujinx (https://github.com/Ryujinx/Ryujinx)
So your point here kinda sums of the whole situation:

For properly written code it should never happen (this is not kernel code where you may do it intentionally)

This basically is kernel code. At least it's emulating one. We want to be able to allocate pages but if the game accesses an unmapped page we want to capture it. As we are emulating switch games it's very rare the game is actually broken and probably the emulator.
Doing the check in code is reasonably simple however every access has to have a page table translation so we need that to be as fast as possible. It's only a few instructions. Adding in validation vastly increases the amount of code so we just can't really do that.
What we want to do is act like a real kernel and intentionally let it throw this exception. This gives us checking without any extra asm.

This also leads on to this question:

Can you elaborate on why 64Kb pages are a problem for you?

If we could use 4kb pages we could directly map the switch (ARM) address space to windows. Meaning we can cut out all page table translation as the hardware will do it for up. On linux this is available and it works giving us the fastest solution possible. On windows you can't do that unless you are a driver, for unknown reason as it internally uses 4kb pages -_-
Sadly even if this were implemented into windows today it wouldn't be useful to us because older versions of windows wont allow it and we want to support them.

Regarding WER
It's just not a very good fit here. Signing an unlicenced open source application just isn't going to go down well. Who would manage the keys, have the right to make a new build etc. Builds are currently just dumped out by appveyor.
There is also the issue that WER just wont ever provide enough info for us like the game the user is running, what they did to create the error etc. While it would be nice to have and may well be added at some point (probably after long talks about how to manage the whole thing) we want something we control so we can work out whatever we need to and builds that aren't signed still provide useful information.

My point
All this shouldn't really matter anyway. It's not a decision .net core should be making for you. I can understand hiding it behind a flag to avoid accidental use but it should be left to the developer to make the final decision. Just like in .net framework. There was nothing broken about it before so why isn't it good enough for .net core.

That was a long post so if I missed out something from your question I apologize, just let me know.

[BaronKiko]

Thinking about it. I should really ask.
Is it going to take significant dev time to add the flag into core? If not then I see no reason not to add it. It's totally optional, disabled by default, increases compatibility with porting from framework and lets you do things you currently just can't do in core but can in other languages and variants of .net.

[cshung]

If you only care about running on Windows - you might be able to "handle" your EXCEPTION_ACCESS_VIOLATION by registering your own VectoredExceptionHandler.

You need to make sure you:

1. Call the AddVectoredExceptionHandler() after CLRAddVectoredExceptionHandler() is called and make sure it is registered as the first handler - now your vectored exception handler get to deal with the EXCEPTION_ACCESS_VIOLATION earlier then CoreCLR can.

2. Identify if the EXCEPTION_ACCESS_VIOLATION is yours - this can be hard - all you have is a CONTEXT.

3. If the EXCEPTION_ACCESS_VIOLATION is yours, do whatever (in native - don't try to call back to managed, that would be a disaster) you want to handle the EXCEPTION_ACCESS_VIOLATION, and you can even return EXCEPTION_CONTINUE_EXECUTION if continue the execution is what you want, or

4. If it is not your EXCEPTION_ACCESS_VIOLATION (or in fact, anything else), simply return EXCEPTION_CONTINUE_SEARCH.

Fingers crossed, it might just work.

Adding a vectored exception handler after CoreCLR did and so it acts as the first one is a total hack. It is neither recommended nor supported, but it might solve your problem. Would love to hear from you if it works.

[BaronKiko]

That does sound like a total hack. I doubt it will be accepted on released code but I will mess around with it tomorrow anyway because it just seems interesting. I would still like a proper solution.
Thank you.

[noahfalk]

Taking this example program: https://gist.github.com/gdkchan/bd18dea1679283ff6e2c152b5f836927
In .Net framework I can simply use a try/catch around line 115. (In 4.0 onward there is some tags needed)

@janvorli am I remembering right that you did a bunch of the earlier exception handling work for .Net Core? Do you recall if there was a conscious decision one way or another to support corrupting state exceptions? Looking in the code I see conflicting evidence, for example FEATURE_CORRUPTING_EXCEPTIONS appears enabled, but the test tree has no tests for it and in practice it doesn't appear to work.

[janvorli]

I've verified that if you set env var COMPlus_legacyCorruptedStateExceptionsPolicy=1, you can catch the exception in .NET Core.

The HandleProcessCorruptedStateExceptionsAttribute should work too, based on what I can see in the runtime here:
https://github.com/dotnet/coreclr/blob/master/src/vm/excep.cpp#L11205-L11257
But for some reason, it doesn't work.

[BaronKiko]

Could it be related to this: dotnet/coreclr#10957 ?

[noahfalk]

@janvorli To the best of your understanding the fact that it doesn't work is simply a bug we should fix (and hopefully add tests for) rather than a feature we deliberately wanted to disable for some reason?

[noahfalk]

dotnet/coreclr#10957 certainly looks related. @jkotas perhaps you could add a little background since you filed #7272?
Assuming we are sticking with the decision in #7272, then we have an open issue that the .Net Core documentation is now in a misleading state that doesn't give any indication that the functionality is deprecated.

[noahfalk]

Reading through #7272 more thoroughly, a few bits of interest:

https://github.com/dotnet/coreclr/issues/9045#issuecomment-292452208

I think it would be reasonable to keep the legacyCorruptedStateExceptionsPolicy config switch that disables the fail fast on corrupting exceptions and let them propagate as regular exceptions.

https://github.com/dotnet/coreclr/issues/9045#issuecomment-290159433

Handling of corrupted state exceptions is set of low-level desktop features that tried to make it possible to write managed code for what would be best written in plain C. We have abandoned this direction and not included features from this set in .NET Core whenever we had opportunity to do so.

https://github.com/dotnet/coreclr/issues/9045#issuecomment-289949603

In particular, any processing of dirty AVs is security liability. It is best to fail fast for them right away.