[API Proposal]: Calling UnmanagedCallersOnly method regardless of GC mode

[DaZombieKiller]

Background and motivation

I am using Windows-specific terminology and APIs in this issue, but the broader idea applies to other platforms as well.

To my knowledge it is not possible today to write a "safe" vectored exception handler in C#, as the runtime will crash when an exception occurs in managed code (due to no GC transition occurring):

static void Main()
{
    unsafe
    {
        AddVectoredExceptionHandler(1, &ExceptionHandler);
        *ReadOnlyOrGuardedPage = 10;
    }
}

[UnmanagedCallersOnly]
static unsafe int ExceptionHandler(EXCEPTION_POINTERS* ex)
{
    Console.WriteLine("ExceptionHandler");
    return EXCEPTION_CONTINUE_SEARCH;
}

This will exit with:

Fatal error.
Invalid Program: attempted to call a UnmanagedCallersOnly method from managed code.

It is possible to bypass this by not using UnmanagedCallersOnly and passing a managed function pointer to AddVectoredExceptionHandler, but I don't think I need to explain why this is a bad idea (and it also now has the opposite issue: it is dangerous if an exception occurs in unmanaged code now).

Additionally, if you define your vectored exception handler in native code, you still have the same problems if you end up calling back into managed code (i.e. through a function pointer).

My use case for this is high-performance dirty page tracking in a C# port of an emulator (this technique is known colloquially among emulator developers as "fastmem", and is also seen in software like VirtualBox). In C++ I allocate pages with PAGE_READONLY protection and catch write exceptions in a vectored exception handler. This handler deals with notifying page listeners and changing the protection to PAGE_READWRITE to suppress exceptions until the page needs to be tracked again. Write watching through GetWriteWatch and MEM_WRITE_WATCH is insufficient as it is a "query" API rather than a "notification" API, so the extremely chatty usage patterns that would be required to use it lead to worse performance by several orders of magnitude.

This issue is similar in spirit to #12077, though that one is limited to catching access violation exceptions. I have use cases for catching other types as well (such as invalid instruction exceptions), though they are less relevant.

API Proposal

I propose adding a new ExtendedGCTransition attribute that can be applied to UnmanagedCallersOnly methods, which causes them to support being invoked regardless of the current GC mode:

namespace System.Runtime.CompilerServices;

[AttributeUsage(AttributeTargets.Method, Inherited = false)]
public sealed class ExtendedGCTransitionAttribute : Attribute
{
    public ExtendedGCTransitionAttribute()
    {
    }
}

API Usage

static void Main()
{
    unsafe
    {
        AddVectoredExceptionHandler(1, &ExceptionHandler);
        *ReadOnlyOrGuardedPage = 2;
    }
}

[ExtendedGCTransition]
[UnmanagedCallersOnly]
static unsafe int ExceptionHandler(EXCEPTION_POINTERS* ex)
{
    Console.WriteLine("ExceptionHandler");
    return EXCEPTION_CONTINUE_SEARCH;
}

Alternative Designs

This could be exposed as a CallConv type, but I don't believe this would make sense as it is not related to how the method is invoked. It could also be possible for the runtime itself to expose a low-level exception handling callback, as it already registers its own VEH that it can dispatch from.

Risks

This is not a common scenario, and may be challenging to support in the runtime (I am not familiar with what would be required to support this kind of method).

[jkotas]

Implementations of vector exception handlers (and signal handlers on Unix) must follow similar set of restrictions as DllMain to be fully reliable. Managed code can run a lot of code related to GC suspension behind the scenes at any point. Even if you are careful to stay within restrictions in your code of the signal handler, you do not have control about the code that runs behind the scenes. The code that runs behind the scenes is not compatible with the DllMain and similar restricitions.

It is not possible to implement fully reliable vector exception handlers in managed code for this reason. This proposal won't change that.

If you need to implement vector exception handlers and signal handlers, you have to use C/C++ or similar language for that.

[huoyaoyuan]

A question here: is it actually possible to execute managed code in the callback without P/Invoke transitions to setup/restore frame information?

[DaZombieKiller]

similar set of restrictions as DllMain

From what I understand, for DllMain these restrictions stem from two things:

• The loader lock (any code that attempts to load another DLL will deadlock)
• Initialization being incomplete (you can touch state that has not been fully initialized yet)

With a VEH, I thought the primary restriction was that the VEH itself cannot throw an exception (though it could technically be invoked from an exception in DllMain, so I suppose it inherits DllMain's restrictions as well in a sense?).

Is that understanding correct?

[jkotas]

it could technically be invoked from an exception in DllMain, so I suppose it inherits DllMain's restrictions as well in a sense?

Right.

There are even scenarios where VEH can be called even before DllMain where thread static local storage is not allocated. We have this in the runtime VEH handlers to handle that.

[DaZombieKiller]

I see now why this is problematic then. For this to work reasonably the runtime essentially needs to support managed code in DllMain (or more accurately, running managed code anywhere, given an exception could occur in any context) which is not a simple change.

Is there any kind of compromise that could theoretically be made here? For example, is it possible to make this work in NativeAOT in some form (maybe by effectively having a "no-GC method" with a very restricted implementation)?

[Sewer56]

@DaZombieKiller As an extra note- I've tried this sort of thing when experimenting with Reloaded-II before; though for a different reasons- safety in hooking OS APIs where I can't guarantee if the caller is in cooperative or preemptive state. (Due to calls into native runtime which can skip transition and then call back into OS function).

I think around a year or two ago; I even wrote a small piece of docs on the issue tracker:

• [Question] Working around Thread Non-PreEmptive Check #82684

(Not strictly as an issue for the .NET team, but as a thing to point interested people towards when they asked me a question similar to this)

When I removed the assert from the assembly, things worked fine if the managed code contained no GC Polls; however encountering GC Polls risked causing the process to lock up. When I asked about the possibility for a 'no-GC' mode (which I was willing to accept), I was informed by Jan it'd be unlikely.

[jkotas]

is it possible to make this work in NativeAOT

We are not interested in building NativeAOT specific interop features like this. Our development story assumes that there is feature parity between JITed and AOTed runtimes.

As @Sewer56 pointed out, this is very niche feature that would be very complicated to build. I do not expect it to ever happen.

[DaZombieKiller]

That makes sense to me, thank you for the clarification.