Mono: a filter or catch nested in the handler of an earlier-listed clause is compiled in the enclosing handler's region (access violation on windows-x64, hang on arm64, LLVM AOT assert) #135641

Description

Mono's JIT mishandles a method whose EH table lists a clause before another clause nested in the first clause's handler. The result is an access violation on windows-x64, a hang on linux-arm64 and osx-arm64, and an assertion in the LLVM AOT compiler. I am preparing a patch (see Other information).

This order is valid: ECMA-335 I.12.4.2.5 constrains the order of nested try blocks only, and CoreCLR's JIT sorts the table to accept it (fgSortEHTable). In Mono, compute_bb_regions (method-to-ir.c) marks regions in table order, and mark_bb_in_region keeps the first handler region a block receives. The blocks of the nested filter or catch handler therefore keep the enclosing handler's region, and the method is compiled as if they were part of the enclosing handler.

mono_find_block_region_notry (mini.c), ip_get_finally_clause_index and the CEE_RETHROW handler lookup (both in method-to-ir.c) also return the first clause that matches. The comment on ip_get_finally_clause_index states the assumption: "The clauses are ordered from inner to outer, so the first match is the innermost one."

This is the Mono counterpart of #135592. It is a separate bug from #135639, which is about invalid IL; the IL here is valid. The CoreCLR fix, #135601, adds the test JIT/Regression/JitBlue/Runtime_135592, which exposed this bug in that PR's CI (build 1631268):

• mono windows-x64 minijit: Runtime_135592 crashed with exit code 0xC0000005, failing the Regression_5 work item.
• mono linux-arm64 minijit: Runtime_135592 hung until the watchdog stopped it (test exit code 110), failing the Regression_5 work item.
• mono llvmaot (linux-x64): the "AOT compile CoreCLR tests" step failed with * Assertion at src/mono/mono/mini/mini-llvm.c:12710, condition 'lhs' not met (OP_ENDFINALLY). When each scenario is compiled as its own assembly, the LLVM AOT compiler asserts only on CatchInEarlierFinally (a catch nested in the handler of an earlier-listed finally).
• mono osx-x64 minijit and osx-x64 interpreter passed.

I am disabling Runtime_135592 on Mono in #135601, with a link to this issue. JIT/Regression/Dev11/Dev11_468598/Test_HndIndex_10_Reordered has the same shape as CatchInEarlierFinally, and it is already disabled on Mono (ActiveIssue "needs triage" and MonoAotIncompatible). The LLVM AOT compiler aborts on it too.

Reproduction Steps

Two of the scenario methods from Runtime_135592 (assemble with ilasm -dll):

.assembly extern System.Runtime { .publickeytoken = (B0 3F 5F 7F 11 D5 0A 3A ) }
.assembly NestedRegion { }
.module NestedRegion.dll

.class public auto ansi abstract sealed beforefieldinit Repro
       extends [System.Runtime]System.Object
{
  // Returns 1 when the nested filter selects the nested handler.
  .method public hidebysig static int32 FilterInEarlierFinally() cil managed noinlining
  {
    .maxstack 2
    .locals init (int32 result)
    ldc.i4.0
    stloc.0
  OuterTry:
    leave Done
  FinallyStart:
    nop
  InnerTry:
    newobj instance void [System.Runtime]System.Exception::.ctor()
    throw
  FilterStart:
    isinst [System.Runtime]System.Exception
    ldnull
    cgt.un
    endfilter
  HandlerStart:
    pop
    ldc.i4.1
    stloc.0
    leave FinallyEnd
  FinallyEnd:
    endfinally
  Done:
    ldloc.0
    ret
    .try OuterTry to FinallyStart finally handler FinallyStart to Done
    .try InnerTry to FilterStart filter FilterStart handler HandlerStart to FinallyEnd
  }

  // Returns 1 when the nested catch runs.
  .method public hidebysig static int32 CatchInEarlierFinally() cil managed noinlining
  {
    .maxstack 1
    .locals init (int32 result)
    ldc.i4.0
    stloc.0
  OuterTry:
    leave Done
  FinallyStart:
    nop
  InnerTry:
    newobj instance void [System.Runtime]System.Exception::.ctor()
    throw
  CatchStart:
    pop
    ldc.i4.1
    stloc.0
    leave FinallyEnd
  FinallyEnd:
    endfinally
  Done:
    ldloc.0
    ret
    .try OuterTry to FinallyStart finally handler FinallyStart to Done
    .try InnerTry to CatchStart catch [System.Runtime]System.Exception handler CatchStart to FinallyEnd
  }
}

Call FilterInEarlierFinally on Mono. The test also has a control, FilterInEarlierFinally_InnerFirst: the same IL with the two .try lines swapped, so that the nested clause is listed first. It passes in every configuration below.

Expected behavior

Both methods return 1.

Actual behavior

I called each of the four scenarios of the compiled Runtime_135592 test by reflection, using the shipped Mono 8.0.28 runtime packs in a self-contained net8.0 host with UseMonoRuntime. Each run used a separate process with a 30 s timeout:

linux-x64 passes, but its region assignment is wrong all the same. With MONO_VERBOSE_METHOD=FilterInEarlierFinally, a Mono build of main (523a36f9) puts the filter block (IL_0016) and its handler (IL_001b) in region 0x112, the finally of clause 0. In FilterInEarlierFinally_InnerFirst, the filter and handler get their own regions (0x181 and 0x121). In CatchInEarlierFinally, the catch handler (IL_0011) is also in region 0x112.

Regression?

Not that I know of. Mono 8.0.28 behaves the same.

Known Workarounds

List the nested clause before the clause whose handler contains it.

Configuration

Shipped Mono 8.0.28 runtime packs on windows-x64, linux-arm64 (WSL2) and osx-arm64; linux-x64 (Ubuntu) with 8.0.28 and with a Mono Release build of main 523a36f9. CI: build 1631268 of #135601.

Other information

A possible fix: let a strictly nested handler or filter region replace the enclosing one in mark_bb_in_region, and pick the innermost enclosing handler in the three lookups above. I am preparing a patch along those lines. In local testing on linux-x64 it gives the nested filter and handler their own regions (0x281 and 0x221), and the LLVM AOT compiles of Runtime_135592 and Test_HndIndex_10_Reordered succeed. On osx-arm64, with a Mono build of main running the whole test under corerun, Runtime_135592 passes in 20 of 20 runs with the patch; without it, all 20 runs hit a 60 s timeout.