source-building .NET 11 vmr crashes during fsharp build on ppc64le

[tmds]

I'm doing builds of the vmr main (.NET 11) branch on ppc64le with a cross-compiled ppc64le SDK.

The builds are overflowing the stack.

I ran into this situation on s390x too. Here changing Mono to allocate 8MB of stack instead of 2MB fixed the issue.

I tried the same on ppc64le, but that did not help. I increased the stack size to 16MB, but the stack overflow remains.

SP:

(gdb) print/x $r1                                                                                                                                     
$1 = 0x7fffe8e20010                                                                                                                                   

proc maps:

7fffe8e10000-7fffe8e20000 ---p   (guard page)
7fffe8e20000-7fffe9e20000 rw-p   (stack)

Per #19285 (comment), the F# compiler recently changed to use .NET's RuntimeHelpers to check that stack space is remaining. This logic is meant to prevent these overflows.

cc @dotnet/fsharp @majocha @jkotas @akoeplinger

[tmds]

Per #19285 (comment), the F# compiler recently changed to use .NET's RuntimeHelpers to check that stack space is remaining. This logic is meant to prevent these overflows.

I'll take a closer look at this next week.

[jkotas]

RuntimeHelpers to check that stack space is remaining. This logic is meant to prevent these overflows.

Sounds like a bug in Mono implementation of these APIs.

[tmds]

Sounds like a bug in Mono implementation of these APIs.

I checked the behavior, and it seems to do something reasonable.

using System.Runtime.CompilerServices;

Thread thread = new Thread(ThreadFunction);
thread.Start();
thread.Join();


unsafe void ThreadFunction()
{
    int localVar = 42;
    nuint address = (nuint)(&localVar);
    Console.WriteLine($"Address of local variable: 0x{address:X}");

    string? region = FindRegionInProcMaps(address, out nuint startAddress, out nuint regionSize);
    Console.WriteLine($"Found in region: {region}. Size: {regionSize} bytes.");

    Console.WriteLine("\nTesting stack depth:");
    (uint callCount, nuint endAddress) = RecursiveStackTest();
    nuint stackUsed = address - endAddress;
    Console.WriteLine($"Recursive calls: {callCount}. Stack used: {stackUsed} bytes. Stack remanining: {endAddress - startAddress}. Bytes per frame: {stackUsed / callCount}.");
}

unsafe (uint, nuint) RecursiveStackTest()
{
    if (!RuntimeHelpers.TryEnsureSufficientExecutionStack())
    {
        int local = 42;
        nuint currentAddress = (nuint)(&local);

        return (1, currentAddress);
    }

    (uint count, nuint endAddress) = RecursiveStackTest();
    return (count + 1, endAddress);
}

static string? FindRegionInProcMaps(nuint address, out nuint startAddress, out nuint size)
{
    startAddress = 0;
    size = 0;
    try
    {
        string[] lines = File.ReadAllLines("/proc/self/maps");

        foreach (string line in lines)
        {
            string[] parts = line.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);

            if (parts.Length < 1)
                continue;

            string[] range = parts[0].Split('-');
            if (range.Length != 2)
                continue;

            nuint start = (nuint)Convert.ToUInt64(range[0], 16);
            nuint end = (nuint)Convert.ToUInt64(range[1], 16);

            if (address >= start && address < end)
            {
                startAddress = start;
                size = end - start;
                return line;
            }
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Error reading /proc/self/maps: {ex.Message}");
    }

    return null;
}

On ppc64le where I compiled Mono with a 16MB stack (instead of the default 2MB), the above outputs:

Address of local variable: 0x7FFF82AFDEF8
Found in region: 7fff81b00000-7fff82c00000 rw-p 00000000 00:00 0 . Size: 17825792 bytes.

Testing stack depth:
Recursive calls: 69048. Stack used: 16571568 bytes. Stack remanining: 197192. Bytes per frame: 240.

So we stay 192 KB away from the stack end.

The same program on x64 (with CoreCLR and 8MB stack) gives:

Address of local variable: 0x7FF7E91FEB1C
Found in region: 7ff7e8a00000-7ff7e9200000 rw-p 00000000 00:00 0 . Size: 8388608 bytes.

Testing stack depth:
Recursive calls: 64469. Stack used: 8252180 bytes. Stack remanining: 131080. Bytes per frame: 128.

So here we are 128 KB away from the stack end.

The difference is due to page size difference: 64K on ppc64le vs 4K on x64.

Note that for this reproducer the "Bytes per frame" is extremely small because there's very little on the stack. If I recall correctly when I looked at the backtrace last week the stack had about 33k frames, so the frame size would have been about double of what it is here.

The ppc64le machine I used last week to debug was a loan so I need to recreate my environment to debug it further.

Before I do that, I'll let our CI do a build with a 2MB stack and increased MONO_MIN_EXECUTION_STACK_SIZE by a factor of 4. This will make TryEnsureSufficientExecutionStack return sooner which may help if the problem is due to larger frame sizes.

[tmds]

I'll let our CI do a build with a 2MB stack and increased MONO_MIN_EXECUTION_STACK_SIZE by a factor of 4. This will make TryEnsureSufficientExecutionStack return sooner which may help if the problem is due to larger frame sizes.

Unfortunately this did not solve the issue. I'll re-create the setup so I can debug it further.

I'll use a build with the 2MB stack size with the increased MONO_MIN_EXECUTION_STACK_SIZE for the debugging.

[tmds]

The backtrace shows 2 functions alternating and consecutively calling themselves for about 160 times in FileContentMapping.fs at locations like:

fsharp/src/Compiler/Driver/GraphChecking/FileContentMapping.fs

Line 365 in e82d6f6

let rec visit (e: SynExpr) (continuation: FileContentEntry list -> FileContentEntry list) : FileContentEntry list =

and

fsharp/src/Compiler/Driver/GraphChecking/FileContentMapping.fs

Line 448 in e82d6f6

visit expr1 (fun nodes1 -> visit expr2 (fun nodes2 -> nodes1 @ nodes2 |> continuation))

Is what happening here also guarded using TryEnsureSufficientExecutionStack?

[majocha]

Is what happening here also guarded using TryEnsureSufficientExecutionStack?

Does not look so. But I think it was intended to be tail recursive, cps style (?)

[jkotas]

tail recursive

I assume that the code uses tail. prefix in IL. Is tail. prefix in IL instructions always respected by Mono?

[tmds]

I guess that if a tail. prefix is used and Mono supports it we shouldn't see 160 consecutive functions for the same function in the backtrace?

I'll let the F# experts answer whether a tail. prefix is used here, and may be a Mono expert can tell what the state of support for tail. is?

If the issue is indeed that Mono doesn't support tail., could we work around it by using the stack guarding that is used elsewhere in the F# compiler?

[tmds]

and may be a Mono expert can tell what the state of support for tail. is?

"expert" Claude:

Based on my exploration of the codebase, yes, Mono's support for the .tail prefix differs significantly per architecture. Here's what I found:

  Architecture Support Summary

  Basic Support Status

  Most architectures have basic tail call support via these flags defined in their header files:

  - x86/x86-64/amd64: ✅ Full support (MONO_ARCH_HAVE_OP_TAILCALL_MEMBASE, MONO_ARCH_HAVE_OP_TAILCALL_REG, MONO_ARCH_HAVE_VOLATILE_NON_PARAM_REGISTER = 1)
    - Locations: src/mono/mono/mini/mini-x86.h:228-229, mini-amd64.h:481-482, 518
  - ARM64: ✅ Full support (all three flags enabled)
    - Location: src/mono/mono/mini/mini-arm64.h:151-152, 191
  - ARM (32-bit): ⚠️  Partial support (has TAILCALL ops, but MONO_ARCH_HAVE_VOLATILE_NON_PARAM_REGISTER = 0)
    - Location: src/mono/mono/mini/mini-arm.h:346-347, 387
    - Cannot tail call with extra generic/interface context
  - s390x: ⚠️  Partial support (same limitation as ARM32)
    - Location: src/mono/mono/mini/mini-s390x.h:80-81, 159
  - RISC-V: ⚠️  Very limited (MONO_ARCH_HAVE_VOLATILE_NON_PARAM_REGISTER = 0, no TAILCALL op flags defined)
    - Location: src/mono/mono/mini/mini-riscv.h:119
  - PowerPC: ⚠️  Very limited (no TAILCALL op flags, MONO_ARCH_HAVE_VOLATILE_NON_PARAM_REGISTER = 0)
    - Location: src/mono/mono/mini/mini-ppc.h:275
  - WebAssembly: ❌ No support - mono_arch_tailcall_supported returns FALSE
    - Location: src/mono/mono/mini/mini-wasm.c:401-404

  Architecture-Specific Restrictions

  Each architecture has unique constraints in their mono_arch_tailcall_supported() implementations:

  x86/amd64 (mini-x86.c:799, mini-amd64.c:1606):
  - Stack frame limit: 1GB
  - AOT compilation: Cannot tail call non-virtual calls in partial AOT mode (due to PLT/GOT using non-volatile ebx)

  ARM32 (mini-arm.c:1824):
  - Stack frame limit: ~4KB (12-bit immediate offset addressing limitation)

  ARM64 (mini-arm64.c:3551):
  - Stack frame limit: 1GB
  - Cannot tail call when valuetype parameters are passed by reference

  RISC-V (mini-riscv.c:1405):
  - Cannot tail call vararg functions
  - Cannot tail call if any parameters are passed on stack
  - Cannot tail call with struct return semantics

  s390x (mini-s390x.c:7724):
  - Complex register-specific limitations (R6 register conflicts)
  - Various restrictions based on parameter types and stack usage

  Common Restrictions (All Architectures)

  The generic tail call logic in is_supported_tailcall() (src/mono/mono/mini/method-to-ir.c:5607-5721) also enforces:

  1. No virtual calls without have_op_tailcall_membase support
  2. No gsharedvt (generic value type sharing)
  3. No byref/pointer parameters (they may point to caller's stack)
  4. Return types must match exactly
  5. No tail calls in try/catch/finally blocks (mentioned in the readme.txt)

  Key Files

  - Generic logic: src/mono/mono/mini/method-to-ir.c:5607-5721
  - Per-arch implementations: src/mono/mono/mini/mini-{x86,amd64,arm,arm64,riscv,s390x,ppc,wasm}.c
  - Test documentation: src/mono/mono/tests/tailcall/readme.txt