[mono][aot] Fix stack overflow normalizing self-referencing struct wrappers

[pavelsavara]

Summary

mono-aot-cross crashed with a native stack overflow (0xC00000FD / exit code -1073741571) on any assembly with a struct that has a field of a generic struct instantiated over the struct itself:

public struct Box<T> { public object? O; }
public struct Self { public Box<Self> F; }

public static class Calls
{
    public static T Id<T>(T value) => value;
    public static Self Call(Self value) => Id(value);
}

Root cause

When building gsharedvt wrapper signatures, get_wrapper_shared_vtype() normalizes each instance field of a struct. get_wrapper_shared_type_full() normalizes each type argument of a generic struct and calls back into get_wrapper_shared_vtype() for struct arguments. For Self -> field Box<Self> -> type argument Self -> ..., nothing stops the recursion.

Generic constraints are not involved, even though the shape in the issue has them; the cycle through the field type argument is enough.

Fix

Pass a stack-allocated chain of the structs whose layout is currently being normalized through both functions. If a struct is reached again while already in the chain, get_wrapper_shared_vtype() returns NULL and the original type is kept. That is the existing "do not share this valuetype" outcome. A depth limit of 16 also bounds shapes that create a new type on every level, such as struct S<T> { Box<S<S<T>>> F; }.

Testing

Tested with a Debug mono-aot-cross built from this branch (-os browser -subset mono+libs -c Debug), comparing against the shipped 10.0.11 compiler:

Input	Shipped 10.0.11	This branch
Minimal library (self-referencing, mutual, growing shapes)	stack overflow	exit 0
Reporter's Blazor repro, app assembly	stack overflow	exit 0
Reporter's Blazor repro, aot-instances.dll (dedup, 31 assemblies)	stack overflow	exit 0

• Added ValueTypeTests.StructWithSelfReferencingGenericFieldPassedToGenericMethodTest with the self-referencing and mutual shapes. The browser AOT leg compiles System.Runtime.Tests, so before this fix the test assembly itself would crash the AOT compiler.
• System.Tests.ValueTypeTests on browser Mono with V8 (interpreter): 23/23 passed.
• Not run locally: a full WASM AOT build and run of System.Runtime.Tests; that is left to the CI browser AOT leg.

Resolves #132071

Note

This PR description was drafted with GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[pavelsavara]

/azp run runtime-wasm

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[pavelsavara]

/ba-g unrelated CI failures

[pavelsavara]

/backport to release/11.0

[github-actions]

Started backporting to release/11.0 (link to workflow run)