Add proactive stack overflow detection via MaxCallDepth + tests

[ModMaker101]

Detect excessive call depth and insufficient native stack before invoking LuaFunction delegates, including direct C# RunAsync reentry, hooks and metamethods. The original guards on VM CALL/TAILCALL also ran for pure Lua calls, which stay inside the VM loop, while direct C# reentry had no guard.

• Add a positive MaxCallDepth policy, shared with existing and newly created coroutines. The default is 100,000 frames per thread. Lua and C# frames count; tail calls reuse a frame.
• Check native stack at the common delegate invocation entry. Retain the existing VM entry check. RuntimeHelpers' native stack check is a runtime heuristic, rather than an exact byte limit.
• Replace tail-call frames in place, preserving pending exception traces and frame depth.
• Make LuaStackOverflowException public. VM execution may wrap it in LuaRuntimeException; it remains available as InnerException.
• Reject zero/negative limits, remove the redundant pcall exception branch, and preserve wrapped C# exception messages in coroutine.resume without changing error(nil).

Validation on .NET 8 x64 Windows: 304 runtime tests (excluding ExpectedFailure), one source-generator test, and Release builds for net10.0/net8.0/net6.0/netstandard2.1 passed. Tests cover real non-tail recursion, tail-call depth, shared coroutine policy, nil errors, and monitored 1 MiB native-stack reentry through both direct C# and Lua calls. Missing guards fail the test without deliberately overflowing the process stack.

Performance comparison against main 991c4fc with BenchmarkDotNet 0.14.0, two launches, five warmups, twelve iterations, and 500 ms iterations. Scripts are compiled in setup and results are validated. These four call-heavy workloads are not a general application benchmark:

Workload	main	revised PR	Difference
100k Lua calls	4.110 ms	4.110 ms	approximately unchanged
420k non-tail recursive calls	20.984 ms	20.971 ms	approximately unchanged
420k tail calls	16.829 ms	16.181 ms	3.9% faster
100k C# calls, separate repeated run	2.655 ms	2.756 ms	3.8% slower

The initial revised C# run was unstable (2.8–5.0 ms); the separate repeat was stable at 2.756 ± 0.0155 ms (99.9% confidence interval). Native-stack protection has a measurable C# call cost. This small cost provides protection for direct C# reentry that main did not check, while ordinary Lua calls remain approximately unchanged and tail recursion improves.

GitHub Ubuntu CI lint and runtime tests also passed on b1453e3 (run 38039314105). The native-stack regression test allows the callback's own frame to cross the runtime's reserve threshold once; the next guarded invocation must throw, and a second low-stack entry stops the test safely if protection is missing.

[ModMaker101]

And solves #168.

[ModMaker101]

@nuskey8 Mind reviewing this?

[Copilot]

Pull request overview

Adds configurable, proactive stack overflow protection to the Lua runtime to avoid process-terminating StackOverflowException and support sandboxing by enforcing a maximum Lua call depth and surfacing a catchable LuaStackOverflowException.

Changes:

• Introduces LuaState.MaxCallDepth and enforces it when pushing call stack frames.
• Adds RuntimeHelpers.TryEnsureSufficientExecutionStack() checks in VM CALL / TAILCALL.
• Makes LuaStackOverflowException public and adds test coverage for max call depth behavior (including pcall).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/Lua.Tests/StackOverflowTests.cs	Adds tests validating max call depth enforcement and pcall behavior.
src/Lua/Standard/BasicLibrary.cs	Updates pcall exception handling to return a stack overflow message.
src/Lua/Runtime/LuaVirtualMachine.cs	Adds stack-availability checks on CALL and TAILCALL.
src/Lua/LuaState.cs	Adds MaxCallDepth and enforces it in PushCallStackFrame.
src/Lua/Exceptions.cs	Makes LuaStackOverflowException public for user catchability.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[akeit0]

I would appreciate it if you could measure the performance changes using recursive call benchmarks.