Windows CI: intermittent STATUS_STACK_OVERFLOW in integration test binaries (interpreter needs ~2 MB of stack)

[logbie]

Summary

Integration Tests (blacksmith-4vcpu-windows-2025) fails intermittently with STATUS_STACK_OVERFLOW (exit code: 0xc00000fd). The binary that aborts varies between runs, which is what makes it look random:

Run	Binary that overflowed
438780a (on main)	concurrent_disconnect_burst_test
affd7eb (PR #676)	concurrent_disconnect_burst_test
f4f72dc (PR #676)	execute_file_test

In the f4f72dc log every individual test prints ok and the process then aborts, so it is stack exhaustion in the harness rather than one assertion failing.

Measurement: this is not new, and not caused by any recent feature work

RUST_MIN_STACK reproduces the Windows condition on Linux. Build normally first (setting it during the build makes rustc itself segfault), then run the built test binary under a constrained stack:

cargo test --test execute_file_test --no-run
RUST_MIN_STACK=1048576 ./target/debug/deps/execute_file_test-<hash> --test-threads=1

Comparing origin/main against the branch for PR #676, same test, same procedure:

Stack	origin/main	PR #676 branch
1,048,576 (1 MB)	overflow	overflow
1,200,000	overflow	overflow
1,400,000	overflow	overflow
1,600,000	overflow	overflow
1,800,000	overflow	overflow
2,097,152 (2 MB)	pass	pass

Identical at every size. The interpreter needs somewhere between 1.8 MB and 2 MB of stack to run execute_file_test today, on main, independent of any in-flight branch. I checked this specifically because the failure first appeared on a PR and looked like a regression; it is not one.

Why it is intermittent rather than constant

The requirement sits just above what the failing configuration provides, so whether a given run tips over depends on which binary runs and how deep its particular program nests. That also explains the varying test name and why main passes on some commits and failed on 438780a.

Root cause direction

Interpreter::execute_statement is a plain async fn that recurses through execute_block for every nested construct. Each await in any match arm enlarges the single generated state machine, and that cost is paid at every level of statement nesting. Deeply nested WFL programs — main loop → check → check → try → … — therefore consume stack quickly. This is a language-level limit, not a test artifact: a sufficiently nested user program will hit it too, and on Windows sooner.

Two directions, not mutually exclusive:

1. Reduce per-level cost. Box the larger arms of execute_statement so their locals do not inflate the shared state machine (PR Add transactions, AEAD encryption, file modes, and TOML support #676 does this for its own new arm, which is why that branch measures no worse than main despite adding a block-bearing statement). Applying the same treatment to the other block-bearing arms would lower the floor for everyone.
2. Give the harness room. Set RUST_MIN_STACK for the Windows CI job so the gate stops flapping while (1) is worked. Mitigation only — it does not help a user running a deeply nested program.

A guard test that runs a deeply nested program under a known stack size would keep the ceiling from quietly regressing.

Context

Found while driving PR #676 to green. Filed separately because it reproduces identically on main and is unrelated to that PR's changes.

[logbie]

A newer Windows failure (5960888) names the individual test, which sharpens the root cause considerably. The original report said "deeply nested programs consume stack". That is not what is happening.

The failing test is test_execute_file_depth_limit_prevents_infinite_recursion, and max_execute_file_depth is 4 (src/exec/budget.rs:150). So four levels of execute wfl file exhaust a 1 MB stack — the depth guard is working correctly and firing at 4, but the stack runs out first.

The cost is per nested interpreter run, not per nested block. Each execute wfl file level runs a complete lex → parse → analyze → typecheck → interpret pipeline inside the parent's already-recursing interpreter, so the frames stack pipeline-on-pipeline.

Measured, isolating that one test (build normally, then run the binary under a constrained stack):

Stack	origin/main	PR #676 branch
1 MB	overflow	overflow
2 MB	pass	pass
3 MB	—	pass
4 MB	—	pass

Identical on both, again. That single test accounts for the entire ~2 MB requirement I measured for the whole binary earlier — roughly 500 KB of stack per nested execute file level.

This changes what a fix should target. Boxing execute_statement's arms (direction 1 in the original report) trims per-block frames and is still worth doing, but it will not move this number much — the dominant cost is the nested pipeline per execute file. More promising:

• Make the execute file recursion heap-allocated (box the nested run's future) so depth costs a pointer rather than a full pipeline frame.
• Or derive max_execute_file_depth from available stack rather than a fixed 4, so the guard fires before the stack does on every platform.

Either way there is a user-facing version of this: a WFL program nesting execute wfl file four deep is within the documented limit and still crashes on Windows rather than getting the depth-limit error it is promised.

Found while driving PR #676; unrelated to it, and confirmed identical on main.

---

Generated by Claude Code