[wasi][R2R] wasmtime OOM-killed at startup for System.Text.Json and RegularExpressions composites

[lewing]

Description

In the CoreCLR WASI composite ReadyToRun library lane (LibraryTestsCoreCLR_WASI_R2R, trimmed), wasmtime is killed with exit code 137 (SIGKILL, out of memory) about 4.5 minutes after start, before any test output, for:

• System.Text.Json.Tests
• System.Text.RegularExpressions.Tests

info: wasmtime 49.0.1 (46c23a87d 2026-09-24)
info: Running /root/helix/work/correlation/wasmtime/wasmtime
info: Process wasmtime exited with 137
fail: Application has finished with exit code 137 but 0 was expected

This is likely wasmtime compiling the very large composed module (the composite R2R code embedded in the per-app wasihost) and exceeding the Helix container's memory. Consistent across runs (Helix jobs e1cac54a-421c-439e-b0cd-8d98aff74c83 and eccd7d69-de48-4cc1-87fd-6036eb264d50). Related: #132855 (browser excludes Text.Json from R2R for function-type size).

Both suites are excluded from the WASI R2R lane in #134813.

Note

This issue was drafted with GitHub Copilot.

[dotnet-policy-service]

Tagging subscribers to 'arch-wasm': @lewing, @pavelsavara
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/crossgen-contrib
See info in area-owners.md if you want to be subscribed.