[browser][CoreCLR][Composite R2R] Int128 benchmark startup aborts with an invalid reverse P/Invoke transition #135368

Description

Wasm CoreCLR composite ReadyToRun execution fails during startup of BenchmarkDotNet benchmarks accepting Int128 arguments. The process aborts with:

Fatal error.
Invalid Program: attempted to call a UnmanagedCallersOnly method from managed code.

The same Perf_Int128.TryFormat(Int128.MinValue) benchmark produces valid measurements with noncomposite ReadyToRun using the same runtime revision, performance revision, BenchmarkDotNet version, and V8 version.

This is a correctness/startup failure, not a performance regression report. The exact offending method and root cause have not yet been identified.

Reproduction Steps

Existing CI reproduction

The failure is retained in dotnet-runtime-perf build 3097911, 20261006.8 (internal build; authentication may be required).

Use:

• Runtime revision: aa595bc05b926d9e5129d13ac82ac7d679a9f2fd.
• Performance revision: de35ca24c8ca1417d24197dc031bf73d1134adf7.
• Pipeline lane: coreclr_r2r_composite_v8.
• Benchmark: System.Tests.Perf_Int128.TryFormat(value: -170141183460469231731687303715884105728).

Failing Helix job: 2df62f0a-7b2c-43d8-86af-3dcf4e1cd74d.
Work item: wasm.x64.micro.net11.0.Partition4.
Original console log.

The CI harness uses benchmarks_ci.py with:

--architecture x64 -f net11.0
--run-isolated --wasm
--wasm-runtime-flavor CoreCLR
--wasm-ready-to-run-composite
--partition=4
--bdn-arguments=--anyCategories Libraries Runtime
  --wasmEngine <v8-15.4.80>
  --cli <original-CI-dotnet>
  --buildTimeout 1200
  --wasmProcessTimeout 20
  "--wasmArgs=--experimental-wasm-exnref"
  --category-exclusion-filter NoWasmCoreCLR NoWASM NoMono
  --logBuildOutput --generateBinLog
  --partition-count 15 --partition-index 4

This is an excerpt of the original arguments, not a standalone installation command. It requires the CI-staged SDK, runtime/tool packs, and Crossgen2Tasks shim. The tested runtime package version is 12.0.0-ci; a stock released SDK/workload is not an equivalent reproduction.

The generated application's failing invocation is:

v8-15.4.80 --experimental-wasm-exnref --module main.mjs -- \
  --run MicroBenchmarks-Wasm-1.dll \
  --ipcDir <generated-ipc-directory> \
  --benchmarkName "System.Tests.Perf_Int128.TryFormat(value: -170141183460469231731687303715884105728)" \
  --job Wasm --diagnoserRunMode 3 --benchmarkId 331

Independent local reproduction

The original SDK/runtime/tool package cohort, performance revision, BenchmarkDotNet nightly, and V8 version were also used to build and execute the actual Perf_Int128 BenchmarkDotNet-generated program under WSL, with generated files retained.

With the retained BrowserWasmCoreCLR artifact from build 3097911 unpacked, the effective environment and command were as follows. Replace the placeholder paths with the extracted original cohort, V8 binary, and writable output locations; invoke the cohort's dotnet, not a stock SDK:

export RestoreAdditionalProjectSources=<cohort>/staging/built-nugets
export PERFLAB_WASM_PACKAGE_VERSION=12.0.0-ci
export PERFLAB_WASM_READY_TO_RUN=true
export PERFLAB_WASM_READY_TO_RUN_COMPOSITE=true
export Crossgen2SdkOverridePropsPath=<cohort>/staging/Crossgen2Tasks/Microsoft.NET.CrossGen.props
export Crossgen2SdkOverrideTargetsPath=<cohort>/staging/Crossgen2Tasks/Microsoft.NET.CrossGen.targets

<cohort>/staging/dotnet-none/dotnet run \
  --project src/benchmarks/micro/MicroBenchmarks.csproj \
  -c Release -f net11.0 -- \
  --filter '*Perf_Int128*' \
  --runtimes wasmnet11_0 --wasmRuntimeFlavor CoreCLR \
  --wasmEngine <v8-15.4.80-d8> \
  --cli <cohort>/staging/dotnet-none/dotnet \
  --artifacts <output> --packages <packages> \
  --job Dry --keepFiles --logBuildOutput --generateBinLog \
  --buildTimeout 1200 --wasmProcessTimeout 2

Of 19 benchmarks:

• All seven Int128-argument cases failed with the fatal message and produced no results: ToString and TryFormat for MinValue, 12345, and MaxValue, and CopySign(1, -1).
• All 12 string-input Parse/TryParse/Span cases produced measurements.
• The benchmark run returned exit code 1.

No local case hung; the 20-minute hang below was observed in the original Helix job.

An independently constructed small marked composite-R2R program exercising Int128 boxing/unboxing, integer-to-Int128 conversions, constructor initialization, and no-op Int128 calls passed on the same package cohort and V8. Therefore that smaller program is a negative control, not a minimal failing reproduction. The currently verified failing reproduction still involves BenchmarkDotNet's generated program.

Expected behavior

The generated runnable should initialize successfully and execute the Int128 benchmarks, producing measurements as it does under noncomposite ReadyToRun.

Actual behavior

The original work item reports:

// BeforeAnythingElse
console.error: Fatal error.
console.error: Invalid Program: attempted to call a UnmanagedCallersOnly method from managed code.
ProcessData operation is interrupted by EarlyProcessExit.
No Workload Results were obtained from the run.
// Benchmark Process 56065 has exited with code 1.

No runtime-environment banner or workload measurements appear for the failing case. The benchmark harness continues with other benchmarks, then returns failure.

Six partitions on six distinct machines failed in the original composite job:

The timeout case is not claimed to have emitted the fatal message.

No actual fatal-error stack or dump has been captured. The retained CI bdn-artifacts.zip contains reports and the benchmark log, but not the generated application or a crash dump.

Regression?

Unknown. There is no verified last-good composite-R2R revision.

Composite performance coverage was added by dotnet/performance#5324, merged October 6, 2026 at 08:14 UTC. Its merge revision is the performance revision used in the failing build. dotnet/runtime#135204 supplied the Crossgen2Tasks support for that lane.

The preceding sampled build, 3096834 (20261005.8, runtime 4cc1f5a863b20a3f06b7d878f256b68c4259cece, queued October 6 at 06:57 UTC), did not have a composite lane.

The next sampled build, 3097041 (20261006.1, runtime 7ede41e0cbfd6d7d2fbabb66ede45118f076c6af, queued October 6 at 11:22 UTC), already shows the identical TryFormat(Int128.MinValue) fatal error in composite job 284dc3ea-dd04-41ab-90fa-b613c4f4cf40, Partition4.

This brackets the introduction of coverage, not the introduction of the runtime defect. Earlier passing noncomposite jobs are not a last-good composite baseline.

Known Workarounds

Use noncomposite ReadyToRun for this benchmark configuration.

In the same original build 3097911, noncomposite job ccc81e0d-4747-438a-a746-96626920785d, Partition4, executed TryFormat(Int128.MinValue) successfully with 15 measurements and a mean of 275.103 ns.

Passing noncomposite console log.

This is verified for that benchmark/configuration, not a claim that all composite-R2R failures can be worked around this way.

Configuration

• Runtime: Wasm CoreCLR, reported as 12.0.0-ci, composite ReadyToRun.
• Runtime source: aa595bc05b926d9e5129d13ac82ac7d679a9f2fd, dotnet/runtime main.
• Performance source: de35ca24c8ca1417d24197dc031bf73d1134adf7.
• SDK actually used: 12.0.100-alpha.1.26471.108.
• Host runtime: 12.0.0-alpha.1.26471.108.
• Target framework: net11.0 (the work-item label and environment SDK request should not be confused with the actual SDK/runtime used).
• Wasm runtime/tool packages: 12.0.0-ci, with the CI-staged Crossgen2Tasks shim.
• BenchmarkDotNet: 0.16.0-nightly.20260801.595.
• JavaScript engine: V8 15.4.80, run via d8/jsvu, not a browser.
• CI host: Linux Ubuntu 22.04.5 LTS, x64; AMD EPYC 9124.
• Local independent reproduction: Linux under WSL, using the retained original package cohort and exact V8 version.
• CI configuration metadata: CompilationMode=wasm;RunKind=micro;RuntimeType=coreclr;R2RType=r2r_composite.

Other information

The fatal text corresponds to a reverse P/Invoke transition guard. In the tested runtime source, JIT_ReversePInvokeEnterRare emits it when the existing thread is already in cooperative GC mode. An interpreter entry path can emit the same text. The message alone does not establish which method was entered, whether it was incorrectly marked, or whether entrypoint selection or an earlier missing transition caused the invalid state.

The failures occur before BenchmarkDotNet prints the environment. Generated runnable construction/argument setup and R2R preparation are investigation candidates, but the passing smaller Int128 control means plain Int128 materialization alone is insufficient to explain the failure.

runtime#123008 reports the same message in browser CoreCLR tests and is closed, but its stack and scenario differ. No equivalence or common fixing commit has been established.

The console also prints Warning: unknown flag --experimental-wasm-exnref. That warning appears in the successful noncomposite control as well. V8 15.4.80 lists exnref as always enabled with no flag, and the obsolete performance argument has a separate cleanup in dotnet/performance#5253. It is not established as a cause of this failure.