Concurrent shared compilations can fail releasing the client mutex on Linux

[JeremyKuhne]

Description

After Csc and Vbc opted into MSBuild multithreaded task execution in dotnet/roslyn#84701, a dotnet/sdk file-based-app test can intermittently fail when -mt builds two project references in the same MSBuild process with shared compilation enabled:

Microsoft.CSharp.Core.targets(97,5): error MSB3883: Unexpected exception:
Microsoft.CSharp.Core.targets(97,5): error : Cannot release a lock that is not owned by the current thread.

The failure is tracked by dotnet/sdk#56238. It occurred on Ubuntu 22.04 x64 with .NET 11 SDK 11.0.100-ci; test history reported a 0.52% failure rate. The failing Helix log shows LibraryB completing while another compiler invocation fails.

Distinguishing trigger

The failing test creates a file-based app with two #:project references and runs:

dotnet build Program.cs -mt

It also sets MSBUILDUSESERVER=0, so the MSBuild server is disabled while Roslyn shared compilation remains enabled. In the same 108-test work item, the ordinary file-based builds passed; only MultiThreadedArgument_BuildsProjectReferencesInProcess(0, -mt) failed. Its -mt:false row passed immediately afterward.

dotnet/sdk#56239 proposes UseSharedCompilation=false for this one test to isolate its MSBuild in-process-node assertion. That is useful test isolation, but it is a mitigation rather than a fix for this failure.

Relevant implementation and history

BuildServerConnection.RunServerBuildRequestAsync serializes server startup with a named client mutex. Its tryConnectToServerAsync local function is deliberately non-async so WaitOne and ReleaseMutex execute on the same thread. However, Roslyn's current tests exercise requests sequentially and do not appear to cover concurrent calls sharing the same pipe/client mutex from one process.

This resembles:

• ReleaseMutex failed. WaitOne Id: {originalThreadId} Release Id: {releaseThreadId} in Microsoft.Extensions.Logging.Generators.Roslyn4.0 roslyn#63863
• dotnet/runtime#53420, closed in July after the failure had not been observed following the .NET 11 wait-subsystem transition
• dotnet/dotnet#4088, where the same raw exception came from ServerNamedMutex.Dispose

The runtime synchronization correction associated with dotnet/dotnet#4088 is present in the runtime used by this failure, so this appears to be a recurrence exposed by the new concurrent compiler-task execution rather than simply a missing backflow.

The current failure also lacks Roslyn's WaitOne Id / Release Id diagnostic. tryConnectToServerAsync catches ApplicationException around mutex disposal, while the .NET 11 Unix implementation surfaced this ownership failure as InvalidOperationException in dotnet/dotnet#4088.

Requested investigation

1. Add a Linux regression/stress test that invokes RunServerBuildRequestAsync concurrently with the same pipe name, or otherwise runs concurrent shared-compilation Csc tasks.
2. Determine whether Roslyn is violating the mutex lifetime contract or whether this is a .NET 11 Unix named-mutex regression; transfer or split the runtime portion if needed.
3. Preserve the acquisition/release thread IDs when this exception occurs on .NET 11 so a future occurrence is actionable.

Note

This report was prepared with GitHub Copilot assistance from the linked failure logs and source investigation.

[JanProvaznik]

Local investigation found an independently reproducible .NET 11 Linux named-mutex ownership defect, and captured a failure through the actual shared-compilation Csc path. This supports a runtime fix, with separate Roslyn regression coverage and diagnostics work.

Observed product failure

Environment: Ubuntu 24.04 x64 under WSL, SDK 11.0.100-rc.1.26453.118, runtime 11.0.0-rc.1.26453.118, MSBuild 18.12.0-1.26453.118, and Roslyn build tasks 5.12.0-1.26453.118. All report VMR commit 3c8d132bfb012af4bbba80de22149a9f58e737f0. The product binaries were not modified.

An ordinary project with 32 project references failed on repetition 7 of:

dotnet build App.csproj -mt -m:16 -nr:false -t:Rebuild \
  -p:PublishAot=false -p:SelfContained=false -p:UseSharedCompilation=true \
  /bl:215.binlog -v:minimal

The binlog identifies Lib17.csproj / CoreCompile / Csc, with MSB3883 and Cannot release a lock that is not owned by the current thread. at Microsoft.CSharp.Core.targets(97,5).

A first-chance exception handler captured the live stack before Roslyn caught the exception (abridged; arguments omitted):

System.Threading.NamedMutexProcessDataBase.ReleaseLock()
System.Threading.WaitSubsystem.ReleaseMutex(...)
System.Threading.Mutex.ReleaseMutex()
Microsoft.CodeAnalysis.CommandLine.ServerNamedMutex.Dispose()
Microsoft.CodeAnalysis.CommandLine.BuildServerConnection
    .<RunServerBuildRequestAsync>g__tryConnectToServerAsync|11_0(...)
Microsoft.CodeAnalysis.CommandLine.BuildServerConnection.RunServerBuildRequestAsync(...)
Microsoft.CodeAnalysis.BuildTasks.ManagedCompiler.ExecuteTool(...)
Microsoft.Build.Utilities.ToolTask.Execute()
Microsoft.Build.BackEnd.TaskExecutionHost.Execute()

A native observer also recorded the first pthread_mutex_trylock following initialization of a shared, robust mutex on that thread:

Event	UTC, 2026-09-15	Identity
Native acquisition returned EBUSY (16)	11:53:02.507844650	PID 1397, native TID 1450
The above Roslyn release path threw InvalidOperationException	11:53:02.5135684	PID 1397, native TID 1450, managed TID 42

/proc/self/maps identified the backing file as:

/tmp/.dotnet/shm/global/GoEi2zpuG0OBx0b3dEif7FaSnKpUMJH0suB8sxbMWKw.client

The observer calls the real pthread functions and returns their results unchanged; it does not deliberately schedule threads, sleep, retry, or manufacture failures. Instrumentation can affect timing, and the interval above includes diagnostic overhead.

This is a local product-build reproduction, not a recovered trace of the original Helix occurrence. The smaller two-reference build passed 200 repetitions. The product trace does not directly capture the competing thread's complete sequence or the managed constructor's createdNew output.

Source-derived diagnosis

In the exact runtime revision:

1. CreateOrOpen publishes the newly initialized mutex in the process-local cache.
2. TryAcquireLock releases the creation/deletion process lock before native acquisition.
3. Another thread in the same process can find the cached object, avoiding the creation/deletion file lock, and acquire the mutex first.
4. The creator's initial zero-timeout acquisition can return TimedOut. CreateOrOpen checks success only with Debug.Assert, then returns with createdNew=true.

A separate controlled runtime experiment observed that last condition directly: the constructor returned createdNew=true after its real native trylock returned EBUSY; release on the original thread then failed.

This contradicts the documented constructor contract: initiallyOwned=true grants ownership if this call creates the named mutex. Multiple wrappers for the same named mutex are explicitly supported.

For comparison, .NET 10's native implementation acquires initial ownership while retaining the creation/deletion locks.

Uninstrumented runtime reproducer

The following tested program uses two threads, one fixed mutex name, and no barriers, sleeps, yields, async code, or native instrumentation:

Runtime on the same Linux host	Result
.NET 11 11.0.0-rc.1.26453.118	Release failed after 4,476 successful operations; created=true, acquisition and release managed thread IDs both 4
.NET 10 10.0.11	All 200,000 operations completed

These are stress observations, not estimates of customer failure rates.

Complete minimal project and program

MutexNatural.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
</Project>

Program.cs:

using System.Runtime.InteropServices;

int iterations = args.Length > 0 ? int.Parse(args[0]) : 100000;
int workerCount = args.Length > 1 ? int.Parse(args[1]) : 2;
string name = @"Global\roslyn85264-natural-" + Guid.NewGuid().ToString("N") + ".client";
int failures = 0;
int createdCount = 0;
int completedCount = 0;
int inside = 0;
int overlapCount = 0;
int protectedValue = 0;
Console.WriteLine($"{RuntimeInformation.FrameworkDescription}; {RuntimeInformation.OSDescription}; workers={workerCount}; iterationsPerWorker={iterations}");
Console.WriteLine($"One name for the entire run: {name}");
Console.WriteLine("No barriers, sleeps, yields, SpinWait, async code, or native instrumentation.");
var threads = Enumerable.Range(0, workerCount).Select(worker => new Thread(() =>
{
    for (int iteration = 0; iteration < iterations && Volatile.Read(ref failures) == 0; iteration++)
    {
        bool created = false;
        string phase = "construct";
        int acquireThread = Environment.CurrentManagedThreadId;
        try
        {
            using var mutex = new Mutex(initiallyOwned: true, name, out created);
            if (created)
                Interlocked.Increment(ref createdCount);
            phase = "wait";
            if (!created)
            {
                try
                {
                    if (!mutex.WaitOne(TimeSpan.FromSeconds(10)))
                        throw new TimeoutException("Mutex acquisition timed out.");
                }
                catch (AbandonedMutexException)
                {
                    // WaitOne grants ownership when reporting abandonment.
                }
            }
            phase = "critical section";
            if (Interlocked.Increment(ref inside) != 1)
                Interlocked.Increment(ref overlapCount);
            protectedValue++;
            Interlocked.Decrement(ref inside);
            phase = "release";
            mutex.ReleaseMutex();
            Interlocked.Increment(ref completedCount);
        }
        catch (Exception ex)
        {
            Interlocked.Increment(ref failures);
            Console.WriteLine($"FAIL worker={worker} iteration={iteration} created={created} phase={phase} acquireThread={acquireThread} releaseThread={Environment.CurrentManagedThreadId}: {ex}");
        }
    }
})).ToArray();
foreach (Thread thread in threads)
    thread.Start();
foreach (Thread thread in threads)
    thread.Join();
Console.WriteLine($"Completed={completedCount}; created={createdCount}; failures={failures}; overlapCount={overlapCount}; protectedValue={protectedValue}");
return failures == 0 && overlapCount == 0 ? 0 : 1;

Build using a .NET 10 SDK, then run the same DLL with each runtime:

dotnet build MutexNatural.csproj -c Release /bl:1.binlog
/path/to/dotnet10/dotnet bin/Release/net10.0/MutexNatural.dll 100000 2
DOTNET_ROLL_FORWARD=Major DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 \
  /path/to/dotnet11/dotnet bin/Release/net10.0/MutexNatural.dll 100000 2

The .NET 11 SDK used for the product trace is available at:
https://ci.dot.net/public/Sdk/11.0.100-rc.1.26453.118/dotnet-sdk-11.0.100-rc.1.26453.118-linux-x64.tar.gz

Suggested split and remaining work

• Runtime: track the identified creation/initial-ownership race separately. The proposed fix is to retain the process lock through the new mutex's nonblocking initial acquisition and ownership bookkeeping, without retaining it across ordinary blocking waits or introducing recursive acquisition of the nonrecursive process lock. This proposal has not been validated against a patched runtime.
• Roslyn: add checked-in Linux concurrent compiler-client coverage and extend the existing acquisition/release thread-ID diagnostic to include InvalidOperationException as well as ApplicationException. The current catch misses this runtime exception.
• A possible Roslyn mitigation is explicit, synchronous acquisition of the client mutex without requesting constructor-time ownership. This would need to preserve timeouts, abandonment, and disposal semantics; it is not sufficient to flip the shared wrapper's constructor flag because server-election and cache callers also depend on its ownership semantics.

The fix associated with dotnet/dotnet#4088 is present: it made the creation/deletion file lock blocking. It does not exclude the same-process cache-hit interleaving above. #53420 is relevant symptom history, but the pre-.NET-11 failures are not explained by this diagnosis; its closure comment explicitly requests a new issue for recurrence.

No production fix or checked-in Roslyn regression test has been made by this investigation. The full local binlog/capture bundle is not attached to this comment; the product trace excerpts and runnable runtime reproducer are included above.

_{Investigation and write-up performed with GitHub Copilot. Observed results, source-derived diagnosis, and unvalidated proposals are distinguished above.}

[JanProvaznik]

@jkoritzinsky Does this analysis sound relevant and reasonable? Is this indeed a runtime bug introduced with the 10->11 change you implemented?

[jkoritzinsky]

The analysis sounds reasonable. I do remember changing the process lock rules here because we hit a deadlock. Will see what change I need to make to get this working.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @VSadov
See info in area-owners.md if you want to be subscribed.

[EgorBo]

Ugh. My EgorBot bot keeps failing because of this bug - I assume we still use the buggy bootstrap SDK.