Fix lock inversion in HttpListener's Windows WebSocket impl

[MihaZupan]

Closes #115559

The TakeLocks helper used elsewhere first takes the sessionHandle lock, then the thisLock.
In these cases, we would call the helper while only holding thisLock, potentially deadlocking with other calls.

runtime/src/libraries/System.Net.HttpListener/src/System/Net/Windows/WebSockets/WebSocketBase.cs

Lines 881 to 888 in cdcc43c

	private void TakeLocks(ref bool thisLockTaken, ref bool sessionHandleLockTaken)
	{
	Debug.Assert(_thisLock != null, "'_thisLock' MUST NOT be NULL.");
	Debug.Assert(SessionHandle != null, "'SessionHandle' MUST NOT be NULL.");
	Monitor.Enter(SessionHandle, ref sessionHandleLockTaken);
	Monitor.Enter(_thisLock, ref thisLockTaken);
	}

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @karelz, @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR addresses a potential deadlock in the Windows HttpListener WebSocket implementation by enforcing a consistent lock acquisition order between the state lock (_thisLock) and the SessionHandle lock, and adds a regression test intended to exercise the problematic concurrent-close interleaving.

Changes:

• Update WebSocketBase.CloseAsyncCore to release _thisLock before starting operations that acquire the SessionHandle lock (CloseOutput, receive processing, Abort).
• Add a Windows-only regression test that concurrently triggers a client close frame and a server CloseAsync to detect deadlocks.
• Add a test helper overload to create a HttpListenerWebSocketContext using a provided ClientWebSocket.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/libraries/System.Net.HttpListener/src/System/Net/Windows/WebSockets/WebSocketBase.cs	Adjusts locking behavior in CloseAsyncCore to avoid lock inversion with SessionHandle-locked receive processing.
src/libraries/System.Net.HttpListener/tests/HttpListenerWebSocketTests.cs	Adds a concurrency regression test and a GetWebSocketContext(ClientWebSocket) helper for setup.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/libraries/System.Net.HttpListener/src/System/Net/Windows/WebSockets/WebSocketBase.cs:648

• CloseAsyncCore now releases _thisLock and then calls CloseOutputAsync(...). If another thread progresses the close handshake in the small window after the lock is released (e.g., state becomes CloseSent), CloseOutputAsync can throw a WebSocketException synchronously (via ThrowOnInvalidState before its first await). That synchronous throw skips the existing closeOutputTask exception handling and flows into the outer catch, which calls Abort() and rethrows. Consider catching WebSocketError.InvalidState around the CloseOutputAsync call and treating it as a benign race when the state has already advanced to CloseSent/terminal, while ensuring _thisLock is re-taken before proceeding.

                        ReleaseLock(_thisLock, ref lockTaken);

                        closeOutputTask = CloseOutputAsync(closeStatus,
                            statusDescription,
                            linkedCancellationToken);

src/libraries/System.Net.HttpListener/tests/HttpListenerWebSocketTests.cs:405

• This test currently swallows all WebSocketException / InvalidOperationException from the three concurrent tasks, which can make the test pass even if the operations fail immediately for an unexpected reason (i.e., it only fails on timeout). Tightening the catch filter to only ignore cancellation/disposal would make the test assert more than just “no deadlock.”

                catch (Exception e) when (e is WebSocketException or InvalidOperationException or ObjectDisposedException or OperationCanceledException)
                {
                }