CI failure in System.Net.Http.Functional.Tests - Operation timeout

Seen in this release/7.0 backport PR: #75373

Please help determine if this needs a fix backported to 7.0.

BTW - I found two issues that looked similar, but they are not quite the same, which is why I opened this one. The issues are:

Details

Queue: Libraries Test Run release coreclr Linux x64 Debug
Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=13080&view=logs&j=b827109d-f627-5869-7deb-edacb438680e&t=85e93d53-1ed9-516b-1635-3169c07ac90d&l=104
Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75373-merge-9a35e1b62af84c0492/System.Net.Http.Functional.Tests.Attempt.3/3/console.b9bce8f9.log?helixlogtype=result
Since logs get deleted, here is an output example:

Callstack

    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Cookies.GetAsync_AddMultipleCookieHeaders_CookiesSent [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(154,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_AddMultipleCookieHeaders_CookiesSent()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.Http2ResponseStreamZeroByteReadTest.ZeroByteRead_BlocksUntilDataIsAvailable(async: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/ResponseStreamZeroByteReadTests.cs(229,0): at System.Net.Http.Functional.Tests.ResponseStreamZeroByteReadTestBase.ZeroByteRead_BlocksUntilDataIsAvailable(Boolean async)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter_Http2.PlaintextStreamFilter_SimpleDelegatingStream_Success(useSsl: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3203,0): at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter.PlaintextStreamFilter_SimpleDelegatingStream_Success(Boolean useSsl)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http11_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Finalization.cs(34,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Finalization_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http2.SendAsync_WithZeroLengthHeaderName_Throws [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Headers.cs(408,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.SendAsync_WithZeroLengthHeaderName_Throws()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http11.LargeSingleHeader_ThrowsException(maxResponseHeadersLength: 1) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.MaxResponseHeadersLength.cs(72,0): at System.Net.Http.Functional.Tests.HttpClientHandler_MaxResponseHeadersLength_Test.LargeSingleHeader_ThrowsException(Int32 maxResponseHeadersLength)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocksProxyTest_Http2.TestLoopbackAsync(scheme: "socks4", useSsl: True, useAuth: True, host: "localhost") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs(43,0): at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http11.SslOptions_CustomPolicy_IgnoresNameMismatch [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3829,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_SecurityTest.SslOptions_CustomPolicy_IgnoresNameMismatch()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Cookies_Http2.GetAsync_AddMultipleCookieHeaders_CookiesSent [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(154,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_AddMultipleCookieHeaders_CookiesSent()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SyncHttpHandlerTest_Cookies.GetAsync_SetCookieContainer_CookieSent(cookieName: "ABC", cookieValue: "123", useCookies: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(69,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_SetCookieContainer_CookieSent(String cookieName, String cookieValue, Boolean useCookies)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http3.SslOptions_InvalidName_Throws [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3798,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_SecurityTest.SslOptions_InvalidName_Throws()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ConnectionPooling_Test.Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnection(timeoutPropertyName: "PooledConnectionLifetime") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs(203,0): at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(1556,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ConnectionPooling_Test.Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnection(String timeoutPropertyName)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Http11_Cancellation_Test.GetAsync_CancelDuringResponseHeadersReceived_TaskCanceledQuickly(connectionClose: True, mode: Token) [FAIL]
      Elapsed time 00:01:07.7107582 should be less than 60 seconds, was 67.7107582
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cancellation.cs(615,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Cancellation_Test.ValidateClientCancellationAsync(Func`1 clientBodyAsync)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cancellation.cs(122,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Cancellation_Test.<>c__DisplayClass2_2.<<GetAsync_CancelDuringResponseHeadersReceived_TaskCanceledQuickly>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(101,0): at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cancellation.cs(108,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Cancellation_Test.GetAsync_CancelDuringResponseHeadersReceived_TaskCanceledQuickly(Boolean connectionClose, CancellationMode mode)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Proxy_DomainJoinedProxyServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Proxy_DomainJoinedProxyServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerUsesWindowsAuthentication_Success [SKIP]
      Condition(s) not met: "IsWindowsServerAvailable"
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
  Finished:    System.Net.Http.Functional.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Http.Functional.Tests  Total: 4171, Errors: 0, Failed: 13, Skipped: 14, Time: 165.557s

Also hit in this other release/7.0 backport PR: #75603

Queue: Libraries Test Run release coreclr Linux x64 Debug
Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=16338&view=logs&j=d883cc86-ff08-5389-6fca-0d01451458f6&t=5fca1690-df52-529b-9100-c473a9350efd&l=104
Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75603-merge-736e589603e2479ea6/System.Net.Http.Functional.Tests.Attempt.3/2/console.acbacc93.log?helixlogtype=result

Callstack

    System.Net.Http.Functional.Tests.SyncHttpHandlerTest_Cookies.GetAsync_SetCookieContainer_CookieSent(cookieName: "ABC", cookieValue: "123", useCookies: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(69,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_SetCookieContainer_CookieSent(String cookieName, String cookieValue, Boolean useCookies)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter_Http2.PlaintextStreamFilter_SimpleDelegatingStream_Success(useSsl: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3203,0): at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter.PlaintextStreamFilter_SimpleDelegatingStream_Success(Boolean useSsl)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandlerTest.PostAsync_ManyDifferentRequestHeaders_SentCorrectly [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.cs(425,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest.PostAsync_ManyDifferentRequestHeaders_SentCorrectly()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http2.ContentLength_DoesNotMatchRequestContentLength_Throws(contentLength: 0, bytesSent: 1) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3692,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.ContentLength_DoesNotMatchRequestContentLength_Throws(Int32 contentLength, Int32 bytesSent)
        --- End of stack trace from previous location ---
      Output:
        Ignored exception:
        System.IO.IOException: Got RST
           at System.Net.Test.Common.Http2LoopbackConnection.ReadBodyAsync(Boolean expectEndOfStream) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 583
           at System.Net.Test.Common.Http2LoopbackConnection.ReadAndParseRequestHeaderAsync(Boolean readBody) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 692
           at System.Net.Test.Common.Http2LoopbackConnection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 976
           at System.Net.Test.Common.Http2LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 153
           at System.Net.Test.Common.Http2LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 153
           at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.<>c__DisplayClass1_0.<<ContentLength_DoesNotMatchRequestContentLength_Throws>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 3711
    System.Net.Http.Functional.Tests.SocksProxyTest_Http2.TestLoopbackAsync(scheme: "socks4", useSsl: True, useAuth: True, host: "localhost") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs(43,0): at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Asynchrony_Test.ExecutionContext_Suppressed_Success [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(37,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Asynchrony_Test.ExecutionContext_Suppressed_Success()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http2.SendAsync_WithZeroLengthHeaderName_Throws [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Headers.cs(408,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.SendAsync_WithZeroLengthHeaderName_Throws()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(authScheme: "NTLM") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(670,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(String authScheme)
        --- End of stack trace from previous location ---
      Output:
        NTLM TlRMTVNTUAABAAAAF4IIoAAAAAAAAAAAAAAAAAAAAAA=
    System.Net.Http.Functional.Tests.Http1ResponseStreamZeroByteReadTest.ZeroByteRead_BlocksUntilDataIsAvailable(async: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/ResponseStreamZeroByteReadTests.cs(229,0): at System.Net.Http.Functional.Tests.ResponseStreamZeroByteReadTestBase.ZeroByteRead_BlocksUntilDataIsAvailable(Boolean async)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http11.ContentLength_DoesNotMatchRequestContentLength_Throws(contentLength: 0, bytesSent: 1) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3692,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.ContentLength_DoesNotMatchRequestContentLength_Throws(Int32 contentLength, Int32 bytesSent)
        --- End of stack trace from previous location ---
      Output:
        Ignored exception:
        System.IO.IOException: Unexpected EOF trying to read request header
           at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestHeaderBytesAsync() in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 720
           at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestDataAsync(Boolean readBody) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 782
           at System.Net.Test.Common.LoopbackServer.Connection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1026
           at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1086
           at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1086
           at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.<>c__DisplayClass1_0.<<ContentLength_DoesNotMatchRequestContentLength_Throws>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 3711
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter_Http11.PlaintextStreamFilter_ExceptionDuringCallback_ThrowsHttpRequestExceptionWithInnerException(useSsl: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3326,0): at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter.PlaintextStreamFilter_ExceptionDuringCallback_ThrowsHttpRequestExceptionWithInnerException(Boolean useSsl)
        --- End of stack trace from previous location ---
      Output:
        Ignored exception:
        System.IO.IOException: Unexpected EOF trying to read request header
           at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestHeaderBytesAsync() in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 720
           at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestDataAsync(Boolean readBody) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 782
           at System.Net.Test.Common.LoopbackServer.Connection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1026
           at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1086
           at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1086
           at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter.<>c__DisplayClass5_0.<<PlaintextStreamFilter_ExceptionDuringCallback_ThrowsHttpRequestExceptionWithInnerException>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 3349
    System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http3.SslOptions_InvalidName_Throws [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3798,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_SecurityTest.SslOptions_InvalidName_Throws()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Async.TestLoopbackAsync(scheme: "socks4", useSsl: True, useAuth: True, host: "localhost") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs(43,0): at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandlerTest.ReadAsStreamAsync_EmptyResponseBody_HandlerProducesWellBehavedResponseStream [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.cs(1212,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest.ReadAsStreamAsync_EmptyResponseBody_HandlerProducesWellBehavedResponseStream()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http2.ThresholdExceeded_ThrowsException(maxResponseHeadersLength: null, headersLengthEstimate: 64512) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.MaxResponseHeadersLength.cs(96,0): at System.Net.Http.Functional.Tests.HttpClientHandler_MaxResponseHeadersLength_Test.ThresholdExceeded_ThrowsException(Nullable`1 maxResponseHeadersLength, Int32 headersLengthEstimate)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Http2.SendAsync_MultipleExpected100Responses_ReceivesCorrectResponse [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.cs(1613,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest.SendAsync_MultipleExpected100Responses_ReceivesCorrectResponse()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Cookies_Http2.GetAsync_AddMultipleCookieHeaders_CookiesSent [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(154,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_AddMultipleCookieHeaders_CookiesSent()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http2_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Finalization.cs(34,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Finalization_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http11.GetAsync_MissingExpires_ReturnNull [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Headers.cs(272,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.GetAsync_MissingExpires_ReturnNull()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http3.ContentLength_DoesNotMatchRequestContentLength_Throws(contentLength: 0, bytesSent: 1) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3692,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.ContentLength_DoesNotMatchRequestContentLength_Throws(Int32 contentLength, Int32 bytesSent)
        --- End of stack trace from previous location ---
      Output:
        Ignored exception:
        System.Net.Quic.QuicException: The connection timed out from inactivity.
           at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
           at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
           at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 589
           at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 611
           at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 640
        --- End of stack trace from previous location ---
           at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 267
           at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 306
           at System.Net.Test.Common.Http3LoopbackStream.ReadIntegerAsync() in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs:line 462
           at System.Net.Test.Common.Http3LoopbackConnection.<EnsureControlStreamAcceptedAsync>g__EnsureControlStreamAcceptedInternalAsync|39_0() in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs:line 149
           at System.Net.Test.Common.Http3LoopbackConnection.AcceptRequestStreamAsync() in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs:line 165
           at System.Net.Test.Common.Http3LoopbackConnection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs:line 231
           at System.Net.Test.Common.Http3LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs:line 93
           at System.Net.Test.Common.Http3LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs:line 93
           at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.<>c__DisplayClass1_0.<<ContentLength_DoesNotMatchRequestContentLength_Throws>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 3711
    System.Net.Http.Functional.Tests.HPackTest.HPack_HeaderEncoding(headerName: ":method", expectedValue: "POST", expectedEncoding: [131]) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(55,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs(177,0): at System.Net.Test.Common.Http2LoopbackServer.CreateClientAndServerAsync(Func`2 clientFunc, Func`2 serverFunc, Http2Options http2Options, Int32 timeout)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HPackTest.cs(37,0): at System.Net.Http.Functional.Tests.HPackTest.HPack_HeaderEncoding(String headerName, String expectedValue, Byte[] expectedEncoding)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.HttpClientHandler_MultipleAuthenticateHeaders_WithSameAuth_Succeeds(authenticateHeader: "WWW-Authenticate: Basic realm=\"hello1\"\r\nWWW-Au"...) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(55,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(160,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass11_0.<<HttpClientHandler_MultipleAuthenticateHeaders_Succeeds>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(101,0): at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(156,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.HttpClientHandler_MultipleAuthenticateHeaders_Succeeds(String authenticateHeader)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(142,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.HttpClientHandler_MultipleAuthenticateHeaders_WithSameAuth_Succeeds(String authenticateHeader)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Proxy_DomainJoinedProxyServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Proxy_DomainJoinedProxyServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerUsesWindowsAuthentication_Success [SKIP]
      Condition(s) not met: "IsWindowsServerAvailable"
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
  Finished:    System.Net.Http.Functional.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Http.Functional.Tests  Total: 4171, Errors: 0, Failed: 22, Skipped: 14, Time: 164.750s
/mnt/work/A5B409BE/w/B2220994/e
----- end Wed Sep 14 15:36:16 UTC 2022 ----- exit code 1 ----------------------------------------------------------

Seen again in an unrelated release/7.0-rc2 backport PR: #75825

Queue: Libraries Test Run release coreclr Linux x64 Debug
Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=21762&view=logs&j=71afe768-e493-597c-228c-037834193aeb&t=fe87d331-02fa-5ca6-e382-69b0a11bb0f6&l=104
Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75825-merge-44ab6e8c6f0e4e49ac/System.Net.Http.Functional.Tests.Attempt.3/3/console.d1f6c924.log?helixlogtype=result

Callstack

System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(authScheme: "NTLM") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(670,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(String authScheme)
        --- End of stack trace from previous location ---
      Output:
        NTLM TlRMTVNTUAABAAAAF4IIoAAAAAAAAAAAAAAAAAAAAAA=
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Http2.SendAsync_MultipleExpected100Responses_ReceivesCorrectResponse [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.cs(1613,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest.SendAsync_MultipleExpected100Responses_ReceivesCorrectResponse()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SyncHttpHandlerTest_Cookies.GetAsync_SetCookieContainer_CookieSent(cookieName: "ABC", cookieValue: "123", useCookies: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(69,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_SetCookieContainer_CookieSent(String cookieName, String cookieValue, Boolean useCookies)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Cookies_Http2.GetAsync_AddMultipleCookieHeaders_CookiesSent [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(154,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_AddMultipleCookieHeaders_CookiesSent()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Async.TestLoopbackAsync(scheme: "socks4", useSsl: True, useAuth: True, host: "localhost") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs(43,0): at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http11.ContentLength_DoesNotMatchRequestContentLength_Throws(contentLength: 0, bytesSent: 1) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3692,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.ContentLength_DoesNotMatchRequestContentLength_Throws(Int32 contentLength, Int32 bytesSent)
        --- End of stack trace from previous location ---
      Output:
        Ignored exception:
        System.IO.IOException: Unexpected EOF trying to read request header
           at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestHeaderBytesAsync() in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 720
           at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestDataAsync(Boolean readBody) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 782
           at System.Net.Test.Common.LoopbackServer.Connection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1026
           at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1086
           at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1086
           at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.<>c__DisplayClass1_0.<<ContentLength_DoesNotMatchRequestContentLength_Throws>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 3711
    System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http2.ContentLength_DoesNotMatchRequestContentLength_Throws(contentLength: 0, bytesSent: 1) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3692,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.ContentLength_DoesNotMatchRequestContentLength_Throws(Int32 contentLength, Int32 bytesSent)
        --- End of stack trace from previous location ---
      Output:
        Ignored exception:
        System.IO.IOException: Got RST
           at System.Net.Test.Common.Http2LoopbackConnection.ReadBodyAsync(Boolean expectEndOfStream) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 583
           at System.Net.Test.Common.Http2LoopbackConnection.ReadAndParseRequestHeaderAsync(Boolean readBody) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 692
           at System.Net.Test.Common.Http2LoopbackConnection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 976
           at System.Net.Test.Common.Http2LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 153
           at System.Net.Test.Common.Http2LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 153
           at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.<>c__DisplayClass1_0.<<ContentLength_DoesNotMatchRequestContentLength_Throws>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 3711
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http2.ThresholdExceeded_ThrowsException(maxResponseHeadersLength: null, headersLengthEstimate: 64512) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.MaxResponseHeadersLength.cs(96,0): at System.Net.Http.Functional.Tests.HttpClientHandler_MaxResponseHeadersLength_Test.ThresholdExceeded_ThrowsException(Nullable`1 maxResponseHeadersLength, Int32 headersLengthEstimate)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http3.ContentLength_DoesNotMatchRequestContentLength_Throws(contentLength: 0, bytesSent: 1) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3692,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.ContentLength_DoesNotMatchRequestContentLength_Throws(Int32 contentLength, Int32 bytesSent)
        --- End of stack trace from previous location ---
      Output:
        Ignored exception:
        System.Net.Quic.QuicException: The connection timed out from inactivity.
           at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
           at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
           at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 589
           at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 611
           at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 640
        --- End of stack trace from previous location ---
           at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 267
           at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 306
           at System.Net.Test.Common.Http3LoopbackStream.ReadIntegerAsync() in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs:line 462
           at System.Net.Test.Common.Http3LoopbackConnection.<EnsureControlStreamAcceptedAsync>g__EnsureControlStreamAcceptedInternalAsync|39_0() in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs:line 149
           at System.Net.Test.Common.Http3LoopbackConnection.AcceptRequestStreamAsync() in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs:line 165
           at System.Net.Test.Common.Http3LoopbackConnection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs:line 231
           at System.Net.Test.Common.Http3LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs:line 93
           at System.Net.Test.Common.Http3LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs:line 93
           at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.<>c__DisplayClass1_0.<<ContentLength_DoesNotMatchRequestContentLength_Throws>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 3711
    System.Net.Http.Functional.Tests.SocketsHttpHandler_PostScenarioTest.DisposeTargetStream_ThrowsObjectDisposedException(knownLength: False) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(530,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_PostScenarioTest.DisposeTargetStream_ThrowsObjectDisposedException(Boolean knownLength)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandlerTest.ReadAsStreamAsync_EmptyResponseBody_HandlerProducesWellBehavedResponseStream [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.cs(1212,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest.ReadAsStreamAsync_EmptyResponseBody_HandlerProducesWellBehavedResponseStream()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ConnectionPooling_Test.Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnection(timeoutPropertyName: "PooledConnectionLifetime") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs(203,0): at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(1556,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ConnectionPooling_Test.Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnection(String timeoutPropertyName)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.HPackTest.HPack_HeaderEncoding(headerName: ":method", expectedValue: "POST", expectedEncoding: [131]) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(55,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs(177,0): at System.Net.Test.Common.Http2LoopbackServer.CreateClientAndServerAsync(Func`2 clientFunc, Func`2 serverFunc, Http2Options http2Options, Int32 timeout)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HPackTest.cs(37,0): at System.Net.Http.Functional.Tests.HPackTest.HPack_HeaderEncoding(String headerName, String expectedValue, Byte[] expectedEncoding)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.HttpClientHandler_MultipleAuthenticateHeaders_WithSameAuth_Succeeds(authenticateHeader: "WWW-Authenticate: Basic realm=\"hello1\"\r\nWWW-Au"...) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(55,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(160,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass11_0.<<HttpClientHandler_MultipleAuthenticateHeaders_Succeeds>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(101,0): at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(156,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.HttpClientHandler_MultipleAuthenticateHeaders_Succeeds(String authenticateHeader)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(142,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.HttpClientHandler_MultipleAuthenticateHeaders_WithSameAuth_Succeeds(String authenticateHeader)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Http11_Cancellation_Test.GetAsync_CancelDuringResponseHeadersReceived_TaskCanceledQuickly(connectionClose: True, mode: Token) [FAIL]
      Elapsed time 00:01:01.5080120 should be less than 60 seconds, was 61.508012
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cancellation.cs(615,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Cancellation_Test.ValidateClientCancellationAsync(Func`1 clientBodyAsync)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cancellation.cs(122,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Cancellation_Test.<>c__DisplayClass2_2.<<GetAsync_CancelDuringResponseHeadersReceived_TaskCanceledQuickly>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(101,0): at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cancellation.cs(108,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Cancellation_Test.GetAsync_CancelDuringResponseHeadersReceived_TaskCanceledQuickly(Boolean connectionClose, CancellationMode mode)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Proxy_DomainJoinedProxyServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Proxy_DomainJoinedProxyServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerUsesWindowsAuthentication_Success [SKIP]
      Condition(s) not met: "IsWindowsServerAvailable"
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
  Finished:    System.Net.Http.Functional.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Http.Functional.Tests  Total: 4171, Errors: 0, Failed: 15, Skipped: 14, Time: 148.691s
/mnt/work/B7930A0F/w/A14B08EC/e

New hit in this release/7.0 dependencies update PR: #75890

Queue: Libraries Test Run release coreclr Linux x64 Debug
Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=26261&view=logs&j=71afe768-e493-597c-228c-037834193aeb&t=fe87d331-02fa-5ca6-e382-69b0a11bb0f6&l=104
Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75890-merge-1ca724650ddc4b5bba/System.Net.Http.Functional.Tests.Attempt.3/3/console.825e15e0.log?helixlogtype=result

Callstack

System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http11.GetAsync_MissingExpires_ReturnNull [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Headers.cs(272,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.GetAsync_MissingExpires_ReturnNull()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Http2.SendAsync_MultipleExpected100Responses_ReceivesCorrectResponse [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.cs(1613,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest.SendAsync_MultipleExpected100Responses_ReceivesCorrectResponse()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http3.ContentLength_DoesNotMatchRequestContentLength_Throws(contentLength: 0, bytesSent: 1) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3692,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.ContentLength_DoesNotMatchRequestContentLength_Throws(Int32 contentLength, Int32 bytesSent)
        --- End of stack trace from previous location ---
      Output:
        Ignored exception:
        System.Net.Quic.QuicException: The connection timed out from inactivity.
           at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
           at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
           at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 589
           at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 611
           at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 640
        --- End of stack trace from previous location ---
           at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 267
           at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 306
           at System.Net.Test.Common.Http3LoopbackStream.ReadIntegerAsync() in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs:line 462
           at System.Net.Test.Common.Http3LoopbackConnection.<EnsureControlStreamAcceptedAsync>g__EnsureControlStreamAcceptedInternalAsync|39_0() in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs:line 149
           at System.Net.Test.Common.Http3LoopbackConnection.AcceptRequestStreamAsync() in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs:line 165
           at System.Net.Test.Common.Http3LoopbackConnection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs:line 231
           at System.Net.Test.Common.Http3LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs:line 93
           at System.Net.Test.Common.Http3LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs:line 93
           at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.<>c__DisplayClass1_0.<<ContentLength_DoesNotMatchRequestContentLength_Throws>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 3711
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http2.ThresholdExceeded_ThrowsException(maxResponseHeadersLength: null, headersLengthEstimate: 64512) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.MaxResponseHeadersLength.cs(96,0): at System.Net.Http.Functional.Tests.HttpClientHandler_MaxResponseHeadersLength_Test.ThresholdExceeded_ThrowsException(Nullable`1 maxResponseHeadersLength, Int32 headersLengthEstimate)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocksProxyTest_Http2.TestLoopbackAsync(scheme: "socks4", useSsl: True, useAuth: True, host: "localhost") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs(43,0): at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http3.SslOptions_InvalidName_Throws [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3798,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_SecurityTest.SslOptions_InvalidName_Throws()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(authScheme: "NTLM") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(670,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(String authScheme)
        --- End of stack trace from previous location ---
      Output:
        NTLM TlRMTVNTUAABAAAAF4IIoAAAAAAAAAAAAAAAAAAAAAA=
    System.Net.Http.Functional.Tests.SyncHttpHandlerTest_Cookies.GetAsync_SetCookieContainer_CookieSent(cookieName: "ABC", cookieValue: "123", useCookies: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(69,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_SetCookieContainer_CookieSent(String cookieName, String cookieValue, Boolean useCookies)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http11.ContentLength_DoesNotMatchRequestContentLength_Throws(contentLength: 0, bytesSent: 1) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3692,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.ContentLength_DoesNotMatchRequestContentLength_Throws(Int32 contentLength, Int32 bytesSent)
        --- End of stack trace from previous location ---
      Output:
        Ignored exception:
        System.IO.IOException: Unexpected EOF trying to read request header
           at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestHeaderBytesAsync() in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 720
           at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestDataAsync(Boolean readBody) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 782
           at System.Net.Test.Common.LoopbackServer.Connection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1026
           at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1086
           at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1086
           at System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest.<>c__DisplayClass1_0.<<ContentLength_DoesNotMatchRequestContentLength_Throws>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 3711
    System.Net.Http.Functional.Tests.Http2ResponseStreamZeroByteReadTest.ZeroByteRead_BlocksUntilDataIsAvailable(async: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/ResponseStreamZeroByteReadTests.cs(229,0): at System.Net.Http.Functional.Tests.ResponseStreamZeroByteReadTestBase.ZeroByteRead_BlocksUntilDataIsAvailable(Boolean async)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http2_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Finalization.cs(34,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Finalization_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter_Http2.PlaintextStreamFilter_SimpleDelegatingStream_Success(useSsl: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3203,0): at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter.PlaintextStreamFilter_SimpleDelegatingStream_Success(Boolean useSsl)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Finalization_Http11_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Finalization.cs(34,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Finalization_Test.IncompleteResponseStream_ResponseDropped_CancelsRequestToServer()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Cookies.GetAsync_AddMultipleCookieHeaders_CookiesSent [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(154,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsync_AddMultipleCookieHeaders_CookiesSent()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Async.TestLoopbackAsync(scheme: "socks4", useSsl: True, useAuth: True, host: "localhost") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocksProxyTest.cs(43,0): at System.Net.Http.Functional.Tests.SocksProxyTest.TestLoopbackAsync(String scheme, Boolean useSsl, Boolean useAuth, String host)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandlerTest.ReadAsStreamAsync_EmptyResponseBody_HandlerProducesWellBehavedResponseStream [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.cs(1212,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest.ReadAsStreamAsync_EmptyResponseBody_HandlerProducesWellBehavedResponseStream()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http11.SslOptions_CustomPolicy_IgnoresNameMismatch [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3829,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_SecurityTest.SslOptions_CustomPolicy_IgnoresNameMismatch()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter_Http11.PlaintextStreamFilter_ExceptionDuringCallback_ThrowsHttpRequestExceptionWithInnerException(useSsl: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3326,0): at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter.PlaintextStreamFilter_ExceptionDuringCallback_ThrowsHttpRequestExceptionWithInnerException(Boolean useSsl)
        --- End of stack trace from previous location ---
      Output:
        Ignored exception:
        System.IO.IOException: Unexpected EOF trying to read request header
           at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestHeaderBytesAsync() in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 720
           at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestDataAsync(Boolean readBody) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 782
           at System.Net.Test.Common.LoopbackServer.Connection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1026
           at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1086
           at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 1086
           at System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_PlaintextStreamFilter.<>c__DisplayClass5_0.<<PlaintextStreamFilter_ExceptionDuringCallback_ThrowsHttpRequestExceptionWithInnerException>b__1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs:line 3349
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Proxy_Test.ProxyTunnelRequest_OriginServerSendsProxyAuthChallenge_NoProxyAuthPerformed [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Proxy.cs(428,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Proxy_Test.ProxyTunnelRequest_OriginServerSendsProxyAuthChallenge_NoProxyAuthPerformed()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ConnectionPooling_Test.Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnection(timeoutPropertyName: "PooledConnectionLifetime") [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs(203,0): at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout)
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(1556,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ConnectionPooling_Test.Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnection(String timeoutPropertyName)
        --- End of stack trace from previous location ---

Another hit in a different release/7.0 dependencies update PR: #75652

Queue: Libraries Test Run release coreclr Linux x64 Debug
Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=29898&view=logs&j=71afe768-e493-597c-228c-037834193aeb&t=fe87d331-02fa-5ca6-e382-69b0a11bb0f6&l=104
Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75652-merge-17fbea841fd04cc080/System.Net.Http.Functional.Tests.Attempt.3/3/console.58b8a868.log?helixlogtype=result

Callstack

===========================================================================================================
/mnt/work/B5500971/w/B059095E/e /mnt/work/B5500971/w/B059095E/e
  Discovering: System.Net.Http.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Http.Functional.Tests (found 1333 of 1604 test cases)
  Starting:    System.Net.Http.Functional.Tests (parallel test collections = on, max threads = 2)
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerUsesWindowsAuthentication_Success [SKIP]
      Condition(s) not met: "IsWindowsServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClient_SelectedSites_Test.RetrieveSite_Succeeds [SKIP]
      Condition(s) not met: "IsSelectedSitesTestEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClient_SelectedSites_Test.RetrieveSite_Debug_Helper [SKIP]
      Condition(s) not met: "IsSelectedSitesTestEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientEKUTest.HttpClient_ClientEKUServerAuth_Fails [SKIP]
      Condition(s) not met: "CanTestCertificates"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientEKUTest.HttpClient_NoEKUServerAuth_Ok [SKIP]
      Condition(s) not met: "CanTestCertificates"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientEKUTest.HttpClient_NoEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "CanTestCertificates"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientEKUTest.HttpClient_ServerEKUClientAuth_Fails [SKIP]
      Condition(s) not met: "CanTestCertificates"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http3.SslOptions_InvalidName_Throws [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs(3798,0): at System.Net.Http.Functional.Tests.SocketsHttpHandler_SecurityTest.SslOptions_InvalidName_Throws()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.Http2ResponseStreamZeroByteReadTest.ZeroByteRead_BlocksUntilDataIsAvailable(async: True) [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/ResponseStreamZeroByteReadTests.cs(229,0): at System.Net.Http.Functional.Tests.ResponseStreamZeroByteReadTestBase.ZeroByteRead_BlocksUntilDataIsAvailable(Boolean async)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SyncHttpHandlerTest_HttpClientHandlerTest_Headers.SendAsync_RequestWithSimpleHeader_ResponseReferencesUnmodifiedRequestHeaders [FAIL]
      System.TimeoutException : The operation has timed out.
      Stack Trace:
        /_/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Headers.cs(31,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.SendAsync_RequestWithSimpleHeader_ResponseReferencesUnmodifiedRequestHeaders()
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Proxy_DomainJoinedProxyServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Proxy_DomainJoinedProxyServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerUsesWindowsAuthentication_Success [SKIP]
      Condition(s) not met: "IsWindowsServerAvailable"
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
  Finished:    System.Net.Http.Functional.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Http.Functional.Tests  Total: 4171, Errors: 0, Failed: 3, Skipped: 14, Time: 169.412s
/mnt/work/B5500971/w/B059095E/e
----- end Mon Sep 26 13:22:00 UTC 2022 ----- exit code 1 ----------------------------------------------------------

Seen again in a release/7.0 backport PR: #79192

Queue: Libraries Test Run release coreclr Linux x64 Debug
Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=125466&view=logs&j=71afe768-e493-597c-228c-037834193aeb&t=fe87d331-02fa-5ca6-e382-69b0a11bb0f6&l=103
Log files:
- SocketsHttpHandler_RequestContentLengthMismatchTest_Http2.ContentLength_DoesNotMatchRequestContentLength_Throws: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-79192-merge-4491004e3e134e2f80/System.Net.Http.Functional.Tests/3/console.9d15c276.log?helixlogtype=result
- SocketsHttpHandler_RequestContentLengthMismatchTest_Http2.ContentLength_DoesNotMatchRequestContentLength_Throws: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-79192-merge-4491004e3e134e2f80/System.Net.Http.Functional.Tests/3/console.9d15c276.log?helixlogtype=result
- SocketsHttpHandler_RequestContentLengthMismatchTest_Http2.ContentLength_DoesNotMatchRequestContentLength_Throws: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-79192-merge-4491004e3e134e2f80/System.Net.Http.Functional.Tests/3/console.9d15c276.log?helixlogtype=result

@rzikm

@rzikm can you please check frequency on 7.0? Is it worth changing there?
If it is across multiple tests, perhaps we need to align the VMs used for servicing and main.

I see it affected 7 PRs in the last 30 days, a total of 62 tests have failed with a timeout.

cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName startswith "System.Net.Http"
        and Outcome == 'Failed' or Outcome == 'PassedOnRerun'
    | extend i = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, i)
    | extend Type = substring(TestName, 0, i - 1)
    | project-away i, TestName
    | where Message has "TimeoutException"
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where Finished > now(-30d)
        | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
        | where (TargetBranch == 'release/7.0')
        | where Type startswith "test/functional/cli/" and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName, Attempt, Build, TargetBranch) by JobId
    | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", "")) 
    | extend Architecture = PropertiesJson.architecture
    | extend DefinitionName = PropertiesJson.DefinitionName
    | project-away JobId
    | extend JobId = extractjson("$.['System.JobId']", Properties)
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').WorkItems
        | project-keep ConsoleUri, WorkItemId) on WorkItemId
    | project-away PropertiesJson
    | project-reorder Started, Type, Method, Arguments, StackTrace, ConsoleUri, OS, Architecture, DefinitionName, Branch, TargetBranch, Attempt, Build, * 
;

What I'm seeing is:

only on RedHat.7.Amd64.Open
only coreclr (no mono failures)
~ 30 distinct failing tests

This might be environmental.

@richlander

If I remember correctly, RedHat.7 still has old OpenSSL 1.0. And it full queue instead of container so timing can be different.

With RHEL.9 out, do we still intend to support RedHat 7 with .NET 8.0 or do we plan to stick to the "two major versions back" @richlander?

Triage: if we intend to support RH 7, we should investigate and fix this, if not, this could be closed when the queues get updated.

No recent hits on CI, closing.

Navigation Menu

Repository navigation

CI failure in System.Net.Http.Functional.Tests - Operation timeout #75611

Details

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CI failure in System.Net.Http.Functional.Tests - Operation timeout #75611

Description

Details

Activity

carlossanlop commented on Sep 14, 2022

carlossanlop commented on Sep 19, 2022

carlossanlop commented on Sep 23, 2022

carlossanlop commented on Sep 26, 2022

carlossanlop commented on Jan 5, 2023

karelz commented on Jan 23, 2023

rzikm commented on Jan 23, 2023

ManickaP commented on Jan 23, 2023

wfurt commented on Jan 23, 2023

ManickaP commented on Feb 9, 2023

rzikm commented on Jul 20, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions