Skip to content

Failures in SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3 #73930

Closed
#74479
Closed
Failures in SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3#73930
#74479
Assignees
wfurt
Labels
area-System.Net.Httpdisabled-testThe test is disabled in source code against the issue
Milestone
8.0.0

Description

@MichalStrehovsky
MichalStrehovsky
opened on Aug 15, 2022

Full log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-72145-merge-c270b69ee20f4e269e/System.Net.Http.Functional.Tests/3/console.142a186b.log?helixlogtype=result

Hit in https://github.com/dotnet/runtime/pull/72145/checks?check_run_id=7832255512

System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.LargeSingleHeader_ThrowsException(maxResponseHeadersLength: 15) [FAIL]
      System.Net.Quic.QuicException : Stream aborted by peer (268).
      Stack Trace:
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs(157,0): at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs(221,0): at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final)
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs(516,0): at System.Net.Quic.QuicStream.HandleEventPeerReceiveAborted(_PEER_RECEIVE_ABORTED_e__Struct& data)
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs(570,0): at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs(601,0): at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
        --- End of stack trace from previous location ---
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs(267,0): at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs(168,0): at System.Net.Test.Common.Http3LoopbackStream.SendFrameAsync(Int64 frameType, ReadOnlyMemory`1 framePayload)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs(123,0): at System.Net.Test.Common.Http3LoopbackStream.SendHeadersFrameAsync(Nullable`1 statusCode, IEnumerable`1 headers, Boolean qpackEncodeStatus)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs(264,0): at System.Net.Test.Common.Http3LoopbackStream.SendResponseHeadersAsync(Nullable`1 statusCode, IEnumerable`1 headers)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs(247,0): at System.Net.Test.Common.Http3LoopbackStream.SendResponseAsync(HttpStatusCode statusCode, IList`1 headers, String content, Boolean isFinal)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs(241,0): at System.Net.Test.Common.Http3LoopbackConnection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs(93,0): at System.Net.Test.Common.Http3LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs(93,0): at System.Net.Test.Common.Http3LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.MaxResponseHeadersLength.cs(84,0): at System.Net.Http.Functional.Tests.HttpClientHandler_MaxResponseHeadersLength_Test.<>c__DisplayClass4_0.<<LargeSingleHeader_ThrowsException>b__1>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(90,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs(111,0): at System.Net.Test.Common.Http3LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.MaxResponseHeadersLength.cs(72,0): at System.Net.Http.Functional.Tests.HttpClientHandler_MaxResponseHeadersLength_Test.LargeSingleHeader_ThrowsException(Int32 maxResponseHeadersLength)
        --- End of stack trace from previous location ---
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Proxy_DomainJoinedProxyServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader [SKIP]
      Condition(s) not met: "IsNtlmInstalled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3.ThresholdExceeded_ThrowsException(maxResponseHeadersLength: 1, headersLengthEstimate: 1024) [FAIL]
      System.Net.Quic.QuicException : Stream aborted by peer (268).
      Stack Trace:
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs(157,0): at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs(221,0): at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final)
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs(516,0): at System.Net.Quic.QuicStream.HandleEventPeerReceiveAborted(_PEER_RECEIVE_ABORTED_e__Struct& data)
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs(570,0): at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs(601,0): at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
        --- End of stack trace from previous location ---
        /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs(267,0): at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs(168,0): at System.Net.Test.Common.Http3LoopbackStream.SendFrameAsync(Int64 frameType, ReadOnlyMemory`1 framePayload)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs(123,0): at System.Net.Test.Common.Http3LoopbackStream.SendHeadersFrameAsync(Nullable`1 statusCode, IEnumerable`1 headers, Boolean qpackEncodeStatus)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs(264,0): at System.Net.Test.Common.Http3LoopbackStream.SendResponseHeadersAsync(Nullable`1 statusCode, IEnumerable`1 headers)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackStream.cs(247,0): at System.Net.Test.Common.Http3LoopbackStream.SendResponseAsync(HttpStatusCode statusCode, IList`1 headers, String content, Boolean isFinal)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackConnection.cs(241,0): at System.Net.Test.Common.Http3LoopbackConnection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs(93,0): at System.Net.Test.Common.Http3LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs(93,0): at System.Net.Test.Common.Http3LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.MaxResponseHeadersLength.cs(128,0): at System.Net.Http.Functional.Tests.HttpClientHandler_MaxResponseHeadersLength_Test.<>c__DisplayClass5_0.<<ThresholdExceeded_ThrowsException>b__1>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(90,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Net/Http/Http3LoopbackServer.cs(111,0): at System.Net.Test.Common.Http3LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.MaxResponseHeadersLength.cs(96,0): at System.Net.Http.Functional.Tests.HttpClientHandler_MaxResponseHeadersLength_Test.ThresholdExceeded_ThrowsException(Nullable`1 maxResponseHeadersLength, Int32 headersLengthEstimate)
        --- End of stack trace from previous location ---

Cc @MihaZupan

Activity

  1. MichalStrehovsky
    added
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    area-System.Net.Quic
    on Aug 15, 2022
  2. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Aug 15, 2022
  4. ManickaP
    added
    area-System.Net.Http
    and removed
    area-System.Net.Quic
    on Aug 15, 2022

  6. MichalStrehovsky commented on Aug 15, 2022

    @MichalStrehovsky
    MichalStrehovsky
    on Aug 15, 2022
    MemberAuthor

    Also failed in https://github.com/dotnet/runtime/pull/73929/checks?check_run_id=7833384695.

  7. ManickaP commented on Aug 15, 2022

    @ManickaP
    ManickaP
    on Aug 15, 2022
    Member

    The error is H3_REQUEST_CANCELLED (0x010c), @MihaZupan could this be recently broken by #73907?

  8. karelz commented on Aug 15, 2022

    @karelz
    karelz
    on Aug 15, 2022
    Member

    I don't see any failures in Kusto -- perhaps they didn't propagate yet?
    Is it 100% failure for you @MichalStrehovsky?

    let failedTests = (testNameSubstring : string, methodName : string, messageSubstr: string, includePR : bool, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | where Branch <> 'refs/pull/73374/merge'
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests(
    '', //testNameSubstring
    'SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http3', //methodName
    '', //messageSubstr
    true,  //includePR
    true); //includePassedOnRerun

  9. MichalStrehovsky commented on Aug 15, 2022

    @MichalStrehovsky
    MichalStrehovsky
    on Aug 15, 2022
    MemberAuthor

    Is it 100% failure for you @MichalStrehovsky?

    100% failure in the two pull requests I was involved in today:

    https://github.com/dotnet/runtime/pull/72145/checks?check_run_id=7832255512
    https://github.com/dotnet/runtime/pull/73929/checks?check_run_id=7833384695

  10. ghost added
    in-prThere is an active PR which will close this issue when it is merged
    on Aug 15, 2022

  11. MihaZupan commented on Aug 15, 2022

    @MihaZupan
    MihaZupan
    on Aug 15, 2022
    Member

    This is definitely related to #73907 - I just merged H2 and H3 versions of this test yesterday.

    Shouldn't be 100% as I never saw this failure on CI on that PR.

    Opened #73937 that may address this

    Edit: It's passing 90% of the time on reruns

  12. MihaZupan commented on Aug 15, 2022

    @MihaZupan
    MihaZupan
    on Aug 15, 2022
    Member

    Hit issues on the PR (#73937 (comment)).

    Disabling the test for H3 until we resolve that: #73945

  13. MihaZupan
    added this to the 7.0.0 milestone on Aug 15, 2022

  14. 5 remaining items

  15. karelz
    removed
    in-prThere is an active PR which will close this issue when it is merged
    on Aug 23, 2022
  16. ghost added
    in-prThere is an active PR which will close this issue when it is merged
    on Aug 23, 2022

  17. karelz commented on Aug 23, 2022

    @karelz
    karelz
    on Aug 23, 2022
    Member

    Triage: Let's try to re-enable the test in main to see if we get still hits.
    @MihaZupan's attempt to reproduce in PR did not succeed so far :(

  18. karelz
    assigned
    MihaZupan
    on Aug 23, 2022
  19. wfurt
    assigned
    wfurt
    and unassigned
    MihaZupan
    on Aug 23, 2022

  20. wfurt commented on Aug 23, 2022

    @wfurt
    wfurt
    on Aug 23, 2022
    Member

    I have local repro and the test still fails. I see no point of trying it in main. I will investigate.

  21. ghost removed
    in-prThere is an active PR which will close this issue when it is merged
    on Aug 23, 2022
  22. ghost added
    in-prThere is an active PR which will close this issue when it is merged
    on Aug 24, 2022

  23. karelz commented on Aug 25, 2022

    @karelz
    karelz
    on Aug 25, 2022
    Member

    Triage: @wfurt analyzed the problem as a problem in test + we need to slightly modify what we fail in this error case (that's a product change) -- neither is important for 7.0.
    Along the way @wfurt added bunch of useful logging that is not critical for 7.0.
    Moving to 8.0.

  24. karelz
    modified the milestones: 7.0.0, 8.0.0 on Aug 25, 2022
  25. wfurt
    closed this as completedin #74479on Aug 26, 2022
  26. ghost removed
    in-prThere is an active PR which will close this issue when it is merged
    on Aug 26, 2022
  27. ghost locked as resolved and limited conversation to collaborators on Sep 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area-System.Net.Httpdisabled-testThe test is disabled in source code against the issue

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions