Long running test: ReadAsync_CancelPendingTask_ThrowsCancellationException and ReadAsync_CancelPendingValueTask_ThrowsCancellationException

[tlakollo]

Long running tests:

• CancelPendingTask - ReadAsync_CancelPendingTask_ThrowsCancellationException (disabled in PR Disable long running ReadAsync_CancelPendingTask_ThrowsCancellationException tests for some Http1 cases #72596 on 7/21)
  • System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingTask_ThrowsCancellationException - Http1CloseResponseStreamConformanceTests
  • System.Net.Http.Functional.Tests.Http1RawResponseStreamConformanceTests.ReadAsync_CancelPendingTask_ThrowsCancellationException - Http1RawResponseStreamConformanceTests
• CancelPendingValueTask - ReadAsync_CancelPendingValueTask_ThrowsCancellationException (disabled in PR Disable ReadAsync_CancelPendingValueTask_ThrowsCancellationException for Http1 #72854 on 7/27)
  • System.Net.Http.Functional.Tests.Http1RawResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException - Http1RawResponseStreamConformanceTests
  • System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException - Http1CloseResponseStreamConformanceTests

Regression on 7/20

Failures 6/30-8/5:

Date	Runs
8/5	1x PR occurrence - likely older source code
8/4	1x PR occurrence - likely older source code
8/3	1x PR occurrence - likely older source code
7/27	24x (1x in Official runs)
7/26	29x (2x in Official runs)
7/25	12x (2x in Official runs)
7/24	8x (3x in Official runs)
7/23	5x (1x in Official runs)
7/22	20x (0x in Official runs)
7/21	61x (5x in Official runs)
7/20	35x (7x in Official runs)

Given the high frequency, we should disable the test ASAP.

let timeouts = (includePR : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').WorkItems
| where Status == "Timeout"
| where FriendlyName startswith "System.Net.Http.Functional.Tests"
//| where Finished > datetime('2021-09-07')
| distinct JobId, WorkItemId, Name, FriendlyName, ConsoleUri
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
    | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")) or (Branch startswith 'refs/heads/release/6.0'))
    | where Type startswith "test/functional/cli/"
        and not(Properties contains "runtime-staging")
    | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
| order by Finished desc
};
timeouts(true);

Original report

Couldn't add runfo information since the work item has the same name as previously fixed issues. But this timeout behavior has occurred in the following rolling build runs:
https://dev.azure.com/dnceng/public/_build/results?buildId=1893373&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=1893617&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=1893157&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=1893763&view=results

The timeout occurs when executing the long-running test System.Net.Http.Functional.Tests.Http1RawResponseStreamConformanceTests.ReadAsync_CancelPendingTask_ThrowsCancellationException
These timeout are occurring on both runtime and runtime-extra-platforms

Related to #72209

{
  "ErrorMessage": "[Long Running Test] 'System.Net.Http.Functional.Tests.Http1RawResponseStreamConformanceTests.ReadAsync_CancelPending",
  "BuildRetry": false
}

Report

Build	Definition	Test	Pull Request
64914	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77169
64782	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77487
64481	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#76859
64009	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77449
63786	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77386
63754	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77261
62760	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77398
62577	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#74820
62186	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
61912	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77398
61472	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77400
60736	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77245
60520	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
60215	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77336
59963	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77336
59761	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77196
59151	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77293
59055	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77196
58799	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#74820
58224	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#72717
57227	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77219
57205	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77220
56479	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77149
56287	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77052
55730	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
55653	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77107
55580	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77127
55028	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#76942
54938	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#62863
54792	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77116
54391	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#76361
54104	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#76361
53859	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#77088

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	18	33

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Couldn't add runfo information since the work item has the same name as previously fixed issues. But this timeout behavior has occurred in the following rolling build runs:
https://dev.azure.com/dnceng/public/_build/results?buildId=1893373&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=1893617&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=1893157&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=1893763&view=results

The timeout occurs when executing the long-running test System.Net.Http.Functional.Tests.Http1RawResponseStreamConformanceTests.ReadAsync_CancelPendingTask_ThrowsCancellationException
These timeout are occurring on both runtime and runtime-extra-platforms

Related to #72209

Author:	tlakollo
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[karelz]

Triage: regression introduced on 7/20. High frequency, we should disable it ASAP.

[karelz]

@stephentoub I asked @rzikm to disable the test. Do you plan to do more (curious about your self-assignment).

[stephentoub]

curious about your self-assignment

I changed this test yesterday.

[stephentoub]

I'm having trouble reproing locally. The changes I made yesterday were:

• Updating the test to be a theory and passing in both 0 and 100 (whereas previously it effectively only did the equivalent of 0)
  

  runtime/src/libraries/Common/tests/StreamConformanceTests/System/IO/StreamConformanceTests.cs

  Lines 1678 to 1679 in 0a7d601

  	[InlineData(0)]
  	[InlineData(100)]

• Changing a call to Cancel() to instead be to CancelAfter(delay) using those 0 or 100 values
  

  runtime/src/libraries/Common/tests/StreamConformanceTests/System/IO/StreamConformanceTests.cs

  Line 542 in 0a7d601

  cts.CancelAfter(cancellationDelay);

Re-reviewing, I think the tests are valid. Most likely there's some race condition somewhere in the implementation of either SocketsHttpHandler or the test stream it's using that causes the waiting for cancellation in these tests to hang. But it's hard to say for sure without a repro.

I've unassigned myself.