Repository navigation
[QUIC] Long running test: System.Net.Quic.Tests.QuicStreamConformanceTests.Parallel_ReadWriteMultipleStreamsConcurrently #73377
Description
Occurrences 7/17-8/7 (incl. PRs):
| Day | Run |
|---|---|
| 7/31 | 2x Rolling run |
| 7/29 | Rolling run |
| 7/28 | 2x Rolling run |
| 7/27 | Rolling run |
| 7/26 | Rolling run |
| 7/25 | Rolling run |
| 7/24 | 3x Rolling run |
| 7/23 | Rolling run |
| 7/22 | 4x Rolling run + 2x PR #72670 |
| 7/18 | 3x Rolling run |
| 7/17 | 3x Rolling run |
| before 7/16 | Logs not available anymore |
let timeouts = (friendlyNamePrefix : string, includePR : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').WorkItems
| where Status == "Timeout"
| where FriendlyName startswith friendlyNamePrefix
//| where Finished > datetime('2021-09-07')
| distinct JobId, WorkItemId, Name, FriendlyName, ConsoleUri
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
| where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")) or (Branch startswith 'refs/heads/release/6.0'))
| where Type startswith "test/functional/cli/"
and not(Properties contains "runtime-staging")
| summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
| extend PropertiesJson = parse_json(Properties)
| extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
| extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
| extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
| extend Architecture = PropertiesJson.architecture
| extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
| project-away PropertiesJson
| order by Finished desc
};
timeouts('System.Net.Quic', true);Example occurrences:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-1b78cb6570b34ab2b4/System.Net.Quic.Functional.Tests/1/console.f63e2e91.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-ef18348474c247e1ac/System.Net.Quic.Functional.Tests/1/console.ff884bf6.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-6c3d3e6a75d84c1f96/System.Net.Quic.Functional.Tests/1/console.c15ed8bb.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-290c3d971e8e4f11b9/System.Net.Quic.Functional.Tests/1/console.b8248e3f.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-46fba0508c4a4e3aab/System.Net.Quic.Functional.Tests/1/console.78a1ce4b.log?helixlogtype=result
Happens about 1-2 per day, and seems to have been happening for some time. Not constrained to a specific OS.
Activity
karelz commentedon Aug 4, 2022
Triage: We should disable the test due to higher impact on CI.
We should figure out why it is happening in 7.0. Perhaps tweak the test to fail instead of hanging.
karelz commentedon Aug 7, 2022
Test disabled in #73446
karelz commentedon Aug 11, 2022
@ManickaP has a local repro, investigating now
karelz commentedon Aug 22, 2022
Based on info from @ManickaP -- it is a product bug. Await gets stuck when GC collects
QuicListener. It just looks like deadlock, though it is not. Still pretty ugly for customers.karelz commentedon Aug 29, 2022
Reopeing to track fix in 7.0 in PR #74740.
karelz commentedon Aug 30, 2022
Fixed in main (8.0) in PR #74450 and in 7.0 (for RC2) in PR #74740.