Skip to content

Crash in System.Net.WebSockets.Client - Illegal memory access #73621

Closed

Description

@karelz

8/9:

  • 13x hits in Runfo last 120 days (older crashes than PR build 1925920 on 8/5 are from different reasons)
  • 18x hits in Kusto 8/5-8/9 (no crashes failures 6/9-8/4 at all)

Failures are on Linux only, with mix of architectures - arm64, amd64. Regression started on 8/5.

2 notable failures are on Rolling runs:

let crashes = (friendlyNamePrefix : string, includePR : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').WorkItems
    | where FriendlyName startswith friendlyNamePrefix
    //| where Queued > ago(7d)
    | where Status == "BadExit"
    | where ExitCode  == 139
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")) or (Branch startswith 'refs/heads/release/6.0'))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    | project-away PropertiesJson
    | order by Finished desc
};
crashes('System.Net.WebSockets.Client', true);

Activity

  1. added this to the 7.0.0 milestone on Aug 9, 2022
  2. added
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    on Aug 9, 2022
  3. CarnaViire commented on Aug 9, 2022

    @CarnaViire
    Member

    #73222 was merged on 8/5 so it might be related. The product change in the PR is minimal, but it added a bunch of tests.

    cc @greenEkatherine

  4. jkotas commented on Aug 9, 2022

    @jkotas
    Member

    This is likely the same root cause as #69125. See my comment at #69125 (comment) . You can confirm it by looking at the stacktrace of the crash using unmanaged debugger (lldb, gdb or windbg).

    I would expect that any test that depend on sockets is going to hit #69125 with some frequency.

  5. wfurt commented on Aug 9, 2022

    @wfurt
    Member

    I'm looking into it...

  6. hoyosjs commented on Aug 13, 2022

    @hoyosjs
    Member
  7. karelz commented on Aug 15, 2022

    @karelz
    MemberAuthor

    Yes, we expect those 2 are dupes -- it was pending verification from @wfurt

  8. wfurt commented on Aug 22, 2022

    @wfurt
    Member

    There is no failure in main since #73972 was merged.

  9. karelz commented on Aug 23, 2022

    @karelz
    MemberAuthor

    Fixed in 8.0 (main) in PR #73972 and in 7.0 (RC1) in PR #74367.

  10. ghost locked as resolved and limited conversation to collaborators on Sep 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area-System.Netblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'os-linuxLinux OS (any supported distro)

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions