Skip to content

System.Net.WebSockets.Client.Tests crash on OSX Mono #75307

Closed
Closed
System.Net.WebSockets.Client.Tests crash on OSX Mono#75307
Labels
area-System.Netblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'os-maccatalystMacCatalyst OSruntime-monospecific to the Mono runtimetest-bugProblem in test source code (most likely)
Milestone
7.0.0

Description

@karelz
karelz
opened on Sep 8, 2022

Frequency in last 30 days in Runfo:

  • 9/8 - 33x failures
    • First occurrence in Rolling run 3461 on 9/2 -- recent regression
    • Each run fails on 2 platforms:
      • net7.0-MacCatalyst-Release-arm64-Mono_Release-OSX.1200.Arm64.Open
      • net7.0-MacCatalyst-Release-x64-Mono_Release-OSX.1200.Amd64.Open
      • Note: There's is most likely unrelated failure on net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open in each of these runs

Failure:

[15:49:31] dbug: Test execution started
[15:49:32] dbug: Test log server listening on: 0.0.0.0:62555
[15:49:32] dbug: Process open exited with 0
[15:49:42] dbug: Test run completed
[15:49:42] dbug: Process log exited with 137
[15:49:42] dbug: Test run started but crashed and no test results were reported
[15:49:42] dbug: No crash reports, waiting 30 seconds for the crash report service...
[15:49:42] dbug: Found 1 new crash report(s)
[15:49:42] dbug: - Adding /Users/helix-runner/Library/Logs/DiagnosticReports/System.Net.WebSockets.Client.Tests-2022-09-02-154941.ips
[15:49:42] dbug: Successfully copied System.Net.WebSockets.Client.Tests-2022-09-02-154941.ips
[15:49:42] fail: Application test run crashed
                 No test log file was produced
                 
                 Check logs for more information
[15:49:42] dbug: Saving diagnostics data to '/tmp/helix/working/9A9B08C3/w/AC6B0966/e/diagnostics.json'
XHarness exit code: 80 (APP_CRASH)

Activity

  1. karelz
    added
    area-System.Net
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    runtime-monospecific to the Mono runtime
    os-maccatalystMacCatalyst OS
    on Sep 8, 2022
  2. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Sep 8, 2022

  4. karelz commented on Sep 8, 2022

    @karelz
    karelz
    on Sep 8, 2022
    MemberAuthor

    @akoeplinger @steveisok this is blocking clean CI -- happens 5x per day. Can you please look into it ASAP? Thanks!

  5. steveisok commented on Sep 8, 2022

    @steveisok
    steveisok
    on Sep 8, 2022
    Member

    Looks like it's an mlaunch issue that can't start the app. We'll need to figure out how widespread this is.

  6. steveisok commented on Sep 9, 2022

    @steveisok
    steveisok
    on Sep 9, 2022
    Member

    @mandel-macaque We haven't bumped xharness/mlaunch in runtime yet. Is this an mlaunch problem or with devices?

    https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-9747453e45bd47beab/System.Net.WebSockets.Client.Tests/1/console.83d2be01.log?helixlogtype=result

    [14:58:25] dbug: [TCP tunnel] Xamarin.Hosting: Device discovery started
[14:58:25] dbug: [TCP tunnel] Xamarin.Hosting: Device discovery event: Connected (fff44:c6:5d:6d:09:3afff)
[14:58:25] dbug: [TCP tunnel] 2022-09-02 14:58:25.562 mlaunch[56071:1486223]
[14:58:25] dbug: [TCP tunnel] Unhandled Exception:
[14:58:25] dbug: [TCP tunnel] Xamarin.Hosting.MobileDeviceException: Read Error (error: 0xe8000004)
[14:58:25] dbug: [TCP tunnel]   at Xamarin.Hosting.RealDevice.Connect () [0x00021] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/RealDevice.cs:57
[14:58:25] dbug: [TCP tunnel]   at Xamarin.Hosting.RealDevice..ctor (System.IntPtr am_device) [0x00022] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/RealDevice.cs:40
[14:58:25] dbug: [TCP tunnel]   at Xamarin.Hosting.DeviceDiscovery+<>c__DisplayClass23_1.<DiscoverNotificationCallback>b__0 () [0x00007] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/DeviceDiscovery.cs:225
[14:58:25] dbug: [TCP tunnel]   at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00014] in

  7. karelz commented on Sep 9, 2022

    @karelz
    karelz
    on Sep 9, 2022
    MemberAuthor

    Note: It happens also in 7.0 branch -- Rolling run 9064 on 9/8 AM (release/7.0) ... so perhaps it is something environmental? Or some shared infra?

  8. akoeplinger commented on Sep 9, 2022

    @akoeplinger
    akoeplinger
    on Sep 9, 2022
    Member

    The MacCatalyst failures shouldn't be related to mlaunch since afaik we don't use mlaunch to run Catalyst apps.

  9. steveisok
    added a commit that references this issue on Sep 9, 2022

    Fix WebSockets.Client.ConnectTest crash for apple mobile platforms

    48dd367

  10. mandel-macaque commented on Sep 9, 2022

    @mandel-macaque
    mandel-macaque
    on Sep 9, 2022

    As @akoeplinger mentioned, mlaunch is not needed for maccatalyst, since it does not need to be added to a device neither it needs the TCP tunnel. Yet, the stack trace from @steveisok points to an issue when creating the tcp tunnel with an apple tv.

    @steveisok there area few things we need to clarify what exactly is happening.

    1. As per logs, there are two different apple tv devices: DNCENGTVOS-091 and DNCENGTVOS-061. Can we have some more info of what are the versions of those devices?
    2. The issue happens when trying to create the tcp tunnel with 3315900bedf22783bd9d2a74d012464b00e0fc99 which is device DNCENGTVOS-061. We need someone to go to the lab and tell use the state of that device. Can we repo the issue with DNCENGTVOS-091 ?
    3. The Xcode version used is Xcode13.app yet the latest stable is 13.4.1 (13F100) we need to verify the version of Xcode is used in the lab. mlaunch has had changes due to issues with the latests Xcodes.
    4. I don't have access to the crash report. Can it be added to the issue?

    Lets try to get answer to the above to keep looking into issues.

  11. akoeplinger commented on Sep 9, 2022

    @akoeplinger
    akoeplinger
    on Sep 9, 2022
    Member

    I looked at this together with @mandel-macaque and the Read Error (error: 0xe8000004) is because mlaunch is trying to connect to the wrong device.

    fff44:c6:5d:6d:09:3afff is DNCENGTVOS-091 which is attached to DNCENGMAC091, but we're running on DNCENGMAC061!

    It looks like somehow Wifi got turned on on DNCENGTVOS-091 and that causes it to be visible on the wrong machine.

    mlaunch --listdev --output-format=xml gets this:

    <?xml version="l.0" encoding="utf-8"?>
<MTouch Version="0" Revi8ion="0">
<SdkRoot>/Applications/Xcodel3.app/Contents/Developer</SdkRoot>
<Device>
    <ActivationState />
    <AmountDataAvailable />
    <BatteryCurrentCapacity />
    <BluetoothAddress />
    <BuildVersion />
    <ChipID />
    <CPUArchitecture />
    <DeveloperStatus />
    <DeviceClass />
    <DeviceColor />
    <DeviceEnclosureColor />
    <DeviceIdentifier>fff44:c6:5d:6d:09:3afff</DeviceIdentifier>
    <DevToolsAvailable />
    <EnableWifiDebugging />
    <HardwareModel />
    <InterfaceType>Wifi</InterfaceType>
    <InternationalMobileEquipmentIdentity />
    <IsPaired>False</IsPaired>
    <IsUsableForDebugging />
    <IsValid>True</IsValid>
    <ModelNumber />
    <Name>DNCENGTVOS-091</Name>
    <PasswordProtected />
    <ProductionSOC />
    <ProductType />
    <ProductVersion />
    <SerialNumber />
    <TotalDataCapacity />
    <UDID />
    <UniqueChipID />
    <WatchCompanionCapability />
    <WiFiAddress />
</Device>
<Device>
    <ActivationState>Activated</ActivationState>
    <AmountDataAvailable>23143993344</AmountDataAvailable>
    <BatteryCurrentCapacity>100</BatteryCurrentCapacity>
    <BluetoothAddress>44:c6:5d:74:bl:e2</BluetoothAddress>
    <BuildVersion>19J346</BuildVersion>
    <ChipID>28672</ChipID>
    <CPUArchitecture>arm64</CPUArchitecture>
    <DeveloperStatus />
    <DeviceClass>AppleTV</DeviceClass>
    <DeviceColor>unknown</DeviceColor>
    <DeviceEnclosureColor>unknown</DeviceEnclosureColor>
    <Devieeldentifier>2a46286a5b932beadleda3d60c823c3f3c4c4761</Deviceidentifier>
    <DevToolsAvailable>Standard</DevToolsAvailable>
    <EnableWifiDebugging />
    <HardwareModel>J42dAP</HardwareModel>
    <InterfaceType>Usb</InterfaceType>
    <InternationalMobileEquipmentIdentity />
    <IsPaired>True</IsPaired>
    <IsUsableForDebugging bugging >True</IsUsableForDebugging>
    <IsValid>True</IsValid>
    <ModelNumber>MR912</ModelNumber>
    <Name>DNCENGTVOS-080</Name>
    <PasswordProtected>False</PasswordProtected>
    <ProductionSOC>l</ProductionSOC>
    <ProductType>AppleTV5,3</ProductType>
    <ProductVersion>15.0</ProductVersion>
    <SerialNumber>C07DK506JG2X</SerialNumber>
    <TotalDataCapacity>28197912576</TotalDataCapacity>
    <UDID>2a46286a5b932beadleda3d60c823c3f3c4c4761</UDID>
    <UniqueChipID>3955605101051942</UniqueChipID>
    <WatchCompanionCapability>False</WatchCompanionCapability>
    <WiFiAddress>44:c6:5d:85:9e:3f</WiFiAddress>
</Device>

    so it is listing two devices, but only the one directly attached has IsPaired=true. @premun can we filter out devices that are not paired?

    I tried shutting down DNCENGTVOS-091 via mlaunch but looks like it just rebooted, though it doesn't show up on the other Macs for now so we should see some improvement.

  12. mandel-macaque commented on Sep 9, 2022

    @mandel-macaque
    mandel-macaque
    on Sep 9, 2022

    @akoeplinger if @premun does not have the time, I can take a quick look, that should be a simple fix in xharness.

  13. steveisok
    added a commit that references this issue on Sep 11, 2022

    Fix WebSockets.Client.ConnectTest crash for apple mobile platforms (#…

    014c36f
  14. github-actions
    added a commit that references this issue on Sep 11, 2022

    Fix WebSockets.Client.ConnectTest crash for apple mobile platforms

    997473a

  15. premun commented on Sep 12, 2022

    @premun
    premun
    on Sep 12, 2022
    Member

    That's a nice find @akoeplinger! Fix is here: dotnet/xharness#946

  16. steveisok
    removed
    untriagedNew issue has not been triaged by the area owner
    on Sep 12, 2022
  17. steveisok
    added a commit that references this issue on Sep 12, 2022

    Fix WebSockets.Client.ConnectTest crash for apple mobile platforms (#…

    8f18a0d

  18. steveisok commented on Sep 12, 2022

    @steveisok
    steveisok
    on Sep 12, 2022
    Member

    The tweak to make the app not crash has been merged into both main and 7.0

  19. steveisok
    closed this as completedon Sep 12, 2022
  20. karelz
    added this to the 7.0.0 milestone on Sep 12, 2022

  21. karelz commented on Sep 12, 2022

    @karelz
    karelz
    on Sep 12, 2022
    MemberAuthor

    Fixed in main (8.0) in PR #75344 and in 7.0 (RC2) in PR #75423

  22. karelz
    added
    test-bugProblem in test source code (most likely)
    on Sep 12, 2022

  23. premun commented on Sep 13, 2022

    @premun
    premun
    on Sep 13, 2022
    Member

    @akoeplinger @mandel-macaque looking at the logs, we already tell mlaunch to not list wireless devices so this might be a bug in mlaunch too:

    /private/tmp/helix/working/B89909A1/p/microsoft.dotnet.xharness.cli/1.0.0-ci/tools/net7.0/any/../../../runtimes/any/native/mlaunch/bin/mlaunch --sdkroot /Applications/Xcode_13.4.app --listdev=/tmp/helix/working/B89909A1/t/tmp7hBqv0.tmp --list-wireless-devices=false --output-format=XML --timeout=0.5 -v -v -v -v -v

    notice the --list-wireless-devices=false

  24. ghost locked as resolved and limited conversation to collaborators on Oct 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-System.Netblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'os-maccatalystMacCatalyst OSruntime-monospecific to the Mono runtimetest-bugProblem in test source code (most likely)

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions